Writing a Pulumi Dynamic Provider in Python

This guide implements a complete dynamic.ResourceProvider in Python that manages a resource against an external REST API — covering create, read, update, delete, and diff, with the CreateResult and DiffResult return types and the serialization caveats that trip people up. It is the hands-on counterpart to Dynamic Providers and Custom Resources in Pulumi, itself part of the broader Pulumi Patterns & Provider Management workflow.

Context

When an API has no Pulumi provider, a dynamic provider gives it managed, idempotent lifecycle handling instead of fire-and-forget imperative calls. The example below manages a "widget" on a fictional api.example.com service: a thing with a name and a size that the API creates, returns by id, updates in place, and deletes. The same shape applies to any CRUD-capable REST or SDK endpoint.

A dynamic provider is not a plugin. There is no separate binary, no gRPC server to register, nothing to publish to a registry, and no schema. pulumi.dynamic.ResourceProvider is an ordinary Python class that the Pulumi SDK serialises — with dill — into the managed resource's own state, under a hidden __provider input, base64-encoded. When the engine later needs to update or delete that resource it reads the string back out of the checkpoint, rehydrates the class, and calls the method inside the same Python interpreter that is executing your program.

Three properties fall straight out of that design, and nearly every surprise later in this guide is a consequence of one of them:

• The class must be picklable at the moment the resource is declared. Anything reachable from self is going into the state file, so a requests.Session, an open file handle, or a lazily built SDK client stored on the instance is a hard failure.
• The code that runs on delete is the code that was serialised on create, unless something forces the __provider property to be rewritten. Your provider is versioned inside the state, not alongside your program.
• There is no protocol boundary. The provider runs with the same interpreter, the same installed packages and the same ambient credentials as the rest of the program. That is convenient, and it is also why a transitive dependency upgrade can break a stack whose source you have not touched.

How the provider reaches — and leaves — state

Exporting the checkpoint makes the mechanism concrete rather than theoretical. The widget resource carries __provider alongside the properties you actually supplied:

# CLI: pulumi stack export | python -m json.tool
pulumi stack export \
  | python -c "import json,sys; d=json.load(sys.stdin)['deployment']['resources']; \
r=[x for x in d if 'Widget' in x['urn']][0]; print(sorted(r['inputs']))"
# ['__provider', 'api_token', 'name', 'remote_id', 'size']

Two practical consequences follow, and the second one bites teams months after the provider was written.

A provider edit is an input change. Because __provider is an ordinary input, altering a single line of WidgetProvider changes the serialised bytes and Pulumi is willing to report an update on every widget the stack manages:

# CLI: pulumi preview --diff
    ~ pulumi-python:dynamic:Resource: (update)
        [id=wgt-8842]
      ~ __provider: "4gIAAAAAAAA..." => "5QIAAAAAAAA..."

Your own diff decides whether that update is recorded. The engine asks the provider, and the implementation in step 2 only inspects name and size. It therefore answers changes=False, no update is written, and the previous serialised class stays in the checkpoint — so the code that eventually executes delete is the version you wrote when the resource was created, not the version in your working tree. If your provider's behaviour matters at teardown time (it usually does — that is where the destructive API call lives), compare __provider explicitly:

# CLI: pulumi preview --diff
# State implication: reporting the __provider change writes today's serialised
# class into the checkpoint, so delete() later runs the code you have now.
from pulumi.dynamic import DiffResult


def diff(self, id_: str, old: dict, new: dict) -> DiffResult:
    fields = ("name", "size")
    changes = [f for f in fields if old.get(f) != new.get(f)]
    provider_changed = old.get("__provider") != new.get("__provider")
    return DiffResult(
        changes=bool(changes) or provider_changed,
        replaces=["name"] if "name" in changes else [],
        stables=["remote_id"],           # never changes without a replace
        delete_before_replace=True,      # the API rejects duplicate names
    )

stables tells the engine which outputs it may treat as fixed during a preview, which keeps downstream previews from showing spurious "output " placeholders. delete_before_replace inverts Pulumi's default create-then-delete ordering — correct when the external API enforces a uniqueness constraint you would otherwise collide with.

Prerequisites

• Python 3.9+ with pulumi>=3.0 and requests: pip install pulumi requests.
• An initialized Pulumi project and stack — verify with pulumi stack ls.
• The external API's base URL and a token, supplied through Pulumi config: pulumi config set --secret apiToken <token>.
• IAM/account permission on the external service to create, read, update, and delete the resource.

Implementation

1. Define typed inputs and the resource client

Keep the API interaction in small functions so the lifecycle methods stay readable. Use a dataclass for the user-facing inputs.

# providers/widget.py — used by __main__.py
# CLI: pulumi up
from dataclasses import dataclass

@dataclass
class WidgetArgs:
    name: str
    size: int  # Provider note: these map one-to-one to the external API's JSON body.

API_BASE = "https://api.example.com/widgets"

2. Implement the ResourceProvider with full CRUD and diff

Every method imports requests locally. The provider class is serialized into state, so it must not capture a module-level client, a connection, or any unpicklable object.

Each lifecycle method has a fixed contract, and the return types are not interchangeable:

The outs dictionary you return is the resource's recorded state — not a merge with what you were given. Returning a partial dictionary from update silently drops every property you omitted, and the next diff will then see those properties as changed. That is why every method below returns {**props, ...} rather than a freshly built dict.

# providers/widget.py
# CLI: pulumi up
from typing import Optional
from pulumi.dynamic import (
    ResourceProvider, CreateResult, ReadResult, UpdateResult, DiffResult,
)

class WidgetProvider(ResourceProvider):
    def _headers(self, props: dict) -> dict:
        # Provider note: token is passed in via props, not captured at class scope.
        return {"Authorization": f"Bearer {props['api_token']}"}

    def create(self, props: dict) -> CreateResult:
        import requests  # serialization caveat: import inside the method.
        r = requests.post(
            API_BASE,
            json={"name": props["name"], "size": props["size"]},
            headers=self._headers(props), timeout=10,
        )
        r.raise_for_status()
        rid = r.json()["id"]
        # State implication: id_ is permanent; outs are this resource's stored state.
        return CreateResult(id_=rid, outs={**props, "remote_id": rid})

    def read(self, id_: str, props: dict) -> ReadResult:
        import requests
        r = requests.get(f"{API_BASE}/{id_}", headers=self._headers(props), timeout=10)
        r.raise_for_status()
        body = r.json()
        # State implication: reconciles checkpoint with live state on `pulumi refresh`.
        return ReadResult(id_=id_, outs={**props, "name": body["name"], "size": body["size"]})

    def diff(self, id_: str, old: dict, new: dict) -> DiffResult:
        fields = ("name", "size")
        changes = [f for f in fields if old.get(f) != new.get(f)]
        # name change forces replacement; size can update in place.
        return DiffResult(changes=bool(changes), replaces=["name"] if "name" in changes else [])

    def update(self, id_: str, old: dict, new: dict) -> UpdateResult:
        import requests
        r = requests.put(
            f"{API_BASE}/{id_}",
            json={"size": new["size"]},
            headers=self._headers(new), timeout=10,
        )
        r.raise_for_status()
        return UpdateResult(outs={**new, "remote_id": id_})

    def delete(self, id_: str, props: dict) -> None:
        import requests
        # State implication: called on resource removal and on replacement teardown.
        requests.delete(f"{API_BASE}/{id_}", headers=self._headers(props), timeout=10).raise_for_status()

3. Expose a typed Resource wrapper

Wrap the provider so consumers get a clean class with typed outputs instead of raw dicts. Feed the secret token from Pulumi config.

# __main__.py
# CLI: pulumi up
import pulumi
from pulumi.dynamic import Resource
from providers.widget import WidgetProvider, WidgetArgs

class Widget(Resource):
    remote_id: pulumi.Output[str]

    def __init__(self, name: str, args: WidgetArgs, opts: Optional[pulumi.ResourceOptions] = None) -> None:
        token = pulumi.Config().require_secret("apiToken")
        props = {**vars(args), "api_token": token, "remote_id": None}
        # State implication: api_token is stored as a secret because it is an Output secret.
        super().__init__(WidgetProvider(), name, props, opts)

widget = Widget("primary", WidgetArgs(name="checkout", size=3))
pulumi.export("widget_id", widget.remote_id)

Verification

Apply twice to prove idempotency, then refresh to prove drift detection works.

# CLI: pulumi up --yes
pulumi up --yes            # creates the widget
pulumi up --yes            # expect "no changes": diff converged
pulumi stack output widget_id
pulumi refresh --yes       # read() reconciles any out-of-band change

The second pulumi up is the load-bearing one. A dynamic provider that reports changes=True on an unchanged program is the single most common defect in this pattern, and it is almost always caused by comparing values the API normalised on the way in — a size the service clamped to a maximum, a name it lowercased, a field it defaulted. Compare against what read returns, not against what you sent.

Drift detection is worth exercising explicitly. Change the widget out of band with curl, then:

# CLI: pulumi refresh --diff
pulumi refresh --diff
#   ~ pulumi-python:dynamic:Resource: (refresh)
#       [id=wgt-8842]
#     ~ size: 3 => 9
pulumi up --yes            # pushes the declared size back to 3

If refresh shows nothing after an out-of-band edit, read is returning the checkpoint's values rather than the API's — usually because the merge order is wrong and {**props} is overwriting the freshly fetched body instead of the other way round.

# CLI: python -m pytest test_widget_provider.py
def test_diff_no_change_is_idempotent() -> None:
    p = WidgetProvider()
    state = {"name": "checkout", "size": 3}
    assert p.diff("id-1", state, dict(state)).changes is False

def test_name_change_forces_replacement() -> None:
    p = WidgetProvider()
    res = p.diff("id-1", {"name": "a", "size": 3}, {"name": "b", "size": 3})
    assert res.replaces == ["name"]

Gotchas & Edge Cases

TypeError: cannot pickle '_io.BufferedReader' (or similar) on pulumi up. The provider captured an unpicklable object. Construct clients and import libraries inside each method, and pass credentials through props rather than storing them on the instance — exactly as the example does with api_token.

Secrets leak into plaintext state. A token passed as a normal string is stored unencrypted. Source it from Config().require_secret(...) so it arrives as a secret Output; Pulumi then encrypts it in the checkpoint. Never hardcode the token in WidgetArgs.

Replacement ordering surprises dependents. A non-empty replaces list makes Pulumi create the new resource first and delete the old one afterwards. That is the safe default for anything with dependents, but it fails outright when the external API enforces a uniqueness constraint on name — the create half of the replacement collides with the resource still sitting there:

# CLI: pulumi up --yes
error: requests.exceptions.HTTPError: 409 Client Error: Conflict for url:
    https://api.example.com/widgets

Set delete_before_replace=True in the DiffResult to invert the ordering, and accept the window of unavailability. If neither ordering is acceptable, the real fix is to stop letting name force a replacement — model it as an updatable field and issue a rename call in update instead.

create that raises after the API call succeeded orphans the resource. Pulumi only records an id when create returns a CreateResult. If the POST succeeds and the next line throws — a KeyError on an unexpected response shape, a timeout while reading the body — the widget exists remotely but the stack has no record of it, and the next pulumi up creates a second one. Do the smallest possible amount of work between the API call and the return, and put any enrichment in read where a failure is recoverable.

Async methods are not supported. ResourceProvider methods are called synchronously. Declaring async def create returns a coroutine object that Pulumi will try to treat as a CreateResult and fail with AttributeError: 'coroutine' object has no attribute 'id'. If the client library you want is async-only, drive it with asyncio.run() inside the synchronous method.

Operational Notes

The hardest part of a dynamic provider is not the create path — it is diff and delete. A diff that reports a change on every run makes previews useless and applies dangerous, so normalise both the old and new values (case, ordering, defaults the API injects) before comparing. A delete that fails when the resource is already gone will strand a stack mid-destroy, so treat a 404 as success.

Because Pulumi serialises the provider class into state, keep it importable and free of closures over live objects, and pin the versions of any HTTP library it uses. When the external system supports it, implement read so pulumi refresh and pulumi import can reconcile out-of-band changes — without it, drift in the external resource is invisible, which defeats the point of managing it as code.

Deleting what is already gone

Treat a 404 on delete as success. Someone will remove a widget through the vendor's console, and a delete that propagates the error strands the stack: the resource stays in the checkpoint, pulumi destroy fails at the same point every time, and the only escape is pulumi state delete. Three lines prevent that:

# providers/widget.py
# CLI: pulumi destroy --yes
# State implication: swallowing 404 lets destroy complete and drop the resource.
def delete(self, id_: str, props: dict) -> None:
    import requests
    resp = requests.delete(
        f"{API_BASE}/{id_}", headers=self._headers(props), timeout=10,
    )
    if resp.status_code == 404:
        return
    resp.raise_for_status()

Retries belong inside the method

The Pulumi engine does not retry a failed provider method; it fails the resource and, with --continue-on-error absent, unwinds the update. External APIs rate-limit, so build the backoff yourself. A requests HTTPAdapter with a Retry policy is picklable in the sense that matters — it is constructed inside the method, so it never reaches self and never reaches the checkpoint.

# providers/widget.py
# CLI: pulumi up --yes
from typing import Any

def _session() -> Any:
    import requests
    from requests.adapters import HTTPAdapter
    from urllib3.util.retry import Retry

    s = requests.Session()
    retry = Retry(
        total=5,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "PUT", "POST", "DELETE"}),
    )
    # Provider note: POST is retried here only because the API accepts an
    # idempotency key; drop it from allowed_methods if yours does not.
    s.mount("https://", HTTPAdapter(max_retries=retry))
    return s

Retrying a POST is only safe when the API deduplicates. If it does not, a 502 on create followed by a retry produces two widgets and one recorded id — the orphan case described above, arrived at from a different direction.

Timeouts and long-running operations

Every request in this guide carries timeout=10 for a reason: requests has no default timeout, and a provider method that blocks forever hangs the whole pulumi up with no output. For an API that returns 202 Accepted and a job handle, poll inside create with an explicit deadline and raise a message an operator can act on, such as TimeoutError("widget wgt-8842 not ACTIVE after 300s"). Pulumi will surface that string verbatim next to the resource in the update log, which is far more useful than a stack trace.

FAQ

When do I need a dynamic provider?

When a resource has no first-class Pulumi provider — a niche API or SaaS. See managing external SaaS resources for an end-to-end example.

What methods must I implement?

At minimum create; add diff, update, delete, and read to support updates, replacement, destroy, and import correctly. Omitting diff makes Pulumi fall back to a structural comparison of the inputs, which will report a change whenever __provider changes and cannot express replacement semantics. Omitting update means every change becomes a replacement.

How do I keep secrets out of state?

Mark sensitive inputs secret so Pulumi encrypts them in state, following securing Pulumi secrets. Note that this encrypts the value at rest in the checkpoint; the plaintext still passes through your provider methods, so avoid logging props wholesale when debugging.

Why does pulumi up want to update every resource after I edit the provider file?

Because the serialised class is stored as the __provider input, so editing the file changes an input on every resource that uses it. This is expected. What matters is whether your diff reports the change: if it does not, the checkpoint keeps the old code and your delete will run a stale implementation.

Can a dynamic provider call boto3 or another cloud SDK?

Yes — import it inside the method rather than at module scope, and let the SDK resolve credentials from the environment instead of storing a client on self. That said, if a first-class Pulumi provider exists for the service, use it: you get schema validation, previews with real values and a proper diff engine for free.

How do I adopt widgets that already exist in the API?

Implement read so that it can populate the full state from an id alone, then run pulumi import against the resource type and the remote id. Without a working read, import has nothing to reconcile against and the resource comes back with empty properties, which the next up will happily "correct" by overwriting the live object.

Related

• Dynamic Providers and Custom Resources in Pulumi — the parent overview of the CRUD lifecycle, diff, and idempotency model.
• Pulumi ComponentResource — use this instead when composing existing resources rather than wrapping an external API.
• Pulumi Patterns & Provider Management — the parent section covering Pulumi workflows and provider strategies.