Dynamic Providers and Custom Resources in Pulumi

A dynamic provider is the hatch Pulumi leaves open for the parts of your estate that no provider models: the internal control plane, the feature-flag platform, the licensing API, the ticketing system that has to be told about every new environment. You implement create, read, update, delete and diff yourself in Python, and in exchange the engine treats the result like any other resource — it appears in the preview, it is recorded in the checkpoint, it is diffed on every run, and it is torn down on pulumi destroy. This topic sits inside Pulumi patterns and provider management and covers the architecture: where the provider code actually runs, what the engine stores about it, which method the engine calls when, and how to decide whether you need one at all.

Problem Framing

The alternative to a dynamic provider is not "no automation" — it is a requests.post somewhere in your Pulumi program, and that is the thing worth naming precisely, because it looks like it works.

An imperative call made during program evaluation creates something real, but the engine never learns about it. There is no entry in the checkpoint, so pulumi preview shows nothing, pulumi destroy leaves the object behind, and a second pulumi up calls the API again and either duplicates the object or fails on a uniqueness constraint. Worse, the call fires during preview as well as during update, because a preview still evaluates your Python top to bottom. Engineers discover this when a dry run creates production records.

The second failure mode is subtler. Someone wraps the call in a pulumi.Output.apply so it only runs when its inputs resolve. That does defer it past preview, but it also buries a side effect inside a callback the engine considers pure, and the object still never enters state. The resource remains invisible to drift detection, to pulumi refresh, to policy checks, and to the next engineer reading the stack.

A dynamic provider fixes all of this by moving the API calls into methods the engine invokes at defined points in the lifecycle. The cost is that you now own semantics a real provider would have given you: deciding what constitutes a change, deciding which changes require replacement rather than an in-place update, deciding what to do when the remote object has vanished, and deciding what is safe to store. Those decisions are the substance of this topic. The concrete implementations live in the child guides: writing a Pulumi dynamic provider in Python for the first working CRUD class, handling diffs and updates in Pulumi dynamic providers for the classification work that keeps previews honest, managing external SaaS resources with Pulumi dynamic providers for the realities of rate limits, archived keys and read-after-write races against a hosted API, and testing Pulumi dynamic providers in Python for proving the lifecycle behaves before it runs against anything live.

Prerequisites

• Python 3.9+ and pulumi>=3.0, in the same virtualenv the CLI resolves — dynamic provider code runs inside your program's interpreter, so anything you import must be importable there.
• A client for the target API, whether that is a vendor SDK or requests, plus credentials supplied through pulumi config or the environment rather than hard-coded.
• An initialised project and stack, and an understanding of where the checkpoint lives, since dynamic resource state — including the serialised provider — is stored alongside every other resource. The backend model is covered in Pulumi stack architecture.
• A read of the target API's documentation for idempotency: whether it supports conditional creates, whether it returns the created object on POST, whether delete is soft, and whether names are reusable. Those four answers determine most of your implementation.

# CLI: confirm the project, the stack and the credential are all in place
pulumi stack ls
pulumi config get apiToken --show-secrets >/dev/null && echo "token configured"
pip show pulumi | grep -i '^Version'
python -c "from pulumi.dynamic import ResourceProvider; print(ResourceProvider)"

Where the Provider Code Actually Runs

This is the architectural fact that explains most of the surprises, and it is different from every other provider you use.

A conventional Pulumi provider is a separate binary — a plugin the CLI downloads into ~/.pulumi/plugins, launched as a subprocess, spoken to over gRPC. A dynamic provider is none of that. It is a Python object living in your program's own process. There is no plugin to download, no version to pin, and no cross-language reuse: a dynamic provider written in Python is usable only from Python programs.

To make lifecycle operations work across separate CLI invocations, Pulumi serialises the provider instance with dill and stores the result in the resource's state under a __provider output. On the next pulumi up, the engine reads that string back, reconstructs the provider, and calls diff or update on the reconstructed object.

Three consequences follow, and each one is a real incident waiting to happen.

The class must be serialisable. Anything captured by the instance has to survive dill. An HTTP session, a thread lock, an open socket, a compiled regex holding a lock, a lambda closing over a module — any of these produce TypeError: cannot pickle '_thread.lock' object at pulumi up, before a single request leaves the machine. The habitual fix is to construct clients inside the methods rather than in __init__, or to hold only plain configuration on the instance and build everything else on demand.

The stored provider can go stale. State holds the version of the provider that was serialised when the resource was last written. If you change the class and run pulumi up, the engine re-serialises it as part of the update. But if you delete the module — during a refactor, or because the resource was removed from the program — pulumi destroy still needs to reconstruct the old provider to call delete, and dill resolves module-level classes by import path. The result is ModuleNotFoundError: No module named 'providers.widget' during a destroy that must now be finished by hand. Keep provider modules around until every stack that referenced them is gone.

Secrets in the instance are secrets in the checkpoint. Whatever the provider holds when it is pickled is written into state as part of __provider. An API token stored on self is therefore in the checkpoint, encrypted only if the surrounding value is marked secret. Read the credential from the environment inside the methods instead, so it is never captured by the serialisation — and note that lifecycle methods execute outside the program's configuration scope, so pulumi.Config() is not a reliable source there either.

# providers/widget.py — a serialisation-safe provider skeleton
# CLI: pulumi up
from typing import Any, Dict

from pulumi.dynamic import CreateResult, ResourceProvider


class WidgetProvider(ResourceProvider):
    def __init__(self, endpoint: str) -> None:
        # Provider note: plain strings only. A requests.Session here would raise
        # TypeError: cannot pickle '_thread.lock' object on the first pulumi up.
        self.endpoint = endpoint

    def _client(self) -> Any:
        import os

        import requests  # built per call, never stored on the instance

        session = requests.Session()
        # State implication: reading the token here rather than holding it on self
        # keeps it out of the pickled __provider blob written to the checkpoint.
        token = os.environ["WIDGET_API_TOKEN"]
        session.headers["Authorization"] = f"Bearer {token}"
        return session

    def create(self, props: Dict[str, Any]) -> CreateResult:
        client = self._client()
        response = client.post(f"{self.endpoint}/widgets", json=props, timeout=15)
        response.raise_for_status()
        widget_id: str = response.json()["id"]
        # State implication: id_ is permanent for this resource; every later diff,
        # update, read and delete receives exactly this string back.
        return CreateResult(id_=widget_id, outs={**props, "widget_id": widget_id})

The Method Contract

ResourceProvider defines six methods. Only create and delete are strictly required for a resource to be manageable; the rest determine how well it behaves. The engine calls them in a fixed order, and knowing that order removes most of the guesswork about why something did or did not run.

check(olds, news) runs first, before anything is compared. It returns CheckResult(inputs, failures) — the normalised inputs the engine should use, plus a list of CheckFailure(property, reason) entries. This is where defaulting and validation belong. A failure here stops the operation with a message naming the property, which is far kinder than a 400 from the remote API halfway through an update.

diff(id, olds, news) decides what happens next. It returns DiffResult, and its four fields are the entire vocabulary the engine has for describing a change. Getting them right is the difference between a preview an operator trusts and one they learn to ignore.

create(inputs) runs when the resource is new or is being replaced. It returns CreateResult(id_, outs). The id_ is the resource's permanent identity; choose something the remote API will still recognise months later, not a value that can be edited.

read(id, props) is called by pulumi refresh. It queries the live object and returns ReadResult(id_, outs). Returning ReadResult(id_=None, outs={}) tells the engine the object no longer exists, and the resource is dropped from state. Without a read implementation, refresh is a no-op and out-of-band drift is invisible.

update(id, olds, news) runs only when diff reported changes=True and did not ask for replacement. It returns UpdateResult(outs), and those outs become the resource's complete new state — not a patch. Omitting a field here silently removes it from state.

delete(id, props) receives the id and the old outputs. During a replacement it is called with the state of the object being discarded, which is a common source of bugs when a provider reads a key that only exists in the new shape.

Diff Is Where the Design Lives

Everything an operator sees in pulumi preview comes from DiffResult, and its four fields encode four separate decisions.

changes is a boolean: does anything need to happen at all. Return False and the engine skips update entirely — including any API call you wrote inside it, with no warning. Return True spuriously and every run churns.

replaces is a list of input property names whose change forces a destroy-and-recreate. It must name inputs the operator can actually edit; listing a computed output produces a replacement nobody can explain because the named property does not appear in their diff. This list is your immutability specification: a property belongs here exactly when the remote API has no way to change it in place.

stables is a promise that the listed properties will never change during an update. Downstream resources consume those values as if they were fixed. Listing something that can move — an endpoint URL that changes when the backing capacity grows, for instance — hands dependents a value the engine believed was constant.

delete_before_replace controls ordering when a replacement happens. The default is create-then-delete, which is safer for availability but fails immediately when the API enforces a unique name: the new create collides with the object still present. Set it to True for name-constrained resources and accept the gap.

# providers/widget.py (continued) — the diff is the immutability specification
# CLI: pulumi preview --diff
from typing import Any, Dict, List

from pulumi.dynamic import DiffResult

IMMUTABLE: frozenset = frozenset({"region", "tier"})       # API cannot change these
MUTABLE: frozenset = frozenset({"display_name", "labels"})  # PATCH-able in place
COMPUTED: frozenset = frozenset({"widget_id", "created_at"})  # never compare these


class WidgetProvider:  # methods shown in isolation
    def diff(self, _id: str, olds: Dict[str, Any], news: Dict[str, Any]) -> DiffResult:
        compared = (IMMUTABLE | MUTABLE)
        changed: List[str] = [
            key for key in compared if olds.get(key) != news.get(key)
        ]
        replaces: List[str] = [key for key in changed if key in IMMUTABLE]
        # State implication: a non-empty replaces list means delete + create, so the
        # remote object's id changes and every dependent is updated in the same run.
        return DiffResult(
            changes=bool(changed),
            replaces=replaces,
            stables=["widget_id"],
            delete_before_replace=bool(replaces),  # the API rejects duplicate names
        )

Two details cause more permanently-dirty previews than anything else. The first is comparing fields the remote API returns but you never sent: if outs contains created_at and your diff loops over news, the missing key reads as a change forever. Compare an explicit set of managed properties, never news.keys(). The second is type drift across the JSON boundary — an API that accepts 1 and returns "1.0" makes every run report a change. Normalise both sides through one coercion helper and unit-test that helper directly.

What Ends Up in the Checkpoint

A dynamic resource's state is not special-cased. It contains the inputs you passed, the outs your methods returned, and the serialised provider — all of it in the same checkpoint as your S3 buckets and Kubernetes deployments.

That has direct security consequences. A signing secret returned in outs is stored, in plaintext unless the value is marked secret. The disciplined pattern is to return a fingerprint instead of the value: a SHA-256 of the secret is enough to detect that it changed, and it leaks nothing if the checkpoint is read. Where the plaintext genuinely must be available to the rest of the program, declare it explicitly with additional_secret_outputs so the engine encrypts it, and follow the handling rules in Pulumi secrets and configuration.

# __main__.py — a typed wrapper, with the secret output declared
# CLI: pulumi up
import hashlib
from dataclasses import dataclass, asdict
from typing import Optional

import pulumi
from pulumi.dynamic import Resource

from providers.widget import WidgetProvider


@dataclass(frozen=True)
class WidgetArgs:
    display_name: str
    region: str
    tier: str


class Widget(Resource):
    widget_id: pulumi.Output[str]
    secret_fingerprint: pulumi.Output[str]

    def __init__(
        self,
        name: str,
        args: WidgetArgs,
        opts: Optional[pulumi.ResourceOptions] = None,
    ) -> None:
        props = {**asdict(args), "widget_id": None, "secret_fingerprint": None}
        # State implication: every key in props is written to the checkpoint, so a
        # None placeholder is required for anything the provider computes.
        super().__init__(WidgetProvider("https://api.internal.example"), name, props, opts)


def fingerprint(value: str) -> str:
    return hashlib.sha256(value.encode()).hexdigest()[:16]


widget = Widget(
    "reports",
    WidgetArgs(display_name="Reports", region="eu-west-1", tier="gold"),
    opts=pulumi.ResourceOptions(additional_secret_outputs=["secret_fingerprint"]),
)
# State implication: the fingerprint is already safe to store, but the same option
# is what encrypts a property in the checkpoint when a real secret must live there.
pulumi.export("widgetId", widget.widget_id)

The other checkpoint concern is size. __provider is a base64 blob on every dynamic resource, so a stack declaring three hundred of them carries three hundred copies. It is rarely fatal, but it is a reason to prefer one dynamic resource type covering a collection over one per item when the API supports batch semantics.

Step-by-Step: From API to Managed Resource

1. Rule out the cheaper options first

Most requests for a dynamic provider are really requests for something else. If you are grouping existing first-class resources into a reusable unit, you want a Pulumi ComponentResource — it has no serialisation constraints and no lifecycle to implement. If you only need to read something external, a plain function call during evaluation is fine, because reads have no lifecycle. A dynamic provider is warranted only when an external object must be created, changed and destroyed in step with the stack.

2. Establish identity before writing any lifecycle code

Decide what id_ will be and never revisit it. It must be stable, unique, and resolvable by the API long after creation. A UUID returned by the service is ideal; a user-editable name is not, because renaming then looks like a different resource. Write down which inputs are immutable at the same time — that list becomes replaces.

3. Implement create and delete, then verify a full round trip

Get pulumi up and pulumi destroy working before adding anything else. A provider that can only create is a leak generator. Make delete tolerate a missing object: a 404 on teardown should be treated as success, because the alternative is a stack that can never be destroyed.

4. Add diff, then update

With the classification from step 2 in hand, diff is mechanical. Only then implement update, and have it return the complete new outs rather than a partial dictionary.

5. Add read last

read is what makes pulumi refresh meaningful. Implement it once the rest is stable, and return an empty result when the object is gone so refresh can prune the entry rather than leaving a phantom.

Verification

The definitive check is that a second run does nothing. Everything else is detail.

# CLI: apply, then prove convergence — the second preview must be empty
pulumi up --yes
pulumi preview --diff          # expect: "no changes"
pulumi refresh --yes           # expect: no unexpected drift if read() is implemented
pulumi stack output widgetId

If the second preview reports an update, the diff is comparing something it should not. Run the preview with diagnostics turned up and read what the engine thinks changed before touching the code.

# CLI: see the engine's property-level view of the diff
pulumi preview --diff --logtostderr -v=3 2>&1 | grep -i 'diff\|replace'

Below that, the provider is a plain Python class and should be tested like one: instantiate it, call the methods with dictionaries, assert on the results. No Pulumi runtime is required for any of it.

# tests/test_widget_provider.py — the lifecycle as ordinary unit tests
# CLI: pytest tests/test_widget_provider.py -q
from pulumi.dynamic import DiffResult

from providers.widget import WidgetProvider


def test_immutable_property_forces_replacement() -> None:
    provider = WidgetProvider("https://api.internal.example")
    result: DiffResult = provider.diff(
        "w-1",
        {"region": "eu-west-1", "tier": "gold", "display_name": "reports"},
        {"region": "us-east-1", "tier": "gold", "display_name": "reports"},
    )
    assert result.changes is True
    assert result.replaces == ["region"]
    assert result.delete_before_replace is True


def test_cosmetic_change_is_an_in_place_update() -> None:
    provider = WidgetProvider("https://api.internal.example")
    result = provider.diff(
        "w-1",
        {"region": "eu-west-1", "tier": "gold", "display_name": "reports"},
        {"region": "eu-west-1", "tier": "gold", "display_name": "Reports"},
    )
    # Provider note: no API call happens in diff; this test needs no network at all.
    assert result.changes is True and result.replaces == []

The full harness — client seams, fakes that fail on the first call, and a proof that the class still pickles — is worked through in testing Pulumi dynamic providers in Python.

Troubleshooting

TypeError: cannot pickle '_thread.lock' object on pulumi up. The provider instance captured something unserialisable, almost always an HTTP session or a client object built in __init__. Move client construction into the methods and keep only plain data on self.

ModuleNotFoundError: No module named 'providers.widget' during pulumi destroy. The serialised provider in state resolves its class by import path, and the module was moved or deleted. Restore the module at its original path, run the destroy, then remove it.

Every pulumi up wants to update the resource. The diff is comparing properties the program does not manage — usually server-computed fields present in outs but absent from news, or values whose type changed crossing JSON. Compare an explicit managed-property set and normalise types on both sides.

A preview says "replace" for a change the API handles in place. The property is in your replaces list when it should be treated as mutable. Move it, and confirm update actually issues the corresponding API call.

pulumi refresh reports nothing even though the object was changed elsewhere. read is not implemented, so the engine has no way to ask. Add it and return the live values.

An update appears to succeed but the remote object is unchanged. diff returned changes=False for that property, so update was never invoked. There is no error in this path — the preview simply says there is nothing to do.

A replacement fails with a 409 Conflict on create. The engine created before deleting and the API enforces unique names. Set delete_before_replace=True in the DiffResult for that class of change.

Teardown fails because the object is already gone. delete propagated a 404. Catch not-found on delete and return normally; deletion of something that does not exist is the desired end state.

FAQ

When is a ComponentResource the better choice than a dynamic provider?

Whenever the thing you are building is a composition of resources that already have providers. A ComponentResource groups them under one logical parent, gets a clean preview for free, and has none of the serialisation constraints. Reach for a dynamic provider only when there is an external API that nothing else can drive.

Can a dynamic provider be used from a Pulumi program written in another language?

No. The provider is a Python object executing inside the Python language host, not a downloadable plugin, so it is reachable only from Python programs. If you need cross-language reuse, the answer is a real provider built with the provider SDK or the Pulumi provider boilerplate.

Does pulumi import work for dynamic resources?

Not through the CLI's import path, which is built around provider plugins that can read an object by id. Adoption of an existing object generally means making create idempotent — detect the existing remote object, return its id rather than creating a second one — and then applying once so the entry lands in state.

Why did my provider stop working after I refactored the package layout?

Because the serialised provider in state names the old module path. Any stack whose resources were created before the move will fail to reconstruct the provider on the next operation. Keep a shim at the old import path until every affected stack has been updated or destroyed.

How do I keep an API token out of the checkpoint?

Do not store it on the provider instance, because the instance is serialised into state. Read it from the environment inside each method — configuration is not in scope where lifecycle methods execute — and if a derived secret must be exposed as an output, return a hash rather than the value and declare the property through additional_secret_outputs so the engine encrypts what it does store.

Is there a performance cost to hundreds of dynamic resources?

Yes, on two axes. Each resource carries its own serialised provider blob in the checkpoint, and each lifecycle call is a Python function making a network request, executed concurrently by the engine. Batch where the API supports it, and use depends_on to serialise deliberately when a rate limit makes concurrency counterproductive.

Related

• Writing a Pulumi Dynamic Provider in Python — the first complete CRUD ResourceProvider, with the serialisation caveats spelled out.
• Handling Diffs and Updates in Pulumi Dynamic Providers — classifying every property and returning a DiffResult an operator can trust.
• Managing External SaaS Resources with Pulumi Dynamic Providers — rate limits, archived keys and read-after-write races against a hosted API.
• Testing Pulumi Dynamic Providers in Python — fakes, client seams and proving the class still pickles.