Dynamic Providers and Custom Resources in Pulumi

Dynamic providers let you manage infrastructure that no off-the-shelf Pulumi provider supports by implementing the create, read, update, delete, and diff lifecycle yourself in Python against an external API. This guide explains when to reach for a dynamic.ResourceProvider, how its CRUD lifecycle maps to state, and how to keep operations idempotent — as part of the broader Pulumi Patterns & Provider Management workflow. When you instead want to bundle several first-class resources into one reusable type, prefer a Pulumi ComponentResource rather than a dynamic provider.

The engine drives the provider's CRUD and diff methods, each of which calls the external API and returns the state Pulumi persists.

Problem Framing

Pulumi's curated providers cover the major clouds, but real systems also depend on SaaS dashboards, internal control planes, and niche APIs that no provider models. Reaching for boto3-style imperative calls inside a Pulumi program creates resources the engine never tracks, so they leak on destroy and drift silently. A dynamic.ResourceProvider closes that gap: you implement the lifecycle, and Pulumi records inputs and outputs in state, diffs them on each run, and calls your delete logic on teardown — giving an unsupported API the same managed semantics as a native resource.

Prerequisites

Python 3.9+ and pulumi>=3.0: pip install pulumi.
A client library or requests for the external API you are wrapping, plus credentials available through Pulumi config or the environment.
An initialized Pulumi project and stack; verify with pulumi stack ls.
Familiarity with the stack and backend model from Pulumi Stack Architecture, since dynamic resource state is stored in the same checkpoint.

# Confirm the project and backend are ready before adding a custom resource.
pulumi stack ls
pulumi config get apiToken

Concept Explanation

The ResourceProvider contract

A dynamic provider subclasses pulumi.dynamic.ResourceProvider and implements create, optionally read, update, delete, and diff. Each method receives plain dictionaries (the resource inputs and prior state) and returns typed results — CreateResult, DiffResult, ReadResult, UpdateResult. The provider itself is serialized and stored, so it must be self-contained: imports must happen inside the methods, not rely on closure state.

# Sketch only — full version in the child guide.
from pulumi.dynamic import ResourceProvider, CreateResult, DiffResult

class WidgetProvider(ResourceProvider):
    def create(self, props: dict) -> CreateResult:
        import requests  # Provider note: import inside the method; the class is serialized.
        # State implication: the returned id and outs become this resource's state.
        return CreateResult(id_="widget-123", outs={**props, "remote_id": "widget-123"})

How `__inputs` and diff drive convergence

Pulumi stores the inputs you last applied and the outputs you returned. On the next run it calls diff with old and new inputs; you return a DiffResult declaring whether anything changed and whether the change forces a replacement. Returning an accurate diff is what makes re-running idempotent — if nothing meaningful changed, you report no change and the engine skips update. Idempotency is a core principle across Pulumi Patterns & Provider Management: the same desired state must converge no matter how many times you apply it.

Read and out-of-band drift

Implementing read lets pulumi refresh ask the external API for the current state and reconcile it against the checkpoint, catching changes made outside Pulumi. Without read, the stored state is the only source of truth and silent drift goes undetected.

Step-by-Step Implementation

1. Decide whether a dynamic provider is the right tool

Goal: avoid building a provider when a simpler construct fits. If you are composing existing resources, use a Pulumi ComponentResource. Only build a dynamic provider when no provider exposes the resource at all.

# Decision is a design step, not code — but encode the inputs as a typed args object.
from dataclasses import dataclass

@dataclass
class WidgetArgs:
    name: str
    size: int  # Provider note: these map directly to the external API's fields.

2. Implement the lifecycle methods

Goal: cover create, diff, update, and delete so the resource is fully managed. The full implementation, including serialization caveats, is in Writing a Pulumi dynamic provider in Python.

# pulumi up
from pulumi.dynamic import ResourceProvider, CreateResult, DiffResult, UpdateResult

class WidgetProvider(ResourceProvider):
    def create(self, props: dict) -> CreateResult:
        import requests
        r = requests.post("https://api.example.com/widgets", json=props, timeout=10)
        rid = r.json()["id"]
        # State implication: id_ becomes the resource id Pulumi tracks forever.
        return CreateResult(id_=rid, outs={**props, "remote_id": rid})

    def diff(self, _id: str, old: dict, new: dict) -> DiffResult:
        changed = [k for k in new if old.get(k) != new.get(k)]
        return DiffResult(changes=bool(changed), replaces=[])

3. Wrap the provider in a typed Resource

Goal: give users a clean, typed resource class instead of raw dictionaries.

# pulumi up
import pulumi
from pulumi.dynamic import Resource

class Widget(Resource):
    remote_id: pulumi.Output[str]

    def __init__(self, name: str, args: WidgetArgs, opts=None) -> None:
        # State implication: vars(args) becomes the resource inputs stored in state.
        super().__init__(WidgetProvider(), name, {**vars(args), "remote_id": None}, opts)

Verification

Confirm the resource is created and tracked, and that a no-op run reports no changes (proving the diff is idempotent).

# First apply creates it; the second should show no changes.
pulumi up --yes
pulumi up --yes   # expect: "no changes" — the diff converged
pulumi stack output   # remote_id should be populated

# A unit assertion against the provider's create result.
def test_create_returns_remote_id() -> None:
    result = WidgetProvider().create({"name": "w1", "size": 1})
    assert result.outs["remote_id"] == result.id

Troubleshooting

TypeError: cannot pickle ... when running pulumi up. The provider class captured something unserializable (a module-level client, an open socket, a lambda). Move all imports and client construction inside the lifecycle methods so the class itself stays picklable.

Every pulumi up wants to update or replace the resource. Your diff returns changes=True spuriously — often because the external API returns fields you did not send, so old and new dicts never match. Compare only the inputs you manage, and normalize values before comparing.

pulumi refresh does nothing useful. You have not implemented read. Without it the engine cannot query the external API, so out-of-band drift stays invisible. Add a read method returning a ReadResult built from a live API lookup.

Frequently Asked Questions

When should I write a dynamic provider versus a ComponentResource? Use a dynamic provider only when no Pulumi provider exposes the resource and you must call an external API to create, read, update, and delete it. If you are merely grouping existing first-class resources into a reusable unit, a Pulumi ComponentResource is simpler and avoids the serialization constraints.

Why must imports go inside the provider methods? Pulumi serializes the provider so it can be reconstructed during later operations. Module-level clients or imports captured in the class body often cannot be pickled, causing TypeError at pulumi up. Importing inside each method keeps the class self-contained and serializable.

How does Pulumi keep a dynamic resource idempotent? The engine stores your last-applied inputs and outputs, then calls diff with the old and new inputs on every run. If your diff accurately reports no change, the engine skips update, so re-running converges to the same state instead of churning.

Can a dynamic provider detect changes made outside Pulumi? Only if you implement read. pulumi refresh calls read to fetch the live state from the external API and reconcile it with the checkpoint; without read, the stored state is assumed correct and drift goes unnoticed.

Writing a Pulumi dynamic provider in Python — a complete CRUD ResourceProvider against an external API with serialization caveats.
Pulumi ComponentResource — the simpler choice when you are composing existing resources rather than calling an unsupported API.
Pulumi Patterns & Provider Management — the parent overview of Pulumi workflows and provider strategies.