Manage Kubernetes CRDs with Pulumi Python

Custom resources are ordinary Kubernetes objects whose schema arrives at runtime, which breaks the assumption that a provider knows every type up front. This guide, part of managing Kubernetes with the Pulumi Python provider under Pulumi patterns and provider management, covers installing a CRD, creating instances of it, and ordering the two correctly.

Context

A CustomResourceDefinition teaches the API server a new type; a custom resource is an instance of that type. Submitting the instance before the definition is registered fails with no matches for kind, and because Pulumi parallelises independent resources, this happens whenever the dependency is not declared. The ordering problem is the whole difficulty — the objects themselves are unremarkable.

A CustomResourceDefinition is itself an ordinary object in the apiextensions.k8s.io/v1 group, and its body is a description of the type it creates. Four parts of that body determine how instances behave.

The versions list is the part people underestimate. Exactly one entry may set storage: true — that is the representation etcd holds — while any number may set served: true to be accepted at the API. When a definition serves v1alpha1 and v1 and stores v1, an instance submitted as v1alpha1 is converted on write and stored as v1, and reading it back through the v1alpha1 endpoint converts it again. The conversion is identity-only unless the definition declares spec.conversion.strategy: Webhook, which is why adding a field in a new version and expecting old objects to acquire it does not work.

The schema matters just as much because apiextensions.k8s.io/v1 requires it to be structural and prunes anything it does not describe. A field you send that the schema does not mention is silently dropped: the API server returns 201 Created, the object is stored without your field, and the controller reconciles something other than what you wrote. That failure has no error message anywhere — it shows up as a spec that does not match the program, which is worth remembering when a Pulumi diff keeps proposing to add a key that never appears to take.

Two behaviours of the provider itself sit underneath everything below. pulumi-kubernetes 4.x applies resources with server-side apply by default, so each field you set is tracked against a field manager and a conflicting owner produces an explicit error rather than a silent overwrite. And the provider maintains a discovery cache of the kinds the API server knows; registering a CRD invalidates it, which is exactly the moment a parallel operation on an unrelated resource can observe a half-refreshed view of the API surface.

Prerequisites

• pulumi-kubernetes>=4.0 with a configured provider
• Cluster-scoped rights to create CustomResourceDefinitions — namespace-scoped RBAC is not enough
• The CRD's API group, version, and kind, which you can read from the definition itself

# CLI: list registered custom types and confirm one is established
kubectl get crd
kubectl get crd certificates.cert-manager.io \
  -o jsonpath='{.status.conditions[?(@.type=="Established")].status}{"\n"}'

The RBAC requirement is stricter than it first appears. Creating a CRD needs create on customresourcedefinitions in the apiextensions.k8s.io group at cluster scope, and afterwards the same principal needs rights on the new type — the grant does not follow automatically from having created the definition. A deploy identity that can install cert-manager's CRDs but cannot create a Certificate will get through step one and fail at step two:

# CLI: check both halves of the permission before running the deploy
kubectl auth can-i create customresourcedefinitions.apiextensions.k8s.io
kubectl auth can-i create certificates.cert-manager.io --namespace apps

Confirm the API group and version from the definition rather than from a blog post or a chart's values file. Group and version travel together in api_version, and a mismatch produces the same no matches for kind message as a missing CRD, which sends people looking for an ordering bug that is not there:

# CLI: authoritative group, versions and kind for a registered type
kubectl get crd certificates.cert-manager.io \
  -o jsonpath='{.spec.group}{" "}{range .spec.versions[*]}{.name}{"(served="}{.served}{",storage="}{.storage}{") "}{end}{.spec.names.kind}{"\n"}'

Implementation

Step 1 — install the definition. For a CRD that ships as YAML, ConfigFile submits it without translating it to Python.

# crd.py — register the definition from the operator's manifest
# CLI: pulumi up
import pulumi
import pulumi_kubernetes as k8s

opts = pulumi.ResourceOptions(provider=k8s_provider)

crds = k8s.yaml.ConfigFile(
    "cert-manager-crds",
    file="manifests/cert-manager-crds.yaml",
    opts=opts,
)
# Provider note: the provider waits for the Established condition before
# treating the CRD as ready, which is what makes the dependency below reliable.

Step 2 — create an instance with CustomResource, declaring the CRD as an explicit dependency so Pulumi cannot reorder them.

# certificate.py — an instance of the custom type
certificate = k8s.apiextensions.CustomResource(
    "api-tls",
    api_version="cert-manager.io/v1",        # must match the CRD's group/version
    kind="Certificate",
    metadata={"name": "api-tls", "namespace": "apps"},
    spec={
        "secretName": "api-tls",
        "dnsNames": ["api.internal"],
        "issuerRef": {"name": "internal-ca", "kind": "ClusterIssuer"},
    },
    opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[crds]),
)
# State implication: spec is untyped — the provider forwards it as-is, so a
# misspelled field is rejected by the API server, not by Python.

The depends_on is not optional. Without it Pulumi may submit the certificate while the CRD is still registering.

Prefer retain_on_delete=True on the definition once instances exist outside this program. It costs nothing on a normal deploy and converts one specific catastrophe — removing the ConfigFile from the program, or renaming its logical name so Pulumi plans a replacement — from a cascading deletion of every object of that kind into an orphaned resource you clean up deliberately.

Step 3 — make the apply wait for something meaningful. The provider treats a custom resource as created once the API server accepts it, because it has no way to know what readiness means for an arbitrary kind. Where the next resource depends on the controller having finished its work, say so with the pulumi.com/waitFor annotation, which takes a condition name or a JSONPath expression the provider polls before completing:

# certificate_ready.py — block the apply until the controller reports Ready
# CLI: pulumi up
import pulumi
import pulumi_kubernetes as k8s

certificate = k8s.apiextensions.CustomResource(
    "api-tls",
    api_version="cert-manager.io/v1",
    kind="Certificate",
    metadata={
        "name": "api-tls",
        "namespace": "apps",
        "annotations": {
            # Provider note: without this the resource completes on acceptance,
            # so a downstream Deployment can mount a Secret that does not exist yet.
            "pulumi.com/waitFor": "condition=Ready",
            "pulumi.com/timeoutSeconds": "600",
        },
    },
    spec={
        "secretName": "api-tls",
        "dnsNames": ["api.internal"],
        "issuerRef": {"name": "internal-ca", "kind": "ClusterIssuer"},
    },
    opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[crds]),
)

The opposite annotation, pulumi.com/skipAwait: "true", is the escape hatch for a kind whose controller only reports status when something external happens — a Certificate issued by an ACME challenge that needs public DNS, for example, will hang an apply in a private environment until the timeout expires. Skipping the wait there is correct; skipping it everywhere hides real failures behind green applies.

Wrapping the untyped spec in a small Pydantic-style dataclass gives back the compile-time checking the generic CustomResource throws away, and keeps the validation on the Python side of the boundary where the error message is useful:

# typed_spec.py — validate the spec before the API server ever sees it
# CLI: pulumi up
from dataclasses import dataclass, field, asdict
from typing import Any, Dict, List


@dataclass(frozen=True)
class IssuerRef:
    name: str
    kind: str = "ClusterIssuer"
    group: str = "cert-manager.io"


@dataclass(frozen=True)
class CertificateSpec:
    secret_name: str
    dns_names: List[str]
    issuer_ref: IssuerRef
    duration: str = "2160h"
    renew_before: str = "360h"

    def to_manifest(self) -> Dict[str, Any]:
        if not self.dns_names:
            raise ValueError("a Certificate needs at least one dnsName")
        return {
            "secretName": self.secret_name,
            "dnsNames": list(self.dns_names),
            "duration": self.duration,
            "renewBefore": self.renew_before,
            "issuerRef": asdict(self.issuer_ref),
        }
# State implication: the manifest this produces is what lands in Pulumi state.
# Renaming a key here shows up as a spec diff, not as a replacement.

Verification

Confirm the type is established, the instance exists, and — crucially — that the operator acted on it. An accepted custom resource that no controller reconciles does nothing.

# CLI: the kind is registered, the instance is Ready, and the secret was produced
kubectl get crd certificates.cert-manager.io -o jsonpath='{.status.conditions[?(@.type=="Established")].status}{"\n"}'
kubectl -n apps get certificate api-tls -o jsonpath='{.status.conditions[*].type}{"\n"}'
kubectl -n apps get secret api-tls -o jsonpath='{.type}{"\n"}'

Then verify the field you sent survived schema pruning, which is the check nobody runs until it has already cost them an afternoon. Compare what the API server stored against what the program declared:

# CLI: read the stored spec back and diff it against intent
kubectl -n apps get certificate api-tls -o jsonpath='{.spec}{"\n"}' | python -m json.tool

A missing key here means the CRD's schema does not describe it — usually an older definition still installed from a previous chart version. Finally, run pulumi preview a second time: a clean preview proves the provider's field manager owns the fields it set, while a preview that keeps proposing the same change points at a controller writing to the same fields, which is the server-side apply conflict described below.

Gotchas & Edge Cases

Deleting a CRD deletes every instance. Removing the definition from the program cascades to all custom resources of that type across the Kubernetes cluster, including ones this stack never created. Treat CRD removal as a destructive migration.

Helm does not upgrade CRDs. A chart installs its CRDs once and ignores them thereafter, so a chart upgrade can leave an old schema in place. Manage CRDs as their own resource, separate from the chart install, and upgrade them deliberately.

CustomResource has no readiness condition it understands. The provider cannot know what Ready means for an arbitrary kind, so it completes on acceptance. Where readiness matters, assert it in verification rather than assuming a green apply proves it.

Finalizers turn a destroy into a hang. Most operators attach a finalizer to the objects they manage, and the API server refuses to remove an object until the finalizer is cleared. Delete the operator before its custom resources — which is exactly what a reverse-dependency destroy does when the operator's Deployment was declared before the instances — and nothing is left to clear them. pulumi destroy sits at deleting... until it times out, and kubectl delete behaves identically because the block is server-side.

Recovering means clearing the finalizer by hand, which is safe only once you have accepted that the controller's cleanup work will not happen:

# CLI: last resort — the object goes away, the external resource it owned may not
kubectl -n apps patch certificate api-tls \
  --type merge -p '{"metadata":{"finalizers":[]}}'

Server-side apply conflicts read as ownership errors. When a controller writes to a field the program also sets, the apply fails with a conflict naming the other field manager. The provider's remedy is the pulumi.com/patchForce: "true" annotation, which takes ownership of the disputed field. Use it only after deciding the program should win — forcing a field the controller manages produces a permanent fight where each side rewrites the other on every reconcile.

A stale discovery cache outlives the CRD install. Client tooling caches the API surface, so a kubectl run seconds after a CRD is registered can still report no matches for kind from ~/.kube/cache/discovery. The Pulumi provider refreshes its own view, but a verification script in the same pipeline may not; kubectl api-resources forces a refresh before the assertion runs.

Operational Notes

CRDs are the longest-lived objects in a Kubernetes estate. Instances come and go with applications, but the definition is shared by every namespace and outlives the program that installed it, which argues for treating it as platform state rather than application state. In practice that means one stack owns the definitions and the operators, and application stacks only ever create instances — the split described in structuring Pulumi stacks per environment applied along a type boundary instead of an environment boundary.

Upgrading a definition is the operation that needs a rehearsal. Count what is at risk first, because the blast radius is every object of that kind across every namespace, not just the ones this program created:

# CLI: how many objects would a bad schema change invalidate?
kubectl get certificates.cert-manager.io --all-namespaces --no-headers | wc -l

Then apply the new definition and read the objects back. A widened schema — a new optional field, a new served version — is safe and takes effect immediately. A narrowed one is not: tightening a pattern, removing an enum value or dropping a field does not retroactively reject stored objects, so the definition and the data silently disagree until something writes an object back and the update is rejected. The symptom is an operator that worked yesterday failing on a routine status update with a validation error against a field nobody touched.

Storage-version migration is the related trap. Flipping storage: true from v1alpha1 to v1 changes what new writes produce, but existing objects stay in etcd in the old representation until each is rewritten. Removing the old version from spec.versions before that rewrite happens leaves objects the API server can no longer decode. Read every object once — a no-op kubectl get -o yaml | kubectl replace -f - per namespace is the crude version — before dropping a served version, and keep the old version served for at least one release cycle.

# crd_ownership.py — definitions belong to the platform stack, not the app stack
# CLI: pulumi up --stack platform-prod
import pulumi
import pulumi_kubernetes as k8s

crds = k8s.yaml.ConfigFile(
    "cert-manager-crds",
    file="manifests/cert-manager-crds.yaml",
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        # State implication: removing this resource from the program orphans the
        # CRD instead of cascading a delete through every Certificate that exists.
        retain_on_delete=True,
    ),
)

pulumi.export("installed_crds", crds.resources.apply(lambda rs: sorted(rs.keys())))

ConfigFile.resources is keyed by <apiVersion>/<kind>::<name>, so the exported list reads as apiextensions.k8s.io/v1/CustomResourceDefinition::certificates.cert-manager.io. Publishing it gives application stacks something concrete to assert against through a stack reference, so an application deploy fails fast with a clear message instead of failing deep inside the API server with no matches for kind.

FAQ

Why does no matches for kind appear even though the CRD is in the same program?

Pulumi creates independent resources in parallel. Without depends_on, the instance can be submitted before the definition is established.

Can the custom resource spec be typed?

Not by the generic CustomResource. You can generate typed classes from a CRD's OpenAPI schema, or validate the spec with a Pydantic model in your own code before passing it through.

Should CRDs live in a separate stack?

Often, yes. CRDs are cluster-scoped and shared, while instances are per-application. Splitting them prevents an application deploy from touching cluster-wide types.

How do I upgrade a CRD safely?

Apply the new definition first and confirm existing instances still validate against it. Schema changes that remove or tighten fields can invalidate stored objects, which surfaces only when something next reads them. Count the affected objects before you start, and keep the previous version served for a release cycle so a rollback is possible.

How do I adopt a CRD that Helm already installed?

Bring it under management with pulumi.ResourceOptions(import_="certificates.cert-manager.io") on a matching resource declaration, so Pulumi records the existing object rather than trying to create it. Expect the first preview to show a diff against whatever the chart set — reconcile the program to the live object before applying, not the other way round.

Why does pulumi destroy hang on a custom resource?

A finalizer is holding it, and the controller that would clear the finalizer has usually already been deleted. Order the destroy so instances go before their operator, or clear the finalizer manually with a merge patch once you accept that the controller's external cleanup will not run.

Related

• Deploy Helm Charts with Pulumi Kubernetes Python — the charts that install CRDs but never upgrade them.
• Managing Kubernetes with the Pulumi Python Provider — the parent topic on provider behaviour.
• Dynamic Providers and Custom Resources — the Pulumi-side equivalent when no Kubernetes controller exists.