Awaiting Kubernetes Rollouts in Pulumi Python

The Pulumi Kubernetes provider blocks until a resource reports ready, which turns a deploy from a submission into an actual verification. This guide, part of managing Kubernetes with the Pulumi Python provider under Pulumi patterns and provider management, explains what the provider waits on, how to tune the timeout, and when skipping the wait is legitimate.

Context

Await logic is why a Pulumi deploy of a broken image takes two minutes and then fails, while kubectl apply of the same manifest succeeds instantly and leaves a crash loop behind. The provider watches each resource's status conditions and only marks it created once they are satisfied — available replicas for a Deployment, bound phase for a PersistentVolumeClaim, an assigned address for a LoadBalancer Service. Knowing which condition is being awaited turns a mysterious hang into a specific diagnosis.

The distinction that matters is between accepted and ready. kubectl apply returns as soon as the API server has validated the object and written it to etcd; everything after that — scheduling, image pulls, probes passing, an endpoint being added to a Service — happens asynchronously and is reported nowhere the command can see. Pulumi's Kubernetes provider closes that gap by keeping the resource step open until the controller responsible for the object has done its work. The consequence is that pulumi up exit code 0 is a much stronger claim than kubectl apply exit code 0, and it is a claim a pipeline can build a deployment gate on.

That strength has a cost, and the cost is that every readiness condition your workload cannot satisfy becomes a blocked deploy. A Deployment with a liveness probe pointing at a path the application does not serve, a PersistentVolumeClaim whose storage class has volumeBindingMode: WaitForFirstConsumer and no pod to trigger it, a Service of type LoadBalancer on a Kubernetes cluster with no cloud controller manager — all of these previously looked like a successful apply followed by a separate, later, someone-else's-problem incident. With await they are a red step in the pipeline that names the resource. Most of the work in this guide is about making that failure fast and legible rather than a twenty-minute hang.

What each kind waits for What each kind waits for: comparison across Ready when, Common hang. Kind Ready when Common hang Deployment available == desired probe never passes Service (LB) ingress address assigned no cloud controller PVC phase == Bound no matching storage class Job succeeded == completions container exits non-zero
A hang is a readiness condition that never became true — the kind tells you which one to inspect.

How the Await Machinery Works

The provider does not poll kubectl get in a loop. For each supported kind it opens a watch against the API server and consumes the object's status subresource as the responsible controller updates it, alongside a second watch on the objects that explain a failure — the ReplicaSets and Pods behind a Deployment, the Endpoints behind a Service. That second stream is why a Pulumi failure message can tell you Back-off pulling image when the object you asked for was a Deployment.

What the provider does while an apply is in flight What the provider does while an apply is in flight: Pulumi engine → k8s provider → API server → Pods. Pulumi engine k8s provider API server Pods create Deployment POST apps/v1 202 accepted watch status + events Ready=True Available=True resource created
The provider holds the step open across a watch stream: the resource is only 'created' once the API server reports the readiness condition.

For a Deployment the condition set is specific: the provider waits for status.observedGeneration to catch up with metadata.generation (proving the controller has seen your update at all), then for Progressing with reason NewReplicaSetAvailable, then for Available to be True with status.availableReplicas matching the desired count. Miss the first of those and you can wait on a stale status from the previous revision; that ordering is the whole reason a rolling update does not report ready the instant it is submitted.

Each kind has its own definition. A StatefulSet is ready when status.readyReplicas equals the desired replica count and status.updateRevision equals status.currentRevision, which is what makes a partially-rolled StatefulSet an honest failure instead of a green deploy. A DaemonSet compares status.numberReady with status.desiredNumberScheduled, so adding a tainted node to the Kubernetes cluster can block an unrelated deploy. A Job waits for status.succeeded to reach spec.completions. Custom resources are the exception: the provider cannot know what "ready" means for a CRD it has never seen, so by default it returns as soon as the object is accepted.

When a wait does expire, the message names the resource and quotes what the controller said. A failed image pull surfaces as a two-part error — the resource-level timed out waiting to be Ready line, and beneath it the underlying event, Back-off pulling image "ghcr.io/acme/api:does-not-exist". A workload that cannot be scheduled surfaces 0/3 nodes are available: 3 Insufficient cpu. A Deployment that Kubernetes itself gave up on reports ProgressDeadlineExceeded, which means progressDeadlineSeconds fired before the Pulumi timeout did — a useful signal, because it tells you the failure was decided by the Kubernetes cluster rather than by your client.

Prerequisites

  • pulumi-kubernetes>=4.0 with a working provider — the pulumi.com/waitFor annotation used below is a 4.x feature.
  • Read access to pod events and logs in the target namespace, for diagnosing a hang. Without get/list on events and pods, the provider still waits correctly but its error message loses the explanatory second half.
  • A rough expectation of normal rollout duration, so an abnormal wait is recognisable. Measure it once with kubectl rollout status before choosing a timeout, rather than guessing.
  • A pipeline whose own job timeout is longer than the sum of the resource timeouts in the stack, so a stuck rollout fails with a Pulumi error rather than a truncated CI log.
# CLI: watch what the provider is waiting for while an apply is in flight
kubectl -n apps get events --sort-by=.lastTimestamp | tail -20
# CLI: measure a normal rollout before you pick a timeout for it
time kubectl -n apps rollout status deployment/api --timeout=10m

Implementation

Step 1 — set a deliberate timeout. The default is generous; a shorter one turns a stuck rollout into a fast, actionable failure in CI. Pick the number from the measurement you took above: roughly three times a healthy rollout is enough to absorb a slow image pull or a node scale-up without making an engineer sit through ten minutes of certain failure. custom_timeouts accepts Go-style duration strings — "5m", "90s", "1h" — and takes a separate value per operation, which matters because a create that has to pull a cold image is legitimately slower than an update that reuses a warm one.

Await loop Await loop: submit object → watch status → condition met? → mark ready → repeat. submit object watch status condition met? mark ready
The provider polls status conditions until they are satisfied or the timeout expires.
# rollout.py — bound how long a deploy may wait before failing
# CLI: pulumi up
import pulumi
import pulumi_kubernetes as k8s

deployment = k8s.apps.v1.Deployment(
    "api",
    metadata={"namespace": "apps"},
    spec={...},
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        custom_timeouts=pulumi.CustomTimeouts(create="5m", update="5m"),
    ),
)
# State implication: on timeout the resource is left in whatever state the
# cluster reached; Pulumi records the failure rather than rolling back.

Note what the timeout does not do. It bounds how long Pulumi waits, not how long Kubernetes tries. When the create timeout fires, the Deployment is still there, the ReplicaSet is still retrying, and the next pulumi up will pick the object up exactly where the Kubernetes cluster left it. That is usually what you want, but it means a timeout is a signal to investigate rather than a rollback.

Step 2 — skip the wait only where readiness is undefined. Some objects have no meaningful ready condition, and waiting on them stalls forever.

# skipaware.py — opt out of awaiting for a kind with no ready condition
configmap = k8s.core.v1.ConfigMap(
    "feature-flags",
    metadata={
        "namespace": "apps",
        "annotations": {"pulumi.com/skipAwait": "true"},
    },
    data={"flags.json": '{"beta": false}'},
    opts=pulumi.ResourceOptions(provider=k8s_provider),
)
# Provider note: skipAwait makes the apply return on acceptance. Use it for
# objects with no readiness semantics, never to paper over a failing rollout.

The annotation is read off the object, not off the Pulumi resource options, which has a useful consequence: it also works on objects you did not author directly — anything rendered by a Helm chart or loaded from a YAML file — as long as you can patch the metadata on the way through.

Step 3 — state the condition explicitly for custom resources. Because the provider has no built-in readiness rule for a CRD, a CustomResource returns immediately even though the operator behind it may take a minute to reconcile. pulumi.com/waitFor supplies the missing rule, either as a status condition type or as a JSONPath expression that must become true.

# waitfor.py — make a custom resource block until its operator reports Ready
# CLI: pulumi up
import pulumi
import pulumi_kubernetes as k8s

certificate = k8s.apiextensions.CustomResource(
    "api-tls",
    api_version="cert-manager.io/v1",
    kind="Certificate",
    metadata={
        "namespace": "apps",
        # Provider note: without this annotation the resource is "created" the
        # instant the API server accepts it, long before a certificate exists.
        "annotations": {"pulumi.com/waitFor": "condition=Ready"},
    },
    spec={
        "secretName": "api-tls",
        "dnsNames": ["api.example.internal"],
        "issuerRef": {"name": "internal-ca", "kind": "ClusterIssuer"},
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        custom_timeouts=pulumi.CustomTimeouts(create="3m"),
    ),
)
# State implication: downstream resources that reference certificate.metadata
# now genuinely order after issuance, not merely after submission.

A JSONPath form — pulumi.com/waitFor: "jsonpath={.status.phase}=Running" — covers operators that never publish a condition. Keep the expression as narrow as the operator's own documentation allows; a wait on a field the controller never writes is indistinguishable from a hang.

Step 4 — standardise the policy in one place. Copying custom_timeouts onto every resource guarantees that one of them will be forgotten. A ComponentResource that applies the same options to its children makes the policy a single reviewable object.

# awaited.py — one place that owns rollout policy for a service
# CLI: pulumi preview
from __future__ import annotations

from dataclasses import dataclass

import pulumi
import pulumi_kubernetes as k8s


@dataclass(frozen=True)
class RolloutPolicy:
    create: str = "5m"
    update: str = "5m"
    delete: str = "2m"


class AwaitedService(pulumi.ComponentResource):
    def __init__(
        self,
        name: str,
        image: str,
        namespace: str,
        provider: k8s.Provider,
        policy: RolloutPolicy = RolloutPolicy(),
        opts: pulumi.ResourceOptions | None = None,
    ) -> None:
        super().__init__("acme:k8s:AwaitedService", name, None, opts)
        child = pulumi.ResourceOptions(
            parent=self,
            provider=provider,
            custom_timeouts=pulumi.CustomTimeouts(
                create=policy.create, update=policy.update, delete=policy.delete
            ),
        )
        labels: dict[str, str] = {"app": name}
        self.deployment = k8s.apps.v1.Deployment(
            name,
            metadata={"namespace": namespace},
            spec={
                "replicas": 3,
                "selector": {"matchLabels": labels},
                # Provider note: progressDeadlineSeconds must be shorter than the
                # create timeout above, so Kubernetes names the failure first.
                "progressDeadlineSeconds": 180,
                "template": {
                    "metadata": {"labels": labels},
                    "spec": {"containers": [{"name": name, "image": image}]},
                },
            },
            opts=child,
        )
        self.register_outputs({"name": self.deployment.metadata["name"]})

Verification

Prove the wait is doing its job by deliberately breaking a rollout and confirming the deploy fails rather than reporting success.

# CLI: point at a nonexistent tag and confirm the apply fails, not passes
pulumi config set apiImageTag does-not-exist
pulumi up --yes ; echo "exit=$?"     # expect a nonzero exit after the timeout
kubectl -n apps describe pod -l app=api | grep -A3 'Events:'

A nonzero exit and an ImagePullBackOff event together confirm the pipeline will catch a bad deploy.

Read the failure text rather than just the exit code, because it tells you which clock fired. A message ending timed out waiting to be Ready is Pulumi's own timeout; one containing ProgressDeadlineExceeded is the Deployment controller giving up first; and [MinimumReplicasUnavailable] Deployment does not have minimum availability. is the Available condition explaining itself. If the run ends without any of those — just a truncated log — the CI job timeout beat both, and no useful diagnosis was recorded.

Restore the good tag and confirm the deploy goes green again, so you have proved the check is sensitive to the fault and not permanently red:

# CLI: undo the broken tag and confirm the same command now succeeds
pulumi config set apiImageTag v1.14.2
pulumi up --yes && kubectl -n apps rollout status deployment/api --timeout=1m

For the custom resource from step 3, verify the annotation actually took effect by checking that the resource's create step spanned the reconcile rather than returning instantly:

# CLI: confirm the Certificate reported Ready before Pulumi moved on
kubectl -n apps get certificate api-tls -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Gotchas & Edge Cases

A LoadBalancer service hangs on clusters without a cloud controller. On bare metal or kind there is nothing to assign an external address, so the await never completes. Use ClusterIP locally, or annotate the service with skipAwait.

Jobs wait for completion, not for start. A long-running Job blocks the apply for its full duration. Model genuinely long tasks as something the deploy triggers rather than something it waits on.

skipAwait hides real failures. Applied broadly it returns the tool to fire-and-forget semantics while still looking like a verified deploy — the worst combination. Restrict it to specific resources and note why in a comment.

Deletes wait too, and finalizers can make them wait forever. custom_timeouts has a delete field for a reason: removing a Namespace blocks until every object inside it is gone, and any object holding a finalizer that no controller is left to clear will pin the Namespace in Terminating indefinitely. Uninstalling an operator before deleting its custom resources is the classic way to produce a pulumi destroy that never returns.

A WaitForFirstConsumer storage class deadlocks a standalone PVC. The provider waits for phase: Bound; the volume binder waits for a pod to be scheduled; if the PVC is created without a workload that mounts it, neither side moves and the step burns the full timeout before failing. Create the PVC and its consumer in the same update, or annotate the claim with skipAwait and let the workload's own readiness be the gate.

Replica count and rollout strategy change what "available" means. With replicas: 1 and the default maxUnavailable: 25%, a rolling update must terminate the only pod before the new one is available, so the Available condition can flip false mid-update on a single-replica Deployment and stay false if the new pod fails. Two replicas plus a PodDisruptionBudget is the difference between a wait that reports a real problem and one that reports your own strategy back at you.

A pulumi refresh does not re-await anything. Refresh reads current state; it does not verify readiness. A workload that has been crash-looping since last week refreshes cleanly and shows no diff, which is why readiness belongs in monitoring as well as in the deploy path.

Operational Notes

Await behaviour is a policy decision that belongs to the platform, not to whoever writes the next stack. Three habits keep it consistent.

Order the deadlines so the most specific one fires first. If progressDeadlineSeconds is 180 and the Pulumi create timeout is 300, a stuck rollout produces ProgressDeadlineExceeded with the ReplicaSet name in it — an actionable message. Reverse the two and you get a generic client-side timeout that says only that something took too long. The CI job timeout sits outside both, and every one of these clocks should be set explicitly rather than inherited.

Every clock that can end a stuck rollout Every clock that can end a stuck rollout: layered from CustomTimeouts on the resource down to timeout-minutes on the CI job. CustomTimeouts on the resource per resource, per operation: create / update / delete pulumi.com/timeoutSeconds annotation travels with the object, survives a Helm chart render progressDeadlineSeconds in the spec Kubernetes itself reports ProgressDeadlineExceeded activeDeadlineSeconds on a Job the Job fails instead of running to the heat death timeout-minutes on the CI job the outer bound; must exceed all of the above
Layer the deadlines deliberately: the innermost clock should fire first so the failure names the real cause.

Be deliberate about custom resources at the provider level rather than object by object. Setting PULUMI_K8S_AWAIT_ALL=true in the deploying environment makes the provider apply its generic readiness heuristic to custom resources as well, which is a reasonable default for a Kubernetes cluster full of operator-backed CRDs — but it is an environment-wide switch, so it changes the behaviour of stacks whose authors never opted in. Prefer explicit pulumi.com/waitFor annotations on the resources that need them, and treat the environment variable as a migration aid rather than a permanent setting.

Finally, budget for the fact that awaiting serialises work that used to overlap. Ten Deployments applied with kubectl return in a second and converge in parallel; ten awaited Deployments in one Pulumi program still converge in parallel, but the update does not finish until the slowest one is ready, and any resource that depends on another's output is genuinely sequential. If a deploy suddenly takes six minutes after you introduce a waitFor on a certificate that the ingress depends on, nothing is broken — you have simply started measuring a wait that was always there. Split independent workloads into separate stacks when that total becomes the pipeline's critical path, and use depends_on sparingly, since every explicit dependency is another rollout the update has to sit through in order.

FAQ

Why did pulumi up take minutes when kubectl apply was instant?

Because Pulumi waited for the workload to become ready. The elapsed time is the rollout you previously were not measuring.

Can the timeout be set globally?

Not directly — custom_timeouts is per resource. A component resource that applies the same options to its children is the usual way to standardise it.

Does a timeout roll back the change?

No. Kubernetes has already accepted the object; Pulumi records the failure and leaves the Kubernetes cluster mid-rollout. Roll forward with a fix, or use the deployment's own rollback.

Is skipAwait ever the right default?

For ConfigMaps, Secrets, and similar objects with no readiness semantics, yes. For anything that runs a container, no.

Why does my custom resource report created immediately?

The provider has no readiness rule for a CRD it does not recognise, so acceptance by the API server is all it can observe. Add pulumi.com/waitFor with a condition type or a JSONPath expression, as in step 3, and the step will hold open until the operator has actually reconciled the object.

How do I make a Helm chart's resources wait?

The chart's rendered objects go through the same provider, so they obey the same annotations — use a transformation to inject pulumi.com/skipAwait or pulumi.com/timeoutSeconds into the metadata of the specific objects you need to change. Chart-level hooks and the Chart resource's own options do not replace per-object readiness.

Does await work against a private API server from CI?

Only if the runner can reach it. The watch is a long-lived HTTPS connection to the API server, so a bastion-only Kubernetes cluster needs a self-hosted runner or a tunnel; without one the step fails on a dial timeout that looks like a readiness problem but is a network one.