Rotating Pulumi Stack Secrets Without Downtime

Rotation is the one secrets operation that has a blast radius, because it is the only one that invalidates something a running process is already holding. Setting a secret for the first time cannot break anything; changing it can take down every consumer that authenticated with the previous value. This guide belongs to Pulumi secrets and configuration in Python within Pulumi patterns and provider management, and it separates the two operations that get filed under the same word — replacing the plaintext credential, and replacing the key that encrypts it — because they fail in completely different ways.

Which rotation are you actually doing? Which rotation are you actually doing?: choose among 3 options. A secret in the stack needschanging value New plaintextcredential,resources change provider Same plaintext, newencryption key both Sequence them, neverin one update
The two operations share a vocabulary and nothing else: one changes what consumers must present, the other changes only how the ciphertext is stored.

Context

pulumi config set --secret myapp:dbPassword does exactly one thing: it encrypts a string with the stack's secrets manager and writes the ciphertext into Pulumi.<stack>.yaml under a secure: key. It does not touch a database, an identity provider, or a running task. The credential in the outside world changes only when a subsequent pulumi up feeds that value into a resource that owns it.

That gap is where outages come from. A single-step replace looks like three commands run back to back — set the new password, run pulumi up, watch it succeed — and it works fine in a stack whose only consumer is the resource Pulumi just updated. It fails the moment a credential is held by processes Pulumi does not manage. Change an RDS master password and the ModifyDBInstance call succeeds in seconds, but every connection pool that authenticated with the old string keeps working until its next reconnect, and then every one of them fails at once with FATAL: password authentication failed for user "appuser". The database never went down. The application did.

The second operation, changing the secrets provider, has no such exposure. It decrypts every secure: value with the old manager and re-encrypts it with the new one. The plaintext is identical afterwards, so no resource input changes and no consumer notices. Its failure mode is the opposite kind: it is a one-way door that needs the old provider to still work, and teams discover that they scheduled the KMS key for deletion first.

Prerequisites

  • Pulumi CLI 3.x with the stack already initialised and an explicit secrets provider — pulumi stack ls --all and pulumi stack export | head will tell you which one is in force.
  • pulumi>=3.0, pulumi-aws>=6.0, and pulumi-random>=4.0 in the project virtualenv for the examples below.
  • Decrypt permission on the current secrets provider from wherever you run the rotation. For an AWS KMS provider that is kms:Decrypt and kms:Encrypt on the key ARN, from both your workstation and the CI role.
  • A list of every consumer of the credential you are about to change, including the ones outside this stack: batch jobs, a colleague's local .env, a monitoring probe, a read replica of the deployment in another region.
  • No pending operations on the stack. pulumi cancel on a half-finished update leaves the checkpoint in a state where a provider change will refuse to run.
# CLI: confirm the secrets provider and that nothing is mid-update
pulumi stack export --stack prod | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print(d['deployment'].get('secrets_providers',{}).get('type')); \
   print('pending:', len(d['deployment'].get('pending_operations', [])))"

Implementation

Step 1 — Rotate the value in two phases, not one

The rule that removes the outage is that the new credential must be accepted by the system before any consumer is asked to present it, and the old credential must remain accepted until no consumer presents it any more. That means three deployments, not one, and the middle one is the only one that touches application code.

Two-phase rotation of a database credential Two-phase rotation of a database credential: Pulumi stack → Database → Running consumers. Pulumi stack Database Runningconsumers phase 1: create standby role old credential still valid phase 2: roll with new role reconnect as new principal phase 3: drop retired role
Phase one adds, phase two switches, phase three removes. A single-step replace collapses all three and invalidates the credential that live processes are holding.

Whether phase one is possible at all depends on how many credentials the target system will hold simultaneously. An RDS master password is singular — setting a new one immediately invalidates the old, so there is no overlap window and the two-phase pattern has to be built on a second principal rather than a second password. An API token, an IAM access key, and an AWS Secrets Manager secret all support two live values, so the overlap is native.

For a Postgres database the durable version is a second role. Keep both roles in configuration, promote one, then drop the other:

# rotation.py — two application roles, one active, both valid during the overlap
# CLI: pulumi up --stack prod
from __future__ import annotations

from dataclasses import dataclass
from typing import Literal

import pulumi
import pulumi_postgresql as postgresql
import pulumi_random as random

Slot = Literal["blue", "green"]


@dataclass(frozen=True)
class RotationState:
    active: Slot          # which role the application is told to use
    retire: Slot | None   # which role is dropped at the end of the rotation


cfg = pulumi.Config()
state = RotationState(
    active=cfg.require("activeRole"),          # "blue" or "green"
    retire=cfg.get("retireRole"),              # unset during the overlap window
)

passwords: dict[Slot, pulumi.Output[str]] = {}
for slot in ("blue", "green"):
    if slot == state.retire:
        continue
    pw = random.RandomPassword(
        f"app-{slot}-password",
        length=32,
        special=True,
        override_special="!#$%&*()-_=+[]{}",
        # Provider note: RandomPassword.result is secret in the provider schema,
        # so it stays masked in the checkpoint without an explicit pulumi.secret().
    )
    postgresql.Role(
        f"app-{slot}",
        name=f"app_{slot}",
        login=True,
        password=pw.result,
        roles=["app_readwrite"],
    )
    passwords[slot] = pw.result

pulumi.export("active_role_name", f"app_{state.active}")
pulumi.export("active_role_password", pulumi.Output.secret(passwords[state.active]))

The retireRole key is the whole trick. While it is unset both roles exist and both authenticate. Setting it to the slot you are leaving is what drops the old role, and you only set it once the deployment that switched activeRole has fully rolled out.

# CLI: phase 1 — create the standby role, change nothing the app reads
pulumi config set myapp:activeRole blue --stack prod
pulumi up --stack prod --yes

# CLI: phase 2 — point consumers at the new role and roll them
pulumi config set myapp:activeRole green --stack prod
pulumi up --stack prod --yes

# CLI: phase 3 — only after every consumer has reconnected
pulumi config set myapp:retireRole blue --stack prod
pulumi up --stack prod --yes

For a credential you are handed rather than generating — a vendor API token, a licence key — phase one is pulumi config set --secret myapp:apiTokenNext, phase two swaps which key the program reads, and phase three is pulumi config rm myapp:apiTokenPrev. Read the value from stdin so it never reaches shell history or the process table:

# CLI: set a secret without it appearing in `ps` output or ~/.bash_history
printf '%s' "$NEW_TOKEN" | pulumi config set --secret myapp:apiTokenNext --stack prod
pulumi config --show-secrets --stack prod | grep apiToken

The stack file gains a second secure: entry whose ciphertext is unrelated to the first — Pulumi re-encrypts per value, so identical plaintext under the same key still produces different ciphertext, and you cannot diff Pulumi.prod.yaml to tell whether two secrets match.

Step 2 — Know which resources update, which replace, and which silently do neither

A rotated value reaches the world through resource inputs, and the three possible behaviours have very different operational consequences.

How a rotated value reaches each resource kind How a rotated value reaches each resource kind: comparison across Engine action, Consumer effect. Resource Engine action Consumer effect rds.Instance.password update in place none until reconnect secretsmanager.SecretVersion replace, labels move next fetch gets new value ecs.TaskDefinition new revision rolling restart of tasks lambda.Function env update in place next cold start k8s Secret via secretKeyRef patch in place none, pods keep old value
Only a resource whose change forces the consumer to restart delivers the rotated value to a running process.

An update is an in-place API call: aws.rds.Instance.password becomes a ModifyDBInstance, aws.lambda_.Function.environment becomes an UpdateFunctionConfiguration. Nothing restarts, which is exactly why running consumers keep failing on stale credentials.

A replace creates a new object before deleting the old one. aws.secretsmanager.SecretVersion treats secret_string as immutable, so a new value produces a new version and the AWSCURRENT staging label moves to it while the previous version keeps AWSPREVIOUS — a native overlap window. aws.ecs.TaskDefinition is the same shape: any change produces a new revision, and the aws.ecs.Service that references it performs a rolling deployment, which is the only mechanism in this list that actually restarts consumers.

The third behaviour is the dangerous one: a successful update that changes nothing a process can see. A Kubernetes Secret patched in place propagates to projected volume mounts within a kubelet sync period, but values injected through env.valueFrom.secretKeyRef are read once at container start and never again. Pulumi reports updated (1), the pods carry on with the old credential, and there is no error anywhere.

# k8s_secret.py — force a rollout instead of a silent in-place patch
# CLI: pulumi up --stack prod
from __future__ import annotations

import pulumi
import pulumi_kubernetes as k8s

cfg = pulumi.Config()
db_password = cfg.require_secret("dbPassword")

db_secret = k8s.core.v1.Secret(
    "app-db",
    string_data={"password": db_password},
    opts=pulumi.ResourceOptions(
        # State implication: replacing on a data change means the auto-named
        # Secret gets a NEW name, so every reference below it changes too.
        replace_on_changes=["stringData", "data"],
        delete_before_replace=False,
    ),
)

k8s.apps.v1.Deployment(
    "app",
    spec=k8s.apps.v1.DeploymentSpecArgs(
        replicas=3,
        selector=k8s.meta.v1.LabelSelectorArgs(match_labels={"app": "api"}),
        template=k8s.core.v1.PodTemplateSpecArgs(
            metadata=k8s.meta.v1.ObjectMetaArgs(labels={"app": "api"}),
            spec=k8s.core.v1.PodSpecArgs(
                containers=[k8s.core.v1.ContainerArgs(
                    name="api",
                    image="ghcr.io/example/api:1.24.0",
                    env=[k8s.core.v1.EnvVarArgs(
                        name="DB_PASSWORD",
                        value_from=k8s.core.v1.EnvVarSourceArgs(
                            secret_key_ref=k8s.core.v1.SecretKeySelectorArgs(
                                # The changed name lands in the pod spec, which
                                # is what makes the ReplicaSet roll.
                                name=db_secret.metadata.name,
                                key="password",
                            ),
                        ),
                    )],
                )],
            ),
        ),
    ),
)

Step 3 — Rotate the secrets provider itself

Changing the provider is a single command, and its safety comes entirely from what you do before running it.

What change-secrets-provider rewrites What change-secrets-provider rewrites: Export checkpoint then Decrypt with old then Re-encrypt all then Preview Export checkpoint rollback point Decrypt with old needs old key live Re-encrypt all config + checkpoint Preview expect no changes
The command touches ciphertext and key material only; a preview that proposes resource changes afterwards means the round trip went wrong.
# CLI: move a stack from the local passphrase manager to a KMS key
export PULUMI_CONFIG_PASSPHRASE="$OLD_PASSPHRASE"
pulumi stack export --stack prod --file prod-checkpoint-pre-rotation.json
pulumi stack change-secrets-provider \
  "awskms://alias/pulumi-prod?region=eu-west-1" --stack prod

Three things are rewritten. Every secure: value in Pulumi.prod.yaml is decrypted and re-encrypted, so the file's diff is large and entirely ciphertext. The stack's encryptedkey or encryptionsalt line is replaced with the new provider's key material. And every secret-marked value inside the checkpoint — resource outputs, stack outputs — is re-encrypted, producing a new state version.

What does not change is any plaintext. Run pulumi preview immediately afterwards and the correct result is no changes. If it proposes updates, the re-encryption round-tripped something incorrectly and you should restore the checkpoint you exported rather than let the update proceed.

The passphrase-to-passphrase case needs both values present, which the CLI reads from two different variables:

# CLI: rotate the passphrase itself, old and new supplied together
export PULUMI_CONFIG_PASSPHRASE="$OLD_PASSPHRASE"
export PULUMI_NEW_CONFIG_PASSPHRASE="$NEW_PASSPHRASE"
pulumi stack change-secrets-provider passphrase --stack prod

Keep the old KMS key enabled for at least one full release cycle. Any checkpoint version predating the change is still encrypted under it, and pulumi stack import of a pre-rotation export will need it. Deleting the key early turns rollback into data loss, with the CLI reporting error: constructing secrets manager of type "cloud": secrets (code=AccessDeniedException) and no path forward.

Verification

The only verification that counts is evidence about the old value, not the new one.

# CLI: nobody is still authenticating as the retired Postgres role
psql -h "$DB_HOST" -U app_green -c \
  "SELECT usename, count(*), max(backend_start) FROM pg_stat_activity GROUP BY usename;"

# CLI: the previous IAM access key has not been used since the cutover
aws iam get-access-key-last-used --access-key-id AKIAIOSFODNN7EXAMPLE \
  --query 'AccessKeyLastUsed.LastUsedDate'

# CLI: which Secrets Manager version each staging label points at
aws secretsmanager describe-secret --secret-id myapp/prod/db \
  --query 'VersionIdsToStages'

# CLI: the old key is gone from configuration, not merely superseded
pulumi config --show-secrets --stack prod | grep -c apiTokenPrev   # expect 0

After a provider rotation, prove the re-encryption is complete rather than partial. Every secret in the stack file must decrypt with the new manager, and no plaintext may have leaked into the file during the rewrite:

# CLI: decrypt every secure: value with the new provider, print nothing
pulumi config --show-secrets --stack prod >/dev/null && echo "all values decrypt"
grep -c 'secure:' Pulumi.prod.yaml
pulumi preview --stack prod --diff | tail -3   # expect: no changes

Gotchas & Edge Cases

Rotating the value does not scrub history. Old checkpoint versions still contain the previous secret, encrypted under whichever key was current at the time. If you are rotating because a value leaked, the rotation limits future use of that value; it does not remove it from state history, from CI logs, or from a colleague's terminal scrollback.

pulumi refresh cannot see a credential drift. Cloud APIs do not return passwords, so the provider has nothing to compare. If someone resets the database password in the console, Pulumi's view of the world stays confidently wrong until the next up overwrites it with the configured value — which is, incidentally, a perfectly good remediation.

A RandomPassword with no keepers never rotates. The resource generates once and then persists in state forever. Rotation requires either replacing it deliberately with pulumi up --target ... --replace or wiring a keepers map whose change forces new material.

The overlap window has to outlive the longest-lived connection. Connection pools with max_lifetime unset, long-running batch jobs, and cached SDK clients can hold a credential for days. Measure the real reconnect distribution before deciding how long phase two lasts; twenty minutes is usually optimistic.

Provider rotation and a code change in the same commit is a debugging trap. If pulumi preview shows resource changes after change-secrets-provider, you need to know instantly whether that is a re-encryption bug or your own edit. Rotate the provider on a commit that changes nothing else.

Two engineers rotating at once corrupt the stack file. pulumi config set writes Pulumi.<stack>.yaml locally; two concurrent rotations produce a merge conflict in ciphertext, which cannot be resolved by reading it. Serialise rotations through one branch.

Operational Notes

Rotation as a repeating operational loop Rotation as a repeating operational loop: Schedule the window → Add standby credential → Switch and roll consumers → Confirm old value unused → Retire and record → repeat. Schedule thewindow Add standbycredential Switch and rollconsumers Confirm oldvalue unused Retire andrecord
Retirement closes one rotation and sets up the next; the loop is only safe while the confirm step gates the retire step.

Rotate on a schedule that the two-phase pattern can absorb, and run the same three phases in a lower environment first — not as a rehearsal, but because a staging rotation exercises exactly the reconnect behaviour you are relying on in production. A rotation that quietly breaks a nightly job in staging tells you something worth knowing a week before it matters.

Automate the phases rather than the whole rotation. The transitions between them are judgement calls that depend on evidence — has every consumer reconnected? — and a script that runs all three phases back to back has reinvented the single-step replace with extra steps. The Pulumi Automation API is a good fit for driving one phase at a time from a workflow that pauses for a check between them.

Record which slot is active somewhere outside the stack file. A stack output naming the live role costs nothing and answers the question every incident responder asks first. Keep the retirement step in the runbook rather than the schedule: an overlap that lasts an extra day is a minor hygiene issue, while retiring early is an outage.

Finally, treat the provider rotation as a distinct change with its own approval. It touches every secret in the stack at once, it is the only operation that can render a stack permanently undecryptable, and it deserves the same care as the key policy it depends on — the ground covered in securing Pulumi secrets with AWS KMS and HashiCorp Vault.

FAQ

Do I have to run pulumi up after pulumi config set --secret?

Yes, for the value to reach anything. config set only rewrites the stack file. Until an update runs, the new ciphertext is inert and every resource still holds the old plaintext, which is precisely why phase one of a rotation is safe to land ahead of time.

Can I rotate a secret for one stack without touching the others?

Yes. Configuration and secrets providers are per stack, so pulumi config set --secret --stack staging changes nothing in prod. The exception is a value referenced through a cross-stack reference, where the consuming stack picks up the new value on its next update, not on yours.

Does pulumi stack change-secrets-provider need a pulumi up afterwards?

No, and it should not produce one. The command rewrites ciphertext only. Run pulumi preview to confirm it reports no changes — if it proposes an update, something round-tripped incorrectly and the safe move is restoring the pre-rotation export.

How do I roll back a rotation that broke consumers?

Set the configuration key back to the previous value and run pulumi up, which works only if the old credential still exists — the entire argument for not retiring it in the same deployment. If the old role or key is already dropped, the fastest path is forward: fix the consumers to read the new value.

What happens to secret stack outputs during a provider rotation?

They are decrypted with the old manager and re-encrypted with the new one, in the same pass as the configuration values. Consumers reading them through a cross-stack reference see identical plaintext and need no change; they do need permission on the new key if they read the checkpoint directly.

Should the application read the secret from Pulumi configuration at runtime?

No. Configuration is deployment-time input, and an application that shells out to pulumi config get has coupled its startup to the CLI, the backend, and decrypt permission on the key. Deliver the value through a mechanism the application already understands, as covered in passing configuration from Pulumi to application code.