Securing Pulumi Secrets with AWS KMS and HashiCorp Vault

Production infrastructure demands cryptographic control over state files. Pulumi's default service-managed encryption lacks audit trails and cross-account portability. Migrating to AWS KMS or HashiCorp Vault enforces compliance boundaries. This guide—part of the AWS Provider Deep Dive within Pulumi Patterns & Provider Management—details atomic provider swaps, strict Python 3.9+ typing patterns, and state recovery workflows. It pairs naturally with managing multi-account AWS environments with Pulumi Python, where each account needs its own KMS key for cross-account state portability.

How Pulumi Encrypts Stack Secrets

Before changing anything, be precise about what the secrets provider actually protects. Pulumi does not send each secret to KMS or Vault. It generates a 32-byte data key locally, uses that key to seal every secure: value in Pulumi.<stack>.yaml and every secret property in the state file with AES-256-GCM, then asks the external provider to encrypt the data key itself. The sealed result is written back into the stack config as encryptedkey, alongside a secretsprovider URL that records which key sealed it.

Two consequences follow, and both shape every decision later in this guide. First, the external provider is called once per operation, not once per secret: a stack with two hundred secret outputs still makes a single kms:Decrypt call during pulumi up, so API cost and throttling are irrelevant while permission errors are fatal. Second, the ciphertext is bound to that specific key material. Delete the KMS key, repoint the alias, or rotate the Vault transit key with the old version pruned, and the encryptedkey in your stack file becomes undecryptable — the state is intact but unreadable, and no pulumi subcommand can recover it.

Environment Isolation & Python 3.9+ Baseline

Virtual Environment & Dependency Pinning

Infrastructure code requires deterministic dependency resolution. Floating versions introduce silent breaking changes during provider upgrades. Pin pulumi, boto3, and hvac in pyproject.toml or requirements.txt. Isolate each stack in a dedicated virtual environment.

# CLI: initialize and activate a clean Python environment for the stack
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Strict Type Checking with mypy

Dynamic typing obscures configuration resolution errors until deployment. Enforce mypy --strict in CI pipelines. Annotate all configuration loaders and resource constructors. Catch None propagation before the Pulumi engine evaluates the dependency graph. The specific bug worth catching statically is the one where an Output[str] is annotated as str: mypy rejects it at the annotation, whereas Pulumi only fails later with a resource property containing the repr of an unresolved output.

IAM & Vault Auth Pre-Flight Checks

Authentication deadlocks halt stack operations mid-execution. Validate the AWS IAM kms:Encrypt and kms:Decrypt permissions before initializing the provider — note that Pulumi's cloud secrets manager does not call kms:GenerateDataKey, so granting it is neither required nor sufficient. For Vault, verify AppRole or TLS certificate validity and that the transit key exists. Run a round-trip through the key to confirm network routing and policy attachment in one step.

# CLI: python scripts/preflight_secrets.py --alias alias/pulumi-secrets-key
# Provider note: exercises the exact two KMS actions Pulumi's secrets manager uses.
from dataclasses import dataclass

import boto3
import hvac
from botocore.exceptions import ClientError


@dataclass(frozen=True)
class PreflightResult:
    backend: str
    ok: bool
    error_code: str = ""


def check_kms(alias: str, region: str) -> PreflightResult:
    kms = boto3.client("kms", region_name=region)
    try:
        sealed = kms.encrypt(KeyId=alias, Plaintext=b"pulumi-preflight")
        kms.decrypt(CiphertextBlob=sealed["CiphertextBlob"])
    except ClientError as exc:
        return PreflightResult("awskms", False, exc.response["Error"]["Code"])
    return PreflightResult("awskms", True)


def check_vault(addr: str, token: str, key: str) -> PreflightResult:
    client = hvac.Client(url=addr, token=token)
    if not client.is_authenticated():
        return PreflightResult("hashivault", False, "invalid_token")
    try:
        client.secrets.transit.read_key(name=key)
    except hvac.exceptions.Forbidden:
        return PreflightResult("hashivault", False, "permission denied")
    except hvac.exceptions.InvalidPath:
        return PreflightResult("hashivault", False, "transit key missing")
    return PreflightResult("hashivault", True)

A failed pre-flight prints the AWS error code verbatim — AccessDeniedException, NotFoundException, or KMSInvalidStateException for a key pending deletion — which is faster to act on than the wrapped message Pulumi emits mid-operation. Credential resolution itself follows the rules in best practices for managing cloud credentials in Python.

Migrating to AWS KMS Secrets Provider

CLI Provider Swap Command

State migration must remain atomic. The change-secrets-provider subcommand re-encrypts ciphertext values without altering resource URNs. Target a specific KMS alias and specify the AWS SDK version to avoid legacy API deprecation. The command needs decrypt access to the old provider and encrypt access to the new one simultaneously, so run it from a session that holds both.

# CLI: create the key, then re-seal the stack against it
aws kms create-key --description "Pulumi stack secrets" --key-usage ENCRYPT_DECRYPT
aws kms create-alias --alias-name alias/pulumi-secrets-key --target-key-id <key-id>
pulumi stack select prod
pulumi stack change-secrets-provider "awskms://alias/pulumi-secrets-key?region=us-east-1&awssdk=v2"

State implication: the swap rewrites encryptedkey and every secure: value in Pulumi.prod.yaml, and rewrites the encrypted properties inside the checkpoint. Commit the changed stack file in the same pull request as the migration, or the next operator's CLI will try to unseal with the previous key.

Prefer an alias over a bare key ARN in the URL. The alias gives you a level of indirection for disaster recovery, but treat repointing it as a destructive act: existing ciphertext was produced by the key the alias used to reference, and the new key cannot read it.

Typed Secret Retrieval in Python

Raw string interpolation bypasses Pulumi's secret masking engine. Wrap sensitive values in pulumi.Output types immediately. config.get_secret() and config.require_secret() both return Output[str]—never assign these to a plain str type annotation.

# CLI: pulumi up --stack prod   (module imported by __main__.py)
import pulumi
from typing import Dict, Optional

def get_db_credentials(config: pulumi.Config) -> Dict[str, pulumi.Output[str]]:
    """Retrieve typed database credentials with explicit secret wrapping.

    Both return values are Output[str], not str—they are resolved asynchronously
    and will appear as [secret] in pulumi preview output.
    """
    username: str = config.require("db_username")
    # State implication: require_secret keeps the value sealed in the checkpoint.
    password: pulumi.Output[str] = config.require_secret("db_password")

    return {
        "user": pulumi.Output.from_input(username),
        "pass": password,
    }

Any transformation of a secret must go through .apply(), and the result stays secret: Pulumi propagates the secret bit through the dependency graph, so a connection string built from a sealed password is itself sealed in state. Use pulumi.Output.secret() to mark a value that arrives from somewhere Pulumi cannot know is sensitive, such as a boto3 call inside an apply. Typed configuration objects layered on top of this are covered in using Pulumi config and typed settings in Python.

IAM Policy Scoping & Least Privilege

Broad KMS permissions violate zero-trust architectures. Scope the key policy to the deploy role and the humans who genuinely run pulumi up, and grant only kms:Encrypt and kms:Decrypt on that one key ARN. Cross-account decryption requires both a key-policy statement naming the external principal and an IAM policy on that principal — a grant on one side alone yields AccessDeniedException with no hint about which side is missing. Consult the AWS Provider Deep Dive for granular IAM policy templates and alias routing strategies.

Integrating HashiCorp Vault Secrets Provider

Vault Transit Engine Configuration

The transit backend provides encryption-as-a-service without persistent secret storage. Enable the transit secrets engine and generate a dedicated keyring. Configure key rotation policies to align with organizational compliance windows, and leave min_decryption_version at 1 so previously sealed data keys stay readable after a rotation.

# CLI: provision the transit path and a dedicated key
vault secrets enable transit
vault write -f transit/keys/pulumi-stack type=aes256-gcm96
vault policy write pulumi-secrets - <<'EOF'
path "transit/encrypt/pulumi-stack" { capabilities = ["update"] }
path "transit/decrypt/pulumi-stack" { capabilities = ["update"] }
EOF

Token & Auth Method Mapping

Pulumi requires persistent authentication during stack operations. Map AppRole, TLS, or Kubernetes service accounts to the transit path. Align token TTLs with maximum deployment durations. Short-lived tokens trigger mid-apply 403 Forbidden failures, and because the unseal happens at the start of the operation while the re-seal happens at the end, a token that expires during a forty-minute database creation fails at the worst possible moment.

# CLI: switch the stack to Vault transit as its secrets provider
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="$(vault write -field=token auth/approle/login \
    role_id="$ROLE_ID" secret_id="$SECRET_ID")"
pulumi stack change-secrets-provider "hashivault://pulumi-stack"

Provider note: on Vault Enterprise, also export VAULT_NAMESPACE; without it the CLI resolves transit/ in the root namespace and fails with a 403 that names a path you can see working in the UI.

Python Fallback Typing Patterns

Dynamic secret resolution often requires conditional fallbacks. Use typing.Optional for values that may be absent. Validate secret presence before passing values to resource constructors.

# CLI: pulumi preview --stack staging
from typing import Dict, Optional, Any
import pulumi

def resolve_vault_secrets(config: pulumi.Config) -> Dict[str, Any]:
    """Dynamically resolve Vault-backed secrets with safe fallback typing."""
    api_key: Optional[pulumi.Output[str]] = config.get_secret("vault_api_key")
    region: str = config.require("deployment_region")

    return {
        "api_key": api_key,
        "region": region,
        "fallback_enabled": api_key is not None,
    }

Note what the is not None test can and cannot tell you: it proves the config key exists, not that the value decrypted successfully. Decryption happens before your program starts, so a bad transit policy never reaches this function — the CLI aborts first.

State Safety, Drift Detection & Safe Rollback

Pre-Migration State Snapshots

Provider transitions introduce cryptographic incompatibilities. Export the current state before executing any migration command. Store snapshots in version-controlled artifact storage. Maintain immutable backups for compliance audits.

# CLI: export stack state to a local artifact before touching the provider
pulumi stack export --file state-pre-migration.json

The export keeps secret values in their sealed form, so the file is safe to store in an artifact repository — but that also means it is only restorable while the old key still exists. Never schedule the previous KMS key for deletion until a full pulumi up has succeeded against the new one.

Drift Detection via pulumi refresh

Post-migration state verification prevents silent configuration divergence. Run pulumi refresh to reconcile the local state file with live infrastructure. Review diff outputs for unexpected resource replacements or property resets. A refresh is also the cheapest end-to-end proof that the new provider works: it unseals the data key, reads every secret input, and writes a re-sealed checkpoint, exercising both Encrypt and Decrypt without changing infrastructure.

Atomic State Import & Rollback

Decryption failures require immediate state restoration. Import the pre-migration snapshot to revert cryptographic bindings. Reference Pulumi Patterns & Provider Management for automated stack lifecycle governance and versioned state recovery pipelines.

# CLI: execute forced state rollback on failure
pulumi stack import --file state-pre-migration.json --force

State implication: --force overwrites the checkpoint including its integrity hash, so the imported snapshot must match the secretsprovider recorded in the current Pulumi.<stack>.yaml. Roll the stack file back in the same step, not afterwards.

Testing Boundaries & Secret Masking Validation

pytest Isolation for IaC

Unit tests must never invoke live cloud providers. Isolate configuration parsing from resource provisioning logic. Mock the Pulumi config to simulate stack evaluation without network calls — the secrets provider is deliberately outside the test boundary, because what you are testing is your typing and fallback logic, not AWS's ability to decrypt.

# CLI: pytest tests/test_secret_typing.py -q
from unittest.mock import patch

import pulumi


def test_credentials_are_output_typed() -> None:
    with patch.object(pulumi.Config, "require", return_value="app_user"), \
         patch.object(pulumi.Config, "require_secret",
                      return_value=pulumi.Output.secret("s3cr3t")):
        creds = get_db_credentials(pulumi.Config())

    assert isinstance(creds["pass"], pulumi.Output)
    # Provider note: no KMS or Vault call happens here — the seal is outside the test.
    assert pulumi.Output.all(creds["pass"]) is not None

Mocking KMS/Vault Responses

Patch pulumi.Config using unittest.mock. Return deterministic values during test execution. Validate type coercion and error handling paths without exposing real credentials.

CLI Output Redaction Verification

Secret masking relies on Pulumi's internal serialization layer. Verify that pulumi preview and pulumi up outputs display [secret] placeholders for values retrieved via require_secret() or get_secret(). Do not attempt synchronous string operations on Output[str] objects—use .apply() for all transformations. A stack output that shows a real value where you expected [secret] means the secret bit was lost somewhere in an apply chain; wrap the result in pulumi.Output.secret() at the point it is constructed.

Common Mistakes & Remediation

The exact string Pulumi emits for a missing permission is worth recognising on sight: error: constructing secrets manager of type "cloud": secrets (code=Unknown): AccessDeniedException: User: arn:aws:iam::123456789012:role/deploy is not authorized to perform: kms:Decrypt. The code=Unknown prefix is Pulumi's wrapper, not an AWS status — read past it to the AWS error code, which is what identifies the fix.

Operational Notes

Give every environment its own key, and give production a key in the same account as the workload rather than a shared tooling account. A single shared key makes the blast radius of one over-permissive IAM policy the whole estate, and it defeats the main reason for leaving service-managed encryption: proving in CloudTrail exactly which principal read production secrets and when. Key material is regional, so a stack sealed with a us-east-1 key cannot be operated from a region-isolated runner during an outage unless you use a multi-Region key, whose replica shares key material and can decrypt the same ciphertext.

Rotation splits into two independent operations that are easy to conflate. Rotating the encryption key — enabling automatic KMS rotation, or vault write -f transit/keys/pulumi-stack/rotate — is transparent, because both providers keep prior key versions available for decryption. Rotating the secret values is a separate exercise covered in rotating Pulumi stack secrets without downtime. Only the second one touches your resources.

In CI, obtain credentials through OIDC role assumption rather than long-lived keys, and make the role's session duration comfortably longer than your slowest stack operation. Add the pre-flight check as the first step of the deploy job so a missing grant fails in two seconds instead of after a fifteen-minute plan. Finally, alarm on kms:Decrypt denials for the stack key in CloudTrail: a denial that is not from a known deploy role is either a misconfigured runner or someone trying to read state they should not.

Key Takeaways

Migrating Pulumi secrets to KMS or Vault is a one-time atomic operation (change-secrets-provider) that significantly improves compliance posture. The ongoing discipline—scoped IAM policies, token TTL alignment, pre-migration snapshots—is what keeps the encrypted state safe after migration. Invest in the validation workflow before executing the migration in production.

FAQ

Can I migrate Pulumi secrets to KMS/Vault without recreating resources? Yes. pulumi stack change-secrets-provider only re-encrypts state values. Resource IDs and URNs remain intact. Validate the operation with pulumi preview before applying changes.

How does drift detection handle rotated KMS keys or Vault tokens? Pulumi does not auto-detect key rotation. Implement CI/CD checks with pulumi refresh and monitor AWS CloudTrail or Vault audit logs for AccessDenied events during stack operations. Align token lifecycles with deployment windows.

What is the testing boundary for mocking KMS/Vault in Python IaC? Use unittest.mock to patch pulumi.Config and return test values. Never mock the actual secrets provider. Test configuration resolution, type safety, and error propagation in strict isolation.

What happens if the KMS key is deleted after the stack is encrypted? The stack becomes unreadable: encryptedkey in Pulumi.<stack>.yaml can no longer be unsealed, and every operation fails before the program runs. There is no recovery path from Pulumi's side, which is why the key should carry deletion protection and a thirty-day pending window at minimum.

Do I need kms:GenerateDataKey on the key policy? No. Pulumi generates the data key locally and calls only kms:Encrypt to seal it and kms:Decrypt to unseal it. Granting GenerateDataKey adds permission without enabling anything, and its absence is never the cause of an AccessDeniedException from a stack operation.

Should each environment use its own KMS key or one shared key? One key per environment, in the account that owns the workload. Separate keys let you revoke production decrypt access without breaking staging deploys, and they make the CloudTrail record unambiguous about which environment's secrets were read.

Related

• AWS Provider Deep Dive — the parent guide on provider initialization, credential routing, and state isolation.
• Managing multi-account AWS environments with Pulumi Python — per-account stacks and assume-role providers that each consume their own KMS key.
• Rotating Pulumi stack secrets without downtime — changing the values themselves once the key that seals them is settled.
• How to Deploy an EKS Cluster with Pulumi (Python) — a workload whose database passwords and tokens you would store as KMS-encrypted secrets.