Choosing a State Backend for Python IaC

Picking the wrong state backend locks your team into the wrong operational model — manual locking, no audit trail, or a single point of failure — so this guide compares S3+DynamoDB, GCS, Terraform Cloud, Pulumi Cloud, and self-managed options against concrete criteria, as part of Managing IaC State for Python Projects within Python IaC Fundamentals & Strategy.

The backend decision is hard to reverse cheaply once dozens of stacks depend on it, so make it deliberately. The right choice depends on which cloud you already run in, whether you want managed locking, and how much you value a hosted UI versus owning the storage.

Context

State backends differ on five axes that matter day to day: where the file lives, how locking is enforced, how secrets are encrypted, whether there is a hosted policy/UI layer, and the operational burden of running it. CDKTF state backends are standard Terraform backends, so the CDKTF-specific wiring is covered in State Backend Configuration for CDKTF; Pulumi backends are selected with pulumi login and organized as described in Pulumi Stack Architecture.

It helps to be concrete about what the backend is storing. A state file is a mapping from the logical name your program gives a resource to the identifier the cloud gave it back, plus the inputs that produced it, the outputs other resources depend on, and the dependency edges between them. That mapping is the only reason pulumi up knows an existing bucket is the bucket your code means rather than something to create again. Lose it and you have infrastructure with no owner; corrupt it and the next update proposes changes nobody wrote.

Two properties follow from that. The state object is small, written rarely, and read on every operation — so throughput never matters and durability always does. And it is the most sensitive file in the repository's orbit: it lists every resource, every configuration value, and depending on the tool, every secret those resources were given.

Prerequisites

• A target cloud account (AWS, GCP) or a Pulumi Cloud / Terraform Cloud organization.
• Python 3.9+ with the pulumi CLI or cdktf CLI plus the Terraform binary.
• IAM permissions to create the storage and lock primitives (S3 bucket + DynamoDB table, or GCS bucket).
• A decision on whether locking must be automatic (managed backends) or self-operated.

Decision Table

How Locking Actually Works

"Managed locking" hides five different mechanisms, and the difference only becomes interesting when a run dies halfway and leaves the lock behind. Knowing which object to inspect turns a twenty-minute outage into a one-line fix.

The Terraform S3 backend writes a single item into DynamoDB keyed by <bucket>/<key>-md5, containing who took the lock and when. A second apply reads it and fails immediately with Error acquiring the state lock: ConditionalCheckFailedException followed by a Lock Info block naming the operation, the user, and the lock ID. terraform force-unlock <ID> deletes that item — and nothing else, which is why forcing a lock while another apply is genuinely still running is how state gets corrupted. Terraform 1.10 added native S3 locking via a .tflock object in the same bucket, and 1.11 deprecated the DynamoDB table; new CDKTF projects should set use_lockfile and skip the table entirely.

# backends/s3_native.py — S3 backend with native locking, no DynamoDB table
# CLI: cdktf synth && terraform -chdir=cdktf.out/stacks/dev init
from __future__ import annotations
from typing import Any, Dict

def s3_backend(bucket: str, region: str, env: str) -> Dict[str, Any]:
    # State implication: use_lockfile writes <key>.tflock next to the state
    # object; a stale lock is cleared by deleting that object, not a table row.
    return {"s3": {
        "bucket": bucket,
        "key": f"iac/{env}/terraform.tfstate",
        "region": region,
        "use_lockfile": True,
        "encrypt": True,
    }}

The GCS backend needs no separate primitive at all: it writes with an object-generation precondition, so a concurrent write fails at the API rather than at a lock table. That is why the comparison above shows no extra infrastructure for GCS — the atomicity is a property of the storage service.

Pulumi's self-managed backends take a different approach again, writing lock entries under .pulumi/locks/ beside the checkpoint. A process killed mid-update leaves one behind, and the next run refuses to start with error: the stack is currently locked by 1 lock(s). pulumi cancel --stack <name> clears it. The important asymmetry: Pulumi's lock is advisory within its own tooling and is not enforced by the object store, so two people with credentials and a determination to break things still can.

Encryption Is Two Separate Decisions

Every backend comparison lists "encryption", which flattens two independent controls. Encrypting the state object at rest protects the bytes if someone reads the bucket. Encrypting individual values protects the secrets if someone reads the state file — including someone who legitimately has read access to the bucket.

Terraform, and therefore CDKTF, stores secret values in the state file in plaintext. A database password passed to an RDS instance is recoverable from terraform state pull by anyone who can read the object. The only mitigations are object-level: SSE-KMS with a key policy that restricts decryption, a bucket policy that denies non-TLS access, and IAM that keeps the reader list short. Treat the state bucket as production-secret material, not as build output.

Pulumi encrypts secret values individually before they are written, using the secrets provider chosen at stack creation — a passphrase, awskms://, gcpkms://, azurekeyvault://, or the service-managed key on Pulumi Cloud. The ciphertext is per-stack, so reading the state object without the key yields nothing useful. The cost of that design is a new failure mode: lose the passphrase and the secrets are unrecoverable, which is a real risk for the DIY backends and the reason a KMS-backed provider beats a passphrase for anything long-lived.

# CLI: harden a self-managed state bucket before anything writes to it
aws s3api put-bucket-encryption --bucket my-iac-state \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":
    {"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"alias/iac-state"}}]}'
aws s3api put-public-access-block --bucket my-iac-state \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
# State implication: versioning plus a lifecycle rule on noncurrent versions gives
# you rollback without unbounded storage growth.

Implementation

1. Configure an S3 + DynamoDB backend (AWS teams)

If your workloads already run in AWS, S3 with a DynamoDB lock table keeps state in the same trust boundary and is the cheapest managed-locking option.

# Provision once; reuse across stacks with a per-environment key.
aws s3api create-bucket --bucket my-iac-state --region us-east-1
aws s3api put-bucket-versioning --bucket my-iac-state \
  --versioning-configuration Status=Enabled
aws dynamodb create-table --table-name iac-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH --billing-mode PAY_PER_REQUEST

from dataclasses import dataclass

@dataclass(frozen=True)
class S3Backend:
    bucket: str
    region: str
    lock_table: str

def render(backend: S3Backend, env: str) -> dict[str, object]:
    # CLI Context: cdktf synth, then terraform -chdir=cdktf.out/stacks/<ns> init
    # State implication: DynamoDB enforces the lock; `encrypt` keeps state ciphertext at rest.
    return {"s3": {
        "bucket": backend.bucket,
        "key": f"iac/{env}/terraform.tfstate",
        "region": backend.region,
        "dynamodb_table": backend.lock_table,
        "encrypt": True,
    }}

2. Select a managed backend for Pulumi

For teams that want zero locking infrastructure and a history UI, Pulumi Cloud is the default; an object store works when you want to own the bytes.

# Hosted managed backend (automatic locking, RBAC, history):
pulumi login
# Or own the storage in S3 (locking via the object backend):
pulumi login s3://my-iac-state
# Provider note: switching backends later requires a stack export/import migration.

3. Choose Terraform Cloud for CDKTF policy/runs

If you want remote execution, run history, and Sentinel/OPA policy gating without self-hosting, Terraform Cloud is the managed CDKTF backend.

from constructs import Construct
from cdktf import TerraformStack

class CloudBackedStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str) -> None:
        super().__init__(scope, ns)
        # State implication: Terraform Cloud owns the state and the lock; runs can
        # execute remotely so local credentials never touch prod state directly.
        self.add_override("terraform.backend", {"remote": {
            "hostname": "app.terraform.io",
            "organization": "my-org",
            "workspaces": {"name": ns},
        }})
        # CLI Context: cdktf synth && cdktf deploy

Verification

# Confirm the backend is remote (not local) and a state object exists.
pulumi whoami                       # shows the active Pulumi backend URL
terraform -chdir=cdktf.out/stacks/dev state list   # non-empty => remote state populated
aws s3api head-object --bucket my-iac-state --key iac/dev/terraform.tfstate
# A successful head-object proves state landed in S3 rather than on local disk.

Then prove the two properties that actually protect you. Locking is only real if a second concurrent operation is refused, and versioning is only real if a previous generation of the object can be listed:

# CLI: confirm concurrency is refused and history exists
terraform -chdir=cdktf.out/stacks/dev apply -auto-approve &   # hold the lock
terraform -chdir=cdktf.out/stacks/dev plan                    # expect a lock error
aws s3api list-object-versions --bucket my-iac-state \
  --prefix iac/dev/terraform.tfstate --query 'Versions[].[VersionId,LastModified]'

For Pulumi, the equivalent evidence is the checkpoint itself. pulumi stack export prints the deployment document; counting resources in it confirms the backend holds the graph you expect rather than an empty stack that would happily recreate everything.

# CLI: confirm the backend holds a populated checkpoint, not an empty one
pulumi stack export --stack dev | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print(len(d['deployment']['resources']))"
# State implication: a count of 1 means only the stack's root resource exists —
# the backend is fresh, and an update would create everything from scratch.

Gotchas & Edge Cases

S3 without a DynamoDB table has no locking. An S3 backend missing dynamodb_table will happily allow concurrent writes and corrupt state. Always pair S3 with a lock table; verify the table name matches exactly.

Pulumi Cloud secrets vs object-backend secrets. Pulumi Cloud encrypts secrets with a managed per-stack key; an S3/GCS Pulumi backend needs a passphrase or KMS key you supply. Losing that passphrase makes encrypted secrets unrecoverable.

Region mismatch causes silent latency, not errors. A state bucket in a distant region adds round-trip latency to every plan. Co-locate the backend with the team and CI runners; it will not error, only slow you down.

Versioning is not backup, and neither is replication. Object versioning protects against a bad write; it does not protect against the bucket being deleted, the KMS key being scheduled for deletion, or an account being closed. A state bucket needs the same deletion protections as a database: s3:DeleteBucket denied outside a break-glass role, MFA delete or an explicit bucket policy on version deletion, and a lifecycle rule that keeps noncurrent versions long enough to notice a problem.

Two tools, one bucket, one prefix collision. Sharing a bucket between Pulumi and CDKTF is fine with distinct prefixes and dangerous without them. A Pulumi checkpoint written over a Terraform state key does not error — it replaces the object, and the next terraform plan reports every resource as needing creation. Enforce the separation in the bucket policy with a Condition on the key prefix per role, not in a naming convention.

Backends have size limits you meet gradually. A stack that accumulates thousands of resources produces a checkpoint measured in tens of megabytes, and every operation reads and writes the whole document. The symptom is not an error but creeping slowness, then occasional timeouts on the largest stacks. That is a signal to split the stack, not to change backend — the fix is fewer resources per state file, covered in structuring Pulumi stacks per environment.

Operational Notes

The backend decision is really about who operates the lock and the storage. A managed backend (Pulumi Cloud or Terraform Cloud) gives you locking, history, and access control with no infrastructure to run, at the cost of a dependency and a bill. A self-hosted S3 backend with a DynamoDB lock table gives you full control and keeps state inside your own account, at the cost of operating those resources yourself.

Whatever you choose, three properties are non-negotiable in a team setting: state must be encrypted at rest, versioned so you can roll back a bad apply, and locked so concurrent applies cannot corrupt it. Local state fails all three and belongs only in throwaway experiments. When you outgrow a backend, migrate deliberately using the no-op-preview procedure in migrating IaC state between backends.

Access control deserves more thought than it usually gets. The identity that runs deployments needs write access to exactly one prefix; humans should have read access at most, and ideally not that. State is the one artefact where read access is close to full disclosure — an attacker who can read it learns your topology, your resource identifiers, and with Terraform, your secrets. Where the pipeline is the only writer, deny writes to everyone else in the bucket policy rather than relying on nobody running terraform apply from a laptop.

Plan the failure drill before you need it. For a self-managed backend that means: how to restore a previous object version, who holds the KMS key policy, and what force-unlock or pulumi cancel requires. For a hosted backend it means knowing what happens when the service is unavailable — you cannot deploy, which is usually acceptable, but you also cannot read outputs other systems may depend on, which sometimes is not. Cache anything a runtime system needs outside the state store, and keep the recovery steps in the repository next to the code they protect.

Cost is rarely a decision factor and occasionally a surprise. S3 plus a lock primitive costs cents per month for the storage and per-request charges that stay invisible; the DynamoDB table, if you still run one, is trivially small on pay-per-request billing. Hosted backends charge per resource under management, which is predictable and can become the largest line in a small team's tooling bill once a few hundred resources exist. Compare against the engineer-hours a self-managed backend consumes over a year, not against zero.

FAQ

Can I use one S3 bucket for both Pulumi and CDKTF? Yes, with different key prefixes — Pulumi writes its checkpoint format and CDKTF writes Terraform .tfstate. Keep the prefixes distinct so the two tools never read each other's objects.

Is a hosted backend worth the cost over S3? If you need RBAC, run history, audit, and policy gates, the hosted backends remove real operational work. For a small team already in AWS, S3+DynamoDB is cheaper and sufficient.

How do I move off a backend I regret choosing? Use an export/import migration with verification, detailed in How to Migrate IaC State Between Backends. Plan for a brief freeze on deploys during cutover.

Does GCS support locking like DynamoDB? Yes — the GCS backend uses native object generation/locking, so you do not need a separate lock table the way S3 does.

What happens if two engineers run apply at the same time? With a locking backend the second run fails immediately with a lock error naming the holder, and nothing is written. Without one — a local file, or S3 with neither a lock table nor use_lockfile — both runs read the same state, both write, and the last writer silently discards the other's resources.

Can I keep state in Git? No. State changes on every apply, contains resolved values including secrets in the Terraform model, and has no locking, so two people applying in parallel produce a merge conflict in a file no human can safely merge. Version the code in Git and let the backend version the state.

Related

• Managing IaC State for Python Projects — the concepts behind backends, locking, and isolation.
• How to Migrate IaC State Between Backends — moving state safely once you change your mind.
• State Backend Configuration for CDKTF — wiring a backend into a CDKTF stack.