Managing IaC State for Python Projects

State is the ledger that maps your Python infrastructure code to the real resources it created; getting it wrong corrupts deployments, so this guide covers the backends, locking, encryption, and per-environment isolation that every Python IaC project needs, as part of the broader Python IaC Fundamentals & Strategy discipline.

Whether you run Pulumi or CDKTF, the same physics apply: a single mutable file (or remote object) records resource IDs, and two runs touching it at once will clobber each other. The differences are in where that ledger lives and how the tool acquires a lock before mutating it.

Three operational questions decide whether state helps or hurts you. Where does the ledger live, and who can read it? What stops two runs from writing it at the same moment? And how do you find out when the ledger and the real world have diverged? This topic answers all three, and hands off to three focused guides for the decisions that deserve their own treatment: Choosing a State Backend for Python IaC weighs S3+DynamoDB against GCS, Terraform Cloud and Pulumi Cloud; How to Migrate IaC State Between Backends covers the export/import cutover once you outgrow your first choice; and Detect and Remediate State Drift in Python IaC shows how to catch out-of-band changes before they turn a routine deploy into a replacement.

Problem Framing

Without managed state, your tool has no memory of what it built. The next run either tries to recreate everything or, worse, two engineers run pulumi up against the same stack simultaneously and one overwrites the other's resource IDs, leaving orphaned cloud resources and a state file that no longer reflects reality. Drift then accumulates silently until an apply deletes something it shouldn't. The fix is a remote, locked, encrypted backend with strict isolation between environments — the same discipline that underpins safe IaC design principles.

The failure is rarely dramatic on the day it happens. A laptop-local terraform.tfstate works perfectly for the engineer who created it; the damage appears weeks later when a second engineer clones the repository, runs terraform plan, sees Plan: 47 to add, 0 to change, 0 to destroy, and realises the tool has no idea the VPC already exists. Apply that plan and you get a second VPC, a duplicate NAT gateway billing by the hour, and two ledgers each convinced it owns the truth. Reconciling them means hand-writing terraform import blocks for every resource — hours of careful work that a five-minute backend decision would have prevented.

The concurrency failure is worse because it corrupts rather than duplicates. State objects are read-modify-write: the tool downloads the whole document, mutates it in memory as resources are created, and uploads the whole document at the end. Two overlapping runs both start from the same base version; the second upload silently discards every resource the first run recorded. Those resources still exist in the cloud account, but nothing tracks them any more. They will not be updated, will not be destroyed, and will not appear in any cost attribution keyed on stack tags.

The third failure mode is disclosure. A state document is not a manifest of names — it holds the last-applied value of every attribute, including RDS master passwords, generated TLS private keys, service-account JSON, and any provider argument marked sensitive. A state bucket with default ACLs and no KMS key is a credential store with no access controls. This is why the baseline for any serious project is the same four properties: remote storage, server-side encryption, mandatory locking, and one state object per environment.

Prerequisites

• Python 3.9+ with your IaC toolchain installed (pulumi CLI or cdktf CLI plus the Terraform binary).
• Cloud credentials with permission to read/write the backend store (e.g. an S3 bucket and DynamoDB table, or a GCS bucket).
• For Pulumi: a chosen backend selected via pulumi login.
• For CDKTF: a configured Terraform backend (S3, GCS, or Terraform Cloud).

# CLI: verify your toolchain and backend selection before doing anything else.
pulumi whoami        # confirms which Pulumi backend you are logged into
cdktf --version      # confirms the CDKTF CLI is on PATH
terraform version    # CDKTF delegates state ops to this binary

Two prerequisites are easy to overlook. First, the identity that runs your deploys needs separate permissions on the state store and on the resources it manages. A CI role that can create EC2 instances but cannot call s3:PutObject on the state bucket will apply successfully and then fail to persist the result — the resources exist, the ledger does not, and the next run tries to build them again. Grant s3:GetObject, s3:PutObject, s3:DeleteObject and s3:ListBucket on the state prefix, plus dynamodb:GetItem, dynamodb:PutItem and dynamodb:DeleteItem on the lock table.

Second, enable bucket versioning before the first apply, not after. Versioning is the only cheap rollback you get: if a partial write or a bad state rm corrupts the object, restoring the previous version is a single API call. Without it, your recovery path is reconstructing the ledger by hand from aws ec2 describe-* output. The equivalent on GCS is object versioning on the bucket; Pulumi Cloud and Terraform Cloud keep their own version history automatically.

# CLI: confirm the backend prerequisites are actually in place before deploying.
aws s3api get-bucket-versioning --bucket my-iac-state
# Expect: {"Status": "Enabled"} — an empty response means versioning is OFF.
aws s3api get-bucket-encryption --bucket my-iac-state
# Expect an SSEAlgorithm of aws:kms; ServerSideEncryptionConfigurationNotFoundError means none.
aws dynamodb describe-table --table-name iac-locks \
  --query 'Table.KeySchema[0].AttributeName'
# Expect: "LockID" — Terraform will not use a table with any other hash key.

Concept Explanation

What state actually stores

State is a serialized graph of every managed resource: its logical name, its provider-assigned ID, its last-known input values, and dependency edges. Pulumi keeps this as a checkpoint in its own backend; CDKTF synthesizes Terraform JSON and lets the Terraform binary own the .tfstate. In both cases the file contains resource IDs and frequently secret values, which is why encryption at rest is non-negotiable.

The logical name matters more than it looks. It is the join key between your Python source and the cloud. In Pulumi it is a URN of the form urn:pulumi:prod::billing::aws:s3/bucket:Bucket::invoices, assembled from the stack name, the project name, the resource type token, and the logical name you passed as the first argument to the constructor. In Terraform state it is an address such as aws_s3_bucket.invoices or, inside a module, module.storage.aws_s3_bucket.invoices. Rename the Python variable and nothing happens; rename the logical name string and the tool sees the old resource disappear and a new one appear, producing a destroy-and-create plan for a bucket that never needed to change. That single fact explains most surprise replacements in a Python IaC codebase.

The state document also records the last-applied inputs, not the current desired inputs. This is what makes a plan cheap: the engine compares the inputs your program produces this run against the inputs stored from last run, and only calls the provider's read API for resources whose comparison is inconclusive or when you explicitly refresh. Where a provider marks an argument as ForceNew (Terraform) or the resource declares it as replace-triggering (Pulumi), a changed input is not an update at all — it is a delete followed by a create, in that order unless create_before_destroy is set.

Locking prevents concurrent corruption

Before mutating state, the tool acquires a lock. Terraform uses a DynamoDB item (with S3 backends) or the native lock of Terraform Cloud; Pulumi Cloud and most Pulumi object backends acquire a per-stack lock automatically. A run that cannot acquire the lock blocks rather than racing.

The mechanism is a conditional write, not a mutex held in memory. Terraform writes an item whose partition key is <bucket>/<key> with a ConditionExpression of attribute_not_exists(LockID). If the item already exists, DynamoDB rejects the write with ConditionalCheckFailedException, and Terraform surfaces it as Error acquiring the state lock along with the ID, Operation, Who, Created and Path fields of the holder. Because the condition is evaluated server-side and DynamoDB writes are strongly consistent for a single item, this is a correct mutual exclusion primitive — no clock skew, no lease renewal, no split brain.

It is also why the lock survives a crashed run. Nothing releases it except a successful DeleteItem at the end of the operation, so a laptop that loses power mid-apply leaves the item behind and every subsequent run blocks. That is the intended trade-off: a stuck lock is an inconvenience you resolve deliberately, whereas an auto-expiring lease would silently permit exactly the concurrent write the lock exists to prevent.

from dataclasses import dataclass

@dataclass(frozen=True)
class BackendConfig:
    """Typed description of a remote, locked, encrypted state backend."""
    bucket: str
    region: str
    lock_table: str          # DynamoDB table for Terraform locking
    kms_key_arn: str | None  # encrypt state at rest

def backend_block(cfg: BackendConfig, env: str) -> dict[str, object]:
    # CLI Context: feed into cdktf via add_override("terraform.backend", ...)
    # State implication: the `key` is namespaced per environment so dev/staging/prod
    # never share one state object — concurrent envs cannot corrupt each other.
    return {
        "s3": {
            "bucket": cfg.bucket,
            "key": f"iac/{env}/terraform.tfstate",
            "region": cfg.region,
            "dynamodb_table": cfg.lock_table,
            "encrypt": True,
            "kms_key_id": cfg.kms_key_arn,
        }
    }

Encryption and isolation

State leaks secrets if stored in plaintext. Enable bucket-level encryption (SSE-KMS on S3, CMEK on GCS) and restrict read access by IAM. Isolation means each environment gets its own state object — a distinct S3 key, GCS prefix, Pulumi stack, or Terraform workspace — so a destroy in dev can never touch prod.

There are two independent layers of encryption and it is worth being precise about which protects what. Bucket-level encryption (encrypt = true plus a kms_key_id) protects the whole document at rest and in transit to the store; anyone who can read the object through the API still sees plaintext. Value-level encryption protects individual fields inside the document: Pulumi encrypts every value you marked with pulumi config set --secret or wrapped in pulumi.Output.secret() using the stack's secrets provider, so those fields remain ciphertext even for someone holding the raw checkpoint. Terraform has no value-level equivalent — a sensitive = true argument is redacted in CLI output but written to state in the clear — which makes bucket-level encryption plus tight IAM the entire control.

Isolation should be enforced by permissions, not by convention. Naming the prod state object iac/prod/terraform.tfstate documents the boundary; a bucket policy that denies the dev CI role any action on arn:aws:s3:::my-iac-state/iac/prod/* enforces it. Terraform workspaces are the weakest form of isolation, because every workspace lives under the same key prefix and the same credentials — convenient for ephemeral feature environments, wrong for the dev/prod boundary.

Reading a State Document by Hand

You will eventually need to look inside state: to confirm a resource is tracked, to find the provider ID of something you are about to import, or to work out why a plan wants to replace a resource you did not touch. Both tools give you a read path that does not involve downloading the raw object.

Always read through the tool rather than fetching the object from the bucket. pulumi stack export decrypts nothing but reproduces the checkpoint exactly as stored; terraform show -json renders the current state with the provider schema applied, which is what you want when an attribute name in the raw JSON does not match the argument name in your Python. Both are safe to pipe into jq.

from __future__ import annotations

import json
import subprocess
from dataclasses import dataclass


@dataclass(frozen=True)
class TrackedResource:
    """One resource as the state ledger currently describes it."""
    urn: str
    resource_type: str
    provider_id: str | None


def export_pulumi_resources(stack: str) -> list[TrackedResource]:
    """Read the selected stack's checkpoint and flatten it to typed rows."""
    # CLI: python inspect_state.py   (wraps `pulumi stack export --stack <stack>`)
    raw = subprocess.run(
        ["pulumi", "stack", "export", "--stack", stack],
        capture_output=True, text=True, check=True,
    ).stdout
    # State implication: export is read-only — it never acquires the stack lock
    # and never mutates the checkpoint, so it is safe to run during a deploy.
    doc = json.loads(raw)
    rows: list[TrackedResource] = []
    for res in doc["deployment"]["resources"]:
        rows.append(TrackedResource(
            urn=res["urn"],
            resource_type=res["type"],
            provider_id=res.get("id"),
        ))
    return rows


def find_untracked_ids(rows: list[TrackedResource]) -> list[str]:
    """Resources with no provider id are pending-create or failed mid-apply."""
    return [r.urn for r in rows if r.provider_id is None]

A resource with no id is the signature of an interrupted apply: the engine recorded its intent to create the resource but never received a provider ID back. Pulumi calls these pending operations and refuses to proceed until you either re-run the operation or remove the entry with pulumi cancel followed by a pulumi refresh.

The Lock Protocol, Step by Step

Locking is the part engineers most often work around and most often regret working around. Following the exact sequence of calls makes it obvious why every step exists and which ones can leave residue.

Note the ordering: the lock is taken before the read, not before the write. If it were taken only at write time, two runs could both read version 1, both plan against it, and the second would still overwrite the first — the plan itself would be computed from stale data. Holding the lock across read, plan and write is what makes the whole operation serialisable.

Two consequences follow for CI design. First, a long apply holds the lock for its entire duration, so a pipeline that fans out five stacks against one state object serialises to a queue; if that hurts, split the state objects rather than disabling the lock. Second, -lock-timeout is a waiting policy, not an override: terraform apply -lock-timeout=10m retries acquisition for ten minutes and then fails. -lock=false is the actual override and belongs in no pipeline that writes state.

# CLI: inspect who holds a stuck lock before deciding to break it.
aws dynamodb get-item --table-name iac-locks \
  --key '{"LockID":{"S":"my-iac-state/iac/prod/terraform.tfstate"}}' \
  --query 'Item.Info.S' --output text | python3 -m json.tool
# Fields returned: ID, Operation, Who, Version, Created, Path.
# State implication: force-unlock deletes this item; if the holder is still
# running it will then write state concurrently with the next run.

Refresh, Drift, and Keeping the Ledger Honest

State is a cache of the cloud, and like every cache it goes stale. Someone widens a security group in the console during an incident, an autoscaling policy rewrites a desired count, a compliance tool retags a bucket. None of that reaches your state object until something reads the live resource back.

Refresh is a state write, not a cloud write: it calls each resource's read API and updates the stored attributes to match what the provider returned. That makes it safe in the sense that no infrastructure changes, and dangerous in the sense that it can rewrite your ledger to record a manual change as the new baseline. The right posture is to refresh on a schedule, diff the result, and decide deliberately whether reality or code is correct — the workflow developed in Detect and Remediate State Drift in Python IaC.

# CLI: detect drift without letting it become the new baseline.
pulumi refresh --diff --preview-only --stack prod
# State implication: --preview-only computes the refresh diff and discards it,
# so the checkpoint is untouched; drop the flag to accept reality into state.

terraform -chdir=cdktf.out/stacks/prod plan -refresh-only -detailed-exitcode
# Exit code 0 = no drift, 2 = drift detected, 1 = the command itself failed.

Deleted resources are the case worth rehearsing. If someone deletes a resource out of band, refresh removes it from state and the next plan proposes to recreate it — usually the correct outcome. If someone deletes it and you have since removed the resource from your Python program, refresh drops it silently and there is no record it ever existed. That is why a drift check belongs in a scheduled pipeline and not only in the pre-apply path.

Step-by-Step Implementation

1. Pick and provision a backend

Choose the store that matches your team's operational model. The trade-offs between S3+DynamoDB, GCS, Terraform Cloud, and Pulumi Cloud are weighed in detail in Choosing a State Backend for Python IaC.

# CLI: provision an S3 backend + DynamoDB lock table once, out of band.
aws s3api create-bucket --bucket my-iac-state --region us-east-1
aws s3api put-bucket-versioning --bucket my-iac-state \
  --versioning-configuration Status=Enabled
aws dynamodb create-table --table-name iac-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST
# State implication: versioning lets you roll back a corrupted state object.

2. Wire the backend into your stack

For CDKTF, the backend is configured in cdktf.json or via add_override. Pulumi selects its backend with pulumi login and isolates with stacks. The CDKTF-specific mechanics — including remote execution — are covered in State Backend Configuration for CDKTF.

from constructs import Construct
from cdktf import TerraformStack

class StatefulStack(TerraformStack):
    def __init__(self, scope: Construct, ns: str, env: str) -> None:
        super().__init__(scope, ns)
        # Provider note: backend must be set before any resource is synthesized.
        self.add_override("terraform.backend", {
            "s3": {
                "bucket": "my-iac-state",
                "key": f"iac/{env}/terraform.tfstate",
                "region": "us-east-1",
                "dynamodb_table": "iac-locks",
                "encrypt": True,
            }
        })
        # CLI Context: cdktf synth && terraform -chdir=cdktf.out/stacks/<ns> init

3. Isolate per environment

For Pulumi, one stack per environment is the idiomatic boundary; the patterns for organizing those stacks live in Pulumi Stack Architecture.

# CLI: create one Pulumi stack per environment.
# Pulumi: each stack is an independent, locked state object.
pulumi stack init dev
pulumi stack init staging
pulumi stack init prod
# State implication: `pulumi up` only ever mutates the currently selected stack.

For CDKTF the equivalent boundary is one synthesized stack per environment, each with its own backend key. Resist the temptation to use a single stack with a terraform workspace per environment unless the environments are genuinely disposable — workspaces share credentials, share the backend configuration, and are selected by an ambient CLI setting that is easy to get wrong in a pipeline.

4. Learn the four surgical state operations

Ninety-five percent of the time you never touch state directly. The remaining five percent is where damage happens, so know exactly what each operation does before you need it.

# CLI: the four operations that edit state without changing infrastructure.
terraform -chdir=cdktf.out/stacks/prod state list          # enumerate tracked addresses
terraform -chdir=cdktf.out/stacks/prod state show aws_s3_bucket.invoices
terraform -chdir=cdktf.out/stacks/prod state mv \
  aws_s3_bucket.old_name aws_s3_bucket.invoices            # rename without replacing
terraform -chdir=cdktf.out/stacks/prod state rm aws_s3_bucket.invoices
# State implication: `state rm` ORPHANS the real bucket — it keeps existing but is
# no longer managed. Pair it with `terraform import` to re-adopt under a new address.

The Pulumi equivalents are pulumi state delete <urn> (with --force when the resource still has dependents), pulumi state rename, and pulumi import <type> <name> <id> to adopt an existing cloud resource. Every one of these should be preceded by an export: pulumi stack export --file pre-surgery.json costs a second and is the difference between a mistake and an outage.

Verification

Confirm that state is remote, locked, and isolated before trusting it.

# CLI: confirm state is remote, populated and isolated.
# Pulumi: list resources recorded in the selected stack's state.
pulumi stack select prod
pulumi stack --show-urns

# CDKTF/Terraform: confirm the backend is remote and inspect the lock table.
terraform -chdir=cdktf.out/stacks/prod state list
aws dynamodb scan --table-name iac-locks --max-items 5
# A populated state list plus a backend that is NOT "local" confirms remote state.

Four assertions are worth automating so they hold on every branch, not just on the day you set the backend up. The backend must not be local. The lock must actually be enforceable. The object must be encrypted with the key you expect. And two environments must not resolve to the same storage path — the check that catches copy-pasted stack definitions.

from __future__ import annotations

import json
import pathlib


def assert_isolated_backends(synth_root: pathlib.Path) -> None:
    """Fail the build if two synthesized stacks share one state object."""
    # CLI: python verify_state.py   (run after `cdktf synth`, before deploy)
    seen: dict[tuple[str, str], str] = {}
    for stack_dir in sorted(synth_root.glob("stacks/*")):
        doc = json.loads((stack_dir / "cdk.tf.json").read_text())
        s3 = doc["terraform"]["backend"]["s3"]
        # Provider note: cdktf writes the backend block verbatim into cdk.tf.json,
        # so this reads exactly what terraform init will consume.
        assert s3.get("encrypt") is True, f"{stack_dir.name}: encrypt must be true"
        assert s3.get("dynamodb_table"), f"{stack_dir.name}: no lock table configured"
        location = (s3["bucket"], s3["key"])
        if location in seen:
            raise AssertionError(
                f"{stack_dir.name} shares state with {seen[location]}: {location}"
            )
        seen[location] = stack_dir.name

Then prove the lock works end to end rather than assuming it. Start a long apply in one terminal and a second one in another; the second must print Error acquiring the state lock within a few seconds. A run that proceeds means the dynamodb_table argument is missing or points at a table with the wrong hash key, and Terraform will have warned once at init time in a message most people scroll past.

Troubleshooting

Error: Error acquiring the state lock / ConditionalCheckFailedException. Cause: a previous run crashed without releasing the DynamoDB lock. Fix: confirm no run is active, then terraform force-unlock <LOCK_ID> (Pulumi: pulumi cancel). Never force-unlock while a deploy is genuinely running.

Symptom: two environments share one state object. Cause: the same key/stack name reused across envs. Fix: namespace the S3 key per environment (or use one Pulumi stack each) and migrate as shown in Migrating IaC State Between Backends.

Symptom: secrets visible in plaintext state. Cause: encryption disabled or secrets passed as plain config. Fix: enable SSE-KMS/CMEK on the bucket and use pulumi config set --secret or CDKTF TerraformVariable(sensitive=True).

Error: Error: Failed to load state: AccessDenied: Access Denied status code: 403. Cause: the deploy identity can list the bucket but lacks s3:GetObject on the specific state key, or the object is encrypted with a KMS key the role has no kms:Decrypt grant for. The distinction matters because the error text is identical. Fix: run aws s3api head-object --bucket my-iac-state --key iac/prod/terraform.tfstate under the same role — if that succeeds, the failure is the KMS grant, so add the role to the key policy and to any kms:ViaService condition on s3.<region>.amazonaws.com.

Error: Error: Resource already managed by Terraform on import, or Pulumi's error: resource 'urn:pulumi:...' already exists. Cause: you are importing a resource that another address in the same state already tracks, usually after a partially completed migration. Fix: terraform state list | grep <id> (or pulumi stack export | jq '.deployment.resources[].id') to find the existing address, then decide whether to state mv it to the address you want or drop the duplicate import.

Symptom: a plan proposes to destroy and recreate a resource nobody edited. Cause: the logical name changed — a refactor renamed the first argument of a construct, or a resource moved into a Python helper function that added a module prefix, so the old address is absent and a new one appears. Fix: do not apply. Restore the identity instead with terraform state mv module.old.aws_db_instance.main aws_db_instance.main, or use Pulumi's aliases=[pulumi.Alias(name="old-name")] resource option to tell the engine the two names refer to one resource.

FAQ

Do Pulumi and CDKTF use the same state format? No. Pulumi serializes its own checkpoint format to a Pulumi backend, while CDKTF produces standard Terraform .tfstate managed by the Terraform binary. Both support remote storage, locking, and encryption, but the files are not interchangeable.

Can I keep state in Git? No. State contains resource IDs and often secrets, mutates on every run, and has no locking in Git — concurrent commits would corrupt it. Use a locked remote object store instead.

How many environments should share a backend bucket? A single bucket is fine; isolation comes from a distinct key/prefix (or stack) per environment, not a separate bucket. Use IAM and KMS policies to fence prod access.

What happens if the lock table is deleted? Terraform falls back to no locking and warns. Recreate the table immediately; until then, serialize runs manually to avoid concurrent state writes.

How do I recover from a corrupted state object? Restore the previous version from the bucket: aws s3api list-object-versions --bucket my-iac-state --prefix iac/prod/ gives you the version IDs, and a get-object --version-id writes the good copy back. With Pulumi, pulumi stack import --file <snapshot>.json replaces the checkpoint wholesale. Both are why versioning and a pre-surgery export are prerequisites rather than nice-to-haves.

Is it safe to run refresh in CI on every merge? Running it in preview mode is safe and useful — it tells you whether reality has moved. Letting a pipeline accept the refresh unattended is not, because it converts an unauthorised console change into the recorded baseline without anyone reviewing it. Gate acceptance behind a human approval, or fail the pipeline on a non-zero drift exit code.

Related

• Choosing a State Backend for Python IaC — compare S3+DynamoDB, GCS, Terraform Cloud, and Pulumi Cloud with a decision table.
• How to Migrate IaC State Between Backends — export, import, and cut over without downtime.
• Detect and Remediate State Drift in Python IaC — scheduled refresh checks and how to decide between code and reality.
• State Backend Configuration for CDKTF — CDKTF-specific backend wiring and remote execution.
• Pulumi Stack Architecture — organizing Pulumi stacks and cross-stack references.