Idempotency and Drift Detection in Python IaC

Idempotency means running the same infrastructure program twice produces the same cloud state—no duplicate resources, no needless replacements. Drift is the gap that opens when reality diverges from that state through manual console edits or out-of-band automation. This guide shows how Pulumi and CDKTF guarantee convergence, how to detect drift with pulumi refresh and cdktf diff, and how to build a typed helper that compares desired against actual state. It is part of the IaC Design Principles within Python IaC Fundamentals & Strategy.

Context

A correct IaC program is a pure function of its inputs: given the same configuration, it must converge on the same resources regardless of how many times you run it. Idempotency is what makes that safe—re-applying after a failed deploy resumes rather than duplicates. Drift breaks the contract from the other direction: someone widens a security group in the console, and your state file no longer describes the live world. Both engines treat the state file as the source of truth, so detecting and reconciling drift is a first-class operation rather than an afterthought. This builds directly on the typed contracts in Python Typing for Cloud Resource Definitions, which catch schema errors before they ever reach the provider.

It helps to separate three ideas that get used interchangeably. Determinism is a property of your Python: the same config produces the same resource graph, with the same logical names, in the same order. Idempotency is a property of the engine's apply: running it against an already-converged world does nothing. Convergence is the property you actually want in production: whatever state the world is in, one apply moves it to the declared state. Determinism is a precondition for the other two — an engine cannot be idempotent about a graph that changes shape between runs.

The three-way comparison every apply performs

Every apply in either engine is a comparison between three versions of the world, and almost every confusing diff comes from not knowing which two of the three are being compared.

• Desired — what your Python program produced this run.
• State — what the checkpoint or terraform.tfstate says exists, recorded at the end of the last successful apply.
• Actual — what the cloud provider reports right now.

The engines differ in a way that matters operationally. Terraform, and therefore CDKTF, refreshes by default: cdktf diff reads every managed resource from the provider before computing the plan, so its output already accounts for the actual world. Pulumi does not — pulumi up compares desired against state and leaves actual alone unless you pass --refresh or run pulumi refresh first. That is why a security group someone widened in the console shows up in cdktf diff immediately but stays invisible to pulumi preview until a refresh happens.

Once the comparison is done, each resource lands in one of five buckets, and the bucket is decided by the provider's schema rather than by your code:

The force-new distinction is the one that causes outages. aws.ec2.Subnet.cidr_block, aws.rds.Instance.engine, and aws.s3.BucketV2.bucket are all force-new: changing them is not an edit, it is a destroy and rebuild. Nothing in Python type-checking will tell you that, because the property is a perfectly valid str either way. The only reliable signal is reading the preview — a line beginning +- in Pulumi, or must be replaced in a Terraform plan — before you approve it.

Prerequisites

• Python 3.9+ with from __future__ import annotations enabled in modules.
• Pinned engine SDKs: pulumi>=3.0, pulumi-aws>=6.0, or cdktf>=0.20 with the matching cdktf-cli.
• A remote state backend with locking already configured (S3 + DynamoDB, Pulumi Cloud, or Terraform Cloud) so refresh operations are serialized.
• IAM credentials with read permission on every resource type under management—refresh and diff call the provider's Describe/Get APIs.
• pytest>=7 for the verification step.

Implementation

1. Make re-runs converge

The engines achieve idempotency by diffing the desired graph against the recorded state and emitting only the delta. Your job is to keep resource identities stable: never derive a resource_name from a timestamp, random value, or list ordering, or every run looks like a new resource and forces a replace.

# infra/idempotent_naming.py
# CLI: pulumi up --stack dev   # run twice — second run must report "0 changes"
from __future__ import annotations

import pulumi
import pulumi_aws as aws

# Provider note: a deterministic logical name keeps the resource's URN stable across runs,
# so the engine matches it to existing state instead of creating a duplicate bucket.
def make_bucket(env: str) -> aws.s3.BucketV2:
    return aws.s3.BucketV2(
        f"artifacts-{env}",            # stable: derived only from config input
        bucket=f"acme-artifacts-{env}",
        tags={"managed-by": "pulumi", "env": env},
    )

# State implication: re-running with identical inputs is a no-op; the checkpoint is unchanged.

Two names are in play here and conflating them is the root of most "why did it replace my bucket" incidents. The first argument, artifacts-dev, is the logical name: it exists only inside the engine, it becomes part of the URN, and it is how the engine matches this declaration to the row already in state. The bucket= argument is the physical name: it is what AWS sees, and for S3 it is force-new. Changing the logical name orphans the state row and creates a second bucket. Changing the physical name destroys and recreates the bucket, taking its contents with it.

That asymmetry explains a Pulumi default that looks wrong at first glance. If you omit bucket= entirely, Pulumi auto-names the resource acme-artifacts-dev-7f3a91b, appending random hex. The random suffix is generated once, recorded in state, and reused on every subsequent run — so it is deterministic from the second apply onward, and it removes an entire class of global-namespace collisions. Auto-naming does not break idempotency; hand-rolling your own suffix with uuid4() does.

The failure patterns worth grepping your codebase for are narrow and concrete:

# infra/antipatterns.py
# CLI: pulumi preview --stack dev --diff   # each of these shows a spurious change
from __future__ import annotations

import datetime
import os
import uuid

BAD_TIMESTAMP = f"deploy-{datetime.datetime.now():%Y%m%d%H%M%S}"   # new URN every run
BAD_RANDOM = f"cache-{uuid.uuid4().hex[:8]}"                        # new URN every run
BAD_SET_ORDER = list({"10.0.1.0/24", "10.0.2.0/24"})                # order not guaranteed
BAD_ENV_LEAK = f"vpc-{os.environ['HOSTNAME']}"                      # differs per CI runner

# State implication: any of these makes the engine see a resource it has never
# recorded, so it creates a new one and schedules the old one for deletion.
GOOD_SORTED = sorted({"10.0.2.0/24", "10.0.1.0/24"})                # ['10.0.1.0/24', ...]

Dictionaries preserve insertion order in Python 3.7+, so a dict built the same way twice is safe. Sets are not, and neither is os.listdir(), glob.glob() without a sort, or anything derived from a concurrent API response. When a value must vary, put it in stack configuration where it is recorded and reviewable, not in runtime entropy.

2. Detect drift against the live cloud

Refresh reconciles the state file with the provider's current view without changing infrastructure. Run it before a deploy in CI and gate on whether anything moved.

# CLI: pulumi refresh --yes --stack prod
# Pulumi: update state from the cloud, then assert nothing drifted
pulumi refresh --yes --stack prod
pulumi preview --diff --expect-no-changes --stack prod   # non-zero exit if drift remains

# CDKTF: synthesize then diff the plan against live state
cdktf diff --stack prod   # exits non-zero when the plan is non-empty

--expect-no-changes is what turns a preview into a gate: without it, pulumi preview reports the diff and exits 0, so a pipeline step happily goes green while production has drifted. With it, any pending change produces:

# CLI: pulumi preview --expect-no-changes --stack prod
error: error: no changes were expected but changes occurred

On the CDKTF side the CLI's exit-code behaviour has moved between releases, so if the gate matters, pin it by dropping into the synthesized directory and using Terraform's documented contract directly. -detailed-exitcode returns 0 for no changes, 1 for an error, and 2 for a non-empty plan:

# CLI: bash scripts/drift_gate.sh
cdktf synth
cd cdktf.out/stacks/prod
terraform init -input=false
terraform plan -detailed-exitcode -refresh=true -out=plan.bin
case $? in
  0) echo "converged" ;;
  2) echo "DRIFT: plan is non-empty"; terraform show -no-color plan.bin; exit 1 ;;
  *) echo "plan failed"; exit 2 ;;
esac

Both paths need read permission on every managed resource type. A refresh that hits AccessDenied on one resource does not fail loudly in every engine version — it can leave that resource's state untouched, which reads as "no drift" for the one resource most likely to have been edited by hand. Audit the refresh role against the resource types in state rather than assuming the deploy role covers it.

3. Compare desired vs actual with a typed helper

For resources the engine cannot fully model—or when you want a custom alert payload—drop to a small comparator. Keep it typed so the comparison fields are explicit and mypy-checked.

# infra/drift_check.py
# CLI: python -m infra.drift_check
from __future__ import annotations

from dataclasses import dataclass
from typing import Mapping


@dataclass(frozen=True)
class ResourceSnapshot:
    resource_id: str
    attributes: Mapping[str, str]


@dataclass(frozen=True)
class DriftReport:
    resource_id: str
    drifted_keys: tuple[str, ...]

    @property
    def has_drift(self) -> bool:
        return len(self.drifted_keys) > 0


def compare(desired: ResourceSnapshot, actual: ResourceSnapshot) -> DriftReport:
    """Return the attribute keys whose live value diverges from the desired value."""
    # Provider note: `actual` is built from a read-only Describe call (e.g. boto3),
    # so this comparison never mutates cloud state.
    drifted = tuple(
        key
        for key, want in desired.attributes.items()
        if actual.attributes.get(key) != want
    )
    return DriftReport(resource_id=desired.resource_id, drifted_keys=drifted)

Verification

A minimal pytest case proves the comparator flags a widened security-group rule and stays quiet when state matches.

# infra/tests/test_drift_check.py
# CLI: pytest infra/tests/test_drift_check.py -q
from __future__ import annotations

from infra.drift_check import ResourceSnapshot, compare


def test_detects_out_of_band_change() -> None:
    desired = ResourceSnapshot("sg-01", {"ingress_cidr": "10.0.0.0/16"})
    actual = ResourceSnapshot("sg-01", {"ingress_cidr": "0.0.0.0/0"})  # console edit
    report = compare(desired, actual)
    assert report.has_drift
    assert report.drifted_keys == ("ingress_cidr",)


def test_converged_state_reports_no_drift() -> None:
    snap = ResourceSnapshot("sg-01", {"ingress_cidr": "10.0.0.0/16"})
    assert not compare(snap, snap).has_drift

Gotchas & Edge Cases

Refresh can hide a destructive deploy. pulumi refresh rewrites state to match reality, so a resource someone deleted in the console will be re-created on the next up rather than flagged. Run preview --expect-no-changes after refresh and fail the pipeline on a non-empty diff instead of auto-applying.

Provider-computed fields create phantom drift. Some attributes (auto-assigned ARNs, default tags injected by the provider, timestamps) differ on every read and will show as drift in a naive comparator. Exclude known computed keys from the desired.attributes map, or compare only the fields you actually manage.

Non-deterministic inputs defeat idempotency silently. A cidr_block pulled from an unsorted set, or a name built from datetime.now(), makes each run look different and triggers replacements that can cause downtime. Sort collections and derive every logical name from configuration, never from runtime entropy.

An interrupted apply leaves resources the state does not know about. If the process is killed after the Create call returns but before the checkpoint is written, the resource exists and the state does not record it. The next run creates a second one. Pulumi mitigates this by writing the checkpoint incrementally per resource, which is why pulumi cancel is safe and kill -9 is not; Terraform writes state at the end of the apply and relies on the lock to prevent a concurrent second attempt. Either way, the correct recovery is refresh plus a targeted import, never a manual state edit.

Idempotent is not the same as safe to run repeatedly. A converged apply is a no-op in the engine, but it still issues read calls against every managed resource. Running a drift gate every five minutes against a stack with two thousand resources is a reliable way to discover your account's DescribeSecurityGroups rate limit. Schedule scans per stack, stagger them, and treat ThrottlingException: Rate exceeded in a scan as a scheduling bug rather than a cloud problem.

Some drift is legitimate and must be excluded. Autoscaling changes desired_capacity. A deployment tool rewrites a task definition revision. A cost-allocation robot adds tags. Comparing those fields guarantees a permanently red gate, and a permanently red gate is the same as no gate.

Operational Notes

The design question a drift gate forces you to answer is which properties you actually own. Everything you declare, you own — and the engine will fight anything else that touches it. Where another system legitimately owns a field, say so explicitly rather than letting the diff argue about it every run.

In Pulumi that is ignore_changes, which takes the names of input properties the engine should read but not reconcile:

# infra/asg.py
# CLI: pulumi up --stack prod
# State implication: desired_capacity is still recorded in state, but a diff on
# it never schedules an update — the autoscaler is the owner of that number.
from __future__ import annotations

import pulumi
import pulumi_aws as aws

group = aws.autoscaling.Group(
    "web-asg",
    max_size=20,
    min_size=2,
    desired_capacity=4,
    vpc_zone_identifiers=["subnet-0a1b2c3d", "subnet-0e4f5a6b"],
    launch_template=aws.autoscaling.GroupLaunchTemplateArgs(id="lt-0f1e2d3c", version="$Latest"),
    opts=pulumi.ResourceOptions(ignore_changes=["desired_capacity", "tags"]),
)

CDKTF exposes the same idea through Terraform's lifecycle block, spelled as a typed argument rather than HCL:

# infra/asg_cdktf.py
# CLI: cdktf diff --stack prod
# Provider note: ignore_changes names the Terraform attribute, not the Python
# argument — desired_capacity, not desiredCapacity.
from cdktf import TerraformResourceLifecycle
from cdktf_cdktf_provider_aws.autoscaling_group import AutoscalingGroup

AutoscalingGroup(
    self,
    "web_asg",
    max_size=20,
    min_size=2,
    desired_capacity=4,
    vpc_zone_identifier=["subnet-0a1b2c3d", "subnet-0e4f5a6b"],
    lifecycle=TerraformResourceLifecycle(
        ignore_changes=["desired_capacity", "tags"],
        prevent_destroy=True,
    ),
)

prevent_destroy=True is worth pairing with it on anything stateful. It converts an accidental force-new into a plan-time failure — Error: Instance cannot be destroyed — rather than a successful apply that took the database with it.

Two habits keep a drift gate useful over time. First, treat every exclusion as a documented decision with an owner, not as a way to quiet a noisy check; a growing ignore_changes list is a signal that something outside IaC is quietly taking over the resource. Second, classify what the gate finds before you page anyone: a changed tag is an audit note, a widened ingress rule is an incident, and a resource that exists in state but not in the cloud is a deletion someone performed without a change record. Route the three to different destinations and the gate will still be switched on a year later.

FAQ

Does pulumi refresh change my infrastructure? No. Refresh only updates the local/remote state file to match what the provider reports; it issues read calls, not writes. The risk is the opposite: after refreshing, a subsequent up may act on the newly-reconciled state, so always preview before applying.

How is CDKTF drift detection different from Pulumi? CDKTF delegates to Terraform: cdktf diff runs terraform plan against the synthesized HCL JSON, comparing the plan to the state backend. Pulumi computes the diff inside its own engine after an optional refresh. Both surface out-of-band changes, but only Pulumi's refresh rewrites state as a distinct step.

Can strong typing prevent drift on its own? Partially. Types catch schema and configuration errors at edit time, but drift originates outside your code—someone editing the cloud directly. Pair the typed contracts from Python Typing for Cloud Resource Definitions with a scheduled refresh-and-diff job to cover both failure directions.

How often should I run drift detection? Run it on every deploy as a pre-apply gate, and on a schedule (hourly or daily) for production stacks via a CI job that calls pulumi refresh + preview --expect-no-changes or cdktf diff and alerts on a non-zero exit. Below roughly fifteen-minute intervals the read calls start to compete with your own applications for the provider's API rate limit.

Why does a property change destroy and recreate the resource instead of updating it? Because the provider's schema marks that property force-new — the cloud API has no way to change it in place. bucket on S3, cidr_block on a subnet and engine on an RDS instance are all in this category. The preview is the only place this is visible; Python types cannot distinguish an updatable str from a force-new one.

Is auto-generated resource naming compatible with idempotency? Yes. Pulumi's random suffix is computed once and then stored in state, so every subsequent run reuses it. What breaks idempotency is generating entropy in your own code — uuid4(), a timestamp, a hostname — because that produces a different value on every execution and therefore a different resource identity.

How do I stop the gate flagging changes another system is supposed to make? Declare the field as not-yours with ignore_changes in Pulumi or TerraformResourceLifecycle(ignore_changes=[...]) in CDKTF. The engine keeps reading the value into state but stops trying to reconcile it, which is exactly right when an autoscaler or a deployment tool owns it.

Related

• Python Typing for Cloud Resource Definitions — edit-time guarantees that complement runtime drift detection.
• How to Structure Python IaC Projects for Scale — where to wire scheduled drift scans into the project layout and CI gates.
• IaC Design Principles — the parent section covering state, locking, and convergence invariants.