Detect and Remediate State Drift in Python IaC

Drift is the gap between what your code says and what the cloud actually holds — a console edit, a manual hotfix, an out-of-band automation. This guide, part of managing IaC state under Python IaC fundamentals and strategy, shows how to detect drift with Pulumi and CDKTF and how to remediate it safely, building on idempotency and drift detection.

Problem Framing

State records what your tool believes exists. When reality diverges — someone widens a security group by hand — the next apply may revert the change, or worse, act on stale assumptions and delete something. Detecting drift on a schedule turns these surprises into reviewable diffs instead of 2 a.m. incidents.

A scheduled refresh compares declared state to reality and surfaces drift for remediation.

Prerequisites

Prerequisites: the stack from the Python interface down to the cloud runtime.

A configured remote state backend so refresh reads the shared source of truth
Read access to the live cloud resources for the refresh credentials
A CI schedule (cron) able to run pulumi refresh or cdktf diff non-interactively

# CLI: dry-run a drift check without changing anything
pulumi preview --refresh --diff

Detecting Drift in CI

Run a refresh-and-preview on a schedule and fail the job when it reports changes. For Pulumi, --expect-no-changes turns any drift into a non-zero exit; for CDKTF, parse the cdktf diff output.

A scheduled job refreshes state, diffs, and alerts when reality has diverged.

# drift_gate.py — fail CI when a Pulumi stack has drifted
# CLI: python drift_gate.py   (run from a scheduled pipeline)
import subprocess, sys

def check(stack: str) -> int:
    # State implication: --refresh updates state from reality before diffing.
    proc = subprocess.run(
        ["pulumi", "preview", "--refresh", "--expect-no-changes", "-s", stack],
        capture_output=True, text=True)
    if proc.returncode != 0:
        sys.stderr.write("DRIFT DETECTED\n" + proc.stdout)
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(check("prod"))

Remediation Strategies

There are two honest responses to drift. Revert — accept your code as the source of truth and pulumi up to bring reality back. Adopt — the manual change was correct, so encode it in code and refresh so state matches. Never leave drift unresolved: an unreconciled diff makes the next unrelated deploy dangerous.

Decide per drift whether code or reality is correct, then reconcile.

Verification

After remediating, a fresh refresh-and-preview must report no changes — the definition of reconciled.

Verification: the test drives the program and asserts on resolved values.

# CLI: prove the stack is reconciled
pulumi preview --refresh --expect-no-changes -s prod && echo "no drift"

Gotchas & Edge Cases

Gotchas & Edge Cases: the boundaries where things break and what to check.

Refresh can mask deletions. If a resource was deleted out of band, refresh removes it from state, and the next apply recreates it — usually what you want, but confirm before applying in production.

Provider defaults look like drift. Some providers normalise values (tags, ordering), producing perpetual diffs. Pin or ignore those fields with resource options rather than fighting them every run.

Secrets in the diff. Drift output can print sensitive values; scrub logs or mark those outputs secret so drift reports stay safe to share.

Idempotency and Drift Detection in Python IaC — the design principles that keep applies safe and repeatable.
Migrating IaC State Between Backends — handling state carefully during larger operations.
Managing IaC State — the parent cluster on state backends and reconciliation.