Detect and Remediate State Drift in Python IaC

Drift is the gap between what your code says and what the cloud actually holds — a console edit, a manual hotfix, an out-of-band automation. This guide, part of managing IaC state under Python IaC fundamentals and strategy, shows how to detect drift with Pulumi and CDKTF and how to remediate it safely, building on idempotency and drift detection.

Problem Framing

State records what your tool believes exists. When reality diverges — someone widens a security group by hand — the next apply may revert the change, or worse, act on stale assumptions and delete something. Detecting drift on a schedule turns these surprises into reviewable diffs instead of 2 a.m. incidents.

Not all drift is the same, and treating it as one phenomenon is why drift reports get muted. Four sources account for nearly all of it, and each wants a different response. A human console edit is the classic case: someone opens the AWS console during an incident, widens a rule, and never comes back. An out-of-band automation — an autoscaler, a certificate renewer, an operator, a backup tool that retags volumes — writes changes continuously and legitimately; that is not a mistake to revert but a field your code should stop claiming. A provider-side mutation happens without anyone acting at all: AWS fills in a default, normalises a policy document's JSON key order, or assigns an ID to a sub-resource. And an out-of-band deletion is the dangerous one, because state still describes a resource that is gone and the next plan will try to build it again.

The cost of not knowing is asymmetric. Drift you have detected is a diff in a pull request. Drift you have not detected is discovered by an unrelated deploy at the worst possible moment, when a routine change to one resource surfaces six months of accumulated divergence in twenty others and the person running the deploy has no context for any of it. The point of a scheduled check is to keep that backlog at zero so that every apply's diff contains only what its author intended.

Prerequisites

• A configured remote state backend so refresh reads the shared source of truth
• Read access to the live cloud resources for the refresh credentials
• A CI schedule (cron) able to run pulumi refresh or cdktf diff non-interactively

# CLI: dry-run a drift check without changing anything
pulumi preview --refresh --diff

What a Refresh Actually Does

A refresh is not a diff against your program. It is a read loop: for every resource in state the engine calls the provider's Read implementation, which becomes a Describe/Get API call against the cloud, and writes whatever comes back into the checkpoint. Your source code plays no part in it. Only afterwards, when a preview runs against the refreshed state, does the comparison with your program happen.

Two consequences follow, and both surprise people. First, a refresh mutates state even when nothing has drifted: the checkpoint is rewritten with a new version, and any concurrent operation will contend for the state lock. Second, a refresh is proportional in cost to the size of the stack, not to the size of the change. A thousand-resource stack means a thousand read calls, which on AWS is the fastest route to ThrottlingException: Rate exceeded — often on a completely unrelated deploy that happens to run at the same minute.

The read path also explains the failure modes. A resource the refresh credentials cannot read is not reported as drifted; it is left alone with a warning, and the stale state persists silently. That is why the drift-checking role needs read permission on every resource type under management, not just the ones you expect to change — a partial-permission drift check reports a clean stack that is anything but.

CDKTF and Terraform separate the same two phases more explicitly. terraform plan -refresh-only performs the read loop and shows what would be written to state without proposing any infrastructure change, and -detailed-exitcode turns the result into a script-friendly signal: 0 for no drift, 2 for drift found, 1 for an error. That three-way exit code is more useful than a boolean, because it lets a scheduled job distinguish "the environment drifted" from "the check itself failed".

# CLI: run the read-only phase against a synthesized CDKTF stack
cdktf synth
terraform -chdir=cdktf.out/stacks/prod init -input=false
terraform -chdir=cdktf.out/stacks/prod plan -refresh-only -detailed-exitcode
# State implication: -refresh-only never touches infrastructure; it only proposes
# state updates. Exit code 2 means reality and state disagree.

Detecting Drift in CI

Run a refresh-and-preview on a schedule and fail the job when it reports changes. For Pulumi, --expect-no-changes turns any drift into a non-zero exit; for CDKTF, parse the cdktf diff output.

# drift_gate.py — fail CI when a Pulumi stack has drifted
# CLI: python drift_gate.py   (run from a scheduled pipeline)
import subprocess, sys

def check(stack: str) -> int:
    # State implication: --refresh updates state from reality before diffing.
    proc = subprocess.run(
        ["pulumi", "preview", "--refresh", "--expect-no-changes", "-s", stack],
        capture_output=True, text=True)
    if proc.returncode != 0:
        sys.stderr.write("DRIFT DETECTED\n" + proc.stdout)
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(check("prod"))

An exit code tells you that something drifted, which is enough to fail a job but not enough to route it. Ask for structured output instead and you can name the resources, count them, and decide whether this particular drift deserves a page or a ticket. pulumi preview --json emits a plan document whose steps array carries one entry per resource with its op and urn, which is all a triage script needs.

# drift_report.py — turn a drift check into a routable summary
# CLI: python drift_report.py prod
from __future__ import annotations

import json
import subprocess
import sys
from dataclasses import dataclass

# Operations that mean "reality and code disagree". "same" is the quiet majority.
DRIFT_OPS: frozenset[str] = frozenset({"update", "replace", "create", "delete"})


@dataclass(frozen=True)
class DriftedResource:
    urn: str
    op: str

    @property
    def resource_type(self) -> str:
        # A URN is urn:pulumi:<stack>::<project>::<type>::<name>
        return self.urn.split("::")[2] if self.urn.count("::") >= 3 else "unknown"


def scan(stack: str) -> list[DriftedResource]:
    # State implication: --refresh rewrites the checkpoint from live reads before planning.
    proc = subprocess.run(
        ["pulumi", "preview", "--refresh", "--json", "-s", stack],
        capture_output=True, text=True, check=False,
    )
    if not proc.stdout.strip():
        raise RuntimeError(f"preview produced no JSON; stderr was: {proc.stderr.strip()}")
    plan: dict = json.loads(proc.stdout)
    return [
        DriftedResource(urn=step["urn"], op=step["op"])
        for step in plan.get("steps", [])
        if step.get("op") in DRIFT_OPS
    ]


if __name__ == "__main__":
    drifted = scan(sys.argv[1])
    for item in drifted:
        print(f"{item.op:<8} {item.resource_type:<40} {item.urn}")
    raise SystemExit(1 if drifted else 0)

Two things make this script survivable in a pipeline. It checks that stdout actually contained JSON before parsing, because a credential failure produces an empty document and a json.JSONDecodeError that hides the real error message. And it groups by resource type, which is what turns "17 resources drifted" into "17 security group rules drifted" — a sentence that names a cause.

Run the check with its own credentials, scoped to read. A drift job that holds write permission is one bad flag away from being a deploy job, and the whole value of the check is that it cannot change anything.

Remediation Strategies

There are two honest responses to drift. Revert — accept your code as the source of truth and pulumi up to bring reality back. Adopt — the manual change was correct, so encode it in code and refresh so state matches. Never leave drift unresolved: an unreconciled diff makes the next unrelated deploy dangerous.

Reverting is the default and needs no special mechanism: state has already been refreshed, your program still describes the intended configuration, and pulumi up plans the update that closes the gap. The judgement call is whether the manual change was load-bearing. Reverting a security group rule someone added at 3 a.m. to unblock an incident will re-break whatever it was fixing, so the revert is a change like any other — it belongs in a pull request with the diff attached, not in a scheduled job that quietly reconciles overnight.

Adopting means editing the program to match reality and then proving the two agree. Do it in that order: change the code first, then run the refresh-and-preview and confirm it reports nothing. If you refresh first and edit afterwards you have no check on whether your edit actually matched what the cloud holds — you have only your reading of a diff.

There is a third response the framing above deliberately leaves out, because it is easy to abuse: exclude the field. When drift is generated continuously by a legitimate actor — an autoscaler moving desired_count, a controller writing an annotation, a provider normalising a policy document — no amount of reverting will help, and adopting is meaningless because the value changes again tomorrow. Tell the engine to stop claiming that field:

# infra/exclusions.py — stop fighting a field another system owns
# CLI: pulumi up
from __future__ import annotations

import pulumi
import pulumi_aws as aws

service = aws.ecs.Service(
    "api",
    cluster=ecs_cluster.arn,
    task_definition=task.arn,
    desired_count=3,
    opts=pulumi.ResourceOptions(
        # Provider note: the autoscaler owns desired_count. Without this the
        # nightly drift check reports a diff every single morning.
        ignore_changes=["desired_count"],
    ),
)

ignore_changes is a claim about ownership, so treat it as documentation: every entry deserves a comment naming the system that owns the field. An unexplained exclusion is indistinguishable from someone silencing an alert.

The deletion case needs its own handling. When a resource is gone from the cloud, refresh removes it from state and the next preview plans a create — which is usually correct, and occasionally catastrophic if the resource held data and the recreate silently produces an empty one. Check delete and create operations in the drift report by hand before applying. If the resource should stay deleted, remove it from the program rather than letting an apply recreate it; if state and reality have diverged in a way you want to reconcile without touching either the cloud or the code, pulumi state delete <urn> and pulumi import are the surgical tools, and both belong in a reviewed runbook rather than in an operator's shell history.

Verification

After remediating, a fresh refresh-and-preview must report no changes — the definition of reconciled.

# CLI: prove the stack is reconciled
pulumi preview --refresh --expect-no-changes -s prod && echo "no drift"

Run the same assertion for a Terraform-backed stack, where the exit code carries more information:

# CLI: 0 = reconciled, 2 = still drifted, 1 = the check itself failed
terraform -chdir=cdktf.out/stacks/prod plan -refresh-only -detailed-exitcode
echo "exit=$?"

One verification is worth doing once per environment rather than once per remediation: prove the check can fail. Introduce a deliberate, harmless divergence — add a tag to a resource through the console — and confirm the scheduled job reports it with the right resource name. A drift check nobody has ever seen fail is a check nobody has evidence works, and the read-permission gap described above produces exactly that: a permanently green job over a stack it cannot see.

Gotchas & Edge Cases

Refresh can mask deletions. If a resource was deleted out of band, refresh removes it from state, and the next apply recreates it — usually what you want, but confirm before applying in production.

Provider defaults look like drift. Some providers normalise values (tags, ordering), producing perpetual diffs. Pin or ignore those fields with resource options rather than fighting them every run.

Secrets in the diff. Drift output can print sensitive values; scrub logs or mark those outputs secret so drift reports stay safe to share.

Refresh contends for the state lock. A nightly drift job and an early-morning deploy will collide, and the loser fails with a lock error naming a run that has already finished. Schedule the check outside your deploy window, or give the job the same concurrency group as the deploy so they serialise instead of racing.

Read permission gaps produce a false green. A drift role missing Describe on one resource type reports that type as clean forever, because refresh cannot read it and leaves the stale value in place. Grant read on everything under management, and re-check the role whenever a stack adds a new service.

Refresh costs API calls on every resource. Large stacks checked hourly can throttle themselves and their neighbours; ThrottlingException: Rate exceeded on an unrelated deploy is a common symptom of an over-eager drift schedule. Reduce the cadence, split the stack, or stagger the schedule across stacks.

A pulumi refresh cannot recover a write-only attribute. Passwords and other values the cloud never returns are not readable, so refresh leaves whatever state holds. Drift in those fields is undetectable by definition, which is an argument for rotating them through an explicit process rather than hoping a check will notice.

Operational Notes

The mechanics above take an afternoon. Keeping the check useful for a year is a different problem, and it is mostly about noise.

Start with cadence. Daily is right for most production stacks: frequent enough that a report covers one day of changes, rare enough that the API cost and lock contention stay invisible. Environments with high blast radius or many hands justify hourly; ephemeral preview environments justify nothing at all, since they are recreated faster than they can drift. Match the schedule to how quickly out-of-band changes actually hurt you rather than to how often the pipeline can run.

Then defend the signal. A drift report that is usually non-empty is a report that gets filtered into an unread folder within a fortnight, at which point you have the cost of the check and none of the benefit. Every recurring, explained diff should be resolved permanently — adopted into code, excluded with a commented ignore_changes, or removed from management altogether — until the steady state is an empty report. That is the only state in which a non-empty report means something.

Route by owner, not by channel. A drift report that lands in a shared operations channel belongs to nobody; one that opens an issue against the team that owns the stack has a name attached. The URN in the report already encodes project and stack, so the mapping from a drifted resource to a team is a lookup table rather than a judgement.

Finally, keep the record. Each remediation should leave a trace of what drifted, when, who changed it and which response you chose, because the pattern across months is more valuable than any single report. Three security group edits in one quarter is not three incidents; it is a missing self-service path, and no amount of reverting will fix it. The audit-trail practices in tracking IaC change ownership turn that history into something you can act on.

FAQ

How often should I check for drift?

Daily is a good default for production; high-blast-radius environments benefit from hourly. Tie the cadence to how quickly out-of-band changes hurt you.

Does refresh change my infrastructure?

No — refresh only updates state to match reality. It is up/apply that changes cloud resources.

Can I detect drift without a schedule?

You can run it ad hoc, but scheduled checks are what make drift visible before it causes an incident.

Why does my drift check report the same tags every morning?

Something else owns those tags — usually a tagging policy, a backup tool, or a cost-allocation job that writes them after your deploy. Reverting them nightly is a losing fight; add the field to ignore_changes with a comment naming the system that writes it, and the report goes quiet for the right reason.

Should the drift job apply the fix automatically?

No. Auto-remediation turns a detection tool into an unreviewed deploy path, and the deletion case makes that genuinely dangerous — an out-of-band delete becomes an automatic recreate with no one watching. Keep the job read-only and let a human choose revert, adopt or exclude.

How do I check drift across many stacks at once?

Iterate the stack list and run the same read-only check per stack, collecting the per-stack exit codes into one summary rather than failing on the first. Stagger the runs so a hundred stacks do not refresh simultaneously against the same account and throttle each other.

Related

• Idempotency and Drift Detection in Python IaC — the design principles that keep applies safe and repeatable.
• Migrating IaC State Between Backends — handling state carefully during larger operations.
• Managing IaC State — the parent topic on state backends and reconciliation.