Tracking IaC Change Ownership and Audit Trails

Nine months after a change ships, somebody asks why a security group allows port 5432 from a peered VPC. Answering that — who made the change, when, under what ticket, and with whose approval — is not one lookup but a join across three systems that were never designed to be joined. This guide, part of IaC cost and governance under Python IaC fundamentals and strategy, shows how to stamp provenance at deploy time from CI metadata and how to walk a cloud audit event back to the pipeline run that caused it.

Context

Three systems each hold part of the answer, and each is missing a different part.

Git knows intent. The commit message, the pull request discussion, and the reviewer approvals are the only place where why is recorded in prose. Git does not know whether the change was ever applied — a merged commit that failed at deploy and was never retried looks identical in history to one that shipped.

The infrastructure tool knows what changed. pulumi stack history lists every update with its result, duration, and the resource counts, and can be exported with the config values that were in effect. It does not know who approved the pull request, and if the update ran from a laptop it records a machine username rather than an identity your organisation recognises.

The cloud provider knows what actually happened to the API. CloudTrail records the exact call, the assumed role session, the source IP, and the request parameters. It has no idea that the call came from a deployment rather than from a person in the console, and its default retention is ninety days — shorter than the interval at which most of these questions get asked.

Three record sources, three blind spots Three record sources, three blind spots: comparison across Answers, Cannot answer. Source Answers Cannot answer Git history why, who reviewed whether it ever applied pulumi stack history what changed, result who approved it CloudTrail exact API call, when that it was a deployment
Each source is authoritative for one column and silent on the next.

The engineering task is to install enough shared identifiers that a query in any one of the three lands you in the other two.

Prerequisites

  • Python 3.9+ with pulumi>=3.0, or the CDKTF equivalent, and pydantic>=2.0 for validating CI metadata
  • A CI system that exposes commit SHA, actor, run ID, and branch as environment variables
  • Deployments running under a dedicated role — one role per environment, assumed with a session name you control
  • A CloudTrail trail delivering to S3, not just the ninety-day event history
# CLI: confirm a durable trail exists rather than relying on event history
aws cloudtrail describe-trails --query 'trailList[].[Name,S3BucketName,IsMultiRegionTrail]'
aws cloudtrail get-trail-status --name org-audit --query 'IsLogging'

Implementation

1. Derive a provenance tag set from CI environment variables

The tag set is the cheapest correlation key available: it rides along on the resource itself and is visible to anyone who opens the console. Build it in one typed place so every stack produces identical keys, and validate it so a missing variable fails loudly at synthesis rather than silently producing commit=None.

Provenance at deploy time Provenance at deploy time: CI variables then BuildContext then default_tags then resource CI variables SHA, actor, run BuildContext pydantic validated default_tags provider level resource carries the index
Validation happens before any resource exists, so a nameless deploy fails early.
# provenance.py — one typed source of truth for change-provenance tags
# CLI: pulumi up --stack prod
from __future__ import annotations

import os
from datetime import datetime, timezone

from pydantic import BaseModel, Field, field_validator


class BuildContext(BaseModel):
    """Populated from CI environment variables; validated before any resource exists."""

    commit_sha: str = Field(min_length=7, max_length=40)
    actor: str = Field(min_length=1)
    run_id: str = Field(min_length=1)
    repository: str = Field(min_length=1)
    ref: str = Field(min_length=1)

    @field_validator("commit_sha")
    @classmethod
    def _hex_only(cls, value: str) -> str:
        int(value, 16)  # raises ValueError on a placeholder like "unknown"
        return value[:12]

    @classmethod
    def from_environment(cls) -> "BuildContext":
        return cls(
            commit_sha=os.environ["GITHUB_SHA"],
            actor=os.environ["GITHUB_ACTOR"],
            run_id=os.environ["GITHUB_RUN_ID"],
            repository=os.environ["GITHUB_REPOSITORY"],
            ref=os.environ["GITHUB_REF_NAME"],
        )

    def as_tags(self, environment: str) -> dict[str, str]:
        return {
            "provenance:commit": self.commit_sha,
            "provenance:actor": self.actor,
            "provenance:run": self.run_id,
            "provenance:repo": self.repository,
            "provenance:ref": self.ref,
            "provenance:applied-at": datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
            "environment": environment,
        }

Two details matter. KeyError on a missing variable is the desired behaviour — a deployment that cannot say who triggered it should not proceed. And applied-at is stamped at synthesis, not at resource creation, so it is the attempt timestamp; the authoritative creation time lives in CloudTrail.

Apply the tag set through the provider's default-tags mechanism rather than resource by resource, so a new resource cannot be added without provenance:

# __main__.py — attach provenance once, at the provider
# CLI: pulumi up --stack prod --yes
from __future__ import annotations

import pulumi
import pulumi_aws as aws

from provenance import BuildContext

context = BuildContext.from_environment()
stack: str = pulumi.get_stack()

provider = aws.Provider(
    "tagged",
    region="eu-west-1",
    default_tags=aws.ProviderDefaultTagsArgs(tags=context.as_tags(environment=stack)),
)
# Provider note: default_tags are merged into every resource created through this
# provider. Resource-level tags win on key collision.

bucket = aws.s3.BucketV2(
    "artifacts",
    opts=pulumi.ResourceOptions(provider=provider),
)
# State implication: because applied-at changes on every run, each deployment shows
# a tag diff on every resource. Pin it to the commit timestamp instead if that noise
# outweighs the value of knowing when the last apply touched the resource.

pulumi.export("provenanceCommit", context.commit_sha)

That state implication is not a footnote — it is the single most common reason teams rip provenance tagging back out. A timestamp that moves every run turns pulumi preview into a wall of updates and trains people to skim diffs. Either derive applied-at from the commit's author date, which is stable for a given commit, or drop it and rely on CloudTrail for timing.

2. Understand why tags are a hint, not an audit trail

A tag is mutable, unversioned, and reflects only the last write. Anyone with ec2:CreateTags can rewrite provenance:actor to any string, and the previous value is gone. There is no history, no signature, and nothing preventing a hand-edit in the console.

Why a tag is not an audit trail Why a tag is not an audit trail: provenance tag with 4 facets. provenance tag Mutable any CreateTags call rewrites it Last write only no previous value survives Uneven coverage not every type is taggable Gone on delete deleted resources hold none
Treat the tag as a fast index into the real record, never as the record itself.

The correct mental model: tags are an index, cheap to read and good enough to point you at the right commit in seconds. The audit trail is CloudTrail, which is append-only, delivered to a bucket you can lock with Object Lock, and integrity-verifiable with digest files. When the two disagree, CloudTrail is right.

The other tag limitation is coverage. Not every resource type supports tags, tag keys and values have length limits (128 and 256 characters on AWS), and a resource deleted last March has no tags at all because it has no existence. Deleted resources are exactly the ones incident reviews ask about.

3. Walk a CloudTrail event back to a pipeline run

The join key is the role session name. When your pipeline assumes its deployment role, set the session name to something derived from the run, and it appears in userIdentity.arn on every event that session produced.

# CLI: assume the deploy role with a session name that encodes the pipeline run
aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/deploy-prod \
  --role-session-name "gha-${GITHUB_RUN_ID}-${GITHUB_ACTOR}" \
  --duration-seconds 3600

Session names are limited to 64 characters and the set [\w+=,.@-], so sanitise the actor before interpolating it. Now a lookup by resource identifier returns events whose ARN contains the run ID:

# trace.py — resolve a resource back to the pipeline run that created it
# CLI: python trace.py sg-0a1b2c3d4e5f67890 --days 400
from __future__ import annotations

import json
import re
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

import boto3

SESSION_RE = re.compile(r"assumed-role/(?P<role>[^/]+)/(?P<session>.+)$")


@dataclass(frozen=True)
class ChangeEvent:
    event_name: str
    event_time: datetime
    role: str
    session: str
    run_id: str | None
    source_ip: str


def _parse_identity(arn: str) -> tuple[str, str]:
    match = SESSION_RE.search(arn)
    return (match["role"], match["session"]) if match else ("unknown", arn)


def trace(resource_id: str, days: int = 90) -> list[ChangeEvent]:
    trail = boto3.client("cloudtrail")
    end = datetime.now(tz=timezone.utc)
    pages = trail.get_paginator("lookup_events").paginate(
        LookupAttributes=[{"AttributeKey": "ResourceName", "AttributeValue": resource_id}],
        StartTime=end - timedelta(days=days),
        EndTime=end,
    )
    # Provider note: lookup_events only reads the 90-day management-event history,
    # regardless of the days argument. Anything older must be queried from the
    # trail's S3 destination with Athena.
    events: list[ChangeEvent] = []
    for page in pages:
        for record in page["Events"]:
            detail = json.loads(record["CloudTrailEvent"])
            role, session = _parse_identity(detail["userIdentity"].get("arn", ""))
            run = session.split("-")[1] if session.startswith("gha-") else None
            events.append(ChangeEvent(
                event_name=record["EventName"],
                event_time=record["EventTime"],
                role=role,
                session=session,
                run_id=run,
                source_ip=detail.get("sourceIPAddress", ""),
            ))
    return sorted(events, key=lambda e: e.event_time)

With the run ID in hand, the pipeline run gives you the commit, the commit gives you the pull request, and the pull request gives you the discussion and the approvals. That is the full chain, and every hop is a deterministic lookup rather than a guess.

Walking an event back to a review Walking an event back to a review: Trail event → Role session → Pipeline run → Pull request. Trail event Role session Pipeline run Pull request userIdentity.arn run id in name commit sha approvals reconcile
Every hop is a deterministic lookup rather than an inference from timestamps.

Verification

Deploy a throwaway resource and prove all three records agree on it.

# CLI: 1 — the tool's own history shows the update and its result
pulumi stack history --stack prod --json | jq '.[0] | {version, result, startTime, environment}'

# CLI: 2 — the tags landed on the resource
aws ec2 describe-tags --filters Name=resource-id,Values=sg-0a1b2c3d4e5f67890 \
  --query 'Tags[?starts_with(Key, `provenance`)]'

# CLI: 3 — CloudTrail attributes the call to the pipeline's session
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0a1b2c3d4e5f67890 \
  --query 'Events[0].[EventName,Username,EventTime]'

The commit SHA in step 2 must match the commit that the run in step 3's session name points at. If it does not, something applied outside the pipeline — which is itself the finding.

Gotchas & Edge Cases

A failed update still leaves CloudTrail events. Pulumi may have created three resources before failing on the fourth; pulumi stack history records result: "failed" while CloudTrail shows three successful creates. Reconcile against the cloud, not against the history.

Local applies bypass the whole scheme. An engineer running pulumi up from a laptop produces no run ID and a session name of their own choosing. Deny the deploy role to anything but the pipeline's OIDC principal, and treat a human-named session in production as an incident.

lookup_events covers management events only. Data-plane calls — s3:PutObject, kms:Decrypt — need data event logging switched on explicitly and are not returned by this API at all.

Tag keys with a colon are conventional, not reserved. provenance:commit is fine on AWS, but aws: is reserved and rejected with InvalidParameterValue: Tag keys starting with 'aws:' are reserved. Some services also lowercase or reject certain characters — verify per service before standardising.

pulumi stack history is bounded by the backend. Self-managed S3 backends keep every checkpoint; the managed service paginates and older entries may need explicit export to retain. Decide retention before you need it.

Operational Notes

Retention is the decision that determines whether any of this works when it matters. Ninety days of CloudTrail event history costs nothing and answers nothing, because the questions arrive at the six-to-eighteen-month mark. A trail delivering compressed JSON to S3 with a lifecycle rule to Glacier Instant Retrieval after ninety days is inexpensive, and an Athena table partitioned by region and date makes multi-year queries practical.

Queryability matters as much as retention. Unpartitioned CloudTrail data in Athena scans the entire prefix for a single lookup, which is slow and billed per byte scanned. Create the table with PARTITIONED BY (region string, dt string) and use partition projection so no partition-repair job is needed.

Keep the git side durable too. Squash-merging discards the individual commits but preserves the pull request number in the message — make that number part of the required commit format, because it is the only reliable link back to review discussion. Protecting the deployment branch and requiring review are what make the git record trustworthy in the first place; the tagging described above just makes it findable. If you also generate infrastructure programmatically, apply the same rule to the generator, using the Automation API deployment patterns to pass build context through explicitly rather than reading it from ambient environment state.

Retention tiers for the audit record Retention tiers for the audit record: layered from Hot: 90 days down to Permanent. Hot: 90 days lookup_events, seconds to query Warm: 13 months S3 standard, Athena partitions Cold: 2 years+ Glacier Instant Retrieval Permanent git history and run metadata
Questions arrive well past the ninety-day window the default retention covers.

FAQ

How far back should I keep CloudTrail data?

Thirteen months covers year-over-year comparisons and most compliance frameworks; two years is common where auditors are involved. Keep the last ninety days in cheap standard storage for fast queries and archive the rest.

Can I put the pull request number in a tag?

Yes, and it is more useful than the commit SHA for humans, because it links straight to the review discussion. Keep both: the SHA is exact, the pull request number is readable.

What identifies a change when several run in parallel?

The pipeline run ID, not the timestamp. Two stacks deploying within the same second are indistinguishable by time but always have distinct run IDs, which is the whole reason the run ID belongs in the session name.

Does this work with CDKTF?

The mechanism is identical — build the tag map in typed Python and pass it to the AWS provider's default_tags block. The difference is that Terraform state records the plan and apply, so you read change history from state versions rather than from pulumi stack history.

Should provenance tags be enforced or advisory?

Enforced. A policy check that fails a deployment missing provenance:commit costs one pipeline failure to fix and permanently removes the "we do not know who did this" answer, in the same way least-privilege rules are enforced in IAM least privilege for Python IaC.

Why not just use the cloud provider's config history?

Configuration history services record resource state over time but not the human intent behind a change, and they cost meaningfully more per recorded item. They complement this chain; they do not replace the git-to-run-to-event join.