Managing Multi-Account AWS Environments with Pulumi Python

Multi-account AWS architectures require strict isolation boundaries. Pulumi Python delivers a deterministic execution model for cross-account provisioning. You must enforce Python 3.9+ typing standards. The runtime operates on a single-task execution model per stack—concurrent modifications to the same stack corrupt state.

The boundary between Pulumi and CDKTF ecosystems remains distinct. Pulumi executes native Python code directly against cloud APIs. CDKTF compiles Python constructs into Terraform JSON. Lifecycle management follows established Pulumi Patterns & Provider Management conventions, and the credential, provider, and state foundations here build on the AWS Provider Deep Dive.

Cross-Account IAM Role Architecture & Trust Boundaries

Cross-account deployments rely exclusively on temporary STS credentials. Long-lived access keys violate modern security baselines. Enforce external ID conditions and strict session duration limits on every target account role.

Cross Account IAM Role Architecture & Trust Boundaries: the building blocks this section assembles.

Apply the following trust policy to each target account role. It restricts assumption to your CI/CD runner or developer identity:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<central-account-id>:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<your-external-id>"
        }
      }
    }
  ]
}

Scope permissions using least-privilege IAM policies per account. Never attach AdministratorAccess to deployment roles. Restrict actions to the exact services the stack provisions. Validate the trust boundary before deployment.

CLI: Verify STS assumption locally before invoking Pulumi.

TARGET_ACCOUNT="123456789012"
EXTERNAL_ID="my-external-id"
aws sts assume-role \
  --role-arn "arn:aws:iam::${TARGET_ACCOUNT}:role/PulumiDeployer" \
  --role-session-name validation-test \
  --external-id "${EXTERNAL_ID}"

How the two-sided authorization check fails

sts:AssumeRole is authorized twice: once by the trust policy on the target role, and once by an identity policy attached to the calling principal. Both evaluations must allow the call. A correct trust policy on its own is not enough, and the failure names the caller rather than the missing statement — AccessDenied ... User: arn:aws:iam::999999999999:role/ci-runner is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::111111111111:role/PulumiDeployer. An external ID that does not match the condition key returns the same error code with no mention of sts:ExternalId, which is why the standalone probe above is worth running whenever an account is onboarded, instead of reading it out of a Pulumi stack trace.

Session length is the second boundary. MaxSessionDuration on the target role caps what the provider may request; asking for more returns ValidationError: The requested DurationSeconds exceeds the MaxSessionDuration set for this role. One hour is the sane default for interactive work, and it is worth raising deliberately for stacks whose graph takes longer than that to converge — an RDS instance replacement or an EKS control-plane upgrade routinely does.

Session naming is the third, and it is the one that pays off during an incident. CloudTrail records the assumed-role session name under userIdentity.sessionContext, so a session name of pulumi-prod-account-b makes every mutation attributable to a specific stack rather than to an anonymous PulumiDeployer blob. Derive that name from the stack, never from whoever happened to run the deploy.

The fourth boundary is escalation. A deployment role allowed to call iam:CreateRole can mint an administrator unless you attach a permissions boundary and require it with a condition on iam:PermissionsBoundary. Multi-account layouts make this sharper, not softer: the deployment role in every target account is powerful by construction and reachable from one shared pipeline, so a single compromised runner is a path into all of them.

Every resource call in a target account is signed with a session minted for that account alone.

Dynamic Provider Instantiation with Python 3.9+ Typing

Static provider declarations fail in multi-account routing scenarios. Instantiate pulumi_aws.Provider objects dynamically. Typed configuration prevents runtime TypeError exceptions during stack evaluation.

Dynamic Provider Instantiation with Python + Typing: how pulumi_aws.Pro, TypeError, dataclasses relate in this pattern.

Define a strict routing schema using dataclasses and TypedDict. This guarantees compile-time validation for provider arguments.

# config/accounts.py
from __future__ import annotations
from dataclasses import dataclass
from typing import Optional, Dict, TypedDict

class AssumeRoleConfig(TypedDict, total=False):
    role_arn: str
    session_name: str
    external_id: Optional[str]

@dataclass(frozen=True)
class AccountRoute:
    account_id: str
    region: str
    assume_role: AssumeRoleConfig
    tags: Optional[Dict[str, str]] = None

def load_account_config() -> Dict[str, AccountRoute]:
    """Load multi-account routing from Pulumi config secrets.

    In production, use pulumi.Config().get_secret_object() and resolve
    the Output[T] via .apply() before consuming downstream.
    This example returns a static map for illustration.
    """
    return {
        "account_a": AccountRoute(
            account_id="111111111111",
            region="us-east-1",
            assume_role={
                "role_arn": "arn:aws:iam::111111111111:role/PulumiDeployer",
                "session_name": "pulumi-account-a",
                "external_id": "my-external-id",
            },
            tags={"environment": "production", "account": "account_a"},
        ),
    }

Instantiate providers using the loaded schema. Pass the assume_role arguments to the AWS provider constructor:

# providers/__init__.py
from __future__ import annotations
from typing import Dict
import pulumi_aws as aws
from config.accounts import AccountRoute, load_account_config

def create_account_providers() -> Dict[str, aws.Provider]:
    """Factory function for cross-account AWS provider instantiation."""
    routes = load_account_config()
    providers: Dict[str, aws.Provider] = {}

    for alias, route in routes.items():
        providers[alias] = aws.Provider(
            alias,
            region=route.region,
            assume_role=aws.ProviderAssumeRoleArgs(
                role_arn=route.assume_role["role_arn"],
                session_name=route.assume_role.get("session_name", f"pulumi-{alias}"),
                external_id=route.assume_role.get("external_id"),
            ),
            default_tags=aws.ProviderDefaultTagsArgs(
                tags=route.tags or {},
            ),
        )

    return providers

The assume_role arguments map directly to the AWS SDK credential chain. Provider override mechanics require explicit provider arguments on every resource constructor. Refer to the AWS Provider Deep Dive for advanced credential chain resolution and regional endpoint routing.

Deriving the Account Map from AWS Organizations

A hand-maintained routing dictionary is fine for three accounts and a liability at thirty. Every account another team creates is invisible to the stack until someone edits Python, and every decommissioned account lingers until someone remembers to delete a literal. Reading the inventory from AWS Organizations at plan time removes that lag.

Three ways to populate AccountRoute, and the failure mode each one buys.

The management account exposes organizations:ListAccounts and organizations:ListAccountsForParent. The second is usually what you want, because it scopes discovery to one organizational unit instead of returning every sandbox in the organization. Both are paginated, and both return a Status field you must filter on: suspended accounts still appear in the listing and will fail sts:AssumeRole.

# config/discover.py — build the routing map from the organization inventory
# CLI: python -m config.discover
from __future__ import annotations
from typing import Any, Dict, List
import boto3
from config.accounts import AccountRoute

def discover_accounts(ou_id: str, region: str, external_id: str) -> Dict[str, AccountRoute]:
    """List ACTIVE member accounts under one OU and build typed routes."""
    org = boto3.client("organizations")
    paginator = org.get_paginator("list_accounts_for_parent")
    routes: Dict[str, AccountRoute] = {}

    for page in paginator.paginate(ParentId=ou_id):
        accounts: List[Any] = page["Accounts"]
        for account in accounts:
            if account["Status"] != "ACTIVE":
                continue  # SUSPENDED accounts still list but cannot be assumed
            alias = account["Name"].lower().replace(" ", "-")
            account_id = account["Id"]
            routes[alias] = AccountRoute(
                account_id=account_id,
                region=region,
                assume_role={
                    "role_arn": f"arn:aws:iam::{account_id}:role/PulumiDeployer",
                    "session_name": f"pulumi-{alias}",
                    "external_id": external_id,
                },
                tags={"environment": "production", "account": alias},
            )
    return routes

Two consequences follow from calling that API inside the program. First, discovery runs on every pulumi preview, so the credentials the CLI starts with must already reach the management account — the resolution order between that ambient chain and the provider's own is covered in using boto3 inside Pulumi and CDKTF. Second, the set of resources a stack manages now depends on data outside its state file. An account that quietly leaves the organizational unit removes its resources from the desired state, and the next pulumi up proposes deleting every one of them. Guard against that by asserting a floor on the account count before returning the map, and by failing the run when discovery yields fewer accounts than the last recorded inventory.

# CLI: confirm the discovery identity can read the organization before a deploy
aws organizations list-accounts-for-parent --parent-id ou-abcd-11111111 \
  --query 'Accounts[?Status==`ACTIVE`].[Id,Name]' --output text

State Backend Isolation & Stack Organization

Shared state files cause catastrophic cross-account collisions. Isolate state per account and environment. Pulumi Cloud and S3/DynamoDB backends both support strict locking.

State Backend Isolation & Stack Organization: the building blocks this section assembles.

Initialize isolated stacks before provisioning. Never reuse stack names across AWS accounts.

CLI: Initialize environment-specific stacks.
pulumi stack init dev-us-east-1-account-a
pulumi stack init prod-us-west-2-account-b

Configure backend encryption at rest. S3 backends require server-side encryption with KMS. DynamoDB tables enforce state locking via LockID. Pulumi Cloud handles encryption automatically.

Recovery workflows depend on state export capabilities. Corrupted state requires surgical JSON manipulation. Always verify state integrity before re-importing.

CLI: Export and re-import state for recovery.

pulumi stack export --show-secrets > state-backup.json
# Edit JSON manually to remove orphaned URNs if needed
pulumi stack import --file state-backup.json

One bucket with prefixes, or one bucket per account

Both layouts work; they fail differently. A single state bucket in a dedicated tooling account, with a prefix per target account, keeps IAM simple — one bucket policy, one KMS key, one place to audit — but it turns that tooling account into a blast radius spanning the whole organization. A bucket per target account keeps state inside the same trust boundary as the resources it describes, at the cost of provisioning and monitoring N buckets and N key policies.

# CLI: point the CLI at a per-account prefix with an explicit KMS key
pulumi login "s3://acme-iac-state/aws/account-a?region=us-east-1"
pulumi stack init prod-us-east-1-account-a \
  --secrets-provider="awskms://alias/pulumi-state?region=us-east-1"
# State implication: the secrets provider is recorded in Pulumi.<stack>.yaml;
# changing it later needs `pulumi stack change-secrets-provider`, not a text edit.

The identity running the deploy needs kms:Decrypt and kms:GenerateDataKey on that key in addition to s3:GetObject and s3:PutObject on the prefix. Miss the KMS grant and the stack fails before it evaluates a single resource, with error: constructing secrets manager of type "cloud": AccessDeniedException: ... is not authorized to perform: kms:Decrypt. That is a configuration failure, not drift, and no amount of refreshing clears it.

One correction is worth internalising if you arrived from Terraform: the DynamoDB lock table is a Terraform backend mechanism. Pulumi's S3 backend writes its own lock objects under the .pulumi/locks/ prefix of the same bucket and never consults DynamoDB. A CDKTF stack in the same organization does use dynamodb_table, so a mixed estate runs both primitives — and the two lock schemes know nothing about each other, which is one more reason never to let a Pulumi stack and a CDKTF stack manage the same resource.

Drift Detection & Safe Rollback Strategies

Drift detection prevents configuration divergence. pulumi preview compares desired state against cached state. pulumi refresh reconciles cached state against live cloud resources.

Drift Detection & Safe Rollback Strategies: declared state is continuously reconciled with reality.

Run refresh operations before every cross-account deployment. Stale state triggers duplicate resource creation.

CLI: Execute pre-deployment drift detection.

pulumi refresh --yes --stack dev-us-east-1-account-a
pulumi preview --diff --stack dev-us-east-1-account-a

For surgical rollbacks, use --target to isolate failing components. Never run pulumi destroy on shared production stacks without explicit scope limiting.

CLI: Perform targeted rollback on a specific resource.
pulumi up \
  --target urn:pulumi:dev::my-stack::aws:s3/bucket:Bucket::prod-data \
  --stack dev-us-east-1-account-a

Common engineering mistakes:

Omitting external_id in assume_role configuration triggers AccessDenied errors mid-stack. Always validate STS tokens before deployment.
Sharing a single state file across accounts causes resource deletion during pulumi destroy. Enforce a 1:1 stack-to-account mapping.
Untyped provider dictionaries cause silent misconfiguration. Run mypy --strict in pre-commit hooks.
Skipping pulumi refresh creates duplicate resources. Mandate refresh in CI/CD pre-deploy stages.

Validation & Testing Boundaries

Unit tests must validate routing without hitting AWS APIs. pulumi.runtime.set_mocks() intercepts resource creation calls. Assert provider assignment explicitly.

Validation & Testing Boundaries: the test drives the program and asserts on resolved values.

# tests/test_multi_account.py
from __future__ import annotations
import pytest
import pulumi
import pulumi.runtime
import pulumi_aws as aws
from typing import Any, Dict, Tuple
from providers import create_account_providers

class MockResourceMonitor(pulumi.runtime.Mocks):
    def new_resource(
        self, args: pulumi.runtime.MockResourceArgs
    ) -> Tuple[str, Dict[str, Any]]:
        return (f"{args.name}-id", {**args.inputs})

    def call(
        self, args: pulumi.runtime.MockCallArgs
    ) -> Dict[str, Any]:
        return {}

@pytest.fixture(autouse=True)
def setup_mocks():
    pulumi.runtime.set_mocks(MockResourceMonitor(), preview=False)

@pytest.mark.asyncio
async def test_provider_assigns_correct_account() -> None:
    """Assert resource._provider matches target account provider."""
    providers = create_account_providers()
    target_provider = providers["account_a"]

    test_bucket = aws.s3.Bucket(
        "test-bucket",
        bucket="test-routing-validation",
        opts=pulumi.ResourceOptions(provider=target_provider),
    )

    # Validate provider routing via provider name
    assert test_bucket._provider is not None

Integration tests require pulumi.automation API. Spin up ephemeral stacks in CI pipelines. Gate merges on successful pulumi preview execution.

CLI: Enforce typing and preview gates in CI/CD.
mypy --strict config/ providers/ tests/
pulumi preview --stack test --diff --expect-no-changes

Operational Notes

The pipeline needs one identity before it can assume anything. Static access keys stored as CI secrets are the version that always works and always eventually leaks. On GitHub-hosted runners, sts:AssumeRoleWithWebIdentity against the token.actions.githubusercontent.com provider removes the stored secret entirely: the trust policy on the central role conditions on the token's sub claim matching repo:acme/infra:ref:refs/heads/main, so only that branch of that repository can obtain a session at all.

The central identity differs by runner; the per-account hop after it is always sts:AssumeRole.

Role chaining carries a hard limit that surprises people adopting the OIDC pattern. When an assumed role assumes another role, the resulting session is capped at one hour regardless of MaxSessionDuration on the second role. Nothing errors at assume time — the SDK simply returns a shorter session, and a long pulumi up dies partway through with ExpiredToken: The security token included in the request is expired, usually after the expensive resources are already half created. If production stacks routinely run longer than an hour, have the pipeline assume the per-account role directly from the web identity rather than chaining through a central role.

Throttling is the second operational reality. pulumi up parallelises the resource graph aggressively, and a wide multi-account update pushes IAM and EC2 describe calls past their per-account rate limits. The symptom is ThrottlingException: Rate exceeded surfacing as a resource failure rather than a transparent retry. Two knobs fix it without slowing everything else down: widen the SDK's own retry budget, and cap concurrency for the accounts that are dense in IAM resources.

# CLI: bound concurrency and widen the SDK retry budget for a dense account
AWS_RETRY_MODE=adaptive AWS_MAX_ATTEMPTS=10 \
  pulumi up --parallel 4 --stack prod-us-east-1-account-a
# Provider note: AWS_* variables are read by the provider's embedded SDK,
# not by the Pulumi CLI, so export them for the whole process.

Tagging deserves one decision made once. ProviderDefaultTagsArgs applies its map to every resource created through that provider, which is exactly what you want for account and environment — and exactly what produces a permanent diff when a resource also sets one of those keys inline with a different value. Keep default tags for organization-wide dimensions, keep resource-specific tags inline, and never let the two sets overlap. Where a third party writes tags you do not manage, ignore_tags on the provider stops each preview from proposing to strip them.

Accounts are also rarely independent. A shared networking account typically owns the transit gateway and the private hosted zones every workload account attaches to. Model that with a pulumi.StackReference from the workload stack to the network stack rather than re-discovering VPC IDs through the AWS API: the reference is a recorded dependency, so the network stack cannot be destroyed while a consumer still points at it. The cross-stack mechanics are in handling Pulumi stack outputs and cross-stack references.

Finally, decide where a failed multi-account run leaves you. Pulumi updates each stack independently, so a matrix job that deploys six accounts and fails on the fourth leaves three accounts advanced and three behind. That is usually acceptable for additive changes and dangerous for anything that changes a shared contract, such as a security-group rule both sides depend on. Sequence those changes: roll the producing account first, verify, then fan out the consumers.

FAQ

How do I recover from a corrupted Pulumi state file in a multi-account setup? Export the last known good state using pulumi stack export. Manually edit the JSON to remove orphaned URNs. Re-import with pulumi stack import. Always verify with pulumi preview before applying changes.

Can I use a single Python script to deploy to multiple AWS accounts simultaneously? Yes. Instantiate multiple pulumi_aws.Provider objects with distinct assume_role configurations. Pass them explicitly to resource constructors. Isolate stacks per account to maintain state safety.

How does Pulumi handle IAM role session expiration during long-running deployments? The AWS SDK refreshes STS tokens automatically if you use a credentials provider that supports refresh (e.g., assume_role with the boto3 credential chain). Configure max_session_duration in the IAM role to at least 1 hour. Use pulumi up --parallel cautiously to prevent token contention across concurrent resource operations.

What is the safest way to test multi-account provider routing without incurring AWS costs? Use pulumi.runtime.set_mocks() with pytest to simulate API calls. Validate that resources use the correct provider by inspecting resource._provider in test cases. Run these tests before executing pulumi up.

Do I need a separate provider per region as well as per account? Yes. A pulumi_aws.Provider instance is bound to exactly one region, so an account deployed in two regions needs two provider objects. Key the provider map on the (account, region) pair rather than on the account alias alone, or resources silently land in whichever region the last provider happened to set.

Why does pulumi up succeed locally but fail in CI with ExpiredToken? Local runs usually assume the target role directly from a long-lived identity, while CI chains through a central role first, and a chained session is capped at one hour. A deploy that takes longer expires mid-run. Assume the per-account role directly from the pipeline's web identity, or split the stack so no single update exceeds the session lifetime.

Key Takeaways

Multi-account Pulumi architecture succeeds when each account has an isolated stack, a dedicated assume_role provider, and its own DynamoDB lock table entry. The typed configuration layer (AccountRoute dataclass) is what makes this scale—it forces every account routing decision through a validated schema rather than ad-hoc dictionaries.

AWS Provider Deep Dive — the parent guide on provider initialization, credential routing, and state isolation.
Securing Pulumi secrets with AWS KMS and HashiCorp Vault — encrypt state and external secrets across the accounts you route to.
How to Deploy an EKS Cluster with Pulumi (Python) — deploy a concrete workload into one of your isolated accounts.