Pulumi Stack Architecture

Designing a robust Pulumi stack architecture requires shifting from monolithic scripts to modular, reusable Python components. This guide details production-ready directory layouts, environment isolation strategies, and provider lifecycle patterns, and it sits within the broader Pulumi Patterns & Provider Management approach. We cover stack configuration, state boundaries, and provider instance structuring for multi-cloud environments — including how to lay out one stack per environment and how to share data through stack outputs and cross-stack references.

Those two guides are the practical halves of this topic. The per-environment guide answers "how do I get dev, staging, and prod out of one codebase without forking it" — stack naming, Pulumi.<stack>.yaml layering, and the shared component package they all import. The outputs guide answers "how does the application stack learn the VPC ID the network stack created" — typed exports, StackReference wiring, and the secret-handling rules that keep a database password from leaking into a plaintext output. Read this page for the decisions; read those for the keystrokes.

Key implementation priorities: modular Python project structures, environment-specific stack configurations, cross-stack dependency management, and provider lifecycle optimization.

Problem Framing

Every Pulumi project starts as one stack and one __main__.py, and that is correct at the start. The architecture question arrives at a specific moment: when two changes to the same stack have different risk profiles. Rotating a container image tag is a thirty-second operation with a trivial rollback. Changing a VPC CIDR is a re-creation that takes the network down. When both live in the same stack, the fast change inherits the slow change's preview time, its lock contention, and its blast radius.

Splitting on resource count is the wrong instinct — a 300-resource stack that one team deploys twice a day is healthier than three 100-resource stacks with tangled references between them. Split on three axes instead. Cadence: does this change weekly or yearly? Ownership: does a different team approve it? Regulatory boundary: does it need a separate state backend, a separate encryption key, or a separate audit trail? A yes to any of those justifies a stack; a no to all three means it belongs where it is, possibly as a new module.

The cost of a split is real and permanent. Two stacks that share data need a StackReference, which introduces an ordering constraint (network must deploy before app), a failure mode (a missing output), and a coupling that no longer shows up in a single preview. You lose the ability to see the whole change at once. Make the split deliberately, and write down which stack owns which resources, because the boundary is far cheaper to draw than to move later.

The other half of the framing is the shape of the code inside a stack. A stack is a Python program, so all the usual structural questions apply — where does shared logic live, how do you avoid copy-paste between environments, how do you keep a change reviewable. Those answers connect to the general advice in how to structure Python IaC projects for scale; what follows is the Pulumi-specific application of them.

Prerequisites

• Pulumi CLI 3.x with the Python runtime, and a project created by pulumi new python
• A state backend you control — Pulumi Cloud, or a self-managed S3/Azure Blob/GCS bucket per environment
• pulumi-aws (or the equivalent provider SDK) pinned in requirements.txt, plus mypy and pytest for the validation steps below
• Credentials that can be scoped per environment: separate AWS accounts, Azure subscriptions, or GCP projects rather than one account with tags

# CLI: confirm the runtime, the backend, and which stacks already exist
pulumi version && pulumi whoami --verbose && pulumi stack ls --all

The last prerequisite is the one teams skip and regret. If dev and prod share a cloud account, no amount of stack discipline stops a mistyped stack name from deleting production resources, because the credentials do not distinguish between them. Environment isolation is a credential boundary first and a Pulumi boundary second. The multi-account patterns in managing multi-account AWS environments with Pulumi Python cover the assume-role wiring that makes this practical.

Modular Project Structure & Component Boundaries

Establish a scalable directory layout that separates core infrastructure from environment-specific configurations. Keep your __main__.py strictly as an orchestration entry point. Extract reusable resource definitions into dedicated Python packages.

Use pulumi.Config to handle runtime environment overrides. Implement factory patterns for resource instantiation to enforce type safety. Always keep sensitive configuration values out of version control by leveraging Pulumi secrets management. Backend encryption must be enforced at the state storage layer.

# infra/network.py — a typed factory, importable and testable on its own
# CLI: pulumi up --stack dev
import pulumi
import pulumi_aws as aws
from dataclasses import dataclass

@dataclass
class VpcConfig:
    cidr_block: str
    enable_dns: bool = True

def create_vpc(name: str, config: VpcConfig) -> aws.ec2.Vpc:
    return aws.ec2.Vpc(
        name,
        cidr_block=config.cidr_block,
        enable_dns_hostnames=config.enable_dns,
        # State implication: pulumi.get_stack() bakes the stack name into the tag,
        # so renaming a stack shows as an update on every tagged resource.
        tags={"Name": name, "Environment": pulumi.get_stack()},
    )

Deploy the stack using pulumi up --stack dev. Unit test the factory function by mocking pulumi.get_stack() and asserting resource properties. Integrate pytest with moto to simulate AWS API responses locally. This approach validates resource attributes without provisioning real infrastructure.

A layout that survives growth looks like this, and the important property is that nothing under infra/ imports from __main__.py:

# CLI: tree -L 2 --dirsfirst
.
├── Pulumi.yaml            # project name, runtime, description
├── Pulumi.dev.yaml        # per-stack config, checked in
├── Pulumi.staging.yaml
├── Pulumi.prod.yaml
├── __main__.py            # 40 lines: read config, call factories, export
├── infra/
│   ├── config.py          # dataclasses that mirror the YAML schema
│   ├── network.py         # VPC, subnets, routing
│   ├── data.py            # RDS, DynamoDB
│   └── compute.py         # ASGs, task definitions
└── tests/
    └── test_network.py

The dependency direction is the whole point. __main__.py may import anything under infra/; nothing under infra/ may import __main__.py or call pulumi.Config() directly. Modules that read configuration themselves cannot be unit tested without a Pulumi runtime, and they cannot be reused by a second stack that names its config keys differently. Push configuration reads to the edge and pass plain dataclasses inward.

Once a group of resources is used by more than one project — not more than one stack, more than one project — promote it from a module to a component resource. A ComponentResource gets its own URN namespace, so its children are grouped in the preview output and can be replaced as a unit. The reusable VPC component walkthrough shows the packaging step. Promoting too early is a common mistake: a component has a public interface you have to maintain, and a plain function does not.

Provider Instantiation & Lifecycle Optimization

Manage cloud provider instances efficiently across stacks to avoid redundant API calls and credential conflicts. Centralized provider configuration ensures consistent resource tagging and region targeting. Explicit provider instantiation prevents unpredictable routing during CI/CD execution.

Refer to the AWS Provider Deep Dive for region aliasing patterns. Consult the GCP Provider Configuration guide for service account delegation strategies. Never hardcode access keys. Rely on environment variables or OIDC federation for credential injection.

# infra/providers.py — explicit provider instances, one per region
# CLI: pulumi up --stack prod
import pulumi
import pulumi_aws as aws

us_east_1 = aws.Provider("us-east-1", region="us-east-1")
eu_west_1 = aws.Provider("eu-west-1", region="eu-west-1")

bucket_eu = aws.s3.Bucket(
    "app-assets",
    # Provider note: without this opts the bucket lands in the DEFAULT provider's
    # region, which comes from `pulumi config get aws:region`.
    opts=pulumi.ResourceOptions(provider=eu_west_1),
)

Configure base routing with pulumi config set aws:region us-east-1. Verify provider routing by checking the resource provider attribute in the preview output. Test provider aliasing by injecting mock provider instances into pytest fixtures.

The mechanism underneath is worth knowing because it explains the failure modes. Pulumi maintains a default provider per package, constructed lazily from the aws: namespace configuration on the stack. Any resource created without an explicit provider in its ResourceOptions binds to that default. An explicit aws.Provider is itself a resource recorded in state, with its own URN, and every resource bound to it records that URN in its state entry.

That last detail is the one that bites. Because the provider reference is stored per resource, changing which provider a resource uses is a state change, and for many resource types it forces a replacement — the provider is part of the resource's identity, not just a transport detail. Moving a bucket from the default provider to an explicit eu_west_1 provider in the same region will still show as a replace in the preview unless you pin the alias.

# infra/providers.py — inheriting a provider through a component's children
# CLI: pulumi preview --stack prod --diff
import pulumi
import pulumi_aws as aws

class RegionalStorage(pulumi.ComponentResource):
    def __init__(self, name: str, provider: aws.Provider,
                 opts: pulumi.ResourceOptions | None = None) -> None:
        super().__init__("acme:storage:Regional", name, None, opts)
        # Provider note: `providers=` on the child opts makes every descendant
        # inherit the instance without repeating it on each resource.
        child = pulumi.ResourceOptions(parent=self, providers=[provider])
        self.bucket = aws.s3.BucketV2(f"{name}-data", opts=child)
        self.register_outputs({"bucket": self.bucket.id})

Provider instances are cheap to create and expensive to churn. Build them once in a providers.py module, import them where needed, and never construct one inside a loop — each construction is a separate resource in state and a separate plugin handshake at preview time.

Stack Configuration & Environment Isolation

Define clear boundaries between development, staging, and production stacks using YAML configuration files and Python type hints. Isolation prevents accidental cross-environment mutations and simplifies CI/CD pipeline routing and audit compliance.

Leverage Pulumi.<stack>.yaml overrides for environment-specific parameters. Implement typed configuration classes with dataclasses to enforce schema validation. Apply stack-level resource naming conventions to guarantee traceability. Enable stack-level state locking to prevent concurrent deployment corruption.

Configure your backend to use isolated prefixes or dedicated storage buckets per environment. Restrict IAM policies to enforce least-privilege access for each stack. Validate configuration schemas during CI linting stages using mypy — this catches type mismatches before deployment execution begins. The state-level mechanics that underpin this isolation — backends, locking, and encryption shared by both Pulumi and Terraform — are covered in Managing IaC State for Python Projects, and the full per-environment layout has its own walkthrough in Structuring Pulumi Stacks per Environment.

The practical technique is to treat Pulumi.<stack>.yaml as untyped input and convert it to a dataclass exactly once, at the top of __main__.py. Everything downstream then works with a checked object, and mypy catches a renamed field before a deploy does.

# infra/config.py — one typed load, validated at import time
# CLI: pulumi config set --path network.cidrBlock 10.20.0.0/16 --stack staging
from dataclasses import dataclass
from typing import Any, Dict
import pulumi

@dataclass(frozen=True)
class NetworkSettings:
    cidr_block: str
    az_count: int
    single_nat_gateway: bool

def load() -> NetworkSettings:
    cfg = pulumi.Config()
    raw: Dict[str, Any] = cfg.require_object("network")
    settings = NetworkSettings(
        cidr_block=raw["cidrBlock"],
        az_count=int(raw.get("azCount", 2)),
        single_nat_gateway=bool(raw.get("singleNatGateway", False)),
    )
    if settings.az_count < 2 and pulumi.get_stack() == "prod":
        raise ValueError("prod requires azCount >= 2")
    return settings

# Pulumi.prod.yaml — only the differences from the program's defaults
# CLI: pulumi config set --path network.azCount 3 --stack prod
config:
  aws:region: eu-west-1
  web:network:
    cidrBlock: 10.20.0.0/16
    azCount: 3
    singleNatGateway: false

Raising a ValueError for an invalid combination is deliberate. The program fails during preview, before the engine registers a single resource, so the operator sees a clear message instead of a half-applied change. Structuring these files across environments — including which keys belong in Pulumi.yaml as project-wide defaults — is covered in structuring per-environment configuration in Pulumi, and the typed-settings pattern itself in using Pulumi config and typed settings in Python.

Secrets never appear in these files as plaintext. pulumi config set --secret db:password encrypts the value with the stack's encryption key and stores the ciphertext in the YAML, which is why Pulumi.prod.yaml is safe to commit. The broader handling — rotation, provider-managed keys, passing values to application code — belongs to Pulumi secrets and configuration.

State Management & Cross-Stack Dependencies

Architect state files to minimize blast radius while enabling secure data sharing between independent infrastructure domains. Proper state segmentation is critical for team-based workflows and compliance auditing. For detailed implementation patterns, review Handling Pulumi stack outputs and cross-stack references.

Segment state files by domain to isolate failure boundaries. Use StackReference for secure data passing between independent deployments. Implement automated state cleanup and drift detection pipelines. Encrypt sensitive outputs at rest and restrict backend access via strict IAM policies.

# __main__.py — consuming another stack's exported outputs
# CLI: pulumi up --stack app-prod
import pulumi
import pulumi_aws as aws

network_stack = pulumi.StackReference("org/network-prod")
# State implication: this records a dependency on the OTHER stack's last
# successful deployment; it reads state, it does not lock or modify it.
vpc_id = network_stack.get_output("vpc_id")

subnet = aws.ec2.Subnet(
    "app-subnet",
    vpc_id=vpc_id,
    cidr_block="10.0.1.0/24",
)

Retrieve outputs safely with pulumi stack output vpc_id --stack prod. Mock StackReference in tests to return dummy outputs and validate subnet creation logic. Use pytest parameterization to simulate missing or malformed cross-stack references to ensure graceful degradation during pipeline execution.

The reference name is fully qualified as <org>/<project>/<stack> when the two stacks belong to different projects, and the shorter <org>/<stack> form only works within one project. Getting this wrong produces a reference that resolves to an empty output set rather than an error, which is why the next snippet uses require_output — it raises instead of silently handing you None.

# infra/refs.py — fail loudly when a contract output is missing
# CLI: pulumi preview --stack app-prod
from typing import Any
import pulumi

def network_ref(env: str) -> pulumi.StackReference:
    return pulumi.StackReference(f"acme/network/{env}")

def required(ref: pulumi.StackReference, key: str) -> pulumi.Output[Any]:
    # State implication: require_output raises during preview if the producing
    # stack has never deployed, so ordering problems surface before any API call.
    return ref.require_output(key)

Treat the set of exported names as a published interface. Renaming an output is a breaking change for every consumer, and unlike a Python function signature nothing checks it at import time — the failure appears at preview in a different repository. Export a small, stable set (vpc_id, private_subnet_ids, cluster_endpoint) and add rather than rename. Where a value is sensitive, wrap it with pulumi.Output.secret() before exporting so it stays encrypted in the producing stack's state and arrives as a secret in the consumer.

Which backend holds that state is a separate decision with its own trade-offs, covered in choosing a state backend for Python IaC. What matters for architecture is that cross-stack references only work between stacks the credentials can read, so a hard security boundary between environments also blocks references across it — by design.

Step-by-Step: Laying Out a Three-Environment Project

The following sequence takes an empty directory to three deployed stacks that share one program. It assumes the credential separation from the prerequisites is already in place.

1. Create the project and the three stacks

# CLI: one project, three stacks, three independent state files
pulumi new python --name web --description "Web platform infrastructure"
pulumi stack init dev && pulumi stack init staging && pulumi stack init prod
pulumi stack ls

Each stack init creates an empty checkpoint and a Pulumi.<stack>.yaml. Nothing is deployed yet, and the three are already isolated — an update to dev cannot touch prod's state.

2. Move resources out of __main__.py into typed factories

Create infra/ as shown above and reduce __main__.py to orchestration. The entry point should read as a table of contents:

# __main__.py — orchestration only; no resource arguments inline
# CLI: pulumi preview --stack dev
import pulumi
from infra import config, network, compute

settings = config.load()
vpc = network.create_vpc("core", network.VpcConfig(cidr_block=settings.cidr_block))
subnets = network.private_subnets("core", vpc, az_count=settings.az_count)
service = compute.web_service("api", subnets=subnets)

pulumi.export("vpc_id", vpc.id)
pulumi.export("private_subnet_ids", [s.id for s in subnets])
pulumi.export("service_url", service.url)

3. Set the per-stack configuration

# CLI: same keys, different values, one command per environment
pulumi config set --path network.cidrBlock 10.10.0.0/16 --stack dev
pulumi config set --path network.azCount 2 --stack dev
pulumi config set --path network.cidrBlock 10.20.0.0/16 --stack prod
pulumi config set --path network.azCount 3 --stack prod

Non-overlapping CIDR ranges per environment are not cosmetic. Overlapping ranges make it impossible to peer dev and prod later, and re-addressing a live VPC is a rebuild.

4. Deploy in dependency order and promote

# CLI: deploy dev, inspect the diff, then promote the identical program
pulumi up --stack dev
pulumi preview --stack staging --diff
pulumi up --stack staging
pulumi preview --stack prod --diff --expect-no-changes || pulumi up --stack prod

The promotion is a stack switch, not a code change. If pulumi preview --stack prod shows a diff you did not expect after staging succeeded, the difference is in configuration, not in code — compare the two YAML files before touching the program. Pipelines that automate this loop, including approval gates between environments, are covered in the Pulumi Automation API.

Verification

Four checks, each catching a different class of layout mistake. Run the first two on every commit and all four before a production promotion.

# CLI: prove the layout, the config, and the exports all behave
pulumi preview --stack staging --diff
mypy infra/ __main__.py
pytest tests/ -q
pulumi stack output --json --stack staging
pulumi refresh --stack staging --expect-no-changes

pulumi preview --diff is the highest-value check because replacements are where stack architecture mistakes become expensive. A +- marker against a VPC or a database is almost always a provider binding or a naming change rather than an intended edit. Read the reason Pulumi prints next to the marker before approving.

pulumi refresh --expect-no-changes exits non-zero when the recorded state disagrees with the cloud, which means somebody changed a resource outside Pulumi. In a CI job that is a hard failure; investigating it is the subject of detecting and remediating state drift in Python IaC.

For the unit layer, drive the factories with pulumi.runtime.set_mocks so the tests never contact a cloud API, as described in unit testing Pulumi programs with mocks.

# tests/test_network.py — assert on resource inputs without any cloud calls
# CLI: pytest tests/test_network.py -q
from typing import Any, Dict, List, Optional, Tuple
import pulumi

class Mocks(pulumi.runtime.Mocks):
    def new_resource(self, args: pulumi.runtime.MockResourceArgs) -> Tuple[str, dict]:
        return f"{args.name}_id", dict(args.inputs)
    def call(self, args: pulumi.runtime.MockCallArgs) -> Dict[str, Any]:
        return {}

pulumi.runtime.set_mocks(Mocks(), preview=False)

from infra import network  # imported AFTER set_mocks

@pulumi.runtime.test
def test_vpc_carries_environment_tag() -> None:
    vpc = network.create_vpc("core", network.VpcConfig(cidr_block="10.10.0.0/16"))
    def check(tags: Optional[Dict[str, str]]) -> None:
        assert tags is not None and "Environment" in tags
    return vpc.tags.apply(check)

The import order is load-bearing: importing a module that creates resources before set_mocks runs raises Exception: Program run without the Pulumi engine available; re-run using the pulumi CLI.

Common Implementation Pitfalls

• Monolithic __main__.py files: Putting all resource definitions in a single entry point creates tight coupling and slows preview times. Extract reusable components into dedicated Python modules.
• Implicit provider defaults causing region mismatches: Relying on CLI-configured defaults leads to unpredictable deployments. Explicitly instantiate providers and pass them via ResourceOptions.
• Hardcoded stack outputs instead of StackReference: Manually copying IDs between stacks breaks automation. Use pulumi.StackReference to dynamically fetch outputs.
• Mixing state backends across environments: Storing all environments in the same bucket complicates access controls. Isolate state files using environment-specific prefixes or separate backend buckets.
• Renaming logical resource names to "clean things up": The logical name is part of the URN. Renaming app-bucket to assets-bucket deletes one resource and creates another. Use pulumi.ResourceOptions(aliases=[...]) when a rename is genuinely required.
• Branching the repository per environment: A prod branch guarantees the environments diverge. Keep one branch and let Pulumi.<stack>.yaml carry every difference.

Troubleshooting

error: no stack named 'staging' found — the stack exists in someone else's backend, or the CLI is logged into a different one. Cause: pulumi login points at a local file backend while the team uses Pulumi Cloud, or the organisation prefix is missing. Fix: run pulumi whoami --verbose to confirm the backend, then pulumi stack ls --all and use the fully qualified <org>/<project>/<stack> name.

error: Missing required configuration variable 'web:network' — the program calls require_object("network") but the stack's YAML has no such key. Cause: a new configuration field was added to infra/config.py and set on dev but never on staging and prod. Fix: pulumi config set --path network.cidrBlock <value> --stack staging, and add a CI step that runs pulumi preview against every stack so a missing key fails on the pull request rather than on the promotion.

error: aws:ec2/subnet:Subnet resource 'app-subnet' has a problem: Missing required property 'vpcId' — a StackReference returned nothing. Cause: the output name in the producing stack does not match the key requested, or the producing stack has never had a successful update. Fix: run pulumi stack output --json --stack acme/network/prod to see the real export names, then switch get_output to require_output so the next occurrence fails with the key name instead of a downstream property error.

error: [409] Conflict: Another update is currently in progress. — two pipelines are updating the same stack. Cause: a merge queue running two jobs against prod, or an earlier job that was killed without releasing the lock. Fix: confirm nothing is genuinely running, then pulumi cancel --stack prod. Prevent recurrence by serialising deployments per stack in the pipeline configuration rather than relying on the lock to arbitrate.

Calling __str__ on an Output[T] is not supported. — a resource ID was interpolated into an f-string. Cause: f"arn:aws:s3:::{bucket.id}" runs before the value resolves. Fix: use bucket.id.apply(lambda v: f"arn:aws:s3:::{v}") or pulumi.Output.concat("arn:aws:s3:::", bucket.id). This surfaces constantly at stack boundaries, because StackReference values are always outputs.

error: creating EC2 Subnet: InvalidVpcID.NotFound: The vpc ID 'vpc-0a1b2c3d' does not exist — a cross-region mismatch. Cause: the subnet bound to the default provider while the VPC came from an explicit regional provider, so the two calls went to different regions. Fix: pass the same provider to both through ResourceOptions(provider=...), or set providers=[...] on the parent component so every child inherits it.

Key Takeaways

Pulumi stack architecture pays dividends when it is boring: predictable directory layout, one stack per environment, typed configuration objects, and StackReference for cross-stack data. Teams that skip these foundations end up with monolithic stacks that take minutes to preview and fail in unpredictable ways during concurrent deployments. The decisions that are expensive to reverse — where the stack boundaries fall, what the exported output names are, and which credentials each environment uses — deserve more design time than the ones that are cheap to change.

FAQ

How do I structure a Pulumi project for multiple environments?

Use a single codebase with environment-specific Pulumi.<stack>.yaml files and typed Python configuration classes to inject environment values at runtime. The program itself contains no if stack == "prod" branches; every difference lives in configuration, which keeps the resource graph identical across environments and makes a promotion reviewable as a config diff.

When should I split infrastructure into separate stacks?

Split when resources have different deployment frequencies, distinct ownership teams, or require isolated state files for compliance and blast-radius reduction. Resource count alone is a poor signal — the real cost of a split is the StackReference coupling and the ordering constraint it introduces between deployments.

How does Python's typing system improve Pulumi stack reliability?

Type hints and dataclasses enable IDE autocomplete, static analysis with mypy, and early detection of misconfigured resource parameters before deployment. The highest-value place to apply them is the boundary where untyped YAML configuration becomes a program value, because that is where a renamed key otherwise fails at deploy time rather than at lint time.

What is the best practice for managing provider credentials in CI/CD?

Use OIDC federation or short-lived roles injected via environment variables, avoiding long-lived access keys entirely. Scope each environment to its own cloud account or subscription so the credentials themselves prevent a mistyped stack name from reaching production.

Can two stacks share the same state backend bucket?

Yes, and Pulumi keys them separately by project and stack name, but a shared bucket means one IAM policy governs both. If the point of the split was a security boundary, use separate buckets or separate accounts so the read path for prod state is not open to whoever can deploy dev.

How do I rename a stack without destroying its resources?

Export the checkpoint with pulumi stack export --stack old > state.json, create the new stack, and import it with pulumi stack import --stack new --file state.json, then delete the old stack without --force. The resources are untouched because URNs contain the project and logical names, not the stack name — but any resource tag that embeds pulumi.get_stack() will show as an update on the next deploy.

Related

• Structuring Pulumi Stacks per Environment — stack naming, Pulumi.<stack>.yaml config, and shared component code for dev, staging, and prod.
• Handling Pulumi Stack Outputs and Cross-Stack References in Python — export typed outputs and consume them safely with StackReference.
• Managing IaC State for Python Projects — the backend, locking, and encryption concepts that state isolation between stacks depends on.
• Pulumi Patterns & Provider Management — the parent set of provider and architecture patterns this design supports.