Using boto3 Inside Pulumi and CDKTF

Pulumi and CDKTF providers cover most AWS resources, but not every lookup or imperative step has a provider equivalent — and reaching for boto3 inside a deployment program is safe only if you keep it read-only or strictly idempotent. This page is part of the Cloud Provider SDKs in Python workflow within Python IaC Fundamentals & Strategy, and it shows exactly when to drop to the AWS SDK and how to do it without corrupting framework state.

The core risk is simple: Pulumi and CDKTF track every resource they create in their own state. A boto3 call that creates or mutates a resource is invisible to that state, so the framework will never plan, diff, or destroy it — you have created untracked drift. The safe pattern is to use boto3 for reads (data the provider has no data source for) and to fence off any genuine side effect behind an idempotency guard.

When to reach for boto3

Use a provider resource or data source first — always. Drop to boto3 only when:

• A data source is missing. You need an attribute the provider does not expose (for example, a quota, a regional service availability flag, or an account-level setting with no Terraform data source).
• A one-off lookup feeds configuration. You want the default VPC ID, the latest AMI matching a custom filter, or an existing KMS key ARN to wire into a resource argument.
• An imperative step has no declarative model. Starting a one-time export, triggering a Lambda for a bootstrap check, or reading a parameter that another team manages out-of-band.

If the task creates persistent infrastructure, write a provider resource instead. boto3 creation belongs in Pulumi dynamic providers, not inline in a stack.

The distinction that actually decides this is not read-versus-write but who owns the lifecycle. A provider resource has four operations — create, read, update, delete — and the framework calls them in response to a diff. A boto3 call has one operation and no diff. So the question to ask at the call site is: if this line disappeared from the program tomorrow, what should happen to the thing it touched? If the answer is "it should be torn down", you need a resource. If the answer is "nothing, it was only a lookup", boto3 is appropriate.

There is a third case that trips people up: a resource that exists but is managed by another team, another account, or another state file. Reading its attributes with boto3 is legitimate and read-only, but it introduces a hidden dependency your framework cannot see. If that resource is deleted upstream, your next deployment fails at evaluation with a KeyError or an empty list, not with a clear message about a missing dependency. Fail loudly on that path — raise a typed error naming the resource and the account — rather than letting an empty response propagate into a resource argument.

Prerequisites

• Python 3.9+ with boto3 >= 1.34 and botocore pinned in your lockfile.
• pulumi >= 3.0 with pulumi-aws, or cdktf >= 0.20 with the generated AWS provider.
• AWS credentials resolvable by the default chain — set up per Best practices for managing cloud credentials in Python.
• IAM permissions for read-only describe calls (e.g. ec2:DescribeVpcs, ec2:DescribeImages) on the principal running the deployment.

Both frameworks and boto3 resolve credentials through the same botocore chain, which is convenient and occasionally surprising. Pulumi's AWS provider is a Go binary with its own credential resolution, and CDKTF hands off to the Terraform AWS provider, also Go — but all three honour AWS_PROFILE, AWS_REGION, the shared config file, and container or instance metadata in broadly the same order. The place they diverge is stack configuration: setting aws:region in Pulumi config or region on AwsProvider in CDKTF configures the provider, not boto3. A boto3 client created without region_name falls back to the environment and may end up pointed at a different region than the resources around it, producing a lookup that succeeds and returns the wrong answer.

Pass the region explicitly, sourced from the same place the provider gets it:

# region.py — one source of truth for the region both layers use
# CLI: pulumi up
import pulumi
import boto3

_cfg = pulumi.Config("aws")
REGION: str = _cfg.require("region")
SESSION: boto3.Session = boto3.Session(region_name=REGION)
# Provider note: reading `aws:region` from Pulumi config keeps the SDK client and
# the provider aligned even when the stack is switched with `pulumi stack select`.

Implementation

1. Read-only lookups in a Pulumi program

A boto3 read runs at program evaluation time, before Pulumi builds its resource graph. The returned value is a plain Python value (not an Output), so you can pass it straight into resource arguments. Wrap the client in a typed helper so the call site stays declarative.

# CLI: pulumi up
from __future__ import annotations
from dataclasses import dataclass
import boto3
import pulumi
import pulumi_aws as aws

@dataclass(frozen=True)
class NetworkLookup:
    region: str

    def default_vpc_id(self) -> str:
        # State implication: this is a READ. boto3 sees nothing Pulumi created;
        # it only reads pre-existing account state, so it adds no untracked drift.
        client = boto3.client("ec2", region_name=self.region)
        resp = client.describe_vpcs(
            Filters=[{"Name": "isDefault", "Values": ["true"]}]
        )
        vpcs = resp.get("Vpcs", [])
        if not vpcs:
            raise RuntimeError(f"No default VPC in {self.region}")
        return vpcs[0]["VpcId"]

lookup = NetworkLookup(region="us-east-1")
sg = aws.ec2.SecurityGroup(
    "app-sg",
    vpc_id=lookup.default_vpc_id(),  # plain str, resolved synchronously
    description="App tier",
)

Provider note: Prefer the native data source when one exists — aws.ec2.get_vpc(default=True) is equivalent here and keeps the lookup inside Pulumi's own provider plumbing. Use boto3 only when no get_* function covers the attribute you need.

2. Read-only lookups in a CDKTF program

CDKTF evaluates Python at synth time. A boto3 read during synth produces a literal that gets baked into the synthesized Terraform JSON — fine for stable values, dangerous for volatile ones (a "latest AMI" lookup will silently change your plan on every synth). Cache or pin volatile reads.

# CLI: cdktf synth
from __future__ import annotations
import boto3
from constructs import Construct
from cdktf import TerraformStack
from cdktf_cdktf_provider_aws.provider import AwsProvider
from cdktf_cdktf_provider_aws.instance import Instance

def latest_ami(region: str, owner: str, name_pattern: str) -> str:
    # State implication: value is frozen into synthesized JSON at synth time.
    # Pin owner+pattern tightly so the resolved AMI is reproducible across CI runs.
    client = boto3.client("ec2", region_name=region)
    images = client.describe_images(
        Owners=[owner],
        Filters=[{"Name": "name", "Values": [name_pattern]}],
    )["Images"]
    newest = max(images, key=lambda i: i["CreationDate"])
    return newest["ImageId"]

class AppStack(TerraformStack):
    def __init__(self, scope: Construct, id_: str) -> None:
        super().__init__(scope, id_)
        AwsProvider(self, "aws", region="us-east-1")
        Instance(
            self, "app",
            ami=latest_ami("us-east-1", "099720109477", "ubuntu/*22.04*"),
            instance_type="t3.micro",
        )

Provider note: The native DataAwsAmi data source resolves the AMI at terraform apply time instead of synth time, which is usually safer for CI. Reach for boto3 only when the filter you need cannot be expressed as a data source.

3. Fencing an imperative side effect

If you genuinely must perform a mutation (a bootstrap that no provider models), guard it so re-runs converge. Check current state first; act only when needed. This is the same idempotency contract a provider gives you for free.

# CLI: python -m bootstrap
from __future__ import annotations
import boto3
from botocore.exceptions import ClientError

def ensure_account_ebs_encryption(region: str) -> bool:
    """Enable default EBS encryption only if not already on. Returns True if changed."""
    client = boto3.client("ec2", region_name=region)
    # Read before write — the guard that makes this safe to re-run.
    current = client.get_ebs_encryption_by_default()["EbsEncryptionByDefault"]
    if current:
        return False  # already converged, no side effect
    # State implication: this mutation is NOT tracked by Pulumi/CDKTF state.
    # Keep such calls out of the deployment program; run them as a separate step.
    client.enable_ebs_encryption_by_default()
    return True

4. Reading when the input is an unresolved Output

The examples above all take literal arguments. The moment the lookup depends on something Pulumi created — a VPC id that does not exist yet, a role ARN produced earlier in the same program — you cannot call boto3 at the top level, because the value is an Output and there is nothing to pass. The read has to move inside an apply, and that changes when it runs.

Inside an apply, the callback executes after the dependency resolves, which during a preview means it does not execute at all for resources that do not yet exist — the value is unknown and Pulumi skips the callback. During an update it runs once the upstream resource is created. So the same helper behaves differently in the two phases, and code that assumes it always returns a string will hit None during preview.

# apply_lookup.py — a boto3 read that depends on a resource Pulumi creates
# CLI: pulumi preview && pulumi up
from typing import Optional
import boto3
import pulumi
import pulumi_aws as aws

vpc = aws.ec2.Vpc("app", cidr_block="10.20.0.0/16")


def _default_route_table(vpc_id: str) -> Optional[str]:
    if pulumi.runtime.is_dry_run():
        # Provider note: during preview the VPC may not exist; skip the API call
        # rather than describing an id that AWS will reject.
        return None
    client = boto3.client("ec2", region_name="us-east-1")
    tables = client.describe_route_tables(
        Filters=[
            {"Name": "vpc-id", "Values": [vpc_id]},
            {"Name": "association.main", "Values": ["true"]},
        ]
    )["RouteTables"]
    return tables[0]["RouteTableId"] if tables else None


main_rt: pulumi.Output[Optional[str]] = vpc.id.apply(_default_route_table)
pulumi.export("mainRouteTableId", main_rt)
# State implication: the exported value is derived, not owned. Pulumi records it
# in state but will not detect drift if the route table is replaced out of band.

CDKTF has no equivalent escape hatch, and that asymmetry is worth stating plainly. CDKTF values are Terraform references — TerraformOutput, Token strings like ${aws_vpc.app.id} — that only acquire a value during terraform apply, in a separate process, long after Python has exited. There is no callback to run boto3 in. If a CDKTF stack needs a value derived from a resource it creates, the answer is a Terraform data source with an explicit depends_on, or a second stack that reads the first through TerraformRemoteState. Attempting the boto3 version produces a describe call filtered on the literal string ${aws_vpc.app.id}, which AWS rejects with An error occurred (InvalidParameterValue) when calling the DescribeRouteTables operation.

Verification

Assert that read helpers are pure reads by mocking boto3 with moto and confirming no create call fires. This mirrors the testing approach in the parent SDK overview.

# CLI: pytest tests/test_lookups.py -v
from moto import mock_aws
import boto3
from infra.lookups import NetworkLookup

@mock_aws
def test_default_vpc_lookup_is_read_only() -> None:
    boto3.client("ec2", region_name="us-east-1").create_default_vpc()
    vpc_id = NetworkLookup(region="us-east-1").default_vpc_id()
    assert vpc_id.startswith("vpc-")
    # No assertion on creation: the helper must never call create_*.

That test proves the helper returns the right value; it does not prove the helper is read-only. Enforce that mechanically with a botocore event hook, which sees every API operation before it leaves the process:

# tests/test_readonly.py — fail the suite if a helper issues a mutating call
# CLI: pytest tests/test_readonly.py -v
from typing import Any
import boto3
import pytest
from moto import mock_aws
from infra.lookups import NetworkLookup

SAFE_PREFIXES = ("Describe", "Get", "List", "Head")


@pytest.fixture
def readonly_session() -> boto3.Session:
    session = boto3.Session(region_name="us-east-1")

    def _reject(model: Any, **kwargs: Any) -> None:
        if not model.name.startswith(SAFE_PREFIXES):
            raise AssertionError(f"mutating call from a lookup helper: {model.name}")

    session.events.register("provide-client-params.*.*", _reject)
    return session


@mock_aws
def test_lookup_issues_no_mutating_calls(readonly_session: boto3.Session) -> None:
    readonly_session.client("ec2").create_default_vpc()   # setup, outside the guard
    assert NetworkLookup(region="us-east-1").default_vpc_id().startswith("vpc-")

The hook is registered on the session, so anything the helper constructs from that session inherits it. Wire the same guard into the deployment program behind an environment flag and a class of accident disappears: a helper that grows a create_tags call during a refactor now fails the build rather than quietly writing to an account.

For CDKTF, verification is different in kind — the thing to assert is that synthesis is reproducible. Synthesize twice and compare:

# CLI: prove a synth-time boto3 read did not make the output non-deterministic
cdktf synth --output out-a >/dev/null
cdktf synth --output out-b >/dev/null
diff -r out-a/stacks out-b/stacks && echo "synthesis is deterministic"

Gotchas & Edge Cases

boto3 reads run before credentials are validated by the provider. If the deployment principal lacks the describe permission, the program crashes at evaluation with AccessDenied before Pulumi or CDKTF prints a plan — confusing because no resource was touched. Grant read permissions explicitly.

Synth-time reads make CDKTF plans non-deterministic. A "latest" lookup changes the synthesized JSON between runs, so terraform plan shows spurious diffs and CI snapshot tests flap. Pin the value, cache it, or switch to a data source that resolves at apply time.

boto3 mutations create invisible drift. Anything you create via boto3 inside a stack is untracked: pulumi destroy and cdktf destroy will leave it behind, and pulumi refresh will never reconcile it. If you need lifecycle management, model it as a provider resource or a Pulumi dynamic provider instead.

A missing region fails before AWS is contacted. boto3.client("ec2") with no region and none in the environment raises botocore.exceptions.NoRegionError: You must specify a region. at client construction. It looks like a network problem and is not — the SDK never opened a socket. This is the single most common difference between a developer machine with ~/.aws/config and a CI runner without one.

Describe calls are throttled per account, not per process. A program that resolves an AMI for twenty instances issues twenty DescribeImages calls, and a pipeline running several stacks in parallel multiplies that. AWS answers with An error occurred (RequestLimitExceeded) when calling the DescribeImages operation: Request limit exceeded. Cache the result in the program and configure adaptive retries rather than adding sleeps.

Every preview pays the cost. pulumi preview evaluates the whole program, so all top-level boto3 reads run again — a plan you thought was free now makes API calls, and in a CI pipeline that previews on every pull request the volume adds up. It also means a lookup that fails intermittently breaks previews on unrelated changes.

Truncated responses look like empty ones. describe_images and friends return at most one page. Code that reads resp["Images"] and takes the newest is correct only if the result fit in one page; beyond that it silently picks the newest of an arbitrary subset. Use client.get_paginator("describe_images").paginate(...) for anything unbounded.

Operational Notes

The default botocore client is tuned for interactive use, not for a deployment that must either succeed or fail cleanly inside a pipeline's timeout. Its default connect and read timeouts are 60 seconds each with legacy retries, so a transient network fault inside a Pulumi program can stall an update for minutes before producing an error. Configure it explicitly.

# client.py — one hardened, cached client factory for the whole program
# CLI: pulumi up
from functools import lru_cache
from typing import Any
import boto3
from botocore.config import Config

_CONFIG = Config(
    retries={"max_attempts": 5, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=20,
    user_agent_extra="iac-lookup/1.0",
)


@lru_cache(maxsize=None)
def client(service: str, region: str) -> Any:
    # One client per (service, region); botocore clients are safe to share.
    # Provider note: user_agent_extra makes these calls distinguishable from the
    # provider's own traffic in CloudTrail, which matters when debugging throttling.
    return boto3.Session(region_name=region).client(service, config=_CONFIG)

Setting mode="adaptive" adds client-side rate limiting on top of the retry count, which is what you want when several stacks deploy concurrently against one account. The user_agent_extra string is a small thing with outsized value: in CloudTrail, provider traffic and your inline lookups become separable, so "who is causing the throttling" stops being guesswork.

Cross-account reads need an explicit assume-role rather than the ambient credentials. boto3.client("sts").assume_role(RoleArn=..., RoleSessionName=...) returns temporary credentials you feed into a new Session; those expire, so cache the session for less than its duration or a long deployment ends with An error occurred (ExpiredToken) when calling the DescribeVpcs operation. For anything more than an occasional lookup, prefer a second provider instance configured with the role and let the framework handle the exchange.

Finally, keep the read helpers in a module the deployment program imports rather than inline in __main__.py. That single structural choice is what makes the moto tests above possible, keeps the boto3 dependency out of the resource-declaration code, and gives you one place to audit when someone asks which API calls a deployment makes.

FAQ

Why call boto3 from inside an IaC program?

For read-only lookups and actions no provider models — fetching an AMI id, checking an account setting — while the provider still owns resource lifecycle.

Does boto3 state get tracked?

No — boto3 calls are imperative and invisible to state. Anything you must manage over time belongs in a resource or a dynamic provider, not a raw SDK call.

When does a boto3 call run?

During program evaluation, before or alongside resource registration, so guard it against unresolved outputs which are not yet known.

Can I use boto3 inside a Pulumi apply()?

Yes, and it is the correct place when the lookup depends on a resource Pulumi creates. Guard it with pulumi.runtime.is_dry_run() so preview does not describe an id that does not exist yet, and accept that the returned value is an Output rather than a plain string.

Why does the same lookup work locally but fail in CI?

Almost always credentials or region. A developer machine has ~/.aws/config with a default region and a profile; a CI runner typically has neither, so the client raises NoRegionError or NoCredentialsError before any request is sent. Set the region explicitly from stack configuration instead of relying on the environment.

Does boto3 respect the provider's assumed role?

No. The Pulumi and Terraform AWS providers perform their own AssumeRole based on provider configuration; boto3 resolves credentials independently through botocore. If the provider assumes a deployment role, a boto3 client in the same program is still using the base credentials unless you assume the role yourself.

Related

• Cloud Provider SDKs in Python — the parent overview of SDK client patterns, idempotency, and the SDK-to-framework boundary.
• Best practices for managing cloud credentials in Python — how the credentials these boto3 calls consume are resolved and rotated safely.
• Python IaC Fundamentals & Strategy — the grandparent overview connecting SDK usage to design principles and tooling choice.