Deploying an AKS Cluster with Pulumi Python

AKS hides most of the control plane but leaves the decisions that matter — identity, networking mode, and node pool shape — to you. This guide, part of Azure provider configuration under Pulumi patterns and provider management, builds a managed cluster with a system-assigned identity and exports a usable kubeconfig.

Context

An AKS cluster is cheap to create and expensive to re-create: the networking mode and the system node pool's VM size are fixed at creation, and changing either replaces the AKS cluster. Deciding them deliberately — rather than accepting defaults and discovering the constraint during a scaling incident — is the whole job. The resulting kubeconfig is also the natural handoff point to the Kubernetes provider, which deploys workloads onto the managed cluster this guide creates.

AKS cluster shape AKS cluster shape: Managed cluster with 4 facets. Managed cluster Identity system-assigned Network azure or kubenet System pool fixed VM size User pool scalable
Networking mode and the system pool's VM size are set at creation; user pools stay flexible.

pulumi-azure-native is generated from the Azure Resource Manager API specifications rather than hand-written, so what you are really declaring is a Microsoft.ContainerService/managedClusters ARM resource with Python names. That has a direct consequence for how a diff behaves: the provider PUTs the whole resource body on every update, and ARM decides whether a given property change is an in-place update, a rolling operation on the node pools, or a replacement. Pulumi's preview reports ~ update or +- replace based on the schema's replaceOnChanges metadata, not on a live call to Azure, so the preview is a reliable warning but not a negotiation — once ARM receives a body it considers incompatible, it returns an error rather than doing something clever.

The most consequential structural decision is where node pools live. agent_pool_profiles is a property of ManagedCluster, and containerservice.AgentPool is a separate top-level resource that manages the same underlying object. Using both for the same pool sets up a fight: Pulumi sees the pool declared inline, ARM reports the pool created out of band by the AgentPool resource, and every pulumi up proposes to remove whichever one it did not create. The workable arrangement is to declare exactly one System pool inline — ARM will not create a managed cluster without one — and every additional pool as its own AgentPool resource, which can then be added, resized and destroyed without touching the AKS cluster body.

Address planning is the other decision that cannot be deferred. With network_plugin="azure", every pod takes a real address from the node's subnet, and each node pre-reserves max_pods + 1 addresses at join time whether or not any pods land on it.

Subnet addresses reserved per node Subnet addresses reserved per node: Azure CNI, max_pods=30, Azure CNI, max_pods=50, Azure CNI, max_pods=110, kubenet node. Azure CNI, max_pods=30 31 IPs Azure CNI, max_pods=50 51 IPs Azure CNI, max_pods=110 111 IPs kubenet node 1 IP
Azure CNI reserves max_pods + 1 addresses per node at join time, before any pod schedules.

That arithmetic decides the subnet size before the first node exists. A /24 offers 251 usable addresses after Azure's five reserved ones, which at the CNI default of 30 pods per node accommodates eight nodes and no more — a limit an autoscaler will discover at exactly the wrong moment. Size the subnet for the maximum node count you would ever let the autoscaler reach, because growing it later means a new subnet and a new node pool, not an edit.

Prerequisites

  • pulumi-azure-native>=2.0 and an authenticated principal with Contributor on the resource group
  • Quota for the chosen VM size in the target region — the default Standard_DS2_v2 is not always available
  • kubectl installed locally to verify the exported kubeconfig
# CLI: confirm the VM size and Kubernetes versions offered in the region
az vm list-sizes --location westeurope --query "[?name=='Standard_DS2_v2']" -o table
az aks get-versions --location westeurope --query 'values[].version' -o tsv | tail -5

Check the vCPU quota as well as the size availability. AKS bring-up consumes quota from two buckets — the regional total and the per-family total for the VM series — and exhausting either produces the same class of failure several minutes into an apply, after the resource group and the managed identity already exist:

# CLI: regional and per-family vCPU headroom in the target location
az vm list-usage --location westeurope \
  --query "[?contains(localName, 'Total Regional vCPUs') || contains(localName, 'DSv2')].[localName,currentValue,limit]" \
  -o table

The Pulumi side needs the subscription selected explicitly rather than inherited from whatever az account set last ran on the machine. Put it in stack configuration so the same program cannot deploy into a different subscription depending on who runs it, using the typed configuration approach from using Pulumi config and typed settings in Python:

# CLI: pin the target subscription and region to the stack, not the workstation
pulumi config set azure-native:subscriptionId 00000000-1111-2222-3333-444444444444
pulumi config set azure-native:location westeurope

Implementation

The build has three steps: create the resource group and the AKS cluster with its mandatory system pool, attach the user pools that will actually run workloads, and fetch a kubeconfig that is safe to store in state.

1. Declare the managed cluster and its system pool

Declare the AKS cluster with a system-assigned managed identity, so there is no service-principal secret attached to the managed cluster itself, and pin the Kubernetes version rather than floating on the region default.

Cluster bring-up Cluster bring-up: resource group then managed cluster then node pools then fetch kubeconfig then deploy workloads resource group managed cluster identity + network node pools system then user fetch kubeconfig marked secret deploy workloads
The kubeconfig is fetched after the AKS cluster exists and is exported as a secret output.
# aks.py — managed cluster with a system-assigned identity
# CLI: pulumi up
import pulumi
import pulumi_azure_native.containerservice as aks
import pulumi_azure_native.resources as resources

rg = resources.ResourceGroup("aks-rg", resource_group_name="platform-aks")

cluster = aks.ManagedCluster(
    "platform",
    resource_group_name=rg.name,
    resource_name_="platform-aks",           # Provider note: trailing underscore avoids
                                             # collision with Pulumi's own resource_name
    kubernetes_version="1.29.4",             # pinned, not the floating region default
    dns_prefix="platform",
    identity=aks.ManagedClusterIdentityArgs(type=aks.ResourceIdentityType.SYSTEM_ASSIGNED),
    network_profile=aks.ContainerServiceNetworkProfileArgs(
        network_plugin="azure",              # immutable: changing this replaces the AKS cluster
    ),
    agent_pool_profiles=[aks.ManagedClusterAgentPoolProfileArgs(
        name="system",
        mode="System",
        count=2,
        vm_size="Standard_DS2_v2",           # immutable on the system pool
        os_type="Linux",
    )],
)
# State implication: network_plugin and the system pool's vm_size are replace-triggering.
# A change to either destroys and rebuilds the managed cluster and everything running on it.

Two arguments are worth adding before this reaches an environment anyone depends on. sku=aks.ManagedClusterSKUArgs(name="Base", tier="Standard") buys the control plane an uptime SLA; the free tier has none and is a poor fit for anything with a paging rotation. And auto_upgrade_profile=aks.ManagedClusterAutoUpgradeProfileArgs(upgrade_channel="patch") lets Azure apply patch releases inside a maintenance window, which keeps the pinned minor version honest without making every CVE a manual deployment. Both are in-place updates, so they can be added to a running AKS cluster.

2. Attach user node pools as separate resources

Workload pools belong outside the ManagedCluster body so their lifecycle is independent of it. A pool declared this way can change VM size — by replacing the pool, not the AKS cluster — and can scale to zero when idle.

Choosing a node pool for a workload Choosing a node pool for a workload: choose among 4 options. Which pool runs this? core System pool:CoreDNS,metrics-server steady User pool, fixedcount bursty User pool,autoscaled to zero special Tainted pool: GPU orspot
Only one System pool is mandatory; everything else belongs in an AgentPool resource.
# pools.py — user pools that can be added, resized and destroyed independently
# CLI: pulumi up --target 'urn:*:AgentPool::*'
from typing import Sequence
import pulumi_azure_native.containerservice as aks

apps = aks.AgentPool(
    "apps",
    agent_pool_name="apps",
    resource_group_name=rg.name,
    resource_name_=cluster.name,             # Provider note: this is the AKS cluster name,
                                             # not the pool's own name
    mode="User",
    os_type="Linux",
    vm_size="Standard_D4s_v5",
    type="VirtualMachineScaleSets",
    enable_auto_scaling=True,
    min_count=0,                             # only a User pool may reach zero
    max_count=10,
    max_pods=30,                             # drives the subnet arithmetic above
    orchestrator_version="1.29.4",           # keep in step with the control plane
    node_labels={"workload": "apps"},
    opts=pulumi.ResourceOptions(
        parent=cluster,
        # State implication: the autoscaler owns `count` at runtime. Without this,
        # every preview after a scale event shows a spurious count diff and an
        # apply scales the pool back to whatever the code last recorded.
        ignore_changes=["count"],
    ),
)

gpu_taints: Sequence[str] = ["workload=gpu:NoSchedule"]

Give a specialised pool a taint at creation rather than after the fact. A taint added later is applied by ARM to the running nodes, but pods already scheduled there are not evicted, so the pool ends up hosting exactly the workloads the taint was meant to exclude until something restarts them.

3. Fetch a kubeconfig and mark it secret

The kubeconfig is fetched with a separate call that depends on the AKS cluster, and must be marked secret — it is a live cluster credential.

User versus admin credentials User versus admin credentials: comparison across Auth type, Revocable, Use for. Call Auth type Revocable Use for User credentials kubelogin exec plugin Yes, via Entra ID Pipelines Admin credentials Client certificate Only by rotation Break-glass Local accounts off User credentials only Yes Hardened stacks
The user credential points at an identity; the admin credential is a standing key.
# kubeconfig.py — fetch and protect the managed cluster credential
import base64
import pulumi
import pulumi_azure_native.containerservice as aks

creds = aks.list_managed_cluster_user_credentials_output(
    resource_group_name=rg.name, resource_name=cluster.name)

kubeconfig = creds.kubeconfigs[0].value.apply(
    lambda v: base64.b64decode(v).decode())

pulumi.export("kubeconfig", pulumi.Output.secret(kubeconfig))
# State implication: without Output.secret the kubeconfig is stored in plaintext
# in the state file and printed by `pulumi stack output`.

list_managed_cluster_user_credentials_output is the right call for a pipeline. When Azure AD integration is enabled it returns a kubeconfig whose users entry is an exec plugin invoking kubelogin, so the credential in state is a pointer to an identity rather than a standing key — revoking the identity revokes the access. The sibling function list_managed_cluster_admin_credentials_output returns a client-certificate kubeconfig that bypasses Azure AD entirely and cannot be revoked short of rotating the AKS cluster's certificates; it also fails outright with Getting static credential is not allowed on any managed cluster created with disable_local_accounts=True.

Passing that kubeconfig into a Kubernetes provider inside the same program is the usual next move, and it must be done through the Output, never through a resolved string:

# k8s_provider.py — hand the credential to the Kubernetes provider without leaking it
# CLI: pulumi up
import pulumi_kubernetes as k8s

k8s_provider = k8s.Provider(
    "aks-k8s",
    kubeconfig=kubeconfig,                  # still an Output — resolution stays inside the engine
    opts=pulumi.ResourceOptions(depends_on=[apps]),
)
# Provider note: depends_on the user pool, not just the AKS cluster. The control plane
# reports Succeeded before the pool has schedulable nodes, so a Deployment created
# too early sits Pending and the rollout wait times out.

Verification

Check that the managed cluster is running the pinned version and that the exported credential actually works.

# CLI: nodes ready on the expected Kubernetes version
pulumi stack output kubeconfig --show-secrets > /tmp/kubeconfig
KUBECONFIG=/tmp/kubeconfig kubectl get nodes -o wide
KUBECONFIG=/tmp/kubeconfig kubectl version -o json | python -c \
  "import json,sys; print(json.load(sys.stdin)['serverVersion']['gitVersion'])"

Two Ready nodes and a server version matching kubernetes_version mean the AKS cluster is usable.

Then check the two things a kubectl get nodes will not tell you. First, that the pools carry the labels and modes you declared, because a pool that silently landed in System mode will refuse to scale to zero later:

# CLI: pool modes, sizes and autoscaler bounds as ARM actually recorded them
az aks nodepool list --resource-group platform-aks --cluster-name platform-aks \
  --query '[].[name,mode,vmSize,enableAutoScaling,minCount,maxCount,orchestratorVersion]' -o table

Second, that a second pulumi preview immediately after a successful pulumi up reports no changes. A non-empty preview here is the signature of a property Azure normalised on the way in — a VM size the API returned in different casing, a node count the autoscaler has already moved — and it will keep proposing that diff on every run until it is either pinned in code or added to ignore_changes.

Gotchas & Edge Cases

resource_name_ is not a typo. The native provider appends an underscore where an ARM property collides with Pulumi's own resource_name. Omitting it silently names the managed cluster after the Pulumi logical name plus a suffix.

The system pool cannot scale to zero. Azure requires at least one node in a System pool. Put burstable workloads in a separate User pool, which may scale to zero.

Version pinning has a floor. Azure retires old Kubernetes versions; a pinned version eventually becomes uncreatable and pulumi up fails on a fresh stack with AgentPoolK8sVersionNotSupported. Review the pin each quarter against az aks get-versions.

Quota failures arrive late and partially applied. Insufficient headroom surfaces as Code="InsufficientVCPUQuota" with a message naming the region, the quota remaining and the quota requested — typically several minutes into the create, once the resource group, the identity and the control plane already exist. Pulumi records what succeeded, so the fix is to raise the quota or shrink the pool and re-run pulumi up; destroying and starting over is unnecessary and slower.

The last System pool cannot be removed. Attempting to delete or convert the only System-mode pool fails with OperationNotAllowed and a message stating that at least one system node pool must remain. This bites when someone tries to migrate the system pool to a larger VM size by deleting and recreating it — create the replacement System pool first, let it come up, then remove the old one.

Subnet exhaustion looks like a scheduling problem. When an Azure CNI pool cannot reserve its max_pods + 1 addresses, the new node never joins and the autoscaler reports NotTriggerScaleUp. The AKS cluster is healthy, the pool is healthy, and pods sit Pending with no obvious cause until you count free addresses in the subnet.

pulumi refresh can import an autoscaler's opinion. Refreshing writes the live node count into state; if ignore_changes=["count"] is absent, the next apply enforces whatever number was current at refresh time. On a pool that scales on a daily curve, that turns a routine refresh into an unplanned scale-down.

Operational Notes

Day-two work on an AKS cluster is mostly version movement, and the ordering is not optional. The control plane upgrades first and may run at most one minor version ahead of its node pools; node pools then upgrade one at a time, each cordoning and draining its nodes in surge batches. Attempting to move a pool more than one minor version, or moving a pool ahead of the control plane, is rejected before anything starts.

Version movement on a live managed cluster Version movement on a live managed cluster: Read get-upgrades → Bump control plane → Roll system pool → Roll user pools → Re-pin in code → repeat. Readget-upgrades Bump controlplane Roll system pool Roll user pools Re-pin in code
Control plane first, one minor version at a time, then each pool in turn.

Ask Azure what is reachable from where you are rather than reading the release notes:

# CLI: the upgrade targets ARM will actually accept for this managed cluster
az aks get-upgrades --resource-group platform-aks --name platform-aks \
  --query 'controlPlaneProfile.upgrades[].kubernetesVersion' -o tsv

In Pulumi terms, the upgrade is a two-commit sequence: change kubernetes_version on the ManagedCluster and apply, wait for the control plane to report Succeeded, then change orchestrator_version on each AgentPool and apply again. Doing both in one commit works, but it puts a fifteen-to-forty-minute node roll inside a single pulumi up, which is a long time to hold a deployment lock and an awkward thing to abandon halfway.

Node image upgrades are the quieter half of the same job. Azure publishes new node images continuously with kernel and runtime patches, and they are applied by az aks nodepool upgrade --node-image-only rather than by any property Pulumi manages. Because the image version is not part of the declared resource body, this is one of the few AKS operations that is correctly left outside infrastructure code entirely — a scheduled job or the node-image auto-upgrade channel, not a Pulumi program.

Give the AKS cluster a maintenance window so Azure's own automation does not roll nodes during business hours:

# maintenance.py — confine Azure-initiated upgrades to a known window
# CLI: pulumi up
import pulumi_azure_native.containerservice as aks

aks.MaintenanceConfiguration(
    "weekly-window",
    config_name="aksManagedAutoUpgradeSchedule",
    resource_group_name=rg.name,
    resource_name_=cluster.name,
    # Provider note: config_name is a fixed Azure-recognised identifier, not a
    # free-text label — an arbitrary name creates a configuration Azure ignores.
    maintenance_window=aks.MaintenanceWindowArgs(
        schedule=aks.ScheduleArgs(
            weekly=aks.WeeklyScheduleArgs(day_of_week="Sunday", interval_weeks=1),
        ),
        duration_hours=4,
        start_time="02:00",
        utc_offset="+00:00",
    ),
)

Finally, treat the state of this stack as a high-value artifact. It holds a decoded kubeconfig, and while Output.secret encrypts it inside the state document, anyone who can run pulumi stack output --show-secrets against the stack holds administrative access to everything running on the AKS cluster. Scope stack access accordingly, and prefer handing downstream stacks a reference to the AKS cluster's identity over copying the credential between stacks.

FAQ

System-assigned or user-assigned identity?

System-assigned is simpler — Azure creates and rotates it with the AKS cluster. Use user-assigned when several resources must share one identity, or when the identity must outlive the managed cluster.

azure or kubenet networking?

azure (CNI) gives pods real VNet addresses, which routing and network policy usually need, at the cost of a larger address plan. kubenet conserves addresses. The choice is fixed at creation.

Why mark the kubeconfig as a secret output?

It is an unrestricted cluster credential. Without Output.secret it is written in plaintext to state and echoed by pulumi stack output.

Can the Kubernetes version be upgraded in place?

Yes — changing kubernetes_version to a supported newer version performs a managed upgrade rather than a replacement. Node pools upgrade separately and should be rolled one at a time. Azure only accepts a jump of one minor version at a time, so moving from 1.28 to 1.30 is two applies with a healthy interval between them.

How do I run kubectl against this managed cluster from CI?

Export the kubeconfig as a secret output and have the pipeline write it to a file with pulumi stack output kubeconfig --show-secrets. With Azure AD integration on, that kubeconfig invokes kubelogin, so the runner also needs kubelogin installed and a workload identity or service principal it can exchange — a bare kubeconfig alone is not enough.

Does destroying the stack delete the workloads too?

It deletes the AKS cluster, which takes the nodes and everything scheduled on them with it, but resources the workloads created outside that boundary survive. Public IPs allocated by a LoadBalancer Service, disks provisioned by a PersistentVolumeClaim with a Retain policy and DNS records written by an in-cluster controller all outlive the destroy and become orphaned spend.