How to Deploy a GKE Cluster with Pulumi (Python)

Deploying a GKE cluster with Pulumi Python means provisioning the GKE control plane, a separately managed node pool, Workload Identity for keyless pod-to-GCP auth, and a typed kubeconfig output — part of the broader GCP Provider Configuration workflow. The pattern that matters is removing the default node pool and managing nodes as their own resource, so you can resize and recreate nodes without touching the control plane.

This guide builds a VPC-native GKE cluster with the default node pool stripped out, a dedicated node pool with autoscaling, Workload Identity enabled end to end, and a kubeconfig assembled from the GKE cluster outputs.

Context

The default GKE cluster is convenient and wrong for production: its node pool is coupled to the GKE cluster resource, so changing node config can force the whole cluster to recreate. Separating the node pool and enabling Workload Identity from the start avoids both an expensive recreate later and the anti-pattern of mounting node service-account keys into pods. Nodes provisioned this way can read a secured GCS bucket through Workload Identity instead of static keys, and the GKE cluster inherits its credential routing from the parent provider configuration.

Context Context: Context with 3 facets. Context GKE key element Workload key element GCS key element
Context: how GKE, Workload, GCS relate in this pattern.

Two structural choices are baked in at creation and cannot be changed later without replacing the GKE cluster: whether it is VPC-native, and whether it is regional or zonal. VPC-native (alias IP) mode gives pods real routable addresses from a secondary range on the subnetwork, which is what makes Private Service Connect, VPC-native load balancing, and cross-project networking work. Routes-based clusters, the older default, consume a custom static route per node and hit VPC route quota long before they hit any interesting scale. Passing an empty ClusterIpAllocationPolicyArgs() opts into VPC-native and lets GKE create the pod and service secondary ranges for you; supplying cluster_secondary_range_name and services_secondary_range_name instead binds the GKE cluster to ranges you already sized deliberately.

Sizing those ranges is a one-way door with arithmetic behind it. The pod range must cover nodes x pods_per_node, and the default 110 pods per node means each node consumes a /24 out of the pod CIDR. A /20 pod range therefore caps the GKE cluster at 16 nodes regardless of what the autoscaler is allowed to do, and the failure arrives as a scheduling stall rather than an obvious error — the node pool scales, the new node comes up without an alias range, and pods stay Pending.

Prerequisites

Prerequisites Prerequisites: layered from GCP_PROJECT down to Python. GCP_PROJECT GCP_REGION mypy kubectl Python
Prerequisites: the building blocks this section assembles.
  • Python 3.9+ with pulumi>=3.0 and pulumi-gcp>=7.0.
  • A GCP project with the Kubernetes Engine API enabled and GCP_PROJECT / GCP_REGION set, or the provider configured per the parent guide.
  • IAM permissions for container.clusters.create, container.clusters.get, and iam.serviceAccounts.create.
  • An existing VPC-native network and subnetwork with secondary ranges for pods and services, or permission to create them.
  • mypy for static checking; kubectl to verify the kubeconfig.

Implementation

1. Define typed cluster configuration

Implementation Implementation: 1. Define typed then 2. Create the GKE then 3. Attach an then 4. Assemble and 1. Define typed 2. Create the GKE 3. Attach an 4. Assemble and
Implementation: the stages run left to right — 1. Define typed, 2. Create the GKE, 3. Attach an, 4. Assemble and.
# infra/gke_config.py
# CLI: mypy --strict infra/
from __future__ import annotations
from dataclasses import dataclass

@dataclass(frozen=True)
class GkeConfig:
    name: str
    location: str = "us-central1"
    network: str = "default"
    subnetwork: str = "default"
    node_machine_type: str = "e2-standard-4"
    min_nodes: int = 1
    max_nodes: int = 3
    # State implication: changing `location` (region vs zone) forces
    # replacement of the GKE cluster.

2. Create the GKE cluster with the default node pool removed

remove_default_node_pool=True plus initial_node_count=1 tells GKE to bootstrap and then delete the default pool, leaving you free to attach a managed one. workload_identity_config binds the GKE cluster to the project workload identity pool.

# infra/gke.py
# CLI: pulumi preview --diff
from __future__ import annotations
import pulumi_gcp as gcp
from infra.gke_config import GkeConfig

def build_cluster(cfg: GkeConfig, project: str) -> gcp.container.Cluster:
    return gcp.container.Cluster(
        cfg.name,
        name=cfg.name,
        location=cfg.location,
        network=cfg.network,
        subnetwork=cfg.subnetwork,
        # Provider note: remove the default pool so nodes are managed
        # independently of the control plane.
        remove_default_node_pool=True,
        initial_node_count=1,
        ip_allocation_policy=gcp.container.ClusterIpAllocationPolicyArgs(),  # VPC-native
        workload_identity_config=gcp.container.ClusterWorkloadIdentityConfigArgs(
            workload_pool=f"{project}.svc.id.goog",
        ),
        deletion_protection=False,  # State implication: set True in prod to block accidental destroy
    )

3. Attach an autoscaling node pool with Workload Identity

The node pool sets workload_metadata_config="GKE_METADATA" so pods cannot reach the node's service-account token through the metadata server — the prerequisite for keyless Workload Identity.

# infra/gke.py (continued)
# CLI: pulumi up
import pulumi_gcp as gcp
from infra.gke_config import GkeConfig

def build_node_pool(cfg: GkeConfig, cluster: gcp.container.Cluster) -> gcp.container.NodePool:
    return gcp.container.NodePool(
        f"{cfg.name}-pool",
        cluster=cluster.name,
        location=cfg.location,
        autoscaling=gcp.container.NodePoolAutoscalingArgs(
            min_node_count=cfg.min_nodes,
            max_node_count=cfg.max_nodes,
        ),
        node_config=gcp.container.NodePoolNodeConfigArgs(
            machine_type=cfg.node_machine_type,
            oauth_scopes=["https://www.googleapis.com/auth/cloud-platform"],
            # Provider note: GKE_METADATA enforces Workload Identity on pods.
            workload_metadata_config=gcp.container.NodePoolNodeConfigWorkloadMetadataConfigArgs(
                mode="GKE_METADATA",
            ),
        ),
    )

4. Assemble and export a typed kubeconfig

Build the kubeconfig from the GKE cluster endpoint and CA certificate using Output.all().apply() so the credentials resolve only after the GKE cluster exists.

# __main__.py
# CLI: pulumi up && pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml
import pulumi
import pulumi_gcp as gcp
from infra.gke_config import GkeConfig
from infra.gke import build_cluster, build_node_pool

project = gcp.config.project or ""
cfg = GkeConfig(name="apps")
cluster = build_cluster(cfg, project)
node_pool = build_node_pool(cfg, cluster)

def kubeconfig(name: str, endpoint: str, ca: str) -> str:
    return f"""apiVersion: v1
clusters:
- cluster: {{certificate-authority-data: {ca}, server: https://{endpoint}}}
  name: {name}
contexts:
- context: {{cluster: {name}, user: {name}}}
  name: {name}
current-context: {name}
users:
- name: {name}
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: gke-gcloud-auth-plugin
      provideClusterInfo: true
"""

kc = pulumi.Output.all(cluster.name, cluster.endpoint,
                       cluster.master_auth.cluster_ca_certificate).apply(
    lambda a: kubeconfig(a[0], a[1], a[2])
)
pulumi.export("kubeconfig", pulumi.Output.secret(kc))  # State implication: secret

Completing the Workload Identity Binding

workload_identity_config on the GKE cluster and GKE_METADATA on the node pool only build the machinery. Nothing authenticates until a specific Kubernetes service account is paired with a specific Google service account, in both directions: an IAM policy binding on the GCP side, and an annotation on the Kubernetes side. Miss either half and pods fail at the first API call with a 403 whose message points at the node's default service account rather than at the identity you intended to use.

How a pod token becomes a GCP access token How a pod token becomes a GCP access token: Pod → GKE metadata server → Google STS → GCS API. Pod GKE metadataserver Google STS GCS API projected KSA token exchange for federated token GSA access token token on metadata path authenticated request
With GKE_METADATA the node's own service account is never handed to the pod.

The exchange happens entirely inside the node. The kubelet projects a short-lived, audience-scoped token for the pod's service account; the GKE metadata server intercepts the well-known metadata endpoint, exchanges that token with Google's STS for a federated token, and swaps that for an access token belonging to the Google service account named in the annotation. The pod never sees a key file, and the tokens expire in minutes rather than never.

# infra/workload_identity.py — bind one KSA to one GSA
# CLI: pulumi up
from __future__ import annotations
import pulumi
import pulumi_gcp as gcp
import pulumi_kubernetes as k8s

def bind_identity(project: str, namespace: str, ksa_name: str,
                  k8s_provider: k8s.Provider) -> gcp.serviceaccount.Account:
    gsa = gcp.serviceaccount.Account(
        f"{ksa_name}-gsa",
        account_id=ksa_name,
        display_name=f"Workload identity for {namespace}/{ksa_name}",
    )
    # Provider note: the member string is the literal principal GKE presents;
    # the square-bracket form is namespace/serviceaccount, not a path.
    gcp.serviceaccount.IAMMember(
        f"{ksa_name}-wi",
        service_account_id=gsa.name,
        role="roles/iam.workloadIdentityUser",
        member=f"serviceAccount:{project}.svc.id.goog[{namespace}/{ksa_name}]",
    )
    k8s.core.v1.ServiceAccount(
        f"{ksa_name}-ksa",
        metadata=k8s.meta.v1.ObjectMetaArgs(
            name=ksa_name,
            namespace=namespace,
            annotations={"iam.gke.io/gcp-service-account": gsa.email},
        ),
        opts=pulumi.ResourceOptions(provider=k8s_provider),
    )
    return gsa

The GSA still needs whatever project-level roles the workload actually uses — roles/storage.objectViewer on a bucket, roles/pubsub.subscriber on a subscription. Grant those to the GSA, never to the node pool's service account, or every pod on the node inherits them. Storage grants in particular pair with the bucket-level IAM described in creating and securing GCS buckets with Pulumi (Python), and the wider binding model is covered in managing GCP IAM bindings with Pulumi (Python).

Ordering matters here in a way Pulumi will not infer for you. The Kubernetes provider needs a reachable API server, which means the GKE cluster must exist before the ServiceAccount resource is created, and in practice you want the node pool up too so the first scheduled pod does not race the metadata daemonset. Pass the kubeconfig Output into the k8s.Provider so the dependency is real rather than implied, and add depends_on=[node_pool] when the manifest schedules work immediately.

Verification

Assert Workload Identity is configured, then confirm the kubeconfig actually reaches the GKE cluster.

Verification Verification: Test → Program → Mock/Cloud. Test Program Mock/Cloud invoke declare resolve assert
Verification: the test drives the program and asserts on resolved values.
# tests/test_gke.py
# CLI: pytest tests/test_gke.py
from __future__ import annotations
import pulumi
from typing import Any, Dict, Tuple

class Mocks(pulumi.runtime.Mocks):
    def new_resource(self, args: pulumi.runtime.MockResourceArgs) -> Tuple[str, Dict[str, Any]]:
        outs = {**args.inputs, "endpoint": "10.0.0.1",
                "masterAuth": {"clusterCaCertificate": "QQ=="}}
        return (f"{args.name}-id", outs)
    def call(self, args: pulumi.runtime.MockCallArgs) -> Dict[str, Any]:
        return {}

pulumi.runtime.set_mocks(Mocks(), preview=False)

import importlib
main = importlib.import_module("__main__")

@pulumi.runtime.test
def test_workload_identity() -> pulumi.Output:
    return main.cluster.workload_identity_config.apply(
        lambda w: None if w and w.workload_pool else (_ for _ in ()).throw(AssertionError("WI not set"))
    )
# CLI: use the exported kubeconfig against the live cluster
pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml
KUBECONFIG=kubeconfig.yaml kubectl get nodes

A green kubectl get nodes proves the kubeconfig and the API server, not the identity path. Verify Workload Identity from inside the GKE cluster, because that is the only place the metadata server exists:

# CLI: prove the pod's identity is the GSA, not the node service account
kubectl run wi-test --rm -it --image=google/cloud-sdk:slim \
  --overrides='{"spec":{"serviceAccountName":"app"}}' -- \
  curl -s -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"
# Expect app@<project>.iam.gserviceaccount.com. Seeing the -compute@ default
# service account means GKE_METADATA or the annotation is missing.

The same check catches the most common half-configured state: the binding exists, the annotation is present, but the node pool was created before GKE_METADATA was set, so pods on the older nodes still see the node identity. Node pool metadata configuration only applies to nodes created after the change, which means recreating the pool or waiting out a rolling upgrade.

Gotchas & Edge Cases

Gotchas & Edge Cases Gotchas & Edge Cases: Where it breaks with 4 facets. Where it breaks location watch this boundary GKE_METADATA watch this boundary deletion_prote watch this boundary Edge Cases watch this boundary
Gotchas & Edge Cases: the boundaries where things break and what to check.

A regional location triples your node count and cost. Setting location to a region (e.g. us-central1) spreads nodes across three zones, so min_node_count=1 means three nodes total — one per zone. Use a zonal location (e.g. us-central1-a) for cheaper dev clusters, and know that switching between regional and zonal forces a GKE cluster replacement.

Workload Identity needs the IAM binding too, not just GKE_METADATA. Enabling workload_identity_config and GKE_METADATA is only half the setup. Each Kubernetes service account must be annotated and bound to a GCP service account via roles/iam.workloadIdentityUser before pods can impersonate it. Without that binding, pod auth fails with a metadata-server permission error.

deletion_protection defaults to blocking pulumi destroy. Recent provider versions default cluster deletion_protection to True, so pulumi destroy fails until you set it to False and run pulumi up first. For dev stacks set it False explicitly; for production leave it on and disable deliberately when you mean to tear down.

Concurrent operations on one GKE cluster serialise, and the second one fails. GKE allows exactly one mutating operation per GKE cluster at a time. Updating the control plane and a node pool in the same pulumi up produces googleapi: Error 400: Cluster is running incompatible operation operation-.... Pulumi surfaces it as a resource failure, not a retry. Split control-plane changes and node-pool changes into separate updates, or give the node pool an explicit depends_on the GKE cluster resource so the engine cannot run them in parallel.

Node pool operations outlive Pulumi's default timeout. Resizing or recreating a pool with slow-draining workloads regularly exceeds twenty minutes, and the update aborts with a timeout while GCP keeps working. The state then disagrees with reality until the next refresh. Set custom_timeouts on the node pool resource (pulumi.CustomTimeouts(create="30m", update="30m", delete="30m")) rather than re-running and hoping.

A pod disruption budget can stall an auto-upgrade forever. Auto-upgrade drains nodes one at a time and respects PDBs. A budget that permits zero disruptions — a single-replica deployment with minAvailable: 1 — blocks the drain indefinitely, and the GKE cluster silently sits on an old node version. Check kubectl get pdb -A before enabling auto-upgrade, not after.

Operational Notes

A GKE cluster is not a resource you create and forget; it is a resource Google keeps changing under you. Release channels decide how fast. RAPID tracks new minor versions within weeks, REGULAR lags by roughly a quarter, STABLE lags further, and omitting release_channel pins you to static versions you must bump by hand. Declaring the channel in code makes the upgrade cadence a reviewed decision instead of an inherited default.

What changes a node without touching the control plane What changes a node without touching the control plane: layered from Release channel down to Pod disruption budgets. Release channel picks the control-plane version track Maintenance window when auto-upgrades may start Surge upgrade settings max_surge / max_unavailable per pool Node auto-repair recreates failed nodes in place Pod disruption budgets can stall a drain indefinitely
Node lifecycle knobs, from the slowest-moving to the one that most often blocks an upgrade.
# infra/gke_lifecycle.py — declare the upgrade cadence and surge behaviour
# CLI: pulumi preview --diff
from __future__ import annotations
import pulumi_gcp as gcp

RELEASE_CHANNEL = gcp.container.ClusterReleaseChannelArgs(channel="REGULAR")

MAINTENANCE = gcp.container.ClusterMaintenancePolicyArgs(
    recurring_window=gcp.container.ClusterMaintenancePolicyRecurringWindowArgs(
        start_time="2026-01-05T02:00:00Z",
        end_time="2026-01-05T06:00:00Z",
        recurrence="FREQ=WEEKLY;BYDAY=SA,SU",
    ),
)

# State implication: max_surge adds nodes before draining, so an upgrade
# temporarily exceeds max_node_count and costs more for its duration.
UPGRADE = gcp.container.NodePoolUpgradeSettingsArgs(max_surge=1, max_unavailable=0)

MANAGEMENT = gcp.container.NodePoolManagementArgs(auto_repair=True, auto_upgrade=True)

max_surge=1, max_unavailable=0 is the setting most teams want and few set: GKE adds one extra node, moves pods onto it, then removes an old node, so capacity never dips during an upgrade. The inverse (max_surge=0, max_unavailable=1) upgrades in place and is cheaper, at the cost of running below desired capacity for the duration. Whichever you pick, remember the autoscaler's max_node_count does not account for surge nodes, so a pool pinned at its ceiling can fail to upgrade with a quota error.

Cost control has two levers worth wiring in from the start. Spot nodes (spot=True in NodePoolNodeConfigArgs) cut compute cost sharply for workloads that tolerate a 30-second eviction notice, and they belong in their own node pool with a taint so only opted-in workloads land there. Node auto-provisioning is the other; it lets GKE create pools sized to pending pods, which is powerful and directly at odds with declarative IaC, because the pools it creates are invisible to your program and show up as unmanaged infrastructure in every audit.

For observability, enable the managed collectors rather than running your own: logging_config and monitoring_config on the GKE cluster resource control which components ship logs and metrics, and turning off workload logging is the single largest lever on Cloud Logging spend for a chatty GKE cluster. Changing either is an in-place update, so it is safe to tune after the fact.

Deleting is where multi-team GKE clusters bite hardest. deletion_protection=True blocks pulumi destroy, which is correct for production and infuriating in ephemeral test stacks — set it False explicitly in throwaway stacks so CI can clean up. Even then, load balancers and persistent disks created by in-cluster controllers are not in your Pulumi state, and deleting the GKE cluster orphans them. Remove Kubernetes Service objects of type LoadBalancer and any dynamically provisioned PersistentVolumeClaim before tearing the GKE cluster down, or expect to reconcile leftover forwarding rules and disks by hand.

FAQ

Why remove the default node pool? The default node pool is part of the GKE cluster resource, so changing node configuration can force the entire cluster to recreate. Setting remove_default_node_pool=True and managing an aws-style separate NodePool lets you resize, upgrade, or replace nodes without disturbing the control plane.

What does Workload Identity actually give me? It lets a Kubernetes service account impersonate a GCP service account, so pods authenticate to GCP APIs with short-lived tokens instead of mounted node keys. You enable workload_pool on the GKE cluster, GKE_METADATA on the node pool, and then bind each KSA to a GSA with roles/iam.workloadIdentityUser.

How do I authenticate kubectl with the exported kubeconfig? The kubeconfig uses the gke-gcloud-auth-plugin exec credential. Install it (gcloud components install gke-gcloud-auth-plugin), then KUBECONFIG=kubeconfig.yaml kubectl get nodes authenticates with your gcloud identity automatically.

Should I use a regional or zonal cluster? Regional clusters replicate the control plane and nodes across zones for high availability at roughly triple the node cost. Zonal clusters are cheaper and fine for development. Choose at creation — switching forces a GKE cluster replacement.

How do I keep the kubeconfig out of plaintext state? Wrap the assembled kubeconfig in pulumi.Output.secret() before exporting, as shown. Pulumi then stores it encrypted in state and masks it in pulumi stack output unless you pass --show-secrets.

Why did my node pool update fail with an incompatible-operation error? GKE serialises mutations per cluster, so a control-plane change and a node-pool change in the same update collide. Order them with depends_on, or apply control-plane changes in a separate pulumi up and let each operation finish before starting the next.