Skip to content

Kubernetes IaC for AKS: Comprehensive Reference

Core Principle: Separation of Concerns

Platform IaC (Terraform/Bicep for AKS cluster, VNet, ACR, Key Vault) and Application IaC (Helm/Kustomize for workload manifests) operate on completely different lifecycles, use different tools, require different credentials, and must live in separate repositories with separate pipelines.

Dimension Platform IaC Application IaC
Changes Infrequently (cluster upgrades, networking) Daily (app deployments)
Blast radius High — can destroy the cluster Namespace-scoped
Credentials Cloud-provider credentials (Terraform/Bicep) Cluster credentials only
Owner Platform team Application teams
Tools Terraform, Bicep Helm, Kustomize

Never mix platform and app resources in one Terraform state. An application change must never be able to accidentally destroy the cluster.

GitOps: The Operational Model

GitOps extends declarative IaC to operations via four principles: 1. Git as single source of truth for all desired state 2. Pull-based deployment — in-cluster agent pulls changes (eliminates stored cluster credentials in CI) 3. Continuous reconciliation — automatically corrects drift 4. Declarative descriptions for everything

Flux CD v2 vs ArgoCD — When to Choose Each

Aspect Flux CD v2 ArgoCD
UI None (CLI only) Rich web dashboard with app visualization, diff views
Architecture Modular Kubernetes-native controllers Monolithic with CRDs
SOPS secrets Native integration Requires plugins
Kustomize + Helm kustomize-controller as post-renderer (unique capability) Separate paths
Multi-cluster Supported ApplicationSets with generators (Git directory, cluster, list, matrix)
After Weaveworks shutdown Continues under AWS, Microsoft, community Stable, CNCF graduated

Choose ArgoCD for teams wanting UI-driven operations and easier onboarding. Choose Flux for platform teams wanting composable, Kubernetes-native controllers with tighter GitOps purity and native SOPS encryption.

GitOps Repository Structure

Separate application source repos from GitOps config repos: - App repo: source code + Dockerfile; CI builds images and updates the config repo - Config repo: Kubernetes manifests; GitOps agent deploys from here

Directory-based promotion on a single branch (not branch-per-environment): 

config-repo/
├── envs/
│   ├── dev/          # kustomization.yaml pointing to base + dev overlay
│   ├── staging/
│   └── prod/
├── base/             # shared manifests
└── platform/         # cluster-level config (RBAC, namespaces, network policies)

Promotion = copying version information between directories via PR. Clear visibility, easy rollback via git revert, no merge conflicts between environments.

Helm Chart Design

Chart Structure

my-chart/
├── Chart.yaml          # metadata, version, appVersion
├── values.yaml         # defaults
├── values.schema.json  # validation schema (ALWAYS include)
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── _helpers.tpl    # reusable named templates
│   └── NOTES.txt
├── charts/             # chart dependencies
└── .helmignore

values.schema.json — Catch Misconfigurations Early

Helm validates values against this schema at install/upgrade time before anything reaches the cluster:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "replicaCount": {
      "type": "integer",
      "minimum": 1
    },
    "image": {
      "type": "object",
      "properties": {
        "repository": { "type": "string" },
        "tag": { "type": "string" }
      },
      "required": ["repository", "tag"]
    }
  },
  "required": ["replicaCount", "image"]
}

_helpers.tpl — Naming and Labels

Define common labels, selectors, and naming in _helpers.tpl and reference with {{ include "mychart.labels" . | nindent 4 }}. This ensures consistency and enables upgrades to find existing resources.

Versioning and Registries

  • Follow SemVer 2.0.0 strictly: MAJOR for breaking value schema changes, MINOR for new features, PATCH for fixes
  • Push charts to OCI-based registries (ACR, GHCR): helm push mychart-1.0.0.tgz oci://myregistry.azurecr.io/charts
  • Use helm registry login with Workload Identity or managed identity credentials

Anti-Patterns to Avoid

  • Over-templating: parameterizing everything "just in case" creates unmaintainable charts
  • God charts: one chart for an entire platform with dozens of conditionally-enabled components
  • :latest tags in default values — breaks reproducibility and rollback
  • Not using .helmignore — test files bloat packaged charts

Kustomize: Overlays and Patches

Base + Overlay Pattern

kubernetes/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml      # references ../base
    │   └── patches/
    │       └── deployment-replicas.yaml
    ├── staging/
    │   └── kustomization.yaml
    └── prod/
        └── kustomization.yaml

base/kustomization.yaml: 

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

overlays/prod/kustomization.yaml: 

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: myapp-prod
resources:
  - ../../base
patches:
  - path: patches/deployment-prod.yaml
images:
  - name: myapp
    newTag: "1.2.3"

Keep namespace declarations out of base resources. Specify namespaces in overlay kustomization.yaml files so the base is reusable across environments without modification.

ConfigMapGenerator — Automatic Rolling Updates

ConfigMapGenerator creates resources with content-hash suffixed names, automatically triggering rolling updates when content changes:

configMapGenerator:
  - name: app-config
    envs:
      - .env.prod
    literals:
      - LOG_LEVEL=info

secretGenerator:
  - name: db-credentials
    files:
      - password.txt

This is a critical advantage over manually managed ConfigMaps — no more "did the deployment pick up the config change?"

Strategic Merge Patches

# patches/deployment-prod.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: myapp
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            memory: "512Mi"

Kustomize + Helm Combined

Flux's kustomize-controller can post-render Helm output with Kustomize patches — combine both tools in a single pipeline. Use Helm for third-party charts (Prometheus, NGINX, cert-manager) and Kustomize for overlaying environment-specific patches.

Terraform for AKS Provisioning

Module Structure (Preferred Over Workspaces)

Directory-based separation is preferred over workspaces for production — workspaces create risk of applying to the wrong environment.

infrastructure/
├── modules/
│   ├── vnet/           # VNet, subnets, NSGs
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── aks/            # AKS cluster, node pools
│   ├── acr/            # Azure Container Registry
│   ├── keyvault/       # Key Vault + RBAC
│   └── monitoring/     # Log Analytics, Managed Prometheus
└── environments/
    ├── dev/
    │   └── main.tf     # composes modules with dev values
    ├── staging/
    └── prod/

Each environment root module composes the modules with different variable values. Bridge environments via Terraform outputs consumed by application configuration.

Remote State in Azure Blob Storage

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-state"
    storage_account_name = "stterraformstate"
    container_name       = "tfstate"
    key                  = "prod/aks.tfstate"
  }
}

Azure Blob Storage natively supports state locking via blob leases — no extra configuration needed. Enable blob versioning for rollback. One state file per environment prevents cross-environment blast radius.

OIDC Authentication for CI/CD (No Stored Secrets)

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=4.14.0"    # Pin exactly; commit .terraform.lock.hcl
    }
  }
}

provider "azurerm" {
  features {}
  use_oidc = true  # Uses OIDC federated credentials from GitHub Actions
}

AKS Cluster Module Example

module "aks" {
  source = "../../modules/aks"

  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  cluster_name        = "aks-${var.environment}"

  # System node pool
  system_node_count  = 3
  system_vm_size     = "Standard_D4s_v5"
  availability_zones = ["1", "2", "3"]

  # Networking — Azure CNI Overlay with Cilium (recommended for new clusters)
  vnet_subnet_id     = module.vnet.aks_subnet_id
  network_plugin     = "azure"
  network_plugin_mode = "overlay"
  network_dataplane  = "cilium"

  # Identity — user-assigned for pre-provisioned role assignments
  identity_type      = "UserAssigned"
  user_assigned_identity_id = azurerm_user_assigned_identity.aks.id

  # Security
  enable_oidc_issuer       = true
  enable_workload_identity = true
  azure_rbac_enabled       = true   # Azure RBAC for Kubernetes authorization
  private_cluster_enabled  = true

  # Add-ons
  enable_azure_policy = true
  enable_key_vault_secrets_provider = true
  enable_monitor       = true
}

Azure Verified Modules

The Azure Verified Modules (Azure/avm-ptn-aks-production/azurerm, released October 2024) provide enterprise-grade, Microsoft-supported AKS provisioning. Use as the starting point for production clusters.

module "aks_production" {
  source  = "Azure/avm-ptn-aks-production/azurerm"
  version = "~> 0.1"

  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  name                = "aks-prod"
}

Bicep: Azure-Native Alternative

Bicep requires no state management — Azure Resource Manager tracks state directly. Eliminates an entire category of operational concerns (state locking, corruption, storage). Day-zero support for new Azure features.

When to choose Bicep: Azure-only teams wanting the simplest experience, teams without Terraform investment.

When to choose Terraform: Multi-cloud environments, non-Azure resources (Datadog, GitHub, DNS providers), or teams with existing Terraform investment.

param location string = resourceGroup().location
param clusterName string

resource aks 'Microsoft.ContainerService/managedClusters@2024-02-01' = {
  name: clusterName
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${aksIdentity.id}': {}
    }
  }
  properties: {
    agentPoolProfiles: [
      {
        name: 'system'
        count: 3
        vmSize: 'Standard_D4s_v5'
        mode: 'System'
        osType: 'Linux'
        availabilityZones: ['1', '2', '3']
        enableAutoScaling: true
        minCount: 3
        maxCount: 9
        nodeTaints: ['CriticalAddonsOnly=true:NoSchedule']
      }
    ]
    networkProfile: {
      networkPlugin: 'azure'
      networkPluginMode: 'overlay'
      networkDataplane: 'cilium'
    }
    oidcIssuerProfile: { enabled: true }
    securityProfile: { workloadIdentity: { enabled: true } }
    enableRBAC: true
    aadProfile: {
      managed: true
      enableAzureRBAC: true
    }
  }
}

Use az deployment group what-if for plan-equivalent previews before applying.

AKS Security Hardening

Pod Security Standards (Replace Deprecated PSPs)

Enforce the Restricted profile on all production namespaces:

apiVersion: v1
kind: Namespace
metadata:
  name: myapp-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.28
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

A Restricted-compliant pod security context:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

Migrate gradually: set warn and audit modes first, then switch to enforce after validating workloads.

Network Policies — Default Deny

Apply a default deny-all NetworkPolicy to every namespace, then explicitly allow required communication:

# 1. Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# 2. Allow DNS (always required)
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

With Cilium on AKS, leverage CiliumNetworkPolicy for L7 and FQDN-based filtering (Azure CNI Overlay + Cilium is the recommended stack for new clusters).

Workload Identity Setup (Replaces Deprecated Pod Identity)

# Enable on cluster
az aks update \
  --resource-group myRG \
  --name myAKS \
  --enable-oidc-issuer \
  --enable-workload-identity

# Create user-assigned managed identity
az identity create \
  --name myapp-identity \
  --resource-group myRG

# Get OIDC issuer URL
OIDC_ISSUER=$(az aks show --resource-group myRG --name myAKS \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

# Create federated credential
az identity federated-credential create \
  --name myapp-federated \
  --identity-name myapp-identity \
  --resource-group myRG \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:myapp:myapp-sa" \
  --audiences api://AzureADTokenExchange

Kubernetes service account annotation: 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: myapp
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"

Pod label to trigger token injection: 

spec:
  serviceAccountName: myapp-sa
  labels:
    azure.workload.identity/use: "true"

OPA Gatekeeper / Azure Policy

Azure Policy for AKS enforces governance at admission time using built-in initiative including Deployment Safeguards. Key policies to enforce:

  • Require resource limits on containers
  • Disallow privileged containers
  • Require non-root user
  • Require readOnlyRootFilesystem
  • Restrict allowed registries (only pull from ACR)
  • Enforce image signing (via Ratify + cosign)

Enable via Terraform: 

resource "azurerm_kubernetes_cluster" "main" {
  # ...
  azure_policy_enabled = true
}

Container Scanning — Trivy

Run Trivy in CI/CD pipeline to fail builds on CRITICAL/HIGH findings:

- name: Scan container image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'myapp:${{ github.sha }}'
    format: 'sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'
    output: 'trivy-results.sarif'

- name: Upload Trivy scan results
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: 'trivy-results.sarif'
  if: always()

Deploy Trivy Operator for continuous in-cluster scanning of running workloads.

Private Cluster Configuration

Production AKS clusters should use API server VNet integration — control plane accessible only from private network:

resource "azurerm_kubernetes_cluster" "main" {
  private_cluster_enabled             = true
  private_cluster_public_fqdn_enabled = false
  api_server_access_profile {
    vnet_integration_enabled = true
    subnet_id                = azurerm_subnet.apiserver.id
  }
}

For management access use Azure Bastion for SSH tunneling, or az aks command invoke to run kubectl without direct network access.

Node Pool Design

Production AKS clusters must separate system and user node pools:

# System node pool — only critical add-ons
resource "azurerm_kubernetes_cluster" "main" {
  default_node_pool {
    name                        = "system"
    vm_size                     = "Standard_D4s_v5"
    node_count                  = 3
    zones                       = ["1", "2", "3"]
    only_critical_addons_enabled = true  # Taint: CriticalAddonsOnly=true:NoSchedule
    os_sku                      = "AzureLinux"
  }
}

# User node pool — application workloads
resource "azurerm_kubernetes_cluster_node_pool" "user" {
  kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id
  name                  = "user"
  vm_size               = "Standard_D8s_v5"
  min_count             = 3
  max_count             = 20
  enable_auto_scaling   = true
  zones                 = ["1", "2", "3"]
  os_sku                = "AzureLinux"
}

Azure Linux 2.0 reaches EOL in November 2025. Migrate to AzureLinux 3 before March 2026.

Kubenet is deprecated with removal scheduled March 2028. Azure CNI Overlay powered by Cilium is the recommended networking stack for new clusters — 30% reduction in service routing latency, replaces kube-proxy with eBPF, built-in network policy.

Upgrade Strategy

Use both cluster auto-upgrade and node OS auto-upgrade with separate maintenance windows:

resource "azurerm_kubernetes_cluster" "main" {
  automatic_upgrade_channel  = "stable"  # Auto-upgrade to latest patch of N-1 minor version
  node_os_upgrade_channel    = "NodeImage"

  maintenance_window_auto_upgrade {
    frequency   = "Weekly"
    interval    = 1
    duration    = 4
    day_of_week = "Sunday"
    start_time  = "02:00"
    utc_offset  = "+00:00"
  }

  maintenance_window_node_os {
    frequency   = "Weekly"
    interval    = 1
    duration    = 4
    day_of_week = "Wednesday"
    start_time  = "02:00"
    utc_offset  = "+00:00"
  }
}

Configure surge upgrades on node pools (max_surge = "33%") to provision extra nodes during rolling upgrades. Always define PodDisruptionBudgets — misconfigured PDBs block the entire upgrade process.

Environment Promotion Pattern

Dev → Staging → Prod via overlay promotion:

  1. CI builds image, tags it with git SHA: myapp:abc1234
  2. CI updates the image tag in envs/dev/kustomization.yaml via PR
  3. GitOps agent deploys to dev
  4. Validate in dev, promote: update envs/staging/kustomization.yaml via PR
  5. Validate in staging, promote: update envs/prod/kustomization.yaml via PR with required approvals

With ArgoCD ApplicationSets: 

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
spec:
  generators:
  - list:
      elements:
      - cluster: dev
        url: https://dev-aks.example.com
      - cluster: staging
        url: https://staging-aks.example.com
      - cluster: prod
        url: https://prod-aks.example.com
  template:
    metadata:
      name: 'myapp-{{cluster}}'
    spec:
      source:
        repoURL: https://github.com/myorg/config-repo
        path: 'envs/{{cluster}}'
      destination:
        server: '{{url}}'
        namespace: myapp

Production Workload Checklist

Every production Deployment should have:

  • Minimum 3 replicas
  • PodDisruptionBudget with maxUnavailable: 1 (never maxUnavailable: 0 — blocks cluster upgrades)
  • HPA with stabilization windows to prevent flapping
  • Topology spread across zones (DoNotSchedule) and nodes (ScheduleAnyway)
  • All three probe types: startup (failureThreshold: 60, slow start tolerance), liveness (lightweight self-check only, never check external deps), readiness (checks dependencies, removes from Service endpoints)
  • Resource requests always set; memory limits always set; CPU limits not set (causes CFS throttling)
  • Non-root user, readOnlyRootFilesystem, no privilege escalation, drop ALL capabilities
  • Dedicated ServiceAccount with automountServiceAccountToken: false
  • ServiceMonitor for Prometheus scraping
  • Distroless or Chainguard base images (no shell, no package manager)
  • Images pinned by digest, not tag: myapp@sha256:abc123...
resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    memory: "256Mi"    # Always set memory limit
    # No CPU limit — allow bursting to avoid CFS throttling

Secret Management Options

Three approaches for different needs:

  1. External Secrets Operator (ESO) — recommended for GitOps. Synchronizes secrets from Azure Key Vault into Kubernetes Secrets via ExternalSecret CRDs safe to commit to Git. Authenticate to Key Vault using Workload Identity.

  2. Azure Key Vault CSI Driver — mounts secrets directly as CSI volumes (secrets never stored in etcd). Ideal when avoiding Kubernetes Secrets entirely. Enable with enable_key_vault_secrets_provider = true in Terraform.

  3. SOPS with Azure Key Vault — encrypts secret values while keeping keys readable. Perfect for GitOps diffs. Native Flux support. Keys are readable in Git; only values are encrypted.

Never store unencrypted secrets in Git. Four defense layers: pre-commit hooks with gitleaks, CI gitleaks scanning, SOPS encryption, ExternalSecret CRDs as the primary pattern.

Plugin: azure-cloud-infra · View SKILL.md on GitHub