Upgrade options and recommendations for Azure Kubernetes Service (AKS) clusters

AKS Standard remains available for scenarios where you need deeper manual control over upgrade mechanics, networking choices, or node pool behavior.

This article gives you a technical foundation for AKS upgrades by covering upgrade options, common scenarios, and recommendations for both AKS Standard.

What this article covers

This technical reference covers:

• Manual versus automated upgrade paths, and when to use each.
• Common upgrade scenarios with specific recommendations.
• Optimization techniques for performance and minimal disruption.
• Validation processes and pre-upgrade checks.

For related guidance:

• For production-oriented AKS upgrade strategy details, see AKS production upgrade strategies.
• For stateful workload upgrade patterns, see Stateful workload upgrade patterns.
• For scenario-led guidance, see AKS upgrade scenarios: Choose your path.
• If you're new to AKS upgrades, start with the upgrade scenarios hub for guided assistance.

Quick navigation

Upgrade operating models

AKS Standard gives you direct control over upgrade sequencing and tuning. You choose and manage:

• Manual or automatic upgrade configuration.
• Upgrade channel selection.
• Node pool and surge behavior.
• Operational procedures around maintenance windows and workload disruption budgets.

Upgrade options

Perform manual upgrades

Applies primarily to AKS Standard, or to specialized operational workflows.

Manual upgrades let you control when your cluster upgrades to a new Kubernetes version. These upgrades are useful for testing, staged rollouts, and targeted version adoption.

• Upgrade an AKS cluster
• Upgrade multiple AKS clusters via Azure Kubernetes Fleet Manager
• Upgrade the node image
• Customize node surge upgrade
• Process node OS updates

Configure automatic upgrades

For AKS Standard, automatic upgrades help keep clusters on supported versions while preserving control over policy and scheduling.

• Automatically upgrade an AKS cluster
• Automatically upgrade multiple AKS clusters via Azure Kubernetes Fleet Manager
• Use planned maintenance to schedule and control upgrades
• Stop AKS cluster upgrades automatically on API breaking changes (preview)
• Automatically upgrade AKS cluster node operating system images
• Apply security updates to AKS nodes automatically by using GitHub actions

Special considerations for node pools that span multiple availability zones

AKS uses best-effort zone balancing in node pools. During an upgrade surge, the zones for surge nodes in virtual machine scale sets are unknown ahead of time, which can temporarily cause an unbalanced zone configuration. AKS deletes surge nodes after the upgrade and restores the original zone balance.

To keep zones balanced, set surge to a multiple of three nodes. Persistent volume claims that use Azure locally redundant storage disks are zone bound and might cause downtime if surge nodes are in a different zone. Use a pod disruption budget (PDB) to maintain high availability during drains.

Optimize upgrades to improve performance and minimize disruptions

Combine planned maintenance window, max surge, PDB, node drain timeout, and node soak time to increase the likelihood of successful, low-disruption upgrades.

In AKS Standard, tune upgrade controls directly:

• Planned maintenance window: Schedule auto-upgrade during low-traffic periods. Use at least four hours.
• Max surge: Higher values speed upgrades but might disrupt workloads. Use 33% for production.
• Max unavailable: Use when capacity is limited.
• Pod disruption budget: Set to limit pods down during upgrades. Validate for your service.
• Node drain timeout: Configure pod eviction wait duration. The default is 30 minutes.
• Node soak time: Stagger upgrades to minimize downtime. The default is 0 minutes.

Validations used in the upgrade process

AKS performs pre-upgrade validations to ensure cluster health:

• API breaking changes: Detects deprecated APIs.
• Kubernetes upgrade version: Ensures a valid upgrade path.
• PDB configuration: Checks for misconfigured PDBs (for example, maxUnavailable=0).
• Quota: Confirms enough quota for surge nodes.
• Subnet: Verifies sufficient IP addresses.
• Certificates/service principals: Detects expired credentials.
• Managed Resource Lock Check: Checks for resource locks applied to the managed cluster resource group.

These checks apply across AKS. In AKS Standard, they're part of your operational workflow.

Common upgrade scenarios and recommendations

Scenario 1: Capacity constraints

If your cluster is limited by product tier or regional capacity, upgrades might fail when surge nodes can't be provisioned. This situation is common with specialized product tiers (like GPU nodes) or in regions with limited resources. Errors such as SKUNotAvailable, AllocationFailed, or OverconstrainedAllocationRequest might occur if maxSurge is set too high for available capacity.

• Use maxUnavailable to upgrade by using existing nodes instead of surging new ones. For more information, see Customize unavailable nodes during upgrade.
• Lower maxSurge to reduce extra capacity needs. For more information, see Customize node surge upgrade.
• For security-only updates, use security patch reimages that don't require surge nodes. For more information, see Apply security and kernel updates to Linux nodes in Azure Kubernetes Service.

Scenario 2: Node drain failures and PDBs

Upgrades require draining nodes (evicting pods). Drains can fail when pods are slow to terminate or strict Pod Disruption Budgets (PDBs) block pod evictions.

Example error:

Code: UpgradeFailed
Message: Drain node ... failed when evicting pod ... Cannot evict pod as it would violate the pod's disruption budget.

AKS Standard guidance

Option 1: Force upgrade, bypass PDB constraints

Use force upgrade only when PDBs prevent critical upgrades and can't be resolved. This action overrides PDB protections and can potentially cause complete service unavailability during the upgrade.

Requirements: Azure CLI 2.79.0+ or AKS API version 2025-09-01+

az aks upgrade \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP_NAME \
  --kubernetes-version $KUBERNETES_VERSION \
  --enable-force-upgrade \
  --upgrade-override-until yyyy-mm-ddT13:00:00Z

Option 2: Handle undrainable nodes while honoring PDBs

Use this conservative approach to honor PDBs while preventing upgrade failures.

Configure undrainable node behavior:

az aks nodepool update \
  --resource-group <resource-group-name> \
  --cluster-name <cluster-name> \
  --name <node-pool-name> \
  --undrainable-node-behavior Cordon \
  --max-blocked-nodes 2 \
  --drain-timeout 30

Behavior options:

• Schedule (default): Deletes blocked node and surges replacement.
• Cordon (recommended): Cordons node and labels it as kubernetes.azure.com/upgrade-status=Quarantined.

Max blocked nodes (preview):

• Specifies how many nodes that fail to drain are tolerated
• Requires undrainable-node-behavior to be set
• Defaults to maxSurge value (typically 10%) if not specified

Prerequisites for max blocked nodes

The Azure CLI aks-preview extension version 18.0.0b9 or later is required to use the max blocked nodes feature.

# Install or update the aks-preview extension
az extension add --name aks-preview
az extension update --name aks-preview

Example configuration with max blocked nodes

az aks nodepool update \
  --cluster-name jizenMC1 \
  --name nodepool1 \
  --resource-group jizenTestMaxBlockedNodesRG \
  --max-surge 1 \
  --undrainable-node-behavior Cordon \
  --max-blocked-nodes 2 \
  --drain-timeout 5

Option 3: Automatic PDB management (preview)

Use the automatic PDB management extension to proactively resolve PDB-blocked drains without bypassing PDB protections or requiring manual cleanup of quarantined nodes. Automatic PDB management detects when a PDB blocks eviction on a cordoned node and temporarily scales up the deployment's replicas so the disruption budget is satisfied. After the drain completes, it scales replicas back to their original count.

Automatic PDB management can also automatically create PDBs for deployments that don't have one, ensuring your workloads are protected during upgrade drains. For installation and configuration details, see Manage Pod Disruption Budgets automatically during AKS upgrades.

Recommendations to prevent drain failures

• Set maxUnavailable in PDBs to allow at least one pod eviction
• Increase pod replicas to meet disruption budget requirements
• Extend drain timeout if workloads need more time. (The default is 30 minutes.)
• Use automatic PDB management to automate PDB creation and replica scaling during drain operations.
• Test PDBs in staging, monitor upgrade events, and use blue-green deployments for critical workloads. For more information, see Blue-green deployment of AKS clusters.

Verify undrainable nodes

• The blocked nodes are unscheduled for pods and marked with the label "kubernetes.azure.com/upgrade-status: Quarantined".

• Verify the label on any blocked nodes when there's a drain node failure on upgrade:

  kubectl get nodes --show-labels=true
  

Resolve undrainable nodes

1. Remove the responsible PDB:

   kubectl delete pdb <pdb-name>
   

2. Remove the kubernetes.azure.com/upgrade-status: Quarantined label:

   kubectl label nodes <node-name> kubernetes.azure.com/upgrade-status-
   

3. Optionally, delete the blocked node:

   az aks nodepool delete-machines --cluster-name <cluster-name> --machine-names <machine-name> --name <node-pool-name> --resource-group <resource-group-name>
   

4. After you finish this step, you can reconcile the cluster status by performing any update operation without the optional fields as outlined in az aks. Alternatively, you can scale the node pool to the same number of nodes as the count of upgraded nodes. This action ensures that the node pool gets to its intended original size. AKS prioritizes the removal of the blocked nodes. This command also restores the cluster provisioning status to Succeeded. In the following example, 2 is the total number of upgraded nodes.

   # Update the cluster to restore the provisioning status
   az aks update --resource-group <resource-group-name> --name <cluster-name>
   
   # Scale the node pool to restore the original size
   az aks nodepool scale --resource-group <resource-group-name> --cluster-name <cluster-name> --name <node-pool-name> --node-count 2
   

Scenario 3: Slow upgrades

Conservative settings or node-level issues can delay upgrades, which affects your ability to stay current with patches and improvements.

Common causes of slow upgrades include:

• Low maxSurge or maxUnavailable values (limits parallelism).

• High soak times (long waits between node upgrades).

• Drain failures (see Node drain failures).

• Use maxSurge=33%, maxUnavailable=1 for production.

• Use maxSurge=50%, maxUnavailable=2 for dev/test.

• Use OS Security Patch for fast, targeted patching (avoids full node reimaging).

• Enable --undrainable-node-behavior to avoid upgrade blockers.

Scenario 4: IP exhaustion

Surge nodes require more IPs. If the subnet is near capacity, node provisioning can fail (for example, Error: SubnetIsFull). This scenario is common with Azure Container Networking Interface, high maxPods, or large node counts.

• Ensure that your subnet has enough IPs for all nodes, surge nodes, and pods. The formula is Total IPs = (Number of nodes + maxSurge) * (1 + maxPods).

• Reclaim unused IPs or expand the subnet (for example, from /24 to /22).

• Lower maxSurge if subnet expansion isn't possible.

  az aks nodepool update \
    --resource-group <resource-group-name> \
    --cluster-name <cluster-name> \
    --name <node-pool-name> \
    --max-surge 10%
  

• Monitor IP usage with Azure Monitor or custom alerts.

• Reduce maxPods per node, clean up orphaned load balancer IPs, and plan subnet sizing for high-scale clusters.

Frequently asked questions

Can I use open-source tools for validation?

Yes. Many open-source tools integrate well with AKS upgrade processes:

• kube-no-trouble (kubent): Scans for deprecated APIs before upgrades.
• Trivy: Security scanning for container images and Kubernetes configurations.
• Sonobuoy: Kubernetes conformance testing and cluster validation.
• kube-bench: Security benchmark checks against Center for Internet Security standards.
• Polaris: Validation of Kubernetes best practices.
• kubectl-neat: Clean up Kubernetes manifests for validation.

How do I validate API compatibility before upgrading?

Run deprecation checks by using tools like kubent:

# Install and run API deprecation scanner
kubectl apply -f https://github.com/doitintl/kube-no-trouble/releases/latest/download/knt-full.yaml

# Check for deprecated APIs in your cluster
kubectl run knt --image=doitintl/knt:latest --rm -it --restart=Never -- \
  -c /kubeconfig -o json > api-deprecation-report.json

# Review findings
cat api-deprecation-report.json | jq '.[] | select(.deprecated==true)'

What makes AKS upgrades different from other Kubernetes platforms?

AKS provides several unique advantages:

• Native Azure integration with Azure Traffic Manager, Azure Load Balancer, and networking.
• Azure Kubernetes Fleet Manager for coordinated multicluster upgrades.
• Automatic node image patching without manual node management.
• Built-in validation for quota, networking, and credentials.
• Azure support for upgrade-related issues.

Choose your upgrade path

This article provided you with a technical foundation. Now select your scenario-based path.

Ready to execute?

Still deciding?

Use the upgrade scenarios hub for a guided decision tree that considers your:

• Downtime tolerance
• Environment complexity
• Risk profile
• Timeline constraints

Final recommendations

• Always check for API breaking changes and validate your workload's compatibility with the target Kubernetes version.
• Test upgrade settings (such as maxSurge, maxUnavailable, and PDBs) in a staging environment to minimize production risk.
• Monitor upgrade events and cluster health throughout the process.