Migrate from cluster autoscaler to node auto-provisioning

Migrate your existing Azure Kubernetes Service (AKS) cluster from cluster autoscaler to node auto-provisioning using the steps in this guide.

Node auto-provisioning (NAP) uses pending pod resource requirements to decide the optimal virtual machine (VM) configuration to run those workloads in the most efficient and cost-effective manner.

Node auto-provisioning is based on the open-source Karpenter project and the AKS Karpenter provider. Node auto-provisioning automatically deploys, configures, and manages Karpenter on your AKS clusters.

Cluster autoscaler vs. node auto-provisioning

Why migrate from cluster autoscaler to node auto-provisioning

Node auto-provisioning improves bin-packing, automates node lifecycle management, and reduces operational overhead compared to cluster autoscaler.

Cluster autoscaler profile settings vs. node auto-provisioning configuration settings

The following table maps cluster autoscaler profile settings to node auto-provisioning configuration settings for the NodePool CRD. This table also shows the cluster autoscaler Azure CLI command and its NAP CRD equivalent.

Before you begin

Limitations

See NAP limitations and unsupported features.

Disable cluster autoscaler

Pre-migration checklist

• Confirm cluster eligibility for node auto-provisioning. For more on NAP requirements, see Overview of NAP documentation.
• Right-size workloads for consolidation.
  • Set proper resource requests/limits, replicas, and pod disruption budgets (PDBs) to allow for a gradual migration. This migration method requires properly set PDBs to ensure well-managed disruption of your workloads.
• Verify your system node pool is active.
  • AKS requires a system node pool for system components (such as CoreDNS and Karpenter). When NAP is enabled, AKS is responsible for autoscaling the system pool.

Disable cluster autoscaler safely

If cluster autoscaler is enabled cluster-wide, disable it at the cluster level using the --disable-cluster-autoscaler flag. Nodes aren’t removed when you disable cluster autoscaler, so your capacity stays steady.

az aks update --resource-group myResourceGroup --name myAKSCluster --disable-cluster-autoscaler

If cluster autoscaler is only enabled on select node pools, disable cluster autoscaler for specific node pools using the --disable-cluster-autoscaler flag.

# Disable CAS on a specific pool
az aks nodepool update \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mypool1 \
  --disable-cluster-autoscaler

You can also set the node count of your node pool to a pinned count as you begin the migration to node auto-provisioning. The following az aks nodepool scale command pins the node count of node pool mypool1 in cluster myAKSCluster to five (5).

# (Optional) Pin to a safe desired count before the switch
az aks nodepool scale \
  --resource-group myResourceGroup \
  --cluster-name myAKSCluster \
  --name mypool1 \
  --node-count 5

Enable node auto-provisioning

Enable node auto-provisioning on an existing cluster

Enable node auto-provisioning on an existing cluster using the az aks update command and set --node-provisioning-mode to Auto.

az aks update --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME --node-provisioning-mode Auto

Define your first NodePool and AKSNodeClass

#nodepool-default.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        intent: apps
    spec:
      nodeClassRef:
        name: default
        group: karpenter.azure.com
        kind: AKSNodeClass
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: [on-demand]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D]
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
    expireAfter: Never
---
apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: default
  annotations:
    kubernetes.io/description: "General purpose AKSNodeClass for running Ubuntu nodes"
spec:
  imageFamily: Ubuntu

kubectl apply -f nodepool-default.yaml

#nodepool-default.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        intent: apps
    spec:
      nodeClassRef:
        apiVersion: karpenter.azure.com/v1beta1
        kind: AKSNodeClass
        name: comprehensive-example
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: [spot, on-demand]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D, E, F]
      expireAfter: Never
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
---
apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: comprehensive-example
spec:
  # Image family configuration
  # Default: Ubuntu
  # Valid values: Ubuntu, AzureLinux
  imageFamily: Ubuntu

  # Virtual network subnet configuration (optional)
  # If not specified, uses the default --vnet-subnet-id from Karpenter installation
  vnetSubnetID: "/subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/my-rg/providers/Microsoft.Network/virtualNetworks/my-vnet/subnets/my-subnet"

  # OS disk size configuration
  # Default: 128 GB
  # Minimum: 30 GB
  osDiskSizeGB: 128

  # Maximum pods per node configuration
  # Default behavior depends on network plugin:
  # - Azure CNI with standard networking: 30 pods
  # - Azure CNI with overlay networking: 250 pods
  # - Other configurations: 110 pods
  # Range: 10-250
  maxPods: 30

  # Azure resource tags (optional)
  # Applied to all VM instances created with this AKSNodeClass
  tags:
    Environment: "production"
    Team: "platform-team"
    Application: "web-service"
    CostCenter: "engineering"

  # Kubelet configuration (optional)
  # All fields are optional with sensible defaults
  kubelet:
    # CPU management policy
    # Default: "none"
    # Valid values: none, static
    cpuManagerPolicy: "static"

    # CPU CFS quota enforcement
    # Default: true
    cpuCFSQuota: true

    # CPU CFS quota period
    # Default: "100ms"
    cpuCFSQuotaPeriod: "100ms"

    # Image garbage collection thresholds
    # imageGCHighThresholdPercent must be greater than imageGCLowThresholdPercent
    # Range: 0-100
    imageGCHighThresholdPercent: 85
    imageGCLowThresholdPercent: 80

    # Topology manager policy
    # Default: "none"
    # Valid values: none, restricted, best-effort, single-numa-node
    topologyManagerPolicy: "best-effort"

    # Container log configuration
    # containerLogMaxSize default: "50Mi"
    containerLogMaxSize: "50Mi"
    
    # containerLogMaxFiles default: 5, minimum: 2
    containerLogMaxFiles: 5

    # Pod process limits
    # Default: -1 (unlimited)
    podPidsLimit: 4096

kubectl apply -f nodepool-default.yaml

Migrate workloads from fixed pools to node auto-provisioning managed nodes

Now scale down user pools gradually (keep the system pool):

# For each user pool, step down to 0 (this command should respect properly set PDBs)
az aks nodepool scale \
  --resource-group <RG> \
  --cluster-name <CLUSTER> \
  --name <USER_POOL> \
  --node-count 0

As pods evict, node auto-provisioning provisions replacement nodes per your NodePool and AKSNodeClass rules. If a user pool must go to zero, remember you can only do that on user pools (not system pool), and with cluster autoscaler disabled, which is already disabled in an earlier step.

To confirm that the scale down is working and workloads are being scheduled to NAP-managed nodes safely, check:

• Custom resource definition files are active
• Karpenter events detailing NAP decisions
• Nodeclaims are created in response to pending pod pressure

Verify node auto-provisioning

Check CRDs and understand NAP fields

Check CRDs to confirm they are in use:

# Verify CRDs
kubectl get crd | grep karpenter

View field descriptions with the kubectl explain command:

# Use help api to describe fields
kubectl explain nodepool.spec

Confirm new NAP-managed nodes are being created

To ensure that NAP is properly provisioning new nodes in response to pending pod pressure, verify that the new nodes are being created. Node auto-provisioning produces cluster events that you can use to monitor deployment and scheduling decisions. View events through the Kubernetes events stream.

kubectl get events -A --field-selector source=karpenter -w

Alternatively, view the NodeClaims that represent the nodes being created:

kubectl get nodeclaims

A populated list confirms NAP is responding to pending pod pressure.

Clean up old autoscaling

• If you're using managed AKS cluster autoscaler only, cluster autoscaler is already disabled with the above steps.
• If you're using self-hosted cluster autoscaler installed in kube-system, scale the cluster autoscaler pods to zero and remove.

kubectl -n kube-system scale deploy/cluster-autoscaler --replicas=0
kubectl -n kube-system delete deploy/cluster-autoscaler

Fine-tune node auto-provisioning post-migration

After you complete your migration, you can fine-tune your cluster with these capabilities.

• Manage disruption behavior - Tune disruption consolidationPolicy and consolidateAfter windows to balance cost vs. virtual machine churn. See the NAP Disruption documentation.
• Multiple NodePools - Split by workload class (for example, Spot vs On-Demand, GPU vs CPU) and use requirements, weights, and taints to control placement. See the NAP NodePool documentation.
• Networking - For more information on managing networking with custom virtual networks, see the NAP networking documentation.
• Observability - Stream Karpenter events and expose NAP control-plane metrics via Azure Monitor managed Prometheus. See the NAP observability documentation.

Next steps

For more information on node auto-provisioning in AKS, see the following articles:

• Use node auto-provisioning in a custom virtual network
• Configure networking for node auto-provisioning on AKS
• Configure node pools for node auto-provisioning on AKS
• Configure disruption policies for node auto-provisioning on AKS
• Upgrade node images for node auto-provisioning on AKS
• Enable/Disable node auto-provisioning on AKS