Configure an AKS cluster

As part of creating an AKS cluster, you may need to customize your cluster configuration to suit your needs. This article introduces a few options for customizing your AKS cluster.

OS configuration

AKS supports Ubuntu 22.04 as the only node operating system (OS) for clusters with Kubernetes 1.25 and higher. Ubuntu 18.04 can also be specified at node pool creation for Kubernetes versions 1.24 and below.

AKS supports Windows Server 2022 as the default operating system (OS) for Windows node pools in clusters with Kubernetes 1.25 and higher. Windows Server 2019 can also be specified at node pool creation for Kubernetes versions 1.32 and below. Windows Server 2019 is being retired after Kubernetes version 1.32 reaches end of life (EOL) and isn't supported in future releases. For more information about this retirement, see the AKS release notes.

Container runtime configuration

A container runtime is software that executes containers and manages container images on a node. The runtime helps abstract away sys-calls or operating system (OS) specific functionality to run containers on Linux or Windows. For Linux node pools, containerd is used on Kubernetes version 1.19 and higher. For Windows Server 2019 and 2022 node pools, containerd is generally available and is the only runtime option on Kubernetes version 1.23 and higher. As of May 2023, Docker is retired and no longer supported. For more information about this retirement, see the AKS release notes.

Containerd is an OCI (Open Container Initiative) compliant core container runtime that provides the minimum set of required functionality to execute containers and manage images on a node. Containerd was donated to the Cloud Native Compute Foundation (CNCF) in March of 2017. AKS uses the current Moby (upstream Docker) version, which is built on top of containerd.

With a containerd-based node and node pools, instead of talking to the dockershim, the kubelet talks directly to containerd using the CRI (container runtime interface) plugin, removing extra hops in the data flow when compared to the Docker CRI implementation. As such, you see better pod startup latency and less resource (CPU and memory) usage.

By using containerd for AKS nodes, pod startup latency improves and node resource consumption by the container runtime decreases. These improvements through this new architecture enable kubelet communicating directly to containerd through the CRI plugin. While in a Moby/docker architecture, kubelet communicates to the dockershim and docker engine before reaching containerd, therefore having extra hops in the data flow. For more details on the origin of the dockershim and its deprecation, see the Dockershim removal FAQ.

Containerd works on every GA version of Kubernetes in AKS, and in every newer Kubernetes version above v1.19, and supports all Kubernetes and AKS features.

containerd limitations/differences

• For containerd, we recommend using crictl as a replacement CLI instead of the Docker CLI for troubleshooting pods, containers, and container images on Kubernetes nodes. For more information on crictl, see General usage and Client configuration options.

  • Containerd doesn't provide the complete functionality of the docker CLI. It's available for troubleshooting only.
  • crictl offers a more Kubernetes-friendly view of containers, with concepts like pods, etc. being present.

• Containerd sets up logging using the standardized cri logging format (which is different from what you currently get from docker's json driver). Your logging solution needs to support the cri logging format (like Azure Monitor for Containers)

• You can no longer access the docker engine, /var/run/docker.sock, or use Docker-in-Docker (DinD).

  • If you currently extract application logs or monitoring data from Docker engine, use Container insights instead. AKS doesn't support running any out of band commands on the agent nodes that could cause instability.
  • Building images and directly using the Docker engine using the methods mentioned earlier aren't recommended. Kubernetes isn't fully aware of those consumed resources, and those methods present numerous issues as described here and here.

• Building images - You can continue to use your current Docker build workflow as normal, unless you're building images inside your AKS cluster. In this case, consider switching to the recommended approach for building images using ACR Tasks, or a more secure in-cluster option like Docker Buildx.

Generation 2 virtual machines

Azure supports Generation 2 (Gen2) virtual machines (VMs). Generation 2 VMs support key features not supported in generation 1 VMs (Gen1). These features include increased memory, Intel Software Guard Extensions (Intel SGX), and virtualized persistent memory (vPMEM).

Generation 2 VMs use the new UEFI-based boot architecture rather than the BIOS-based architecture used by generation 1 VMs. Only specific SKUs and sizes support Gen2 VMs. Check the list of supported sizes, to see if your SKU supports or requires Gen2.

Additionally, not all VM images support Gen2 VMs. On AKS, Gen2 VMs use AKS Ubuntu 22.04 or 18.04 image or AKS Windows Server 2022 image. These images support all Gen2 SKUs and sizes.

Gen2 VMs are supported on Linux. Gen2 VMs on Windows are supported for WS2022 only.

Generation 2 virtual machines on Windows

Limitations

• Generation 2 VMs are supported on Windows for WS2022 only.
• Generation 2 VMs are default for Windows clusters greater than or equal to Kubernetes 1.25.
• If you select a vm size which supports both Gen 1 and Gen 2, the default for windows node pools will be Gen 1. To specify Gen 2, use custom header UseWindowsGen2VM=true.

Add a Windows node pool with a generation 2 VM

• Add a node pool with generation 2 VMs on Windows using the az aks nodepool add command.

  az aks nodepool add --resource-group myResourceGroup --cluster-name myAKSCluster --name gen2np --node-vm-size Standard_D32_v4 --os-type Windows --aks-custom-headers UseWindowsGen2VM=true
  

The above example will create a WS2022 node pool with a Gen 2 VM. If you're using a vm size which only supports Gen 2, you do not need to add the custom header. If you're using a kubernetes version where Windows Server 2022 is not default, you need to specify --os-sku.

• Check whether you're using generation 1 or generation 2 using the az aks nodepool show command, and check that the nodeImageVersion contains gen2.

  az aks nodepool show
  

• Check available generation 2 VM sizes using the az vm list command.

  az vm list -skus -l $region
  

For more information, see Support for generation 2 VMs on Azure.

Default OS disk sizing

When you create a new cluster or add a new node pool to an existing cluster, the number for vCPUs by default determines the OS disk size. The number of vCPUs is based on the VM SKU, and in the following table we list the default values:

VM SKU Cores (vCPUs)	Default OS Disk Tier	Provisioned IOPS	Provisioned Throughput (Mbps)
1 - 7	P10/128G	500	100
8 - 15	P15/256G	1100	125
16 - 63	P20/512G	2300	150
64+	P30/1024G	5000	200

Use Ephemeral OS on new clusters

Configure the cluster to use ephemeral OS disks when the cluster is created. Use the --node-osdisk-type argument to set Ephemeral OS as the OS disk type for the new cluster.

az aks create --name myAKSCluster --resource-group myResourceGroup -s Standard_DS3_v2 --node-osdisk-type Ephemeral

If you want to create a regular cluster using network-attached OS disks, you can do so by specifying the --node-osdisk-type=Managed argument. You can also choose to add other ephemeral OS node pools, which we cover in the following section.

Use Ephemeral OS on existing clusters

Configure a new node pool to use Ephemeral OS disks. Use the --node-osdisk-type argument to set as the OS disk type as the OS disk type for that node pool.

az aks nodepool add --name ephemeral --cluster-name myAKSCluster --resource-group myResourceGroup -s Standard_DS3_v2 --node-osdisk-type Ephemeral

If you want to create node pools with network-attached OS disks, you can do so by specifying --node-osdisk-type Managed.

Azure Linux container host for AKS

You can deploy the Azure Linux container host for through Azure CLI or ARM templates.

Prerequisites

1. You need the Azure CLI version 2.44.1 or later installed and configured. Run az --version to find the version currently installed. If you need to install or upgrade, see Install Azure CLI.
2. If you don't already have kubectl installed, install it through Azure CLI using az aks install-cli or follow the upstream instructions.

Deploy an Azure Linux AKS cluster with Azure CLI

Use the following example commands to create an Azure Linux cluster.

az group create --name AzureLinuxTest --location chinaeast2

az aks create --name testAzureLinuxCluster --resource-group AzureLinuxTest --os-sku AzureLinux --generate-ssh-keys

az aks get-credentials --resource-group AzureLinuxTest --name testAzureLinuxCluster

kubectl get pods --all-namespaces

Deploy an Azure Linux AKS cluster with an ARM template

To add Azure Linux to an existing ARM template, you need to make the following changes:

• Add "osSKU": "AzureLinux" and "mode": "System" to agentPoolProfiles property.
• Set the apiVersion to 2021-03-01 or newer: "apiVersion": "2021-03-01"

The following deployment uses the ARM template azurelinuxaksarm.json.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.1",
  "parameters": {
    "clusterName": {
      "type": "string",
      "defaultValue": "azurelinuxakscluster",
      "metadata": {
        "description": "The name of the Managed Cluster resource."
      }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "The location of the Managed Cluster resource."
      }
    },
    "dnsPrefix": {
      "type": "string",
      "defaultValue": "azurelinux",
      "metadata": {
        "description": "Optional DNS prefix to use with hosted Kubernetes API server FQDN."
      }
    },
    "osDiskSizeGB": {
      "type": "int",
      "defaultValue": 0,
      "minValue": 0,
      "maxValue": 1023,
      "metadata": {
        "description": "Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 will apply the default disk size for that agentVMSize."
      }
    },
    "agentCount": {
      "type": "int",
      "defaultValue": 3,
      "minValue": 1,
      "maxValue": 50,
      "metadata": {
        "description": "The number of nodes for the cluster."
      }
    },
    "agentVMSize": {
      "type": "string",
      "defaultValue": "Standard_DS2_v2",
      "metadata": {
        "description": "The size of the Virtual Machine."
      }
    },
    "linuxAdminUsername": {
      "type": "string",
      "metadata": {
        "description": "User name for the Linux Virtual Machines."
      }
    },
    "sshRSAPublicKey": {
      "type": "string",
      "metadata": {
        "description": "Configure all linux machines with the SSH RSA public key string. Your key should include three parts, for example 'ssh-rsa AAAAB...snip...UcyupgH azureuser@linuxvm'"
      }
    },
    "osType": {
      "type": "string",
      "defaultValue": "Linux",
      "allowedValues": [
        "Linux"
      ],
      "metadata": {
        "description": "The type of operating system."
      }
    },
    "osSKU": {
      "type": "string",
      "defaultValue": "azurelinux",
      "allowedValues": [
        "AzureLinux",
        "Ubuntu",
      ],
      "metadata": {
        "description": "The Linux SKU to use."
      }
    }
  },
  "resources": [
    {
      "type": "Microsoft.ContainerService/managedClusters",
      "apiVersion": "2021-03-01",
      "name": "[parameters('clusterName')]",
      "location": "[parameters('location')]",
      "properties": {
        "dnsPrefix": "[parameters('dnsPrefix')]",
        "agentPoolProfiles": [
          {
            "name": "agentpool",
            "mode": "System",
            "osDiskSizeGB": "[parameters('osDiskSizeGB')]",
            "count": "[parameters('agentCount')]",
            "vmSize": "[parameters('agentVMSize')]",
            "osType": "[parameters('osType')]",
            "osSKU": "[parameters('osSKU')]"
          }
        ],
        "linuxProfile": {
          "adminUsername": "[parameters('linuxAdminUsername')]",
          "ssh": {
            "publicKeys": [
              {
                "keyData": "[parameters('sshRSAPublicKey')]"
              }
            ]
          }
        }
      },
      "identity": {
          "type": "SystemAssigned"
      }
    }
  ],
  "outputs": {
    "controlPlaneFQDN": {
      "type": "string",
      "value": "[reference(parameters('clusterName')).fqdn]"
    }
  }
}

Create this file on your system and include the settings defined in the azurelinuxaksarm.json file.

az group create --name AzureLinuxTest --location chinaeast2

az deployment group create --resource-group AzureLinuxTest --template-file azurelinuxaksarm.json --parameters linuxAdminUsername=azureuser sshRSAPublicKey=`<contents of your id_rsa.pub>`

az aks get-credentials --resource-group AzureLinuxTest --name testAzureLinuxCluster

kubectl get pods --all-namespaces

Deploy an Azure Linux AKS cluster with Terraform

To deploy an Azure Linux cluster with Terraform, you first need to set your azurerm provider to version 2.76 or higher.

required_providers {
  azurerm = {
    source = "hashicorp/azurerm"
    version = "~> 2.76"
  }
}

Once you've updated your azurerm provider, you can specify the AzureLinux os_sku in default_node_pool.

default_node_pool {
  name = "default"
  node_count = 2
  vm_size = "Standard_D2_v2"
  os_sku = "AzureLinux"
}

Similarly, you can specify the AzureLinux os_sku in azurerm_kubernetes_cluster_node_pool.

Custom resource group name

When you deploy an Azure Kubernetes Service cluster in Azure, it also creates a second resource group for the worker nodes. By default, AKS names the node resource group MC_resourcegroupname_clustername_location, but you can specify a custom name.

To specify a custom resource group name, install the aks-preview Azure CLI extension version 0.3.2 or later. When using the Azure CLI, include the --node-resource-group parameter with the az aks create command to specify a custom name for the resource group. To deploy an AKS cluster with an Azure Resource Manager template, you can define the resource group name by using the nodeResourceGroup property.

az aks create --name myAKSCluster --resource-group myResourceGroup --node-resource-group myNodeResourceGroup

The Azure resource provider in your subscription automatically creates the secondary resource group. You can only specify the custom resource group name during cluster creation.

As you work with the node resource group, keep in mind that you can't:

• Specify an existing resource group for the node resource group.
• Specify a different subscription for the node resource group.
• Change the node resource group name after creating the cluster.
• Specify names for the managed resources within the node resource group.
• Modify or delete Azure-created tags of managed resources within the node resource group.

Next steps

• Learn how to upgrade the node images in your cluster.
• Review Baseline architecture for an Azure Kubernetes Service (AKS) cluster to learn about our recommended baseline infrastructure architecture.
• See Upgrade an Azure Kubernetes Service (AKS) cluster to learn how to upgrade your cluster to the latest version of Kubernetes.
• Read more about containerd and Kubernetes
• See the list of Frequently asked questions about AKS to find answers to some common AKS questions.
• Read more about Ephemeral OS disks.