Azure CNI networking in Azure Kubernetes Service (AKS) overview

By default, AKS clusters use kubenet and create a virtual network and subnet. With kubenet, nodes get an IP address from a virtual network subnet. Network address translation (NAT) is then configured on the nodes, and pods receive an IP address "hidden" behind the node IP. This approach reduces the number of IP addresses that you need to reserve in your network space for pods to use.

With Azure Container Networking Interface (CNI), every pod gets an IP address from the subnet and can be accessed directly. Systems in the same virtual network as the AKS cluster see the pod IP as the source address for any traffic from the pod. Systems outside the AKS cluster virtual network see the node IP as the source address for any traffic from the pod. These IP addresses must be unique across your network space and must be planned in advance. Each node has a configuration parameter for the maximum number of pods that it supports. The equivalent number of IP addresses per node are then reserved up front for that node. This approach requires more planning, and often leads to IP address exhaustion or the need to rebuild clusters in a larger subnet as your application demands grow.

Prerequisites

• The virtual network for the AKS cluster must allow outbound internet connectivity.

• AKS clusters can't use 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24 for the Kubernetes service address range, pod address range, or cluster virtual network address range.

• The cluster identity used by the AKS cluster must have at least Network Contributor permissions on the subnet within your virtual network. If you wish to define a custom role instead of using the built-in Network Contributor role, the following permissions are required:

  • Microsoft.Network/virtualNetworks/subnets/join/action

  • Microsoft.Network/virtualNetworks/subnets/read

  • Microsoft.Authorization/roleAssignments/write

• The subnet assigned to the AKS node pool can't be a delegated subnet.

• AKS doesn't apply Network Security Groups (NSGs) to its subnet and doesn't modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range. For more information, see Network security groups.

Plan IP addressing for your cluster

Clusters configured with Azure CNI networking require extra planning. The size of your virtual network and its subnet must accommodate the number of pods you plan to run and the number of nodes for the cluster.

IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the virtual network. Each node is configured with a primary IP address. Azure CNI preconfigures 30 extra IP addresses by default. These IP addresses are assigned to pods scheduled on the node. When you scale out your cluster, each node is similarly configured with IP addresses from the subnet. You can also view the maximum pods per node.

If you expect your nodes to run the maximum number of pods, and regularly destroy and deploy pods, you should also factor in some extra IP addresses per node. A few seconds can be required to delete a service and release its IP address for a new service to be deployed and acquire the address. These extra IP addresses consider this possibility.

The IP address plan for an AKS cluster consists of a virtual network, at least one subnet for nodes and pods, and a Kubernetes service address range.

Address range / Azure resource	Limits and sizing
Virtual network	The Azure virtual network can be as large as /8, but is limited to 65,536 configured IP addresses. Consider all your networking needs, including communicating with services in other virtual networks, before configuring your address space. For example, if you configure too large of an address space, you might run into issues with overlapping other address spaces within your network.
Subnet	Must be large enough to accommodate the nodes, pods, and all Kubernetes and Azure resources that might be provisioned in your cluster. For example, if you deploy an internal Azure Load Balancer, its front-end IPs are allocated from the cluster subnet, not public IPs. The subnet size should also take into account upgrade operations or future scaling needs. Use the following equation to calculate the minimum subnet size including an extra node for upgrade operations: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure) Example for a 50 node cluster: (51) + (51 * 30 (default)) = 1,581 (/21 or larger) Example for a 50 node cluster that also includes preparation to scale up an extra 10 nodes: (61) + (61 * 30 (default)) = 1,891 (/21 or larger) If you don't specify a maximum number of pods per node when you create your cluster, the maximum number of pods per node is set to 30. The minimum number of IP addresses required is based on that value. If you calculate your minimum IP address requirements on a different maximum value, see Maximum pods per node to set this value when you deploy your cluster.
Kubernetes service address range	Any network element on or connected to this virtual network must not use this range. Service address CIDR must be smaller than /12. You can reuse this range across different AKS clusters.
Kubernetes DNS service IP address	IP address within the Kubernetes service address range that is used by cluster service discovery. Don't use the first IP address in your address range. The first address in your subnet range is used for the kubernetes.default.svc.cluster.local address.

Maximum pods per node

The maximum number of pods per node in an AKS cluster is 250. The default maximum number of pods per node varies between kubenet and Azure CNI networking, and the method of cluster deployment.

Deployment method	Kubenet default	Azure CNI default	Configurable at deployment
Azure CLI	110	30	Yes (up to 250)
Resource Manager template	110	30	Yes (up to 250)
Portal	110	110 (configurable in the Node Pools tab)	Yes (up to 250)

Configure maximum pods per node for new clusters

You can configure the maximum number of pods per node either at cluster deployment time or as you add new node pools. You can set the maximum pods per node value as high as 250.

If you don't specify maxPods when creating new node pools, you receive a default value of 30 for Azure CNI.

A minimum value for maximum pods per node is enforced to guarantee space for system pods critical to cluster health. The minimum value that can be set for maximum pods per node is 10 if and only if the configuration of each node pool has space for a minimum of 30 pods. For example, setting the maximum pods per node to the minimum of 10 requires each individual node pool to have a minimum of three nodes. This requirement applies for each new node pool created as well, so if 10 is defined as maximum pods per node each subsequent node pool added must have at least three nodes.

Networking	Minimum	Maximum
Azure CNI	10	250
Kubenet	10	250

• Azure CLI: Specify the --max-pods argument when you deploy a cluster with the az aks create command. The maximum value is 250.
• Resource Manager template: Specify the maxPods property in the ManagedClusterAgentPoolProfile object when you deploy a cluster with a Resource Manager template. The maximum value is 250.
• Azure portal: Change the Max pods per node field in the node pool settings when creating a cluster or adding a new node pool.

Configure maximum pods per node for existing clusters

The maxPods per node setting can be defined when you create a new node pool. If you need to increase the maxPods setting on an existing cluster, add a new node pool with the new desired maxPods count. After migrating your pods to the new pool, delete the older pool. To delete any older pool in a cluster, ensure you're setting node pool modes as defined in the system node pools document.

Deployment parameters

When you create an AKS cluster, the following parameters are configurable for Azure CNI networking:

Virtual network: The virtual network into which you want to deploy the Kubernetes cluster. If you want to create a new virtual network for your cluster, select Create new and follow the steps in the Create virtual network section. If you want to select an existing virtual network, make sure it's in the same location and Azure subscription as your Kubernetes cluster. For information about the limits and quotas for an Azure virtual network, see Azure subscription and service limits, quotas, and constraints.

Subnet: The subnet within the virtual network where you want to deploy the cluster. If you want to create a new subnet in the virtual network for your cluster, select Create new and follow the steps in the Create subnet section. For hybrid connectivity, the address range shouldn't overlap with any other virtual networks in your environment.

Azure Network Plugin: When Azure network plugin is used, the internal LoadBalancer service with "externalTrafficPolicy=Local" can't be accessed from VMs with an IP in clusterCIDR that doesn't belong to AKS cluster.

Kubernetes service address range: This parameter is the set of virtual IPs that Kubernetes assigns to internal services in your cluster. This range can't be updated after you create your cluster. You can use any private address range that satisfies the following requirements:

• Must not be within the virtual network IP address range of your cluster
• Must not overlap with any other virtual networks with which the cluster virtual network peers
• Must not overlap with any on-premises IPs
• Must not be within the ranges 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24

Although it's technically possible to specify a service address range within the same virtual network as your cluster, doing so isn't recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article. For more information on Kubernetes services, see Services in the Kubernetes documentation.

Kubernetes DNS service IP address: The IP address for the cluster's DNS service. This address must be within the Kubernetes service address range. Don't use the first IP address in your address range. The first address in your subnet range is used for the kubernetes.default.svc.cluster.local address.

Frequently asked questions

• Can I deploy VMs in my cluster subnet?

  Yes. But for Azure CNI for dynamic IP allocation, the VMs cannot be deployed in pod's subnet.

• What source IP do external systems see for traffic that originates in an Azure CNI-enabled pod?

  Systems in the same virtual network as the AKS cluster see the pod IP as the source address for any traffic from the pod. Systems outside the AKS cluster virtual network see the node IP as the source address for any traffic from the pod.

  But for Azure CNI for dynamic IP allocation, no matter the connection is inside the same virtual network or cross virtual networks, the pod IP is always the source address for any traffic from the pod. This is because the Azure CNI for dynamic IP allocation implements Azure Container Networking infrastructure, which gives end-to-end experience. Hence, it eliminates the use of ip-masq-agent, which is still used by traditional Azure CNI.

• Can I configure per-pod network policies?

  Yes, Kubernetes network policy is available in AKS. To get started, see Secure traffic between pods by using network policies in AKS.

• Is the maximum number of pods deployable to a node configurable?

  Yes, when you deploy a cluster with the Azure CLI or a Resource Manager template. See Maximum pods per node.

  You can't change the maximum number of pods per node on an existing cluster.

• How do I configure additional properties for the subnet that I created during AKS cluster creation? For example, service endpoints.

  The complete list of properties for the virtual network and subnets that you create during AKS cluster creation can be configured in the standard virtual network configuration page in the Azure portal.

• Can I use a different subnet within my cluster virtual network for the Kubernetes service address range?

  It's not recommended, but this configuration is possible. The service address range is a set of virtual IPs (VIPs) that Kubernetes assigns to internal services in your cluster. Azure Networking has no visibility into the service IP range of the Kubernetes cluster. The lack of visibility into the cluster's service address range can lead to issues. It's possible to later create a new subnet in the cluster virtual network that overlaps with the service address range. If such an overlap occurs, Kubernetes could assign a service an IP that's already in use by another resource in the subnet, causing unpredictable behavior or failures. By ensuring you use an address range outside the cluster's virtual network, you can avoid this overlap risk.

Next step

Learn more about networking in AKS in the following articles:

• Use a static IP address with the Azure Kubernetes Service (AKS) load balancer

• Use an internal load balancer with Azure Kubernetes Service (AKS)

• Use the application routing addon in Azure Kubernetes Service (AKS)