Overview of Azure CNI Overlay networking in Azure Kubernetes Service (AKS)

Azure CNI Overlay is a networking model for Azure Kubernetes Service (AKS) that provides efficient IP address management and high-performance pod communication. This article provides an overview of Azure CNI Overlay, including its architecture, IP address planning, and differences from the traditional kubenet networking model.

Overview of overlay networking

The traditional Azure Container Networking Interface (CNI) assigns a virtual network IP address to every pod. It assigns this IP address from a reserved set of IPs on every node or a separate subnet reserved for pods. This approach requires IP address planning and might lead to address exhaustion, which introduces difficulties scaling your clusters as your application demands grow.

In overlay networking, only the Kubernetes cluster nodes are assigned IPs from subnets. Pods receive IPs from a private Classless Inter-Domain Routing (CIDR) range provided at the time of cluster creation. Each node is assigned a /24 address space carved out from the same CIDR. Extra nodes created when you scale out a cluster automatically receive /24 address spaces from the same CIDR. Azure CNI assigns IPs to pods from this /24 space.

A separate routing domain is created in the Azure networking stack for the pod's private CIDR space. This domain creates an overlay network for direct communication between pods. There's no need to provision custom routes on the cluster subnet or use an encapsulation method to tunnel traffic between pods, which provides connectivity performance between pods on par with virtual machines (VMs) in a virtual network. Workloads running within the pods aren't even aware that network address manipulation is happening.

Communication with endpoints outside the cluster, such as on-premises and peered virtual networks, uses the node IP through network address translation (NAT). Azure CNI translates the source IP (overlay IP of the pod) of the traffic to the primary IP address of the VM. This translation enables the Azure networking stack to route the traffic to the destination.

Endpoints outside the cluster can't connect to a pod directly. You have to publish the pod's application as a Kubernetes Load Balancer service to make it reachable on the virtual network.

You can provide outbound (egress) connectivity to the internet for overlay pods by using a standard load balancer or managed NAT gateway. You can also control egress traffic by directing it to a firewall via user-defined routes on the cluster subnet.

You can configure ingress connectivity to the cluster by using an ingress controller, such as Application Gateway for Containers, NGINX, or the application routing add-on.

Differences between kubenet and Azure CNI Overlay

Like Azure CNI Overlay, kubenet assigns IP addresses to pods from an address space that's logically different from the virtual network, but it has scaling and other limitations. The following table provides a detailed comparison between kubenet and Azure CNI Overlay:

IP address planning

The following sections provide guidance on how to plan your IP address space for Azure CNI Overlay.

Cluster nodes

When you set up your AKS cluster, make sure that your virtual network subnets have enough room to grow for future scaling. You can assign each node pool to a dedicated subnet. A /24 subnet can fit up to 251 nodes because the first three IP addresses are reserved for management tasks.

Pods

The /24 size that Azure CNI Overlay assigns is fixed and can't be increased or decreased. You can run up to 250 pods on a node. When you plan the pod address space, ensure that the private CIDR is large enough to provide /24 address spaces for new nodes to support future cluster expansion.

When you plan IP address space for pods, consider the following factors:

• You can use the same pod CIDR space on multiple independent AKS clusters in the same virtual network.
• Pod CIDR space must not overlap with the cluster subnet range.
• Pod CIDR space must not overlap with directly connected networks, like virtual network peering, Azure ExpressRoute, or VPN. If external traffic has source IPs in the pod CIDR range, it needs translation to a non-overlapping IP via Source Network Address Translation (SNAT) to communicate with the cluster.
• Pod CIDR space can only be expanded.

Kubernetes service address range

The size of the service address CIDR depends on the number of cluster services that you plan to create. It must be smaller than /12. This range shouldn't overlap with the pod CIDR range, cluster subnet range, and IP range used in peered virtual networks and on-premises networks.

Kubernetes service IP address for DNS

The IP address for DNS is within the Kubernetes service address range that cluster service discovery uses. Don't use the first IP address in your address range, because this address is used for the kubernetes.default.svc.cluster.local address.

Network security groups

Pod-to-pod traffic with Azure CNI Overlay isn't encapsulated, and subnet network security group (NSG) rules are applied. If the subnet NSG contains deny rules that would affect the pod CIDR traffic, make sure that the following rules are in place to ensure proper cluster functionality (in addition to all AKS egress requirements):

• Traffic from the node CIDR to the node CIDR on all ports and protocols
• Traffic from the node CIDR to the pod CIDR on all ports and protocols (required for service traffic routing)
• Traffic from the pod CIDR to the pod CIDR on all ports and protocols (required for pod-to-pod and pod-to-service traffic, including DNS)

Traffic from a pod to any destination outside the pod CIDR block uses SNAT to set the source IP to the IP of the node where the pod runs.

If you want to restrict traffic between workloads in the cluster, we recommend using network policies.

Maximum pods per node

You can configure the maximum number of pods per node at the time of cluster creation or when you add a new node pool. The default for Azure CNI Overlay is 250. The maximum value that you can specify in Azure CNI Overlay is 250, and the minimum value is 10. The value for maximum pods per node that you configure during creation of a node pool applies to the nodes in that node pool only.

Choosing a network model

Azure CNI offers two IP addressing options for pods: overlay networking and the traditional configuration that assigns virtual network IPs to pods. The choice of which option to use for your AKS cluster is a balance between flexibility and advanced configuration needs. The following considerations help outline when each network model might be the most appropriate.

Use overlay networking when:

• You want to scale to a large number of pods but are limited by IP address space in your virtual network.
• Most of the pod communication is within the cluster.
• You don't need advanced AKS features, such as virtual nodes.

Use the traditional virtual network option when:

• You have available IP address space.
• Most of the pod communication is to resources outside the cluster.
• Resources outside the cluster need to reach pods directly.
• You need AKS advanced features, such as virtual nodes.

Limitations with Azure CNI Overlay

Azure CNI Overlay has the following limitations:

• VM availability sets aren't supported.
• If you're using your own subnet to deploy the cluster, the names of the subnet, the virtual network, and the resource group that contains the virtual network must be 63 characters or fewer. These names are used as labels in AKS worker nodes, so they're subject to Kubernetes syntax rules for labels.

Related content

To get started with Azure CNI Overlay in AKS, see the following articles:

• Configure Azure CNI Overlay networking in Azure Kubernetes Service (AKS)
• Expand pod CIDR space in Azure CNI Overlay Azure Kubernetes Service (AKS) clusters