Customize cluster egress with outbound types in Azure Kubernetes Service (AKS)

  • Article

You can customize egress for an AKS cluster to fit specific scenarios. By default, AKS provisions a standard SKU load balancer to be set up and used for egress. However, the default setup may not meet the requirements of all scenarios if public IPs are disallowed or extra hops are required for egress.

This article covers the various types of outbound connectivity that are available in AKS clusters.

Note

You can now update the outboundType after cluster creation.

Important

In non-private clusters, API server cluster traffic is routed and processed through the clusters outbound type. To prevent API server traffic from being processed as public traffic, consider using a private cluster, or check out the API Server VNet Integration feature.

Limitations

  • Setting outboundType requires AKS clusters with a vm-set-type of VirtualMachineScaleSets and load-balancer-sku of Standard.

Outbound types in AKS

You can configure an AKS cluster using the following outbound types: load balancer, NAT gateway, or user-defined routing. The outbound type impacts only the egress traffic of your cluster. For more information, see setting up ingress controllers.

Note

You can use your own [route table][byo-route-table] with UDR and kubenet networking. Make sure your cluster identity (service principal or managed identity) has Contributor permissions to the custom route table.

Outbound type of loadBalancer

The load balancer is used for egress through an AKS-assigned public IP. An outbound type of loadBalancer supports Kubernetes services of type loadBalancer, which expect egress out of the load balancer created by the AKS resource provider.

If loadBalancer is set, AKS automatically completes the following configuration:

  • A public IP address is provisioned for cluster egress.
  • The public IP address is assigned to the load balancer resource.
  • Backend pools for the load balancer are set up for agent nodes in the cluster.

Diagram shows ingress I P and egress I P, where the ingress I P directs traffic to a load balancer, which directs traffic to and from an internal cluster and other traffic to the egress I P, which directs traffic to the Internet, M C R, Azure required services, and the A K S Control Plane.

For more information, see using a standard load balancer in AKS.

Outbound type of managedNatGateway or userAssignedNatGateway

If managedNatGateway or userAssignedNatGateway are selected for outboundType, AKS relies on Azure Networking NAT gateway for cluster egress.

  • Select managedNatGateway when using managed virtual networks. AKS provisions a NAT gateway and attach it to the cluster subnet.
  • Select userAssignedNatGateway when using bring-your-own virtual networking. This option requires that you have provisioned a NAT gateway before cluster creation.

For more information, see using NAT gateway with AKS.

Outbound type of userDefinedRouting

Note

The userDefinedRouting outbound type is an advanced networking scenario and requires proper network configuration.

If userDefinedRouting is set, AKS won't automatically configure egress paths. The egress setup must be done by you.

You must deploy the AKS cluster into an existing virtual network with a subnet that has been previously configured. Since you're not using a standard load balancer (SLB) architecture, you must establish explicit egress. This architecture requires explicitly sending egress traffic to an appliance like a firewall, gateway, proxy or to allow NAT to be done by a public IP assigned to the standard load balancer or appliance.

For more information, see configuring cluster egress via user-defined routing.

Updating outboundType after cluster creation

Changing the outbound type after cluster creation will deploy or remove resources as required to put the cluster into the new egress configuration.

The following tables show the supported migration paths between outbound types for managed and BYO virtual networks.

Supported Migration Paths for Managed VNet

Each row shows whether the outbound type can be migrated to the types listed across the top. "Supported" means migration is possible, while "Not Supported" or "N/A" means it isn’t.

From|To loadBalancer managedNATGateway userAssignedNATGateway userDefinedRouting
loadBalancer N/A Supported Not Supported Not Supported
managedNATGateway Supported N/A Not Supported Not Supported
userAssignedNATGateway Not Supported Not Supported N/A Not Supported

Supported Migration Paths for BYO VNet

From|To loadBalancer managedNATGateway userAssignedNATGateway userDefinedRouting
loadBalancer N/A Not Supported Supported Supported
managedNATGateway Not Supported N/A Not Supported Not Supported
userAssignedNATGateway Supported Not Supported N/A Supported
userDefinedRouting Supported Not Supported Supported N/A

Migration is only supported between loadBalancer, managedNATGateway (if using a managed virtual network), userAssignedNATGateway and userDefinedRouting (if using a custom virtual network).

Warning

Migrating the outbound type to user managed types (userAssignedNATGateway and userDefinedRouting) will change the outbound public IP addresses of the cluster. if Authorized IP ranges is enabled, please make sure new outbound ip range is appended to authorized ip range.

Warning

Changing the outbound type on a cluster is disruptive to network connectivity and will result in a change of the cluster's egress IP address. If any firewall rules have been configured to restrict traffic from the cluster, you need to update them to match the new egress IP address.

Update cluster to use a new outbound type

Note

You must use a version >= 2.56 of Azure CLI to migrate outbound type. Use az upgrade to update to the latest version of Azure CLI.

  • Update the outbound configuration of your cluster using the az aks update command.

Update cluster from loadbalancer to managedNATGateway

az aks update --resource-group <resourceGroup> --name <clusterName> --outbound-type managedNATGateway --nat-gateway-managed-outbound-ip-count <number of managed outbound ip>

Update cluster from managedNATGateway to loadbalancer

az aks update --resource-group <resourceGroup> --name <clusterName> \
--outbound-type loadBalancer \
<--load-balancer-managed-outbound-ip-count <number of managed outbound ip>| --load-balancer-outbound-ips <outbound ip ids> | --load-balancer-outbound-ip-prefixes <outbound ip prefix ids> >

Warning

Do not reuse an IP address that is already in use in prior outbound configurations.

Update cluster from managedNATGateway to userDefinedRouting

az aks update --resource-group <resourceGroup> --name <clusterName> --outbound-type userDefinedRouting

Update cluster from loadbalancer to userAssignedNATGateway in BYO vnet scenario

az aks update --resource-group <resourceGroup> --name <clusterName> --outbound-type userAssignedNATGateway

Next steps