Create an Azure Kubernetes Service cluster with API Server VNet Integration (Preview)

  • Article

An Azure Kubernetes Service (AKS) cluster configured with API Server VNet Integration projects the API server endpoint directly into a delegated subnet in the VNet where AKS is deployed. API Server VNet Integration enables network communication between the API server and the cluster nodes without requiring a private link or tunnel. The API server is available behind an internal load balancer VIP in the delegated subnet, which the nodes are configured to utilize. By using API Server VNet Integration, you can ensure network traffic between your API server and your node pools remains on the private network only.

API server connectivity

The control plane or API server is in an AKS-managed Azure subscription. Your cluster or node pool is in your Azure subscription. The server and the virtual machines that make up the cluster nodes can communicate with each other through the API server VIP and pod IPs that are projected into the delegated subnet.

API Server VNet Integration is supported for public or private clusters. You can add or remove public access after cluster provisioning. Unlike non-VNet integrated clusters, the agent nodes always communicate directly with the private IP address of the API server internal load balancer (ILB) IP without using DNS. All node to API server traffic is kept on private networking, and no tunnel is required for API server to node connectivity. Out-of-cluster clients needing to communicate with the API server can do so normally if public network access is enabled. If public network access is disabled, you should follow the same private DNS setup methodology as standard private clusters.

Region availability

API Server VNet Integration is available in all global Azure regions.

Prerequisites

  • Azure CLI with aks-preview extension 0.5.97 or later.
  • If using ARM or the REST API, the AKS API version must be 2022-04-02-preview or later.

Install the aks-preview Azure CLI extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

  • Install the aks-preview extension using the az extension add command.

    az extension add --name aks-preview

  • Update to the latest version of the extension released using the az extension update command.

    az extension update --name aks-preview

Register the 'EnableAPIServerVnetIntegrationPreview' feature flag

  1. Register the EnableAPIServerVnetIntegrationPreview feature flag using the az feature register command.

    az feature register --namespace "Microsoft.ContainerService" --name "EnableAPIServerVnetIntegrationPreview"

    It takes a few minutes for the status to show Registered.

  2. Verify the registration status using the az feature show command:

    az feature show --namespace "Microsoft.ContainerService" --name "EnableAPIServerVnetIntegrationPreview"

  3. When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider using the az provider register command.

    az provider register --namespace Microsoft.ContainerService

Create an AKS cluster with API Server VNet Integration using managed VNet

You can configure your AKS clusters with API Server VNet Integration in managed VNet or bring-your-own VNet mode. You can create them as public clusters (with API server access available via a public IP) or private clusters (where the API server is only accessible via private VNet connectivity). You can also toggle between a public and private state without redeploying your cluster.

Create a resource group

  • Create a resource group using the az group create command.

    az group create --location chinanorth3 --name <resource-group>

Deploy a public cluster

  • Deploy a public AKS cluster with API Server VNet integration for managed VNet using the az aks create command with the --enable-api-server-vnet-integration flag.

    az aks create --name <cluster-name> \
    --resource-group <resource-group> \
    --location <location> \
    --network-plugin azure \
    --enable-apiserver-vnet-integration \
    --generate-ssh-keys

Deploy a private cluster

  • Deploy a private AKS cluster with API Server VNet integration for managed VNet using the az aks create command with the --enable-api-server-vnet-integration and --enable-private-cluster flags.

    az aks create --name <cluster-name> \
    --resource-group <resource-group> \
    --location <location> \
    --network-plugin azure \
    --enable-private-cluster \
    --enable-apiserver-vnet-integration \
    --generate-ssh-keys

Create a private AKS cluster with API Server VNet Integration using bring-your-own VNet

When using bring-your-own VNet, you must create and delegate an API server subnet to Microsoft.ContainerService/managedClusters, which grants the AKS service permissions to inject the API server pods and internal load balancer into that subnet. You can't use the subnet for any other workloads, but you can use it for multiple AKS clusters located in the same virtual network. The minimum supported API server subnet size is a /28.

The cluster identity needs permissions to both the API server subnet and the node subnet. Lack of permissions at the API server subnet can cause a provisioning failure.

Warning

An AKS cluster reserves at least 9 IPs in the subnet address space. Running out of IP addresses may prevent API server scaling and cause an API server outage.

Create a resource group

az group create --location <location> --name <resource-group>

Create a virtual network

  1. Create a virtual network using the az network vnet create command.

    az network vnet create --name <vnet-name> \
--resource-group <resource-group> \
--location <location> \
--address-prefixes 172.19.0.0/16

  2. Create an API server subnet using the az network vnet subnet create command.

    az network vnet subnet create --resource-group <resource-group> \
--vnet-name <vnet-name> \
--name <apiserver-subnet-name> \
--delegations Microsoft.ContainerService/managedClusters \
--address-prefixes 172.19.0.0/28

  3. Create a cluster subnet using the az network vnet subnet create command.

    az network vnet subnet create --resource-group <resource-group> \
--vnet-name <vnet-name> \
--name <cluster-subnet-name> \
--address-prefixes 172.19.1.0/24

Create a managed identity and give it permissions on the virtual network

  1. Create a managed identity using the az identity create command.

    az identity create --resource-group <resource-group> --name <managed-identity-name> --location <location>

  2. Assign the Network Contributor role to the API server subnet using the az role assignment create command.

    az role assignment create --scope <apiserver-subnet-resource-id> \
--role "Network Contributor" \
--assignee <managed-identity-client-id>

  3. Assign the Network Contributor role to the cluster subnet using the az role assignment create command.

    az role assignment create --scope <cluster-subnet-resource-id> \
--role "Network Contributor" \
--assignee <managed-identity-client-id>

Deploy a public cluster

  • Deploy a public AKS cluster with API Server VNet integration using the az aks create command with the --enable-api-server-vnet-integration flag.

    az aks create --name <cluster-name> \
    --resource-group <resource-group> \
    --location <location> \
    --network-plugin azure \
    --enable-apiserver-vnet-integration \
    --vnet-subnet-id <cluster-subnet-resource-id> \
    --apiserver-subnet-id <apiserver-subnet-resource-id> \
    --assign-identity <managed-identity-resource-id> \
    --generate-ssh-keys

Deploy a private cluster

  • Deploy a private AKS cluster with API Server VNet integration using the az aks create command with the --enable-api-server-vnet-integration and --enable-private-cluster flags.

    az aks create --name <cluster-name> \
--resource-group <resource-group> \
--location <location> \
--network-plugin azure \
--enable-private-cluster \
--enable-apiserver-vnet-integration \
--vnet-subnet-id <cluster-subnet-resource-id> \
--apiserver-subnet-id <apiserver-subnet-resource-id> \
--assign-identity <managed-identity-resource-id> \
--generate-ssh-keys

Convert an existing AKS cluster to API Server VNet Integration

You can convert existing public/private AKS clusters to API Server VNet Integration clusters by supplying an API server subnet that meets the requirements listed earlier. These requirements include: in the same VNet as the cluster nodes, permissions granted for the AKS cluster identity, not used by other resources like private endpoint, and size of at least /28. Converting your cluster is a one-way migration. Clusters can't have API Server VNet Integration disabled after it's been enabled.

This upgrade performs a node-image version upgrade on all node pools and restarts all workloads while they undergo a rolling image upgrade.

Warning

Converting a cluster to API Server VNet Integration results in a change of the API Server IP address, though the hostname remains the same. If the IP address of the API server has been configured in any firewalls or network security group rules, those rules may need to be updated.

  • Update your cluster to API Server VNet Integration using the az aks update command with the --enable-apiserver-vnet-integration flag.

    az aks update --name <cluster-name> \
--resource-group <resource-group> \
--enable-apiserver-vnet-integration \
--apiserver-subnet-id <apiserver-subnet-resource-id>

Enable or disable private cluster mode on an existing cluster with API Server VNet Integration

AKS clusters configured with API Server VNet Integration can have public network access/private cluster mode enabled or disabled without redeploying the cluster. The API server hostname doesn't change, but public DNS entries are modified or removed if necessary.

Note

--disable-private-cluster is currently in preview. For more information, see Reference and support levels.

Enable private cluster mode

  • Enable private cluster mode using the az aks update command with the --enable-private-cluster flag.

    az aks update --name <cluster-name> \
--resource-group <resource-group> \
--enable-private-cluster

Disable private cluster mode

  • Disable private cluster mode using the az aks update command with the --disable-private-cluster flag.

    az aks update --name <cluster-name> \
--resource-group <resource-group> \
--disable-private-cluster

Connect to cluster using kubectl

  • Configure kubectl to connect to your cluster using the az aks get-credentials command.

    az aks get-credentials --resource-group <resource-group> --name <cluster-name>

NSG security rules

All traffic within the VNet is allowed by default. But if you have added NSG rules to restrict traffic between different subnets, ensure that the NSG security rules permit the following types of communication:

Destination Source Protocol Port Use
APIServer Subnet CIDR Cluster Subnet TCP 443 and 4443 Required to enable communication between Nodes and the API server.
APIServer Subnet CIDR Azure Load Balancer TCP 9988 Required to enable communication between Azure Load Balancer and the API server. You can also enable all communicatios between the Azure Load Balancer and the API Server Subnet CIDR.

Next steps

For associated best practices, see Best practices for network connectivity and security in AKS.