Limit network traffic with Azure Firewall in Azure Kubernetes Service (AKS)

Learn how to use the Outbound network and FQDN rules for AKS clusters to control egress traffic using the Azure Firewall in AKS. To simplify this configuration, Azure Firewall provides an Azure Kubernetes Service (AzureKubernetesService) Fully Qualified Domain Name (FQDN) tag that restricts outbound traffic from the AKS cluster. This article shows how you can configure your AKS Cluster traffic rules through Azure firewall.

The following information provides an example architecture of the deployment:

• Public ingress is forced to flow through firewall filters
  • AKS agent nodes are isolated in a dedicated subnet
  • Azure Firewall is deployed in its own subnet
  • A DNAT rule translates the firewall public IP into the load balancer frontend IP
• Outbound requests start from agent nodes to the Azure Firewall internal IP using a user-defined route (UDR)
  • Requests from AKS agent nodes follow a UDR that has been placed on the subnet the AKS cluster was deployed into
  • Azure Firewall egresses out of the virtual network from a public IP frontend
  • Access to the public internet or other Azure services flows to and from the firewall frontend IP address
  • Access to the AKS control plane can be protected by API server authorized IP ranges, including the firewall public frontend IP address
• Internal traffic
  • You can use an internal load balancer for internal traffic, which you could isolate on its own subnet, instead of or alongside a public load balancer

Configure environment variables

Define a set of environment variables to be used in resource creations.

PREFIX="aks-egress"
RG="${PREFIX}-rg"
LOC="chinaeast2"
PLUGIN=azure
AKSNAME="${PREFIX}"
VNET_NAME="${PREFIX}-vnet"
AKSSUBNET_NAME="aks-subnet"
# DO NOT CHANGE FWSUBNET_NAME - This is currently a requirement for Azure Firewall.
FWSUBNET_NAME="AzureFirewallSubnet"
FWNAME="${PREFIX}-fw"
FWPUBLICIP_NAME="${PREFIX}-fwpublicip"
FWIPCONFIG_NAME="${PREFIX}-fwconfig"
FWROUTE_TABLE_NAME="${PREFIX}-fwrt"
FWROUTE_NAME="${PREFIX}-fwrn"
FWROUTE_NAME_INTERNET="${PREFIX}-fwinternet"

Create a virtual network with multiple subnets

Provision a virtual network with two separate subnets: one for the cluster and one for the firewall. Optionally, you can create one for internal service ingress.

1. Create a resource group using the az group create command.

   az group create --name $RG --location $LOC
   

2. Create a virtual network with two subnets to host the AKS cluster and the Azure Firewall using the az network vnet create and az network vnet subnet create commands.

   # Dedicated virtual network with AKS subnet
   az network vnet create \
       --resource-group $RG \
       --name $VNET_NAME \
       --location $LOC \
       --address-prefixes 10.42.0.0/16 \
       --subnet-name $AKSSUBNET_NAME \
       --subnet-prefix 10.42.1.0/24
   
   # Dedicated subnet for Azure Firewall (Firewall name can't be changed)
   az network vnet subnet create \
       --resource-group $RG \
       --vnet-name $VNET_NAME \
       --name $FWSUBNET_NAME \
       --address-prefix 10.42.2.0/24
   

Create and set up an Azure Firewall

You need to configure Azure Firewall inbound and outbound rules. The main purpose of the firewall is to enable organizations to configure granular ingress and egress traffic rules into and out of the AKS cluster.

1. Create a standard SKU public IP resource using the az network public-ip create command. This resource will be used as the Azure Firewall frontend address.

   az network public-ip create --resource-group $RG -n $FWPUBLICIP_NAME --location $LOC --sku "Standard"
   

2. Register the Azure Firewall CLI extension to create an Azure Firewall using the az extension add command.

   az extension add --name azure-firewall
   

3. Create an Azure Firewall and enable DNS proxy using the az network firewall create command and setting the --enable-dns-proxy to true.

   az network firewall create --resource-group $RG --name $FWNAME --location $LOC --enable-dns-proxy true
   

Setting up the public IP address to the Azure Firewall may take a few minutes. Once it's ready, the IP address created earlier can be assigned to the firewall front end.

4. Create an Azure Firewall IP configuration using the az network firewall ip-config create command.

   az network firewall ip-config create --resource-group $RG --firewall-name $FWNAME --name $FWIPCONFIG_NAME --public-ip-address $FWPUBLICIP_NAME --vnet-name $VNET_NAME
   

5. Once the previous command succeeds, save the firewall frontend IP address for configuration later.

   FWPUBLIC_IP=$(az network public-ip show --resource-group $RG --name $FWPUBLICIP_NAME --query "ipAddress" -o tsv)
   FWPRIVATE_IP=$(az network firewall show --resource-group $RG --name $FWNAME --query "ipConfigurations[0].privateIPAddress" -o tsv)
   

Create a route with a hop to Azure Firewall

Azure automatically routes traffic between Azure subnets, virtual networks, and on-premises networks. If you want to change any of Azure's default routing, you can create a route table.

1. Create an empty route table to be associated with a given subnet using the az network route-table create command. The route table will define the next hop as the Azure Firewall created above. Each subnet can have zero or one route table associated to it.

   az network route-table create --resource-group $RG --location $LOC --name $FWROUTE_TABLE_NAME
   

2. Create routes in the route table for the subnets using the az network route-table route create command.

   az network route-table route create --resource-group $RG --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP
   
   az network route-table route create --resource-group $RG --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME --address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet
   

For information on how to override Azure's default system routes or add additional routes to a subnet's route table, see the virtual network route table documentation.

Add firewall rules

This section covers three network rules and an application rule you can use to configure on your firewall. You may need to adapt these rules based on your deployment.

• The first network rule allows access to port 9000 via TCP.
• The second network rule allows access to port 1194 and 123 via UDP. If you're deploying to Azure operated by 21Vianet, see the Azure operated by 21Vianet required network rules. Both these rules will only allow traffic destined to the Azure Region CIDR in this article, which is China East.
• The third network rule opens port 123 to ntp.ubuntu.com FQDN via UDP. Adding an FQDN as a network rule is one of the specific features of Azure Firewall, so you'll need to adapt it when using your own options.
• The fourth and fifth network rules allow access to pull containers from GitHub Container Registry (ghcr.io) and Docker Hub (docker.io).

1. Create the network rules using the az network firewall network-rule create command.

   az network firewall network-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwnr' --name 'apiudp' --protocols 'UDP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 1194 --action allow --priority 100
   
   az network firewall network-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwnr' --name 'apitcp' --protocols 'TCP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 9000
   
   az network firewall network-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwnr' --name 'time' --protocols 'UDP' --source-addresses '*' --destination-fqdns 'ntp.ubuntu.com' --destination-ports 123
   
   az network firewall network-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwnr' --name 'ghcr' --protocols 'TCP' --source-addresses '*' --destination-fqdns ghcr.io pkg-containers.githubusercontent.com --destination-ports '443'
   
   az network firewall network-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwnr' --name 'docker' --protocols 'TCP' --source-addresses '*' --destination-fqdns docker.io registry-1.docker.io production.cloudflare.docker.com --destination-ports '443'
   

2. Create the application rule using the az network firewall application-rule create command.

   az network firewall application-rule create --resource-group $RG --firewall-name $FWNAME --collection-name 'aksfwar' --name 'fqdn' --source-addresses '*' --protocols 'http=80' 'https=443' --fqdn-tags "AzureKubernetesService" --action allow --priority 100
   

To learn more about Azure Firewall, see the Azure Firewall documentation.

Associate the route table to AKS

To associate the cluster with the firewall, the dedicated subnet for the cluster's subnet must reference the route table created above. Use the az network vnet subnet update command to associate the route table to AKS.

az network vnet subnet update --resource-group $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --route-table $FWROUTE_TABLE_NAME

Deploy an AKS cluster that follows your outbound rules

Now, you can deploy an AKS cluster into the existing virtual network. You will use the userDefinedRouting outbound type, which ensures that any outbound traffic is forced through the firewall and no other egress paths will exist. The loadBalancer outbound type can also be used.

The target subnet to be deployed into is defined with the environment variable, $SUBNETID. Set the value for the subnet ID using the following command:

SUBNETID=$(az network vnet subnet show --resource-group $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --query id -o tsv)

You'll define the outbound type to use the UDR that already exists on the subnet. This configuration will enable AKS to skip the setup and IP provisioning for the load balancer.

az aks create --resource-group $RG --name $AKSNAME --location $LOC \
    --node-count 3 \
    --network-plugin azure \
    --outbound-type userDefinedRouting \
    --vnet-subnet-id $SUBNETID \
    --api-server-authorized-ip-ranges $FWPUBLIC_IP \
    --generate-ssh-keys

az aks create \
    --resource-group $RG \
    --name $AKSNAME \
    --location $LOC \
    --node-count 3 \
    --network-plugin kubenet \
    --outbound-type userDefinedRouting \
    --vnet-subnet-id $SUBNETID \
    --api-server-authorized-ip-ranges $FWPUBLIC_IP
    --assign-identity <identity-resource-id> \
    --assign-kubelet-identity <kubelet-identity-resource-id> \
    --generate-ssh-keys

Enable developer access to the API server

If you used authorized IP ranges for your cluster in the previous step, you need to add your developer tooling IP addresses to the AKS cluster list of approved IP ranges so you access the API server from there. You can also configure a jumpbox with the needed tooling inside a separate subnet in the firewall's virtual network.

1. Retrieve your IP address using the following command:

   CURRENT_IP=$(dig @resolver1.opendns.com ANY myip.opendns.com +short)
   

2. Add the IP address to the approved ranges using the az aks update command.

   az aks update --resource-group $RG --name $AKSNAME --api-server-authorized-ip-ranges $CURRENT_IP/32
   

3. Configure kubectl to connect to your AKS cluster using the az aks get-credentials command.

   az aks get-credentials --resource-group $RG --name $AKSNAME
   

Deploy a public service on AKS

You can now start exposing services and deploying applications to this cluster. In this example, we'll expose a public service, but you also might want to expose an internal service using an internal load balancer.

1. Review the AKS Store Demo quickstart manifest to see all the resources that will be created.

2. Deploy the service using the kubectl apply command.

   kubectl apply -f https://raw.githubusercontent.com/Azure-Samples/aks-store-demo/main/aks-store-quickstart.yaml
   

Allow inbound traffic through Azure Firewall

To configure inbound connectivity, you need to write a DNAT rule to the Azure Firewall. To test connectivity to your cluster, a rule is defined for the firewall frontend public IP address to route to the internal IP exposed by the internal service. The destination address can be customized. The translated address must be the IP address of the internal load balancer. The translated port must be the exposed port for your Kubernetes service. You also need to specify the internal IP address assigned to the load balancer created by the Kubernetes service.

1. Get the internal IP address assigned to the load balancer using the kubectl get services command.

   kubectl get services
   

   The IP address will be listed in the EXTERNAL-IP column, as shown in the following example output:

   NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)              AGE
   kubernetes        ClusterIP      10.0.0.1       <none>        443/TCP              9m10s
   order-service     ClusterIP      10.0.104.144   <none>        3000/TCP             11s
   product-service   ClusterIP      10.0.237.60    <none>        3002/TCP             10s
   rabbitmq          ClusterIP      10.0.161.128   <none>        5672/TCP,15672/TCP   11s
   store-front       LoadBalancer   10.0.89.139    20.39.18.6    80:32271/TCP         10s
   

2. Get the service IP using the kubectl get svc store-front command.

   SERVICE_IP=$(kubectl get svc store-front -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
   

3. Add the NAT rule using the az network firewall nat-rule create command.

   az network firewall nat-rule create --collection-name exampleset --destination-addresses $FWPUBLIC_IP --destination-ports 80 --firewall-name $FWNAME --name inboundrule --protocols Any --resource-group $RG --source-addresses '*' --translated-port 80 --action Dnat --priority 100 --translated-address $SERVICE_IP
   

Validate connectivity

Navigate to the Azure Firewall frontend IP address in a browser to validate connectivity.

You should see the AKS store app. In this example, the firewall public IP was 52.253.228.132.

On this page, you can view products, add them to your cart, and then place an order.

Clean up resources

To clean up Azure resources, delete the AKS resource group using the az group delete command.

az group delete --name $RG

Next steps

In this article, you learned how to secure your outbound traffic using Azure Firewall. If needed, you can generalize the steps above to forward the traffic to your preferred egress solution following the Outbound Type userDefinedRoute documentation.