Install an Application Gateway Ingress Controller (AGIC) using an existing Application Gateway

Article
09/29/2024

The Application Gateway Ingress Controller (AGIC) is a pod within your Azure Kubernetes Service (AKS) cluster. AGIC monitors the Kubernetes Ingress resources, and creates and applies Application Gateway config based on the status of the Kubernetes cluster.

Outline

Prerequisites
Azure Resource Manager authentication
- Option 1: Set up Microsoft Entra Workload ID and create Azure Identity on ARMs
- Option 2: Set up a Service Principal
Install Ingress Controller using Helm
Shared Application Gateway: Install AGIC in an environment, where Application Gateway is shared between one AKS cluster and/or other Azure components.

Prerequisites

This document assumes you already have the following tools and infrastructure installed:

An AKS cluster with Azure Container Networking Interface (CNI)
Application Gateway v2 in the same virtual network as the AKS cluster
Microsoft Entra Workload ID configured for your AKS cluster

Backup your Application Gateway's configuration before installing AGIC:

From the Azure portal, navigate to your Application Gateway instance.
Under the Automation section, select Export template and then select Download.

The zip file you downloaded contains JSON templates, bash, and PowerShell scripts you could use to restore App Gateway should that become necessary

Install Helm

Helm is a package manager for Kubernetes, used to install the application-gateway-kubernetes-ingress package.

Install Helm and run the following:

Kubernetes RBAC enabled AKS cluster

kubectl create serviceaccount --namespace kube-system tiller-sa
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
helm init --tiller-namespace kube-system --service-account tiller-sa

Azure Resource Manager Authentication

AGIC communicates with the Kubernetes API server and the Azure Resource Manager. It requires an identity to access these APIs.

Set up Microsoft Entra Workload ID

Microsoft Entra Workload ID is an identity you assign to a software workload, to authenticate and access other services and resources. This identity enables your AKS pod to use this identity and authenticate with other Azure resources. For this configuration, we need authorization for the AGIC pod to make HTTP requests to ARM.

Use the Azure CLI az account set command to set a specific subscription to be the current active subscription. Then use the az identity create command to create a managed identity. The identity needs to be created in the node resource group. The node resource group is assigned a name by default, such as MC_myResourceGroup_myAKSCluster_chinanorth2.
```
az account set --subscription "subscriptionID"
```
```
az identity create --name "userAssignedIdentityName" --resource-group "resourceGroupName" --location "location" --subscription "subscriptionID"
```

For the role assignment, run the following command to identify the principalId for the newly created identity:

$resourceGroup="resource-group-name"
$identityName="identity-name"
az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv

Grant the identity Contributor access to your Application Gateway. You need the ID of the Application Gateway, which looks like: /subscriptions/A/resourceGroups/B/providers/Microsoft.Network/applicationGateways/C. First, get the list of Application Gateway IDs in your subscription by running the following command:

az network application-gateway list --query '[].id'

To assign the identity Contributor access, run the following command:

$resourceGroup="resource-group-name"
$identityName="identity-Name"
# Get the Application Gateway ID
$AppGatewayID=$(az network application-gateway list --query '[].id' -o tsv)
$role="contributor"
# Get the principal ID for the User assigned identity
$principalId=$(az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv)
az role assignment create --assignee $principalId --role $role --scope $AppGatewayID

Grant the identity Reader access to the Application Gateway resource group. The resource group ID looks like: /subscriptions/A/resourceGroups/B. You can get all resource groups with: az group list --query '[].id'

$resourceGroup="resource-group-name"
$identityName="identity-Name"
# Get the Application Gateway resource group
$AppGatewayResourceGroup=$(az network application-gateway list --query '[].resourceGroup' -o tsv)
# Get the Application Gateway resource group ID
$AppGatewayResourceGroupID=$(az group show --name $AppGatewayResourceGroup --query id -o tsv)
$role="Reader"
# Get the principal ID for the User assigned identity
$principalId=$(az identity list -g $resourceGroup --query "[?name == '$identityName'].principalId | [0]" -o tsv)
# Assign the Reader role to the User assigned identity at the resource group scope
az role assignment create --role $role --assignee $principalId  --scope $AppGatewayResourceGroupID

Note

Please ensure the identity used by AGIC has the Microsoft.Network/virtualNetworks/subnets/join/action permission delegated to the subnet where Application Gateway is deployed. If a custom role is not defined with this permission, you can use the built-in Network Contributor role, which contains the Microsoft.Network/virtualNetworks/subnets/join/action permission.

Using a Service Principal

It's also possible to provide AGIC access to ARM using a Kubernetes secret.

Create an Active Directory Service Principal and encode with base64. The base64 encoding is required for the JSON blob to be saved to Kubernetes.
```
az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0
```
Add the base64 encoded JSON blob to the helm-config.yaml file. More information on helm-config.yaml is in the next section.
```
armAuth:
    type: servicePrincipal
    secretJSON: <Base64-Encoded-Credentials>
```

Deploy the Azure Application Gateway Ingress Controller Add-on

Create an Ingress Controller deployment manifest

---
# file: pet-supplies-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pet-supplies-ingress
spec:
  ingressClassName: azure-application-gateway
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: store-front
            port:
              number: 80
      - path: /order-service
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 3000
      - path: /product-service
        pathType: Prefix
        backend:
          service:
            name: product-service
            port:
              number: 3002

Deploy Ingress Controller

$namespace="namespace"
$file="pet-supplies-ingress.yaml"
kubectl apply -f $file -n $namespace

Install Ingress Controller as a Helm Chart

In the first few steps, we installed Helm's Tiller on your Kubernetes cluster. Install the AGIC Helm package:

Perform a helm update
```
helm repo update
```

Download helm-config.yaml, which configures AGIC:

wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml

Or copy the following YAML file:

# This file contains the essential configs for the ingress controller helm chart

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

################################################################################
# Specify which application gateway the ingress controller must manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: <resourceGroupName>
    name: <applicationGatewayName>
    environment: AzureChinaCloud

    # Setting appgw.shared to "true" creates an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false

################################################################################
# Specify which kubernetes namespace the ingress controller must watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all accessible namespaces.
#
# kubernetes:
#   watchNamespace: <namespace>

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: Azure-AD-workload-identity
armAuth:
    type: workloadIdentity
    identityClientID:  <identityClientId>

## Alternatively you can use Service Principal credentials
# armAuth:
#    type: servicePrincipal
#    secretJSON: <<Generate this value with: "az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0" >>

################################################################################
# Specify if the cluster is Kubernetes RBAC enabled or not
rbac:
    enabled: false # true/false

# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
    apiServerAddress: <aks-api-server-address>

Edit helm-config.yaml and fill in the values for appgw and armAuth.

Note

The <identity-client-id> is a property of the Microsoft Entra Workload ID you setup in the previous section. You can retrieve this information by running the following command: az identity show -g <resourcegroup> -n <identity-name>, where <resourcegroup> is the resource group hosting the infrastructure resources related to the AKS cluster, Application Gateway and managed identity.

Install Helm chart with the helm-config.yaml configuration from the previous step

helm install agic-controller oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure --version 1.7.5 -f helm-config.yaml

Alternatively you can combine the helm-config.yaml and the Helm command in one step:

helm install oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure \
     --name agic-controller \
     --version 1.7.5 \
     --namespace default \
     --debug \
     --set appgw.name=applicationgatewayABCD \
     --set appgw.resourceGroup=your-resource-group \
     --set appgw.subscriptionId=subscription-uuid \
     --set appgw.environment=AzureChinaCloud \
     --set appgw.shared=false \
     --set armAuth.type=servicePrincipal \
     --set armAuth.secretJSON=$(az ad sp create-for-rbac --role Contributor --sdk-auth | base64 -w0) \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set kubernetes.watchNamespace=default \
     --set aksClusterConfiguration.apiServerAddress=aks-abcdefg.hcp.chinanorth2.azmk8s.io

Check the log of the newly created pod to verify if it started properly

Refer to this how-to guide to understand how you can expose an AKS service over HTTP or HTTPS, to the internet, using an Azure Application Gateway.

Shared Application Gateway

By default AGIC assumes full ownership of the Application Gateway it's linked to. AGIC version 0.8.0 and later can share a single Application Gateway with other Azure components. For instance, we could use the same Application Gateway for an app hosted on Virtual Machine Scale Set and an AKS cluster.

Backup your Application Gateway's configuration before enabling this setting:

From the Azure portal, navigate to your Application Gateway instance
Under the Automation section, select Export template and then select Download.

The zip file you downloaded contains JSON templates, bash, and PowerShell scripts you could use to restore Application Gateway

Example Scenario

Let's look at an imaginary Application Gateway, which manages traffic for two web sites:

dev.contoso.com - hosted on a new AKS cluster, using Application Gateway and AGIC
prod.contoso.com - hosted on an Azure Virtual Machine Scale Set

With default settings, AGIC assumes 100% ownership of the Application Gateway it's pointed to. AGIC overwrites all of App Gateway's configuration. If you manually create a listener for prod.contoso.com (on Application Gateway) without defining it in the Kubernetes Ingress, AGIC deletes the prod.contoso.com config within seconds.

To install AGIC and also serve prod.contoso.com from our Virtual Machine Scale Set machines, we must constrain AGIC to configuring dev.contoso.com only. This is facilitated by instantiating the following CRD:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: prod-contoso-com
spec:
  hostname: prod.contoso.com
EOF

The command above creates an AzureIngressProhibitedTarget object. This makes AGIC (version 0.8.0 and later) aware of the existence of Application Gateway config for prod.contoso.com and explicitly instructs it to avoid changing any configuration related to that hostname.

Enable with new AGIC installation

To limit AGIC (version 0.8.0 and later) to a subset of the Application Gateway configuration, modify the helm-config.yaml template. Under the appgw: section, add shared key and set it to true.

appgw:
    subscriptionId: <subscriptionId>    # existing field
    resourceGroup: <resourceGroupName>  # existing field
    name: <applicationGatewayName>      # existing field
    environment: AzureChinaCloud        # existing field
    shared: true                        # <<<<< Add this field to enable shared Application Gateway >>>>>

Apply the Helm changes:

Ensure the AzureIngressProhibitedTarget CRD is installed with:

kubectl apply -f https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/7b55ad194e7582c47589eb9e78615042e00babf3/crds/AzureIngressProhibitedTarget-v1-CRD-v1.yaml

Update Helm:

helm upgrade \
    --recreate-pods \
    -f helm-config.yaml \
    agic-controller
    oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure

As a result, your AKS cluster has a new instance of AzureIngressProhibitedTarget called prohibit-all-targets:

kubectl get AzureIngressProhibitedTargets prohibit-all-targets -o yaml

The object prohibit-all-targets, as the name implies, prohibits AGIC from changing config for any host and path. Helm install with appgw.shared=true deploys AGIC, but doesn't make any changes to Application Gateway.

Broaden permissions

Since Helm with appgw.shared=true and the default prohibit-all-targets blocks AGIC from applying a config, broaden AGIC permissions:

Create a new YAML file named AzureIngressProhibitedTarget with the following snippet containing your specific setup:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: your-custom-prohibitions
spec:
  hostname: your.own-hostname.com
EOF

Only after you have created your own custom prohibition, you can delete the default one, which is too broad:
```
kubectl delete AzureIngressProhibitedTarget prohibit-all-targets
```

Enable for an existing AGIC installation

Let's assume that we already have a working AKS cluster, Application Gateway, and configured AGIC in our cluster. We have an Ingress for prod.contoso.com and are successfully serving traffic for it from the cluster. We want to add staging.contoso.com to our existing Application Gateway, but need to host it on a VM. We're going to reuse the existing Application Gateway and manually configure a listener and backend pools for staging.contoso.com. But manually tweaking Application Gateway config (using portal, ARM APIs or Terraform) would conflict with AGIC's assumptions of full ownership. Shortly after we apply changes, AGIC overwrites or deletes them.

We can prohibit AGIC from making changes to a subset of configuration.

Create a new YAML file named AzureIngressProhibitedTarget with the following snippet:

cat <<EOF | kubectl apply -f -
apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: manually-configured-staging-environment
spec:
  hostname: staging.contoso.com
EOF

View the newly created object:

kubectl get AzureIngressProhibitedTargets

Modify Application Gateway config from the Azure portal - add listeners, routing rules, backends etc. The new object we created (manually-configured-staging-environment) prohibits AGIC from overwriting Application Gateway configuration related to staging.contoso.com.