Create an Application Gateway Ingress Controller (AGIC) in Azure Kubernetes Service (AKS) using Terraform

Azure Kubernetes Service (AKS) manages your hosted Kubernetes environment. AKS makes it quick and easy to deploy and manage containerized applications without container orchestration expertise. AKS also eliminates the burden of taking applications offline for operational and maintenance tasks. With AKS, you can provision, upgrade, and scale resources on-demand.

Azure Application Gateway provides Application Gateway Ingress Controller (AGIC). AGIC enables various features for Kubernetes services, including reverse proxy, configurable traffic routing, and TLS termination. Kubernetes Ingress resources help configure the ingress rules for individual Kubernetes services. An ingress controller allows a single IP address to route traffic to multiple services in a Kubernetes cluster.

Terraform enables the definition, preview, and deployment of cloud infrastructure. Using Terraform, you create configuration files using HCL syntax. The HCL syntax allows you to specify the cloud provider - such as Azure - and the elements that make up your cloud infrastructure. After you create your configuration files, you create an execution plan that allows you to preview your infrastructure changes before they're deployed. Once you verify the changes, you apply the execution plan to deploy the infrastructure.

In this article, you learn how to:

Before you get started, you need to install and configure the following tools:

• Terraform
• kubectl command-line tool
• Helm package manager
• GNU wget command-line tool

1. Create a directory to test sample Terraform code and make it your working directory.
2. Create a file named providers.tf and copy in the following code:

terraform {
  required_version = ">=1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

3. Create a file named main.tf and copy in the following code:

    resource "random_pet" "rg_name" {
      prefix = var.resource_group_name_prefix
    }
   
    resource "azurerm_resource_group" "rg" {
      name     = random_pet.rg_name.id
      location = var.resource_group_location
    }
   
    # Locals block for hardcoded names
    locals {
      backend_address_pool_name      = "${azurerm_virtual_network.vnet.name}-beap"
      frontend_port_name             = "${azurerm_virtual_network.vnet.name}-feport"
      frontend_ip_configuration_name = "${azurerm_virtual_network.vnet.name}-feip"
      http_setting_name              = "${azurerm_virtual_network.vnet.name}-be-htst"
      listener_name                  = "${azurerm_virtual_network.vnet.name}-httplstn"
      request_routing_rule_name      = "${azurerm_virtual_network.vnet.name}-rqrt"
    }
   
    # Subnets
    data "azurerm_subnet" "kubesubnet" {
      name                 = var.aks_subnet_name
      virtual_network_name = azurerm_virtual_network.vnet.name
      resource_group_name  = azurerm_resource_group.rg.name
    }
   
    data "azurerm_subnet" "appgwsubnet" {
      name                 = var.appgw_subnet_name
      virtual_network_name = azurerm_virtual_network.vnet.name
      resource_group_name  = azurerm_resource_group.rg.name
    }
   
    data "azurerm_user_assigned_identity" "ingress" {
      name                = "ingressapplicationgateway-${azurerm_kubernetes_cluster.aks.name}"
      resource_group_name = azurerm_kubernetes_cluster.aks.node_resource_group
    }
   
    # Virtual network (vnet)
    resource "azurerm_virtual_network" "vnet" {
      name                = var.virtual_network_name
      location            = azurerm_resource_group.rg.location
      resource_group_name = azurerm_resource_group.rg.name
      address_space       = [var.virtual_network_address_prefix]
   
      subnet {
        name           = var.aks_subnet_name
        address_prefix = var.aks_subnet_address_prefix
      }
   
      subnet {
        name           = var.appgw_subnet_name
        address_prefix = var.app_gateway_subnet_address_prefix
      }
    }
   
    resource "azurerm_user_assigned_identity" "aks" {
      name                = "aks-${var.aks_cluster_name}"
      resource_group_name = azurerm_resource_group.rg.name
      location            = azurerm_resource_group.rg.location
    }
   
    # AKS cluster
    resource "azurerm_kubernetes_cluster" "aks" {
      name                              = var.aks_cluster_name
      location                          = azurerm_resource_group.rg.location
      resource_group_name               = azurerm_resource_group.rg.name
      dns_prefix                        = var.aks_cluster_name
      private_cluster_enabled           = var.aks_private_cluster
      role_based_access_control_enabled = var.aks_enable_rbac
      sku_tier                          = var.aks_sku_tier
   
      default_node_pool {
        name            = "agentpool"
        node_count      = var.aks_node_count
        vm_size         = var.aks_vm_size
        os_disk_size_gb = var.aks_os_disk_size
        max_pods        = 100
        vnet_subnet_id  = data.azurerm_subnet.kubesubnet.id
      }
   
      identity {
        type         = "UserAssigned"
        identity_ids = [azurerm_user_assigned_identity.aks.id]
      }
   
   
      network_profile {
        network_plugin = "azure"
        dns_service_ip = var.aks_dns_service_ip
        service_cidr   = var.aks_service_cidr
      }
   
      ingress_application_gateway {
        gateway_id = azurerm_application_gateway.appgw.id
      }
   
      depends_on = [
        azurerm_application_gateway.appgw
      ]
    }
   
    resource "azurerm_public_ip" "pip" {
      name                = "appgw-pip"
      resource_group_name = azurerm_resource_group.rg.name
      location            = azurerm_resource_group.rg.location
      allocation_method   = "Static"
      sku                 = "Standard"
    }
   
    resource "azurerm_application_gateway" "appgw" {
      name                = var.app_gateway_name
      resource_group_name = azurerm_resource_group.rg.name
      location            = azurerm_resource_group.rg.location
   
      sku {
        name     = var.app_gateway_tier
        tier     = var.app_gateway_tier
        capacity = 1
      }
   
      gateway_ip_configuration {
        name      = "appGatewayIpConfig"
        subnet_id = data.azurerm_subnet.appgwsubnet.id
      }
   
      frontend_port {
        name = local.frontend_port_name
        port = 80
      }
   
      frontend_ip_configuration {
        name                 = local.frontend_ip_configuration_name
        public_ip_address_id = azurerm_public_ip.pip.id
      }
   
      backend_address_pool {
        name = local.backend_address_pool_name
      }
   
      backend_http_settings {
        name                  = local.http_setting_name
        cookie_based_affinity = "Disabled"
        port                  = 80
        protocol              = "Http"
        request_timeout       = 1
      }
   
      http_listener {
        name                           = local.listener_name
        frontend_ip_configuration_name = local.frontend_ip_configuration_name
        frontend_port_name             = local.frontend_port_name
        protocol                       = "Http"
      }
   
      request_routing_rule {
        name                       = local.request_routing_rule_name
        priority                   = 1
        rule_type                  = "Basic"
        http_listener_name         = local.listener_name
        backend_address_pool_name  = local.backend_address_pool_name
        backend_http_settings_name = local.http_setting_name
      }
   
      # Since this sample is creating an Application Gateway 
      # that is later managed by an Ingress Controller, there is no need 
      # to create a backend address pool (BEP). However, the BEP is still 
      # required by the resource. Therefore, "lifecycle:ignore_changes" is 
      # used to prevent TF from managing the gateway.
      lifecycle {
        ignore_changes = [
          tags,
          backend_address_pool,
          backend_http_settings,
          http_listener,
          probe,
          request_routing_rule,
        ]
      }
    }
   
    # Role assignments
    resource "azurerm_role_assignment" "ra1" {
      scope                = azurerm_resource_group.rg.id
      role_definition_name = "Reader"
      principal_id         = data.azurerm_user_assigned_identity.ingress.principal_id
    }
   
    resource "azurerm_role_assignment" "ra2" {
      scope                = azurerm_virtual_network.vnet.id
      role_definition_name = "Network Contributor"
      principal_id         = data.azurerm_user_assigned_identity.ingress.principal_id
    }
   
    resource "azurerm_role_assignment" "ra3" {
      scope                = azurerm_application_gateway.appgw.id
      role_definition_name = "Contributor"
      principal_id         = data.azurerm_user_assigned_identity.ingress.principal_id
    }
   

4. Create a file named variables.tf and copy in the following code:

    variable "resource_group_location" {
      type        = string
      default     = "eastus"
      description = "Location for all resources."
    }
   
    variable "resource_group_name_prefix" {
      type        = string
      default     = "rg"
      description = "Prefix of the resource group name that's combined with a random value so name is unique in your Azure subscription."
    }
   
    variable "virtual_network_name" {
      type        = string
      description = "Virtual network name."
      default     = "aksVirtualNetwork"
    }
   
    variable "virtual_network_address_prefix" {
      type        = string
      description = "VNET address prefix."
      default     = "10.1.0.0/18"
    }
   
    variable "aks_subnet_name" {
      type        = string
      description = "Name of the subset."
      default     = "akssubnet"
    }
   
    variable "appgw_subnet_name" {
      type        = string
      description = "Name of the subset."
      default     = "appgwsubnet"
    }
   
    variable "aks_cluster_name" {
      type        = string
      description = "The name of the Managed Kubernetes Cluster to create."
      default     = "aks-cluster"
    }
   
    variable "aks_os_disk_size" {
      type        = number
      description = "(Optional) The size of the OS Disk which should be used for each agent in the Node Pool."
      default     = 50
    }
   
    variable "aks_node_count" {
      type        = number
      description = "(Optional) The initial number of nodes which should exist in this Node Pool."
      default     = 3
    }
   
    variable "aks_sku_tier" {
      type        = string
      description = "(Optional) The SKU tier that should be used for this Kubernetes Cluster. Possible values are Free and Paid (which includes the Uptime SLA)."
      default     = "Free"
      validation {
        condition     = contains(["Free", "Paid"], var.aks_sku_tier)
        error_message = "Invalid SKU tier. The value should be one of the following: 'Free','Paid'."
      }
    }
   
    variable "aks_vm_size" {
      type        = string
      description = "The size of the virtual machine."
      default     = "Standard_D3_v2"
    }
   
    variable "kubernetes_version" {
      type        = string
      description = "(Optional) Version of Kubernetes specified when creating the AKS managed cluster."
      default     = "1.19.11"
    }
   
    variable "aks_service_cidr" {
      type        = string
      description = "(Optional) The Network Range used by the Kubernetes service."
      default     = "192.168.0.0/20"
    }
   
    variable "aks_dns_service_ip" {
      type        = string
      description = "(Optional) IP address within the Kubernetes service address range that will be used by cluster service discovery (kube-dns)."
      default     = "192.168.0.10"
    }
   
    variable "aks_private_cluster" {
      type        = bool
      description = "(Optional) Should this Kubernetes Cluster have its API server only exposed on internal IP addresses? This provides a Private IP Address for the Kubernetes API on the Virtual Network where the Kubernetes Cluster is located."
      default     = false
    }
   
    variable "aks_subnet_address_prefix" {
      description = "Subnet address prefix."
      type        = string
      default     = "10.1.0.0/22"
    }
   
    variable "app_gateway_subnet_address_prefix" {
      type        = string
      description = "Subnet address prefix."
      default     = "10.1.4.0/24"
    }
   
    variable "app_gateway_name" {
      description = "Name of the Application Gateway"
      type        = string
      default     = "ApplicationGateway1"
    }
   
    variable "app_gateway_tier" {
      description = "Tier of the Application Gateway tier."
      type        = string
      default     = "Standard_v2"
    }
   
    variable "aks_enable_rbac" {
      description = "(Optional) Is Role Based Access Control based on Azure AD enabled?"
      type        = bool
      default     = false
    }
   

5. Create a file named outputs.tf and copy in the following code:

    output "resource_group_name" {
      value = azurerm_resource_group.rg.name
    }
   
    output "aks_cluster_name" {
      value = azurerm_kubernetes_cluster.aks.name
    }
   
    output "application_gateway_name" {
      value = azurerm_application_gateway.appgw.name
    }
   
    output "identity_name" {
      value = azurerm_user_assigned_identity.aks.name
    }
   
    output "identity_resource_id" {
      value = azurerm_user_assigned_identity.aks.id
    }
   
    output "identity_client_id" {
      value = azurerm_user_assigned_identity.aks.client_id
    }
   
    output "application_ip_address" {
      value = azurerm_public_ip.pip.ip_address
    }
   
    output "client_key" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.client_key
      sensitive = true
    }
   
    output "client_certificate" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate
      sensitive = true
    }
   
    output "cluster_ca_certificate" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate
      sensitive = true
    }
   
    output "cluster_username" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.username
      sensitive = true
    }
   
    output "cluster_password" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.password
      sensitive = true
    }
   
    output "kube_config" {
      value     = azurerm_kubernetes_cluster.aks.kube_config_raw
      sensitive = true
    }
   
    output "host" {
      value     = azurerm_kubernetes_cluster.aks.kube_config.0.host
      sensitive = true
    }
   

Run terraform init to initialize the Terraform deployment. This command downloads the Azure provider required to manage your Azure resources.

terraform init -upgrade

Key points:

• The -upgrade parameter upgrades the necessary provider plugins to the newest version that complies with the configuration's version constraints.

Run terraform plan to create an execution plan.

terraform plan -out main.tfplan

Key points:

• The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
• The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
• To read more about persisting execution plans and security, see the security warning section.

Run terraform apply to apply the execution plan to your cloud infrastructure.

terraform apply main.tfplan

Key points:

• The example terraform apply command assumes you previously ran terraform plan -out main.tfplan.
• If you specified a different filename for the -out parameter, use that same filename in the call to terraform apply.
• If you didn't use the -out parameter, call terraform apply without any parameters.

1. Get the Azure resource group name.

   resource_group_name=$(terraform output -raw resource_group_name)
   

2. Get the AKS cluster name.

   aks_cluster_name=$(terraform output -raw aks_cluster_name)
   

3. Get the Kubernetes configuration and access credentials for the cluster using the az aks get-credentials command.

   az aks get-credentials \
       --name $aks_cluster_name  \
       --resource-group $resource_group_name \
       --overwrite-existing
   

4. Verify the health of the cluster using the kubectl get command.

   kubectl get nodes
   

   Key points:

   • The details of your worker nodes display with a status of Ready.

   

Azure Active Directory (Azure AD) Pod Identity provides token-based access to Azure Resource Manager.

Azure AD Pod Identity adds the following components to your Kubernetes cluster:

• Kubernetes CRDs: AzureIdentity, AzureAssignedIdentity, AzureIdentityBinding
• Managed Identity Controller (MIC) component
• Node Managed Identity (NMI) component

To install Azure AD Pod Identity on your cluster, you need to know if RBAC is enabled or disabled. RBAC is disabled by default for this demo. Enabling or disabling RBAC is done in the variables.tf file via the aks_enable_rbac block's default value.

• If RBAC is enabled, run the following kubectl create command.

  kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
  

• If RBAC is disabled, run the following kubectl create command.

  kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml
  

1. Add the AGIC Helm repo using the helm repo add command.

   helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.chinacloudapi.cn/ingress-azure-helm-package/
   

2. Update the AGIC Helm repo using the helm repo update command.

   helm repo update
   

1. Download helm-config.yaml to configure AGIC using the wget command.

   wget https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/sample-helm-config.yaml -O helm-config.yaml
   

2. Open helm-config.yaml in a text editor.

3. Enter the following value for the top level keys:

   • verbosityLevel: Specify the verbosity level of the AGIC logging infrastructure. For more information about logging levels, see logging Levels section of Application Gateway Kubernetes Ingress.

4. Enter the following values for the appgw block:

   • appgw.subscriptionId: Specify the Azure subscription ID used to create the App Gateway.
   • appgw.resourceGroup: Get the resource group name using the echo "$(terraform output -raw resource_group_name)" command.
   • appgw.name: Get the Application Gateway name using the echo "$(terraform output -raw application_gateway_name)" command.
   • appgw.shared: This boolean flag defaults to false. Set it to true if you need a Shared App Gateway.

5. Enter the following value for the kubernetes block:

   • kubernetes.watchNamespace: Specify the name space, which AGIC should watch. The namespace can be a single string value or a comma-separated list of namespaces. Leaving this variable commented out or setting it to a blank or an empty string results in the Ingress controller observing all accessible namespaces.

6. Enter the following values for the armAuth block:

   • If you specify armAuth.type as aadPodIdentity:

     • armAuth.identityResourceID: Get the Identity resource ID by running echo "$(terraform output -raw identity_resource_id)".
     • armAuth.identityClientId: Get the Identity client ID by running echo "$(terraform output -raw identity_client_id)".

   • If you specify armAuth.type as servicePrincipal, see Using a service principal.

1. Install the AGIC package using the helm install command.

   helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --generate-name
   

2. Get the Azure resource group name.

   resource_group_name=$(terraform output -raw resource_group_name)
   

3. Get the identity name.

   identity_name=$(terraform output -raw identity_name)
   

4. Get the key values from your identity using the az identity show command.

   az identity show -g $resource_group_name -n $identity_name
   

1. Download the YAML file using the curl command.

   curl https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml -o aspnetapp.yaml
   

2. Apply the YAML file using the kubectl apply command.

   kubectl apply -f aspnetapp.yaml
   

1. Get the app IP address.

   echo "$(terraform output -raw application_ip_address)"
   

2. In a browser, navigate to the IP address from the output of the previous step.

   

When you no longer need the resources created via Terraform, do the following steps:

1. Run terraform plan and specify the destroy flag.

   terraform plan -destroy -out main.destroy.tfplan
   

   Key points:

   • The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
   • The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
   • To read more about persisting execution plans and security, see the security warning section.

2. Run terraform apply to apply the execution plan.

   terraform apply main.destroy.tfplan
   

Troubleshoot common problems when using Terraform on Azure