Migrate to Azure Kubernetes Service (AKS)

Article
12/06/2024

To help you plan and execute a successful migration to Azure Kubernetes Service (AKS), this guide provides details for the current recommended AKS configuration. While this article doesn't cover every scenario, it contains links to more detailed information for planning a successful migration.

In this article, we summarize migration details for:

AKS with Azure Load Balancer (Standard) and Virtual Machine Scale Sets
Existing attached Azure services
Ensure valid quotas
High availability and business continuity
Considerations for stateless applications
Considerations for stateful applications
Deployment of your cluster configuration

Note

Depending on your scenario, the following open-source tools might help with your migration:

Velero (Requires Kubernetes 1.7+)
Azure Kube CLI extension

Before you begin

Ensure your target Kubernetes version is within the supported window for AKS. Older versions may not be within the supported range and require a version upgrade for AKS support. For more information, see AKS supported Kubernetes versions.
If you're migrating to a newer version of Kubernetes, review the Kubernetes version and version skew support policy.

An important practice that you should include as part of your migration process is remembering to follow commonly used deployment and testing patterns. Testing your application before deployment is an important step to ensure its quality, functionality, and compatibility with the target environment. It can help you identify and fix any errors, bugs, or issues that might affect the performance, security, or usability of the application or underlying infrastructure.

AKS with Standard Load Balancer and Virtual Machine Scale Sets

AKS is a managed service offering unique capabilities with lower management overhead. Since AKS is a managed service, you must select from a set of AKS-supported regions. You may need to modify your existing applications to keep them healthy on the AKS-managed control plane during the transition from your existing cluster to AKS.

We recommend using AKS clusters backed by Virtual Machine Scale Sets and Load Balancer (Standard) to ensure you get the following features:

AKS clusters backed by virtual machine (VM) availability sets lack support for many of these features.

Create an AKS cluster with Load Balancer (Standard) and Virtual Machine Scale Sets

Azure CLI
Terraform

The following example creates an AKS cluster with single node pool backed by a Virtual Machine Scale Set. It enables the cluster autoscaler on the node pool for the cluster and sets a minimum of one and a maximum of three nodes.

Create a resource group using the az group create command.

az group create --name myResourceGroup --location eastus

Create an AKS cluster using the az aks create command.

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --node-count 1 \
    --vm-set-type VirtualMachineScaleSets \
    --load-balancer-sku standard \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 3 \
    --generate-ssh-keys

The following code creates a resource group and a Kubernetes cluster in Azure, with auto-scaling enabled and specific network settings, using Terraform.

Note

The sample code for this article is located in the Azure Terraform GitHub repo. You can view the log file containing the test results from current and previous versions of Terraform.

Create a directory in which to test and run the sample Terraform code, and make it the current directory.
Create a file named main.tf, and insert the following code:

resource "random_pet" "rg_name" {
  prefix = var.resource_group_name_prefix
}

resource "azurerm_resource_group" "rg" {
  location = var.resource_group_location
  name     = random_pet.rg_name.id
}

resource "random_string" "aks_cluster_name" {
  length  = 12
  special = false
}

resource "azurerm_kubernetes_cluster" "aks_cluster" {
  name                = random_string.aks_cluster_name.result
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "myakscluster"

  default_node_pool {
    type                = "VirtualMachineScaleSets"
    name                = "default"
    node_count          = 1
    max_count           = 3
    min_count           = 1
    vm_size             = "Standard_DS2_v2"
    enable_auto_scaling = true
    upgrade_settings {
      drain_timeout_in_minutes      = 90
      max_surge                     = "10%"
      node_soak_duration_in_minutes = 0
    }
  }

  network_profile {
    network_plugin    = "azure"
    load_balancer_sku = "standard"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Production"
  }
}

Create a file named outputs.tf, and insert the following code:

output "resource_group_name" {
  value = azurerm_resource_group.rg.name
}

output "aks_cluster_name" {
  value = azurerm_kubernetes_cluster.aks_cluster.name
}

Create a file named providers.tf, and insert the following code:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~>3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

Create a file named variables.tf, and insert the following code:

variable "resource_group_location" {
  type        = string
  default     = "eastus"
  description = "Location of the resource group."
}

variable "resource_group_name_prefix" {
  type        = string
  default     = "rg"
  description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription."
}

Initialize Terraform.

Run terraform init to initialize the Terraform deployment. This command downloads the Azure provider required to manage your Azure resources.
```
terraform init -upgrade
```
Key points:
- The -upgrade parameter upgrades the necessary provider plugins to the newest version that complies with the configuration's version constraints.
Create a Terraform execution plan.

Run terraform plan to create an execution plan.
```
terraform plan -out main.tfplan
```
Key points:
- The terraform plan command creates an execution plan, but doesn't execute it. Instead, it determines what actions are necessary to create the configuration specified in your configuration files. This pattern allows you to verify whether the execution plan matches your expectations before making any changes to actual resources.
- The optional -out parameter allows you to specify an output file for the plan. Using the -out parameter ensures that the plan you reviewed is exactly what is applied.
Apply a Terraform execution plan.

Run terraform apply to apply the execution plan to your cloud infrastructure.
```
terraform apply main.tfplan
```
Key points:
- The example terraform apply command assumes you previously ran terraform plan -out main.tfplan.
- If you specified a different filename for the -out parameter, use that same filename in the call to terraform apply.
- If you didn't use the -out parameter, call terraform apply without any parameters.

Existing attached Azure Services

When migrating clusters, you may have attached external Azure services. While the following services don't require resource recreation, they require updating connections from previous to new clusters to maintain functionality:

Azure Container Registry
Azure Log Analytics
Azure Application Insights
Azure Traffic Manager
Azure Storage account
External databases

Ensure valid quotas

Since other VMs are deployed into your subscription during migration, you should verify your quotas and limits are sufficient for these resources. If necessary, request an increase in vCPU quota.

You may need to request an increase for network quotas to ensure you don't exhaust IPs. For more information, see networking and IP ranges for AKS.

For more information, see Azure subscription and service limits. To check your current quotas, in the Azure portal, go to the subscriptions blade, select your subscription, and then select Usage + quotas.

High availability and business continuity

If your application can't handle downtime, you need to follow best practices for high availability migration scenarios. Read more about Best practices for complex business continuity planning, disaster recovery, and maximizing uptime in Azure Kubernetes Service (AKS).

For complex applications, you typically migrate over time rather than all at once, meaning the old and new environments might need to communicate over the network. Applications previously using ClusterIP services to communicate might need to be exposed as type LoadBalancer and secured appropriately.

To complete the migration, you want to point clients to the new services that run on AKS. We recommend you redirect traffic by updating DNS to point to the load balancer sitting in front of your AKS cluster.

Azure Traffic Manager can direct customers to the desired Kubernetes cluster and application instance. Traffic Manager is a DNS-based traffic load balancer that can distribute network traffic across regions. For the best performance and redundancy, direct all application traffic through Traffic Manager before it goes to your AKS cluster.

In a multi-cluster deployment, customers should connect to a Traffic Manager DNS name that points to the services on each AKS cluster. Define these services by using Traffic Manager endpoints. Each endpoint is the service load balancer IP. Use this configuration to direct network traffic from the Traffic Manager endpoint in one region to the endpoint in a different region.

AKS with Traffic Manager

Considerations for stateless applications

Stateless application migration involves the following steps:

Apply your resource definitions (YAML or Helm) to the new cluster.
Ensure everything works as expected.
Redirect traffic to activate your new cluster.

Considerations for stateful applications

Carefully plan your migration of stateful applications to avoid data loss or unexpected downtime.

If you use Azure Files, you can mount the file share as a volume into the new cluster. See Mount Static Azure Files as a Volume.
If you use Azure Managed Disks, you can only mount the disk if unattached to any VM. See Mount Static Azure Disk as a Volume.
If neither of those approaches work, you can use a backup and restore options. See Velero on Azure.

Azure Files

Unlike disks, Azure Files can be mounted to multiple hosts concurrently. In your AKS cluster, Azure and Kubernetes don't prevent you from creating a pod that your AKS cluster still uses. To prevent data loss and unexpected behavior, ensure the clusters don't simultaneously write to the same files.

If your application can host multiple replicas that point to the same file share, follow the stateless migration steps and deploy your YAML definitions to your new cluster.

If not, a possible migration approach involves the following steps:

Validate your application is working correctly.
Point your live traffic to your new AKS cluster.
Disconnect the old cluster.

If you want to start with an empty share and make a copy of the source data, you can use the az storage file copy command to migrate your data.

Migrating persistent volumes

If you're migrating existing persistent volumes to AKS, you generally follow these steps:

Quiesce writes to the application.
- This step is optional and requires downtime.
Take snapshots of the disks.
Create new managed disks from the snapshots.
Create persistent volumes in AKS.
Update pod specifications to use existing volumes rather than PersistentVolumeClaims (static provisioning).
Deploy your application to AKS.
Validate your application is working correctly.
Point your live traffic to your new AKS cluster.

Important

If you choose not to quiesce writes, you need to replicate data to the new deployment. Otherwise, you miss the data that was written after you took the disk snapshots.

The following open-source tools can help you create managed disks and migrate volumes between Kubernetes clusters:

Azure CLI Disk Copy extension copies and converts disks across resource groups and Azure regions.
Azure Kube CLI extension enumerates ACS Kubernetes volumes and migrates them to an AKS cluster.

Deployment of your cluster configuration

We recommend you use your existing continuous integration and continuous delivery pipeline to deploy a known-good configuration to AKS. You can use Azure Pipelines to build and deploy your applications to AKS. Clone your existing deployment tasks and ensure kubeconfig points to the new AKS cluster.

If that's not possible, export resource definitions from your existing Kubernetes cluster, and then apply them to AKS. You can use kubectl to export objects. For example:

kubectl get deployment -o yaml > deployments.yaml

Be sure to examine the output and remove any unnecessary live data fields.

Moving existing resources to another region

You might want to move your AKS cluster to a different region supported by AKS. We recommend you create a new cluster in the other region and then deploy your resources and applications to your new cluster.

If you have any services running on your AKS cluster, you need to install and configure those services on your cluster in the new region.

In this article, we summarized migration details for:

AKS with Load Balancer (Standard) and Virtual Machine Scale Sets
Existing attached Azure services
Ensuring valid quotas
High availability and business continuity
Considerations for stateless applications
Considerations for stateful applications
Deploying your cluster configuration