Enable monitoring for Kubernetes clusters

Article
04/09/2024

This article describes how to enable complete monitoring of your Kubernetes clusters using the following Azure Monitor features:

Container insights for log collection

Using the Azure portal, you can enable all of the features at the same time. You can also enable them individually by using the Azure CLI, Azure Resource Manager template, Terraform, or Azure Policy. Each of these methods is described in this article.

Important

Kubernetes clusters generate a lot of log data, which can result in significant costs if you aren't selective about the logs that you collect. Before you enable monitoring for your cluster, see the following articles to ensure that your environment is optimized for cost and that you limit your log collection to only the data that you require:

Configure data collection and cost optimization in Container insights using data collection rule
Details on customizing log collection once you've enabled monitoring, including using preset cost optimization configurations.
Best practices for monitoring Kubernetes with Azure Monitor
Best practices for monitoring Kubernetes clusters organized by the five pillars of the Azure Well-Architected Framework, including cost optimization.
Cost optimization in Azure Monitor
Best practices for configuring all features of Azure Monitor to optimize you costs and limit the amount of data that you collect.

Supported clusters

This article provides onboarding guidance for the following types of clusters. Any differences in the process for each type are noted in the relevant sections.

Prerequisites

Permissions

You require at least Contributor access to the cluster for onboarding.
You require Monitoring Reader or Monitoring Contributor to view data after monitoring is enabled.

Arc-Enabled Kubernetes clusters prerequisites

Prerequisites for Azure Arc-enabled Kubernetes cluster extensions.
Verify the firewall requirements in addition to the Azure Arc-enabled Kubernetes network requirements.
If you previously installed monitoring for AKS, ensure that you have disabled monitoring before proceeding to avoid issues during the extension install.
If you previously installed monitoring on a cluster using a script without cluster extensions, follow the instructions at Disable monitoring of your Kubernetes cluster to delete this Helm chart.

Workspaces

The following table describes the workspaces that are required to support Container insights. You can create each workspace as part of the onboarding process or use an existing workspace. See Design a Log Analytics workspace architecture for guidance on how many workspaces to create and where they should be placed.

Feature	Workspace	Notes
Container insights	Log Analytics workspace	You can attach an AKS cluster to a Log Analytics workspace in a different Azure subscription in the same Microsoft Entra tenant, but you must use the Azure CLI or an Azure Resource Manager template. You can't currently perform this configuration with the Azure portal. If you're connecting an existing AKS cluster to a Log Analytics workspace in another subscription, the Microsoft.ContainerService resource provider must be registered in the subscription with the Log Analytics workspace. For more information, see Register resource provider.

Enable Container insights

Use one of the following methods to enable Container insights on your cluster. Once this is complete, see Configure agent data collection for Container insights to customize your configuration to ensure that you aren't collecting more data than you require.

Use one of the following commands to enable monitoring of your AKS and Arc-enabled clusters. If you don't specify an existing Log Analytics workspace, the default workspace for the resource group will be used. If a default workspace doesn't already exist in the cluster's region, one will be created with a name in the format DefaultWorkspace-<GUID>-<Region>.

Prerequisites

Azure CLI version 2.43.0 or higher
Managed identity authentication is default in CLI version 2.49.0 or higher.
Azure k8s-extension version 1.3.7 or higher
Managed identity authentication is the default in k8s-extension version 1.43.0 or higher.
Managed identity authentication is not supported for Arc-enabled Kubernetes clusters with ARO (Azure Red Hat Openshift) or Windows nodes. Use legacy authentication.
For CLI version 2.54.0 or higher, the logging schema will be configured to ContainerLogV2 using ConfigMap.

AKS cluster

### Use default Log Analytics workspace
az aks enable-addons -a monitoring -n <cluster-name> -g <cluster-resource-group-name>

### Use existing Log Analytics workspace
az aks enable-addons -a monitoring -n <cluster-name> -g <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id>

Example

az aks enable-addons -a monitoring -n "my-cluster" -g "my-resource-group" --workspace-resource-id "/subscriptions/my-subscription/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace"

Arc-enabled cluster

### Use default Log Analytics workspace
az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers

### Use existing Log Analytics workspace
az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID=<workspace-resource-id>

### Use managed identity authentication (default as k8s-extension version 1.43.0)
az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings amalogs.useAADAuth=true

### Use advanced configuration settings
az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings  amalogs.resources.daemonset.limits.cpu=150m amalogs.resources.daemonset.limits.memory=600Mi amalogs.resources.deployment.limits.cpu=1 amalogs.resources.deployment.limits.memory=750Mi

### With custom mount path for container stdout & stderr logs
### Custom mount path not required for Azure Stack Edge version > 2318. Custom mount path must be /home/data/docker for Azure Stack Edge cluster with version <= 2318
az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings amalogs.logsettings.custommountpath=<customMountPath>

See the resource requests and limits section of Helm chart for the available configuration settings.

Example

az k8s-extension create --name azuremonitor-containers --cluster-name "my-cluster" --resource-group "my-resource-group" --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID="/subscriptions/my-subscription/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace"

Arc-enabled cluster with forward proxy

If the cluster is configured with a forward proxy, then proxy settings are automatically applied to the extension. In the case of a cluster with AMPLS + proxy, proxy config should be ignored. Onboard the extension with the configuration setting amalogs.ignoreExtensionProxySettings=true.

az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings amalogs.ignoreExtensionProxySettings=true

Arc-enabled cluster with ARO or OpenShift or Windows nodes

Managed identity authentication is not supported for Arc-enabled Kubernetes clusters with ARO (Azure Red Hat OpenShift) or OpenShift or Windows nodes. Use legacy authentication by specifying amalogs.useAADAuth=false as in the following example.

az k8s-extension create --name azuremonitor-containers --cluster-name <cluster-name> --resource-group <resource-group> --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings amalogs.useAADAuth=false

Delete extension instance

The following command only deletes the extension instance, but doesn't delete the Log Analytics workspace. The data in the Log Analytics resource is left intact.

az k8s-extension delete --name azuremonitor-containers --cluster-type connectedClusters --cluster-name <cluster-name> --resource-group <resource-group>

Both ARM and Bicep templates are provided in this section.

Prerequisites

The template must be deployed in the same resource group as the cluster.

Download and install template

Download and edit template and parameter file

AKS cluster ARM
- Template file: https://aka.ms/aks-enable-monitoring-msi-onboarding-template-file
- Parameter file: https://aka.ms/aks-enable-monitoring-msi-onboarding-template-parameter-file
AKS cluster Bicep
- Template file (Syslog): https://aka.ms/enable-monitoring-msi-syslog-bicep-template
- Parameter file (No Syslog): https://aka.ms/enable-monitoring-msi-syslog-bicep-parameters
- Template file (No Syslog): https://aka.ms/enable-monitoring-msi-bicep-template
- Parameter file (No Syslog): https://aka.ms/enable-monitoring-msi-bicep-parameters
Arc-enabled cluster ARM
- Template file: https://aka.ms/arc-k8s-azmon-extension-msi-arm-template
- Parameter file: https://aka.ms/arc-k8s-azmon-extension-msi-arm-template-params
- Template file (legacy authentication): https://aka.ms/arc-k8s-azmon-extension-arm-template
- Parameter file (legacy authentication): https://aka.ms/arc-k8s-azmon-extension-arm-template-params

Edit the following values in the parameter file. The same set of values are used for both the ARM and Bicep templates. Retrieve the resource ID of the resources from the JSON View of their Overview page.

Parameter	Description
AKS: `aksResourceId` Arc: `clusterResourceId`	Resource ID of the cluster.
AKS: `aksResourceLocation` Arc: `clusterRegion`	Location of the cluster.
AKS: `workspaceResourceId` Arc: `workspaceResourceId`	Resource ID of the Log Analytics workspace.
Arc: `workspaceRegion`	Region of the Log Analytics workspace.
Arc: `workspaceDomain`	Domain of the Log Analytics workspace. `opinsights.azure.com` for Azure public cloud `opinsights.azure.cn` for Microsoft Azure operated by 21Vianet.
AKS: `resourceTagValues`	Tag values specified for the existing Container insights extension data collection rule (DCR) of the cluster and the name of the DCR. The name will be `MSCI-<clusterName>-<clusterRegion>` and this resource created in an AKS clusters resource group. For first time onboarding, you can set arbitrary tag values.

Deploy the template with the parameter file by using any valid method for deploying Resource Manager templates. For examples of different methods, see Deploy the sample templates.

New AKS cluster

Download Terraform template file depending on whether you want to enable Syslog collection.

Syslog
- https://aka.ms/enable-monitoring-msi-syslog-terraform
No Syslog
- https://aka.ms/enable-monitoring-msi-terraform
Adjust the azurerm_kubernetes_cluster resource in main.tf based on your cluster settings.

Update parameters in variables.tf to replace values in "<>"

Parameter	Description
`aks_resource_group_name`	Use the values on the AKS Overview page for the resource group.
`resource_group_location`	Use the values on the AKS Overview page for the resource group.
`cluster_name`	Define the cluster name that you would like to create.
`workspace_resource_id`	Use the resource ID of your Log Analytics workspace.
`workspace_region`	Use the location of your Log Analytics workspace.
`resource_tag_values`	Match the existing tag values specified for the existing Container insights extension data collection rule (DCR) of the cluster and the name of the DCR. The name will match `MSCI-<clusterName>-<clusterRegion>` and this resource is created in the same resource group as the AKS clusters. For first time onboarding, you can set the arbitrary tag values.
`enabledContainerLogV2`	Set this parameter value to be true to use the default recommended ContainerLogV2.
Cost optimization parameters	Refer to Data collection parameters

Run terraform init -upgrade to initialize the Terraform deployment.
Run terraform plan -out main.tfplan to initialize the Terraform deployment.
Run terraform apply main.tfplan to apply the execution plan to your cloud infrastructure.

Existing AKS cluster

Import the existing cluster resource first with the command: terraform import azurerm_kubernetes_cluster.k8s <aksResourceId>

Add the oms_agent add-on profile to the existing azurerm_kubernetes_cluster resource.

oms_agent {
    log_analytics_workspace_id = var.workspace_resource_id
    msi_auth_for_monitoring_enabled = true
  }

Copy the DCR and DCRA resources from the Terraform templates
Run terraform plan -out main.tfplan and make sure the change is adding the oms_agent property. Note: If the azurerm_kubernetes_cluster resource defined is different during terraform plan, the existing cluster will get destroyed and recreated.
Run terraform apply main.tfplan to apply the execution plan to your cloud infrastructure.

Tip

Edit the main.tf file appropriately before running the terraform template
Data will start flowing after 10 minutes since the cluster needs to be ready first
WorkspaceID needs to match the format /subscriptions/12345678-1234-9876-4563-123456789012/resourceGroups/example-resource-group/providers/Microsoft.OperationalInsights/workspaces/workspaceValue
If resource group already exists, run terraform import azurerm_resource_group.rg /subscriptions/<Subscription_ID>/resourceGroups/<Resource_Group_Name> before terraform plan

Azure Portal

From the Definitions tab of the Policy menu in the Azure portal, create a policy definition with the following details.
- Definition location: Azure subscription where the policy definition should be stored.
- Name: AKS-Monitoring-Addon
- Description: Azure custom policy to enable the Monitoring Add-on onto Azure Kubernetes clusters.
- Category: Select Use existing and then Kubernetes from the dropdown list.
- Policy rule: Replace the existing sample JSON with the contents of https://aka.ms/aks-enable-monitoring-custom-policy.
Select the new policy definition AKS Monitoring Addon.
Select Assign and specify a Scope of where the policy should be assigned.
Select Next and provide the resource ID of the Log Analytics workspace.
Create a remediation task if you want to apply the policy to existing AKS clusters in the selected scope.
Select Review + create to create the policy assignment.

Azure CLI

Download Azure Policy template and parameter files.
- Template file: https://aka.ms/enable-monitoring-msi-azure-policy-template
- Parameter file: https://aka.ms/enable-monitoring-msi-azure-policy-parameters

Create the policy definition using the following CLI command:

az policy definition create --name "AKS-Monitoring-Addon-MSI" --display-name "AKS-Monitoring-Addon-MSI" --mode Indexed --metadata version=1.0.0 category=Kubernetes --rules azure-policy.rules.json --params azure-policy.parameters.json

Create the policy definition using the following CLI command:

az policy assignment create --name aks-monitoring-addon --policy "AKS-Monitoring-Addon-MSI" --assign-identity --identity-scope /subscriptions/<subscriptionId> --role Contributor --scope /subscriptions/<subscriptionId> --location <location> --role Contributor --scope /subscriptions/<subscriptionId> -p "{ \"workspaceResourceId\": { \"value\":  \"/subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/microsoft.operationalinsights/workspaces/<workspaceName>\" } }"

After the policy is assigned to the subscription, whenever you create a new cluster without Prometheus enabled, the policy will run and deploy to enable Prometheus monitoring.

Enable full monitoring with Azure portal

Using the Azure portal.

New AKS cluster

When you create a new AKS cluster in the Azure portal, you can enable Container insights from the Integrations tab. In the Azure Monitor section, select either Default configuration or Custom configuration if you want to specify which workspaces to use. You can perform additional configuration once the cluster is created.

Existing cluster

This option enables Container insights on an existing AKS cluster.

Either select Insights from the cluster's menu OR select Containers from the Monitor menu, Unmonitored clusters tab, and click Enable next to a cluster.
1. If Container insights isn't enabled for the cluster, then you're presented with a screen identifying which of the features have been enabled. Click Configure monitoring.
1. If Container insights has already been enabled on the cluster, select the Monitoring Settings button to modify the configuration.
Click Advanced settings to select alternate workspaces or create new ones. The Cost presets setting allows you to modify the default collection details to reduce your monitoring costs. See Enable cost optimization settings in Container insights for details.
Click Configure to save the configuration.

Verify deployment

Use the kubectl command line tool to verify that the agent is deployed properly.

Container insights

Verify that the DaemonSets were deployed properly on the Linux node pools

kubectl get ds ama-logs --namespace=kube-system

The number of pods should be equal to the number of Linux nodes on the cluster. The output should resemble the following example:

User@aksuser:~$ kubectl get ds ama-logs --namespace=kube-system
NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ama-logs   2         2         2         2            2           <none>          1d

Verify that Windows nodes were deployed properly

kubectl get ds ama-logs-windows --namespace=kube-system

The number of pods should be equal to the number of Windows nodes on the cluster. The output should resemble the following example:

User@aksuser:~$ kubectl get ds ama-logs-windows --namespace=kube-system
NAME                   DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR     AGE
ama-logs-windows           2         2         2         2            2       <none>            1d

Verify deployment of the Container insights solution

kubectl get deployment ama-logs-rs --namespace=kube-system

The output should resemble the following example:

User@aksuser:~$ kubectl get deployment ama-logs-rs --namespace=kube-system
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
ama-logs-rs   1/1     1            1           24d

View configuration with CLI

Use the aks show command to find out whether the solution is enabled, the Log Analytics workspace resource ID, and summary information about the cluster.

az aks show -g <resourceGroupofAKSCluster> -n <nameofAksCluster>

The command will return JSON-formatted information about the solution. The addonProfiles section should include information on the omsagent as in the following example:

"addonProfiles": {
    "omsagent": {
        "config": {
            "logAnalyticsWorkspaceResourceID": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/my-resource-group/providers/microsoft.operationalinsights/workspaces/my-workspace",
            "useAADAuth": "true"
        },
        "enabled": true,
        "identity": null
    },
}

Resources provisioned

When you enable monitoring, the following resources are created in your subscription:

Resource Name	Resource Type	Resource Group	Region/Location	Description
`MSCI-<aksclusterregion>-<clustername>`	Data Collection Rule	Same as cluster	Same as Log Analytics workspace	This data collection rule is for log collection by Azure Monitor agent, which uses the Log Analytics workspace as destination, and is associated to the AKS cluster resource.
`MSPROM-<aksclusterregion>-<clustername>`	Data Collection Rule	Same as cluster	Same as Azure Monitor workspace	This data collection rule is for prometheus metrics collection by metrics addon, which has the chosen Azure monitor workspace as destination, and also it is associated to the AKS cluster resource
`MSPROM-<aksclusterregion>-<clustername>`	Data Collection endpoint	Same as cluster	Same as Azure Monitor workspace	This data collection endpoint is used by the above data collection rule for ingesting Prometheus metrics from the metrics addon

When you create a new Azure Monitor workspace, the following additional resources are created as part of it

Resource Name	Resource Type	Resource Group	Region/Location	Description
`<azuremonitor-workspace-name>`	Data Collection Rule	MA_<azuremonitor-workspace-name>_<azuremonitor-workspace-region>_managed	Same as Azure Monitor Workspace	DCR created when you use OSS Prometheus server to Remote Write to Azure Monitor Workspace.
`<azuremonitor-workspace-name>`	Data Collection Endpoint	MA_<azuremonitor-workspace-name>_<azuremonitor-workspace-region>_managed	Same as Azure Monitor Workspace	DCE created when you use OSS Prometheus server to Remote Write to Azure Monitor Workspace.

Differences between Windows and Linux clusters

The main differences in monitoring a Windows Server cluster compared to a Linux cluster include:

Windows doesn't have a Memory RSS metric. As a result, it isn't available for Windows nodes and containers. The Working Set metric is available.
Disk storage capacity information isn't available for Windows nodes.
Only pod environments are monitored, not Docker environments.
With the preview release, a maximum of 30 Windows Server containers are supported. This limitation doesn't apply to Linux containers.

Note

Container insights support for the Windows Server 2022 operating system is in preview.

The containerized Linux agent (replicaset pod) makes API calls to all the Windows nodes on Kubelet secure port (10250) within the cluster to collect node and container performance-related metrics. Kubelet secure port (:10250) should be opened in the cluster's virtual network for both inbound and outbound for Windows node and container performance-related metrics collection to work.

If you have a Kubernetes cluster with Windows nodes, review and configure the network security group and network policies to make sure the Kubelet secure port (:10250) is open for both inbound and outbound in the cluster's virtual network.

Next steps

If you experience issues while you attempt to onboard the solution, review the Troubleshooting guide.
With monitoring enabled to collect health and resource utilization of your AKS cluster and workloads running on them, learn how to use Container insights.

Enable monitoring for Kubernetes clusters

Supported clusters

Prerequisites

Workspaces

Enable Container insights

Prerequisites

AKS cluster

Arc-enabled cluster

Enable full monitoring with Azure portal

New AKS cluster

Existing cluster

Verify deployment

Container insights

Resources provisioned

Differences between Windows and Linux clusters

Next steps

Additional resources