Enable monitoring for AKS clusters

Prerequisites

• Managed identity authentication is default in CLI version 2.49.0 or higher.
• The aks-preview extension must be uninstalled from AKS clusters by using the command az extension remove --name aks-preview.

Prometheus metrics

Use the -enable-azure-monitor-metrics option with az aks create or az aks update depending whether you're creating a new cluster or updating an existing cluster to install the metrics add-on that scrapes Prometheus metrics. This will use the configuration described in Default Prometheus metrics configuration in Azure Monitor. To modify this configuration, see Customize scraping of Prometheus metrics in Azure Monitor managed service for Prometheus.

See the following examples.

### Use default Azure Monitor workspace
az aks create/update --enable-azure-monitor-metrics --name <cluster-name> --resource-group <cluster-resource-group>

### Use existing Azure Monitor workspace
az aks create/update --enable-azure-monitor-metrics --name <cluster-name> --resource-group <cluster-resource-group> --azure-monitor-workspace-resource-id <workspace-name-resource-id>

### Use an existing Azure Monitor workspace and link with an existing Grafana workspace
az aks create/update --enable-azure-monitor-metrics --name <cluster-name> --resource-group <cluster-resource-group> --azure-monitor-workspace-resource-id <azure-monitor-workspace-name-resource-id> --grafana-resource-id  <grafana-workspace-name-resource-id>

### Use optional parameters
az aks create/update --enable-azure-monitor-metrics --name <cluster-name> --resource-group <cluster-resource-group> --ksm-metric-labels-allow-list "namespaces=[k8s-label-1,k8s-label-n]" --ksm-metric-annotations-allow-list "pods=[k8s-annotation-1,k8s-annotation-n]"

Example

az aks create/update --enable-azure-monitor-metrics --name "my-cluster" --resource-group "my-resource-group" --azure-monitor-workspace-resource-id "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/my-resource-group/providers/microsoft.monitor/accounts/my-workspace"

Optional parameters

Each of the commands above allow the following optional parameters. The parameter name is different for each, but their use is the same.

Container logs

Use the --addon monitoring option with az aks create for a new cluster or az aks enable-addon to update an existing cluster to enable collection of container logs. See below to modify the log collection settings.

See the following examples.

### Use default Log Analytics workspace
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name>

### Use existing Log Analytics workspace
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id>

### Use custom log configuration file
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id> --data-collection-settings dataCollectionSettings.json

### Use legacy authentication
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id> --enable-msi-auth-for-monitoring false

Example

az aks enable-addons --addon monitoring --name "my-cluster" --resource-group "my-resource-group" --workspace-resource-id "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace"

Log configuration file

To customize log collection settings for the cluster, you can provide the configuration as a JSON file using the following format. If you don't provide a configuration file, the default settings identified in the table below are used.

{
  "interval": "1m",
  "namespaceFilteringMode": "Include",
  "namespaces": ["kube-system"],
  "enableContainerLogV2": true, 
  "streams": ["Microsoft-Perf", "Microsoft-ContainerLogV2"]
}

Each of the settings in the configuration is described in the following table.

Prerequisites

• The Azure Monitor workspace and Azure Managed Grafana instance must already be created.
• The template must be deployed in the same resource group as the Azure Managed Grafana instance.
• If the Azure Managed Grafana instance is in a subscription other than the Azure Monitor workspace subscription, register the Azure Monitor workspace subscription with the Microsoft.Dashboard resource provider using the guidance at Register resource provider.
• Users with the User Access Administrator role in the subscription of the AKS cluster can enable the Monitoring Reader role directly by deploying the template.

Prometheus metrics

Retrieve required values for Grafana resource

If the Azure Managed Grafana instance is already linked to an Azure Monitor workspace, then you must include this list in the template or it will be overwritten. On the Overview page for the Azure Managed Grafana instance in the Azure portal, select JSON view, and copy the value of azureMonitorWorkspaceIntegrations which will look similar to the sample below. If it doesn't exist, then the instance hasn't been linked with any Azure Monitor workspace.

"properties": {
    "grafanaIntegrations": {
        "azureMonitorWorkspaceIntegrations": [
            {
                "azureMonitorWorkspaceResourceId": "full_resource_id_1"
            },
            {
                "azureMonitorWorkspaceResourceId": "full_resource_id_2"
            }
        ]
    }
}

Download and edit template and parameter file

1. Download the required files.

   Bicep

   • Template file: https://aka.ms/azureprometheus-enable-bicep-template
   • Parameter file: https://aka.ms/azureprometheus-enable-bicep-template-parameters
   • DCRA module: https://aka.ms/nested_azuremonitormetrics_dcra_clusterResourceId
   • Profile module: https://aka.ms/nested_azuremonitormetrics_profile_clusterResourceId
   • Azure Managed Grafana Role Assignment module: https://aka.ms/nested_grafana_amw_role_assignment

   JSON

   • Template file: https://aka.ms/azureprometheus-enable-arm-template
   • Parameter file: https://aka.ms/azureprometheus-enable-arm-template-parameters

2. Edit the following values in the parameter file. The same set of values are used for both the ARM and Bicep templates. Retrieve the resource ID of the resources from the JSON View of their Overview page.

   Parameter	Value
   azureMonitorWorkspaceResourceId	Resource ID for the Azure Monitor workspace. Retrieve from the JSON view on the Overview page for the Azure Monitor workspace.
   azureMonitorWorkspaceLocation	Location of the Azure Monitor workspace. Retrieve from the JSON view on the Overview page for the Azure Monitor workspace.
   clusterResourceId	Resource ID for the AKS cluster. Retrieve from the JSON view on the Overview page for the cluster.
   clusterLocation	Location of the AKS cluster. Retrieve from the JSON view on the Overview page for the cluster.
   metricLabelsAllowlist	Comma-separated list of Kubernetes labels keys to be used in the resource's labels metric.
   metricAnnotationsAllowList	Comma-separated list of more Kubernetes label keys to be used in the resource's annotations metric.
   grafanaResourceId	Resource ID for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance.
   grafanaLocation	Location for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance.
   grafanaSku	SKU for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance. Use the sku.name.

3. Open the template file and update the grafanaIntegrations property at the end of the file with the values that you retrieved from the Grafana instance. This will look similar to the following samples. In these samples, full_resource_id_1 and full_resource_id_2 were already in the Azure Managed Grafana resource JSON. The final azureMonitorWorkspaceResourceId entry is already in the template and is used to link to the Azure Monitor workspace resource ID provided in the parameters file.

   Bicep

       resource grafanaResourceId_8 'Microsoft.Dashboard/grafana@2022-08-01' = {
           name: split(grafanaResourceId, '/')[8]
           sku: {
               name: grafanaSku
           }
           identity: {
               type: 'SystemAssigned'
           }
           location: grafanaLocation
           properties: {
               grafanaIntegrations: {
                   azureMonitorWorkspaceIntegrations: [
                       {
                           azureMonitorWorkspaceResourceId: 'full_resource_id_1'
                       }
                       {
                           azureMonitorWorkspaceResourceId: 'full_resource_id_2'
                       }
                       {
                           azureMonitorWorkspaceResourceId: azureMonitorWorkspaceResourceId
                       }
                   ]
               }
           }
       }
   

   JSON

   {
       "type": "Microsoft.Dashboard/grafana",
       "apiVersion": "2022-08-01",
       "name": "[split(parameters('grafanaResourceId'),'/')[8]]",
       "sku": {
           "name": "[parameters('grafanaSku')]"
       },
       "location": "[parameters('grafanaLocation')]",
       "properties": {
           "grafanaIntegrations": {
           "azureMonitorWorkspaceIntegrations": [
               {
                   "azureMonitorWorkspaceResourceId": "full_resource_id_1"
               },
               {
                   "azureMonitorWorkspaceResourceId": "full_resource_id_2"
               },
               {
                   "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"
               }
           ]
           }
       }
   }
   

4. Deploy the template with the parameter file by using any valid method for deploying Resource Manager templates. For examples of different methods, see Deploy the sample templates.

Container logs

Prerequisites

• The template must be deployed in the same resource group as the cluster.

Download and install template

1. Download and edit template and parameter file.

   Bicep

   • Template file (Syslog): https://aka.ms/enable-monitoring-msi-syslog-bicep-template
   • Parameter file (Syslog): https://aka.ms/enable-monitoring-msi-syslog-bicep-parameters
   • Template file (No Syslog): https://aka.ms/enable-monitoring-msi-bicep-template
   • Parameter file (No Syslog): https://aka.ms/enable-monitoring-msi-bicep-parameters

   ARM

   • Template file: https://aka.ms/aks-enable-monitoring-msi-onboarding-template-file
   • Parameter file: https://aka.ms/aks-enable-monitoring-msi-onboarding-template-parameter-file

2. Edit the following values in the parameter file. The same set of values are used for both the ARM and Bicep templates. Retrieve the resource ID of the resources from the JSON View of their Overview page.

   Parameter	Description
   aksResourceId	Resource ID of the cluster.
   aksResourceLocation	Location of the cluster.
   workspaceResourceId	Resource ID of the Log Analytics workspace.
   resourceTagValues	Tag values specified for the existing Container insights extension data collection rule (DCR) of the cluster and the name of the DCR. The name will be MSCI-<clusterName>-<clusterRegion> and this resource created in an AKS clusters resource group. For first time onboarding, you can set arbitrary tag values.
   enableRetinaNetworkFlowLogs	Flag to indicate whether to enable Retina Network Flow Logs.
   enableContainerLogV2	Boolean flag to enable ContainerLogV2 schema. If set to true, the stdout/stderr Logs are sent to ContainerLogV2 table. If not, the container logs are sent to ContainerLog table, unless otherwise specified in the ConfigMap. When specifying the individual streams, you must include the corresponding table for ContainerLog or ContainerLogV2.
   enableSyslog	Specifies whether Syslog collection should be enabled.
   syslogLevels	If Syslog collection is enabled, specifies the log levels to collect.
   dataCollectionInterval	Determines how often the agent collects data. Valid values are 1m - 30m in 1m intervals The default value is 1m. If the value is outside the allowed range, then it defaults to 1 m.
   namespaceFilteringModeForDataCollection	Include: Collects only data from the values in the namespaces field. Exclude: Collects data from all namespaces except for the values in the namespaces field. Off: Ignores any namespace selections and collect data on all namespaces.
   namespacesForDataCollection	Array of comma separated Kubernetes namespaces to collect inventory and perf data based on the namespaceFilteringMode. For example, namespaces = ["kube-system", "default"] with an Include setting collects only these two namespaces. With an Exclude setting, the agent collects data from all other namespaces except for kube-system and default. With an Off setting, the agent collects data from all namespaces including kube-system and default. Invalid and unrecognized namespaces are ignored.
   streams	An array of table streams. See Stream values for a list of the valid streams and their corresponding tables. To enable high scale mode for container logs, use Microsoft-ContainerLogV2-HighScale.
   useAzureMonitorPrivateLinkScope	Specifies whether to use private link for the cluster connection to Azure Monitor.
   azureMonitorPrivateLinkScopeResourceId	If private link is used, resource ID of the private link scope.

3. Deploy the template with the parameter file by using any valid method for deploying Resource Manager templates. For examples of different methods, see Deploy the sample templates.

Prerequisites

• The Azure Monitor workspace and Azure Managed Grafana workspace must already be created.
• The template needs to be deployed in the same resource group as the Azure Managed Grafana workspace.
• Users with the User Access Administrator role in the subscription of the AKS cluster can enable the Monitoring Reader role directly by deploying the template.
• If the Azure Managed Grafana instance is in a subscription other than the Azure Monitor Workspaces subscription, register the Azure Monitor Workspace subscription with the Microsoft.Dashboard resource provider by following this documentation.

Retrieve required values for Grafana resource

If the Azure Managed Grafana instance is already linked to an Azure Monitor workspace, then you must include this list in the template or it will be overwritten. On the Overview page for the Azure Managed Grafana instance in the Azure portal, select JSON view, and copy the value of azureMonitorWorkspaceIntegrations which will look similar to the sample below. If it doesn't exist, then the instance hasn't been linked with any Azure Monitor workspace.

"properties": {
    "grafanaIntegrations": {
        "azureMonitorWorkspaceIntegrations": [
            {
                "azureMonitorWorkspaceResourceId": "full_resource_id_1"
            },
            {
                "azureMonitorWorkspaceResourceId": "full_resource_id_2"
            }
        ]
    }
}

Update the azure_monitor_workspace_integrations block in main.tf with the list of grafana integrations.

  azure_monitor_workspace_integrations {
    resource_id  = var.monitor_workspace_id[var.monitor_workspace_id1, var.monitor_workspace_id2]
  }

Download and edit templates

New AKS cluster

1. Download all files under AddonTerraformTemplate.
2. Edit the variables in variables.tf file with the correct parameter values.
3. Run terraform init -upgrade to initialize the Terraform deployment.
4. Run terraform plan -out main.tfplan to initialize the Terraform deployment.
5. Run terraform apply main.tfplan to apply the execution plan to your cloud infrastructure.

Note: Pass the variables for annotations_allowed and labels_allowed keys in main.tf only when those values exist. These are optional blocks.

Container logs

New AKS cluster

1. Download Terraform template file depending on whether you want to enable Syslog collection.

   • Syslog: https://aka.ms/enable-monitoring-msi-syslog-terraform
   • No Syslog: https://aka.ms/enable-monitoring-msi-terraform

2. Adjust the azurerm_kubernetes_cluster resource in main.tf based on your cluster settings.

3. Update parameters in variables.tf to replace values in "<>"

   Parameter	Description
   aks_resource_group_name	Use the values on the AKS Overview page for the resource group.
   resource_group_location	Use the values on the AKS Overview page for the resource group.
   cluster_name	Define the cluster name that you would like to create.
   workspace_resource_id	Use the resource ID of your Log Analytics workspace.
   workspace_region	Use the location of your Log Analytics workspace.
   resource_tag_values	Match the existing tag values specified for the existing Container insights extension data collection rule (DCR) of the cluster and the name of the DCR. The name will match MSCI-<clusterName>-<clusterRegion> and this resource is created in the same resource group as the AKS clusters. For first time onboarding, you can set the arbitrary tag values.
   enabledContainerLogV2	Set this parameter value to be true to use the default recommended ContainerLogV2.
   Cost optimization parameters	Refer to Data collection parameters
   streams	Streams for data collection. For retina networkflow logs feature, include Microsoft-RetinaNetworkFlowLogs. For high scale mode, replace the stream Microsoft-ContainerLogV2 with Microsoft-ContainerLogV2-HighScale in the template.
   use_azure_monitor_private_link_scope	Flag to indicate whether to configure Azure Monitor Private Link Scope.
   azure_monitor_private_link_scope_resource_id	Azure Resource ID of the Azure Monitor Private Link Scope.

4. Run terraform init -upgrade to initialize the Terraform deployment.

5. Run terraform plan -out main.tfplan to initialize the Terraform deployment.

6. Run terraform apply main.tfplan to apply the execution plan to your cloud infrastructure.

Existing AKS cluster

1. Import the existing cluster resource first with the command: terraform import azurerm_kubernetes_cluster.k8s <aksResourceId>
2. Add the oms_agent add-on profile to the existing azurerm_kubernetes_cluster resource.

   oms_agent {
       log_analytics_workspace_id = var.workspace_resource_id
       msi_auth_for_monitoring_enabled = true
     }
   

3. Copy the DCR and DCRA resources from the Terraform templates
4. Run terraform plan -out main.tfplan and make sure the change is adding the oms_agent property. Note: If the azurerm_kubernetes_cluster resource defined is different during terraform plan, the existing cluster will get destroyed and recreated.
5. Run terraform apply main.tfplan to apply the execution plan to your cloud infrastructure.

You can enable Prometheus metrics and container logs when you create a new AKS cluster or on an existing cluster in the Azure portal. In both cases, the configuration experience is the same.

New AKS cluster

When you create a new AKS cluster in the Azure portal, configure monitoring in the Monitoring tab.

Existing cluster

Navigate to your cluster in the Azure portal. In the service menu, select Monitor and then Monitor Settings.

Configuration options

Configuration options are the same for both new and existing clusters. The only difference is you may need to select Advanced settings to view all options for an existing cluster.

3. Prometheus metrics, Grafana and Container Logs and events are selected for you. If you have existing Azure Monitor workspace, Grafana workspace and Log Analytics workspace, then they're selected for you.
4. Select Advanced settings if you want to select alternate workspaces or create new ones. The Logging profiles and Classic profiles setting allows you to modify the default collection details to reduce your monitoring costs. See Enable cost optimization settings in Container insights for details.
5. Select Configure.

For container logs, you must select a logging profile, which defines which logs will be collected and at what frequency. The available profiles are listed in the following table.

If you want to customize the settings, click Edit collection settings. Each of these settings is described in the following table.

The Collected data option allows you to select the tables that are populated for the cluster. The tables are grouped by the most common scenarios.

Prometheus metrics

1. Download Azure Policy template and parameter files.

   • Template file: https://aka.ms/AddonPolicyMetricsProfile
   • Parameter file: https://aka.ms/AddonPolicyMetricsProfile.parameters

2. Create the policy definition using the following CLI command:

   az policy definition create --name "Prometheus Metrics addon" --display-name "Prometheus Metrics addon" --mode Indexed --metadata version=1.0.0 category=Kubernetes --rules AddonPolicyMetricsProfile.rules.json --params AddonPolicyMetricsProfile.parameters.json

3. After you create the policy definition, in the Azure portal, select Policy and then Definitions. Select the policy definition you created.

4. Select Assign and fill in the details on the Parameters tab. Select Review + Create.

5. If you want to apply the policy to an existing cluster, create a Remediation task for that cluster resource from Policy Assignment.

After the policy is assigned to the subscription, whenever you create a new cluster without Prometheus enabled, the policy will run and deploy to enable Prometheus monitoring.

Azure portal

1. From the Definitions tab of the Policy menu in the Azure portal, create a policy definition with the following details.

   • Definition location: Azure subscription where the policy definition should be stored.
   • Name: AKS-Monitoring-Addon
   • Description: Azure custom policy to enable the Monitoring Add-on onto Azure Kubernetes clusters.
   • Category: Select Use existing and then Kubernetes from the dropdown list.
   • Policy rule: Replace the existing sample JSON with the contents of https://aka.ms/aks-enable-monitoring-custom-policy.

2. Select the new policy definition AKS Monitoring Addon.

3. Select Assign and specify a Scope of where the policy should be assigned.

4. Select Next and provide the resource ID of the Log Analytics workspace.

5. Create a remediation task if you want to apply the policy to existing AKS clusters in the selected scope.

6. Select Review + create to create the policy assignment.

Azure CLI

1. Download Azure Policy template and parameter files.

   • Template file: https://aka.ms/enable-monitoring-msi-azure-policy-template
   • Parameter file: https://aka.ms/enable-monitoring-msi-azure-policy-parameters

2. Create the policy definition using the following CLI command:

   az policy definition create --name "AKS-Monitoring-Addon-MSI" --display-name "AKS-Monitoring-Addon-MSI" --mode Indexed --metadata version=1.0.0 category=Kubernetes --rules azure-policy.rules.json --params azure-policy.parameters.json
   

3. Create the policy definition using the following CLI command:

   az policy assignment create --name aks-monitoring-addon --policy "AKS-Monitoring-Addon-MSI" --assign-identity --identity-scope /subscriptions/<subscriptionId> --role Contributor --scope /subscriptions/<subscriptionId> --location <location> -p "{ \"workspaceResourceId\": { \"value\": \"/subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/microsoft.operationalinsights/workspaces/<workspaceName>\" }, \"resourceTagValues\": { \"value\": {} }, \"workspaceRegion\": { \"value\": \"<location>\" }}"
   

After the policy is assigned to the subscription, whenever you create a new cluster without Container insights enabled, the policy will run and deploy to enable Container insights monitoring.

Use the az monitor diagnostic-settings create command to create a diagnostic setting with the Azure CLI. See the documentation for this command for descriptions of its parameters.

The following example creates a diagnostic setting that sends all Kubernetes categories to a Log Analytics workspace. This includes resource-specific mode to send the logs to specific tables listed in Supported resource logs for Microsoft.ContainerService/fleets.

az monitor diagnostic-settings create \
--name 'Collect control plane logs' \
--resource  /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.ContainerService/managedClusters/<cluster-name> \
--workspace /subscriptions/<subscription ID>/resourcegroups/<resource group name>/providers/microsoft.operationalinsights/workspaces/<log analytics workspace name> \
--logs '[{"category": "karpenter-events","enabled": true},{"category": "kube-audit","enabled": true},
{"category": "kube-apiserver","enabled": true},{"category": "kube-audit-admin","enabled": true},{"category": "kube-controller-manager","enabled": true},{"category": "kube-scheduler","enabled": true},{"category": "cluster-autoscaler","enabled": true},{"category": "cloud-controller-manager","enabled": true},{"category": "guard","enabled": true},{"category": "csi-azuredisk-controller","enabled": true},{"category": "csi-azurefile-controller","enabled": true},{"category": "csi-snapshot-controller","enabled": true},{"category": "fleet-member-agent","enabled": true},{"category": "fleet-member-net-controller-manager","enabled": true},{"category": "fleet-mcs-controller-manager","enabled": true}]'
--metrics '[{"category": "AllMetrics","enabled": true}]' \
--export-to-resource-specific true

Following are sample template and parameter files to create a diagnostic setting for control plane logs. Modify the templates to collect different categories or to send the logs to a different destination.

Bicep

param clusterName string
param workspaceId string
param settingName string

resource cluster 'Microsoft.ContainerService/managedClusters@2021-05-01-preview' existing = {
  name: clusterName
}

resource setting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: settingName
  scope: cluster
  properties: {
    workspaceId: workspaceId
    logs: [
      {
        category: 'kube-apiserver'
        enabled: true
      }
      {
        category: 'kube-audit'
        enabled: true
      }
      {
        category: 'kube-audit-admin'
        enabled: true
      }
      {
        category: 'kube-controller-manager'
        enabled: true
      }
      {
        category: 'kube-scheduler'
              }
      {
        category: 'cluster-autoscaler'
        enabled: true
      }
      {
        category: 'guard'
        enabled: true
      }
    ]
  }
}

JSON

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "clusterName": {
            "type": "String"
        },
        "workspaceId": {
            "type": "String"
        },
        "settingName": {
            "type": "String"
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/diagnosticSettings",
            "apiVersion": "2021-05-01-preview",
            "scope": "[format('Microsoft.ContainerService/managedClusters/{0}', parameters('clusterName'))]",
            "name": "[parameters('settingName')]",
            "properties": {
                "workspaceId": "[parameters('workspaceId')]",
                "logs": [
                    {
                        "category": "kube-apiserver",
                        "enabled": true
                    },
                    {
                        "category": "kube-audit",
                        "enabled": true
                    },
                    {
                        "category": "kube-audit-admin",
                        "enabled": true
                    },
                    {
                        "category": "kube-controller-manager",
                        "enabled": true
                    },
                    {
                        "category": "kube-scheduler",
                        "enabled": false
                    },
                    {
                        "category": "cluster-autoscaler",
                        "enabled": true
                    },
                    {
                        "category": "guard",
                        "enabled": true
                    }
                ]
            }
        }
    ]
}

Parameter file

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "settingName": {
        "value": "<cluster-name>"
    },
    "workspaceId": {
      "value": "/subscriptions/<subscription id>/resourcegroups/<resourcegroup name>/providers/microsoft.operationalinsights/workspaces/<workspace name>"
    },
    "scope": {
      "value": "Microsoft.<resource type>/<resourceName>"
    }
  }
}

Use the following template to create a diagnostic setting for control plane logs. Modify the template to collect different categories or to send the logs to a different destination.

  features {}
}

variable "setting_name" {
  type        = string
  description = "Name for the diagnostic setting."
}

variable "workspace_id" {
  type        = string
  description = "Resource ID of the Log Analytics workspace."
}

variable "cluster_id" {
  type        = string
  description = "Resource ID of the AKS cluster to attach diagnostics to."
}

resource "azurerm_monitor_diagnostic_setting" "aks" {
  name                       = var.setting_name
  target_resource_id         = var.cluster_id
  log_analytics_workspace_id = var.workspace_id

  log {
    category = "kube-apiserver"
    enabled  = true
  }

  log {
    category = "kube-audit"
    enabled  = true
  }

  log {
    category = "kube-audit-admin"
    enabled  = true
  }

  log {
    category = "kube-controller-manager"
    enabled  = true
  }

  log {
    category = "kube-scheduler"
    enabled  = false
  }

  log {
    category = "cluster-autoscaler"
    enabled  = true
  }

  log {
    category = "guard"
    enabled  = true
  }
}

Select Diagnostic settings from the Monitoring section of the menu for the cluster. Then select + Add diagnostic setting.

Select Send to Log Analytics workspace and select the same workspace where you send your container logs. Then select the different Categories that you want to collect. Give the Diagnostic setting name a descriptive name such as Collect Control Plane Logs.

Verify deployment

Within a few minutes after enabling monitoring, you should be able to use the following methods to verify that the monitoring features are enabled.

• The cluster should move from the Unmonitored clusters view to the Monitored clusters view in Container insights multi-cluster view.
• The Monitor view for the cluster should start to populate with data and no longer provide an option to enable monitoring. This includes the Nodes, Workloads, and Containers tabs.

See Create diagnostic settings at scale using built-in Azure Policies for details about using Azure Policy to create diagnostic settings at scale.