Send Prometheus data to Azure Monitor by using managed identity authentication

  • Article

This article describes how to set up remote write to send data from a self-managed Prometheus server running in your Azure Kubernetes Service (AKS) cluster or Azure Arc-enabled Kubernetes cluster by using managed identity authentication and a side car container provided by Azure Monitor. You can either use an existing identity that's created by AKS or create your own. Both options are described here.

Note

If you are using the user-assigned managed identity, we recommend that you directly configure Prometheus running on your Kubernetes cluster to remote-write into Azure Monitor Workspace. See Send Prometheus data to Azure Monitor using user-assigned managed identity to learn more. The steps below use the Azure Monitor side car container.

Cluster configurations

This article applies to the following cluster configurations:

  • Azure Kubernetes Service cluster
  • Azure Arc-enabled Kubernetes cluster

Note

For information about setting up remote write for a Kubernetes cluster running in a different cloud or on-premises, see Send Prometheus data to Azure Monitor by using Microsoft Entra authentication.

Prerequisites

Supported versions

Prometheus versions greater than v2.45 are required for managed identity authentication.

Azure Monitor workspace

This article covers sending Prometheus metrics to an Azure Monitor workspace. To create an Azure monitor workspace, see Manage an Azure Monitor workspace.

Permissions

Administrator permissions for the cluster or resource are required to complete the steps in this article.

Set up an application for managed identity

The process to set up Prometheus remote write for an application by using managed identity authentication involves completing the following tasks:

  1. Get the name of the AKS node resource group.
  2. Get the client ID of the user-assigned managed identity.
  3. Assign the Monitoring Metrics Publisher role on the workspace data collection rule to the managed identity.
  4. Give the AKS cluster access to the managed identity.
  5. Deploy a sidecar container to set up remote write.

The tasks are described in the following sections.

Get the name of the AKS node resource group

The node resource group of the AKS cluster contains resources that you use in other steps in this process. This resource group has the name MC_<AKS-RESOURCE-GROUP>_<AKS-CLUSTER-NAME>_<REGION>. You can find the resource group name by using the Resource groups menu in the Azure portal.

Screenshot that shows a list of resource groups.

Get the client ID of the user-assigned managed identity

You must get the client ID of the identity that you're going to use. Copy the client ID to use later in the process.

Instead of creating your own client ID, you can use one of the identities that are created by AKS. To learn more about the identities, see Use a managed identity in Azure Kubernetes Service.

This article uses the kubelet identity. The name of this identity is <AKS-CLUSTER-NAME>-agentpool, and it's in the node resource group of the AKS cluster.

Screenshot that shows a list of resources that are in the node resource group.

Select the <AKS-CLUSTER-NAME>-agentpool managed identity. On the Overview page, copy the value for Client ID. For more information, see Manage user-assigned managed identities.

Screenshot that shows a client ID on an overview page for a managed identity.

Assign the Monitoring Metrics Publisher role on the workspace data collection rule to the managed identity

The managed identity must be assigned the Monitoring Metrics Publisher role on the data collection rule that is associated with your Azure Monitor workspace.

  1. On the resource menu for your Azure Monitor workspace, select Overview. For Data collection rule, select the link.

    Screenshot that shows the data collection rule that's associated with an Azure Monitor workspace.

  2. On the resource menu for the data collection rule, select Access control (IAM).

  3. Select Add, and then select Add role assignment.

    Screenshot that shows adding a role assignment on Access control pages.

  4. Select the Monitoring Metrics Publisher role, and then select Next.

    Screenshot that shows a list of role assignments.

  5. Select Managed Identity, and then choose Select members. Select the subscription that contains the user-assigned identity, and then select User-assigned managed identity. Select the user-assigned identity that you want to use, and then choose Select.

    Screenshot that shows selecting a user-assigned managed identity.

  6. To complete the role assignment, select Review + assign.

Give the AKS cluster access to the managed identity

This step isn't required if you're using an AKS identity. An AKS identity already has access to the cluster.

Important

To complete the steps in this section, you must have owner or user access administrator permissions for the cluster.

  1. Identify the virtual machine scale sets in the node resource group for your AKS cluster.

    Screenshot that shows virtual machine scale sets in the node resource group.

  2. For each virtual machine scale set, run the following command in the Azure CLI:

    az vmss identity assign -g <AKS-NODE-RESOURCE-GROUP> -n <AKS-VMSS-NAME> --identities <USER-ASSIGNED-IDENTITY-RESOURCE-ID>

Deploy a sidecar container to set up remote write

  1. Copy the following YAML and save it to a file. The YAML uses port 8081 as the listening port. If you use a different port, modify the port in the YAML.

    prometheus:
  prometheusSpec:
    externalLabels:
          cluster: <AKS-CLUSTER-NAME>

    ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write    
    remoteWrite:
    - url: 'http://localhost:8081/api/v1/write'
  ## Azure Managed Prometheus currently exports some default mixins in Grafana. 
  ## These mixins are compatible with Azure Monitor agent on your Azure Kubernetes Service cluster. 
  ## However, these mixins aren't compatible with Prometheus metrics scraped by the Kube Prometheus stack. 
  ## In order to make these mixins compatible, uncomment remote write relabel configuration below:

  ## writeRelabelConfigs:
  ##   - sourceLabels: [metrics_path]
  ##     regex: /metrics/cadvisor
  ##     targetLabel: job
  ##     replacement: cadvisor
  ##     action: replace
  ##   - sourceLabels: [job]
  ##     regex: 'node-exporter'
  ##     targetLabel: job
  ##     replacement: node
  ##     action: replace
    containers:
    - name: prom-remotewrite
      image: <CONTAINER-IMAGE-VERSION>
      imagePullPolicy: Always
      ports:
        - name: rw-port
          containerPort: 8081
      livenessProbe:
        httpGet:
          path: /health
          port: rw-port
        initialDelaySeconds: 10
        timeoutSeconds: 10
      readinessProbe:
        httpGet:
          path: /ready
          port: rw-port
        initialDelaySeconds: 10
        timeoutSeconds: 10
      env:
      - name: INGESTION_URL
        value: <INGESTION_URL>
      - name: LISTENING_PORT
        value: '8081'
      - name: IDENTITY_TYPE
        value: userAssigned
      - name: AZURE_CLIENT_ID
        value: <MANAGED-IDENTITY-CLIENT-ID>
      # Optional parameter
      - name: CLUSTER
        value: <CLUSTER-NAME>

  2. Replace the following values in the YAML:

    Value Description
    <AKS-CLUSTER-NAME> The name of your AKS cluster.
    <CONTAINER-IMAGE-VERSION> mcr.microsoft.com/azuremonitor/containerinsights/ciprod/prometheus-remote-write/images:prom-remotewrite-20240617.1
    The remote write container image version.
    <INGESTION-URL> The value for Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace.
    <MANAGED-IDENTITY-CLIENT-ID> The value for Client ID from the Overview page for the managed identity.
    <CLUSTER-NAME> Name of the cluster that Prometheus is running on.

    Important

    For Microsoft Azure operated by 21Vianet cloud, add the following environment variables in the env section of the YAML file:

    - name: INGESTION_AAD_AUDIENCE value: https://monitor.azure.cn/

  1. Use Helm to apply the YAML file and update your Prometheus configuration:

    # set context to your cluster 
az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name> 

# use Helm to update your remote write config 
helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack --namespace <namespace where Prometheus pod resides>

Verification and troubleshooting

For verification and troubleshooting information, see Troubleshooting remote write and Azure Monitor managed service for Prometheus remote write.

Next steps