将 Prometheus 指标发送到多个 Azure Monitor 工作区

项目
2024/05/24

可以通过创建额外的数据收集规则将不同的指标路由到更多 Azure Monitor 工作区。

可以创建具有相应数据收集终结点的数据收集规则，以便将不同的指标从同一个 Kubernetes 群集发送到其他 Azure Monitor 工作区。目前，只能通过资源管理器模板以加入的方式做到这一点。可以遵循常规加入流程，然后编辑相同的资源管理器模板，以便为其他 Azure Monitor 工作区添加其他 DCR 和 DCE。需要编辑模板，为每个额外的 Azure Monitor 工作区添加一个额外的参数，为每个额外的 Azure Monitor 工作区添加另一个 DCR，添加另一个 DCE，为新的 Azure Monitor 工作区添加监视器读取者角色，并为 Grafana 添加一个额外的 Azure Monitor 工作区集成。

添加以下参数：

"parameters": {
  "azureMonitorWorkspaceResourceId2": {
    "type": "string"
  },
  "azureMonitorWorkspaceLocation2": {
    "type": "string",
    "defaultValue": "",
    "allowedValues": [
      "chinanorth3"
    ]
  },
...
}

添加一个额外的数据收集终结点。必须替换 <dceName>：

  {
    "type": "Microsoft.Insights/dataCollectionEndpoints",
    "apiVersion": "2021-09-01-preview",
    "name": "[variables('dceName')]",
    "location": "[parameters('azureMonitorWorkspaceLocation2')]",
    "kind": "Linux",
    "properties": {}
  }

添加使用新数据收集终结点的额外 DCR。必须替换 <dcrName>：

{
  "type": "Microsoft.Insights/dataCollectionRules",
  "apiVersion": "2021-09-01-preview",
  "name": "<dcrName>",
  "location": "[parameters('azureMonitorWorkspaceLocation2')]",
  "kind": "Linux",
  "properties": {
    "dataCollectionEndpointId": "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
    "dataFlows": [
      {
        "destinations": ["MonitoringAccount2"],
        "streams": ["Microsoft-PrometheusMetrics"]
      }
    ],
    "dataSources": {
      "prometheusForwarder": [
        {
          "name": "PrometheusDataSource",
          "streams": ["Microsoft-PrometheusMetrics"],
          "labelIncludeFilter": 
                  "microsoft_metrics_include_label": "MonitoringAccountLabel2"
        }
      ]
    },
    "description": "DCR for Azure Monitor Metrics Profile (Managed Prometheus)",
    "destinations": {
      "monitoringAccounts": [
        {
          "accountResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]",
          "name": "MonitoringAccount2"
        }
      ]
    }
  },
  "dependsOn": [
    "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]"
  ]
}

使用相关的数据收集规则 (DCR) 添加其他数据收集规则关联 (DCRA)。这会将 DCR 与群集相关联。必须替换 <dcraName>：

{
 "type": "Microsoft.Resources/deployments",
 "name": "<dcraName>",
 "apiVersion": "2017-05-10",
 "subscriptionId": "[variables('clusterSubscriptionId')]",
 "resourceGroup": "[variables('clusterResourceGroup')]",
 "dependsOn": [
   "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
   "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
 ],
 "properties": {
   "mode": "Incremental",
   "template": {
     "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
     "contentVersion": "1.0.0.0",
     "parameters": {},
     "variables": {},
     "resources": [
       {
         "type": "Microsoft.ContainerService/managedClusters/providers/dataCollectionRuleAssociations",
         "name": "[concat(variables('clusterName'),'/microsoft.insights/', variables('dcraName'))]",
         "apiVersion": "2021-09-01-preview",
         "location": "[parameters('clusterLocation')]",
         "properties": {
           "description": "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster.",
           "dataCollectionRuleId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
         }
       }
     ]
   },
   "parameters": {}
 }
}

添加额外的 Grafana 集成：

{
      "type": "Microsoft.Dashboard/grafana",
      "apiVersion": "2022-08-01",
      "name": "[split(parameters('grafanaResourceId'),'/')[8]]",
      "sku": {
        "name": "[parameters('grafanaSku')]"
      },
      "location": "[parameters('grafanaLocation')]",
      "properties": {
        "grafanaIntegrations": {
          "azureMonitorWorkspaceIntegrations": [
            // Existing azureMonitorWorkspaceIntegrations values (if any)
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"
            },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]"
            }
          ]
        }
      }
    }

分配 Monitoring Data Reader 角色以从新的 Azure Monitor 工作区读取数据：

{
  "type": "Microsoft.Authorization/roleAssignments",
  "apiVersion": "2022-04-01",
  "name": "[parameters('roleNameGuid')]",
  "scope": "[parameters('azureMonitorWorkspaceResourceId2')]",
  "properties": {
      "roleDefinitionId": "[concat('/subscriptions/', variables('clusterSubscriptionId'), '/providers/Microsoft.Authorization/roleDefinitions/', 'b0d8363b-8ddd-447d-831f-62ca05bff136')]",
      "principalId": "[reference(resourceId('Microsoft.Dashboard/grafana', split(parameters('grafanaResourceId'),'/')[8]), '2022-08-01', 'Full').identity.principalId]"
  }
}

然后配置将哪些指标路由到哪个工作区，方法是将一个额外的预定义标签 microsoft_metrics_account 添加到指标。值应与该工作区的 DCR 中相应的 microsoft_metrics_include_label 相同。若要将标签添加到指标，可以在抓取配置中使用 relabel_configs。若要将一个作业的所有指标发送到某个工作区，请添加以下重建标签配置：

relabel_configs:
- source_labels: [__address__]
  target_label: microsoft_metrics_account
  action: replace
  replacement: "MonitoringAccountLabel2"

源标签为 __address__，因为此标签始终存在。因此始终会应用此重建标签配置。目标标签始终为 microsoft_metrics_account，应将其值替换为工作区的相应标签值。

示例

如果你要配置三个不同的作业以将指标发送到三个不同的工作区，请在每个数据收集规则中包含以下内容：

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel1"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel2"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel3"
}

然后在抓取配置中，为每个规则包含相同的标签值：

scrape_configs:
- job_name: prometheus_ref_app_1
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-1"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel1"
- job_name: prometheus_ref_app_2
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-2"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel2"
- job_name: prometheus_ref_app_3
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-3"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel3"

AI 技能盛会

通过

将 Prometheus 指标发送到多个 Azure Monitor 工作区

示例

后续步骤

其他资源