教程:在 Azure 数据资源管理器中引入和查询监视数据Tutorial: Ingest and query monitoring data in Azure Data Explorer

本教程介绍如何在不编写代码的情况下,将数据从诊断日志和活动日志引入到 Azure 数据资源管理器群集。This tutorial will teach you how to ingest data from diagnostic and activity logs to an Azure Data Explorer cluster without writing code. 使用这种简单的引入方法,可快速开始查询 Azure 数据资源管理器,从而进行数据分析。With this simple ingestion method, you can quickly begin querying Azure Data Explorer for data analysis.

本教程介绍以下操作:In this tutorial, you'll learn how to:

  • 在 Azure 数据资源管理器数据库中创建表和引入映射。Create tables and ingestion mapping in an Azure Data Explorer database.
  • 使用更新策略设置引入数据的格式。Format the ingested data by using an update policy.
  • 创建事件中心并将其连接到 Azure 数据资源管理器。Create an event hub and connect it to Azure Data Explorer.
  • 将 Azure Monitor 诊断指标和日志以及活动日志中的数据流式传输到事件中心。Stream data to an event hub from Azure Monitor diagnostic metrics and logs and activity logs.
  • 使用 Azure 数据资源管理器查询引入的数据。Query the ingested data by using Azure Data Explorer.

备注

在同一 Azure 位置或区域中创建所有资源。Create all resources in the same Azure location or region.

先决条件Prerequisites

Azure Monitor 数据提供程序:诊断指标和日志以及活动日志Azure Monitor data provider: diagnostic metrics and logs and activity logs

查看并了解以下 Azure Monitor 诊断指标和日志以及活动日志提供的数据。View and understand the data provided by the Azure Monitor diagnostic metrics and logs and activity logs below. 你将基于这些数据架构创建引入管道。You'll create an ingestion pipeline based on these data schemas. 请注意,日志中的每个事件都具有记录数组。Note that each event in a log has an array of records. 稍后在本教程中,将拆分此记录数组。This array of records will be split later in the tutorial.

诊断指标和日志以及活动日志的示例Examples of diagnostic metrics and logs and activity logs

Azure 诊断指标和日志以及活动日志是由 Azure 服务发出的,用于提供与该服务的操作相关的数据。Azure diagnostic metrics and logs and activity logs are emitted by an Azure service and provide data about the operation of that service.

示例Example

诊断指标以 1 分钟的时间粒度聚合。Diagnostic metrics are aggregated with a time grain of 1 minute. 下面是有关查询持续时间的 Azure 数据资源管理器指标事件架构示例:Following is an example of an Azure Data Explorer metric-event schema on query duration:

{
    "records": [
    {
        "count": 14,
        "total": 0,
        "minimum": 0,
        "maximum": 0,
        "average": 0,
        "resourceId": "/SUBSCRIPTIONS/<subscriptionID>/RESOURCEGROUPS/<resource-group>/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/<cluster-name>",
        "time": "2018-12-20T17:00:00.0000000Z",
        "metricName": "QueryDuration",
        "timeGrain": "PT1M"
    },
    {
        "count": 12,
        "total": 0,
        "minimum": 0,
        "maximum": 0,
        "average": 0,
        "resourceId": "/SUBSCRIPTIONS/<subscriptionID>/RESOURCEGROUPS/<resource-group>/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/<cluster-name>",
        "time": "2018-12-21T17:00:00.0000000Z",
        "metricName": "QueryDuration",
        "timeGrain": "PT1M"
    }
    ]
}

在 Azure 数据资源管理器中设置引入管道Set up an ingestion pipeline in Azure Data Explorer

设置 Azure 数据资源管理器管道需要执行若干步骤,如表创建和数据引入Setting up an Azure Data Explorer pipeline involves several steps, such as table creation and data ingestion. 此外,你可以处理、映射和更新数据。You can also manipulate, map, and update the data.

连接到 Azure 数据资源管理器 Web UIConnect to the Azure Data Explorer Web UI

在 Azure 数据资源管理器的 TestDatabase 数据库中,选择“查询”打开 Azure 数据资源管理器 Web UI 。In your Azure Data Explorer TestDatabase database, select Query to open the Azure Data Explorer Web UI.

查询页

创建目标表Create the target tables

Azure Monitor 日志的结构不是表格。The structure of the Azure Monitor logs isn't tabular. 你将操纵数据并将每个事件扩展到一个或多个记录。You'll manipulate the data and expand each event to one or more records. 将原始数据引入到活动日志的中间表 ActivityLogsRawRecords 以及诊断指标和日志的中间表 DiagnosticRawRecordsThe raw data will be ingested to an intermediate table named ActivityLogsRawRecords for activity logs and DiagnosticRawRecords for diagnostic metrics and logs. 此时,数据已经过处理和扩展。At that time, the data will be manipulated and expanded. 接着,使用更新策略将扩展的数据引入到活动日志的 ActivityLogs 表、诊断指标的 DiagnosticMetrics 表以及诊断日志的 DiagnosticLogs 表。Using an update policy, the expanded data will then be ingested into the ActivityLogs table for activity logs, DiagnosticMetrics for diagnostic metrics and DiagnosticLogs for diagnostic logs. 这意味着需要创建两个单独的表来引入活动日志,并创建三个单独的表来引入诊断指标和日志。This means that you'll need to create two separate tables for ingesting activity logs and three separate tables for ingesting diagnostic metrics and logs.

使用 Azure 数据资源管理器 Web UI 在 Azure 数据资源管理器数据库中创建目标表。Use the Azure Data Explorer Web UI to create the target tables in the Azure Data Explorer database.

创建诊断指标表Create tables for the diagnostic metrics

  1. TestDatabase 数据库中,创建名为 DiagnosticMetrics 的表来存储诊断指标记录。In the TestDatabase database, create a table named DiagnosticMetrics to store the diagnostic metrics records. 使用以下 .create table 控制命令:Use the following .create table control command:

    .create table DiagnosticMetrics (Timestamp:datetime, ResourceId:string, MetricName:string, Count:int, Total:double, Minimum:double, Maximum:double, Average:double, TimeGrain:string)
    
  2. 选择“运行”以创建该表 。Select Run to create the table.

    运行查询

  3. 使用以下查询在 TestDatabase 数据库中创建名为 DiagnosticRawRecords 的中间数据表,以进行数据操作。Create the intermediate data table named DiagnosticRawRecords in the TestDatabase database for data manipulation using the following query. 选择“运行”以创建该表 。Select Run to create the table.

    .create table DiagnosticRawRecords (Records:dynamic)
    
  4. 为中间表设置零保留策略Set zero retention policy for the intermediate table:

    .alter-merge table DiagnosticRawRecords policy retention softdelete = 0d
    

创建表映射Create table mappings

由于数据格式为 json,因此需要数据映射。Because the data format is json, data mapping is required. json 映射将每个 JSON 路径映射到表列名。The json mapping maps each json path to a table column name.

将诊断指标和日志映射到表Map diagnostic metrics and logs to the table

若要将诊断指标和日志数据映射到表,请使用以下查询:To map the diagnostic metric and log data to the table, use the following query:

.create table DiagnosticRawRecords ingestion json mapping 'DiagnosticRawRecordsMapping' '[{"column":"Records","Properties":{"path":"$.records"}}]'

为指标和日志数据创建更新策略Create the update policy for metric and log data

为诊断指标创建数据更新策略Create data update policy for diagnostics metrics

  1. 创建一个函数来扩展诊断指标记录集合,使集合中的每个值收到一个单独的行。Create a function that expands the collection of diagnostic metric records so that each value in the collection receives a separate row. 使用 mv-expand 运算符:Use the mv-expand operator:

    .create function DiagnosticMetricsExpand() {
       DiagnosticRawRecords
       | mv-expand events = Records
       | where isnotempty(events.metricName)
       | project
           Timestamp = todatetime(events['time']),
           ResourceId = tostring(events.resourceId),
           MetricName = tostring(events.metricName),
           Count = toint(events['count']),
           Total = todouble(events.total),
           Minimum = todouble(events.minimum),
           Maximum = todouble(events.maximum),
           Average = todouble(events.average),
           TimeGrain = tostring(events.timeGrain)
    }
    
  2. 更新策略添加到目标表。Add the update policy to the target table. 此策略将针对 DiagnosticRawRecords 中间数据表中任何新引入的数据自动运行查询,并将查询结果引入到 DiagnosticMetrics 表中:This policy will automatically run the query on any newly ingested data in the DiagnosticRawRecords intermediate data table and ingest its results into the DiagnosticMetrics table:

    .alter table DiagnosticMetrics policy update @'[{"Source": "DiagnosticRawRecords", "Query": "DiagnosticMetricsExpand()", "IsEnabled": "True", "IsTransactional": true}]'
    

创建一个 Azure 事件中心命名空间Create an Azure Event Hubs namespace

通过 Azure 诊断设置,能够将指标和日志导出到存储帐户或事件中心。Azure diagnostic settings enable exporting metrics and logs to a storage account or to an event hub. 本教程通过事件中心路由指标和日志。In this tutorial, we'll route the metrics and logs via an event hub. 按照以下步骤,为诊断指标和日志创建事件中心命名空间和事件中心。You'll create an Event Hubs namespace and an event hub for the diagnostic metrics and logs in the following steps. Azure Monitor 将为活动日志创建事件中心 insights-operational-logs 。Azure Monitor will create the event hub insights-operational-logs for the activity logs.

  1. 在 Azure 门户中使用 Azure 资源管理器模板创建事件中心。Create an event hub by using an Azure Resource Manager template in the Azure portal. 若要执行本文的剩余步骤,请右键单击“部署到 Azure”,然后选择“在新窗口中打开” 。To follow the rest of the steps in this article, right-click the Deploy to Azure button, and then select Open in new window. 单击“部署到 Azure”按钮可转到 Azure 门户。 The Deploy to Azure button takes you to the Azure portal.

    “部署到 Azure”按钮Deploy to Azure button

  2. 为诊断日志创建事件中心命名空间和事件中心。Create an Event Hubs namespace and an event hub for the diagnostic logs.

    创建事件中心

  3. 使用以下信息填写窗体。Fill out the form with the following information. 对于下表中未列出的任何设置,请使用默认值。For any settings not listed in the following table, use the default values.

    设置Setting 建议的值Suggested value 说明Description
    订阅Subscription 订阅Your subscription 选择要用于事件中心的 Azure 订阅。Select the Azure subscription that you want to use for your event hub.
    资源组Resource group test-resource-grouptest-resource-group 创建新的资源组。Create a new resource group.
    位置Location 选择最符合需求的区域。Select the region that best meets your needs. 在其他资源所在的同一位置创建事件中心命名空间。Create the Event Hubs namespace in the same location as other resources.
    命名空间名称Namespace name AzureMonitoringDataAzureMonitoringData 选择用于标识命名空间的唯一名称。Choose a unique name that identifies your namespace.
    事件中心名称Event hub name DiagnosticDataDiagnosticData 事件中心位于命名空间下,该命名空间提供唯一的范围容器。The event hub sits under the namespace, which provides a unique scoping container.
    使用者组名称Consumer group name adxpipelineadxpipeline 创建使用者组名称。Create a consumer group name. 使用者组允许多个使用应用程序各自具有事件流的单独视图。Consumer groups enable multiple consuming applications to each have a separate view of the event stream.

将 Azure Monitor 指标和日志连接到事件中心Connect Azure Monitor metrics and logs to your event hub

现在需要将诊断指标和日志以及活动日志连接到事件中心。Now you need to connect your diagnostic metrics and logs and your activity logs to the event hub.

将诊断指标和日志连接到事件中心Connect diagnostic metrics and logs to your event hub

选择要从其中导出指标的资源。Select a resource from which to export metrics. 支持导出诊断数据的资源类型有多种,包括事件中心命名空间、Azure Key Vault、Azure IoT 中心和 Azure 数据资源管理器群集。Several resource types support exporting diagnostic data, including Event Hubs namespace, Azure Key Vault, Azure IoT Hub, and Azure Data Explorer clusters. 在本教程中,我们将使用 Azure 数据资源管理器群集作为资源,并查看查询性能指标和引入结果日志。In this tutorial, we'll use an Azure Data Explorer cluster as our resource, we'll review query performance metrics and ingestion results logs.

  1. 在 Azure 门户中选择你的 Kusto 群集。Select your Kusto cluster in the Azure portal.

  2. 选择“诊断设置”,然后选择“启用诊断”链接 。Select Diagnostic settings, and then select the Turn on diagnostics link.

    诊断设置

  3. “诊断设置”窗格打开 。The Diagnostics settings pane opens. 执行以下步骤:Take the following steps:

    1. 将诊断日志数据命名为 ADXExportedData 。Give your diagnostics log data the name ADXExportedData.

    2. 在“日志”下,选中“SucceededIngestion”和“FailedIngestion”复选框 。Under LOG, select both SucceededIngestion and FailedIngestion check boxes.

    3. 在“指标”下,选中“查询性能”复选框 。Under METRIC, select the Query performance check box.

    4. 选择“流式传输到事件中心”复选框 。Select the Stream to an event hub check box.

    5. 选择“配置” 。Select Configure.

      诊断设置窗格

  4. 在“选择事件中心”窗格中,配置将数据从诊断日志导出到所创建事件中心的方法 :In the Select event hub pane, configure how to export data from diagnostic logs to the event hub you created:

    1. 在“选择事件中心命名空间”列表中,选择 AzureMonitoringData 。In the Select event hub namespace list, select AzureMonitoringData.
    2. 在“选择事件中心名称”列表中,选择“DiagnosticData” 。In the Select event hub name list, select DiagnosticData.
    3. 在“选择事件中心策略名称”列表中,选择 RootManagerSharedAccessKey 。In the Select event hub policy name list, select RootManagerSharedAccessKey.
    4. 选择“确定” 。Select OK.
  5. 选择“保存” 。Select Save.

查看流入事件中心的数据See data flowing to your event hubs

  1. 需等待几分钟,才能定义连接,完成活动日志到事件中心的数据导出。Wait a few minutes until the connection is defined, and the activity-log export to the event hub is finished. 转到事件中心命名空间,查看创建的事件中心。Go to your Event Hubs namespace to see the event hubs you created.

    创建的事件中心

  2. 查看流入事件中心的数据:See data flowing to your event hub:

    事件中心的数据

将事件中心连接到 Azure 数据资源管理器Connect an event hub to Azure Data Explorer

现在需要为诊断指标和日志以及活动日志创建数据连接。Now you need to create the data connections for your diagnostic metrics and logs and activity logs.

为诊断指标和日志以及活动日志创建数据连接Create the data connection for diagnostic metrics and logs and activity logs

  1. 在名为 kustodocs 的 Azure 数据资源管理器群集中,选择左侧菜单中的“数据库” 。In your Azure Data Explorer cluster named kustodocs, select Databases in the left menu.

  2. 在“数据库”窗口中,选择 TestDatabase 数据库 。In the Databases window, select your TestDatabase database.

  3. 在左侧菜单中,选择“数据引入” 。In the left menu, select Data ingestion.

  4. 在“数据引入”窗口中,单击“+ 添加数据连接” 。In the Data ingestion window, click + Add Data Connection.

  5. 在“数据连接”窗口中输入以下信息: In the Data connection window, enter the following information:

    事件中心数据连接

  1. 在“数据连接”窗口中使用以下设置 :Use the following settings in the Data Connection window:

    数据源:Data source:

    设置Setting 建议的值Suggested value 字段说明Field description
    数据连接名称Data connection name DiagnosticsLogsConnectionDiagnosticsLogsConnection 要在 Azure 数据资源管理器中创建的连接的名称。The name of the connection you want to create in Azure Data Explorer.
    事件中心命名空间Event hub namespace AzureMonitoringDataAzureMonitoringData 先前选择的用于标识命名空间的名称。The name you chose earlier that identifies your namespace.
    事件中心Event hub DiagnosticDataDiagnosticData 你创建的事件中心。The event hub you created.
    使用者组Consumer group adxpipelineadxpipeline 在创建的事件中心定义的使用者组。The consumer group defined in the event hub you created.

    目标表:Target table:

    有两个路由选项:静态和动态。 There are two options for routing: static and dynamic. 本教程将使用静态路由(默认),需在其中指定表名、数据格式和映射。For this tutorial, you'll use static routing (the default), where you specify the table name, the data format, and the mapping. 让“我的数据包含路由信息”保持取消选中状态。 Leave My data includes routing info unselected.

    设置Setting 建议的值Suggested value 字段说明Field description
    Table DiagnosticRawRecordsDiagnosticRawRecords 在 TestDatabase 数据库中创建的表 。The table you created in the TestDatabase database.
    数据格式Data format JSONJSON 表中使用的格式。The format used in the table.
    列映射Column mapping DiagnosticRawRecordsMappingDiagnosticRawRecordsMapping TestDatabase 数据库中创建的映射,它将传入的 JSON 数据映射到 DiagnosticRawRecords 表的列名和数据类型。The mapping you created in the TestDatabase database, which maps incoming JSON data to the column names and data types of the DiagnosticRawRecords table.
  2. 选择“创建” 。Select Create.

查询新表Query the new tables

现已创建用于流送数据的管道。You now have a pipeline with data flowing. 默认情况下,通过群集引入数据需要 5 分钟,因此,请先让数据流动几分钟,然后再开始查询。Ingestion via the cluster takes 5 minutes by default, so allow the data to flow for a few minutes before beginning to query.

查询诊断指标表Query the diagnostic metrics table

以下查询分析 Azure 数据资源管理器诊断指标记录中的查询持续时间数据:The following query analyzes query duration data from diagnostic metric records in Azure Data Explorer:

DiagnosticMetrics
| where Timestamp > ago(15m) and MetricName == 'QueryDuration'
| summarize avg(Average)

查询结果:Query results:

avg_Averageavg_Average
00:06.15600:06.156

后续步骤Next steps