使用诊断日志监视 Azure 数据资源管理器引入操作Monitor Azure Data Explorer ingestion operations using diagnostic logs

• 05/09/2020
• • 
  • 

Azure 数据资源管理器是一项快速、完全托管的数据分析服务，用于实时分析从应用程序、网站和 IoT 设备等资源流式传输的海量数据。Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. 若要使用 Azure 数据资源管理器，请先创建群集，再在该群集中创建一个或多个数据库。To use Azure Data Explorer, you first create a cluster, and create one or more databases in that cluster. 然后将数据引入（加载）到数据库中的某个表，以便对其运行查询。Then you ingest (load) data into a table in a database so that you can run queries against it. Azure Monitor 诊断日志提供有关 Azure 资源操作的数据。Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure 数据资源管理器使用诊断日志获取有关引入成功和失败的见解。Azure Data Explorer uses diagnostic logs for insights on ingestion successes and failures. 可将操作日志导出到 Azure 存储、事件中心或 Log Analytics 以监视引入状态。You can export operation logs to Azure Storage, Event Hub, or Log Analytics to monitor ingestion status. 可将 Azure 存储和 Azure 事件中心的日志路由到 Azure 数据资源管理器群集中的某个表，以进一步分析。Logs from Azure Storage and Azure Event Hub can be routed to a table in your Azure Data Explorer cluster for further analysis.

先决条件Prerequisites

• 如果没有 Azure 订阅，请创建一个试用 Azure 帐户。If you don't have an Azure subscription, create a trial Azure account.
• 创建群集和数据库。Create a cluster and database.

登录到 Azure 门户Sign in to the Azure portal

登录到 Azure 门户。Sign in to the Azure portal.

设置 Azure 数据资源管理器群集的诊断日志Set up diagnostic logs for an Azure Data Explorer cluster

诊断日志可用于配置以下日志数据的收集：Diagnostic logs can be used to configure the collection of the following log data:

• 成功的引入操作：这些日志包含有关已成功完成的引入操作的信息。Successful ingestion operations: These logs have information about successfully completed ingestion operations.
• 失败的引入操作：这些日志包含有关失败的引入操作的详细信息，包括错误详细信息。Failed ingestion operations: These logs have detailed information about failed ingestion operations including error details.

然后可根据规范将数据存档到存储帐户、流式传输到事件中心，或发送到 Log Analytics。The data is then archived into a Storage account, streamed to an Event Hub, or sent to Log Analytics, as per your specifications.

启用诊断日志Enable diagnostic logs

诊断日志默认已禁用。Diagnostic logs are disabled by default. 若要启用诊断日志，请执行以下步骤：To enable diagnostic logs, do the following steps:

1. 在 Azure 门户中，选择要监视的 Azure 数据资源管理器群集资源。In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.

2. 在“监视”下，选择“诊断设置” 。Under Monitoring, select Diagnostic settings.

   

3. 选择“添加诊断设置”。 Select Add diagnostic setting.

4. 在“诊断设置”窗口中： In the Diagnostics settings window:

   

   1. 选择诊断设置的名称。Select Name for your diagnostic setting.
   2. 选择一个或多个目标：存储帐户、事件中心或 Log Analytics。Select one or more targets: a Storage account, Event Hub, or Log Analytics.
   3. 选择要收集的日志：SucceededIngestion 或 FailedIngestion。Select logs to be collected: SucceededIngestion or FailedIngestion.
   4. 选择要收集的指标（可选）。Select metrics to be collected (optional).
   5. 选择“保存”以保存新的诊断日志设置和指标。 Select Save to save the new diagnostic logs settings and metrics.

在几分钟内即会完成新的设置。New settings will be set in a few minutes. 日志随后会显示在配置的存档目标（存储帐户、事件中心或 Log Analytics）中。Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).

诊断日志架构Diagnostic logs schema

所有 Azure Monitor 诊断日志共享一个通用的顶级架构。All Azure Monitor diagnostic logs share a common top-level schema. Azure 数据资源管理器对其自身的事件使用唯一属性。Azure Data Explorer has unique properties for their own events. 所有日志均以 JSON 格式存储。All logs are stored in a JSON format.

引入日志架构Ingestion logs schema

日志 JSON 字符串包含下表中列出的元素：Log JSON strings include elements listed in the following table:

成功引入操作日志Successful ingestion operation log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "SucceededIngestion",
    "properties":
    {
        "succeededOn": "2019-05-27 07:55:05.3693628",
        "operationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events8347293.json",
        "rootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
    }
}

成功操作诊断日志的属性Properties of a successful operation diagnostic log

失败引入操作日志Failed ingestion operation log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "FailedIngestion",
    "properties":
    {
        "failedOn": "2019-05-27 08:57:05.4273524",
        "operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events5725592.json",
        "rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
        "details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
        "errorCode": "Download_SourceNotFound",
        "failureStatus": "Permanent",
        "originatesFromUpdatePolicy": false,
        "shouldRetry": false
    }
}

失败操作诊断日志的属性Properties of a failed operation diagnostic log

后续步骤Next steps

• 教程：在 Azure 数据资源管理器中引入和查询监视数据Tutorial: Ingest and query monitoring data in Azure Data Explorer
• 使用指标来监视群集运行状况Use metrics to monitor cluster health