使用诊断日志监视 Azure 数据资源管理器的引入、命令和查询Monitor Azure Data Explorer ingestion, commands, and queries using diagnostic logs

Azure 数据资源管理器是一项快速、完全托管的数据分析服务,用于实时分析从应用程序、网站和 IoT 设备等资源流式传输的海量数据。Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Monitor 诊断日志提供有关 Azure 资源操作的数据。Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure 数据资源管理器使用诊断日志获取有关引入成功、引入失败、命令和查询操作的见解。Azure Data Explorer uses diagnostic logs for insights on ingestion successes, ingestion failures, commands, and query operations. 可将操作日志导出到 Azure 存储、事件中心或 Log Analytics 以监视引入、命令和查询状态。You can export operation logs to Azure Storage, Event Hub, or Log Analytics to monitor ingestion, commands, and query status. 可将 Azure 存储和 Azure 事件中心的日志路由到 Azure 数据资源管理器群集中的某个表,以进一步分析。Logs from Azure Storage and Azure Event Hub can be routed to a table in your Azure Data Explorer cluster for further analysis.

重要

诊断日志数据可能包含敏感数据。Diagnostic log data may contain sensitive data. 请根据监视需求限制日志目标的权限。Restrict permissions of the logs destination according to your monitoring needs.

先决条件Prerequisites

设置 Azure 数据资源管理器群集的诊断日志Set up diagnostic logs for an Azure Data Explorer cluster

诊断日志可用于配置以下日志数据的收集:Diagnostic logs can be used to configure the collection of the following log data:

  • 成功的引入操作 :这些日志包含有关已成功完成的引入操作的信息。Successful ingestion operations : These logs have information about successfully completed ingestion operations.
  • 失败的引入操作 :这些日志包含有关失败的引入操作的详细信息,包括错误详细信息。Failed ingestion operations : These logs have detailed information about failed ingestion operations including error details.

然后可根据规范将数据存档到存储帐户、流式传输到事件中心,或发送到 Log Analytics。The data is then archived into a Storage account, streamed to an Event Hub, or sent to Log Analytics, as per your specifications.

启用诊断日志Enable diagnostic logs

诊断日志默认已禁用。Diagnostic logs are disabled by default. 若要启用诊断日志,请执行以下步骤:To enable diagnostic logs, do the following steps:

  1. Azure 门户中,选择要监视的 Azure 数据资源管理器群集资源。In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.

  2. 在“监视”下,选择“诊断设置” 。Under Monitoring , select Diagnostic settings.

    添加诊断日志

  3. 选择“添加诊断设置”。 Select Add diagnostic setting.

  4. 在“诊断设置”窗口中: In the Diagnostics settings window:

    配置诊断设置

    1. 选择诊断设置的 名称Select Name for your diagnostic setting.
    2. 选择一个或多个目标:存储帐户、事件中心或 Log Analytics。Select one or more targets: a Storage account, Event Hub, or Log Analytics.
    3. 选择要收集的日志:SucceededIngestionFailedIngestionCommandQuerySelect logs to be collected: SucceededIngestion, FailedIngestion, Command, or Query.
    4. 选择要收集的指标(可选)。Select metrics to be collected (optional).
    5. 选择“保存”以保存新的诊断日志设置和指标。 Select Save to save the new diagnostic logs settings and metrics.

在几分钟内即会完成新的设置。New settings will be set in a few minutes. 日志随后会显示在配置的存档目标(存储帐户、事件中心或 Log Analytics)中。Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).

备注

如果将日志发送到 Log Analytics,则 SucceededIngestionFailedIngestionCommandQuery 日志将分别存储在名为 SucceededIngestionFailedIngestionADXCommandADXQuery 的 Log Analytics 表中。If you send logs to Log Analytics, the SucceededIngestion, FailedIngestion, Command, and Query logs will be stored in Log Analytics tables named: SucceededIngestion, FailedIngestion, ADXCommand, ADXQuery, respectively.

诊断日志架构Diagnostic logs schema

所有 Azure Monitor 诊断日志共享一个通用的顶级架构All Azure Monitor diagnostic logs share a common top-level schema. Azure 数据资源管理器对其自身的事件使用唯一属性。Azure Data Explorer has unique properties for their own events. 所有日志均以 JSON 格式存储。All logs are stored in a JSON format.

引入日志架构Ingestion logs schema

日志 JSON 字符串包含下表中列出的元素:Log JSON strings include elements listed in the following table:

名称Name 说明Description
timetime 报告时间Time of the report
ResourceIdresourceId Azure Resource Manager 资源 IDAzure Resource Manager resource ID
operationNameoperationName 操作名称:'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'
operationVersionoperationVersion 架构版本:'1.0'Schema version: '1.0'
categorycategory 操作类别。Category of the operation. SucceededIngestionFailedIngestionSucceededIngestion or FailedIngestion. 成功的操作失败的操作的属性不同。Properties differ for successful operation or failed operation.
propertiesproperties 操作的详细信息。Detailed information of the operation.

成功引入操作日志Successful ingestion operation log

示例:Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "SucceededIngestion",
    "properties":
    {
        "succeededOn": "2019-05-27 07:55:05.3693628",
        "operationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events8347293.json",
        "rootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
    }
}

成功操作诊断日志的属性Properties of a successful operation diagnostic log

名称Name 说明Description
succeededOnsucceededOn 引入完成时间Time of ingestion completion
operationIdoperationId Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
databasedatabase 目标数据库的名称Name of the target database
table 目标表的名称Name of the target table
ingestionSourceIdingestionSourceId 引入数据源的 IDID of the ingestion data source
ingestionSourcePathingestionSourcePath 引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
rootActivityIdrootActivityId 活动 IDActivity ID

失败引入操作日志Failed ingestion operation log

示例:Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "FailedIngestion",
    "properties":
    {
        "failedOn": "2019-05-27 08:57:05.4273524",
        "operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events5725592.json",
        "rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
        "details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
        "errorCode": "Download_SourceNotFound",
        "failureStatus": "Permanent",
        "originatesFromUpdatePolicy": false,
        "shouldRetry": false
    }
}

失败操作诊断日志的属性Properties of a failed operation diagnostic log

名称Name 说明Description
failedOnfailedOn 引入完成时间Time of ingestion completion
operationIdoperationId Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
databasedatabase 目标数据库的名称Name of the target database
table 目标表的名称Name of the target table
ingestionSourceIdingestionSourceId 引入数据源的 IDID of the ingestion data source
ingestionSourcePathingestionSourcePath 引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
rootActivityIdrootActivityId 活动 IDActivity ID
详细信息details 失败和错误消息的详细说明Detailed description of the failure and error message
errorCodeerrorCode 错误代码Error code
failureStatusfailureStatus PermanentTransientPermanent or Transient. 重试暂时性故障可能会成功。Retry of a transient failure may succeed.
originatesFromUpdatePolicyoriginatesFromUpdatePolicy 如果故障源自更新策略,则为 TrueTrue if failure originates from an update policy
shouldRetryshouldRetry 如果重试可以成功,则为 TrueTrue if retry may succeed

后续步骤Next steps