使用诊断日志监视 Azure 数据资源管理器的引入、命令、查询和表Monitor Azure Data Explorer ingestion, commands, queries, and tables using diagnostic logs

Azure 数据资源管理器是一项快速、完全托管的数据分析服务,用于实时分析从应用程序、网站和 IoT 设备等资源流式传输的海量数据。Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Monitor 诊断日志提供有关 Azure 资源操作的数据。Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure 数据资源管理器使用诊断日志获取有关引入、命令、查询和表的见解。Azure Data Explorer uses diagnostic logs for insights on ingestion, commands, query, and tables. 可将操作日志导出到 Azure 存储、事件中心或 Log Analytics 以监视引入、命令和查询状态。You can export operation logs to Azure Storage, Event Hub, or Log Analytics to monitor ingestion, commands, and query status. 可将 Azure 存储和 Azure 事件中心的日志路由到 Azure 数据资源管理器群集中的某个表,以进一步分析。Logs from Azure Storage and Azure Event Hub can be routed to a table in your Azure Data Explorer cluster for further analysis.

重要

诊断日志数据可能包含敏感数据。Diagnostic log data may contain sensitive data. 请根据监视需求限制日志目标的权限。Restrict permissions of the logs destination according to your monitoring needs.

先决条件Prerequisites

设置 Azure 数据资源管理器群集的诊断日志Set up diagnostic logs for an Azure Data Explorer cluster

诊断日志可用于配置以下日志数据的收集:Diagnostic logs can be used to configure the collection of the following log data:

备注

对于使用 SDK、数据连接和连接器将排队的引入内容引入到引入终结点,引入日志受支持。Ingestion logs are supported for queued ingestion to the ingestion endpoint using SDKs, data connections, and connectors.

对于流式引入、目标为引擎的直接引入、从查询进行的引入或者设置或追加命令,引入日志不受支持。Ingestion logs aren't supported for streaming ingestion, direct ingestion to the engine, ingestion from query, or set-or-append commands.

备注

只会针对引入操作的最终状态报告“失败引入”日志,这与引入结果指标不同,后者是针对在内部重试的暂时性故障发出的。Failed ingestion logs are only reported for the final state of an ingest operation, unlike the Ingestion result metric, which is emitted for transient failures that are retried internally.

  • 成功的引入操作:这些日志包含有关已成功完成的引入操作的信息。Successful ingestion operations: These logs have information about successfully completed ingestion operations.
  • 失败的引入操作:这些日志包含有关失败的引入操作的详细信息,包括错误详细信息。Failed ingestion operations: These logs have detailed information about failed ingestion operations including error details.
  • 引入批处理操作:这些日志详细说明了可用于引入的批处理的统计信息(持续时间、批大小和 blob 计数)。Ingestion batching operations: These logs have detailed statistics of batches ready for ingestion (duration, batch size and blobs count).

然后可根据规范将数据存档到存储帐户、流式传输到事件中心,或发送到 Log Analytics。The data is then archived into a Storage account, streamed to an Event Hub, or sent to Log Analytics, as per your specifications.

启用诊断日志Enable diagnostic logs

诊断日志默认已禁用。Diagnostic logs are disabled by default. 若要启用诊断日志,请执行以下步骤:To enable diagnostic logs, do the following steps:

  1. Azure 门户中,选择要监视的 Azure 数据资源管理器群集资源。In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.

  2. 在“监视”下,选择“诊断设置” 。Under Monitoring, select Diagnostic settings.

    添加诊断日志

  3. 选择“添加诊断设置”。 Select Add diagnostic setting.

  4. 在“诊断设置”窗口中,执行以下操作:In the Diagnostic settings window:

    配置诊断设置

    1. 输入一个诊断设置名称。Enter a Diagnostic setting name.
    2. 选择一个或多个目标:Log Analytics 工作区、存储帐户或事件中心。Select one or more targets: a Log Analytics workspace, a storage account, or an Event Hub.
    3. 选择要收集的日志:SucceededIngestionFailedIngestionIngestionBatchingCommandQueryTableUsageStatisticsTableDetailsSelect logs to be collected: SucceededIngestion, FailedIngestion, IngestionBatching, Command, or Query, TableUsageStatistics, or TableDetails.
    4. 选择要收集的指标(可选)。Select metrics to be collected (optional).
    5. 选择“保存”以保存新的诊断日志设置和指标。 Select Save to save the new diagnostic logs settings and metrics.

在几分钟内即会完成新的设置。New settings will be set in a few minutes. 日志随后会显示在配置的存档目标(存储帐户、事件中心或 Log Analytics)中。Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).

备注

如果将日志发送到 Log Analytics,则 SucceededIngestionFailedIngestionIngestionBatchingCommandQueryTableUsageStatisticsTableDetails 日志会分别存储在名为 SucceededIngestionFailedIngestionADXIngestionBatchingADXCommandADXQueryADXTableUsageStatisticsADXTableDetails 的 Log Analytics 表中。If you send logs to Log Analytics, the SucceededIngestion, FailedIngestion, IngestionBatching, Command, Query, TableUsageStatistics and TableDetails logs will be stored in Log Analytics tables named: SucceededIngestion, FailedIngestion, ADXIngestionBatching, ADXCommand, ADXQuery, ADXTableUsageStatistics and ADXTableDetails respectively.

诊断日志架构Diagnostic logs schema

所有 Azure Monitor 诊断日志共享一个通用的顶级架构All Azure Monitor diagnostic logs share a common top-level schema. Azure 数据资源管理器对其自身的事件使用唯一属性。Azure Data Explorer has unique properties for their own events. 所有日志均以 JSON 格式存储。All logs are stored in a JSON format.

引入日志架构Ingestion logs schema

日志 JSON 字符串包含下表中列出的元素:Log JSON strings include elements listed in the following table:

名称Name 说明Description
timetime 报告时间Time of the report
ResourceIdresourceId Azure Resource Manager 资源 IDAzure Resource Manager resource ID
operationNameoperationName 操作名称:'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'
operationVersionoperationVersion 架构版本:'1.0'Schema version: '1.0'
categorycategory 操作类别。Category of the operation. SucceededIngestionFailedIngestionIngestionBatchingSucceededIngestion, FailedIngestion or IngestionBatching. 成功的操作失败的操作批处理操作的属性不同。Properties differ for successful operation, failed operation or batching operation.
propertiesproperties 操作的详细信息。Detailed information of the operation.

成功引入操作日志Successful ingestion operation log

示例:Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "SucceededIngestion",
    "properties":
    {
        "SucceededOn": "2019-05-27 07:55:05.3693628",
        "OperationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
        "Database": "Samples",
        "Table": "StormEvents",
        "IngestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
        "IngestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events8347293.json",
        "RootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
    }
}

成功操作诊断日志的属性Properties of a successful operation diagnostic log

名称Name 说明Description
SucceededOnSucceededOn 引入完成时间Time of ingestion completion
OperationIdOperationId Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
数据库Database 目标数据库的名称Name of the target database
Table 目标表的名称Name of the target table
IngestionSourceIdIngestionSourceId 引入数据源的 IDID of the ingestion data source
IngestionSourcePathIngestionSourcePath 引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
RootActivityIdRootActivityId 活动 IDActivity ID

失败引入操作日志Failed ingestion operation log

示例:Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "FailedIngestion",
    "properties":
    {
        "failedOn": "2019-05-27 08:57:05.4273524",
        "operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events5725592.json",
        "rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
        "details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
        "errorCode": "Download_SourceNotFound",
        "failureStatus": "Permanent",
        "originatesFromUpdatePolicy": false,
        "shouldRetry": false
    }
}

失败操作诊断日志的属性Properties of a failed operation diagnostic log

名称Name 说明Description
FailedOnFailedOn 引入完成时间Time of ingestion completion
OperationIdOperationId Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
数据库Database 目标数据库的名称Name of the target database
Table 目标表的名称Name of the target table
IngestionSourceIdIngestionSourceId 引入数据源的 IDID of the ingestion data source
IngestionSourcePathIngestionSourcePath 引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
RootActivityIdRootActivityId 活动 IDActivity ID
详细信息Details 失败和错误消息的详细说明Detailed description of the failure and error message
ErrorCodeErrorCode 错误代码Error code
FailureStatusFailureStatus PermanentTransientPermanent or Transient. 重试暂时性故障可能会成功。Retry of a transient failure may succeed.
OriginatesFromUpdatePolicyOriginatesFromUpdatePolicy 如果故障源自更新策略,则为 TrueTrue if failure originates from an update policy
ShouldRetryShouldRetry 如果重试可以成功,则为 TrueTrue if retry may succeed

引入批处理操作日志Ingestion batching operation log

示例:Example:

{
  "resourceId": "/SUBSCRIPTIONS/12534EB3-8109-4D84-83AD-576C0D5E1D06/RESOURCEGROUPS/KEREN/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/KERENEUS",
  "time": "2020-05-27T07:55:05.3693628Z",
  "operationVersion": "1.0",
  "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGESTIONBATCHING/ACTION",
  "category": "IngestionBatching",
  "correlationId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735",
  "properties": {
    "Database": "Samples",
    "Table": "StormEvents",
    "BatchingType": "Size",
    "SourceCreationTime": "2020-05-27 07:52:04.9623640",
    "BatchTimeSeconds": 215.5,
    "BatchSizeBytes": 2356425,
    "DataSourcesInBatch": 4,
    "RootActivityId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735"
  }
}

引入批处理操作诊断日志的属性Properties of an ingestion batching operation diagnostic log

名称Name 说明Description
TimeGeneratedTimeGenerated 生成此事件的时间 (UTC)The time (UTC) at which this event was generated
数据库Database 保存目标表的数据库的名称Name of the database holding the target table
Table 数据引入到的目标表的名称Name of the target table into which the data is ingested
BatchingTypeBatchingType 批处理类型:批处理是否达到批处理策略设置的批处理时间、数据大小或文件数限制Type of batching: whether the batch reached batching time, data size, or number of files limit set by batching policy
SourceCreationTimeSourceCreationTime 此批中 blob 的最早创建时间 (UTC)Minimal time (UTC) at which blobs in this batch were created
BatchTimeSecondsBatchTimeSeconds 此批的总批处理时间(秒)Total batching time of this batch (seconds)
BatchSizeBytesBatchSizeBytes 此批中数据的未压缩大小总计(字节)Total uncompressed size of data in this batch (bytes)
DataSourcesInBatchDataSourcesInBatch 此批中的数据源数Number of data sources in this batch
RootActivityIdRootActivityId 操作的活动 IDThe operation's activity ID

后续步骤Next steps