使用诊断日志监视 Azure 数据资源管理器的引入、命令和查询Monitor Azure Data Explorer ingestion, commands, and queries using diagnostic logs

09/30/2020

Azure 数据资源管理器是一项快速、完全托管的数据分析服务，用于实时分析从应用程序、网站和 IoT 设备等资源流式传输的海量数据。Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Monitor 诊断日志提供有关 Azure 资源操作的数据。Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure 数据资源管理器使用诊断日志获取有关引入成功、引入失败、命令和查询操作的见解。Azure Data Explorer uses diagnostic logs for insights on ingestion successes, ingestion failures, commands, and query operations. 可将操作日志导出到 Azure 存储、事件中心或 Log Analytics 以监视引入、命令和查询状态。You can export operation logs to Azure Storage, Event Hub, or Log Analytics to monitor ingestion, commands, and query status. 可将 Azure 存储和 Azure 事件中心的日志路由到 Azure 数据资源管理器群集中的某个表，以进一步分析。Logs from Azure Storage and Azure Event Hub can be routed to a table in your Azure Data Explorer cluster for further analysis.

重要

诊断日志数据可能包含敏感数据。Diagnostic log data may contain sensitive data. 请根据监视需求限制日志目标的权限。Restrict permissions of the logs destination according to your monitoring needs.

先决条件Prerequisites

如果没有 Azure 订阅，请创建一个试用 Azure 帐户。If you don't have an Azure subscription, create a trial Azure account.
登录到 Azure 门户。Sign in to the Azure portal.
创建群集和数据库。Create a cluster and database.

设置 Azure 数据资源管理器群集的诊断日志Set up diagnostic logs for an Azure Data Explorer cluster

诊断日志可用于配置以下日志数据的收集：Diagnostic logs can be used to configure the collection of the following log data:

引流Ingestion
命令和查询Commands and Queries

成功的引入操作 ：这些日志包含有关已成功完成的引入操作的信息。Successful ingestion operations : These logs have information about successfully completed ingestion operations.
失败的引入操作 ：这些日志包含有关失败的引入操作的详细信息，包括错误详细信息。Failed ingestion operations : These logs have detailed information about failed ingestion operations including error details.

然后可根据规范将数据存档到存储帐户、流式传输到事件中心，或发送到 Log Analytics。The data is then archived into a Storage account, streamed to an Event Hub, or sent to Log Analytics, as per your specifications.

启用诊断日志Enable diagnostic logs

诊断日志默认已禁用。Diagnostic logs are disabled by default. 若要启用诊断日志，请执行以下步骤：To enable diagnostic logs, do the following steps:

在 Azure 门户中，选择要监视的 Azure 数据资源管理器群集资源。In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.
在“监视”下，选择“诊断设置” 。Under Monitoring , select Diagnostic settings.
选择“添加诊断设置”。 Select Add diagnostic setting.
在“诊断设置”窗口中： In the Diagnostics settings window:
1. 选择诊断设置的名称。Select Name for your diagnostic setting.
2. 选择一个或多个目标：存储帐户、事件中心或 Log Analytics。Select one or more targets: a Storage account, Event Hub, or Log Analytics.
3. 选择要收集的日志：SucceededIngestion、FailedIngestion、Command 或 Query。Select logs to be collected: SucceededIngestion, FailedIngestion, Command, or Query.
4. 选择要收集的指标（可选）。Select metrics to be collected (optional).
5. 选择“保存”以保存新的诊断日志设置和指标。 Select Save to save the new diagnostic logs settings and metrics.

在几分钟内即会完成新的设置。New settings will be set in a few minutes. 日志随后会显示在配置的存档目标（存储帐户、事件中心或 Log Analytics）中。Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).

备注

如果将日志发送到 Log Analytics，则 SucceededIngestion、FailedIngestion、Command 和 Query 日志将分别存储在名为 SucceededIngestion、FailedIngestion、ADXCommand 和 ADXQuery 的 Log Analytics 表中。If you send logs to Log Analytics, the SucceededIngestion, FailedIngestion, Command, and Query logs will be stored in Log Analytics tables named: SucceededIngestion, FailedIngestion, ADXCommand, ADXQuery, respectively.

诊断日志架构Diagnostic logs schema

所有 Azure Monitor 诊断日志共享一个通用的顶级架构。All Azure Monitor diagnostic logs share a common top-level schema. Azure 数据资源管理器对其自身的事件使用唯一属性。Azure Data Explorer has unique properties for their own events. 所有日志均以 JSON 格式存储。All logs are stored in a JSON format.

引流Ingestion
命令和查询Commands and Queries

引入日志架构Ingestion logs schema

日志 JSON 字符串包含下表中列出的元素：Log JSON strings include elements listed in the following table:

名称Name	说明Description
timetime	报告时间Time of the report
ResourceIdresourceId	Azure Resource Manager 资源 IDAzure Resource Manager resource ID
operationNameoperationName	操作名称：'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION'
operationVersionoperationVersion	架构版本：'1.0'Schema version: '1.0'
categorycategory	操作类别。Category of the operation. `SucceededIngestion` 或 `FailedIngestion`。`SucceededIngestion` or `FailedIngestion`. 成功的操作或失败的操作的属性不同。Properties differ for successful operation or failed operation.
propertiesproperties	操作的详细信息。Detailed information of the operation.

成功引入操作日志Successful ingestion operation log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "SucceededIngestion",
    "properties":
    {
        "succeededOn": "2019-05-27 07:55:05.3693628",
        "operationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events8347293.json",
        "rootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
    }
}

成功操作诊断日志的属性Properties of a successful operation diagnostic log

名称Name	说明Description
succeededOnsucceededOn	引入完成时间Time of ingestion completion
operationIdoperationId	Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
databasedatabase	目标数据库的名称Name of the target database
表table	目标表的名称Name of the target table
ingestionSourceIdingestionSourceId	引入数据源的 IDID of the ingestion data source
ingestionSourcePathingestionSourcePath	引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
rootActivityIdrootActivityId	活动 IDActivity ID

失败引入操作日志Failed ingestion operation log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
    "operationVersion": "1.0",
    "category": "FailedIngestion",
    "properties":
    {
        "failedOn": "2019-05-27 08:57:05.4273524",
        "operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
        "database": "Samples",
        "table": "StormEvents",
        "ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
        "ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events5725592.json",
        "rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
        "details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
        "errorCode": "Download_SourceNotFound",
        "failureStatus": "Permanent",
        "originatesFromUpdatePolicy": false,
        "shouldRetry": false
    }
}

失败操作诊断日志的属性Properties of a failed operation diagnostic log

名称Name	说明Description
failedOnfailedOn	引入完成时间Time of ingestion completion
operationIdoperationId	Azure 数据资源管理器引入操作 IDAzure Data Explorer ingestion operation ID
databasedatabase	目标数据库的名称Name of the target database
表table	目标表的名称Name of the target table
ingestionSourceIdingestionSourceId	引入数据源的 IDID of the ingestion data source
ingestionSourcePathingestionSourcePath	引入数据源或 Blob URI 的路径Path of the ingestion data source or blob URI
rootActivityIdrootActivityId	活动 IDActivity ID
详细信息details	失败和错误消息的详细说明Detailed description of the failure and error message
errorCodeerrorCode	错误代码Error code
failureStatusfailureStatus	`Permanent` 或 `Transient`。`Permanent` or `Transient`. 重试暂时性故障可能会成功。Retry of a transient failure may succeed.
originatesFromUpdatePolicyoriginatesFromUpdatePolicy	如果故障源自更新策略，则为 TrueTrue if failure originates from an update policy
shouldRetryshouldRetry	如果重试可以成功，则为 TrueTrue if retry may succeed

命令和查询日志架构Commands and Queries logs schema

日志 JSON 字符串包含下表中列出的元素：Log JSON strings include elements listed in the following table:

名称Name	说明Description
timetime	报告时间Time of the report
ResourceIdresourceId	Azure Resource Manager 资源 IDAzure Resource Manager resource ID
operationNameoperationName	操作名称：“MICROSOFT.KUSTO/CLUSTERS/COMMAND/ACTION”或“MICROSOFT.KUSTO/CLUSTERS/QUERY/ACTION”。Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/COMMAND/ACTION' or "MICROSOFT.KUSTO/CLUSTERS/QUERY/ACTION". 命令和查询日志的属性不同。Properties differ for commands and query logs.
operationVersionoperationVersion	架构版本：'1.0'Schema version: '1.0'
categorycategory	操作类别：`Command` 或 `Query`。Category of the operation: `Command` or `Query`. 命令和查询日志的属性不同。Properties differ for commands and query logs.
propertiesproperties	操作的详细信息。Detailed information of the operation.

命令日志Command log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/COMMAND/ACTION",
    "operationVersion": "1.0",
    "category": "Command",
    "properties":
    {
        "RootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d",
        "StartedOn": "2020-09-12T18:06:04.6898353Z",
        "LastUpdatedOn": "2020-09-12T18:06:04.6898353Z",
        "Database": "DatabaseSample",
        "State": "Completed",
        "FailureReason": "XXX",
        "TotalCpu": "00:01:30.1562500",
        "CommandType": "ExtentsDrop",
        "Application": "Kusto.WinSvc.DM.Svc",
        "ResourceUtilization": "{\"CacheStatistics\":{\"Memory\":{\"Hits\":0,\"Misses\":0},\"Disk\":{\"Hits\":0,\"Misses\":0},\"Shards\":{\"Hot\":{\"HitBytes\":0,\"MissBytes\":0,\"RetrieveBytes\":0},\"Cold\":{\"HitBytes\":0,\"MissBytes\":0,\"RetrieveBytes\":0},\"BypassBytes\":0}},\"TotalCpu\":\"00:00:00\",\"MemoryPeak\":0,\"ScannedExtentsStatistics\":{\"MinDataScannedTime\":null,\"MaxDataScannedTime\":null,\"TotalExtentsCount\":0,\"ScannedExtentsCount\":0,\"TotalRowsCount\":0,\"ScannedRowsCount\":0}}",
        "Duration": "00:03:30.1562500",
        "User": "AAD app id=0571b364-eeeb-4f28-ba74-90a8b4132b53",
        "Principal": "aadapp=0571b364-eeeb-4f28-ba74-90a3b4136b53;5c443533-c927-4410-a5d6-4d6a5443b64f"
    }
}

命令诊断日志的属性Properties of a command diagnostic log

名称Name	说明Description
RootActivityIdRootActivityId	根活动 IDThe root activity ID
StartedOnStartedOn	该命令开始的时间 (UTC)Time (UTC) at which this command started
LastUpdatedOnLastUpdatedOn	该命令结束的时间 (UTC)Time (UTC) at which this command ended
数据库Database	运行命令的数据库的名称The name of the database the command ran on
状态State	命令结束的状态The state the command ended with
FailureReasonFailureReason	失败原因The failure reason
TotalCpuTotalCpu	CPU 总持续时间Total CPU duration
CommandTypeCommandType	命令类型Command type
应用程序Application	调用了命令的应用程序的名称Application name that invoked the command
ResourceUtilizationResourceUtilization	命令资源利用率Command resource utilization
DurationDuration	命令持续时间Command duration
用户User	调用了查询的用户The user that invoked the query
主体Principal	调用了查询的主体The principal that invoked the query

查询日志Query log

示例：Example:

{
    "time": "",
    "resourceId": "",
    "operationName": "MICROSOFT.KUSTO/CLUSTERS/QUERY/ACTION",
    "operationVersion": "1.0",
    "category": "Query",
    "properties": {
        "RootActivityId": "3e6e8814-e64f-455a-926d-bf16229f6d2d",
        "StartedOn": "2020-09-04T13:45:45.331925Z",
        "LastUpdatedOn": "2020-09-04T13:45:45.3484372Z",
        "Database": "DatabaseSample",
        "State": "Completed",
        "FailureReason": "[none]",
        "TotalCpu": "00:00:00",
        "ApplicationName": "MyApp",
        "MemoryPeak": 0,
        "Duration": "00:00:00.0165122",
        "User": "AAD app id=0571b364-eeeb-4f28-ba74-90a8b4132b53",
        "Principal": "aadapp=0571b364-eeeb-4f28-ba74-90a8b4132b53;5c823e4d-c927-4010-a2d8-6dda2449b6cf",
        "ScannedExtentsStatistics": {
            "MinDataScannedTime": "2020-07-27T08:34:35.3299941",
            "MaxDataScannedTime": "2020-07-27T08:34:41.991661",
            "TotalExtentsCount": 64,
            "ScannedExtentsCount": 64,
            "TotalRowsCount": 320,
            "ScannedRowsCount": 320
        },
        "CacheStatistics": {
            "Memory": {
                "Hits": 192,
                "Misses": 0
          },
            "Disk": {
                "Hits": 0,
                "Misses": 0
      },
            "Shards": {
                "Hot": {
                    "HitBytes": 0,
                    "MissBytes": 0,
                    "RetrieveBytes": 0
        },
                "Cold": {
                    "HitBytes": 0,
                    "MissBytes": 0,
                    "RetrieveBytes": 0
        },
            "BypassBytes": 0
      }
    },
    "ResultSetStatistics": {
        "TableCount": 1,
        "TablesStatistics": [
        {
          "RowCount": 1,
          "TableSize": 9
        }
      ]
    }
  }
}

查询诊断日志的属性Properties of a query diagnostic log

名称Name	说明Description
RootActivityIdRootActivityId	根活动 IDThe root activity ID
StartedOnStartedOn	该命令开始的时间 (UTC)Time (UTC) at which this command started
LastUpdatedOnLastUpdatedOn	该命令结束的时间 (UTC)Time (UTC) at which this command ended
数据库Database	运行命令的数据库的名称The name of the database the command ran on
状态State	命令结束的状态The state the command ended with
FailureReasonFailureReason	失败原因The failure reason
TotalCpuTotalCpu	CPU 总持续时间Total CPU duration
ApplicationNameApplicationName	调用了查询的应用程序的名称Application name that invoked the query
MemoryPeakMemoryPeak	内存峰值Memory peak
DurationDuration	命令持续时间Command duration
用户User	调用了查询的用户The user that invoked the query
主体Principal	调用了查询的主体The principal that invoked the query
ScannedExtentsStatisticsScannedExtentsStatistics	包含扫描的盘区统计信息Contains scanned extent statistics
MinDataScannedTimeMinDataScannedTime	最短数据扫描时间Minimum data scan time
MaxDataScannedTimeMaxDataScannedTime	最长数据扫描时间Maximum data scan time
TotalExtentsCountTotalExtentsCount	盘区总数Total extent count
ScannedExtentsCountScannedExtentsCount	扫描的盘区计数Scanned extent count
TotalRowsCountTotalRowsCount	总行计数Total rows count
ScannedRowsCountScannedRowsCount	扫描的行计数Scanned rows count
CacheStatisticsCacheStatistics	包含缓存统计信息Contains cache statistics
内存Memory	包含缓存内存统计信息Contains cache memory statistics
命中数Hits	内存缓存命中数Memory cache hits
未命中数Misses	内存缓存未命中数Memory cache misses
磁盘Disk	包含缓存磁盘统计信息Contains cache disk statistics
命中数Hits	磁盘缓存命中数Disk cache hits
未命中数Misses	磁盘缓存未命中数Disk cache misses
ShardsShards	包含热和冷分片缓存统计信息Contains hot and cold shards cache statistics
热Hot	包含热分片缓存统计信息Contains hot shards cache statistics
HitBytesHitBytes	分片热缓存命中数Shards hot cache hits
MissBytesMissBytes	分片热缓存未命中数Shards hot cache misses
RetrieveBytesRetrieveBytes	分片热缓存检索的字节数Shards hot cache retrieved bytes
冷Cold	包含冷分片缓存统计信息Contains cold shards cache statistics
HitBytesHitBytes	分片冷缓存命中数Shards cold cache hits
MissBytesMissBytes	分片冷缓存未命中数Shards cold cache misses
RetrieveBytesRetrieveBytes	分片冷缓存检索的字节数Shards cold cache retrieved bytes
BypassBytesBypassBytes	分片缓存绕过字节数Shards cache bypass bytes
ResultSetStatisticsResultSetStatistics	包含结果集统计信息Contains result set statistics
TableCountTableCount	结果集表计数Result set table count
TablesStatisticsTablesStatistics	包含结果集表统计信息Contains result set table statistics
RowCountRowCount	结果集表行计数Result set table row count
TableSizeTableSize	结果集表行计数Result set table row count

后续步骤Next steps

使用指标来监视群集运行状况Use metrics to monitor cluster health
教程：在 Azure 数据资源管理器中引入和查询监视数据，可帮助获取引入诊断日志Tutorial: Ingest and query monitoring data in Azure Data Explorer for ingestion diagnostic logs

使用诊断日志监视 Azure 数据资源管理器的引入、命令和查询Monitor Azure Data Explorer ingestion, commands, and queries using diagnostic logs

先决条件Prerequisites

设置 Azure 数据资源管理器群集的诊断日志Set up diagnostic logs for an Azure Data Explorer cluster

启用诊断日志Enable diagnostic logs

诊断日志架构Diagnostic logs schema

引入日志架构Ingestion logs schema

成功引入操作日志Successful ingestion operation log

失败引入操作日志Failed ingestion operation log

命令和查询日志架构Commands and Queries logs schema

命令日志Command log

查询日志Query log

后续步骤Next steps

此页面有帮助吗?

此页面有帮助吗?