Monitor Azure Data Explorer ingestion, commands, queries, and tables using diagnostic logs
Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Monitor diagnostic logs provide data about the operation of Azure resources. Azure Data Explorer uses diagnostic logs for insights on ingestion, commands, query, and tables. You can export operation logs to Azure Storage, event hub, or Log Analytics to monitor ingestion, commands, and query status. Logs from Azure Storage and Azure Event Hubs can be routed to a table in your Azure Data Explorer cluster for further analysis.
Important
Diagnostic log data may contain sensitive data. Restrict permissions of the logs destination according to your monitoring needs.
Prerequisites
- An Azure subscription. Create a Azure account.
- Sign in to the Azure portal.
- Create a cluster and database.
Set up diagnostic logs for an Azure Data Explorer cluster
Diagnostic logs can be used to configure the collection of the following log data:
Note
Ingestion logs are supported for queued ingestion to the ingestion endpoint using SDKs, data connections, and connectors.
Ingestion logs aren't supported for streaming ingestion, direct ingestion to the engine, ingestion from query, or set-or-append commands.
Note
Failed ingestion logs are only reported for the final state of an ingest operation, unlike the Ingestion result metric, which is emitted for transient failures that are retried internally.
- Successful ingestion operations: These logs have information about successfully completed ingestion operations.
- Failed ingestion operations: These logs have detailed information about failed ingestion operations including error details.
- Ingestion batching operations: These logs have detailed statistics of batches ready for ingestion (duration, batch size, blobs count, and batching types).
The data is then archived into a Storage account, streamed to an event hub, or sent to Log Analytics, as per your specifications.
Enable diagnostic logs
Diagnostic logs are disabled by default. To enable diagnostic logs, do the following steps:
In the Azure portal, select the Azure Data Explorer cluster resource that you want to monitor.
Under Monitoring, select Diagnostic settings.
Select Add diagnostic setting.
In the Diagnostic settings window:
- Enter a Diagnostic setting name.
- Select one or more targets: a Log Analytics workspace, a storage account, or an event hub.
- Select logs to be collected:
SucceededIngestion,FailedIngestion,IngestionBatching,Command, orQuery,TableUsageStatistics, orTableDetails. - Select metrics to be collected (optional).
- Select Save to save the new diagnostic logs settings and metrics.
New settings will be set in a few minutes. Logs then appear in the configured archival target (Storage account, Event Hub, or Log Analytics).
Note
If you send logs to Log Analytics, the SucceededIngestion, FailedIngestion, IngestionBatching, Command, Query, TableUsageStatistics and TableDetails logs will be stored in Log Analytics tables named: SucceededIngestion, FailedIngestion, ADXIngestionBatching, ADXCommand, ADXQuery, ADXTableUsageStatistics and ADXTableDetails respectively.
Diagnostic logs schema
All Azure Monitor diagnostic logs share a common top-level schema. Azure Data Explorer has unique properties for their own events. All logs are stored in a JSON format.
Ingestion logs schema
Log JSON strings include elements listed in the following table:
| Name | Description |
|---|---|
| time | Time of the report |
| resourceId | Azure Resource Manager resource ID |
| operationName | Name of the operation: 'MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION' |
| operationVersion | Schema version: '1.0' |
| category | Category of the operation. SucceededIngestion, FailedIngestion or IngestionBatching. Properties differ for successful operation, failed operation or batching operation. |
| properties | Detailed information of the operation. |
Successful ingestion operation log
Example:
{
"time": "2019-05-27 07:55:05.3693628",
"resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
"operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
"operationVersion": "1.0",
"category": "SucceededIngestion",
"properties":
{
"SucceededOn": "2019-05-27 07:55:05.3693628",
"OperationId": "b446c48f-6e2f-4884-b723-92eb6dc99cc9",
"Database": "Samples",
"Table": "StormEvents",
"IngestionSourceId": "66a2959e-80de-4952-975d-b65072fc571d",
"IngestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events8347293.json",
"RootActivityId": "d0bd5dd3-c564-4647-953e-05670e22a81d"
}
}
Properties of a successful operation diagnostic log
| Name | Description |
|---|---|
| SucceededOn | Time of ingestion completion |
| OperationId | Azure Data Explorer ingestion operation ID |
| Database | Name of the target database |
| Table | Name of the target table |
| IngestionSourceId | ID of the ingestion data source |
| IngestionSourcePath | Path of the ingestion data source or blob URI |
| RootActivityId | Activity ID |
Failed ingestion operation log
Example:
{
"time": "2019-05-27 08:57:05.4273524",
"resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
"operationName": "MICROSOFT.KUSTO/CLUSTERS/INGEST/ACTION",
"operationVersion": "1.0",
"category": "FailedIngestion",
"properties":
{
"failedOn": "2019-05-27 08:57:05.4273524",
"operationId": "5956515d-9a48-4544-a514-cf4656fe7f95",
"database": "Samples",
"table": "StormEvents",
"ingestionSourceId": "eee56f8c-2211-4ea4-93a6-be556e853e5f",
"ingestionSourcePath": "https://kustoingestionlogs.blob.core.chinacloudapi.cn/sampledata/events5725592.json",
"rootActivityId": "52134905-947a-4231-afaf-13d9b7b184d5",
"details": "Permanent failure downloading blob. URI: ..., permanentReason: Download_SourceNotFound, DownloadFailedException: 'Could not find file ...'",
"errorCode": "Download_SourceNotFound",
"failureStatus": "Permanent",
"originatesFromUpdatePolicy": false,
"shouldRetry": false
}
}
Properties of a failed operation diagnostic log
| Name | Description |
|---|---|
| FailedOn | Time of ingestion completion |
| OperationId | Azure Data Explorer ingestion operation ID |
| Database | Name of the target database |
| Table | Name of the target table |
| IngestionSourceId | ID of the ingestion data source |
| IngestionSourcePath | Path of the ingestion data source or blob URI |
| RootActivityId | Activity ID |
| Details | Detailed description of the failure and error message |
| ErrorCode | Ingestion error code |
| FailureStatus | Permanent or RetryAttemptsExceeded indicates that operation has exceeded the retry attempts limit or timespan limit following a recurring transient error. |
| OriginatesFromUpdatePolicy | True if failure originates from an update policy |
| ShouldRetry | True if retry may succeed |
Ingestion batching operation log
Example:
{
"resourceId": "/SUBSCRIPTIONS/12534000-8109-4D84-83AD-576C0D5E1AAA/RESOURCEGROUPS/myResourceGroup/PROVIDERS/MICROSOFT.KUSTO/CLUSTERS/mycluster",
"time": "2021-04-18T19:19:57.0211782Z",
"operationVersion": "1.0",
"operationName": "MICROSOFT.KUSTO/CLUSTERS/INGESTIONBATCHING/ACTION",
"category": "IngestionBatching",
"correlationId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735",
"properties": {
"Timestamp": "2021-04-18T19:19:57.0211782Z",
"Database": "Samples",
"Table": "StormEvents",
"BatchingType": "Time",
"SourceCreationTime": "2021-04-18T19:14:53.9543732Z",
"BatchTimeSeconds": 302.1449075,
"BatchSizeBytes": 3988,
"DataSourcesInBatch": 2,
"RootActivityId": "2bb51038-c7dc-4ebd-9d7f-b34ece4cb735"
}
}
Properties of an ingestion batching operation diagnostic log
| Name | Description |
|---|---|
| Timestamp | The time of the batching reporting |
| Database | Name of the database holding the target table |
| Table | Name of the target table into which the data is ingested |
| BatchingType | The trigger for sealing a batch. For a complete list of batching types, see Batching types. |
| SourceCreationTime | Minimal time (UTC) at which blobs in this batch were created |
| BatchTimeSeconds | Total batching time of this batch (seconds) |
| BatchSizeBytes | Total uncompressed size of data in this batch (bytes) |
| DataSourcesInBatch | Number of data sources in this batch |
| RootActivityId | The operation's activity ID |
Next steps
- Use metrics to monitor cluster health
- Tutorial: Ingest and query monitoring data in Azure Data Explorer for ingestion diagnostic logs