为存档到存储帐户的 Azure Monitor 资源日志的格式更改做准备Prepare for format change to Azure Monitor resource logs archived to a storage account

警告

如果使用资源诊断设置将 Azure 资源资源日志或指标发送到存储帐户,则存储帐户中的数据格式将于 2018 年 11 月 1 日更改为 JSON Lines。If you are sending Azure resource resource logs or metrics to a storage account using resource diagnostic settings, the format of the data in the storage account will change to JSON Lines on Nov. 1, 2018. 以下说明介绍了影响以及如何更新工具以处理新格式。The instructions below describe the impact and how to update your tooling to handle the new format.

有什么变化What is changing

Azure Monitor 提供的功能可将资源诊断数据和活动日志数据发送到 Azure 存储帐户、事件中心命名空间或 Azure Monitor 中的 Log Analytics 工作区。Azure Monitor offers a capability that enables you to send resource diagnostic data and activity log data into an Azure storage account, Event Hubs namespace, or into a Log Analytics workspace in Azure Monitor. 为解决系统性能问题,发送到 blob 存储的日志数据格式将于 2018 年 11 月 1 日凌晨 12 点 (UTC) 发生更改 。In order to address a system performance issue, on November 1, 2018 at 12:00 midnight UTC the format of log data send to blob storage will change. 如果工具从 blob 存储中读取数据,则需要更新工具以了解新的数据格式。If you have tooling that is reading data out of blob storage, you need to update your tooling to understand the new data format.

  • Blob 格式将于 2018 年 11 月 1 日星期四凌晨 12 点 (UTC) 更改为 JSON LinesOn Thursday, November 1, 2018 at 12:00 midnight UTC, the blob format will change to be JSON Lines. 这意味着每个记录将由换行符分隔,JSON 记录之间没有外部记录数组和逗号。This means each record will be delimited by a newline, with no outer records array and no commas between JSON records.
  • 将同时更改所有订阅中所有诊断设置的 blob 格式。The blob format changes for all diagnostic settings across all subscriptions at once. 11 月 1 日发出的第一个 PT1H.json 文件将使用此新格式。The first PT1H.json file emitted for November 1 will use this new format. Blob 和容器名称保持不变。The blob and container names remain the same.
  • 从现在到 11 月 1 日之间设置诊断设置将继续以当前格式发出数据,直到 11 月 1 日为止。Setting a diagnostic setting between now and November 1 continues to emit data in the current format until November 1.
  • 此更改将同时在所有公有云区域中发生。This change will occur at once across all public cloud regions. Azure 中国、Azure 德国或 Azure 中国云中还不会进行此更改。The change will not occur in Azure China, Azure Germany, or Azure China Cloud clouds yet.
  • 此更改会影响以下数据类型:This change impacts the following data types:
  • 此更改不会影响:This change does not impact:
    • 网络流日志Network flow logs
    • 尚未通过 Azure Monitor 可用的 Azure 服务日志(例如,Azure 应用服务资源日志、存储分析日志)Azure service logs not made available through Azure Monitor yet (for example, Azure App Service resource logs, storage analytics logs)
    • 将 Azure 资源日志和活动日志路由到其他目标(事件中心、Log Analytics)Routing of Azure resource logs and activity logs to other destinations (Event Hubs, Log Analytics)

如何查看是否受到影响How to see if you are impacted

只有在以下情况下,才会受到此更改的影响:You are only impacted by this change if you:

  1. 使用资源诊断设置将日志数据发送到 Azure 存储帐户,以及Are sending log data to an Azure storage account using a resource diagnostic setting, and
  2. 拥有依赖于存储中这些日志的 JSON 结构的工具。Have tooling that depends on the JSON structure of these logs in storage.

若要确定是否具有将数据发送到 Azure 存储帐户的资源诊断设置,可导航到门户的“监视”部分,单击“诊断设置”,并识别所有将“诊断状态”设置为“已启用”的资源 :To identify if you have resource diagnostic settings that are sending data to an Azure storage account, you can navigate to the Monitor section of the portal, click on Diagnostic Settings, and identify any resources that have Diagnostic Status set to Enabled:

“Azure Monitor 诊断设置”边栏选项卡

如果“诊断状态”设置为已启用,则该资源上具有活动诊断设置。If Diagnostic Status is set to enabled, you have an active diagnostic setting on that resource. 单击资源以查看是否有诊断设置将数据发送到存储帐户:Click on the resource to see if any diagnostic settings are sending data to a storage account:

已启用存储帐户

如果确实有资源使用这些资源诊断设置将数据发送到存储帐户,则此更改将影响该存储帐户中的数据格式。If you do have resources sending data to a storage account using these resource diagnostic settings, the format of the data in that storage account will be impacted by this change. 除非拥有运行这些存储帐户的自定义工具,否则格式更改将不会产生影响。Unless you have custom tooling that operates off of these storage accounts, the format change will not impact you.

格式更改的详细信息Details of the format change

Azure blob 存储中 PT1H.json 文件的当前格式使用 JSON 数组记录。The current format of the PT1H.json file in Azure blob storage uses a JSON array of records. 以下是 KeyVault 日志文件的示例:Here is a sample of a KeyVault log file now:

{
    "records": [
        {
            "time": "2016-01-05T01:32:01.2691226Z",
            "resourceId": "/SUBSCRIPTIONS/361DA5D4-A47A-4C79-AFDD-XXXXXXXXXXXX/RESOURCEGROUPS/CONTOSOGROUP/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSOKEYVAULT",
            "operationName": "VaultGet",
            "operationVersion": "2015-06-01",
            "category": "AuditEvent",
            "resultType": "Success",
            "resultSignature": "OK",
            "resultDescription": "",
            "durationMs": "78",
            "callerIpAddress": "104.40.82.76",
            "correlationId": "",
            "identity": {
                "claim": {
                    "http://schemas.microsoft.com/identity/claims/objectidentifier": "d9da5048-2737-4770-bd64-XXXXXXXXXXXX",
                    "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "live.com#username@outlook.com",
                    "appid": "1950a258-227b-4e31-a9cf-XXXXXXXXXXXX"
                }
            },
            "properties": {
                "clientInfo": "azure-resource-manager/2.0",
                "requestUri": "https://control-prod-wus.vaultcore.azure.net/subscriptions/361da5d4-a47a-4c79-afdd-XXXXXXXXXXXX/resourcegroups/contosoresourcegroup/providers/Microsoft.KeyVault/vaults/contosokeyvault?api-version=2015-06-01",
                "id": "https://contosokeyvault.vault.azure.cn/",
                "httpStatusCode": 200
            }
        },
        {
            "time": "2016-01-05T01:33:56.5264523Z",
            "resourceId": "/SUBSCRIPTIONS/361DA5D4-A47A-4C79-AFDD-XXXXXXXXXXXX/RESOURCEGROUPS/CONTOSOGROUP/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSOKEYVAULT",
            "operationName": "VaultGet",
            "operationVersion": "2015-06-01",
            "category": "AuditEvent",
            "resultType": "Success",
            "resultSignature": "OK",
            "resultDescription": "",
            "durationMs": "83",
            "callerIpAddress": "104.40.82.76",
            "correlationId": "",
            "identity": {
                "claim": {
                    "http://schemas.microsoft.com/identity/claims/objectidentifier": "d9da5048-2737-4770-bd64-XXXXXXXXXXXX",
                    "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "live.com#username@outlook.com",
                    "appid": "1950a258-227b-4e31-a9cf-XXXXXXXXXXXX"
                }
            },
            "properties": {
                "clientInfo": "azure-resource-manager/2.0",
                "requestUri": "https://control-prod-wus.vaultcore.azure.net/subscriptions/361da5d4-a47a-4c79-afdd-XXXXXXXXXXXX/resourcegroups/contosoresourcegroup/providers/Microsoft.KeyVault/vaults/contosokeyvault?api-version=2015-06-01",
                "id": "https://contosokeyvault.vault.azure.cn/",
                "httpStatusCode": 200
            }
        }
    ]
}

新格式使用 JSON lines,其中每个事件都是一行,换行符表示新事件。The new format uses JSON lines, where each event is a line and the newline character indicates a new event. 以下是更改后以上示例在 PT1H.json 文件中的外观:Here is what the above sample will look like in the PT1H.json file after the change:

{"time": "2016-01-05T01:32:01.2691226Z","resourceId": "/SUBSCRIPTIONS/361DA5D4-A47A-4C79-AFDD-XXXXXXXXXXXX/RESOURCEGROUPS/CONTOSOGROUP/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSOKEYVAULT","operationName": "VaultGet","operationVersion": "2015-06-01","category": "AuditEvent","resultType": "Success","resultSignature": "OK","resultDescription": "","durationMs": "78","callerIpAddress": "104.40.82.76","correlationId": "","identity": {"claim": {"http://schemas.microsoft.com/identity/claims/objectidentifier": "d9da5048-2737-4770-bd64-XXXXXXXXXXXX","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "live.com#username@outlook.com","appid": "1950a258-227b-4e31-a9cf-XXXXXXXXXXXX"}},"properties": {"clientInfo": "azure-resource-manager/2.0","requestUri": "https://control-prod-wus.vaultcore.azure.net/subscriptions/361da5d4-a47a-4c79-afdd-XXXXXXXXXXXX/resourcegroups/contosoresourcegroup/providers/Microsoft.KeyVault/vaults/contosokeyvault?api-version=2015-06-01","id": "https://contosokeyvault.vault.azure.cn/","httpStatusCode": 200}}
{"time": "2016-01-05T01:33:56.5264523Z","resourceId": "/SUBSCRIPTIONS/361DA5D4-A47A-4C79-AFDD-XXXXXXXXXXXX/RESOURCEGROUPS/CONTOSOGROUP/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSOKEYVAULT","operationName": "VaultGet","operationVersion": "2015-06-01","category": "AuditEvent","resultType": "Success","resultSignature": "OK","resultDescription": "","durationMs": "83","callerIpAddress": "104.40.82.76","correlationId": "","identity": {"claim": {"http://schemas.microsoft.com/identity/claims/objectidentifier": "d9da5048-2737-4770-bd64-XXXXXXXXXXXX","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "live.com#username@outlook.com","appid": "1950a258-227b-4e31-a9cf-XXXXXXXXXXXX"}},"properties": {"clientInfo": "azure-resource-manager/2.0","requestUri": "https://control-prod-wus.vaultcore.azure.net/subscriptions/361da5d4-a47a-4c79-afdd-XXXXXXXXXXXX/resourcegroups/contosoresourcegroup/providers/Microsoft.KeyVault/vaults/contosokeyvault?api-version=2015-06-01","id": "https://contosokeyvault.vault.azure.cn/","httpStatusCode": 200}}

通过此新格式,Azure Monitor 能够使用追加 blob 来推送日志文件,这对于连续追加新事件数据更加高效。This new format enables Azure Monitor to push log files using append blobs, which are more efficient for continuously appending new event data.

更新方法How to update

如果具有引入这些日志文件以进行进一步处理的自定义工具,则只需进行更新即可。You only need to make updates if you have custom tooling that ingests these log files for further processing. 如果正在使用外部日志分析或 SIEM 工具,则建议改为使用事件中心来引入此数据If you are making use of an external log analytics or SIEM tool, we recommend using event hubs to ingest this data instead. 在处理许多服务中的日志以及为特定日志中的位置添加书签方面,事件中心集成更加容易。Event hubs integration is easier in terms of processing logs from many services and bookmarking location in a particular log.

应更新自定义工具以处理当前格式和上述 JSON Lines 格式。Custom tools should be updated to handle both the current format and the JSON Lines format described above. 这将确保当数据开始以新格式显示时,工具不会中断。This will ensure that when data starts to appear in the new format, your tools do not break.

后续步骤Next steps