Azure 资源日志Azure resource logs

Azure 资源日志是平台日志,可以通过它深入了解已在 Azure 资源中执行的操作。Azure resource logs are platform logs that provide insight into operations that were performed within an Azure resource. 资源日志的内容因 Azure 服务和资源类型而异。The content of resource logs varies by the Azure service and resource type. 默认不会收集资源日志。Resource logs are not collected by default. 必须为每个 Azure 资源创建诊断设置,以便将其资源日志发送到 Log Analytics 工作区与 Azure Monitor 日志一起使用,发送到 Azure 事件中心以转发到 Azure 外部,或者发送到 Azure 存储进行存档。You must create a diagnostic setting for each Azure resource to send its resource logs to a Log Analytics workspace to use with Azure Monitor Logs, Azure Event Hubs to forward outside of Azure, or to Azure Storage for archiving.

有关创建诊断设置的详细信息,请参阅创建诊断设置以将平台日志和指标发送到不同的目标。若要详细了解如何使用 Azure Policy 为你创建的每个 Azure 资源自动创建诊断设置,请参阅使用 Azure Policy 大规模部署 Azure MonitorSee Create diagnostic settings to send platform logs and metrics to different destinations for details on creating a diagnostic setting and Deploy Azure Monitor at scale using Azure Policy for details on using Azure Policy to automatically create a diagnostic setting for each Azure resource you create.

发送到 Log Analytics 工作区Send to Log Analytics workspace

将资源日志发送到 Log Analytics 工作区,以启用 Azure Monitor 日志的功能,其中包括以下内容:Send resource logs to a Log Analytics workspace to enable the features of Azure Monitor Logs which includes the following:

  • 将资源日志数据与 Azure Monitor 收集的其他监视数据关联。Correlate resource log data with other monitoring data collected by Azure Monitor.
  • 将来自多个 Azure 资源、订阅和租户的日志项合并到一个位置一起进行分析。Consolidate log entries from multiple Azure resources, subscriptions, and tenants into one location for analysis together.
  • 使用日志查询执行复杂分析,并深入了解日志数据。Use log queries to perform complex analysis and gain deep insights on log data.
  • 使用带有复杂警报逻辑的日志警报。Use log alerts with complex alerting logic.

创建诊断设置,以便将资源日志发送到 Log Analytics 工作区。Create a diagnostic setting to send resource logs to a Log Analytics workspace. Azure Monitor 日志的结构中所述,此数据将存储在表中。This data is stored in tables as described in Structure of Azure Monitor Logs. 资源日志使用的表取决于资源使用的收集类型:The tables used by resource logs depend on what type of collection the resource is using:

  • Azure 诊断 - 所有数据将写入到 AzureDiagnostics 表中。Azure diagnostics - All data written is to the AzureDiagnostics table.
  • 特定于资源 - 每个类别的资源的数据将写入到单独的表中。Resource-specific - Data is written to individual table for each category of the resource.

Azure 诊断模式Azure diagnostics mode

在此模式下,任何诊断设置中的所有数据都将收集到 AzureDiagnostics 表中。In this mode, all data from any diagnostic setting will be collected in the AzureDiagnostics table. 这是当今大多数 Azure 服务使用的传统方法。This is the legacy method used today by most Azure services. 由于多个资源类型会将数据发送到同一个表,因此其架构是所收集的所有不同数据类型的架构的超集。Since multiple resource types send data to the same table, its schema is the superset of the schemas of all the different data types being collected.

在以下示例中,以下数据类型的诊断设置将收集到同一个工作区中:Consider the following example where diagnostic settings are being collected in the same workspace for the following data types:

  • 服务 1 的审核日志(架构由列 A、B 和 C 组成)Audit logs of service 1 (having a schema consisting of columns A, B, and C)
  • 服务 1 的错误日志(架构由列 D、E 和 F 组成)Error logs of service 1 (having a schema consisting of columns D, E, and F)
  • 服务 2 的审核日志(架构由列 G、H 和 I 组成)Audit logs of service 2 (having a schema consisting of columns G, H, and I)

AzureDiagnostics 表的外观如下所示:The AzureDiagnostics table will look as follows:

ResourceProviderResourceProvider CategoryCategory AA BB CC DD EE FF GG HH II
Microsoft.Service1Microsoft.Service1 AuditLogsAuditLogs x1x1 y1y1 z1z1
Microsoft.Service1Microsoft.Service1 ErrorLogsErrorLogs q1q1 w1w1 e1e1
Microsoft.Service2Microsoft.Service2 AuditLogsAuditLogs j1j1 k1k1 l1l1
Microsoft.Service1Microsoft.Service1 ErrorLogsErrorLogs q2q2 w2w2 e2e2
Microsoft.Service2Microsoft.Service2 AuditLogsAuditLogs j3j3 k3k3 l3l3
Microsoft.Service1Microsoft.Service1 AuditLogsAuditLogs x5x5 y5y5 z5z5
......

特定于资源Resource-specific

在此模式下,将会根据在诊断设置中选择的每个类别,在所选工作区中创建各个表。In this mode, individual tables in the selected workspace are created for each category selected in the diagnostic setting. 建议使用此方法,因为它可以大幅简化日志查询中数据的处理、更好地发现架构及其结构、改善引入延迟和查询时间方面的性能、可以授予针对特定表的 Azure RBAC 权限,等等。This method is recommended since it makes it much easier to work with the data in log queries, provides better discoverability of schemas and their structure, improves performance across both ingestion latency and query times, and the ability to grant Azure RBAC rights on a specific table. 所有 Azure 服务最终都会迁移到特定于资源的模式。All Azure services will eventually migrate to the Resource-Specific mode.

以上示例会创建三个表:The example above would result in three tables being created:

  • 如下所示的表 Service1AuditLogsTable Service1AuditLogs as follows:

    资源提供程序Resource Provider CategoryCategory AA BB CC
    Service1Service1 AuditLogsAuditLogs x1x1 y1y1 z1z1
    Service1Service1 AuditLogsAuditLogs x5x5 y5y5 z5z5
    ......
  • 如下所示的表 Service1ErrorLogsTable Service1ErrorLogs as follows:

    资源提供程序Resource Provider CategoryCategory DD EE FF
    Service1Service1 ErrorLogsErrorLogs q1q1 w1w1 e1e1
    Service1Service1 ErrorLogsErrorLogs q2q2 w2w2 e2e2
    ......
  • 如下所示的表 Service2AuditLogsTable Service2AuditLogs as follows:

    资源提供程序Resource Provider CategoryCategory GG HH II
    Service2Service2 AuditLogsAuditLogs j1j1 k1k1 l1l1
    Service2Service2 AuditLogsAuditLogs j3j3 k3k3 l3l3
    ......

选择收集模式Select the collection mode

大多数 Azure 资源在“Azure 诊断”或“特定于资源”模式下将数据写入工作区,而不允许用户选择模式。 Most Azure resources will write data to the workspace in either Azure Diagnostic or Resource-Specific mode without giving you a choice. 有关使用哪种模式的详细信息,请参阅每个服务的文档See the documentation for each service for details on which mode it uses. 所有 Azure 服务最终都会使用特定于资源的模式。All Azure services will eventually use Resource-Specific mode. 在进行这种过渡期间,某些资源允许在诊断设置中选择模式。As part of this transition, some resources will allow you to select a mode in the diagnostic setting. 为任何新的诊断设置指定特定于资源的模式,因为这可以更轻松地管理数据,并可帮助你在今后避免复杂的迁移。Specify resource-specific mode for any new diagnostic settings since this makes the data easier to manage and may help you to avoid complex migrations at a later date.

诊断设置模式选择器

备注

有关使用资源管理器模板设置收集模式的示例,请参阅 Azure Monitor 中的诊断设置的资源管理器模板示例For an example setting the collection mode using a resource manager template, see Resource Manager template samples for diagnostic settings in Azure Monitor.

可将现有的诊断设置修改为特定于资源的模式。You can modify an existing diagnostic setting to resource-specific mode. 在这种情况下,已收集的数据将保留在 AzureDiagnostics 表中,直到根据工作区的保留设置删除了这些数据。In this case, data that was already collected will remain in the AzureDiagnostics table until it's removed according to your retention setting for the workspace. 新数据将收集到专用表中。New data will be collected in the dedicated table. 可以使用 union 运算符跨两个表查询数据。Use the union operator to query data across both tables.

有关支持特定于资源模式的 Azure 服务的公告,请继续阅读 Azure 更新博客。Continue to watch Azure Updates blog for announcements about Azure services supporting Resource-Specific mode.

AzureDiagnostics 中的列限制Column limit in AzureDiagnostics

Azure Monitor 日志中的任何一个表限制为 500 个属性。There is a 500 property limit for any table in Azure Monitor Logs. 一旦达到该限制,在引入时,包含不属于前 500 个属性的数据的所有行将被删除。Once this limit is reached, any rows containing data with any property outside of the first 500 will be dropped at ingestion time. AzureDiagnostics 表特别容易超过此限制,因为它包含写入到其中的所有 Azure 服务的属性。The AzureDiagnostics table is in particular susceptible to this limit since it includes properties for all Azure services writing to it.

如果从多个服务收集资源日志,AzureDiagnostics 可能会超过此限制,因此数据将会丢失。If you're collecting resource logs from multiple services, AzureDiagnostics may exceed this limit, and data will be missed. 在所有 Azure 服务都支持特定于资源的模式之前,应将资源配置为写入到多个工作区,以减少达到 500 列限制的可能性。Until all Azure services support resource-specific mode, you should configure resources to write to multiple workspaces to reduce the possibility of reaching the 500 column limit.

Azure 数据工厂Azure Data Factory

由于包含一组详细的日志,因此 Azure 数据工厂是一种已知会写入大量列并可能导致 AzureDiagnostics 超过其限制的服务。Azure Data Factory, because of a detailed set of logs, is a service that is known to write a large number of columns and potentially cause AzureDiagnostics to exceed its limit. 对于在启用特定于资源的模式之前配置的任何诊断设置,将会针对任一活动,为每个唯一命名的用户参数创建一个新列。For any diagnostic settings configured before the resource-specific mode was enabled, there will be a new column created for every uniquely named user parameter against any activity. 由于活动输入和输出的详细特性,将会创建更多列。More columns will be created because of the verbose nature of activity inputs and outputs.

应尽快迁移日志,以使用特定于资源的模式。You should migrate your logs to use the resource-specific mode as soon as possible. 如果无法立即做到这一点,一种临时的替代方案是将 Azure 数据工厂日志隔离到其自身的工作区,以尽量减少这些日志影响工作区中收集的其他日志类型的可能性。If you are unable to do so immediately, an interim alternative is to isolate Azure Data Factory logs into their own workspace to minimize the chance of these logs impacting other log types being collected in your workspaces.

发送到 Azure 事件中心Send to Azure Event Hubs

将资源日志发送到事件中心可将其发送到 Azure 之外,例如,发送到第三方 SIEM 或其他日志分析解决方案。Send resource logs to an event hub to send them outside of Azure, for example to a third-party SIEM or other log analytics solutions. 来自事件中心的资源日志以 JSON 格式使用,其中的 records 元素包含每个有效负载中的记录。Resource logs from event hubs are consumed in JSON format with a records element containing the records in each payload. 架构取决于资源类型,如 Azure 资源日志的通用架构和特定于服务的架构中所述。The schema depends on the resource type as described in Common and service-specific schema for Azure Resource Logs.

下面是事件中心的资源日志输出数据示例:Following is sample output data from Event Hubs for a resource log:

{
    "records": [
        {
            "time": "2019-07-15T18:00:22.6235064Z",
            "workflowId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330013509921957/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Error",
            "operationName": "Microsoft.Logic/workflows/workflowActionCompleted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T17:58:55.048482Z",
                "endTime": "2016-07-15T18:00:22.4109204Z",
                "status": "Failed",
                "code": "BadGateway",
                "resource": {
                    "subscriptionId": "00000000-0000-0000-0000-000000000000",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330013509921957",
                    "location": "chinanorth",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "29a9862f-969b-4c70-90c4-dfbdc814e413",
                    "clientTrackingId": "08587330013509921958"
                }
            }
        },
        {
            "time": "2019-07-15T18:01:15.7532989Z",
            "workflowId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330012106702630/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Information",
            "operationName": "Microsoft.Logic/workflows/workflowActionStarted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T18:01:15.5828115Z",
                "status": "Running",
                "resource": {
                    "subscriptionId": "00000000-0000-0000-0000-000000000000",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330012106702630",
                    "location": "chinanorth",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "042fb72c-7bd4-439e-89eb-3cf4409d429e",
                    "clientTrackingId": "08587330012106702632"
                }
            }
        }
    ]
}

发送到 Azure 存储Send to Azure Storage

将资源日志发送到 Azure 存储,将其保留以供存档。Send resource logs to Azure storage to retain it for archiving. 创建诊断设置以后,一旦在已启用的日志类别之一中出现事件,就会在存储帐户中创建存储容器。Once you have created the diagnostic setting, a storage container is created in the storage account as soon as an event occurs in one of the enabled log categories. 容器中的 blob 使用以下命名约定:The blobs within the container use the following naming convention:

insights-logs-{log category name}/resourceId=/SUBSCRIPTIONS/{subscription ID}/RESOURCEGROUPS/{resource group name}/PROVIDERS/{resource provider name}/{resource type}/{resource name}/y={four-digit numeric year}/m={two-digit numeric month}/d={two-digit numeric day}/h={two-digit 24-hour clock hour}/m=00/PT1H.json

例如,网络安全组的 blob 的名称可能如下所示:For example, the blob for a network security group might have a name similar to the following:

insights-logs-networksecuritygrouprulecounter/resourceId=/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/TESTRESOURCEGROUP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUP/TESTNSG/y=2016/m=08/d=22/h=18/m=00/PT1H.json

每个 PT1H.json blob 都包含一个 JSON blob,其中的事件为在 blob URL 中指定的小时(例如 h=12)内发生的。Each PT1H.json blob contains a JSON blob of events that occurred within the hour specified in the blob URL (for example, h=12). 在当前的小时内发生的事件将附加到 PT1H.json 文件。During the present hour, events are appended to the PT1H.json file as they occur. 分钟值始终为 00 (m=00),因为资源日志事件按小时细分成单个 blob。The minute value (m=00) is always 00, since resource log events are broken into individual blobs per hour.

在 PT1H.json 文件中,每个事件都按以下格式存储。Within the PT1H.json file, each event is stored with the following format. 这将使用通用顶级架构,但该架构对于每个 Azure 服务都是独一无二的,如资源日志架构中所述。This will use a common top-level schema but be unique for each Azure service as described in Resource logs schema.

{"time": "2016-07-01T00:00:37.2040000Z","systemId": "46cdbb41-cb9c-4f3d-a5b4-1d458d827ff1","category": "NetworkSecurityGroupRuleCounter","resourceId": "/SUBSCRIPTIONS/s1id1234-5679-0123-4567-890123456789/RESOURCEGROUPS/TESTRESOURCEGROUP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/TESTNSG","operationName": "NetworkSecurityGroupCounters","properties": {"vnetResourceGuid": "{12345678-9012-3456-7890-123456789012}","subnetPrefix": "10.3.0.0/24","macAddress": "000123456789","ruleName": "/subscriptions/ s1id1234-5679-0123-4567-890123456789/resourceGroups/testresourcegroup/providers/Microsoft.Network/networkSecurityGroups/testnsg/securityRules/default-allow-rdp","direction": "In","type": "allow","matchedConnections": 1988}}

后续步骤Next steps