将 Azure 诊断日志流式传输到事件中心Stream Azure Diagnostic Logs to an event hub

可将 Azure 诊断日志 以近实时方式流式传输到任何应用程序,方法是使用门户中的内置“导出到事件中心”选项,或者通过 Azure PowerShell Cmdlet 或 Azure CLI 在诊断设置中启用事件中心授权规则 ID。Azure diagnostic logs can be streamed in near real time to any application using the built-in “Export to Event Hubs” option in the Portal, or by enabling the Event Hub Authorization Rule ID in a diagnostic setting via the Azure PowerShell Cmdlets or Azure CLI.

可以对诊断日志和事件中心执行的操作What you can do with diagnostics logs and Event Hubs

可以通过下述几种方式将流式处理功能用于诊断日志:Here are just a few ways you might use the streaming capability for Diagnostic Logs:

  • 将日志流式传输到第三方日志记录和遥测系统 - 可以将所有诊断日志流式传输到单个事件中心,以便将日志数据通过管道传送到第三方 SIEM 或日志分析工具。Stream logs to 3rd party logging and telemetry systems – You can stream all of your diagnostic logs to a single event hub to pipe log data to a third-party SIEM or log analytics tool.

  • 通过将“热路径”数据流式传输到 Power BI 查看服务运行状况 – 可以通过事件中心、流分析和 Power BI 在 Azure 服务中轻松将诊断数据转化成准实时分析结果。View service health by streaming “hot path” data to Power BI – Using Event Hubs, Stream Analytics, and Power BI, you can easily transform your diagnostics data in to near real-time insights on your Azure services. 下面是有关如何设置诊断日志的一些提示:Here are a few tips for getting set up with diagnostic logs:

    • 在门户中选中相关选项或通过 PowerShell 启用相关选项以后,即可自动创建用于某类诊断日志的事件中心,因此需在命名空间中选择名称以 insights- 开头的事件中心。An event hub for a category of diagnostic logs is created automatically when you check the option in the portal or enable it through PowerShell, so you want to select the event hub in the namespace with the name that starts with insights-.

    • 以下 SQL 代码是一个流分析查询示例,可用于将所有日志数据解析成 Power BI 表:The following SQL code is a sample Stream Analytics query that you can use to parse all the log data in to a Power BI table:

      SELECT
      records.ArrayValue.[Properties you want to track]
      INTO
      [OutputSourceName – the Power BI source]
      FROM
      [InputSourceName] AS e
      CROSS APPLY GetArrayElements(e.records) AS records
      
  • 生成自定义遥测和日志记录平台 – 如果已经有一个自定义生成的遥测平台,或者正想生成一个,则可利用事件中心高度可缩放的发布-订阅功能,灵活地引入诊断日志。Build a custom telemetry and logging platform – If you already have a custom-built telemetry platform or are just thinking about building one, the highly scalable publish-subscribe nature of Event Hubs allows you to flexibly ingest diagnostic logs.

启用诊断日志的流式处理Enable streaming of diagnostic logs

可以通过门户或使用 Azure Monitor REST API 以编程方式启用诊断日志的流式处理。You can enable streaming of diagnostic logs programmatically, via the portal, or using the Azure Monitor REST APIs. 无论采用哪种方式,都可以创建一个诊断设置并在其中指定事件中心命名空间,以及要发送到该命名空间的日志类别和指标。Either way, you create a diagnostic setting in which you specify an Event Hubs namespace and the log categories and metrics you want to send in to the namespace. 在该命名空间中针对每个启用的日志类别创建一个事件中心。An event hub is created in the namespace for each log category you enable. 诊断日志类别是一类可供资源收集的日志。A diagnostic log category is a type of log that a resource may collect.

Warning

从计算资源(例如,VM 或 Service Fabric)启用诊断日志并对其进行流式处理需要另一组步骤Enabling and streaming diagnostic logs from Compute resources (for example, VMs or Service Fabric) requires a different set of steps.

只要配置设置的用户同时拥有两个订阅的相应 RBAC 访问权限并且这两个订阅都是属于同一个 AAD 租户,事件中心命名空间就不必与资源发出日志位于同一订阅中。The Event Hubs namespace does not have to be in the same subscription as the resource emitting logs as long as the user who configures the setting has appropriate RBAC access to both subscriptions and both subscriptions are part of the same AAD tenant.

Note

当前不支持通过诊断设置发送多维指标。Sending multi-dimensional metrics via diagnostic settings is not currently supported. 多维指标将按平展后的单维指标导出,并跨维值聚合。Metrics with dimensions are exported as flattened single dimensional metrics, aggregated across dimension values.

例如:可以基于每个队列级别浏览和绘制事件中心上的“传入消息”指标。For example: The 'Incoming Messages' metric on an Event Hub can be explored and charted on a per queue level. 但是,当通过诊断设置导出时,该指标将表示为事件中心的所有队列中的所有传入消息。However, when exported via diagnostic settings the metric will be represented as all incoming messages across all queues in the Event Hub.

使用门户流式传输诊断日志Stream diagnostic logs using the portal

  1. 在门户中,导航到 Azure Monitor 并单击“诊断设置”In the portal, navigate to Azure Monitor and click on Diagnostic Settings

    Azure Monitor 的“监视”部分

  2. (可选)按资源组或资源类型筛选列表,并单击要为其设置诊断设置的资源。Optionally filter the list by resource group or resource type, then click on the resource for which you would like to set a diagnostic setting.

  3. 如果选定的资源上不存在任何设置,系统会提示创建设置。If no settings exist on the resource you have selected, you are prompted to create a setting. 单击“启用诊断”。Click "Turn on diagnostics."

    添加诊断设置 - 没有现有的设置

    如果资源上有现有的设置,则会看到已在此资源上配置的设置列表。If there are existing settings on the resource, you will see a list of settings already configured on this resource. 单击“添加诊断设置”。Click "Add diagnostic setting."

    添加诊断设置 - 现有的设置

  4. 为设置命名,选中“流式传输到事件中心”对应的复选框,并选择一个事件中心命名空间。Give your setting a name and check the box for Stream to an event hub, then select an Event Hubs namespace.

    添加诊断设置 - 现有的设置

    选择的命名空间将是创建事件中心的地方(如果这是用户第一次流式传输诊断日志),或者是将诊断日志流式传输到事件中心的地方(如果已有资源将日志类别流式传输到该命名空间),而策略则定义流式传输机制所具有的权限。The namespace selected will be where the event hub is created (if this is your first time streaming diagnostic logs) or streamed to (if there are already resources that are streaming that log category to this namespace), and the policy defines the permissions that the streaming mechanism has. 目前,流式传输到事件中心需要“管理”、“发送”和“侦听”权限。Today, streaming to an event hub requires Manage, Send, and Listen permissions. 在门户中针对命名空间的“配置”选项卡下,可以创建或修改事件中心命名空间的共享访问策略。You can create or modify Event Hubs namespace shared access policies in the portal under the Configure tab for your namespace. 若要更新这些诊断设置的其中一个,必须在事件中心授权规则中让客户端拥有 ListKey 权限。To update one of these diagnostic settings, the client must have the ListKey permission on the Event Hubs authorization rule. 还可以选择指定事件中心名称。You can also optionally specify an event hub name. 如果指定事件中心名称,日志将被路由到该事件中心,而不是为每个日志类别新创建的事件中心。If you specify an event hub name, logs are routed to that event hub rather than to a newly created event hub per log category.

  5. 单击“保存” 。Click Save.

几分钟后,新设置会显示在此资源的设置列表中,只要生成新的事件数据,就会立即将诊断日志流式传输到该事件中心。After a few moments, the new setting appears in your list of settings for this resource, and diagnostic logs are streamed to that event hub as soon as new event data is generated.

通过 PowerShell CmdletVia PowerShell Cmdlets

Note

本文进行了更新,以便使用新的 Azure PowerShell Az 模块。This article has been updated to use the new Azure PowerShell Az module. 你仍然可以使用 AzureRM 模块,至少在 2020 年 12 月之前,它将继续接收 bug 修补程序。You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. 若要详细了解新的 Az 模块和 AzureRM 兼容性,请参阅新 Azure Powershell Az 模块简介To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. 有关 Az 模块安装说明,请参阅安装 Azure PowerShellFor Az module installation instructions, see Install Azure PowerShell.

若要通过 Azure PowerShell Cmdlet 启用流式传输,可以使用 Set-AzDiagnosticSetting cmdlet 并设置以下参数:To enable streaming via the Azure PowerShell Cmdlets, you can use the Set-AzDiagnosticSetting cmdlet with these parameters:

Set-AzDiagnosticSetting -ResourceId [your resource ID] -EventHubAuthorizationRuleId [your Event Hub namespace auth rule ID] -Enabled $true

事件中心授权规则 ID 是以下格式的字符串:{Event Hub namespace resource ID}/authorizationrules/{key name},例如 /subscriptions/{subscription ID}/resourceGroups/{resource group}/providers/Microsoft.EventHub/namespaces/{Event Hub namespace}/authorizationrules/RootManageSharedAccessKeyThe Event Hub Authorization Rule ID is a string with this format: {Event Hub namespace resource ID}/authorizationrules/{key name}, for example, /subscriptions/{subscription ID}/resourceGroups/{resource group}/providers/Microsoft.EventHub/namespaces/{Event Hub namespace}/authorizationrules/RootManageSharedAccessKey. 目前无法使用 PowerShell 选择特定事件中心名称。You cannot currently select a particular event hub name with PowerShell.

通过 Azure CLIVia Azure CLI

若要通过 Azure CLI 启用流式传输,可以使用 az monitor diagnostic-settings create 命令。To enable streaming via the Azure CLI, you can use the az monitor diagnostic-settings create command.

az monitor diagnostic-settings create --name <diagnostic name> \
    --event-hub <event hub name> \
    --event-hub-rule <event hub rule ID> \
    --resource <target resource object ID> \
    --logs '[
    {
        "category": <category name>,
        "enabled": true
    }
    ]'

可以通过将字典添加到以 --logs 参数传递的 JSON 数组来将其他类别添加到诊断日志。You can add additional categories to the diagnostic log by adding dictionaries to the JSON array passed as the --logs parameter.

--event-hub-rule 参数使用与事件中心授权规则 ID 相同的格式,如 PowerShell Cmdlet 的说明所述。The --event-hub-rule argument uses the same format as the Event Hub Authorization Rule ID as explained for the PowerShell Cmdlet.

如何使用事件中心的日志数据?How do I consume the log data from Event Hubs?

下面是事件中心的输出数据示例:Here is sample output data from Event Hubs:

{
    "records": [
        {
            "time": "2016-07-15T18:00:22.6235064Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330013509921957/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Error",
            "operationName": "Microsoft.Logic/workflows/workflowActionCompleted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T17:58:55.048482Z",
                "endTime": "2016-07-15T18:00:22.4109204Z",
                "status": "Failed",
                "code": "BadGateway",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330013509921957",
                    "location": "westus",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "29a9862f-969b-4c70-90c4-dfbdc814e413",
                    "clientTrackingId": "08587330013509921958"
                }
            }
        },
        {
            "time": "2016-07-15T18:01:15.7532989Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330012106702630/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Information",
            "operationName": "Microsoft.Logic/workflows/workflowActionStarted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T18:01:15.5828115Z",
                "status": "Running",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330012106702630",
                    "location": "westus",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "042fb72c-7bd4-439e-89eb-3cf4409d429e",
                    "clientTrackingId": "08587330012106702632"
                }
            }
        }
    ]
}
元素名称Element Name 说明Description
recordsrecords 此有效负载中所有日志事件的数组。An array of all log events in this payload.
timetime 发生事件的时间。Time at which the event occurred.
categorycategory 此事件的日志类别。Log category for this event.
ResourceIdresourceId 生成此事件的资源的资源 ID。Resource ID of the resource that generated this event.
operationNameoperationName 操作的名称。Name of the operation.
级别level 可选。Optional. 指示日志事件级别。Indicates the log event level.
propertiesproperties 事件的属性。Properties of the event.

可以在此处查看支持流式传输到事件中心的所有资源提供程序的列表。You can view a list of all resource providers that support streaming to Event Hubs here.

对来自计算资源的数据进行流式处理Stream data from Compute resources

还可以使用 Windows Azure 诊断代理对来自计算资源的诊断日志进行流式处理。You can also stream diagnostic logs from Compute resources using the Windows Azure Diagnostics agent. 请参阅本文 了解如何进行设置。See this article for how to set that up.

后续步骤Next steps