将 Azure 资源日志流式传输到 Azure 事件中心Stream Azure resource logs to Azure Event Hubs

Azure 中的资源日志提供有关 Azure 资源内部操作的丰富、频繁的数据。Resource logs in Azure provide rich, frequent data about the internal operation of an Azure resource. 本文介绍如何将资源日志流式传输到事件中心,以便将数据发送到外部系统,例如第三方 SIEM 和其他日志分析解决方案。This article describes streaming resource logs to event hubs to send data to external systems such as third-party SIEMs and other log analytics solutions.

如何处理发送到事件中心的资源日志What you can do with resource logs sent to an event hub

将 Azure 中的资源日志流式传输到事件中心可提供以下功能:Stream resource logs in Azure to event hubs to provide the following functionality:

  • 将日志流式传输到第三方日志记录和遥测系统 - 将所有资源日志流式传输到单个事件中心,以便将日志数据通过管道传送到第三方 SIEM 或日志分析工具。Stream logs to 3rd party logging and telemetry systems – Stream all of your resource logs to a single event hub to pipe log data to a third-party SIEM or log analytics tool.

  • 生成自定义遥测和日志记录平台 - 可利用事件中心高度可缩放的发布-订阅功能,灵活地将资源日志引入自定义 teletry 平台。Build a custom telemetry and logging platform – The highly scalable publish-subscribe nature of event hubs allows you to flexibly ingest resource logs into a custom teletry platform. 有关详细信息,请参阅 Designing and Sizing a Global Scale Telemetry Platform on Azure Event Hubs(在 Azure 事件中心设计全球规模的遥测平台并设置其大小)。See Designing and Sizing a Global Scale Telemetry Platform on Azure Event Hubs for details.

  • 通过将数据流式传输到 Power BI 查看服务运行状况 - 通过事件中心、流分析和 Power BI 在 Azure 服务中将诊断数据转化成准实时见解。View service health by streaming data to Power BI – Use Event Hubs, Stream Analytics, and Power BI to transform your diagnostics data into near real-time insights on your Azure services.

    以下 SQL 代码是一个流分析查询示例,可用于将所有日志数据解析成 Power BI 表:The following SQL code is a sample Stream Analytics query that you can use to parse all the log data in to a Power BI table:

    SELECT
    records.ArrayValue.[Properties you want to track]
    INTO
    [OutputSourceName – the Power BI source]
    FROM
    [InputSourceName] AS e
    CROSS APPLY GetArrayElements(e.records) AS records
    

先决条件Prerequisites

创建事件中心(如果还没有)。You need to create an event hub if you don't already have one. 如果先前已将资源日志流式传输到此事件中心命名空间,则会重用该事件中心。If you previously streamed resource logs to this Event Hubs namespace, then that event hub will be reused.

命名空间的共享访问策略定义流式处理机制具有的权限。The shared access policy for the namespace defines the permissions that the streaming mechanism has. 流式传输到事件中心需要“管理”、“发送”和“侦听”权限。Streaming to Event Hubs requires Manage, Send, and Listen permissions. 在 Azure 门户中事件中心命名空间的“配置”选项卡下,可以创建或修改共享访问策略。You can create or modify shared access policies in the Azure portal under the Configure tab for your Event Hubs namespace.

若要更新诊断设置,使之包括流式传输,则必须在事件中心授权规则中拥有 ListKey 权限。To update the diagnostic setting to include streaming, you must have the ListKey permission on that Event Hubs authorization rule. 只要配置设置的用户同时拥有两个订阅的相应 RBAC 访问权限并且这两个订阅都在同一个 AAD 租户中,事件中心命名空间就不必与发出日志的订阅位于同一订阅中。The Event Hubs namespace does not have to be in the same subscription as the subscription that's emitting logs, as long as the user who configures the setting has appropriate RBAC access to both subscriptions and both subscriptions are in the same AAD tenant.

创建诊断设置Create a diagnostic setting

默认不会收集资源日志。Resource logs are not collected by default. 需要通过创建 Azure 资源的诊断设置,将它们发送到事件中心和其他目标。Send them to an event hub and other destinations by creating a diagnostic setting for an Azure resource. 有关详细信息,请参阅创建诊断设置以收集 Azure 中的日志和指标See Create diagnostic setting to collect logs and metrics in Azure for details.

对来自计算资源的数据进行流式处理Stream data from compute resources

本文中的过程适用于非计算资源,详见 Azure 资源日志概述The process in this article is for non-compute resources as described in Overview of Azure resource logs. 使用 Windows Azure 诊断代理对来自 Azure 计算资源的资源日志进行流式处理。Stream resource logs from Azure compute resources using the Windows Azure Diagnostics agent. 有关详细信息,请参阅使用事件中心流式处理热路径中的 Azure 诊断数据See Streaming Azure Diagnostics data in the hot path by using Event Hubs for details.

使用事件中心的日志数据Consuming log data from event hubs

使用事件中心的资源日志时,这些日志为 JSON 格式,其中包含下表中的元素。When you consume resource logs from event hubs, it will be is JSON format with the elements in the following table.

元素名称Element Name 说明Description
recordsrecords 此有效负载中所有日志事件的数组。An array of all log events in this payload.
timetime 发生事件的时间。Time at which the event occurred.
categorycategory 此事件的日志类别。Log category for this event.
ResourceIdresourceId 生成此事件的资源的资源 ID。Resource ID of the resource that generated this event.
operationNameoperationName 操作的名称。Name of the operation.
levellevel 可选。Optional. 指示日志事件级别。Indicates the log event level.
propertiesproperties 事件的属性。Properties of the event. 这些属性因 Azure 服务而异,详见 These will vary for each Azure service as described in .

下面是事件中心的输出数据示例:Following is sample output data from Event Hubs:

{
    "records": [
        {
            "time": "2016-07-15T18:00:22.6235064Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330013509921957/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Error",
            "operationName": "Microsoft.Logic/workflows/workflowActionCompleted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T17:58:55.048482Z",
                "endTime": "2016-07-15T18:00:22.4109204Z",
                "status": "Failed",
                "code": "BadGateway",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330013509921957",
                    "location": "China East",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "29a9862f-969b-4c70-90c4-dfbdc814e413",
                    "clientTrackingId": "08587330013509921958"
                }
            }
        },
        {
            "time": "2016-07-15T18:01:15.7532989Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330012106702630/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Information",
            "operationName": "Microsoft.Logic/workflows/workflowActionStarted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T18:01:15.5828115Z",
                "status": "Running",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330012106702630",
                    "location": "China East",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "042fb72c-7bd4-439e-89eb-3cf4409d429e",
                    "clientTrackingId": "08587330012106702632"
                }
            }
        }
    ]
}

后续步骤Next steps