将 Azure 平台日志流式传输到 Azure 事件中心Stream Azure platform logs to Azure Event Hubs

Azure 中的平台日志(包括 Azure 活动日志和资源日志)提供 Azure 资源及其所依赖的 Azure 平台的详细诊断和审核信息。Platform logs in Azure, including Azure Activity log and resource logs, provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on. 本文介绍如何将平台日志流式传输到事件中心,以便将数据发送到外部系统,例如第三方 SIEM 和其他日志分析解决方案。This article describes streaming platform logs to event hubs to send data to external systems such as third-party SIEMs and other log analytics solutions.

如何处理发送到事件中心的平台日志What you can do with platform logs sent to an event hub

将 Azure 中的平台日志流式传输到事件中心可提供以下功能:Stream platform logs in Azure to event hubs to provide the following functionality:

  • 将日志流式传输到第三方日志记录和遥测系统 - 将所有平台日志流式传输到单个事件中心,以便将日志数据通过管道传送到第三方 SIEM 或日志分析工具。Stream logs to 3rd party logging and telemetry systems – Stream all of your platform logs to a single event hub to pipe log data to a third-party SIEM or log analytics tool.

  • 生成自定义遥测和日志记录平台 - 可利用事件中心高度可缩放的发布-订阅功能,灵活地将平台日志引入自定义 teletry 平台。Build a custom telemetry and logging platform – The highly scalable publish-subscribe nature of event hubs allows you to flexibly ingest platform logs into a custom teletry platform.

  • 通过将数据流式传输到 Power BI 查看服务运行状况 - 通过事件中心、流分析和 Power BI 在 Azure 服务中将诊断数据转化成准实时见解。View service health by streaming data to Power BI – Use Event Hubs, Stream Analytics, and Power BI to transform your diagnostics data into near real-time insights on your Azure services.

    以下 SQL 代码是一个流分析查询示例,可用于将所有日志数据解析成 Power BI 表:The following SQL code is a sample Stream Analytics query that you can use to parse all the log data in to a Power BI table:

    SELECT
    records.ArrayValue.[Properties you want to track]
    INTO
    [OutputSourceName – the Power BI source]
    FROM
    [InputSourceName] AS e
    CROSS APPLY GetArrayElements(e.records) AS records
    

先决条件Prerequisites

创建事件中心(如果还没有)。You need to create an event hub if you don't already have one. 如果先前已将资源日志流式传输到此事件中心命名空间,则会重用该事件中心。If you previously streamed resource logs to this Event Hubs namespace, then that event hub will be reused.

命名空间的共享访问策略定义流式处理机制具有的权限。The shared access policy for the namespace defines the permissions that the streaming mechanism has. 流式传输到事件中心需要“管理”、“发送”和“侦听”权限。Streaming to Event Hubs requires Manage, Send, and Listen permissions. 在 Azure 门户中事件中心命名空间的“配置”选项卡下,可以创建或修改共享访问策略。You can create or modify shared access policies in the Azure portal under the Configure tab for your Event Hubs namespace.

若要更新诊断设置,使之包括流式传输,则必须在事件中心授权规则中拥有 ListKey 权限。To update the diagnostic setting to include streaming, you must have the ListKey permission on that Event Hubs authorization rule. 只要配置设置的用户同时拥有两个订阅的相应 RBAC 访问权限并且这两个订阅都在同一个 AAD 租户中,事件中心命名空间就不必与发出日志的订阅位于同一订阅中。The Event Hubs namespace does not have to be in the same subscription as the subscription that's emitting logs, as long as the user who configures the setting has appropriate RBAC access to both subscriptions and both subscriptions are in the same AAD tenant.

创建诊断设置Create a diagnostic setting

通过创建 Azure 资源的诊断设置,将平台日志发送到事件中心和其他目标。Send platform logs to an event hub and other destinations by creating a diagnostic setting for an Azure resource. 有关详细信息,请参阅创建诊断设置以收集 Azure 中的日志和指标See Create diagnostic setting to collect logs and metrics in Azure for details.

对来自计算资源的数据进行收集Collect data from compute resources

诊断设置将如同收集任何其他资源一样收集 Azure 计算资源的资源日志,但不是其来宾操作系统或工作负荷。Diagnostic settings will collect resource logs for Azure compute resources like any other resource, but not their guest operating system or workloads. 若要收集此数据,请安装 Log Analytics 代理To collect this data, install the Log Analytics agent.

使用事件中心的日志数据Consuming log data from event hubs

事件中心的平台日志以 JSON 格式使用,其中包含下表中的元素。Platform logs from event hubs are consumed in JSON format with the elements in the following table.

元素名称Element Name 说明Description
recordsrecords 此有效负载中所有日志事件的数组。An array of all log events in this payload.
timetime 发生事件的时间。Time at which the event occurred.
categorycategory 此事件的日志类别。Log category for this event.
ResourceIdresourceId 生成此事件的资源的资源 ID。Resource ID of the resource that generated this event.
operationNameoperationName 操作的名称。Name of the operation.
levellevel 可选。Optional. 指示日志事件级别。Indicates the log event level.
propertiesproperties 事件的属性。Properties of the event. 这些属性因 Azure 服务而异,详见 These will vary for each Azure service as described in .

下面是事件中心的资源日志输出数据示例:Following is sample output data from Event Hubs for a resource log:

{
    "records": [
        {
            "time": "2016-07-15T18:00:22.6235064Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330013509921957/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Error",
            "operationName": "Microsoft.Logic/workflows/workflowActionCompleted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T17:58:55.048482Z",
                "endTime": "2016-07-15T18:00:22.4109204Z",
                "status": "Failed",
                "code": "BadGateway",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330013509921957",
                    "location": "China East",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "29a9862f-969b-4c70-90c4-dfbdc814e413",
                    "clientTrackingId": "08587330013509921958"
                }
            }
        },
        {
            "time": "2016-07-15T18:01:15.7532989Z",
            "workflowId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA",
            "resourceId": "/SUBSCRIPTIONS/DF602C9C-7AA0-407D-A6FB-EB20C8BD1192/RESOURCEGROUPS/JOHNKEMTEST/PROVIDERS/MICROSOFT.LOGIC/WORKFLOWS/JOHNKEMTESTLA/RUNS/08587330012106702630/ACTIONS/SEND_EMAIL",
            "category": "WorkflowRuntime",
            "level": "Information",
            "operationName": "Microsoft.Logic/workflows/workflowActionStarted",
            "properties": {
                "$schema": "2016-04-01-preview",
                "startTime": "2016-07-15T18:01:15.5828115Z",
                "status": "Running",
                "resource": {
                    "subscriptionId": "df602c9c-7aa0-407d-a6fb-eb20c8bd1192",
                    "resourceGroupName": "JohnKemTest",
                    "workflowId": "243aac67fe904cf195d4a28297803785",
                    "workflowName": "JohnKemTestLA",
                    "runId": "08587330012106702630",
                    "location": "China East",
                    "actionName": "Send_email"
                },
                "correlation": {
                    "actionTrackingId": "042fb72c-7bd4-439e-89eb-3cf4409d429e",
                    "clientTrackingId": "08587330012106702632"
                }
            }
        }
    ]
}

后续步骤Next steps