将 Azure 监视数据流式传输到事件中心以便外部工具使用Stream Azure monitoring data to an event hub for consumption by an external tool

本文将演练如何将 Azure 环境中的不同数据层设置为发送到单个事件中心命名空间或事件中心,以便由外部工具收集。This article walks through setting up different tiers of data from your Azure environment to be sent to a single Event Hubs namespace or event hub, where it can be collected by an external tool.

可将哪些数据发送到事件中心?What data can I send into an event hub?

在 Azure 环境中,有多“层”监视数据,访问每层数据的方法略有不同。Within your Azure environment, there are several 'tiers' of monitoring data, and the method of accessing data from each tier varies slightly. 通常情况下,这些层可描述为:Typically, these tiers can be described as:

  • 应用程序监视数据: 有关已编写并在 Azure 上运行的代码的性能和功能的数据。Application monitoring data: Data about the performance and functionality of the code you have written and are running on Azure. 应用程序监视数据的示例包括性能跟踪、应用程序日志及用户遥测数据。Examples of application monitoring data include performance traces, application logs, and user telemetry. 通常以下列的一种方式收集应用程序监视数据:Application monitoring data is usually collected in one of the following ways:
  • 来宾 OS 监视数据: 有关运行应用程序的操作系统的数据。Guest OS monitoring data: Data about the operating system on which your application is running. 来宾 OS 监视数据的示例有 Linux syslog 或 Windows 系统日志。Examples of guest OS monitoring data would be Linux syslog or Windows system events. 若要收集此类型的数据,需安装代理,如 Windows Azure 诊断代理Linux Azure 诊断代理To collect this type of data, you need to install an agent such as the Windows Azure Diagnostic Agent or Linux Azure Diagnostic Agent.
  • Azure 资源监视数据: 有关 Azure 资源操作的数据。Azure resource monitoring data: Data about the operation of an Azure resource. 对于某些 Azure 资源类型(如虚拟机),该 Azure 服务中会监视来宾 OS 和应用程序。For some Azure resource types, such as virtual machines, there is a guest OS and application(s) to monitor inside of that Azure service. 对于其他 Azure 资源(如网络安全组),资源监视数据是可用数据的最高层(因为没有 来宾 OS 或应用程序在这些资源中运行)。For other Azure resources, such as Network Security Groups, the resource monitoring data is the highest tier of data available (since there is no guest OS or application running in those resources). 可以使用资源诊断设置收集这些数据。This data can be collected using resource diagnostic settings.
  • Azure 订阅监视数据: 有关 Azure 订阅操作和管理的数据,以及有关 Azure 本身运行状况和操作的数据。Azure subscription monitoring data: Data about the operation and management of an Azure subscription, as well as data about the health and operation of Azure itself. 活动日志包含大多数订阅监视数据,例如服务运行状况事件和 Azure 资源管理器审核。The activity log contains most subscription monitoring data, such as service health incidents and Azure Resource Manager audits. 可以使用日志配置文件收集此数据。You can collect this data using a Log Profile.
  • Azure 租户监视数据: 有关租户级 Azure 服务(例如 Azure Active Directory)操作的数据。Azure tenant monitoring data: Data about the operation of tenant-level Azure services, such as Azure Active Directory. Azure Active Directory 审核和登录是租户监视数据的示例。The Azure Active Directory audits and sign-ins are examples of tenant monitoring data. 可以使用租户诊断设置收集此数据。This data can be collected using a tenant diagnostic setting.

可将任何层的数据发送到事件中心,以便将其拉取到合作伙伴工具。Data from any tier can be sent into an event hub, where it can be pulled into a partner tool. 可将某些源配置为直接向事件中心发送数据,同时,可能需要使用另一个进程(例如逻辑应用)来检索所需的数据。Some sources can be configured to send data directly to an event hub while another process such as a Logic App may be required to retrieve the required data. 以下各节描述了如何将每层数据配置为流式传输到事件中心。The next sections describe how you can configure data from each tier to be streamed to an event hub. 这些步骤假定你拥有处于要监视的层的资产。The steps assume that you already have assets at that tier to be monitored.

设置事件中心命名空间Set up an Event Hubs namespace

开始之前,需创建事件中心命名空间和事件中心Before you begin, you need to create an Event Hubs namespace and event hub. 此命名空间和事件中心是所有监视数据的目标。This namespace and event hub is the destination for all of your monitoring data. 事件中心命名空间是共享相同访问策略的事件中心的逻辑分组,就像存储帐户中有各个 blob 一样。An Event Hubs namespace is a logical grouping of event hubs that share the same access policy, much like a storage account has individual blobs within that storage account. 请注意有关所创建的事件中心命名空间和事件中心的一些详细信息:Please note a few details about the event hubs namespace and event hubs that you create:

  • 我们建议使用标准事件中心命名空间。We recommend using a Standard Event Hubs namespace.
  • 通常,只需要一个吞吐量单位。Typically, only one throughput unit is necessary. 如果需要在日志使用量增加时纵向扩展,以后始终可以手动增加命名空间的吞吐量单位数或启用自动膨胀。If you need to scale up as your log usage increases, you can always manually increase the number of throughput units for the namespace later or enable auto inflation.
  • 使用吞吐量单位数,可增加事件中心的吞吐量规模。The number of throughput units allows you to increase throughput scale for your event hubs. 使用分区数可以在多个使用者之间并行使用。The number of partitions allows you to parallelize consumption across many consumers. 单个分区最多可以执行 20MBps,或者大约每秒 20,000 条消息。A single partition can do up to 20MBps, or approximately 20,000 messages per second. 不一定支持从多个分区使用,具体取决于使用数据的工具。Depending on the tool consuming the data, it may or may not support consuming from multiple partitions. 如果不确定要设置的分区数,我们建议从四个分区开始。If you're not sure about the number of partitions to set, we recommend starting with four partitions.
  • 我们建议将事件中心的消息保留期设置为 7 天。We recommend that you set message retention on your event hub to 7 days. 如果使用的工具多天出现故障,这可确保该工具可以从它中断的位置重新开始(因为事件最多可保存 7 天)。If your consuming tool goes down for more than a day, this ensures that the tool can pick up where it left off (for events up to 7 days old).
  • 我们建议将默认使用者组用于事件中心。We recommend using the default consumer group for your event hub. 除非你打算使用两个不同的工具使用同一事件中心内的相同数据,否则无需创建其他使用者组或使用单独的使用者组。There is no need to create other consumer groups or use a separate consumer group unless you plan to have two different tools consume the same data from the same event hub.
  • 对于 Azure 活动日志,可选择事件中心命名空间,Azure Monitor 将在该命名空间内创建名为“insights-logs-operationallogs”的事件中心。For the Azure Activity Log, you pick an Event Hubs namespace and Azure Monitor creates an event hub within that namespace called 'insights-logs-operationallogs.' 对于其他日志类型,可以选择现有事件中心(可以重复使用同一 insights-logs-operationallogs 事件中心),也可以让 Azure Monitor 为每个日志类别创建一个事件中心。For other log types, you can either choose an existing event hub (allowing you to reuse the same insights-logs-operationallogs event hub) or have Azure Monitor create an event hub per log category.
  • 通常,必须在使用事件中心数据的计算机上打开端口 5671 和端口 5672。Typically, port 5671 and 5672 must be opened on the machine consuming data from the event hub.

另请参阅 Azure 事件中心常见问题解答Please also see the Azure Event Hubs FAQ.

Azure 租户监视数据Azure tenant monitoring data

Azure 租户监视数据目前仅适用于 Azure Active Directory。Azure tenant monitoring data is currently only available for Azure Active Directory. 可以使用 Azure Active Directory 报告中的数据,其中包含特定租户中的登录活动历史记录和更改审核跟踪。You can use the data from Azure Active Directory reporting, which contains the history of sign-in activity and audit trail of changes made within a particular tenant.

Azure Active Directory 数据Azure Active Directory data

若要将 Azure Active Directory 日志中的数据发送到事件中心命名空间,请在 AAD 租户上设置租户诊断设置。To send data from the Azure Active Directory log into an Event Hubs namespace, you set up a tenant diagnostic setting on your AAD tenant.

Azure 订阅监视数据Azure subscription monitoring data

Azure 订阅监视数据可以在 Azure 活动日志中找到。Azure subscription monitoring data is available in the Azure activity log. 此日志包含来自资源管理器的创建、更新和删除操作;Azure 服务运行状况中可能影响订阅中资源的更改;资源运行状况状态转换;以及若干其他类型的订阅级别事件。This contains the create, update, and delete operations from Resource Manager, the changes in Azure service health that may impact resources in your subscription, the resource health state transitions, and several other types of subscription-level events. 本文详细介绍了 Azure 活动日志中显示的所有事件类别This article details all categories of events that appear in the Azure activity log.

活动日志数据Activity log data

若要将数据从 Azure 活动日志发送到事件中心命名空间,请在订阅上设置日志配置文件。To send data from the Azure activity log into an Event Hubs namespace, you set up a Log Profile on your subscription. 对要监视每个订阅执行一次此操作。Do this once per subscription you want to monitor.

Tip

日志配置文件当前仅允许选择一个事件中心命名空间,并将在其中创建名为“insights-operational-logs”的事件日志。A Log Profile currently only allows you to select an Event Hubs namespace, in which an event hub is created with the name 'insights-operational-logs.' 尚不可在日志配置文件中指定自己的事件中心名称。It is not yet possible to specify your own event hub name in a Log Profile.

Azure 资源指标和诊断日志Azure resource metrics and diagnostics logs

Azure 资源将发出两种类型的监视数据:Azure resources emit two types of monitoring data:

  1. 资源诊断日志Resource diagnostic logs
  2. 度量值Metrics

使用资源诊断设置将两种类型的数据发送到事件中心。Both types of data are sent to an event hub using a resource diagnostic setting. 按照本指南在特定资源上设置资源诊断设置。Follow this guide to set up a resource diagnostic setting on a particular resource. 在要从其收集日志的每个资源上设置资源诊断设置。Set a resource diagnostic setting on each resource from which you want to collect logs.

Tip

可使用 Azure Policy,在策略规则中使用 DeployIfNotExists 效果,确保特定范围内的每个资源始终设置了诊断设置。You can use Azure Policy to ensure that every resource within a certain scope is always set up with a diagnostic setting by using the DeployIfNotExists effect in the policy rule.

来宾 OS 数据Guest OS data

需要安装代理以将来宾 OS 监视数据发送到事件中心。You need to install an agent to send guest OS monitoring data into an event hub. 对于 Windows 或 Linux,请在配置文件中指定要发送到事件中心的数据,以及应将数据发送到的事件中心,并将该配置文件传递给在 VM 上运行的代理。For either Windows or Linux, you specify the data you want to be sent to the event hub as well as the event hub to which the data should be sent in a configuration file and pass that configuration file to the agent running on the VM.

Linux 数据Linux data

Linux Azure 诊断代理用于将来自 Linux 计算机的监视数据发送到事件中心。The Linux Azure Diagnostic agent can be used to send monitoring data from a Linux machine to an event hub. 在 LAD 配置文件保护的设置 JSON 中添加事件中心作为接收器,即可完成此操作。Do this by adding the event hub as a sink in your LAD configuration file protected settings JSON. 参阅此文章,详细了解如何向 Linux Azure 诊断代理添加事件中心接收器See this article to learn more about adding the event hub sink to your Linux Azure Diagnostic agent.

Note

不能在门户中将来宾 OS 监视数据设置为流式传输到事件中心。You cannot set up streaming of guest OS monitoring data to an event hub in the portal. 相反,必须手动编辑配置文件。Instead, you must manually edit the configuration file.

Windows 数据Windows data

Windows Azure 诊断代理用于将来自 Windows 计算机的监视数据发送到事件中心。The Windows Azure Diagnostic agent can be used to send monitoring data from a Windows machine to an event hub. 在 WAD 配置文件的 privateConfig 部分添加事件中心作为接收器,即可完成此操作。Do this by adding the event hub as a sink in your privateConfig section of the WAD configuration file. 参阅此文章,详细了解如何向 Windows Azure 诊断代理添加事件中心接收器See this article to learn more about adding the event hub sink to your Windows Azure Diagnostic agent.

Note

不能在门户中将来宾 OS 监视数据设置为流式传输到事件中心。You cannot set up streaming of guest OS monitoring data to an event hub in the portal. 相反,必须手动编辑配置文件。Instead, you must manually edit the configuration file.

应用程序监视数据Application monitoring data

应用程序监视数据要求代码经过 SDK 检测,因此没有将应用程序监视数据路由到 Azure 中事件中心的通用解决方案。Application monitoring data requires that your code is instrumented with an SDK, so there isn't a general-purpose solution to routing application monitoring data to an event hub in Azure. 如果使用 Application Insights,可通过执行以下操作,将监视数据流式传输到事件中心:If you are using Application Insights, you can stream monitoring data to an event hub by doing the following:

  1. 将 Application Insights 数据设置为连续导出)到存储帐户。Set up continuous export of the Application Insights data to a storage account.

  2. 设置计时器触发逻辑应用,从 blob 存储拉取数据将其作为消息推送到事件中心Set up a timer-triggered Logic App that pulls data from blob storage and pushes it as a message to the event hub.

可对发送到事件中心的监视数据执行什么操作?What can I do with the monitoring data being sent to my event hub?

通过 Azure Monitor 将监视数据路由到事件中心,可与合作伙伴 SIEM 和监视工具轻松集成。Routing your monitoring data to an event hub with Azure Monitor enables you to easily integrate with partner SIEM and monitoring tools. 大多数工具需要事件中心连接字符串和对 Azure 订阅的某些权限,才能从事件中心读取数据。Most tools require the event hub connection string and certain permissions to your Azure subscription to read data from the event hub. 下面是与 Azure Monitor 集成的工具的不完整列表:Here is a non-exhaustive list of tools with Azure Monitor integration:

后续步骤Next Steps