了解和调整流式处理单元Understand and adjust Streaming Units

流单元 (SU) 表示分配用于执行流分析作业的计算资源。Streaming Units (SUs) represents the computing resources that are allocated to execute a Stream Analytics job. SU 数目越大,为作业分配的 CPU 和内存资源就越多。The higher the number of SUs, the more CPU and memory resources are allocated for your job. 此容量使你能够专注于查询逻辑,并且无需管理及时运行流分析作业所需的硬件。This capacity lets you focus on the query logic and abstracts the need to manage the hardware to run your Stream Analytics job in a timely manner.

为了实现低延迟流式处理,Azure 流分析作业将执行内存中的所有处理。To achieve low latency stream processing, Azure Stream Analytics jobs perform all processing in memory. 内存不足时,流式处理作业会失败。When running out of memory, the streaming job fails. 因此,对于生产作业,请务必监视流式处理作业的资源使用情况,并确保分配有足够的资源来保持作业的全天候运行。As a result, for a production job, it’s important to monitor a streaming job’s resource usage, and make sure there is enough resource allocated to keep the jobs running 24/7.

SU 利用率指标的范围为 0% 到 100%,描述工作负荷的内存消耗量。The SU % utilization metric, which ranges from 0% to 100%, describes the memory consumption of your workload. 对于占用最小内存的流式处理作业,此指标通常介于 10% 到 20%。For a streaming job with minimal footprint, this metric is usually between 10% to 20%. 如果 SU 利用率较低并且输入事件积压,则可能表示工作负荷需更多的计算资源,这就需要增加 SU 的数目。If SU% utilization is low and input events get backlogged, your workload likely requires more compute resources, which requires you to increase the number of SUs. 最好保持低于 80% 的 SU 指标,以应对偶发的峰值。It’s best to keep the SU metric below 80% to account for occasional spikes. Microsoft 建议针对 SU 利用率指标达到 80% 设置警报,以防止资源耗尽。Microsoft recommends setting an alert on 80% SU Utilization metric to prevent resource exhaustion. 有关详细信息,请参阅教程:为 Azure 流分析作业设置警报For more information, see Tutorial: Set up alerts for Azure Stream Analytics jobs.

配置流分析流式处理单元 (SU)Configure Stream Analytics Streaming Units (SUs)

  1. 登录到 Azure 门户Sign in to Azure portal

  2. 在资源列表中,找到要缩放的流分析作业,然后将其打开。In the list of resources, find the Stream Analytics job that you want to scale and then open it. 

  3. 在作业页中的“配置”标题下,选择“缩放”。 In the job page, under the Configure heading, select Scale. 

    Azure 门户流分析作业配置

  4. 使用滑块设置作业的 SU。Use the slider to set the SUs for the job. 请注意,只能设置特定的 SU。Notice that you are limited to specific SU settings. 

  5. 即使在作业运行时,也可以更改分配给作业的 SU 数量。You can change the number of SUs assigned to your job even when it is running. 如果作业使用未分区输出或包含一个使用不同 PARTITION BY 值的多步骤查询,则这是不可能的。This is not possible if your job uses a non-partitioned output or has a multi-step query with different PARTITION BY values. 作业还应该至少有 6 个 SU,以便在作业运行时更改此设置。You job should also have at least 6 SUs in order to change this setting when the job is running. 在作业运行时,你可能只能从一组 SU 值中进行选择。You maybe restricted to choosing from a set of SU values when the job is running.

监视作业性能Monitor job performance

使用 Azure 门户时,可以跟踪作业的吞吐量:Using the Azure portal, you can track the throughput of a job:

Azure 流分析监视作业

计算工作负荷的预期吞吐量。Calculate the expected throughput of the workload. 如果吞吐量低于预期,则可调整输入分区和查询,并为作业添加 SU。If the throughput is less than expected, tune the input partition, tune the query, and add SUs to your job.

一个作业需要多少 SU?How many SUs are required for a job?

为特定作业选择所需的 SU 数量时,需要根据输入的分区配置以及在作业内定义的查询来决定。Choosing the number of required SUs for a particular job depends on the partition configuration for the inputs and the query that's defined within the job. 可以使用“缩放” 页设置正确的 SU 数量。The Scale page allows you to set the right number of SUs. 分配的 SU 数最好超过所需的数量。It is a best practice to allocate more SUs than needed. 流分析处理引擎会针对延迟和吞吐量进行优化,不过,代价是需要分配额外的内存。The Stream Analytics processing engine optimizes for latency and throughput at the cost of allocating additional memory.

通常情况下,最佳做法是一开始为未使用 PARTITION BY 的查询分配 6 个 SU。In general, the best practice is to start with 6 SUs for queries that don't use PARTITION BY. 然后,在传递了具有代表性的数据量并检查了 SU 利用率指标后,使用修改 SU 数量的试用和错误方法来确定最佳数量。Then determine the sweet spot by using a trial and error method in which you modify the number of SUs after you pass representative amounts of data and examine the SU% Utilization metric. 流分析作业所能使用的最大流单元数取决于为作业定义的查询中的步骤数,以及每一步中的分区数。The maximum number of streaming units that can be used by a Stream Analytics job depends on the number of steps in the query defined for the job and the number of partitions in each step. 可在此处了解更多有关限制的信息。You can learn more about the limits here.

有关如何选择正确的 SU 数量的详细信息,请参阅此页:扩展 Azure 流分析作业以增加吞吐量For more information about choosing the right number of SUs, see this page: Scale Azure Stream Analytics jobs to increase throughput

Note

选择特定作业所需的 SU 数目时,需根据输入的分区配置以及为作业定义的查询来决定。Choosing how many SUs are required for a particular job depends on the partition configuration for the inputs and on the query defined for the job. 可为作业选择的最大数目为 SU 配额。You can select up to your quota in SUs for a job. 默认情况下,每个 Azure 订阅的配额为最多 200 个 SU,这适用于特定区域的所有分析作业。By default, each Azure subscription has a quota of up to 200 SUs for all the analytics jobs in a specific region. 若要增加订阅的 SU 数,使其超过此配额,请联系 Azure 支持部门To increase SUs for your subscriptions beyond this quota, contact Azure Support. 每个作业的 SU 有效值以 1、3、6 开始,往上再按 6 递增。Valid values for SUs per job are 1, 3, 6, and up in increments of 6.

SU 利用率提高的因素Factors that increase SU% utilization 

时态(时间导向)的查询元素是流分析提供的有状态运算符的核心集。Temporal (time-oriented) query elements are the core set of stateful operators provided by Stream Analytics. 流分析通过管理内存消耗量、为复原创建检查点,并在服务升级期间恢复状态,代表用户在内部管理这些操作的状态。Stream Analytics manages the state of these operations internally on user’s behalf, by managing memory consumption, checkpointing for resiliency, and state recovery during service upgrades. 尽管流分析能够全面管理状态,但用户还是应该考虑一些最佳做法建议。Even though Stream Analytics fully manages the states, there are a number of best practice recommendations that users should consider.

请注意,具有复杂查询逻辑的作业即使在不连续接收输入事件时也可能具有较高的 SU% 利用率。Note that a job with complex query logic could have high SU% utilization even when it is not continuously receiving input events. 这可能发生在输入和输出事件突然激增之后。This can happen after a sudden spike in input and output events. 如果查询很复杂,作业可能会继续在内存中维护状态。The job might continue to maintain state in memory if the query is complex.

在回到预期水平之前,SU% 利用率可能会在短时间内突然降至 0。SU% utilization may suddenly drop to 0 for a short period before coming back to expected levels. 发生这种情况是由于暂时性错误或系统启动升级。This happens due to transient errors or system initiated upgrades.

时态元素中的有状态查询逻辑Stateful query logic in temporal elements

Azure 流分析作业的独有功能之一是执行有状态的处理,如开窗聚合、临时联接和临时分析函数。One of the unique capability of Azure Stream Analytics job is to perform stateful processing, such as windowed aggregates, temporal joins, and temporal analytic functions. 其中的每个运算符都会保存状态信息。Each of these operators keeps state information. 这些查询元素的最大窗口大小为 7 天。 The maximum window size for these query elements is seven days.

多个流分析查询元素中都出现了时态窗口的概念:The temporal window concept appears in several Stream Analytics query elements:

  1. 开窗聚合:翻转窗口、跳跃窗口和滑动窗口 GROUP BYWindowed aggregates: GROUP BY of Tumbling, Hopping, and Sliding windows

  2. 时态联接:JOIN with DATEDIFF 函数Temporal joins: JOIN with DATEDIFF function

  3. 时态分析函数:ISFIRST、LAST 和 LAG with LIMIT DURATIONTemporal analytic functions: ISFIRST, LAST, and LAG with LIMIT DURATION

以下因素影响流分析作业使用的内存(流单元指标部分):The following factors influence the memory used (part of streaming units metric) by Stream Analytics jobs:

开窗聚合Windowed aggregates

开窗聚合的消耗内存(状态大小)并不始终与窗口大小成正比。The memory consumed (state size) for a windowed aggregate is not always directly proportional to the window size. 消耗内存与数据基数或者每个时间窗口中的组数成正比。Instead, the memory consumed is proportional to the cardinality of the data, or the number of groups in each time window.

例如,在以下查询中,与 clusterid 关联的数字就是查询的基数。For example, in the following query, the number associated with clusterid is the cardinality of the query. 

SELECT count(*)
FROM input 
GROUP BY  clusterid, tumblingwindow (minutes, 5)

若要缓解前面查询中由高基数导致的任何问题,可以将事件发送到按 clusterid 分区的事件中心,并通过允许系统使用 PARTITION BY 分别处理每个输入分区来横向扩展查询,如以下示例所示:In order to mitigate any issues caused by high cardinality in the previous query, you can send events to Event Hub partitioned by clusterid, and scale out the query by allowing the system to process each input partition separately using PARTITION BY as shown in the example below:

SELECT count(*) 
FROM input PARTITION BY PartitionId
GROUP BY PartitionId, clusterid, tumblingwindow (minutes, 5)

将查询分区后,它会分散到多个节点中。Once the query is partitioned out, it is spread out over multiple nodes. 因此,可以通过减少依据运算符分组的基数来减少传入每个节点的 clusterid 值数。As a result, the number of clusterid values coming into each node is reduced thereby reducing the cardinality of the group by operator. 

事件中心分区应根据分组键进行分区,以避免减少步骤的需要。Event Hub partitions should be partitioned by the grouping key to avoid the need for a reduce step. 有关详细信息,请参阅事件中心概述For more information, see Event Hubs overview. 

时态联接Temporal joins

时态联接的消耗内存(状态大小)与联接的时态调整空间中的事件数量(即事件输入速率乘以调整空间大小)成正比。The memory consumed (state size) of a temporal join is proportional to the number of events in the temporal wiggle room of the join, which is event input rate multiplied by the wiggle room size. 换而言之,联接消耗的内存与 DateDiff 时间范围乘以平均事件速率的结果成正比。In other words, the memory consumed by joins is proportional to the DateDiff time range multiplied by average event rate.

联接中的不匹配事件数会影响查询的内存利用率。The number of unmatched events in the join affect the memory utilization for the query. 以下查询将查找产生点击量的广告印象:The following query is looking to find the ad impressions that generate clicks:

SELECT clicks.id
FROM clicks 
INNER JOIN impressions ON impressions.id = clicks.id AND DATEDIFF(hour, impressions, clicks) between 0 AND 10.

在本示例中,有可能显示了很多广告,但很少有人点击它们,并且需要保留该时间范围内的所有事件。In this example, it is possible that lots of ads are shown and few people click on it and it is required to keep all the events in the time window. 内存消耗量与时间范围大小和事件发生速率成比例。Memory consumed is proportional to the window size and event rate. 

若要修正此问题,请将事件发送到按联接键(在本示例中为 ID)分区的事件中心,并通过允许系统使用 PARTITION BY 单独处理每个输入分区来横向扩展查询,如下所示:To remediate this, send events to Event Hub partitioned by the join keys (ID in this case), and scale out the query by allowing the system to process each input partition separately using PARTITION BY as shown:

SELECT clicks.id
FROM clicks PARTITION BY PartitionId
INNER JOIN impressions PARTITION BY PartitionId 
ON impression.PartitionId = clicks.PartitionId AND impressions.id = clicks.id AND DATEDIFF(hour, impressions, clicks) between 0 AND 10 

将查询分区后,它会分散到多个节点中。Once the query is partitioned out, it is spread out over multiple nodes. 因此,可以通过减小保留在联接窗口中状态的大小来减少传入每个节点的事件数。As a result the number of events coming into each node is reduced thereby reducing the size of the state kept in the join window. 

时态分析函数Temporal analytic functions

时态分析函数的消耗内存(状态大小)与事件速率和持续时间的乘积成正比。The memory consumed (state size) of a temporal analytic function is proportional to the event rate multiply by the duration. 分析函数消耗的内存与窗口大小不成正比,而是与每个时间窗口中的分区计数成正比。 The memory consumed by analytic functions is not proportional to the window size, but rather partition count in each time window.

修正的方法类似于临时联接。The remediation is similar to temporal join. 你可以使用 PARTITION BY 来横向扩展查询。You can scale out the query using PARTITION BY. 

无序缓冲区Out of order buffer 

在“事件排序配置”窗格中,用户可以配置无序的缓冲区大小。User can configure the out of order buffer size in the Event Ordering configuration pane. 可以使用缓冲区来保留窗口持续时间内的输入,并对其进行重新排序。The buffer is used to hold inputs for the duration of the window, and reorder them. 缓冲区的大小与事件输入速率和无序窗口大小的乘积成正比。The size of the buffer is proportional to the event input rate multiply by the out of order window size. 默认窗口大小为 0。The default window size is 0. 

若要修正失序缓冲区溢出,请使用 PARTITION BY 横向扩展查询。To remediate overflow of the out of order buffer, scale out query using PARTITION BY. 将查询分区后,它会分散到多个节点中。Once the query is partitioned out, it is spread out over multiple nodes. 因此,可以通过减少每个重新排序缓冲区中的事件数来减少传入每个节点的事件数。As a result, the number of events coming into each node is reduced thereby reducing the number of events in each reorder buffer. 

输入分区计数Input partition count 

作业输入的每个输入分区都有一个缓冲区。Each input partition of a job input has a buffer. 输入分区数量越大,作业所消耗的资源越多。The larger number of input partitions, the more resource the job consumes. 对于每个流单元,Azure 流分析大致可以处理 1 MB/秒的输入。For each streaming unit, Azure Stream Analytics can process roughly 1 MB/s of input. 因此,可以通过将流分析流单元数与事件中心内的分区数进行匹配来进行优化。Therefore, you can optimize by matching the number of Stream Analytics streaming units with the number of partitions in your Event Hub.

通常,使用一个流单元配置的作业足以满足包含两个分区(事件中心至少包含两个分区)的事件中心。Typically, a job configured with one streaming unit is sufficient for an Event Hub with two partitions (which is the minimum for Event Hub). 如果事件中心具有更多分区,流分析作业将耗用更多资源,但不一定使用事件中心提供的额外吞吐量。If the Event Hub has more partitions, your Stream Analytics job consumes more resources, but not necessarily uses the extra throughput provided by Event Hub.

对于包含 6 个流单元的作业,可能需要事件中心的 4 个或 8 个分区。For a job with 6 streaming units, you may need 4 or 8 partitions from the Event Hub. 但是,请避免过多的不必要分区,否则可能会超出资源用量。However, avoid too many unnecessary partitions since that causes excessive resource usage. 例如,在包含 1 个流单元的流分析作业中,使用包含 16 个分区的事件中心或更大的事件中心。For example, an Event Hub with 16 partitions or larger in a Stream Analytics job that has 1 streaming unit.

引用数据Reference data 

ASA 中的引用数据会被加载到内存中,以便快速查找。Reference data in ASA are loaded into memory for fast lookup. 在当前的实现中,每个带有引用数据的联接操作都在内存中保留有一份引用数据,即使你多次使用相同的引用数据进行联接也是如此。With the current implementation, each join operation with reference data keeps a copy of the reference data in memory, even if you join with the same reference data multiple times. 对于使用 PARTITION BY 的查询,每个分区都有一份引用数据,因此,这些分区是完全分离的。For queries with PARTITION BY, each partition has a copy of the reference data, so the partitions are fully decoupled. 通过倍增效应,如果多次使用多个分区联接引用数据,内存使用率很快就会变得非常高。With the multiplier effect, memory usage can quickly get very high if you join with reference data multiple times with multiple partitions.  

使用 UDF 函数Use of UDF functions

当添加 UDF 函数时,Azure 流分析会将 JavaScript 运行时加载到内存中。When you add a UDF function, Azure Stream Analytics loads the JavaScript runtime into memory. 这将影响 SU%。This will affect the SU%.

后续步骤Next steps