容量策略Capacity policy

容量策略用于控制群集上数据管理操作的计算资源。A capacity policy is used for controlling the compute resources of data management operations on the cluster.

容量策略对象The capacity policy object

容量策略由以下内容组成:The capacity policy is made of:

引入容量Ingestion capacity

属性Property 类型Type 说明Description
ClusterMaximumConcurrentOperationsClusterMaximumConcurrentOperations longlong 群集中并发引入操作数的最大值。A maximal value for the number of concurrent ingestion operations in a cluster.
CoreUtilizationCoefficientCoreUtilizationCoefficient Doubledouble 计算引入容量时使用的核心百分比系数。A coefficient for the percentage of cores to use when calculating the ingestion capacity. 计算得出的结果将始终由 ClusterMaximumConcurrentOperations 进行标准化The calculation's result will always be normalized by ClusterMaximumConcurrentOperations
群集总引入容量(由 .show capacity 显示)的计算方式为:The cluster's total ingestion capacity, as shown by .show capacity, is calculated by:
Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

备注

在具有三个或以上节点的群集中,管理节点不参与引入操作。In clusters with three or more nodes, the admin node doesn't participate in ingestion operations. Number of nodes in cluster 减少 1 个。The Number of nodes in cluster is reduced by one.

盘区合并容量Extents merge capacity

属性Property 类型Type 说明Description
MinimumConcurrentOperationsPerNodeMinimumConcurrentOperationsPerNode longlong 单个节点上并发盘区合并/重新生成操作数的最小值。A minimal value for the number of concurrent extents merge/rebuild operations on a single node. 默认值为 1Default is 1
MaximumConcurrentOperationsPerNodeMaximumConcurrentOperationsPerNode longlong 单个节点上并发盘区合并/重新生成操作数的最大值。A maximal value for the number of concurrent extents merge/rebuild operations on a single node. 默认值为 3Default is 3

群集总盘区合并容量(由 .show capacity 显示)的计算方式为:The cluster's total extents merge capacity, as shown by .show capacity, is calculated by:

Number of nodes in cluster x Concurrent operations per nodeNumber of nodes in cluster x Concurrent operations per node

在 [MinimumConcurrentOperationsPerNode,MaximumConcurrentOperationsPerNode] 范围内,系统会自动调整 Concurrent operations per node 的有效值,前提是合并操作的成功率高于 90%。The effective value for Concurrent operations per node gets automatically adjusted by the system in the range [MinimumConcurrentOperationsPerNode,MaximumConcurrentOperationsPerNode], as long as the success rate of the merge operations is above 90%.

备注

在具有三个或以上节点的群集中,管理节点不参与合并操作。In clusters with three or more nodes, the admin node doesn't participate in doing merge operations. Number of nodes in cluster 减少 1 个。The Number of nodes in cluster is reduced by one.

盘区清除重新生成容量Extents purge rebuild capacity

属性Property 类型Type 说明Description
MaximumConcurrentOperationsPerNodeMaximumConcurrentOperationsPerNode longlong 单个节点上清除操作的并发重新生成盘区数的最大值A maximal value for the number of concurrent rebuild extents for purge operations on a single node

群集总盘区清除重新生成容量(由 .show capacity 显示)的计算方式为:The cluster's total extents purge rebuild capacity (as shown by .show capacity) is calculated by:

Number of nodes in cluster x MaximumConcurrentOperationsPerNodeNumber of nodes in cluster x MaximumConcurrentOperationsPerNode

备注

在具有三个或以上节点的群集中,管理节点不参与合并操作。In clusters with three or more nodes, the admin node doesn't participate in doing merge operations. Number of nodes in cluster 减少 1 个。The Number of nodes in cluster is reduced by one.

导出容量Export capacity

属性Property 类型Type 说明Description
ClusterMaximumConcurrentOperationsClusterMaximumConcurrentOperations longlong 群集中并发导出操作数的最大值。A maximal value for the number of concurrent export operations in a cluster.
CoreUtilizationCoefficientCoreUtilizationCoefficient Doubledouble 计算导出容量时使用的核心百分比系数。A coefficient for the percentage of cores to use when calculating the export capacity. 计算得出的结果将始终由 ClusterMaximumConcurrentOperations 进行标准化。The calculation's result will always be normalized by ClusterMaximumConcurrentOperations.

群集总导出容量(由 .show capacity 显示)的计算方式为:The cluster's total export capacity, as shown by .show capacity, is calculated by:

Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))Minimum(ClusterMaximumConcurrentOperations, Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

备注

在具有三个或以上节点的群集中,管理节点不参与导出操作。In clusters with three or more nodes, the admin node doesn't participate in export operations. Number of nodes in cluster 减少 1 个。The Number of nodes in cluster is reduced by one.

盘区分区容量Extents partition capacity

属性Property 类型Type 说明Description
ClusterMinimumConcurrentOperationsClusterMinimumConcurrentOperations longlong 群集中并发盘区分区操作数的最小值。A minimal value for the number of concurrent extents partition operations in a cluster. 默认值:1Default: 1
ClusterMaximumConcurrentOperationsClusterMaximumConcurrentOperations longlong 群集中并发盘区分区操作数的最大值。A maximal value for the number of concurrent extents partition operations in a cluster. 默认值:16Default: 16

群集总盘区分区容量(由 .show capacity 显示)。The cluster's total extents partition capacity (as shown by .show capacity).

在 [ClusterMinimumConcurrentOperations,ClusterMaximumConcurrentOperations] 范围内,系统会自动调整 Concurrent operations 的有效值,前提是分区操作的成功率高于 90%。The effective value for Concurrent operations is automatically adjusted by the system in the range [ClusterMinimumConcurrentOperations,ClusterMaximumConcurrentOperations], as long as the success rate of the partitioning operations is above 90%.

具体化视图容量策略Materialized views capacity policy

使用 alter cluster policy capacity 更改容量策略。Change the capacity policy using the alter cluster policy capacity. 此更改需要 AllDatabasesAdmin 权限。This change requires AllDatabasesAdmin permissions. 该策略可用于更改具体化视图的并发设置。The policy can be used to change concurrency settings for materialized views. 如果在一个群集上定义了多个具体化视图,并且群集无法跟上所有视图的具体化,则可能需要进行此更改。This change may be required when there's more than a single materialized view defined on a cluster, and the cluster can't keep up with the materialization of all views. 默认情况下,并发设置相对较低,以确保具体化不会影响群集的性能。By default, concurrency settings are relatively low to ensure that materialization doesn't impact cluster's performance.

警告

仅当群集资源良好(低 CPU、有可用内存)时,才应增加具体化视图容量策略。The materialized view capacity policy should only be increased if the cluster's resources are well (low CPU, available memory). 当资源有限时,增加这些值可能会导致资源耗尽,并对群集的性能产生不良影响。Increasing these values when resources are limited may result in resources exhaustion and will badly impact the cluster's performance.

具体化视图容量策略是群集容量策略的一部分,并且具有以下 JSON 表示形式:The materialized views capacity policy is part of the cluster's capacity policy, and has the following JSON representation:

{
   "MaterializedViewsCapacity": {
    "ClusterMaximumConcurrentOperations": 1,
    "ExtentsRebuildCapacity": {
      "ClusterMaximumConcurrentOperations": 50,
      "MaximumConcurrentOperationsPerNode": 5
    }
  }
}

属性Properties

propertiesProperty 说明Description
ClusterMaximumConcurrentOperations 群集可并发具体化的具体化视图的最大数目。The maximum number of materialized views that the cluster can materialize concurrently. 虽然(单个视图的)具体化本身可能会运行多个并发操作,但默认情况下该值为 1。This value is 1 by default, while materialization itself (of a single individual view) may run many concurrent operations. 如果在群集上定义了多个具体化视图,并且群集的资源处于良好状态,建议增加该值。If there's more than a single materialized view defined on the cluster, and if the cluster's resources are in good state, it's recommended to increase this value.
ExtentsRebuildCapacity 确定在具体化过程中为所有具体化视图执行的并发盘区重新生成操作的数目。Determines the number of concurrent extents rebuild operations, executed for all materialized views during the materialization process. 如果多个视图并发执行,则由于 ClusterMaximumConcurrentOperation 大于 1,它们将共享此属性定义的配额。If several views are executing concurrently, since ClusterMaximumConcurrentOperation is greater than 1, they'll share the quota defined by this property. 并发盘区重新生成操作的最大数目不会超过此值。The maximum number of concurrent extents rebuild operations won't exceed this value.

盘区重新生成Extents rebuild

若要详细了解盘区重新生成操作,请参阅具体化视图的工作原理To learn more about extents rebuild operations, see how materialized views work. 盘区重新生成的最大数目的计算方式如下:The maximum number of extents rebuild is calculated by:

Maximum(`ClusterMaximumConcurrentOperations`, `Number of nodes in cluster` * `MaximumConcurrentOperationsPerNode`)
  • 默认值为总共 50 个并发重新生成,每个节点最多 5 个。Default values are 50 total concurrency rebuilds and maximum 5 per node.
  • ExtentsRebuildCapacity 策略仅用作上限。The ExtentsRebuildCapacity policy serves as an upper limit only. 系统根据当前群集的条件(内存、CPU)和重新生成操作所需的资源量的估计值,动态确定所使用的实际值。The actual value used is dynamically determined by the system, based on current cluster's conditions (memory, CPU) and an estimation of the amount of resources required by the rebuild operation. 在实际情况下,并发会远远低于容量策略中指定的值。In practice, concurrency can be much lower than the value specified in capacity policy.
    • MaterializedViewExtentsRebuild 指标提供有关在每个具体化循环中重新生成盘区的数量的信息。The MaterializedViewExtentsRebuild metric provides information about how many extents were rebuilt in each materialization cycle. 有关详细信息,请参阅具体化视图监视For more information, see materialized views monitoring.

默认值Defaults

默认容量策略具有以下 JSON 表示形式:The default capacity policy has the following JSON representation:

{
  "IngestionCapacity": {
    "ClusterMaximumConcurrentOperations": 512,
    "CoreUtilizationCoefficient": 0.75
  },
  "ExtentsMergeCapacity": {
    "MinimumConcurrentOperationsPerNode": 1,
    "MaximumConcurrentOperationsPerNode": 3
  },
  "ExtentsPurgeRebuildCapacity": {
    "MaximumConcurrentOperationsPerNode": 1
  },
  "ExportCapacity": {
    "ClusterMaximumConcurrentOperations": 100,
    "CoreUtilizationCoefficient": 0.25
  },
  "ExtentsPartitionCapacity": {
    "ClusterMinimumConcurrentOperations": 1,
    "ClusterMaximumConcurrentOperations": 16
  }
}

控制命令Control commands

警告

更改容量策略之前,请咨询 Azure 数据资源管理器团队。Consult with the Azure Data Explorer team before altering a capacity policy.

限制Throttling

Kusto 限制以下用户启动命令的并发请求数:Kusto limits the number of concurrent requests for the following user-initiated commands:

  • 引入(包括此处列出的所有命令)Ingestions (includes all the commands that are listed here)
  • 清除Purges
    • 当前全局固定为每个群集一个。Global is currently fixed at one per cluster.
    • 清除重新生成容量在内部用于确定使用清除命令期间的并发重新生成操作数。The purge rebuild capacity is used internally to determine the number of concurrent rebuild operations during purge commands. 清除命令不会因此过程而被阻止/限制,但将以更快或更慢的速度运行,具体取决于清除重新生成容量。Purge commands won't be blocked/throttled because of this process, but will work faster or slower depending on the purge rebuild capacity.
  • 导出Exports

当群集检测到某个操作具有超过允许数量的并发操作时,它将使用 429“受限制”HTTP 代码进行响应。When the cluster detects that an operation has exceeded the permitted concurrent operation, it will respond with a 429, "Throttled", HTTP code. 回退后重试操作。Retry the operation after some backoff.