分区策略Partitioning policy

分区策略定义了对于特定表,是否以及如何对区(数据分片)进行分区。The partitioning policy defines if and how extents (data shards) should be partitioned for a specific table.

该策略的主要目的是提高可以缩小数据集的查询的性能,例如,基于分区列进行筛选的查询、使用聚合的查询,或基于高基数字符串列进行联接的查询。The main purpose of the policy is to improve performance of queries that narrow the data set; For example, queries that filter on partitioned columns, use an aggregate, or join on a high cardinality string column. 该策略还可能会导致更好的数据压缩。The policy may also result in better data compression.

注意

对于定义了分区策略的表数量,没有设置硬编码限制。There are no hard-coded limits set on the number of tables with the partitioning policy defined. 但是,每个附加表都会增加在群集节点上运行的后台数据分区过程的开销。However, every additional table adds overhead to the background data partitioning process that runs on the cluster's nodes. 添加表可能会导致使用更多的群集资源。Adding tables may result in more cluster resources being used. 有关详细信息,请参阅监视容量For more information, see monitoring and capacity.

分区键Partition keys

支持以下类型的分区键。The following kinds of partition keys are supported.

种类Kind 列类型Column Type 分区属性Partition properties 分区值Partition value
哈希Hash string Function, MaxPartitionCount, SeedFunction, MaxPartitionCount, Seed Function(ColumnName, MaxPartitionCount, Seed)Function(ColumnName, MaxPartitionCount, Seed)
统一范围Uniform range datetime RangeSize, ReferenceRangeSize, Reference bin_at(ColumnName, RangeSize, Reference)bin_at(ColumnName, RangeSize, Reference)

哈希分区键Hash partition key

备注

仅在以下实例中对表中的 string 类型列应用哈希分区键:Apply a hash partition key on a string-type column in a table only in the following instances:

  • 大多数查询都使用相等筛选器 (==, in())。If the majority of queries use equality filters (==, in()).
  • 大多数查询在大维度(基数为 10M 或更高)的特定 string 类型列(例如 application_IDtenant_IDuser_ID)上聚合/联接。The majority of queries aggregate/join on a specific string-typed column of large-dimension (cardinality of 10M or higher) such as an application_ID, a tenant_ID, or a user_ID.
  • 哈希取模函数用于对数据进行分区。A hash-modulo function is used to partition the data.
  • 同类(已分区)盘区中的数据按哈希分区键排序。Data in homogeneous (partitioned) extents is ordered by the hash partition key.
    • 如果表上定义了哈希分区键,则不需要在行顺序策略中包含哈希分区键。You don't need to include the hash partition key in the row order policy, if one is defined on the table.
  • 如果查询使用无序策略,并且按照此策略,joinsummarizemake-series 中使用的 shuffle key 是表的哈希分区键,那么查询的性能将会更好,因为需要跨群集节点移动的数据量减少。Queries that use the shuffle strategy, and in which the shuffle key used in join, summarize or make-series is the table's hash partition key, are expected to perform better because the amount of data required to move across cluster nodes is reduced.

分区属性Partition properties

propertiesProperty 说明Description 支持的值Supported value(s) 建议的值Recommended value
Function 要使用的哈希取模函数的名称。The name of a hash-modulo function to use. XxHash64
MaxPartitionCount 每个时间段要创建的最大分区数(哈希取模函数的取模参数)。The maximum number of partitions to create (the modulo argument to the hash-modulo function) per time period. (1,2048] 范围内。In the range (1,2048].
大于群集中节点数的 5 倍,小于列的基数。Larger than five times the number of nodes in the cluster, and smaller than the cardinality of the column.
值越大,群集节点上数据分区过程的开销就越大,每个时间段的区数量也就越大。Higher values lead to greater overhead of the data partitioning process on the cluster's nodes, and a higher number of extents for each time period. 对于少于 50 个节点的群集,请从 256 开始。For clusters with fewer than 50 nodes, start with 256. 基于这些注意事项,或者基于查询性能的优势与数据引入后分区的开销,调整该值。Adjust the value based on these considerations, or based on the benefit in query performance vs. the overhead of partitioning the data post-ingestion.
Seed 用于将哈希值随机化。Use for randomizing the hash value. 正整数。A positive integer. 1,这也是默认值。1, which is also the default value.
PartitionAssignmentMode 用于将分区分配给群集中节点的模式。The mode used for assigning partitions to nodes in the cluster. Default:属于同一分区的所有同类(已分区)盘区都分配给同一个节点。Default: All homogeneous (partitioned) extents that belong to the same partition are assigned to the same node.
Uniform:将忽略区的分区值。Uniform: An extents' partition values are disregarded. 区将统一分配给群集的节点。Extents are assigned uniformly to the cluster's nodes.
如果查询不根据哈希分区键联接或聚合,请使用 UniformIf queries don't join or aggregate on the hash partition key, use Uniform. 否则使用 DefaultOtherwise, use Default.

哈希分区键示例Hash partition key example

名为 tenant_idstring 类型列的哈希分区键。A hash partition key over a string-typed column named tenant_id. 它使用 XxHash64 哈希函数,其中 MaxPartitionCount256,默认的 Seed1It uses the XxHash64 hash function, with a MaxPartitionCount of 256, and the default Seed of 1.

{
  "ColumnName": "tenant_id",
  "Kind": "Hash",
  "Properties": {
    "Function": "XxHash64",
    "MaxPartitionCount": 256,
    "Seed": 1,
    "PartitionAssignmentMode": "Default"
  }
}

统一范围日期/时间分区键Uniform range datetime partition key

备注

只有当引入到表中的数据不太可能根据此列排序时,才对表中的 datetime 类型的列应用统一范围的日期/时间分区键。Only apply a uniform range datetime partition key on a datetime-typed column in a table when data ingested into the table is unlikely to be ordered according to this column.

在这些情况下,你可以在区之间重新组织数据,以便每个区都包括有限时间范围的记录。In these cases, you can reshuffle the data between extents so that each extent includes records from a limited time range. 此过程会使 datetime 列上的筛选器在查询时更有效。This process results in filters on the datetime column being more effective at query time.

使用的分区函数是 bin_at(),该函数不可自定义。The partition function used is bin_at() and isn't customizable.

分区属性Partition properties

propertiesProperty 说明Description 建议的值Recommended value
RangeSize 一个 timespan 标量常量,指示每个日期/时间分区的大小。A timespan scalar constant that indicates the size of each datetime partition. 从值 1.00:00:00(一天)开始。Start with the value 1.00:00:00 (one day). 不要设置更短的值,因为这可能会导致表中有大量无法合并的小区。Don't set a shorter value, because it may result in the table having a large number of small extents that can't be merged.
Reference 一个 datetime 标量常量,指示一个固定的时间点,根据该时间点调整日期/时间分区。A datetime scalar constant that indicates a fixed point in time, according to which datetime partitions are aligned. 1970-01-01 00:00:00 开始。Start with 1970-01-01 00:00:00. 如果存在日期/时间分区键具有 null 值的记录,则将其分区值设置为 ReferenceIf there are records in which the datetime partition key has null values, their partition value is set to the value of Reference.

统一范围日期/时间分区示例Uniform range datetime partition example

代码片段在名为 timestampdatetime 类型列上显示了统一日期/时间范围分区键。The code snippet shows a uniform datetime range partition key over a datetime typed column named timestamp. 它使用 datetime(1970-01-01) 作为参考点,每个分区的大小为 1dIt uses datetime(1970-01-01) as its reference point, with a size of 1d for each partition.

{
  "ColumnName": "timestamp",
  "Kind": "UniformRange",
  "Properties": {
    "Reference": "1970-01-01T00:00:00",
    "RangeSize": "1.00:00:00"
  }
}

策略对象The policy object

表的数据分区策略默认为 null,在这种情况下,将不会对表中的数据进行分区。By default, a table's data partitioning policy is null, in which case data in the table won't be partitioned.

数据分区策略具有以下主要属性:The data partitioning policy has the following main properties:

  • PartitionKeysPartitionKeys :

  • EffectiveDateTimeEffectiveDateTime :

    • 策略生效的 UTC 日期/时间。The UTC datetime from which the policy is effective.
    • 此属性是可选的。This property is optional. 如果未指定它,则策略将对应用策略后引入的数据生效。If it isn't specified, the policy will take effect on data ingested after the policy was applied.
    • 分区过程会忽略由于保留而可能丢弃的任何非同类(非分区)区。Any non-homogeneous (non-partitioned) extents that may be dropped because of retention are ignored by the partitioning process. 这些区会被忽略,因为它们的创建时间早于表的有效软删除周期的 90%。The extents are ignored because their creation time precedes 90% of the table's effective soft-delete period.

      备注

      你可以设置过去的一个日期/时间值,并对已引入的数据分区。You can set a datetime value in the past and partition already-ingested data. 但是,这种做法可能会显著增加分区过程中使用的资源。However, this practice may significantly increase resources used in the partitioning process.

数据分区示例Data partitioning example

具有两个分区键的数据分区策略对象。Data partitioning policy object with two partition keys.

  1. 名为 tenant_idstring 类型列的哈希分区键。A hash partition key over a string-typed column named tenant_id.
    • 它使用 XxHash64 哈希函数,其中 MaxPartitionCount 为 256,默认的 Seed1It uses the XxHash64 hash function, with a MaxPartitionCount of 256, and the default Seed of 1.
  2. 名为 timestampdatetime 类型列的统一日期/时间范围分区键。A uniform datetime range partition key over a datetime type column named timestamp.
    • 它使用 datetime(1970-01-01) 作为参考点,每个分区的大小为 1dIt uses datetime(1970-01-01) as its reference point, with a size of 1d for each partition.
{
  "PartitionKeys": [
    {
      "ColumnName": "tenant_id",
      "Kind": "Hash",
      "Properties": {
        "Function": "XxHash64",
        "MaxPartitionCount": 256,
        "Seed": 1,
        "PartitionAssignmentMode": "Default"
      }
    },
    {
      "ColumnName": "timestamp",
      "Kind": "UniformRange",
      "Properties": {
        "Reference": "1970-01-01T00:00:00",
        "RangeSize": "1.00:00:00"
      }
    }
  ]
}

其他属性Additional properties

可以将以下属性定义为策略的一部分。The following properties can be defined as part of the policy. 这些属性是可选的,建议不要更改它们。These properties are optional and we recommend not changing them.

propertiesProperty 说明Description 建议的值Recommended value 默认值Default value
MinRowCountPerOperationMinRowCountPerOperation 单个数据分区操作的源盘区行数总和的最小目标值。Minimum target for the sum of row count of the source extents of a single data partitioning operation. 0
MaxRowCountPerOperationMaxRowCountPerOperation 单个数据分区操作的源盘区行数总和的最大目标值。Maximum target for the sum of the row count of the source extents of a single data partitioning operation. 如果看到单个分区操作消耗了大量的内存或 CPU,请设置一个小于 5M 的值。Set a value lower than 5M if you see that the partitioning operations consume a large amount of memory or CPU per operation. 有关详细信息,请参阅监视For more information, see monitoring. 0,默认目标是 5,000,000 条记录。0, with a default target of 5,000,000 records.

数据分区过程The data partitioning process

  • 数据分区在群集中作为引入后的后台进程运行。Data partitioning runs as a post-ingestion background process in the cluster.
    • 持续引入的表始终具有尚未分区的数据的“尾部”(非同类盘区)。A table that is continuously ingested into is expected to always have a "tail" of data that is yet to be partitioned (non-homogeneous extents).
  • 无论策略中 EffectiveDateTime 属性的值如何,数据分区只在热盘区上运行。Data partitioning runs only on hot extents, regardless of the value of the EffectiveDateTime property in the policy.
    • 如果需要对冷区进行分区,则需要临时调整缓存策略If partitioning cold extents is required, you need to temporarily adjust the caching policy.

监视分区Monitor partitioning

使用 .show diagnostics 命令监视群集中分区的进度或状态。Use the .show diagnostics command to monitor the progress or state of partitioning in a cluster.

.show diagnostics
| project MinPartitioningPercentageInSingleTable, TableWithMinPartitioningPercentage

输出包括:The output includes:

  • MinPartitioningPercentageInSingleTable:群集中具有数据分区策略的所有表中分区数据的最小百分比。MinPartitioningPercentageInSingleTable: The minimal percentage of partitioned data across all tables that have a data partitioning policy in the cluster.
    • 如果此百分比持续低于 90%,则评估群集的分区容量If this percentage remains constantly under 90%, then evaluate the cluster's partitioning capacity.
  • TableWithMinPartitioningPercentage:上面显示了分区百分比的表的完全限定名。TableWithMinPartitioningPercentage: The fully qualified name of the table whose partitioning percentage is shown above.

使用 .show 命令监视分区命令及其资源使用情况。Use .show commands to monitor the partitioning commands and their resource use. 例如:For example:

.show commands 
| where StartedOn > ago(1d)
| where CommandType == "ExtentsPartition"
| parse Text with ".partition async table " TableName " extents" *
| summarize count(), sum(TotalCpu), avg(tolong(ResourcesUtilization.MemoryPeak)) by TableName, bin(StartedOn, 15m)
| render timechart with(ysplit = panels)

分区容量Partition capacity

  • 数据分区过程会导致创建更多的盘区。The data partitioning process results in the creation of more extents. 群集可以逐渐增加其区合并容量,以便合并区的过程可以赶上进度。The cluster may gradually increase its extents merge capacity, so that the process of merging extents can keep up.
  • 如果有较高的引入吞吐量,或者有足够多的表定义了分区策略,则群集可能会逐渐增加其盘区分区容量,以便分区盘区的过程可以保持。If there's a high ingestion throughput, or a large enough number of tables that have a partitioning policy defined, then the cluster may gradually increase its Extents partition capacity, so that the process of partitioning extents can keep up.
  • 为了避免消耗过多资源,对这些动态增加进行了限制。To avoid consuming too many resources, these dynamic increases are capped. 如果它们全部耗尽,则可能需要逐渐线性地增加,以超过上限。You may be required to gradually and linearly increase them beyond the cap, if they're used up entirely.
    • 如果增加容量会显著增加群集资源的使用,则可以手动或通过启用自动缩放来横向/纵向扩展群集If increasing the capacities causes a significant increase in the use of the cluster's resources, you can scale the cluster up/out, either manually, or by enabling autoscale.

已分区列中的离群值Outliers in partitioned columns

  • 以下情况可能会导致群集节点上的数据分布不均衡,并降低查询性能:The following situations can contribute to imbalanced distribution of data across the cluster's nodes, and degrade query performance:
    • 如果某个哈希分区键包含比其他值要普遍得多的值,例如,空字符串或泛型值(例如 nullN/A)。If a hash partition key includes values that are much more prevalent than others, for example, an empty string, or a generic value (such as null or N/A).
    • 这些值表示在数据集中更为普遍的实体(例如 tenant_id)。The values represent an entity (such as tenant_id) that is more prevalent in the data set.
  • 如果统一范围日期/时间分区键包含的大部分值与列中的大多数值“相差甚远”,则会增加数据分区过程的开销,并可能会导致群集需要跟踪多个小区。If a uniform range datetime partition key has a large enough percentage of values that are "far" from the majority of the values in the column, the overhead of the data partitioning process is increased and may lead to many small extents that the cluster will need to keep track of. 这种情况的一个示例是来自遥远的过去或未来的日期/时间值。An example of such a situation is datetime values from the distant past or future.

在这两种情况下,要么“修复”数据,要么在数据引入之前或引入时筛选掉数据中不相关的记录,以减少群集上数据分区的开销。In both of these cases, either "fix" the data, or filter out any irrelevant records in the data before or at ingestion time, to reduce the overhead of the data partitioning on the cluster. 例如,使用更新策略For example, use an update policy.

后续步骤Next steps

使用分区策略控制命令管理表的数据分区策略。Use the partitioning policy control commands to manage data partitioning policies for tables.