盘区(数据分片)Extents (data shards)

概述Overview

Kusto 可以支持含有大量记录(行)和大量数据的表。Kusto is built to support tables with a huge number of records (rows) and large amounts of data. 为了处理此类大型表,每个表的数据被划分为较小的“块”,称为数据分片或盘区(这两个术语是同义词) 。To handle such large tables, each table's data is divided into smaller "tablets" called data shards or extents (the two terms are synonymous). 表的所有盘区的并集保存表的数据。The union of all the table's extents holds the table's data. 单个盘区将保持小于单个节点的容量,并且盘区分散在群集的节点上,从而实现横向扩展。Individual extents are kept smaller than a single node's capacity, and the extents are spread over the cluster's nodes, achieving scale-out.

盘区就像一种小型表。An extent is a like a type of mini-table. 它包含数据和元数据,以及其创建时间和与其数据关联的可选标记等信息。It contains data and metadata and information such as its creation time and optional tags that are associated with its data. 此外,盘区通常保存能让 Kusto 高效查询数据的信息。Additionally, the extent usually holds information that lets Kusto query the data efficiently. 例如盘区中每列数据的索引和编码字典(如果已对列数据进行编码)。For example, an index for each column of data in the extent, and an encoding dictionary, if column data is encoded. 因此,表的数据是表的盘区中所有数据的并集。As a result, the table's data is the union of all the data in the table's extents.

盘区是不可变的,并且永远不能修改。Extents are immutable and can never be modified. 只能查询它、将它重新分配给另一个节点或从表中删除它。It may only be queried, reassigned to a different node, or dropped out of the table. 可通过以下方式进行数据修改:创建一个或多个新盘区,并将旧盘区与新盘区进行事务交换。Data modification happens by creating one or more new extents and transactionally swapping old extents with new ones.

盘区保存物理上按列排列的记录的集合。Extents hold a collection of records that are physically arranged in columns. 此方法称为分栏式存储。This technique is called columnar store. 利用它可以高效地对数据进行编码和压缩,因为同一列中的不同值经常彼此类似。It enables efficient encoding and compression of the data, because different values from the same column often "resemble" each other. 它还支持更高效地查询大量数据,因为只需加载查询所使用的列。It also makes querying large spans of data more efficient, because only the columns used by the query need to be loaded. 在内部,盘区内的每个数据列将细分为若干个段,段分为若干个块。Internally, each column of data in the extent is subdivided into segments, and the segments into blocks. 这种分隔对查询而言并不能观察到,这可使 Kusto 优化列压缩和索引编制。This division isn't observable to queries, and lets Kusto optimize column compression and indexing.

为了保持查询效率,较小的盘区将合并到较大的盘区中。To maintain query efficiency, smaller extents are merged into larger extents. 根据配置的合并策略分片策略,合并作为后台过程自动完成。The merge is done automatically, as a background process, according to the configured merge policy and sharding policy. 合并盘区可减少跟踪大量盘区带来的管理开销。更重要的是,它可使 Kusto 优化其索引并改进压缩。Merging extents reduces the management overhead of having a large number of extents to track. More importantly, it allows Kusto to optimize its indexes and improve compression.

某个盘区达到特定限制(如大小)后,盘区合并将停止,因为超过特定点后,合并非但不会提高效率,反而会降低效率。Extent merging stops once an extent reaches certain limits, such as size, since beyond a certain point, merging reduces rather than increases efficiency.

在表上定义数据分区策略时,盘区在创建后将经历另一个后台过程(后期引入)。When a Data partitioning policy is defined on a table, extents go through another background process after they're created (post-ingestion). 此过程重新引入源盘区中的数据,并创建“同类”盘区,其中作为表的“分区键”的列的值全部属于同一分区。This process reingests the data from the source extents and creates homogeneous extents, in which the values of the column that is the table's partition key all belong to the same partition. 如果策略包括“哈希分区键”,属于同一分区的所有同类盘区将被分配到群集中的同一数据节点。If the policy includes a hash partition key , all homogeneous extents that belong to the same partition will be assigned to the same data node in the cluster.

备注

盘区级别的操作(如合并或更改盘区标记)不会修改现有盘区。Extent-level operations, such as merging or altering extent tags, don't modify existing extents. 而是基于现有源盘区在这些操作中创建新的盘区。Instead, new extents are created in these operations, based on existing source extents. 新的盘区将在单个事务中替换其前身。The new extents replace their forefathers in a single transaction.

常见的盘区生命周期:The common extent lifecycle is:

  1. 盘区由“引入”操作创建。The extent is created by an ingestion operation.
  2. 盘区与其他盘区合并。The extent is merged with other extents. 如果要合并的盘区较小,Kusto 实际上会对其执行引入过程,称为“重新生成”。When the extents being merged are small, Kusto actually carries out an ingestion process on them, called rebuild. 在盘区达到一定大小后,只对索引执行合并。Once extents reach a certain size, merging is done only for indexes. 不会修改存储中盘区的数据项目。The extents' data artifacts in storage aren't modified.
  3. 由于保留策略的原因,合并的盘区(该盘区的世系可能追溯到其他合并的盘区)最终会被删除。The merged extent (possibly one that tracks its lineage to other merged extents, and so on) is eventually dropped because of a retention policy. 删除盘区时,根据时间(旧的盘区 x 小时/天),在计算中使用合并盘区中最新盘区的创建日期。When extents are dropped, based on time (older x hours / days), then the creation date of the newest extent inside the merged one is used in the calculation.

盘区创建时间Extent Creation time

对于每个盘区而言,创建时间是关键信息之一。One of the more important pieces of information for each extent is its creation time. 此时间用于:This time is used for:

  1. 保留 - 较早创建的盘区将较早被删除。Retention - Extents that were created earlier will be dropped earlier.
  2. 缓存 - 最近创建的盘区将保存在热缓存Caching - Extents that were created recently will be kept in hot cache)
  3. 采样 - 使用查询操作(例如 take)时,建议使用最近的盘区Sampling - Recent extents are favored, when using query operations such as take

事实上,Kusto 会跟踪每个区的两个 datetime 值:MinCreatedOnMaxCreatedOnIn fact, Kusto tracks two datetime values per extent: MinCreatedOn and MaxCreatedOn. 最初,这两个值是相同的。Initially, the two values are the same. 如果该盘区与其他盘区合并,则新值将与合并盘区的原始最小值和最大值保持一致。When the extent is merged with other extents, the new values are according to the original minimum and maximum values of the merged extents.

通常,会根据引入盘区中的数据的时间来设置盘区的创建时间。Normally, an extent's creation time is set according to the time in which the data in the extent is ingested. 客户端可以选择覆盖该盘区的创建时间,方法是在 ingestion 属性中提供备用创建时间。Clients can optionally overwrite the extent's creation time, by providing an alternative creation time in the ingestion properties. 如果出于保留目的,客户端想要重新引入数据并且不希望它看起来像“迟到”的数据,则覆盖很有用。Overwriting is useful, for example for retention purposes, if the client wants to reingest data and doesn't want it to appear as if it arrived late.

盘区标记Extent Tagging

Kusto 支持将多个可选盘区标记附加到盘区,作为其元数据的一部分。Kusto supports attaching multiple optional extent tags to the extent, as part of its metadata. 盘区标记(或只是标记)是与盘区关联的字符串。An extent tag (or simply tag ), is a string that is associated with the extent. 可以使用 .show extents 命令查看与某个盘区关联的标记,并使用 extent-tags() 函数查看与某个盘区内的记录相关联的标记。You can use the .show extents commands to see the tags associated with an extent, and the extent-tags() function to see the tags associated with records in an extent. 盘区标记可用于高效地描述盘区内的所有数据所共有的属性。Extent tags can be used to efficiently describe properties that are common to all of the data in the extent. 例如,可以在引入期间添加盘区标记,指示引入数据的源,并在稍后使用该标记。For example, you could add an extent tag during ingestion, that indicates the source of the ingested data, and use that tag later. 因为盘区描述数据,因此当两个或多个盘区合并时,它们的相关标记也将合并。Since the extents describe data, when two or more merge, their associated tags also merge. 生成的盘区的标记将是这些合并盘区的所有标记的并集。The resulting extent's tags will be the union of all the tags of those merged extents.

Kusto 向所有其值具有格式“前缀 后缀” 的盘区标记分配特殊含义,其中“前缀”是以下项之一:Kusto assigns a special meaning to all extent tags whose value has the format prefix suffix , where prefix is one of:

  • drop-by:
  • ingest-by:

'drop-by:' 盘区标记'drop-by:' extent tags

drop-by: 前缀开头的标记可用于控制要合并的其他盘区。Tags that start with a drop-by: prefix can be used to control which other extents to merge with. 具有给定 drop-by: 标记的盘区可以合并在一起,但不会与其他盘区合并。Extents that have a given drop-by: tag can be merged together, but they won't be merged with other extents. 然后,可以根据其 drop-by: 标记发出命令以删除盘区。You can then issue a command to drop extents according to their drop-by: tag.

例如:For example:

.ingest ... with @'{"tags":"[\"drop-by:2016-02-17\"]"}'

.drop extents <| .show table MyTable extents where tags has "drop-by:2016-02-17" 

性能说明Performance notes

  • 不要滥用 drop-by 标记。Don't overuse drop-by tags. 以上述方式删除数据仅适用于很少发生的事件。Dropping data in the manner mentioned above is meant for rarely occurring events. 它不用于替换记录级数据,并且要求以这种方式标记的数据很庞大。It isn't for replacing record-level data, and it relies on the fact that the data tagged in this manner is bulky. 如果尝试为每个记录或少量记录提供不同的标记,可能会对性能产生严重影响。Attempting to give a different tag for each record, or small number of records, might result in a severe impact on performance.
  • 如果引入数据后,在一段时间内不需要 drop-by 标记,建议删除标记If drop-by tags aren't needed for a period of time after data is ingested, we recommend that you drop the tags.

'ingest-by:' 盘区标记'ingest-by:' extent tags

ingest-by: 前缀开头的标记可用于确保数据仅引入一次。Tags that start with an ingest-by: prefix can be used to ensure that data is only ingested once. 如果已存在具有此特定 ingest-by: 标记的盘区,则可以发出 ingestIfNotExists 属性命令阻止数据引入。You can issue an ingestIfNotExists property command that prevents the data from being ingested if there already exists an extent with this specific ingest-by: tag. tagsingestIfNotExists 的值都是字符串数组,序列化为 JSON。The values for both tags and ingestIfNotExists are arrays of strings, serialized as JSON.

以下示例仅引入数据一次。The following example ingests data only once. 第二个和第三个命令不执行任何操作:The 2nd and 3rd commands do nothing:

.ingest ... with (tags = '["ingest-by:2016-02-17"]')

.ingest ... with (ingestIfNotExists = '["2016-02-17"]')

.ingest ... with (ingestIfNotExists = '["2016-02-17"]', tags = '["ingest-by:2016-02-17"]')

备注

通常,引入命令可能同时包含 ingest-by: 标记和 ingestIfNotExists 属性,并设置为相同的值,如上述第三个命令中所示。Generally, an ingest command is likely to include both an ingest-by: tag and an ingestIfNotExists property, set to the same value, as shown in the 3rd command above.

性能说明Performance notes

  • 不建议过度使用 ingest-by 标记。Overusing ingest-by tags isn't recommended. 如果已知馈送 Kusto 的管道有数据重复,我们建议在将数据引入 Kusto 之前,尽可能解决这些重复项。If the pipeline feeding Kusto is known to have data duplications, we recommend that you solve these duplications as much as possible, before ingesting the data into Kusto. 此外,仅当引入到 Kusto 的部分可能会自行引入重复项时(例如,有一种重试机制可能会与正在进行的引入调用重叠),才在 Kusto 中使用 ingest-by 标记。Also, use ingest-by tags in Kusto only when the part that ingests to Kusto might introduce duplicates by itself (for example, there's a retry mechanism that can overlap with already-in-progress ingestion calls). 尝试为每个引入调用设置唯一的 ingest-by 标记可能会对性能产生严重影响。Attempting to set a unique ingest-by tag for each ingestion call might result in a severe impact on performance.
  • 如果在数据引入后的某个时间段内不需要此类标记,则建议删除区标记If such tags aren't required for some period of time after the data is ingested, we recommend that you drop extent tags.