Extent tags retention policy

Applies to: ✅ Azure Data Explorer

The extent tags retention policy controls the mechanism that automatically removes extent tags from tables, based on the age of the extents.

It's recommended to remove any tags that are no longer helpful, or were used temporarily as part of an ingestion pipeline, and may limit the system from reaching optimal performance. For example: old drop-by: tags, which prevent merging extents together.

The policy can be set at the table-level, or at the database-level. A database-level policy applies to all tables in the database that don't override the policy.

Note

The deletion time is imprecise. The system guarantees that tags won't be deleted before the limit is exceeded, but deletion isn't immediate following that point.

The policy object

The extent tags retention policy is an array of policy objects. Each object includes the following properties:

Property name Type Description Example
TagPrefix string The prefix of the tags to be automatically deleted, once RetentionPeriod is exceeded. The prefix must include a colon (:) as its final character, and may only include one colon. drop-by:, ingest-by:, custom_prefix:
RetentionPeriod timespan The duration for which it's guaranteed that the tags aren't dropped. This period is measured starting from the extent's creation time. 1.00:00:00

Example

The following policy will have any drop-by: tags older than three days and any ingest-by: tags older than two hours automatically dropped:

[
    {
        "TagPrefix": "drop-by:",
        "RetentionPeriod": "3.00:00:00"
    },
    {
        "TagPrefix": "ingest-by:",
        "RetentionPeriod": "02:00:00"
    }
]

Defaults

By default, when the policy isn't defined, extent tags of any kind are retained as long as the extent isn't dropped.

Management commands

The following management commands can be used to manage the extent tags retention policy: