.ingest-from-storage-queued command (preview)

Applies to: ✅ Azure Data Explorer

The .ingest-from-storage-queued command is used with the .list blobs command to queue blobs for ingestion into a table. This command supports bulk ingestion of an entire storage container, a specific folder within a container, or all blobs that match a given prefix and suffix.

Note

This ingestion method is intended for exploration and prototyping. Don't use it in production or high-volume scenarios. For more information about ingestion options, see Data ingestion overview.

Note

Queued ingestion commands are run on the data ingestion URI endpoint https://ingest-<YourClusterName><Region>.kusto.chinacloudapi.cn.

Permissions

You must have at least Table Ingestor permissions to run this command.

Syntax

.ingest-from-storage-queued into table [database(DatabaseName).]TableName[EnableTracking=EnableTracking] [SkipBatching=SkipBatching] [CompressionFactor=CompressionFactor] [with (IngestionPropertyName = IngestionPropertyValue [, ...])] <| IngestionSource

Learn more about syntax conventions.

Parameters

Name Type Required Description
DatabaseName string The name of the database into which to ingest data. If no database name is provided, the request's context database is used.
TableName string ✔️ The name of the table into which to ingest data.
EnableTracking boolean Determines whether to track the blob ingestion. For more information, see .show queued ingestion operations command. The default is false.
SkipBatching boolean If set to true, the blobs are ingested individually rather than batched together with other blobs. The default value is false.
CompressionFactor real The compression factor (ratio) between the original size and the compressed size of blobs. Compression factor is used to estimate the original size of the data for batching purposes, when blobs are provided in a compressed format.
IngestionPropertyName, IngestionPropertyValue string Optional ingestion properties. For more information about ingestion properties, see Data ingestion properties.
IngestionSource table ✔️ The ingestion source. The source is a list of blobs returned using the .list blobs command.

Note

The .list blobs command can be used with the .ingest-from-storage-queued command to return the blobs you want to ingest. For detailed information about the command and a full list of its parameters, see .list blobs command.

Returns

The result of the command is a table with one row and one column.

Name Type Description
IngestionOperationId string A unique ID used to track the set of blobs, whether or not tracking is enabled.
ClientRequestId string The client request ID of the command.
OperationInfo string Displays the command to run to retrieve the current status of the operation.

Note

This command doesn't modify the schema of the target table. If necessary, the data is converted to fit the table's schema during ingestion. Extra columns are ignored and missing columns are treated as null values.

Example

The example in this section shows how to use the syntax to help you get started.

Ingest all blobs in a folder

The following example queues all blobs within a folder for ingestion using the cluster's system managed identity.

.ingest-from-storage-queued into table database('LogDb').RawLogs
EnableTracking=true
with (
  format='csv',
  ingestionMappingReference='LogMapping',
  ignoreFirstRecord=false  
)
<|
  .list blobs (
      "https://mystorageaccount.blob.core.chinacloudapi.cn/datasets/myfolder;managed_identity=system"
  )