Data formats supported for ingestion

Applies to: ✅ Azure Data Explorer

Data ingestion adds data to a table and makes it available for query. For all ingestion methods, other than ingest-from-query, the data must be in one of the supported formats. The following table lists and describes the formats that are supported for data ingestion.

Note

Before you ingest data, make sure that your data is properly formatted and defines the expected fields. We recommend using your preferred validator to confirm the format is valid. For example, you may find the following validators useful to check CSV or JSON files:

To learn why ingestion might fail, see Ingestion failures.

and Ingestion error codes in Azure Data Explorer.

Format Extension Description
ApacheAvro .avro An Avro format that supports logical types. Supported compression codecs: null, deflate, and snappy. The reader implementation of the apacheavro format is based on the official Apache Avro library. For details on ingesting Event Hubs Capture Avro files, see Ingesting Event Hubs Capture Avro files.
Avro .avro A legacy implementation of the Avro format based on the .NET library. Supported compression codecs: null and deflate. To use snappy, use the ApacheAvro data format.
CSV .csv A text file with comma-separated values (,). See RFC 4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files.
JSON .json A text file with JSON objects delimited by \n or \r\n. See JSON Lines (JSONL).
MultiJSON .multijson A text file with a JSON array of property bags (each representing a record), or any number of property bags delimited by whitespace, \n, or \r\n. Each property bag can span multiple lines.
ORC .orc An ORC file.
Parquet .parquet A Parquet file.
PSV .psv A text file with pipe-separated values (|).
RAW .raw A text file whose entire contents are a single string value.
SCsv .scsv A text file with semicolon-separated values (;).
SOHsv .sohsv A text file with SOH-separated values. (SOH is ASCII codepoint 1; this format is used by Hive on HDInsight.)
TSV .tsv A text file with tab-separated values (\t).
TSVE .tsv A text file with tab-separated values (\t). A backslash character (\) is used for escaping.
TXT .txt A text file with lines delimited by \n. Empty lines are skipped.
W3CLOGFILE .log Web log file format standardized by the W3C.

Note

  • Ingestion from data storage systems that provide ACID functionality on top of regular Parquet format files (for example, Apache Iceberg, Apache Hudi, and Delta Lake) isn't supported.
  • Schemaless Avro isn't supported.

For more information about ingesting data by using the json or multijson formats, see Ingest JSON formats.

Supported data compression formats

Compress blobs and files with these algorithms:

Compression Extension
gzip .gz
zip .zip

Indicate compression by appending the extension to the blob or file name.

For example:

  • MyData.csv.zip indicates a blob or file formatted as CSV, compressed with zip (archive or single file).
  • MyData.json.gz indicates a blob or file formatted as JSON, compressed with gzip.

Blob or file names that include only the compression extension (for example, MyData.zip) are also supported. In this case, specify the file format as an ingestion property because it can't be inferred.

Note

  • Some compression formats store the original file extension in the compressed stream. Ignore this extension when you determine the file format. If you can't determine the file format from the compressed blob or file name, specify it with the format ingestion property.
  • Don't confuse these with internal chunk-level compression codecs used by Parquet, AVRO, and ORC formats. The internal compression name is usually added before the file format extension (for example, file1.gz.parquet, file1.snappy.avro).
  • The Deflate64/Enhanced Deflate zip compression method isn't supported. Windows built-in zip compressor can use this method on files larger than 2 GB.

Related content