Parquet format in Azure Data Factory and Azure Synapse Analytics

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Follow this article when you want to parse the Parquet files or write the data into Parquet format.

Parquet format is supported for the following connectors:

• Amazon S3
• Amazon S3 Compatible Storage
• Azure Blob
• Azure Data Lake Storage Gen2
• Azure Files
• File System
• FTP
• Google Cloud Storage
• HDFS
• HTTP
• Oracle Cloud Storage
• SFTP

For a list of supported features for all available connectors, visit the Connectors Overview article.

Using Self-hosted Integration Runtime

For copy running on Self-hosted IR with Parquet file serialization/deserialization, the service locates the Java runtime by firstly checking the registry (SOFTWARE\JavaSoft\Java Runtime Environment\{Current Version}\JavaHome) for JRE, if not found, secondly checking system variable JAVA_HOME for OpenJDK.

• To use JRE: The 64-bit IR requires 64-bit JRE. You can find it from here.
• To use JDK: The 64-but IR requires 64-bit JDK 23. You can find it from here. Be sure to update the JAVA_HOME system variable to the root folder of the JDK 23 installation i.e. C:\Program Files\Java\jdk-23, and add the path to both the C:\Program Files\Java\jdk-23\bin and C:\Program Files\Java\jdk-23\bin\server folders to the Path system variable.
• To use OpenJDK: It's supported since IR version 3.13. Package the jvm.dll with all other required assemblies of OpenJDK into Self-hosted IR machine, and set system environment variable JAVA_HOME accordingly, and then restart Self-hosted IR for taking effect immediately. To download the Microsoft Build of OpenJDK, see Microsoft Build of OpenJDK�.

Example: set variable _JAVA_OPTIONS with value -Xms256m -Xmx16g. The flag Xms specifies the initial memory allocation pool for a Java Virtual Machine (JVM), while Xmx specifies the maximum memory allocation pool. This means that JVM will be started with Xms amount of memory and will be able to use a maximum of Xmx amount of memory. By default, the service uses min 64 MB and max 1G.

Dataset properties

For a full list of sections and properties available for defining datasets, see the Datasets article. This section provides a list of properties supported by the Parquet dataset.

Below is an example of Parquet dataset on Azure Blob Storage:

{
    "name": "ParquetDataset",
    "properties": {
        "type": "Parquet",
        "linkedServiceName": {
            "referenceName": "<Azure Blob Storage linked service name>",
            "type": "LinkedServiceReference"
        },
        "schema": [ < physical schema, optional, retrievable during authoring > ],
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "container": "containername",
                "folderPath": "folder/subfolder",
            },
            "compressionCodec": "snappy"
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by the Parquet source and sink.

Parquet as source

The following properties are supported in the copy activity *source* section.

Parquet as sink

The following properties are supported in the copy activity *sink* section.

Supported Parquet write settings under formatSettings:

Mapping data flow properties

In mapping data flows, you can read and write to parquet format in the following data stores: Azure Blob Storage, Azure Data Lake Storage Gen2 and SFTP, and you can read parquet format in Amazon S3.

Source properties

The below table lists the properties supported by a parquet source. You can edit these properties in the Source options tab.

Source example

The below image is an example of a parquet source configuration in mapping data flows.

The associated data flow script is:

source(allowSchemaDrift: true,
    validateSchema: false,
    rowUrlColumn: 'fileName',
    format: 'parquet') ~> ParquetSource

Sink properties

The below table lists the properties supported by a parquet sink. You can edit these properties in the Settings tab.

Sink example

The below image is an example of a parquet sink configuration in mapping data flows.

The associated data flow script is:

ParquetSource sink(
    format: 'parquet',
    filePattern:'output[n].parquet',
    truncate: true,
    allowSchemaDrift: true,
    validateSchema: false,
    skipDuplicateMapInputs: true,
    skipDuplicateMapOutputs: true) ~> ParquetSink

Data type mapping for Parquet

When reading data from the source connector in Parquet format, the following mappings are used from Parquet data types to interim data types used by the service internally.

When writing data to the sink connector in Parquet format, the following mappings are used from interim data types used by the service internally to Parquet data types.

To learn about how the copy activity maps the source schema and data type to the sink, see Schema and data type mappings.

Parquet complex data types (e.g. MAP, LIST, STRUCT) are currently supported only in Data Flows, not in Copy Activity. To use complex types in data flows, do not import the file schema in the dataset, leaving schema blank in the dataset. Then, in the Source transformation, import the projection.

Related content

• Copy activity overview
• Mapping data flow
• Lookup activity
• GetMetadata activity