Mapping data flow transformation overview

  • Article

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

Data flows are available both in Azure Data Factory and Azure Synapse Pipelines. This article applies to mapping data flows. If you are new to transformations, please refer to the introductory article Transform data using a mapping data flow.

Below is a list of the transformations currently supported in mapping data flow. Click on each transformations to learn its configuration details.

Name Category Description
Aggregate Schema modifier Define different types of aggregations such as SUM, MIN, MAX, and COUNT grouped by existing or computed columns.
Alter row Row modifier Set insert, delete, update, and upsert policies on rows.
Assert Row modifier Set assert rules for each row.
Cast Schema modifier Change column data types with type checking.
Conditional split Multiple inputs/outputs Route rows of data to different streams based on matching conditions.
Derived column Schema modifier Generate new columns or modify existing fields using the data flow expression language.
External call Schema modifier Call external endpoints inline row-by-row.
Exists Multiple inputs/outputs Check whether your data exists in another source or stream.
Filter Row modifier Filter a row based upon a condition.
Flatten Formatters Take array values inside hierarchical structures such as JSON and unroll them into individual rows.
Flowlet Flowlets Build and include custom re-usable transformation logic.
Join Multiple inputs/outputs Combine data from two sources or streams.
Lookup Multiple inputs/outputs Reference data from another source.
New branch Multiple inputs/outputs Apply multiple sets of operations and transformations against the same data stream.
Parse Formatters Parse text columns in your data stream that are strings of JSON, delimited text, or XML formatted text.
Pivot Schema modifier An aggregation where one or more grouping columns has its distinct row values transformed into individual columns.
Rank Schema modifier Generate an ordered ranking based upon sort conditions
Select Schema modifier Alias columns and stream names, and drop or reorder columns
Sink - A final destination for your data
Sort Row modifier Sort incoming rows on the current data stream
Source - A data source for the data flow
Stringify Formatters Turn complex types into plain strings
Surrogate key Schema modifier Add an incrementing non-business arbitrary key value
Union Multiple inputs/outputs Combine multiple data streams vertically
Unpivot Schema modifier Pivot columns into row values
Window Schema modifier Define window-based aggregations of columns in your data streams.