Create and get relationships using the REST API
In this tutorial, you learn how to use Purview REST APIs to:
- Create assets and lineage relationships between the data assets.
- Query lineage relationships/paths.
APIs referenced in this article:
Prerequisites
The use of these APIs requires Data Curator and Data Reader roles. Check this tutorial for details on how to obtain the access token.
Concepts
Asset
In Purview, we have two types of base assets: DataSet and Process.
- DataSet Assets that contain data such as Azure SQL Table, Table etc. should be inherited from DataSet.
- Process Assets that process data, such as a data pipeline, query, function etc. should be inherited from Process.
Refer to assets and types to understand type definitions of DataSets and Processes.
Relationships & Lineage
In Microsoft Purview, we define three types of relationships for lineage:
- dataset_process_inputs: connects DataSet to Process, which means the DataSet is the input of the Process
- process_dataset_outputs: connects Process to DataSet, which means the Process produces the DataSet
- direct_lineage_dataset_dataset: connects DataSet1 to DataSet2, which means DataSet1 is the upstream of DataSet2, although we wouldn’t know exactly which Process is between them
Example 1: Here’s an example of lineage with 2 DataSets, one Process and two lineage relationships:
Dataset ---> dataset_process_inputs ---> Process ---> process_dataset_outputs ---> DataSet
Example 2: Another example of lineage with 2 DataSets and one lineage relationship:
Dataset ---> direct_lineage_dataset_dataset ---> DataSet
Create assets and lineage relationships between the data assets
In the following sections, let’s take hive_table as an example type for DataSet, and hive_query_process as an example type for Process. We'll create assets with those two types and create lineages between them. You can use any other type, which inherits from DataSet or Process to create lineages.
Example 1
Create assets via API
If the assets you want to create lineage for aren't created in Microsoft Purview yet, you can call the following API to create them.
API: Bulk Create Assets
Task: Create two hive_tables as datasets - table1 and table2, each with 2 hive_columns - column1 and column2.
POST {endpoint}/datamap/api/atlas/v2/entity/bulk
With the body:
{
"entities": [
{
"typeName": "hive_table",
"attributes": {
"qualifiedName": "test_lineage.table1",
"name": "table1"
},
"relationshipAttributes": {
"columns": [
{
"guid": "-11",
"typeName": "hive_column"
},{
"guid": "-12",
"typeName": "hive_column"
}
]
},
"guid": "-1"
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table1#column1",
"name": "column1",
"type": "int"
},
"guid": "-11",
"relationshipAttributes": {
"table": {
"guid": "-1",
"typeName": "hive_table"
}
}
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table1#column2",
"name": "column2",
"type": "int"
},
"guid": "-12",
"relationshipAttributes": {
"table": {
"guid": "-1",
"typeName": "hive_table"
}
}
},
{
"typeName": "hive_table",
"attributes": {
"qualifiedName": "test_lineage.table2",
"name": "table2"
},
"relationshipAttributes": {
"columns": [
{
"guid": "-21",
"typeName": "hive_column"
},{
"guid": "-22",
"typeName": "hive_column"
}
]
},
"guid": "-2"
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table2#column1",
"name": "column1",
"type": "int"
},
"guid": "-21",
"relationshipAttributes": {
"table": {
"guid": "-2",
"typeName": "hive_table"
}
}
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table2#column2",
"name": "column2",
"type": "int"
},
"guid": "-22",
"relationshipAttributes": {
"table": {
"guid": "-2",
"typeName": "hive_table"
}
}
}
]
}
Task: Create the process asset 'hive_view_query'
POST {endpoint}/datamap/api/atlas/v2/entity/bulk
With the body:
{
"entities": [
{
"typeName": "hive_view_query",
"attributes": {
"qualifiedName": "test_lineage.HiveQuery1",
"name": "HiveQuery1",
"columnMapping": "[{\"DatasetMapping\":{\"Source\":\"test_lineage.table1\",\"Sink\":\"test_lineage.table2\"},\"ColumnMapping\":[{\"Source\":\"column1\",\"Sink\":\"column1\"},{\"Source\":\"column2\",\"Sink\":\"column2\"}]}]"
},
"guid": "-1"
}
]
}
The above API calls result in creation of two hive_tables(Datasets) and one hive_view_query(Process).
Create lineage relationships between Datasets and the Process
API: Create Relationship
Task: Create lineage from table1 -> HiveQuery1 (that is, Dataset -> Process)
POST {endpoint}/datamap/api/atlas/v2/relationship
With the body:
{
"typeName": "dataset_process_inputs",
"guid": "-1",
"end1": {
"typeName": "hive_table",
"uniqueAttributes": {
"qualifiedName": "test_lineage.table1"
}
},
"end2": {
"typeName": "Process",
"uniqueAttributes": {
"qualifiedName": "test_lineage.HiveQuery1"
}
}
}
Task: Create lineage from HiveQuery1 -> table2 (that is, Process -> Dataset)
POST {endpoint}/datamap/api/atlas/v2/relationship
With the body:
{
"typeName": "process_dataset_outputs",
"guid": "-2",
"end1": {
"typeName": "Process",
"uniqueAttributes": {
"qualifiedName": "test_lineage.HiveQuery1"
}
},
"end2": {
"typeName": "hive_table",
"uniqueAttributes": {
"qualifiedName": "test_lineage.table2"
}
}
}
View lineage
Once the assets and lineage relationships are created, you can check the lineage graph in Microsoft Purview:
Example 2
Create a hive table, table3, with two columns
API: Bulk Create Assets
Task: Create table3, with 2 hive_columns, column1 and column2
POST {endpoint}/datamap/api/atlas/v2/entity/bulk
With the body:
{
"entities": [
{
"typeName": "hive_table",
"attributes": {
"qualifiedName": "test_lineage.table3",
"name": "table3"
},
"relationshipAttributes": {
"columns": [
{
"guid": "-31",
"typeName": "hive_column"
},{
"guid": "-32",
"typeName": "hive_column"
}
]
},
"guid": "-3"
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table3#column1",
"name": "column1",
"type": "int"
},
"guid": "-31",
"relationshipAttributes": {
"table": {
"guid": "-3",
"typeName": "hive_table"
}
}
},
{
"typeName": "hive_column",
"attributes": {
"qualifiedName": "test_lineage.table3#column2",
"name": "column2",
"type": "int"
},
"guid": "-32",
"relationshipAttributes": {
"table": {
"guid": "-3",
"typeName": "hive_table"
}
}
}
]
}
Create direct lineage between table 2 and table 3, with column mapping
API: Create Relationship
Task: Create lineage from table2 -> table3 (that is, Dataset -> Dataset) with column mapping
POST {endpoint}/datamap/api/atlas/v2/relationship
With the body:
{
"typeName": "direct_lineage_dataset_dataset",
"guid": "-1",
"end1": {
"typeName": "hive_table",
"uniqueAttributes": {
"qualifiedName": "test_lineage.table2"
}
},
"end2": {
"typeName": " hive_table ",
"uniqueAttributes": {
"qualifiedName": "test_lineage.table3"
}
},
"attributes": {
"columnMapping": "[{\"Source\":\"column1\",\"Sink\":\"column1\"},{\"Source\":\"column2\",\"Sink\":\"column2\"}]"
}
}
View lineage
Now the lineage graph (together from Example 1 & Example 2 above) becomes:
Notice that table2 is directly linked to table3, without a HiveQuery between them.
Query lineage relationships/paths
API: Get Lineage by GUID
Task: Get lineage of table2 via REST API
GET {{endpoint}}/api/atlas/v2/lineage/{{guid_of_table2}}?direction=BOTH
Below you can JSON response payload:
{
"baseEntityGuid": "2a12b3ff-5816-4222-833a-035bf82e06e0",
"lineageDirection": "BOTH",
"lineageDepth": 3,
"lineageWidth": -1,
"childrenCount": -1,
"guidEntityMap": {
"16b93b78-8683-4f88-9651-24c4a9d797b0": {
"typeName": "hive_table",
"attributes": {
"temporary": false,
"lastAccessTime": 0,
"createTime": 0,
"qualifiedName": "test_lineage.table3",
"name": "table3",
"retention": 0
},
"lastModifiedTS": "1",
"guid": "16b93b78-8683-4f88-9651-24c4a9d797b0",
"status": "ACTIVE",
"displayText": "table3",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": [],
"isIndexed": true
},
"cb22ba23-47a2-4149-ade6-e3d9642fe592": {
"typeName": "hive_table",
"attributes": {
"temporary": false,
"lastAccessTime": 0,
"createTime": 0,
"qualifiedName": "test_lineage.table1",
"name": "table1",
"retention": 0
},
"lastModifiedTS": "1",
"guid": "cb22ba23-47a2-4149-ade6-e3d9642fe592",
"status": "ACTIVE",
"displayText": "table1",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": [],
"isIndexed": true
},
"bbeacce6-5bde-46f7-8fe4-689cbb36ba51": {
"typeName": "hive_view_query",
"attributes": {
"qualifiedName": "test_lineage.HiveQuery1",
"name": "HiveQuery1",
"columnMapping": "[{\"DatasetMapping\":{\"Source\":\"test_lineage.table1\",\"Sink\":\"test_lineage.table2\"},\"ColumnMapping\":[{\"Source\":\"column1\",\"Sink\":\"column1\"},{\"Source\":\"column2\",\"Sink\":\"column2\"}]}]"
},
"lastModifiedTS": "1",
"guid": "bbeacce6-5bde-46f7-8fe4-689cbb36ba51",
"status": "ACTIVE",
"displayText": "HiveQuery1",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": [],
"isIndexed": true
},
"2a12b3ff-5816-4222-833a-035bf82e06e0": {
"typeName": "hive_table",
"attributes": {
"temporary": false,
"lastAccessTime": 0,
"createTime": 0,
"qualifiedName": "test_lineage.table2",
"name": "table2",
"retention": 0
},
"lastModifiedTS": "1",
"guid": "2a12b3ff-5816-4222-833a-035bf82e06e0",
"status": "ACTIVE",
"displayText": "table2",
"classificationNames": [],
"meaningNames": [],
"meanings": [],
"isIncomplete": false,
"labels": [],
"isIndexed": true
}
},
"includeParent": false,
"relations": [
{
"fromEntityId": "2a12b3ff-5816-4222-833a-035bf82e06e0",
"toEntityId": "16b93b78-8683-4f88-9651-24c4a9d797b0",
"relationshipId": "23df8e3e-b066-40b2-be29-9fd90693c51b",
"columnMapping": "[{\"Source\":\"column1\",\"Sink\":\"column1\"},{\"Source\":\"column2\",\"Sink\":\"column2\"}]"
},
{
"fromEntityId": "bbeacce6-5bde-46f7-8fe4-689cbb36ba51",
"toEntityId": "2a12b3ff-5816-4222-833a-035bf82e06e0",
"relationshipId": "5fe8d378-39cd-4c6b-8ced-91b0152d3014"
},
{
"fromEntityId": "cb22ba23-47a2-4149-ade6-e3d9642fe592",
"toEntityId": "bbeacce6-5bde-46f7-8fe4-689cbb36ba51",
"relationshipId": "73e084bf-98a3-45fb-a1e4-c56cc40661b8"
}
],
"parentRelations": [],
"widthCounts": {
"INPUT": {
"cb22ba23-47a2-4149-ade6-e3d9642fe592": 0,
"bbeacce6-5bde-46f7-8fe4-689cbb36ba51": 1,
"2a12b3ff-5816-4222-833a-035bf82e06e0": 1
},
"OUTPUT": {
"16b93b78-8683-4f88-9651-24c4a9d797b0": 0,
"2a12b3ff-5816-4222-833a-035bf82e06e0": 1
}
}
}
Other Resources
Type Definitions
All Assets/Entities and Relationships are defined in the type system.
Call list all type definitions API to get current type definitions in your Microsoft Purview instance. If you’re not sure which typeName to use in the API calls described, you can check the type definitions API to find the appropriate Asset/Entity type.
Following is a sample response of this API:
You can see that in the entityDefs in the above response, asset types are defined. A further look at the definition of the asset shows that the asset is inherited DataSet type.
Similarly, you can find that a process is inherited from “Process” type as shown:
Create New Custom Types
If you want to create custom assets or processes, you can use the following APIs.
API: Create Typedef API
Task: Create a Custom Process Type
POST {endpoint}/datamap/api/atlas/v2/types/typedefs
With the body:
{
"enumDefs": [],
"structDefs": [],
"classificationDefs": [],
"entityDefs": [
{
"name": "MyCustomServiceProcess",
"superTypes": [
"Process"
],
"typeVersion": "1.0",
"attributeDefs": []
}
],
"relationshipDefs": []
}
Task: Create a Custom DataSet Type
POST {endpoint}/datamap/api/atlas/v2/types/typedefs
With the body:
{
"enumDefs": [],
"structDefs": [],
"classificationDefs": [],
"entityDefs": [
{
"name": "MyCustomDataSet",
"superTypes": [
"DataSet"
],
"typeVersion": "1.0",
"attributeDefs": []
}
],
"relationshipDefs": []
}