Connect to Azure Data Lake Storage in Microsoft Purview

This article outlines the process to register and govern an Azure Data Lake Storage (ADLS Gen2) data source in Microsoft Purview including instructions to authenticate and interact with the ADLS Gen2 source.

Supported capabilities

Metadata Extraction	Full Scan	Incremental Scan	Scoped Scan	Classification	Labeling	Access Policy	Lineage	Data Sharing	Live view
Yes	Yes	Yes	Yes	Yes	No	Yes (preview)	Limited*	No	Yes

* Lineage is supported if dataset is used as a source/sink in Data Factory or Synapse pipeline.

When scanning Azure Data Lake Storage Gen2 source, Microsoft Purview supports extracting technical metadata including:

• Storage account
• Data Lake Storage Gen2 Service
• File system (container)
• Folders
• Files
• Resource sets

When setting up scan, you can choose to scan the entire ADLS Gen2 or selective folders. Learn about the supported file format here.

Prerequisites

• An Azure account with an active subscription. Create an account.

• An active Microsoft Purview account.

• You'll need to be a Data Source Administrator and Data Reader to register a source and manage it in the Microsoft Purview governance portal. See our Microsoft Purview Permissions page for details.

• You need to have at least Reader permission on the ADLS Gen 2 account to be able to register it.

Register

This section will enable you to register the ADLS Gen2 data source for scan and data share in Purview.

Prerequisites for register

• You'll need to be a Data Source Admin and one of the other Purview roles (for example, Data Reader or Data Share Contributor) to register a source and manage it in the Microsoft Purview governance portal. See our Microsoft Purview Permissions page for details.

Steps to register

It's important to register the data source in Microsoft Purview prior to setting up a scan for the data source.

1. Go to the Microsoft Purview governance portal by:

   • Browsing directly to https://web.purview.azure.cn and selecting your Microsoft Purview account.
   • Opening the Azure portal, searching for and selecting the Microsoft Purview account. Select the the Microsoft Purview governance portal button.

2. Navigate to the Data Map --> Sources

   

   

3. Create the Collection hierarchy using the Collections menu and assign permissions to individual subcollections, as required

   

4. Navigate to the appropriate collection under the Sources menu and select the Register icon to register a new ADLS Gen2 data source

   

5. Select the Azure Data Lake Storage Gen2 data source and select Continue

   

6. Provide a suitable Name for the data source, select the relevant Azure subscription, existing Data Lake Store account name and the collection and select Apply. Leave the Data Policy Enforcement toggle on the disabled position.

   

7. The ADLS Gen2 storage account will be shown under the selected Collection

   

Scan

Authentication for a scan

Your Azure network may allow for communications between your Azure resources, but if you've set up firewalls, private endpoints, or virtual networks within Azure, you'll need to follow one of these configurations below.

Networking constraints	Integration runtime type	Available credential types
No private endpoints or firewalls	Azure IR	Managed identity (Recommended), service principal, or account key
Firewall enabled but no private endpoints	Azure IR	Managed identity
Private endpoints enabled	*Self-Hosted IR	Service principal, account key

Create the scan

1. Open your Microsoft Purview account and select the Open Microsoft Purview governance portal

2. Navigate to the Data map --> Sources to view the collection hierarchy

3. Select the New Scan icon under the ADLS Gen2 data source registered earlier

   

4. Choose either the Azure integration runtime if your source is publicly accessible, a self-hosted integration runtime if your source is in a private virtual network. For more information about which integration runtime to use, see the choose the right integration runtime configuration article.

Scope and run the scan

1. You can scope your scan to specific folders and subfolders by choosing the appropriate items in the list.

   

2. Then select a scan rule set. You can choose between the system default, existing custom rule sets, or create a new rule set inline.

   

3. If creating a new scan rule set, select the file types to be included in the scan rule.

   

4. You can select the classification rules to be included in the scan rule

   

   

5. Choose your scan trigger. You can set up a schedule or run the scan once.

   

6. Review your scan and select Save and run.

   

View your scans and scan runs

To view existing scans:

1. Go to the Microsoft Purview portal. On the left pane, select Data map.
2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
3. Select the scan that has results you want to view. The pane shows you all the previous scan runs, along with the status and metrics for each scan run.
4. Select the run ID to check the scan run details.

Manage your scans

To edit, cancel, or delete a scan:

1. Go to the Microsoft Purview portal. On the left pane, select Data Map.

2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

3. Select the scan that you want to manage. You can then:

   • Edit the scan by selecting Edit scan.
   • Cancel an in-progress scan by selecting Cancel scan run.
   • Delete your scan by selecting Delete scan.

Access policy pre-requisites on Azure Storage accounts

Region support

• All Microsoft Purview regions are supported.

If needed, you can create a new Storage account by following this guide.

Configure the subscription where the Azure Storage account resides for policies from Microsoft Purview

This step is only necessary in certain regions (see prior section). To enable Microsoft Purview to manage policies for one or more Azure Storage accounts, execute the following PowerShell commands in the subscription where you'll deploy your Azure Storage account. These PowerShell commands will enable Microsoft Purview to manage policies on all Azure Storage accounts in that subscription.

If you’re executing these commands locally, be sure to run PowerShell as an administrator.

# Install the Az module
Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force
# Login into the subscription
Connect-AzAccount -Environment AzureChinaCloud -Subscription <SubscriptionID>
# Register the feature
Register-AzProviderFeature -FeatureName AllowPurviewPolicyEnforcement -ProviderNamespace Microsoft.Storage

If the output of the last command shows RegistrationState as Registered, then your subscription is enabled for access policies. If the output is Registering, wait at least 10 minutes, and then retry the command. Do not continue unless the RegistrationState shows as Registered.

Configure the Microsoft Purview account for policies

Register the data source in Microsoft Purview

Before a policy can be created in Microsoft Purview for a data resource, you must register that data resource in Microsoft Purview Studio. You will find the instructions related to registering the data resource later in this guide.

Configure permissions to enable Data policy enforcement on the data source

Once a resource is registered, but before a policy can be created in Microsoft Purview for that resource, you must configure permissions. A set of permissions are needed to enable the Data policy enforcement. This applies to data sources, resource groups, or subscriptions. To enable Data policy enforcement, you must have both specific Identity and Access Management (IAM) privileges on the resource as well as specific Microsoft Purview privileges:

• You must have either one of the following IAM role combinations on the resource's Azure Resource Manager path or any parent of it (that is, using IAM permission inheritance):

  • IAM Owner
  • Both IAM Contributor and IAM User Access Administrator

  To configure Azure role-based access control (RBAC) permissions, follow this guide. The following screenshot shows how to access the Access Control section in the Azure portal for the data resource to add a role assignment.

  

  Note

  The IAM Owner role for a data resource can be inherited from a parent resource group, a subscription, or a subscription management group. Check which Microsoft Entra users, groups, and service principals hold or are inheriting the IAM Owner role for the resource.

• You also need to have the Microsoft Purview Data source admin role for the collection or a parent collection (if inheritance is enabled). For more information, see the guide on managing Microsoft Purview role assignments.

  The following screenshot shows how to assign the Data source admin role at the root collection level.

  

Configure Microsoft Purview permissions to create, update, or delete access policies

To create, update or delete policies, you need to get the Policy author role in Microsoft Purview at root collection level:

• The Policy author role can create, update, and delete DevOps and Data Owner policies.
• The Policy author role can delete self-service access policies.

For more information about managing Microsoft Purview role assignments, see Create and manage collections in the Microsoft Purview Data Map.

In addition, to easily search Microsoft Entra users or groups when creating or updating the subject of a policy, you can greatly benefit from getting the Directory Readers permission in Microsoft Entra ID. This is a common permission for users in an Azure tenant. Without the Directory Reader permission, the Policy Author will have to type the complete username or email for all the principals included in the subject of a data policy.

Delegate access provisioning responsibility to roles in Microsoft Purview

After a resource has been enabled for Data policy enforcement, any Microsoft Purview user with the Policy author role at the root collection level can provision access to that data source from Microsoft Purview.

If a Microsoft Purview account with published policies is deleted, such policies will stop being enforced within an amount of time that depends on the specific data source. This change can have implications on both security and data access availability. The Contributor and Owner roles in IAM can delete Microsoft Purview accounts. You can check these permissions by going to the Access control (IAM) section for your Microsoft Purview account and selecting Role Assignments. You can also use a lock to prevent the Microsoft Purview account from being deleted through Resource Manager locks.

Register the data source in Microsoft Purview for Data Policy Enforcement

The Azure Storage resource needs to be registered first with Microsoft Purview before you can create access policies. To register your resource, follow the Prerequisites and Register sections of this guide:

• Register Azure Data Lake Storage (ADLS) Gen2 in Microsoft Purview

After you've registered the data source, you'll need to enable Data Policy Enforcement. This is a pre-requisite before you can create policies on the data source. Data Policy Enforcement can impact the security of your data, as it delegates to certain Microsoft Purview roles managing access to the data sources. Go through the secure practices related to Data Policy Enforcement in this guide: How to enable Data Policy Enforcement

Once your data source has the Data Policy Enforcement option set to Enabled, it will look like this screenshot:

Create a policy

To create policies that cover all data sources inside a resource group or Azure subscription you can refer to this section.

Next steps

Follow the below guides to learn more about Microsoft Purview and your data.

• Data Estate Insights in Microsoft Purview
• Lineage in Microsoft Purview