Data governance with Unity Catalog

  • Article

This guide shows how to manage data and AI object access in Azure Databricks. For information on Azure Databricks security, see the Security guide. Azure Databricks provides centralized governance for data and AI with Unity Catalog and Delta Sharing.

Centralize access control using Unity Catalog

Unity Catalog is a fine-grained governance solution for data and AI on the Databricks platform. It helps simplify security and governance of your data and AI assets by providing a central place to administer and audit access to data and AI assets.

In most accounts, Unity Catalog is enabled by default when you create a workspace. For details, see Automatic enablement of Unity Catalog.

For a discussion of how to use Unity Catalog effectively, see Unity Catalog best practices.

Track data lineage using Unity Catalog

You can use Unity Catalog to capture runtime data lineage across queries in any language executed on an Azure Databricks cluster or SQL warehouse. Lineage is captured down to the column level, and includes notebooks, jobs, and dashboards related to the query. To learn more, see Capture and view data lineage using Unity Catalog.

Discover data using Catalog Explorer

Databricks Catalog Explorer provides a UI to explore and manage data and AI assets, including schemas (databases), tables, volumes (non-tabular data), and registered ML models, along with asset permissions, data owners, external locations, and credentials. You can use the Insights tab in Catalog Explorer to view the most frequent recent queries and users of any table registered in Unity Catalog.

Share data using Delta Sharing

Delta Sharing is an open protocol developed by Databricks for secure data and AI asset sharing with other organizations, or with other teams within your organization, regardless of which computing platforms they use.

Configure audit logging

Databricks provides access to audit logs of activities performed by Databricks users, allowing your enterprise to monitor detailed Databricks usage patterns.

Configure identity

Every good data governance story starts with a strong identity foundation. To learn how to best configure identity in Azure Databricks, see Identity best practices.

Legacy data governance solutions

Azure Databricks also provides these legacy governance models:

  • Table access control is a legacy data governance model that lets you programmatically grant and revoke access to objects managed by your workspace's built-in Hive metastore. Databricks recommends that you use Unity Catalog instead of table access control. Unity Catalog simplifies security and governance of your data by providing a central place to administer and audit data access across multiple workspaces in your account.

  • Azure Data Lake Storage credential passthrough (legacy) is also a legacy data governance feature that allows you authenticate automatically to Azure Storage from Azure Databricks clusters using the same Microsoft Entra ID identity that you use to log into Azure Databricks. Databricks recommends that you use Unity Catalog instead.