Manage Unity Catalog metastores

  • Article

This article shows how to update, delete, and manage the behavior of Unity Catalog metastores in your Azure Databricks account.

To learn about Unity Catalog metastores and how to create them, see Create a Unity Catalog metastore.

Enable a metastore to be automatically assigned to new workspaces

To assign an existing Unity Catalog metastore automatically to new workspaces in that metastore's region, an account admin can enable workspace auto-assignment for the metastore. If this setting is not selected, the admin who creates a workspace in the same region as the metastore must manually enable the workspace for Unity Catalog and select the metastore from a drop-down.

Before an account admin enables this option, they should be sure to understand the following impacts on new workspaces:

To enable automatic assignment:

  1. As an account admin, go to the Azure Databricks account console.
  2. Click Catalog icon Catalog.
  3. Select your metastore.
  4. On the Configuration tab, under Workspace assignment, select Automatically assign new workspaces in <region> to this metastore.
  5. On the confirmation dialog, click Enable auto-assignment.

Add managed storage to an existing metastore

Metastore-level managed storage is optional, and it is not included for metastores that were created automatically. You might want to add metastore-level storage to your metastore if you prefer a data isolation model that stores data centrally for multiple workspaces. You need metastore-level storage if you want to share notebooks using Delta Sharing or if you are an Azure Databricks partner who uses personal staging locations.

See also Specify a managed storage location in Unity Catalog.

Requirements

  • You must have at least one workspace attached to the Unity Catalog metastore.
  • Azure Databricks permissions required:
    • To create an external location, you must be a metastore admin or user with the CREATE EXTERNAL LOCATION and CREATE STORAGE CREDENTIAL privileges.
    • To add the storage location to the metastore definition, you must be an account admin. For instructions on enabling the account admin role in your account, see Establish your first account admin.
  • Azure tenant permissions required:
    • Permission to create a storage account to use with Azure Data Lake Storage Gen2. This storage account must have a hierarchical namespace. See Create a storage account to use with Azure Data Lake Storage Gen2.
    • Permission to create a new resource to hold a system-assigned managed identity. This requires that you be a Contributor or Owner of a resource group in any subscription in the tenant.

Step 1: Create the storage location

Follow the instructions in Step 1 (Optional): Create a storage container for metastore-level managed storage and Step 2 (Optional): Create a managed identity to access the managed storage location to create a storage container in Azure Data Lake Storage Gen2 and an Azure Databricks access connector that holds a managed identity that has access to the storage container.

Step 2: Create an external location in Unity Catalog

In this step, you create an external location in Unity Catalog that references the ADLS Gen 2 path that you just created.

  1. Create a storage credential.

    The storage credential will represent the Azure managed identity that you created in Step 1: Create the storage location.

    Follow the instructions in Create a storage credential for connecting to Azure Data Lake Storage Gen2.

  2. Create an external location that references the storage credential that you created in the previous step and the ADLS Gen 2 storage container that you created in Step 1: Create the storage location.

    Follow the instructions in Create an external location to connect cloud storage to Azure Databricks

  3. Grant yourself the CREATE MANAGED STORAGE privilege on the external location.

    1. Click the external location name to open the details pane.
    2. On the Permissions tab, click Grant.
    3. On the Grant on <external location> dialog, select yourself in the Principals field and select CREATE MANAGED STORAGE.
    4. Click Grant.

Step 3: Add the storage location to the metastore

After you have created an external location that represents the metastore storage bucket, you can add it to the metastore.

  1. As an account admin, log in to the account console.

  2. Click Catalog icon Catalog.

  3. Click the metastore name.

  4. Confirm that you are the Metastore Admin.

    If you are not, click Edit and assign yourself as the metastore admin. You can unassign yourself when you are done with this procedure.

  5. On the Configuration tab, next to ADLS Gen 2 path, click Set.

  6. On the Set metastore root dialog, enter the ADLS Gen 2 path that you used to create the external location, and click Update.

    You cannot modify this path once you set it.

Add a metastore admin

Metastore admins are optional, but there are situations where you might want one for your metastore. See Assign a metastore admin.

Delete a metastore

If you are closing your Azure Databricks account or have another reason to delete access to data managed by your Unity Catalog metastore, you can delete the metastore.

Warning

All objects managed by the metastore will become inaccessible using Azure Databricks workspaces. This action cannot be undone.

Managed table data and metadata will be auto-deleted after 30 days. External table data in your cloud storage is not affected by metastore deletion.

To delete a metastore:

  1. As a metastore admin, log in to the account console.
  2. Click Catalog icon Catalog.
  3. Click the metastore name.
  4. On the Configuration tab, click the three-button menu at the far upper right and select Delete.
  5. On the confirmation dialog, enter the name of the metastore and click Delete.