Configure HSM customer-managed keys for DBFS using the Azure CLI

  • Article

Note

This feature is available only in the Premium plan.

You can use the Azure CLI to configure your own encryption key to encrypt the workspace storage account. This article describes how to configure your own key from Azure Key Vault Managed HSM. For instructions on using a key from Azure Key Vault vaults, see Configure customer-managed keys for DBFS using the Azure CLI.

Important

The Key Vault must be in the same Azure tenant as your Azure Databricks workspace.

For more information about customer-managed keys for DBFS, see Customer-managed keys for DBFS root.

Install the Azure Databricks CLI extension

  1. Install the Azure CLI.

  2. Install the Azure Databricks CLI extension.

    az extension add --name databricks

Prepare a new or existing Azure Databricks workspace for encryption

Replace the placeholder values in brackets with your own values. The <workspace-name> is the resource name as displayed in the Azure portal.

az cloud set -n AzureChinaCloud
az login
# az cloud set -n AzureCloud   //means return to Public Azure.
az account set --subscription <subscription-id>

Prepare for encryption during workspace creation:

az databricks workspace create --name <workspace-name> --location <workspace-location> --resource-group <resource-group> --sku premium --prepare-encryption

Prepare an existing workspace for encryption:

az databricks workspace update --name <workspace-name> --resource-group <resource-group> --prepare-encryption

Note the principalId field in the storageAccountIdentity section of the command output. You will provide it as the managed identity value when you configure the role assignment on your Key Vault.

For more information about Azure CLI commands for Azure Databricks workspaces, see the az databricks workspace command reference.

Create an Azure Key Vault Managed HSM and an HSM key

You can use an existing Azure Key Vault Managed HSM or create and activate a new one following Quickstart: Provision and activate a Managed HSM using Azure CLI. The Azure Key Vault Managed HSM must have Purge Protection enabled.

To create an HSM key, follow Create an HSM key.

Configure the Managed HSM role assignment

Configure a role assignment for the Key Vault Managed HSM so that your Azure Databricks workspace has permission to access it. Replace the placeholder values in brackets with your own values.

az keyvault role assignment create \
        --role "Managed HSM Crypto Service Encryption User" \
        --scope "/" \
        --hsm-name <hsm-name> \
        --assignee-object-id <managed-identity>

Replace <managed-identity> with the principalId value that you noted when you prepared your workspace for encryption.

Configure DBFS encryption with customer-managed keys

Configure your Azure Databricks workspace to use the key you created in your Azure Key Vault.

Replace the placeholder values with your own values.

az databricks workspace update --name <workspace-name> --resource-group <resource-group> --key-source Microsoft.KeyVault --key-name <key> --key-vault <hsm-uri> --key-version <key-version>

Disable customer-managed keys

When you disable customer-managed keys, your storage account is once again encrypted with Azure-managed keys.

Replace the placeholder values in brackets with your own values and use the variables defined in the previous steps.

az databricks workspace update --name <workspace-name> --resource-group <resource-group> --key-source Default