Enable customer-managed keys for managed services

Article
09/16/2024

Note

This feature requires the Premium plan.

For additional control of your data, you can add your own key to protect and control access to some types of data. Azure Databricks has multiple customer-managed key features. To compare the related features, see Customer-managed keys for encryption.

Tip

This article describes how to configure your own key from Azure Key Vault vaults for managed services. For instructions on using a key from Azure Key Vault Managed HSM, see Enable HSM customer-managed keys for managed services.

Managed services data in the Azure Databricks control plane is encrypted at rest. You can add a customer-managed key for managed services to help protect and control access to the following types of encrypted data:

Notebook source in the Azure Databricks control plane.
Notebook results for notebooks run interactively (not as jobs) that are stored in the control plane. By default, larger results are also stored in your workspace root bucket. You can configure Azure Databricks to store all interactive notebook results in your cloud account.
Secrets stored by the secret manager APIs.
Databricks SQL queries and query history.
Personal access tokens (PAT) or other credentials used to set up Git integration with Databricks Git folders.

After you add a customer-managed key encryption for a workspace, Azure Databricks uses your key to control access to the key that encrypts future write operations to your workspace's managed services data. Existing data is not re-encrypted. The data encryption key is cached in memory for several read and write operations and evicted from memory at a regular interval. New requests for that data require another request to your cloud service's key management system. If you delete or revoke your key, reading or writing to the protected data fails at the end of the cache time interval.

You can rotate (update) the customer-managed key at a later time. See Rotate the key at a later time.

This feature does not encrypt data stored outside of the control plane. For other customer-managed key features, see Customer-managed keys for encryption

Requirements

To use the Azure CLI for these tasks, install the Azure CLI tool and install the Databricks extension:
```
az extension add --name databricks
```
To use Powershell for these tasks, install Azure PowerShell and install the Databricks Powershell module. You must also log in:
```
Connect-AzAccount -Environment AzureChinaCloud
```
To log in to your Azure account as a user, see PowerShell login with an Azure Databricks user account. To log in to your Azure account as a service principal, see PowerShell login with a Microsoft Entra ID service principal.

Step 1: Set up a Key Vault

You must create an Azure Key Vault instance and set its permissions. You can do this through the Azure portal, CLI, or APIs.

Important

The Key Vault must be in the same Azure tenant as your Azure Databricks workspace.

These instructions offer details multiple deployment options:

Use the Azure portal
Use the Azure CLI
Use Azure Powershell

Use the Azure portal

Create or select a Key Vault:
- To create a Key Vault, go to the Azure portal page for creating a Key Vault. Click + Create. Enter the resource group name, Key Vault name, region, and pricing tier. Click Review + create and then click Create.
- To use an existing Key Vault, copy its Key Vault name for the next step.
Get the object ID of the AzureDatabricks application:
1. In the Azure portal, go to Microsoft Entra ID.
2. Select Enterprise Applications from the sidebar menu.
3. Search for AzureDatabricks and click the Enterprise Application in the results.
4. From Properties, copy the object ID.
Add an access policy to Key Vault using the Azure portal:
1. Navigate to the Azure Key Vault that you will use to configure customer managed keys for managed services for your workspace.
2. Click the Access policies tab from the left-side panel.
3. Select the Create button found at the top of the page.
4. Under the Key permissions section in the Permissions tab, enable Get, Unwrap Key, and Wrap key.
5. Click Next.
6. On the Principal tab, type AzureDatabricks and scroll to the first Enterprise Application result that has an Application ID of 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d and select it.
7. Continue to the Review + create tab and click b.

Use the Azure CLI

Use the Azure CLI to complete the following instructions.

Create a Key Vault or select an existing Key Vault:
- To create a Key Vault, use the following Azure CLI command and replace the items in brackets with your region, Key Vault name, resource group name, and location:
```
az keyvault create --location <region> \
                   --name <key-vault-name> \
                   --resource-group <resource-group-name> \
                   --location <location> \
                   --enable-purge-protection
```
- To use an existing Key Vault, copy the Key Vault name for the next step.

Get the object ID of the AzureDatabricks application with the Azure CLI.

az ad sp show --id "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d" \
              --query "id" \
              --output tsv

Confirm that you are using the correct Azure subscription:
```
az account set --subscription {subscription_id}
```

Use Azure Powershell

You can create a new Key Vault or use an existing one.

Create a Key Vault:

$keyVault = New-AzKeyVault -Name <key-vault-name> \
-ResourceGroupName <resource-group-name> \
-Location <location> \
-sku <sku> \
-EnablePurgeProtection

Use an existing Key Vault:

$keyVault = Get-AzKeyVault -VaultName <key-vault-name>

Step 2: Prepare a key

You can create a key or use an existing key. Use whatever tooling you prefer to use: Azure portal, Azure CLI, or other tooling.

Use Azure CLI

Create a key under the Key Vault. The KeyType must be RSA.

To create the key in CLI, run this command:

az keyvault key create --name <key-name> \
                       --vault-name <key-vault-name> \
                       --protection software

Make note of the following values, which you can get from the key ID in the kid property in the response. You will use them in subsequent steps:

Key vault URL: The beginning part of the key ID that includes the Key Vault name. It has the form https://<key-vault-name>.vault.azure.cn.
Key name: Name of your key.
Key version: Version of the key.

The full key ID usually has the form https://<key-vault-name>.vault.azure.cn/keys/<key-name>/<key-version>. Azure Key Vault keys that are in a non-public cloud have a different form.

To use an existing key instead of creating one, get and copy these values for your key so you can use them in the next steps. Check to confirm that your existing key is enabled before proceeding.

Use Azure Powershell

If you plan to create a key, you might need to set the access policy, depending on how and when you created it. For example, if you recently created the Key Vault using PowerShell, your new Key Vault may lack the access policy required to create a key. The following example uses the EmailAddress parameter to set the policy. For related details, see the Microsoft article about Set-AzKeyVaultAccessPolicy.

Set the access policy on a new Key Vault:
```
Set-AzKeyVaultAccessPolicy \
-VaultName $keyVault.VaultName \
-PermissionsToKeys all \
-EmailAddress <email-address>
```

You can either create a key or retrieve an existing key:

Create a key:

$key = Add-AzKeyVaultKey \
-VaultName $keyVault.VaultName \
-Name <key-name> \
-Destination 'Software'

Retrieve an existing key:

$key = Get-AzKeyVaultKey \
-VaultName $keyVault.VaultName \
-Name <key-name>

Add an access policy with permissions to the Key Vault:

$managedService = Get-AzureADServicePrincipal \
-Filter "appId eq '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d'"

Set-AzKeyVaultAccessPolicy -VaultName $keyVault.VaultName \
-ObjectId $managedService.ObjectId \
-PermissionsToKeys wrapkey,unwrapkey,get

Step 3: Add a key to a workspace

You can deploy a new workspace with a customer-managed key for managed services or add a key to an existing workspace. You can do both with Azure CLI, Powershell, ARM templates, Azure portal, or other tools. This section includes details for multiple deployment options:

Use the Azure portal with no template
Use the Azure CLI with no template
Use Powershell with no template
Apply changes with an ARM template

Use the Azure portal with no template

Go to the Azure Portal homepage.
Click Create a resource in the top left corner of the page.
Within the search bar, type Azure Databricks and click the Azure Databricks option.
Click Create in the Azure Databricks widget.
Enter values for the input fields on the Basics and Networking tabs.
After you reach the Encryption tab:
- For creating a workspace, enable Use your own key in the Managed Services section.
- For updating a workspace, enable Managed Services.
Set the encryption fields.
- In the Key Identifier field, paste the Key Identifier of your Azure Key Vault key.
- In the Subscription dropdown, enter the subscription name of your Azure Key Vault key.
Complete the remaining tabs and click Review + Create (for new workspace) or Save (for updating a workspace).

Important

If you rotate the key, you must keep the old key available for 24 hours.

Use the Azure CLI with no template

Add an access policy to Key Vault with the following command. Replace <key-vault-name> with the vault name that you used in the previous step and replace <object-id> with the object ID of the AzureDatabricks application.
```
az keyvault set-policy -n <key-vault-name> \
                       --key-permissions get wrapKey unwrapKey  \
                       --object-id <object-id>
```

Create or update a workspace:

For both creation and update, add these fields to the command:

managed-services-key-name: Key name
managed-services-key-vault: Key vault URI
managed-services-key-version: Key version

Example creation of a workspace using these fields:

az databricks workspace create --name <workspace-name> \
--resource-group <resource-group-name> \
--location <location> \
--sku premium \
--managed-services-key-name <key-name> \
--managed-services-key-vault <key-vault-uri> \
--managed-services-key-version <key-version>

Example update of a workspace using these fields:

az databricks workspace update --name <workspace-name> \
--resource-group <resource-group-name> \
--managed-services-key-name <key-name> \
--managed-services-key-vault <key-vault-uri> \
--managed-services-key-version <key-version>

Important

If you rotate the key, you must keep the old key available for 24 hours.

Use Powershell with no template

To create or update a workspace, add the following parameters to the command for your new key:

ManagedServicesKeyVaultPropertiesKeyName: Key name
ManagedServicesKeyVaultPropertiesKeyVaultUri: Key URI
ManagedServicesKeyVaultPropertiesKeyVersion: Key version

Example workspace creation with these fields:

New-AzDatabricksWorkspace -Name <workspace-name> \
-ResourceGroupName <resource-group-name> \
-location $keyVault.Location \
-sku premium \
-ManagedServicesKeyVaultPropertiesKeyName $key.Name \
-ManagedServicesKeyVaultPropertiesKeyVaultUri $keyVault.VaultUri \
-ManagedServicesKeyVaultPropertiesKeyVersion $key.Version

Example workspace update with these fields:

Update-AzDatabricksWorkspace -Name <workspace-name> \
-ResourceGroupName <resource-group-name> \
-sku premium \
-ManagedServicesKeyVaultPropertiesKeyName $key.Name \
-ManagedServicesKeyVaultPropertiesKeyVaultUri $keyVault.VaultUri \
-ManagedServicesKeyVaultPropertiesKeyVersion $key.Version

Important

If you rotate the key, you must keep the old key available for 24 hours.

Apply changes with an ARM template

The following ARM template creates a new workspace with a customer-managed key, using the API version 2023-02-01 for resource Microsoft.Databricks/workspaces. Save this text locally to a file named databricks-cmk-template.json.

This example template does not include all possible Azure Databricks features, such as providing your own VNet in which to deploy the workspace.

Important

If you already use a template, merge this template's extra parameters, resources, and outputs into your existing template.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "workspaceName": {
      "type": "string",
      "metadata": {
        "description": "The name of the Azure Databricks workspace to create."
      }
    },
    "pricingTier": {
      "type": "string",
      "defaultValue": "premium",
      "allowedValues": [
        "standard",
        "premium"
      ],
      "metadata": {
        "description": "The pricing tier of workspace."
      }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "Location for all resources."
      }
    },
    "apiVersion": {
      "type": "string",
      "defaultValue": "2023-02-01",
      "allowedValues":[
        "2023-02-01",
        "2021-04-01-preview"
      ],
      "metadata": {
        "description": "The api version to create the workspace resources"
      }
    },
    "keyvaultUri": {
      "type": "string",
      "metadata": {
        "description": "The Key Vault URI for customer-managed key for managed services"
      }
    },
    "keyName": {
      "type": "string",
      "metadata": {
        "description": "The key name used for customer-managed key for managed services"
      }
    },
    "keyVersion": {
      "type": "string",
      "metadata": {
        "description": "The key version used for customer-managed key for managed services"
      }
    }
  },
  "variables": {
    "managedResourceGroupName": "[concat('databricks-rg-', parameters('workspaceName'), '-', uniqueString(parameters('workspaceName'), resourceGroup().id))]"
  },
  "resources": [
    {
      "type": "Microsoft.Databricks/workspaces",
      "name": "[parameters('workspaceName')]",
      "location": "[parameters('location')]",
      "apiVersion": "[parameters('apiVersion')]",
      "sku": {
        "name": "[parameters('pricingTier')]"
      },
      "properties": {
        "ManagedResourceGroupId": "[concat(subscription().id, '/resourceGroups/', variables('managedResourceGroupName'))]",
        "encryption": {
          "entities": {
             "managedServices": {
                "keySource": "Microsoft.Keyvault",
                "keyVaultProperties": {
                   "keyVaultUri": "[parameters('keyvaultUri')]",
                   "keyName": "[parameters('keyName')]",
                   "keyVersion": "[parameters('keyVersion')]"
                }
             }
          }
        }
      }
    }
  ],
  "outputs": {
    "workspace": {
      "type": "object",
      "value": "[reference(resourceId('Microsoft.Databricks/workspaces', parameters('workspaceName')))]"
    }
  }
}

If you use another template already, you can merge this template's parameters, resources, and outputs into your existing template.

To use this template to create or update a workspace, choose one of these deployment options:

Apply a template with the Azure CLI
Apply a template with the Azure portal

Apply a template with the Azure CLI

To create a new workspace with Azure CLI, run the following command:

az deployment group create --resource-group <resource-group-name>  \
                           --template-file <file-name>.json \
                           --parameters workspaceName=<new-workspace-name> \
                           keyvaultUri=<keyvaultUrl> \
                           keyName=<keyName> keyVersion=<keyVersion>

To update an existing workspace to use a customer-managed key workspace (or to rotate the existing key) using Azure CLI:

If your ARM template that deployed the workspace never added customer-managed keys, add the resources.properties.encryption section and its related parameters. See the template earlier in this article.
1. Add the resources.properties.encryption section from the template.
2. In the parameters section, add three new parameters keyvaultUri, keyName, and keyVersion from the template.
3. In the parameters section, remove "type": "string", from the template.
Run the same command as for creating a new workspace. As long as the resource group name and the workspace name are identical to your existing workspace, this command updates the existing workspace rather than creating a new workspace.
```
az deployment group create --resource-group <existing-resource-group-name>  \
                           --template-file <file-name>.json \
                           --parameters workspaceName=<existing-workspace-name> \
                           keyvaultUri=<keyvaultUrl> \
                           keyName=<keyName> keyVersion=<keyVersion>
```
Other than changes in the key-related parameters, use the same parameters that were used for creating the workspace.

Important

If you rotate the key, you must keep the old key available for 24 hours.

Apply a template with the Azure portal

To use the template in the Azure portal to create or update a workspace:

Go to the Custom deployment page.
Click Build your own template in the editor.
Paste in the JSON.
Click Save.
Fill in the parameters.

To update an existing workspace, use the same parameters that you used to create the workspace. To add a key for the first time, add the three key-related parameters. To rotate the key, change some or all of the key-related parameters. Ensure the resource group name and the workspace name are identical to your existing workspace. If they are the same, this command updates the existing workspace rather than creating a new workspace.

Other than changes in the key-related parameters, use the same parameters that were used for creating the workspace.
Click Review + Create.
If there are no validation issues, click Create.

Important

If you rotate the key, you must keep the old key available for 24 hours.

For more details, see the Azure article Quickstart: Create and deploy ARM templates by using the Azure portal.

Step 4 (optional): Re-import notebooks

After you initially add a key for managed services for an existing workspace, only future write operations use your key. Existing data is not re-encrypted.

You can export all notebooks and then re-import them so the key that encrypts the data is protected and controlled by your key. You can use the Export and Import Workspace APIs.

Rotate the key at a later time

If you are already using a customer-managed key for managed services, you can update the workspace with a new key version, or an entirely new key. This is called key rotation.

Create a new key or rotate your existing key in the Key Vault. See Step 1: Set up a Key Vault.

Ensure the new key has the proper permissions.
Confirm that your template has the correct API version. It must be equal to or higher than 2021-04-01-preview.
Update the workspace with your new key using the portal, CLI, or PowerShell. See Step 3: Add a key to a workspace and follow instructions for workspace update. Be sure that you use the same values for the resource group name and the workspace name so it updates the existing workspace, rather than creating a new workspace. Other than changes in the key-related parameters, use the same parameters that were used for creating the workspace.

Important

If you rotate the key, you must keep the old key available for 24 hours.
Optionally export and re-import existing notebooks to ensure that all your existing notebooks use your new key.

Troubleshooting

Accidental deletion of a key

If you delete your key in the Azure Key Vault, the workspace login will start failing and no notebooks will be readable by Azure Databricks. To avoid this, we recommend that you enable soft deletes. This option ensures that if a key is deleted, it can be recovered within a 30 day period. If soft delete is enabled, you can simply re-enable the key in order to resolve the issue.

Key update failure due to Key Vault permissions

If you have trouble creating your workspace, check if your Key Vault has correct permissions. The error that is returned from Azure may not correctly indicate this as the root cause. Also, the required permissions are get, wrapKey, and unwrapKey. See Step 1: Set up a Key Vault.

Lost keys are unrecoverable

If you lose your key and cannot recover it, all the notebook data encrypted by the key is unrecoverable.

Enable customer-managed keys for managed services

Requirements

Step 1: Set up a Key Vault

Use the Azure portal

Use the Azure CLI

Use Azure Powershell

Step 2: Prepare a key

Use Azure CLI

Use Azure Powershell

Step 3: Add a key to a workspace

Use the Azure portal with no template

Use the Azure CLI with no template

Use Powershell with no template

Apply changes with an ARM template

Apply a template with the Azure CLI

Apply a template with the Azure portal

Step 4 (optional): Re-import notebooks

Rotate the key at a later time

Troubleshooting

Accidental deletion of a key

Key update failure due to Key Vault permissions

Lost keys are unrecoverable

Additional resources