Set up and use Azure managed identities authentication for Azure Databricks automation

Article
09/09/2024

Follow this article's steps to authenticate managed identities for Azure resources (formerly Managed Service Identities (MSI)) to automate your Azure Databricks accounts and workspaces.

Azure automatically manages identities in Microsoft Entra ID for applications to use when connecting to resources that support Microsoft Entra ID authentication. These resources include Azure Databricks accounts and workspaces. Azure managed identities authentication for Azure Databricks uses managed identities to obtain Microsoft Entra ID tokens without having to manage any credentials.

For more information about managed identities, see What are managed identities for Azure resources?.
For more information about Azure managed identities authentication for Azure Databricks, see Azure managed identities authentication.

Note

Managed identities for Azure resources are different than Microsoft Entra ID managed service principals, which Azure Databricks also supports for authentication. To learn how to use Microsoft Entra ID managed service principals for Azure Databricks authentication instead of managed identities for Azure resources, see:

This article demonstrates how to set up and use Azure managed identities authentication as follows:

Create a user-assigned managed identity. Azure supports system-assigned and user-assigned managed identities. Databricks recommends that you use user-assigned managed identities for Azure managed identities authentication with Azure Databricks.
Assign your managed identity to your Azure Databricks account and to an Azure Databricks workspace in that account.
Create and log in to an Azure virtual machine (Azure VM). You must use a resource that supports managed identities such as an Azure VM, with a managed identity assigned to that Azure VM, to programmatically call Azure Databricks account and workspace operations.
Assign your user-assigned managed identity to your Azure VM.
Install the Databricks CLI on your Azure VM and then configure the Databricks CLI for Azure managed identities authentication for Azure Databricks by using the assigned managed identity.
Run commands with the Databricks CLI to automate your Azure Databricks account and workspace by using Azure managed identities authentication for Azure Databricks with the assigned managed identity.

Requirements

To create and assign user-assigned managed identities, you must have the minimum required Azure role-based access control (Azure RBAC) permissions in your Azure subscription. See Which Azure RBAC permissions are required to use a managed identity on a resource?.
To assign a managed identity to your Azure Databricks account, you must be an admin of that account. See Assign account admin roles to a user.
To assign a managed identity to your Azure Databricks workspace, you must be an admin of that workspace. See Assign the workspace admin role to a user using the workspace admin settings page.
To create an Azure VM and assign a user-managed identity to it, you must have at minimum the Virtual Machine Contributor and Managed Identity Operator role assignments in your Azure subscription.

Step 1: Create a user-assigned managed identity

In this step, you create a user-assigned managed identity for Azure resources. Azure supports system-assigned and user-assigned managed identities. Databricks recommends that you use user-assigned managed identities for Azure managed identities authentication with Azure Databricks. See also Manage user-assigned managed identities.

Sign in to the Azure portal.

Note

The portal to use is different depending on whether you use the Azure public cloud or a national or sovereign cloud. For more information, see National clouds.
If you have access to multiple tenants, subscriptions, or directories, click the gear (Settings) icon in the top menu to switch to the directory in which you want to create the managed identity.
In Search resources, services, and docs, search for and select the Azure service named Managed Identities.
Click + Create.
On the Basics tab, for Resource group, choose an existing resource group to add this managed identity to, or click Create new to create a new resource group to add this managed identity to. For information about resource groups, see Manage Azure resource groups by using the Azure portal.
For Region, choose the appropriate region to add this managed identity to. For information about regions, see Choose the right Azure region for you.
For Name, enter some unique name for this managed identity that's easy for you to remember.
On the Review + create tab, click Create.
Click Go to resource.
Copy the value of the Client ID field, as you will need it later for Steps 2, 3, and 8:

If you forget to copy this value, you can return to your managed identity's overview page later to get this value. To return to your managed identity's overview page, in Search resources, services, and docs, search for and select your managed identity's name. Then, on the managed identity's settings page, click Overview in the sidebar.

Step 2: Assign your managed identity to your Azure Databricks account

In this step, you give your managed identity access to your Azure Databricks account. If you do not want to give your managed identity access to your Azure Databricks account, skip ahead to Step 3.

In your Azure Databricks workspace, click your username in the top bar and click Manage account.

Alternatively, go directly to your Azure Databricks account console, at https://accounts.databricks.azure.cn.
Sign in to your Azure Databricks account, if prompted.
On the sidebar, click User management.
Click the Service principals tab.

Note

Although this tab is labeled Service principals, this tab works with managed identities as well. Azure Databricks treats managed identities as service principals in your Azure Databricks account.
Click Add service principal.
Enter some unique Name for the service principal that's easy for you to remember.
For UUID, enter the Client ID value for your managed identity from Step 1.
Click Add. Your managed identity is added as a service principal in your Azure Databricks account.
Assign any account-level permissions that you want the service principal to have:
1. On the Service principals tab, click the name of your service principal.
2. On the Roles tab, toggle to enable or disable each target role that you want this service principal to have.
3. On the Permissions tab, grant access to any Azure Databricks users, service principals, and account group roles that you want to manage and use this service principal. See Manage roles on a service principal.

Step 3: Assign your managed identity to your Azure Databricks workspace

In this step, you give your managed identity access to your Azure Databricks workspace.

If your workspace is enabled for identity federation:

In your Azure Databricks workspace, click your username in the top bar and click Settings.
Click Service principals.

Note

Although this tab is labeled Service principals, this tab works with managed identities as well. Azure Databricks treats managed identities as service principals in your Azure Databricks account.
Click Add service principal.
Select your service principal from Step 2 and click Add. Your service principal is added as a service principal in your Azure Databricks workspace.
Assign any workspace-level permissions that you want the service principal to have:
1. On the Service principals tab, click the name of your service principal.
2. On the Configurations tab, select or clear to grant or revoke each target status or entitlement that you want this service principal to have.
3. On the Permissions tab, grant access to any Azure Databricks users, service principals, and account group roles that you want to manage and use this service principal. See Manage roles on a service principal.

Skip ahead to Step 4.

If your workspace is not enabled for identity federation:

In your Azure Databricks workspace, click your username in the top bar and click Settings.
Click Service principals.

Note

Although this tab is labelled Service principals, this tab works with managed identities as well. Azure Databricks treats managed identities as service principals in your Azure Databricks workspace.
Click Add service principal.
In the Service Principal list, select Add new service principal.
For ApplicationId, enter the Client ID for your managed identity from Step 1.
Enter some Display Name that's easy for you to remember for the new service principal, and click Add. Your managed identity is added as a service principal in your Azure Databricks workspace.
Assign any workspace-level permissions that you want the service principal to have:
1. On the Service principals tab, click the name of your service principal.
2. On the Configurations tab, select or clear to grant or revoke each target status or entitlement that you want this service principal to have.
3. On the Permissions tab, grant access to any Azure Databricks users, service principals, and account group roles that you want to manage and use this service principal. See Manage roles on a service principal.

Step 4: Get the Azure resource ID for your Azure Databricks workspace

In this step, you get the resource ID that Azure assigns to your Azure Databricks workspace. You will need this Azure resource ID later to help Azure managed identities authentication determine the specific Azure resource that Azure associates with your Azure Databricks workspace.

In your Azure Databricks workspace, click your username in the top bar and click Azure Portal.
On the side pane, in the Settings section, click Properties.

In the Essentials section, copy the Id value, as you will need it later in Step 8. It should look similar to the following:

/subscriptions/<subscription-id>/resourceGroups/<resource-group-id>/providers/Microsoft.Databricks/workspaces/<workspace-id>

Step 5: Create and log in to an Azure VM

In this step, you create and log in to an Azure virtual machine (Azure VM). Azure VMs are one of the resource types that support managed identities. See also Quickstart: Create a Linux virtual machine in the Azure portal.

This Azure VM is intended only for demonstration purposes. This Azure VM uses settings that are not necessarily optimized for your ongoing usage needs. After you are done experimenting with this Azure VM, you should delete it as shown later in Step 11.

In the Azure portal that you signed in to in Step 1, in Search resources, services, and docs, search for and select the Azure service named Virtual machines.
Click + Create > Azure virtual machine.
On the Basics tab, for Resource group, choose an existing resource group to add this Azure VM to, or click Create new to create a new resource group to add this Azure VM to. For information about resource groups, see Manage Azure resource groups by using the Azure portal.
For Virtual machine name, enter some unique name for this Azure VM that's easy for you to remember.
For Region, choose the appropriate region to add this Azure VM to. For information about regions, see Choose the right Azure region for you.
For Image, choose Ubuntu Server 22.04 LTS - x64 Gen 2.
For Authentication type, select SSH public key.
For Username, enter azureuser.
For SSH public key source, leave the default of Generate new key pair.
For Key pair name, enter myKey.
For Public inbound ports, select Allow selected ports.
For Select inbound ports, select HTTP (80) and SSH (22).
Leave the remaining defaults.
On the Review + create tab, click Create.
Click Download private key and create resource. Your key file is download to your local development machine as myKey.pem. Make a note of where this myKey.pem file is downloaded, as you will need it to log in to the Azure VM later in this Step.

If you lose your key file, you can return to your Azure VM's settings page later to get a replacement key file. To return to your Azure VM's settings page, in Search resources, services, and docs, search for and select your Azure VM's name. To get a replacement key file from your Azure VM's settings page, do the following:
1. In the Help section on the side pane, click Reset password.
2. Select Add SSH public key.
3. For Key pair name, enter some unique name.
4. Click Update.
5. Click Download + create. Your key file is download with a .pem extension. Make a note of where this .pem file is downloaded, as you will need it to log in to the Azure VM later in this Step.
After the Azure VM is created, click Go to resource.
Copy the value of the Public IP address field, as you will need it to log in to the Azure VM later in this Step.

If you forget to copy this value, you can return to your Azure VM's overview page later to get this value. To return to your Azure VM's overview page, in Search resources, services, and docs, search for and select your Azure VM's name. Then, on the Azure VM's settings page, click Overview in the sidebar and look for the Public IP address field.
If your local development machine runs Linux, macOS, or WSL on Windows, check that you have read-only access to the private key you just downloaded. To do this, run the following command from your local development machine's terminal or command prompt. In this command, replace the following values:
- Replace </path/to> with the path to your downloaded .pem file.
- Replacce myKey.pem with the filename of your downloaded .pem file.
```
chmod 400 </path/to>/myKey.pem
```
Log in to the Azure VM. To do this, from your local development machine's terminal or command prompt, run the following command. In this command, replace the following values:
- Replace </path/to> with the path to your downloaded .pem file.
- Replacce myKey.pem with the filename of your downloaded .pem file.
- Replace <public-ip-address> with the value of the Public IP address field that you copied earlier in this Step.
```
ssh -i </path/to>/myKey.pem azureuser@<public-ip-address>
```
For example:
```
ssh -i ./myKey.pem azureuser@192.0.2.0
```
If you have never connected to this Azure VM before, you are prompted to verify the the host's fingerprint. To do this, follow the on-screen prompts. Databricks recommends that you always validate the host's fingerprint.
The terminal or command prompt changes to azureuser@<your-azure-vm-name>:~$.
To exit the Azure VM at any time, run the command logout or exit. The terminal or command prompt then changes back to normal.

Step 6: Assign your managed identity to your Azure VM

In this step, you associate your managed identity with your Azure VM. This enables Azure to use the managed identity for authentication as needed while the Azure VM is running. See also Assign a user-assigned managed identity to an existing VM.

In the Azure portal that you signed in to in Step 1, on the Azure VM's settings page, in the Settings section on the side pane, click Identity.

To return to your Azure VM's overview page if you closed it earlier, in Search resources, services, and docs, search for and select your Azure VM's name.
On the User assigned tab, click + Add.
Select your user-assigned managed identity that you created in Step 1, and click Add.

Step 7: Install the Databricks CLI on your Azure VM

In this step, you install the Databricks CLI so that you can use it to run commands that automate your Azure Databricks accounts and workspaces.

Tip

You can also use the Databricks Terraform provider or the Databricks SDK for Go along with Azure managed identities authentication to automate your Azure Databricks accounts and workspaces by running HCL or Go code. See the Databricks SDK for Go and Azure managed identities authentication.

With the terminal or command prompt still open and logged in to your Azure VM from Step 5, install the Databricks CLI by running the following two commands:
```
sudo apt install unzip
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh
```
Confirm that the Databricks CLI is installed by running the following command, which prints the installed Databricks CLI version:
```
databricks -v
```

Step 8: Configure the Databricks CLI for Azure managed identities authentication

In this step, you set up the Databricks CLI to use Azure managed identities authentication for Azure Databricks with your managed identity's settings. To do this, you create a file with a default filename in a default location where the Databricks CLI expects to find the authentication settings it needs.

With the terminal or command prompt still open and logged in to your Azure VM from Step 5, use the vi text editor to create and open a file named .databrickscfg for editing in the logged-in user's home directory, by running the following command:
```
vi ~/.databrickscfg
```
Begin editing the .databrickscfg file by pressing the editor key combination Esc followed by i. The command prompt disappears, the vi editor starts, and the word -- INSERT -- appears at the bottom of the editor to indicate that the .databrickscfg file is in editing mode.
Enter the following content. In this content, replace the following values:
- Replace <account-console-url> with your Azure Databricks account console URL, such as https://accounts.databricks.azure.cn.
- Replace <account-id> with your Azure Databricks account ID. See Locate your account ID.
- Replace <azure-managed-identity-application-id> with Client ID value for your managed identity from Step 1.
- Replace <workspace-url> with your per-workspace URL, for example https://adb-1234567890123456.7.databricks.azure.cn.
- Replace <azure-workspace-resource-id> with the Azure resource ID from Step 4.
- You can replace the suggested configuration profile names AZURE_MI_ACCOUNT and AZURE_MI_WORKSPACE with different configuration profile names if desired. These specific names are not required.
If you do not want to run account-level operations, you can omit the [AZURE_MI_ACCOUNT] section in the following content.
```
[AZURE_MI_ACCOUNT]
host            = <account-console-url>
account_id      = <account-id>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

[AZURE_MI_WORKSPACE]
host                        = <workspace-url>
azure_workspace_resource_id = <azure-workspace-resource-id>
azure_client_id             = <azure-managed-identity-application-id>
azure_use_msi               = true
```
Save your edits to the .databrickscfg file by pressing the editor key combination Esc, followed by entering :wq, followed by Enter. The vi editor closes and the command prompt reappears.

Step 9: Run an account-level command

In this step, you use the Databricks CLI to run a command that automates the Azure Databricks account that was configured in Step 8.

If you do not want to run account-level commands, skip ahead to Step 10.

With the terminal or command prompt still open and logged in to your Azure VM from Step 5, run the following command to list all available users in your Azure Databricks account. If you renamed AZURE_MI_ACCOUNT in Step 8, be sure to replace it here.

databricks account users list -p AZURE_MI_ACCOUNT

Step 10: Run a workspace-level command

In this step, you use the Databricks CLI to run a command that automates the Azure Databricks workspace that was configured in Step 8.

With the terminal or command prompt still open and logged in to your Azure VM from Step 5, run the following command to list all available users in your Azure Databricks workspace. If you renamed AZURE_MI_WORKSPACE in Step 8, be sure to replace it here.

databricks users list -p AZURE_MI_WORKSPACE

Step 11: Clean up

This step is optional. It deletes the Azure VM to save costs, and it deletes the managed identity if you no longer want to keep using it. This Step also removes the deleted managed identity from your Azure Databricks account and workspace for completeness.

Delete the Azure VM

If the terminal or command prompt is still open and logged in to your Azure VM from Step 5, exit the Azure VM by running the command logout or exit. The terminal or command prompt then changes back to normal.
In the Azure portal that you signed in to in Step 1, return to your Azure VM's overview page if you closed it earlier. To do this, in Search resources, services, and docs, search for and select your Azure VM's name.
On your Azure VM's overview page's menu bar, click Delete.
Select the I have read and understand checkbox, and click Delete.

Delete the managed identity from your Azure subscription

In the Azure portal that you signed in to in Step 1, return to your managed identity's overview page if you closed it earlier. To do this, in Search resources, services, and docs, search for and select your managed identity's name.
On your managed identity's overview page's menu bar, click Delete.
Select the I have read and understand checkbox, and click Delete.

Remove the managed identity from your Azure Databricks account

In your Azure Databricks account, on the sidebar, click User management.
Click the Service principals tab.
Click the name of the service principal that you added in Step 2. If the service principal's name is not visible, use Filter service principals to find it.
Click the ellipses button, and then click Delete.
Click Confirm delete.

Remove the managed identity from your Azure Databricks workspace

In your Azure Databricks workspace, click your username in the top bar and click Settings.
Click the Service principals tab.
Click the name of the service principal that you added in Step 3. If the service principal's name is not visible, use Filter service principals to find it.
Click Delete.
In the confirmation dialog, click Delete.