Authentication for the Databricks CLI

This article describes how to set up authentication between the Databricks CLI and your Azure Databricks accounts and workspaces. See What is the Databricks CLI?.

This article assumes that you have already installed the Databricks CLI. See Install or update the Databricks CLI.

Before you can run Databricks CLI commands, you must set up authentication between the Databricks CLI and your Azure Databricks accounts, workspaces, or a combination of these, depending on the types of CLI commands that you want to run.

You must authenticate the Databricks CLI to the relevant resources at run time in order to run Azure Databricks automation commands within an Azure Databricks account or workspace. Depending on whether you want to call Azure Databricks workspace-level commands, Azure Databricks account-level commands, or both, you must authenticate to the Azure Databricks workspace, account, or both. For a list of Azure Databricks workspace-level and account-level CLI command groups, run the command databricks -h. For a list of Azure Databricks workspace-level and account-level REST API operations that the Databricks CLI commands cover, see the Databricks REST API.

The following sections provide information about how to set up authentication between the Databricks CLI and Azure Databricks:

• Azure Databricks personal access token authentication

• Azure managed identities authentication

• Microsoft Entra ID service principal authentication

• Azure CLI authentication

• Authentication order of evaluation

Azure Databricks personal access token authentication

Azure Databricks personal access token authentication uses an Azure Databricks personal access token to authenticate the target Azure Databricks entity, such as an Azure Databricks user account. See Azure Databricks personal access token authentication.

To create a personal access token, do the following:

1. In your Azure Databricks workspace, click your Azure Databricks username in the top bar, and then select Settings from the drop down.
2. Click Developer.
3. Next to Access tokens, click Manage.
4. Click Generate new token.
5. (Optional) Enter a comment that helps you to identify this token in the future, and change the token's default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).
6. Click Generate.
7. Copy the displayed token to a secure location, and then click Done.

To configure and use Azure Databricks personal access token authentication, do the following:

1. Use the Databricks CLI to run the following command:

   databricks configure
   

2. For the prompt Databricks Host, enter your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.databricks.azure.cn.

3. For the prompt Personal Access Token, enter the Azure Databricks personal access token for your workspace.

   After you enter your Azure Databricks personal access token, a corresponding configuration profile is added to your .databrickscfg file. If the Databricks CLI cannot find this file in its default location, it creates this file for you first and then adds this configuration profile to the new file. The default location for this file is in your ~ (your user home) folder on Unix, Linux, or macOS, or your %USERPROFILE% (your user home) folder on Windows.

4. You can now use the Databricks CLI's --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks clusters list -p <configuration-profile-name>.

Azure managed identities authentication

Azure managed identities authentication uses managed identities for Azure resources (formerly Managed Service Identities (MSI)) for authentication. See What are managed identities for Azure resources?. See also Azure managed identities authentication.

To create an Azure user-assigned managed identity, do the following:

1. Create or identify an Azure VM and install the Databricks CLI on it, then assign your managed identity to your Azure VM and your target Azure Databricks accounts, workspaces, or both. See Set up and use Azure managed identities authentication for Azure Databricks automation.

2. On the Azure VM, create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

   For account-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host            = <account-console-url>
   account_id      = <account-id>
   azure_client_id = <azure-managed-identity-application-id>
   azure_use_msi   = true
   

   For workspace-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host            = <workspace-url>
   azure_client_id = <azure-managed-identity-application-id>
   azure_use_msi   = true
   

   For workspace-level commands, if the target identity has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID, instead of host along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource.

   Note

   The default location for the .databrickscfg file is in the user's home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

3. On the Azure VM, use the Databricks CLI's --profile or -p option followed by the name of your configuration profile to set the profile for Databricks to use, for example, databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

   Tip

   You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Microsoft Entra ID service principal authentication

Microsoft Entra ID service principal authentication uses the credentials of a Microsoft Entra ID service principal to authenticate. To create and manage service principals for Azure Databricks, see Manage service principals. See also MS Entra service principal authentication.

To configure and use Microsoft Entra ID service principal authentication, you must have the Azure CLI authentication installed locally. You must also do the following:

1. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

   For account-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host                = <account-console-url>
   account_id          = <account-id>
   azure_tenant_id     = <azure-service-principal-tenant-id>
   azure_client_id     = <azure-service-principal-application-id>
   azure_client_secret = <azure-service-principal-client-secret>
   

   For workspace-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host                = <workspace-url>
   azure_tenant_id     = <azure-service-principal-tenant-id>
   azure_client_id     = <azure-service-principal-application-id>
   azure_client_secret = <azure-service-principal-client-secret>
   

   For workspace-level commands, if the target Microsoft Entra ID service principal has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID, instead of host along with the workspace URL. In this case, the target Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource.

   Note

   The default location for the .databrickscfg file is in the user's home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

2. Use the Databricks CLI's --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

   Tip

   You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Azure CLI authentication

Azure CLI authentication uses the Azure CLI to authenticate the signed-in entity. See also Azure CLI authentication.

To configure Azure CLI authentication, you must do the following:

1. Have the Azure CLI installed locally.

2. Use the Azure CLI to log in to Azure Databricks by running the az login command. See Azure CLI login with an Azure Databricks user account.

3. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

   For account-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host       = <account-console-url>
   account_id = <account-id>
   

   For workspace-level commands, set the following values in your .databrickscfg file:

   [<some-unique-configuration-profile-name>]
   host = <workspace-url>
   

   Note

   The default location for the .databrickscfg file is in the user's home directory. This is ~ for Linux and macOS, and %USERPROFILE% for Windows.

4. Use the Databricks CLI's --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

   Tip

   You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Authentication order of evaluation

Whenever the Databricks CLI needs to gather the settings that are required to attempt to authenticate with an Azure Databricks workspace or account, it searches for these settings in the following locations, in the following order.

1. For any command run from the bundle working directory (the bundle root and any nested path), the values of fields within a project's bundle setting files. (Bundle setting files do not support the direct inclusion of access credential values.)
2. The values of environment variables, as listed within this article and in Environment variables and fields for client unified authentication.
3. Configuration profile field values within the .databrickscfg file, as listed previously within this article.

Whenever the Databricks CLI finds the required settings that it needs, it stops searching in other locations. For example:

• The Databricks CLI needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is set, and the .databrickscfg file also contains multiple personal access tokens. In this example, the Databricks CLI uses the value of the DATABRICKS_TOKEN environment variable and does not search the .databrickscfg file.
• The databricks bundle deploy -t dev command needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project's bundle settings file contains a dev environment declaration that references through its profile field a configuration profile named DEV. In this example, the Databricks CLI searches the .databrickscfg file for a profile named DEV and uses the value of that profile's token field.
• The databricks bundle run -t dev hello-job command needs the value of an Azure Databricks personal access token. A DATABRICKS_TOKEN environment variable is not set, and the .databrickscfg file contains multiple personal access tokens. The project's bundle settings file contains a dev environment declaration that references through its host field a specific Azure Databricks workspace URL. In this example, the Databricks CLI searches through the configuration profiles within the .databrickscfg file for a profile that contains a host field with a matching workspace URL. The Databricks CLI finds a matching host field and then uses that profile's token field value.