Enable workload identity federation for CircleCI

Databricks OAuth token federation, also known as OpenID Connect (OIDC), allows your automated workloads running outside of Databricks to securely access Databricks without the need for Databricks secrets. See Authenticate access to Azure Databricks using OAuth token federation.

To enable workload identity federation for CircleCI:

1. Create a federation policy

2. Configure the CircleCI YAML

After you enable workload identity federation, the Databricks SDKs and the Databricks CLI automatically fetch workload identity tokens from CircleCI and exchange them for Databricks OAuth tokens.

Create a federation policy

First, create a custom workload identity federation policy. For instructions, see Configure a service principal federation policy. For CircleCI, set the following values for the policy:

• Issuer URL: https://oidc.circleci.com/org/<org_id>, where <org-id> is your organization ID
• Audiences: Your CircleCI organization ID
• Subject: The CircleCI project ID
• Subject claim: oidc.circleci.com/project-id

For example, the following Databricks CLI command creates a federation policy for an organization ID 1234 and a Databricks service principal numeric ID of 5581763342009999:

databricks account service-principal-federation-policy create 5581763342009999 --json '{
  "oidc_policy": {
	"issuer": "https://oidc.circleci.com/org/1234",
	"audiences": [
  	  "1234"
	],
	"subject": "5678",
  "subject_claim": "oidc.circleci.com/project-id"
  }
}'

Configure the CircleCI YAML

Next, modify the CircleCI configuration file. In addition to the following variables, set DATABRICKS_OIDC_TOKEN_ENV in your CircleCI config.yml file to instruct the Databricks SDKs or CLI to search for the token in CIRCLE_OIDC_TOKEN_V2. (You can also use the older CIRCLE_OIDC_TOKEN environment variable.)

• DATABRICKS_AUTH_TYPE: env-oidc
• DATABRICKS_HOST: your Databricks workspace URL
• DATABRICKS_CLIENT_ID: the service principal (application) ID

version: 2.1

jobs:
  build:
    docker:
  	  - image: cimg/base:current
    environment:
      DATABRICKS_HOST: https://my-workspace.cloud.databricks.com/
      DATABRICKS_CLIENT_ID: a1b2c3d4-ee42-1eet-1337-f00b44r
      DATABRICKS_OIDC_TOKEN_ENV: CIRCLE_OIDC_TOKEN
      DATABRICKS_AUTH_TYPE: env-oidc
    steps:
      - checkout
      - run:
          name: Install Databricks CLI
          command: |
            curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh
            databricks --version
      - run:
          name: Run Databricks CLI commands
          command: databricks current-user me