Configure back-end private connectivity to Azure Databricks

Use Azure Private Link to create a secure back-end connection for your Azure Databricks workspace. This connection secures traffic between clusters on the classic compute plane and core services on the Azure Databricks control plane.

• To enable hybrid access, front-end connectivity to Azure Databricks, see Configure private connectivity to Azure Databricks.

Why choose a back-end connection only?

Configuring a back-end Private Link connection provides critical security and compliance benefits for your data processing environment:

• Enhanced security: Prevents your Azure Databricks clusters from communicating with the control plane over the public internet, isolating your data workloads from public networks.
• Compliance requirements: Helps meet strict regulatory and corporate compliance mandates that require all internal cloud traffic to remain on a private network.
• Data exfiltration control: By securing the connection from the compute plane, where data is actively processed, you add a powerful layer of protection against data exfiltration.
• Eliminate public IP addresses: Works with secure cluster connectivity to enable a fully private compute environment with no public IP addresses on your clusters.

Architecture overview

In a back-end Private Link setup, you deploy a private endpoint directly in your workspace VNet (virtual network). This endpoint provides a private, secure route for your Azure Databricks clusters to communicate with the control plane services, such as the secure cluster connectivity relay. All traffic flows over the Microsoft Azure backbone network, never touches the public internet.

Requirements

• The workspace must be on the Premium plan.
• You have an Azure Databricks workspace deployed with VNet injection.
• You must have Azure permissions to create private endpoints and manage DNS records.

Networking configuration

• A transit VNet configured for the following:
  • It acts as the primary transit point for all user/client traffic connecting to your Azure network.
  • It provides centralized connectivity for on-premises or other external networks.
  • It manages shared services and contains the primary route for outbound internet traffic (egress).
• A dedicated subnet must exist in your workspace VNet specifically for private endpoints. If one doesn't exist, create it.
• Your private DNS zones are managed by Azure DNS.

Best practices

Azure Databricks recommends the following for a resilient and manageable setup:

• Architecture: Your network must follow the Microsoft-recommended hub-spoke architecture. See Hub-spoke network topology in Azure.
• Isolated authentication workspace: For improved resilience, create a separate browser authentication workspace inside your transit VNet. This dedicated workspace hosts the browser_authentication private endpoint and prevents a single point of failure if other workspaces are deleted.

Configuration paths

Choose the path that matches your scenario:

Option 1: Configure for a new workspace

Option 1: Configure for a new workspace

Follow these steps to deploy a new Azure Databricks workspace with a back-end Private Link connection.

Step 1: Create a resource group

1. In the Azure portal, select Resource groups and click Create.
2. Name the resource group. Verify the Region matches the region where your VNet is deployed.

Step 2: Create a VNet

Create a customer-managed VNet, or workspace VNet, in your own Azure subscription. When you choose VNet injection, Azure Databricks deploys its compute resources directly inside this private, secure network that you own. You need this VNet in Step 3.

1. Go to Virtual networks and click Create.
2. Assign it to the resource group you just created and provide a descriptive name.
3. Select the region where you want to host your Azure Databricks workspace.
4. Define the IP address space for the VNet, such as 10.10.0.0/16. You are prompted to create an initial subnet with its own range, such as 10.10.1.0/24.
5. Select Review + Create then Create.

Step 3: Create a new workspace

1. In the Azure portal, search for and select Azure Databricks. Click Create.
2. Choose the resource group you just created. Verify the Region matches the resource group and VNet.
3. Azure Databricks recommends adding "managed" as a prefix in the Managed resource group name.
4. On the Networking tab, configure the following:
   1. Deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP): Select Yes.
   2. Deploy Azure Databricks workspace in your own Virtual Network (VNet): Select Yes.
   3. Select the workspace VNet you created previously.
   4. Create two subnets for the workspace. Azure Databricks recommends naming each subnet something easily identifiable, such as private-worker-subnet or public-host-subnet.
   5. Allow Public Network Access: Select Enabled.
   6. Required NSG Rules: Select No Azure Databricks Rules.
   7. Deploy Azure Databricks workspace with NAT Gateway: Select No.

Step 4: Create a private endpoint

1. Go to Private endpoints.
   1. Under Private endpoints, click Add.
2. Configure the endpoint for databricks_ui_api:
   1. Name: Enter a descriptive name, such as private-endpoint-front-end-ui.
   2. Target sub-resource: Select databricks_ui_api.
   3. Virtual network: Select your workspace VNet.
      • The workspace VNet and its dedicated private endpoint subnet provide a secure, direct network path, allowing your workspace to access its back-end services privately.
   4. Region: Verify the region matches your Azure Databricks workspace.

After deployment, proceed to the Shared Steps.

Option 2: Configure an existing workspace

Option 2: Configure an existing workspace

Step 1: Update workspace networking settings

1. Go to your Azure Databricks workspace in the Azure portal.
2. Under Settings, click Networking.
3. Confirm that Secure cluster connectivity (No Public IP) is set to Yes.
4. Change Required NSG rules to NoAzureDatabricksRules.
5. Click Save. The network update can take over 15 minutes to complete.

Step 2: Create a private endpoint on the workspace VNet

1. From your workspace's Networking tab, select Private endpoint connections.
2. Click Private endpoint.
3. Select the resource group for the endpoint, provide a name such as my-workspace-fe-pe, and verify that the region matches your workspace.
4. On the Resources tab, set Target sub-resource to databricks_ui_api.
5. Select the VNet associated with your workspace.
6. Select your private endpoint subnet.
7. Verify that Integrate with private DNS zone is set to Yes. Azure automatically selects the privatelink.databricks.azure.cn zone. If one doesn't already exist, Azure shows a new privatelink.databricks.azure.cn zone.

Shared steps

Step 1: Verify DNS

If you use the built-in Azure private DNS integration, Azure automatically creates a DNS address record for your workspace.

1. Verify private DNS zone records:
   1. In the Azure portal, search for and navigate to the Private DNS zone named privatelink.databricks.azure.cn.
   2. Verify the following A records exist and point to the private IP addresses of your endpoints:
      1. Workspace UI/API record:
         • Name: Your unique workspace ID, like adb-xxxxxxxxxxxxxxxx.x
         • Value: The private IP address of your databricks_ui_api private endpoint.
      2. Browser authentication record:
         • Name: Choose a descriptive name like pl-auth.<your_region>.
         • Value: The private IP address of your browser_authentication private endpoint.

Step 2: Verify the back-end connection

Verify the back-end connection by launching a cluster.

1. Log in to your Azure Databricks workspace.
2. Go to Compute in the sidebar.
3. Click Create compute, give it a name, and click Create.

The cluster launch can take several minutes. A successful cluster start is a good first check, but you must run a simple query or notebook command to fully confirm that the connection works.

• If the cluster fails to start after 10-15 minutes and the cluster event log shows an error like Cluster terminated. Reason: Control Plane Request Failure, your Private Link setup has a misconfiguration. Review the steps on this page to troubleshoot the issue.