Enable Azure Private Link back-end and front-end connections
This article summarizes the use of Azure Private Link to enable private connectivity between users and their Databricks workspaces, and also between clusters on the classic compute plane and the core services on the control plane within the Databricks workspace infrastructure.
Overview
Private Link provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing the traffic to the public network. Azure Databricks supports the following Private Link connection types:
- Front-end Private Link, also known as user to workspace: A front-end Private Link connection allows users to connect to the Azure Databricks web application, REST API, and Databricks Connect API over a VNet interface endpoint. The front-end connection is also used by JDBC/ODBC and Power BI integrations. The network traffic for a front-end Private Link connection between a transit VNet and the workspace's Azure Databricks control plane traverses the Microsoft backbone network.
- Back-end Private Link, also known as compute plane to control plane: Databricks Runtime clusters in a customer-managed VNet (the compute plane) connect to an Azure Databricks workspace's core services (the control plane) in the Azure Databricks cloud account. This enables private connectivity from the clusters to the secure cluster connectivity relay endpoint and REST API endpoint.
- Browser authentication private endpoint: To support private front-end connections to the Azure Databricks web application for clients that have no public internet connectivity, you must add a browser authentication private endpoint to support single sign-on (SSO) login callbacks to the Azure Databricks web application from Microsoft Entra ID. If you allow connections from your network to the public internet, adding a browser authentication private endpoint is recommended but not required. A browser authentication private endpoint is a private connection with sub-resource type
browser_authentication
.
If you implement Private Link for both front-end and back-end connections, you can optionally mandate private connectivity for the workspace, which means Azure Databricks rejects any connections over the public network. If you decline to implement both front-end or back-end connection types, you cannot enforce this requirement.
Sample Unity Catalog datasets and Azure Databricks datasets are not available when back-end Private Link is configured. See Sample datasets.
Most of this article is about creating a new workspace, but you can enable or disable Private Link on an existing workspace. See Enable or disable Azure Private Link on an existing workspace.
Terminology
The following table describes important terminology.
Terminology | Description |
---|---|
Azure Private Link | An Azure technology that provides private connectivity from Azure VNets and on-premises networks to Azure services without exposing the traffic to the public network. |
Azure Private Link service | A service that can be the destination for a Private Link connection. Each Azure Databricks control plane instance publishes an Azure Private Link service. |
Azure private endpoint | An Azure private endpoint enables a private connection between a VNet and a Private Link service. For front-end and back-end connectivity, the target of an Azure private endpoint is the Azure Databricks control plane. |
For general information about private endpoints, see the Azure article What is a private endpoint?.
Choose standard or simplified deployment
There are two types of Private Link deployment that Azure Databricks supports, and you must choose one:
- Standard deployment (recommended): For improved security, Databricks recommends you use a separate private endpoint for your front-end connection from a separate transit VNet. You can implement both front-end and back-end Private Link connections or just the back-end connection. Use a separate VNet to encapsulate user access, separate from the VNet that you use for your compute resources in the classic compute plane. Create separate Private Link endpoints for back-end and front-end access. Follow the instructions in Enable Azure Private Link as a standard deployment.
- Simplified deployment: Some organizations cannot use the standard deployment for various network policy reasons, such as disallowing more than one private endpoint or discouraging separate transit VNets. You can alternatively use the Private Link simplified deployment. No separate VNet separates user access from the VNet you use for your compute resources in the classic compute plane. Instead, a transit subnet in the compute plane VNet is used for user access. There is only a single Private Link endpoint. Typically, both front-end and back-end connectivity are configured. You can optionally configure only the back-end connection. You cannot choose to use only the front-end connections in this deployment type. Follow the instructions in Enable Azure Private Link as a simplified deployment.
Requirements
Azure subscription
Your Azure Databricks workspace must be on the Premium plan.
Azure Databricks workspace network architecture
- Your Azure Databricks workspace must use VNet injection to add any Private Link connection (even a front-end-only connection).
- If you implement the back-end Private Link connection, your Azure Databricks workspace must use secure cluster connectivity (SCC / No Public IP / NPIP).
- You need a VNet that satisfies the requirements of VNet injection.
- You must define two subnets (referred to in the UI as the public subnet and the private subnet). The VNet and subnet IP ranges that you use for Azure Databricks defines the maximum number of cluster nodes that you can use at one time.
- To implement front-end Private Link, back-end Private Link, or both, your workspace VNet needs a third subnet that contains the Private Link endpoint and its IP address range must not overlap with the range of your other workspace subnets. This article refers to this third subnet as the private endpoint subnet. Examples and screenshots assume the subnet name
private-link
. This can be as small as CIDR range/27
. Do not define any NSG rules for a subnet that contains private endpoints. - If you use the UI to create objects, you need to create the network and subnets manually before creating the Azure Databricks workspace. If you want to use a template, the template that Azure Databricks provides creates a VNet and appropriate subnets for you, including the two regular subnets plus another for private endpoints.
- If you have a Network Security Groups policy enabled on the private endpoint, you must allow ports 443, 6666, 3306, and 8443-8451 for Inbound Security Rules in the network security group on the subnet where the private endpoint is deployed.
- To connect between your network and the Azure portal and its services, you might need to add Azure portal URLs to your allowlist. See Allow the Azure portal URLs on your firewall or proxy server
Front-end connection network architecture
For front-end Private Link only, for users to access the workspace from your on-premises network, you must add private connectivity from that network to your Azure network. Add this connectivity before configuring Private Link. The details vary based on whether you choose the Private Link standard deployment or the simplified deployment.
- For the standard deployment, you would create or use an existing transit VNet, sometimes called a bastion VNet or hub VNet. This VNet must be reachable from the on-premises user environment using Expressroute or a VPN gateway connection. For front-end Private Link, Databricks recommends creating a separate VNet for your connectivity to the control plane, rather than sharing the workspace VNet. Note that the transit VNet and its subnet can be in the same region, zone, and resource group as your workspace VNet and its subnets, but they do not have to match. Create a resource group for the separate transit VNet and use a different private DNS zone for that private endpoint. If you use two separate private endpoints, you cannot share the DNS zone.
- For the simplified deployment, you create a transit subnet in your workspace VNet. In this deployment, the transit subnet does not have a separate private endpoint. The transit subnet in the workspace VNet uses a single private endpoint for both back-end and front-end connections.
Azure user permissions
As Azure user, you must have read/write permissions sufficient to:
- Provision a new Azure Databricks workspace.
- Create Azure Private Link endpoints in your workspace VNet and also (for front-end usage) your transit VNet.
If the user who created the private endpoint for the transit VNet does not have owner/contributor permissions for the workspace, then a separate user with owner/contributor permissions for the workspace must manually approve the private endpoint creation request.
Enable or disable Azure Private Link on an existing workspace
You can enable Private Link on an existing workspace. The upgrade requires that the workspace uses VNet Injection, secure cluster connectivity, and Premium pricing tier. You can update to secure cluster connectivity and to the Premium pricing tier during the update.
You can use an ARM template or azurerm
Terraform provider version 3.41.0+. You can use the Azure portal to apply a custom template and modify the parameter in the UI. However, there is no Azure Portal user interface support for this upgrade on the Azure Databricks workspace instance itself.
If something goes wrong with the upgrade and you can repeat the workspace update step but instead set the fields to disable Private Link.
Although the focus of this section is enabling Private Link on an existing workspace, you can disable it on an existing workspace by using the same workspace update call with the ARM template or a Terraform update. See to the step Step 4: Apply the workspace update for details.
Step 1: Read the requirements and documentation on this page
Before you attempt an upgrade to Private Link there are important concepts and requirements that you should read:
- Read this article including concepts and requirements before proceeding.
- Determine whether you want to use the standard deployment or the simplified deployment.
- On the page for standard deployment or the simplified deployment (whichever approach you use), carefully review the page including the various scenarios. Find the scenario that matches your use case. Write down which values you intend to use for
publicNetworkAccess
andrequiredNsgRules
. For the recommended configuration of both front-end and back-end Private Link with front-end connectivity locked down, use the settingspublicNetworkAccess=Disabled
andrequiredNsgRules=NoAzureDatabricksRules
Step 2: Stop all compute resources
Before attempting this upgrade, you must stop all compute resources such as clusters, pools, or classic SQL warehouses. No workspace compute resources can be running or the upgrade attempt fails. Databricks recommends planning the timing of the upgrade for down time.
Important
Do not attempt to start any compute resources during the update. If Azure Databricks determines that compute resources were started (or are still starting), Azure Databricks terminates them after the update.
Step 3: Create subnet and private endpoints
Add a subnet to your workspace VNet for your back-end private endpoints.
Open the article for standard deployment or the simplified deployment (whichever approach you use).
Follow the instructions on that page to create the private endpoints that match your type of deployment.
Create all your private endpoints for back-end support before doing the workspace update.
For UI access, create a private endpoint with subresource
databricks_ui_api
to support SSO from your transit VNet. If you have more than one transit VNet that accesses the workspace for front-end access, create multiple private endpoint with subresourcedatabricks_ui_api
.
Step 4: Apply the workspace update
Instead of creating a new workspace, you need to apply the workspace update.
You must update the publicNetworkAccess
and requiredNsgRules
parameters to the values that you chose in a previous step.
Use one of these methods:
Apply an update using Terraform
Use Azure portal UI (with no template)
Go to your Azure Databricks Service instance in the Azure portal.
In the left navigation under Settings, click Networking.
Set Allow Public Network Access and Required NSG Rules to the appropriate values based on your use case. See Step 1: Read the requirements and documentation on this page.
Note
To enable Azure Private Link, the workspace must enable secure cluster connectivity (sometimes called No Public IP). If it's not yet enabled, at the same time as the Private Link update, you can choose to enable secure cluster connectivity by setting No Public IP to True. However, Azure Databricks recommends enabling secure cluster connectivity as a separate step before enabling Private Link so you can separately validate success.
Click Save.
The network update might take over 15 minutes to complete.
Apply an updated ARM template using Azure portal
Note
If your managed resource group has a custom name, you must modify the template accordingly. Contact your Azure Databricks account team for more information.
Copy the following upgrade ARM template JSON:
{ "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "location": { "defaultValue": "[resourceGroup().location]", "type": "String", "metadata": { "description": "Location for all resources." } }, "workspaceName": { "type": "String", "metadata": { "description": "The name of the Azure Databricks workspace to create." } }, "apiVersion": { "defaultValue": "2023-02-01", "allowedValues": [ "2018-04-01", "2020-02-15", "2022-04-01-preview", "2023-02-01" ], "type": "String", "metadata": { "description": "2018-03-15 for 'full region isolation control plane' and 2020-02-15 for 'FedRAMP certified' regions" } }, "publicNetworkAccess": { "defaultValue": "Enabled", "allowedValues": [ "Enabled", "Disabled" ], "type": "String", "metadata": { "description": "Whether the workspace allows access from the public Internet" } }, "requiredNsgRules": { "defaultValue": "AllRules", "allowedValues": [ "AllRules", "NoAzureDatabricksRules" ], "type": "String", "metadata": { "description": "The security rules that are applied to the security group of the Vnet" } }, "enableNoPublicIp": { "defaultValue": true, "type": "Bool" }, "pricingTier": { "defaultValue": "premium", "allowedValues": [ "premium", "standard", "trial" ], "type": "String", "metadata": { "description": "The pricing tier of workspace." } }, "privateSubnetName": { "defaultValue": "private-subnet", "type": "String", "metadata": { "description": "The name of the private subnet." } }, "publicSubnetName": { "defaultValue": "public-subnet", "type": "String", "metadata": { "description": "The name of the public subnet." } }, "vnetId": { "type": "String", "metadata": { "description": "The virtual network Resource ID." } } }, "variables": { "managedResourceGroupName": "[concat('databricks-rg-', parameters('workspaceName'), '-', uniqueString(parameters('workspaceName'), resourceGroup().id))]", "managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', variables('managedResourceGroupName'))]" }, "resources": [ { "type": "Microsoft.Databricks/workspaces", "apiVersion": "[parameters('apiVersion')]", "name": "[parameters('workspaceName')]", "location": "[parameters('location')]", "sku": { "name": "[parameters('pricingTier')]" }, "properties": { "ManagedResourceGroupId": "[variables('managedResourceGroupId')]", "publicNetworkAccess": "[parameters('publicNetworkAccess')]", "requiredNsgRules": "[parameters('requiredNsgRules')]", "parameters": { "enableNoPublicIp": { "value": "[parameters('enableNoPublicIp')]" }, "customVirtualNetworkId": { "value": "[parameters('vnetId')]" }, "customPublicSubnetName": { "value": "[parameters('publicSubnetName')]" }, "customPrivateSubnetName": { "value": "[parameters('privateSubnetName')]" } } } } ] }
Go to the Azure portal Custom deployment page.
Click Build your own template in the editor.
Paste in the JSON for the template that you copied.
Click Save.
To enable Private Link, set
publicNetworkAccess
andrequiredNsgRules
parameters according to your use case.To disable Private Link, set
publicNetworkAccess
totrue
and setrequiredNsgRules
toAllRules
.For other fields, use the same parameters that you used to create the workspace, such as subscription, region, workspace name, subnet names, resource ID of the existing VNet.
Important
The resource group name, workspace name, and subnet names must be identical to your existing workspace so that this command updates the existing workspace rather than creating a new workspace.
Click Review + Create.
If there are no validation issues, click Create.
The network update might take over 15 minutes to complete.
Apply an update using Terraform
For workspaces created with Terraform, you can update the workspace to use Private Link.
Important
You must use terraform-provider-azurerm
version 3.41.0 or later, so upgrade your Terraform provider version as needed. Earlier versions attempt to recreate the workspace if you change any of these settings.
The high-level steps are:
Change the following workspace settings:
public_network_access_enabled
: Set totrue
(Enabled) orfalse
(Disabled)network_security_group_rules_required
: Set toAllRules
orNoAzureDatabricksRules
.
The network update might take over 15 minutes to complete.
Create your private endpoints.
For a detailed guide for how to enable Private Link and create the private endpoints:
- For a standard deployment, see the Terraform guide for Azure Private Link standard deployment
- For a simplified deployment, see the Terraform guide for Azure Private Link simplified deployment
The network update might take over 15 minutes to complete.
Step 5: Test user SSO authentication and back-end connectivity
Follow your main deployment page for details on how to:
- Test user SSO authentication to your workspace.
- Test back-end Private Link connection (required for a back-end connection)
Step 6: Validate the update
- Go to your Azure Databricks Service instance in the Azure portal.
- In the left navigation under Settings, click Networking.
- Confirm that the value for Allow Public Network Access matches the value that you set.
- Confirm that the value for Required NSG Rules matches the value that you set.
Failure recovery
If a workspace update fails, the workspace might be marked as a Failed state, which means that the workspace is unable to perform compute operations. To restore a failed workspace back to Active state, review the instructions in the status message of the update operation. Once you fix any issues, redo the update on the failed workspace. Repeat the steps until the update successfully completes. If you have questions, contact your Azure Databricks account team.