Configure Azure virtual network service endpoint policies for storage access from classic compute

This page explains how to use Azure virtual network service endpoint policies to filter outbound traffic from the classic compute plane, ensuring connections are only made to specific Azure Storage accounts.

What are service endpoints and service endpoint policies?

Service endpoints and policies work together to create a mutually approved, private connection.

• Azure virtual network service endpoints: To secure traffic to Azure Storage, enable a service endpoint on your virtual network's subnet. This creates a secure, direct connection to the Azure Storage service over the Azure backbone network, keeping traffic off the public internet.
• Azure virtual network service endpoint policies: You apply a service endpoint policy to that same subnet as your service endpoint. This policy acts as a filter on top of the endpoint, allowing you to restrict connections to only the specific Azure Storage accounts you define. This combination prevents data exfiltration to unauthorized storage accounts.

How do they work?

The following image shows an overview of how to configure service endpoints and service endpoint policies:

1. Create an Azure virtual network service policy. See Step 1: Create a service endpoint policy.
2. Add storage accounts that you need to access to your policy. For example, the default workspace and Unity Catalog storage accounts. Storage accounts that use private endpoints do not need to be added to the policy.
3. Attach your service policy to the workspace subnet. See Step 2: Attach the policy to the workspace subnet.
4. Connect the Microsoft.Storage Azure virtual network service endpoint to the workspace subnet. The service endpoint policy of the subnet is applied to the service endpoint.
5. Unsanctioned storage cannot be accessed because it is not allowed by the policy.

The service endpoint routes all network traffic from your Azure Databricks workspace, allowing connections to storage accounts defined in the policy and blocking all unauthorized attempts.

Benefits of service endpoints

• Route priority and security: Service endpoints create a high-priority, direct routing path to Azure services over the Azure backbone. This optimized route overrides most user-defined routes (UDRs), helping prevent traffic from being unintentionally forced through a network virtual appliance (NVA) or to the internet. This behavior strengthens your security posture and helps prevent data exfiltration.
• Cost optimization: Because traffic to Azure services stays in the Azure backbone, you avoid charges associated with egressing through a NAT gateway or a network virtual appliance.

See Azure virtual network service endpoints and Virtual network service endpoint policies for Azure Storage for more details.

Benefits of service endpoint policies

• Global reach: While a service endpoint policy is a regional resource, the rules in it can target Azure Storage accounts in any region, subscription, or tenant. This allows a regional policy to manage global storage access.
• Additive policies: You can associate multiple policies with a single subnet. Policy assignments are additive, which means the subnet gains access to the combined set of all allowed storage accounts from all attached policies. This is useful for modeling different access requirements.

Best practices

For each Azure Databricks workspace, configure a service endpoint policy that targets its root DBFS storage account and any associated Unity Catalog external locations. You must also add the /services/Azure/Databricks service alias to this policy to allow access to Azure Databricks essential storage. To apply different rule sets, you can create additional policies for specific environments such as development and production.

Requirements

Your Azure Databricks workspace must meet the following requirements:

• Be deployed in your own Azure virtual network (VNet), also known as VNet injection. See Deploy Azure Databricks in your Azure virtual network (VNet injection).
• Use secure cluster connectivity (SCC). See Enable secure cluster connectivity.
• The workspace's resource group is unlocked, and the virtual network has no custom configurations or policies that block this feature.

Configure a service endpoint policy

Before you attach a service endpoint policy to your workspace's public subnet, create policy rules for all storage accounts that your classic compute resources access using service endpoints. Any storage account without a corresponding policy rule is blocked.

Step 1: Create a service endpoint policy

1. In the Azure portal, search for Service Endpoint Policy and select Create a service endpoint policy.
2. On the Basics tab:
   • Select the Subscription and Region that match your workspace's VNet.
   • Enter a descriptive Name for the policy, such as databricks-workspace-prod-chinaeast2.
   • Click Next: Policy definitions.
3. On the Policy definitions tab, add the Azure Databricks managed storage:
   • Click + Add an Alias.
   • Select /services/Azure/Databricks.
   • Click Add.
4. Add your own storage accounts:
   • Click + Add a resource.
   • For Scope, select Single account.
   • Add your workspace's default DBFS storage account, Unity Catalog storage accounts, and any other required storage accounts one by one.
   • You can add or remove storage accounts from the policy at any time.
5. Click Review + create, then Create.

Step 2: Attach the policy to the workspace subnet

A service endpoint policy can be attached to both the public and private subnets of your workspace, but only the public subnet communicates with storage.

1. In the Azure portal, go to your workspace's VNet.
2. In the sidebar under Settings, click Subnets.
3. Select your public subnet.
4. In the subnet settings, scroll to the Service Endpoints section.
5. From the Services drop-down menu, select Microsoft.Storage.
6. Scroll to the Service Endpoint Policies section.
7. From the Policies drop-down menu, select the policy that you created earlier.
8. Click Save.

Validation

To validate the configuration:

• Start an Azure Databricks cluster in the workspace.
• Run a workload that reads from or writes to a storage account included in your policy. The operation will succeed.
• Attempt to access a storage account that is not included in your policy. Any request to a storage account not defined in the policy will fail with an authorization error.