Networking

  • Article

This article introduces networking configurations for the deployment and management of Azure Databricks accounts and workspaces.

Important

Starting December 4, 2024, Databricks will begin charging for networking costs on serverless workloads that connect to external resources. Billing will be implemented gradually, and you might not be charged until after December 4, 2024. You won't be charged retroactively for usage before billing is enabled. After billing is enabled, you may be charged for:

  • Private connectivity to your resources over Private Link. Data processing charges for private connectivity to your resources over Private Link are waived indefinitely. Per-hour charges will apply.
  • Public connectivity to your resources over NAT gateway.
  • Data transfer charges incurred, such as when serverless compute and the target resource are in different regions.

Azure Databricks architecture overview

Azure Databricks operates out of a control plane and a compute plane.

  • The control plane includes the backend services that Azure Databricks manages in your Azure Databricks account. The web application is in the control plane.

  • The compute plane is where your data is processed. There are two types of compute planes depending on the compute that you are using.

    • For classic Azure Databricks compute, the compute resources are in your Azure subscription in what is called the classic compute plane. This refers to the network in your Azure subscription and its resources. Classic compute plan resources are in the region that your workspace is in.

To learn more about classic compute and serverless compute, see Types of compute. For additional architecture information, see Azure Databricks architecture overview.

Secure network connectivity

Azure Databricks provides a secure networking environment by default, but if your organization has additional needs, you can configure network connectivity features between the different networking connections shown in the diagram below.

Network connectivity overview diagram

  1. Users and applications to Azure Databricks: You can configure features to control access and provide private connectivity between users and their Azure Databricks workspaces. See Users to Azure Databricks networking.
  2. The control plane and the classic compute plane: Classic compute resources, such as clusters, are deployed in are in your Azure subscription and connect to the control plane. You can use classic network connectivity features to deploy classic compute plane resources in your own virtual networks and to enable private connectivity from the clusters to the control plane. See Classic compute plane networking.

You can configure Azure storage networking features, such as private endpoints to secure the connection between the classic compute plane and your Azure resources. See Grant your Azure Databricks workspace access to Azure Data Lake Storage Gen2 and Networking recommendations for Lakehouse Federation.

You can also enable firewall support for the workspace storage account to limit access to the account from authorized networks and connections.

Connectivity between the control plane and the serverless compute plane is always over the Azure network backbone and not the public internet.