Notebook compute resources
This article covers the options for notebook compute resources. You can run a notebook on a Databricks cluster, serverless compute, or, for SQL commands, you can use a SQL warehouse, a type of compute-optimized for SQL analytics.
Attach a notebook to a cluster
To attach a notebook to a cluster, you need the CAN ATTACH TO cluster-level permission.
Important
As long as a notebook is attached to a cluster, any user with the CAN RUN permission on the notebook has implicit permission to access the cluster.
To attach a notebook to a cluster, click the compute selector in the notebook toolbar and select a cluster from the dropdown menu.
The menu shows a selection of clusters you have used recently or are currently running.
To select from all available clusters, click More…. Click the cluster name to display a dropdown menu, and select an existing cluster.
You can also create a new cluster by selecting Create new resource… from the dropdown menu.
Important
An attached notebook has the following Apache Spark variables defined.
Class | Variable Name |
---|---|
SparkContext |
sc |
SQLContext /HiveContext |
sqlContext |
SparkSession (Spark 2.x) |
spark |
Do not create a SparkSession
, SparkContext
, or SQLContext
. Doing so will lead to inconsistent behavior.
Use a notebook with a SQL warehouse
When a notebook is attached to a SQL warehouse, you can run SQL and Markdown cells. Running a cell in any other language (such as Python or R) throws an error. SQL cells executed on a SQL warehouse appear in the SQL warehouse's query history. The user who ran a query can view the query profile from the notebook by clicking the elapsed time at the bottom of the output.
Running a notebook requires a pro SQL warehouse. You must have access to the workspace and the SQL warehouse.
To attach a notebook to a SQL warehouse do the following:
Click the compute selector in the notebook toolbar. The dropdown menu shows compute resources that are currently running or that you have used recently. SQL warehouses are marked with .
From the menu, select a SQL warehouse.
To see all available SQL warehouses, select More… from the dropdown menu. A dialog appears showing compute resources available for the notebook. Select SQL Warehouse, choose the warehouse you want to use, and click Attach.
You can also select a SQL warehouse as the compute resource for a SQL notebook when you create a workflow or scheduled job.
SQL warehouse limitations
See Known limitations Databricks notebooks for more information.
Detach a notebook
To detach a notebook from a compute resource, click the compute selector in the notebook toolbar and hover over the attached cluster or SQL warehouse in the list to display a side menu. From the side menu, select Detach.
You can also detach notebooks from a cluster using the Notebooks tab on the cluster details page.
Tip
Azure Databricks recommends that you detach unused notebooks from clusters. This frees up memory space on the driver.