Run shell commands in Azure Databricks web terminal

Article
10/21/2024

The Azure Databricks web terminal provides a convenient and highly interactive way to run shell commands in a command-line interface (CLI), including Databricks CLI commands, to take actions on Databricks objects programmatically. It's especially useful for advanced use cases, such as batch operations on multiple files, which existing user interfaces (UIs) might not fully support.

Multiple users can use the web terminal on one compute.

You can use the web terminal to do the following:

Make quick file edits using Vim or Emacs.
Monitor resource usage with commands like htop (cluster usage) or nvidia-smi (GPU usage).
Run non-Spark Python scripts.
Execute file operations with shell commands such as mv and mkdir.
Install and manage libraries on compute.
Use the Databricks CLI to automate various aspects of Azure Databricks.

Requirements

Warning

Azure Databricks proxies the web terminal service from port 7681 on the compute's Spark driver. This web proxy is intended for use only with the web terminal. If the port is occupied when the compute starts or there is some sort of conflict, the web terminal might not work as expected. If other web services are launched on port 7681, compute users might be exposed to potential security exploits. Neither Databricks nor Azure is responsible for any issues that result from the installation of unsupported software on a compute.

Web terminal is disabled by default for all workspace users. To enable it, see Enable the web terminal.
CAN ATTACH TO permission on a compute.
The compute must use an access mode of Single user or No isolation shared.

Launch the web terminal

You can launch the web terminal from the compute details page or from a notebook.

From a notebook

To launch the web terminal from a notebook:

Connect the notebook to compute.
At the bottom of the notebook's right sidebar, click the Open bottom panel icon .
Alternatively, click the attached compute drop-down, hover over the attached compute, then click Web Terminal.

The web terminal opens in a panel at the bottom of the screen. The buttons at the upper-right of the panel allow you to:

Open a new terminal session in a new tab .
Reload a terminal session .
Close the bottom panel . To reopen the panel, click at the bottom of the right sidebar.

From the compute details page

To launch the web terminal from the compute details page:

On the workspace's sidebar, click Compute.
On the All-purpose compute tab, click the name of the compute.
Click Start to start the compute.
On the Apps tab, click Web Terminal.

A new tab opens with the web terminal UI and the Bash prompt.

Use web terminal

In the web terminal, you can run commands from root inside the container of the compute driver node.

Each user can have up to 100 active web terminal sessions (tabs) open. Idle web terminal sessions may time out and the web terminal web application will reconnect, resulting in a new shell process. If you want to keep your Bash session, Databricks recommends using tmux.

Run Databricks CLI commands

You can also use the web terminal to run Databricks CLI commands. The available CLI is always the latest version, and authentication is based on the current user. The compute must meet the following requirements:

The cluster must have Databricks Runtime 15.0 or above installed.
The workspace must not be enabled for Private Link.

Launch the web terminal and run the following command to output information about the current user:

   databricks current-user me

Bundle commands are also available, which allows you to create and manage your Databricks Asset Bundles directly from the web terminal within the Databricks workspace. For example, to create, deploy, and run a simple bundle using the default template:

From the web terminal root, navigate to your workspace home and run bundle init:
```
cd /Workspace/Users/someone@example.com
databricks bundle init
```
Accept the default template prompts, then change to the bundle directory and deploy it:
```
cd my_project
databricks bundle deploy
```
You can view the deployed my_project bundle in the Databricks workspace UI.
Finally, run the default job in the bundle:
```
databricks bundle run my_project_job
```
Navigate to Job Runs to see the running job.

Limitations

Azure Databricks does not support running Spark jobs from the web terminal.
Azure Databricks web terminal is not available in the following compute types:
- Job compute
- Compute launched with the DISABLE_WEB_TERMINAL=true environment variable set.
- Compute launched with access mode set to Shared.
- Compute launched with the Spark configuration spark.databricks.pyspark.enableProcessIsolation set to true.
There is a hard limit of 12 hours since the initial page load, after which any connection, even if active, will be terminated. You can refresh the web terminal to reconnect. Databricks recommends using tmux to preserve your shell session.
ARM compute resources cannot use web terminals to access workspace files, including files in Git folders.
Enabling Docker Container Services disables web terminal.