Install libraries from workspace files

  • Article

This article walks you through the steps required to upload package or requirements.txt files to workspace files and install them onto clusters in Azure Databricks. You can install libraries onto all-purpose compute or job compute.

Important

This article describes storing libraries as workspace files. This is different than workspace libraries which are deprecated.

For more information about workspace files, see Navigate the workspace.

Databricks Runtime 15.0 or above is required to upload requirements.txt files to workspace files. For full library compatibility details, see Libraries.

Load libraries to workspace files

You can load libraries to workspace files the same way you load other files.

To load a library to workspace files:

  1. Click Workspace Icon Workspace in the left sidebar.
  2. Navigate to the location in the workspace where you want to upload the library.
  3. Click the Kebab menu in the upper right and choose Import.
  4. The Import dialog appears. For Import from: choose File or URL. Drag and drop or browse to the file(s) you want to upload, or provide the URL path to the file.
  5. Click Import.

Install libraries from workspace files onto a cluster

When you install a library onto a cluster, all notebooks running on that cluster have access to the library.

To install a library from workspace files onto a cluster:

  1. Click compute icon Compute in the left sidebar.
  2. Click the name of the cluster in the cluster list.
  3. Click the Libraries tab.
  4. Click Install new. The Install library dialog appears.
  5. For Library Source, select Workspace.
  6. Upload the library or requirements.txt file, browse to the library or requirements.txt in the workspace, or enter its workspace location in the Workspace File Path field, such as the following: /Workspace/Users/someone@example.com/<path-to-library>/<library-name>.<ext>
  7. Click Install.

Add dependent libraries to workflow tasks from workspace files

You can add dependent libraries to tasks from workspace files. See Configure dependent libraries.

To configure a workflow task with a dependent library from workspace files:

  1. Select an existing task in a workflow or create a new task.
  2. Next to Dependent libraries, click + Add.
  3. In the Add dependent library dialog, select Workspace for Library Source.
  4. Upload the library or requirements.txt file, browse to the library or requirements.txt file in the workspace, or enter its workspace location in the Workspace File Path field, such as the following: /Workspace/Users/someone@example.com/<path-to-library>/<library-name>.<ext>
  5. Click Install.

Install libraries from workspace files to a notebook

You can install Python libraries directly to a notebook to create custom environments that are specific to the notebook. For example, you can use a specific version of a library in a notebook, without affecting other users on the cluster who may need a different version of the same library. For more information, see notebook-scoped libraries.

When you install a library to a notebook, only the current notebook and any jobs associated with that notebook have access to that library. Other notebooks attached to the same cluster are not affected.