Install libraries from a volume

  • Article

This article walks you through the steps required to upload libraries or requirements.txt files to volumes and install them onto clusters in Azure Databricks. You can install libraries onto all-purpose compute or job compute.

For more information about volumes, see What are Unity Catalog volumes?. For information about working with Unity Catalog, including controlling access and creating objects, see What is Unity Catalog?.

For full library compatibility details, see Cluster-scoped libraries.

Load libraries to a volume

To load a library to a volume:

  1. Click Catalog icon Catalog in the left sidebar.
  2. In the Catalog Explorer tree, navigate to the volume.
  3. Click +Add, then select Upload to this volume.
  4. The Upload files to volume dialog appears. Drag and drop or browse to the file(s) you want to upload, and click Upload.

Install libraries from a volume onto a cluster

When you install a library onto a cluster, all notebooks running on that cluster have access to the library.

To install a library from a volume onto a cluster:

  1. Click compute icon Compute in the left sidebar.
  2. Click the name of the cluster in the cluster list.
  3. Click the Libraries tab.
  4. Click Install new. The Install library dialog appears.
  5. For Library Source, select Volumes.
  6. Upload the library or requirements.txt file, browse to the library or requirements.txt file in the Volumes browser, or enter its location in the Volumes File Path field, such as the following: /Volumes/<catalog>/<schema>/<volume>/<path-to-library>/<file-name>.<ext>.
  7. Click Install.

Add dependent libraries to workflow tasks from a volume

To configure a workflow task with a dependent library from a volume:

  1. Select an existing task in a workflow or create a new task.
  2. Next to Dependent libraries, click + Add.
  3. In the Add dependent library dialog, select Volumes for Library Source.
  4. Upload the library or requirements.txt file, browse to the library or requirements.txt file in the Volumes browser, or enter its location in the Volumes File Path field, such as the following: /Volumes/<catalog>/<schema>/<volume>/<path-to-library>/<file-name>.<ext>.
  5. Click Install.

Install libraries from a volume to a notebook

You can install Python libraries directly to a notebook to create custom Python environments that are specific to the notebook. For example, you can use a specific version of a library in a notebook, without affecting other users on the cluster who may need a different version of the same library. For more information, see notebook-scoped libraries.

When you install a library to a notebook, only the current notebook and any jobs associated with that notebook have access to that library. Other notebooks attached to the same cluster are not affected.

The following code shows how to install a Python wheel file from a volume to a notebook as a notebook-scoped library.

%pip install /Volumes/<catalog>/<schema>/<volume>/<path-to-library>/mypackage-0.0.1-py3-none-any.whl

or

%pip install /Volumes/<catalog>/<schema>/<volume>/<path-to-project>/requirements.txt