Prepare environment for ML

This section describes how to prepare your data and your Azure Databricks environment for machine learning and deep learning.

Databricks Runtime for Machine Learning (Databricks Runtime ML) is a ready-to-go environment optimized for machine learning and data science. Databricks Runtime ML includes many external libraries, including TensorFlow, PyTorch, Horovod, scikit-learn and XGBoost, and provides extensions to improve performance, including GPU acceleration in XGBoost, distributed deep learning using HorovodRunner, and model checkpointing using a Databricks File System (DBFS) FUSE mount.

To use Databricks Runtime ML, select the ML version of the runtime when you create your cluster.

Note

To access data in Unity Catalog for machine learning workflows, the access mode for the cluster must be single user (assigned). Shared clusters are not compatible with Databricks Runtime for Machine Learning.

Install libraries

You can install additional libraries to create a custom environment for your notebook or cluster.

Use GPU clusters

You can create GPU clusters to accelerate deep learning tasks. For information about creating Azure Databricks GPU clusters, see GPU-enabled compute. Databricks Runtime ML includes GPU hardware drivers and NVIDIA libraries such as CUDA.