Databricks Runtime 10.1 for ML (EoS)

Databricks Runtime 10.1 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 10.1 (EoS). Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. It also supports distributed deep learning training using Horovod.

For more information, including instructions for creating a Databricks Runtime ML cluster, see AI and machine learning on Databricks.

New features and improvements

Databricks Runtime 10.1 ML is built on top of Databricks Runtime 10.1. For information on what's new in Databricks Runtime 10.1, including Apache Spark MLlib and SparkR, see the Databricks Runtime 10.1 (EoS) release notes.

Enhancements to Databricks AutoML

In Databricks Runtime 10.1, Databricks AutoML includes improved semantic type detection, new alerts for potential data issues during training, new capabilities to prevent overfitting models, and the ability to split the input dataset into train, validation, and test sets chronologically.

Additional semantic type detections

AutoML now supports additional semantic type detection:

• Numeric columns that contain categorical labels are treated as a categorical type.
• String columns that contain English text are treated as a text feature.

You can now also add annotations to specify a column data type. For details, see Semantic type detection.

Alerts during training for potential data issues

AutoML now detects and generates alerts for potential issues with the dataset. Example alerts include unsupported column types and high cardinality columns. These alerts appear on the experiment page under the new Alerts tab. Additional information on alerts is included in the data exploration notebook. For more information, see Run the experiment and monitor the results.

Reduced model overfitting

Two new capabilities reduce the chances of overfitting a model when using AutoML:

• AutoML now reports test metrics in additional to validation and training metrics.
• AutoML now uses early stopping. It stops training and tuning models if the validation metric is no longer improving.

Split dataset into train/validation/test sets chronologically

For classification and regression problems, you can split the dataset into train, validation, and test sets chronologically. See Split data into train, validation, and test sets for details.

Enhancements to Databricks Feature Store

Databricks Feature Store now supports additional data types for feature tables: BinaryType, DecimalType, and MapType.

Mlflow

The following improvements are available starting in Mlflow version 1.21.0, which is included in Databricks Runtime 10.1 ML.

• [Models] Upgrade the fastai model flavor to support fastai v2 (2.4.1 and above).
• [Models] Introduce an mlflow.prophet model flavor for Prophet time series models.
• [Scoring] Fix a schema enforcement error that incorrectly cast date-like strings to datetime objects.

Hyperopt

SparkTrials now supports the early_stopping_fn parameter for fmin. You can use the early stopping function to specify conditions when Hyperopt should stop hyperparameter tuning before the maximum number of evaluations is reached. For example, you can use this parameter to end tuning if the objective function is no longer decreasing. For details, see fmin().

Major changes to Databricks Runtime ML Python environment

Python packages upgraded

• automl 1.3.1 => 1.4.1
• feature_store 0.3.4 => 0.3.5
• holidays 0.11.2 => 0.11.3.1
• horovod 0.22.1 => 0.23.0
• hyperopt 0.2.5.db2 => 0.2.5.db4
• imbalanced-learn 0.8.0 => 0.8.1
• lightgbm 3.1.1 => 3.3.0
• mlflow 1.20.2 => 1.21.0
• petastorm 0.11.2 => 0.11.3
• plotly 5.1.0 => 5.3.0
• pytorch 1.9.0 => 1.9.1
• spacy 3.1.2 => 3.1.3
• sparkdl 2.2.0_db3 => 2.2.0_db4
• torchvision 0.10.0 => 0.10.1
• transformers 4.9.2 => 4.11.3

Python packages added

• fasttext => 0.9.2
• tensorboard-plugin-profile => 2.5.0

Deprecations

MLlib automated MLflow tracking is deprecated on clusters that run Databricks Runtime 10.1 ML and above. Instead, use MLflow PySpark ML autologging by calling mlflow.pyspark.ml.autolog(). Autologging is enabled by default with Databricks Autologging.

System environment

The system environment in Databricks Runtime 10.1 ML differs from Databricks Runtime 10.1 as follows:

• DBUtils: Databricks Runtime ML does not include Library utility (dbutils.library) (legacy). Use %pip commands instead. See Notebook-scoped Python libraries.
• For GPU clusters, Databricks Runtime ML includes the following NVIDIA GPU libraries:
  • CUDA 11.0
  • cuDNN 8.0.5.39
  • NCCL 2.10.3
  • TensorRT 7.2.2

Libraries

The following sections list the libraries included in Databricks Runtime 10.1 ML that differ from those included in Databricks Runtime 10.1.

In this section:

• Top-tier libraries
• Python libraries
• R libraries
• Java and Scala libraries (Scala 2.12 cluster)

Top-tier libraries

Databricks Runtime 10.1 ML includes the following top-tier libraries:

• GraphFrames
• Horovod and HorovodRunner
• MLflow
• PyTorch
• spark-tensorflow-connector
• TensorFlow
• TensorBoard

Python libraries

Databricks Runtime 10.1 ML uses Virtualenv for Python package management and includes many popular ML packages.

In addition to the packages specified in the in the following sections, Databricks Runtime 10.1 ML also includes the following packages:

• hyperopt 0.2.5.db4
• sparkdl 2.2.0-db4
• feature_store 0.3.5
• automl 1.4.0

#!/bin/bash

set -e

pip install --upgrade "scikit-learn>=1.0,<1.1"

Python libraries on CPU clusters

Python libraries on GPU clusters

Spark packages containing Python modules

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 10.1.

Java and Scala libraries (Scala 2.12 cluster)

In addition to Java and Scala libraries in Databricks Runtime 10.1, Databricks Runtime 10.1 ML contains the following JARs:

CPU clusters

GPU clusters