用于机器学习的 Databricks Runtime 5.4（不受支持）Databricks Runtime 5.4 for Machine Learning (Unsupported)

• 12/01/2020
• • m

Databricks 于 2019 年 6 月发布了此映像。Databricks released this image in June 2019.

用于机器学习的 Databricks Runtime 5.4 基于 Databricks Runtime 5.4（不受支持），为机器学习和数据科学提供了随时可用的环境。Databricks Runtime 5.4 for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 5.4 (Unsupported). Databricks Runtime ML 包含许多常用的机器学习库，包括 TensorFlow、PyTorch、Keras 和 XGBoost。Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, Keras, and XGBoost. 它还支持使用 Horovod 进行分布式深度学习训练。It also supports distributed deep learning training using Horovod.

有关详细信息，包括有关如何创建 Databricks Runtime ML 群集的说明，请参阅用于机器学习的 Databricks Runtime。For more information, including instructions for creating a Databricks Runtime ML cluster, see Databricks Runtime for Machine Learning.

新增功能New features

Databricks Runtime 5.4 ML 是基于 Databricks Runtime 5.4 构建的。Databricks Runtime 5.4 ML is built on top of Databricks Runtime 5.4. 若要了解 Databricks Runtime 5.4 中的新增功能，请参阅 Databricks Runtime 5.4（不受支持）发行说明。For information on what’s new in Databricks Runtime 5.4, see the Databricks Runtime 5.4 (Unsupported) release notes.

除了库更新，Databricks Runtime 5.4 ML 还引入了以下新功能：In addition to library updates, Databricks Runtime 5.4 ML introduces the following new features:

分布式 Hyperopt + 自动化 MLflow 跟踪Distributed Hyperopt + automated MLflow tracking

Databricks Runtime 5.4 ML 引入了由 Apache Spark 提供支持的 Hyperopt 的新实现，用于缩放和简化超参数优化。Databricks Runtime 5.4 ML introduces a new implementation of Hyperopt powered by Apache Spark to scale and simplify hyperparameter tuning. 实现新的 Trials 类 SparkTrials 是为了使用 Apache Spark 在多个计算机和节点之间分发 Hyperopt 试用版运行。A new Trials class SparkTrials is implemented to distribute Hyperopt trial runs among multiple machines and nodes using Apache Spark. 此外，所有优化试验以及优化后的超参数和目标指标会自动记录到 MLflow 运行。In addition, all tuning experiments, along with the tuned hyperparameters and targeted metrics, are automatically logged to MLflow runs. 请参阅使用 scikit-learn 和 MLflow 并行优化超参数。See Parallelize hyperparameter tuning with scikit-learn and MLflow.

Apache Spark MLlib + 自动化 MLflow 跟踪Apache Spark MLlib + automated MLflow tracking

对于使用 PySpark 优化算法 CrossValidator 和 TrainValidationSplit 拟合的模型，Databricks Runtime 5.4 ML 支持自动记录 MLflow 运行。Databricks Runtime 5.4 ML supports automatic logging of MLflow runs for models fit using PySpark tuning algorithms CrossValidator and TrainValidationSplit. 请参阅 Apache Spark MLlib 和自动化 MLflow 跟踪。See Apache Spark MLlib and automated MLflow tracking. 此功能在 Databricks Runtime 5.4 ML 中默认处于启用状态，但在 Databricks Runtime 5.3 ML 中则默认处于关闭状态。This feature is on by default in Databricks Runtime 5.4 ML but was off by default in Databricks Runtime 5.3 ML.

HorovodRunner 改进HorovodRunner improvement

从 Horovod 发送到 Spark 驱动程序节点的输出现在显示在笔记本单元中。Output sent from Horovod to the Spark driver node is now visible in notebook cells.

XGBoost Python 包更新XGBoost Python package update

已安装 XGBoost Python 包 0.80。XGBoost Python package 0.80 is installed.

系统环境System environment

Databricks Runtime 5.4 ML 中的系统环境在以下方面不同于 Databricks Runtime 5.4：The system environment in Databricks Runtime 5.4 ML differs from Databricks Runtime 5.4 as follows:

• Python：2.7.15 适用于 Python 2 群集，3.6.5 适用于 Python 3 群集。Python: 2.7.15 for Python 2 clusters and 3.6.5 for Python 3 clusters.
• DBUtils：Databricks Runtime 5.4 ML 未包含库实用工具。DBUtils: Databricks Runtime 5.4 ML does not contain Library utilities.
• 对于 GPU 群集，以下 NVIDIA GPU 库：For GPU clusters, the following NVIDIA GPU libraries:
  • Tesla 驱动程序 396.44Tesla driver 396.44
  • CUDA 9.2CUDA 9.2
  • CUDNN 7.2.1CUDNN 7.2.1

库 Libraries

以下部分列出了 Databricks Runtime 5.4 ML 中包含的库，这些库不同于 Databricks Runtime 5.4 中包含的库。The following sections list the libraries included in Databricks Runtime 5.4 ML that differ from those included in Databricks Runtime 5.4.

顶层库Top-tier libraries

Databricks Runtime 5.4 ML 包含以下顶层库：Databricks Runtime 5.4 ML includes the following top-tier libraries:

• GraphFramesGraphFrames
• Horovod 和 HorovodRunnerHorovod and HorovodRunner
• PyTorchPyTorch
• spark-tensorflow-connectorspark-tensorflow-connector
• TensorFlowTensorFlow
• TensorBoardTensorBoard

Python 库Python libraries

Databricks Runtime 5.4 ML 使用 Conda 进行 Python 包管理。Databricks Runtime 5.4 ML uses Conda for Python package management. 因此，已安装的 Python 库相对于 Databricks Runtime 有很大区别。As a result, there are major differences in installed Python libraries compared to Databricks Runtime. 下面是所提供的 Python 包和使用 Conda 包管理器安装的版本的完整列表。The following is a full list of provided Python packages and versions installed using Conda package manager.

此外，以下 Spark 包还包括 Python 模块：In addition, the following Spark packages include Python modules:

R 库R libraries

R 库与 Databricks Runtime 5.4 中的 R 库完全相同。The R libraries are identical to the R Libraries in Databricks Runtime 5.4.

Java 库和 Scala 库（Scala 2.11 群集）Java and Scala libraries (Scala 2.11 cluster)

除了 Databricks Runtime 5.4 中的 Java 库和 Scala 库之外，Databricks Runtime 5.4 ML 还包含以下 JAR：In addition to Java and Scala libraries in Databricks Runtime 5.4, Databricks Runtime 5.4 ML contains the following JARs: