用于机器学习的 Databricks Runtime 16.1 （EoS）

用于机器学习的 Databricks Runtime 16.1 提供了基于 Databricks Runtime 16.1（EoS） 的机器学习和数据科学的现成的环境。 Databricks Runtime ML 包含许多常用的机器学习库，包括 TensorFlow、PyTorch 和 XGBoost。 Databricks Runtime ML 包括 AutoML 工具，可用于自动训练机器学习管道。 Databricks Runtime ML 还支持使用 TorchDistributor、DeepSpeed 和 Ray 进行分布式深度学习训练。

新功能和改进

Databricks Runtime 16.1 ML 基于 Databricks Runtime 16.1 构建。 有关 Databricks Runtime 16.1 中的新增功能（包括 Apache Spark MLlib 和 SparkR）的信息，请参阅 Databricks Runtime 16.1 （EoS） 发行说明。

系统环境

Databricks Runtime 16.1 ML 中的系统环境不同于 Databricks Runtime 16.1，如下所示。

• 对于 GPU 群集，Databricks Runtime ML 包含以下 NVIDIA GPU 库：
  • CUDA 12.6
  • cublas 12.6.0.22-1
  • cusolver 11.6.4.38-1
  • cupti 12.6.37-1
  • cusparse 12.5.2.23-1
  • cuDNN 9.3.0.75-1
  • NCCL 2.22.3
  • TensorRT 10.2.0.19-1

图书馆

以下部分列出了 Databricks Runtime 16.1 ML 中包含的库，这些库不同于 Databricks Runtime 16.1 中包含的库。

本节内容：

• 顶层库
• Python 库
• R 库
• Java 库和 Scala 库（Scala 2.12 群集）

顶级图书馆

Databricks Runtime 16.1 ML 包含以下顶级 库：

• 数据集
• GraphFrames
• MLflow
• PyTorch
• spark-tensorflow-connector
• Scikit-learn
• 流媒体
• TensorFlow
• TensorBoard
• 转换器

Python 库

Databricks Runtime 16.1 ML 用于 virtualenv Python 包管理，并包括许多常用的 ML 包。

除了以下部分中指定的包外，Databricks Runtime 16.1 ML 还包括以下包：

• hyperopt 0.2.9+db5
• automl 1.29.0

若要在本地 Python 虚拟环境中重现 Databricks Runtime ML Python 环境，请下载 requirements-16.1.txt 文件并运行 pip install -r requirements-16.1.txt。 此命令安装 Databricks Runtime ML 使用的所有开源库，但不安装 Databricks 开发的库，例如 databricks-automl、databricks-feature-engineering 或 hyperopt 的 Databricks 分支。

CPU 群集上的 Python 库

|greenlet |3.0.1 |grpcio |1.60.0 |grpcio-status |1.60.0 | |gunicorn |20.1.0 |gviz-api |1.10.0 |gymnasium |0.28.1 | |h11 |0.14.0 |h5py |3.11.0 |hjson |3.1.0 | |holidays |0.54 |htmlmin |0.1.12 |httpcore |1.0.5 | |httplib2 |0.20.4 |httpx |0.28.0 |huggingface-hub |0.24.5 | |idna |3.7 |ImageHash |4.3.1 |imageio |2.33.1 | |imbalanced-learn |0.12.3 |importlib-metadata |6.0.0 |importlib_resources |6.4.5 | |inflect |7.3.1 |ipyflow-core |0.0.201 |ipykernel |6.28.0 | |ipython |8.25.0 |ipython-genutils |0.2.0 |ipywidgets |7.7.2 | |isodate |0.6.1 |isoduration |20.11.0 |itsdangerous |2.2.0 | |jaraco.context |5.3.0 |jaraco.functools |4.0.1 |jaraco.text |3.12.1 | |jax-jumpy |1.0.0 |jedi |0.19.1 |Jinja2 |3.1.4 | |jiter |0.8.0 |jmespath |1.0.1 |joblib |1.4.2 | |joblibspark |0.5.1 |json5 |0.9.6 |jsonpatch |1.33 | |jsonpointer |3.0.0 |jsonschema |4.19.2 |jsonschema-specifications |2023.7.1 | |jupyter-events |0.10.0 |jupyter-lsp |2.2.0 |jupyter_client |8.6.0 | |jupyter_core |5.7.2 |jupyter_server |2.14.1 |jupyter_server_terminals |0.4.4 | |jupyterlab |4.0.11 |jupyterlab-pygments |0.1.2 |jupyterlab_server |2.25.1 | |keras |3.6.0 |kiwisolver |1.4.4 |langchain |0.3.8 | |langchain-core |0.2.41 |langchain-text-splitters |0.2.4 |langcodes |3.4.1 | |langsmith |0.1.129 |language_data |1.2.0 |launchpadlib |1.11.0 | |lazr.restfulclient |0.14.6 |lazr.uri |1.0.6 |lazy_loader |0.4 | |libclang |15.0.6.1 |librosa |0.10.2 |lightgbm |4.5.0 | |lightning-utilities |0.11.9 |linkify-it-py |2.0.0 |llvmlite |0.42.0 | |lz4 |4.3.2 |Mako |1.2.0 |marisa-trie |1.2.0 | |Markdown |3.4.1 |markdown-it-py |2.2.0 |MarkupSafe |2.1.3 | |matplotlib |3.8.4 |matplotlib-inline |0.1.6 |mccabe |0.7.0 | |mdit-py-plugins |0.3.0 |mdurl |0.1.0 |memray |1.14.0 | |mistune |2.0.4 |ml-dtypes |0.4.1 |mlflow-skinny |2.15.1 | |more-itertools |10.3.0 |mosaicml-cli |0.6.41 |mosaicml-streaming |0.8.0 | |mpmath |1.3.0 |msal |1.31.1 |msal-extensions |1.2.0 | |msgpack |1.1.0 |multidict |6.0.4 |multimethod |1.12 | |multiprocess |0.70.16 |murmurhash |1.0.11 |mypy |1.10.0 | |mypy-extensions |1.0.0 |namex |0.0.8 |nbclient |0.8.0 | |nbconvert |7.10.0 |nbformat |5.9.2 |nest-asyncio |1.6.0 | |networkx |3.2.1 |ninja |1.11.1.1 |nltk |3.8.1 | |nodeenv |1.9.1 |notebook |7.0.8 |notebook_shim |0.2.3 | |numba |0.59.1 |numpy |1.26.4 |nvidia-ml-py |12.560.30 | |oauthlib |3.2.0 |oci |2.139.0 |openai |1.55.1 | |opencensus |0.11.4 |opencensus-context |0.1.3 |opentelemetry-api |1.28.2 | |opentelemetry-sdk |1.28.2 |opentelemetry-semantic-conventions |0.49b2 |opt_einsum |3.4.0 | |optree |0.12.1 |optuna |3.6.1 |optuna-integration |3.6.0 | |orjson |3.10.12 |overrides |7.4.0 |packaging |24.1 | |pandas |1.5.3 |pandocfilters |1.5.0 |paramiko |3.4.0 | |parso |0.8.3 |pathspec |0.10.3 |patsy |0.5.6 | |pexpect |4.8.0 |phik |0.12.4 |pillow |10.3.0 | |pip |24.2 |platformdirs |3.10.0 |plotly |5.22.0 | |pluggy |1.0.0 |pmdarima |2.0.4 |pooch |1.8.2 | |portalocker |2.10.1 |preshed |3.0.9 |prometheus-client |0.14.1 | |prompt-toolkit |3.0.43 |prophet |1.1.5 |proto-plus |1.24.0 | |protobuf |4.24.1 |psutil |5.9.0 |psycopg2 |2.9.3 | |ptyprocess |0.7.0 |pure-eval |0.2.2 |py-cpuinfo |9.0.0 | |py-spy |0.3.14 |pyarrow |15.0.2 |pyarrow-hotfix |0.6 | |pyasn1 |0.4.8 |pyasn1-modules |0.2.8 |pybind11 |2.13.6 | |pyccolo |0.0.65 |pycparser |2.21 |pydantic |2.8.2 | |pydantic_core |2.20.1 |pyflakes |3.2.0 |Pygments |2.15.1 | |PyGObject |3.48.2 |PyJWT |2.7.0 |PyNaCl |1.5.0 | |pyodbc |5.0.1 |pyOpenSSL |24.0.0 |pyparsing |3.0.9 | |pyright |1.1.294 |pytesseract |0.3.10 |python-dateutil |2.9.0.post0 | |python-editor |1.0.4 |python-json-logger |2.0.7 |python-lsp-jsonrpc |1.1.2 | |python-lsp-server |1.10.0 |python-snappy |0.6.1 |pytoolconfig |1.2.6 | |pytorch-ranger |0.1.1 |pytz |2024.1 |PyWavelets |1.5.0 | |PyYAML |6.0.1 |pyzmq |25.1.2 |questionary |1.10.0 | |ray |2.35.0 |referencing |0.30.2 |regex |2023.10.3 | |requests |2.32.2 |requests-oauthlib |1.3.1 |rfc3339-validator |0.1.4 | |rfc3986-validator |0.1.1 |rich |13.3.5 |rope |1.12.0 | |rpds-py |0.10.6 |rsa |4.9 |ruamel.yaml |0.18.6 | |ruamel.yaml.clib |0.2.8 |s3transfer |0.10.2 |safetensors |0.4.4 | |scikit-image |0.23.2 |scikit-learn |1.4.2 |scipy |1.13.1 | |seaborn |0.13.2 |Send2Trash |1.8.2 |sentence-transformers |3.3.1 | |sentencepiece |0.2.0 |setuptools |74.0.0 |shap |0.46.0 | |shellingham |1.5.4 |simplejson |3.17.6 |six |1.16.0 | |slicer |0.0.8 |smart-open |5.2.1 |smmap |5.0.0 | |sniffio |1.3.0 |soundfile |0.12.1 |soupsieve |2.5 | |soxr |0.5.0.post1 |spacy |3.7.5 |spacy-legacy |3.0.12 | |spacy-loggers |1.0.5 |SQLAlchemy |2.0.30 |sqlparse |0.4.2 | |srsly |2.4.8 |ssh-import-id |5.11 |stack-data |0.2.0 | |stanio |0.5.1 |statsmodels |0.14.2 |sympy |1.13.1 | |tabulate |0.9.0 |tangled-up-in-unicode |0.2.0 |tenacity |8.2.2 | |tensorboard |2.18.0 |tensorboard-data-server |0.7.2 |tensorboard-plugin-profile |2.18.0 | |tensorboardX |2.6.2.2 |tensorflow |2.18.0 |tensorflow-estimator |2.15.0 | |termcolor |2.4.0 |terminado |0.17.1 |textual |0.87.1 | |tf_keras |2.18.0 |thinc |8.2.5 |threadpoolctl |2.2.0 | |tifffile |2023.4.12 |tiktoken |0.7.0 |tinycss2 |1.2.1 | |tokenize-rt |4.2.1 |tokenizers |0.20.0 |tomli |2.0.1 | |torch |2.5.0+cpu |torch-optimizer |0.3.0 |torcheval |0.0.7 | |torchmetrics |1.4.0.post0 |torchvision |0.20.0+cpu |tornado |6.4.1 | |tqdm |4.66.4 |traitlets |5.14.3 |transformers |4.46.3 | |typeguard |4.3.0 |typer |0.12.5 |types-protobuf |3.20.3 | |types-psutil |5.9.0 |types-pytz |2023.3.1.1 |types-PyYAML |6.0.0 | |types-requests |2.31.0.0 |types-setuptools |68.0.0.0 |types-six |1.16.0 | |types-urllib3 |1.26.25.14 |typing_extensions |4.11.0 |uc-micro-py |1.0.1 | |ujson |5.10.0 |unattended-upgrades |0.1 |uri-template |1.3.0 | |urllib3 |1.26.16 |validators |0.34.0 |virtualenv |20.26.2 | |visions |0.7.5 |wadllib |1.3.6 |wasabi |1.1.3 | |wcwidth |0.2.5 |weasel |0.4.1 |webcolors |24.8.0 | |webencodings |0.5.1 |websocket-client |1.8.0 |websockets |11.0.3 | |Werkzeug |3.0.3 |whatthepatch |1.0.2 |wheel |0.43.0 | |wordcloud |1.9.3 |wrapt |1.14.1 |xgboost |2.0.3 | |xgboost-ray |0.1.19 |xxhash |3.4.1 |yapf |0.33.0 | |yarl |1.9.3 |ydata-profiling |4.9.0 |zipp |3.17.0 | |zstd |1.5.5.1 | | | | |

GPU 群集上的 Python 库

|graphql-core |3.2.4 |greenlet |3.0.1 |grpcio |1.60.0 | |grpcio-status |1.60.0 |gunicorn |20.1.0 |gviz-api |1.10.0 | |gymnasium |0.28.1 |h11 |0.14.0 |h5py |3.11.0 | |hjson |3.1.0 |holidays |0.54 |htmlmin |0.1.12 | |httpcore |1.0.5 |httplib2 |0.20.4 |httpx |0.27.2 | |huggingface-hub |0.24.5 |idna |3.7 |ImageHash |4.3.1 | |imageio |2.33.1 |imbalanced-learn |0.12.3 |importlib-metadata |6.0.0 | |importlib_resources |6.4.5 |inflect |7.3.1 |ipyflow-core |0.0.201 | |ipykernel |6.28.0 |ipython |8.25.0 |ipython-genutils |0.2.0 | |ipywidgets |7.7.2 |isodate |0.6.1 |isoduration |20.11.0 | |itsdangerous |2.2.0 |jaraco.context |5.3.0 |jaraco.functools |4.0.1 | |jaraco.text |3.12.1 |jax-jumpy |1.0.0 |jedi |0.19.1 | |Jinja2 |3.1.4 |jiter |0.8.0 |jmespath |1.0.1 | |joblib |1.4.2 |joblibspark |0.5.1 |json5 |0.9.6 | |jsonpatch |1.33 |jsonpointer |3.0.0 |jsonschema |4.19.2 | |jsonschema-specifications |2023.7.1 |jupyter-events |0.10.0 |jupyter-lsp |2.2.0 | |jupyter_client |8.6.0 |jupyter_core |5.7.2 |jupyter_server |2.14.1 | |jupyter_server_terminals |0.4.4 |jupyterlab |4.0.11 |jupyterlab-pygments |0.1.2 | |jupyterlab_server |2.25.1 |keras |3.6.0 |kiwisolver |1.4.4 | |langchain |0.3.8 |langchain-core |0.2.41 |langchain-text-splitters |0.2.4 | |langcodes |3.4.1 |langsmith |0.1.129 |language_data |1.2.0 | |launchpadlib |1.11.0 |lazr.restfulclient |0.14.6 |lazr.uri |1.0.6 | |lazy_loader |0.4 |libclang |15.0.6.1 |librosa |0.10.2 | |lightgbm |4.5.0 |lightning-utilities |0.11.9 |linkify-it-py |2.0.0 | |llvmlite |0.42.0 |lz4 |4.3.2 |Mako |1.2.0 | |marisa-trie |1.2.0 |Markdown |3.4.1 |markdown-it-py |2.2.0 | |MarkupSafe |2.1.3 |matplotlib |3.8.4 |matplotlib-inline |0.1.6 | |mccabe |0.7.0 |mdit-py-plugins |0.3.0 |mdurl |0.1.0 | |memray |1.14.0 |mistune |2.0.4 |ml-dtypes |0.4.1 | |mlflow-skinny |2.15.1 |more-itertools |10.3.0 |mosaicml-cli |0.6.41 | |mosaicml-streaming |0.8.0 |mpmath |1.3.0 |msal |1.31.1 | |msal-extensions |1.2.0 |msgpack |1.1.0 |multidict |6.0.4 | |multimethod |1.12 |multiprocess |0.70.16 |murmurhash |1.0.11 | |mypy |1.10.0 |mypy-extensions |1.0.0 |namex |0.0.8 | |nbclient |0.8.0 |nbconvert |7.10.0 |nbformat |5.9.2 | |nest-asyncio |1.6.0 |networkx |3.2.1 |ninja |1.11.1.1 | |nltk |3.8.1 |nodeenv |1.9.1 |notebook |7.0.8 | |notebook_shim |0.2.3 |numba |0.59.1 |numpy |1.26.4 | |nvidia-cublas-cu12 |12.4.5.8 |nvidia-cuda-cupti-cu12 |12.4.127 |nvidia-cuda-nvrtc-cu12 |12.4.127 | |nvidia-cuda-runtime-cu12 |12.4.127 |nvidia-cudnn-cu12 |9.1.0.70 |nvidia-cufft-cu12 |11.2.1.3 | |nvidia-curand-cu12 |10.3.5.147 |nvidia-cusolver-cu12 |11.6.1.9 |nvidia-cusparse-cu12 |12.3.1.170 | |nvidia-ml-py |12.560.30 |nvidia-nccl-cu12 |2.21.5 |nvidia-nvjitlink-cu12 |12.4.127 | |nvidia-nvtx-cu12 |12.4.127 |oauthlib |3.2.0 |oci |2.139.0 | |openai |1.55.1 |opencensus |0.11.4 |opencensus-context |0.1.3 | |opentelemetry-api |1.28.2 |opentelemetry-sdk |1.28.2 |opentelemetry-semantic-conventions |0.49b2 | |opt_einsum |3.4.0 |optree |0.12.1 |optuna |3.6.1 | |optuna-integration |3.6.0 |orjson |3.10.12 |overrides |7.4.0 | |packaging |24.1 |pandas |1.5.3 |pandocfilters |1.5.0 | |paramiko |3.4.0 |parso |0.8.3 |pathspec |0.10.3 | |patsy |0.5.6 |pexpect |4.8.0 |phik |0.12.4 | |pillow |10.3.0 |pip |24.2 |platformdirs |3.10.0 | |plotly |5.22.0 |plugin |1.0.0 |pmdarima |2.0.4 | |pooch |1.8.2 |portalocker |2.10.1 |preshed |3.0.9 | |prometheus-client |0.14.1 |prompt-toolkit |3.0.43 |prophet |1.1.5 | |proto-plus |1.24.0 |protobuf |4.24.1 |psutil |5.9.0 | |psycopg2 |2.9.3 |ptyprocess |0.7.0 |pure-eval |0.2.2 | |py-cpuinfo |9.0.0 |py-spy |0.3.14 |pyarrow |15.0.2 | |pyarrow-hotfix |0.6 |pyasn1 |0.4.8 |pyasn1-modules |0.2.8 | |pybind11 |2.13.6 |pyccolo |0.0.65 |pycparser |2.21 | |pydantic |2.8.2 |pydantic_core |2.20.1 |pyflakes |3.2.0 | |Pygments |2.15.1 |PyGObject |3.48.2 |PyJWT |2.7.0 | |PyNaCl |1.5.0 |pyodbc |5.0.1 |pyOpenSSL |24.0.0 | |pyparsing |3.0.9 |pyright |1.1.294 |pytesseract |0.3.10 | |python-dateutil |2.9.0.post0 |python-editor |1.0.4 |python-json-logger |2.0.7 | |python-lsp-jsonrpc |1.1.2 |python-lsp-server |1.10.0 |python-snappy |0.6.1 | |pytoolconfig |1.2.6 |pytorch-ranger |0.1.1 |pytz |2024.1 | |PyWavelets |1.5.0 |PyYAML |6.0.1 |pyzmq |25.1.2 | |questionary |1.10.0 |ray |2.35.0 |quoting |0.30.2 | |regex |2023.10.3 |requests |2.32.2 |requests-oauthlib |1.3.1 | |rfc3339-validator |0.1.4 |rfc3986-validator |0.1.1 |rich |13.3.5 | |rope |1.12.0 |rpds-py |0.10.6 |rsa |4.9 | |ruamel.yaml |0.18.6 |ruamel.yaml.clib |0.2.8 |s3transfer |0.10.2 | |safetensors |0.4.4 |scikit-image |0.23.2 |scikit-learn |1.4.2 | |scipy |1.13.1 |seaborn |0.13.2 |Send2Trash |1.8.2 | |sentence-transformers |3.3.1 |sentencepiece |0.2.0 |setuptools |74.0.0 | |shap |0.46.0 |shellingham |1.5.4 |simplejson |3.17.6 | |six |1.16.0 |slicer |0.0.8 |smart-open |5.2.1 | |smmap |5.0.0 |sniffio |1.3.0 |soundfile |0.12.1 | |soupsieve |2.5 |soxr |0.5.0.post1 |spacy |3.7.5 | |spacy-legacy |3.0.12 |spacy-loggers |1.0.5 |SQLAlchemy |2.0.30 | |sqlparse |0.4.2 |srsly |2.4.8 |ssh-import-id |5.11 | |stack-data |0.2.0 |stanio |0.5.1 |statsmodels |0.14.2 | |sympy |1.13.1 |tabulate |0.9.0 |tangled-up-in-unicode |0.2.0 | |tenacity |8.2.2 |tensorboard |2.17.0 |tensorboard-data-server |0.7.2 | |tensorboard-plugin-profile |2.18.0 |tensorboardX |2.6.2.2 |tensorflow |2.18.0 | |tensorflow-estimator |2.15.0 |termcolor |2.4.0 |terminado |0.17.1 | |textual |0.87.1 |tf_keras |2.17.0 |thinc |8.2.5 | |threadpoolctl |2.2.0 |tifffile |2023.4.12 |tiktoken |0.7.0 | |tinycss2 |1.2.1 |tokenize-rt |4.2.1 |tokenizers |0.20.0 | |tomli |2.0.1 |torch |2.5.0+cu124 |torch-optimizer |0.3.0 | |torcheval |0.0.7 |torchmetrics |1.4.0.post0 |torchvision |0.20.0+cu124 | |tornado |6.4.1 |tqdm |4.66.4 |traitlets |5.14.3 | |transformers |4.46.3 |triton |3.1.0 |typeguard |4.3.0 | |typer |0.14.0 |types-protobuf |3.20.3 |types-psutil |5.9.0 | |types-pytz |2023.3.1.1 |types-PyYAML |6.0.0 |types-requests |2.31.0.0 | |types-setuptools |68.0.0.0 |types-six |1.16.0 |types-urllib3 |1.26.25.14 | |typing_extensions |4.11.0 |uc-micro-py |1.0.1 |ujson |5.10.0 | |unattended-upgrades |0.1 |uri-template |1.3.0 |urllib3 |1.26.16 | |validators |0.34.0 |virtualenv |20.26.2 |visions |0.7.5 | |wadllib |1.3.6 |wasabi |1.1.3 |wcwidth |0.2.5 | |weasel |0.4.1 |webcolors |24.8.0 |webencodings |0.5.1 | |websocket-client |1.8.0 |websockets |11.0.3 |Werkzeug |3.0.3 | |whatthepatch |1.0.2 |wheel |0.43.0 |wordcloud |1.9.3 | |wrapt |1.14.1 |xgboost |2.0.3 |xgboost-ray |0.1.19 | |xxhash |3.4.1 |yapf |0.33.0 |yarl |1.9.3 | |ydata-profiling |4.9.0 |zipp |3.17.0 |zstd |1.5.5.1 |

R 软件库

R 库与 Databricks Runtime 16.1 中的 R 库 相同。

Java 库和 Scala 库（Scala 2.12 群集）

除了 Databricks Runtime 16.1 中的 Java 和 Scala 库外，Databricks Runtime 16.1 ML 还包含以下 JAR：

CPU 群集

GPU 群集