Databricks Runtime 5.5 LTS for ML (EoS)

  • Article

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in July 2019. Support ended on July 27, 2021. Databricks Runtime 5.5 ML Extended Support (EoS) extends 5.5 ML support through December 2021. It uses Ubuntu 18.04.5 LTS instead of the deprecated Ubuntu 16.04.6 LTS distribution used in the original Databricks Runtime 5.5 ML LTS. Ubuntu 16.04.6 LTS support ceased on April 1, 2021.

Databricks Runtime 5.5 LTS for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 5.5 LTS (EoS). Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, Keras, and XGBoost. It also supports distributed deep learning training using Horovod.

For more information, including instructions for creating a Databricks Runtime ML cluster, see AI and machine learning on Databricks.

New features

Databricks Runtime 5.5 LTS for Machine Learning is built on top of Databricks Runtime 5.5 LTS. For information on what's new in Databricks Runtime 5.5 LTS, see the Databricks Runtime 5.5 LTS (EoS) release notes.

In addition to library updates, Databricks Runtime 5.5 LTS for Machine Learning introduces the following new features:

  • Added the MLflow 1.0 Python package

Improvements

  • Upgraded machine learning libraries

    • TensorFlow upgraded from 1.12.0 to 1.13.1
    • PyTorch upgraded from 0.4.1 to 1.1.0
    • scikit-learn upgraded from 0.19.1 to 0.20.3

  • Single-node operation for HorovodRunner

    Enabled HorovodRunner to run on only the driver node. Previously, to use HorovodRunner you would have to run a driver and at least one worker node. With this change, you can now distribute training within a single node (that is, a multi-GPU node) and thus use compute resources more efficiently.

Deprecation

In the hyperopt library, we deprecated the following properties of hyperopt.SparkTrials:

  • SparkTrials.successful_trials_count
  • SparkTrials.failed_trials_count
  • SparkTrials.cancelled_trials_count
  • SparkTrials.total_trials_count

and replaced the properties with the following functions:

  • SparkTrials.count_successful_trials()
  • SparkTrials.count_failed_trials()
  • SparkTrials.count_cancelled_trials()
  • SparkTrials.count_total_trials()

System environment

The system environment in Databricks Runtime 5.5 LTS for Machine Learning differs from Databricks Runtime 5.5 as follows:

  • Python: 3.6.5 for Python 3 clusters and 2.7.15 for Python 2 clusters.
  • DBUtils: Does not contain Library utility (dbutils.library) (legacy).
  • For GPU clusters, the following NVIDIA GPU libraries:
    • CUDA 10.0
    • CUDNN 7.6.0

Libraries

The following sections list the libraries included in Databricks Runtime 5.5 LTS for Machine Learning that differ from those included in Databricks Runtime 5.5.

Top-tier libraries

Databricks Runtime 5.5 LTS for Machine Learning includes the following top-tier libraries:

Python libraries

Databricks Runtime 5.5 LTS for Machine Learning uses Conda for Python package management. As a result, there are major differences in installed Python libraries compared to Databricks Runtime. The following sections describe the Conda environments for Databricks Runtime 5.5 LTS for Machine Learning clusters using Python 2 or 3, and CPU or GPU-enabled machines.

Python 3 on CPU clusters

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36hb0f60f5_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36h17d85b1_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36he6710b0_0
  - py-xgboost-cpu=0.90=py36_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow=1.13.1=mkl_py36h27d456a_0
  - tensorflow-base=1.13.1=mkl_py36h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36h8cadb63_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py3.6_cpu_0
  - torchvision-cpu=0.3.0=py36_cuNone_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 3 on GPU clusters

name: null
channels:
  - pytorch
  - Databricks
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py36h2903d8e_0
  - tensorflow-base=1.13.1.db1=gpu_py36he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - backcall=0.1.0=py36_0
  - backports=1.0=py_2
  - bcrypt=3.1.6=py36h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py36_0
  - boto=2.48.0=py36_1
  - boto3=1.7.62=py36h28b3542_1
  - botocore=1.10.62=py36h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36he75722e_1
  - chardet=3.0.4=py36_1
  - click=7.0=py36_0
  - cloudpickle=0.8.0=py36_0
  - colorama=0.3.9=py36h489cec4_0
  - configparser=3.7.3=py36_1
  - cryptography=2.2.2=py36h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py36_0
  - cython=0.28.2=py36h14c3975_0
  - decorator=4.3.0=py36_0
  - docutils=0.14=py36_0
  - entrypoints=0.2.3=py36_2
  - et_xmlfile=1.0.1=py36hd6bccc3_0
  - flask=1.0.2=py36_1
  - freetype=2.8=hab7d2ae_1
  - gast=0.2.2=py36_0
  - gitdb2=2.0.5=py36_0
  - gitpython=2.1.11=py36_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py36hdbcaa40_0
  - gunicorn=19.9.0=py36_0
  - h5py=2.8.0=py36h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py36_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py36h82fb2a8_1
  - intel-openmp=2018.0.0=8
  - ipython=6.4.0=py36_1
  - ipython_genutils=0.2.0=py36hb52b0d5_0
  - itsdangerous=0.24=py36_1
  - jdcal=1.4=py36_0
  - jedi=0.12.0=py36_1
  - jinja2=2.10=py36_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter_client=5.2.3=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py36_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - llvmlite=0.23.1=py36hdbcaa40_0
  - lxml=4.2.1=py36h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py36_0
  - markupsafe=1.0=py36h14c3975_1
  - mistune=0.8.3=py36h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - mock=3.0.5=py36_0
  - msgpack-python=0.5.6=py36h6bb024c_1
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36h31c9010_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py36hfd86e86_0
  - numba=0.38.0=py36h637b7d7_0
  - numpy=1.16.2=py36h7e9f1db_0
  - numpy-base=1.16.2=py36hde5b4d6_0
  - olefile=0.45.1=py36_0
  - openpyxl=2.5.3=py36_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py36h637b7d7_0
  - pandocfilters=1.4.2=py36_1
  - paramiko=2.4.2=py36_0
  - parso=0.2.0=py36_0
  - pathlib2=2.3.2=py36_0
  - patsy=0.5.0=py36_0
  - pexpect=4.5.0=py36_0
  - pickleshare=0.7.4=py36h63277f8_0
  - pillow=5.1.0=py36h3deb7b8_0
  - pip=10.0.1=py36_0
  - ply=3.11=py36_0
  - prompt_toolkit=1.0.15=py36_0
  - protobuf=3.8.0=py36he6710b0_0
  - psycopg2=2.7.5=py36hb7f436b_0
  - ptyprocess=0.5.2=py36h69acd42_0
  - py-xgboost=0.90=py36h688424c_0
  - py-xgboost-gpu=0.90=py36h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py36_1
  - pygments=2.2.0=py36_0
  - pynacl=1.3.0=py36h7b6447c_0
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36_1
  - pysocks=1.6.8=py36_0
  - python=3.6.5=hc3d631a_2
  - python-dateutil=2.7.3=py36_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py36_0
  - pyyaml=5.1=py36h7b6447c_0
  - pyzmq=17.0.0=py36h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py36he2e5f8d_1
  - s3transfer=0.1.13=py36_0
  - scikit-learn=0.20.3=py36hd81dba3_0
  - scipy=1.1.0=py36h7c811a0_2
  - setuptools=39.1.0=py36_0
  - simplegeneric=0.8.1=py36_2
  - simplejson=3.16.0=py36h14c3975_0
  - singledispatch=3.4.0.3=py36h7a266c3_0
  - six=1.11.0=py36_1
  - smmap2=2.0.5=py36_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py36h035aef0_0
  - tabulate=0.8.3=py36_0
  - tensorboard=1.13.1=py36hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py36_1
  - testpath=0.3.1=py36_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36h14c3975_0
  - traitlets=4.3.2=py36h674d592_0
  - urllib3=1.22=py36hbe7ace6_0
  - virtualenv=16.0.0=py36_0
  - wcwidth=0.1.7=py36hdf4376a_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.31.1=py36_0
  - wrapt=1.11.1=py36h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py3.6_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py36_cu10.0.130_1
  - pip:
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - future==0.17.1
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python3

Python 2 on CPU clusters

name: null
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=2.0=cpu_0
  - _tflow_select=2.3.0=mkl
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27h5cde069_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cycler=0.10.0=py27hc7354d3_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=he6710b0_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27he6710b0_0
  - py-xgboost-cpu=0.90=py27_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow=1.13.1=mkl_py27h74ee40f_0
  - tensorflow-base=1.13.1=mkl_py27h7ce6ba3_0
  - tensorflow-estimator=1.13.0=py_0
  - tensorflow-mkl=1.13.1=h4fcabd2_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27hc38d2c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27h9e3e1ab_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch-cpu=1.1.0=py2.7_cpu_0
  - torchvision-cpu=0.3.0=py27_cuNone_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - torchvision==0.3.0
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Python 2 on GPU clusters

name: null
channels:
  - Databricks
  - pytorch
  - defaults
dependencies:
  - tensorflow=1.13.1.db1=gpu_py27h8e347d7_0
  - tensorflow-base=1.13.1.db1=gpu_py27he292aa2_0
  - tensorflow-gpu=1.13.1.db1=h0d30ee6_0
  - _libgcc_mutex=0.1=main
  - _py-xgboost-mutex=1.0=gpu_0
  - _tflow_select=2.1.0=gpu
  - absl-py=0.7.1=py27_0
  - asn1crypto=0.24.0=py27_0
  - astor=0.7.1=py27_0
  - backports=1.0=py_2
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports.weakref=1.0.post1=py_1
  - backports_abc=0.5=py_0
  - bcrypt=3.1.6=py27h7b6447c_0
  - blas=1.0=mkl
  - bleach=2.1.3=py27_0
  - boto=2.48.0=py27_1
  - boto3=1.7.62=py27h28b3542_1
  - botocore=1.10.62=py27h28b3542_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py27_0
  - cffi=1.11.5=py27he75722e_1
  - chardet=3.0.4=py27_1
  - click=7.0=py27_0
  - cloudpickle=0.8.0=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.7.3=py27_1
  - cryptography=2.2.2=py27h14c3975_0
  - cudnn=7.6.0=cuda10.0_0
  - cupti=10.0.130=0
  - cycler=0.10.0=py27_0
  - cython=0.28.2=py27h14c3975_0
  - decorator=4.3.0=py27_0
  - docutils=0.14=py27hae222c1_0
  - entrypoints=0.2.3=py27_2
  - enum34=1.1.6=py27_1
  - et_xmlfile=1.0.1=py27h75840f5_0
  - flask=1.0.2=py27_1
  - freetype=2.8=hab7d2ae_1
  - funcsigs=1.0.2=py27_0
  - functools32=3.2.3.2=py27_1
  - future=0.17.1=py27_0
  - futures=3.2.0=py27_0
  - gast=0.2.2=py27_0
  - gitdb2=2.0.5=py27_0
  - gitpython=2.1.11=py27_0
  - gmp=6.1.2=h6c8ec71_1
  - grpcio=1.12.1=py27hdbcaa40_0
  - gunicorn=19.9.0=py27_0
  - h5py=2.8.0=py27h989c5e5_3
  - hdf5=1.10.2=hba1933b_1
  - html5lib=1.0.1=py27_0
  - icu=58.2=h9c2bf20_1
  - idna=2.6=py27h5722d68_1
  - intel-openmp=2018.0.0=8
  - ipaddress=1.0.22=py27_0
  - ipython=5.7.0=py27_0
  - ipython_genutils=0.2.0=py27h89fb69b_0
  - itsdangerous=0.24=py27_1
  - jdcal=1.4=py27_0
  - jinja2=2.10=py27_0
  - jmespath=0.9.4=py_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py27h7ed5aa4_0
  - jupyter_client=5.2.3=py27_0
  - jupyter_core=4.4.0=py27_0
  - keras=2.2.4=0
  - keras-applications=1.0.8=py_0
  - keras-base=2.2.4=py27_0
  - keras-preprocessing=1.1.0=py_1
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.3.0=hdf63c60_0
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libpng=1.6.34=hb9fc6fc_0
  - libpq=10.4=h1ad7b7a_0
  - libprotobuf=3.8.0=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - libtiff=4.0.9=he85c1e1_2
  - libxgboost=0.90=h688424c_0
  - libxml2=2.9.8=h26e45fe_1
  - libxslt=1.1.32=h1312cb7_0
  - linecache2=1.0.0=py27_0
  - llvmlite=0.23.1=py27hdbcaa40_0
  - lxml=4.2.1=py27h23eabaa_0
  - mako=1.0.10=py_0
  - markdown=3.1.1=py27_0
  - markupsafe=1.0=py27h14c3975_1
  - mistune=0.8.3=py27h14c3975_1
  - mkl=2019.4=243
  - mkl_fft=1.0.12=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - mock=3.0.5=py27_0
  - msgpack-python=0.5.6=py27h6bb024c_1
  - nbconvert=5.3.1=py27_0
  - nbformat=4.4.0=py27hed7f2b2_0
  - ncurses=6.1=he6710b0_1
  - ninja=1.9.0=py27hfd86e86_0
  - numba=0.38.0=py27h637b7d7_0
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.45.1=py27_0
  - openpyxl=2.5.3=py27_0
  - openssl=1.0.2o=h14c3975_1
  - pandas=0.23.0=py27h637b7d7_0
  - pandocfilters=1.4.2=py27_1
  - paramiko=2.4.2=py27_0
  - pathlib2=2.3.2=py27_0
  - patsy=0.5.0=py27_0
  - pexpect=4.5.0=py27_0
  - pickleshare=0.7.4=py27h09770e1_0
  - pillow=5.1.0=py27h3deb7b8_0
  - pip=10.0.1=py27_0
  - ply=3.11=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.8.0=py27he6710b0_0
  - psycopg2=2.7.5=py27hb7f436b_0
  - ptyprocess=0.5.2=py27h4ccb14c_0
  - py-xgboost=0.90=py27h688424c_0
  - py-xgboost-gpu=0.90=py27h28bbb66_0
  - pyasn1=0.4.5=py_0
  - pycparser=2.18=py27_1
  - pygments=2.2.0=py27_0
  - pynacl=1.3.0=py27h7b6447c_0
  - pyopenssl=18.0.0=py27_0
  - pyparsing=2.2.0=py27_1
  - pysocks=1.6.8=py27_0
  - python=2.7.15=h1571d57_0
  - python-dateutil=2.7.3=py27_0
  - python-editor=1.0.4=py_0
  - pytz=2018.4=py27_0
  - pyyaml=5.1=py27h7b6447c_0
  - pyzmq=17.0.0=py27h14c3975_3
  - readline=7.0=h7b6447c_5
  - requests=2.18.4=py27hc5b0589_1
  - s3transfer=0.1.13=py27_0
  - scandir=1.7=py27h14c3975_0
  - scikit-learn=0.20.3=py27hd81dba3_0
  - scipy=1.1.0=py27h7c811a0_2
  - setuptools=39.1.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - simplejson=3.16.0=py27h14c3975_0
  - singledispatch=3.4.0.3=py27h9bcb476_0
  - six=1.11.0=py27_1
  - smmap2=2.0.5=py27_0
  - sqlite=3.23.1=he433501_0
  - sqlparse=0.3.0=py_0
  - statsmodels=0.9.0=py27h035aef0_0
  - tabulate=0.8.3=py27_0
  - tensorboard=1.13.1=py27hf484d3e_0
  - tensorflow-estimator=1.13.0=py_0
  - termcolor=1.1.0=py27_1
  - testpath=0.3.1=py27_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py27h14c3975_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27hd6ce930_0
  - unittest2=1.1.0=py27_0
  - urllib3=1.22=py27ha55213b_0
  - virtualenv=16.0.0=py27_0
  - wcwidth=0.1.7=py27_0
  - webencodings=0.5.1=py27_1
  - werkzeug=0.14.1=py27_0
  - wheel=0.31.1=py27_0
  - wrapt=1.11.1=py27h7b6447c_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.2.5=hf484d3e_1
  - zlib=1.2.11=h7b6447c_3
  - pytorch=1.1.0=py2.7_cuda10.0.130_cudnn7.5.1_0
  - torchvision=0.3.0=py27_cu10.0.130_1
  - pip:
    - backports.functools-lru-cache==1.5
    - backports.ssl-match-hostname==3.7.0.1
    - databricks-cli==0.8.7
    - docker==4.0.2
    - fusepy==2.0.4
    - horovod==0.16.4
    - hyperopt==0.1.2.db6
    - kiwisolver==1.1.0
    - matplotlib==2.2.2
    - mleap==0.8.1
    - mlflow==1.0.0
    - msgpack==0.5.6
    - networkx==2.2
    - nose==1.3.7
    - nose-exclude==0.5.0
    - psutil==5.6.3
    - pyarrow==0.13.0
    - pymongo==3.8.0
    - querystring-parser==1.2.3
    - seaborn==0.8.1
    - subprocess32==3.5.4
    - tensorboardx==1.7
    - tqdm==4.32.2
    - websocket-client==0.56.0
prefix: /databricks/python2

Spark packages containing Python modules

Spark Package Python Module Version
graphframes graphframes 0.7.0-db1-spark2.4
spark-deep-learning sparkdl 1.5.0-db4-spark2.4
tensorframes tensorframes 0.7.0-s_2.11

R libraries

The R libraries are identical to the R Libraries in Databricks Runtime 5.5.

Java and Scala libraries (Scala 2.11 cluster)

In addition to Java and Scala libraries in Databricks Runtime 5.5, Databricks Runtime 5.5 LTS for Machine Learning contains the following JARs:

Group ID Artifact ID Version
com.databricks spark-deep-learning 1.5.0-db4-spark2.4
com.typesafe.akka akka-actor_2.11 2.3.11
ml.combust.mleap mleap-databricks-runtime_2.11 0.13.0
ml.dmlc xgboost4j 0.90
ml.dmlc xgboost4j-spark 0.90
org.graphframes graphframes_2.11 0.7.0-db1-spark2.4
org.tensorflow libtensorflow 1.13.1
org.tensorflow libtensorflow_jni 1.13.1
org.tensorflow spark-tensorflow-connector_2.11 1.13.1
org.tensorflow tensorflow 1.13.1
org.tensorframes tensorframes 0.7.0-s_2.11