带有 Conda 的 Databricks Runtime 5.4 (Beta)Databricks Runtime 5.4 with Conda (Beta)

重要

带有 Conda 的 Databricks Runtime 5.4 以 Beta 版本提供。Databricks Runtime 5.4 with Conda is in Beta. 在即将发布的 Beta 版本中,支持的环境的内容可能会发生变化。The contents of the supported environments may change in upcoming Beta releases. 更改可能包括包列表或已安装包的版本的列表。Changes can include the list of packages or versions of installed packages.

很高兴引入带有 Conda 的 Databricks Runtime 5.4,它可以让你利用 Conda 来管理 Python 库和环境。We’re excited to introduce Databricks Runtime 5.4 with Conda, which lets you take advantage of Conda to manage Python libraries and environments. 此运行时在创建群集时提供两个 Conda 根环境选项:This runtime offers two root Conda environment options at cluster creation:

  • Databricks Standard 环境包括许多常用 Python 包的更新版本。Databricks Standard environment includes updated versions of many popular Python packages. 此环境旨在替代在 Databricks Runtime 上运行的现有笔记本。This environment is intended as a drop-in replacement for existing notebooks that run on Databricks Runtime. 这是基于 Databricks Conda 的默认运行时环境。This is the default Databricks Conda-based runtime environment.
  • Databricks Minimal 环境包含 PySpark 和 Databricks Python 笔记本功能所需的最小数量的包。Databricks Minimal environment contains a minimum number of packages that are required for PySpark and Databricks Python notebook functionality. 这个环境非常适合使用各种 Python 包进行运行时自定义。This environment is ideal if you want to customize the runtime with various Python packages.

两者都包括对 Databricks 库实用工具的支持。Both include support for Databricks Library utilities.

备注

带有 Conda 的 Databricks Runtime 5.4 中的 Scala、Java 和 R 库与 Databricks Runtime 5.4 中的相同。The Scala, Java, and R libraries in Databricks Runtime 5.4 with Conda are identical to those in Databricks Runtime 5.4. 有关详细信息,请参阅 Databricks Runtime 5.4(不受支持)发行说明。For details, see the Databricks Runtime 5.4 (Unsupported) release notes. 有关如何使用带有 Conda 的 Databricks Runtime 的信息,请参阅带有 Conda 的 Databricks RuntimeFor information about how to use Databricks Runtime with Conda, see Databricks Runtime with Conda.

系统环境System environment

带有 Conda 的 Databricks Runtime 5.4 中的系统环境与 Databricks Runtime 5.4 不同,如下所示:The system environment in Databricks Runtime 5.4 with Conda differs from Databricks Runtime 5.4 as follows:

  • Python:3.7.x.Python: 3.7.x. 仅支持 Python 3。Only Python 3 is supported.

Libraries

下面是带有 Conda 的 Databricks Runtime 5.4 上的默认根环境的导出的 environment.yml 文件。The following is the exported environment.yml file for default root environments on Databricks Runtime 5.4 with Conda.

Databricks StandardDatabricks Standard

name: databricks-standard
channels:
  - defaults
dependencies:
  - asn1crypto=0.24.0=py37_0
  - backcall=0.1.0=py37_0
  - blas=1.0=openblas
  - boto=2.49.0=py37_0
  - boto3=1.9.111=py_0
  - botocore=1.12.112=py_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.8.24=py37_1
  - cffi=1.11.5=py37he75722e_1
  - chardet=3.0.4=py37_1
  - cryptography=2.3.1=py37hc365091_0
  - cython=0.28.5=py37hf484d3e_0
  - decorator=4.3.0=py37_0
  - docutils=0.14=py37_0
  - idna=2.7=py37_0
  - ipython=6.5.0=py37_0
  - ipython_genutils=0.2.0=py37_0
  - jedi=0.12.1=py37_0
  - jmespath=0.9.4=py_0
  - krb5=1.16.1=hc83ff2d_6
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libopenblas=0.3.3=h5a2b251_3
  - libpq=10.5=h1ad7b7a_0
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - nomkl=3.0=0
  - numpy=1.15.1=py37h99e49ec_0
  - numpy-base=1.15.1=py37h2f8d375_0
  - openssl=1.0.2p=h14c3975_0
  - pandas=0.23.4=py37h04863e7_0
  - parso=0.3.1=py37_0
  - patsy=0.5.0=py37_0
  - pexpect=4.6.0=py37_0
  - pickleshare=0.7.4=py37_0
  - pip=10.0.1=py37_0
  - prompt_toolkit=1.0.15=py37_0
  - psycopg2=2.7.5=py37hb7f436b_0
  - ptyprocess=0.6.0=py37_0
  - pycparser=2.18=py37_1
  - pygments=2.2.0=py37_0
  - pyopenssl=18.0.0=py37_0
  - pysocks=1.6.8=py37_0
  - python=3.7.0=hc3d631a_0
  - python-dateutil=2.7.3=py37_0
  - pytz=2018.5=py37_0
  - readline=7.0=h7b6447c_5
  - requests=2.19.1=py37_0
  - s3transfer=0.2.0=py37_0
  - scikit-learn=0.19.2=py37h22eb022_0
  - scipy=1.1.0=py37he2b7bc3_2
  - setuptools=40.2.0=py37_0
  - simplegeneric=0.8.1=py37_2
  - six=1.11.0=py37_1
  - sqlite=3.24.0=h84994c4_0
  - statsmodels=0.9.0=py37h035aef0_0
  - tk=8.6.8=hbc83047_0
  - traitlets=4.3.2=py37_0
  - urllib3=1.23=py37_0
  - wcwidth=0.1.7=py37_0
  - wheel=0.31.1=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - cycler==0.10.0
    - kiwisolver==1.1.0
    - matplotlib==2.2.3
    - pyarrow==0.12.0
    - pyparsing==2.4.0
    - seaborn==0.9.0
prefix: /databricks/conda/envs/databricks-standard

Databricks MinimalDatabricks Minimal

name: databricks-minimal
channels:
  - defaults
dependencies:
  - backcall=0.1.0=py37_0
  - blas=1.0=openblas
  - ca-certificates=2018.03.07=0
  - certifi=2018.8.24=py37_1
  - decorator=4.3.0=py37_0
  - ipython=6.5.0=py37_0
  - ipython_genutils=0.2.0=py37_0
  - jedi=0.12.1=py37_0
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libopenblas=0.3.3=h5a2b251_3
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - nomkl=3.0=0
  - numpy=1.15.1=py37h99e49ec_0
  - numpy-base=1.15.1=py37h2f8d375_0
  - openssl=1.0.2p=h14c3975_0
  - pandas=0.23.4=py37h04863e7_0
  - parso=0.3.1=py37_0
  - pexpect=4.6.0=py37_0
  - pickleshare=0.7.4=py37_0
  - pip=10.0.1=py37_0
  - prompt_toolkit=1.0.15=py37_0
  - ptyprocess=0.6.0=py37_0
  - pygments=2.2.0=py37_0
  - python=3.7.0=hc3d631a_0
  - python-dateutil=2.7.3=py37_0
  - pytz=2018.5=py37_0
  - readline=7.0=h7b6447c_5
  - setuptools=40.2.0=py37_0
  - simplegeneric=0.8.1=py37_2
  - six=1.11.0=py37_1
  - sqlite=3.24.0=h84994c4_0
  - tk=8.6.8=hbc83047_0
  - traitlets=4.3.2=py37_0
  - wcwidth=0.1.7=py37_0
  - wheel=0.31.1=py37_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
  - pip:
    - pyarrow==0.12.0
prefix: /databricks/conda/envs/databricks-minimal