Databricks Runtime 6.0（不受支持） Databricks Runtime 6.0 (Unsupported)

• 08/10/2020
• • m

Databricks 于 2019 年 10 月发布了此映像。Databricks released this image in October 2019.

以下发行说明提供了由 Apache Spark 支持的 Databricks Runtime 6.0 的相关信息。The following release notes provide information about Databricks Runtime 6.0, powered by Apache Spark.

新功能 New features

Python 环境 Python environment

Databricks Runtime 6.0 包括了对 Python 和 Python 环境配置方式的重大更改，包括将 Python 升级到 3.7.3、优化已安装的 Python 包的列表，以及将那些安装的包升级到较新版本。Databricks Runtime 6.0 includes major changes to Python and the way Python environments are configured, including upgrading Python to 3.7.3, refining the list of installed Python packages, and upgrading those installed packages to newer versions. 有关详细信息，请参阅已安装的 Python 库。For details, see Installed Python libraries.

此外，如之前所公告的那样，Databricks Runtime 6.0 不支持 Python 2。In addition, as was previously announced, Databricks Runtime 6.0 does not support Python 2.

重大更改包括：Major changes include:

• 已将 Python 从 3.5.2 升级到 3.7.3。Upgraded Python from 3.5.2 to 3.7.3. 某些旧版本的 Python 包可能不能前向兼容 Python 3.7，因为它们依赖于与 Python 3.7 不兼容的旧版本 Cython。Some old versions of Python packages may not be forward compatible with Python 3.7 because they depend on old versions of Cython that are not compatible with Python 3.7. 安装此类包可能会触发类似于 ‘PyThreadState’ {‘struct _ts’} has no member named ‘exc_type’ 的错误（有关详细信息，请参阅 GitHub 问题 1978）。Installing such a package can trigger errors similar to ‘PyThreadState’ {‘struct _ts’} has no member named ‘exc_type’ (see GitHub issue 1978 for details). 请改为安装兼容 Python 3.7 的 Python 包版本。Install Python 3.7 compatible versions of Python packages instead.
• 重大包升级：Major package upgrades:
  • 将 boto3 升级到了 1.9.162boto3 to 1.9.162
  • 将 ipython 升级到了 7.4.0ipython to 7.4.0
  • 将 matplotlib 升级到了 3.0.3matplotlib to 3.0.3
  • 将 numpy 升级到了 1.16.2numpy to 1.16.2
  • 将 pandas 升级到了 0.24.2pandas to 0.24.2
  • 将 pyarrow 升级到了 0.13.0pyarrow to 0.13.0
• 与 Databricks Runtime 5.5 LTS 相比，新增了以下 Python 包：asn1crypto、backcall、jedi、kiwisolver、parso 和 PySocks。Compared with Databricks Runtime 5.5 LTS, the following Python packages are newly included: asn1crypto, backcall, jedi, kiwisolver, parso, and PySocks.
• 与 Databricks Runtime 5.5 LTS 相比，未安装以下 Python 包：ansi2html、brewer2mpl、colorama、configobj、enum34、et-xmlfile、freetype-py、funcsigs、fusepy、ggplot、html5lib、ipaddress、jdcal、Jinja2、llvmlite、lxml、MarkupSafe、mpld3、msgpack-python、ndg-httpsclient、numba、openpyxl、pathlib2、Pillow、ply、pyasn1、pypng、python-geohash、scour、simplejson 和 singledispatch。Compared with Databricks Runtime 5.5 LTS, the following Python packages are not installed: ansi2html, brewer2mpl, colorama, configobj, enum34, et-xmlfile, freetype-py, funcsigs, fusepy, ggplot, html5lib, ipaddress, jdcal, Jinja2, llvmlite, lxml, MarkupSafe, mpld3, msgpack-python, ndg-httpsclient, numba, openpyxl, pathlib2, Pillow, ply, pyasn1, pypng, python-geohash, scour, simplejson, and singledispatch.
• 不再支持 Python ggplot 对象上的 display 函数，因为 ggplot 包不兼容较新版本的 pandas。The display function on Python ggplot objects is no longer supported because the ggplot package is not compatible with newer version of pandas.
• 不支持将 PYSPARK_PYTHON 设置为 /databricks/python2/bin/python，因为 Databricks Runtime 6.0 不支持 Python 2。Setting PYSPARK_PYTHON to /databricks/python2/bin/python is not supported because Databricks Runtime 6.0 does not support Python 2. 具有此类设置的群集仍可以启动。A cluster with such setting can still start. 但是，Python 笔记本和 Python 命令将不工作，也就是说，Python 命令单元格将失败并出现“已取消”错误，且 Python shell failed to start 错误会出现在驱动程序日志中。However, Python notebooks and Python commands will not work, that is, Python command cells will fail with a “Canceled” error and a Python shell failed to start error will appear in the driver logs.
• 如果 PYSPARK_PYTHON 指向由 Virtualenv 管理的环境中的一个 Python 可执行文件，则会为初始化脚本和笔记本激活此环境。If PYSPARK_PYTHON points to a Python executable that is in an environment managed by Virtualenv , this environment will be activated for init scripts and notebooks. 你可以直接使用在激活的环境中定义的 python 和 pip 命令，不必指定这些命令的绝对位置。You can use python and pip commands that are defined in the activated environment directly without having to specify the absolute locations of these commands. 默认情况下，PYSPARK_PYTHON 设置为 /databricks/python3/bin/python。By default, PYSPARK_PYTHON is set to /databricks/python3/bin/python. 因此，默认情况下，对于初始化脚本和笔记本，python 指向 /databricks/python3/bin/python，pip 指向 /databricks/python3/bin/pip。Thus, by default, python points to /databricks/python3/bin/python and pip points to /databricks/python3/bin/pip for init scripts and notebooks. 如果你的 PYSPARK_PYTHON 指向的 Python 可执行文件不在由 Virtualenv 管理的环境中，或者你要编写初始化脚本来创建 PYSPARK_PYTHON 指定的 Python，则需要使用绝对路径来访问正确的 python 和 pip。If your PYSPARK_PYTHON points to a Python executable that is not in an environment managed by Virtualenv or if you are writing an init script to create the Python specified by PYSPARK_PYTHON, you will need to use absolute paths to access the correct python and pip. 启用了 Python 库隔离时（默认情况下启用），激活的环境仍是与 PYSPARK_PYTHON 关联的环境。When Python library isolation is enabled (it is enabled by default), the activated environment is still the environment that PYSPARK_PYTHON is associated with. 建议使用库实用工具修改与 Python 笔记本关联的独立环境。We recommend that you use Library utilities to modify the isolated environment associated with a Python notebook.

与 Delta Lake DML 命令对应的 Scala 和 Java APIScala and Java APIs for Delta Lake DML commands

你现在可以使用用于删除、更新和合并的编程 API 来修改 Delta 表中的数据。You can now modify data in Delta tables using programmatic APIs for delete, update, and merge. 这些 API 镜像了其对应 SQL 命令的语法和语义，非常适用于许多工作负荷，例如，缓变维度 (SCD) 操作、合并更改数据以进行复制，以及从流式处理查询进行更新插入。These APIs mirror the syntax and semantics of their corresponding SQL commands and are great for many workloads, for example, slowly changing dimension (SCD) operations, merging change data for replication, and upserts from streaming queries.

有关详细信息，请参阅表删除、更新和合并。For details, see Table deletes, updates, and merges.

与 Delta Lake 实用工具命令对应的 Scala 和 Java APIScala and Java APIs for Delta Lake utility commands

Databricks Runtime 现在具有与 vacuum 和 history 实用工具命令对应的编程 API。Databricks Runtime now has programmatic APIs for the vacuum and history utility commands. 这些 API 镜像了 Databricks Runtime 早期版本中提供的对应 SQL 命令的语法和语义。These APIs mirror the syntax and semantics of their corresponding SQL commands available in earlier versions of Databricks Runtime.

你可以对表运行 vacuum，以清理 Delta 表不再引用的在保留期阈值之前创建的文件。You can clean up files that are no longer referenced by a Delta table and are older than the retention threshold by running vacuum on the table. 对表运行 vacuum 命令将以递归方式清理与 Delta 表关联的目录。Running the vacuum command on the table recursively vacuums the directories associated with the Delta table. 文件的默认保留期阈值为 7 天。The default retention threshold for the files is 7 days. 运行 vacuum 后，无法再按时间顺序查看在保留期之前创建的版本。The ability to time travel back to a version older than the retention period is lost after running vacuum. vacuum 不会自动触发。vacuum is not triggered automatically.

通过运行 history 命令，可检索每次写入 Delta 表的操作、用户和时间戳等内容的相关信息。You can retrieve information on the operations, user, timestamp, and so on for each write to a Delta table by running the history command. 按时间倒序返回返回操作。The operations are returned in reverse chronological order. 默认情况下，表历史记录保留 30 天。By default, table history is retained for 30 days.

有关详细信息，请参阅表实用工具命令。For details, see Table utility commands.

适用于 Azure Lsv2 实例的 Delta 缓存Delta cache available for Azure Lsv2 instances

默认情况下，现在会为所有 Lsv2 实例启用 Delta 缓存。The Delta cache is now enabled by default for all Lsv2 instances.

使用本地文件 API 的优化存储Optimized storage using local file APIs

本地文件 API 非常有用，因为它们允许你像访问本地文件那样访问基础分布式对象存储中的文件。Local file APIs are useful as they allow you to access files from underlying distributed object storage as local files. 在 Databricks Runtime 6.0 中，我们增强了 FUSE 装载，使本地文件 API 能够满足关键限制。In Databricks Runtime 6.0, we enhanced the FUSE mount that enables local file APIs to address key limitations. Databricks Runtime 6.0 极大地提高了读写速度，并支持大于 2 GB 的文件。Databricks Runtime 6.0 significantly improves read and write speed and supports files that are larger than 2 GB. 如果需要更快、更可靠的读取和写入（例如针对分布式模型训练的读取和写入），则你会发现此增强功能特别有用。If you need faster and more reliable reads and writes such as for distributed model training, you would find this enhancement particularly useful. 此外，你将不需要将数据加载到工作负荷的本地存储，因此可以节省成本并提高工作效率。In addition, you would not need to load data to a local storage for your workloads, both saving cost and improving productivity.

有关详细信息，请参阅本地文件 API。For details, see Local file APIs.

每个笔记本单元格多个 matplotlib 绘图Multiple matplotlib plots per notebook cell

现在可以为每个笔记本单元格显示多个 matplotlib 绘图：You can now display multiple matplotlib plots per notebook cell:

改进 Improvements

• 已将 AWS SDK 升级到 1.11.596。Upgraded AWS SDK to 1.11.596.
• 已将 WASB 驱动程序中的 Azure 存储 SDK 升级到 7.0。Upgraded Azure Storage SDK in the WASB driver to 7.0.
• OPTIMIZE 现在提供了指标值（例如添加的文件数、删除的文件数，以及最大和最小文件大小）的摘要。OPTIMIZE now provides a summary of metrics such as number of files added, number of files removed, and max and min file size. 请参阅压缩（二进制打包）。See Compaction (bin-packing).

删除Removal

Databricks ML 模型导出已删除。Databricks ML Model Export is removed. 请改用 MLeap 来导入和导出模型。Use MLeap for importing and exporting models instead.

Apache SparkApache Spark

Databricks Runtime 6.0 包括 Apache Spark 2.4.3。Databricks Runtime 6.0 includes Apache Spark 2.4.3. 此版本包括 Databricks Runtime 5.5 LTS 中包含的所有 Spark 修复和改进，以及对 Spark 进行的以下其他 bug 修复和改进：This release includes all Spark fixes and improvements included in Databricks Runtime 5.5 LTS, as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-27992][SPARK-28881][PYTHON] 允许 Python 与连接线程联接以传播错误[SPARK-27992][SPARK-28881][PYTHON] Allow Python to join with connection thread to propagate errors
• [SPARK-27330][SS] 在 foreach 编写器中支持任务中止（6.0、5.x）[SPARK-27330][SS]support task abort in foreach writer (6.0, 5.x)
• [SPARK-28642][SQL] 在 SHOW CREATE TABLE 中隐藏凭据[SPARK-28642][SQL] Hide credentials in SHOW CREATE TABLE
• [SPARK-28699][CORE] 修复了中止不确定阶段的个别案例[SPARK-28699][CORE] Fix a corner case for aborting indeterminate stage
• [SPARK-28647][WEBUI] 恢复其他指标功能[SPARK-28647][WEBUI] Recover additional metric feature
• [SPARK-28766][R] [DOC] 修复对无效 URL 的 CRAN 传入可行性警告[SPARK-28766][R][DOC] Fix CRAN incoming feasibility warning on invalid URL
• [SPARK-28486][CORE] [PYTHON] 将 PythonBroadcast 的数据文件映射到 BroadcastBlock，以避免被系统通过垃圾回收删除[SPARK-28486][CORE][PYTHON] Map PythonBroadcast’s data file to a BroadcastBlock to avoid delete by GC
• [SPARK-25035][CORE] 避免在复制磁盘存储的块时进行内存映射[SPARK-25035][CORE] Avoiding memory mapping at disk-stored blocks replication
• [SPARK-27234][SS][PYTHON] 对 EpochTracker 中的当前 epoch 使用 InheritableThreadLocal（以支持 Python UDF）[SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
• [SPARK-28638][WEBUI] 任务摘要应该只包含成功任务的指标[SPARK-28638][WEBUI] Task summary should only contain successful tasks’ metrics
• [SPARK-28153][PYTHON] 在 InputFileBlockHolder 上使用 AtomicReference（以通过 Python UDF 支持 input_file_name）[SPARK-28153][PYTHON] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)
• [SPARK-28564][CORE] 访问历史记录应用程序默认为最后一个尝试 ID[SPARK-28564][CORE] Access history application defaults to the last attempt id
• [SPARK-28260] 群集可能会在 thriftserver 查询仍在提取结果时自动终止[SPARK-28260] Cluster can auto-terminate while thriftserver query is still fetching results
• [SPARK-26152][CORE] 将辅助角色清理与辅助角色关闭同步[SPARK-26152][CORE] Synchronize Worker Cleanup with Worker Shutdown
• [SPARK-28545][SQL] 将哈希映射大小添加到 ObjectAggregationIterator 的定向日志中[SPARK-28545][SQL] Add the hash map size to the directional log of ObjectAggregationIterator
• [SPARK-28489][SS] 修复了 KafkaOffsetRangeCalculator.getRanges 可能会删除偏移量的 bug[SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets
• [SPARK-28421][ML] SparseVector.apply 性能优化[SPARK-28421][ML] SparseVector.apply performance optimization
• [SPARK-28156][SQL] 自联接不应未命中缓存视图[SPARK-28156][SQL] Self-join should not miss cached view
• [SPARK-28152][SQL] 将 ShortType 映射到 SMALLINT，将 FloatType 映射到 REAL（适用于 MsSqlServerDialect）[SPARK-28152][SQL] Mapped ShortType to SMALLINT and FloatType to REAL for MsSqlServerDialect
• [SPARK-28054][SQL] 修复当分区名称为大写的情况下动态插入 Hive 分区表时的错误[SPARK-28054][SQL] Fix error when insert Hive partitioned table dynamically where partition name is upper case
• [SPARK-27159][SQL] 更新 mssql server 方言以支持二进制类型[SPARK-27159][SQL] update mssql server dialect to support binary type
• [SPARK-28355][CORE] [PYTHON] 将 Spark conf 用作阈值…[SPARK-28355][CORE][PYTHON] Use Spark conf for threshold at which com…
• [SPARK-27989][CORE] 在连接到 k8s 的驱动程序时添加了重试[SPARK-27989][CORE] Added retries on the connection to the driver for k8s
• [SPARK-27416][SQL] UnsafeMapData & UnsafeArrayData Kryo 序列化 …[SPARK-27416][SQL] UnsafeMapData & UnsafeArrayData Kryo serialization …
• [SPARK-28430][UI] 修复缺少某些任务的指标时的阶段表呈现[SPARK-28430][UI] Fix stage table rendering when some tasks’ metrics are missing
• [SPARK-27485] EnsureRequirements.reorder 应该会妥善处理重复表达式[SPARK-27485] EnsureRequirements.reorder should handle duplicate expressions gracefully
• [SPARK-28404][SS] 修复 RateStreamContinuousPartitionReader 中的负超时值[SPARK-28404][SS] Fix negative timeout value in RateStreamContinuousPartitionReader
• [SPARK-28378][PYTHON] 删除 cgi.escape 的使用[SPARK-28378][PYTHON] Remove usage of cgi.escape
• [SPARK-28371][SQL] 使 Parquet“StartsWith”筛选器 null 安全[SPARK-28371][SQL] Make Parquet “StartsWith” filter null-safe
• [SPARK-28015][SQL] 检查 stringToDate() 是否使用 yyyy 和 yyyy-[m]m 格式的整个输入[SPARK-28015][SQL] Check stringToDate() consumes entire input for the yyyy and yyyy-[m]m formats
• [SPARK-28302][CORE] 确保为 Windows 上的 SparkLauncher 生成唯一的输出文件[SPARK-28302][CORE] Make sure to generate unique output file for SparkLauncher on Windows
• [SPARK-28308][CORE] 解析 CalendarInterval 亚秒部分之前应将其填充[SPARK-28308][CORE] CalendarInterval sub-second part should be padded before parsing
• [SPARK-28170][ML] [PYTHON] 统一矢量和矩阵文档[SPARK-28170][ML][PYTHON] Uniform Vectors and Matrix documentation
• [SPARK-28160][CORE] 修复当未选中的异常缺失时回调函数可能挂起的 bug[SPARK-28160][CORE] Fix a bug that callback function may hang when unchecked exception missed
• [SPARK-27839][SQL] 更改 UTF8String.replace() 以对 UTF8 字节进行操作[SPARK-27839][SQL] Change UTF8String.replace() to operate on UTF8 bytes
• [SPARK-28157][CORE] 使 SHS 清除阻止列表中的条目的 KVStore LogInfo[SPARK-28157][CORE] Make SHS clear KVStore LogInfo for the blacklisted entries
• [SPARK-28128][PYTHON][SQL] pandas 分组的 UDF 会跳过空分区[SPARK-28128][PYTHON][SQL] pandas Grouped UDFs skip empty partitions
• [SPARK-28012][SQL] Hive UDF 支持结构类型可折叠表达式[SPARK-28012][SQL] Hive UDF supports struct type foldable expression
• [SPARK-28164] 修复 start-slave.sh 的用法描述[SPARK-28164] Fix usage description of start-slave.sh
• [SPARK-27100][SQL] 在 FilePartition 中使用 Array 而非 Seq 以防止 StackOverflowError[SPARK-27100][SQL] Use Array instead of Seq in FilePartition to prevent StackOverflowError
• [SPARK-28154][ML] GMM 修复双重缓存问题[SPARK-28154][ML] GMM fix double caching

维护更新Maintenance updates

请参阅 Databricks Runtime 6.0 维护更新。See Databricks Runtime 6.0 maintenance updates.

系统环境System environment

• 操作系统：Ubuntu 16.04.6 LTSOperating System: Ubuntu 16.04.6 LTS
• Java：1.8.0_232Java: 1.8.0_232
• Scala：2.11.12Scala: 2.11.12
• Python：3.7.3Python: 3.7.3
• R：R 版本 3.6.1 (2019-07-05)R: R version 3.6.1 (2019-07-05)
• Delta Lake：0.3.0Delta Lake: 0.3.0

已安装的 Python 库Installed Python libraries

已安装的 R 库 Installed R libraries

已安装的 Java 库和 Scala 库（Scala 2.11 群集版本）Installed Java and Scala libraries (Scala 2.11 cluster version)