Databricks Runtime 6.5Databricks Runtime 6.5

• 08/04/2020
• • m

Databricks 于 2020 年 4 月发布了此映像。Databricks released this image in April 2020.

以下发行说明提供了由 Apache Spark 支持的 Databricks Runtime 6.5 的相关信息。The following release notes provide information about Databricks Runtime 6.5, powered by Apache Spark.

新增功能New features

对 Delta 表进行的所有写入、更新和删除操作的操作指标现在都显示在表历史记录中Operation metrics for all writes, updates, and deletes on a Delta table now shown in table history

现在，在表的历史记录中，你可以看到对 Delta 表进行的所有写入、更新和删除操作的操作指标（例如，更改的文件和行的数量）。You can now see operation metrics (for example, number of files and rows changed) for all writes, updates, and deletes on a Delta table in the table’s history. 生成这些指标的功能在默认情况下是启用的。The generation of these metrics is enabled by default. 可以通过运行以下命令来查看历史记录：You can see the history by running:

DESCRIBE HISTORY deltaTableName

对 Delta Lake 流式处理微批中处理的数据进行速率限制Rate-limit the data processed in Delta Lake streaming micro-batches

现在，在使用结构化流式处理从 Delta Lake 中处理数据时，可以设置 maxBytesPerTrigger 选项，以便对每个微批中处理多少数据进行速率限制。When you process data from Delta Lake using Structured Streaming, you can now set the option maxBytesPerTrigger to rate-limit how much data gets processed in each micro-batch. 这会设置“软最大值”，即批处理大约可以处理这一数量的数据，并且可能处理超出该限制的数据量。This sets a “soft max,” meaning that a batch processes approximately this amount of data and may process more than the limit.

如果将 Trigger.Once 用于流式处理，此选项将被忽略。If you use Trigger.Once for your streaming, this option is ignored. 另外，如果将此选项与 maxFilesPerTrigger 结合使用，微批会持续处理数据，直至达到 maxFilesPerTrigger 或 maxBytesPerTrigger 限制。In addition, if you use this option in conjunction with maxFilesPerTrigger, the micro-batch processes data until either the maxFilesPerTrigger or maxBytesPerTrigger limit is reached.

改进Improvements

Snowflake 连接器已更新到 2.5.9Snowflake connector updated to 2.5.9

该更新后的连接器会将 Snowflake JDBC 驱动程序升级到 3.12.0。The updated connector promotes the Snowflake JDBC driver to 3.12.0.

Java 已升级到 1.8.0_242Java upgraded to 1.8.0_242

此版本引入了对信任定位点证书的新检查，以确保信任定位点是 CA 证书并包含正确的扩展。This version introduces new checks on trust anchor certificates to ensure that trust anchors are CA certificates and contain proper extensions. 信任定位点证书必须包含 ca 字段设置为 true 的基本约束扩展。Trust anchor certificates must include a Basic Constraints extension with the ca field set to true. 另外，如果这些证书包含密钥用法扩展，则必须设置 keyCertSign 位。In addition, if they include a Key Usage extension, the keyCertSign bit must be set.

如果使用自签名的信任定位点证书，可能会遇到这些新检查引入的以下异常：If you use self-signed trust anchor certificates, you may hit the following exceptions introduced by the new checks:

TrustAnchor with subject "..." is not a CA certificate

TrustAnchor with subject "..." does not have keyCertSign bit set in KeyUsage extension

若要继续使用自签名信任定位点证书，可以选择使用以下两种方法之一来解决上述异常：To keep using self-signed trust anchor certificates, you have two options to address the above exceptions:

• 将通过 Java 1.9.0_242 引入的 Java 系统属性 jdk.security.allowNonCaAnchor 设置为 true，以还原以前的行为。Set Java system property jdk.security.allowNonCaAnchor introduced with Java 1.9.0_242 to true to restore the previous behavior. 可使用 spark.driver.extraJavaOptions 和 spark.executor.extraJavaOptions 将此系统属性分别设置为 Spark 驱动程序和执行程序。You can use spark.driver.extraJavaOptions and spark.executor.extraJavaOptions to set this system property to Spark driver and executors, respectively.

• 在信任定位点证书中包含 ca 字段设置为 true 的基本约束扩展。Include a Basic Constraints extension with the ca field set to true to your trust anchor certificates. 如果信任定位点证书包含密钥用法扩展，请设置 keyCertSign 位。If your trust anchor certificates include a Key Usage extension, set keyCertSign bit.

  如果使用 Java Keytool 生成公钥/私钥对，请在生成信任定位点证书时设置 -ext BasicConstraints=ca:true。If you use Java Keytool to generate public/private key pairs, set -ext BasicConstraints=ca:true when generating trust anchor certificates. 如果信任定位点证书包含密钥用法扩展，请在生成信任定位点证书时设置 keyCertSign 位 -ext KeyUsage=...,keyCertSign。If your trust anchor certificates include a Key Usage extension, set keyCertSign bit -ext KeyUsage=...,keyCertSign when generating trust anchor certificates.

升级了 R 库Upgraded R libraries

升级了多个已安装的 R 库。Upgraded several installed R libraries. 请参阅已安装的 R 库。See Installed R Libraries.

弃用功能Deprecations

• 已弃用 boto 包，并且在即将发布的 Databricks Runtime 主版本中将会删除该包。The boto package is deprecated and will be removed in an upcoming major release of Databricks Runtime. Azure Databricks 建议改用 boto3。Azure Databricks recommends that you use boto3 instead.
• 在即将发布的 Databricks Runtime 的主版本中，将会删除以下 R 包：The following R packages will be removed in an upcoming major release of Databricks Runtime:
  • abindabind
  • bitopsbitops
  • carcar
  • carDatacarData
  • doMCdoMC
  • gbmgbm
  • h2oh2o
  • littlerlittler
  • lme4lme4
  • mapprojmapproj
  • mapsmaps
  • maptoolsmaptools
  • MatrixModelsMatrixModels
  • minqaminqa
  • mvtnormmvtnorm
  • nloptrnloptr
  • openxlsxopenxlsx
  • pbkrtestpbkrtest
  • pkgKittenpkgKitten
  • quantregquantreg
  • R.methodsS3R.methodsS3
  • R.ooR.oo
  • R.utilsR.utils
  • RcppEigenRcppEigen
  • RCurlRCurl
  • riorio
  • spsp
  • SparseMSparseM
  • statmodstatmod
  • zipzip

Azure Databricks 建议使用以下代码来安装并加载 Databricks Runtime 未安装的 R 包：Azure Databricks recommends using the following code to install and load R packages that are not installed by Databricks Runtime:

if (!require("package_name")) {
  install.packages("package_name")
  library("package_name")
}

Apache SparkApache Spark

Databricks Runtime 6.5 包括 Apache Spark 2.4.5。Databricks Runtime 6.5 includes Apache Spark 2.4.5. 此版本包含 Databricks Runtime 6.4 中包含的所有 Spark 修复和改进，以及对 Spark 进行的以下更多 bug 修复和改进：This release includes all Spark fixes and improvements included in Databricks Runtime 6.4, as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-27099] [SQL] 添加“xxhash64”以便将任意列哈希处理为长整型[SPARK-27099] [SQL] Add ‘xxhash64’ for hashing arbitrary columns to Long
• [SPARK-26151] [SQL] 返回错误 CSV 记录的部分结果[SPARK-26151] [SQL] Return partial results for bad CSV records
• [SPARK-30927] [SS] StreamingQueryManager 应避免保持对已终止的 StreamingQuery 的引用[SPARK-30927] [SS] StreamingQueryManager should avoid keeping reference to terminated StreamingQuery
• [SPARK-22590] 将 sparkContext.localproperties 复制到 BroadcastExchangeExec.executionContext 中的子线程[SPARK-22590] Copy sparkContext.localproperties to child thread in BroadcastExchangeExec.executionContext
• [SPARK-30671] emptyDataFrame 应该使用 LocalRelation[SPARK-30671] emptyDataFrame should use a LocalRelation
• [SPARK-30556] [SQL] 重置 SQLExecution withThreadLocalCaptured 中更改的状态[SPARK-30556] [SQL] Reset the status changed in SQLExecution withThreadLocalCaptured
• [SPARK-30763] [SQL] 为 regexp_extract 修复 java.lang.IndexOutOfBoundsException No group 1[SPARK-30763] [SQL] Fix java.lang.IndexOutOfBoundsException No group 1 for regexp_extract
• [SPARK-30806] [SQL] 在 UnboundedWindowFunctionFrame 中，每个组评估一次[SPARK-30806] [SQL] Evaluate once per group in UnboundedWindowFunctionFrame
• [SPARK-30731] 更新了已弃用的 Mkdocs 选项[SPARK-30731] Update deprecated Mkdocs option
• [SPARK-30857] [SQL] 修复了将 epoch 之前的时间戳截断到小时和天的问题[SPARK-30857] [SQL] Fix truncations of timestamps before the epoch to hours and days
• [SPARK-30798] 在 QueryExecution 中限定 Session.active 的范围[SPARK-30798] Scope Session.active in QueryExecution
• [SPARK-30793] [SQL] 修复了将 epoch 之前的时间戳截断到分钟和秒的问题[SPARK-30793] [SQL] Fix truncations of timestamps before the epoch to minutes and seconds
• [SPARK-30826] [SQL] 遵循向下推送到 parquet 的 StringStartsWith 中的引用大小写[SPARK-30826] [SQL] Respect reference case in StringStartsWith pushed down to parquet
• [SPARK-30797] [SQL] 在截断表中设置 ACL 时，将传统用户/组/其他权限设置为 ACL 条目[SPARK-30797] [SQL] Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table
• [SPARK-30065] [SQL] DataFrameNaFunctions.drop 应该处理重复的列[SPARK-30065] [SQL] DataFrameNaFunctions.drop should handle duplicate columns
• [SPARK-29890] [SQL] DataFrameNaFunctions.fill 应该处理重复的列[SPARK-29890] [SQL] DataFrameNaFunctions.fill should handle duplicate columns
• [SPARK-30310] [CORE] 解决了 SparkUncaughtExceptionHandler 中缺失匹配大小写的问题并添加了测试[SPARK-30310] [CORE] Resolve missing match case in SparkUncaughtExceptionHandler and added tests
• [SPARK-30512] 添加了专用的 boss 事件循环组[SPARK-30512] Added a dedicated boss event loop group

维护更新Maintenance updates

请参阅 Databricks Runtime 6.5 维护更新。See Databricks Runtime 6.5 maintenance updates.

系统环境System environment

• 操作系统：Ubuntu 16.04.6 LTSOperating System: Ubuntu 16.04.6 LTS
• Java：1.8.0_252Java: 1.8.0_252
• Scala：2.11.12Scala: 2.11.12
• Python：3.7.3Python: 3.7.3
• R：R 版本 3.6.3 (2020-02-29)R: R version 3.6.3 (2020-02-29)
• Delta Lake：0.5.0Delta Lake: 0.5.0

本节内容：In this section:

• 已安装的 Python 库Installed Python libraries
• 已安装的 R 库Installed R libraries
• 已安装的 Java 库和 Scala 库（Scala 2.11 群集版本）Installed Java and Scala libraries (Scala 2.11 cluster version)

已安装的 Python 库Installed Python libraries

已安装的 R 库Installed R libraries

R 库安装自 2020-02-26 的 Microsoft CRAN 快照。R libraries are installed from Microsoft CRAN snapshot on 2020-02-26.

已安装的 Java 库和 Scala 库（Scala 2.11 群集版本）Installed Java and Scala libraries (Scala 2.11 cluster version)