Databricks Runtime 7.0（不受支持）Databricks Runtime 7.0 (Unsupported)

• 01/14/2021
• • m

Databricks 于 2020 年 6 月发布了此映像。Databricks released this image in June 2020.

以下发行说明提供 Apache Spark 3.0 支持的 Databricks Runtime 7.0 的相关信息。The following release notes provide information about Databricks Runtime 7.0, powered by Apache Spark 3.0.

新增功能New features

Databricks Runtime 7.0 包括以下新功能：Databricks Runtime 7.0 includes the following new features:

• Scala 2.12Scala 2.12

  Databricks Runtime 7.0 将 Scala 从 2.11.12 升级到 2.12.10。Databricks Runtime 7.0 upgrades Scala from 2.11.12 to 2.12.10. Scala 2.12 和 2.11 之间的更改列表位于 Scala 2.12.0 发行说明中。The change list between Scala 2.12 and 2.11 is in the Scala 2.12.0 release notes.

• 在 Databricks Runtime 6.4 中发布的 自动加载程序（公共预览版） 已在 Databricks Runtime 7.0 中得到改进Auto Loader (Public Preview), released in Databricks Runtime 6.4, has been improved in Databricks Runtime 7.0

  当新数据文件在 ETL 过程中到达云 blob 存储时，你可以借助自动加载程序更高效地以增量方式处理这些数据文件。Auto Loader gives you a more efficient way to process new data files incrementally as they arrive on a cloud blob store during ETL. 这是对基于文件的结构化流式处理的改进，它通过重复列出云目录和跟踪已看到的文件来标识新文件。随着目录的增长，它的效率可能会非常低。This is an improvement over file-based structured streaming, which identifies new files by repeatedly listing the cloud directory and tracking the files that have been seen, and can be very inefficient as the directory grows. 与基于文件通知的结构化流式处理相比，自动加载程序也更方便、更有效，这要求你在云中手动配置文件通知服务，而不允许回填现有文件。Auto Loader is also more convenient and effective than file-notification-based structured streaming, which requires that you manually configure file-notification services on the cloud and doesn’t let you backfill existing files. 有关详细信息，请参阅使用自动加载程序从 Azure Blob 存储、Azure Data Lake Storage Gen1（受限）或 Azure Data Lake Storage Gen2 加载文件。For details, see Load files from Azure Blob storage, Azure Data Lake Storage Gen1 (limited), or Azure Data Lake Storage Gen2 using Auto Loader.

  在 Databricks Runtime 7.0 上，不再需要请求自定义 Databricks Runtime 映像即可使用自动加载程序。On Databricks Runtime 7.0 you no longer need to request a custom Databricks Runtime image in order to use Auto Loader.

• COPY INTO（公共预览版） ，已在 Databricks Runtime 7.0 中得到改进，它允许你通过幂等重试将数据加载到 Delta Lake。COPY INTO (Public Preview), which lets you load data into Delta Lake with idempotent retries, has been improved in Databricks Runtime 7.0

  COPY INTO SQL 命令在 6.4 Databricks Runtime 中以公共预览版形式发布，允许你通过幂等重试将数据加载到 Delta Lake。Released as a Public Preview in Databricks Runtime 6.4, the COPY INTO SQL command lets you load data into Delta Lake with idempotent retries. 若要将数据加载到 Delta Lake，现在必须使用 Apache Spark 数据帧 API。To load data into Delta Lake today you have to use Apache Spark DataFrame APIs. 如果在加载过程中出现故障，则必须有效地处理它们。If there are failures during loads, you have to handle them effectively. 新的 COPY INTO 命令提供了一种熟悉的声明性接口，用于通过 SQL 加载数据。The new COPY INTO command provides a familiar declarative interface to load data in SQL. 此命令会跟踪以前加载的文件，在出现故障时你可以安全地重新运行它。The command keeps track of previously loaded files and you safely re-run it in case of failures. 有关详细信息，请参阅 COPY INTO（Azure Databricks 上的 Delta Lake）。For details, see COPY INTO (Delta Lake on Azure Databricks).

改进Improvements

• Azure Synapse（以前称为 SQL 数据仓库）连接器支持 COPY 语句。Azure Synapse (formerly SQL Data Warehouse) connector supports the COPY statement.

  COPY 的主要优点是较低权限的用户可以将数据写入到 Azure Synapse，而无需在 Azure Synapse 上拥有严格的 CONTROL 权限。The main benefit of COPY is that lower privileged users can write data to Azure Synapse without needing strict CONTROL permissions on Azure Synapse.

• 在笔记本单元格中以内联方式显示 Matplolib 对象不再需要 %matplotlib inline magic 命令。The %matplotlib inline magic command is no longer required to display Matplolib objects inline in notebook cells. 默认情况下，它们始终以内联方式显示。They are always displayed inline by default.

• 现在可以使用 transparent=False 来呈现 Matplolib 图，这样就不会丢失用户指定的背景。Matplolib figures are now rendered with transparent=False, so that user-specified backgrounds are not lost. 可以通过设置 Spark 配置 spark.databricks.workspace.matplotlib.transparent true 来重写此行为。This behavior can be overridden by setting Spark configuration spark.databricks.workspace.matplotlib.transparent true.

• 在“高并发性”模式群集上运行结构化流式处理生产作业时，作业的重启有时会失败，因为以前运行的作业未正确终止。When running Structured Streaming production jobs on High Concurrency mode clusters, restarts of a job would occasionally fail, because the previously running job wasn’t terminated properly. Databricks Runtime 6.3 引入了在群集上设置 SQL 配置 spark.sql.streaming.stopActiveRunOnRestart true 的功能，以确保以前的运行停止。Databricks Runtime 6.3 introduced the ability to set the SQL configuration spark.sql.streaming.stopActiveRunOnRestart true on your cluster to ensure that the previous run stops. 默认情况下，会在 Databricks Runtime 7.0 中设置此配置。This configuration is set by default in Databricks Runtime 7.0.

主要库更改Major library changes

Python 包Python packages

升级的主要 Python 包：Major Python packages upgraded:

• boto3 1.9.162 -> 1.12.0boto3 1.9.162 -> 1.12.0
• matplotlib 3.0.3 -> 3.1.3matplotlib 3.0.3 -> 3.1.3
• numpy 1.16.2 -> 1.18.1numpy 1.16.2 -> 1.18.1
• pandas 0.24.2 -> 1.0.1pandas 0.24.2 -> 1.0.1
• pip 19.0.3 -> 20.0.2pip 19.0.3 -> 20.0.2
• pyarrow 0.13.0 -> 0.15.1pyarrow 0.13.0 -> 0.15.1
• psycopg2 2.7.6 -> 2.8.4psycopg2 2.7.6 -> 2.8.4
• scikit-learn 0.20.3 -> 0.22.1scikit-learn 0.20.3 -> 0.22.1
• scipy 1.2.1 -> 1.4.1scipy 1.2.1 -> 1.4.1
• seaborn 0.9.0 -> 0.10.0seaborn 0.9.0 -> 0.10.0

删除的 Python 包：Python packages removed:

• boto（使用 boto3）boto (use boto3)
• pycurlpycurl

R 包R packages

添加的 R 包：R packages added:

• broombroom
• highrhighr
• isobandisoband
• knitrknitr
• markdownmarkdown
• modelrmodelr
• reprexreprex
• rmarkdownrmarkdown
• rvestrvest
• selectrselectr
• tidyversetidyverse
• tinytextinytex
• xfunxfun

删除的 R 包：R packages removed:

• abindabind
• bitopsbitops
• carcar
• carDatacarData
• doMCdoMC
• gbmgbm
• h2oh2o
• littlerlittler
• lme4lme4
• mapprojmapproj
• mapsmaps
• maptoolsmaptools
• MatrixModelsMatrixModels
• minqaminqa
• mvtnormmvtnorm
• nloptrnloptr
• openxlsxopenxlsx
• pbkrtestpbkrtest
• pkgKittenpkgKitten
• quantregquantreg
• R.methodsS3R.methodsS3
• R.ooR.oo
• R.utilsR.utils
• RcppEigenRcppEigen
• RCurlRCurl
• riorio
• spsp
• SparseMSparseM
• statmodstatmod
• zipzip

Java 和 Scala 库Java and Scala libraries

• 用于处理 Hive 用户定义函数和 Hive SerDes 的 Apache Hive 版本已升级到 2.3。Apache Hive version used for handling Hive user-defined functions and Hive SerDes upgraded to 2.3.
• 以前，Azure 存储和 Key Vault jar 作为 Databricks Runtime 的一部分打包，这会阻止你使用附加到群集的这些库的不同版本。Previously Azure Storage and Key Vault jars were packaged as part of Databricks Runtime, which would prevent you from using different versions of those libraries attached to clusters. com.microsoft.azure.storage 和 com.microsoft.azure.keyvault 下的类不再位于 Databricks Runtime 中的类路径上。Classes under com.microsoft.azure.storage and com.microsoft.azure.keyvault are no longer on the class path in Databricks Runtime. 如果你依赖于这些类路径中的任何一个，则现在必须将 Azure 存储 SDK 或 Azure Key Vault SDK 附加到群集。If you depend on either of those class paths, you must now attach Azure Storage SDK or Azure Key Vault SDK to your clusters.

行为更改Behavior changes

本部分列出从 Databricks Runtime 6.6 到 Databricks Runtime 7.0 的行为变更。This section lists behavior changes from Databricks Runtime 6.6 to Databricks Runtime 7.0. 在将工作负载从较低的 Databricks Runtime 版本迁移到 Databricks Runtime 7.0 及更高版本时，应注意到这些行为变更。You should be aware of these as you migrate workloads from lower Databricks Runtime releases to Databricks Runtime 7.0 and above.

Spark 行为变更Spark behavior changes

由于 Databricks Runtime 7.0 是在 Spark 3.0 上构建的第一个 Databricks Runtime，因此从在 Spark 2.4 上构建的 Databricks Runtime 5.5 LTS 或 6.x 迁移工作负载时，有许多更改应予以注意。Because Databricks Runtime 7.0 is the first Databricks Runtime built on Spark 3.0, there are many changes that you should be aware of when you migrate workloads from Databricks Runtime 5.5 LTS or 6.x, which are built on Spark 2.4. 此发行说明文章的 Apache Spark 部分的每个功能区域的“行为变更”部分中列出了这些更改：These changes are listed in the “Behavior changes” section of each functional area in the Apache Spark section of this release notes article:

• Spark Core、Spark SQL 和结构化流式处理的行为变更Behavior changes for Spark core, Spark SQL, and Structured Streaming
• MLlib 的行为变更Behavior changes for MLlib
• SparkR 的行为变更Behavior changes for SparkR

其他行为变更Other behavior changes

• 升级到 Scala 2.12 涉及以下更改：The upgrade to Scala 2.12 involves the following changes:

  • 包单元序列化的处理方式不同。Package cell serialization is handled differently. 下面的示例演示了行为变更以及如何处理它。The following example illustrates the behavior change and how to handle it.

    运行以下包单元中定义的 foo.bar.MyObjectInPackageCell.run() 会触发“java.lang.NoClassDefFoundError: Could not initialize class foo.bar.MyObjectInPackageCell$”错误Running foo.bar.MyObjectInPackageCell.run() as defined in the following package cell will trigger the error java.lang.NoClassDefFoundError: Could not initialize class foo.bar.MyObjectInPackageCell$

    package foo.bar
    
    case class MyIntStruct(int: Int)
    
    import org.apache.spark.sql.SparkSession
    import org.apache.spark.sql.functions._
    import org.apache.spark.sql.Column
    
    object MyObjectInPackageCell extends Serializable {
    
      // Because SparkSession cannot be created in Spark executors,
      // the following line triggers the error
      // Could not initialize class foo.bar.MyObjectInPackageCell$
      val spark = SparkSession.builder.getOrCreate()
    
      def foo: Int => Option[MyIntStruct] = (x: Int) => Some(MyIntStruct(100))
    
      val theUDF = udf(foo)
    
      val df = {
        val myUDFInstance = theUDF(col("id"))
        spark.range(0, 1, 1, 1).withColumn("u", myUDFInstance)
      }
    
      def run(): Unit = {
        df.collect().foreach(println)
      }
    }
    

    若要解决此错误，可以将 MyObjectInPackageCell 包装到可序列化的类中。To work around this error, you can wrap MyObjectInPackageCell inside a serializable class.

  • 某些使用 DataStreamWriter.foreachBatch 的情况将需要源代码更新。Certain cases using DataStreamWriter.foreachBatch will require a source code update. 此更改的原因是 Scala 2.12 会自动将 lambda 表达式转换为 SAM 类型，而这可能会导致多义性。This change is due to the fact that Scala 2.12 has automatic conversion from lambda expressions to SAM types and can cause ambiguity.

    例如，以下 Scala 代码无法编译：For example, the following Scala code can’t compile:

    streams
      .writeStream
      .foreachBatch { (df, id) => myFunc(df, id) }
    

    若要修复此编译错误，请将 foreachBatch { (df, id) => myFunc(df, id) } 更改为 foreachBatch(myFunc _) 或显式使用 Java API：foreachBatch(new VoidFunction2 ...)。To fix the compilation error, change foreachBatch { (df, id) => myFunc(df, id) } to foreachBatch(myFunc _) or use the Java API explicitly: foreachBatch(new VoidFunction2 ...).

• 由于用于处理 Hive 用户定义函数和 Hive SerDes 的 Apache Hive 版本已升级到 2.3，因此需要两项更改：Because the Apache Hive version used for handling Hive user-defined functions and Hive SerDes is upgraded to 2.3, two changes are required:

  • Hive 的 SerDe 接口由 AbstractSerDe 抽象类取代。Hive’s SerDe interface is replaced by an abstract class AbstractSerDe. 对于任何自定义 Hive SerDe 实现，需要迁移到 AbstractSerDe。For any custom Hive SerDe implementation, migrating to AbstractSerDe is required.
  • 将 spark.sql.hive.metastore.jars 设置为 builtin 意味着将使用 Hive 2.3 元存储客户端来访问 Databricks Runtime 7.0 的元存储。Setting spark.sql.hive.metastore.jars to builtin means that the Hive 2.3 metastore client will be used to access metastores for Databricks Runtime 7.0. 如果需要访问基于 Hive 1.2 的外部元存储，请将 spark.sql.hive.metastore.jars 设置为包含 Hive 1.2 jar 的文件夹。If you need to access Hive 1.2 based external metastores, set spark.sql.hive.metastore.jars to the folder that contains Hive 1.2 jars.

弃用的功能和删除的功能Deprecations and removals

• “跳过数据”索引已在 Databricks Runtime 4.3 中弃用，并已在 Databricks Runtime 7.0 中删除。Data skipping index was deprecated in Databricks Runtime 4.3 and removed in Databricks Runtime 7.0. 建议改用 Delta 表，它提供了经过改进的数据跳过功能。We recommend that you use Delta tables instead, which offer improved data skipping capabilities.
• 在 Databricks Runtime 7.0 中，Apache Spark 的基础版本使用 Scala 2.12。In Databricks Runtime 7.0, the underlying version of Apache Spark uses Scala 2.12. 由于针对 Scala 2.11 编译的库可能会以意外的方式禁用 Databricks Runtime 7.0 群集，运行 Databricks Runtime 7.0 及更高版本的群集不会安装配置为在所有群集上安装的库。Since libraries compiled against Scala 2.11 can disable Databricks Runtime 7.0 clusters in unexpected ways, clusters running Databricks Runtime 7.0 and above do not install libraries configured to be installed on all clusters. 群集“库”选项卡显示状态“Skipped”和一条说明库处理方式更改的“弃用”消息。The cluster Libraries tab shows a status Skipped and a deprecation message that explains the changes in library handling. 但是，如果你的群集是在早期版本的 Databricks Runtime（在 Azure Databricks 平台 3.20 版发布到工作区之前的版本）上创建的，并且你现在将该群集编辑为使用 Databricks Runtime 7.0，则已配置为在所有群集上安装的任何库都将安装在该群集上。However, if you have a cluster that was created on an earlier version of Databricks Runtime before Azure Databricks platform version 3.20 was released to your workspace, and you now edit that cluster to use Databricks Runtime 7.0, any libraries that were configured to be installed on all clusters will be installed on that cluster. 在这种情况下，已安装的库中的任何不兼容的 JAR 都可能导致群集被禁用。In this case, any incompatible JARs in the installed libraries can cause the cluster to be disabled. 解决方法是克隆群集或创建新群集。The workaround is either to clone the cluster or to create a new cluster.

Apache Spark Apache Spark

Databricks Runtime 7.0 包括 Apache Spark 3.0。Databricks Runtime 7.0 includes Apache Spark 3.0.

本节内容：In this section:

• Core、Spark SQL、结构化流式处理Core, Spark SQL, Structured Streaming
• MLlibMLlib
• SparkRSparkR
• GraphXGraphX
• 弃用功能Deprecations
• 已知问题Known issues

Core、Spark SQL、结构化流式处理Core, Spark SQL, Structured Streaming

亮点Highlights

• （Hydrogen 项目）加速器感知型计划程序 (SPARK-24615)(Project Hydrogen) Accelerator-aware Scheduler (SPARK-24615)
• 自适应查询执行 (SPARK-31412)Adaptive Query Execution (SPARK-31412)
• 动态分区修剪 (SPARK-11150)Dynamic Partition Pruning (SPARK-11150)
• 重新设计的带类型提示的 pandas UDF API (SPARK-28264)Redesigned pandas UDF API with type hints (SPARK-28264)
• 结构化流式处理 UI (SPARK-29543)Structured Streaming UI (SPARK-29543)
• 目录插件 API (SPARK-31121)Catalog plugin API (SPARK-31121)
• 更好的 ANSI SQL 兼容性Better ANSI SQL compatibility

性能增强功能Performance enhancements

• 自适应查询执行 (SPARK-31412)Adaptive Query Execution (SPARK-31412)
  • 基本框架 (SPARK-23128)Basic framework (SPARK-23128)
  • 后期无序分区号调整 (SPARK-28177)Post shuffle partition number adjustment (SPARK-28177)
  • 动态子查询重用 (SPARK-28753)Dynamic subquery reuse (SPARK-28753)
  • 本地无序读取器 (SPARK-28560)Local shuffle reader (SPARK-28560)
  • 倾斜联接优化 (SPARK-29544)Skew join optimization (SPARK-29544)
  • 优化读取连续无序块的操作 (SPARK-9853)Optimize reading contiguous shuffle blocks (SPARK-9853)
• 动态分区修剪 (SPARK-11150)Dynamic Partition Pruning (SPARK-11150)
• 其他优化器规则Other optimizer rules
  • ReuseSubquery 规则 (SPARK-27279)Rule ReuseSubquery (SPARK-27279)
  • PushDownLeftSemiAntiJoin 规则 (SPARK-19712)Rule PushDownLeftSemiAntiJoin (SPARK-19712)
  • PushLeftSemiLeftAntiThroughJoin 规则 (SPARK-19712)Rule PushLeftSemiLeftAntiThroughJoin (SPARK-19712)
  • ReplaceNullWithFalse 规则 (SPARK-25860)Rule ReplaceNullWithFalse (SPARK-25860)
  • 在联接/聚合的子查询中，Eliminate 规则在排序时不会受到限制 (SPARK-29343)Rule Eliminate sorts without limit in the subquery of Join/Aggregation (SPARK-29343)
  • PruneHiveTablePartitions 规则 (SPARK-15616)Rule PruneHiveTablePartitions (SPARK-15616)
  • 从“生成”中删除不必要的嵌套字段 (SPARK-27707)Pruning unnecessary nested fields from Generate (SPARK-27707)
  • RewriteNonCorrelatedExists 规则 (SPARK-29800)Rule RewriteNonCorrelatedExists (SPARK-29800)
• 最小化表缓存同步成本 (SPARK-26917)、(SPARK-26617)、(SPARK-26548)Minimize table cache synchronization costs (SPARK-26917), (SPARK-26617), (SPARK-26548)
• 将聚合代码拆分为小函数 (SPARK-21870)Split aggregation code into small functions (SPARK-21870)
• 在 INSERT 和 ALTER TABLE ADD PARTITION 命令中添加批处理 (SPARK-29938)Add batching in INSERT and ALTER TABLE ADD PARTITION command (SPARK-29938)

扩展性增强功能Extensibility enhancements

• 目录插件 API (SPARK-31121)Catalog plugin API (SPARK-31121)
• 数据源 V2 API 重构 (SPARK-25390)Data source V2 API refactoring (SPARK-25390)
• Hive 3.0 和 3.1 元存储支持 (SPARK-27970)、(SPARK-24360)Hive 3.0 and 3.1 metastore support (SPARK-27970),(SPARK-24360)
• 将 Spark 插件接口扩展到驱动程序 (SPARK-29396)Extend Spark plugin interface to driver (SPARK-29396)
• 使用执行程序插件通过用户定义的指标扩展 Spark 指标系统 (SPARK-28091)Extend Spark metrics system with user-defined metrics using executor plugins (SPARK-28091)
• 用于提供扩展的列式处理支持的开发人员 API (SPARK-27396)Developer APIs for extended Columnar Processing Support (SPARK-27396)
• 使用 DSV2 的内置源迁移：parquet、ORC、CSV、JSON、Kafka、Text、Avro (SPARK-27589)Built-in source migration using DSV2: parquet, ORC, CSV, JSON, Kafka, Text, Avro (SPARK-27589)
• 允许 SparkExtensions 中的 FunctionInjection (SPARK-25560)Allow FunctionInjection in SparkExtensions (SPARK-25560)
• 允许聚合器注册为 UDAF (SPARK-27296)Allows Aggregator to be registered as a UDAF (SPARK-27296)

连接器增强功能Connector enhancements

• 通过非确定性表达式进行的列修剪 (SPARK-29768)Column pruning through nondeterministic expressions (SPARK-29768)
• 支持数据源表中的 spark.sql.statistics.fallBackToHdfs (SPARK-25474)Support spark.sql.statistics.fallBackToHdfs in data source tables (SPARK-25474)
• 允许在文件源上使用子查询筛选器进行分区修剪 (SPARK-26893)Allow partition pruning with subquery filters on file source (SPARK-26893)
• 避免在数据源筛选器中下推子查询 (SPARK-25482)Avoid pushdown of subqueries in data source filters (SPARK-25482)
• 从文件源加载递归数据 (SPARK-27990)Recursive data loading from file sources (SPARK-27990)
• Parquet/ORCParquet/ORC
  • 下推析取谓词 (SPARK-27699)Pushdown of disjunctive predicates (SPARK-27699)
  • 通用化嵌套列修剪 (SPARK-25603) 并在默认情况下启用 (SPARK-29805)Generalize Nested Column Pruning (SPARK-25603) and turned on by default (SPARK-29805)
  • 仅 ParquetParquet only
    • 嵌套字段的 Parquet 谓词下推 (SPARK-17636)Parquet predicate pushdown for nested fields (SPARK-17636)
  • 仅 ORCORC only
    • 支持 ORC 的合并架构 (SPARK-11412)Support merge schema for ORC (SPARK-11412)
    • ORC 的嵌套架构修剪 (SPARK-27034)Nested schema pruning for ORC (SPARK-27034)
    • ORC 的谓词转换复杂度降低（SPARK-27105、SPARK-28108）Predicate conversion complexity reduction for ORC (SPARK-27105, SPARK-28108)
    • 将 Apache ORC 升级到 1.5.9 (SPARK-30695)Upgrade Apache ORC to 1.5.9 (SPARK-30695)
• CSVCSV
  • 支持在 CSV 数据源中向下推送筛选器 (SPARK-30323)Support filters pushdown in CSV datasource (SPARK-30323)
• Hive SerDeHive SerDe
  • 在读取使用原生数据源的 Hive serde 表时不进行架构推理 (SPARK-27119)No schema inference when reading Hive serde table with native data source (SPARK-27119)
  • Hive CTAS 命令应使用数据源（如果数据源是可转换的）(SPARK-25271)Hive CTAS commands should use data source if it is convertible (SPARK-25271)
  • 使用原生数据源优化插入已分区 Hive 表的操作 (SPARK-28573)Use native data source to optimize inserting partitioned Hive table (SPARK-28573)
• Apache KafkaApache Kafka
  • 添加对 Kafka 标头的支持 (SPARK-23539)Add support for Kafka headers (SPARK-23539)
  • 添加 Kafka 委派令牌支持 (SPARK-25501)Add Kafka delegation token support (SPARK-25501)
  • 引入 Kafka 源的新选项：按时间戳进行偏移（开始/结束）(SPARK-26848)Introduce new option to Kafka source: offset by timestamp (starting/ending) (SPARK-26848)
  • 在 Kafka 批处理源和流式处理源 v1 中支持 minPartitions 选项 (SPARK-30656)Support the minPartitions option in Kafka batch source and streaming source v1 (SPARK-30656)
  • 将 Kafka 升级到 2.4.1 (SPARK-31126)Upgrade Kafka to 2.4.1 (SPARK-31126)
• 新的内置数据源New built-in data sources
  • 新的内置二进制文件数据源 (SPARK-25348)New built-in binary file data sources (SPARK-25348)
  • 新的无操作批处理数据源 (SPARK-26550) 和无操作流式处理接收器 (SPARK-26649)New no-op batch data sources (SPARK-26550) and no-op streaming sink (SPARK-26649)

功能增强Feature enhancements

• [Hydrogen] 加速器感知型计划程序 (SPARK-24615)[Hydrogen] Accelerator-aware Scheduler (SPARK-24615)
• 引入一组完整的联接提示 (SPARK-27225)Introduce a complete set of Join Hints (SPARK-27225)
• 为 SQL 查询添加 PARTITION BY 提示 (SPARK-28746)Add PARTITION BY hint for SQL queries (SPARK-28746)
• Thrift 服务器中的元数据处理 (SPARK-28426)Metadata Handling in Thrift Server (SPARK-28426)
• 将更高阶的函数添加到 scala API (SPARK-27297)Add higher order functions to scala API (SPARK-27297)
• 在屏障任务上下文中支持简单的全部收集 (SPARK-30667)Support simple all gather in barrier task context (SPARK-30667)
• Hive UDF 支持 UDT 类型 (SPARK-28158)Hive UDFs supports the UDT type (SPARK-28158)
• 支持 Catalyst 中的 DELETE/UPDATE/MERGE 运算符（SPARK-28351、SPARK-28892、SPARK-28893）Support DELETE/UPDATE/MERGE Operators in Catalyst (SPARK-28351, SPARK-28892, SPARK-28893)
• 实现 DataFrame.tail (SPARK-30185)Implement DataFrame.tail (SPARK-30185)
• 新的内置函数New built-in functions
  • sinh、cosh、tanh、asinh、acosh、atanh (SPARK-28133)sinh, cosh, tanh, asinh, acosh, atanh (SPARK-28133)
  • any、every、some (SPARK-19851)any, every, some (SPARK-19851)
  • bit_and、bit_or (SPARK-27879)bit_and, bit_or (SPARK-27879)
  • bit_count (SPARK-29491)bit_count (SPARK-29491)
  • bit_xor (SPARK-29545)bit_xor (SPARK-29545)
  • bool_and、bool_or (SPARK-30184)bool_and, bool_or (SPARK-30184)
  • count_if (SPARK-27425)count_if (SPARK-27425)
  • date_part (SPARK-28690)date_part (SPARK-28690)
  • extract (SPARK-23903)extract (SPARK-23903)
  • forall (SPARK-27905)forall (SPARK-27905)
  • from_csv (SPARK-25393)from_csv (SPARK-25393)
  • make_date (SPARK-28432)make_date (SPARK-28432)
  • make_interval (SPARK-29393)make_interval (SPARK-29393)
  • make_timestamp (SPARK-28459)make_timestamp (SPARK-28459)
  • map_entries (SPARK-23935)map_entries (SPARK-23935)
  • map_filter (SPARK-23937)map_filter (SPARK-23937)
  • map_zip_with (SPARK-23938)map_zip_with (SPARK-23938)
  • max_by、min_by (SPARK-27653)max_by, min_by (SPARK-27653)
  • schema_of_csv (SPARK-25672)schema_of_csv (SPARK-25672)
  • to_csv (SPARK-25638)to_csv (SPARK-25638)
  • transform_keys (SPARK-23939)transform_keys (SPARK-23939)
  • transform_values (SPARK-23940)transform_values (SPARK-23940)
  • typeof (SPARK-29961)typeof (SPARK-29961)
  • version (SPARK-29554)version (SPARK-29554)
  • xxhash64 (SPARK-27099)xxhash64 (SPARK-27099)
• 对现有内置函数的改进Improvements on existing built-in functions
  • 内置日期-时间函数/操作改进 (SPARK-31415)Built-in date-time functions/operations improvement (SPARK-31415)
  • 对 from_json 支持 FAILFAST 模式 (SPARK-25243)Support FAILFAST mode for from_json (SPARK-25243)
  • array_sort 添加新的比较运算符参数 (SPARK-29020)array_sort adds a new comparator parameter (SPARK-29020)
  • 筛选器现在可以将索引用作输入和元素 (SPARK-28962)Filter can now take the index as input as well as the element (SPARK-28962)

SQL 兼容性增强功能SQL compatibility enhancements

• 切换到前公历 (SPARK-26651)Switch to Proleptic Gregorian calendar (SPARK-26651)
• 生成 Spark 自己的日期/时间模式定义 (SPARK-31408)Build Spark’s own datetime pattern definition (SPARK-31408)
• 引入用于表插入的 ANSI 存储分配策略 (SPARK-28495)Introduce ANSI store assignment policy for table insertion (SPARK-28495)
• 默认情况下，在表插入中遵循 ANSI 存储分配规则 (SPARK-28885)Follow ANSI store assignment rule in table insertion by default (SPARK-28885)
• 添加 SQLConf spark.sql.ansi.enabled (SPARK-28989)Add a SQLConf spark.sql.ansi.enabled (SPARK-28989)
• 支持用于聚合表达式的 ANSI SQL 筛选子句 (SPARK-27986)Support ANSI SQL filter clause for aggregate expression (SPARK-27986)
• 支持 ANSI SQL OVERLAY 函数 (SPARK-28077)Support ANSI SQL OVERLAY function (SPARK-28077)
• 支持 ANSI 嵌套的带括号注释 (SPARK-28880)Support ANSI nested bracketed comments (SPARK-28880)
• 整数溢出时引发异常 (SPARK-26218)Throw exception on overflow for integers (SPARK-26218)
• 区间算术运算的溢出检查 (SPARK-30341)Overflow check for interval arithmetic operations (SPARK-30341)
• 将无效字符串强制转换为数值类型时引发异常 (SPARK-30292)Throw Exception when invalid string is cast to numeric type (SPARK-30292)
• 使区间乘法和除法的溢出行为与其他运算一致 (SPARK-30919)Make interval multiply and divide’s overflow behavior consistent with other operations (SPARK-30919)
• 为 char 和 decimal 添加 ANSI 类型的别名 (SPARK-29941)Add ANSI type aliases for char and decimal (SPARK-29941)
• SQL 分析程序定义符合 ANSI 标准的保留关键字 (SPARK-26215)SQL Parser defines ANSI compliant reserved keywords (SPARK-26215)
• 当 ANSI 模式处于开启状态时，禁止将保留关键字用作标识符 (SPARK-26976)Forbid reserved keywords as identifiers when ANSI mode is on (SPARK-26976)
• 支持 ANSI SQL LIKE ... ESCAPE 语法 (SPARK-28083)Support ANSI SQL LIKE ... ESCAPE syntax (SPARK-28083)
• 支持 ANSI SQL 布尔值-谓词语法 (SPARK-27924)Support ANSI SQL Boolean-Predicate syntax (SPARK-27924)
• 更好地支持相关子查询处理 (SPARK-18455)Better support for correlated subquery processing (SPARK-18455)

监视和可调试性增强功能Monitoring and debugability enhancements

• 新结构化流式处理 UI (SPARK-29543)New Structured Streaming UI (SPARK-29543)
• SHS：允许正在运行的流式处理应用的事件日志滚动更新 (SPARK-28594)SHS: Allow event logs for running streaming apps to be rolled over (SPARK-28594)
• 添加一个 API，以便用户定义和观察批处理和流式处理查询的任意指标 (SPARK-29345)Add an API that allows a user to define and observe arbitrary metrics on batch and streaming queries (SPARK-29345)
• 用于跟踪每个查询的计划时间的检测 (SPARK-26129)Instrumentation for tracking per-query planning time (SPARK-26129)
• 将基本的无序指标置于 SQL 交换运算符中 (SPARK-26139)Put the basic shuffle metrics in the SQL exchange operator (SPARK-26139)
• SQL 语句显示在 SQL 选项卡中，而不是 callsite 中 (SPARK-27045)SQL statement is shown in SQL Tab instead of callsite (SPARK-27045)
• 向 SparkUI 添加工具提示 (SPARK-29449)Add tooltip to SparkUI (SPARK-29449)
• 提高历史记录服务器的并发性能 (SPARK-29043)Improve the concurrent performance of History Server (SPARK-29043)
• EXPLAIN FORMATTED 命令 (SPARK-27395)EXPLAIN FORMATTED command (SPARK-27395)
• 支持将截断的计划和生成的代码转储到文件 (SPARK-26023)Support Dumping truncated plans and generated code to a file (SPARK-26023)
• 增强用于描述查询输出的框架 (SPARK-26982)Enhance describe framework to describe the output of a query (SPARK-26982)
• 添加 SHOW VIEWS 命令 (SPARK-31113)Add SHOW VIEWS command (SPARK-31113)
• 改进 SQL 分析程序的错误消息 (SPARK-27901)Improve the error messages of SQL parser (SPARK-27901)
• 支持以原生方式进行 Prometheus 监视 (SPARK-29429)Support Prometheus monitoring natively (SPARK-29429)

PySpark 增强功能PySpark enhancements

• 重新设计的带类型提示的 pandas UDF (SPARK-28264)Redesigned pandas UDFs with type hints (SPARK-28264)
• Pandas UDF 管道 (SPARK-26412)Pandas UDF pipeline (SPARK-26412)
• 支持将 StructType 作为标量 Pandas UDF 的参数和返回类型 (SPARK-27240)Support StructType as arguments and return types for Scalar Pandas UDF (SPARK-27240 )
• 通过 Pandas UDF 支持数据帧协同组 (SPARK-27463)Support Dataframe Cogroup via Pandas UDFs (SPARK-27463)
• 添加 mapInPandas 以允许数据帧的迭代器 (SPARK-28198)Add mapInPandas to allow an iterator of DataFrames (SPARK-28198)
• 某些 SQL 函数也应采用列名 (SPARK-26979)Certain SQL functions should take column names as well (SPARK-26979)
• 使 PySpark SQL 异常更具 Python 特性 (SPARK-31849)Make PySpark SQL exceptions more Pythonic (SPARK-31849)

文档和测试覆盖范围增强功能Documentation and test coverage enhancements

• 构建 SQL 参考 (SPARK-28588)Build a SQL Reference (SPARK-28588)
• 构建 WebUI 的用户指南 (SPARK-28372)Build a user guide for WebUI (SPARK-28372)
• 构建 SQL 配置文档的页面 (SPARK-30510)Build a page for SQL configuration documentation (SPARK-30510)
• 添加 Spark 配置的版本信息 (SPARK-30839)Add version information for Spark configuration (SPARK-30839)
• 从 PostgreSQL 移植回归测试 (SPARK-27763)Port regression tests from PostgreSQL (SPARK-27763)
• Thrift 服务器测试覆盖范围 (SPARK-28608)Thrift-server test coverage (SPARK-28608)
• UDF（python UDF、pandas UDF、scala UDF）的测试覆盖范围 (SPARK-27921)Test coverage of UDFs (python UDF, pandas UDF, scala UDF) (SPARK-27921)

其他值得注意的更改Other notable changes

• 内置 Hive 执行从 1.2.1 升级到 2.3.6（SPARK-23710、SPARK-28723、SPARK-31381）Built-in Hive execution upgrade from 1.2.1 to 2.3.6 (SPARK-23710, SPARK-28723, SPARK-31381)
• 默认情况下，使用 Apache Hive 2.3 依赖项 (SPARK-30034)Use Apache Hive 2.3 dependency by default (SPARK-30034)
• 正式发布 Scala 2.12，删除了 2.11 (SPARK-26132)GA Scala 2.12 and remove 2.11 (SPARK-26132)
• 改进动态分配中让执行程序超时的逻辑 (SPARK-20286)Improve logic for timing out executors in dynamic allocation (SPARK-20286)
• 磁盘持久保存的 RDD 块由无序处理服务提供，在进行动态分配时会被忽略 (SPARK-27677)Disk-persisted RDD blocks served by shuffle service and ignored for Dynamic Allocation (SPARK-27677)
• 获取新的执行程序，避免因加入阻止列表而挂起 (SPARK-22148)Acquire new executors to avoid hang because of blacklisting (SPARK-22148)
• 允许共享 Netty 的内存池分配器 (SPARK-24920)Allow sharing of Netty’s memory pool allocators (SPARK-24920)
• 修复 TaskMemoryManager 和 UnsafeExternalSorter$SpillableIterator 之间的死锁 (SPARK-27338)Fix deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator (SPARK-27338)
• 引入用于 StructuredStreaming 的 AdmissionControl API (SPARK-30669)Introduce AdmissionControl APIs for StructuredStreaming (SPARK-30669)
• Spark 历史记录主页性能改善 (SPARK-25973)Spark History Main page performance improvement (SPARK-25973)
• 提高 SQL 侦听器中的指标聚合速度，并降低该指标聚合所占用的内存 (SPARK-29562)Speed up and slim down metric aggregation in SQL listener (SPARK-29562)
• 从同一主机提取无序块时避免使用网络 (SPARK-27651)Avoid the network when shuffle blocks are fetched from the same host (SPARK-27651)
• 改进 DistributedFileSystem 的文件列表操作 (SPARK-27801)Improve file listing for DistributedFileSystem (SPARK-27801)

Spark Core、Spark SQL 和结构化流式处理的行为变更Behavior changes for Spark core, Spark SQL, and Structured Streaming

以下迁移指南列出了 Apache Spark 2.4 和 3.0 之间的行为变更。The following migration guides list behavior changes between Apache Spark 2.4 and 3.0. 这些变更可能会要求更新一直在较低 Databricks Runtime 版本上运行的作业：These changes may require updates to jobs that you have been running on lower Databricks Runtime versions:

• 迁移指南：Spark CoreMigration Guide: Spark Core
• 迁移指南：SQL、数据集和数据帧Migration Guide: SQL, Datasets and DataFrame
• 迁移指南：结构化流式处理Migration Guide: Structured Streaming
• 迁移指南：PySpark（Spark 上的 Python）Migration Guide: PySpark (Python on Spark)

这些迁移指南不涵盖以下行为变更：The following behavior changes are not covered in these migration guides:

• 在 Spark 3.0 中，已删除弃用的类 org.apache.spark.sql.streaming.ProcessingTime。In Spark 3.0, the deprecated class org.apache.spark.sql.streaming.ProcessingTime has been removed. 请改用 org.apache.spark.sql.streaming.Trigger.ProcessingTime。Use org.apache.spark.sql.streaming.Trigger.ProcessingTime instead. 同样，删除 org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger 是为了支持 Trigger.Continuous，隐藏 org.apache.spark.sql.execution.streaming.OneTimeTrigger 是为了支持 Trigger.Once。Likewise, org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger has been removed in favor of Trigger.Continuous, and org.apache.spark.sql.execution.streaming.OneTimeTrigger has been hidden in favor of Trigger.Once. (SPARK-28199)(SPARK-28199)
• 在 Databricks Runtime 7.0 中读取 Hive SerDe 表时，默认情况下 Spark 不允许读取并非表分区的子目录下的文件。In Databricks Runtime 7.0, when reading a Hive SerDe table, by default Spark disallows reading files under a subdirectory that is not a table partition. 若要允许它，请将配置 spark.databricks.io.hive.scanNonpartitionedDirectory.enabled 设置为 true。To enable it, set the configuration spark.databricks.io.hive.scanNonpartitionedDirectory.enabled as true. 这不影响 Spark 原生表读取器和文件读取器。This does not affect Spark native table readers and file readers.

编程指南：Programming guides:

• Spark RDD 编程指南Spark RDD Programming Guide
• Spark SQL、数据帧和数据集指南Spark SQL, DataFrames and Datasets Guide
• 结构化流式处理编程指南。Structured Streaming Programming Guide.

MLlibMLlib

亮点Highlights

• 已将多列支持添加到 Binarizer (SPARK-23578)、StringIndexer (SPARK-11215)、StopWordsRemover (SPARK-29808) 和 PySpark QuantileDiscretizer (SPARK-22796)Multiple columns support was added to Binarizer (SPARK-23578), StringIndexer (SPARK-11215), StopWordsRemover (SPARK-29808) and PySpark QuantileDiscretizer (SPARK-22796)
• 支持基于树的功能转换 (SPARK-13677)Support tree-based feature transformation(SPARK-13677)
• 添加了两个新计算器：MultilabelClassificationEvaluator (SPARK-16692) 和 RankingEvaluator (SPARK-28045)Two new evaluators MultilabelClassificationEvaluator (SPARK-16692) and RankingEvaluator (SPARK-28045) were added
• 已将示例权重支持添加到 DecisionTreeClassifier/Regressor (SPARK-19591)、RandomForestClassifier/Regressor (SPARK-9478)、GBTClassifier/Regressor (SPARK-9612)、RegressionEvaluator (SPARK-24102)、BinaryClassificationEvaluator (SPARK-24103)、BisectingKMeans (SPARK-30351)、KMeans (SPARK-29967) 和 GaussianMixture (SPARK-30102)Sample weights support was added in DecisionTreeClassifier/Regressor (SPARK-19591), RandomForestClassifier/Regressor (SPARK-9478), GBTClassifier/Regressor (SPARK-9612), RegressionEvaluator (SPARK-24102), BinaryClassificationEvaluator (SPARK-24103), BisectingKMeans (SPARK-30351), KMeans (SPARK-29967) and GaussianMixture (SPARK-30102)
• 添加了适用于 PowerIterationClustering 的 R API (SPARK-19827)R API for PowerIterationClustering was added (SPARK-19827)
• 添加了 Spark ML 侦听器，用于跟踪 ML 管道状态 (SPARK-23674)Added Spark ML listener for tracking ML pipeline status (SPARK-23674)
• 已将带验证集的拟合添加到 Python 中的梯度提升树 (SPARK-24333)Fit with validation set was added to Gradient Boosted Trees in Python (SPARK-24333)
• 已添加 RobustScaler 转换器 (SPARK-28399)RobustScaler transformer was added (SPARK-28399)
• 已添加因式分解计算机分类器和回归量 (SPARK-29224)Factorization Machines classifier and regressor were added (SPARK-29224)
• 已添加高斯朴素贝叶斯 (SPARK-16872) 和补足朴素贝叶斯 (SPARK-29942)Gaussian Naive Bayes (SPARK-16872) and Complement Naive Bayes (SPARK-29942) were added
• Scala 和 Python 之间的 ML 函数奇偶校验 (SPARK-28958)ML function parity between Scala and Python (SPARK-28958)
• predictRaw 在所有分类模型中都是公共的。predictRaw is made public in all the Classification models. predictProbability 在除了 LinearSVCModel 之外的所有分类模型中都是公共的 (SPARK-30358)predictProbability is made public in all of the Classification models except LinearSVCModel (SPARK-30358)

MLlib 的行为变更Behavior changes for MLlib

以下迁移指南列出了 Apache Spark 2.4 和 3.0 之间的行为变更。The following migration guide lists behavior changes between Apache Spark 2.4 and 3.0. 这些变更可能会要求更新一直在较低 Databricks Runtime 版本上运行的作业：These changes may require updates to jobs that you have been running on lower Databricks Runtime versions:

• 迁移指南：MLlib（机器学习）Migration Guide: MLlib (Machine Learning)

此迁移指南不涵盖以下行为变更：The following behavior changes are not covered in the migration guide:

• 在 Spark 3.0 中，Pyspark 中的多类逻辑回归现在会（正确地）返回 LogisticRegressionSummary 而不是子类 BinaryLogisticRegressionSummary。In Spark 3.0, a multiclass logistic regression in Pyspark will now (correctly) return LogisticRegressionSummary, not the subclass BinaryLogisticRegressionSummary. 在这种情况下，BinaryLogisticRegressionSummary 公开的其他方法仍不起作用。The additional methods exposed by BinaryLogisticRegressionSummary would not work in this case anyway. (SPARK-31681)(SPARK-31681)
• 在 Spark 3.0 中，pyspark.ml.param.shared.Has* mixin 不再提供任何 set*(self, value) 资源库方法，请改用相应的 self.set(self.*, value)。In Spark 3.0, pyspark.ml.param.shared.Has* mixins do not provide any set*(self, value) setter methods anymore, use the respective self.set(self.*, value) instead. 有关详细信息，请参阅 SPARK-29093。See SPARK-29093 for details. (SPARK-29093)(SPARK-29093)

编程指南Programming guide

• 机器学习库 (MLlib) 指南Machine Learning Library (MLlib) Guide

SparkRSparkR

• SparkR 的互操作性方面的 Arrow 优化 (SPARK-26759)Arrow optimization in SparkR’s interoperability (SPARK-26759)
• 通过向量化的 R gapply()、dapply()、createDataFrame、collect() 进行的性能增强Performance enhancement via vectorized R gapply(), dapply(), createDataFrame, collect()
• “预先执行”，适用于 R shell、IDE (SPARK-24572)“Eager execution” for R shell, IDE (SPARK-24572)
• 适用于 Power 迭代聚类分析的 R API (SPARK-19827)R API for Power Iteration Clustering (SPARK-19827)

SparkR 的行为变更Behavior changes for SparkR

以下迁移指南列出了 Apache Spark 2.4 和 3.0 之间的行为变更。The following migration guide lists behavior changes between Apache Spark 2.4 and 3.0. 这些变更可能会要求更新一直在较低 Databricks Runtime 版本上运行的作业：These changes may require updates to jobs that you have been running on lower Databricks Runtime versions:

• 迁移指南：SparkR（Spark 上的 R）Migration Guide: SparkR (R on Spark)

编程指南Programming guide

• SparkR（Spark 上的 R）。SparkR (R on Spark).

GraphXGraphX

编程指南：GraphX 编程指南。Programming guide: GraphX Programming Guide.

弃用功能Deprecations

• 弃用 Python 2 支持 (SPARK-27884)Deprecate Python 2 support (SPARK-27884)
• 弃用 R < 3.4 的支持 (SPARK-26014)Deprecate R < 3.4 support (SPARK-26014)

已知问题Known issues

• 如果 year 字段缺失，则使用模式字母“D”分析年份中的天会返回错误的结果。Parsing day of year using pattern letter ‘D’ returns the wrong result if the year field is missing. 这种情况可能发生在 to_timestamp 等 SQL 函数中，它使用模式字符串将日期/时间字符串分析为日期/时间值。This can happen in SQL functions like to_timestamp which parses datetime string to datetime values using a pattern string. (SPARK-31939)(SPARK-31939)
• 如果键的值为 -0.0 和 0.0，则在子查询中进行联接/窗口/聚合操作可能会导致错误的结果。Join/Window/Aggregate inside subqueries may lead to wrong results if the keys have values -0.0 and 0.0. (SPARK-31958)(SPARK-31958)
• 窗口查询可能会由于意外的歧义自联接错误而失败。A window query may fail with ambiguous self-join error unexpectedly. (SPARK-31956)(SPARK-31956)
• 使用 dropDuplicates 运算符的流式处理查询可能无法使用通过 Spark 2.x 编写的检查点重启。Streaming queries with dropDuplicates operator may not be able to restart with the checkpoint written by Spark 2.x. (SPARK-31990)(SPARK-31990)

维护更新Maintenance updates

请参阅 Databricks Runtime 7.0 维护更新。See Databricks Runtime 7.0 maintenance updates.

系统环境System environment

• 操作系统：Ubuntu 18.04.4 LTSOperating System: Ubuntu 18.04.4 LTS
• Java：1.8.0_252Java: 1.8.0_252
• Scala：2.12.10Scala: 2.12.10
• Python：3.7.5Python: 3.7.5
• R：R 版本 3.6.3 (2020-02-29)R: R version 3.6.3 (2020-02-29)
• Delta Lake 0.7.0Delta Lake 0.7.0

已安装的 Python 库Installed Python libraries

已安装的 R 库 Installed R libraries

R 库安装自（2020-04-22 的 Microsoft CRAN 快照）。R libraries are installed from (Microsoft CRAN snapshot on 2020-04-22).

已安装的 Java 库和 Scala 库（Scala 2.12 群集版本）Installed Java and Scala libraries (Scala 2.12 cluster version)