Databricks Runtime 7.1 (EoS)

Databricks released this version in July 2020.

The following release notes provide information about Databricks Runtime 7.1, powered by Apache Spark 3.0.

New features

Databricks Runtime 7.1 includes the following new features:

Delta Lake

• You can now set user-defined metadata in each commit made by Delta operations. For details, see Enrich Delta Lake tables with custom metadata.

• You can now query the SQL configuration spark.databricks.delta.lastCommitVersionInSession to get the version of the last commit written by the current SparkSession.

  If no commits have been made by the SparkSession, querying the key returns an empty value. This feature is not supported in streaming or multithreaded cases.

• CONVERT TO DELTA can now convert Parquet tables that were created by Structured Streaming using the transaction log found in the _spark_metadata folder. This prevents uncommitted files through the streaming query from being considered in your Delta table. You can then continue your stream using the same checkpoint location.

  Note

  There is a small chance that data from the last micro-batch may be committed again to your table after the conversion process.

• This release contains performance improvements for MERGE INTO operations that have only matched clauses. That is, they have only UPDATE and DELETE actions and no INSERT action.

pip commands to support notebook-scoped Python libraries

Starting with Databricks Runtime 7.1, you can use %pip commands to manage Python libraries installed in a notebook session. For more information, see Notebook-scoped Python libraries.

Improvements

• Adds a display method to PySpark, pandas, and Koalas DataFrames, so you can now run df.display() for easier exploratory data analysis. See Create a new visualization.
• Koalas is now installed.

Apache Spark

Databricks Runtime 7.1 includes Apache Spark 3.0.0. This release includes all Spark fixes and improvements included in Databricks Runtime 7.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-32057] ExecuteStatement: cancel and close should not transiently ERROR
• [SPARK-32167] [SQL] Fix GetArrayStructFields to respect inner field's nullability together
• [SPARK-31935] Hadoop file system config should be effective in data source options
• [SPARK-32038] [SQL] NormalizeFloatingNumbers should also work on distinct aggregate
• [SPARK-31705] [SQL] Push more possible predicates through Join via CNF conversion
• [SPARK-31945] [SQL][PYSPARK] Enable cache for the same Python function
• [SPARK-8981] [CORE] Add MDC support in Executor
• [SPARK-31986] [SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps
• [SPARK-31959] [SQL] Fix Gregorian-Julian micros rebasing while switching standard time zone offset
• [SPARK-26905] [SQL] Follow the SQL:2016 reserved keywords
• [SPARK-31990] [SS] Use toSet.toSeq in Dataset.dropDuplicates
• [SPARK-31992] [SQL] Benchmark the EXCEPTION rebase mode
• [SPARK-31983] [WEBUI] Fix sorting for duration column in structured streaming tab
• [SPARK-31593] [SS] Remove unnecessary streaming query progress update
• [SPARK-31968] [SQL] Duplicate partition columns check when writing data
• [SPARK-31632] [CORE][WEBUI] Enrich the exception message when application summary is unavailable
• [SPARK-31916] [SQL] StringConcat can lead to StringIndexOutOfBoundsException
• [SPARK-31935] [SQL] Hadoop file system config should be effective in data source options
• [SPARK-31954] [SQL] Delete duplicate testcase in HiveQuerySuite
• [SPARK-28199] [SS] Remove package private in class/object in sql.execution package
• [SPARK-31956] [SQL] Do not fail if there is no ambiguous self join
• [SPARK-31941] [CORE] Replace SparkException to NoSuchElementException for applicationInfo in AppStatusStore
• [SPARK-26905] [SQL] Add TYPE in the ANSI non-reserved list
• [SPARK-31921] [CORE] Fix the wrong warning: "App app-xxx requires more resource than any of Workers could have"
• [SPARK-31220] [SQL] repartition obeys initialPartitionNum when adaptiveExecutionEnabled
• [SPARK-29295] [SQL] Dynamic partition map parsed from partition path should be case insensitive
• [SPARK-31849] [PYTHON][SQL] More correct error message in Python UDF exception message
• [SPARK-31923] [CORE] Ignore internal accumulators that use unrecognized types rather than crashing
• [SPARK-31849] [PYTHON][SQL] Deduplicate and reuse Utils.exceptionString in Python exception handling

Maintenance updates

See Databricks Runtime 7.1 maintenance updates.

System environment

• Operating System: Ubuntu 18.04.4 LTS
• Java: 1.8.0_252
• Scala: 2.12.10
• Python: 3.7.5
• R: R version 3.6.3 (2020-02-29)
• Delta Lake 0.7.0

Installed Python libraries

Installed R libraries

R libraries are installed from Microsoft CRAN snapshot on 2020-04-22.

Installed Java and Scala libraries (Scala 2.12 cluster version)