Databricks Runtime 6.2 (EoS)

Databricks released this version in December 2019.

The following release notes provide information about Databricks Runtime 6.2, powered by Apache Spark.

New features

Delta Lake insert-only merge is optimized

The performance of Delta Lake merge operations that have no update clauses has been optimized. Writes made by such insert-only merge operations now only append data and do not modify existing data, thus allowing downstream streaming queries to continuously read the appended data. This better supports merge-based deduplication: batch or streaming ETL jobs can write deduplicated data into tables using insert-only merge, and downstream streaming queries can continuously read that deduplicated data.

Improvements

• display() of BinaryType columns are automatically truncated to the base64 encoding of first 128 bytes, instead of all bytes. If the field is truncated, you will see "(truncated)" at the end.
• Upgraded several installed R libraries. See Installed R libraries.

Deprecations

display() of spark.mllib models is deprecated and it will be removed in the next major release. Azure Databricks recommends using spark.ml models.

Apache Spark

Databricks Runtime 6.2 includes Apache Spark 2.4.4. This release includes all Spark fixes and improvements included in Databricks Runtime 6.1 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-29875][PYTHON][SQL] Avoid to use deprecated pyarrow.open_stream API in Spark 2.4.x
• [SPARK-29001][CORE] Print events that take too long time to process
• [SPARK-29850][SQL] sort-merge-join an empty table should not memory leak
• [SPARK-29790][DOC] Note required port for Kube API
• [SPARK-29743][SQL] sample should set needCopyResult to true if its child is
• [SPARK-29651][SQL] Fix parsing of interval seconds fraction
• [SPARK-29604][SQL] Force initialize SessionState before initializing HiveClient in SparkSQLEnv
• [SPARK-29637][CORE] Add description to Job SHS web API
• [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture
• [SPARK-29498][SQL] CatalogTable to HiveTable should not change the table's ownership
• [SPARK-29530][SQL] Make SQLConf in SQL parse process thread safe
• [SPARK-21287][SQL] Remove requirement of fetch_size>=0 from JDBCOptions
• [SPARK-29081]Replace calls to SerializationUtils.clone on properties with a faster implementation
• [SPARK-21492][SQL][FOLLOW UP] Reimplement UnsafeExternalRowSorter in database style iterator
• [SPARK-21492][SQL] Fix memory leak in SortMergeJoin
• [SPARK-29556][CORE] Avoid putting request path in error response in ErrorServlet
• [SPARK-29494][SQL] Fix for ArrayOutofBoundsException while converting string to timestamp
• [SPARK-27259][CORE] Allow setting -1 as length for FileBlock
• [SPARK-29445][CORE] Bump netty-all from 4.1.39.Final to 4.1.42.Final
• [SPARK-29410] Update commons-beanutils to 1.9.4
• [SPARK-28917][CORE] Synchronize access to RDD mutable state
• [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
• [SPARK-29048] Improve performance on Column.isInCollection() with a large size collection
• [SPARK-28962][SQL] Provide index argument to filter lambda functions
• [SPARK-25753][CORE] Fix reading small files via BinaryFileRDD
• [SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors
• [SPARK-29244][CORE] Prevent freed page in BytesToBytesMap free again
• [SPARK-29055][CORE] Update driver/executors' storage memory when block is removed from BlockManager
• [SPARK-29186][SQL] AliasIdentifier should be converted to Json in prettyJson
• [SPARK-29186][SQL] AliasIdentifier should be converted to Json in prettyJson
• [SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state
• [SPARK-29263][SCHEDULER] Update availableSlots in resourceOffers() before checking available slots for barrier taskSet
• [SPARK-29240][PYTHON] Pass Py4J column instance to support PySpark column in element_at function
• [SPARK-29213][SQL] Generate extra IsNotNull predicate in FilterExec
• [SPARK-29229][SQL] Change the additional remote repository in IsolatedClientLoader to google minor
• [SPARK-28599][SQL] Fix Duration column sorting for ThriftServerSessionPage
• [SPARK-25903][CORE] TimerTask should be synchronized on ContextBarrierState
• [SPARK-29053][WEBUI] Sort does not work on some columns
• [SPARK-29177][CORE] fix zombie tasks after stage abort

Maintenance updates

See Databricks Runtime 6.2 maintenance updates.

System environment

• Operating System: Ubuntu 16.04.6 LTS
• Java: 1.8.0_232
• Scala: 2.11.12
• Python: 3.7.3
• R: R version 3.6.1 (2019-07-05)
• Delta Lake: 0.4.0

In this section:

• Installed Python libraries
• Installed R libraries
• Installed Java and Scala libraries (Scala 2.11 cluster version)

Installed Python libraries

Installed R libraries

Installed Java and Scala libraries (Scala 2.11 cluster version)