Databricks Runtime 3.4 (EoS)

Databricks released this version in November 2017.

The following release notes provide information about the Databricks Runtime 3.4 powered by Apache Spark.

Changes and Improvements

• Introduced new vectorized UDFs for PySpark.
• Capable of limiting the max number of partitions in query watchdog.
• Improved error message communication when installing CRAN packages using Databricks Library UI and API.
• MySQL JDBC driver has been replaced by MariaDB JDBC driver.
• Upgraded Python library setuptools from 36.2.7 to 36.6.0, wheel from 0.30.0a0 to 0.30.0.
• Upgraded R from 3.2.3 to 3.4.2.
• Upgraded several pre-installed R libraries. For details, see Pre-installed R Libraries.
• Added AES encryption for customers who require HIPAA-compliant clusters

Apache Spark

The Databricks Runtime 3.4 includes Apache Spark 2.2.0. Databricks Runtime 3.4 also includes the following extra bug fixes and improvements made to Spark:

• [SPARK-21718][SQL] Heavy log of type: "Skipping partition based on stats …"
• [SPARK-22278][SS] Expose current event time watermark and current processing time in GroupState
• [SPARK-22187][SS] Update unsaferow format for saved state such that we can set timeouts when state is null
• [SPARK-22136][SS] Evaluate one-sided conditions early in stream-stream joins
• [SPARK-20396][SQL][PYSPARK] groupby().apply() with pandas udf
• [SPARK-22124][SQL] Sample and Limit should also defer input evaluation under codegen
• [SPARK-21782][CORE] Repartition creates skews when numPartitions is a power of 2
• [SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache
• [SPARK-22176][SQL] Fix overflow issue in Dataset.show
• [SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than partitions
• [SPARK-21517][CORE] Avoid copying memory when transfer chunks remotely
• [SPARK-22238]Fix plan resolution bug caused by EnsureStatefulOpPartitioning
• [SPARK-22159][SQL] Make config names consistently end with "enabled".
• [SPARK-20331][SQL] Enhanced Hive partition pruning predicate pushdown
• [SPARK-22223][SQL] ObjectHashAggregate should not introduce unnecessary shuffle
• [SPARK-22263][SQL] Refactor deterministic as lazy value
• [SPARK-22257][SQL] Reserve all non-deterministic expressions in ExpressionSet
• [SPARK-21907][CORE] Fix oom during spill
• [SPARK-22252][SPARK-21165]FileFormatWriter bug fixes
• [SPARK-22230]Swap per-row order in state store restore.
• [SPARK-21988][SS] Implement StreamingRelation.computeStats to fix explain
• [SPARK-22170][SQL] Reduce memory consumption in broadcast joins
• [SPARK-22206][SQL][SPARKR] gapply in R can't work on empty grouping columns
• [SPARK-20466][CORE] HadoopRDD#addLocalConfiguration throws NPE
• [SPARK-22178][SQL] Refresh Persistent Views by REFRESH TABLE Command
• [SPARK-22158][SQL] convertMetastore should not ignore table property
• [SPARK-22146]FileNotFoundException while reading ORC files containing special characters
• [SPARK-22161][SQL] Add Impala-modified TPC-DS queries
• [SPARK-22129][SPARK-22138] Release script improvements
• [SPARK-22143][SQL] Fix memory leak in OffHeapColumnVector
• [SPARK-22140]Add TPCDSQuerySuite
• [SPARK-22214][SQL] Refactor the list hive partitions code
• [SPARK-22136][SS] Implement stream-stream outer joins.
• [SPARK-22053][SS] Stream-stream inner join in Append Mode
• [SPARK-22153][SQL] Rename ShuffleExchange -ShuffleExchangeExec
• [SPARK-22160][SQL] Make sample points per partition (in range partitioner) configurable and bump the default value up to 100
• [SPARK-22159][SQL] Make config names consistently end with "enabled".
• [SPARK-22141][SQL] Propagate empty relation before checking Cartesian products
• [SPARK-22120][SQL] TestHiveSparkSession.reset() should clean out Hive warehouse directory
• [SPARK-22083][CORE] Release locks in MemoryStore.evictBlocksToFreeSpace
• [SPARK-22107]Change as to alias in python quickstart
• [SPARK-22109][SQL] Resolves type conflicts between strings and timestamps in partition column
• [SPARK-22092]Reallocation in OffHeapColumnVector.reserveInternal corrupts struct and array data
• [SPARK-18136]Fix SPARK_JARS_DIR for Python pip install on Windows
• [SPARK-22094][SS] processAllAvailable should check the query state
• [SPARK-21928][CORE] Set classloader on SerializerManager's private kryo
• [SPARK-22076][SQL] Expand.projections should not be a Stream
• [SPARK-22052]Incorrect Metric assigned in MetricsReporter.scala
• [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
• [SPARK-21953]Show both memory and disk bytes spilled if either is present
• [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs
• [SPARK-18608][ML][FOLLOWUP] Fix double caching for PySpark OneVsRest.
• [SPARK-21980][SQL] References in grouping functions should be indexed with semanticEquals
• [SPARK-22103]Move HashAggregateExec parent consume to a separate function
• [SPARK-22141][SQL] Propagate empty relation before checking Cartesian products
• [SPARK-21977]SinglePartition optimizations break certain Streaming Stateful Aggregation requirements

Known Issues

• Log links on the executor page are not set correctly. Please use the worker page to access stdout and stderr links of an executor for now.

Maintenance Updates

See Databricks Runtime 3.4 maintenance updates.

System Environment

• Operating System: Ubuntu 16.04.3 LTS
• Java: 1.8.0_131
• Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
• Python: 2.7.12 (or 3.5.2 if using Python 3)
• R: R version 3.4.2 (2017-09-28)
• For GPU clusters, the following NVIDIA GPU libraries are installed:
  • CUDA 8.0
  • CUDNN 6.0

Pre-installed Python Libraries

Pre-installed R Libraries

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)