Databricks Runtime 3.5 LTS (EoS)

Databricks released this version in December 2017. It was declared Long Term Support (LTS) in January 2018. Support ended on January 2, 2020.

The following release notes provide information about Databricks Runtime 3.5, powered by Apache Spark.

Changes and improvements

• Added repartitionByRange to Dataset API.

• Added "Heap Histogram" to Spark UI's Executors page for driver and executors. A heap histogram shows the total size and instance count for each class in the heap, which allows uses to easily understand how different kinds of classes consume heap memory of a driver or an executor.

  

  

• Improved window function performance.

• Upgraded Python library setuptools from 36.6.0 to 38.2.3.

• Upgraded several pre-installed R libraries. For details, see Pre-installed R Libraries.

• Bug fixes and stability improvements.

Apache Spark

Databricks Runtime 3.5 includes Apache Spark 2.2.1. This release includes all fixes and improvements included in Databricks Runtime 3.4 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-20557][SQL] Only support TIMESTAMP WITH TIME ZONE for Oracle Dialect
• [SPARK-21652][SQL] Fix rule conflict between InferFiltersFromConstraints and the other Filter Reduction Rules
• [SPARK-22721] BytesToBytesMap peak memory not updated.
• [SPARK-22431][SQL] Ensure that the datatype in the schema for the table/view metadata is parseable by Spark before persisting it
• [SPARK-22686][SQL] DROP TABLE IF EXISTS should not show AnalysisException
• [SPARK-21652][SQL] Fix rule conflict between InferFiltersFromConstraints and ConstantPropagation
• [SPARK-22162] Executors and the driver should use consistent JobIDs in the RDD commit protocol
• [SPARK-22635][SQL][ORC] FileNotFoundException while reading ORC files containing special characters
• [SPARK-22601][SQL] Data load is getting displayed successful on providing non existing nonlocal file path
• [SPARK-22653] executorAddress registered in CoarseGrainedSchedulerBac…
• [SPARK-22373] Bump Janino dependency version to fix thread safety issue…
• [SPARK-22637][SQL] Only refresh a logical plan once.
• [SPARK-22603][SQL] Fix 64KB JVM bytecode limit problem with FormatString
• [SPARK-22595][SQL] fix flaky test: CastSuite.SPARK-22500: cast for struct should not generate codes beyond 64KB
• [SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW
• [SPARK-17920][SPARK-19580][SPARK-19878][SQL] Support writing to Hive table which uses Avro schema url 'avro.schema.url'
• [SPARK-22548][SQL] Incorrect nested AND expression pushed down to JDBC data source
• [SPARK-22500][SQL] Fix 64KB JVM bytecode limit problem with cast
• [SPARK-22550][SQL] Fix 64KB JVM bytecode limit problem with elt
• [SPARK-22508][SQL] Fix 64KB JVM bytecode limit problem with GenerateUnsafeRowJoiner.create()
• [SPARK-22549][SQL] Fix 64KB JVM bytecode limit problem with concat_ws
• [SPARK-22498][SQL] Fix 64KB JVM bytecode limit problem with concat
• [SPARK-22544][SS] FileStreamSource should use its own hadoop conf to call globPathIfNecessary
• [SPARK-22538][ML] SQLTransformer should not unpersist possibly cached input dataset
• [SPARK-22540][SQL] Ensure HighlyCompressedMapStatus calculates correct avgSize
• [SPARK-22535][PYSPARK] Sleep before killing the python worker in PythRunner.MonitorThread (branch-2.2)
• [SPARK-22501][SQL] Fix 64KB JVM bytecode limit problem with in
• [SPARK-22494][SQL] Fix 64KB limit exception with Coalesce and AtleastNNonNulls
• [SPARK-22499][SQL] Fix 64KB JVM bytecode limit problem with least and greatest
• [SPARK-22479][SQL] Exclude credentials from SaveintoDataSourceCommand.simpleString
• [SPARK-22469][SQL] Accuracy problem in comparison with string and numeric
• [SPARK-22721] BytesToBytesMap peak memory usage not accurate after reset()
• [SPARK-22614] Dataset API: repartitionByRange(…)
• [SPARK-22471][SQL] SQLListener consumes much memory causing OutOfMemoryError
• [SPARK-22442][SQL] ScalaReflection should produce correct field names for special characters
• [SPARK-21694][R][ML] Reduce max iterations in Linear SVM test in R to speed up AppVeyor build
• [SPARK-22464][SQL] No pushdown for Hive metastore partition predicates containing null-safe equality
• [SPARK-22488][SQL] Fix the view resolution issue in the SparkSession internal table() API
• [SPARK-21720][SQL] Fix 64KB JVM bytecode limit problem with AND or OR
• [SPARK-21667][STREAMING] ConsoleSink should not fail streaming query with checkpointLocation option
• [SPARK-19644][SQL] Clean up Scala reflection garbage after creating Encoder (branch-2.2)
• [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem in calculating hash for nested structs
• [SPARK-22243][DSTREAM] spark.yarn.jars should reload from config when checkpoint recovery
• [SPARK-22344][SPARKR] clean up install dir if running test as source package
• [SPARK-22472][SQL] add null check for top-level primitive values
• [SPARK-22403][SS] Add optional checkpointLocation argument to StructuredKafkaWordCount example
• [SPARK-22281][SPARKR] Handle R method breaking signature changes
• [SPARK-22417][PYTHON] Fix for createDataFrame from pandas.DataFrame with timestamp
• [SPARK-22315][SPARKR] Warn if SparkR package version doesn't match SparkContext
• [SPARK-22429][STREAMING] Streaming checkpointing code does not retry after failure
• [SPARK-22211][SQL] Remove incorrect FOJ limit pushdown
• [SPARK-22306][SQL] alter table schema should not erase the bucketing metadata at hive side
• [SPARK-22333][SQL] timeFunctionCall(CURRENT_DATE, CURRENT_TIMESTAMP) has conflicts with columnReference
• [SPARK-19611][SQL] set dataSchema correctly in HiveMetastoreCatalog.convertToLogicalRelation
• [SPARK-22291][SQL] Conversion error when transforming array types of uuid, inet and cidr to StingType in PostgreSQL
• [SPARK-22344][SPARKR] Set java.io.tmpdir for SparkR tests
• [SPARK-19727][SQL] Fix for round function that modifies original column
• [SPARK-22445][SQL] Respect stream-side child's needCopyResult in BroadcastHashJoin
• [SPARK-22542][SQL] remove unused features in ColumnarBatch
• [SPARK-17310] Add an option to disable record-level filter in Parquet-side
• [SPARK-22514][SQL] move ColumnVector.Array and ColumnarBatch.Row to individual files
• [SPARK-22222][SPARK-22033] Fix the ARRAY_MAX in BufferHolder and add a test
• [SPARK-10365][SQL] Support Parquet logical type TIMESTAMP_MICROS
• [SPARK-22356][SQL] data source table should support overlapped columns between data and partition schema
• [SPARK-22355][SQL] Dataset.collect is not threadsafe
• [SPARK-22328][CORE] ClosureCleaner should not miss referenced superclass fields
• [SPARK-17902][R] Revive stringsAsFactors option for collect() in SparkR
• [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now tolerates temp files
• [SPARK-22319][CORE] call loginUserFromKeytab before accessing hdfs
• [SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator
• [SPARK-22249][SQL] Check if list of value for IN is empty in the optimizer
• [SPARK-22271][SQL] mean overflows and returns null for some decimal variables
• [SPARK-22249][SQL] isin with empty list throws exception on cached DataFrame
• [SPARK-22223][SQL] ObjectHashAggregate should not introduce unnecessary shuffle
• [SPARK-21549][CORE] Respect OutputFormats with no/invalid output directory provided
• [SPARK-22273][SQL] Fix key/value schema field names in HashMapGenerators.
• [SPARK-14387][SPARK-16628][SPARK-18355][SQL] Use Spark schema to read ORC table instead of ORC file schema
• [SPARK-22252][SQL] FileFormatWriter should respect the input query schema
• [SPARK-22217][SQL] ParquetFileFormat to support arbitrary OutputCommitters
• [SPARK-21907][CORE] oom during spill
• [SPARK-22218] spark shuffle services fails to update secret on app re-attempts
• [SPARK-21549][CORE] Respect OutputFormats with no output directory provided
• [SPARK-22445][SQL] move CodegenContext.copyResult to CodegenSupport
• [SPARK-22408][SQL] RelationalGroupedDataset's distinct pivot value calculation launches unnecessary stages
• [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in UnsafeExternalSorter and ShuffleExternalSorter
• [SPARK-22385][SQL] MapObjects should not access list element by index
• [SPARK-22355][SQL] Dataset.collect is not threadsafe
• [SPARK-21055][SQL] replace grouping__id with grouping_id()
• [SPARK-21762][SQL] FileFormatWriter/BasicWriteTaskStatsTracker metrics collection fails if a new file isn't yet visible

Maintenance updates

See Databricks Runtime 3.5 maintenance updates.

System environment

• Operating System: Ubuntu 16.04.3 LTS
• Java: 1.8.0_151
• Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)
• Python: 2.7.12 (or 3.5.2 if using Python 3)
• R: R version 3.4.2 (2017-09-28)
• GPU clusters: The following NVIDIA GPU libraries are installed:
• Tesla driver 375.66
• CUDA 8.0
• CUDNN 6.0

Pre-installed Python libraries

Pre-installed R libraries

Pre-installed Java and Scala libraries (Scala 2.10 cluster version)

Pre-installed Java and Scala libraries (Scala 2.11 cluster version)