Databricks Runtime 4.1Databricks Runtime 4.1

Databricks 于 2018 年 5 月发布了此映像。Databricks released this image in May 2018.

重要

此版本已于 2019 年 1 月 17 日弃用。This release was deprecated on January 17, 2019. 有关 Databricks Runtime 弃用策略和计划的详细信息，请参阅 Databricks Runtime 支持生命周期。For more information about the Databricks Runtime deprecation policy and schedule, see Databricks runtime support lifecycle.

以下发行说明提供了由 Apache Spark 支持的 Databricks Runtime 4.1 的相关信息。The following release notes provide information about Databricks Runtime 4.1, powered by Apache Spark.

Delta LakeDelta Lake

Databricks Runtime 版本 4.1 为 Delta Lake 添加了重大质量改进和功能。Databricks Runtime version 4.1 adds major quality improvements and functionality to Delta Lake. Databricks 强烈建议将所有 Delta Lake 客户升级到新的运行时。Databricks highly recommends that all Delta Lake customers upgrade to the new runtime. 此版本仍以个人预览版提供，但它代表了即将发布的正式发布版 (GA) 的候选发行版。This release remains in Private Preview, but it represents a candidate release in anticipation of the upcoming general availability (GA) release.

Delta Lake 现在也提供 Azure Databricks 用户的个人预览版。Delta Lake is now also available in Private Preview to Azure Databricks users. 请联系你的帐户管理员或在 https://databricks.com/product/databricks-delta 进行注册。Contact your account manager or sign up at https://databricks.com/product/databricks-delta.

中断性变更Breaking changes

  • Databricks Runtime 4.1 包括对事务协议的更改，以便启用新功能，如验证。Databricks Runtime 4.1 includes changes to the transaction protocol to enable new features, such as validation. 使用 Databricks Runtime 4.1 创建的表会自动使用新版本，且旧版本的 Databricks Runtime 无法写入。Tables created with Databricks Runtime 4.1 automatically use the new version and cannot be written to by older versions of Databricks Runtime. 必须升级现有表才能利用这些改进。You must upgrade existing tables in order to take advantage of these improvements. 若要升级现有表，请先升级写入表的所有作业。To upgrade an existing table, first upgrade all jobs that are writing to the table. 然后运行：Then run:

    com.databricks.delta.Delta.upgradeTableProtocol("<path-to-table>" or "<tableName>")

    有关详细信息，请参阅表版本控制。See Table versioning for more information.

  • 现在，将针对表的当前架构对写入操作进行验证，而不是像以前一样自动添加目标表中缺少的列。Writes are now validated against the current schema of the table rather than, as before, automatically adding columns that are missing from the destination table. 若要启用以前的行为，请将 mergeSchema 选项设置为 true。To enable the previous behavior, set the mergeSchema option to true. 请参阅架构验证。See Schema validation.

  • 如果运行更早版本的 Databricks Delta，则需要升级所有作业才能使用 Databricks Runtime 4.1。If you are running earlier versions of Databricks Delta, you must upgrade all jobs before you use Databricks Runtime 4.1. 如果看到以下任一错误，请升级到 Databricks Runtime 4.1：If you see either of these errors, upgrade to Databricks Runtime 4.1:

    java.lang.NumberFormatException: For input string: "00000000000000....crc"
scala.MatchError

  • 表不能再具有仅因大小写而不同的列。Tables can no longer have columns that differ only by case.

  • Delta 特定的表配置现在必须以 delta. 为前缀Delta-specific table configurations must now be prefixed with delta.

新增功能New features

  • 架构管理 - Databricks Delta 现在验证对现有表的追加和覆盖，以确保所写入的架构与架构匹配。Schema management - Databricks Delta now validates appends and overwrites to an existing table to ensure that the schema being written matches the schema.

    • Databricks Delta 继续支持自动架构迁移。Databricks Delta continues to support automatic schema migration.
    • Databricks Delta 现在支持以下 DDL 以显式修改架构：Databricks Delta now supports the following DDL to modify schema explicitly:
      • 向表中添加新列的 ALTER TABLE ADD COLUMNALTER TABLE ADD COLUMN to add new columns to a table
      • 更改列顺序的 ALTER TABLE CHANGE COLUMNSALTER TABLE CHANGE COLUMNS to change column ordering
      • ALTER TABLE [SET|UNSET] TBLPROPERTIES

    有关详细信息，请参阅架构验证。For details, see Schema validation.

  • 增强的 DDL 和表支持Enhanced DDL and table support

    • 表 DDL 和 saveAsTable() 的完全支持。Full support for table DDL and saveAsTable(). save() 和 saveAsTable() 现在具有相同的语义。save() and saveAsTable() now have identical semantics.
    • 所有 DDL 和 DML 命令都支持名称和 delta. 。All DDL and DML commands support both table name and delta. .
    • SHOW PARTITIONS
    • SHOW COLUMNS
    • DESC TABLE
    • 详细表信息 - 你可以通过运行 DESCRIBE DETAIL 查看表当前的读取器和编写器版本。Detailed table information - You can see the current reader and writer versions of a table by running DESCRIBE DETAIL. 请参阅表版本控制。See Table versioning.
    • 表详细信息 - 现在，每次写入表时都提供出处信息。Table details - Provenance information is now available for each write to a table. 数据边栏还显示 Databricks Delta 表的详细表信息和历史记录。The Data sidebar also shows detailed table information and history for Databricks Delta tables. 请参阅表元数据。See Table metadata.
    • 流式处理表 - 可使用 spark.readStream.format("delta").table("<table-name>") 创建流式处理数据帧。Streaming tables - Streaming DataFrames can be created using spark.readStream.format("delta").table("<table-name>").
    • 仅追加表 - Databricks Delta 现在支持基本数据治理。Append-only tables - Databricks Delta now supports basic data governance. 可通过设置表属性 delta.appendOnly=true 来阻止对表的删除和修改。You can block deletes and modifications to a table by setting the table property delta.appendOnly=true.
    • MERGE INTO 源 - 对 MERGE 的查询规范添加更全面的支持。MERGE INTO source - Adds more comprehensive support to the source query specification of MERGE. 例如，可以在源中指定 LIMIT、ORDER BY 和 INLINE TABLE。For example, you can specify LIMIT, ORDER BY and INLINE TABLE in the source.
    • 表 ACL 的完全支持。Full support for Table ACLs.

性能改进Performance improvements

  • 降低了统计数据收集开销 - 统计数据收集的效率已提高，现在仅为可配置的数量的列收集统计信息（默认情况下设置为 32）。Reduced stats collection overhead - The efficiency of stats collection has been improved and stats are now only collected for a configurable number of columns, set to 32 by default. 由于统计信息收集开销的减少，Databricks Delta 写入性能提高了 2 倍。Databricks Delta write performance has been improved by up to 2x due to the reduction in stats collection overhead. 若要配置列数，请设置表属性 delta.dataSkippingNumIndexedCols=<number-of-columns>。To configure the number of columns, set the table property delta.dataSkippingNumIndexedCols=<number-of-columns>.
  • 支持限制下推 - 统计信息用于限制针对某些查询进行扫描的文件的数量，这些查询的分区列上方具有 LIMIT 和谓词。Support for limit push down - Statistics are used to limit the number of files scanned for queries that have LIMIT and predicates over partition columns. 这适用于笔记本中的查询，因为所有笔记本命令都使用隐式 limit=1000。This is applicable to queries in notebooks due to the implicit limit=1000 in effect for all notebook commands.
  • 流式源中的筛选器下推 - 流式查询现在在启动新流时使用分区来跳过不相关的数据。Filter pushdown in the streaming source - Streaming queries now use partitioning when starting a new stream to skip irrelevant data.
  • 改进的 OPTIMIZE - OPTIMIZE 的并行度现在作为单个 Spark 任务运行，并将使用群集上可用的所有并行度（以前限制为一次 100 个压缩文件）。Improved parallelism for OPTIMIZE - OPTIMIZE now runs as a single Spark task and will use all parallelism available on the cluster (previously was limited to 100 compacted files at a time).
  • DML 中的数据跳过 - UPDATE、DELETE 和 MERGE 现在在发现需要重写的文件时使用统计信息。Data Skipping in DML - UPDATE, DELETE, and MERGE now use statistics when locating files that need to be rewritten.
  • 减少了检查点保留 - 检查点现在保留两天（历史记录仍保留 30 天），以减少事务日志的存储成本。Decreased checkpoint retention - checkpoints are now retained for two days (history is still retained for 30) to decrease storage costs for the transaction log.

API 行为API behavior

  • Databricks Delta 中的 insertInto(<table-name>) 行为和其他数据源相同。The behavior of insertInto(<table-name>) in Databricks Delta is the same as other data sources.
    • 如果未指定模式，或 mode 是 ErrorIfExists、Ignore 或 Append，则会将数据帧中的数据追加到 Databricks Delta 表。If no mode is specified or mode is ErrorIfExists, Ignore, or Append, appends the data in the DataFrame to the Databricks Delta table.
    • 如果 mode 是 Overwrite，则删除现有表中的所有数据，并将数据帧中的数据插入 Databricks Delta 表中。If mode is Overwrite, deletes all data in the existing table and inserts the data from the DataFrame into the Databricks Delta table.
  • 如果已缓存，则必须对 MERGE 的目标表手动取消缓存。If cached, the destination table of MERGE must be manually uncached.

可用性改进Usability improvements

  • 工作负载迁移验证 - 将工作负载迁移到 Databricks Delta 时所犯的常见错误现在引发异常而不是失败：Workload migration validations - Common mistakes made when migrating workloads to Databricks Delta now throw an exception rather than failing:
    • 使用 format("parquet") 读取或写入表。Using format("parquet") to read or write a table.
    • 直接读取或写入分区（即 /path/to/delta/part=1）。Reading or writing directly to a partition (that is, /path/to/delta/part=1).
    • 清扫表的子目录。Vacuuming subdirectories of a table.
    • INSERT OVERWRITE DIRECTORY 对表使用 Parquet。INSERT OVERWRITE DIRECTORY using Parquet on a table.
  • 不区分大小写的配置 - 数据帧读取器/写入器和表属性的选项现在不区分大小写（包括读取路径和写入路径）。Case insensitive configuration - Options to the DataFrame Reader/Writer and table properties are now case insensitive (including both read path and write path).
  • 列名 - 表列名现在可以包含点。Column names - Table column names can now include dots.

已知问题Known issues

  • 多插入语句的插入位置位于不同的工作单位中，而不是同一个事务中。The inserts of multi-insert statements are in different units of work, rather than the same transaction.

Bug 修复Bug fixes

  • 在快速更新表上启动新流时，已修复无限循环。An infinite loop when starting a new stream on a fast updating table has been fixed.

弃用功能Deprecations

结构化流式处理不处理非追加的输入，并且会在对用作源的表进行了任何修改时引发异常。Structured Streaming does not handle input that is not an append and throws an exception if any modifications occur on the table being used as a source. 以前，你可以使用 ignoreFileDeletion 标志替代此行为，但现在已弃用此操作。Previously you could override this behavior using the ignoreFileDeletion flag, but it is now deprecated. 改为使用 ignoreDeletes 或 ignoreChanges。Instead use ignoreDeletes or ignoreChanges. 请参阅用作流源的 Delta 表。See Delta table as a stream source.

其他更改和改进Other Changes and Improvements

  • 已为使用 UI 创建的所有用途的群集启用了查询监视器。Query Watchdog is enabled for all all-purpose clusters created using the UI.
  • 已改进 DBIO 缓存的驱动程序端性能Improved driver-side performance for the DBIO cache
  • 已通过新的本机 Parquet 解码器提升 Parquet 解码性能Improved performance for Parquet decoding via a new native Parquet decoder
  • 已提升常见子表达式消除性能Improved performance for common subexpression elimination
  • 已提升连接小表的大表的数据跳过性能（事实维度表联接）Improved data skipping performance for large table joining small tables (fact-dimension table joins)
  • display() 现在以富 HTML 的形式呈现包含图像数据类型的列。display() now renders columns containing image data types as rich HTML. 请参阅图像。See Images.
  • 导出和导入模型的改进Improvements to Export and import models
    • 已将 dbml-local 升级到最新版本 0.4.1Upgraded dbml-local to latest release 0.4.1
    • 已修复和指定的 threshold 参数一起导出的模型的 bugFixed bug with models exported with threshold parameter specified
    • 已添加对导出 OneVsRestModel、GBTClassificationModel 的支持Added support for exporting OneVsRestModel, GBTClassificationModel
  • 升级了一些已安装的 Python 库：Upgraded some installed Python libraries:
    • pip：从 9.0.1 升级到 10.0.0b2pip: from 9.0.1 to 10.0.0b2
    • setuptools：从 38.5.1 升级到 39.0.1setuptools: from 38.5.1 to 39.0.1
    • tornado：4.5.3 升级到 5.0.1tornado: 4.5.3 to 5.0.1
    • wheel：0.30.0 升级到 0.31.0wheel: 0.30.0 to 0.31.0
  • 升级了多个已安装的 R 库。Upgraded several installed R libraries. 请参阅已安装的 R 库。See Installed R Libraries.
  • 已将 Azure Data Lake Store SDK 从 2.0.11 升级到 2.2.8。Upgraded Azure Data Lake Store SDK from 2.0.11 to 2.2.8.
  • 已将 GPU 群集的 CUDA 从 8.0 升级到 9.0，CUDNN 从 6.0 升级到 7.0。Upgraded CUDA to 9.0 from 8.0 and CUDNN to 7.0 from 6.0 for GPU clusters.

Apache SparkApache Spark

Databricks Runtime 4.1 包括 Apache Spark 2.3.0。Databricks Runtime 4.1 includes Apache Spark 2.3.0. 此版本包括 Databricks Runtime 4.0 中包含的所有修复和改进，以及对 Spark 进行的以下其他 bug 修复和改进：This release includes all fixes and improvements included in Databricks Runtime 4.0, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-24007][SQL] FloatType 和 DoubleType 的 EqualNullSafe 可能会按 codegen 生成错误的结果。[SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.
  • [SPARK-23942][PYTHON][SQL] 使 PySpark 中的收集成为查询执行器侦听器的操作[SPARK-23942][PYTHON][SQL] Makes collect in PySpark as action for a query executor listener
  • [SPARK-23815][CORE] Spark 写入器动态分区覆盖模式可能无法在多级分区上写入输出[SPARK-23815][CORE] Spark writer dynamic partition overwrite mode may fail to write output on multi level partition
  • [SPARK-23748][SS] 修复 SS 连续进程不支持 SubqueryAlias 问题[SPARK-23748][SS] Fix SS continuous process doesn’t support SubqueryAlias issue
  • [SPARK-23963][SQL] 在针对基于文本的 Hive 表的查询中正确处理大量列[SPARK-23963][SQL] Properly handle large number of columns in query on text-based Hive table
  • [SPARK-23867][SCHEDULER] 在 logWarning 中使用 droppedCount[SPARK-23867][SCHEDULER] use droppedCount in logWarning
  • [SPARK-23816][CORE] 终止的任务应忽略 FetchFailures。[SPARK-23816][CORE] Killed tasks should ignore FetchFailures.
  • [SPARK-23809][SQL] 应当通过 getOrCreate 设置活动 SparkSession[SPARK-23809][SQL] Active SparkSession should be set by getOrCreate
  • [SPARK-23966][SS] 在通用 CheckpointFileManager 接口中重构所有检查点文件写入逻辑[SPARK-23966][SS] Refactoring all checkpoint file writing logic in a common CheckpointFileManager interface
  • [SPARK-21351][SQL] 基于子输出更新可空性[SPARK-21351][SQL] Update nullability based on children’s output
  • [SPARK-23847][PYTHON][SQL] 将 asc_nulls_first、asc_nulls_last 添加到 PySpark[SPARK-23847][PYTHON][SQL] Add asc_nulls_first, asc_nulls_last to PySpark
  • [SPARK-23822][SQL] 改善有关 Parquet 架构不匹配的错误消息[SPARK-23822][SQL] Improve error message for Parquet schema mismatches
  • [SPARK-23823][SQL] 在 transformExpression 中保留来源[SPARK-23823][SQL] Keep origin in transformExpression
  • [SPARK-23838][WEBUI] 正在运行的 SQL 查询在 SQL 选项卡中显示为“已完成”[SPARK-23838][WEBUI] Running SQL query is displayed as “completed” in SQL tab
  • [SPARK-23802][SQL] PropagateEmptyRelation 可能在未解决的状态下退出查询计划[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state
  • [SPARK-23727][SQL] 支持在 parquet 中向下推送 DateType 的筛选器[SPARK-23727][SQL] Support for pushing down filters for DateType in parquet
  • [SPARK-23574][SQL] 当正好有 1 个数据读取器工厂时，在 DataSourceV2ScanExec 中报告 SinglePartition。[SPARK-23574][SQL] Report SinglePartition in DataSourceV2ScanExec when there’s exactly 1 data reader factory.
  • [SPARK-23533][SS] 添加对 ContinuousDataReader 的 startOffset 的更改支持[SPARK-23533][SS] Add support for changing ContinuousDataReader’s startOffset
  • [SPARK-23491][SS] 从 ContinuousExecution 重新配置中删除显式作业取消[SPARK-23491][SS] Remove explicit job cancellation from ContinuousExecution reconfiguring
  • [SPARK-23040][CORE] 返回随机读取器的可中断迭代器[SPARK-23040][CORE] Returns interruptible iterator for shuffle reader
  • [SPARK-23827][SS] StreamingJoinExec 应确保将输入数据分区成特定数目的分区[SPARK-23827][SS] StreamingJoinExec should ensure that input data is partitioned into specific number of partitions
  • [SPARK-23639][SQL] 在初始化 SparkSQL CLI 中的元存储客户端之前获取令牌[SPARK-23639][SQL] Obtain token before init metastore client in SparkSQL CLI
  • [SPARK-23806]使用…时，Broadcast.unpersist 可能导致严重异常[SPARK-23806]Broadcast.unpersist can cause fatal exception when used…
  • [SPARK-23599][SQL] 在 Uuid 表达式中使用 RandomUUIDGenerator[SPARK-23599][SQL] Use RandomUUIDGenerator in Uuid expression
  • [SPARK-23599][SQL] 从伪随机数添加 UUID 生成器[SPARK-23599][SQL] Add a UUID generator from Pseudo-Random Numbers
  • [SPARK-23759][UI] 无法将 Spark UI 绑定到特定的主机名/IP[SPARK-23759][UI] Unable to bind Spark UI to specific host name / IP
  • [SPARK-23769][CORE] 删除不必要地禁用 Scalastyle 检查的注释[SPARK-23769][CORE] Remove comments that unnecessarily disable Scalastyle check
  • [SPARK-23614][SQL] 修复使用缓存时错误地重复使用交换的问题[SPARK-23614][SQL] Fix incorrect reuse exchange when caching is used
  • [SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs 应正确保存/还原 CSE 状态[SPARK-23760][SQL] CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly
  • [SPARK-23729][CORE] 解析 glob 时遵循 URI 分段[SPARK-23729][CORE] Respect URI fragment when resolving globs
  • [SPARK-23550][CORE] 清理 Utils[SPARK-23550][CORE] Cleanup Utils
  • [SPARK-23288][SS] 修正 parquet 接收器的输出指标[SPARK-23288][SS] Fix output metrics with parquet sink
  • [SPARK-23264][SQL] 修复 literals.sql.out 中的 scala.MatchError[SPARK-23264][SQL] Fix scala.MatchError in literals.sql.out
  • [SPARK-23649][SQL] 跳过 UTF-8 中禁止的字符[SPARK-23649][SQL] Skipping chars disallowed in UTF-8
  • [SPARK-23691][PYTHON] 尽量在 PySpark 测试中使用 sql_conf 实用工具[SPARK-23691][PYTHON] Use sql_conf util in PySpark tests where possible
  • [SPARK-23644][CORE][UI] 对 SHS 中的 REST 调用使用绝对路径[SPARK-23644][CORE][UI] Use absolute path for REST call in SHS
  • [SPARK-23706][PYTHON] spark.conf.get(value, default=None) 应在 PySpark 中生成 None[SPARK-23706][PYTHON] spark.conf.get(value, default=None) should produce None in PySpark
  • [SPARK-23623][SS] 避免在 CachedKafkaConsumer 中并发使用缓存的使用者[SPARK-23623][SS] Avoid concurrent use of cached consumers in CachedKafkaConsumer
  • [SPARK-23670][SQL] 修复 SparkPlanGraphWrapper 上的内存泄漏[SPARK-23670][SQL] Fix memory leak on SparkPlanGraphWrapper
  • [SPARK-23608][CORE][WEBUI] 在 attachSparkUI 和 detachSparkUI 函数之间添加 SHS 同步，以避免 Jetty 处理程序的并发修改问题[SPARK-23608][CORE][WEBUI] Add synchronization in SHS between attachSparkUI and detachSparkUI functions to avoid concurrent modification issue to Jetty Handlers
  • [SPARK-23671][CORE] 修复状态以启用 SHS 线程池。[SPARK-23671][CORE] Fix condition to enable the SHS thread pool.
  • [SPARK-23658][LAUNCHER] InProcessAppHandle 在 getLogger 中使用错误的类[SPARK-23658][LAUNCHER] InProcessAppHandle uses the wrong class in getLogger
  • [SPARK-23642][DOCS] 修复 AccumulatorV2 子类 isZero scaladoc[SPARK-23642][DOCS] AccumulatorV2 subclass isZero scaladoc fix
  • [SPARK-22915][MLLIB] spark.ml.feature 的从 N 到 Z 的流测试[SPARK-22915][MLLIB] Streaming tests for spark.ml.feature, from N to Z
  • [SPARK-23598][SQL] 使 BufferedRowIterator 中的方法成为公共方法，以避免大型查询出现运行时错误[SPARK-23598][SQL] Make methods in BufferedRowIterator public to avoid runtime error for a large query
  • [SPARK-23546][SQL] 在 CodegenContext 中重构无状态方法/值[SPARK-23546][SQL] Refactor stateless methods/values in CodegenContext
  • [SPARK-23523][SQL] 修复规则 OptimizeMetadataOnlyQuery 导致的错误结果[SPARK-23523][SQL] Fix the incorrect result caused by the rule OptimizeMetadataOnlyQuery
  • [SPARK-23462][SQL] 改善 StructType 中的缺少字段错误消息[SPARK-23462][SQL] improve missing field error message in StructType
  • [SPARK-23624][SQL] 在 Datasource V2 中修订方法 pushFilters 的文档[SPARK-23624][SQL] Revise doc of method pushFilters in Datasource V2
  • [SPARK-23173][SQL] 从 JSON 加载数据时避免创建损坏的 parquet 文件[SPARK-23173][SQL] Avoid creating corrupt parquet files when loading data from JSON
  • [SPARK-23436][SQL] 仅当分区可以强制转换为日期时，才推断分区[SPARK-23436][SQL] Infer partition as Date only if it can be casted to Date
  • [SPARK-23406][SS] 启用流间自联接[SPARK-23406][SS] Enable stream-stream self-joins
  • [SPARK-23490][SQL] 在 CreateTable 中检查包含现有表的 storage.locationUri[SPARK-23490][SQL] Check storage.locationUri with existing table in CreateTable
  • [SPARK-23524]不应检查大型本地随机块是否已损坏。[SPARK-23524]Big local shuffle blocks should not be checked for corruption.
  • [SPARK-23525][SQL] 支持外部 hive 表的 ALTER TABLE CHANGE COLUMN COMMENT[SPARK-23525][SQL] Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table
  • [SPARK-23434][SQL] Spark 不应针对 HDFS 文件路径的元数据目录发出警告[SPARK-23434][SQL] Spark should not warn metadata directory for a HDFS file path
  • [SPARK-23457][SQL] 首先在 ParquetFileFormat 中注册任务完成侦听器[SPARK-23457][SQL] Register task completion listeners first in ParquetFileFormat
  • [SPARK-23329][SQL] 修正三角函数的文档[SPARK-23329][SQL] Fix documentation of trigonometric functions
  • [SPARK-23569][PYTHON] 允许 pandas_udf 使用 python3 样式的类型批注函数[SPARK-23569][PYTHON] Allow pandas_udf to work with python3 style type-annotated functions
  • [SPARK-23570][SQL] 在 HiveExternalCatalogVersionsSuite 中添加 Spark 2.3.0[SPARK-23570][SQL] Add Spark 2.3.0 in HiveExternalCatalogVersionsSuite
  • [SPARK-23517][PYTHON] 使 _pyspark.util.xception_message 通过 Py4JJavaError 生成 Java 端的跟踪[SPARK-23517][PYTHON] Make _pyspark.util.exception_message produce the trace from Java side by Py4JJavaError
  • [SPARK-23508][CORE] 修复“blockManagerIdCache 导致 oom”情况下的 BlockmanagerId[SPARK-23508][CORE] Fix BlockmanagerId in case blockManagerIdCache cause oom
  • [SPARK-23448][SQL] 阐明文档中的 JSON 和 CSV 解析器行为[SPARK-23448][SQL] Clarify JSON and CSV parser behavior in document
  • [SPARK-23365][CORE] 终止空闲执行器时，不要调整 num 执行器。[SPARK-23365][CORE] Do not adjust num executors when killing idle executors.
  • [SPARK-23438][DSTREAMS] 修复驱动程序崩溃时因使用 WAL 产生的 DStreams 数据丢失问题[SPARK-23438][DSTREAMS] Fix DStreams data loss with WAL when driver crashes
  • [SPARK-23475][UI] 还显示跳过的阶段[SPARK-23475][UI] Show also skipped stages
  • [SPARK-23518][SQL] 避免在用户只想读取和写入数据帧时发生元存储访问[SPARK-23518][SQL] Avoid metastore access when the users only want to read and write data frames
  • [SPARK-23406][SS] 启用流间自联接[SPARK-23406][SS] Enable stream-stream self-joins
  • [SPARK-23541][SS] 允许 Kafka 源采用比文章分区数量更大的并行度来读取数据[SPARK-23541][SS] Allow Kafka source to read data with greater parallelism than the number of article-partitions
  • [SPARK-23097][SQL][SS] 将文本套接字源迁移到 V2[SPARK-23097][SQL][SS] Migrate text socket source to V2
  • [SPARK-23362][SS] 将 Kafka Microbatch 源迁移到 v2[SPARK-23362][SS] Migrate Kafka Microbatch source to v2
  • [SPARK-23445]ColumnStat 重构[SPARK-23445]ColumnStat refactoring
  • [SPARK-23092][SQL] 将 MemoryStream 迁移到 DataSourceV2 API[SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 APIs
  • [SPARK-23447][SQL] 清理文本的 codegen 模板[SPARK-23447][SQL] Cleanup codegen template for Literal
  • [SPARK-23366]提高 ReadAheadInputStream 中的热读取路径[SPARK-23366]Improve hot reading path in ReadAheadInputStream
  • [SPARK-22624][PYSPARK] 公开范围分区随机排序[SPARK-22624][PYSPARK] Expose range partitioning shuffle

维护更新Maintenance Updates

请参阅 Databricks Runtime 4.1 维护更新。See Databricks Runtime 4.1 maintenance updates.

系统环境System Environment

  • 操作系统：Ubuntu 16.04.4 LTSOperating System: Ubuntu 16.04.4 LTS
  • Java：1.8.0_162Java: 1.8.0_162
  • Scala：2.11.8Scala: 2.11.8
  • Python：2.7.12 适用于 Python 2 群集，3.5.2 适用于 Python 3 群集。Python: 2.7.12 for Python 2 clusters and 3.5.2 for Python 3 clusters. 有关详细信息，请参阅 Python 版本。For details, see Python version.
  • R：R 版本 3.4.4 (2018-03-15)R: R version 3.4.4 (2018-03-15)
  • GPU 群集：安装了以下 NVIDIA GPU 库：GPU clusters: The following NVIDIA GPU libraries are installed:
    • Tesla 驱动程序 375.66Tesla driver 375.66
    • CUDA 9.0CUDA 9.0
    • cuDNN 7.0cuDNN 7.0

已安装的 Python 库Installed Python Libraries

库Library 版本Version 库Library 版本Version 库Library 版本Version
ansi2htmlansi2html 1.1.11.1.1 argparseargparse 1.2.11.2.1 backports-abcbackports-abc 0.50.5
botoboto 2.42.02.42.0 boto3boto3 1.4.11.4.1 botocorebotocore 1.4.701.4.70
brewer2mplbrewer2mpl 1.4.11.4.1 certificertifi 2016.2.282016.2.28 cfficffi 1.7.01.7.0
chardetchardet 2.3.02.3.0 coloramacolorama 0.3.70.3.7 configobjconfigobj 5.0.65.0.6
密码系统cryptography 1.51.5 cyclercycler 0.10.00.10.0 CythonCython 0.24.10.24.1
decoratordecorator 4.0.104.0.10 docutilsdocutils 0.140.14 enum34enum34 1.1.61.1.6
et-xmlfileet-xmlfile 1.0.11.0.1 freetype-pyfreetype-py 1.0.21.0.2 funcsigsfuncsigs 1.0.21.0.2
fusepyfusepy 2.0.42.0.4 Futurefutures 3.2.03.2.0 ggplotggplot 0.6.80.6.8
html5libhtml5lib 0.9990.999 idnaidna 2.12.1 ipaddressipaddress 1.0.161.0.16
ipythonipython 2.2.02.2.0 ipython-genutilsipython-genutils 0.1.00.1.0 jdcaljdcal 1.21.2
Jinja2Jinja2 2.82.8 jmespathjmespath 0.9.00.9.0 llvmlitellvmlite 0.13.00.13.0
lxmllxml 3.6.43.6.4 MarkupSafeMarkupSafe 0.230.23 matplotlibmatplotlib 1.5.31.5.3
mpld3mpld3 0.20.2 msgpack-pythonmsgpack-python 0.4.70.4.7 ndg-httpsclientndg-httpsclient 0.3.30.3.3
numbanumba 0.28.10.28.1 numpynumpy 1.11.11.11.1 openpyxlopenpyxl 2.3.22.3.2
pandaspandas 0.19.20.19.2 pathlib2pathlib2 2.1.02.1.0 patsypatsy 0.4.10.4.1
pexpectpexpect 4.0.14.0.1 picklesharepickleshare 0.7.40.7.4 PillowPillow 3.3.13.3.1
pippip 10.0.0b210.0.0b2 plyply 3.93.9 prompt-toolkitprompt-toolkit 1.0.71.0.7
psycopg2psycopg2 2.6.22.6.2 ptyprocessptyprocess 0.5.10.5.1 py4jpy4j 0.10.30.10.3
pyarrowpyarrow 0.8.00.8.0 pyasn1pyasn1 0.1.90.1.9 pycparserpycparser 2.142.14
PygmentsPygments 2.1.32.1.3 PyGObjectPyGObject 3.20.03.20.0 pyOpenSSLpyOpenSSL 16.0.016.0.0
pyparsingpyparsing 2.2.02.2.0 pypngpypng 0.0.180.0.18 PythonPython 2.7.122.7.12
python-dateutilpython-dateutil 2.5.32.5.3 python-geohashpython-geohash 0.8.50.8.5 pytzpytz 2016.6.12016.6.1
请求requests 2.11.12.11.1 s3transfers3transfer 0.1.90.1.9 scikit-learnscikit-learn 0.18.10.18.1
scipyscipy 0.18.10.18.1 scourscour 0.320.32 seabornseaborn 0.7.10.7.1
setuptoolssetuptools 39.0.139.0.1 simplejsonsimplejson 3.8.23.8.2 simples3simples3 1.01.0
singledispatchsingledispatch 3.4.0.33.4.0.3 6six 1.10.01.10.0 statsmodelsstatsmodels 0.6.10.6.1
tornadotornado 5.0.15.0.1 traitletstraitlets 4.3.04.3.0 urllib3urllib3 1.19.11.19.1
virtualenvvirtualenv 15.0.115.0.1 wcwidthwcwidth 0.1.70.1.7 wheelwheel 0.31.00.31.0
wsgirefwsgiref 0.1.20.1.2

已安装的 R 库Installed R Libraries

库Library 版本Version 库Library 版本Version 库Library 版本Version
abindabind 1.4-51.4-5 assertthatassertthat 0.2.00.2.0 backportsbackports 1.1.21.1.2
basebase 3.4.43.4.4 BHBH 1.66.0-11.66.0-1 bindrbindr 0.1.10.1.1
bindrcppbindrcpp 0.2.20.2.2 bitbit 1.1-121.1-12 bit64bit64 0.9-70.9-7
bitopsbitops 1.0-61.0-6 blobblob 1.1.11.1.1 bootboot 1.3-201.3-20
brewbrew 1.0-61.0-6 broombroom 0.4.40.4.4 carcar 3.0-03.0-0
carDatacarData 3.0-13.0-1 caretcaret 6.0-796.0-79 cellrangercellranger 1.1.01.1.0
chronchron 2.3-522.3-52 classclass 7.3-147.3-14 clicli 1.0.01.0.0
clustercluster 2.0.72.0.7 codetoolscodetools 0.2-150.2-15 colorspacecolorspace 1.3-21.3-2
commonmarkcommonmark 1.41.4 compilercompiler 3.4.43.4.4 crayoncrayon 1.3.41.3.4
curlcurl 3.23.2 CVSTCVST 0.2-10.2-1 data.tabledata.table 1.10.4-31.10.4-3
datasetsdatasets 3.4.43.4.4 DBIDBI 0.80.8 ddalphaddalpha 1.3.1.11.3.1.1
DEoptimRDEoptimR 1.0-81.0-8 descdesc 1.1.11.1.1 devtoolsdevtools 1.13.51.13.5
dichromatdichromat 2.0-02.0-0 digestdigest 0.6.150.6.15 dimReddimRed 0.1.00.1.0
doMCdoMC 1.3.51.3.5 dplyrdplyr 0.7.40.7.4 DRRDRR 0.0.30.0.3
forcatsforcats 0.3.00.3.0 foreachforeach 1.4.41.4.4 foreignforeign 0.8-690.8-69
gbmgbm 2.1.32.1.3 ggplot2ggplot2 2.2.12.2.1 git2rgit2r 0.21.00.21.0
glmnetglmnet 2.0-162.0-16 glueglue 1.2.01.2.0 gowergower 0.1.20.1.2
graphicsgraphics 3.4.43.4.4 grDevicesgrDevices 3.4.43.4.4 gridgrid 3.4.43.4.4
gsubfngsubfn 0.70.7 gtablegtable 0.2.00.2.0 h2oh2o 3.16.0.23.16.0.2
havenhaven 1.1.11.1.1 hmshms 0.4.20.4.2 httrhttr 1.3.11.3.1
hwriterhwriter 1.3.21.3.2 hwriterPlushwriterPlus 1.0-31.0-3 ipredipred 0.9-60.9-6
iteratorsiterators 1.0.91.0.9 jsonlitejsonlite 1.51.5 kernlabkernlab 0.9-250.9-25
KernSmoothKernSmooth 2.23-152.23-15 labelinglabeling 0.30.3 latticelattice 0.20-350.20-35
lavalava 1.6.11.6.1 lazyevallazyeval 0.2.10.2.1 littlerlittler 0.3.30.3.3
lme4lme4 1.1-171.1-17 lubridatelubridate 1.7.31.7.3 magrittrmagrittr 1.51.5
mapprojmapproj 1.2.61.2.6 mapsmaps 3.3.03.3.0 maptoolsmaptools 0.9-20.9-2
MASSMASS 7.3-497.3-49 矩阵Matrix 1.2-131.2-13 MatrixModelsMatrixModels 0.4-10.4-1
memoisememoise 1.1.01.1.0 methodsmethods 3.4.43.4.4 mgcvmgcv 1.8-231.8-23
mimemime 0.50.5 minqaminqa 1.2.41.2.4 mnormtmnormt 1.5-51.5-5
ModelMetricsModelMetrics 1.1.01.1.0 munsellmunsell 0.4.30.4.3 mvtnormmvtnorm 1.0-71.0-7
nlmenlme 3.1-1373.1-137 nloptrnloptr 1.0.41.0.4 nnetnnet 7.3-127.3-12
numDerivnumDeriv 2016.8-12016.8-1 opensslopenssl 1.0.11.0.1 openxlsxopenxlsx 4.0.174.0.17
parallelparallel 3.4.43.4.4 pbkrtestpbkrtest 0.4-70.4-7 pillarpillar 1.2.11.2.1
pkgconfigpkgconfig 2.0.12.0.1 pkgKittenpkgKitten 0.1.40.1.4 plogrplogr 0.2.00.2.0
plyrplyr 1.8.41.8.4 praisepraise 1.0.01.0.0 prettyunitsprettyunits 1.0.21.0.2
pROCpROC 1.11.01.11.0 prodlimprodlim 1.6.11.6.1 protoproto 1.0.01.0.0
psychpsych 1.8.3.31.8.3.3 purrrpurrr 0.2.40.2.4 quantregquantreg 5.355.35
R.methodsS3R.methodsS3 1.7.11.7.1 R.ooR.oo 1.21.01.21.0 R.utilsR.utils 2.6.02.6.0
R6R6 2.2.22.2.2 randomForestrandomForest 4.6-144.6-14 RColorBrewerRColorBrewer 1.1-21.1-2
RcppRcpp 0.12.160.12.16 RcppEigenRcppEigen 0.3.3.4.00.3.3.4.0 RcppRollRcppRoll 0.2.20.2.2
RCurlRCurl 1.95-4.101.95-4.10 readrreadr 1.1.11.1.1 readxlreadxl 1.0.01.0.0
recipesrecipes 0.1.20.1.2 rematchrematch 1.0.11.0.1 reshape2reshape2 1.4.31.4.3
riorio 0.5.100.5.10 rlangrlang 0.2.00.2.0 robustbaserobustbase 0.92-80.92-8
RODBCRODBC 1.3-151.3-15 roxygen2roxygen2 6.0.16.0.1 rpartrpart 4.1-134.1-13
rprojrootrprojroot 1.3-21.3-2 RserveRserve 1.7-31.7-3 RSQLiteRSQLite 2.1.02.1.0
rstudioapirstudioapi 0.70.7 scalesscales 0.5.00.5.0 sfsmiscsfsmisc 1.1-21.1-2
spsp 1.2-71.2-7 SparkRSparkR 2.3.02.3.0 SparseMSparseM 1.771.77
spatialspatial 7.3-117.3-11 splinessplines 3.4.43.4.4 sqldfsqldf 0.4-110.4-11
SQUAREMSQUAREM 2017.10-12017.10-1 statmodstatmod 1.4.301.4.30 statsstats 3.4.43.4.4
stats4stats4 3.4.43.4.4 stringistringi 1.1.71.1.7 stringrstringr 1.3.01.3.0
survivalsurvival 2.41-32.41-3 tcltktcltk 3.4.43.4.4 TeachingDemosTeachingDemos 2.102.10
testthattestthat 2.0.02.0.0 tibbletibble 1.4.21.4.2 tidyrtidyr 0.8.00.8.0
tidyselecttidyselect 0.2.40.2.4 timeDatetimeDate 3043.1023043.102 toolstools 3.4.43.4.4
utf8utf8 1.1.31.1.3 utilsutils 3.4.43.4.4 viridisLiteviridisLite 0.3.00.3.0
whiskerwhisker 0.3-20.3-2 withrwithr 2.1.22.1.2 xml2xml2 1.2.01.2.0

已安装的 Java 库和 Scala 库（Scala 2.11 群集版本）Installed Java and Scala libraries (Scala 2.11 cluster version)

组 IDGroup ID 项目 IDArtifact ID 版本Version
antlrantlr antlrantlr 2.7.72.7.7
com.amazonawscom.amazonaws amazon-kinesis-clientamazon-kinesis-client 1.7.31.7.3
com.amazonawscom.amazonaws aws-java-sdk-autoscalingaws-java-sdk-autoscaling 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudformationaws-java-sdk-cloudformation 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudfrontaws-java-sdk-cloudfront 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudhsmaws-java-sdk-cloudhsm 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudsearchaws-java-sdk-cloudsearch 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudtrailaws-java-sdk-cloudtrail 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudwatchaws-java-sdk-cloudwatch 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cloudwatchmetricsaws-java-sdk-cloudwatchmetrics 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-codedeployaws-java-sdk-codedeploy 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cognitoidentityaws-java-sdk-cognitoidentity 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-cognitosyncaws-java-sdk-cognitosync 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-configaws-java-sdk-config 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-coreaws-java-sdk-core 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-datapipelineaws-java-sdk-datapipeline 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-directconnectaws-java-sdk-directconnect 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-directoryaws-java-sdk-directory 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-dynamodbaws-java-sdk-dynamodb 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-ec2aws-java-sdk-ec2 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-ecsaws-java-sdk-ecs 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-efsaws-java-sdk-efs 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-elasticacheaws-java-sdk-elasticache 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-elasticbeanstalkaws-java-sdk-elasticbeanstalk 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-elasticloadbalancingaws-java-sdk-elasticloadbalancing 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-elastictranscoderaws-java-sdk-elastictranscoder 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-emraws-java-sdk-emr 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-glacieraws-java-sdk-glacier 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-iamaws-java-sdk-iam 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-importexportaws-java-sdk-importexport 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-kinesisaws-java-sdk-kinesis 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-kmsaws-java-sdk-kms 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-lambdaaws-java-sdk-lambda 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-logsaws-java-sdk-logs 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-machinelearningaws-java-sdk-machinelearning 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-opsworksaws-java-sdk-opsworks 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-rdsaws-java-sdk-rds 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-redshiftaws-java-sdk-redshift 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-route53aws-java-sdk-route53 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-s3aws-java-sdk-s3 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-sesaws-java-sdk-ses 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-simpledbaws-java-sdk-simpledb 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-simpleworkflowaws-java-sdk-simpleworkflow 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-snsaws-java-sdk-sns 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-sqsaws-java-sdk-sqs 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-ssmaws-java-sdk-ssm 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-storagegatewayaws-java-sdk-storagegateway 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-stsaws-java-sdk-sts 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-supportaws-java-sdk-support 1.11.3131.11.313
com.amazonawscom.amazonaws aws-java-sdk-swf-librariesaws-java-sdk-swf-libraries 1.11.221.11.22
com.amazonawscom.amazonaws aws-java-sdk-workspacesaws-java-sdk-workspaces 1.11.3131.11.313
com.amazonawscom.amazonaws jmespath-javajmespath-java 1.11.3131.11.313
com.carrotsearchcom.carrotsearch hppchppc 0.7.20.7.2
com.chuusaicom.chuusai shapeless_2.11shapeless_2.11 2.3.22.3.2
com.clearspring.analyticscom.clearspring.analytics 流 (stream)stream 2.7.02.7.0
com.databrickscom.databricks RserveRserve 1.8-31.8-3
com.databrickscom.databricks dbml-local_2.11dbml-local_2.11 0.4.1-db1-spark2.30.4.1-db1-spark2.3
com.databrickscom.databricks dbml-local_2.11-testsdbml-local_2.11-tests 0.4.1-db1-spark2.30.4.1-db1-spark2.3
com.databrickscom.databricks jets3tjets3t 0.7.1-00.7.1-0
com.databricks.scalapbcom.databricks.scalapb compilerplugin_2.11compilerplugin_2.11 0.4.15-90.4.15-9
com.databricks.scalapbcom.databricks.scalapb scalapb-runtime_2.11scalapb-runtime_2.11 0.4.15-90.4.15-9
com.esotericsoftwarecom.esotericsoftware kryo-shadedkryo-shaded 3.0.33.0.3
com.esotericsoftwarecom.esotericsoftware minlogminlog 1.3.01.3.0
com.fasterxmlcom.fasterxml classmateclassmate 1.0.01.0.0
com.fasterxml.jackson.corecom.fasterxml.jackson.core jackson-annotationsjackson-annotations 2.6.72.6.7
com.fasterxml.jackson.corecom.fasterxml.jackson.core jackson-corejackson-core 2.6.72.6.7
com.fasterxml.jackson.corecom.fasterxml.jackson.core jackson-databindjackson-databind 2.6.7.12.6.7.1
com.fasterxml.jackson.dataformatcom.fasterxml.jackson.dataformat jackson-dataformat-cborjackson-dataformat-cbor 2.6.72.6.7
com.fasterxml.jackson.datatypecom.fasterxml.jackson.datatype jackson-datatype-jodajackson-datatype-joda 2.6.72.6.7
com.fasterxml.jackson.modulecom.fasterxml.jackson.module jackson-module-paranamerjackson-module-paranamer 2.6.72.6.7
com.fasterxml.jackson.modulecom.fasterxml.jackson.module jackson-module-scala_2.11jackson-module-scala_2.11 2.6.7.12.6.7.1
com.github.fommilcom.github.fommil jniloaderjniloader 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib corecore 1.1.21.1.2
com.github.fommil.netlibcom.github.fommil.netlib native_ref-javanative_ref-java 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib native_ref-java-nativesnative_ref-java-natives 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib native_system-javanative_system-java 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib native_system-java-nativesnative_system-java-natives 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib netlib-native_ref-linux-x86_64-nativesnetlib-native_ref-linux-x86_64-natives 1.11.1
com.github.fommil.netlibcom.github.fommil.netlib netlib-native_system-linux-x86_64-nativesnetlib-native_system-linux-x86_64-natives 1.11.1
com.github.lubencom.github.luben zstd-jnizstd-jni 1.3.2-21.3.2-2
com.github.rwlcom.github.rwl jtransformsjtransforms 2.4.02.4.0
com.google.code.findbugscom.google.code.findbugs jsr305jsr305 2.0.12.0.1
com.google.code.gsoncom.google.code.gson gsongson 2.2.42.2.4
com.google.guavacom.google.guava guavaguava 15.015.0
com.google.protobufcom.google.protobuf protobuf-javaprotobuf-java 2.6.12.6.1
com.googlecode.javaewahcom.googlecode.javaewah JavaEWAHJavaEWAH 0.3.20.3.2
com.h2databasecom.h2database h2h2 1.3.1741.3.174
com.jamesmurty.utilscom.jamesmurty.utils java-xmlbuilderjava-xmlbuilder 1.11.1
com.jcraftcom.jcraft jschjsch 0.1.500.1.50
com.jolboxcom.jolbox bonecpbonecp 0.8.0.RELEASE0.8.0.RELEASE
com.mchangecom.mchange c3p0c3p0 0.9.5.10.9.5.1
com.mchangecom.mchange mchange-commons-javamchange-commons-java 0.2.100.2.10
com.microsoft.azurecom.microsoft.azure azure-data-lake-store-sdkazure-data-lake-store-sdk 2.2.82.2.8
com.microsoft.sqlservercom.microsoft.sqlserver mssql-jdbcmssql-jdbc 6.2.2.jre86.2.2.jre8
com.ningcom.ning compress-lzfcompress-lzf 1.0.31.0.3
com.sun.mailcom.sun.mail javax.mailjavax.mail 1.5.21.5.2
com.thoughtworks.paranamercom.thoughtworks.paranamer paranamerparanamer 2.82.8
com.trueaccord.lensescom.trueaccord.lenses lenses_2.11lenses_2.11 0.30.3
com.twittercom.twitter chill-javachill-java 0.8.40.8.4
com.twittercom.twitter chill_2.11chill_2.11 0.8.40.8.4
com.twittercom.twitter parquet-hadoop-bundleparquet-hadoop-bundle 1.6.01.6.0
com.twittercom.twitter util-app_2.11util-app_2.11 6.23.06.23.0
com.twittercom.twitter util-core_2.11util-core_2.11 6.23.06.23.0
com.twittercom.twitter util-jvm_2.11util-jvm_2.11 6.23.06.23.0
com.typesafecom.typesafe configconfig 1.2.11.2.1
com.typesafe.scala-loggingcom.typesafe.scala-logging scala-logging-api_2.11scala-logging-api_2.11 2.1.22.1.2
com.typesafe.scala-loggingcom.typesafe.scala-logging scala-logging-slf4j_2.11scala-logging-slf4j_2.11 2.1.22.1.2
com.univocitycom.univocity univocity-parsersunivocity-parsers 2.5.92.5.9
com.vlkancom.vlkan flatbuffersflatbuffers 1.2.0-3f79e0551.2.0-3f79e055
com.zaxxercom.zaxxer HikariCPHikariCP 2.4.12.4.1
commons-beanutilscommons-beanutils commons-beanutilscommons-beanutils 1.7.01.7.0
commons-beanutilscommons-beanutils commons-beanutils-corecommons-beanutils-core 1.8.01.8.0
commons-clicommons-cli commons-clicommons-cli 1.21.2
commons-codeccommons-codec commons-codeccommons-codec 1.101.10
commons-collectionscommons-collections commons-collectionscommons-collections 3.2.23.2.2
commons-configurationcommons-configuration commons-configurationcommons-configuration 1.61.6
commons-dbcpcommons-dbcp commons-dbcpcommons-dbcp 1.41.4
commons-digestercommons-digester commons-digestercommons-digester 1.81.8
commons-httpclientcommons-httpclient commons-httpclientcommons-httpclient 3.13.1
commons-iocommons-io commons-iocommons-io 2.42.4
commons-langcommons-lang commons-langcommons-lang 2.62.6
commons-loggingcommons-logging commons-loggingcommons-logging 1.1.31.1.3
commons-netcommons-net commons-netcommons-net 2.22.2
commons-poolcommons-pool commons-poolcommons-pool 1.5.41.5.4
info.ganglia.gmetric4jinfo.ganglia.gmetric4j gmetric4jgmetric4j 1.0.71.0.7
io.airliftio.airlift aircompressoraircompressor 0.80.8
io.dropwizard.metricsio.dropwizard.metrics metrics-coremetrics-core 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-gangliametrics-ganglia 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-graphitemetrics-graphite 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-healthchecksmetrics-healthchecks 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-jetty9metrics-jetty9 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-jsonmetrics-json 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-jvmmetrics-jvm 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-log4jmetrics-log4j 3.1.53.1.5
io.dropwizard.metricsio.dropwizard.metrics metrics-servletsmetrics-servlets 3.1.53.1.5
io.nettyio.netty nettynetty 3.9.9.Final3.9.9.Final
io.nettyio.netty netty-allnetty-all 4.1.17.Final4.1.17.Final
io.prometheusio.prometheus simpleclientsimpleclient 0.0.160.0.16
io.prometheusio.prometheus simpleclient_commonsimpleclient_common 0.0.160.0.16
io.prometheusio.prometheus simpleclient_dropwizardsimpleclient_dropwizard 0.0.160.0.16
io.prometheusio.prometheus simpleclient_servletsimpleclient_servlet 0.0.160.0.16
io.prometheus.jmxio.prometheus.jmx 收集器collector 0.70.7
javax.activationjavax.activation activationactivation 1.1.11.1.1
javax.annotationjavax.annotation javax.annotation-apijavax.annotation-api 1.21.2
javax.eljavax.el javax.el-apijavax.el-api 2.2.42.2.4
javax.jdojavax.jdo jdo-apijdo-api 3.0.13.0.1
javax.servletjavax.servlet javax.servlet-apijavax.servlet-api 3.1.03.1.0
javax.servlet.jspjavax.servlet.jsp jsp-apijsp-api 2.12.1
javax.transactionjavax.transaction jtajta 1.11.1
javax.validationjavax.validation validation-apivalidation-api 1.1.0.Final1.1.0.Final
javax.ws.rsjavax.ws.rs javax.ws.rs-apijavax.ws.rs-api 2.0.12.0.1
javax.xml.bindjavax.xml.bind jaxb-apijaxb-api 2.2.22.2.2
javax.xml.streamjavax.xml.stream stax-apistax-api 1.0-21.0-2
javolutionjavolution javolutionjavolution 5.5.15.5.1
jlinejline jlinejline 2.112.11
joda-timejoda-time joda-timejoda-time 2.9.32.9.3
log4jlog4j apache-log4j-extrasapache-log4j-extras 1.2.171.2.17
log4jlog4j log4jlog4j 1.2.171.2.17
net.hydromaticnet.hydromatic eigenbase-propertieseigenbase-properties 1.1.51.1.5
net.ihardernet.iharder base64base64 2.3.82.3.8
net.java.dev.jets3tnet.java.dev.jets3t jets3tjets3t 0.9.40.9.4
net.razorvinenet.razorvine pyrolitepyrolite 4.134.13
net.sf.jpamnet.sf.jpam jpamjpam 1.11.1
net.sf.opencsvnet.sf.opencsv opencsvopencsv 2.32.3
net.sf.supercsvnet.sf.supercsv super-csvsuper-csv 2.2.02.2.0
net.sourceforge.f2jnet.sourceforge.f2j arpack_combined_allarpack_combined_all 0.10.1
org.acpltorg.acplt oncrpconcrpc 1.0.71.0.7
org.antlrorg.antlr ST4ST4 4.0.44.0.4
org.antlrorg.antlr antlr-runtimeantlr-runtime 3.43.4
org.antlrorg.antlr antlr4-runtimeantlr4-runtime 4.74.7
org.antlrorg.antlr stringtemplatestringtemplate 3.2.13.2.1
org.apache.antorg.apache.ant antant 1.9.21.9.2
org.apache.antorg.apache.ant ant-jschant-jsch 1.9.21.9.2
org.apache.antorg.apache.ant ant-launcherant-launcher 1.9.21.9.2
org.apache.arroworg.apache.arrow arrow-formatarrow-format 0.8.00.8.0
org.apache.arroworg.apache.arrow arrow-memoryarrow-memory 0.8.00.8.0
org.apache.arroworg.apache.arrow arrow-vectorarrow-vector 0.8.00.8.0
org.apache.avroorg.apache.avro avroavro 1.7.71.7.7
org.apache.avroorg.apache.avro avro-ipcavro-ipc 1.7.71.7.7
org.apache.avroorg.apache.avro avro-ipc-testsavro-ipc-tests 1.7.71.7.7
org.apache.avroorg.apache.avro avro-mapred-hadoop2avro-mapred-hadoop2 1.7.71.7.7
org.apache.calciteorg.apache.calcite calcite-avaticacalcite-avatica 1.2.0-incubating1.2.0-incubating
org.apache.calciteorg.apache.calcite calcite-corecalcite-core 1.2.0-incubating1.2.0-incubating
org.apache.calciteorg.apache.calcite calcite-linq4jcalcite-linq4j 1.2.0-incubating1.2.0-incubating
org.apache.commonsorg.apache.commons commons-compresscommons-compress 1.4.11.4.1
org.apache.commonsorg.apache.commons commons-cryptocommons-crypto 1.0.01.0.0
org.apache.commonsorg.apache.commons commons-lang3commons-lang3 3.53.5
org.apache.commonsorg.apache.commons commons-math3commons-math3 3.4.13.4.1
org.apache.curatororg.apache.curator curator-clientcurator-client 2.7.12.7.1
org.apache.curatororg.apache.curator curator-frameworkcurator-framework 2.7.12.7.1
org.apache.curatororg.apache.curator curator-recipescurator-recipes 2.7.12.7.1
org.apache.derbyorg.apache.derby derbyderby 10.12.1.110.12.1.1
org.apache.directory.apiorg.apache.directory.api api-asn1-apiapi-asn1-api 1.0.0-M201.0.0-M20
org.apache.directory.apiorg.apache.directory.api api-utilapi-util 1.0.0-M201.0.0-M20
org.apache.directory.serverorg.apache.directory.server apacheds-i18napacheds-i18n 2.0.0-M152.0.0-M15
org.apache.directory.serverorg.apache.directory.server apacheds-kerberos-codecapacheds-kerberos-codec 2.0.0-M152.0.0-M15
org.apache.hadooporg.apache.hadoop hadoop-annotationshadoop-annotations 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-authhadoop-auth 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-clienthadoop-client 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-commonhadoop-common 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-hdfshadoop-hdfs 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-mapreduce-client-apphadoop-mapreduce-client-app 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-mapreduce-client-commonhadoop-mapreduce-client-common 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-mapreduce-client-corehadoop-mapreduce-client-core 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-mapreduce-client-jobclienthadoop-mapreduce-client-jobclient 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-mapreduce-client-shufflehadoop-mapreduce-client-shuffle 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-yarn-apihadoop-yarn-api 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-yarn-clienthadoop-yarn-client 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-yarn-commonhadoop-yarn-common 2.7.32.7.3
org.apache.hadooporg.apache.hadoop hadoop-yarn-server-commonhadoop-yarn-server-common 2.7.32.7.3
org.apache.htraceorg.apache.htrace htrace-corehtrace-core 3.1.0-incubating3.1.0-incubating
org.apache.httpcomponentsorg.apache.httpcomponents httpclienthttpclient 4.5.44.5.4
org.apache.httpcomponentsorg.apache.httpcomponents httpcorehttpcore 4.4.84.4.8
org.apache.ivyorg.apache.ivy ivyivy 2.4.02.4.0
org.apache.orcorg.apache.orc orc-core-nohiveorc-core-nohive 1.4.11.4.1
org.apache.orcorg.apache.orc orc-mapreduce-nohiveorc-mapreduce-nohive 1.4.11.4.1
org.apache.parquetorg.apache.parquet parquet-columnparquet-column 1.8.2-databricks11.8.2-databricks1
org.apache.parquetorg.apache.parquet parquet-commonparquet-common 1.8.2-databricks11.8.2-databricks1
org.apache.parquetorg.apache.parquet parquet-encodingparquet-encoding 1.8.2-databricks11.8.2-databricks1
org.apache.parquetorg.apache.parquet parquet-formatparquet-format 2.3.12.3.1
org.apache.parquetorg.apache.parquet parquet-hadoopparquet-hadoop 1.8.2-databricks11.8.2-databricks1
org.apache.parquetorg.apache.parquet parquet-jacksonparquet-jackson 1.8.2-databricks11.8.2-databricks1
org.apache.thriftorg.apache.thrift libfb303libfb303 0.9.30.9.3
org.apache.thriftorg.apache.thrift libthriftlibthrift 0.9.30.9.3
org.apache.xbeanorg.apache.xbean xbean-asm5-shadedxbean-asm5-shaded 4.44.4
org.apache.zookeeperorg.apache.zookeeper zookeeperzookeeper 3.4.63.4.6
org.bouncycastleorg.bouncycastle bcprov-jdk15onbcprov-jdk15on 1.581.58
org.codehaus.jacksonorg.codehaus.jackson jackson-core-asljackson-core-asl 1.9.131.9.13
org.codehaus.jacksonorg.codehaus.jackson jackson-jaxrsjackson-jaxrs 1.9.131.9.13
org.codehaus.jacksonorg.codehaus.jackson jackson-mapper-asljackson-mapper-asl 1.9.131.9.13
org.codehaus.jacksonorg.codehaus.jackson jackson-xcjackson-xc 1.9.131.9.13
org.codehaus.janinoorg.codehaus.janino commons-compilercommons-compiler 3.0.83.0.8
org.codehaus.janinoorg.codehaus.janino janinojanino 3.0.83.0.8
org.datanucleusorg.datanucleus datanucleus-api-jdodatanucleus-api-jdo 3.2.63.2.6
org.datanucleusorg.datanucleus datanucleus-coredatanucleus-core 3.2.103.2.10
org.datanucleusorg.datanucleus datanucleus-rdbmsdatanucleus-rdbms 3.2.93.2.9
org.eclipse.jettyorg.eclipse.jetty jetty-clientjetty-client 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-continuationjetty-continuation 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-httpjetty-http 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-iojetty-io 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-jndijetty-jndi 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-plusjetty-plus 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-proxyjetty-proxy 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-securityjetty-security 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-serverjetty-server 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-servletjetty-servlet 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-servletsjetty-servlets 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-utiljetty-util 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-webappjetty-webapp 9.3.20.v201705319.3.20.v20170531
org.eclipse.jettyorg.eclipse.jetty jetty-xmljetty-xml 9.3.20.v201705319.3.20.v20170531
org.fusesource.leveldbjniorg.fusesource.leveldbjni leveldbjni-allleveldbjni-all 1.81.8
org.glassfish.hk2org.glassfish.hk2 hk2-apihk2-api 2.4.0-b342.4.0-b34
org.glassfish.hk2org.glassfish.hk2 hk2-locatorhk2-locator 2.4.0-b342.4.0-b34
org.glassfish.hk2org.glassfish.hk2 hk2-utilshk2-utils 2.4.0-b342.4.0-b34
org.glassfish.hk2org.glassfish.hk2 osgi-resource-locatorosgi-resource-locator 1.0.11.0.1
org.glassfish.hk2.externalorg.glassfish.hk2.external aopalliance-repackagedaopalliance-repackaged 2.4.0-b342.4.0-b34
org.glassfish.hk2.externalorg.glassfish.hk2.external javax.injectjavax.inject 2.4.0-b342.4.0-b34
org.glassfish.jersey.bundles.repackagedorg.glassfish.jersey.bundles.repackaged jersey-guavajersey-guava 2.22.22.22.2
org.glassfish.jersey.containersorg.glassfish.jersey.containers jersey-container-servletjersey-container-servlet 2.22.22.22.2
org.glassfish.jersey.containersorg.glassfish.jersey.containers jersey-container-servlet-corejersey-container-servlet-core 2.22.22.22.2
org.glassfish.jersey.coreorg.glassfish.jersey.core jersey-clientjersey-client 2.22.22.22.2
org.glassfish.jersey.coreorg.glassfish.jersey.core jersey-commonjersey-common 2.22.22.22.2
org.glassfish.jersey.coreorg.glassfish.jersey.core jersey-serverjersey-server 2.22.22.22.2
org.glassfish.jersey.mediaorg.glassfish.jersey.media jersey-media-jaxbjersey-media-jaxb 2.22.22.22.2
org.hibernateorg.hibernate hibernate-validatorhibernate-validator 5.1.1.Final5.1.1.Final
org.iq80.snappyorg.iq80.snappy snappysnappy 0.20.2
org.javassistorg.javassist javassistjavassist 3.18.1-GA3.18.1-GA
org.jboss.loggingorg.jboss.logging jboss-loggingjboss-logging 3.1.3.GA3.1.3.GA
org.jdbiorg.jdbi jdbijdbi 2.63.12.63.1
org.jodaorg.joda joda-convertjoda-convert 1.71.7
org.joddorg.jodd jodd-corejodd-core 3.5.23.5.2
org.json4sorg.json4s json4s-ast_2.11json4s-ast_2.11 3.2.113.2.11
org.json4sorg.json4s json4s-core_2.11json4s-core_2.11 3.2.113.2.11
org.json4sorg.json4s json4s-jackson_2.11json4s-jackson_2.11 3.2.113.2.11
org.lz4org.lz4 lz4-javalz4-java 1.4.01.4.0
org.mariadb.jdbcorg.mariadb.jdbc mariadb-java-clientmariadb-java-client 2.1.22.1.2
org.mockitoorg.mockito mockito-allmockito-all 1.9.51.9.5
org.objenesisorg.objenesis objenesisobjenesis 2.12.1
org.postgresqlorg.postgresql postgresqlpostgresql 42.1.442.1.4
org.roaringbitmaporg.roaringbitmap RoaringBitmapRoaringBitmap 0.5.110.5.11
org.rocksdborg.rocksdb rocksdbjnirocksdbjni 5.2.15.2.1
org.rosuda.REngineorg.rosuda.REngine REngineREngine 2.1.02.1.0
org.scala-langorg.scala-lang scala-compiler_2.11scala-compiler_2.11 2.11.82.11.8
org.scala-langorg.scala-lang scala-library_2.11scala-library_2.11 2.11.82.11.8
org.scala-langorg.scala-lang scala-reflect_2.11scala-reflect_2.11 2.11.82.11.8
org.scala-langorg.scala-lang scalap_2.11scalap_2.11 2.11.82.11.8
org.scala-lang.modulesorg.scala-lang.modules scala-parser-combinators_2.11scala-parser-combinators_2.11 1.0.21.0.2
org.scala-lang.modulesorg.scala-lang.modules scala-xml_2.11scala-xml_2.11 1.0.51.0.5
org.scala-sbtorg.scala-sbt test-interfacetest-interface 1.01.0
org.scalacheckorg.scalacheck scalacheck_2.11scalacheck_2.11 1.12.51.12.5
org.scalanlporg.scalanlp breeze-macros_2.11breeze-macros_2.11 0.13.20.13.2
org.scalanlporg.scalanlp breeze_2.11breeze_2.11 0.13.20.13.2
org.scalatestorg.scalatest scalatest_2.11scalatest_2.11 2.2.62.2.6
org.slf4jorg.slf4j jcl-over-slf4jjcl-over-slf4j 1.7.161.7.16
org.slf4jorg.slf4j jul-to-slf4jjul-to-slf4j 1.7.161.7.16
org.slf4jorg.slf4j slf4j-apislf4j-api 1.7.161.7.16
org.slf4jorg.slf4j slf4j-log4j12slf4j-log4j12 1.7.161.7.16
org.spark-project.hiveorg.spark-project.hive hive-beelinehive-beeline 1.2.1.spark21.2.1.spark2
org.spark-project.hiveorg.spark-project.hive hive-clihive-cli 1.2.1.spark21.2.1.spark2
org.spark-project.hiveorg.spark-project.hive hive-exechive-exec 1.2.1.spark21.2.1.spark2
org.spark-project.hiveorg.spark-project.hive hive-jdbchive-jdbc 1.2.1.spark21.2.1.spark2
org.spark-project.hiveorg.spark-project.hive hive-metastorehive-metastore 1.2.1.spark21.2.1.spark2
org.spark-project.sparkorg.spark-project.spark unusedunused 1.0.01.0.0
org.spire-mathorg.spire-math spire-macros_2.11spire-macros_2.11 0.13.00.13.0
org.spire-mathorg.spire-math spire_2.11spire_2.11 0.13.00.13.0
org.springframeworkorg.springframework spring-corespring-core 4.1.4.RELEASE4.1.4.RELEASE
org.springframeworkorg.springframework spring-testspring-test 4.1.4.RELEASE4.1.4.RELEASE
org.tukaaniorg.tukaani xzxz 1.01.0
org.typelevelorg.typelevel machinist_2.11machinist_2.11 0.6.10.6.1
org.typelevelorg.typelevel macro-compat_2.11macro-compat_2.11 1.1.11.1.1
org.xerialorg.xerial sqlite-jdbcsqlite-jdbc 3.8.11.23.8.11.2
org.xerial.snappyorg.xerial.snappy snappy-javasnappy-java 1.1.2.61.1.2.6
org.yamlorg.yaml snakeyamlsnakeyaml 1.161.16
orooro orooro 2.0.82.0.8
software.amazon.ionsoftware.amazon.ion ion-javaion-java 1.0.21.0.2
staxstax stax-apistax-api 1.0.11.0.1
xmlencxmlenc xmlencxmlenc 0.520.52