已存档的发行说明Archived release notes

Important

Linux 是 HDInsight 3.4 或更高版本上使用的唯一操作系统。Linux is the only operating system used on HDInsight version 3.4 or greater. 有关详细信息,请参阅 HDInsight 版本控制文章For more information, see HDInsight versioning article.

摘要Summary

Azure HDInsight 是 Azure 中最受企业客户青睐的开源 Apache Hadoop 和 Apache Spark 分析服务之一。Azure HDInsight is one of the most popular services among enterprise customers for open-source Apache Hadoop and Apache Spark analytics on Azure.

发行日期:2019/12/17Release date: 12/17/2019

此发行版适用于 HDInsight 3.6 和 4.0。This release applies both for HDInsight 3.6 and 4.0.

新增功能New features

服务标记Service tags

服务标记通过使你轻松限制 Azure 服务的网络访问,以此简化 Azure 虚拟机和 Azure 虚拟网络的安全性。Service tags simplify security for Azure virtual machines and Azure virtual networks by enabling you to easily restrict network access to the Azure services. 你可在网络安全组 (NSG) 规则中使用服务标记,允许或拒绝全局或每个 Azure 区域的特定 Azure 服务流量。You can use service tags in your network security group (NSG) rules to allow or deny traffic to a specific Azure service globally or per Azure region. Azure 会对每个标记下面的 IP 地址进行维护。Azure provides the maintenance of IP addresses underlying each tag. 网络安全组 (NSG) 的 HDInsight 服务标记是运行状况和管理服务的 IP 地址组。HDInsight service tags for network security groups (NSGs) are groups of IP addresses for health and management services. 这些组有助于尽量降低创建安全规则时的复杂性。These groups help minimize complexity for security rule creation. HDInsight 客户可通过 Azure 门户、PowerShell 和 REST API 启用服务标记。HDInsight customers can enable service tag through Azure Portal, PowerShell, and REST API. 有关详细信息,请参阅 Azure HDInsight 的网络安全组 (NSG) 服务标记For more information, see Network security group (NSG) service tags for Azure HDInsight.

自定义 Ambari DBCustom Ambari DB

通过 HDInsight,你现在可将自己的 SQL DB 用于 Apache Ambari。HDInsight now allows you to use your own SQL DB for Apache Ambari. 你可通过 Azure 门户或资源管理器模板配置此自定义 Ambari DB。You can configure this custom Ambari DB from the Azure portal or through resource manager template. 此功能可让你为处理和容量需求选择合适的 SQL DB。This feature allows you to choose the right SQL DB for your processing and capacity needs. 你还可轻松升级以匹配业务增长需求。You can also upgrade easily to match business growth requirements. 有关详细信息,请参阅使用自定义 Ambari 数据库设置 HDInsight 群集For more information, see Set up HDInsight clusters with a custom Ambari DB.

自定义 Ambari DB

弃用Deprecation

此版本无弃用。No deprecations for this release. 若要为即将到来的弃用做好准备,请参阅即将推出的变更To get ready for upcoming deprecations, see Upcoming changes.

行为更改Behavior changes

此版本无行为变更。No behavior changes for this release. 若要为即将到来的行为变更做好准备,请参阅即将推出的变更To get ready for upcoming behavior changes, see Upcoming changes.

即将推出的更改Upcoming changes

即将发布的版本中将推出以下变更。The following changes will happen in upcoming releases.

传输层安全性 (TLS) 1.2 强制措施Transport Layer Security (TLS) 1.2 enforcement

传输层安全性 (TLS) 和安全套接字层 (SSL) 是提供计算机网络通信安全的加密协议。Transport Layer Security (TLS) and Secure Sockets Layer (SSL) are cryptographic protocols that provide communications security over a computer network. 有关详细信息,请参阅传输层安全性For more information, see Transport Layer Security. 虽然 Azure HDInsight 群集接受公共 HTTPS 终结点上的 TLS 1.2 连接,但仍支持 TLS 1.1,以便实现与旧客户端的后向兼容性。While Azure HDInsight clusters accept TLS 1.2 connections on public HTTPS endpoints, TLS 1.1 is still supported for backward compatibility with older clients.

从下一个版本开始,你将可以选择加入和配置新的 HDInsight 群集,以仅接受 TLS 1.2 连接。Starting from the next release, you will be able to opt-in and configure your new HDInsight clusters to only accept TLS 1.2 connections.

今年晚些时候,预计从 2020/6/30 开始,Azure HDInsight 将为所有 HTTPS 连接强制实行 TLS 2.1 或更高版本。Later in the year, starting on 6/30/2020, Azure HDInsight will enforce TLS 1.2 or later versions for all HTTPS connections. 我们建议你确保所有客户端都已准备好处理 TLS 1.2 或更高版本。We recommend that you ensure that all your clients are ready to handle TLS 1.2 or later versions.

迁移到 Azure 虚拟机规模集Moving to Azure virtual machine scale sets

HDInsight 目前使用 Azure 虚拟机来预配群集。HDInsight now uses Azure virtual machines to provision the cluster. 从 2020 年 2 月开始(将在稍后传达确切日期),HDInsight 将改为使用 Azure 虚拟机规模集。Starting from February 2020 (exact date will be communicated later), HDInsight will use Azure virtual machine scale sets instead. 详细了解 Azure 虚拟机规模集See more about Azure virtual machine scale sets.

HBase 2.0 到 2.1HBase 2.0 to 2.1

在即将推出的 HDInsight 4.0 版本中,HBase 版本将从 2.0 升级到 2.1。In the upcoming HDInsight 4.0 release, HBase version will be upgraded from version 2.0 to 2.1.

Bug 修复Bug fixes

HDInsight 会持续改善群集的可靠性和性能。HDInsight continues to make cluster reliability and performance improvements.

组件版本更改Component version change

我们已将 HDInsight 3.6 支持延长到 2020 年 12 月 31 日。We've extended HDInsight 3.6 support to December 31, 2020. 如需了解更多详情,请参阅支持的 HDInsight 版本You can find more details in Supported HDInsight versions.

HDInsight 4.0 无组件版本变更。No component version change for HDInsight 4.0.

HDInsight 3.6 上的 Apache Zeppelin:0.7.0-->0.7.3.Apache Zeppelin on HDInsight 3.6: 0.7.0-->0.7.3.

可以在此文档中获取最新的组件版本。You can find the most up-to-date component versions from this doc.

发行日期:11/07/2019Release date: 11/07/2019

此发行版适用于 HDInsight 3.6 和 4.0。This release applies both for HDInsight 3.6 and 4.0.

新增功能New features

HDInsight 标识中介 (HIB)(预览版)HDInsight Identity Broker (HIB) (Preview)

HDInsight 标识中介 (HIB) 可让用户使用多重身份验证 (MFA) 登录到 Apache Ambari,并获取所需的 Kerberos 票证,而无需在 Azure Active Directory 域服务 (AAD-DS) 中提供密码哈希。HDInsight Identity Broker (HIB) enables users to sign in to Apache Ambari using multi-factor authentication (MFA) and get the required Kerberos tickets without needing password hashes in Azure Active Directory Domain Services (AAD-DS). 目前,HIB 仅适用于通过 Azure 资源管理 (ARM) 模板部署的群集。Currently HIB is only available for clusters deployed through Azure Resource Management (ARM) template.

Kafka Rest API 代理(预览版)Kafka Rest API Proxy (Preview)

Kafka Rest API 代理通过受保护的 AAD 授权和 OAuth 协议,为 Kafka 群集提供一键式的高可用性 REST 代理部署。Kafka Rest API Proxy provides one-click deployment of highly available REST proxy with Kafka cluster via secured AAD authorization and OAuth protocol.

自动缩放Auto scale

适用于 Azure HDInsight 的自动缩放功能现已在所有区域中针对 Apache Spark 和 Hadoop 群集类型推出正式版。Autoscale for Azure HDInsight is now generally available across all regions for Apache Spark and Hadoop cluster types. 使用此功能能够以更具成本效益且高效的方式管理大数据分析工作负荷。This feature makes it possible to manage big data analytics workloads in a more cost-efficient and productive way. 现在,你可以优化 HDInsight 群集的使用,并且只需为所用的资源付费。Now you can optimize use of your HDInsight clusters and only pay for what you need.

可以根据要求,在基于负载和基于计划的自动缩放之间进行选择。Depending on your requirements, you can choose between load-based and schedule-based autoscaling. 基于负载的自动缩放可根据当前资源需求增大和缩小群集大小,而基于计划的自动缩放可根据预定义的计划更改群集大小。Load-based Autoscale can scale the cluster size up and down based on the current resource needs while schedule-based Autoscale can change the cluster size based on a predefined schedule.

适用于 HBase 和 LLAP 工作负荷的自动缩放支持也推出了公共预览版。Autoscale support for HBase and LLAP workload is also public preview. 有关详细信息,请参阅自动缩放 Azure HDInsight 群集For more information, see Automatically scale Azure HDInsight clusters.

适用于 Apache HBase 的 HDInsight 加速写入HDInsight Accelerated Writes for Apache HBase

加速写入使用 Azure 高级 SSD 托管磁盘,可以改善 Apache HBase 预写日志 (WAL) 的性能。Accelerated Writes uses Azure premium SSD managed disks to improve performance of the Apache HBase Write Ahead Log (WAL). 有关详细信息,请参阅面向 Apache HBase 的 Azure HDInsight 加速写入For more information, see Azure HDInsight Accelerated Writes for Apache HBase.

自定义 Ambari DBCustom Ambari DB

HDInsight 现在提供新的容量,使客户能够使用自己的适用于 Ambari 的 SQL 数据库。HDInsight now offers a new capacity to enable customers to use their own SQL DB for Ambari. 现在,客户可以选择适当的用于 Ambari 的 SQL 数据库,并根据自己的业务增长需求轻松对其进行升级。Now customers can choose the right SQL DB for Ambari and  easily upgrade it based on their own business growth requirement. 部署是使用 Azure 资源管理器模板完成的。The deployment is done with an Azure Resource Manager template. 有关详细信息,请参阅使用自定义 Ambari 数据库设置 HDInsight 群集For more information, see Set up HDInsight clusters with a custom Ambari DB.

现已推出适用于 HDInsight 的 F 系列虚拟机F-series virtual machines are now available with HDInsight

F 系列虚拟机 (VM) 非常适合用于体验处理要求很低的 HDInsight。F-series virtual machines(VMs) are good choice to get started with HDInsight with light processing requirements. 根据每个 vCPU 的 Azure 计算单位 (ACU),F 系列以较低的小时定价,在 Azure 产品组合中具有最高性价比。At a lower per-hour list price, the F-series is the best value in price-performance in the Azure portfolio based on the Azure Compute Unit (ACU) per vCPU. 有关详细信息,请参阅为 Azure HDInsight 群集选择适当的 VM 大小For more information, see Selecting the right VM size for your Azure HDInsight cluster.

弃用Deprecation

G 系列虚拟机已弃用G-series virtual machine deprecation

从此发行版开始,HDInsight 中不再提供 G 系列 VM。From this release, G-series VMs are no longer offered in HDInsight.

Dv1 虚拟机已弃用Dv1 virtual machine deprecation

从此发行版开始,已弃用包含 HDInsight 的 Dv1 VM。From this release, the use of Dv1 VMs with HDInsight is deprecated. 客户提出的 Dv1 请求将自动以 Dv2 来满足。Any customer request for Dv1 will be served with Dv2 automatically. Dv1 与 Dv2 VM 的价格没有差别。There is no price difference between Dv1 and Dv2 VMs.

行为更改Behavior changes

群集托管磁盘大小更改Cluster managed disk size change

HDInsight 在群集中提供托管磁盘空间。HDInsight provides managed disk space with the cluster. 从此发行版开始,新建的群集中每个节点的托管磁盘大小将更改为 128 GB。From this release, the managed disk size of each node in the new created cluster is changed to 128 GB.

即将推出的更改Upcoming changes

即将发布的版本中将发生以下更改。The following changes will happen in the upcoming releases.

迁移到 Azure 虚拟机规模集Moving to Azure virtual machine scale sets

HDInsight 目前使用 Azure 虚拟机来预配群集。HDInsight now uses Azure virtual machines to provision the cluster. 从 12 月开始,HDInsight 将改用 Azure 虚拟机规模集。Starting from December, HDInsight will use Azure virtual machine scale sets instead. 详细了解 Azure 虚拟机规模集See more about Azure virtual machine scale sets.

HBase 2.0 到 2.1HBase 2.0 to 2.1

在即将推出的 HDInsight 4.0 版本中,HBase 版本将从 2.0 升级到 2.1。In the upcoming HDInsight 4.0 release, HBase version will be upgraded from version 2.0 to 2.1.

Bug 修复Bug fixes

HDInsight 会持续改善群集的可靠性和性能。HDInsight continues to make cluster reliability and performance improvements.

组件版本更改Component version change

此版本未发生组件版本更改。There is no component version change for this release. 可在此处查找 HDInsight 4.0 和 HDInsight 3.6 的当前组件版本。You could find the current component versions for HDInsight 4.0 and HDInsight 3.6 here.

发布日期:2019/08/07Release Date: 08/07/2019

组件版本Component versions

下面提供了所有 HDInsight 4.0 组件的正式 Apache 版本。The official Apache versions of all HDInsight 4.0 components are given below. 列出的组件是可用的最新稳定版本。The components listed are releases of the most recent stable versions available.

  • Apache Ambari 2.7.1Apache Ambari 2.7.1
  • Apache Hadoop 3.1.1Apache Hadoop 3.1.1
  • Apache HBase 2.0.0Apache HBase 2.0.0
  • Apache Hive 3.1.0Apache Hive 3.1.0
  • Apache Kafka 1.1.1, 2.1.0Apache Kafka 1.1.1, 2.1.0
  • Apache Mahout 0.9.0+Apache Mahout 0.9.0+
  • Apache Oozie 4.2.0Apache Oozie 4.2.0
  • Apache Phoenix 4.7.0Apache Phoenix 4.7.0
  • Apache Pig 0.16.0Apache Pig 0.16.0
  • Apache Ranger 0.7.0Apache Ranger 0.7.0
  • Apache Slider 0.92.0Apache Slider 0.92.0
  • Apache Spark 2.3.1, 2.4.0Apache Spark 2.3.1, 2.4.0
  • Apache Sqoop 1.4.7Apache Sqoop 1.4.7
  • Apache TEZ 0.9.1Apache TEZ 0.9.1
  • Apache Zeppelin 0.8.0Apache Zeppelin 0.8.0
  • Apache ZooKeeper 3.4.6Apache ZooKeeper 3.4.6

除了上面所列的版本以外,Apache 组件的较高版本有时也会捆绑在 HDP 分发版中。Later versions of Apache components are sometimes bundled in the HDP distribution in addition to the versions listed above. 在这种情况下,这些较高版本会列在“技术预览”表中,并且在生产环境中不应替换为上述列表中的 Apache 组件版本。In this case, these later versions are listed in the Technical Previews table and should not substitute for the Apache component versions of the above list in a production environment.

Apache 修补程序信息Apache patch information

有关 HDInsight 4.0 中可用的修补程序的详细信息,请参阅下表中适用于每个产品的修补程序列表。For more information on patches available in HDInsight 4.0, see the patch listing for each product in the table below.

产品名称Product name 修补程序信息Patch information
AmbariAmbari Ambari 修补程序信息Ambari patch information
HadoopHadoop Hadoop 修补程序信息Hadoop patch information
HBaseHBase HBase 修补程序信息HBase patch information
HiveHive 此版本提供 Hive 3.1.0,但不提供其他 Apache 修补程序。This release provides Hive 3.1.0 with no additional Apache patches.
KafkaKafka 此版本提供 Kafka 1.1.1,但不提供其他 Apache 修补程序。This release provides Kafka 1.1.1 with no additional Apache patches.
OozieOozie Oozie 修补程序信息Oozie patch information
PhoenixPhoenix Phoenix 修补程序信息Phoenix patch information
PigPig Pig 修补程序信息Pig patch information
RangerRanger Ranger 修补程序信息Ranger patch information
SparkSpark Spark 修补程序信息Spark patch information
SqoopSqoop 此版本提供 Sqoop 1.4.7,但不提供其他 Apache 修补程序。This release provides Sqoop 1.4.7 with no additional Apache patches.
TezTez 此版本提供 Tez 0.9.1,但不提供其他 Apache 修补程序。This release provides Tez 0.9.1 with no additional Apache patches.
ZeppelinZeppelin 此版本提供 Zeppelin 0.8.0,但不提供其他 Apache 修补程序。This release provides Zeppelin 0.8.0 with no additional Apache patches.
ZookeeperZookeeper Zookeeper 修补程序信息Zookeeper patch information

修复了常见漏洞和透露Fixed Common Vulnerabilities and Exposures

有关此版本中已解决的安全问题的详细信息,请参阅 Hortonworks 编写的修复了 HDP 3.0.1 的常见漏洞和透露For more information on security issues resolved in this release, see Hortonworks' Fixed Common Vulnerabilities and Exposures for HDP 3.0.1.

已知问题Known issues

使用默认安装选项时 Secure HBase 的复制中断Replication is broken for Secure HBase with default installation

对于 HDInsight 4.0,请执行以下步骤:For HDInsight 4.0, do the following steps:

  1. 启用群集间通信。Enable inter-cluster communication.

  2. 登录到活动的头节点。Sign in to the active headnode.

  3. 使用以下命令下载一个脚本以启用复制:Download a script to enable replication with the following command:

    sudo wget https://raw.githubusercontent.com/Azure/hbase-utils/master/replication/hdi_enable_replication.sh
    
  4. 键入命令 sudo kinit <domainuser>Type the command sudo kinit <domainuser>.

  5. 键入以下命令以运行该脚本:Type the following command to run the script:

    sudo bash hdi_enable_replication.sh -m <hn0> -s <srclusterdns> -d <dstclusterdns> -sp <srcclusterpasswd> -dp <dstclusterpasswd> -copydata
    

对于 HDInsight 3.6,请执行以下步骤:For HDInsight 3.6, do the following:

  1. 登录到活动的 HMaster ZK。Sign in to active HMaster ZK.

  2. 使用以下命令下载一个脚本以启用复制:Download a script to enable replication with the following command:

    sudo wget https://raw.githubusercontent.com/Azure/hbase-utils/master/replication/hdi_enable_replication.sh
    
  3. 键入命令 sudo kinit -k -t /etc/security/keytabs/hbase.service.keytab hbase/<FQDN>@<DOMAIN>Type the command sudo kinit -k -t /etc/security/keytabs/hbase.service.keytab hbase/<FQDN>@<DOMAIN>.

  4. 输入以下命令:Type the following command:

    sudo bash hdi_enable_replication.sh -s <srclusterdns> -d <dstclusterdns> -sp <srcclusterpasswd> -dp <dstclusterpasswd> -copydata
    

将 HBase 群集迁移到 HDInsight 4.0 后,Phoenix Sqlline 停止工作Phoenix Sqlline stops working after migrating HBase cluster to HDInsight 4.0

执行以下步骤:Do the following steps:

  1. 删除以下 Phoenix 表:Drop the following Phoenix tables:
    1. SYSTEM.FUNCTION
    2. SYSTEM.SEQUENCE
    3. SYSTEM.STATS
    4. SYSTEM.MUTEX
    5. SYSTEM.CATALOG
  2. 如果无法删除其中的任何表,请重启 HBase 以清除与表建立的任何连接。If you can't delete any of the tables, restart HBase to clear any connections to the tables.
  3. 再次运行 sqlline.pyRun sqlline.py again. Phoenix 将重新创建在步骤 1 中删除的所有表。Phoenix will re-create all of the tables that were deleted in step 1.
  4. 重新生成 HBase 数据的 Phoenix 表和视图。Regenerate Phoenix tables and views for your HBase data.

将 HBase Phoenix 元数据从 HDInsight 3.6 复制到 4.0 之后,Phoenix Sqlline 停止工作Phoenix Sqlline stops working after replicating HBase Phoenix metadata from HDInsight 3.6 to 4.0

执行以下步骤:Do the following steps:

  1. 在执行复制之前,请转到目标 4.0 群集并执行 sqlline.pyBefore doing the replication, go to the destination 4.0 cluster and execute sqlline.py. 此命令将生成类似于 SYSTEM.MUTEXSYSTEM.LOG 且只存在于 4.0 中的 Phoenix 表。This command will generate Phoenix tables like SYSTEM.MUTEX and SYSTEM.LOG that only exist in 4.0.
  2. 删除以下表:Drop the following tables:
    1. SYSTEM.FUNCTION
    2. SYSTEM.SEQUENCE
    3. SYSTEM.STATS
    4. SYSTEM.CATALOG
  3. 启动 HBase 复制Start the HBase replication

弃用Deprecation

Apache Storm 和机器学习服务在 HDInsight 4.0 中不可用。Apache Storm and ML services aren't available in HDInsight 4.0.

发布日期:2019/04/14Release Date: 04/14/2019

新增功能New features

新的更新和功能划分为以下类别:The new updates and capabilities fall in to the following categories:

  • 更新 Hadoop 和其他开源项目 - 除了超过 20 个开源项目的 1000 多个 bug 修复,此更新还包含 Spark (2.3)Kafka (1.0) 的新版本。Update Hadoop and other open-source projects – In addition to 1000+ bug fixes across 20+ open-source projects, this update contains a new version of Spark (2.3) and Kafka (1.0).

    a.a. Apache Spark 2.3 中的新增功能New features in Apache Spark 2.3

    b.b. Apache Kafka 1.0 中的新增功能New features in Apache Kafka 1.0

  • 支持 Azure Data Lake Storage Gen2 - HDInsight 将支持 Azure Data Lake Storage Gen2 的预览版本。Support for Azure Data Lake Storage Gen2 – HDInsight will support the Preview release of Azure Data Lake Storage Gen2. 在可用区域中,客户可以选择将 ADLS Gen2 帐户作为 HDInsight 群集的主要存储或辅助存储。In the available regions, customers will be able to choose an ADLS Gen2 account as the Primary or Secondary store for their HDInsight clusters.

组件版本Component versions

下面列出了所有 HDInsight 3.6 组件的正式 Apache 版本。The official Apache versions of all HDInsight 3.6 components are listed below. 此处列出的所有组件是最新稳定版本的正式 Apache 发行版。All components listed here are official Apache releases of the most recent stable versions available.

  • Apache Hadoop 2.7.3Apache Hadoop 2.7.3

  • Apache HBase 1.1.2Apache HBase 1.1.2

  • Apache Hive 1.2.1Apache Hive 1.2.1

  • Apache Hive 2.1.0Apache Hive 2.1.0

  • Apache Kafka 1.0.0Apache Kafka 1.0.0

  • Apache Mahout 0.9.0+Apache Mahout 0.9.0+

  • Apache Oozie 4.2.0Apache Oozie 4.2.0

  • Apache Phoenix 4.7.0Apache Phoenix 4.7.0

  • Apache Pig 0.16.0Apache Pig 0.16.0

  • Apache Ranger 0.7.0Apache Ranger 0.7.0

  • Apache Slider 0.92.0Apache Slider 0.92.0

  • Apache Spark 2.2.0/2.3.0Apache Spark 2.2.0/2.3.0

  • Apache Sqoop 1.4.6Apache Sqoop 1.4.6

  • Apache Storm 1.1.0Apache Storm 1.1.0

  • Apache TEZ 0.7.0Apache TEZ 0.7.0

  • Apache Zeppelin 0.7.3Apache Zeppelin 0.7.3

  • Apache ZooKeeper 3.4.6Apache ZooKeeper 3.4.6

除了上面所列的版本以外,某些 Apache 组件的较高版本有时也捆绑在 HDP 分发版中。Later versions of a few Apache components are sometimes bundled in the HDP distribution in addition to the versions listed above. 在这种情况下,这些较高版本会列在“技术预览”表中,并且在生产环境中不应替换为上述列表中的 Apache 组件版本。In this case, these later versions are listed in the Technical Previews table and should not substitute for the Apache component versions of the above list in a production environment.

Apache 修补程序信息Apache patch information

HadoopHadoop

此版本提供 Hadoop Common 2.7.3 和以下 Apache 修补程序:This release provides Hadoop Common 2.7.3 and the following Apache patches:

  • HADOOP-13190:在 KMS HA 文档中提到 LoadBalancingKMSClientProvider。HADOOP-13190: Mention LoadBalancingKMSClientProvider in KMS HA documentation.

  • HADOOP-13227:AsyncCallHandler 应使用事件驱动的体系结构来处理异步调用。HADOOP-13227: AsyncCallHandler should use an event driven architecture to handle async calls.

  • HADOOP-14104:客户端应始终请求 kms 提供程序路径的名称节点。HADOOP-14104: Client should always ask namenode for kms provider path.

  • HADOOP-14799:将 nimbus-jose-jwt 更新为 4.41.1。HADOOP-14799: Update nimbus-jose-jwt to 4.41.1.

  • HADOOP-14814:将 FsServerDefaults 上不兼容的 API 更改修复为 HADOOP-14104。HADOOP-14814: Fix incompatible API change on FsServerDefaults to HADOOP-14104.

  • HADOOP-14903:将 json-smart 显式添加到 pom.xml。HADOOP-14903: Add json-smart explicitly to pom.xml.

  • HADOOP-15042:当 numberOfPagesRemaining 为 0 时,Azure PageBlobInputStream.skip() 可以返回负值。HADOOP-15042: Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.

  • HADOOP-15255:LdapGroupsMapping 中的组名称支持大小写转换。HADOOP-15255: Upper/Lower case conversion support for group names in LdapGroupsMapping.

  • HADOOP-15265:从 hadoop-auth pom.xml 中显式排除 json-smart。HADOOP-15265: exclude json-smart explicitly from hadoop-auth pom.xml.

  • HDFS-7922:ShortCircuitCache#close 未发布 ScheduledThreadPoolExecutors。HDFS-7922: ShortCircuitCache#close is not releasing ScheduledThreadPoolExecutors.

  • HDFS-8496:在持有 FSDatasetImpl 锁的情况下调用 stopWriter() 可能会阻止其他线程 (cmccabe)。HDFS-8496: Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe).

  • HDFS-10267:FsDatasetImpl#recoverAppend 和 FsDatasetImpl#recoverClose 上的附加“synchronized”。HDFS-10267: Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose.

  • HDFS-10489:弃用 HDFS 加密区域的 dfs.encryption.key.provider.uri。HDFS-10489: Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones.

  • HDFS-11384:添加让均衡器分散 getBlocks 调用的选项,以避免 NameNode 的 rpc.CallQueueLength 峰值。HDFS-11384: Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike.

  • HDFS-11689:DFSClient%isHDFSEncryptionEnabled 引发的新异常破坏 hacky hive 代码。HDFS-11689: New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code.

  • HDFS-11711:发生“打开的文件过多”异常时 DN 不应删除块。HDFS-11711: DN should not delete the block On "Too many open files" Exception.

  • HDFS-12347:TestBalancerRPCDelay#testBalancerRPCDelay 非常频繁地失败。HDFS-12347: TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently.

  • HDFS-12781:数据节点关闭后,在“名称节点”UI 中,“数据节点”选项卡引发警告消息。HDFS-12781: After Datanode down, In Namenode UI Datanode tab is throwing warning message.

  • HDFS-13054:处理 DFSClient 中的 PathIsNotEmptyDirectoryException 删除调用。HDFS-13054: Handling PathIsNotEmptyDirectoryException in DFSClient delete call.

  • HDFS-13120:concat 后,快照差异可能会损坏。HDFS-13120: Snapshot diff could be corrupted after concat.

  • YARN-3742:如果 ZKClient 创建超时,YARN RM 会关闭。YARN-3742: YARN RM will shut down if ZKClient creation times out.

  • YARN-6061:为 RM 中的关键线程添加 UncaughtExceptionHandler。YARN-6061: Add an UncaughtExceptionHandler for critical threads in RM.

  • YARN-7558:如果启用 UI 身份验证,获取运行中容器的日志的 yarn logs 命令会失败。YARN-7558: yarn logs command fails to get logs for running containers if UI authentication is enabled.

  • YARN-7697:即使日志聚合已完成,提取已完成应用程序的日志也会失败。YARN-7697: Fetching logs for finished application fails even though log aggregation is complete.

HDP 2.6.4 提供 Hadoop Common 2.7.3 和以下 Apache 修补程序:HDP 2.6.4 provided Hadoop Common 2.7.3 and the following Apache patches:

  • HADOOP-13700:从 TrashPolicy#initialize 和 #getInstance 签名中删除未引发的 IOException。HADOOP-13700: Remove unthrown IOException from TrashPolicy#initialize and #getInstance signatures.

  • HADOOP-13709:能够清理进程退出时由 Shell 生成的子进程。HADOOP-13709: Ability to clean up subprocesses spawned by Shell when the process exits.

  • HADOOP-14059:s3a rename(self, subdir) 错误消息中存在拼写错误。HADOOP-14059: typo in s3a rename(self, subdir) error message.

  • HADOOP-14542:添加接受 slf4j 记录器 API 的 IOUtils.cleanupWithLogger。HADOOP-14542: Add IOUtils.cleanupWithLogger that accepts slf4j logger API.

  • HDFS-9887:WebHdfs 套接字超时应可配置。HDFS-9887: WebHdfs socket timeouts should be configurable.

  • HDFS-9914:修复可配置的 WebhDFS 连接/读取超时。HDFS-9914: Fix configurable WebhDFS connect/read timeout.

  • MAPREDUCE-6698:增大 TestUnnecessaryBlockingOnHist oryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser 中的超时。MAPREDUCE-6698: Increase timeout on TestUnnecessaryBlockingOnHist oryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser.

  • YARN-4550:TestContainerLanch 中的某些测试在非英语区域设置环境中失败。YARN-4550: Some tests in TestContainerLanch fail on non-english locale environment.

  • YARN-4717:由于清理后发生 IllegalArgumentException,TestResourceLocalizationService.testPublicResourceInitializesLocalDir 间歇性失败。YARN-4717: TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup.

  • YARN-5042:将 /sys/fs/cgroup 作为只读装载项装入 Docker 容器。YARN-5042: Mount /sys/fs/cgroup into Docker containers as readonly mount.

  • YARN-5318:修复 TestRMAdminService#te stRefreshNodesResourceWithFileSystemBasedConfigurationProvider 的间歇性测试失败。YARN-5318: Fix intermittent test failure of TestRMAdminService#te stRefreshNodesResourceWithFileSystemBasedConfigurationProvider.

  • YARN-5641:容器完成后,本地化程序留下 tarball。YARN-5641: Localizer leaves behind tarballs after container is complete.

  • YARN-6004:重构 TestResourceLocalizationService#testDownloadingResourcesOnContainer,使其小于 150 行。YARN-6004: Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines.

  • YARN-6078:容器停滞在“正在本地化”状态。YARN-6078: Containers stuck in Localizing state.

  • YARN-6805:由于 null PrivilegedOperationException 退出代码,LinuxContainerExecutor 中出现 NPE。YARN-6805: NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code.

HBaseHBase

此版本提供 HBase 1.1.2 和以下 Apache 修补程序。This release provides HBase 1.1.2 and the following Apache patches.

  • HBASE-13376:Stochastic 负载均衡器改进。HBASE-13376: Improvements to Stochastic load balancer.

  • HBASE-13716:停止使用 Hadoop 的 FSConstants。HBASE-13716: Stop using Hadoop's FSConstants.

  • HBASE-13848:通过凭据提供程序 API 访问 InfoServer SSL 密码。HBASE-13848: Access InfoServer SSL passwords through Credential Provider API.

  • HBASE-13947:在 AssignmentManager 中使用 MasterServices 而不是 Server。HBASE-13947: Use MasterServices instead of Server in AssignmentManager.

  • HBASE-14135:HBase 备份/还原阶段 3:合并备份映像。HBASE-14135: HBase Backup/Restore Phase 3: Merge backup images.

  • HBASE-14473:并行计算区域位置。HBASE-14473: Compute region locality in parallel.

  • HBASE-14517:在主状态页中显示 regionserver 的版本。HBASE-14517: Show regionserver's version in master status page.

  • HBASE-14606:apache 上的主干版本中的 TestSecureLoadIncrementalHFiles 测试超时。HBASE-14606: TestSecureLoadIncrementalHFiles tests timed out in trunk build on apache.

  • HBASE-15210:撤消每毫秒记录数十个行的激进负载均衡器日志记录。HBASE-15210: Undo aggressive load balancer logging at tens of lines per millisecond.

  • HBASE-15515:改进均衡器中的 LocalityBasedCandidateGenerator。HBASE-15515: Improve LocalityBasedCandidateGenerator in Balancer.

  • HBASE-15615:RegionServerCallable 需要重试时休眠时间错误。HBASE-15615: Wrong sleep time when RegionServerCallable need retry.

  • HBASE-16135:已删除的对等方的 rs 下的 PeerClusterZnode 可能永远无法删除。HBASE-16135: PeerClusterZnode under rs of removed peer may never be deleted.

  • HBASE-16570:启动时并行计算区域位置。HBASE-16570: Compute region locality in parallel at startup.

  • HBASE-16810:当 regionservers 在 /hbase/draining znode 中且已卸载时,HBase 均衡器引发 ArrayIndexOutOfBoundsException。HBASE-16810: HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers are in /hbase/draining znode and unloaded.

  • HBASE-16852:TestDefaultCompactSelection 在 branch-1.3 上失败。HBASE-16852: TestDefaultCompactSelection failed on branch-1.3.

  • HBASE-17387:在 multi() 的 RegionActionResult 中减少异常报告的开销。HBASE-17387: Reduce the overhead of exception report in RegionActionResult for multi().

  • HBASE-17850:备份系统修复实用工具。HBASE-17850: Backup system repair utility.

  • HBASE-17931:将系统表分配到具有最高版本的服务器。HBASE-17931: Assign system tables to servers with highest version.

  • HBASE-18083:使大/小文件清理线程数在 HFileCleaner 中可配置。HBASE-18083: Make large/small file clean thread number configurable in HFileCleaner.

  • HBASE-18084:改进 CleanerChore,以便从消耗更多磁盘空间的目录中清理数据。HBASE-18084: Improve CleanerChore to clean from directory which consumes more disk space.

  • HBASE-18164:位置成本函数和候选生成器的速度快得多。HBASE-18164: Much faster locality cost function and candidate generator.

  • HBASE-18212:在独立模式下出现本地文件系统 HBase 日志警告消息:无法在类 org.apache.hadoop.fs.FSDataInputStream 中调用“unbuffer”方法。HBASE-18212: In Standalone mode with local filesystem HBase logs Warning message: Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream.

  • HBASE-18808:配置签入 BackupLogCleaner#getDeletableFiles() 的效率低下。HBASE-18808: Ineffective config check in BackupLogCleaner#getDeletableFiles().

  • HBASE-19052:FixedFileTrailer 应识别 branch-1.x 中的 CellComparatorImpl 类。HBASE-19052: FixedFileTrailer should recognize CellComparatorImpl class in branch-1.x.

  • HBASE-19065:HRegion#bulkLoadHFiles() 应等待并发 Region#flush() 完成。HBASE-19065: HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish.

  • HBASE-19285:添加每个表的延迟直方图。HBASE-19285: Add per-table latency histograms.

  • HBASE-19393:使用 SSL 访问 HBase UI 时的 HTTP 413 FULL 标头。HBASE-19393: HTTP 413 FULL head while accessing HBase UI using SSL.

  • HBASE-19395:[branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting 失败并出现 NPE。HBASE-19395: [branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting fails with NPE.

  • HBASE-19421:branch-1 不会针对 Hadoop 3.0.0 进行编译。HBASE-19421: branch-1 does not compile against Hadoop 3.0.0.

  • HBASE-19934:启用读取副本并在区域拆分后创建联机快照时,发生 HBaseSnapshotException。HBASE-19934: HBaseSnapshotException when read replicas is enabled and online snapshot is taken after region splitting.

  • HBASE-20008:拆分区域后还原快照时[向后移植] NullPointerException。HBASE-20008: [backport] NullPointerException when restoring a snapshot after splitting a region.

HiveHive

除以下修补程序以外,此版本还提供 Hive 1.2.1 和 Hive 2.1.0:This release provides Hive 1.2.1 and Hive 2.1.0 in addition to the following patches:

Hive 1.2.1 Apache 修补程序:Hive 1.2.1 Apache patches:

  • HIVE-10697:ObjectInspectorConvertors#UnionConvertor 执行错误的转换。HIVE-10697: ObjectInspectorConvertors#UnionConvertor does a faulty conversion.

  • HIVE-11266:基于外部表的表统计信息 count(*) 错误结果。HIVE-11266: count(*) wrong result based on table statistics for external tables.

  • HIVE-12245:支持基于 HBase 的表的列注释。HIVE-12245: Support column comments for an HBase backed table.

  • HIVE-12315:修复矢量化双精度除零。HIVE-12315: Fix Vectorized double divide by zero.

  • HIVE-12360:使用谓词下推在未压缩的 ORC 中进行错误查找。HIVE-12360: Bad seek in uncompressed ORC with predicate pushdown.

  • HIVE-12378:HBaseSerDe.serialize 二进制字段中出现异常。HIVE-12378: Exception on HBaseSerDe.serialize binary field.

  • HIVE-12785:包含联合类型和结构 UDF 的视图破坏。HIVE-12785: View with union type and UDF to the struct is broken.

  • HIVE-14013:描述表未正确显示 unicode。HIVE-14013: Describe table doesn't show unicode properly.

  • HIVE-14205:Hive 不支持 AVRO 文件格式的联合类型。HIVE-14205: Hive doesn't support union type with AVRO file format.

  • HIVE-14421:FS.deleteOnExit 包含对 _tmp_space.db 文件的引用。HIVE-14421: FS.deleteOnExit holds references to _tmp_space.db files.

  • HIVE-15563:忽略 SQLOperation.runQuery 中的非法操作状态转换异常,以公开实际异常。HIVE-15563: Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.

  • HIVE-15680:在 MR 模式下,当 hive.optimize.index.filter=true 且在查询中两次引用同一个 ORC 表时,出现错误的结果。HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query, in MR mode.

  • HIVE-15883:在 Hive 中的 HBase 映射表内插入十进制数失败。HIVE-15883: HBase mapped table in Hive insert fail for decimal.

  • HIVE-16232:QuotedIdentifier 中的列支持统计信息计算。HIVE-16232: Support stats computation for columns in QuotedIdentifier.

  • HIVE-16828:启用 CBO 后,基于分区视图进行查询会引发 IndexOutOfBoundException。HIVE-16828: With CBO enabled, Query on partitioned views throws IndexOutOfBoundException.

  • HIVE-17013:使用基于视图选择的子查询删除请求。HIVE-17013: Delete request with a subquery based on select over a view.

  • HIVE-17063:在首先删除分区的情况下,将覆盖分区插入外部表失败。HIVE-17063: insert overwrite partition onto an external table fail when drop partition first.

  • HIVE-17259:Hive JDBC 无法识别 UNIONTYPE 列。HIVE-17259: Hive JDBC does not recognize UNIONTYPE columns.

  • HIVE-17419:ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS 命令显示掩码表的计算统计信息。HIVE-17419: ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command shows computed stats for masked tables.

  • HIVE-17530:转换 uniontype 时发生 ClassCastException。HIVE-17530: ClassCastException when converting uniontype.

  • HIVE-17621:执行 HCatInputFormat 拆分计算期间忽略 Hive-site 设置。HIVE-17621: Hive-site settings are ignored during HCatInputFormat split-calculation.

  • HIVE-17636:为 Blob 存储添加 multiple_agg.q 测试。HIVE-17636: Add multiple_agg.q test for blobstores.

  • HIVE-17729:添加 Database 和 Explain 相关的 Blob 存储测试。HIVE-17729: Add Database and Explain related blobstore tests.

  • HIVE-17731:将面向外部用户的向后兼容选项添加到 HIVE-11985。HIVE-17731: add a backward compat option for external users to HIVE-11985.

  • HIVE-17803:使用 Pig 多重查询时,将 2 个 HCatStorer 写入同一个表会破坏彼此的输出。HIVE-17803: With Pig multi-query, 2 HCatStorers writing to the same table will trample each other's outputs.

  • HIVE-17829:ArrayIndexOutOfBoundsException - Hive2 中采用 Avro 架构的基于 HBASE 的表。HIVE-17829: ArrayIndexOutOfBoundsException - HBASE-backed tables with Avro schema in Hive2.

  • HIVE-17845:如果目标表列不是小写,插入将会失败。HIVE-17845: insert fails if target table columns are not lowercase.

  • HIVE-17900:分析压缩器触发的列中的统计信息生成包含多个分区列的、格式不当的 SQL。HIVE-17900: analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column.

  • HIVE-18026:Hive webhcat 主体配置优化。HIVE-18026: Hive webhcat principal configuration optimization.

  • HIVE-18031:支持“更改数据库”操作的复制。HIVE-18031: Support replication for Alter Database operation.

  • HIVE-18090:通过 hadoop 凭据连接元存储时,acid 检测信号失败。HIVE-18090: acid heartbeat fails when metastore is connected via hadoop credential.

  • HIVE-18189:将 hive.groupby.orderby.position.alias 设置为 true 时,Hive 查询返回错误结果。HIVE-18189: Hive query returning wrong results when set hive.groupby.orderby.position.alias to true.

  • HIVE-18258:矢量化:包含重复列的化简端 GROUP BY MERGEPARTIAL 已破坏。HIVE-18258: Vectorization: Reduce-Side GROUP BY MERGEPARTIAL with duplicate columns is broken.

  • HIVE-18293:Hive 无法压缩运行 HiveMetaStore 的标识不拥有的某个文件夹中的表。HIVE-18293: Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore.

  • HIVE-18327:删除 MiniHiveKdc 的不必要的 HiveConf 依赖项。HIVE-18327: Remove the unnecessary HiveConf dependency for MiniHiveKdc.

  • HIVE-18341:添加复制负载支持,以便能够使用相同的加密密钥添加 TDE 的“原始”命名空间。HIVE-18341: Add repl load support for adding "raw" namespace for TDE with same encryption keys.

  • HIVE-18352:引入执行 REPL DUMP 时的 METADATAONLY 选项,以便与其他工具集成。HIVE-18352: introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools.

  • HIVE-18353:CompactorMR 应调用 jobclient.close() 来触发清理(通过 Thejas Nair 的 Prabhu Joseph)。HIVE-18353: CompactorMR should call jobclient.close() to trigger cleanup (Prabhu Joseph via Thejas Nair).

  • HIVE-18390:查询 ColumnPruner 中的分区视图时发生 IndexOutOfBoundsException。HIVE-18390: IndexOutOfBoundsException when query a partitioned view in ColumnPruner.

  • HIVE-18429:压缩应处理不生成任何输出的情况。HIVE-18429: Compaction should handle a case when it produces no output.

  • HIVE-18447:JDBC:提供某种方式让 JDBC 用户通过连接字符串传递 Cookie 信息。HIVE-18447: JDBC: Provide a way for JDBC users to pass cookie info via connection string.

  • HIVE-18460:压缩器不会将表属性传递给 Orc 写入器。HIVE-18460: Compactor doesn't pass Table properties to the Orc writer.

  • HIVE-18467:支持整个仓库转储/加载 + 创建/删除数据库事件(Anishek Agarwal,由 Sankar Hariappan 审阅)。HIVE-18467: support whole warehouse dump / load + create/drop database events (Anishek Agarwal, reviewed by Sankar Hariappan).

  • HIVE-18551:矢量化:VectorMapOperator 尝试为 Hybrid Grace 写入过许多的矢量列。HIVE-18551: Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace.

  • HIVE-18587:插入 DML 事件可能尝试针对目录计算校验和。HIVE-18587: insert DML event may attempt to calculate a checksum on directories.

  • HIVE-18613:扩展 JsonSerDe 以支持 BINARY 类型。HIVE-18613: Extend JsonSerDe to support BINARY type.

  • HIVE-18626:复制负载“with”子句不会将配置传递给任务。HIVE-18626: Repl load "with" clause does not pass config to tasks.

  • HIVE-18660:PCR 无法区分分区和虚拟列。HIVE-18660: PCR doesn't distinguish between partition and virtual columns.

  • HIVE-18754:REPL STATUS 应支持“with”子句。HIVE-18754: REPL STATUS should support 'with' clause.

  • HIVE-18754:REPL STATUS 应支持“with”子句。HIVE-18754: REPL STATUS should support 'with' clause.

  • HIVE-18788:清理 JDBC PreparedStatement 中的输入。HIVE-18788: Clean up inputs in JDBC PreparedStatement.

  • HIVE-18794:复制负载“with”子句不会将配置传递给非分区表的任务。HIVE-18794: Repl load "with" clause does not pass config to tasks for non-partition tables.

  • HIVE-18808:统计信息更新失败时使压缩更加可靠。HIVE-18808: Make compaction more robust when stats update fails.

  • HIVE-18817:读取 ACID 表期间发生 ArrayIndexOutOfBounds 异常。HIVE-18817: ArrayIndexOutOfBounds exception during read of ACID table.

  • HIVE-18833:“作为 orcfile 插入目录时”自动合并失败。HIVE-18833: Auto Merge fails when "insert into directory as orcfile".

  • HIVE-18879:如果 xercesImpl.jar 在类路径中,需要能够禁止在 UDFXPathUtil 中使用嵌入式元素。HIVE-18879: Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath.

  • HIVE-18907:创建实用工具来解决 HIVE-18817 中的 acid 键索引问题。HIVE-18907: Create utility to fix acid key index issue from HIVE-18817.

Hive 1.2.0 Apache 修补程序Hive 2.1.0 Apache Patches:

  • HIVE-14013:描述表未正确显示 unicode。HIVE-14013: Describe table doesn't show unicode properly.

  • HIVE-14205:Hive 不支持 AVRO 文件格式的联合类型。HIVE-14205: Hive doesn't support union type with AVRO file format.

  • HIVE-15563:忽略 SQLOperation.runQuery 中的非法操作状态转换异常,以公开实际异常。HIVE-15563: Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.

  • HIVE-15680:在 MR 模式下,当 hive.optimize.index.filter=true 且在查询中两次引用同一个 ORC 表时,出现错误的结果。HIVE-15680: Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query, in MR mode.

  • HIVE-15883:在 Hive 中的 HBase 映射表内插入十进制数失败。HIVE-15883: HBase mapped table in Hive insert fail for decimal.

  • HIVE-16757:删除对已弃用 AbstractRelNode.getRows 的调用。HIVE-16757: Remove calls to deprecated AbstractRelNode.getRows.

  • HIVE-16828:启用 CBO 后,基于分区视图进行查询会引发 IndexOutOfBoundException。HIVE-16828: With CBO enabled, Query on partitioned views throws IndexOutOfBoundException.

  • HIVE-17063:在首先删除分区的情况下,将覆盖分区插入外部表失败。HIVE-17063: insert overwrite partition onto an external table fail when drop partition first.

  • HIVE-17259:Hive JDBC 无法识别 UNIONTYPE 列。HIVE-17259: Hive JDBC does not recognize UNIONTYPE columns.

  • HIVE-17530:转换 uniontype 时发生 ClassCastException。HIVE-17530: ClassCastException when converting uniontype.

  • HIVE-17600:使用户可设置 OrcFile 的 enforceBufferSize。HIVE-17600: Make OrcFile's enforceBufferSize user-settable.

  • HIVE-17601:改进 LlapServiceDriver 中的错误处理。HIVE-17601: improve error handling in LlapServiceDriver.

  • HIVE-17613:删除较短的相同线程分配的对象池。HIVE-17613: remove object pools for short, same-thread allocations.

  • HIVE-17617:空结果集汇总应包含空组集的分组。HIVE-17617: Rollup of an empty resultset should contain the grouping of the empty grouping set.

  • HIVE-17621:执行 HCatInputFormat 拆分计算期间忽略 Hive-site 设置。HIVE-17621: Hive-site settings are ignored during HCatInputFormat split-calculation.

  • HIVE-17629:CachedStore:提供允许列表/方块列表配置,以允许对表/分区进行选择性的缓存,并允许在预热时读取。HIVE-17629: CachedStore: Have a whitelist/blacklist config to allow selective caching of tables/partitions and allow read while prewarming.

  • HIVE-17636:为 Blob 存储添加 multiple_agg.q 测试。HIVE-17636: Add multiple_agg.q test for blobstores.

  • HIVE-17702:在 ORC 中的十进制读取器内进行错误的 isRepeating 处理。HIVE-17702: incorrect isRepeating handling in decimal reader in ORC.

  • HIVE-17729:添加 Database 和 Explain 相关的 Blob 存储测试。HIVE-17729: Add Database and Explain related blobstore tests.

  • HIVE-17731:将面向外部用户的向后兼容选项添加到 HIVE-11985。HIVE-17731: add a backward compat option for external users to HIVE-11985.

  • HIVE-17803:使用 Pig 多重查询时,将 2 个 HCatStorer 写入同一个表会破坏彼此的输出。HIVE-17803: With Pig multi-query, 2 HCatStorers writing to the same table will trample each other's outputs.

  • HIVE-17845:如果目标表列不是小写,插入将会失败。HIVE-17845: insert fails if target table columns are not lowercase.

  • HIVE-17900:分析压缩器触发的列中的统计信息生成包含多个分区列的、格式不当的 SQL。HIVE-17900: analyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column.

  • HIVE-18006:优化 HLLDenseRegister 的内存占用量。HIVE-18006: Optimize memory footprint of HLLDenseRegister.

  • HIVE-18026:Hive webhcat 主体配置优化。HIVE-18026: Hive webhcat principal configuration optimization.

  • HIVE-18031:支持“更改数据库”操作的复制。HIVE-18031: Support replication for Alter Database operation.

  • HIVE-18090:通过 hadoop 凭据连接元存储时,acid 检测信号失败。HIVE-18090: acid heartbeat fails when metastore is connected via hadoop credential.

  • HIVE-18189:禁用 cbo 时无法按位置排序。HIVE-18189: Order by position does not work when cbo is disabled.

  • HIVE-18258:矢量化:包含重复列的化简端 GROUP BY MERGEPARTIAL 已破坏。HIVE-18258: Vectorization: Reduce-Side GROUP BY MERGEPARTIAL with duplicate columns is broken.

  • HIVE-18269:LLAP:使用慢速处理管道进行快速 llap io 可能导致 OOM。HIVE-18269: LLAP: Fast llap io with slow processing pipeline can lead to OOM.

  • HIVE-18293:Hive 无法压缩运行 HiveMetaStore 的标识不拥有的某个文件夹中的表。HIVE-18293: Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore.

  • HIVE-18318:即使在未阻塞时,LLAP 记录读取器也应检查中断。HIVE-18318: LLAP record reader should check interrupt even when not blocking.

  • HIVE-18326:LLAP Tez 计划程序 - 仅当任务之间存在依赖关系时,才预先清空任务。HIVE-18326: LLAP Tez scheduler - only preempt tasks if there's a dependency between them.

  • HIVE-18327:删除 MiniHiveKdc 的不必要的 HiveConf 依赖项。HIVE-18327: Remove the unnecessary HiveConf dependency for MiniHiveKdc.

  • HIVE-18331:添加 TGT 过期时重新登录和一些日志记录/lambda。HIVE-18331: Add relogin when TGT expire and some logging/lambda.

  • HIVE-18341:添加复制负载支持,以便能够使用相同的加密密钥添加 TDE 的“原始”命名空间。HIVE-18341: Add repl load support for adding "raw" namespace for TDE with same encryption keys.

  • HIVE-18352:引入执行 REPL DUMP 时的 METADATAONLY 选项,以便与其他工具集成。HIVE-18352: introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools.

  • HIVE-18353:CompactorMR 应调用 jobclient.close() 来触发清理。HIVE-18353: CompactorMR should call jobclient.close() to trigger cleanup.

  • HIVE-18384:log4j2.x 库中发生 ConcurrentModificationException。HIVE-18384: ConcurrentModificationException in log4j2.x library.

  • HIVE-18390:查询 ColumnPruner 中的分区视图时发生 IndexOutOfBoundsException。HIVE-18390: IndexOutOfBoundsException when query a partitioned view in ColumnPruner.

  • HIVE-18447:JDBC:提供某种方式让 JDBC 用户通过连接字符串传递 Cookie 信息。HIVE-18447: JDBC: Provide a way for JDBC users to pass cookie info via connection string.

  • HIVE-18460:压缩器不会将表属性传递给 Orc 写入器。HIVE-18460: Compactor doesn't pass Table properties to the Orc writer.

  • HIVE-18462:(使用映射联接的查询的格式化解释包含 columnExprMap,其中包含格式不当的列名称)。HIVE-18462: (Explain formatted for queries with map join has columnExprMap with unformatted column name).

  • HIVE-18467:支持整个仓库转储/加载 + 创建/删除数据库事件。HIVE-18467: support whole warehouse dump / load + create/drop database events.

  • HIVE-18488:LLAP ORC 读取器缺少一些 null 检查。HIVE-18488: LLAP ORC readers are missing some null checks.

  • HIVE-18490:使用 EXISTS 和 NOT EXISTS 且包含 non-equi 谓词的查询可能生成错误结果。HIVE-18490: Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong result.

  • HIVE-18506:LlapBaseInputFormat - 负数组索引。HIVE-18506: LlapBaseInputFormat - negative array index.

  • HIVE-18517:矢量化:修复 VectorMapOperator,以接受 VRB 并正确检查矢量化标志来支持 LLAP 缓存。HIVE-18517: Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching).

  • HIVE-18523:在没有输入的情况下修复摘要行。HIVE-18523: Fix summary row in case there are no inputs.

  • HIVE-18528:聚合 ObjectStore 中的统计信息时收到错误的结果。HIVE-18528: Aggregate stats in ObjectStore get wrong result.

  • HIVE-18530:复制应暂时跳过 MM 表。HIVE-18530: Replication should skip MM table (for now).

  • HIVE-18548:修复 log4j 导入。HIVE-18548: Fix log4j import.

  • HIVE-18551:矢量化:VectorMapOperator 尝试为 Hybrid Grace 写入过许多的矢量列。HIVE-18551: Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace.

  • HIVE-18577:SemanticAnalyzer.validate 包含一些毫无意义的元存储调用。HIVE-18577: SemanticAnalyzer.validate has some pointless metastore calls.

  • HIVE-18587:插入 DML 事件可能尝试针对目录计算校验和。HIVE-18587: insert DML event may attempt to calculate a checksum on directories.

  • HIVE-18597:LLAP:始终打包 org.apache.log4j 的 log4j2 API jar。HIVE-18597: LLAP: Always package the log4j2 API jar for org.apache.log4j.

  • HIVE-18613:扩展 JsonSerDe 以支持 BINARY 类型。HIVE-18613: Extend JsonSerDe to support BINARY type.

  • HIVE-18626:复制负载“with”子句不会将配置传递给任务。HIVE-18626: Repl load "with" clause does not pass config to tasks.

  • HIVE-18643:不检查 ACID 操作的已存档分区。HIVE-18643: don't check for archived partitions for ACID ops.

  • HIVE-18660:PCR 无法区分分区和虚拟列。HIVE-18660: PCR doesn't distinguish between partition and virtual columns.

  • HIVE-18754:REPL STATUS 应支持“with”子句。HIVE-18754: REPL STATUS should support 'with' clause.

  • HIVE-18788:清理 JDBC PreparedStatement 中的输入。HIVE-18788: Clean up inputs in JDBC PreparedStatement.

  • HIVE-18794:复制负载“with”子句不会将配置传递给非分区表的任务。HIVE-18794: Repl load "with" clause does not pass config to tasks for non-partition tables.

  • HIVE-18808:统计信息更新失败时使压缩更加可靠。HIVE-18808: Make compaction more robust when stats update fails.

  • HIVE-18815:删除 HPL/SQL 中的未使用功能。HIVE-18815: Remove unused feature in HPL/SQL.

  • HIVE-18817:读取 ACID 表期间发生 ArrayIndexOutOfBounds 异常。HIVE-18817: ArrayIndexOutOfBounds exception during read of ACID table.

  • HIVE-18833:“作为 orcfile 插入目录时”自动合并失败。HIVE-18833: Auto Merge fails when "insert into directory as orcfile".

  • HIVE-18879:如果 xercesImpl.jar 在类路径中,需要能够禁止在 UDFXPathUtil 中使用嵌入式元素。HIVE-18879: Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath.

  • HIVE-18944:DPP 期间错误地设置了分组集位置。HIVE-18944: Grouping sets position is set incorrectly during DPP.

KafkaKafka

此版本提供 Kafka 1.0.0 和以下 Apache 修补程序。This release provides Kafka 1.0.0 and the following Apache patches.

  • KAFKA-4827:Kafka 连接:连接器名称中的特殊字符导致错误。KAFKA-4827: Kafka connect: error with special characters in connector name.

  • KAFKA-6118:kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials 中发生暂时性失败。KAFKA-6118: Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials.

  • KAFKA-6156:JmxReporter 无法处理 Windows 样式的目录路径。KAFKA-6156: JmxReporter can't handle windows style directory paths.

  • KAFKA-6164:如果在加载日志时遇到错误,ClientQuotaManager 线程会阻止关闭。KAFKA-6164: ClientQuotaManager threads prevent shutdown when encountering an error loading logs.

  • KAFKA-6167:streams 目录中的时间戳包含冒号,这是非法字符。KAFKA-6167: Timestamp on streams directory contains a colon, which is an illegal character.

  • KAFKA-6179:RecordQueue.clear() 不会清除 MinTimestampTracker 的维护列表。KAFKA-6179: RecordQueue.clear() does not clear MinTimestampTracker's maintained list.

  • KAFKA-6185:执行向下转换时,选择器内存泄漏并很有可能出现 OOM。KAFKA-6185: Selector memory leak with high likelihood of OOM in case of down conversion.

  • KAFKA-6190:使用事务消息时,GlobalKTable 永远不会完成还原。KAFKA-6190: GlobalKTable never finishes restoring when consuming transactional messages.

  • KAFKA-6210:如果将 1.0.0 用于 inter.broker.protocol.version 或 log.message.format.version,则会发生 IllegalArgumentException。KAFKA-6210: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version.

  • KAFKA-6214:结合内存中状态存储使用备用副本会导致流崩溃。KAFKA-6214: Using standby replicas with an in memory state store causes Streams to crash.

  • KAFKA-6215:KafkaStreamsTest 在主干中失败。KAFKA-6215: KafkaStreamsTest fails in trunk.

  • KAFKA-6238:应用滚动升级到 1.0.0 时,协议版本出现问题。KAFKA-6238: Issues with protocol version when applying a rolling upgrade to 1.0.0.

  • KAFKA-6260:AbstractCoordinator 不会明显处理 NULL 异常。KAFKA-6260: AbstractCoordinator not clearly handles NULL Exception.

  • KAFKA-6261:如果 ack=0,则请求日志记录会引发异常。KAFKA-6261: Request logging throws exception if acks=0.

  • KAFKA-6274:改善自动生成的 KTable 源状态存储名称。KAFKA-6274: Improve KTable Source state store auto-generated names.

MahoutMahout

HDP 2.3.x 和 2.4.x 未随附 Mahout 的特定 Apache 版本,而是同步到 Apache Mahout 主干中的特定修订版本点。In HDP-2.3.x and 2.4.x, instead of shipping a specific Apache release of Mahout, we synchronized to a particular revision point on Apache Mahout trunk. 此修订版本点在 0.9.0 版本之后、0.10.0 版本之前。This revision point is after the 0.9.0 release, but before the 0.10.0 release. 与 0.9.0 版本相比,此版本提供大量的 bug 修复和功能增强,并且在完成转换到 0.10.0 中基于 Spark 的 Mahout 之前,提供 Mahout 功能的稳定版本。This provides a large number of bug fixes and functional enhancements over the 0.9.0 release, but provides a stable release of the Mahout functionality before the complete conversion to new Spark-based Mahout in 0.10.0.

为 HDP 2.3.x 和 2.4.x 中的 Mahout 选择的修订版本点在截至 2014 年 12 月 19 日,来自 GitHub 中 Apache Mahout 的“mahout-0.10.x”分支的 0f037cb03e77c096。The revision point chosen for Mahout in HDP 2.3.x and 2.4.x is from the "mahout-0.10.x" branch of Apache Mahout, as of 19 December 2014, revision 0f037cb03e77c096 in GitHub.

在 HDP 2.5.x 和 2.6.x 中,我们从 Mahout 中删除了“commons-httpclient”库(因为我们认为它是具有潜在安全问题的已过时库),并已将 Mahout 中的 Hadoop 客户端升级到了版本 2.7.3(HDP-2.5 中使用的相同版本)。In HDP-2.5.x and 2.6.x, we removed the "commons-httpclient" library from Mahout because we view it as an obsolete library with possible security issues, and upgraded the Hadoop-Client in Mahout to version 2.7.3, the same version used in HDP-2.5. 因此:As a result:

  • 以前编译的 Mahout 作业需要在 HDP-2.5 或 2.6 环境中重新编译。Previously compiled Mahout jobs will need to be recompiled in the HDP-2.5 or 2.6 environment.

  • 还有一种较小的可能性:某些 Mahout 作业可能会遇到与“org.apache.commons.httpclient”、“net.java.dev.jets3t”或类名前缀相关的“ClassNotFoundException”或“无法加载类”错误。There is a small possibility that some Mahout jobs may encounter "ClassNotFoundException" or "could not load class" errors related to "org.apache.commons.httpclient", "net.java.dev.jets3t", or related class name prefixes. 如果环境中可以接受已过时库中安全问题带来的风险,在发生这些错误时,可以考虑是否在作业的类路径中手动安装所需的 jar。If these errors happen, you may consider whether to manually install the needed jars in your classpath for the job, if the risk of security issues in the obsolete library is acceptable in your environment.

  • 还存在一种更小的可能性:由于二进制兼容问题,在 Mahout 对 hadoop-common 库的 hbase-client 代码调用中,某些 Mahout 作业可能遇到崩溃。There is an even smaller possibility that some Mahout jobs may encounter crashes in Mahout's hbase-client code calls to the hadoop-common libraries, due to binary compatibility problems. 遗憾的是,没有任何方法可以解决此问题,而只能还原到 Mahout 的 HDP-2.4.2 版本,这可能会造成安全问题。Regrettably, there is no way to resolve this issue except revert to the HDP-2.4.2 version of Mahout, which may have security issues. 同样,这种情况极少见,在任何给定的 Mahout 作业套件中都不太可能会发生。Again, this should be very unusual, and is unlikely to occur in any given Mahout job suite.

OozieOozie

此版本提供 Oozie 4.2.0 和以下 Apache 修补程序。This release provides Oozie 4.2.0 with the following Apache patches.

  • OOZIE-2571:添加 spark.scala.binary.version Maven 属性,以便可以使用 Scala 2.11。OOZIE-2571: Add spark.scala.binary.version Maven property so that Scala 2.11 can be used.

  • OOZIE-2606:设置 spark.yarn.jars 以修复包含 Oozie 的 Spark 2.0。OOZIE-2606: Set spark.yarn.jars to fix Spark 2.0 with Oozie.

  • OOZIE-2658:--driver-class-path 可以覆盖 SparkMain 中的类路径。OOZIE-2658: --driver-class-path can overwrite the classpath in SparkMain.

  • OOZIE-2787:Oozie 分配应用程序 jar 两次,导致 spark 作业失败。OOZIE-2787: Oozie distributes application jar twice making the spark job fail.

  • OOZIE-2792:当 Hive 位于 Spark 中时,Hive2 操作不会正确分析日志文件中的 Spark 应用程序 ID。OOZIE-2792: Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark.

  • OOZIE-2799:为 hive 上的 spark sql 设置日志位置。OOZIE-2799: Setting log location for spark sql on hive.

  • OOZIE-2802:由于重复的 sharelibs,Spark 操作在 Spark 2.1.0 上失败。OOZIE-2802: Spark action failure on Spark 2.1.0 due to duplicate sharelibs.

  • OOZIE-2923:改善 Spark 选项分析。OOZIE-2923: Improve Spark options parsing.

  • OOZIE-3109:SCA:跨站点脚本:已反映。OOZIE-3109: SCA: Cross-Site Scripting: Reflected.

  • OOZIE-3139:Oozie 不正确地验证工作流。OOZIE-3139: Oozie validates workflow incorrectly.

  • OOZIE-3167:升级 Oozie 4.3 分支上的 tomcat 版本。OOZIE-3167: Upgrade tomcat version on Oozie 4.3 branch.

PhoenixPhoenix

此版本提供 Phoenix 4.7.0 和以下 Apache 修补程序:This release provides Phoenix 4.7.0 and the following Apache patches:

  • PHOENIX-1751:在 preScannerNext 而不是 postScannerOpen 中执行聚合、排序,等等。PHOENIX-1751: Perform aggregations, sorting, etc., in the preScannerNext instead of postScannerOpen.

  • PHOENIX-2714:在 BaseResultIterators 中正确估计字节并公开为接口。PHOENIX-2714: Correct byte estimate in BaseResultIterators and expose as interface.

  • PHOENIX-2724:无统计信息相比,具有大量 guideposts 的查询速度更慢。PHOENIX-2724: Query with large number of guideposts is slower compared to no stats.

  • PHOENIX-2855:不会对 HBase 1.2 序列化的递增 TimeRange 的解决方法。PHOENIX-2855: Workaround Increment TimeRange not being serialized for HBase 1.2.

  • PHOENIX-3023:按默认方式并行执行限制查询时性能降低。PHOENIX-3023: Slow performance when limit queries are executed in parallel by default.

  • PHOENIX-3040:不要使用 guideposts 串行执行查询。PHOENIX-3040: Don't use guideposts for executing queries serially.

  • PHOENIX-3112:无法正确处理部分行扫描。PHOENIX-3112: Partial row scan not handled correctly.

  • PHOENIX-3240:Pig 加载程序中出现 ClassCastException。PHOENIX-3240: ClassCastException from Pig loader.

  • PHOENIX-3452:NULLS FIRST/NULL LAST 不应影响 GROUP BY 是否为顺序保留。PHOENIX-3452: NULLS FIRST/NULL LAST should not impact whether GROUP BY is order preserving.

  • PHOENIX-3469:NULLS LAST/NULLS FIRST 的 DESC 主键的排序顺序错误。PHOENIX-3469: Incorrect sort order for DESC primary key for NULLS LAST/NULLS FIRST.

  • PHOENIX-3789:在 postBatchMutateIndispensably 中执行跨区域索引维护调用。PHOENIX-3789: Execute cross region index maintenance calls in postBatchMutateIndispensably.

  • PHOENIX-3865:未针对第一个列系列筛选时,IS NULL 不会返回正确的结果。PHOENIX-3865: IS NULL does not return correct results when first column family not filtered against.

  • PHOENIX-4290:使用包含不可变索引的表对 DELETE 执行全表扫描。PHOENIX-4290: Full table scan performed for DELETE with table having immutable indexes.

  • PHOENIX-4373:执行更新插入时,本地索引变量长度键可能包含尾随的 null 值。PHOENIX-4373: Local index variable length key can have trailing nulls while upserting.

  • PHOENIX-4466:java.lang.RuntimeException:响应代码 500 - 执行 spark 作业以连接到 phoenix 查询服务器并加载数据。PHOENIX-4466: java.lang.RuntimeException: response code 500 - Executing a spark job to connect to phoenix query server and load data.

  • PHOENIX-4489:Phoenix MR 作业中的 HBase 连接泄漏。PHOENIX-4489: HBase Connection leak in Phoenix MR Jobs.

  • PHOENIX-4525:GroupBy 执行中的整数溢出。PHOENIX-4525: Integer overflow in GroupBy execution.

  • PHOENIX-4560:如果 pk 列中包含 WHERE, ORDER BY 和 GROUP BY 不起作用。PHOENIX-4560: ORDER BY with GROUP BY doesn't work if there is WHERE on pk column.

  • PHOENIX-4586:UPSERT SELECT 不会考虑子查询的比较运算符。PHOENIX-4586: UPSERT SELECT doesn't take in account comparison operators for subqueries.

  • PHOENIX-4588:如果表达式的子级包含 Determinism.PER_INVOCATION,则也会克隆表达式。PHOENIX-4588: Clone expression also if its children have Determinism.PER_INVOCATION.

PigPig

此版本提供 Pig 0.16.0 和以下 Apache 修补程序。This release provides Pig 0.16.0 with the following Apache patches.

  • PIG-5159:修复 Pig 不保存 grunt 历史记录的问题。PIG-5159: Fix Pig not saving grunt history.

  • PIG-5175:将 jruby 升级到 1.7.26。PIG-5175: Upgrade jruby to 1.7.26.

RangerRanger

此版本提供 Ranger 0.7.0 和以下 Apache 修补程序:This release provides Ranger 0.7.0 and the following Apache patches:

  • RANGER-1805:遵循 js 中的最佳做法改善代码。RANGER-1805: Code improvement to follow best practices in js.

  • RANGER-1960:考虑删除快照的表名称。RANGER-1960: Take snapshot's table name into consideration for deletion.

  • RANGER-1982:改善 Ranger Admin 和 Ranger KMS 分析指标的错误。RANGER-1982: Error Improvement for Analytics Metric of Ranger Admin and Ranger KMS.

  • RANGER-1984:Hbase 审核日志记录无法显示与访问过的列相关联的所有标记。RANGER-1984: Hbase audit log records may not show all tags associated with accessed column.

  • RANGER-1988:修复不安全的随机性。RANGER-1988: Fix insecure randomness.

  • RANGER-1990:在 Ranger Admin 中添加单向 SSL MySQL 支持。RANGER-1990: Add One-way SSL MySQL support in Ranger Admin.

  • RANGER-2006:修复静态代码分析在 ldap 同步源的 ranger usersync 中检测到的问题。RANGER-2006: Fix problems detected by static code analysis in ranger usersync for ldap sync source.

  • RANGER-2008:策略评估对于多行策略条件失败。RANGER-2008: Policy evaluation is failing for multiline policy conditions.

滑块Slider

此版本提供了 Slider 0.92.0,但未提供其他 Apache 修补程序。This release provides Slider 0.92.0 with no additional Apache patches.

SparkSpark

此版本提供 Spark 2.3.0 和以下 Apache 修补程序:This release provides Spark 2.3.0 and the following Apache patches:

  • SPARK-13587:支持 pyspark 中的 virtualenv。SPARK-13587: Support virtualenv in pyspark.

  • SPARK-19964:避免从 SparkSubmitSuite 中的远程存储库读取。SPARK-19964: Avoid reading from remote repos in SparkSubmitSuite.

  • SPARK-22882:结构化流的机器学习测试:ml.classification。SPARK-22882: ML test for structured streaming: ml.classification.

  • SPARK-22915:spark.ml.feature 的从 N 到 Z 的流测试。SPARK-22915: Streaming tests for spark.ml.feature, from N to Z.

  • SPARK-23020:在进程内启动程序测试中修复另一种争用情况。SPARK-23020: Fix another race in the in-process launcher test.

  • SPARK-23040:返回随机读取器的可中断迭代器。SPARK-23040: Returns interruptible iterator for shuffle reader.

  • SPARK-23173:从 JSON 加载数据时避免创建损坏的 parquet 文件。SPARK-23173: Avoid creating corrupt parquet files when loading data from JSON.

  • SPARK-23264:修复 literals.sql.out 中的 scala.MatchError。SPARK-23264: Fix scala.MatchError in literals.sql.out.

  • SPARK-23288:修正 parquet 接收器的输出指标。SPARK-23288: Fix output metrics with parquet sink.

  • SPARK-23329:修正三角函数的文档。SPARK-23329: Fix documentation of trigonometric functions.

  • SPARK-23406:为 branch-2.3 启用流到流的自联接。SPARK-23406: Enable stream-stream self-joins for branch-2.3.

  • SPARK-23434:Spark 不应针对 HDFS 文件路径的`元数据目录`发出警告。SPARK-23434: Spark should not warn `metadata directory` for a HDFS file path.

  • SPARK-23436:仅当分区可以强制转换为日期时才将其推断为日期。SPARK-23436: Infer partition as Date only if it can be cast to Date.

  • SPARK-23457:首先在 ParquetFileFormat 中注册任务完成侦听器。SPARK-23457: Register task completion listeners first in ParquetFileFormat.

  • SPARK-23462:改善 `StructType` 中的缺少字段错误消息。SPARK-23462: improve missing field error message in `StructType`.

  • SPARK-23490:在 CreateTable 中检查包含现有表的 storage.locationUri。SPARK-23490: Check storage.locationUri with existing table in CreateTable.

  • SPARK-23524:不应检查大型本地随机块是否已损坏。SPARK-23524: Big local shuffle blocks should not be checked for corruption.

  • SPARK-23525:支持外部 hive 表的 ALTER TABLE CHANGE COLUMN COMMENT。SPARK-23525: Support ALTER TABLE CHANGE COLUMN COMMENT for external hive table.

  • SPARK-23553:测试不应假设 `spark.sql.sources.default` 的默认值。SPARK-23553: Tests should not assume the default value of `spark.sql.sources.default`.

  • SPARK-23569:允许 pandas_udf 使用 python3 样式的类型批注函数。SPARK-23569: Allow pandas_udf to work with python3 style type-annotated functions.

  • SPARK-23570:在 HiveExternalCatalogVersionsSuite 中添加 Spark 2.3.0。SPARK-23570: Add Spark 2.3.0 in HiveExternalCatalogVersionsSuite.

  • SPARK-23598:使 BufferedRowIterator 中的方法成为公共方法,以避免大型查询出现运行时错误。SPARK-23598: Make methods in BufferedRowIterator public to avoid runtime error for a large query.

  • SPARK-23599:从伪随机数添加 UUID 生成器。SPARK-23599: Add a UUID generator from Pseudo-Random Numbers.

  • SPARK-23599:在 Uuid 表达式中使用 RandomUUIDGenerator。SPARK-23599: Use RandomUUIDGenerator in Uuid expression.

  • SPARK-23601:从版本中删除 .md5 文件。SPARK-23601: Remove .md5 files from release.

  • SPARK-23608:在 attachSparkUI 和 detachSparkUI 函数之间添加 SHS 同步,以避免 Jetty 处理程序的并发修改问题。SPARK-23608: Add synchronization in SHS between attachSparkUI and detachSparkUI functions to avoid concurrent modification issue to Jetty Handlers.

  • SPARK-23614:修复使用缓存时错误地重复使用交换的问题。SPARK-23614: Fix incorrect reuse exchange when caching is used.

  • SPARK-23623:避免在 CachedKafkaConsumer (branch-2.3) 中并发使用缓存的使用者。SPARK-23623: Avoid concurrent use of cached consumers in CachedKafkaConsumer (branch-2.3).

  • SPARK-23624:在 Datasource V2 中修订方法 pushFilters 的文档。SPARK-23624: Revise doc of method pushFilters in Datasource V2.

  • SPARK-23628:calculateParamLength 不应返回 1 + 表达式数目。SPARK-23628: calculateParamLength should not return 1 + num of expressions.

  • SPARK-23630:允许用户的 hadoop 配置自定义项生效。SPARK-23630: Allow user's hadoop conf customizations to take effect.

  • SPARK-23635:Spark 执行器环境变量由同名的 AM 环境变量覆盖。SPARK-23635: Spark executor env variable is overwritten by same name AM env variable.

  • SPARK-23637:如果多次终止同一个执行器,Yarn 可能会分配更多的资源。SPARK-23637: Yarn might allocate more resource if a same executor is killed multiple times.

  • SPARK-23639:在初始化 SparkSQL CLI 中的元存储客户端之前获取令牌。SPARK-23639: Obtain token before init metastore client in SparkSQL CLI.

  • SPARK-23642:修复 AccumulatorV2 子类 isZero scaladoc。SPARK-23642: AccumulatorV2 subclass isZero scaladoc fix.

  • SPARK-23644:对 SHS 中的 REST 调用使用绝对路径。SPARK-23644: Use absolute path for REST call in SHS.

  • SPARK-23645:添加文档 RE `pandas_udf` 和关键字参数。SPARK-23645: Add docs RE `pandas_udf` with keyword args.

  • SPARK-23649:跳过 UTF-8 中禁止的字符。SPARK-23649: Skipping chars disallowed in UTF-8.

  • SPARK-23658:InProcessAppHandle 在 getLogger 中使用错误的类。SPARK-23658: InProcessAppHandle uses the wrong class in getLogger.

  • SPARK-23660:修复应用程序快速结束时 yarn 群集模式的异常。SPARK-23660: Fix exception in yarn cluster mode when application ended fast.

  • SPARK-23670:修复 SparkPlanGraphWrapper 上的内存泄漏。SPARK-23670: Fix memory leak on SparkPlanGraphWrapper.

  • SPARK-23671:修复状态以启用 SHS 线程池。SPARK-23671: Fix condition to enable the SHS thread pool.

  • SPARK-23691:尽量在 PySpark 测试中使用 sql_conf 实用工具。SPARK-23691: Use sql_conf util in PySpark tests where possible.

  • SPARK-23695:修正 Kinesis 流测试的错误消息。SPARK-23695: Fix the error message for Kinesis streaming tests.

  • SPARK-23706:spark.conf.get(value, default=None) 应在 PySpark 中生成 None。SPARK-23706: spark.conf.get(value, default=None) should produce None in PySpark.

  • SPARK-23728:修复运行流测试时机器学习测试出现预期异常的问题。SPARK-23728: Fix ML tests with expected exceptions running streaming tests.

  • SPARK-23729:解析 glob 时遵循 URI 分段。SPARK-23729: Respect URI fragment when resolving globs.

  • SPARK-23759:无法将 Spark UI 绑定到特定的主机名/IP。SPARK-23759: Unable to bind Spark UI to specific host name / IP.

  • SPARK-23760:CodegenContext.withSubExprEliminationExprs 应正确保存/还原 CSE 状态。SPARK-23760: CodegenContext.withSubExprEliminationExprs should save/restore CSE state correctly.

  • SPARK-23769:删除不必要地禁用 Scalastyle 检查的注释。SPARK-23769: Remove comments that unnecessarily disable Scalastyle check.

  • SPARK-23788:修复 StreamingQuerySuite 中的争用问题。SPARK-23788: Fix race in StreamingQuerySuite.

  • SPARK-23802:PropagateEmptyRelation 可能在未解决的状态下退出查询计划。SPARK-23802: PropagateEmptyRelation can leave query plan in unresolved state.

  • SPARK-23806:与动态分配配合使用时,Broadcast.unpersist 可能导致严重异常。SPARK-23806: Broadcast.unpersist can cause fatal exception when used with dynamic allocation.

  • SPARK-23808:在仅限测试的 spark 会话中设置默认的 Spark 会话。SPARK-23808: Set default Spark session in test-only spark sessions.

  • SPARK-23809:Active SparkSession 应由 getOrCreate 设置。SPARK-23809: Active SparkSession should be set by getOrCreate.

  • SPARK-23816:已终止的任务应忽略 FetchFailures。SPARK-23816: Killed tasks should ignore FetchFailures.

  • SPARK-23822:改善有关 Parquet 架构不匹配的错误消息。SPARK-23822: Improve error message for Parquet schema mismatches.

  • SPARK-23823:在 transformExpression 中保留来源。SPARK-23823: Keep origin in transformExpression.

  • SPARK-23827:StreamingJoinExec 应确保将输入数据分区成特定数目的分区。SPARK-23827: StreamingJoinExec should ensure that input data is partitioned into specific number of partitions.

  • SPARK-23838:正在运行的 SQL 查询在 SQL 选项卡中显示为“已完成”。SPARK-23838: Running SQL query is displayed as "completed" in SQL tab.

  • SPARK-23881:修复测试 JobCancellationSuite 中出现的怪异消息“随机读取器的不间断迭代器”。SPARK-23881: Fix flaky test JobCancellationSuite."interruptible iterator of shuffle reader".

SqoopSqoop

此版本提供了 Sqoop 1.4.6,但未提供其他 Apache 修补程序。This release provides Sqoop 1.4.6 with no additional Apache patches.

StormStorm

此版本提供 Storm 1.1.1 和以下 Apache 修补程序:This release provides Storm 1.1.1 and the following Apache patches:

  • STORM-2652:JmsSpout 打开方法中引发异常。STORM-2652: Exception thrown in JmsSpout open method.

  • STORM-2841:testNoAcksIfFlushFails UT 失败并出现 NullPointerException。STORM-2841: testNoAcksIfFlushFails UT fails with NullPointerException.

  • STORM-2854:公开 IEventLogger,使事件日志记录可插入。STORM-2854: Expose IEventLogger to make event logging pluggable.

  • STORM-2870:FileBasedEventLogger 泄漏非守护程序 ExecutorService,从而阻止进程完成。STORM-2870: FileBasedEventLogger leaks non-daemon ExecutorService which prevents process to be finished.

  • STORM-2960:更好地强调为 Storm 进程设置正确 OS 帐户的重要性。STORM-2960: Better to stress importance of setting up proper OS account for Storm processes.

TezTez

此版本提供 Tez 0.7.0 和以下 Apache 修补程序:This release provides Tez 0.7.0 and the following Apache patches:

  • TEZ-1526:运行大型作业时,TezTaskID 的 LoadingCache 速度较慢。TEZ-1526: LoadingCache for TezTaskID slow for large jobs.

ZeppelinZeppelin

此版本提供了 Zeppelin 0.7.3,但未提供其他 Apache 修补程序。This release provides Zeppelin 0.7.3 with no additionalApache patches.

ZooKeeperZooKeeper

此版本提供 ZooKeeper 3.4.6 和以下 Apache 修补程序:This release provides ZooKeeper 3.4.6 and the following Apache patches:

  • ZOOKEEPER-1256:ClientPortBindTest 在 Mac OS X 上失败。ZOOKEEPER-1256: ClientPortBindTest is failing on Mac OS X.

  • ZOOKEEPER-1901:[JDK8] 将子级排序,以便在 AsyncOps 测试中进行比较。ZOOKEEPER-1901: [JDK8] Sort children for comparison in AsyncOps tests.

  • ZOOKEEPER-2423:由于安全漏洞而升级 Netty 版本 (CVE-2014-3488)。ZOOKEEPER-2423: Upgrade Netty version due to security vulnerability (CVE-2014-3488).

  • ZOOKEEPER-2693:针对 wchp/wchc 四字母单词展开 DOS 攻击 (4lw)。ZOOKEEPER-2693: DOS attack on wchp/wchc four letter words (4lw).

  • ZOOKEEPER-2726:造成潜在争用条件的问题的修补程序。ZOOKEEPER-2726: Patch for introduces potential race condition.

修复了常见漏洞和透露Fixed Common Vulnerabilities and Exposures

本部分介绍此版本中已解决的所有常见漏洞和透露 (CVE) 问题。This section covers all Common Vulnerabilities and Exposures (CVE) that are addressed in this release.

CVE-2017-7676CVE-2017-7676

摘要:  Apache Ranger 策略评估忽略“*”通配符后面的字符Summary: Apache Ranger policy evaluation ignores characters after ‘*’ wildcard character
严重性:  关键Severity: Critical
供应商:  HortonworksVendor: Hortonworks
受影响的版本:  HDInsight 3.6 版本,包括 Apache Ranger 版本 0.5.x/0.6.x/0.7.0Versions Affected: HDInsight 3.6 versions including Apache Ranger versions 0.5.x/0.6.x/0.7.0
受影响的用户:  使用在“*”通配符后面包含字符的 Ranger 策略的环境 – 如 my*test、test*.txtUsers affected: Environments that use Ranger policies with characters after ‘*’ wildcard character – like my*test, test*.txt
影响:  策略资源匹配程序将忽略“*”通配符后面的字符,从而可能导致意外的行为。Impact: Policy resource matcher ignores characters after ‘*’ wildcard character, which can result in unintended behavior.
修复详细信息:  Ranger 策略资源匹配程序已更新,可以正确处理通配符匹配。Fix detail: Ranger policy resource matcher was updated to correctly handle wildcard matches.
建议的操作:  升级到 HDI 3.6(使用 Apache Ranger 0.7.1+)。Recommended Action: Upgrade to HDI 3.6 (with Apache Ranger 0.7.1+).

CVE-2017-7677CVE-2017-7677

摘要:  指定了外部位置时,Apache Ranger Hive 授权者应检查 RWX 权限Summary: Apache Ranger Hive Authorizer should check for RWX permission when external location is specified
严重性:  关键Severity: Critical
供应商:  HortonworksVendor: Hortonworks
受影响的版本:  HDInsight 3.6 版本,包括 Apache Ranger 版本 0.5.x/0.6.x/0.7.0Versions Affected: HDInsight 3.6 versions including Apache Ranger versions 0.5.x/0.6.x/0.7.0
受影响的用户:  对 hive 表使用外部位置的环境Users affected: Environments that use external location for hive tables
影响:  在对 hive 表使用外部位置的环境中,Apache Ranger Hive 授权者应检查指定用于创建表的外部位置的 RWX 权限。Impact: In environments that use external location for hive tables, Apache Ranger Hive Authorizer should check for RWX permission for the external location specified for create table.
修复详细信息:  Ranger Hive 授权者已更新,可以使用外部位置正确处理权限检查。Fix detail: Ranger Hive Authorizer was updated to correctly handle permission check with external location.
建议的操作:  用户应升级到 HDI 3.6(使用 Apache Ranger 0.7.1+)。Recommended Action: Users should upgrade to HDI 3.6 (with Apache Ranger 0.7.1+).

CVE-2017-9799CVE-2017-9799

摘要:  可能以 Apache Storm 中错误用户的身份执行代码Summary: Potential execution of code as the wrong user in Apache Storm
严重性:  重要Severity: Important
供应商:  HortonworksVendor: Hortonworks
受影响的版本:  HDP-2.4.0、HDP-2.5.0、HDP-2.6.0Versions Affected: HDP 2.4.0, HDP-2.5.0, HDP-2.6.0
受影响的用户:  在安全模式下使用 Storm,并使用 Blob 存储来分发基于拓扑的项目或使用 Blob 存储来分发任何拓扑资源的用户。Users affected: Users who use Storm in secure mode and are using blobstore to distribute topology based artifacts or using the blobstore to distribute any topology resources.
影响:  在某些情况下使用 storm 的配置时,在理论上,拓扑的所有者能够以不同的非 root 用户身份欺骗监督程序来启动辅助角色。Impact: Under some situations and configurations of storm it is theoretically possible for the owner of a topology to trick the supervisor to launch a worker as a different, non-root, user. 在最坏的情况下,这可能导致其他用户的安全凭据泄密。In the worst case, this could lead to secure credentials of the other user being compromised. 此漏洞仅适用于已启用安全性的 Apache Storm 安装。This vulnerability only applies to Apache Storm installations with security enabled.
缓解措施:  目前没有解决方法,只能升级到 HDP 2.6.2.1。Mitigation: Upgrade to HDP-2.6.2.1 as there are currently no workarounds.

CVE-2016-4970CVE-2016-4970

摘要:  4.0.37.Final 之前的 Netty 4.0.x 以及 4.1.1.Final 之前的 4.1.x 中的 handler/ssl/OpenSslEngine.java 允许远程攻击者造成拒绝服务(无限循环)Summary: handler/ssl/OpenSslEngine.java in Netty 4.0.x before 4.0.37.Final and 4.1.x before 4.1.1.Final allows remote attackers to cause a denial of service (infinite loop)
严重性: 中等Severity: Moderate
供应商:  HortonworksVendor: Hortonworks
受影响的版本:  从 2.3.x 开始的 HDP 2.x.xVersions Affected: HDP 2.x.x since 2.3.x
受影响的用户:  使用 HDFS 的所有用户。Users Affected: All users that use HDFS.
影响:  影响较低,因为 Hortonworks 不直接在 Hadoop 代码库中使用 OpenSslEngine.java。Impact: Impact is low as Hortonworks does not use OpenSslEngine.java directly in Hadoop codebase.
建议的操作:  升级到 HDP 2.6.3。Recommended Action: Upgrade to HDP 2.6.3.

CVE-2016-8746CVE-2016-8746

摘要:  策略评估中存在 Apache Ranger 路径匹配问题Summary: Apache Ranger path matching issue in policy evaluation
严重性:  普通Severity: Normal
供应商: HortonworksVendor: Hortonworks
受影响的版本:  所有 HDP 2.5 版本,包括 Apache Ranger 版本 0.6.0/0.6.1/0.6.2Versions Affected: All HDP 2.5 versions including Apache Ranger versions 0.6.0/0.6.1/0.6.2
受影响的用户:  Ranger 策略管理工具的所有用户。Users affected: All users of the ranger policy admin tool.
影响:  在某些情况下,当策略包含通配符和递归标志时,Ranger 策略引擎不会正确匹配路径。Impact: Ranger policy engine incorrectly matches paths in certain conditions when a policy contains wildcards and recursive flags.
修复详细信息: 已修复策略评估逻辑Fix detail: Fixed policy evaluation logic
建议的操作:  用户应升级到 HDP 2.5.4+(使用 Apache Ranger 0.6.3+)或 HDP 2.6+(使用 Apache Ranger 0.7.0+)Recommended Action: Users should upgrade to HDP 2.5.4+ (with Apache Ranger 0.6.3+) or HDP 2.6+ (with Apache Ranger 0.7.0+)

CVE-2016-8751CVE-2016-8751

摘要:  Apache Ranger 存储跨站点脚本问题Summary: Apache Ranger stored cross site scripting issue
严重性:  普通Severity: Normal
供应商:  HortonworksVendor: Hortonworks
受影响的版本:  所有 HDP 2.3/2.4/2.5 版本,包括 Apache Ranger 版本 0.5.x/0.6.0/0.6.1/0.6.2Versions Affected: All HDP 2.3/2.4/2.5 versions including Apache Ranger versions 0.5.x/0.6.0/0.6.1/0.6.2
受影响的用户:  Ranger 策略管理工具的所有用户。Users affected: All users of the ranger policy admin tool.
影响:  进入自定义的策略条件时,Apache Ranger 容易受到存储跨站点脚本攻击。Impact: Apache Ranger is vulnerable to a Stored Cross-Site Scripting when entering custom policy conditions. 当普通用户登录和访问策略时,管理员用户可能会存储一些任意 javascript 代码执行。Admin users can store some arbitrary javascript code execute when normal users sign in and access policies.
修复详细信息:  添加了逻辑来清理用户输入。Fix detail: Added logic to sanitize the user input.
建议的操作:  用户应升级到 HDP 2.5.4+(使用 Apache Ranger 0.6.3+)或 HDP 2.6+(使用 Apache Ranger 0.7.0+)Recommended Action: Users should upgrade to HDP 2.5.4+ (with Apache Ranger 0.6.3+) or HDP 2.6+ (with Apache Ranger 0.7.0+)

修复了支持问题Fixed issues for support

修复的问题代表以前通过 Hortonworks 支持记录的、但现已在当前版本中解决的选定问题。Fixed issues represent selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. 这些问题可能已在先前版本的“已知问题”部分中报告;这意味着,这些问题已由客户报告,或者由 Hortonworks 质量工程团队识别。These issues may have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

不正确的结果Incorrect Results

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100019BUG-100019 YARN-8145YARN-8145 yarn rmadmin -getGroups 不返回已更新的用户组yarn rmadmin -getGroups doesn't return updated groups for user
BUG-100058BUG-100058 PHOENIX-2645PHOENIX-2645 通配符与换行符不匹配Wildcard characters do not match newline characters
BUG-100266BUG-100266 PHOENIX-3521、 PHOENIX-4190PHOENIX-3521PHOENIX-4190 包含本地索引的结果错误Results wrong with local indexes
BUG-88774BUG-88774 HIVE-17617、 HIVE-18413、 HIVE-18523HIVE-17617HIVE-18413HIVE-18523 query36 失败,行计数不匹配query36 failing, row count mismatch
BUG-89765BUG-89765 HIVE-17702HIVE-17702 在 ORC 中的十进制读取器内进行错误的 isRepeating 处理。incorrect isRepeating handling in decimal reader in ORC
BUG-92293BUG-92293 HADOOP-15042HADOOP-15042 当 numberOfPagesRemaining 为 0 时,Azure PageBlobInputStream.skip() 可以返回负值。Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0
BUG-92345BUG-92345 ATLAS-2285ATLAS-2285 UI:已使用日期属性重命名保存的搜索。UI: Renamed saved search with date attribute.
BUG-92563BUG-92563 HIVE-17495、 HIVE-18528HIVE-17495HIVE-18528 聚合 ObjectStore 中的统计信息时收到错误的结果Aggregate stats in ObjectStore get wrong result
BUG-92957BUG-92957 HIVE-11266HIVE-11266 基于外部表的表统计信息 count(*) 错误结果count(*) wrong result based on table statistics for external tables
BUG-93097BUG-93097 RANGER-1944RANGER-1944 用于管理审核的操作筛选器不起作用Action filter for Admin Audit is not working
BUG-93335BUG-93335 HIVE-12315HIVE-12315 vectorization_short_regress.q 在执行双精度计算时出现错误结果问题vectorization_short_regress.q has a wrong result issue for a double calculation
BUG-93415BUG-93415 HIVE-18258、 HIVE-18310HIVE-18258HIVE-18310 矢量化:包含重复列的化简端 GROUP BY MERGEPARTIAL 已破坏Vectorization: Reduce-Side GROUP BY MERGEPARTIAL with duplicate columns is broken
BUG-93939BUG-93939 ATLAS-2294ATLAS-2294 创建类型时添加了额外的参数“description”Extra parameter "description" added when creating a type
BUG-94007BUG-94007 PHOENIX-1751、 PHOENIX-3112PHOENIX-1751PHOENIX-3112 由于存在 HBase 部分行,Phoenix 查询返回 Null 值Phoenix Queries returns Null values due to HBase Partial rows
BUG-94266BUG-94266 HIVE-12505HIVE-12505 同一加密区域中的插入覆盖操作以无提示方式失败,以删除某些现有文件Insert overwrite in same encrypted zone silently fails to remove some existing files
BUG-94414BUG-94414 HIVE-15680HIVE-15680 当 hive.optimize.index.filter=true 且在查询中两次引用同一个 ORC 表时,出现错误的结果Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
BUG-95048BUG-95048 HIVE-18490HIVE-18490 使用 EXISTS 和 NOT EXISTS 且包含 non-equi 谓词的查询可能生成错误结果Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong result
BUG-95053BUG-95053 PHOENIX-3865PHOENIX-3865 未针对第一个列系列筛选时,IS NULL 不会返回正确的结果IS NULL does not return correct results when first column family not filtered against
BUG-95476BUG-95476 RANGER-1966RANGER-1966 在某些情况下,策略引擎初始化不会创建上下文扩充器Policy engine initialization does not create context enrichers in some cases
BUG-95566BUG-95566 SPARK-23281SPARK-23281 当复合 order by 子句引用原始列和别名时,查询以错误的顺序生成结果Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases
BUG-95907BUG-95907 PHOENIX-3451、 PHOENIX-3452、 PHOENIX-3469、 PHOENIX-4560PHOENIX-3451PHOENIX-3452PHOENIX-3469PHOENIX-4560 修复当查询包含聚合时 ORDER BY ASC 出现的问题Fixing issues with ORDER BY ASC when query has aggregation
BUG-96389BUG-96389 PHOENIX-4586PHOENIX-4586 UPSERT SELECT 不会考虑子查询的比较运算符。UPSERT SELECT doesn't take in account comparison operators for subqueries.
BUG-96602BUG-96602 HIVE-18660HIVE-18660 PCR 无法区分分区和虚拟列PCR doesn't distinguish between partition and virtual columns
BUG-97686BUG-97686 ATLAS-2468ATLAS-2468 [基本搜索]:当 NEQ 用于数字类型时,OR 大小写出现问题[Basic Search] Issue with OR cases when NEQ is used with numeric types
BUG-97708BUG-97708 HIVE-18817HIVE-18817 读取 ACID 表期间发生 ArrayIndexOutOfBounds 异常。ArrayIndexOutOfBounds exception during read of ACID table.
BUG-97864BUG-97864 HIVE-18833HIVE-18833 “作为 orcfile 插入目录时”自动合并失败Auto Merge fails when "insert into directory as orcfile"
BUG-97889BUG-97889 RANGER-2008RANGER-2008 策略评估对于多行策略条件失败。Policy evaluation is failing for multiline policy conditions.
BUG-98655BUG-98655 RANGER-2066RANGER-2066 Hbase 列系列由该列系列中的某个标记列授权Hbase column family access is authorized by a tagged column in the column family
BUG-99883BUG-99883 HIVE-19073、 HIVE-19145HIVE-19073HIVE-19145 StatsOptimizer 可能损坏常量列StatsOptimizer may mangle constant columns

其他Other

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100267BUG-100267 HBASE-17170HBASE-17170 由于类加载程序的差异,HBase 也正在重试 DoNotRetryIOException。HBase is also retrying DoNotRetryIOException because of class loader differences.
BUG-92367BUG-92367 YARN-7558YARN-7558 如果启用 UI 身份验证,获取运行中容器的日志的“yarn logs”命令会失败。"yarn logs" command fails to get logs for running containers if UI authentication is enabled.
BUG-93159BUG-93159 OOZIE-3139OOZIE-3139 Oozie 不正确地验证工作流Oozie validates workflow incorrectly
BUG-93936BUG-93936 ATLAS-2289ATLAS-2289 将从 KafkaNotification 实现中移出嵌入的 kafka/zookeeper 服务器启动/停止代码Embedded kafka/zookeeper server start/stop code to be moved out of KafkaNotification implementation
BUG-93942BUG-93942 ATLAS-2312ATLAS-2312 使用 ThreadLocal DateFormat 对象避免从多个线程同时使用Use ThreadLocal DateFormat objects to avoid simultaneous use from multiple threads
BUG-93946BUG-93946 ATLAS-2319ATLAS-2319 UI:在平面结构和树结构中的标记列表内删除第 25 个位置以后的某个标记需要刷新,这样才能从该列表中删除该标记。UI: Deleting a tag which at 25+ position in the tag list in both Flat and Tree structure needs a refresh to remove the tag from the list.
BUG-94618BUG-94618 YARN-5037、 YARN-7274YARN-5037YARN-7274 能够在叶队列级别禁用弹性Ability to disable elasticity at leaf queue level
BUG-94901BUG-94901 HBASE-19285HBASE-19285 添加每个表的延迟直方图Add per-table latency histograms
BUG-95259BUG-95259 HADOOP-15185、 HADOOP-15186HADOOP-15185HADOOP-15186 将 adls 连接器更新为使用 ADLS SDK 的当前版本Update adls connector to use the current version of ADLS SDK
BUG-95619BUG-95619 HIVE-18551HIVE-18551 矢量化:VectorMapOperator 尝试为 Hybrid Grace 写入过许多的矢量列Vectorization: VectorMapOperator tries to write too many vector columns for Hybrid Grace
BUG-97223BUG-97223 SPARK-23434SPARK-23434 Spark 不应针对 HDFS 文件路径的`元数据目录`发出警告Spark should not warn `metadata directory` for a HDFS file path

“性能”Performance

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-83282BUG-83282 HBASE-13376、 HBASE-14473、 HBASE-15210、 HBASE-15515、 HBASE-16570、 HBASE-16810、 HBASE-18164HBASE-13376HBASE-14473HBASE-15210HBASE-15515HBASE-16570HBASE-16810HBASE-18164 均衡器中的快速位置计算Fast locality computation in balancer
BUG-91300BUG-91300 HBASE-17387HBASE-17387 在 multi() 的 RegionActionResult 中减少异常报告的开销Reduce the overhead of exception report in RegionActionResult for multi()
BUG-91804BUG-91804 TEZ-1526TEZ-1526 运行大型作业时,TezTaskID 的 LoadingCache 速度较慢LoadingCache for TezTaskID slow for large jobs
BUG-92760BUG-92760 ACCUMULO-4578ACCUMULO-4578 取消压缩 FATE 操作不会释放命名空间锁Cancel compaction FATE operation does not release namespace lock
BUG-93577BUG-93577 RANGER-1938RANGER-1938 用于审核设置的 Solr 不会有效使用 DocValuesSolr for Audit setup doesn't use DocValues effectively
BUG-93910BUG-93910 HIVE-18293HIVE-18293 Hive 无法压缩运行 HiveMetaStore 的标识不拥有的某个文件夹中的表Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
BUG-94345BUG-94345 HIVE-18429HIVE-18429 压缩应处理不生成任何输出的情况Compaction should handle a case when it produces no output
BUG-94381BUG-94381 HADOOP-13227、 HDFS-13054HADOOP-13227HDFS-13054 处理 RequestHedgingProxyProvider RetryAction 顺序:FAIL < RETRY < FAILOVER_AND_RETRY。Handling RequestHedgingProxyProvider RetryAction order: FAIL < RETRY < FAILOVER_AND_RETRY.
BUG-94432BUG-94432 HIVE-18353HIVE-18353 CompactorMR 应调用 jobclient.close() 来触发清理CompactorMR should call jobclient.close() to trigger cleanup
BUG-94869BUG-94869 PHOENIX-4290、 PHOENIX-4373PHOENIX-4290PHOENIX-4373 为本地索引加盐 phoenix 表请求 Get on HRegion 的超出范围的行。Requested row out of range for Get on HRegion for local indexed salted phoenix table.
BUG-94928BUG-94928 HDFS-11078HDFS-11078 修复 LazyPersistFileScrubber 中的 NPEFix NPE in LazyPersistFileScrubber
BUG-94964BUG-94964 HIVE-18269、 HIVE-18318、 HIVE-18326HIVE-18269HIVE-18318HIVE-18326 多项 LLAP 修复Multiple LLAP fixes
BUG-95669BUG-95669 HIVE-18577、 HIVE-18643HIVE-18577HIVE-18643 针对 ACID 分区表运行更新/删除查询时,HS2 读取所有分区。When run update/delete query on ACID partitioned table, HS2 read all each partitions.
BUG-96390BUG-96390 HDFS-10453HDFS-10453 对大型群集中同一文件执行复制和删除造成的资源争用可能会导致 ReplicationMonitor 线程长时间停滞。ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
BUG-96625BUG-96625 HIVE-16110HIVE-16110 还原“矢量化:支持 2 值 CASE WHEN,而不是故障回复到 VectorUDFAdaptor”Revert of "Vectorization: Support 2 Value CASE WHEN instead of fall back to VectorUDFAdaptor"
BUG-97109BUG-97109 HIVE-16757HIVE-16757 使用已弃用的 getRows() 而不是新的 estimateRowCount(RelMetadataQuery...) 会造成严重的性能影响Use of deprecated getRows() instead of new estimateRowCount(RelMetadataQuery...) has serious performance impact
BUG-97110BUG-97110 PHOENIX-3789PHOENIX-3789 在 postBatchMutateIndispensably 中执行跨区域索引维护调用Execute cross region index maintenance calls in postBatchMutateIndispensably
BUG-98833BUG-98833 YARN-6797YARN-6797 TimelineWriter 不完全使用 POST 响应TimelineWriter does not fully consume the POST response
BUG-98931BUG-98931 ATLAS-2491ATLAS-2491 更新 Hive 挂钩,以使用 Atlas v2 通知Update Hive hook to use Atlas v2 notifications

可能的数据丢失Potential Data Loss

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-95613BUG-95613 HBASE-18808HBASE-18808 配置签入 BackupLogCleaner#getDeletableFiles() 的效率低下Ineffective config check in BackupLogCleaner#getDeletableFiles()
BUG-97051BUG-97051 HIVE-17403HIVE-17403 非托管表和事务表的串联失败Fail concatenation for unmanaged and transactional tables
BUG-97787BUG-97787 HIVE-18460HIVE-18460 压缩器不会将表属性传递给 Orc 写入器Compactor doesn't pass Table properties to the Orc writer
BUG-97788BUG-97788 HIVE-18613HIVE-18613 扩展 JsonSerDe 以支持 BINARY 类型Extend JsonSerDe to support BINARY type

查询失败Query Failure

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100180BUG-100180 CALCITE-2232CALCITE-2232 调整聚合索引时 AggregatePullUpConstantsRule 出现断言错误Assertion error on AggregatePullUpConstantsRule while adjusting Aggregate indices
BUG-100422BUG-100422 HIVE-19085HIVE-19085 FastHiveDecimal abs(0) 将符号设置为 +veFastHiveDecimal abs(0) sets sign to +ve
BUG-100834BUG-100834 PHOENIX-4658PHOENIX-4658 IllegalStateException:无法对 ReversedKeyValueHeap 调用 requestSeekIllegalStateException: requestSeek cannot be called on ReversedKeyValueHeap
BUG-102078BUG-102078 HIVE-17978HIVE-17978 TPCDS 查询 58 和 83 在矢量化中生成异常。TPCDS queries 58 and 83 generate exceptions in vectorization.
BUG-92483BUG-92483 HIVE-17900HIVE-17900 分析压缩器触发的列中的统计信息生成包含多个分区列的、格式不当的 SQLanalyze stats on columns triggered by Compactor generates malformed SQL with > 1 partition column
BUG-93135BUG-93135 HIVE-15874、 HIVE-18189HIVE-15874HIVE-18189 将 hive.groupby.orderby.position.alias 设置为 true 时,Hive 查询返回错误结果Hive query returning wrong results when set hive.groupby.orderby.position.alias to true
BUG-93136BUG-93136 HIVE-18189HIVE-18189 禁用 cbo 时无法按位置排序Order by position does not work when cbo is disabled
BUG-93595BUG-93595 HIVE-12378、 HIVE-15883HIVE-12378HIVE-15883 在 Hive 中的 HBase 映射表内插入十进制和二进制列失败HBase mapped table in Hive insert fail for decimal and binary columns
BUG-94007BUG-94007 PHOENIX-1751、 PHOENIX-3112PHOENIX-1751PHOENIX-3112 由于存在 HBase 部分行,Phoenix 查询返回 Null 值Phoenix Queries returns Null values due to HBase Partial rows
BUG-94144BUG-94144 HIVE-17063HIVE-17063 在首先删除分区的情况下,将覆盖分区插入外部表失败insert overwrite partition onto an external table fail when drop partition first
BUG-94280BUG-94280 HIVE-12785HIVE-12785 包含联合类型的视图和用于`强制转换`结构的 UDF 破坏View with union type and UDF to `cast` the struct is broken
BUG-94505BUG-94505 PHOENIX-4525PHOENIX-4525 GroupBy 执行中的整数溢出Integer overflow in GroupBy execution
BUG-95618BUG-95618 HIVE-18506HIVE-18506 LlapBaseInputFormat - 负数组索引LlapBaseInputFormat - negative array index
BUG-95644BUG-95644 HIVE-9152HIVE-9152 CombineHiveInputFormat:Tez 中的 Hive 查询失败并出现 java.lang.IllegalArgumentException 异常CombineHiveInputFormat: Hive query is failing in Tez with java.lang.IllegalArgumentException exception
BUG-96762BUG-96762 PHOENIX-4588PHOENIX-4588 如果表达式的子级包含 Determinism.PER_INVOCATION,则也会克隆表达式Clone expression also if its children have Determinism.PER_INVOCATION
BUG-97145BUG-97145 HIVE-12245、 HIVE-17829HIVE-12245HIVE-17829 支持基于 HBase 的表的列注释Support column comments for an HBase backed table
BUG-97741BUG-97741 HIVE-18944HIVE-18944 DPP 期间错误地设置分组集位置Groupping sets position is set incorrectly during DPP
BUG-98082BUG-98082 HIVE-18597HIVE-18597 LLAP:始终打包 org.apache.log4j 的 log4j2 API jarLLAP: Always package the log4j2 API jar for org.apache.log4j
BUG-99849BUG-99849 不适用N/A 通过文件向导创建一个新表,以尝试使用默认数据库Create a new table from a file wizard tries to use default database

安全性Security

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100436BUG-100436 RANGER-2060RANGER-2060 使用 knox-sso 的 Knox 代理不适用于 rangerKnox proxy with knox-sso is not working for ranger
BUG-101038BUG-101038 SPARK-24062SPARK-24062 Zeppelin %Spark 解释器发生“连接被拒绝”错误,HiveThriftServer 中出现“必须指定机密密钥...”错误Zeppelin %Spark interpreter "Connection refused" error, "A secret key must be specified..." error in HiveThriftServer
BUG-101359BUG-101359 ACCUMULO-4056ACCUMULO-4056 在发布时将 commons-collection 的版本更新为 3.2.2Update version of commons-collection to 3.2.2 when released
BUG-54240BUG-54240 HIVE-18879HIVE-18879 如果 xercesImpl.jar 在类路径中,需要能够禁止在 UDFXPathUtil 中使用嵌入式元素Disallow embedded element in UDFXPathUtil needs to work if xercesImpl.jar in classpath
BUG-79059BUG-79059 OOZIE-3109OOZIE-3109 转义日志流的特定于 HTML 的字符Escape log-streaming's HTML-specific characters
BUG-90041BUG-90041 OOZIE-2723OOZIE-2723 JSON.org 许可证现在为 CatXJSON.org license is now CatX
BUG-93754BUG-93754 RANGER-1943RANGER-1943 集合为空或 null 时,将跳过 Ranger Solr 授权Ranger Solr authorization is skipped when collection is empty or null
BUG-93804BUG-93804 HIVE-17419HIVE-17419 ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS 命令显示掩码表的计算统计信息ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command shows computed stats for masked tables
BUG-94276BUG-94276 ZEPPELIN-3129ZEPPELIN-3129 Zeppelin UI 不会在 IE 中注销Zeppelin UI does not sign out in IE
BUG-95349BUG-95349 ZOOKEEPER-1256、 ZOOKEEPER-1901ZOOKEEPER-1256ZOOKEEPER-1901 升级 nettyUpgrade netty
BUG-95483BUG-95483 不适用N/A CVE 2017-15713 的修复Fix for CVE-2017-15713
BUG-95646BUG-95646 OOZIE-3167OOZIE-3167 升级 Oozie 4.3 分支上的 tomcat 版本Upgrade tomcat version on Oozie 4.3 branch
BUG-95823BUG-95823 不适用N/A Knox:升级 BeanutilsKnox: Upgrade Beanutils
BUG-95908BUG-95908 RANGER-1960RANGER-1960 在删除快照时,HBase 身份验证不考虑表命名空间HBase auth does not take table namespace into consideration for deleting snapshot
BUG-96191BUG-96191 FALCON-2322、 FALCON-2323FALCON-2322FALCON-2323 升级 Jackson 和 Spring 版本以避免安全漏洞Upgrade Jackson and Spring versions to avoid security vulnerabilities
BUG-96502BUG-96502 RANGER-1990RANGER-1990 在 Ranger Admin 中添加单向 SSL MySQL 支持Add One-way SSL MySQL support in Ranger Admin
BUG-96712BUG-96712 FLUME-3194FLUME-3194 将 derby 升级到最新版本 (1.14.1.0)upgrade derby to the latest (1.14.1.0) version
BUG-96713BUG-96713 FLUME-2678FLUME-2678 将 xalan 升级到 2.7.2 以处理 CVE-2014-0107 漏洞Upgrade xalan to 2.7.2 to take care of CVE-2014-0107 vulnerability
BUG-96714BUG-96714 FLUME-2050FLUME-2050 升级到 log4j2(推出正式版时)Upgrade to log4j2 (when GA)
BUG-96737BUG-96737 不适用N/A 使用 java io 文件系统方法来问本地文件Use java io filesystem methods to access local files
BUG-96925BUG-96925 不适用N/A 将 Hadoop 中的 Tomcat 从 6.0.48 升级到 6.0.53Upgrade Tomcat from 6.0.48 to 6.0.53 in Hadoop
BUG-96977BUG-96977 FLUME-3132FLUME-3132 升级 tomcat jasper 库依赖项Upgrade tomcat jasper library dependencies
BUG-97022BUG-97022 HADOOP-14799、 HADOOP-14903、 HADOOP-15265HADOOP-14799HADOOP-14903HADOOP-15265 升级具有 4.39 以上版本的 Nimbus JOSE JWT 库Upgrading Nimbus-JOSE-JWT library with version above 4.39
BUG-97101BUG-97101 RANGER-1988RANGER-1988 修复不安全的随机性Fix insecure randomness
BUG-97178BUG-97178 ATLAS-2467ATLAS-2467 Spring 和 nimbus-jose-jwt 的依赖项升级Dependency upgrade for Spring and nimbus-jose-jwt
BUG-97180BUG-97180 不适用N/A 升级 Nimbus-jose-jwtUpgrade Nimbus-jose-jwt
BUG-98038BUG-98038 HIVE-18788HIVE-18788 清理 JDBC PreparedStatement 中的输入Clean up inputs in JDBC PreparedStatement
BUG-98353BUG-98353 HADOOP-13707HADOOP-13707 还原“如果在未配置 HTTP SPNEGO 的情况下启用 kerberos,则无法访问某些链接”Revert of "If kerberos is enabled while HTTP SPNEGO is not configured, some links cannot be accessed"
BUG-98372BUG-98372 HBASE-13848HBASE-13848 通过凭据提供程序 API 访问 InfoServer SSL 密码Access InfoServer SSL passwords through Credential Provider API
BUG-98385BUG-98385 ATLAS-2500ATLAS-2500 将其他标头添加到 Atlas 响应。Add additional headers to Atlas response.
BUG-98564BUG-98564 HADOOP-14651HADOOP-14651 将 Okhttp 版本更新为 2.7.5Update okhttp version to 2.7.5
BUG-99440BUG-99440 RANGER-2045RANGER-2045 使用“desc table”命令列出没有显式允许策略的 Hive 表列Hive table columns with no explicit allow policy are listed with 'desc table' command
BUG-99803BUG-99803 不适用N/A Oozie 应禁用 HBase 动态类加载Oozie should disable HBase dynamic class loading

稳定性Stability

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100040BUG-100040 ATLAS-2536ATLAS-2536 Atlas Hive 挂钩中 NPENPE in Atlas Hive Hook
BUG-100057BUG-100057 HIVE-19251HIVE-19251 具有 LIMIT 的 ObjectStore.getNextNotification 应使用较少的内存ObjectStore.getNextNotification with LIMIT should use less memory
BUG-100072BUG-100072 HIVE-19130HIVE-19130 REPL LOAD 应用删除分区事件后引发 NPE。NPE is thrown when REPL LOAD applied drop partition event.
BUG-100073BUG-100073 不适用N/A 从 hiveserver 到数据节点的 close_wait 连接过多too many close_wait connections from hiveserver to data node
BUG-100319BUG-100319 HIVE-19248HIVE-19248 如果文件复制失败,REPL LOAD 不会引发错误。REPL LOAD doesn't throw error if file copy fails.
BUG-100352BUG-100352 不适用N/A CLONE - RM 过于频繁地清除逻辑扫描/注册表 znodeCLONE - RM purging logic scans /registry znode too frequently
BUG-100427BUG-100427 HIVE-19249HIVE-19249 复制:WITH 子句无法在所有情况下正确地向任务传递配置Replication: WITH clause is not passing the configuration to Task correctly in all cases
BUG-100430BUG-100430 HIVE-14483HIVE-14483 java.lang.ArrayIndexOutOfBoundsException org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.commonReadByteArraysjava.lang.ArrayIndexOutOfBoundsException org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.commonReadByteArrays
BUG-100432BUG-100432 HIVE-19219HIVE-19219 如果请求的事件已清理,增量 REPL DUMP 应引发错误。Incremental REPL DUMP should throw error if requested events are cleaned-up.
BUG-100448BUG-100448 SPARK-23637、 SPARK-23802、 SPARK-23809、 SPARK-23816、 SPARK-23822、 SPARK-23823、 SPARK-23838、 SPARK-23881SPARK-23637SPARK-23802SPARK-23809SPARK-23816SPARK-23822SPARK-23823SPARK-23838SPARK-23881 将 Spark2 更新到 2.3.0+ (4/11)Update Spark2 to 2.3.0+ (4/11)
BUG-100740BUG-100740 HIVE-16107HIVE-16107 JDBC:发生 NoHttpResponseException 时 HttpClient 应重试一次或多次JDBC: HttpClient should retry one more time on NoHttpResponseException
BUG-100810BUG-100810 HIVE-19054HIVE-19054 Hive 函数复制失败Hive Functions replication fails
BUG-100937BUG-100937 MAPREDUCE-6889MAPREDUCE-6889 添加 Job#close API 以关闭 MR 客户端服务。Add Job#close API to shutdown MR client services.
BUG-101065BUG-101065 ATLAS-2587ATLAS-2587 为 HA 中的 /apache_atlas/active_server_info znode 设置读取 ACL,以便读取 Knox 代理。Set read ACL for /apache_atlas/active_server_info znode in HA for Knox proxy to read.
BUG-101093BUG-101093 STORM-2993STORM-2993 使用时间轮换策略时,Storm HDFS Bolt 引发 ClosedChannelExceptionStorm HDFS bolt throws ClosedChannelException when Time rotation policy is used
BUG-101181BUG-101181 不适用N/A PhoenixStorageHandler 不会正确处理谓词中的 ANDPhoenixStorageHandler doesn't handle AND in predicate correctly
BUG-101266BUG-101266 PHOENIX-4635PHOENIX-4635 org.apache.phoenix.hive.mapreduce.PhoenixInputFormat 中的 HBase 连接泄漏HBase Connection leak in org.apache.phoenix.hive.mapreduce.PhoenixInputFormat
BUG-101458BUG-101458 HIVE-11464HIVE-11464 存在多个输出时缺少沿袭信息lineage info missing if there are multiple outputs
BUG-101485BUG-101485 不适用N/A hive metastore thrift api 速度缓慢,导致客户端超时hive metastore thrift api is slow and causing client timeout
BUG-101628BUG-101628 HIVE-19331HIVE-19331 Hive 增量复制到云失败。Hive incremental replication to cloud failed.
BUG-102048BUG-102048 HIVE-19381HIVE-19381 FunctionTask 的 Hive 函数复制到云失败Hive Function Replication to cloud fails with FunctionTask
BUG-102064BUG-102064 不适用N/A ReplCopyTask 中的 Hive 复制 [本地到本地] 测试失败Hive Replication [ onprem to onprem ] tests failed in ReplCopyTask
BUG-102137BUG-102137 HIVE-19423HIVE-19423 ReplCopyTask 中的 Hive 复制 [本地到云] 测试失败Hive Replication [ Onprem to Cloud ] tests failed in ReplCopyTask
BUG-102305BUG-102305 HIVE-19430HIVE-19430 HS2 和 hive 元存储 OOM 转储HS2 and hive metastore OOM dumps
BUG-102361BUG-102361 不适用N/A 复制到目标 hive 群集(本地到 s3)的单个插入内容中存在多个插入结果multiple insert results in single insert replicated to target hive cluster ( onprem - s3 )
BUG-87624BUG-87624 不适用N/A 启用 storm 事件日志记录导致工作线程持续关闭Enabling storm event logging causes workers to continuously die
BUG-88929BUG-88929 HBASE-15615HBASE-15615 RegionServerCallable 需要重试时休眠时间错误Wrong sleep time when RegionServerCallable need retry
BUG-89628BUG-89628 HIVE-17613HIVE-17613 删除较短的相同线程分配的对象池remove object pools for short, same-thread allocations
BUG-89813BUG-89813 不适用N/A SCA:代码正确性:非同步方法重写同步方法SCA: Code Correctness: Non-Synchronized Method Overrides Synchronized Method
BUG-90437BUG-90437 ZEPPELIN-3072ZEPPELIN-3072 如果存在过多的笔记本,Zeppelin UI 将会变慢/无响应Zeppelin UI becomes slow/unresponsive if there are too many notebooks
BUG-90640BUG-90640 HBASE-19065HBASE-19065 HRegion#bulkLoadHFiles() 应等待并发 Region#flush() 完成HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish
BUG-91202BUG-91202 HIVE-17013HIVE-17013 使用基于视图选择的子查询删除请求Delete request with a subquery based on select over a view
BUG-91350BUG-91350 KNOX-1108KNOX-1108 NiFiHaDispatch 不会故障转移NiFiHaDispatch not failing over
BUG-92054BUG-92054 HIVE-13120HIVE-13120 生成 ORC 拆分时传播 doAspropagate doAs when generating ORC splits
BUG-92373BUG-92373 FALCON-2314FALCON-2314 将 TestNG 版本升级到 6.13.1 以避免 BeanShell 依赖关系Bump TestNG version to 6.13.1 to avoid BeanShell dependency
BUG-92381BUG-92381 不适用N/A testContainerLogsWithNewAPI 和 testContainerLogsWithOldAPI UT 失败testContainerLogsWithNewAPI and testContainerLogsWithOldAPI UT fails
BUG-92389BUG-92389 STORM-2841STORM-2841 testNoAcksIfFlushFails UT 失败并出现 NullPointerExceptiontestNoAcksIfFlushFails UT fails with NullPointerException
BUG-92586BUG-92586 SPARK-17920、 SPARK-20694、 SPARK-21642、 SPARK-22162、 SPARK-22289、 SPARK-22373、 SPARK-22495、 SPARK-22574、 SPARK-22591、 SPARK-22595、 SPARK-22601、 SPARK-22603、 SPARK-22607、 SPARK-22635、 SPARK-22637、 SPARK-22653、 SPARK-22654、 SPARK-22686、 SPARK-22688、 SPARK-22817、 SPARK-22862、 SPARK-22889、 SPARK-22972、 SPARK-22975、 SPARK-22982、 SPARK-22983、 SPARK-22984、 SPARK-23001、 SPARK-23038、 SPARK-23095SPARK-17920SPARK-20694SPARK-21642SPARK-22162SPARK-22289SPARK-22373SPARK-22495SPARK-22574SPARK-22591SPARK-22595SPARK-22601SPARK-22603SPARK-22607SPARK-22635SPARK-22637SPARK-22653SPARK-22654SPARK-22686SPARK-22688SPARK-22817SPARK-22862SPARK-22889SPARK-22972SPARK-22975SPARK-22982SPARK-22983SPARK-22984SPARK-23001SPARK-23038SPARK-23095 将最新的 Spark2 更新到 2.2.1(1 月 16 日版)Update Spark2 up-to-date to 2.2.1 (Jan. 16)
BUG-92680BUG-92680 ATLAS-2288ATLAS-2288 通过 Hive 创建 hbase 表后运行 import-hive 脚本时出现 NoClassDefFoundError 异常NoClassDefFoundError Exception while running import-hive script when hbase table is created via Hive
BUG-92760BUG-92760 ACCUMULO-4578ACCUMULO-4578 取消压缩 FATE 操作不会释放命名空间锁Cancel compaction FATE operation does not release namespace lock
BUG-92797BUG-92797 HDFS-10267、 HDFS-8496HDFS-10267HDFS-8496 在某些用例中减少数据节点锁争用Reducing the datanode lock contentions on certain use cases
BUG-92813BUG-92813 FLUME-2973FLUME-2973 hdfs 接收器中出现死锁Deadlock in hdfs sink
BUG-92957BUG-92957 HIVE-11266HIVE-11266 基于外部表的表统计信息 count(*) 错误结果count(*) wrong result based on table statistics for external tables
BUG-93018BUG-93018 ATLAS-2310ATLAS-2310 在 HA 中,被动节点使用错误的 URL 编码重定向请求In HA, the passive node redirects the request with wrong URL encoding
BUG-93116BUG-93116 RANGER-1957RANGER-1957 启用增量同步时,Ranger Usersync 不会定期同步用户或组。Ranger Usersync is not syncing users or groups periodically when incremental sync is enabled.
BUG-93361BUG-93361 HIVE-12360HIVE-12360 使用谓词下推在未压缩的 ORC 中进行错误查找Bad seek in uncompressed ORC with predicate pushdown
BUG-93426BUG-93426 CALCITE-2086CALCITE-2086 在某些情况下,大型授权标头导致 HTTP/413HTTP/413 in certain circumstances due to large Authorization headers
BUG-93429BUG-93429 PHOENIX-3240PHOENIX-3240 Pig 加载程序中出现 ClassCastExceptionClassCastException from Pig loader
BUG-93485BUG-93485 不适用N/A 无法获取表 mytestorg.apache.hadoop.hive.ql.metadata.InvalidTableException:针对 LLAP 中的列运行分析表时找不到表Cannot get table mytestorg.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found when running analyze table on columns in LLAP
BUG-93512BUG-93512 PHOENIX-4466PHOENIX-4466 java.lang.RuntimeException:响应代码 500 - 执行 spark 作业以连接到 phoenix 查询服务器并加载数据java.lang.RuntimeException: response code 500 - Executing a spark job to connect to phoenix query server and load data
BUG-93550BUG-93550 不适用N/A 由于 scala 版本不匹配,Zeppelin %spark.r 不适用于 spark1Zeppelin %spark.r does not work with spark1 due to scala version mismatch
BUG-93910BUG-93910 HIVE-18293HIVE-18293 Hive 无法压缩运行 HiveMetaStore 的标识不拥有的某个文件夹中的表Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
BUG-93926BUG-93926 ZEPPELIN-3114ZEPPELIN-3114 >1d 压力测试后,笔记本和解释器不会在 zeppelin 中保存Notebooks and interpreters are not getting saved in zeppelin after >1d stress testing
BUG-93932BUG-93932 ATLAS-2320ATLAS-2320 包含查询的分类“*”引发 500 内部服务器异常。classification "*" with query throws 500 Internal server exception.
BUG-93948BUG-93948 YARN-7697YARN-7697 由于日志聚合泄露 (part#1),NM 关闭并出现 OOMNM goes down with OOM due to leak in log-aggregation (part#1)
BUG-93965BUG-93965 ATLAS-2229ATLAS-2229 DSL 搜索:orderby 非字符串属性引发异常DSL search: orderby non-string attribute throws exception
BUG-93986BUG-93986 YARN-7697YARN-7697 由于日志聚合泄露 (part#2),NM 关闭并出现 OOMNM goes down with OOM due to leak in log-aggregation (part#2)
BUG-94030BUG-94030 ATLAS-2332ATLAS-2332 使用具有嵌套集合数据类型的属性创建类型失败Creation of type with attributes having nested collection datatype fails
BUG-94080BUG-94080 YARN-3742、 YARN-6061YARN-3742YARN-6061 两个 RM 在安全群集中处于待机状态Both RM are in standby in secure cluster
BUG-94081BUG-94081 HIVE-18384HIVE-18384 log4j2.x 库中发生 ConcurrentModificationExceptionConcurrentModificationException in log4j2.x library
BUG-94168BUG-94168 不适用N/A Yarn RM 关闭并且服务注册表处于错误的 ERROR 状态Yarn RM goes down with Service Registry is in wrong state ERROR
BUG-94330BUG-94330 HADOOP-13190、 HADOOP-14104、 HADOOP-14814、 HDFS-10489、 HDFS-11689HADOOP-13190HADOOP-14104HADOOP-14814HDFS-10489HDFS-11689 HDFS 应受多个 KMS RUI 的支持HDFS should support for multiple KMS Uris
BUG-94345BUG-94345 HIVE-18429HIVE-18429 压缩应处理不生成任何输出的情况Compaction should handle a case when it produces no output
BUG-94372BUG-94372 ATLAS-2229ATLAS-2229 DSL 查询:hive_table name = ["t1","t2"] 引发 DSL 查询无效异常DSL query: hive_table name = ["t1","t2"] throws invalid DSL query exception
BUG-94381BUG-94381 HADOOP-13227、 HDFS-13054HADOOP-13227HDFS-13054 处理 RequestHedgingProxyProvider RetryAction 顺序:FAIL < RETRY < FAILOVER_AND_RETRY。Handling RequestHedgingProxyProvider RetryAction order: FAIL < RETRY < FAILOVER_AND_RETRY.
BUG-94432BUG-94432 HIVE-18353HIVE-18353 CompactorMR 应调用 jobclient.close() 来触发清理CompactorMR should call jobclient.close() to trigger cleanup
BUG-94575BUG-94575 SPARK-22587SPARK-22587 如果 fs.defaultFS 和应用程序 jar 是不同的 url,Spark 作业将会失败Spark job fails if fs.defaultFS and application jar are different url
BUG-94791BUG-94791 SPARK-22793SPARK-22793 Spark Thrift 服务器出现内存泄漏Memory leak in Spark Thrift Server
BUG-94928BUG-94928 HDFS-11078HDFS-11078 修复 LazyPersistFileScrubber 中的 NPEFix NPE in LazyPersistFileScrubber
BUG-95013BUG-95013 HIVE-18488HIVE-18488 LLAP ORC 读取器缺少一些 null 检查LLAP ORC readers are missing some null checks
BUG-95077BUG-95077 HIVE-14205HIVE-14205 Hive 不支持 AVRO 文件格式的联合类型Hive doesn't support union type with AVRO file format
BUG-95200BUG-95200 HDFS-13061HDFS-13061 SaslDataTransferClient#checkTrustAndSend 不应信任部分信任的通道SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel
BUG-95201BUG-95201 HDFS-13060HDFS-13060 为 TrustedChannelResolver 添加 BlacklistBasedTrustedChannelResolverAdding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver
BUG-95284BUG-95284 HBASE-19395HBASE-19395 [branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting 失败并出现 NPE[branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting fails with NPE
BUG-95301BUG-95301 HIVE-18517HIVE-18517 矢量化:修复 VectorMapOperator,以接受 VRB 并正确检查矢量化标志来支持 LLAP 缓存Vectorization: Fix VectorMapOperator to accept VRBs and check vectorized flag correctly to support LLAP Caching
BUG-95542BUG-95542 HBASE-16135HBASE-16135 已删除的对等方的 rs 下的 PeerClusterZnode 可能永远无法删除PeerClusterZnode under rs of removed peer may never be deleted
BUG-95595BUG-95595 HIVE-15563HIVE-15563 忽略 SQLOperation.runQuery 中的非法操作状态转换异常,以公开实际异常。Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.
BUG-95596BUG-95596 YARN-4126、 YARN-5750YARN-4126YARN-5750 TestClientRMService 失败TestClientRMService fails
BUG-96019BUG-96019 HIVE-18548HIVE-18548 修复 log4j 导入Fix log4j import
BUG-96196BUG-96196 HDFS-13120HDFS-13120 concat 后,快照差异可能会损坏Snapshot diff could be corrupted after concat
BUG-96289BUG-96289 HDFS-11701HDFS-11701 未解析的主机中的 NPE 导致永久性 DFSInputStream 失败NPE from Unresolved Host causes permanent DFSInputStream failures
BUG-96291BUG-96291 STORM-2652STORM-2652 JmsSpout 打开方法中引发异常Exception thrown in JmsSpout open method
BUG-96363BUG-96363 HIVE-18959HIVE-18959 避免在 LLAP 中创建额外的线程池Avoid creating extra pool of threads within LLAP
BUG-96390BUG-96390 HDFS-10453HDFS-10453 对大型群集中同一文件执行复制和删除造成的资源争用可能会导致 ReplicationMonitor 线程长时间停滞。ReplicationMonitor thread could be stuck for a long time due to the race between replication and delete of the same file in a large cluster.
BUG-96454BUG-96454 YARN-4593YARN-4593 AbstractService.getConfig() 中出现死锁Deadlock in AbstractService.getConfig()
BUG-96704BUG-96704 FALCON-2322FALCON-2322 submitAndSchedule 馈送时发生 ClassCastExceptionClassCastException while submitAndSchedule feed
BUG-96720BUG-96720 SLIDER-1262SLIDER-1262 Kerberized 环境中的 Slider 函数测试失败Slider functests are failing in Kerberized environment
BUG-96931BUG-96931 SPARK-23053、 SPARK-23186、 SPARK-23230、 SPARK-23358、 SPARK-23376、 SPARK-23391SPARK-23053SPARK-23186SPARK-23230SPARK-23358SPARK-23376SPARK-23391 更新到最新的 Spark2(2 月 19 日版)Update Spark2 up-to-date (Feb. 19)
BUG-97067BUG-97067 HIVE-10697HIVE-10697 ObjectInspectorConvertors#UnionConvertor 执行错误的转换ObjectInspectorConvertors#UnionConvertor does a faulty conversion
BUG-97244BUG-97244 KNOX-1083KNOX-1083 HttpClient 默认超时应是一个有意义的值HttpClient default timeout should be a sensible value
BUG-97459BUG-97459 ZEPPELIN-3271ZEPPELIN-3271 用于禁用计划程序的选项Option for disabling scheduler
BUG-97511BUG-97511 KNOX-1197KNOX-1197 当服务中的 authentication=Anonymous 时不会添加 AnonymousAuthFilterAnonymousAuthFilter is not added when authentication=Anonymous in service
BUG-97601BUG-97601 HIVE-17479HIVE-17479 不会针对更新/删除查询清理临时目录Staging directories do not get cleaned up for update/delete queries
BUG-97605BUG-97605 HIVE-18858HIVE-18858 提交 MR 作业时不会解析作业配置中的系统属性System properties in job configuration not resolved when submitting MR job
BUG-97674BUG-97674 OOZIE-3186OOZIE-3186 Oozie 无法使用通过 jceks://file/... 链接的配置Oozie is unable to use configuration linked using jceks://file/...
BUG-97743BUG-97743 不适用N/A 部署 storm 拓扑时发生 java.lang.NoClassDefFoundError 异常java.lang.NoClassDefFoundError exception while deploying storm topology
BUG-97756BUG-97756 PHOENIX-4576PHOENIX-4576 修复主分支中失败的 LocalIndexSplitMergeIT 测试Fix LocalIndexSplitMergeIT tests failing in master branch
BUG-97771BUG-97771 HDFS-11711HDFS-11711 发生“打开的文件过多”异常时 DN 不应删除块DN should not delete the block On "Too many open files" Exception
BUG-97869BUG-97869 KNOX-1190KNOX-1190 Google OIDC 的 Knox SSO 支持已中断。Knox SSO support for Google OIDC is broken.
BUG-97879BUG-97879 PHOENIX-4489PHOENIX-4489 Phoenix MR 作业中的 HBase 连接泄漏HBase Connection leak in Phoenix MR Jobs
BUG-98392BUG-98392 RANGER-2007RANGER-2007 ranger-tagsync 的 Kerberos 票证无法续订ranger-tagsync's Kerberos ticket fails to renew
BUG-98484BUG-98484 不适用N/A Hive 增量复制到云无法正常进行Hive Incremental Replication to Cloud not working
BUG-98533BUG-98533 HBASE-19934、 HBASE-20008HBASE-19934HBASE-20008 由于出现 Null 指针异常,Hbase 快照还原失败Hbase snapshot restore is failing due to Null pointer exception
BUG-98555BUG-98555 PHOENIX-4662PHOENIX-4662 重新发送缓存时 TableResultIterator.java 中发生 NullPointerExceptionNullPointerException in TableResultIterator.java on cache resend
BUG-98579BUG-98579 HBASE-13716HBASE-13716 停止使用 Hadoop 的 FSConstantsStop using Hadoop's FSConstants
BUG-98705BUG-98705 KNOX-1230KNOX-1230 向 Knox 发送许多并发请求导致 URL 损坏Many Concurrent Requests to Knox causes URL Mangling
BUG-98983BUG-98983 KNOX-1108KNOX-1108 NiFiHaDispatch 不会故障转移NiFiHaDispatch not failing over
BUG-99107BUG-99107 HIVE-19054HIVE-19054 函数复制应使用“hive.repl.replica.functions.root.dir”作为根Function replication shall use "hive.repl.replica.functions.root.dir" as root
BUG-99145BUG-99145 RANGER-2035RANGER-2035 使用 Oracle 后端访问包含空 implClass 的 servicedefs 时出错Errors accessing servicedefs with empty implClass with Oracle backend
BUG-99160BUG-99160 SLIDER-1259SLIDER-1259 Slider 在多宿主环境中无法工作Slider does not work in multi-homed environments
BUG-99239BUG-99239 ATLAS-2462ATLAS-2462 由于未在命令中提供任何表,针对所有表执行 Sqoop 导入引发 NPESqoop import for all tables throws NPE for no table provided in command
BUG-99301BUG-99301 ATLAS-2530ATLAS-2530 hive_process 和 hive_column_lineage 的名称属性开头存在换行符Newline at the beginning of the name attribute of a hive_process and hive_column_lineage
BUG-99453BUG-99453 HIVE-19065HIVE-19065 元存储客户端兼容性检查应包括 syncMetaStoreClientMetastore client compatibility check should include syncMetaStoreClient
BUG-99521BUG-99521 不适用N/A 重新实例化迭代器时,未重新创建 HashJoin 的 ServerCacheServerCache for HashJoin is not re-created when iterators are re-instantiated
BUG-99590BUG-99590 PHOENIX-3518PHOENIX-3518 RenewLeaseTask 中出现内存泄漏Memory Leak in RenewLeaseTask
BUG-99618BUG-99618 SPARK-23599、 SPARK-23806SPARK-23599SPARK-23806 将 Spark2 更新到 2.3.0+ (3/28)Update Spark2 to 2.3.0+ (3/28)
BUG-99672BUG-99672 ATLAS-2524ATLAS-2524 Hive 与 V2 通知挂钩 - 不正确地处理“alter view as”操作Hive hook with V2 notifications - incorrect handling of 'alter view as' operation
BUG-99809BUG-99809 HBASE-20375HBASE-20375 在 hbase spark 模块中删除 getCurrentUserCredentials 的使用Remove use of getCurrentUserCredentials in hbase-spark module

可支持性Supportability

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-87343BUG-87343 HIVE-18031HIVE-18031 支持“更改数据库”操作的复制。Support replication for Alter Database operation.
BUG-91293BUG-91293 RANGER-2060RANGER-2060 使用 knox-sso 的 Knox 代理不适用于 rangerKnox proxy with knox-sso is not working for ranger
BUG-93116BUG-93116 RANGER-1957RANGER-1957 启用增量同步时,Ranger Usersync 不会定期同步用户或组。Ranger Usersync is not syncing users or groups periodically when incremental sync is enabled.
BUG-93577BUG-93577 RANGER-1938RANGER-1938 用于审核设置的 Solr 不会有效使用 DocValuesSolr for Audit setup doesn't use DocValues effectively
BUG-96082BUG-96082 RANGER-1982RANGER-1982 改善 Ranger Admin 和 Ranger KMS 分析指标的错误Error Improvement for Analytics Metric of Ranger Admin and Ranger Kms
BUG-96479BUG-96479 HDFS-12781HDFS-12781 数据节点关闭后,在“名称节点”UI 中,“数据节点”选项卡引发警告消息。After Datanode down, In Namenode UI Datanode tab is throwing warning message.
BUG-97864BUG-97864 HIVE-18833HIVE-18833 “作为 orcfile 插入目录时”自动合并失败Auto Merge fails when "insert into directory as orcfile"
BUG-98814BUG-98814 HDFS-13314HDFS-13314 如果检测到 FsImage 损坏,NameNode 应可选择性退出NameNode should optionally exit if it detects FsImage corruption

升级Upgrade

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100134BUG-100134 SPARK-22919SPARK-22919 还原“升级 Apache httpclient 版本”Revert of "Bump Apache httpclient versions"
BUG-95823BUG-95823 不适用N/A Knox:升级 BeanutilsKnox: Upgrade Beanutils
BUG-96751BUG-96751 KNOX-1076KNOX-1076 将 nimbus-jose-jwt 更新为 4.41.2Update nimbus-jose-jwt to 4.41.2
BUG-97864BUG-97864 HIVE-18833HIVE-18833 “作为 orcfile 插入目录时”自动合并失败Auto Merge fails when "insert into directory as orcfile"
BUG-99056BUG-99056 HADOOP-13556HADOOP-13556 将 Configuration.getPropsWithPrefix 更改为使用 getProps 而不是迭代器Change Configuration.getPropsWithPrefix to use getProps instead of iterator
BUG-99378BUG-99378 ATLAS-2461、 ATLAS-2554ATLAS-2461ATLAS-2554 用于在 Titan 图形数据库中导出 Atlas 数据的迁移实用工具Migration utility to export Atlas data in Titan graph DB

可用性Usability

Hortonworks Bug IDHortonworks Bug ID Apache JIRAApache JIRA 摘要Summary
BUG-100045BUG-100045 HIVE-19056HIVE-19056 当 ORC 文件包含 0 行时,FixAcidKeyIndex 中发生 IllegalArgumentExceptionIllegalArgumentException in FixAcidKeyIndex when ORC file has 0 rows
BUG-100139BUG-100139 KNOX-1243KNOX-1243 规范化在 KnoxToken 服务中配置的所需 DNNormalize the required DNs that are Configured in KnoxToken Service
BUG-100570BUG-100570 ATLAS-2557ATLAS-2557 进行修复,以便当 UGI 中的组已错误设置或不为空时,能够查找 hadoop ldap 组Fix to allow to lookup hadoop ldap groups when are groups from UGI are wrongly set or are not empty
BUG-100646BUG-100646 ATLAS-2102ATLAS-2102 Atlas UI 改进:搜索结果页Atlas UI Improvements: Search results page
BUG-100737BUG-100737 HIVE-19049HIVE-19049 添加为 Druid 更改表和添加列的支持Add support for Alter table add columns for Druid
BUG-100750BUG-100750 KNOX-1246KNOX-1246 更新 Knox 中的服务配置,以支持 Ranger 的最新配置。Update service config in Knox to support latest configurations for Ranger.
BUG-100965BUG-100965 ATLAS-2581ATLAS-2581 使用 V2 Hive 挂钩通知回归:将表移到不同的数据库Regression with V2 Hive hook notifications: Moving table to a different database
BUG-84413BUG-84413 ATLAS-1964ATLAS-1964 UI:支持对搜索表中的列排序UI: Support to order columns in Search table
BUG-90570BUG-90570 HDFS-11384、 HDFS-12347HDFS-11384HDFS-12347 添加让均衡器分散 getBlocks 调用的选项,以避免 NameNode 的 rpc.CallQueueLength 峰值Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
BUG-90584BUG-90584 HBASE-19052HBASE-19052 FixedFileTrailer 应识别 branch-1.x 中的 CellComparatorImpl 类FixedFileTrailer should recognize CellComparatorImpl class in branch-1.x
BUG-90979BUG-90979 KNOX-1224KNOX-1224 用于支持 HA 中的 Atlas 的 Knox 代理 HADispatcher。Knox Proxy HADispatcher to support Atlas in HA.
BUG-91293BUG-91293 RANGER-2060RANGER-2060 使用 knox-sso 的 Knox 代理不适用于 rangerKnox proxy with knox-sso is not working for ranger
BUG-92236BUG-92236 ATLAS-2281ATLAS-2281 使用 null/非 null 筛选器保存标记/类型属性筛选器查询。Saving Tag/Type attribute filter queries with null/not null filters.
BUG-92238BUG-92238 ATLAS-2282ATLAS-2282 如果存在 25 个以上的收藏项,只在创建后刷新时才显示保存的收藏搜索。Saved favorite search appears only on refresh after creation when there are 25+ favorite searches.
BUG-92333BUG-92333 ATLAS-2286ATLAS-2286 预生成的类型“kafka_topic”不应将“topic”属性声明为唯一属性Pre-built type 'kafka_topic' should not declare 'topic' attribute as unique
BUG-92678BUG-92678 ATLAS-2276ATLAS-2276 hdfs_path 类型实体的路径值在 hive-bridge 中设置为小写。Path value for hdfs_path type entity is set to lower case from hive-bridge.
BUG-93097BUG-93097 RANGER-1944RANGER-1944 用于管理审核的操作筛选器不起作用Action filter for Admin Audit is not working
BUG-93135BUG-93135 HIVE-15874、 HIVE-18189HIVE-15874HIVE-18189 将 hive.groupby.orderby.position.alias 设置为 true 时,Hive 查询返回错误结果Hive query returning wrong results when set hive.groupby.orderby.position.alias to true
BUG-93136BUG-93136 HIVE-18189HIVE-18189 禁用 cbo 时无法按位置排序Order by position does not work when cbo is disabled
BUG-93387BUG-93387 HIVE-17600HIVE-17600 使用户可设置 OrcFile 的“enforceBufferSize”。Make OrcFile's "enforceBufferSize" user-settable.
BUG-93495BUG-93495 RANGER-1937RANGER-1937 Ranger tagsync 应处理 ENTITY_CREATE 通知,以支持 Atlas 导入功能Ranger tagsync should process ENTITY_CREATE notification, to support Atlas import feature
BUG-93512BUG-93512 PHOENIX-4466PHOENIX-4466 java.lang.RuntimeException:响应代码 500 - 执行 spark 作业以连接到 phoenix 查询服务器并加载数据java.lang.RuntimeException: response code 500 - Executing a spark job to connect to phoenix query server and load data
BUG-93801BUG-93801 HBASE-19393HBASE-19393 使用 SSL 访问 HBase UI 时的 HTTP 413 FULL 标头。HTTP 413 FULL head while accessing HBase UI using SSL.
BUG-93804BUG-93804 HIVE-17419HIVE-17419 ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS 命令显示掩码表的计算统计信息ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command shows computed stats for masked tables
BUG-93932BUG-93932 ATLAS-2320ATLAS-2320 包含查询的分类“*”引发 500 内部服务器异常。classification "*" with query throws 500 Internal server exception.
BUG-93933BUG-93933 ATLAS-2286ATLAS-2286 预生成的类型“kafka_topic”不应将“topic”属性声明为唯一属性Pre-built type 'kafka_topic' should not declare 'topic' attribute as unique
BUG-93938BUG-93938 ATLAS-2283、 ATLAS-2295ATLAS-2283ATLAS-2295 分类的 UI 更新UI updates for classifications
BUG-93941BUG-93941 ATLAS-2296、 ATLAS-2307ATLAS-2296ATLAS-2307 基本搜索增强,可以选择性地排除子类型实体和子分类类型Basic search enhancement to optionally exclude sub-type entities and sub-classification-types
BUG-93944BUG-93944 ATLAS-2318ATLAS-2318 UI:单击子标记两次会选择父标记UI: Clicking on child tag twice , parent tag is selected
BUG-93946BUG-93946 ATLAS-2319ATLAS-2319 UI:在平面结构和树结构中的标记列表内删除第 25 个位置以后的某个标记需要刷新,这样才能从该列表中删除该标记。UI: Deleting a tag which at 25+ position in the tag list in both Flat and Tree structure needs a refresh to remove the tag from the list.
BUG-93977BUG-93977 HIVE-16232HIVE-16232 QuotedIdentifier 中的列支持统计信息计算Support stats computation for column in QuotedIdentifier
BUG-94030BUG-94030 ATLAS-2332ATLAS-2332 使用具有嵌套集合数据类型的属性创建类型失败Creation of type with attributes having nested collection datatype fails
BUG-94099BUG-94099 ATLAS-2352ATLAS-2352 Atlas 服务器应提供配置来指定 Kerberos DelegationToken 的有效性Atlas server should provide configuration to specify validity for Kerberos DelegationToken
BUG-94280BUG-94280 HIVE-12785HIVE-12785 包含联合类型的视图和用于`强制转换`结构的 UDF 破坏View with union type and UDF to `cast` the struct is broken
BUG-94332BUG-94332 SQOOP-2930SQOOP-2930 Sqoop 作业 exec 不重写已保存的作业泛型属性Sqoop job exec not overriding the saved job generic properties
BUG-94428BUG-94428 不适用N/A 数据平面探查器代理 REST API Knox 支持Dataplane Profiler Agent REST API Knox support
BUG-94514BUG-94514 ATLAS-2339ATLAS-2339 UI:基本搜索结果视图中“列”内的修改也影响 DSL。UI: Modifications in "columns" in Basic search result view affects DSL also.
BUG-94515BUG-94515 ATLAS-2169ATLAS-2169 配置硬删除时删除请求失败Delete request fails when hard delete is configured
BUG-94518BUG-94518 ATLAS-2329ATLAS-2329 如果用户单击另一个错误的标记,会显示 Atlas UI 多个悬停标记Atlas UI Multiple Hovers appears if user click on another tag which is incorrect
BUG-94519BUG-94519 ATLAS-2272ATLAS-2272 使用保存搜索 API 保存已拖动列的状态。Save the state of dragged columns using save search API.
BUG-94627BUG-94627 HIVE-17731HIVE-17731 将面向外部用户的向后兼容选项添加到 HIVE-11985add a backward compat option for external users to HIVE-11985
BUG-94786BUG-94786 HIVE-6091HIVE-6091 为连接创建/关闭创建空的 pipeout 文件Empty pipeout files are created for connection create/close
BUG-94793BUG-94793 HIVE-14013HIVE-14013 描述表未正确显示 unicodeDescribe table doesn't show unicode properly
BUG-94900BUG-94900 OOZIE-2606、 OOZIE-2658、 OOZIE-2787、 OOZIE-2802OOZIE-2606OOZIE-2658OOZIE-2787OOZIE-2802 设置 spark.yarn.jars 以修复包含 Oozie 的 Spark 2.0Set spark.yarn.jars to fix Spark 2.0 with Oozie
BUG-94901BUG-94901 HBASE-19285HBASE-19285 添加每个表的延迟直方图Add per-table latency histograms
BUG-94908BUG-94908 ATLAS-1921ATLAS-1921 UI:使用实体和特征属性的搜索:UI 不执行范围检查,并允许为整型和浮点数据类型提供超限值。UI: Search using entity and trait attributes: UI doesn't perform range check and allows providing out of bounds values for integral and float data types.
BUG-95086BUG-95086 RANGER-1953RANGER-1953 对用户组页列表做了改进improvement on user-group page listing
BUG-95193BUG-95193 SLIDER-1252SLIDER-1252 python 2.7.5-58 中 Slider 代理失败并出现 SSL 验证错误Slider agent fails with SSL validation errors with python 2.7.5-58
BUG 95314BUG-95314 YARN-7699YARN-7699 queueUsagePercentage 即将用作 getApp REST api 调用的 INFqueueUsagePercentage is coming as INF for getApp REST api call
BUG-95315BUG-95315 HBASE-13947、 HBASE-14517、 HBASE-17931HBASE-13947HBASE-14517HBASE-17931 将系统表分配到具有最高版本的服务器Assign system tables to servers with highest version
BUG-95392BUG-95392 ATLAS-2421ATLAS-2421 通知更新为支持 V2 数据结构Notification updates to support V2 data structures
BUG-95476BUG-95476 RANGER-1966RANGER-1966 在某些情况下,策略引擎初始化不会创建上下文扩充器Policy engine initialization does not create context enrichers in some cases
BUG-95512BUG-95512 HIVE-18467HIVE-18467 支持整个仓库转储/加载 + 创建/删除数据库事件support whole warehouse dump / load + create/drop database events
BUG-95593BUG-95593 不适用N/A 扩展 Oozie DB 实用工具以支持 Spark2 sharelib 创建Extend Oozie DB utils to support Spark2 sharelib creation
BUG-95595BUG-95595 HIVE-15563HIVE-15563 忽略 SQLOperation.runQuery 中的非法操作状态转换异常,以公开实际异常。Ignore Illegal Operation state transition exception in SQLOperation.runQuery to expose real exception.
BUG-95685BUG-95685 ATLAS-2422ATLAS-2422 导出:支持基于类型的导出Export: Support type-based Export
BUG-95798BUG-95798 PHOENIX-2714、 PHOENIX-2724、 PHOENIX-3023、 PHOENIX-3040PHOENIX-2714PHOENIX-2724PHOENIX-3023PHOENIX-3040 不要使用 guideposts 串行执行查询Don't use guideposts for executing queries serially
BUG-95969BUG-95969 HIVE-16828、 HIVE-17063、 HIVE-18390HIVE-16828HIVE-17063HIVE-18390 分区视图失败并出现 FAILED:IndexOutOfBoundsException 索引:1,大小:1Partitioned view fails with FAILED: IndexOutOfBoundsException Index: 1, Size: 1
BUG-96019BUG-96019 HIVE-18548HIVE-18548 修复 log4j 导入Fix log4j import
BUG-96288BUG-96288 HBASE-14123、 HBASE-14135、 HBASE-17850HBASE-14123HBASE-14135HBASE-17850 向后移植 Hbase 备份/还原 2.0Backport Hbase Backup/Restore 2.0
BUG-96313BUG-96313 KNOX-1119KNOX-1119 Pac4J OAuth/OpenID 主体需可配置Pac4J OAuth/OpenID Principal Needs to be Configurable
BUG-96365BUG-96365 ATLAS-2442ATLAS-2442 对实体资源拥有只读权限的用户无法执行基本搜索User with read-only permission on entity resource not able perform basic search
BUG-96479BUG-96479 HDFS-12781HDFS-12781 数据节点关闭后,在“名称节点”UI 中,“数据节点”选项卡引发警告消息。After Datanode down, In Namenode UI Datanode tab is throwing warning message.
BUG-96502BUG-96502 RANGER-1990RANGER-1990 在 Ranger Admin 中添加单向 SSL MySQL 支持Add One-way SSL MySQL support in Ranger Admin
BUG-96718BUG-96718 ATLAS-2439ATLAS-2439 更新 Sqoop 挂钩以使用 V2 通知Update Sqoop hook to use V2 notifications
BUG-96748BUG-96748 HIVE-18587HIVE-18587 插入 DML 事件可能尝试针对目录计算校验和insert DML event may attempt to calculate a checksum on directories
BUG-96821BUG-96821 HBASE-18212HBASE-18212 在独立模式下出现本地文件系统 HBase 日志警告消息:无法在类 org.apache.hadoop.fs.FSDataInputStream 中调用“unbuffer”方法In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
BUG-96847BUG-96847 HIVE-18754HIVE-18754 REPL STATUS 应支持“with”子句REPL STATUS should support 'with' clause
BUG-96873BUG-96873 ATLAS-2443ATLAS-2443 在传出的 DELETE 消息中捕获所需的实体属性Capture required entity attributes in outgoing DELETE messages
BUG-96880BUG-96880 SPARK-23230SPARK-23230 当 hive.default.fileformat 是其他种类的文件类型时,创建文本文件表会导致 serde 错误When hive.default.fileformat is other kinds of file types, create textfile table cause a serde error
BUG-96911BUG-96911 OOZIE-2571、 OOZIE-2792、 OOZIE-2799、 OOZIE-2923OOZIE-2571OOZIE-2792OOZIE-2799OOZIE-2923 改善 Spark 选项分析Improve Spark options parsing
BUG-97100BUG-97100 RANGER-1984RANGER-1984 Hbase 审核日志记录无法显示与访问过的列相关联的所有标记Hbase audit log records may not show all tags associated with accessed column
BUG-97110BUG-97110 PHOENIX-3789PHOENIX-3789 在 postBatchMutateIndispensably 中执行跨区域索引维护调用Execute cross region index maintenance calls in postBatchMutateIndispensably
BUG-97145BUG-97145 HIVE-12245、 HIVE-17829HIVE-12245HIVE-17829 支持基于 HBase 的表的列注释Support column comments for an HBase backed table
BUG-97409BUG-97409 HADOOP-15255HADOOP-15255 LdapGroupsMapping 中的组名称支持大小写转换Upper/Lower case conversion support for group names in LdapGroupsMapping
BUG-97535BUG-97535 HIVE-18710HIVE-18710 将 inheritPerms 扩展到 Hive 2.X 中的 ACIDextend inheritPerms to ACID in Hive 2.X
BUG-97742BUG-97742 OOZIE-1624OOZIE-1624 sharelib JAR 的排除模式Exclusion pattern for sharelib JARs
BUG-97744BUG-97744 PHOENIX-3994PHOENIX-3994 索引 RPC 优先级仍依赖于 hbase-site.xml 中的控制器工厂属性Index RPC priority still depends on the controller factory property in hbase-site.xml
BUG-97787BUG-97787 HIVE-18460HIVE-18460 压缩器不会将表属性传递给 Orc 写入器Compactor doesn't pass Table properties to the Orc writer
BUG-97788BUG-97788 HIVE-18613HIVE-18613 扩展 JsonSerDe 以支持 BINARY 类型Extend JsonSerDe to support BINARY type
BUG-97899BUG-97899 HIVE-18808HIVE-18808 统计信息更新失败时使压缩更加可靠Make compaction more robust when stats update fails
BUG-98038BUG-98038 HIVE-18788HIVE-18788 清理 JDBC PreparedStatement 中的输入Clean up inputs in JDBC PreparedStatement
BUG-98383BUG-98383 HIVE-18907HIVE-18907 创建实用工具来解决 HIVE-18817 中的 acid 键索引问题Create utility to fix acid key index issue from HIVE-18817
BUG-98388BUG-98388 RANGER-1828RANGER-1828 合理的编码做法 - 在 ranger 中添加其他标头Good coding practice-add additional headers in ranger
BUG-98392BUG-98392 RANGER-2007RANGER-2007 ranger-tagsync 的 Kerberos 票证无法续订ranger-tagsync's Kerberos ticket fails to renew
BUG-98533BUG-98533 HBASE-19934、 HBASE-20008HBASE-19934HBASE-20008 由于出现 Null 指针异常,Hbase 快照还原失败Hbase snapshot restore is failing due to Null pointer exception
BUG-98552BUG-98552 HBASE-18083、 HBASE-18084HBASE-18083HBASE-18084 使大/小文件清理线程数在 HFileCleaner 中可配置Make large/small file clean thread number configurable in HFileCleaner
BUG-98705BUG-98705 KNOX-1230KNOX-1230 向 Knox 发送许多并发请求导致 URL 损坏Many Concurrent Requests to Knox causes URL Mangling
BUG-98711BUG-98711 不适用N/A 在未修改 service.xml 的情况下,NiFi 调度无法使用双向 SSLNiFi dispatch can't use two-way SSL without service.xml modifications
BUG-98880BUG-98880 OOZIE-3199OOZIE-3199 让系统属性限制可配置Let system property restriction configurable
BUG-98931BUG-98931 ATLAS-2491ATLAS-2491 更新 Hive 挂钩,以使用 Atlas v2 通知Update Hive hook to use Atlas v2 notifications
BUG-98983BUG-98983 KNOX-1108KNOX-1108 NiFiHaDispatch 不会故障转移NiFiHaDispatch not failing over
BUG-99088BUG-99088 ATLAS-2511ATLAS-2511 提供相应的选项用于选择地将数据库/表从 Hive 导入 AtlasProvide options to selectively import database / tables from Hive into Atlas
BUG-99154BUG-99154 OOZIE-2844、 OOZIE-2845、 OOZIE-2858、 OOZIE-2885OOZIE-2844OOZIE-2845OOZIE-2858OOZIE-2885 Spark 查询失败并出现“java.io.FileNotFoundException: hive-site.xml (权限被拒绝)”异常Spark query failed with "java.io.FileNotFoundException: hive-site.xml (Permission denied)" exception
BUG-99239BUG-99239 ATLAS-2462ATLAS-2462 由于未在命令中提供任何表,针对所有表执行 Sqoop 导入引发 NPESqoop import for all tables throws NPE for no table provided in command
BUG-99636BUG-99636 KNOX-1238KNOX-1238 修复网关的自定义信任存储设置Fix Custom Truststore Settings for Gateway
BUG-99650BUG-99650 KNOX-1223KNOX-1223 Zeppelin 的 Knox 代理不按预期重定向 /api/ticketZeppelin's Knox proxy doesn't redirect /api/ticket as expected
BUG-99804BUG-99804 OOZIE-2858OOZIE-2858 HiveMain、ShellMain 和 SparkMain 不应在本地覆盖属性和配置文件HiveMain, ShellMain and SparkMain should not overwrite properties and config files locally
BUG-99805BUG-99805 OOZIE-2885OOZIE-2885 运行 Spark 操作不应该需要在类路径中使用 HiveRunning Spark actions should not need Hive on the classpath
BUG-99806BUG-99806 OOZIE-2845OOZIE-2845 替换在 HiveConf 中设置变量的基于反射的代码Replace reflection-based code which sets variable in HiveConf
BUG-99807BUG-99807 OOZIE-2844OOZIE-2844 当 log4j.properties 缺失或不可读时提高 Oozie 操作的稳定性Increase stability of Oozie actions when log4j.properties is missing or not readable
RMP-9995RMP-9995 AMBARI-22222AMBARI-22222 切换 druid 以使用 /var/druid 目录而不是本地磁盘上的 /apps/druidSwitch druid to use /var/druid directory instead of /apps/druid on local disk

行为变更Behavioral changes

Apache 组件Apache Component Apache JIRAApache JIRA 摘要Summary 详细信息Details
Spark 2.3Spark 2.3 不适用N/A Apache Spark 发行说明中所述的更改Changes as documented in the Apache Spark release notes - 提供了“弃用”文档和“行为变更”指南: https://spark.apache.org/releases/spark-release-2-3-0.html#deprecations- There is a "Deprecation" document and a "Change of behavior" guide, https://spark.apache.org/releases/spark-release-2-3-0.html#deprecations

- 对于 SQL 部分,提供了另一篇详细“迁移”指南(从 2.3 到 2.2): https://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-22-to-23- For SQL part, there is another detailed "Migration" guide (from 2.2 to 2.3), https://spark.apache.org/docs/latest/sql-programming-guide.html#upgrading-from-spark-sql-22-to-23
SparkSpark HIVE-12505HIVE-12505 Spark 作业成功完成,但出现 HDFS 磁盘配额已满错误Spark job completes successfully but there is an HDFS disk quota full error 场景: 当运行 insert overwrite 命令的用户的回收站文件夹中设置了配额时运行该命令。Scenario: Running insert overwrite when a quota is set on the Trash folder of the user who runs the command.

以前的行为: 作业会成功,但无法将数据移到回收站。Previous Behavior: The job succeeds even though it fails to move the data to the Trash. 结果可能错误地包含表中以前存在的一些数据。The result can wrongly contain some of the data previously present in the table.

新行为: 如果移到回收站失败,会永久删除文件。New Behavior: When the move to the Trash folder fails, the files are permanently deleted.
Kafka 1.0Kafka 1.0 不适用N/A Apache Spark 发行说明中所述的更改Changes as documented in the Apache Spark release notes https://kafka.apache.org/10/documentation.html#upgrade_100_notable
Hive/RangerHive/ Ranger INSERT OVERWRITE 需要其他 ranger hive 策略Additional ranger hive policies required for INSERT OVERWRITE 场景: INSERT OVERWRITE 需要其他 ranger hive 策略Scenario: Additional ranger hive policies required for INSERT OVERWRITE

以前的行为: Hive INSERT OVERWRITE 查询像往常一样成功。Previous behavior: Hive INSERT OVERWRITE queries succeed as usual.

新行为: 升级到 HDP 2.6.x 之后,Hive INSERT OVERWRITE 查询意外失败并出现错误:New behavior: Hive INSERT OVERWRITE queries are unexpectedly failing after upgrading to HDP-2.6.x with the error:

编译语句时出错:失败:HiveAccessControlException 权限被拒绝: 用户 jdoe 对 /tmp/ 没有写入特权*(状态=42000,代码=40000)Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user jdoe does not have WRITE privilege on /tmp/*(state=42000,code=40000)

从 HDP-2.6.0 开始,Hive INSERT OVERWRITE 查询需要 Ranger URI 策略才能允许写入操作,即使已通过 HDFS 策略为用户授予了写入特权。As of HDP-2.6.0, Hive INSERT OVERWRITE queries require a Ranger URI policy to allow write operations, even if the user has write privilege granted through HDFS policy.

解决方法/预期的客户操作:Workaround/Expected Customer Action:

1.在 Hive 存储库下创建新策略。1. Create a new policy under the Hive repository.
2.在显示“数据库”的下拉列表中,选择“URI”。2. In the dropdown where you see Database, select URI.
3.更新路径(示例:/tmp/*)3. Update the path (Example: /tmp/*)
4.添加用户和组并保存。4. Add the users and group and save.
5.重试 insert 查询。5. Retry the insert query.
HDFSHDFS 不适用N/A HDFS 应受多个 KMS RUI 的支持HDFS should support for multiple KMS Uris 以前的行为: dfs.encryption.key.provider.uri 属性用于配置 KMS 提供程序路径。Previous Behavior: dfs.encryption.key.provider.uri property was used to configure the KMS provider path.

新行为: 现已弃用 dfs.encryption.key.provider.uri,改用 hadoop.security.key.provider.path 来配置 KMS 提供程序路径。New Behavior: dfs.encryption.key.provider.uri is now deprecated in favor of hadoop.security.key.provider.path to configure the KMS provider path.
ZeppelinZeppelin ZEPPELIN-3271ZEPPELIN-3271 用于禁用计划程序的选项Option for disabling scheduler 受影响的组件: Zeppelin-ServerComponent Affected: Zeppelin-Server

以前的行为: 以前的 Zeppelin 版本未提供用于禁用计划程序的选项。Previous Behavior: In previous releases of Zeppelin, there was no option for disabling scheduler.

新行为: 默认情况下,用户不再会看到计划程序,因为它默认已禁用。New Behavior: By default, users will no longer see scheduler, as it is disabled by default.

解决方法/预期的客户操作: 若要启用计划程序,需要通过 Ambari 在 Zeppelin 中的自定义 zeppelin 站点设置下添加值为 true 的 azeppelin.notebook.cron.enable。Workaround/Expected Customer Action: If you want to enable scheduler, you will need to add azeppelin.notebook.cron.enable with value of true under custom zeppelin site in Zeppelin settings from Ambari.

已知问题Known issues

  • Spark 2.3Spark 2.3

    • [SPARK-23523][SQL] 规则 OptimizeMetadataOnlyQuery 导致错误的结果[SPARK-23523][SQL] Incorrect result caused by the rule OptimizeMetadataOnlyQuery

    • [SPARK-23406] 流到流的自联接中存在 Bug[SPARK-23406] Bugs in stream-stream self-joins

    • 如果 Azure Data Lake Storage(第 2 代)是群集的默认存储,则 Spark 示例笔记本不可用。Spark sample notebooks are not available when Azure Data Lake Storage (Gen2) is default storage of the cluster.

  • 企业安全性套餐Enterprise Security Package

    • Spark Thrift 服务器不接受来自 ODBC 客户端的连接。Spark Thrift Server does not accept connections from ODBC clients. 解决方法步骤:Workaround steps:
      1. 创建群集后等待大约 15 分钟。Wait for about 15 minutes after cluster creation.
      2. 检查 ranger UI 中是否存在 hivesampletable_policy。Check ranger UI for existence of hivesampletable_policy.
      3. 重启 Spark 服务。Restart Spark service. 现在,STS 连接应该工作。STS connection should work now.
  • Ranger 服务检查失败的解决方法Workaround for Ranger service check failure

    • RANGER-1607:从以前的 HDP 版本升级到 HDP 2.6.2 时 Ranger 服务检查失败的解决方法。RANGER-1607: Workaround for Ranger service check failure while upgrading to HDP 2.6.2 from previous HDP versions.

      Note

      仅当已在 Ranger 中启用 SSL 时。Only when Ranger is SSL enabled.

    尝试通过 Ambari 从以前的 HDP 版本升级到 HDP 2.6.1 时会出现此问题。This issue arises when attempting to upgrade to HDP-2.6.1 from previous HDP versions through Ambari. Ambari 使用 curl 调用对 Ambari 中的 Ranger 服务执行服务检查。Ambari uses a curl call to do a service check to Ranger service in Ambari. 如果 Ambari 使用的 JDK 版本是 JDK-1.7,curl 调用将会失败并出现以下错误:If the JDK version used by Ambari is JDK-1.7, the curl call will fail with the below error:

    curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

    出现此错误的原因是 Ranger 中使用的 tomcat 版本是 Tomcat-7.0.7*。The reason for this error is the tomcat version used in Ranger is Tomcat-7.0.7*. 使用 JDK-1.7 会与 Tomcat-7.0.7* 中提供的默认加密法产生冲突。Using JDK-1.7 conflicts with default ciphers provided in Tomcat-7.0.7*.

    可通过两种方式解决此问题:You can resolve this issue in two ways:

    • 将 Ambari 中使用的 JDK 从 JDK 1.7 更新到 JDK 1.8(请参阅 Ambari 参考指南中的更改 JDK 版本部分)。Update the JDK used in Ambari from JDK-1.7 to JDK-1.8 (see the section Change the JDK Version in the Ambari Reference Guide).

    • 如果想要继续支持 JDK-1.7 环境:If you want to continue supporting a JDK-1.7 environment:

      1. 在 Ambari Ranger 配置中的 ranger-admin-site 部分添加具有以下值的属性 ranger.tomcat.ciphers:Add the property ranger.tomcat.ciphers in the ranger-admin-site section in your Ambari Ranger configuration with the below value:

        SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHASSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA

      2. 如果为 Ranger-KMS 配置了环境,请在 Ambari Ranger 配置中的 theranger-kms-site 部分添加具有以下值的属性 ranger.tomcat.ciphers:If your environment is configured for Ranger-KMS, add the property ranger.tomcat.ciphers in theranger-kms-site section in your Ambari Ranger configuration with the below value:

        SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHASSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA

    Note

    所述的值是工作示例,可能不会反映你的环境。The noted values are working examples and may not be indicative of your environment. 确保设置这些属性的方式与配置环境的方式相匹配。Ensure that the way you set these properties matches how your environment is configured.

  • RangerUI:转义在策略窗体中输入的策略条件文本RangerUI: Escape of policy condition text entered in the policy form

    受影响的组件: RangerComponent Affected: Ranger

    问题说明Description of Problem

    如果用户想要创建包含自定义策略条件和表达式的策略,或文本中含有特殊字符,则无法强制实施策略。If a user wants to create policy with custom policy conditions and the expression or text contains special characters, then policy enforcement will not work. 在数据库中保存策略之前,特殊字符将转换为 ASCII。Special characters are converted into ASCII before saving the policy into the database.

    特殊字符: & < > " ` 'Special Characters: & < > " ` '

    例如,保存策略后,条件 tags.attributes['type']='abc' 将转换为以下内容。For example, the condition tags.attributes['type']='abc' would get converted to the following once the policy is saved.

    tags.attds[&#x27;dsds&#x27;]=&#x27;cssdfs&#x27;tags.attds[&#x27;dsds&#x27;]=&#x27;cssdfs&#x27;

    可以通过在编辑模式下打开策略,查看包含这些字符的策略条件。You can see the policy condition with these characters by opening the policy in edit mode.

    解决方法Workaround

    • 选项# 1:通过 Ranger Rest API 创建/更新策略Option #1: Create/Update policy via Ranger Rest API

      REST URL: http://<host>:6080/service/plugins/policiesREST URL: http://<host>:6080/service/plugins/policies

      创建包含策略条件的策略:Creating policy with policy condition:

      以下示例将创建标记为 `tags-test` 的策略,并通过选择 select、update、create、drop、alter、index、lock、all 等所有 hive 组件权限,将该策略分配到策略条件为 astags.attr['type']=='abc' 的 `public` 组。The following example will create policy with tags as `tags-test` and assign it to `public` group with policy condition astags.attr['type']=='abc' by selecting all hive component permissions like select, update, create, drop, alter, index, lock, all.

      示例:Example:

        curl -H "Content-Type: application/json" -X POST http://localhost:6080/service/plugins/policies -u admin:admin -d '{"policyType":"0","name":"P100","isEnabled":true,"isAuditEnabled":true,"description":"","resources":{"tag":{"values":["tags-test"],"isRecursive":"","isExcludes":false}},"policyItems":[{"groups":["public"],"conditions":[{"type":"accessed-after-expiry","values":[]},{"type":"tag-expression","values":["tags.attr['type']=='abc'"]}],"accesses":[{"type":"hive:select","isAllowed":true},{"type":"hive:update","isAllowed":true},{"type":"hive:create","isAllowed":true},{"type":"hive:drop","isAllowed":true},{"type":"hive:alter","isAllowed":true},{"type":"hive:index","isAllowed":true},{"type":"hive:lock","isAllowed":true},{"type":"hive:all","isAllowed":true}]}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"service":"tagdev"}'
      

      更新包含策略条件的现有策略:Update existing policy with policy condition:

      以下示例将更新标记为 `tags-test` 的策略,并通过选择 select、update、create、drop、alter、index、lock、all 等所有 hive 组件权限,将该策略分配到策略条件为 astags.attr['type']=='abc' 的 `public` 组。The following example will update policy with tags as `tags-test` and assign it to `public` group with policy condition astags.attr['type']=='abc' by selecting all hive component permissions like select, update, create, drop, alter, index, lock, all.

      REST URL: http://<host-name>:6080/service/plugins/policies/<policy-id>REST URL: http://<host-name>:6080/service/plugins/policies/<policy-id>

      示例:Example:

        curl -H "Content-Type: application/json" -X PUT http://localhost:6080/service/plugins/policies/18 -u admin:admin -d '{"id":18,"guid":"ea78a5ed-07a5-447a-978d-e636b0490a54","isEnabled":true,"createdBy":"Admin","updatedBy":"Admin","createTime":1490802077000,"updateTime":1490802077000,"version":1,"service":"tagdev","name":"P0101","policyType":0,"description":"","resourceSignature":"e5fdb911a25aa7f77af5a9546938d9ed","isAuditEnabled":true,"resources":{"tag":{"values":["tags"],"isExcludes":false,"isRecursive":false}},"policyItems":[{"accesses":[{"type":"hive:select","isAllowed":true},{"type":"hive:update","isAllowed":true},{"type":"hive:create","isAllowed":true},{"type":"hive:drop","isAllowed":true},{"type":"hive:alter","isAllowed":true},{"type":"hive:index","isAllowed":true},{"type":"hive:lock","isAllowed":true},{"type":"hive:all","isAllowed":true}],"users":[],"groups":["public"],"conditions":[{"type":"ip-range","values":["tags.attributes['type']=abc"]}],"delegateAdmin":false}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"dataMaskPolicyItems":[],"rowFilterPolicyItems":[]}'
      
    • 选项# 2:应用 Javascript 更改Option #2: Apply Javascript changes

      更新 JS 文件的步骤:Steps to update JS file:

      1. 在 /usr/hdp/current/ranger-admin 下找到 PermissionList.js 文件Find out PermissionList.js file under /usr/hdp/current/ranger-admin

      2. 找到 renderPolicyCondtion 函数的定义(行号:404)。Find out definition of renderPolicyCondtion function (line no:404).

      3. 从该函数中删除以下行,即Remove following line from that function i.e display 函数下的行(行号:434)under display function(line no:434)

        val = _.escape(val);//Line No:460val = _.escape(val);//Line No:460

        删除上述行之后,Ranger UI 将允许你创建策略条件可以包含特殊字符的策略,并且针对同一策略执行策略评估将会成功。After removing the above line, the Ranger UI will allow you to create policies with policy condition that can contain special characters and policy evaluation will be successful for the same policy.

正在升级Upgrading

所有这些功能已在 HDInsight 3.6 中提供。All of these features are available in HDInsight 3.6. 若要获取最新版本的 Spark、Kafka 和 R Server(机器学习服务),请在创建 HDInsight 3.6 群集时选择 Spark、Kafka 和 机器学习服务版本。To get the latest version of Spark, Kafka and R Server (Machine Learning Services), please choose the Spark, Kafka, ML Services version when you create a HDInsight 3.6 cluster. 若要获取 ADLS 支持,可以选择 ADLS 存储类型作为选项。To get support for ADLS, you can choose the ADLS storage type as an option. 现有群集不会自动升级到这些版本。Existing clusters will not be upgraded to these versions automatically.

在 2018 年 6 月后创建的所有新群集将自动获取所有开源项目的 1000 多个 bug 修复。All new clusters created after June 2018 will automatically get the 1000+ bug fixes across all the open-source projects. 请遵循此指南,获取有关升级到较新 HDInsight 版本的最佳做法。Please follow this guide for best practices around upgrading to a newer HDInsight version.