Azure HDInsight 发行说明Azure HDInsight release notes

本文提供有关 最新 Azure HDInsight 版本更新的信息。This article provides information about the most recent Azure HDInsight release updates. 有关较早版本的信息,请参阅 HDInsight 发行说明存档For information on earlier releases, see HDInsight Release Notes Archive.

摘要Summary

Azure HDInsight 是 Azure 中最受企业客户青睐的开源分析服务之一。Azure HDInsight is one of the most popular services among enterprise customers for open-source analytics on Azure.

如果要订阅发行说明,请查看此 GitHub 存储库上的版本。If you would like to subscribe on release notes, watch releases on this GitHub repository.

发行日期:2020/11/18Release date: 11/18/2020

此版本适用于 HDInsight 3.6 和 HDInsight 4.0。This release applies for both HDInsight 3.6 and HDInsight 4.0. HDInsight 发行版在几天后即会在所有区域中推出。HDInsight release is made available to all regions over several days. 此处的发行日期是指在第一个区域中的发行日期。The release date here indicates the first region release date. 如果看不到以下更改,请耐心等待,几天后发行版会在你所在的区域推出。If you don't see below changes, wait for the release being live in your region in several days.

新增功能New features

为客户管理的密钥静态加密自动轮替密钥Auto key rotation for customer managed key encryption at rest

从此发行版开始,客户可使用不限 Azure KeyValut 版本的加密密钥 URL 来管理客户管理的密钥静态加密。Starting from this release, customers can use Azure KeyValut version-less encryption key URLs for customer managed key encryption at rest. 密钥过期时,HDInsight 会自动轮替密钥,或将其替换为新的版本。HDInsight will automatically rotate the keys as they expire or replaced with new versions. 请访问此处了解更多详细信息。Learn more details here.

能够为 Spark、Hadoop 和 ML 服务选择不同的 Zookeeper 虚拟机大小Ability to select different Zookeeper virtual machine sizes for Spark, Hadoop, and ML Services

HDInsight 之前不支持为 Spark、Hadoop 和 ML 服务群集类型自定义 Zookeeper 节点大小。HDInsight previously didn't support customizing Zookeeper node size for Spark, Hadoop, and ML Services cluster types. 默认情况下为 A2_v2/A2 虚拟机大小(免费提供)。It defaults to A2_v2/A2 virtual machine sizes, which are provided free of charge. 从此版本开始,你可以选择最适合自己方案的 Zookeeper 虚拟机大小。From this release, you can select a Zookeeper virtual machine size that is most appropriate for your scenario. 虚拟机大小不是 A2_v2/A2 的 Zookeeper 节点需要付费。Zookeeper nodes with virtual machine size other than A2_v2/A2 will be charged. A2_v2 和 A2 虚拟机仍免费提供。A2_v2 and A2 virtual machines are still provided free of charge.

迁移到 Azure 虚拟机规模集Moving to Azure virtual machine scale sets

HDInsight 目前使用 Azure 虚拟机来预配群集。HDInsight now uses Azure virtual machines to provision the cluster. 从此版本开始,该服务将逐渐迁移到 Azure 虚拟机规模集Starting from this release, the service will gradually migrate to Azure virtual machine scale sets. 整个过程可能需要几个月。The entire process may take months. 迁移区域和订阅后,新创建的 HDInsight 群集将在虚拟机规模集上运行,而无需客户执行任何操作。After your regions and subscriptions are migrated, newly created HDInsight clusters will run on virtual machine scale sets without customer actions. 预计不会有中断性变更。No breaking change is expected.

弃用Deprecation

弃用 HDInsight 3.6 ML 服务群集Deprecation of HDInsight 3.6 ML Services cluster

HDInsight 3.6 ML 服务群集类型将于 2020 年 12 月 31 日终止支持。HDInsight 3.6 ML Services cluster type will be end of support by December 31 2020. 2020 年 12 月 31 日之后,客户将不会创建新的 3.6 ML 服务群集。Customers won't create new 3.6 ML Services clusters after December 31 2020. 现有群集将在没有 Microsoft 支持的情况下按原样运行。Existing clusters will run as is without the support from Microsoft. 请在此处检查 HDInsight 版本的有效期限和群集类型。Check the support expiration for HDInsight versions and cluster types here.

禁用的 VM 大小Disabled VM sizes

自 2020 年 11 月 16 日起,HDInsight 将阻止新客户使用 standand_A8、standand_A9、standand_A10 和 standand_A11 VM 大小创建群集。Starting from November 16 2020, HDInsight will block new customers creating clusters using standand_A8, standand_A9, standand_A10 and standand_A11 VM sizes. 过去三个月内使用过这些 VM 大小的现有客户将不会受到影响。Existing customers who have used these VM sizes in the past three months won't be affected. 自 2021 年 1 月 9 日起,HDInsight 将阻止所有客户使用 standand_A8、standand_A9、standand_A10 和 standand_A11 VM 大小创建群集。Starting form January 9 2021, HDInsight will block all customers creating clusters using standand_A8, standand_A9, standand_A10 and standand_A11 VM sizes. 现有群集将照常运行。Existing clusters will run as is. 请考虑迁移到 HDInsight 4.0,避免出现潜在的系统/支持中断。Consider moving to HDInsight 4.0 to avoid potential system/support interruption.

行为更改Behavior changes

添加在缩放操作前进行的 NSG 规则检查Add NSG rule checking before scaling operation

HDInsight 为缩放操作添加了网络安全组 (NSG) 和用户定义的路由 (UDR) 检查。HDInsight added network security groups (NSGs) and user-defined routes (UDRs) checking with scaling operation. 除了群集创建外,还会对群集缩放执行相同的验证。The same validation is done for cluster scaling besides of cluster creation. 此验证有助于防止不可预知的错误。This validation helps prevent unpredictable errors. 如果验证未通过,则缩放会失败。If validation doesn't pass, scaling fails. 若要详细了解如何正确配置 NSG 和 UDR,请参阅 HDInsight 管理 IP 地址Learn more about how to configure NSGs and UDRs correctly, refer to HDInsight management IP addresses.

即将推出的更改Upcoming changes

即将发布的版本中将推出以下变更。The following changes will happen in upcoming releases.

默认群集版本将更改为 4.0Default cluster version will be changed to 4.0

自 2021 年 2 月起,HDInsight 群集的默认版本将从 3.6 更改为 4.0。Starting February 2021, the default version of HDInsight cluster will be changed from 3.6 to 4.0. 有关可用版本的详细信息,请参阅可用版本For more information about available versions, see available versions. 详细了解 HDInsight 4.0 中的新增功能Learn more about what is new in HDInsight 4.0

将于 2021 年 6 月 30 日终止支持 HDInsight 3.6HDInsight 3.6 end of support on June 30 2021

将终止支持 HDInsight 3.6。HDInsight 3.6 will be end of support. 自 2021 年 6 月 30 日起,客户无法创建新的 HDInsight 3.6 群集。Starting form June 30 2021, customers can't create new HDInsight 3.6 clusters. 现有群集将在没有 Microsoft 支持的情况下按原样运行。Existing clusters will run as is without the support from Microsoft. 请考虑迁移到 HDInsight 4.0,避免出现潜在的系统/支持中断。Consider moving to HDInsight 4.0 to avoid potential system/support interruption.

Bug 修复Bug fixes

HDInsight 会持续改善群集的可靠性和性能。HDInsight continues to make cluster reliability and performance improvements.

组件版本更改Component version change

此发行版未发生组件版本更改。No component version change for this release. 可以在此文档中查找 HDInsight 4.0 和 HDInsight 3.6 的当前组件版本。You can find the current component versions for HDInsight 4.0 and HDInsight 3.6 in this doc.

已知问题Known issues

防止 HDInsight 群集 VM 定期重启Prevent HDInsight cluster VMs from rebooting periodically

你可能已注意到,从 2020 年 11 月月中开始,HDInsight 群集 VM 会定期重启。Starting from mid November 2020, you may have noticed HDInsight cluster VMs getting rebooted on a regular basis. 这可能是由以下原因导致的:This could be caused by:

  1. 在群集上启用了 Clamav。Clamav is enabled on your cluster. 新的 azsec-clamav 包消耗大量的内存,这会触发节点重启。The new azsec-clamav package consumes large amount of memory that triggers node rebooting.
  2. 每天会安排一个 CRON 作业,该作业监视 Azure 服务使用的证书颁发机构 (CA) 列表的变更。A CRON job is scheduled daily that monitors for changes to the list of certificate authorities (CAs) used by Azure services. 当有新的 CA 证书可用时,该脚本会将证书添加到 JDK 信任存储,并安排重启。When a new CA certificate is available, the script adds the certificate to the JDK trust store and schedules a reboot.

对于这两个问题,HDInsight 正在为所有正在运行的群集部署修补程序并应用补丁。HDInsight is deploying fixes and applying patch for all running clusters for both issues. 若要立即应用修补程序,避免意外的 VM 重启,可以在所有群集节点上以持久脚本操作的形式运行以下脚本操作。To apply the fix immediately and avoid unexpected VMs rebooting, you can run below script actions on all cluster nodes as a persistent script action. 在修补程序和修补完成后,HDInsight 会发布另一个通知。HDInsight will post another notice after the fix and patching complete.

https://hdiconfigactions.blob.core.windows.net/linuxospatchingrebootconfigv02/replace_cacert_script.sh
https://healingscriptssa.blob.core.windows.net/healingscripts/ChangeOOMPolicyAndApplyLatestConfigForClamav.sh