通过 Apache Kafka on HDInsight 实现数据的高可用性High availability of your data with Apache Kafka on HDInsight

了解如何为 Apache Kafka 主题配置分区副本,充分利用基础硬件机架配置。Learn how to configure partition replicas for Apache Kafka topics to take advantage of underlying hardware rack configuration. 此配置可确保存储在 HDInsight 上的 Apache Kafka 中的数据的可用性。This configuration ensures the availability of data stored in Apache Kafka on HDInsight.

Apache Kafka 的容错域和更新域Fault and update domains with Apache Kafka

容错域是 Azure 数据中心基础硬件的逻辑分组。A fault domain is a logical grouping of underlying hardware in an Azure data center. 每个容错域共享公用电源和网络交换机。Each fault domain shares a common power source and network switch. 在 HDInsight 群集中实现节点的虚拟机和托管磁盘跨这些容错域分布。The virtual machines and managed disks that implement the nodes within an HDInsight cluster are distributed across these fault domains. 此体系结构可限制物理硬件故障造成的潜在影响。This architecture limits the potential impact of physical hardware failures.

每个 Azure 区域都有特定数量的容错域。Each Azure region has a specific number of fault domains. 如需域的列表及其所含容错域的数目,请参阅可用性集文档。For a list of domains and the number of fault domains they contain, see the Availability sets documentation.

重要

Kafka 不识别容错域。Kafka is not aware of fault domains. 在 Kafka 中创建主题时,可能会将所有分区副本存储在同一容错域中。When you create a topic in Kafka, it may store all partition replicas in the same fault domain. 为了解决此问题,HDInsight 提供了 Kafka 分区重新均衡工具To solve this problem, HDInsight provides the Kafka partition rebalance tool.

何时重新均衡分区副本When to rebalance partition replicas

若要确保 Kafka 数据的最高可用性,应该在以下时间为主题重新均衡分区副本:To ensure the highest availability of your Kafka data, you should rebalance the partition replicas for your topic at the following times:

  • 创建新主题或分区时When a new topic or partition is created

  • 纵向扩展群集时When you scale up a cluster

复制因子Replication factor

重要

建议使用包含三个容错域的 Azure 区域,并使用 3 作为复制因子。We recommend using an Azure region that contains three fault domains, and using a replication factor of 3.

如果必须使用只包含两个容错域的区域,请使用 4 作为复制因子,将副本均衡地分布到两个容错域中。If you must use a region that contains only two fault domains, use a replication factor of 4 to spread the replicas evenly across the two fault domains.

如需创建主题和设置复制因子的示例,请参阅 HDInsight 上的 Apache Kafka 入门文档。For an example of creating topics and setting the replication factor, see the Start with Apache Kafka on HDInsight document.

如何重新均衡分区副本How to rebalance partition replicas

使用 Apache Kafka 分区重新均衡工具以重新均衡所选主题。Use the Apache Kafka partition rebalance tool to rebalance selected topics. 必须通过 SSH 会话运行此工具,以便连接到 Kafka 群集的头节点。This tool must be ran from an SSH session to the head node of your Kafka cluster.

若要详细了解如何使用 SSH 连接到 HDInsight,请参阅将 SSH 与 HDInsight 配合使用文档。For more information on connecting to HDInsight using SSH, see the Use SSH with HDInsight document.

后续步骤Next steps