使用 Apache Kafka 应用程序中的 Azure 事件中心Use Azure Event Hubs from Apache Kafka applications

事件中心提供 Kafka 终结点,现有的基于 Kafka 的应用程序可将该终结点用作运行你自己的 Kafka 群集的替代方案。Event Hubs provides a Kafka endpoint that can be used by your existing Kafka based applications as an alternative to running your own Kafka cluster. 事件中心支持 Apache Kafka 协议 1.0 及更高版本,并且可与现有 Kafka 应用程序(包括 MirrorMaker)配合使用。Event Hubs supports Apache Kafka protocol 1.0 and later, and works with your existing Kafka applications, including MirrorMaker.

适用于 Kafka 的事件中心可提供什么?What does Event Hubs for Kafka provide?

适用于 Kafka 功能的事件中心基于 Azure 事件中心提供协议头,该协议头与 Kafka 1.0 和更高版本二进制兼容,可以读取和写入 Kafka 主题。The Event Hubs for Kafka feature provides a protocol head on top of Azure Event Hubs that is binary compatible with Kafka versions 1.0 and later for both reading from and writing to Kafka topics. 可以通过应用程序开始使用 Kafka 终结点,而无需更改任何代码,只需进行最小的配置更改。You may start using the Kafka endpoint from your applications with no code change but a minimal configuration change. 更新配置中的连接字符串以指向事件中心公开的 Kafka 终结点,而不是指向 Kafka 群集。You update the connection string in configurations to point to the Kafka endpoint exposed by your event hub instead of pointing to your Kafka cluster. 然后,可以开始将使用 Kafka 协议的应用程序中的事件流式传输到事件中心。Then, you can start streaming events from your applications that use the Kafka protocol into Event Hubs. 此集成还支持 Kafka Connect 等框架,该框架目前处于预览状态。This integration also supports frameworks like Kafka Connect, which is currently in preview.

从概念上讲,Kafka 和事件中心几乎完全相同:它们都是为流式传输数据而生成的分区日志。Conceptually Kafka and Event Hubs are nearly identical: they're both partitioned logs built for streaming data. 下表映射 Kafka 和事件中心之间的概念。The following table maps concepts between Kafka and Event Hubs.

Kafka 和事件中心概念映射Kafka and Event Hub conceptual mapping

Kafka 概念Kafka Concept 事件中心概念Event Hubs Concept
群集Cluster 命名空间Namespace
主题Topic 事件中心Event Hub
分区Partition 分区Partition
使用者组Consumer Group 使用者组Consumer Group
OffsetOffset OffsetOffset

Kafka 和事件中心之间的主要区别Key differences between Kafka and Event Hubs

Apache Kafka 是可在所选的任意位置运行的软件,而事件中心是类似 Azure Blob 存储的云服务。While Apache Kafka is software, which you can run wherever you choose, Event Hubs is a cloud service similar to Azure Blob Storage. 没有需要管理的服务器或网络,也没有需要配置的代理。There are no servers or networks to manage and no brokers to configure. 你可创建一个命名空间(它是主题所在的 FQDN),然后在该命名空间内创建事件中心或主题。You create a namespace, which is an FQDN in which your topics live, and then create Event Hubs or topics within that namespace. 有关事件中心和命名空间的详细信息,请参阅事件中心功能For more information about Event Hubs and namespaces, see Event Hubs features. 作为云服务,事件中心使用单一稳定的虚拟 IP 地址作为终结点,因此客户端无需了解群集中代理或计算机的情况。As a cloud service, Event Hubs uses a single stable virtual IP address as the endpoint, so clients don't need to know about the brokers or machines within a cluster.

事件中心的规模由购买的吞吐量单位数量控制,每个吞吐量单位可提供每秒 1 MB 的速度或每秒 1,000 个事件的流入量。Scale in Event Hubs is controlled by how many throughput units you purchase, with each throughput unit entitling you to 1 MB per second, or 1000 events per second of ingress. 默认情况下,如果达到限制,事件中心会使用自动膨胀功能按比例增加吞吐量单位;此功能也可与适用于 Kafka 功能的事件中心配合使用。By default, Event Hubs scales up throughput units when you reach your limit with the Auto-Inflate feature; this feature also works with the Event Hubs for Kafka feature.

安全性和身份验证Security and authentication

每次你发布或使用来自用于 Kafka 的事件中心的事件时,客户端都会尝试访问事件中心资源。Every time you publish or consume events from an Event Hubs for Kafka, your client is trying to access the Event Hubs resources. 你希望确保使用已授权的实体来访问资源。You want to ensure that the resources are accessed using an authorized entity. 在客户端上使用 Apache Kafka 协议时,可以使用 SASL 机制设置用于身份验证和加密的配置。When using Apache Kafka protocol with your clients, you can set your configuration for authentication and encryption using the SASL mechanisms. 使用用于 Kafka 的事件中心时,需要进行 TLS 加密(因为使用事件中心传输的所有数据均经过 TLS 加密)。When using Event Hubs for Kafka requires the TLS-encryption (as all data in transit with Event Hubs is TLS encrypted). 可以在配置文件中指定 SASL_SSL 选项来完成此设置。It can be done specifying the SASL_SSL option in your configuration file.

Azure 事件中心提供了多个选项来授予对安全资源的访问权限。Azure Event Hubs provides multiple options to authorize access to your secure resources.

  • OAuthOAuth
  • 共享访问签名 (SAS)Shared access signature (SAS)

OAuthOAuth

事件中心会与 Azure Active Directory (Azure AD) 集成,后者提供了与 OAuth 2.0 兼容的集中式授权服务器。Event Hubs integrates with Azure Active Directory (Azure AD), which provides a OAuth 2.0 compliant centralized authorization server. 使用 Azure AD,可以通过基于角色的访问控制 (RBAC) 向客户端标识授予细粒度权限。With Azure AD, you can use role-based access control (RBAC) to grant fine grained permissions to your client identities. 可以指定“SASL_SSL”作为协议,并指定“OAUTHBEARER”作为机制,通过这种方式将此功能用于 Kafka 客户端。You can use this feature with your Kafka clients by specifying SASL_SSL for the protocol and OAUTHBEARER for the mechanism. 有关 RBAC 角色和范围访问级别的详细信息,请参阅使用 Azure AD 授予访问权限For details about RBAC roles and levels for scoping access, see Authorize access with Azure AD.

bootstrap.servers=NAMESPACENAME.servicebus.chinacloudapi.cn:9093
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
sasl.login.callback.handler.class=CustomAuthenticateCallbackHandler;

共享访问签名 (SAS)Shared Access Signature (SAS)

事件中心还提供了共享访问签名 (SAS),方便你对用于 Kafka 的事件中心资源进行委派访问。Event Hubs also provides the Shared Access Signatures (SAS) for delegated access to Event Hubs for Kafka resources. 与 SAS 相比,使用 OAuth 2.0 基于令牌的机制授予访问权限具有更好的安全性和易用性。Authorizing access using OAuth 2.0 token-based mechanism provides superior security and ease of use over SAS. 内置角色还可以消除基于 ACL 的授权(用户必须对其进行维护和管理)的需要。The built-in roles can also eliminate the need for ACL-based authorization, which has to be maintained and managed by the user. 可以指定“SASL_SSL”作为协议,并指定“PLAIN”作为机制,通过这种方式将此功能用于 Kafka 客户端。You can use this feature with your Kafka clients by specifying SASL_SSL for the protocol and PLAIN for the mechanism.

bootstrap.servers=NAMESPACENAME.servicebus.chinacloudapi.cn:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";

示例Samples

有关创建事件中心并使用 SAS 或 OAuth 对其进行访问的分步说明教程,请参阅快速入门:使用 Kafka 协议通过事件中心进行数据流式传输For a tutorial with step-by-step instructions to create an event hub and access it using SAS or OAuth, see Quickstart: Data streaming with Event Hubs using the Kafka protocol.

如需更多示例来演示如何在用于 Kafka 的事件中心使用 OAuth,请参阅 GitHub 上的示例For more samples that show how to use OAuth with Event Hubs for Kafka, see samples on GitHub.

Kafka 可用的其他事件中心功能Other Event Hubs features available for Kafka

借助适用于 Kafka 功能的事件中心,你能够使用一种协议写入并使用另一种协议读取,以便当前 Kafka 生成方可通过 Kafka 继续发布,而你可使用 Azure 流分析或 Azure Functions 等事件中心添加读取器。The Event Hubs for Kafka feature enables you to write with one protocol and read with another, so that your current Kafka producers can continue publishing via Kafka, and you can add readers with Event Hubs, such as Azure Stream Analytics or Azure Functions. 此外,捕获等事件中心功能也在用于 Kafka 的事件中心功能中使用。Additionally, Event Hubs features such as Capture also work with the Event Hubs for Kafka feature.

尚不支持的功能Features that are not yet supported

下面是尚不支持的 Kafka 功能列表:Here is the list of Kafka features that are not yet supported:

  • 幂等生成方Idempotent producer
  • 事务Transaction
  • 压缩Compression
  • 基于大小的保留Size-based retention
  • 日志压缩Log compaction
  • 将分区添加到现有主题Adding partitions to an existing topic
  • HTTP Kafka API 支持HTTP Kafka API support
  • Kafka StreamKafka Streams

后续步骤Next steps

本文介绍了适用于 Kafka 的事件中心。This article provided an introduction to Event Hubs for Kafka. 若要了解详细信息,请参阅针对 Azure 事件中心的 Apache Kafka 开发人员指南To learn more, see Apache Kafka developer guide for Azure Event Hubs.