消息复制任务和应用程序Message replication tasks and applications

消息复制和跨区域联合一文中所述,在服务总线实体对之间以及服务总线与其他消息源和目标之间的消息序列复制通常依赖于 Azure Functions。As explained in the message replication and cross-region federation article, replication of message sequences between pairs of Service Bus entities and between Service Bus and other message sources and targets generally leans on Azure Functions.

Azure Functions 是一种可缩放且可靠的执行环境,用于配置和运行无服务器应用程序,包括消息复制和联合任务。Azure Functions is a scalable and reliable execution environment for configuring and running serverless applications, including message replication and federation tasks.

在本概述中,你将了解 Azure Functions 针对此类应用程序提供的内置功能、可为转换任务调整和修改的代码块,以及如何配置 Azure Functions 应用程序,使其与服务总线和其他 Azure 消息传送服务完美集成。In this overview, you will learn about Azure Functions' built-in capabilities for such applications, about code blocks that you can adapt and modify for transformation tasks, and about how to configure an Azure Functions application such that it integrates ideally with Service Bus and other Azure Messaging services. 有关更多详细信息,本文将指向 Azure Functions 文档。For many details, this article will point to the Azure Functions documentation.

What is a replication task?

A replication task receives events from a source and forwards them to a target. Most replication tasks will forward events unchanged and at most perform mapping between metadata structures if the source and target protocols differ.

Replication tasks are generally stateless, meaning that they do not share state or other side-effects across sequential or parallel executions of a task. That is also true for batching and chaining, which can both be implemented on top of the existing state of a stream.

This makes replication tasks different from aggregation tasks, which are generally stateful, and are the domain of analytics frameworks and services like Azure Stream Analytics.

Replication applications and tasks in Azure Functions

In Azure Functions, a replication task is implemented using a trigger that acquires one or more input message from a configured source and an output binding that forwards messages copied from the source to a configured target.

Trigger Output
Azure Event Hubs trigger Azure Event hubs output binding
Azure Service Bus trigger Azure Service Bus output binding
Azure IoT Hub trigger Azure IoT Hub output binding
Azure Event Grid trigger Azure Event Grid output binding
Azure Queue Storage trigger Azure Queue Storage output binding
Apache Kafka trigger Apache Kafka output binding
RabbitMQ trigger RabbitMQ output binding
Azure Notification Hubs output binding
Azure SignalR service output binding
Twilio SendGrid output binding

Replication tasks are deployed as into the replication application through the same deployment methods as any other Azure Functions application. You can configure multiple tasks into the same application.

With Azure Functions Premium, multiple replication applications can share the same underlying resource pool, called an App Service Plan. That means you can easily collocate replication tasks written in .NET with replication tasks that are written in Java, for instance. That will matter if you want to take advantage of specific libraries such as Apache Camel that are only available for Java and if those are the best option for a particular integration path, even though you would commonly prefer a different language and runtime for you other replication tasks.

Whenever available, you should prefer the batch-oriented triggers over triggers that deliver individual events or messages and you should always obtain the complete event or message structure rather than rely on Azure Function's parameter binding expressions.

The name of the function should reflect the pair of source and target you are connecting, and you should prefix references to connection strings or other configuration elements in the application configuration files with that name.

Data and metadata mapping

Once you've decided on a pair of input trigger and output binding, you will have to perform some mapping between the different event or message types, unless the type of your trigger and the output is the same.

For simple replication tasks that copy messages between Event Hubs and Service Bus, you do not have to write your own code, but can lean on a utility library that is provided with the replication samples.

Retry policy

To avoid data loss during availability event on either side of a replication function, you need to configure the retry policy to be robust. Refer to the Azure Functions documentation on retries to configure the retry policy.

The policy settings chosen for the example projects in the sample repository configure an exponential backoff strategy with retry intervals from 5 seconds to 15 minutes with infinite retries to avoid data loss.

For Service Bus, review the "using retry support on top of trigger resilience" section to understand the interaction of triggers and the maximum delivery count defined for the queue.

Setting up a replication application host

A replication application is an execution host for one or more replication tasks.

It's an Azure Functions application that is configured to run either on the consumption plan or (recommended) on an Azure Functions Premium plan. All replication applications must run under a system- or user-assigned managed identity.

The linked Azure Resource Manager (ARM) templates create and configure a replication application with:

  • an Azure Storage account for tracking the replication progress and for logs,
  • a system-assigned managed identity, and
  • Azure Monitoring and Application Insights integration for monitoring.

Replication applications that must access Event Hubs bound to an Azure virtual network (VNet) must use the Azure Functions Premium plan and be configured to attach to the same VNet, which is also one of the available options.

Deploy Visualize
Azure Functions Premium Plan Deploy To Azure Visualize
Azure Functions Premium Plan with VNet Deploy To Azure Visualize

Examples

The samples repository contains several examples of replication tasks that copy events between Event Hubs and/or between Service Bus entities.

For copying event between Event Hubs, you use an Event Hub Trigger with an Event Hub output binding:

[FunctionName("telemetry")]
[ExponentialBackoffRetry(-1, "00:00:05", "00:05:00")]
public static Task Telemetry(
    [EventHubTrigger("telemetry", ConsumerGroup = "$USER_FUNCTIONS_APP_NAME.telemetry", Connection = "telemetry-source-connection")] EventData[] input,
    [EventHub("telemetry-copy", Connection = "telemetry-target-connection")] EventHubClient outputClient,
    ILogger log)
{
    return EventHubReplicationTasks.ForwardToEventHub(input, outputClient, log);
}

For copying messages between Service Bus entities, you use the Service Bus trigger and output binding:

[FunctionName("jobs-transfer")]
[ExponentialBackoffRetry(-1, "00:00:05", "00:05:00")]
public static Task JobsTransfer(
    [ServiceBusTrigger("jobs-transfer", Connection = "jobs-transfer-source-connection")] Message[] input,
    [ServiceBus("jobs", Connection = "jobs-target-connection")] IAsyncCollector<Message> output,
    ILogger log)
{
    return ServiceBusReplicationTasks.ForwardToServiceBus(input, output, log);
}

The helper methods can make it easy to replicate between Event Hubs and Service Bus:

Source Target Entry Point
Event Hub Event Hub Azure.Messaging.Replication.EventHubReplicationTasks.ForwardToEventHub
Event Hub Service Bus Azure.Messaging.Replication.EventHubReplicationTasks.ForwardToServiceBus
Service Bus Event Hub Azure.Messaging.Replication.ServiceBusReplicationTasks.ForwardToEventHub
Service Bus Service Bus Azure.Messaging.Replication.ServiceBusReplicationTasks.ForwardToServiceBus

Monitoring

To learn how you can monitor your replication app, please refer to the monitoring section of the Azure Functions documentation.

A particularly useful visual tool for monitoring replication tasks is the Application Insights Application Map, which is automatically generated from the captured monitoring information and allows exploring the reliability and performance of the replication task source and target transfers.

For immediate diagnostic insights, you can work with the Live Metrics portal tool, which provides low latency visualization of log details.

Next steps