创建运行管道的触发器来响应自定义事件(预览版)Create a trigger that runs a pipeline in response to a custom event (Preview)

适用于: Azure 数据工厂 Azure Synapse Analytics

本文介绍可以在数据工厂管道中创建的自定义事件触发器。This article describes the Custom Event Triggers that you can create in your Data Factory pipelines.

事件驱动的体系结构 (EDA) 是一种常见数据集成模式,其中涉及到事件的生成、检测、消耗和响应。Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. 数据集成方案通常要求数据工厂客户基于发生的特定事件触发管道。Data integration scenarios often require Data Factory customers to trigger pipelines based on certain events happening. 数据工厂与 Azure 事件网格的原生集成现在涵盖了自定义事件:客户向事件网格主题发送任意事件,然后数据工厂将订阅并侦听该主题,并相应地触发管道。Data Factory native integration with Azure Event Grid now covers Custom Events: customers send arbitrary events to an event grid topic, and Data Factory subscribes and listens to the topic and triggers pipelines accordingly.

备注

本文中所介绍的集成依赖于 Azure 事件网格The integration described in this article depends on Azure Event Grid. 请确保订阅已注册事件网格资源提供程序。Make sure that your subscription is registered with the Event Grid resource provider. 有关详细信息,请参阅资源提供程序和类型For more info, see Resource providers and types. 你必须能够执行 Microsoft.EventGrid/eventSubscriptions/* 操作。You must be able to do the Microsoft.EventGrid/eventSubscriptions/* action. 此操作属于 EventGrid EventSubscription 参与者内置角色。This action is part of the EventGrid EventSubscription Contributor built-in role.

此外,将管道参数和自定义事件触发器相结合,客户可以在管道运行中分析和引用自定义数据有效负载。Furthermore, combining pipeline parameters and Custom Event Trigger, customers can parse and reference custom data payload in pipeline runs. 自定义事件有效负载中的数据字段是自由格式的 JSON 键-值结构,可让客户最大程度地控制事件驱动的管道运行。data field in Custom Event payload is a free-form json key-value structure, giving customers maximum control over the event driven pipeline runs.

重要

在参数化中引用的键会不时地在自定义事件有效负载中丢失。Every so often, a key referenced in parameterization may be missing in custom event payload. 触发器运行将会失败并出现错误,指出由于属性 keyName 不存在,无法对表达式求值。The trigger run will fail with an error, stating that expression cannot be evaluated because property keyName doesn't exist. 事件不会触发任何管道运行。No pipeline run will be triggered by the event.

设置事件网格自定义主题Setup Event Grid Custom Topic

若要在数据工厂中使用自定义事件触发器,首先需要在事件网格中设置一个自定义主题To use the Custom Event Trigger in Data Factory, you need to first set up a Custom Topic in Event Grid. 该工作流与使用存储事件触发器时不同,对于后者,数据工厂将为你设置主题。The workflow is different from Storage Event Trigger, where Data Factory will set up the topic for you. 此处你需要在 Azure 事件网格中导航并自行创建主题。Here you need to navigate the Azure Event Grid and create the topic yourself. 有关如何创建自定义主题的详细信息,请参阅 Azure 事件网格门户教程CLI 教程For more information on how to create the custom topic, see Azure Event Grid Portal Tutorials and CLI Tutorials

数据工厂预期事件会遵循网格事件架构Data Factories expect the events to follow Event Grid event schema. 请确保事件有效负载具有以下字段。Make sure event payloads have following fields.

[
  {
    "topic": string,
    "subject": string,
    "id": string,
    "eventType": string,
    "eventTime": string,
    "data":{
      object-unique-to-each-publisher
    },
    "dataVersion": string,
    "metadataVersion": string
  }
]

数据工厂 UIData Factory UI

本部分介绍如何在 Azure 数据工厂用户界面中创建存储事件触发器。This section shows you how to create a storage event trigger within the Azure Data Factory User Interface.

  1. 切换到“编辑”选项卡(显示为铅笔符号)。Switch to the Edit tab, shown with a pencil symbol.

  2. 在菜单上选择“触发器”,然后选择“新建/编辑” 。Select Trigger on the menu, then select New/Edit.

  3. 在“添加触发器”页上,选择“选择触发器...”,然后选择“+新建” 。On the Add Triggers page, select Choose trigger..., then select +New.

  4. 选择触发器类型“自定义事件”Select trigger type Custom Events

    数据工厂 UI 中用于新建自定义事件触发器的“创作”页的屏幕截图。

  5. 从 Azure 订阅下拉列表中选择你的自定义主题,或手动输入事件主题范围。Select your custom topic from the Azure subscription dropdown or manually enter the event topic scope.

    备注

    若要新建自定义事件触发器或修改现有的自定义事件触发器,用于登录到数据工厂并发布存储事件触发器的 Azure 帐户必须对主题拥有相应的基于角色的访问控制 (Azure RBAC) 权限。To create a new or modify an existing Custom Event Trigger, the Azure account used to log into Data Factory and publish the storage event trigger must have appropriate role based access control (Azure RBAC) permission on topic. 不需要其他权限:Azure 数据工厂的服务主体不需要对事件网格拥有特殊权限。No additional permission is required: Service Principal for the Azure Data Factory does not need special permission to Event Grid. 有关访问控制的详细信息,请参阅基于角色的访问控制部分。For more information about access control, see Role based access control section.

  6. 使用“主题开头为”和“主题结尾为”属性可以筛选你要对其触发管道的事件。 The Subject begins with and Subject ends with properties allow you to filter events for which you want to trigger pipeline. 这两个属性都是可选的。Both properties are optional.

  7. 使用“+ 新建”添加要用作筛选依据的 事件类型Use + New to add Event Types you want to filter on. 自定义事件触发器对列表采用 OR 关系:如果某个自定义事件具有与此处所列任一项匹配的 eventType 属性,则会触发管道运行。Custom Event trigger employee an OR relationship for the list: if a custom event has an eventType property that matches any listed here, it will trigger a pipeline run. 事件类型不区分大小写。The event type is case insensitive. 例如,在以下屏幕截图中,触发器匹配主题以 factories 开头的所有 copycompletedcopysucceeded 事件For instance, in the screenshot below, the trigger matches all copycompleted or copysucceeded events with subject starts with factories

    “编辑触发器”页的屏幕截图,用于说明数据工厂 UI 中的“事件类型”和“主题”筛选。

  8. 自定义事件触发器可以分析自定义数据有效负载并将其发送到管道中。Custom event trigger can parse and send custom data payload to your pipeline. 首先创建管道参数,并在“参数”页上填写值。First create the pipeline parameters, and fill in the values on the Parameters page. 使用 @triggerBody().event.data.keyName 格式分析数据有效负载,并将值传递给管道参数。Use format @triggerBody().event.data.keyName to parse the data payload, and pass values to pipeline parameters. 有关详细说明,请参阅在管道中引用触发器元数据自定义事件触发器中的系统变量For detailed explanation, see Reference Trigger Metadata in Pipelines and System Variables in Custom Event Trigger

    管道“参数”设置的屏幕截图。

    用于在自定义事件中引用数据有效负载的“参数”页的屏幕截图。

  9. 完成后,单击“确定”。Click OK once you are done.

JSON 架构JSON schema

下表概述了与自定义事件触发器相关的架构元素:The following table provides an overview of the schema elements that are related to custom event triggers:

JSON 元素JSON Element 说明Description 类型Type 允许的值Allowed Values 必需Required
作用域scope 事件网格主题的 Azure 资源管理器资源 ID。The Azure Resource Manager resource ID of the event grid topic. 字符串String Azure 资源管理器 IDAzure Resource Manager ID Yes
eventsevents 导致此触发器触发的事件的类型。The type of events that cause this trigger to fire. 字符串数组Array of strings 是,至少应有一个值Yes, at least one value is expected
subjectBeginsWithsubjectBeginsWith 主题字段必须使用为要触发的触发器提供的模式开头。Subject field must begin with the pattern provided for the trigger to fire. 例如,factories 只会对以 factories 开头的事件主题触发触发器。For example, factories only fires the trigger for event subject starting with factories. 字符串String No
subjectEndsWithsubjectEndsWith 主题字段必须使用为要触发的触发器提供的模式结尾。Subject field must end with the pattern provided for the trigger to fire. 字符串String No

基于角色的访问控制Role-based access control

Azure 数据工厂使用 Azure 基于角色的访问控制 (Azure RBAC) 确保严格禁止通过未经授权的访问来侦听自定义事件、订阅自定义事件更新以及触发链接到自定义事件的管道。Azure Data Factory uses Azure role-based access control (Azure RBAC) to ensure that unauthorized access to listen to, subscribe to updates from, and trigger pipelines linked to custom events, are strictly prohibited.

  • 若要成功新建自定义事件触发器或更新现有的自定义事件触发器,登录到数据工厂的 Azure 帐户需要对相关存储帐户拥有相应访问权限。To successfully create a new or update an existing Custom Event Trigger, the Azure account signed into the Data Factory needs to have appropriate access to the relevant storage account. 否则,操作将会失败并出现“拒绝访问”消息。Otherwise, the operation with fail with Access Denied.
  • 数据工厂不需要对事件网格拥有特殊权限,并且你无需为操作的数据工厂服务主体分配特殊的 Azure RBAC 权限。Data Factory needs no special permission to your Event Grid, and you do not need to assign special Azure RBAC permission to Data Factory service principal for the operation.

具体而言,客户需要对 /subscriptions/####/resourceGroups//####/providers/Microsoft.EventGrid/topics/someTopics 拥有 Microsoft.EventGrid/EventSubscriptions/Write 权限Specifically, customer needs Microsoft.EventGrid/EventSubscriptions/Write permission on /subscriptions/####/resourceGroups//####/providers/Microsoft.EventGrid/topics/someTopics

后续步骤Next steps