创建按翻转窗口运行管道的触发器Create a trigger that runs a pipeline on a tumbling window

适用于: Azure 数据工厂 Azure Synapse Analytics

本文提供了创建、启动和监视翻转窗口触发器的步骤。This article provides steps to create, start, and monitor a tumbling window trigger. 有关触发器和支持的类型的一般信息,请参阅管道执行和触发器For general information about triggers and the supported types, see Pipeline execution and triggers.

翻转窗口触发器是一类可以在保留状态的同时按周期性的时间间隔(从指定的开始时间算起)触发的触发器。Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. 翻转窗口是一系列固定大小、非重叠且连续的时间间隔。Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. 翻转窗口触发器与管道存在一对一关系,一个这样的触发器只能引用一个管道。A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline. 翻转窗口触发器是计划触发器的权重更高的替代方案,为复杂方案提供一套功能(依赖于其他翻转窗口触发器重新运行失败的作业以及为管道设置用户重试)。Tumbling window trigger is a more heavy weight alternative for schedule trigger offering a suite of features for complex scenarios(dependency on other tumbling window triggers, rerunning a failed job and set user retry for pipelines). 若要进一步了解计划触发器和翻转窗口触发器之间的差异,请访问此处To further understand the difference between schedule trigger and tumbling window trigger, please visit here.

数据工厂 UIData Factory UI

  1. 若要在数据工厂 UI 中创建翻转窗口触发器,请选择“触发器”选项卡,然后选择“新建”。To create a tumbling window trigger in the Data Factory UI, select the Triggers tab, and then select New.
  2. 在触发器配置窗格打开后,选择“翻转窗口”,然后定义翻转窗口触发器属性。After the trigger configuration pane opens, select Tumbling Window, and then define your tumbling window trigger properties.
  3. 完成后,选择“保存”。When you're done, select Save.

在 Azure 门户中创建翻转窗口触发器

翻转窗口触发器类型属性Tumbling window trigger type properties

翻转窗口具有以下触发器类型属性:A tumbling window has the following trigger type properties:

{
    "name": "MyTriggerName",
    "properties": {
        "type": "TumblingWindowTrigger",
        "runtimeState": "<<Started/Stopped/Disabled - readonly>>",
        "typeProperties": {
            "frequency": <<Minute/Hour>>,
            "interval": <<int>>,
            "startTime": "<<datetime>>",
            "endTime": <<datetime – optional>>,
            "delay": <<timespan – optional>>,
            "maxConcurrency": <<int>> (required, max allowed: 50),
            "retryPolicy": {
                "count": <<int - optional, default: 0>>,
                "intervalInSeconds": <<int>>,
            },
            "dependsOn": [
                {
                    "type": "TumblingWindowTriggerDependencyReference",
                    "size": <<timespan – optional>>,
                    "offset": <<timespan – optional>>,
                    "referenceTrigger": {
                        "referenceName": "MyTumblingWindowDependency1",
                        "type": "TriggerReference"
                    }
                },
                {
                    "type": "SelfDependencyTumblingWindowTriggerReference",
                    "size": <<timespan – optional>>,
                    "offset": <<timespan>>
                }
            ]
        },
        "pipeline": {
            "pipelineReference": {
                "type": "PipelineReference",
                "referenceName": "MyPipelineName"
            },
            "parameters": {
                "parameter1": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowStartTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "parameter2": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowEndTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "parameter3": "https://mydemo.chinacloudsites.cn/api/demoapi"
            }
        }
    }
}

下表概述了与翻转窗口触发器中的循环和计划相关的主要 JSON 元素:The following table provides a high-level overview of the major JSON elements that are related to recurrence and scheduling of a tumbling window trigger:

JSON 元素JSON element 说明Description 类型Type 允许的值Allowed values 必选Required
typetype 触发器的类型。The type of the trigger. 类型为固定值“TumblingWindowTrigger”。The type is the fixed value "TumblingWindowTrigger". StringString "TumblingWindowTrigger""TumblingWindowTrigger" Yes
runtimeStateruntimeState 触发器运行时的当前状态。The current state of the trigger run time.
注意:此元素是 <readOnly>。Note: This element is <readOnly>.
StringString “Started”、“Stopped”、“Disabled”"Started," "Stopped," "Disabled" Yes
frequencyfrequency 一个字符串,表示触发器重复出现的频率单位(分钟或小时)。A string that represents the frequency unit (minutes or hours) at which the trigger recurs. 如果 startTime 日期值粒度比 frequency 值更细,则会在计算窗口边界时考虑 startTime 日期。If the startTime date values are more granular than the frequency value, the startTime dates are considered when the window boundaries are computed. 例如:如果 frequency 值为每小时,startTime 值为 2017-09-01T10:10:10Z,则第一个窗口为 (2017-09-01T10:10:10Z, 2017-09-01T11:10:10Z)。For example, if the frequency value is hourly and the startTime value is 2017-09-01T10:10:10Z, the first window is (2017-09-01T10:10:10Z, 2017-09-01T11:10:10Z). StringString “minute”、“hour”"minute," "hour" Yes
intervalinterval 一个正整数,表示 frequency 值对应的时间间隔,决定了触发器的运行频率。A positive integer that denotes the interval for the frequency value, which determines how often the trigger runs. 例如,如果 interval 为 3,frequency 为“hour”,则触发器每 3 小时重复触发一次。For example, if the interval is 3 and the frequency is "hour," the trigger recurs every 3 hours.
注意:最小窗口间隔为 5 分钟。Note: The minimum window interval is 5 minutes.
IntegerInteger 正整数。A positive integer. Yes
startTimestartTime 第一个匹配项,可以是过去的时间。The first occurrence, which can be in the past. 第一个触发器间隔是 (startTime, startTime + interval)。The first trigger interval is (startTime, startTime + interval). DateTimeDateTime 一个日期时间值。A DateTime value. Yes
endTimeendTime 最后一个匹配项,可以是过去的时间。The last occurrence, which can be in the past. DateTimeDateTime 一个日期时间值。A DateTime value. Yes
delaydelay 延迟窗口数据处理开始的时间量。The amount of time to delay the start of data processing for the window. 管道运行在预期的执行时间加上 delay 的量之后启动。The pipeline run is started after the expected execution time plus the amount of delay. delay 的定义是:在预期的执行时间过后,触发器在触发新的运行之前等待的时间。The delay defines how long the trigger waits past the due time before triggering a new run. delay 不改变窗口 startTimeThe delay doesn’t alter the window startTime. 例如,值为 00:10:00 的 delay 意味着 10 分钟的延迟。For example, a delay value of 00:10:00 implies a delay of 10 minutes. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
一个时间跨度值,默认值为 00:00:00。A timespan value where the default is 00:00:00. No
maxConcurrencymaxConcurrency 同时针对已就绪窗口触发的触发器运行数。The number of simultaneous trigger runs that are fired for windows that are ready. 例如,若要每小时回填,昨天的运行会产生 24 个 windows。For example, to back fill hourly runs for yesterday results in 24 windows. 如果 maxConcurrency = 10,则仅针对头 10 个窗口 (00:00-01:00 - 09:00-10:00) 触发触发器事件。If maxConcurrency = 10, trigger events are fired only for the first 10 windows (00:00-01:00 - 09:00-10:00). 在头 10 个触发的管道运行完成后,将针对接下来的 10 个窗口 (10:00-11:00 - 19:00-20:00) 触发触发器运行。After the first 10 triggered pipeline runs are complete, trigger runs are fired for the next 10 windows (10:00-11:00 - 19:00-20:00). 继续进行 maxConcurrency = 10 的此示例,如果有 10 个窗口就绪,则总共有 10 个管道运行。Continuing with this example of maxConcurrency = 10, if there are 10 windows ready, there are 10 total pipeline runs. 如果只有 1 个窗口就绪,则只有 1 管道运行。If there's only 1 window ready, there's only 1 pipeline run. IntegerInteger 一个介于 1 到 50 之间的整数。An integer between 1 and 50. Yes
retryPolicy: CountretryPolicy: Count 将管道运行标记为“失败”之前的重试次数。The number of retries before the pipeline run is marked as "Failed." IntegerInteger 一个整数,其默认值为 0(不重试)。An integer, where the default is 0 (no retries). No
retryPolicy: intervalInSecondsretryPolicy: intervalInSeconds 重试之间的延迟(以秒为单位指定)。The delay between retry attempts specified in seconds. IntegerInteger 秒数,其默认值为 30。The number of seconds, where the default is 30. No
dependsOn: typedependsOn: type TumblingWindowTriggerReference 的类型。The type of TumblingWindowTriggerReference. 如果设置了依赖项,则为必需。Required if a dependency is set. StringString "TumblingWindowTriggerDependencyReference", "SelfDependencyTumblingWindowTriggerReference""TumblingWindowTriggerDependencyReference", "SelfDependencyTumblingWindowTriggerReference" No
dependsOn: sizedependsOn: size 依赖项翻转窗口的大小。The size of the dependency tumbling window. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
一个正的时间跨度值,其中默认值为子触发器的窗口大小A positive timespan value where the default is the window size of the child trigger No
dependsOn: offsetdependsOn: offset 依赖项触发器的偏移量。The offset of the dependency trigger. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
在自我依赖项中必须为负的时间跨度值。A timespan value that must be negative in a self-dependency. 如果未指定任何值,则该窗口与触发器本身相同。If no value specified, the window is the same as the trigger itself. 自我依赖项:是Self-Dependency: Yes
其他:否Other: No

备注

发布翻转窗口触发器后,无法编辑“间隔”和“频率” 。After a tumbling window trigger is published, interval and frequency can't be edited.

WindowStart 和 WindowEnd 系统变量WindowStart and WindowEnd system variables

可以在 管道 定义中(即,作为查询的一部分),使用翻转窗口触发器的 WindowStartWindowEnd 系统变量。You can use the WindowStart and WindowEnd system variables of the tumbling window trigger in your pipeline definition (that is, for part of a query). 触发器 定义中将系统变量作为参数传递给管道。Pass the system variables as parameters to your pipeline in the trigger definition. 下面的示例演示如何将这些变量作为参数传递:The following example shows you how to pass these variables as parameters:

{
    "name": "MyTriggerName",
    "properties": {
        "type": "TumblingWindowTrigger",
            ...
        "pipeline": {
            "pipelineReference": {
                "type": "PipelineReference",
                "referenceName": "MyPipelineName"
            },
            "parameters": {
                "MyWindowStart": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowStartTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "MyWindowEnd": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowEndTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                }
            }
        }
    }
}

若要在管道定义中使用 WindowStartWindowEnd 系统变量值,请相应地使用“MyWindowStart”和“MyWindowEnd”参数。To use the WindowStart and WindowEnd system variable values in the pipeline definition, use your "MyWindowStart" and "MyWindowEnd" parameters, accordingly.

回填方案中的窗口执行顺序Execution order of windows in a backfill scenario

如果触发器的 startTime 为过去时间,那么根据公式 M=(CurrentTime- TriggerStartTime)/TumblingWindowSize,触发器将在执行未来运行之前生成 {M} backfill(past) 次并行运行,以保证触发器并发性。If the startTime of trigger is in the past, then based on this formula, M=(CurrentTime- TriggerStartTime)/TumblingWindowSize, the trigger will generate {M} backfill(past) runs in parallel, honoring trigger concurrency, before executing the future runs. 窗口的执行顺序是确定的(从最旧到最新的时间间隔)。The order of execution for windows is deterministic, from oldest to newest intervals. 当前无法修改此行为。Currently, this behavior can't be modified.

现有 TriggerResource 元素Existing TriggerResource elements

以下几点适用于现有 TriggerResource 元素的更新:The following points apply to update of existing TriggerResource elements:

  • 创建触发器后,便无法更改触发器的 frequency 元素(或窗口大小)与 interval 元素的值。The value for the frequency element (or window size) of the trigger along with interval element cannot be changed once the trigger is created. 这是 triggerRun 重新运行和依赖项评估正常运行所必需的This is required for proper functioning of triggerRun reruns and dependency evaluations
  • 如果触发器的 endTime 元素的值更改(添加或更新),则已处理窗口的状态不会重置。If the value for the endTime element of the trigger changes (added or updated), the state of the windows that are already processed is not reset. 触发器会遵循新的 endTime 值。The trigger honors the new endTime value. 如果新的 endTime 值在已执行的窗口之前,则触发器会停止。If the new endTime value is before the windows that are already executed, the trigger stops. 否则,触发器会在遇到新的 endTime 值停止。Otherwise, the trigger stops when the new endTime value is encountered.

用户分配的管道重试次数User assigned retries of pipelines

如果出现管道故障,翻转窗口触发器可以使用相同的输入参数自动重试引用的管道的执行,而无需用户干预。In case of pipeline failures, tumbling window trigger can retry the execution of the referenced pipeline automatically, using the same input parameters, without the user intervention. 可以在触发器定义中使用属性“retryPolicy”来指定此项。This can be specified using the property "retryPolicy" in the trigger definition.

翻转窗口触发器依赖项Tumbling window trigger dependency

若要确保仅在数据工厂中成功执行另一个翻转窗口触发器后才执行翻转窗口触发器,请创建翻转窗口触发器依赖项If you want to make sure that a tumbling window trigger is executed only after the successful execution of another tumbling window trigger in the data factory, create a tumbling window trigger dependency.

取消翻转窗口运行Cancel tumbling window run

如果特定窗口处于“正在等待”、“正在等待依赖项”或“正在运行”状态,则可以取消翻转窗口触发器的运行。You can cancel runs for a tumbling window trigger, if the specific window is in Waiting, Waiting on Dependency, or Running state

  • 如果窗口处于“正在运行”状态,则取消关联的“管道运行”,然后触发器运行会被标记为“已取消”If the window is in Running state, cancel the associated Pipeline Run, and the trigger run will be marked as Canceled afterwards
  • 如果窗口处于“正在等待”或“正在等待依赖项”状态,则可以取消该窗口的监视:If the window is in Waiting or Waiting on Dependency state, you can cancel the window from Monitoring:

从“监视”页取消翻转窗口触发器

还可以重新运行已取消的窗口。You can also rerun a canceled window. 重新运行会采用触发器的最新发布定义,并且指定窗口的依赖项会在重新运行时重新进行评估The rerun will take the latest published definitions of the trigger, and dependencies for the specified window will be re-evaluated upon rerun

为之前取消的运行重新运行翻转窗口触发器

Azure PowerShell 示例Sample for Azure PowerShell

备注

本文已经过更新,以便使用 Azure Az PowerShell 模块。This article has been updated to use the Azure Az PowerShell module. 若要与 Azure 交互,建议使用的 PowerShell 模块是 Az PowerShell 模块。The Az PowerShell module is the recommended PowerShell module for interacting with Azure. 若要开始使用 Az PowerShell 模块,请参阅安装 Azure PowerShellTo get started with the Az PowerShell module, see Install Azure PowerShell. 若要了解如何迁移到 Az PowerShell 模块,请参阅 将 Azure PowerShell 从 AzureRM 迁移到 AzTo learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

本部分展示了如何使用 Azure PowerShell 创建、启动和监视触发器。This section shows you how to use Azure PowerShell to create, start, and monitor a trigger.

  1. 在 C:\ADFv2QuickStartPSH\ 文件夹中创建一个名为 MyTrigger.json 的且包含以下内容的 JSON 文件:Create a JSON file named MyTrigger.json in the C:\ADFv2QuickStartPSH\ folder with the following content:

    重要

    保存 JSON 文件之前,请将 startTime 元素的值设置为当前 UTC 时间。Before you save the JSON file, set the value of the startTime element to the current UTC time. endTime 元素的值设置为比当前 UTC 时间早一小时。Set the value of the endTime element to one hour past the current UTC time.

    {
      "name": "PerfTWTrigger",
      "properties": {
        "type": "TumblingWindowTrigger",
        "typeProperties": {
          "frequency": "Minute",
          "interval": "15",
          "startTime": "2017-09-08T05:30:00Z",
          "delay": "00:00:01",
          "retryPolicy": {
            "count": 2,
            "intervalInSeconds": 30
          },
          "maxConcurrency": 50
        },
        "pipeline": {
          "pipelineReference": {
            "type": "PipelineReference",
            "referenceName": "DynamicsToBlobPerfPipeline"
          },
          "parameters": {
            "windowStart": "@trigger().outputs.windowStartTime",
            "windowEnd": "@trigger().outputs.windowEndTime"
          }
        },
        "runtimeState": "Started"
      }
    }
    
  2. 使用 Set-AzDataFactoryV2Trigger cmdlet 创建一个触发器:Create a trigger by using the Set-AzDataFactoryV2Trigger cmdlet:

    Set-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger" -DefinitionFile "C:\ADFv2QuickStartPSH\MyTrigger.json"
    
  3. 使用 Get-AzDataFactoryV2Trigger cmdlet 确认触发器的状态为 StoppedConfirm that the status of the trigger is Stopped by using the Get-AzDataFactoryV2Trigger cmdlet:

    Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  4. 使用 Start-AzDataFactoryV2Trigger cmdlet 启动触发器:Start the trigger by using the Start-AzDataFactoryV2Trigger cmdlet:

    Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  5. 使用 Get-AzDataFactoryV2Trigger cmdlet 确认触发器的状态为 StartedConfirm that the status of the trigger is Started by using the Get-AzDataFactoryV2Trigger cmdlet:

    Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  6. 使用 Get-AzDataFactoryV2TriggerRun cmdlet 在 Azure PowerShell 中获取触发器运行。Get the trigger runs in Azure PowerShell by using the Get-AzDataFactoryV2TriggerRun cmdlet. 若要获取有关触发器运行的信息,请定期执行以下命令。To get information about the trigger runs, execute the following command periodically. 更新 TriggerRunStartedAfterTriggerRunStartedBefore 值以匹配触发器定义中的值:Update the TriggerRunStartedAfter and TriggerRunStartedBefore values to match the values in your trigger definition:

    Get-AzDataFactoryV2TriggerRun -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -TriggerName "MyTrigger" -TriggerRunStartedAfter "2017-12-08T00:00:00" -TriggerRunStartedBefore "2017-12-08T01:00:00"
    

若要在 Azure 门户中监视触发器运行和管道运行,请参阅监视管道运行To monitor trigger runs and pipeline runs in the Azure portal, see Monitor pipeline runs.

后续步骤Next steps