创建按翻转窗口运行管道的触发器Create a trigger that runs a pipeline on a tumbling window

适用于:是 Azure 数据工厂是 Azure Synapse Analytics(预览版)APPLIES TO: yesAzure Data Factory yesAzure Synapse Analytics (Preview)

本文提供了创建、启动和监视翻转窗口触发器的步骤。This article provides steps to create, start, and monitor a tumbling window trigger. 有关触发器和支持的类型的一般信息,请参阅管道执行和触发器For general information about triggers and the supported types, see Pipeline execution and triggers.

翻转窗口触发器是一类可以在保留状态的同时按周期性的时间间隔(从指定的开始时间算起)触发的触发器。Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. 翻转窗口是一系列固定大小、非重叠且连续的时间间隔。Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. 翻转窗口触发器与管道存在一对一关系,一个这样的触发器只能引用一个管道。A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline.

数据工厂 UIData Factory UI

  1. 若要在数据工厂 UI 中创建翻转窗口触发器,请选择“触发器”**** 选项卡,然后选择“新建”****。To create a tumbling window trigger in the Data Factory UI, select the Triggers tab, and then select New.
  2. 在触发器配置窗格打开后,选择“翻转窗口”****,然后定义翻转窗口触发器属性。After the trigger configuration pane opens, select Tumbling Window, and then define your tumbling window trigger properties.
  3. 完成后,选择“保存”****。When you're done, select Save.

在 Azure 门户中创建翻转窗口触发器

翻转窗口触发器类型属性Tumbling window trigger type properties

翻转窗口具有以下触发器类型属性:A tumbling window has the following trigger type properties:

{
    "name": "MyTriggerName",
    "properties": {
        "type": "TumblingWindowTrigger",
        "runtimeState": "<<Started/Stopped/Disabled - readonly>>",
        "typeProperties": {
            "frequency": <<Minute/Hour>>,
            "interval": <<int>>,
            "startTime": "<<datetime>>",
            "endTime: <<datetime – optional>>,
            "delay": <<timespan – optional>>,
            "maxConcurrency": <<int>> (required, max allowed: 50),
            "retryPolicy": {
                "count": <<int - optional, default: 0>>,
                "intervalInSeconds": <<int>>,
            },
            "dependsOn": [
                {
                    "type": "TumblingWindowTriggerDependencyReference",
                    "size": <<timespan – optional>>,
                    "offset": <<timespan – optional>>,
                    "referenceTrigger": {
                        "referenceName": "MyTumblingWindowDependency1",
                        "type": "TriggerReference"
                    }
                },
                {
                    "type": "SelfDependencyTumblingWindowTriggerReference",
                    "size": <<timespan – optional>>,
                    "offset": <<timespan>>
                }
            ]
        },
        "pipeline": {
            "pipelineReference": {
                "type": "PipelineReference",
                "referenceName": "MyPipelineName"
            },
            "parameters": {
                "parameter1": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowStartTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "parameter2": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowEndTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "parameter3": "https://mydemo.chinacloudsites.cn/api/demoapi"
            }
        }
    }
}

下表概述了与翻转窗口触发器中的循环和计划相关的主要 JSON 元素:The following table provides a high-level overview of the major JSON elements that are related to recurrence and scheduling of a tumbling window trigger:

JSON 元素JSON element 说明Description 类型Type 允许的值Allowed values 必须Required
typetype 触发器的类型。The type of the trigger. 类型为固定值“TumblingWindowTrigger”。The type is the fixed value "TumblingWindowTrigger". StringString "TumblingWindowTrigger""TumblingWindowTrigger" Yes
runtimeStateruntimeState 触发器运行时的当前状态。The current state of the trigger run time.
注意:此元素是 <readOnly>。Note: This element is <readOnly>.
StringString “Started”、“Stopped”、“Disabled”"Started," "Stopped," "Disabled" Yes
frequencyfrequency 一个字符串,表示触发器重复出现的频率单位(分钟或小时)。A string that represents the frequency unit (minutes or hours) at which the trigger recurs. 如果 startTime 日期值粒度比 frequency 值更细,则会在计算窗口边界时考虑 startTime 日期。If the startTime date values are more granular than the frequency value, the startTime dates are considered when the window boundaries are computed. 例如:如果 frequency 值为每小时,startTime 值为 2017-09-01T10:10:10Z,则第一个窗口为 (2017-09-01T10:10:10Z, 2017-09-01T11:10:10Z)。For example, if the frequency value is hourly and the startTime value is 2017-09-01T10:10:10Z, the first window is (2017-09-01T10:10:10Z, 2017-09-01T11:10:10Z). StringString “minute”、“hour”"minute," "hour" Yes
intervalinterval 一个正整数,表示 frequency 值对应的时间间隔,决定了触发器的运行频率。A positive integer that denotes the interval for the frequency value, which determines how often the trigger runs. 例如,如果 interval 为 3,frequency 为“hour”,则触发器每 3 小时重复触发一次。For example, if the interval is 3 and the frequency is "hour," the trigger recurs every 3 hours.
注意:最小窗口间隔为 5 分钟。Note: The minimum window interval is 5 minutes.
IntegerInteger 正整数。A positive integer. Yes
startTimestartTime 第一个匹配项,可以是过去的时间。The first occurrence, which can be in the past. 第一个触发器间隔是 (startTime, startTime + interval)。The first trigger interval is (startTime, startTime + interval). DateTimeDateTime 一个日期时间值。A DateTime value. Yes
endTimeendTime 最后一个匹配项,可以是过去的时间。The last occurrence, which can be in the past. DateTimeDateTime 一个日期时间值。A DateTime value. Yes
delaydelay 延迟窗口数据处理开始的时间量。The amount of time to delay the start of data processing for the window. 管道运行在预期的执行时间加上 delay 的量之后启动。The pipeline run is started after the expected execution time plus the amount of delay. delay 的定义是:在预期的执行时间过后,触发器在触发新的运行之前等待的时间。The delay defines how long the trigger waits past the due time before triggering a new run. delay 不改变窗口 startTimeThe delay doesn’t alter the window startTime. 例如,值为 00:10:00 的 delay 意味着 10 分钟的延迟。For example, a delay value of 00:10:00 implies a delay of 10 minutes. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
一个时间跨度值,默认值为 00:00:00。A timespan value where the default is 00:00:00. No
maxConcurrencymaxConcurrency 同时针对已就绪窗口触发的触发器运行数。The number of simultaneous trigger runs that are fired for windows that are ready. 例如,若要每小时回填,昨天的运行会产生 24 个 windows。For example, to back fill hourly runs for yesterday results in 24 windows. 如果 maxConcurrency = 10,则仅针对头 10 个窗口 (00:00-01:00 - 09:00-10:00) 触发触发器事件。If maxConcurrency = 10, trigger events are fired only for the first 10 windows (00:00-01:00 - 09:00-10:00). 在头 10 个触发的管道运行完成后,将针对接下来的 10 个窗口 (10:00-11:00 - 19:00-20:00) 触发触发器运行。After the first 10 triggered pipeline runs are complete, trigger runs are fired for the next 10 windows (10:00-11:00 - 19:00-20:00). 继续进行 maxConcurrency = 10 的此示例,如果有 10 个窗口就绪,则总共有 10 个管道运行。Continuing with this example of maxConcurrency = 10, if there are 10 windows ready, there are 10 total pipeline runs. 如果只有 1 个窗口就绪,则只有 1 管道运行。If there's only 1 window ready, there's only 1 pipeline run. IntegerInteger 一个介于 1 到 50 之间的整数。An integer between 1 and 50. Yes
retryPolicy: CountretryPolicy: Count 将管道运行标记为“失败”之前的重试次数。The number of retries before the pipeline run is marked as "Failed." IntegerInteger 一个整数,其默认值为 0(不重试)。An integer, where the default is 0 (no retries). No
retryPolicy: intervalInSecondsretryPolicy: intervalInSeconds 重试之间的延迟(以秒为单位指定)。The delay between retry attempts specified in seconds. IntegerInteger 秒数,其默认值为 30。The number of seconds, where the default is 30. No
dependsOn: typedependsOn: type TumblingWindowTriggerReference 的类型。The type of TumblingWindowTriggerReference. 如果设置了依赖项,则为必需。Required if a dependency is set. StringString "TumblingWindowTriggerDependencyReference", "SelfDependencyTumblingWindowTriggerReference""TumblingWindowTriggerDependencyReference", "SelfDependencyTumblingWindowTriggerReference" No
dependsOn: sizedependsOn: size 依赖项翻转窗口的大小。The size of the dependency tumbling window. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
一个正的时间跨度值,其中默认值为子触发器的窗口大小A positive timespan value where the default is the window size of the child trigger No
dependsOn: offsetdependsOn: offset 依赖项触发器的偏移量。The offset of the dependency trigger. TimespanTimespan
(hh:mm:ss)(hh:mm:ss)
在自我依赖项中必须为负的时间跨度值。A timespan value that must be negative in a self-dependency. 如果未指定任何值,则该窗口与触发器本身相同。If no value specified, the window is the same as the trigger itself. 自我依赖项:是Self-Dependency: Yes
其他:否Other: No

Note

发布翻转窗口触发器后,无法编辑“间隔”和“频率”**** ****。After a tumbling window trigger is published, interval and frequency can't be edited.

WindowStart 和 WindowEnd 系统变量WindowStart and WindowEnd system variables

可以在管道定义中(即,作为查询的一部分),使用翻转窗口触发器的 WindowStartWindowEnd 系统变量。You can use the WindowStart and WindowEnd system variables of the tumbling window trigger in your pipeline definition (that is, for part of a query). 触发器定义中将系统变量作为参数传递给管道。Pass the system variables as parameters to your pipeline in the trigger definition. 下面的示例演示如何将这些变量作为参数传递:The following example shows you how to pass these variables as parameters:

{
    "name": "MyTriggerName",
    "properties": {
        "type": "TumblingWindowTrigger",
            ...
        "pipeline": {
            "pipelineReference": {
                "type": "PipelineReference",
                "referenceName": "MyPipelineName"
            },
            "parameters": {
                "MyWindowStart": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowStartTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                },
                "MyWindowEnd": {
                    "type": "Expression",
                    "value": "@{concat('output',formatDateTime(trigger().outputs.windowEndTime,'-dd-MM-yyyy-HH-mm-ss-ffff'))}"
                }
            }
        }
    }
}

若要在管道定义中使用 WindowStartWindowEnd 系统变量值,请相应地使用“MyWindowStart”和“MyWindowEnd”参数。To use the WindowStart and WindowEnd system variable values in the pipeline definition, use your "MyWindowStart" and "MyWindowEnd" parameters, accordingly.

回填方案中的窗口执行顺序Execution order of windows in a backfill scenario

有多个窗口需要执行时(尤其是在回填的情况下),窗口的执行顺序是确定的,将根据时间间隔的新旧按从旧到新的顺序执行。When there are multiple windows up for execution (especially in a backfill scenario), the order of execution for windows is deterministic, from oldest to newest intervals. 当前无法修改此行为。Currently, this behavior can't be modified.

现有 TriggerResource 元素Existing TriggerResource elements

以下各点适用于现有 TriggerResource 元素:The following points apply to existing TriggerResource elements:

  • 如果触发器的 frequency 元素(或窗口大小)的值更改,则已处理窗口的状态不会** 重置。If the value for the frequency element (or window size) of the trigger changes, the state of the windows that are already processed is not reset. 触发器会根据新的窗口大小继续触发一段时间(从上次执行开始计算)。The trigger continues to fire for the windows from the last window that it executed by using the new window size.
  • 如果触发器的 endTime 元素的值更改(添加或更新),则已处理窗口的状态不会** 重置。If the value for the endTime element of the trigger changes (added or updated), the state of the windows that are already processed is not reset. 触发器会遵循新的 endTime 值。The trigger honors the new endTime value. 如果新的 endTime 值在已执行的窗口之前,则触发器会停止。If the new endTime value is before the windows that are already executed, the trigger stops. 否则,触发器会在遇到新的 endTime 值停止。Otherwise, the trigger stops when the new endTime value is encountered.

翻转窗口触发器依赖项Tumbling window trigger dependency

若要确保仅在数据工厂中成功执行另一个翻转窗口触发器后才执行翻转窗口触发器,请创建翻转窗口触发器依赖项If you want to make sure that a tumbling window trigger is executed only after the successful execution of another tumbling window trigger in the data factory, create a tumbling window trigger dependency.

Azure PowerShell 示例Sample for Azure PowerShell

Note

本文进行了更新,以便使用新的 Azure PowerShell Az 模块。This article has been updated to use the new Azure PowerShell Az module. 你仍然可以使用 AzureRM 模块,至少在 2020 年 12 月之前,它将继续接收 bug 修补程序。You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. 若要详细了解新的 Az 模块和 AzureRM 兼容性,请参阅新 Azure Powershell Az 模块简介To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. 有关 Az 模块安装说明,请参阅安装 Azure PowerShellFor Az module installation instructions, see Install Azure PowerShell.

本部分展示了如何使用 Azure PowerShell 创建、启动和监视触发器。This section shows you how to use Azure PowerShell to create, start, and monitor a trigger.

  1. 在 C:\ADFv2QuickStartPSH\ 文件夹中创建一个名为 MyTrigger.json 的且包含以下内容的 JSON 文件:Create a JSON file named MyTrigger.json in the C:\ADFv2QuickStartPSH\ folder with the following content:

    Important

    保存 JSON 文件之前,请将 startTime 元素的值设置为当前 UTC 时间。Before you save the JSON file, set the value of the startTime element to the current UTC time. endTime 元素的值设置为比当前 UTC 时间早一小时。Set the value of the endTime element to one hour past the current UTC time.

    {
      "name": "PerfTWTrigger",
      "properties": {
        "type": "TumblingWindowTrigger",
        "typeProperties": {
          "frequency": "Minute",
          "interval": "15",
          "startTime": "2017-09-08T05:30:00Z",
          "delay": "00:00:01",
          "retryPolicy": {
            "count": 2,
            "intervalInSeconds": 30
          },
          "maxConcurrency": 50
        },
        "pipeline": {
          "pipelineReference": {
            "type": "PipelineReference",
            "referenceName": "DynamicsToBlobPerfPipeline"
          },
          "parameters": {
            "windowStart": "@trigger().outputs.windowStartTime",
            "windowEnd": "@trigger().outputs.windowEndTime"
          }
        },
        "runtimeState": "Started"
      }
    }
    
  2. 使用 Set-AzDataFactoryV2Trigger cmdlet 创建一个触发器:Create a trigger by using the Set-AzDataFactoryV2Trigger cmdlet:

    Set-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger" -DefinitionFile "C:\ADFv2QuickStartPSH\MyTrigger.json"
    
  3. 使用 Get-AzDataFactoryV2Trigger cmdlet 确认触发器的状态为 StoppedConfirm that the status of the trigger is Stopped by using the Get-AzDataFactoryV2Trigger cmdlet:

    Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  4. 使用 Start-AzDataFactoryV2Trigger cmdlet 启动触发器:Start the trigger by using the Start-AzDataFactoryV2Trigger cmdlet:

    Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  5. 使用 Get-AzDataFactoryV2Trigger cmdlet 确认触发器的状态为 StartedConfirm that the status of the trigger is Started by using the Get-AzDataFactoryV2Trigger cmdlet:

    Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name "MyTrigger"
    
  6. 使用 Get-AzDataFactoryV2TriggerRun cmdlet 在 Azure PowerShell 中获取触发器运行。Get the trigger runs in Azure PowerShell by using the Get-AzDataFactoryV2TriggerRun cmdlet. 若要获取有关触发器运行的信息,请定期执行以下命令。To get information about the trigger runs, execute the following command periodically. 更新 TriggerRunStartedAfterTriggerRunStartedBefore 值以匹配触发器定义中的值:Update the TriggerRunStartedAfter and TriggerRunStartedBefore values to match the values in your trigger definition:

    Get-AzDataFactoryV2TriggerRun -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -TriggerName "MyTrigger" -TriggerRunStartedAfter "2017-12-08T00:00:00" -TriggerRunStartedBefore "2017-12-08T01:00:00"
    

若要在 Azure 门户中监视触发器运行和管道运行,请参阅监视管道运行To monitor trigger runs and pipeline runs in the Azure portal, see Monitor pipeline runs.

后续步骤Next steps