通过 PowerShell 在 Azure 数据工厂中使用“执行 SSIS 包”活动运行 SSIS 包
适用于:Azure 数据工厂 Azure Synapse Analytics
本文介绍如何使用 PowerShell 和“执行 SSIS 包”活动在 Azure 数据工厂管道中运行 SQL Server Integration Services (SSIS) 包。
先决条件
注意
建议使用 Azure Az PowerShell 模块与 Azure 交互。 请参阅安装 Azure PowerShell 以开始使用。 若要了解如何迁移到 Az PowerShell 模块,请参阅 将 Azure PowerShell 从 AzureRM 迁移到 Az。
如果还没有 Azure-SSIS Integration Runtime (IR),请按照以下文章中的分步说明创建 IR:教程:预配 Azure-SSIS IR。
创建包含 Azure-SSIS IR 的数据工厂
可以使用已预配 Azure-SSIS IR 的现有数据工厂,或者创建包含 Azure-SSIS IR 的新数据工厂。 按照教程:通过 PowerShell 将 SSIS 包部署到 Azure 中的分步说明,创建包含 Azure-SSIS IR 的新 ADF。
使用“执行 SSIS 包”活动创建管道
在此步骤中创建包含“执行 SSIS 包”活动的管道。 该活动运行 SSIS 包。
在
C:\ADF\RunSSISPackage
文件夹中创建一个名为RunSSISPackagePipeline.json
的 JSON 文件,使其包含类似于以下示例的内容。重要
在保存该文件之前,请替换对象名称、说明、路径、属性或参数值、密码及其他变量值。
{ "name": "RunSSISPackagePipeline", "properties": { "activities": [{ "name": "MySSISActivity", "description": "My SSIS package/activity description", "type": "ExecuteSSISPackage", "typeProperties": { "connectVia": { "referenceName": "MyAzureSSISIR", "type": "IntegrationRuntimeReference" }, "executionCredential": { "domain": "MyExecutionDomain", "username": "MyExecutionUsername", "password": { "type": "SecureString", "value": "MyExecutionPassword" } }, "runtime": "x64", "loggingLevel": "Basic", "packageLocation": { "type": "SSISDB", "packagePath": "MyFolder/MyProject/MyPackage.dtsx" }, "environmentPath": "MyFolder/MyEnvironment", "projectParameters": { "project_param_1": { "value": "123" }, "project_param_2": { "value": { "value": "@pipeline().parameters.MyProjectParameter", "type": "Expression" } } }, "packageParameters": { "package_param_1": { "value": "345" }, "package_param_2": { "value": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "myAKV", "type": "LinkedServiceReference" }, "secretName": "MyPackageParameter" } } }, "projectConnectionManagers": { "MyAdonetCM": { "username": { "value": "MyConnectionUsername" }, "password": { "value": { "type": "SecureString", "value": "MyConnectionPassword" } } } }, "packageConnectionManagers": { "MyOledbCM": { "username": { "value": { "value": "@pipeline().parameters.MyConnectionUsername", "type": "Expression" } }, "password": { "value": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "myAKV", "type": "LinkedServiceReference" }, "secretName": "MyConnectionPassword", "secretVersion": "MyConnectionPasswordVersion" } } } }, "propertyOverrides": { "\\Package.MaxConcurrentExecutables": { "value": 8, "isSensitive": false } } }, "policy": { "timeout": "0.01:00:00", "retry": 0, "retryIntervalInSeconds": 30 } }] } }
若要执行存储在文件系统/Azure 文件存储中的包,请输入包和日志位置属性的值,如下所示:
{ { { { "packageLocation": { "type": "File", "packagePath": "//MyStorageAccount.file.core.chinacloudapi.cn/MyFileShare/MyPackage.dtsx", "typeProperties": { "packagePassword": { "type": "SecureString", "value": "MyEncryptionPassword" }, "accessCredential": { "domain": "Azure", "username": "MyStorageAccount", "password": { "type": "SecureString", "value": "MyAccountKey" } } } }, "logLocation": { "type": "File", "logPath": "//MyStorageAccount.file.core.chinacloudapi.cn/MyFileShare/MyLogFolder", "typeProperties": { "accessCredential": { "domain": "Azure", "username": "MyStorageAccount", "password": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "myAKV", "type": "LinkedServiceReference" }, "secretName": "MyAccountKey" } } } } } } } }
若要执行存储在文件系统/Azure 文件存储中的项目内的包,请输入包位置属性的值,如下所示:
{ { { { "packageLocation": { "type": "File", "packagePath": "//MyStorageAccount.file.core.chinacloudapi.cn/MyFileShare/MyProject.ispac:MyPackage.dtsx", "typeProperties": { "packagePassword": { "type": "SecureString", "value": "MyEncryptionPassword" }, "accessCredential": { "domain": "Azure", "userName": "MyStorageAccount", "password": { "type": "SecureString", "value": "MyAccountKey" } } } } } } } }
若要执行嵌入包,请输入包位置属性的值,如下所示:
{ { { { "packageLocation": { "type": "InlinePackage", "typeProperties": { "packagePassword": { "type": "SecureString", "value": "MyEncryptionPassword" }, "packageName": "MyPackage.dtsx", "packageContent":"My compressed/uncompressed package content", "packageLastModifiedDate": "YYYY-MM-DDTHH:MM:SSZ UTC-/+HH:MM" } } } } } }
若要执行存储在包存储中的包,请输入包和配置位置属性的值,如下所示:
{ { { { "packageLocation": { "type": "PackageStore", "packagePath": "myPackageStore/MyFolder/MyPackage", "typeProperties": { "packagePassword": { "type": "SecureString", "value": "MyEncryptionPassword" }, "accessCredential": { "domain": "Azure", "username": "MyStorageAccount", "password": { "type": "SecureString", "value": "MyAccountKey" } }, "configurationPath": "//MyStorageAccount.file.core.chinacloudapi.cn/MyFileShare/MyConfiguration.dtsConfig", "configurationAccessCredential": { "domain": "Azure", "userName": "MyStorageAccount", "password": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "myAKV", "type": "LinkedServiceReference" }, "secretName": "MyAccountKey" } } } } } } } }
在 Azure PowerShell 中,切换到
C:\ADF\RunSSISPackage
文件夹。若要创建管道 RunSSISPackagePipeline,请运行 Set-AzDataFactoryV2Pipeline cmdlet。
$DFPipeLine = Set-AzDataFactoryV2Pipeline -DataFactoryName $DataFactory.DataFactoryName ` -ResourceGroupName $ResGrp.ResourceGroupName ` -Name "RunSSISPackagePipeline" -DefinitionFile ".\RunSSISPackagePipeline.json"
下面是示例输出:
PipelineName : Adfv2QuickStartPipeline ResourceGroupName : <resourceGroupName> DataFactoryName : <dataFactoryName> Activities : {CopyFromBlobToBlob} Parameters : {[inputPath, Microsoft.Azure.Management.DataFactory.Models.ParameterSpecification], [outputPath, Microsoft.Azure.Management.DataFactory.Models.ParameterSpecification]}
运行管道
使用 Invoke-AzDataFactoryV2Pipeline cmdlet 运行该管道。 此 cmdlet 返回管道运行 ID,用于将来的监视。
$RunId = Invoke-AzDataFactoryV2Pipeline -DataFactoryName $DataFactory.DataFactoryName `
-ResourceGroupName $ResGrp.ResourceGroupName `
-PipelineName $DFPipeLine.Name
监视管道
运行以下 PowerShell 脚本,持续检查管道运行状态,直到完成数据复制为止。 在 PowerShell 窗口中复制或粘贴以下脚本,然后按 Enter。
while ($True) {
$Run = Get-AzDataFactoryV2PipelineRun -ResourceGroupName $ResGrp.ResourceGroupName `
-DataFactoryName $DataFactory.DataFactoryName `
-PipelineRunId $RunId
if ($Run) {
if ($run.Status -ne 'InProgress') {
Write-Output ("Pipeline run finished. The status is: " + $Run.Status)
$Run
break
}
Write-Output "Pipeline is running...status: InProgress"
}
Start-Sleep -Seconds 10
}
还可使用 Azure 门户监视管道。 有关分步说明,请参阅监视管道。
使用触发器计划管道
在上一步骤中,已按需运行了管道。 还可创建一个计划触发器,按计划(例如每小时或每天)运行管道。
在
C:\ADF\RunSSISPackage
文件夹中创建一个名为MyTrigger.json
的 JSON 文件,使其包含以下内容:{ "properties": { "name": "MyTrigger", "type": "ScheduleTrigger", "typeProperties": { "recurrence": { "frequency": "Hour", "interval": 1, "startTime": "2017-12-07T00:00:00-08:00", "endTime": "2017-12-08T00:00:00-08:00" } }, "pipelines": [{ "pipelineReference": { "type": "PipelineReference", "referenceName": "RunSSISPackagePipeline" }, "parameters": {} }] } }
在 Azure PowerShell 中,切换到
C:\ADF\RunSSISPackage
文件夹。运行 Set-AzDataFactoryV2Trigger cmdlet,以创建触发器。
Set-AzDataFactoryV2Trigger -ResourceGroupName $ResGrp.ResourceGroupName ` -DataFactoryName $DataFactory.DataFactoryName ` -Name "MyTrigger" -DefinitionFile ".\MyTrigger.json"
默认情况下,触发器处于停止状态。 运行 Start-AzDataFactoryV2Trigger cmdlet 以启动该触发器。
Start-AzDataFactoryV2Trigger -ResourceGroupName $ResGrp.ResourceGroupName ` -DataFactoryName $DataFactory.DataFactoryName ` -Name "MyTrigger"
通过运行 Get-AzDataFactoryV2Trigger cmdlet 确认该触发器已启动。
Get-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName ` -DataFactoryName $DataFactoryName ` -Name "MyTrigger"
在下一小时后运行以下命令。 例如,如果当前时间为下午 3:25 (UTC),则在下午 4:00 (UTC) 运行该命令。
Get-AzDataFactoryV2TriggerRun -ResourceGroupName $ResourceGroupName ` -DataFactoryName $DataFactoryName ` -TriggerName "MyTrigger" ` -TriggerRunStartedAfter "2017-12-06" ` -TriggerRunStartedBefore "2017-12-09"
在 SQL Server 中针对 SSISDB 数据库运行以下查询,验证是否执行了该包。
select * from catalog.executions