使用 Azure Monitor 监视数据工厂和发警报Monitor and Alert Data Factory by using Azure Monitor

适用于:是 Azure 数据工厂否 Azure Synapse Analytics(预览版)APPLIES TO: yesAzure Data Factory noAzure Synapse Analytics (Preview)

云应用程序比较复杂,包含很多移动部件。Cloud applications are complex and have many moving parts. 监视功能可以提供数据来确保应用程序始终处于正常运行状态。Monitors provide data to help ensure that your applications stay up and running in a healthy state. 监视功能还有助于避免潜在问题,或者排查以往的问题。Monitors also help you avoid potential problems and troubleshoot past ones. 可以使用监视数据深入了解应用程序的情况。You can use monitoring data to gain deep insights about your applications. 了解这些情况有助于改进应用程序的性能或可维护性。This knowledge helps you improve application performance and maintainability. 此外,还有助于实现本来需要手动干预的操作的自动化。It also helps you automate actions that otherwise require manual intervention.

Azure Monitor 针对大多数 Azure 服务提供基本级别的基础结构指标和日志。Azure Monitor provides base-level infrastructure metrics and logs for most Azure services. Azure 诊断日志由资源发出,提供与该资源的操作相关的各种频繁生成的数据。Azure diagnostic logs are emitted by a resource and provide rich, frequent data about the operation of that resource. Azure 数据工厂 (ADF) 可以在 Azure Monitor 中写入诊断日志。Azure Data Factory (ADF) can write diagnostic logs in Azure Monitor.

有关更多详细信息,请参阅 Azure Monitor 概述For further details, see Azure Monitor overview.

保留 Azure 数据工厂指标和管道运行数据Keeping Azure Data Factory metrics and pipeline-run data

数据工厂仅将管道运行数据存储 45 天。Data Factory stores pipeline-run data for only 45 days. 若要将这些数据保留更长时间,请使用 Azure Monitor。Use Azure Monitor if you want to keep that data for a longer time. 使用 Monitor,可以将诊断日志路由到多个不同目标进行分析。With Monitor, you can route diagnostic logs for analysis to multiple different targets.

  • 存储帐户:将诊断日志保存到存储帐户进行审核或手动检查。Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. 可以使用诊断设置指定保留时间(天)。You can use the diagnostic settings to specify the retention time in days.
  • 事件中心:将日志流式传输到 Azure 事件中心。Event Hub: Stream the logs to Azure Event Hubs. 日志可用作合作伙伴服务/自定义分析解决方案(例如 Power BI)的输入。The logs become input to a partner service/custom analytics solution like Power BI.
  • Log Analytics:使用 Log Analytics 分析日志。Log Analytics: Analyze the logs with Log Analytics. 在以下情况下,将数据工厂与 Azure Monitor 集成非常有用:The Data Factory integration with Azure Monitor is useful in the following scenarios:
    • 需要针对由数据工厂发布到 Monitor 的丰富指标集编写复杂查询。You want to write complex queries on a rich set of metrics that are published by Data Factory to Monitor. 可以通过 Monitor 创建针对这些查询的自定义警报。You can create custom alerts on these queries via Monitor.
    • 你希望跨数据工厂进行监视。You want to monitor across data factories. 可将来自多个数据工厂的数据路由到单个 Monitor 工作区。You can route data from multiple data factories to a single Monitor workspace.

还可使用与发出日志的资源不同的订阅中的存储帐户或事件中心命名空间。You can also use a storage account or event-hub namespace that isn't in the subscription of the resource that emits logs. 配置此设置的用户必须对两个订阅都具有适当的基于角色的访问控制 (RBAC) 访问权限。The user who configures the setting must have appropriate role-based access control (RBAC) access to both subscriptions.

配置诊断设置和工作区Configure diagnostic settings and workspace

为数据工厂创建或添加诊断设置。Create or add diagnostic settings for your data factory.

  1. 在门户中,转到“数据工厂”。In the portal, go to Data factory. 选择“诊断设置”。Select Diagnostic settings.

  2. 选择“添加诊断设置”。Select Add diagnostic setting.

    如果存在设置,则添加诊断设置

  3. 为设置指定名称,选择“发送到 Log Analytics”,然后从 Log Analytics 工作区中选择一个工作区。Give your setting a name, select Send to Log Analytics, and then select a workspace from Log Analytics Workspace.

如果选择 AllMetrics,则可使用多种 ADF 指标来进行监视或发出警报,包括针对 ADF 活动、管道和触发器运行的指标以及针对 SSIS IR 操作和 SSIS 包执行的指标。If you select AllMetrics, various ADF metrics will be made available for you to monitor or raise alerts on, including the metrics for ADF activity, pipeline, and trigger runs, as well as for SSIS IR operations and SSIS package executions.

命名设置并选择 log-analytics 工作区

  1. 选择“保存” 。Select Save.

几分钟后,新设置将出现在此数据工厂的设置列表中。After a few moments, the new setting appears in your list of settings for this data factory. 生成新的事件数据后,诊断日志将立即流式传输到该工作区。Diagnostic logs are streamed to that workspace as soon as new event data is generated. 发出事件后可能需要最多 15 分钟的时间该事件才会出现在 Log Analytics 中。Up to 15 minutes might elapse between when an event is emitted and when it appears in Log Analytics.

数据工厂指标Data Factory Metrics

使用 Monitor 可以洞察 Azure 工作负荷的性能与运行状况。With Monitor, you can gain visibility into the performance and health of your Azure workloads. 最重要的 Monitor 数据类型是指标(也称为性能计数器)。The most important type of Monitor data is the metric, which is also called the performance counter. 大多数 Azure 资源都会发出指标。Metrics are emitted by most Azure resources. Monitor 提供多种方式来配置和使用这些指标,以便进行监视与故障排除。Monitor provides several ways to configure and consume these metrics for monitoring and troubleshooting.

下面是 Azure 数据工厂版本 2 发出的一些指标:Here are some of the metrics emitted by Azure Data Factory version 2:

指标Metric 指标显示名称Metric display name 单位Unit 聚合类型Aggregation type 说明Description
ActivityCancelledRunsActivityCancelledRuns 已取消的活动运行指标数Cancelled activity runs metrics 计数Count 总计Total 在一分钟时段内取消的活动运行总数。The total number of activity runs that were cancelled within a minute window.
ActivityFailedRunsActivityFailedRuns 失败的活动运行数指标Failed activity runs metrics 计数Count 总计Total 在一分钟时段内失败的活动运行总数。The total number of activity runs that failed within a minute window.
ActivitySucceededRunsActivitySucceededRuns 成功的活动运行数指标Succeeded activity runs metrics 计数Count 总计Total 在一分钟时段内成功的活动运行总数。The total number of activity runs that succeeded within a minute window.
PipelineCancelledRunsPipelineCancelledRuns 已取消的管道运行指标数Cancelled pipeline runs metrics 计数Count 总计Total 在一分钟时段内取消的管道运行总数。The total number of pipeline runs that were cancelled within a minute window.
PipelineFailedRunsPipelineFailedRuns 失败的管道运行数指标Failed pipeline runs metrics 计数Count 总计Total 在一分钟时段内失败的管道运行总数。The total number of pipeline runs that failed within a minute window.
PipelineSucceededRunsPipelineSucceededRuns 成功的管道运行数指标Succeeded pipeline runs metrics 计数Count 总计Total 在一分钟时段内成功的管道运行总数。The total number of pipeline runs that succeeded within a minute window.
TriggerCancelledRunsTriggerCancelledRuns 已取消的触发器运行指标数Cancelled trigger runs metrics 计数Count 总计Total 在一分钟时段内取消的触发器运行总数。The total number of trigger runs that were cancelled within a minute window.
TriggerFailedRunsTriggerFailedRuns 失败的触发器运行数指标Failed trigger runs metrics 计数Count 总计Total 在一分钟时段内失败的触发器运行总数。The total number of trigger runs that failed within a minute window.
TriggerSucceededRunsTriggerSucceededRuns 成功的触发器运行数指标Succeeded trigger runs metrics 计数Count 总计Total 在一分钟时段内成功的触发器运行总数。The total number of trigger runs that succeeded within a minute window.
SSISIntegrationRuntimeStartCancelledSSISIntegrationRuntimeStartCancelled 已取消的 SSIS IR 启动指标Cancelled SSIS IR start metrics 计数Count 总计Total 在一分钟时段内取消的 SSIS IR 启动总数。The total number of SSIS IR starts that were cancelled within a minute window.
SSISIntegrationRuntimeStartFailedSSISIntegrationRuntimeStartFailed 失败的 SSIS IR 启动指标Failed SSIS IR start metrics 计数Count 总计Total 在一分钟时段内失败的 SSIS IR 启动总数。The total number of SSIS IR starts that failed within a minute window.
SSISIntegrationRuntimeStartSucceededSSISIntegrationRuntimeStartSucceeded 已成功的 SSIS IR 启动指标Succeeded SSIS IR start metrics 计数Count 总计Total 在一分钟时段内成功的 SSIS IR 启动总数。The total number of SSIS IR starts that succeeded within a minute window.
SSISIntegrationRuntimeStopStuckSSISIntegrationRuntimeStopStuck 停滞的 SSIS IR 停止指标Stuck SSIS IR stop metrics 计数Count 总计Total 在一分钟时段内停滞的 SSIS IR 停止总数。The total number of SSIS IR stops that were stuck within a minute window.
SSISIntegrationRuntimeStopSucceededSSISIntegrationRuntimeStopSucceeded 已成功的 SSIS IR 停止指标Succeeded SSIS IR stop metrics 计数Count 总计Total 在一分钟时段内成功的 SSIS IR 停止总数。The total number of SSIS IR stops that succeeded within a minute window.
SSISPackageExecutionCancelledSSISPackageExecutionCancelled 已取消的 SSIS 包执行指标Cancelled SSIS package execution metrics 计数Count 总计Total 在一分钟时段内取消的 SSIS 包执行总数。The total number of SSIS package executions that were cancelled within a minute window.
SSISPackageExecutionFailedSSISPackageExecutionFailed 失败的 SSIS 包执行指标Failed SSIS package execution metrics 计数Count 总计Total 在一分钟时段内失败的 SSIS 包执行总数。The total number of SSIS package executions that failed within a minute window.
SSISPackageExecutionSucceededSSISPackageExecutionSucceeded 已成功的 SSIS 包执行指标Succeeded SSIS package execution metrics 计数Count 总计Total 在一分钟时段内成功的 SSIS 包执行总数。The total number of SSIS package executions that succeeded within a minute window.

若要访问指标,请参阅 Azure Monitor 数据平台中的说明。To access the metrics, complete the instructions in Azure Monitor data platform.

备注

仅发出已完成和已触发的活动以及管道运行事件。Only events from completed, triggered activity and pipeline runs are emitted. 不会发出正在进行的运行和调试运行。In progress and debug runs are not emitted. 另一方面,将发出所有 SSIS 包执行事件,包括已完成和正在进行的事件,而无论使用何种调用方法。On the other hand, events from all SSIS package executions are emitted, including those that are completed and in progress, regardless of their invocation methods. 例如,可以在 SSMS、SQL Server 代理或其他指定工具上通过 T-SQL 调用包执行,也可以作为 ADF 管道中“执行 SSIS 包”的已触发运行或调试运行进行调用。For example, you can invoke package executions via T-SQL on SSMS, SQL Server Agent, or other designated tools, and as triggered or debug runs of Execute SSIS Package activities in ADF pipelines.

数据工厂警报Data Factory Alerts

登录到 Azure 门户,选择“Monitor” > “警报”以创建警报。Sign in to the Azure portal and select Monitor > Alerts to create alerts.

门户菜单中的警报

创建警报Create Alerts

  1. 选择“+ 创建新的预警规则”,创建新的警报。Select + New Alert rule to create a new alert.

    新建警报规则

  2. 定义警报条件。Define the alert condition.

    备注

    请务必在“按资源类型筛选”下拉列表中选择“所有”。Make sure to select All in the Filter by resource type drop-down list.

    “定义警报条件”>“选择目标”,此时会打开“选择资源”窗格"Define alert condition" > "Select target", which opens the "Select a resource" pane

    “定义警报条件”>“添加条件”,此时会打开“配置信号逻辑”窗格

    “配置信号类型”窗格

  3. 定义警报详细信息。Define the alert details.

    警报详细信息

  4. 定义操作组。Define the action group.

    创建规则的屏幕截图,其中突出显示了“新建操作组”

    创建新的操作组

    配置电子邮件、短信、推送和语音

    删除操作组

通过 Azure Monitor REST API 设置诊断日志Set up diagnostic logs via the Azure Monitor REST API

诊断设置Diagnostic settings

可以使用诊断设置来配置非计算资源的诊断日志。Use diagnostic settings to configure diagnostic logs for non-compute resources. 用于资源控制的诊断设置具有以下功能:The settings for a resource control have the following features:

  • 指定要将诊断日志发送到何处。They specify where diagnostic logs are sent. 例如,发送到 Azure 存储帐户、Azure 事件中心或 Monitor 日志。Examples include an Azure storage account, an Azure event hub, or Monitor logs.
  • 指定要发送的日志类别。They specify which log categories are sent.
  • 指定要将每个日志类别保留在存储帐户中多长时间。They specify how long each log category should be kept in a storage account.
  • 保留期为 0 天表示永久保留日志。A retention of zero days means logs are kept forever. 如果不需要永久保留,可将该值设置为 1 到 2,147,483,647 的任意天数。Otherwise, the value can be any number of days from 1 through 2,147,483,647.
  • 如果设置了保留策略,但禁止将日志存储在存储帐户中,则保留策略无效。If retention policies are set but storing logs in a storage account is disabled, the retention policies have no effect. 例如,如果仅选择了“事件中心”或“Monitor 日志”选项,此可能会发生这种情况。For example, this condition can happen when only Event Hubs or Monitor logs options are selected.
  • 保留策略按天应用。Retention policies are applied per day. 一天的结束时间可能出现在协调世界时 (UTC) 的午夜。The boundary between days occurs at midnight Coordinated Universal Time (UTC). 在一天结束时,会删除当天已超过保留策略期限的日志。At the end of a day, logs from days that are beyond the retention policy are deleted. 例如,如果保留策略的期限为一天,则在今天开始时,会删除前天的日志。For example, if you have a retention policy of one day, at the beginning of today the logs from before yesterday are deleted.

通过 Azure Monitor REST API 启用诊断日志Enable diagnostic logs via the Azure Monitor REST API

在 Monitor REST API 中创建或更新诊断设置Create or update a diagnostics setting in the Monitor REST API

请求Request
PUT
https://management.chinacloudapi.cn/{resource-id}/providers/microsoft.insights/diagnosticSettings/service?api-version={api-version}
头文件Headers
  • {api-version} 替换为 2016-09-01Replace {api-version} with 2016-09-01.
  • {resource-id} 替换为要编辑其诊断设置的资源的 ID。Replace {resource-id} with the ID of the resource for which you want to edit diagnostic settings. 有关详细信息,请参阅使用资源组管理 Azure 资源For more information, see Using Resource groups to manage your Azure resources.
  • Content-Type 标头设置为 application/jsonSet the Content-Type header to application/json.
  • 将授权标头设置为从 Azure Active Directory (Azure AD) 获取的 JSON Web 令牌。Set the authorization header to the JSON web token that you got from Azure Active Directory (Azure AD). 有关详细信息,请参阅对请求进行身份验证For more information, see Authenticating requests.
正文Body
{
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Storage/storageAccounts/<storageAccountName>",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.EventHub/namespaces/<eventHubName>/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<LogAnalyticsName>",
        "metrics": [
        ],
        "logs": [
                {
                    "category": "PipelineRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                },
                {
                    "category": "TriggerRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                },
                {
                    "category": "ActivityRuns",
                    "enabled": true,
                    "retentionPolicy": {
                        "enabled": false,
                        "days": 0
                    }
                }
            ]
    },
    "location": ""
}
属性Property 类型Type 说明Description
storageAccountIdstorageAccountId StringString 要将诊断日志发送到的存储帐户的资源 ID。The resource ID of the storage account to which you want to send diagnostic logs.
serviceBusRuleIdserviceBusRuleId StringString 服务总线命名空间的服务总线规则 ID。你要在该服务总线命名空间中创建事件中心,以便流式传输诊断日志。The service-bus rule ID of the service-bus namespace in which you want to have Event Hubs created for streaming diagnostic logs. 规则 ID 的格式为 {service bus resource ID}/authorizationrules/{key name}The rule ID has the format {service bus resource ID}/authorizationrules/{key name}.
workspaceIdworkspaceId 复杂类型Complex Type 指标时间粒度及其保留策略的数组。An array of metric time grains and their retention policies. 此属性的值为空。This property's value is empty.
指标metrics 要传递到被调用管道的管道运行的参数值Parameter values of the pipeline run to be passed to the invoked pipeline 将参数名映射为自变量值的 JSON 对象。A JSON object that maps parameter names to argument values.
logslogs 复杂类型Complex Type 某个资源类型的诊断日志类别的名称。The name of a diagnostic-log category for a resource type. 若要获取资源的诊断日志类别列表,请先执行 GET 诊断设置操作。To get the list of diagnostic-log categories for a resource, perform a GET diagnostic-settings operation.
categorycategory StringString 日志类别及其保留策略的数组。An array of log categories and their retention policies.
timeGraintimeGrain StringString 以 ISO 8601 持续时间格式捕获的指标的粒度。The granularity of metrics, which are captured in ISO 8601 duration format. 该属性值必须为 PT1M(1 分钟)。The property value must be PT1M, which specifies one minute.
enabledenabled 布尔Boolean 指定是否为此资源启用了该指标或日志类别的收集。Specifies whether collection of the metric or log category is enabled for this resource.
retentionPolicyretentionPolicy 复杂类型Complex Type 描述指标或日志类别的保留策略。Describes the retention policy for a metric or log category. 此属性仅用于存储帐户。This property is used for storage accounts only.
daysdays intInt 指标或日志的保留天数。The number of days to keep the metrics or logs. 如果该属性值为 0,则永久保留日志。If the property value is 0, the logs are kept forever. 此属性仅用于存储帐户。This property is used for storage accounts only.
响应Response

200 正常。200 OK.

{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Storage/storageAccounts/<storageAccountName>",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.EventHub/namespaces/<eventHubName>/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/<resourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<LogAnalyticsName>",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}

在 Monitor REST API 中获取有关诊断设置的信息Get information about diagnostics settings in the Monitor REST API

请求Request
GET
https://management.chinacloudapi.cn/{resource-id}/providers/microsoft.insights/diagnosticSettings/service?api-version={api-version}
头文件Headers
  • {api-version} 替换为 2016-09-01Replace {api-version} with 2016-09-01.
  • {resource-id} 替换为要编辑其诊断设置的资源的 ID。Replace {resource-id} with the ID of the resource for which you want to edit diagnostic settings. 有关详细信息,请参阅使用资源组管理 Azure 资源For more information, see Using Resource groups to manage your Azure resources.
  • Content-Type 标头设置为 application/jsonSet the Content-Type header to application/json.
  • 将授权标头设置为从 Azure AD 获取的 JSON Web 令牌。Set the authorization header to a JSON web token that you got from Azure AD. 有关详细信息,请参阅对请求进行身份验证For more information, see Authenticating requests.
响应Response

200 正常。200 OK.

{
    "id": "/subscriptions/<subID>/resourcegroups/adf/providers/microsoft.datafactory/factories/shloadobetest2/providers/microsoft.insights/diagnosticSettings/service",
    "type": null,
    "name": "service",
    "location": null,
    "kind": null,
    "tags": null,
    "properties": {
        "storageAccountId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.Storage/storageAccounts/azmonlogs",
        "serviceBusRuleId": "/subscriptions/<subID>/resourceGroups/shloprivate/providers/Microsoft.EventHub/namespaces/shloeventhub/authorizationrules/RootManageSharedAccessKey",
        "workspaceId": "/subscriptions/<subID>/resourceGroups/ADF/providers/Microsoft.OperationalInsights/workspaces/mihaipie",
        "eventHubAuthorizationRuleId": null,
        "eventHubName": null,
        "metrics": [],
        "logs": [
            {
                "category": "PipelineRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "TriggerRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            },
            {
                "category": "ActivityRuns",
                "enabled": true,
                "retentionPolicy": {
                    "enabled": false,
                    "days": 0
                }
            }
        ]
    },
    "identity": null
}

有关详细信息,请参阅诊断设置For more information, see Diagnostic Settings.

日志和事件的架构Schema of logs and events

监视架构Monitor schema

活动运行日志属性Activity-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "activityRunId":"",
   "pipelineRunId":"",
   "resourceId":"",
   "category":"ActivityRuns",
   "level":"Informational",
   "operationName":"",
   "pipelineName":"",
   "activityName":"",
   "start":"",
   "end":"",
   "properties":
       {
          "Input": "{
              "source": {
                "type": "BlobSource"
              },
              "sink": {
                "type": "BlobSink"
              }
           }",
          "Output": "{"dataRead":121,"dataWritten":121,"copyDuration":5,
               "throughput":0.0236328132,"errors":[]}",
          "Error": "{
              "errorCode": "null",
              "message": "null",
              "failureType": "null",
              "target": "CopyBlobtoBlob"
        }
    }
}
属性Property 类型Type 描述Description 示例Example
级别Level StringString 诊断日志的级别。The level of the diagnostic logs. 对于活动运行日志,请将该属性值设置为 4。For activity-run logs, set the property value to 4. 4
correlationIdcorrelationId StringString 用于跟踪特定请求的唯一 ID。The unique ID for tracking a particular request. 319dc6b4-f348-405e-b8d7-aafc77b73e77
timetime StringString 事件的时间,以时间跨度表示,采用 UTC 格式 YYYY-MM-DDTHH:MM:SS.00000ZThe time of the event in the timespan UTC format YYYY-MM-DDTHH:MM:SS.00000Z. 2017-06-28T21:00:27.3534352Z
activityRunIdactivityRunId StringString 活动运行的 ID。The ID of the activity run. 3a171e1f-b36e-4b80-8a54-5625394f4354
pipelineRunIdpipelineRunId StringString 管道运行的 ID。The ID of the pipeline run. 9f6069d6-e522-4608-9f99-21807bfc3c70
resourceIdresourceId StringString 与数据工厂资源关联的 ID。The ID associated with the data-factory resource. /SUBSCRIPTIONS/<subID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>
categorycategory StringString 诊断日志的类别。The category of the diagnostic logs. 请将该属性值设置为 ActivityRunsSet the property value to ActivityRuns. ActivityRuns
levellevel StringString 诊断日志的级别。The level of the diagnostic logs. 请将该属性值设置为 InformationalSet the property value to Informational. Informational
operationNameoperationName StringString 活动的名称及其状态。The name of the activity with its status. 如果活动是启动检测信号,则属性值为 MyActivity -If the activity is the start heartbeat, the property value is MyActivity -. 如果活动是结束检测信号,则属性值为 MyActivity - SucceededIf the activity is the end heartbeat, the property value is MyActivity - Succeeded. MyActivity - Succeeded
pipelineNamepipelineName StringString 管道的名称。The name of the pipeline. MyPipeline
activityNameactivityName StringString 活动的名称。The name of the activity. MyActivity
startstart StringString 活动运行的开始时间,以时间跨度表示,采用 UTC 格式。The start time of the activity runs in timespan UTC format. 2017-06-26T20:55:29.5007959Z
endend StringString 活动运行的结束时间,以时间跨度表示,采用 UTC 格式。The end time of the activity runs in timespan UTC format. 如果诊断日志显示活动已启动但尚未结束,则该属性值为 1601-01-01T00:00:00ZIf the diagnostic log shows that an activity has started but not yet ended, the property value is 1601-01-01T00:00:00Z. 2017-06-26T20:55:29.5007959Z

管道运行日志属性Pipeline-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "runId":"",
   "resourceId":"",
   "category":"PipelineRuns",
   "level":"Informational",
   "operationName":"",
   "pipelineName":"",
   "start":"",
   "end":"",
   "status":"",
   "properties":
    {
      "Parameters": {
        "<parameter1Name>": "<parameter1Value>"
      },
      "SystemParameters": {
        "ExecutionStart": "",
        "TriggerId": "",
        "SubscriptionId": ""
      }
    }
}
属性Property 类型Type 描述Description 示例Example
级别Level StringString 诊断日志的级别。The level of the diagnostic logs. 对于活动运行日志,请将该属性值设置为 4。For activity-run logs, set the property value to 4. 4
correlationIdcorrelationId StringString 用于跟踪特定请求的唯一 ID。The unique ID for tracking a particular request. 319dc6b4-f348-405e-b8d7-aafc77b73e77
timetime StringString 事件的时间,以时间跨度表示,采用 UTC 格式 YYYY-MM-DDTHH:MM:SS.00000ZThe time of the event in the timespan UTC format YYYY-MM-DDTHH:MM:SS.00000Z. 2017-06-28T21:00:27.3534352Z
runIdrunId StringString 管道运行的 ID。The ID of the pipeline run. 9f6069d6-e522-4608-9f99-21807bfc3c70
resourceIdresourceId StringString 与数据工厂资源关联的 ID。The ID associated with the data-factory resource. /SUBSCRIPTIONS/<subID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>
categorycategory StringString 诊断日志的类别。The category of the diagnostic logs. 请将该属性值设置为 PipelineRunsSet the property value to PipelineRuns. PipelineRuns
levellevel StringString 诊断日志的级别。The level of the diagnostic logs. 请将该属性值设置为 InformationalSet the property value to Informational. Informational
operationNameoperationName StringString 管道的名称及其状态。The name of the pipeline along with its status. 管道运行完成后,该属性值为 Pipeline - SucceededAfter the pipeline run is finished, the property value is Pipeline - Succeeded. MyPipeline - Succeeded.MyPipeline - Succeeded.
pipelineNamepipelineName StringString 管道的名称。The name of the pipeline. MyPipeline
startstart StringString 活动运行的开始时间,以时间跨度表示,采用 UTC 格式。The start time of the activity runs in timespan UTC format. 2017-06-26T20:55:29.5007959Z.2017-06-26T20:55:29.5007959Z.
endend StringString 活动运行的结束时间,以时间跨度表示,采用 UTC 格式。The end time of the activity runs in timespan UTC format. 如果诊断日志显示活动已启动但尚未结束,则该属性值为 1601-01-01T00:00:00ZIf the diagnostic log shows an activity has started but not yet ended, the property value is 1601-01-01T00:00:00Z. 2017-06-26T20:55:29.5007959Z
statusstatus StringString 管道运行的最终状态。The final status of the pipeline run. 可能的属性值为 SucceededFailedPossible property values are Succeeded and Failed. Succeeded

触发器运行日志属性Trigger-run log attributes

{
   "Level": "",
   "correlationId":"",
   "time":"",
   "triggerId":"",
   "resourceId":"",
   "category":"TriggerRuns",
   "level":"Informational",
   "operationName":"",
   "triggerName":"",
   "triggerType":"",
   "triggerEvent":"",
   "start":"",
   "status":"",
   "properties":
   {
      "Parameters": {
        "TriggerTime": "",
       "ScheduleTime": ""
      },
      "SystemParameters": {}
    }
}
属性Property 类型Type 描述Description 示例Example
级别Level StringString 诊断日志的级别。The level of the diagnostic logs. 对于活动运行日志,请将该属性值设置为 4。For activity-run logs, set the property value to 4. 4
correlationIdcorrelationId StringString 用于跟踪特定请求的唯一 ID。The unique ID for tracking a particular request. 319dc6b4-f348-405e-b8d7-aafc77b73e77
timetime StringString 事件的时间,以时间跨度表示,采用 UTC 格式 YYYY-MM-DDTHH:MM:SS.00000ZThe time of the event in the timespan UTC format YYYY-MM-DDTHH:MM:SS.00000Z. 2017-06-28T21:00:27.3534352Z
triggerIdtriggerId StringString 触发器运行的 ID。The ID of the trigger run. 08587023010602533858661257311
resourceIdresourceId StringString 与数据工厂资源关联的 ID。The ID associated with the data-factory resource. /SUBSCRIPTIONS/<subID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>
categorycategory StringString 诊断日志的类别。The category of the diagnostic logs. 请将该属性值设置为 PipelineRunsSet the property value to PipelineRuns. PipelineRuns
levellevel StringString 诊断日志的级别。The level of the diagnostic logs. 请将该属性值设置为 InformationalSet the property value to Informational. Informational
operationNameoperationName StringString 触发器的名称及其最终状态(指示是否已成功激发该触发器)。The name of the trigger with its final status, which indicates whether the trigger successfully fired. 如果检测信号成功,则该属性值为 MyTrigger - SucceededIf the heartbeat was successful, the property value is MyTrigger - Succeeded. MyTrigger - Succeeded
triggerNametriggerName StringString 触发器的名称。The name of the trigger. MyTrigger
triggerTypetriggerType StringString 触发器的类型。The type of the trigger. 可能的属性值为 Manual TriggerSchedule TriggerPossible property values are Manual Trigger and Schedule Trigger. ScheduleTrigger
triggerEventtriggerEvent StringString 触发器的事件。The event of the trigger. ScheduleTime - 2017-07-06T01:50:25Z
startstart StringString 激发触发器的开始时间,以时间跨度表示,采用 UTC 格式。The start time of the trigger firing in timespan UTC format. 2017-06-26T20:55:29.5007959Z
statusstatus StringString 最终状态(指示是否已成功激发该触发器)。The final status showing whether the trigger successfully fired. 可能的属性值为 SucceededFailedPossible property values are Succeeded and Failed. Succeeded

SSIS Integration Runtime 日志特性SSIS integration runtime log attributes

下面是 SSIS IR 启动/停止/维护操作的日志特性。Here are the log attributes of SSIS IR start/stop/maintenance operations.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "resultType": "",
   "properties": {
      "message": ""
   },
   "resourceId": ""
}
属性Property 类型Type 描述Description 示例Example
timetime StringString 事件的时间,采用 UTC 格式 YYYY-MM-DDTHH:MM:SS.00000ZThe time of event in UTC format: YYYY-MM-DDTHH:MM:SS.00000Z 2017-06-28T21:00:27.3534352Z
operationNameoperationName StringString SSIS IR 操作的名称The name of your SSIS IR operation Start/Stop/Maintenance
categorycategory StringString 诊断日志的类别The category of diagnostic logs SSISIntegrationRuntimeLogs
correlationIdcorrelationId StringString 用于跟踪特定操作的唯一 IDThe unique ID for tracking a particular operation f13b159b-515f-4885-9dfa-a664e949f785Deprovision0059035558
dataFactoryNamedataFactoryName StringString ADF 的名称The name of your ADF MyADFv2
integrationRuntimeNameintegrationRuntimeName StringString SSIS IR 的名称The name of your SSIS IR MySSISIR
levellevel StringString 诊断日志的级别The level of diagnostic logs Informational
resultTyperesultType StringString SSIS IR 操作的结果The result of your SSIS IR operation Started/InProgress/Succeeded/Failed
messagemessage StringString SSIS IR 操作的输出消息The output message of your SSIS IR operation The stopping of your SSIS integration runtime has succeeded.
resourceIdresourceId StringString ADF 资源的唯一 IDThe unique ID of your ADF resource /SUBSCRIPTIONS/<subscriptionID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>

SSIS 事件消息上下文日志特性SSIS event message context log attributes

下面是与 SSIS IR 上的 SSIS 包执行生成的事件消息相关的条件的日志特性。Here are the log attributes of conditions related to event messages that are generated by SSIS package executions on your SSIS IR. 它们传递的信息与显示许多 SSIS 包属性的运行时值的 SSIS 目录 (SSISDB) 事件消息上下文表或视图类似。They convey similar information as SSIS catalog (SSISDB) event message context table or view that shows run-time values of many SSIS package properties. 这些日志在你选择“Basic/Verbose”日志记录级别时生成,可用于调试/合规性检查。They're generated when you select Basic/Verbose logging level and useful for debugging/compliance checking.

{
   "time": "",
   "operationName": "",
   "category": "",
   "correlationId": "",
   "dataFactoryName": "",
   "integrationRuntimeName": "",
   "level": "",
   "properties": {
      "operationId": "",
      "contextDepth": "",
      "packagePath": "",
      "contextType": "",
      "contextSourceName": "",
      "contextSourceId": "",
      "propertyName": "",
      "propertyValue": ""
   },
   "resourceId": ""
}
属性Property 类型Type 描述Description 示例Example
timetime StringString 事件的时间,采用 UTC 格式 YYYY-MM-DDTHH:MM:SS.00000ZThe time of event in UTC format: YYYY-MM-DDTHH:MM:SS.00000Z 2017-06-28T21:00:27.3534352Z
operationNameoperationName StringString 此项设置为 YourSSISIRName-SSISPackageEventMessageContextThis is set to YourSSISIRName-SSISPackageEventMessageContext mysqlmissisir-SSISPackageEventMessageContext
categorycategory StringString 诊断日志的类别The category of diagnostic logs SSISPackageEventMessageContext
correlationIdcorrelationId StringString 用于跟踪特定操作的唯一 IDThe unique ID for tracking a particular operation e55700df-4caf-4e7c-bfb8-78ac7d2f28a0
dataFactoryNamedataFactoryName StringString ADF 的名称The name of your ADF MyADFv2
integrationRuntimeNameintegrationRuntimeName StringString SSIS IR 的名称The name of your SSIS IR MySSISIR
levellevel StringString 诊断日志的级别The level of diagnostic logs Informational
operationIdoperationId StringString 用于跟踪 SSISDB 中的特定操作的唯一 IDThe unique ID for tracking a particular operation in SSISDB 1(1 表示与未存储在 SSISDB 中的包/通过 T-SQL 调用的包相关的操作)1 (1 signifies operations related to packages not stored in SSISDB/invoked via T-SQL)
contextDepthcontextDepth StringString 事件消息上下文的深度The depth of your event message context 0(0 表示在包执行开始之前的上下文,1 表示发生错误时的上下文,该数字将随着上下文离错误的距离的增加而增大)0 (0 signifies the context before package execution starts, 1 signifies the context when an error occurs, and it increases as the context is further from the error)
packagePathpackagePath StringString 作为事件消息上下文源的包对象的路径The path of package object as your event message context source \Package
contextTypecontextType StringString 作为事件消息上下文源的包对象的类型The type of package object as your event message context source 60(请参阅更多上下文类型60(see more context types)
contextSourceNamecontextSourceName StringString 作为事件消息上下文源的包对象的名称The name of package object as your event message context source MyPackage
contextSourceIdcontextSourceId StringString 作为事件消息上下文源的包对象的唯一 IDThe unique ID of package object as your event message context source {E2CF27FB-EA48-41E9-AF6F-3FE938B4ADE1}
propertyNamepropertyName StringString 事件消息上下文源的包属性的名称The name of package property for your event message context source DelayValidation
propertyValuepropertyValue StringString 事件消息上下文源的包属性的值The value of package property for your event message context source False
resourceIdresourceId StringString ADF 资源的唯一 IDThe unique ID of your ADF resource /SUBSCRIPTIONS/<subscriptionID>/RESOURCEGROUPS/<resourceGroupName>/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/<dataFactoryName>

Log Analytics 架构Log Analytics schema

Log Analytics 从 Monitor 继承架构,但存在以下例外情况:Log Analytics inherits the schema from Monitor with the following exceptions:

  • 每个列名中的第一个字母会大写。The first letter in each column name is capitalized. 例如,Monitor 中的列名“correlationId”在 Log Analytics 中为“CorrelationId”。For example, the column name "correlationId" in Monitor is "CorrelationId" in Log Analytics.

  • 没有“Level”列。There's no "Level" column.

  • 动态“properties”列保留为以下动态 JSON Blob 类型。The dynamic "properties" column is preserved as the following dynamic JSON blob type.

    Azure Monitor 列Azure Monitor column Log Analytics 列Log Analytics column 类型Type
    $.properties.UserProperties$.properties.UserProperties UserPropertiesUserProperties 动态Dynamic
    $.properties.Annotations$.properties.Annotations 批注Annotations 动态Dynamic
    $.properties.Input$.properties.Input 输入Input 动态Dynamic
    $.properties.Output$.properties.Output 输出Output 动态Dynamic
    $.properties.Error.errorCode$.properties.Error.errorCode ErrorCodeErrorCode intint
    $.properties.Error.message$.properties.Error.message ErrorMessageErrorMessage stringstring
    $.properties.Error$.properties.Error 错误Error 动态Dynamic
    $.properties.Predecessors$.properties.Predecessors PredecessorsPredecessors 动态Dynamic
    $.properties.Parameters$.properties.Parameters parametersParameters 动态Dynamic
    $.properties.SystemParameters$.properties.SystemParameters SystemParametersSystemParameters 动态Dynamic
    $.properties.Tags$.properties.Tags TagsTags 动态Dynamic

使用 Azure Monitor 监视 SSIS 操作Monitor SSIS operations with Azure Monitor

要直接迁移 SSIS 工作负载,可以在 ADF 中预配 SSIS IR,以支持:To lift & shift your SSIS workloads, you can provision SSIS IR in ADF that supports:

  • 运行部署在由 Azure SQL 数据库服务器/托管实例托管的 SSIS 目录 (SSISDB) 中的包(项目部署模型)Running packages deployed into SSIS catalog (SSISDB) hosted by Azure SQL Database server/Managed Instance (Project Deployment Model)
  • 运行部署在由 Azure SQL 托管实例托管的文件系统、Azure 文件存储或 SQL Server 数据库 (MSDB) 中的包(包部署模型)Running packages deployed into file system, Azure Files, or SQL Server database (MSDB) hosted by Azure SQL Managed Instance (Package Deployment Model)

预配后,可以通过 Azure PowerShell 或 ADF 门户的“监视器”中心检查 SSIS IR 操作状态Once provisioned, you can check SSIS IR operational status using Azure PowerShell or on the Monitor hub of ADF portal. 使用项目部署模型时,SSIS 包执行日志存储在 SSISDB 内部表或视图中,因此可以使用 SSMS 之类的指定工具对其进行查询、分析和直观显示。With Project Deployment Model, SSIS package execution logs are stored in SSISDB internal tables or views, so you can query, analyze, and visually present them using designated tools like SSMS. 使用包部署模型时,可以将 SSIS 包执行日志作为 CSV 文件存储在文件系统或 Azure 文件存储中,仍需要使用其他指定工具对这些文件进行分析和处理,然后才能对其进行查询、分析和直观显示。With Package Deployment Model, SSIS package execution logs can be stored in file system or Azure Files as CSV files that you still need to parse and process using other designated tools before you can query, analyze, and visually present them.

现在,通过 Azure Monitor 集成,可在 Azure 门户上查询、分析和直观显示从 SSIS IR 操作和 SSIS 包执行生成的所有指标和日志。Now with Azure Monitor integration, you can query, analyze, and visually present all metrics and logs generated from SSIS IR operations and SSIS package executions on Azure portal. 此外,还可发出相关警报。Additionally, you can also raise alerts on them.

为 SSIS 操作配置诊断设置和工作区Configure diagnostic settings and workspace for SSIS operations

要将从 SSIS IR 操作和 SSIS 包执行生成的所有指标和日志发送到 Azure Monitor,需要为 ADF 配置诊断设置和工作区To send all metrics and logs generated from SSIS IR operations and SSIS package executions to Azure Monitor, you need to configure diagnostics settings and workspace for your ADF.

SSIS 操作指标SSIS operational metrics

SSIS 操作指标是性能计数器或数字值,用于描述特定时间点的 SSIS IR 启动和停止操作以及 SSIS 包执行的状态。SSIS operational metrics are performance counters or numerical values that describe the status of SSIS IR start and stop operations, as well as SSIS package executions at a particular point in time. 它们是 Azure Monitor 中 ADF 指标的一部分。They're part of ADF metrics in Azure Monitor.

在 Azure Monitor 上为 ADF 配置诊断设置和工作区时,选中“AllMetrics”复选框将使 SSIS 操作指标可用于使用 Azure 指标资源管理器进行的交互分析在 Azure 仪表板上呈现以及近实时警报When you configure diagnostic settings and workspace for your ADF on Azure Monitor, selecting the AllMetrics check box will make SSIS operational metrics available for interactive analysis using Azure Metrics Explorer, presentation on Azure dashboard, and near-real time alerts.

命名设置并选择 log-analytics 工作区

SSIS 操作警报SSIS operational alerts

若要从 ADF 门户基于 SSIS 操作指标引发警报,请选择 ADF“监视器”中心的“警报和指标”页面,并按照提供的分步说明进行操作To raise alerts on SSIS operational metrics from ADF portal, select the Alerts & metrics page of ADF Monitor hub and follow the step-by-step instructions provided.

从 ADF 门户引发 SSIS 操作警报

若要从 Azure 门户基于 SSIS 操作指标引发警报,请选择 Azure“监视器”中心的“警报”页面,并按照提供的分步说明进行操作To raise alerts on SSIS operational metrics from Azure portal, select the Alerts page of Azure Monitor hub and follow the step-by-step instructions provided.

从 Azure 门户引发 SSIS 操作警报

SSIS 操作日志SSIS operational logs

SSIS 操作日志是由 SSIS IR 操作和 SSIS 包执行生成的事件,这些事件提供了有关已识别问题的充足上下文,对根本原因分析很有用。SSIS operational logs are events generated by SSIS IR operations and SSIS package executions that provide enough context on any identified issues and are useful for root cause analysis.

在 Azure Monitor 上为 ADF 配置诊断设置和工作区时,可以选择相关 SSIS 操作日志,并将其发送到基于 Azure 数据资源管理器的 Log Analytics。When you configure diagnostic settings and workspace for your ADF on Azure Monitor, you can select the relevant SSIS operational logs and send them to Log Analytics that's based on Azure Data Explorer. 在这里,它们可用于使用丰富的查询语言进行的分析在 Azure 仪表板上呈现以及近实时警报In there, they'll be made available for analysis using rich query language, presentation on Azure dashboard, and near-real time alerts.

命名设置并选择 log-analytics 工作区

Azure Monitor 和 Log Analytics 中的 SSIS 包执行日志的架构和内容类似于 SSISDB 内部表或视图中的架构。The schemas and content of SSIS package execution logs in Azure Monitor and Log Analytics are similar to the schemas of SSISDB internal tables or views.

Azure Monitor 日志类别Azure Monitor log categories Log Analytics 表Log Analytics tables SSISDB 内部表/视图SSISDB internal tables/views
SSISIntegrationRuntimeLogs ADFSSISIntegrationRuntimeLogs
SSISPackageEventMessageContext ADFSSISPackageEventMessageContext [internal].[event_message_context]
SSISPackageEventMessages ADFSSISPackageEventMessages [internal].[event_messages]
SSISPackageExecutableStatistics ADFSSISPackageExecutableStatistics [internal].[executable_statistics]
SSISPackageExecutionComponentPhases ADFSSISPackageExecutionComponentPhases [internal].[execution_component_phases]
SSISPackageExecutionDataStatistics ADFSSISPackageExecutionDataStatistics [internal].[execution_data_statistics]

有关 SSIS 操作日志特性/属性的详细信息,请参阅 Azure Monitor 和 Log Analytics 为 ADF 使用的架构For more info on SSIS operational log attributes/properties, see Azure Monitor and Log Analytics schemas for ADF.

无论使用哪种调用方法,所选 SSIS 包执行日志始终会发送到 Log Analytics。Your selected SSIS package execution logs are always sent to Log Analytics regardless of their invocation methods. 例如,可以在 SSMS、SQL Server 代理或其他指定工具上通过 T-SQL 调用包执行,也可以作为 ADF 管道中“执行 SSIS 包”的已触发运行或调试运行进行调用。For example, you can invoke package executions via T-SQL on SSMS, SQL Server Agent, or other designated tools, and as triggered or debug runs of Execute SSIS Package activities in ADF pipelines.

在 Logs Analytics 上查询 SSIS IR 操作日志时,可以使用 OperationName 和 ResultType 属性,这两个属性分别设置为 Start/Stop/MaintenanceStarted/InProgress/Succeeded/FailedWhen querying SSIS IR operation logs on Logs Analytics, you can use OperationName and ResultType properties that are set to Start/Stop/Maintenance and Started/InProgress/Succeeded/Failed, respectively.

在 Log Analytics 上查询 SSIS IR 操作日志

在 Logs Analytics 上查询 SSIS 包执行日志时,可以使用 OperationId/ExecutionId/CorrelationId 属性联接它们 。When querying SSIS package execution logs on Logs Analytics, you can join them using OperationId/ExecutionId/CorrelationId properties. 对于与未存储在 SSISDB 中的包/通过 T-SQL 调用的包相关的所有操作/执行,OperationId/ExecutionId 始终设置为 1OperationId/ExecutionId are always set to 1 for all operations/executions related to packages not stored in SSISDB/invoked via T-SQL.

在 Log Analytics 上查询 SSIS 包执行日志

后续步骤Next steps

以编程方式监视和管理管道Monitor and manage pipelines programmatically