使用 Azure 数据工厂将数据复制到 Azure 数据资源管理器Copy data to Azure Data Explorer by using Azure Data Factory

Azure 数据资源管理器是一个快速、完全托管的数据分析服务。Azure Data Explorer is a fast, fully managed, data-analytics service. 它可以实时分析从应用程序、网站和 IoT 设备等许多源流式传输的大量数据。It offers real-time analysis on large volumes of data that stream from many sources, such as applications, websites, and IoT devices. 使用 Azure 数据资源管理器,可以迭代方式浏览数据,识别模式和异常以改进产品、增强客户体验、监视设备,以及提升操作性能。With Azure Data Explorer, you can iteratively explore data and identify patterns and anomalies to improve products, enhance customer experiences, monitor devices, and boost operations. 它可以帮助你探讨新的问题,并在短时间内获得解答。It helps you explore new questions and get answers in minutes.

Azure 数据工厂是一个完全托管的基于云的数据集成服务。Azure Data Factory is a fully managed, cloud-based, data-integration service. 可以使用它在 Azure 数据资源管理器数据库中填充现有系统中的数据。You can use it to populate your Azure Data Explorer database with data from your existing system. 它可以帮助你节省生成分析解决方案所花费的时间。It can help you save time when you're building analytics solutions.

将数据载入 Azure 数据资源管理器时,数据工厂可提供以下优势:When you load data into Azure Data Explorer, Data Factory provides the following benefits:

  • 轻松设置:使用直观的五步骤向导,无需编写脚本。Easy setup: Get an intuitive, five-step wizard with no scripting required.
  • 丰富的数据存储支持:对一组丰富的本地和基于云的数据存储的内置支持。Rich data store support: Get built-in support for a rich set of on-premises and cloud-based data stores. 有关详细列表,请参阅表支持的数据存储For a detailed list, see the table of Supported data stores.
  • 安全且合规:通过 HTTPS 或 Azure ExpressRoute 传输数据。Secure and compliant: Data is transferred over HTTPS or Azure ExpressRoute. 存在全局服务可确保数据永远不会离开地理边界。The global service presence ensures that your data never leaves the geographical boundary.
  • 高性能:向 Azure 数据资源管理器载入数据的速度高达 1 GBps。High performance: The data-loading speed is up to 1 gigabyte per second (GBps) into Azure Data Explorer. 有关详细信息,请参阅复制活动性能For more information, see Copy activity performance.

在本文中,你将使用数据工厂复制数据工具,将数据从 Amazon 简单存储服务 (S3) 载入 Azure 数据资源管理器。In this article, you use the Data Factory Copy Data tool to load data from Amazon Simple Storage Service (S3) into Azure Data Explorer. 可以遵循类似的步骤,从下述其他数据存储复制数据:You can follow a similar process to copy data from other data stores, such as:


创建数据工厂Create a data factory

  1. 登录到 Azure 门户Sign in to the Azure portal.

  2. 在左窗格中,选择“创建资源” > “Analytics” > “数据工厂”。In the left pane, select Create a resource > Analytics > Data Factory.

    在 Azure 门户中创建数据工厂

  3. 在“新建数据工厂”窗格中,提供下表中的字段值:In the New data factory pane, provide values for the fields in the following table:


    设置Setting 要输入的值Value to enter
    名称Name 在框中输入数据工厂的全局唯一名称。In the box, enter a globally unique name for your data factory. 如果收到错误“数据工厂名称 "LoadADXDemo" 不可用”,请输入不同的数据工厂名称。If you receive an error, Data factory name "LoadADXDemo" is not available, enter a different name for the data factory. 有关数据工厂项目的命名规则,请参阅数据工厂命名规则For rules about naming Data Factory artifacts, see Data Factory naming rules.
    订阅Subscription 在下拉列表中,选择要在其中创建数据工厂的 Azure 订阅。In the drop-down list, select the Azure subscription in which to create the data factory.
    资源组Resource Group 选择“新建”,然后输入新资源组的名称。Select Create new, and then enter the name of a new resource group. 如果已有一个资源组,请选择“使用现有项”。If you already have a resource group, select Use existing.
    版本Version 在下拉列表中选择“V2”。In the drop-down list, select V2.
    位置Location 在下拉列表中选择数据工厂的位置。In the drop-down list, select the location for the data factory. 该列表仅显示支持的位置。Only supported locations are displayed in the list. 数据工厂使用的数据存储可位于其他位置或区域中。The data stores that are used by the data factory can exist in other locations or regions.
  4. 选择“创建” 。Select Create.

  5. 若要监视创建过程,请在工具栏上选择“通知”。To monitor the creation process, select Notifications on the toolbar. 创建数据工厂后,将其选中。After you've created the data factory, select it.

    此时会打开“数据工厂”窗格。The Data Factory pane opens.


  6. 若要在单独的窗格中打开应用程序,请选择“创作和监视”磁贴。To open the application in a separate pane, select the Author & Monitor tile.

将数据载入 Azure 数据资源管理器Load data into Azure Data Explorer

可将许多类型的数据存储中的数据载入 Azure 数据资源管理器。You can load data from many types of data stores into Azure Data Explorer. 本文介绍如何从 Amazon S3 加载数据。This article discusses how to load data from Amazon S3.

可通过以下任一方式加载数据:You can load your data in either of the following ways:

从 Amazon S3(源)复制数据Copy data from Amazon S3 (source)

  1. 在“开始使用”窗格中,选择“复制数据”打开复制数据工具。 In the Let's get started pane, open the Copy Data tool by selecting Copy Data.


  2. 在“属性”窗格中的“任务名称”框内输入名称,然后选择“下一步”。 In the Properties pane, in the Task name box, enter a name, and then select Next.


  3. 在“源数据存储”窗格中,选择“创建新连接”。 In the Source data store pane, select Create new connection.


  4. 依次选择“Amazon S3”、“继续”。 Select Amazon S3, and then select Continue.


  5. 在“新建链接服务(Amazon S3)”窗格中执行以下操作:In the New Linked Service (Amazon S3) pane, do the following:

    指定 Amazon S3 链接服务

    a.a. 在“名称”框中,输入新链接服务的名称。In the Name box, enter the name of your new linked service.

    b.b. 在“通过集成运行时进行连接”下拉列表中选择值。In the Connect via integration runtime drop-down list, select the value.

    c.c. 在“访问密钥 ID”框中输入值。In the Access Key ID box, enter the value.


    若要在 Amazon S3 中查找访问密钥,请在导航栏上选择自己的 Amazon 用户名,然后选择“我的安全凭据”。In Amazon S3, to locate your access key, select your Amazon username on the navigation bar, and then select My Security Credentials.

    d.d. 在“机密访问密钥”中输入值。In the Secret Access Key box, enter a value.

    e.e. 若要测试创建的链接服务连接,请选择“测试连接”。To test the linked service connection you created, select Test Connection.

    f.f. 选择“完成”。Select Finish.

    “源数据存储”窗格中会显示新的 AmazonS31 连接。The Source data store pane displays your new AmazonS31 connection.

  6. 选择“下一步”。Select Next.

    源数据存储 - 创建的连接

  7. 在“选择输入文件或文件夹”窗格中执行以下步骤:In the Choose the input file or folder pane, do the following steps:

    a.a. 浏览到要复制的文件或文件夹并将其选中。Browse to the file or folder that you want to copy, and then select it.

    b.b. 选择所需的复制行为。Select the copy behavior that you want. 请确保未选中“二进制副本”复选框。Make sure that the Binary copy check box is cleared.

    c.c. 选择“下一步”。Select Next.


  8. 在“文件格式设置”窗格中选择文件的相关设置。In the File format settings pane, select the relevant settings for your file. 然后选择“下一步”。and then select Next.


将数据复制到 Azure 数据资源管理器(目标)Copy data into Azure Data Explorer (destination)

现已创建新的 Azure 数据资源管理器链接服务,用于将数据复制到本部分指定的 Azure 数据资源管理器目标表(接收器)。The new Azure Data Explorer linked service is created to copy the data into the Azure Data Explorer destination table (sink) that's specified in this section.

创建 Azure 数据资源管理器链接服务Create the Azure Data Explorer linked service

若要创建 Azure 数据资源管理器链接服务,请执行以下步骤:To create the Azure Data Explorer linked service, do the following steps:

  1. 若要使用现有的数据存储连接或指定新的数据存储,请在“目标数据存储”窗格中选择“创建新连接”。 To use an existing data store connection or specify a new data store, in the Destination data store pane, select Create new connection.


  2. 在“新建链接服务”窗格中选择“Azure 数据资源管理器”,然后选择“继续”。 In the New Linked Service pane, select Azure Data Explorer, and then select Continue.


  3. 在“新建链接服务(Azure 数据资源管理器)”窗格中执行以下步骤:In the New Linked Service (Azure Data Explorer) pane, do the following steps:

    Azure 数据资源管理器的“新建链接服务”窗格

    a.a. 在“名称”框中,输入 Azure 数据资源管理器链接服务的名称。In the Name box, enter a name for the Azure Data Explorer linked service.

    b.b. 在“帐户选择方法”下,选择以下选项之一:Under Account selection method, choose one of the following options:

    • 选择“从 Azure 订阅”,然后在下拉列表中,选择你的 Azure 订阅群集Select From Azure subscription and then, in the drop-down lists, select your Azure subscription and your Cluster.


      “群集”下拉控件只会列出与你的订阅关联的群集。The Cluster drop-down control lists only clusters that are associated with your subscription.

    • 选择“手动输入”,然后输入你的终结点Select Enter manually, and then enter your Endpoint.

    c.c. 在“租户”框中输入租户名称。In the Tenant box, enter the tenant name.

    d.d. 在“服务主体 ID”框中输入服务主体 ID。In the Service principal ID box, enter the service principal ID.

    e.e. 选择“服务主体密钥”,然后在“服务主体密钥”框中输入密钥的值。 Select Service principal key and then, in the Service principal key box, enter the value for the key.

    f.f. 在“数据库”下拉列表中选择数据库名称。In the Database drop-down list, select your database name. 或者,选中“编辑”复选框,然后输入数据库名称。Alternatively, select the Edit check box, and then enter the database name.

    g.g. 若要测试创建的链接服务连接,请选择“测试连接”。To test the linked service connection you created, select Test Connection. 如果可以连接到该链接服务,该窗格将显示绿色的勾选标记和“连接成功”消息。If you can connect to your linked service, the pane displays a green checkmark and a Connection successful message.

    h.如果该值不存在,请单击“添加行”。h. 选择“完成”以完成链接服务的创建过程。Select Finish to complete the linked service creation.


    Azure 数据工厂使用服务主体来访问 Azure 数据资源管理器服务。The service principal is used by Azure Data Factory to access the Azure Data Explorer service. 若要创建服务主体,请参阅创建 Azure Active Directory (Azure AD) 服务主体To create a service principal, go to create an Azure Active Directory (Azure AD) service principal. 不要使用 Azure Key Vault 方法。Do not use the Azure Key Vault method.

配置 Azure 数据资源管理器数据连接Configure the Azure Data Explorer data connection

创建链接服务连接后,“目标数据存储”窗格将会打开,创建的连接可供使用。After you've created the linked service connection, the Destination data store pane opens, and the connection you created is available for use. 若要配置连接,请执行以下步骤:To configure the connection, do the following steps:

  1. 选择“下一步”。Select Next.

    Azure 数据资源管理器的“目标数据存储”窗格

  2. 在“表映射”窗格中设置目标表名称,然后选择“下一步”。 In the Table mapping pane, set the destination table name, and then select Next.


  3. 在“列映射”窗格中,将发生以下映射:In the Column mapping pane, the following mappings take place:

    a.a. 第一个映射由 Azure 数据工厂根据 Azure 数据工厂架构映射执行。The first mapping is performed by Azure Data Factory according to the Azure Data Factory schema mapping. 请执行以下操作:Do the following:

    • 设置 Azure 数据工厂目标表的列映射Set the Column mappings for the Azure Data Factory destination table. 将显示从源到 Azure 数据工厂目标表的默认映射。The default mapping is displayed from source to the Azure Data Factory destination table.

    • 取消选择不需要定义列映射的列。Cancel the selection of the columns that you don't need to define your column mapping.

    b.b. 将此表格数据引入 Azure 数据资源管理器时,会发生第二个映射。The second mapping occurs when this tabular data is ingested into Azure Data Explorer. 映射是根据 CSV 映射规则执行的。Mapping is performed according to CSV mapping rules. 即使源数据不采用 CSV 格式,Azure 数据工厂也会将数据转换为表格格式。Even if the source data isn't in CSV format, Azure Data Factory converts the data into a tabular format. 因此,在此阶段,只有 CSV 映射才是相关的映射。Therefore, CSV mapping is the only relevant mapping at this stage. 请执行以下操作:Do the following:

    • (可选)在“Azure 数据资源管理器(Kusto)接收器属性”下,添加相关的“引入映射名称”,以便可以使用列映射。 (Optional) Under Azure Data Explorer (Kusto) sink properties, add the relevant Ingestion mapping name so that column mapping can be used.

    • 如果未指定“引入映射名称”,将使用“列映射”部分定义的“按名称”映射顺序。If Ingestion mapping name isn't specified, the by-name mapping order that's defined in the Column mappings section will be used. 如果“按名称”映射失败,Azure 数据资源管理器会尝试“按列位置”顺序(即,默认按位置映射)引入数据。 If by-name mapping fails, Azure Data Explorer tries to ingest the data in a by-column position order (that is, it maps by position as the default).

    • 选择“下一步”。Select Next.


  4. 在“设置”窗格中执行以下步骤:In the Settings pane, do the following steps:

    a.a. 在“容错设置”下,输入相关设置。Under Fault tolerance settings, enter the relevant settings.

    b.b. 在“性能设置”下,“启用暂存”的选项不适用,“高级设置”包括成本考虑因素。 Under Performance settings, Enable staging doesn't apply, and Advanced settings includes cost considerations. 如果没有具体的要求,请将这些设置保留原样。If you have no specific requirements, leave these settings as is.

    c.c. 选择“下一步”。Select Next.


  5. 在“摘要”窗格中检查设置,然后选择“下一步”。 In the Summary pane, review the settings, and then select Next.


  6. 在“部署已完成”窗格中执行以下操作:In the Deployment complete pane, do the following:

    a.a. 若要切换到“监视”选项卡并查看管道的状态(进度、错误和数据流),选择“监视”。 To switch to the Monitor tab and view the status of the pipeline (that is, progress, errors, and data flow), select Monitor.

    b.b. 若要编辑链接服务、数据集和管道,请选择“编辑管道”。To edit linked services, datasets, and pipelines, select Edit Pipeline.

    c.c. 选择“完成”以完成数据复制任务。Select Finish to complete the copy data task.


后续步骤Next steps