使用 Azure 数据工厂向/从文件系统复制数据Copy data to or from a file system by using Azure Data Factory

• 01/04/2021
• • W

适用于： Azure 数据工厂 Azure Synapse Analytics

本文概述了如何将数据复制到文件系统以及从文件系统复制数据。This article outlines how to copy data to and from file system. 若要了解 Azure 数据工厂，请阅读介绍性文章。To learn about Azure Data Factory, read the introductory article.

支持的功能Supported capabilities

以下活动支持此文件系统连接器：This file system connector is supported for the following activities:

• 带有支持的源或接收器矩阵的复制活动Copy activity with supported source/sink matrix
• Lookup 活动Lookup activity
• GetMetadata 活动GetMetadata activity
• Delete 活动Delete activity

具体而言，此文件系统连接器支持：Specifically, this file system connector supports:

• 从/向本地计算机或网络文件共享复制文件。Copying files from/to local machine or network file share. 若要使用 Linux 文件共享，请在 Linux 服务器上安装 Samba。To use a Linux file share, install Samba on your Linux server.
• 使用 Windows 身份验证复制文件。Copying files using Windows authentication.
• 按原样复制文件，或者使用支持的文件格式和压缩编解码器分析/生成文件。Copying files as-is or parsing/generating files with the supported file formats and compression codecs.

先决条件Prerequisites

如果数据存储位于本地网络、Azure 虚拟网络或 Amazon Virtual Private Cloud 内部，则需要配置自承载集成运行时才能连接到该数据存储。If your data store is located inside an on-premises network, an Azure virtual network, or Amazon Virtual Private Cloud, you need to configure a self-hosted integration runtime to connect to it.

另外，如果数据存储是托管的云数据服务，可以使用 Azure 集成运行时。Alternatively, if your data store is a managed cloud data service, you can use Azure integration runtime. 如果访问范围限制为防火墙规则中允许的 IP，你可以选择将 Azure Integration Runtime IP 添加到允许列表。If the access is restricted to IPs that are approved in the firewall rules, you can add Azure Integration Runtime IPs into the allow list.

要详细了解网络安全机制和数据工厂支持的选项，请参阅数据访问策略。For more information about the network security mechanisms and options supported by Data Factory, see Data access strategies.

入门Getting started

若要使用管道执行复制活动，可以使用以下工具或 SDK 之一：To perform the Copy activity with a pipeline, you can use one of the following tools or SDKs:

• 复制数据工具The Copy Data tool
• Azure 门户The Azure portal
• .NET SDKThe .NET SDK
• Python SDKThe Python SDK
• Azure PowerShellAzure PowerShell
• REST APIThe REST API
• Azure 资源管理器模板The Azure Resource Manager template

对于特定于文件系统的数据工厂实体，以下部分提供了有关用于定义这些实体的属性详细信息。The following sections provide details about properties that are used to define Data Factory entities specific to file system.

链接服务属性Linked service properties

文件系统链接服务支持以下属性：The following properties are supported for file system linked service:

链接服务和数据集定义示例Sample linked service and dataset definitions

示例：Example:

{
    "name": "FileLinkedService",
    "properties": {
        "type": "FileServer",
        "typeProperties": {
            "host": "<host>",
            "userid": "<domain>\\<user>",
            "password": {
                "type": "SecureString",
                "value": "<password>"
            }
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

数据集属性Dataset properties

有关可用于定义数据集的各部分和属性的完整列表，请参阅数据集一文。For a full list of sections and properties available for defining datasets, see the Datasets article.

Azure 数据工厂支持以下文件格式。Azure Data Factory supports the following file formats. 请参阅每一篇介绍基于格式的设置的文章。Refer to each article for format-based settings.

• Avro 格式Avro format
• 二进制格式Binary format
• 带分隔符的文本格式Delimited text format
• Excel 格式Excel format
• JSON 格式JSON format
• ORC 格式ORC format
• Parquet 格式Parquet format
• XML 格式XML format

在基于格式的数据集中的 location 设置下，文件系统支持以下属性：The following properties are supported for file system under location settings in format-based dataset:

示例：Example:

{
    "name": "DelimitedTextDataset",
    "properties": {
        "type": "DelimitedText",
        "linkedServiceName": {
            "referenceName": "<File system linked service name>",
            "type": "LinkedServiceReference"
        },
        "schema": [ < physical schema, optional, auto retrieved during authoring > ],
        "typeProperties": {
            "location": {
                "type": "FileServerLocation",
                "folderPath": "root/folder/subfolder"
            },
            "columnDelimiter": ",",
            "quoteChar": "\"",
            "firstRowAsHeader": true,
            "compressionCodec": "gzip"
        }
    }
}

复制活动属性Copy activity properties

有关可用于定义活动的各部分和属性的完整列表，请参阅管道一文。For a full list of sections and properties available for defining activities, see the Pipelines article. 本部分提供文件系统源和接收器支持的属性列表。This section provides a list of properties supported by file system source and sink.

文件系统作为源File system as source

Azure 数据工厂支持以下文件格式。Azure Data Factory supports the following file formats. 请参阅每一篇介绍基于格式的设置的文章。Refer to each article for format-based settings.

• Avro 格式Avro format
• 二进制格式Binary format
• 带分隔符的文本格式Delimited text format
• Excel 格式Excel format
• JSON 格式JSON format
• ORC 格式ORC format
• Parquet 格式Parquet format
• XML 格式XML format

在基于格式的复制源中的 storeSettings 设置下，文件系统支持以下属性：The following properties are supported for file system under storeSettings settings in format-based copy source:

示例：Example:

"activities":[
    {
        "name": "CopyFromFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<Delimited text input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "DelimitedTextSource",
                "formatSettings":{
                    "type": "DelimitedTextReadSettings",
                    "skipLineCount": 10
                },
                "storeSettings":{
                    "type": "FileServerReadSettings",
                    "recursive": true,
                    "wildcardFolderPath": "myfolder*A",
                    "wildcardFileName": "*.csv"
                }
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

文件系统作为接收器File system as sink

Azure 数据工厂支持以下文件格式。Azure Data Factory supports the following file formats. 请参阅每一篇介绍基于格式的设置的文章。Refer to each article for format-based settings.

• Avro 格式Avro format
• 二进制格式Binary format
• 带分隔符的文本格式Delimited text format
• JSON 格式JSON format
• ORC 格式ORC format
• Parquet 格式Parquet format

在基于格式的复制接收器中的 storeSettings 设置下，文件系统支持以下属性：The following properties are supported for file system under storeSettings settings in format-based copy sink:

示例：Example:

"activities":[
    {
        "name": "CopyToFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<Parquet output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "ParquetSink",
                "storeSettings":{
                    "type": "FileServerWriteSettings",
                    "copyBehavior": "PreserveHierarchy"
                }
            }
        }
    }
]

文件夹和文件筛选器示例Folder and file filter examples

本部分介绍使用通配符筛选器生成文件夹路径和文件名的行为。This section describes the resulting behavior of the folder path and file name with wildcard filters.

文件列表示例File list examples

本部分介绍了在复制活动源中使用文件列表路径时的结果行为。This section describes the resulting behavior of using file list path in copy activity source.

假设有以下源文件夹结构，并且要复制加粗显示的文件：Assuming you have the following source folder structure and want to copy the files in bold:

recursive 和 copyBehavior 示例recursive and copyBehavior examples

本节介绍了将 recursive 和 copyBehavior 值进行不同组合所产生的复制操作行为。This section describes the resulting behavior of the Copy operation for different combinations of recursive and copyBehavior values.

Lookup 活动属性Lookup activity properties

若要了解有关属性的详细信息，请查看 Lookup 活动。To learn details about the properties, check Lookup activity.

GetMetadata 活动属性GetMetadata activity properties

若要了解有关属性的详细信息，请查看 GetMetadata 活动To learn details about the properties, check GetMetadata activity

Delete 活动属性Delete activity properties

若要详细了解这些属性，请查看 Delete 活动To learn details about the properties, check Delete activity

旧模型Legacy models

旧数据集模型Legacy dataset model

示例：Example:

{
    "name": "FileSystemDataset",
    "properties": {
        "type": "FileShare",
        "linkedServiceName":{
            "referenceName": "<file system linked service name>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "folder/subfolder/",
            "fileName": "*",
            "modifiedDatetimeStart": "2018-12-01T05:00:00Z",
            "modifiedDatetimeEnd": "2018-12-01T06:00:00Z",
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "\n"
            },
            "compression": {
                "type": "GZip",
                "level": "Optimal"
            }
        }
    }
}

旧复制活动源模型Legacy copy activity source model

示例：Example:

"activities":[
    {
        "name": "CopyFromFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<file system input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "FileSystemSource",
                "recursive": true
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

旧复制活动接收器模型Legacy copy activity sink model

示例：Example:

"activities":[
    {
        "name": "CopyToFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<file system output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "FileSystemSink",
                "copyBehavior": "PreserveHierarchy"
            }
        }
    }
]

后续步骤Next steps

有关 Azure 数据工厂中复制活动支持作为源和接收器的数据存储的列表，请参阅支持的数据存储。For a list of data stores supported as sources and sinks by the copy activity in Azure Data Factory, see supported data stores.