Azure 数据工厂中的 JSON 格式JSON format in Azure Data Factory
Azure 数据工厂
Azure Synapse Analytics
如果要 分析 JSON 文件或以 JSON 格式写入数据,请遵循本文中的说明。Follow this article when you want to parse the JSON files or write the data into JSON format.
以下连接器支持 JSON 格式:Amazon S3、Azure Blob、Azure Data Lake Storage Gen2、Azure 文件存储、文件系统、FTP、Google 云存储、HDFS、HTTP 和 SFTP。JSON format is supported for the following connectors: Amazon S3, Azure Blob, Azure Data Lake Storage Gen2, Azure File Storage, File System, FTP, Google Cloud Storage, HDFS, HTTP, and SFTP.
数据集属性Dataset properties
有关可用于定义数据集的各部分和属性的完整列表,请参阅数据集一文。For a full list of sections and properties available for defining datasets, see the Datasets article. 本部分提供 JSON 数据集支持的属性列表。This section provides a list of properties supported by the JSON dataset.
属性Property | 说明Description | 必须Required |
---|---|---|
typetype | 数据集的 type 属性必须设置为 Json。The type property of the dataset must be set to Json. | 是Yes |
locationlocation | 文件的位置设置。Location settings of the file(s). 每个基于文件的连接器在 location 下都有其自己的位置类型和支持的属性。Each file-based connector has its own location type and supported properties under location . 请在连接器文章 -> 数据集属性部分中查看详细信息。See details in connector article -> Dataset properties section. |
是Yes |
encodingNameencodingName | 用于读取/写入测试文件的编码类型。The encoding type used to read/write test files. 可用的值如下:"UTF-8"、"UTF-16"、"UTF-16BE"、"UTF-32"、"UTF-32BE"、"US-ASCII"、"UTF-7"、"BIG5"、"EUC-JP"、"EUC-KR"、"GB2312"、"GB18030"、"JOHAB"、"SHIFT-JIS"、"CP875"、"CP866"、"IBM00858"、"IBM037"、"IBM273"、"IBM437"、"IBM500"、"IBM737"、"IBM775"、"IBM850"、"IBM852"、"IBM855"、"IBM857"、"IBM860"、"IBM861"、"IBM863"、"IBM864"、"IBM865"、"IBM869"、"IBM870"、"IBM01140"、"IBM01141"、"IBM01142"、"IBM01143"、"IBM01144"、"IBM01145"、"IBM01146"、"IBM01147"、"IBM01148"、"IBM01149"、"ISO-2022-JP"、"ISO-2022-KR"、"ISO-8859-1"、"ISO-8859-2"、"ISO-8859-3"、"ISO-8859-4"、"ISO-8859-5"、"ISO-8859-6"、"ISO-8859-7"、"ISO-8859-8"、"ISO-8859-9"、"ISO-8859-13"、"ISO-8859-15"、"WINDOWS-874"、"WINDOWS-1250"、"WINDOWS-1251"、"WINDOWS-1252"、"WINDOWS-1253"、"WINDOWS-1254"、"WINDOWS-1255"、"WINDOWS-1256"、"WINDOWS-1257"、"WINDOWS-1258"。Allowed values are as follows: "UTF-8", "UTF-16", "UTF-16BE", "UTF-32", "UTF-32BE", "US-ASCII", "UTF-7", "BIG5", "EUC-JP", "EUC-KR", "GB2312", "GB18030", "JOHAB", "SHIFT-JIS", "CP875", "CP866", "IBM00858", "IBM037", "IBM273", "IBM437", "IBM500", "IBM737", "IBM775", "IBM850", "IBM852", "IBM855", "IBM857", "IBM860", "IBM861", "IBM863", "IBM864", "IBM865", "IBM869", "IBM870", "IBM01140", "IBM01141", "IBM01142", "IBM01143", "IBM01144", "IBM01145", "IBM01146", "IBM01147", "IBM01148", "IBM01149", "ISO-2022-JP", "ISO-2022-KR", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-13", "ISO-8859-15", "WINDOWS-874", "WINDOWS-1250", "WINDOWS-1251", "WINDOWS-1252", "WINDOWS-1253", "WINDOWS-1254", "WINDOWS-1255", "WINDOWS-1256", "WINDOWS-1257", "WINDOWS-1258". |
否No |
compressioncompression | 用来配置文件压缩的属性组。Group of properties to configure file compression. 如果需要在活动执行期间进行压缩/解压缩,请配置此部分。Configure this section when you want to do compression/decompression during activity execution. | 否No |
typetype (在 compression 下)(under compression ) |
用来读取/写入 JSON 文件的压缩编解码器。The compression codec used to read/write JSON files. 允许的值为 bzip2、gzip、deflate、ZipDeflate、TarGzip、Tar、snappy 或 lz4 。Allowed values are bzip2, gzip, deflate, ZipDeflate, TarGzip, Tar, snappy, or lz4. 默认设置是不压缩。Default is not compressed. 注意 当前,复制活动不支持“snappy”和“lz4”,映射数据流不支持“ZipDeflate”、“TarGzip”和“Tar”。Note currently Copy activity doesn't support "snappy" & "lz4", and mapping data flow doesn't support "ZipDeflate"", "TarGzip" and "Tar". 注意,使用复制活动解压缩 ZipDeflate/TarGzip/Tar 文件并将其写入基于文件的接收器数据存储时,默认情况下文件将提取到 <path specified in dataset>/<folder named as source compressed file>/ 文件夹,对复制活动源使用 preserveZipFileNameAsFolder /preserveCompressionFileNameAsFolder 来控制是否以文件夹结构形式保留压缩文件名 。Note when using copy activity to decompress ZipDeflate/TarGzip/Tar file(s) and write to file-based sink data store, by default files are extracted to the folder:<path specified in dataset>/<folder named as source compressed file>/ , use preserveZipFileNameAsFolder /preserveCompressionFileNameAsFolder on copy activity source to control whether to preserve the name of the compressed file(s) as folder structure. |
否。No. |
levellevel (在 compression 下)(under compression ) |
压缩率。The compression ratio. 允许的值为 Optimal 或 Fastest。Allowed values are Optimal or Fastest. - Fastest:尽快完成压缩操作,不过,无法以最佳方式压缩生成的文件。- Fastest: The compression operation should complete as quickly as possible, even if the resulting file is not optimally compressed. - Optimal:以最佳方式完成压缩操作,不过,需要耗费更长的时间。- Optimal: The compression operation should be optimally compressed, even if the operation takes a longer time to complete. 有关详细信息,请参阅 Compression Level(压缩级别)主题。For more information, see Compression Level topic. |
否No |
下面是 Azure Blob 存储上的 JSON 数据集的示例:Below is an example of JSON dataset on Azure Blob Storage:
{
"name": "JSONDataset",
"properties": {
"type": "Json",
"linkedServiceName": {
"referenceName": "<Azure Blob Storage linked service name>",
"type": "LinkedServiceReference"
},
"schema": [ < physical schema, optional, retrievable during authoring > ],
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "containername",
"folderPath": "folder/subfolder",
},
"compression": {
"type": "gzip"
}
}
}
}
复制活动属性Copy activity properties
有关可用于定义活动的各部分和属性的完整列表,请参阅管道一文。For a full list of sections and properties available for defining activities, see the Pipelines article. 本部分提供 JSON 源和接收器支持的属性列表。This section provides a list of properties supported by the JSON source and sink.
若要了解如何从 JSON 文件中提取数据并将其映射到接收器数据存储/格式(反之亦然),请参阅架构映射。Learn about how to extract data from JSON files and map to sink data store/format or vice versa from schema mapping.
以 JSON 作为源JSON as source
复制活动的 __source* * 节支持以下属性。The following properties are supported in the copy activity __source** section.
属性Property | 说明Description | 必需Required |
---|---|---|
typetype | 复制活动源的 type 属性必须设置为 JSONSource。The type property of the copy activity source must be set to JSONSource. | 是Yes |
formatSettingsformatSettings | 一组属性。A group of properties. 请参阅下面的“JSON 读取设置”表。Refer to JSON read settings table below. | 否No |
storeSettingsstoreSettings | 有关如何从数据存储读取数据的一组属性。A group of properties on how to read data from a data store. 每个基于文件的连接器在 storeSettings 下都有其自己支持的读取设置。Each file-based connector has its own supported read settings under storeSettings . 请在连接器文章 -> 复制活动属性部分中查看详细信息。See details in connector article -> Copy activity properties section. |
否No |
formatSettings
下支持的“JSON 读取设置”:Supported JSON read settings under formatSettings
:
属性Property | 说明Description | 必需Required |
---|---|---|
typetype | formatSettings 的 type 必须设置为“JsonReadSettings”。The type of formatSettings must be set to JsonReadSettings. | 是Yes |
compressionPropertiescompressionProperties | 一组属性,指示如何为给定的压缩编解码器解压缩数据。A group of properties on how to decompress data for a given compression codec. | 否No |
preserveZipFileNameAsFolderpreserveZipFileNameAsFolder (在 compressionProperties ->type 下为 ZipDeflateReadSettings )(under compressionProperties ->type as ZipDeflateReadSettings ) |
当输入数据集配置了 ZipDeflate 压缩时适用。Applies when input dataset is configured with ZipDeflate compression. 指示是否在复制过程中以文件夹结构形式保留源 zip 文件名。Indicates whether to preserve the source zip file name as folder structure during copy. - 当设置为 true(默认值)时,数据工厂会将已解压缩的文件写入 <path specified in dataset>/<folder named as source zip file>/ 。- When set to true (default), Data Factory writes unzipped files to <path specified in dataset>/<folder named as source zip file>/ .- 当设置为 false 时,数据工厂会直接将未解压缩的文件写入 <path specified in dataset> 。- When set to false, Data Factory writes unzipped files directly to <path specified in dataset> . 请确保不同的源 zip 文件中没有重复的文件名,以避免产生冲突或出现意外行为。Make sure you don't have duplicated file names in different source zip files to avoid racing or unexpected behavior. |
否No |
preserveCompressionFileNameAsFolderpreserveCompressionFileNameAsFolder (在 compressionProperties ->type 下为 TarGZipReadSettings 或 TarReadSettings )(under compressionProperties ->type as TarGZipReadSettings or TarReadSettings ) |
当输入数据集配置了 TarGzip/Tar 压缩时适用 。Applies when input dataset is configured with TarGzip/Tar compression. 指示是否在复制过程中以文件夹结构形式保留源压缩文件名。Indicates whether to preserve the source compressed file name as folder structure during copy. - 当设置为 true(默认值)时,数据工厂会将已解压缩的文件写入 <path specified in dataset>/<folder named as source compressed file>/ 。- When set to true (default), Data Factory writes decompressed files to <path specified in dataset>/<folder named as source compressed file>/ . - 当设置为 false 时,数据工厂会直接将已解压缩的文件写入 <path specified in dataset> 。- When set to false, Data Factory writes decompressed files directly to <path specified in dataset> . 请确保不同的源文件中没有重复的文件名,以避免产生冲突或出现意外行为。Make sure you don't have duplicated file names in different source files to avoid racing or unexpected behavior. |
否No |
JSON 作为接收器JSON as sink
复制活动的 _sink* 节支持以下属性。The following properties are supported in the copy activity __sink** section.
propertiesProperty | 说明Description | 必需Required |
---|---|---|
typetype | 复制活动源的 type 属性必须设置为 JSONSink。The type property of the copy activity source must be set to JSONSink. | 是Yes |
formatSettingsformatSettings | 一组属性。A group of properties. 请参阅下面的“JSON 写入设置”表。Refer to JSON write settings table below. | 否No |
storeSettingsstoreSettings | 有关如何将数据写入到数据存储的一组属性。A group of properties on how to write data to a data store. 每个基于文件的连接器在 storeSettings 下都有其自身支持的写入设置。Each file-based connector has its own supported write settings under storeSettings . 请在连接器文章 -> 复制活动属性部分中查看详细信息。See details in connector article -> Copy activity properties section. |
否No |
formatSettings
下支持的 JSON 写入设置:Supported JSON write settings under formatSettings
:
属性Property | 说明Description | 必需Required |
---|---|---|
typetype | formatSettings 的类型必须设置为 JsonWriteSettings。The type of formatSettings must be set to JsonWriteSettings. | 是Yes |
filePatternfilePattern | 指示每个 JSON 文件中存储的数据模式。Indicate the pattern of data stored in each JSON file. 允许的值为:setOfObjects (JSON Lines) 和 arrayOfObjects。Allowed values are: setOfObjects (JSON Lines) and arrayOfObjects. 默认 值为 setOfObjects。The default value is setOfObjects. 请参阅 JSON 文件模式部分,详细了解这些模式。See JSON file patterns section for details about these patterns. | 否No |
JSON 文件模式JSON file patterns
从 JSON 文件复制数据时,复制活动可自动检测并分析以下 JSON 文件模式。When copying data from JSON files, copy activity can automatically detect and parse the following patterns of JSON files. 将数据写入 JSON 文件时,可以在复制活动接收器上配置文件模式。When writing data to JSON files, you can configure the file pattern on copy activity sink.
类型 I:setOfObjectsType I: setOfObjects
每个文件都包含单一对象、JSON Lines 或串联的对象。Each file contains single object, JSON lines, or concatenated objects.
单一对象 JSON 示例single object JSON example
{ "time": "2015-04-29T07:12:20.9100000Z", "callingimsi": "466920403025604", "callingnum1": "678948008", "callingnum2": "567834760", "switch1": "China", "switch2": "Germany" }
JSON Lines(接收器的默认值)JSON Lines (default for sink)
{"time":"2015-04-29T07:12:20.9100000Z","callingimsi":"466920403025604","callingnum1":"678948008","callingnum2":"567834760","switch1":"China","switch2":"Germany"} {"time":"2015-04-29T07:13:21.0220000Z","callingimsi":"466922202613463","callingnum1":"123436380","callingnum2":"789037573","switch1":"US","switch2":"UK"} {"time":"2015-04-29T07:13:21.4370000Z","callingimsi":"466923101048691","callingnum1":"678901578","callingnum2":"345626404","switch1":"Germany","switch2":"UK"}
串连的 JSON 示例concatenated JSON example
{ "time": "2015-04-29T07:12:20.9100000Z", "callingimsi": "466920403025604", "callingnum1": "678948008", "callingnum2": "567834760", "switch1": "China", "switch2": "Germany" } { "time": "2015-04-29T07:13:21.0220000Z", "callingimsi": "466922202613463", "callingnum1": "123436380", "callingnum2": "789037573", "switch1": "US", "switch2": "UK" } { "time": "2015-04-29T07:13:21.4370000Z", "callingimsi": "466923101048691", "callingnum1": "678901578", "callingnum2": "345626404", "switch1": "Germany", "switch2": "UK" }
类型 II:arrayOfObjectsType II: arrayOfObjects
每个文件包含对象的数组。Each file contains an array of objects.
[ { "time": "2015-04-29T07:12:20.9100000Z", "callingimsi": "466920403025604", "callingnum1": "678948008", "callingnum2": "567834760", "switch1": "China", "switch2": "Germany" }, { "time": "2015-04-29T07:13:21.0220000Z", "callingimsi": "466922202613463", "callingnum1": "123436380", "callingnum2": "789037573", "switch1": "US", "switch2": "UK" }, { "time": "2015-04-29T07:13:21.4370000Z", "callingimsi": "466923101048691", "callingnum1": "678901578", "callingnum2": "345626404", "switch1": "Germany", "switch2": "UK" } ]
映射数据流属性Mapping data flow properties
在映射数据流中,你可以在下列数据存储中以 JSON 格式进行读取和写入:Azure Blob 存储和 Azure Data Lake Storage Gen2。In mapping data flows, you can read and write to JSON format in the following data stores: Azure Blob Storage, and Azure Data Lake Storage Gen2.
源属性Source properties
下表列出了 json 源支持的属性。The below table lists the properties supported by a json source. 你可以在“源选项”选项卡中编辑这些属性。You can edit these properties in the Source options tab.
名称Name | 说明Description | 必需Required | 允许的值Allowed values | 数据流脚本属性Data flow script property |
---|---|---|---|---|
通配符路径Wild card paths | 将处理与通配符路径匹配的所有文件。All files matching the wildcard path will be processed. 替代在数据集中设置的文件夹和文件路径。Overrides the folder and file path set in the dataset. | 否no | String[]String[] | wildcardPathswildcardPaths |
分区根路径Partition root path | 对于已分区的文件数据,你可以输入分区根路径,以便将已分区的文件夹作为列读取For file data that is partitioned, you can enter a partition root path in order to read partitioned folders as columns | 否no | 字符串String | partitionRootPathpartitionRootPath |
文件列表List of files | 源是否指向列出了要处理的文件的文本文件Whether your source is pointing to a text file that lists files to process | 否no | true 或 false true or false |
fileListfileList |
用于存储文件名的列Column to store file name | 使用源文件名称和路径创建一个新列Create a new column with the source file name and path | 否no | 字符串String | rowUrlColumnrowUrlColumn |
完成后After completion | 在处理后删除或移动文件。Delete or move the files after processing. 文件路径从容器根目录开始File path starts from the container root | 否no | Delete:true 或 false Delete: true or false Move: ['<from>', '<to>'] Move: ['<from>', '<to>'] |
purgeFilespurgeFiles moveFilesmoveFiles |
按上次修改时间筛选Filter by last modified | 选择根据上次更改时间筛选文件Choose to filter files based upon when they were last altered | 否no | 时间戳Timestamp | ModifiedAftermodifiedAfter modifiedBeforemodifiedBefore |
单个文档Single document | 映射数据流从每个文件中读取一个 JSON 文档Mapping data flows read one JSON document from each file | 否no | true 或 false true or false |
singleDocumentsingleDocument |
不带引号的列名称Unquoted column names | 如果选择了“不带引号的列名称”,则映射数据流会读取不在引号中的 JSON 列。If Unquoted column names is selected, mapping data flows reads JSON columns that aren't surrounded by quotes. | 否no | true 或 false true or false |
unquotedColumnNamesunquotedColumnNames |
具有注释Has comments | 如果 JSON 数据具有 C 或 C++ 样式的注释,请选择“具有注释”Select Has comments if the JSON data has C or C++ style commenting | 否no | true 或 false true or false |
asCommentsasComments |
带单引号Single quoted | 读取不在引号中的 JSON 列Reads JSON columns that aren't surrounded by quotes | 否no | true 或 false true or false |
singleQuotedsingleQuoted |
使用反斜杠转义Backslash escaped | 如果使用反斜杠对 JSON 数据中的字符进行转义,请选择“使用反斜杠转义”Select Backslash escaped if backslashes are used to escape characters in the JSON data | 否no | true 或 false true or false |
backslashEscapebackslashEscape |
允许找不到文件Allow no files found | 如果为 true,则找不到文件时不会引发错误If true, an error is not thrown if no files are found | 否no | true 或 false true or false |
ignoreNoFilesFoundignoreNoFilesFound |
源格式选项Source format options
在数据流中使用 JSON 数据集作为源时,你可以设置五个额外的设置。Using a JSON dataset as a source in your data flow allows you to set five additional settings. 可以在“源选项”选项卡的“JSON 设置”可折叠列表下找到这些设置。 。These settings can be found under the JSON settings accordion in the Source Options tab.
默认Default
默认情况下,将使用以下格式读取 JSON 数据。By default, JSON data is read in the following format.
{ "json": "record 1" }
{ "json": "record 2" }
{ "json": "record 3" }
单个文档Single document
如果选择了“单个文档”,则映射数据流会从每个文件中读取一个 JSON 文档。If Single document is selected, mapping data flows read one JSON document from each file.
File1.json
{
"json": "record 1"
}
File2.json
{
"json": "record 2"
}
File3.json
{
"json": "record 3"
}
备注
如果预览你的 JSON 数据时数据流引发了“corrupt_record”错误,则你的数据可能在 JSON 文件中包含单个文档。If data flows throw an error stating "corrupt_record" when previewing your JSON data, it is likely that your data contains contains a single document in your JSON file. 设置“单个文档”应当能够清除该错误。Setting "single document" should clear that error.
不带引号的列名称Unquoted column names
如果选择了“不带引号的列名称”,则映射数据流会读取不在引号中的 JSON 列。If Unquoted column names is selected, mapping data flows reads JSON columns that aren't surrounded by quotes.
{ json: "record 1" }
{ json: "record 2" }
{ json: "record 3" }
具有注释Has comments
如果 JSON 数据具有 C 或 C++ 样式的注释,请选择“具有注释”。Select Has comments if the JSON data has C or C++ style commenting.
{ "json": /** comment **/ "record 1" }
{ "json": "record 2" }
{ /** comment **/ "json": "record 3" }
带单引号Single quoted
如果 JSON 字段和值使用单引号而非双引号,请选择“带单引号”。Select Single quoted if the JSON fields and values use single quotes instead of double quotes.
{ 'json': 'record 1' }
{ 'json': 'record 2' }
{ 'json': 'record 3' }
使用反斜杠转义Backslash escaped
如果使用反斜杠对 JSON 数据中的字符进行转义,请选择“使用反斜杠转义”。Select Backslash escaped if backslashes are used to escape characters in the JSON data.
{ "json": "record 1" }
{ "json": "\} \" \' \\ \n \\n record 2" }
{ "json": "record 3" }
接收器属性Sink Properties
下表列出了 json 接收器支持的属性。The below table lists the properties supported by a json sink. 可以在“设置”选项卡中编辑这些属性。You can edit these properties in the Settings tab.
名称Name | 说明Description | 必需Required | 允许的值Allowed values | 数据流脚本属性Data flow script property |
---|---|---|---|---|
清除文件夹Clear the folder | 在写入之前是否清除目标文件夹If the destination folder is cleared prior to write | 否no | true 或 false true or false |
truncatetruncate |
文件名选项File name option | 写入的数据的命名格式。The naming format of the data written. 默认情况下,每个分区有一个 part-#####-tid-<guid> 格式的文件By default, one file per partition in format part-#####-tid-<guid> |
否no | 模式:StringPattern: String 按分区:String[]Per partition: String[] 作为列中的数据:StringAs data in column: String 输出到单个文件: ['<fileName>'] Output to single file: ['<fileName>'] |
filePatternfilePattern partitionFileNamespartitionFileNames rowUrlColumnrowUrlColumn partitionFileNamespartitionFileNames |
在派生列中创建 JSON 结构Creating JSON structures in a derived column
你可以通过派生列表达式生成器将复杂的列添加到数据流中。You can add a complex column to your data flow via the derived column expression builder. 在派生列转换中,添加一个新列并通过单击蓝色框打开表达式生成器。In the derived column transformation, add a new column and open the expression builder by clicking on the blue box. 若要使列成为复杂列,可以手动输入 JSON 结构,或者使用 UX 以交互方式添加子列。To make a column complex, you can enter the JSON structure manually or use the UX to add subcolumns interactively.
使用表达式生成器 UXUsing the expression builder UX
在输出架构侧窗格中,将鼠标指针悬停在某个列上,并单击加号图标。In the output schema side pane, hover over a column and click the plus icon. 选择“添加子列”,使列成为复杂类型。Select Add subcolumn to make the column a complex type.
你可以通过相同的方式添加更多的列和子列。You can add additional columns and subcolumns in the same way. 对于每个非复杂字段,可以在右侧的表达式编辑器中添加表达式。For each non-complex field, an expression can be added in the expression editor to the right.
手动输入 JSON 结构Entering the JSON structure manually
若要手动添加 JSON 结构,请添加一个新列,然后在编辑器中输入表达式。To manually add a JSON structure, add a new column and enter the expression in the editor. 表达式采用以下常规格式:The expression follows the following general format:
@(
field1=0,
field2=@(
field1=0
)
)
如果为名为“complexColumn”的列输入了此表达式,则会将其作为以下 JSON 写入到接收器:If this expression were entered for a column named "complexColumn", then it would be written to the sink as the following JSON:
{
"complexColumn": {
"field1": 0,
"field2": {
"field1": 0
}
}
}
完整分层定义的示例手动脚本Sample manual script for complete hierarchical definition
@(
title=Title,
firstName=FirstName,
middleName=MiddleName,
lastName=LastName,
suffix=Suffix,
contactDetails=@(
email=EmailAddress,
phone=Phone
),
address=@(
line1=AddressLine1,
line2=AddressLine2,
city=City,
state=StateProvince,
country=CountryRegion,
postCode=PostalCode
),
ids=[
toString(CustomerID), toString(AddressID), rowguid
]
)