文本拆分认知技能

文本拆分技能将文本分解为文本区块。 你可以指定是要将文件分解为句子还是特定长度的页面。 位置元数据（如偏移量和序号位置）也可用作输出。 如果下游的其他技能（例如将数据区块传递给 Azure OpenAI 和其他模型提供程序的嵌入模型的嵌入技能）有最大文本长度要求，则此技能将非常有用。 有关这种情况的更多信息，请参阅为矢量搜索将大型文档分块。

有几个参数是特定于版本的。 技能参数表注明了引入参数的 API 版本，以便了解是否需要版本升级。 要使用特定版本的功能，如 2024-09-01-preview 中的“标记分块”，可以使用 Azure 门户，或以 REST API 版本为目标，或者查看 Azure SDK 更改日志，以确定它是否支持该功能。

Azure 门户支持大多数预览功能，并可用于创建或更新技能组。 要更新文本拆分技能，请编辑技能组 JSON 定义以添加新的预览参数。

@odata.type

Microsoft.Skills.Text.SplitSkill

技能参数

参数区分大小写。

技能输入

技能输出

示例定义

{
    "name": "SplitSkill", 
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill", 
    "description": "A skill that splits text into chunks", 
    "context": "/document", 
    "defaultLanguageCode": "en", 
    "textSplitMode": "pages", 
    "unit": "azureOpenAITokens", 
    "azureOpenAITokenizerParameters":{ 
        "encoderModelName":"cl100k_base", 
        "allowedSpecialTokens": [ 
            "[START]", 
            "[END]" 
        ] 
    },
    "maximumPageLength": 512,
    "inputs": [
        {
            "name": "text",
            "source": "/document/text"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "pages"
        }
    ]
}

示例输入

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Azure employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several pages...",
                "languageCode": "en"
            }
        }
    ]
}

示例输出

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "pages": [
                    "This is the loan...",
                    "In the next section, we continue..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    }
                ],
                "lengths": [
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    },
                    {
                        "utf8": 211,
                        "utf16": 211,
                        "codePoint": 211
                    }
                ],
                "ordinalPositions" : [
                    1,
                    2
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "pages": [
                    "This is the second document...",
                    "In the next section of the second doc..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    }
                ],
                "lengths": [
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    },
                    {
                        "utf8": 209,
                        "utf16": 209,
                        "codePoint": 209
                    }
                ],
                 "ordinalPositions" : [
                    1,
                    2
                ]
            }
        }
    ]
}

分块和矢量化示例

此示例适用于集成矢量化。

• pageOverlapLength：重叠文本在数据分块方案中非常有用，因为它可以保留从同一文档生成的块之间的连续性。

• maximumPagesToTake：对页面输入的限制在矢量化场景中非常有用，因为它有助于确保不会超过提供矢量化的嵌入模型的最大输入限制。

示例定义

此定义将 pageOverlapLength 增加了 100 个字符，并且将 maximumPagesToTake 加 1。

假设 maximumPageLength 为 5,000 个字符（默认值），则 "maximumPagesToTake": 1 会处理每个源文档的前 5,000 个字符。

此示例将 textItems 设置为 myPages 至 targetName。 由于已设置 targetName，因此应使用 myPages 值来从文本拆分技能中选择输出。 /document/myPages/* 用于下游技能、索引器输出字段映射、知识存储投影和索引投影。

{
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
    "textSplitMode" : "pages", 
    "maximumPageLength": 1000,
    "pageOverlapLength": 100,
    "maximumPagesToTake": 1,
    "defaultLanguageCode": "en",
    "inputs": [
        {
            "name": "text",
            "source": "/document/content"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "myPages"
        }
    ]
}

示例输入（与前面的示例相同）

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Azure employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several sections...",
                "languageCode": "en"
            }
        }
    ]
}

示例输出（注意重叠内容）

在每个“textItems”数组中，第一项的尾随文本将被复制到第二项的开头。

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "myPages": [
                    "This is the loan...Here is the overlap part",
                    "Here is the overlap part...In the next section, we continue..."
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "myPages": [
                    "This is the second document...Here is the overlap part...",
                    "Here is the overlap part...In the next section of the second doc..."
                ]
            }
        }
    ]
}

错误案例

如果某语言不受支持，则会生成警告。

另请参阅

• 内置技能
• 如何定义技能组