教程:通过查看终结点话语来修复不确定的预测Tutorial: Fix unsure predictions by reviewing endpoint utterances

本教程介绍如何通过验证或更正 LUIS 不确定的、通过 LUIS HTTPS 终结点收到的言语来改进应用预测。In this tutorial, improve app predictions by verifying or correcting utterances, received via the LUIS HTTPS endpoint, that LUIS is unsure of. 在日常的计划性 LUIS 维护过程中,应该评审终结点言语。You should review endpoint utterances as a regular part of your scheduled LUIS maintenance.

此评审过程使 LUIS 能够了解你的应用域。This review process allows LUIS to learn your app domain. LUIS 将选择评审列表中显示的言语。LUIS selects the utterances that appear in the review list. 此列表具有以下特点:This list is:

  • 特定于应用。Specific to the app.
  • 旨在改进应用的预测准确性。Is meant to improve the app's prediction accuracy.
  • 应该定期进行审核。Should be reviewed on a periodic basis.

可以通过审核终结点表述来验证或纠正表述的已预测意向。By reviewing the endpoint utterances, you verify or correct the utterance's predicted intent.

本教程介绍如何执行下列操作:In this tutorial, you learn how to:

  • 导入示例应用Import example app
  • 查看终结点话语Review endpoint utterances
  • 训练和发布应用Train and publish app
  • 查询应用终结点以查看 LUIS JSON 响应Query endpoint of app to see LUIS JSON response

在本文中,可以使用免费 LUIS 帐户来创作 LUIS 应用程序。For this article, you can use the free LUIS account in order to author your LUIS application.

下载适用于应用的 JSON 文件Download JSON file for app

下载并保存应用 JSON 文件Download and save app JSON file.

导出适用于应用的 JSON 文件Import JSON file for app

  1. LUIS 门户上的“我的应用”页上,选择“+ 新建对话应用”,然后选择“导入为 JSON”。In the LUIS portal, on the My apps page, select + New app for conversation , then Import as JSON . 查找上一步中保存的 JSON 文件。Find the saved JSON file from the previous step. 无需更改应用的名称。You don't need to change the name of the app. 选择“完成”Select Done

  2. 选择“生成”,然后选择“意向”,以查看意向(LUIS 应用的主要构建基块) 。Select Build then Intents to see the intents, the main building blocks of a LUIS app.

    从“版本”页切换到“意向”页。

训练应用以将实体更改应用于应用Train the app to apply the entity changes to the app

  1. 在 LUIS 网站的右上方,选择“训练”按钮。In the top-right side of the LUIS website, select the Train button.

    “训练”按钮

  2. 当“训练”按钮上的状态指示器为绿色时,即表示训练完成。Training is complete when status indicator on the Train button is green.

发布应用以从 HTTP 终结点访问它Publish the app to access it from the HTTP endpoint

若要在聊天机器人或其他客户端应用程序中接收 LUIS 预测,需要将应用发布到预测终结点。In order to receive a LUIS prediction in a chat bot or other client application, you need to publish the app to the prediction endpoint.

  1. 在右上方的导航栏中选择“发布”。Select Publish in the top-right navigation.

    右上方菜单中的“LUIS 发布到终结点”按钮的屏幕截图

  2. 选择“生产”槽,然后选择“完成” 。Select the Production slot, then select Done.

    “LUIS 发布到终结点”的屏幕截图Screenshot of LUIS publish to endpoint

  3. 在通知中选择“访问终结点 URL”链接,转到“Azure 资源”页。 Select the Access your endpoint URLs link in the notification to go to the Azure Resources page. 终结点 URL 作为“示例查询”列出。The endpoint URLs are listed as the Example Query.

在终结点上添加言语Add utterances at the endpoint

此应用中包含意向和实体,但不包含任何终结点使用情况信息。In this app, you have intents and entities but you don't have any endpoint usage. 若要使用终结点言语评审来改进应用,必须提供此终结点使用情况信息。This endpoint usage is required to improve the app with the endpoint utterance review.

  1. 在“Azure 资源”页(左侧菜单)的“管理”部分(右上方菜单)中,复制“示例查询”URL,然后粘贴到新的浏览器选项卡中 。In the Manage section (top-right menu), on the Azure Resources page (left menu), copy the Example Query URL then paste into a new browser tab.

    终结点 URL 的格式如下所示,其中 APP-ID 和 KEY-ID 将替换为你自己的自定义子域、应用 ID 和终结点密钥:The endpoint URL looks like the following format, with your own custom subdomain, app ID, and endpoint key replacing APP-ID, and KEY-ID:

    https://YOUR-CUSTOM-SUBDOMAIN.api.cognitive.azure.cn/luis/prediction/v3.0/apps/APP-ID/slots/production/predict?subscription-key=KEY-ID&verbose=true&show-all-intents=true&log=true&query=YOUR_QUERY_HERE
    
  2. 转到地址栏中 URL 的末尾,将 YOUR_QUERY_HERE 替换为下表中的言语。Go to the end of the URL in the address bar and replace YOUR_QUERY_HERE with the utterances in the following table. 对于每个言语,请提交言语并获取结果。For each utterance, submit the utterance, and get the result. 然后,将结尾的言语替换为下一个言语。Then replace the utterance at the end with the next utterance.

    终结点言语Endpoint utterance 已调整意向Aligned intent
    I'm looking for a job with Natural Language Processing GetJobInformation
    I want to cancel on March 3 Utilities.Cancel
    When were HRF-123456 and hrf-234567 published in the last year? FindForm
    shift 123-45-6789 from Z-1242 to T-54672 MoveEmployee
    Please relocation jill-jones@mycompany.com from x-2345 to g-23456 MoveEmployee
    Here is my c.v. for the programmer job ApplyForJob
    This is the lead welder paperwork. ApplyForJob
    does form hrf-123456 cover the new dental benefits and medical plan FindForm
    Jill Jones work with the media team on the public portal was amazing EmployeeFeedback

    要审核的话语只有一个池,不管经常编辑哪个版本的话语,也不管在终结点上发布哪个版本的应用。There is a single pool of utterances to review, regardless of which version you are actively editing or which version of the app was published at the endpoint.

查看终结点话语Review endpoint utterances

评审终结点言语,使意向经过适当的调整。Review the endpoint utterances for correctly aligned intent. 尽管在所有版本中只需评审单个言语池,但适当调整意向的过程只会将示例言语添加到当前的活动模型。While there is a single pool of utterances to review across all versions, the process of correctly aligning the intent adds the example utterance to the current active model only.

  1. 在门户的“生成”部分,从左侧导航栏中选择“评审终结点言语”。 From the Build section of the portal, select Review endpoint utterances from the left navigation. 列表会筛选出 ApplyForJob 意向。The list is filtered for the ApplyForJob intent.

    左侧导航栏中“评审终结点言语”按钮的屏幕截图。

    I'm looking for a job with Natural Language Processing 这一言语不在正确的意向 GetJobInformation 中。This utterance, I'm looking for a job with Natural Language Processing, is not in the correct intent, GetJobInformation . 由于两个意向中作业名称和谓词的相似性,因此它被错误预测为 ApplyForJob。It has been mispredicted as ApplyForJob because of the similarity of job names and verbs in the two intents.

  2. 若要调整此言语,请选择正确的“已调整意向”:GetJobInformationTo align this utterance, select the correct Aligned Intent of GetJobInformation. 选择勾选标记将更改的言语添加到应用。Add the changed utterance to the app by selecting the checkmark.

    评审此意向中的剩余言语,并根据需要更正已调整的意向。Review the remaining utterances in this intent, correcting the aligned intent as needed. 使用本教程中的初始言语表来查看已调整的意向。Use the initial utterance table in this tutorial to view the aligned intent.

    “评审终结点言语”列表应不再包含已更正的言语。The Review endpoint utterances list should no longer have the corrected utterances. 如果显示了其他言语,请继续在列表中更正已调整的意向,直到列表为空。If more utterances appear, continue to work through the list, correcting aligned intents until the list is empty.

    对实体标签进行的任何更正是在调整意向后,通过“意向详细信息”页完成的。Any correction of entity labeling is done after the intent is aligned, from the Intent details page.

  3. 再次定型并发布应用。Train and publish the app again.

从终结点获取意向预测Get intent prediction from endpoint

若要验证适当调整的示例言语是否已改进应用的预测,请尝试使用一个与已更正的言语接近的言语。To verify the correctly aligned example utterances improved the app's prediction, try an utterance close to the corrected utterance.

  1. 在“Azure 资源”页(左侧菜单)的“管理”部分(右上方菜单)中,复制“示例查询”URL,然后粘贴到新的浏览器选项卡中 。In the Manage section (top-right menu), on the Azure Resources page (left menu), copy the Example Query URL then paste into a new browser tab.

    终结点 URL 的格式如下所示,其中 APP-ID 和 KEY-ID 将替换为你自己的自定义子域、应用 ID 和终结点密钥:The endpoint URL looks like the following format, with your own custom subdomain, app ID, and endpoint key replacing APP-ID, and KEY-ID:

    https://YOUR-CUSTOM-SUBDOMAIN.api.cognitive.azure.cn/luis/prediction/v3.0/apps/APP-ID/slots/production/predict?subscription-key=KEY-ID&verbose=true&show-all-intents=true&log=true&query=YOUR_QUERY_HERE
    
  2. 转到地址栏中 URL 的末尾,将 YOUR_QUERY_HERE 替换为 Are there any natural language processing jobs in my department right now?Go to the end of the URL in the address bar and replace YOUR_QUERY_HERE with Are there any natural language processing jobs in my department right now?.

     {
         "query": "Are there any natural language processing jobs in my department right now?",
         "prediction": {
             "topIntent": "GetJobInformation",
             "intents": {
                 "GetJobInformation": {
                     "score": 0.901367366
                 },
                 "ApplyForJob": {
                     "score": 0.0307973567
                 },
                 "EmployeeFeedback": {
                     "score": 0.0296942145
                 },
                 "MoveEmployee": {
                     "score": 0.00739785144
                 },
                 "FindForm": {
                     "score": 0.00449316856
                 },
                 "Utilities.Stop": {
                     "score": 0.00417657848
                 },
                 "Utilities.StartOver": {
                     "score": 0.00407167152
                 },
                 "Utilities.Help": {
                     "score": 0.003662492
                 },
                 "None": {
                     "score": 0.00335733569
                 },
                 "Utilities.Cancel": {
                     "score": 0.002225436
                 },
                 "Utilities.Confirm": {
                     "score": 0.00107437756
                 }
             },
             "entities": {
                 "keyPhrase": [
                     "natural language processing jobs",
                     "department"
                 ],
                 "datetimeV2": [
                     {
                         "type": "datetime",
                         "values": [
                             {
                                 "timex": "PRESENT_REF",
                                 "resolution": [
                                     {
                                         "value": "2020-07-02 21:45:50"
                                     }
                                 ]
                             }
                         ]
                     }
                 ],
                 "$instance": {
                     "keyPhrase": [
                         {
                             "type": "builtin.keyPhrase",
                             "text": "natural language processing jobs",
                             "startIndex": 14,
                             "length": 32,
                             "modelTypeId": 2,
                             "modelType": "Prebuilt Entity Extractor",
                             "recognitionSources": [
                                 "model"
                             ]
                         },
                         {
                             "type": "builtin.keyPhrase",
                             "text": "department",
                             "startIndex": 53,
                             "length": 10,
                             "modelTypeId": 2,
                             "modelType": "Prebuilt Entity Extractor",
                             "recognitionSources": [
                                 "model"
                             ]
                         }
                     ],
                     "datetimeV2": [
                         {
                             "type": "builtin.datetimeV2.datetime",
                             "text": "right now",
                             "startIndex": 64,
                             "length": 9,
                             "modelTypeId": 2,
                             "modelType": "Prebuilt Entity Extractor",
                             "recognitionSources": [
                                 "model"
                             ]
                         }
                     ]
                 }
             }
         }
     }
    

    适当调整不确定的言语后,将会预测正确的意向并返回 较高的评分Now that the unsure utterances are correctly aligned, the correct intent was predicted with a high score .

是否可以通过添加更多的表述来代替审核?Can reviewing be replaced by adding more utterances?

你可能会想,为何不添加更多的示例表述呢?You may wonder why not add more example utterances. 审核终结点表述的目的是什么?What is the purpose of reviewing endpoint utterances? 在实际的 LUIS 应用中,终结点表述来自各个用户,其遣词造句方式与你并不相同。In a real-world LUIS app, the endpoint utterances are from users with word choice and arrangement you haven't used yet. 如果同样的词汇和句式多次使用,则原始预测的百分比会更高。If you had used the same word choice and arrangement, the original prediction would have a higher percentage.

为何顶级意向会包含在表述列表中?Why is the top intent on the utterance list?

某些终结点表述在审核列表中的预测分数会很高。Some of the endpoint utterances will have a high prediction score in the review list. 仍需审核并验证这些表述。You still need to review and verify those utterances. 这些顶级意向之所以在列表中,是因为次高分数意向的分数与顶级意向的分数很接近。They are on the list because the next highest intent had a score too close to the top intent score. 你希望两个顶级意向之间存在大约 15% 的差异。You want about 15% difference between the two top intents.

清理资源Clean up resources

不再需要 LUIS 应用时,请将其删除。When no longer needed, delete the LUIS app. 为此,请在左上角的菜单中选择“我的应用”。To do so, select My apps from the top left menu. 在应用列表中选择应用名称右侧的省略号 (...),然后选择“删除”。Select the ellipsis (...) to the right of the app name in the app list, select Delete. 在弹出的“删除应用?”对话框中,选择“确定”。On the pop-up dialog Delete app?, select Ok.

后续步骤Next steps

在本教程中,你已审核了在终结点提交的、LUIS 不确定的表述。In this tutorial, you reviewed utterances submitted at the endpoint, that LUIS was unsure of. 在验证这些表述并将其作为示例表述移动到正确的意向中后,LUIS 将可以提高预测准确度。Once these utterances have been verified and moved into the correct intents as example utterances, LUIS will improve the prediction accuracy.