Quickstart: custom question answering

You can create a custom question answering project from your own content, such as FAQs or product manuals. This article includes an example of creating a custom question answering project from a product manual, to answer questions.

Prerequisites

Create your first custom question answering project

1. Sign in to the Language Studio with your Azure credentials.

2. Scroll down to the Answer questions section and select Open custom question answering.

   

3. If your resource is not yet connected to Azure Search select Connect to Azure Search. This will open a new browser tab to Features pane of your resource in the Azure portal.

   

4. Select Enable custom question answering, choose the Azure Search resource to link to, and then select Apply.

   

5. Return to the Language Studio tab. You may need to refresh this page for it to register the change to your resource. Select Create new project.

6. Choose the option I want to set the language for all projects created in this resource > select English > Select Next.

7. Enter a project name of Sample-project, a description of My first question answering project, and leave the default answer with a setting of No answer found.

8. Review your choices and select Create project

9. From the Manage sources page select Add source > URLS.

10. Select Add url enter the following values and then select Add all:

    URL Name	URL Value
    Surface Book User Guide	https://download.microsoft.com/download/7/B/1/7B10C82E-F520-4080-8516-5CF0D803EEE0/surface-book-user-guide-EN.pdf

    The extraction process takes a few moments to read the document and identify questions and answers.

    After successfully adding the source, you can then edit the source contents to add more custom question answer sets.

Test your project

1. Select the link to your source, this will open the edit project page.

2. Select Test from the menu bar > Enter the question How do I setup my surface book?. An answer will be generated based on the question answer pairs that were automatically identified and extracted from your source URL:

   

   If you check the box for include short answer response you will also see a precise answer, if available, along with the answer passage in the test pane when you ask a question.

3. Select Inspect to examine the response in more detail. The test window is used to test your changes to your project before deploying your project.

   

   From the Inspect interface, you can see the level of confidence that this response will answer the question and directly edit a given question and answer response pair.

Deploy your project

1. Select the Deploy project icon to enter the deploy project menu.

   

   When you deploy a project, the contents of your project move from the test index to a prod index in Azure Search.

2. Select Deploy > and then when prompted select Deploy again.

   

   Your project is now successfully deployed. You can use the endpoint to answer questions in your own custom application to answer or in a bot.

Prerequisites

• The current version of cURL. Several command-line switches are used in the quickstarts, which are noted in the cURL documentation.
• Azure subscription - Create one for trial
• Custom question answering requires a Language resource with the custom question answering feature enabled to generate an API key and endpoint.
  • After your Language resource deploys, select Go to resource. You will need the key and endpoint from the resource you create to connect to the API. Paste your key and endpoint into the code below later in the quickstart.
• To create a Language resource with Azure CLI provide the following additional properties: --api-properties qnaAzureSearchEndpointId=/subscriptions/<azure-subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Search/searchServices/<azure-search-service-name> qnaAzureSearchEndpointKey=<azure-search-service-auth-key>
• An existing project to query. If you have not setup a project, you can follow the instructions in the Language Studio quickstart. Or add a project that uses this Surface User Guide URL as a data source.

Setting up

Create environment variables

Your application must be authenticated to send API requests. For production, use a secure way of storing and accessing your credentials. In this example, you will write your credentials to environment variables on the local machine running the application.

To set the environment variable for your Language resource key, open a console window, and follow the instructions for your operating system and development environment.

• To set the LANGUAGE_KEY environment variable, replace your-key with one of the keys for your resource.
• To set the LANGUAGE_ENDPOINT environment variable, replace your-endpoint with the endpoint for your resource.

setx LANGUAGE_KEY your-key

setx LANGUAGE_ENDPOINT your-endpoint

export LANGUAGE_KEY=your-key

export LANGUAGE_ENDPOINT=your-endpoint

export LANGUAGE_KEY=your-key

export LANGUAGE_ENDPOINT=your-endpoint

Query a project

Generate an answer from a project

To query a custom question answering project with the REST APIs and cURL, you need the following information:

The cURL command is executed from a BASH shell. Edit this command with your own resource name, resource key, and JSON values and size of JSON.

curl -X POST -H "Ocp-Apim-Subscription-Key: $LANGUAGE_KEY" -H "Content-Type: application/json" -d '{
  "question": "How much battery life do I have left?"
  }'  '$LANGUAGE_ENDPOINT.api.cognitive.azure.cn/language/:query-knowledgebases?projectName={YOUR_PROJECT_NAME}&api-version=2021-10-01&deploymentName={DEPLOYMENT_NAME}'

When you run the code above, if you are using the data source from the prerequisites you will get an answer that looks as follows:

{
"answers": [
    {
      "questions": [
        "Check battery level"
      ],
      "answer": "If you want to see how much battery you have left, go to **Start  **> **Settings  **> **Devices  **> **Bluetooth & other devices  **, then find your pen. The current battery level will appear under the battery icon.",
      "confidenceScore": 0.9185,
      "id": 101,
      "source": "https://support.microsoft.com/surface/how-to-use-your-surface-pen-8a403519-cd1f-15b2-c9df-faa5aa924e98",
      "metadata": {},
      "dialog": {
        "isContextOnly": false,
        "prompts": []
      }
    }
  ]
}

The confidenceScore returns a value between 0 and 1. You can think of this like a percentage and multiply by 100 so a confidence score of 0.9185 means custom question answering is 91.85% confident this is the correct answer to the question based on the project.

If you want to exclude answers where the confidence score falls below a certain threshold, you can add the confidenceScoreThreshold parameter.

curl -X POST -H "Ocp-Apim-Subscription-Key: $LANGUAGE_KEY" -H "Content-Type: application/json" -d '{
  "question": "How much battery life do I have left?",
  "confidenceScoreThreshold": "0.95",
  }'  '$LANGUAGE_ENDPOINT.api.cognitive.azure.cn//language/:query-knowledgebases?projectName=Sample-project&api-version=2021-10-01&deploymentName={DEPLOYMENT_NAME}'

Since we know from our previous execution of the code that our confidence score is: .9185 setting the threshold to .95 will result in the default answer being returned.

{
  "answers": [
    {
      "questions": [],
      "answer": "No good match found in KB",
      "confidenceScore": 0.0,
      "id": -1,
      "metadata": {}
    }
  ]
}

Query text without a project

You can also use custom question answering without a project with the prebuilt custom question answering REST API, which is called via query-text. In this case, you provide question answering with both a question and the associated text records you would like to search for an answer at the time the request is sent.

For this example, you only need to modify the variables for API KEY and ENDPOINT.

curl -X POST -H "Ocp-Apim-Subscription-Key: $LANGUAGE_KEY" -H "Content-Type: application/json" -d '{
"question":"How long does it takes to charge a surface?",
"records":[
{"id":"doc1","text":"Power and charging.It takes two to four hours to charge the Surface Pro 4 battery fully from an empty state. It can take longer if you\u0027re using your Surface for power-intensive activities like gaming or video streaming while you\u0027re charging it"},
{"id":"doc2","text":"You can use the USB port on your Surface Pro 4 power supply to charge other devices, like a phone, while your Surface charges. The USB port on the power supply is only for charging, not for data transfer. If you want to use a USB device, plug it into the USB port on your Surface."}],
"language":"en",
"stringIndexType":"Utf16CodeUnit"
}'  '$LANGUAGE_ENDPOINT.api.cognitive.azure.cn/language/:query-text?&api-version=2021-10-01'

This example will return a result of:

{  
"answers": [
    {
      "answer": "Power and charging.It takes two to four hours to charge the Surface Pro 4 battery fully from an empty state. It can take longer if you're using your Surface for power-intensive activities like gaming or video streaming while you're charging it",
      "confidenceScore": 0.9118788838386536,
      "id": "doc1",
      "answerSpan": {
        "text": "two to four hours",
        "confidenceScore": 0.9850527,
        "offset": 27,
        "length": 18
      },
      "offset": 0,
      "length": 243
    },
    {
      "answer": "It can take longer if you're using your Surface for power-intensive activities like gaming or video streaming while you're charging it",
      "confidenceScore": 0.052793052047491074,
      "id": "doc1",
      "answerSpan": {
        "text": "longer",
        "confidenceScore": 0.6694634,
        "offset": 11,
        "length": 7
      },
      "offset": 109,
      "length": 134
    },
    {
      "answer": "You can use the USB port on your Surface Pro 4 power supply to charge other devices, like a phone, while your Surface charges. The USB port on the power supply is only for charging, not for data transfer. If you want to use a USB device, plug it into the USB port on your Surface.",
      "confidenceScore": 0.017600709572434425,
      "id": "doc2",
      "answerSpan": {
        "text": "USB port on your Surface Pro 4 power supply to charge other devices, like a phone, while your Surface charges. The USB port on the power supply is only for charging",
        "confidenceScore": 0.1544854,
        "offset": 15,
        "length": 165
      },
      "offset": 0,
      "length": 280
    }
  ]
}

Use this quickstart for the custom question answering client library for .NET to:

• Get an answer from a project.
• Get an answer from a body of text that you send along with your question.
• Get the confidence score for the answer to your question.

Reference documentation | Package (NuGet) | Additional samples | Library source code

Prerequisites

• Azure subscription - Create one for trial
• The Visual Studio IDE or current version of .NET Core.
• Custom question answering requires a Language resource with the custom question answering feature enabled to generate an API key and endpoint.
  • After your Language resource deploys, select Go to resource. You will need the key and endpoint from the resource you create to connect to the API. Paste your key and endpoint into the code below later in the quickstart.
• To create a Language resource with Azure CLI provide the following additional properties: --api-properties qnaAzureSearchEndpointId=/subscriptions/<azure-subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Search/searchServices/<azure-search-service-name> qnaAzureSearchEndpointKey=<azure-search-service-auth-key>
• An existing project to query. If you have not set up a project, you can follow the instructions in the Language Studio quickstart. Or add a project that uses this Surface User Guide URL as a data source.

Setting up

Create environment variables

Your application must be authenticated to send API requests. For production, use a secure way of storing and accessing your credentials. In this example, you will write your credentials to environment variables on the local machine running the application.

To set the environment variable for your Language resource key, open a console window, and follow the instructions for your operating system and development environment.

• To set the LANGUAGE_KEY environment variable, replace your-key with one of the keys for your resource.
• To set the LANGUAGE_ENDPOINT environment variable, replace your-endpoint with the endpoint for your resource.

setx LANGUAGE_KEY your-key

setx LANGUAGE_ENDPOINT your-endpoint

export LANGUAGE_KEY=your-key

export LANGUAGE_ENDPOINT=your-endpoint

export LANGUAGE_KEY=your-key

export LANGUAGE_ENDPOINT=your-endpoint

CLI

In a console window (such as cmd, PowerShell, or Bash), use the dotnet new command to create a new console app with the name question-answering-quickstart. This command creates a simple "Hello World" C# project with a single source file: program.cs.

dotnet new console -n question-answering-quickstart

Change your directory to the newly created app folder. You can build the application with:

dotnet build

The build output should contain no warnings or errors.

...
Build succeeded.
 0 Warning(s)
 0 Error(s)
...

Within the application directory, install the custom question answering client library for .NET with the following command:

dotnet add package Azure.AI.Language.QuestionAnswering

Query a project

Generate an answer from a project

The example below will allow you to query a project using GetAnswers to get an answer to your question.

You will need to update the code below and provide your own values for the following variables.

From the project directory, open the program.cs file and replace with the following code:

using Azure;
using Azure.AI.Language.QuestionAnswering;
using System;

namespace question_answering
{
    class Program
    {
        static void Main(string[] args)
        {

            // This example requires environment variables named "LANGUAGE_KEY" and "LANGUAGE_ENDPOINT"
            Uri endpoint = new Uri("LANGUAGE_ENDPOINT");
            AzureKeyCredential credential = new AzureKeyCredential("LANGUAGE_KEY");
            string projectName = "{YOUR-PROJECT-NAME}";
            string deploymentName = "production";

            string question = "How long should my Surface battery last?";

            QuestionAnsweringClient client = new QuestionAnsweringClient(endpoint, credential);
            QuestionAnsweringProject project = new QuestionAnsweringProject(projectName, deploymentName);

            Response<AnswersResult> response = client.GetAnswers(question, project);

            foreach (KnowledgeBaseAnswer answer in response.Value.Answers)
            {
                Console.WriteLine($"Q:{question}");
                Console.WriteLine($"A:{answer.Answer}");
            }
        }
    }
}

While we are hard coding the variables for our example. For production, consider using a secure way of storing and accessing your credentials. For example, Azure key vault provides secure key storage.

After updating Program.cs with the code above and substituting in the correct variable values. Run the application with the dotnet run command from your application directory.

dotnet run

The response will look as follows:

Q: How much battery life do I have left?
A: If you want to see how much battery you have left, go to **Start  **> **Settings  **> **Devices  **> **Bluetooth & other devices  **, then find your pen. The current battery level will appear under the battery icon.

For information on how confident custom question answering is that this is the correct response add an additional print statement underneath the existing print statements:

Console.WriteLine($"Q:{question}");
Console.WriteLine($"A:{answer.Answer}");
Console.WriteLine($"({answer.Confidence})"); // add this line

If you execute dotnet run again, you will now receive a result with a confidence score:

Q:How much battery life do I have left?
A:If you want to see how much battery you have left, go to **Start  **> **Settings  **> **Devices  **> **Bluetooth & other devices  **, then find your pen. The current battery level will appear under the battery icon.
(0.9185)

The confidence score returns a value between 0 and 1. You can think of this like a percentage and multiply by 100 so a confidence score of 0.9185 means custom question answering is 91.85% confident this is the correct answer to the question based on the project.

If you want to exclude answers where the confidence score falls below a certain threshold, you use AnswerOptions to add the ConfidenceScoreThreshold property.

QuestionAnsweringClient client = new QuestionAnsweringClient(endpoint, credential);
QuestionAnsweringProject project = new QuestionAnsweringProject(projectName, deploymentName);
AnswersOptions options = new AnswersOptions(); //Add this line
options.ConfidenceThreshold = 0.95; //Add this line

Response<AnswersResult> response = client.GetAnswers(question, project, options); //Add the additional options parameter

Since we know from our previous execution of the code that our confidence score is: .9185 setting the threshold to .95 will result in the default answer being returned.

Q:How much battery life do I have left?
A:No good match found in KB
(0)

Query text without a project

You can also use custom question answering without a project with GetAnswersFromText. In this case, you provide custom question answering with both a question and the associated text records you would like to search for an answer at the time the request is sent.

For this example, you only need to modify the variables for endpoint and credential.

using Azure;
using Azure.AI.Language.QuestionAnswering;
using System;
using System.Collections.Generic;


namespace questionansweringcsharp
{
    class Program
    {
        static void Main(string[] args)
        {

            Uri endpoint = new Uri("https://{YOUR-ENDPOINT}.api.cognitive.azure.cn/");
            AzureKeyCredential credential = new AzureKeyCredential("YOUR-LANGUAGE-RESOURCE-KEY");
            QuestionAnsweringClient client = new QuestionAnsweringClient(endpoint, credential);

            IEnumerable<TextDocument> records = new[]
            {
                new TextDocument("doc1", "Power and charging.It takes two to four hours to charge the Surface Pro 4 battery fully from an empty state. " +
                         "It can take longer if you're using your Surface for power-intensive activities like gaming or video streaming while you're charging it"),
                new TextDocument("doc2", "You can use the USB port on your Surface Pro 4 power supply to charge other devices, like a phone, while your Surface charges. " +
                         "The USB port on the power supply is only for charging, not for data transfer. If you want to use a USB device, plug it into the USB port on your Surface."),
            };

            AnswersFromTextOptions options = new AnswersFromTextOptions("How long does it takes to charge a surface?", records);
            Response<AnswersFromTextResult> response = client.GetAnswersFromText(options);

           foreach (TextAnswer answer in response.Value.Answers)
            {
                if (answer.Confidence > .9)
                {
                    string BestAnswer = response.Value.Answers[0].Answer;

                    Console.WriteLine($"Q:{options.Question}");
                    Console.WriteLine($"A:{BestAnswer}");
                    Console.WriteLine($"Confidence Score: ({response.Value.Answers[0].Confidence:P2})"); //:P2 converts the result to a percentage with 2 decimals of accuracy. 
                    break;
                }
                else
                {
                    Console.WriteLine($"Q:{options.Question}");
                    Console.WriteLine("No answers met the requested confidence score.");
                    break;
                }
            }

        }
    }
}

To run the code above, replace the Program.cs with the contents of the script block above and modify the endpoint and credential variables to correspond to the language resource you created as part of the prerequisites.

In this case, we iterate through all responses and only return the response with the highest confidence score that is greater than 0.9. To understand more about the options available with GetAnswersFromText.

Use this quickstart for the custom question answering client library for Python to:

• Get an answer from a project.
• Get an answer from a body of text that you send along with your question.
• Get the confidence score for the answer to your question.

Reference documentation | Package (PyPI) | Additional samples | Library source code

Prerequisites

• Azure subscription - Create one for trial
• Python 3.x
• Custom question answering requires a Language resource with the custom question answering feature enabled to generate an API key and endpoint.
  • After your Language resource deploys, select Go to resource. You need the key and endpoint from the resource you create to connect to the API. Paste your key and endpoint into the code below later in the quickstart.
• To create a Language resource with Azure CLI provide the following other properties: --api-properties qnaAzureSearchEndpointId=/subscriptions/<azure-subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Search/searchServices/<azure-search-service-name> qnaAzureSearchEndpointKey=<azure-search-service-auth-key>
• An existing project to query. If you have not set up a project, you can follow the instructions in the Language Studio quickstart. Or add a project that uses this Surface User Guide URL as a data source.

Setting up

Install the client library

After installing Python, you can install the client library with:

pip install azure-ai-language-questionanswering

Query a project

Generate an answer from a project

The example below will allow you to query a project using get_answers to get an answer to your question. You can copy this code into a dedicated .py file or into a cell in Jupyter Notebook/Lab.

You need to update the code below and provide your own values for the following variables.

from azure.core.credentials import AzureKeyCredential
from azure.ai.language.questionanswering import QuestionAnsweringClient

endpoint = "https://{YOUR-ENDPOINT}.api.cognitive.azure.cn/"
credential = AzureKeyCredential("{YOUR-LANGUAGE-RESOURCE-KEY}")
knowledge_base_project = "{YOUR-PROJECT-NAME}"
deployment = "production"

def main():
    client = QuestionAnsweringClient(endpoint, credential)
    with client:
        question="How much battery life do I have left?"
        output = client.get_answers(
            question = question,
            project_name=knowledge_base_project,
            deployment_name=deployment
        )
    print("Q: {}".format(question))
    print("A: {}".format(output.answers[0].answer))

if __name__ == '__main__':
    main()

While we're hard coding the variables for our example. For production, consider using a secure way of storing and accessing your credentials. For example, Azure key vault provides secure key storage.

When you run the code above, if you're using the data source from the prerequisites you get an answer that looks as follows:

Q: How much battery life do I have left?
A: If you want to see how much battery you have left, go to **Start  **> **Settings  **> **Devices  **> **Bluetooth & other devices  **, then find your pen. The current battery level will appear under the battery icon.

For information on how confident custom question answering is that this is the correct response add another print statement underneath the existing print statements:

print("Q: {}".format(question))
print("A: {}".format(output.answers[0].answer))
print("Confidence Score: {}".format(output.answers[0].confidence)) # add this line 

You'll now receive a result with a confidence score:

Q: How much battery life do I have left?
A: If you want to see how much battery you have left, go to **Start  **> **Settings  **> **Devices  **> **Bluetooth & other devices  **, then find your pen. The current battery level will appear under the battery icon.
Confidence Score: 0.9185

The confidence score returns a value between 0 and 1. You can think of this like a percentage and multiply by 100 so a confidence score of 0.9185 means custom question answering is 91.85% confident this is the correct answer to the question based on the project.

If you want to exclude answers where the confidence score falls below a certain threshold, you can modify the AnswerOptions to add the confidence_threshold parameter.

        output = client.get_answers(
            confidence_threshold = 0.95, #add this line
            question = question,
            project_name=knowledge_base_project,
            deployment_name=deployment
        )

Since we know from our previous execution of the code that our confidence score is: .9185 setting the threshold to .95 results in the default answer being returned.

Q: How much battery life do I have left?
A: No good match found in KB
Confidence Score: 0.0

Query text without a project

You can also use custom question answering without a project with get_answers_from_text. In this case, you provide custom question answering with both a question and the associated text records you would like to search for an answer at the time the request is sent.

For this example, you only need to modify the variables for endpoint and credential.

import os
from azure.core.credentials import AzureKeyCredential
from azure.ai.language.questionanswering import QuestionAnsweringClient
from azure.ai.language.questionanswering import models as qna

endpoint = "https://{YOUR-ENDPOINT}.api.cognitive.azure.cn/"
credential = AzureKeyCredential("YOUR-LANGUAGE-RESOURCE-KEY")

def main():
    client = QuestionAnsweringClient(endpoint, credential)
    with client:
        question="How long does it takes to charge a surface?"
        input = qna.AnswersFromTextOptions(
            question=question,
            text_documents=[
                "Power and charging. It takes two to four hours to charge the Surface Pro 4 battery fully from an empty state. " +
                "It can take longer if you're using your Surface for power-intensive activities like gaming or video streaming while you're charging it.",
                "You can use the USB port on your Surface Pro 4 power supply to charge other devices, like a phone, while your Surface charges. " +
                "The USB port on the power supply is only for charging, not for data transfer. If you want to use a USB device, plug it into the USB port on your Surface.",
            ]
        )


        output = client.get_answers_from_text(input)

    best_answer = [a for a in output.answers if a.confidence > 0.9][0]
    print(u"Q: {}".format(input.question))
    print(u"A: {}".format(best_answer.answer))
    print("Confidence Score: {}".format(output.answers[0].confidence))

if __name__ == '__main__':
    main()

You can copy this code into a dedicated .py file or into a new cell in Jupyter Notebook/Lab. This example returns a result of:

Q: How long does it takes to charge surface?
A: Power and charging. It takes two to four hours to charge the Surface Pro 4 battery fully from an empty state. It can take longer if you're using your Surface for power-intensive activities like gaming or video streaming while you're charging it.
Confidence Score: 0.9254655838012695

In this case, we iterate through all responses and only return the response with the highest confidence score that is greater than 0.9. To understand more about the options available with get_answers_from_text, review the AnswersFromTextOptions parameters.