Hybrid search using vectors and full text in Azure AI Search

  • Article

Hybrid search is a combination of full text and vector queries that execute against a search index that contains both searchable plain text content and generated embeddings. For query purposes, hybrid search is:

  • A single query request that includes both search and vectors query parameters
  • Executing in parallel
  • With merged results in the query response, scored using Reciprocal Rank Fusion (RRF)

This article explains the concepts, benefits, and limitations of hybrid search. Watch this embedded video for an explanation and short demos of how hybrid retrieval contributes to high quality chat-style and copilot apps.

How does hybrid search work?

In Azure AI Search, vector fields containing embeddings can live alongside textual and numerical fields, allowing you to formulate hybrid queries that execute in parallel. Hybrid queries can take advantage of existing functionality like filtering, faceting, sorting and scoring profiles in a single search request.

Hybrid search combines results from both full text and vector queries, which use different ranking functions such as BM25 and HNSW. A Reciprocal Rank Fusion (RRF) algorithm merges the results. The query response provides just one result set, using RRF to pick the most relevant matches from each query.

Structure of a hybrid query

Hybrid search is predicated on having a search index that contains fields of various data types, including plain text and numbers, geo coordinates for geospatial search, and vectors for a mathematical representation of a chunk of text. You can use almost all query capabilities in Azure AI Search with a vector query, except for client-side interactions such as autocomplete and suggestions.

A representative hybrid query might be as follows (notice the vector is trimmed for brevity):

POST https://{{searchServiceName}}.search.azure.cn/indexes/hotels-vector-quickstart/docs/search?api-version=2024-07-01
  content-type: application/JSON
{
    "count": true,
    "search": "historic hotel walk to restaurants and shopping",
    "select": "HotelId, HotelName, Category, Description, Address/City, Address/StateProvince",
    "filter": "geo.distance(Location, geography'POINT(-77.03241 38.90166)') le 300",
    "facets": [ "Address/StateProvince"], 
    "vectors": [
        {
            "value": [ <array of embeddings> ]
            "k": 7,
            "fields": "DescriptionVector"
        },
        {
            "value": [ <array of embeddings> ]
            "k": 7,
            "fields": "Description_frVector"
        }
    ],

}

Key points include:

  • search specifies a full text search query.
  • vectors for vector queries, which can be multiple, targeting multiple vector fields. If the embedding space includes multi-lingual content, vector queries can find the match with no language analyzers or translation required.
  • select specifies which fields to return in results, which can be text fields that are human readable.
  • filters can specify geospatial search or other include and exclude criteria, such as whether parking is included. The geospatial query in this example finds hotels within a 300-kilometer radius of Washington D.C.
  • facets can be used to compute facet buckets over results that are returned from hybrid queries.

Filters and facets target data structures within the index that are distinct from the inverted indexes used for full text search and the vector indexes used for vector search. As such, when filters and faceted operations execute, the search engine can apply the operational result to the hybrid search results in the response.

Notice how there's no orderby in the query. Explicit sort orders override relevanced-ranked results, so if you want similarity and BM25 relevance, omit sorting in your query.

A response from the above query might look like this:

{
    "@odata.count": 3,
    "@search.facets": {
        "Address/StateProvince": [
            {
                "count": 1,
                "value": "NY"
            },
            {
                "count": 1,
                "value": "VA"
            }
        ]
    },
    "value": [
        {
            "@search.score": 0.03333333507180214,
            "@search.rerankerScore": 2.5229012966156006,
            "HotelId": "49",
            "HotelName": "Old Carrabelle Hotel",
            "Description": "Spacious rooms, glamorous suites and residences, rooftop pool, walking access to shopping, dining, entertainment and the city center.",
            "Category": "Luxury",
            "Address": {
                "City": "Arlington",
                "StateProvince": "VA"
            }
        },
        {
            "@search.score": 0.032522473484277725,
            "@search.rerankerScore": 2.111117362976074,
            "HotelId": "48",
            "HotelName": "Nordick's Motel",
            "Description": "Only 90 miles (about 2 hours) from the nation's capital and nearby most everything the historic valley has to offer.  Hiking? Wine Tasting? Exploring the caverns?  It's all nearby and we have specially priced packages to help make our B&B your home base for fun while visiting the valley.",
            "Category": "Boutique",
            "Address": {
                "City": "Washington D.C.",
                "StateProvince": null
            }
        }
    ]
}

Hybrid search combines the strengths of vector search and keyword search. The advantage of vector search is finding information that's conceptually similar to your search query, even if there are no keyword matches in the inverted index. The advantage of keyword or full text search is precision. Some scenarios - such as querying over product codes, highly specialized jargon, dates, and people's names - can perform better with keyword search because it can identify exact matches.

See also

Outperform vector search with hybrid retrieval and ranking (Tech blog)