Manage indexing policies in Azure Cosmos DB

APPLIES TO: NoSQL

In Azure Cosmos DB, data is indexed following indexing policies that are defined for each container. The default indexing policy for newly created containers enforces range indexes for any string or number. You can override this policy with your own custom indexing policy.

Indexing policy examples

Here are some examples of indexing policies shown in their JSON format. They're exposed on the Azure portal in JSON format. The same parameters can be set through the Azure CLI or any SDK.

Opt-out policy to selectively exclude some property paths

{
    "indexingMode": "consistent",
    "includedPaths": [
        {
            "path": "/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/path/to/single/excluded/property/?"
        },
        {
            "path": "/path/to/root/of/multiple/excluded/properties/*"
        }
    ]
}

Opt-in policy to selectively include some property paths

{
    "indexingMode": "consistent",
    "includedPaths": [
        {
            "path": "/path/to/included/property/?"
        },
        {
            "path": "/path/to/root/of/multiple/included/properties/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/*"
        }
    ]
}

Using a spatial index on a specific property path only

{
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
        {
            "path": "/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/_etag/?"
        }
    ],
    "spatialIndexes": [
        {
                    "path": "/path/to/geojson/property/?",
            "types": [
                "Point",
                "Polygon",
                "MultiPolygon",
                "LineString"
            ]
        }
    ]
}

Vector indexing policy examples

In addition to including or excluding paths for individual properties, you can also specify a vector index. In general, vector indexes should be specified whenever the VectorDistance system function is used to measure similarity between a query vector and a vector property.

{
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
        {
            "path": "/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/_etag/?"
        },
        {
            "path": "/vector/*"
        }
    ],
    "vectorIndexes": [
        {
            "path": "/vector",
            "type": "quantizedFlat"
        }
    ]
}

You can define the following types of vector index policies:

The flat and quantizedFlat index types leverage Azure Cosmos DB's index to store and read each vector when performing a vector search. Vector searches with a flat index are brute-force searches and produce 100% accuracy. However, there is a limitation of 505 dimensions for vectors on a flat index.

The quantizedFlat index stores quantized or compressed vectors on the index. Vector searches with quantizedFlat index are also brute-force searches, however their accuracy might be slightly less than 100% since the vectors are quantized before adding to the index. However, vector searches with quantized flat should have lower latency, higher throughput, and lower RU cost than vector searches on a flat index. This is a good option for scenarios where you are using query filters to narrow down the vector search to a relatively small set of vectors.

The diskANN index is a separate index defined specifically for vectors leveraging DiskANN, a suite of highly performant vector indexing algorithms developed by Microsoft Research. DiskANN indexes can offer some of the lowest latency, highest query-per-second (QPS), and lowest RU cost queries at high accuracy. However, since DiskANN is an approximate nearest neighbors (ANN) index, the accuracy may be lower than quantizedFlat or flat.

The diskANN and quantizedFlat indexes can take optional index build parameters that can be used to tune the accuracy vs latency trade off that applies to every Approximate Nearest Neighbors vector index.

• quantizationByteSize: Sets the size (in bytes) for product quantization. Min=1, Default=dynamic (system decides), Max=512. Setting this larger may result in higher accuracy vector searches at expense of higher RU cost and higher latency. This applies to both quantizedFlat and DiskANN index types.
• indexingSearchListSize: Sets how many vectors to search over during index build construction. Min=10, Default=100, Max=500. Setting this larger may result in higher accuracy vector searches at the expense of longer index build times and higher vector ingest latencies. This applies to DiskANN indexes only.

Tuple indexing policy examples

This example indexing policy defines a tuple index on events.name and events.category

{  
    "automatic":true,
    "indexingMode":"Consistent",
    "includedPaths":[  
        {"path":"/*"}, 
        {"path":"/events/[]/{name,category}/?"} 
    ],
    "excludedPaths":[],
    "compositeIndexes":[]
}

The above index is used for the below query.

SELECT * 
FROM root r 
WHERE 
   EXISTS (SELECT VALUE 1 FROM ev IN r.events 
           WHERE ev.name = ‘M&M’ AND ev.category = ‘Candy’) 

Composite indexing policy examples

In addition to including or excluding paths for individual properties, you can also specify a composite index. To perform a query that has an ORDER BY clause for multiple properties, a composite index is required on those properties. If the query includes filters along with sorting on multiple properties, you may need more than one composite index.

Composite indexes also have a performance benefit for queries that have multiple filters or both a filter and an ORDER BY clause.

Composite index defined for (name asc, age desc)

{  
    "automatic":true,
    "indexingMode":"Consistent",
    "includedPaths":[  
        {  
            "path":"/*"
        }
    ],
    "excludedPaths":[],
    "compositeIndexes":[  
        [  
            {  
                "path":"/name",
                "order":"ascending"
            },
            {  
                "path":"/age",
                "order":"descending"
            }
        ]
    ]
}

The composite index on name and age is required for the following queries:

Query #1:

SELECT *
FROM c
ORDER BY c.name ASC, c.age DESC

Query #2:

SELECT *
FROM c
ORDER BY c.name DESC, c.age ASC

This composite index benefits the following queries and optimizes the filters:

Query #3:

SELECT *
FROM c
WHERE c.name = "Tim"
ORDER BY c.name DESC, c.age ASC

Query #4:

SELECT *
FROM c
WHERE c.name = "Tim" AND c.age > 18

Composite index defined for (name ASC, age ASC) and (name ASC, age DESC)

You can define multiple composite indexes within the same indexing policy.

{  
    "automatic":true,
    "indexingMode":"Consistent",
    "includedPaths":[  
        {  
            "path":"/*"
        }
    ],
    "excludedPaths":[],
    "compositeIndexes":[  
        [  
            {  
                "path":"/name",
                "order":"ascending"
            },
            {  
                "path":"/age",
                "order":"ascending"
            }
        ],
        [  
            {  
                "path":"/name",
                "order":"ascending"
            },
            {  
                "path":"/age",
                "order":"descending"
            }
        ]
    ]
}

Composite index defined for (name ASC, age ASC)

It's optional to specify the order. If not specified, the order is ascending.

{  
    "automatic":true,
    "indexingMode":"Consistent",
    "includedPaths":[  
        {  
            "path":"/*"
        }
    ],
    "excludedPaths":[],
    "compositeIndexes":[  
        [  
            {  
               "path":"/name"
            },
            {  
               "path":"/age"
            }
        ]
    ]
}

Exclude all property paths but keeping indexing active

You can use this policy where the Time-to-Live (TTL) feature is active but no other indexes are necessary to use Azure Cosmos DB as a pure key-value store.

{
    "indexingMode": "consistent",
    "includedPaths": [],
    "excludedPaths": [{
        "path": "/*"
    }]
}

No indexing

This policy turns off indexing. If indexingMode is set to none, you can't set a TTL on the container.

{
    "indexingMode": "none"
}

Updating indexing policy

In Azure Cosmos DB, the indexing policy can be updated using any of the following methods:

• From the Azure portal
• Using the Azure CLI
• Using PowerShell
• Using one of the SDKs

An indexing policy update triggers an index transformation. The progress of this transformation can also be tracked from the SDKs.

Use the Azure portal

Azure Cosmos DB containers store their indexing policy as a JSON document that the Azure portal lets you directly edit.

1. Sign in to the Azure portal.

2. Create a new Azure Cosmos DB account or select an existing account.

3. Open the Data Explorer pane and select the container that you want to work on.

4. Select Scale & Settings.

5. Modify the indexing policy JSON document, as shown in these examples.

6. Select Save when you're done.

Use the Azure CLI

To create a container with a custom indexing policy, see Create a container with a custom index policy using CLI.

Use PowerShell

To create a container with a custom indexing policy, see Create a container with a custom index policy using PowerShell.

Use the .NET SDK

// Retrieve the container's details
ContainerResponse containerResponse = await client.GetContainer("database", "container").ReadContainerAsync();
// Set the indexing mode to consistent
containerResponse.Resource.IndexingPolicy.IndexingMode = IndexingMode.Consistent;
// Add an included path
containerResponse.Resource.IndexingPolicy.IncludedPaths.Add(new IncludedPath { Path = "/*" });
// Add an excluded path
containerResponse.Resource.IndexingPolicy.ExcludedPaths.Add(new ExcludedPath { Path = "/name/*" });
// Add a spatial index
SpatialPath spatialPath = new SpatialPath
{
    Path = "/locations/*"
};
spatialPath.SpatialTypes.Add(SpatialType.Point);
containerResponse.Resource.IndexingPolicy.SpatialIndexes.Add(spatialPath);
// Add a composite index
containerResponse.Resource.IndexingPolicy.CompositeIndexes.Add(new Collection<CompositePath> { new CompositePath() { Path = "/name", Order = CompositePathSortOrder.Ascending }, new CompositePath() { Path = "/age", Order = CompositePathSortOrder.Descending } });
// Update container with changes
await client.GetContainer("database", "container").ReplaceContainerAsync(containerResponse.Resource);

// retrieve the container's details
ContainerResponse containerResponse = await client.GetContainer("database", "container").ReadContainerAsync(new ContainerRequestOptions { PopulateQuotaInfo = true });
// retrieve the index transformation progress from the result
long indexTransformationProgress = long.Parse(containerResponse.Headers["x-ms-documentdb-collection-index-transformation-progress"]);

await client.GetDatabase("database").DefineContainer(name: "container", partitionKeyPath: "/myPartitionKey")
    .WithIndexingPolicy()
        .WithIncludedPaths()
            .Path("/*")
        .Attach()
        .WithExcludedPaths()
            .Path("/name/*")
        .Attach()
        .WithSpatialIndex()
            .Path("/locations/*", SpatialType.Point)
        .Attach()
        .WithCompositeIndex()
            .Path("/name", CompositePathSortOrder.Ascending)
            .Path("/age", CompositePathSortOrder.Descending)
        .Attach()
    .Attach()
    .CreateIfNotExistsAsync();

// Retrieve the container's details
ResourceResponse<DocumentCollection> containerResponse = await client.ReadDocumentCollectionAsync(UriFactory.CreateDocumentCollectionUri("database", "container"));
// Set the indexing mode to consistent
containerResponse.Resource.IndexingPolicy.IndexingMode = IndexingMode.Consistent;
// Add an included path
containerResponse.Resource.IndexingPolicy.IncludedPaths.Add(new IncludedPath { Path = "/*" });
// Add an excluded path
containerResponse.Resource.IndexingPolicy.ExcludedPaths.Add(new ExcludedPath { Path = "/name/*" });
// Add a spatial index
containerResponse.Resource.IndexingPolicy.SpatialIndexes.Add(new SpatialSpec() { Path = "/locations/*", SpatialTypes = new Collection<SpatialType>() { SpatialType.Point } } );
// Add a composite index
containerResponse.Resource.IndexingPolicy.CompositeIndexes.Add(new Collection<CompositePath> {new CompositePath() { Path = "/name", Order = CompositePathSortOrder.Ascending }, new CompositePath() { Path = "/age", Order = CompositePathSortOrder.Descending }});
// Update container with changes
await client.ReplaceDocumentCollectionAsync(containerResponse.Resource);

// retrieve the container's details
ResourceResponse<DocumentCollection> container = await client.ReadDocumentCollectionAsync(UriFactory.CreateDocumentCollectionUri("database", "container"), new RequestOptions { PopulateQuotaInfo = true });
// retrieve the index transformation progress from the result
long indexTransformationProgress = container.IndexTransformationProgress;

Use the Java SDK

The DocumentCollection object from the Java SDK exposes the getIndexingPolicy() and setIndexingPolicy() methods. The IndexingPolicy object they manipulate lets you change the indexing mode and add or remove included and excluded paths. For more information, see Quickstart: Build a Java app to manage Azure Cosmos DB for NoSQL data.

// Retrieve the container's details
Observable<ResourceResponse<DocumentCollection>> containerResponse = client.readCollection(String.format("/dbs/%s/colls/%s", "database", "container"), null);
containerResponse.subscribe(result -> {
DocumentCollection container = result.getResource();
IndexingPolicy indexingPolicy = container.getIndexingPolicy();

// Set the indexing mode to consistent
indexingPolicy.setIndexingMode(IndexingMode.Consistent);

// Add an included path

Collection<IncludedPath> includedPaths = new ArrayList<>();
IncludedPath includedPath = new IncludedPath();
includedPath.setPath("/*");
includedPaths.add(includedPath);
indexingPolicy.setIncludedPaths(includedPaths);

// Add an excluded path

Collection<ExcludedPath> excludedPaths = new ArrayList<>();
ExcludedPath excludedPath = new ExcludedPath();
excludedPath.setPath("/name/*");
excludedPaths.add(excludedPath);
indexingPolicy.setExcludedPaths(excludedPaths);

// Add a spatial index

Collection<SpatialSpec> spatialIndexes = new ArrayList<SpatialSpec>();
Collection<SpatialType> collectionOfSpatialTypes = new ArrayList<SpatialType>();

SpatialSpec spec = new SpatialSpec();
spec.setPath("/locations/*");
collectionOfSpatialTypes.add(SpatialType.Point);
spec.setSpatialTypes(collectionOfSpatialTypes);
spatialIndexes.add(spec);

indexingPolicy.setSpatialIndexes(spatialIndexes);

// Add a composite index

Collection<ArrayList<CompositePath>> compositeIndexes = new ArrayList<>();
ArrayList<CompositePath> compositePaths = new ArrayList<>();

CompositePath nameCompositePath = new CompositePath();
nameCompositePath.setPath("/name");
nameCompositePath.setOrder(CompositePathSortOrder.Ascending);

CompositePath ageCompositePath = new CompositePath();
ageCompositePath.setPath("/age");
ageCompositePath.setOrder(CompositePathSortOrder.Descending);

compositePaths.add(ageCompositePath);
compositePaths.add(nameCompositePath);

compositeIndexes.add(compositePaths);
indexingPolicy.setCompositeIndexes(compositeIndexes);

// Update the container with changes

 client.replaceCollection(container, null);
});

To track the index transformation progress on a container, pass a RequestOptions object that requests the quota info to be populated. Retrieve the value from the x-ms-documentdb-collection-index-transformation-progress response header.

// set the RequestOptions object
RequestOptions requestOptions = new RequestOptions();
requestOptions.setPopulateQuotaInfo(true);
// retrieve the container's details
Observable<ResourceResponse<DocumentCollection>> containerResponse = client.readCollection(String.format("/dbs/%s/colls/%s", "database", "container"), requestOptions);
containerResponse.subscribe(result -> {
    // retrieve the index transformation progress from the response headers
    String indexTransformationProgress = result.getResponseHeaders().get("x-ms-documentdb-collection-index-transformation-progress");
});

Use the Node.js SDK

The ContainerDefinition interface from Node.js SDK exposes an indexingPolicy property that lets you change the indexingMode and add or remove includedPaths and excludedPaths. For more information, see Quickstart - Azure Cosmos DB for NoSQL client library for Node.js.

Retrieve the container's details:

const containerResponse = await client.database('database').container('container').read();

Set the indexing mode to consistent:

containerResponse.body.indexingPolicy.indexingMode = "consistent";

Add included path including a spatial index:

containerResponse.body.indexingPolicy.includedPaths.push({
    includedPaths: [
      {
        path: "/age/*",
        indexes: [
          {
            kind: cosmos.DocumentBase.IndexKind.Range,
            dataType: cosmos.DocumentBase.DataType.String
          },
          {
            kind: cosmos.DocumentBase.IndexKind.Range,
            dataType: cosmos.DocumentBase.DataType.Number
          }
        ]
      },
      {
        path: "/locations/*",
        indexes: [
          {
            kind: cosmos.DocumentBase.IndexKind.Spatial,
            dataType: cosmos.DocumentBase.DataType.Point
          }
        ]
      }
    ]
  });

Add excluded path:

containerResponse.body.indexingPolicy.excludedPaths.push({ path: '/name/*' });

Update the container with changes:

const replaceResponse = await client.database('database').container('container').replace(containerResponse.body);

To track the index transformation progress on a container, pass a RequestOptions object that sets the populateQuotaInfo property to true. Retrieve the value from the x-ms-documentdb-collection-index-transformation-progress response header.

// retrieve the container's details
const containerResponse = await client.database('database').container('container').read({
    populateQuotaInfo: true
});
// retrieve the index transformation progress from the response headers
const indexTransformationProgress = replaceResponse.headers['x-ms-documentdb-collection-index-transformation-progress'];

Add a composite index:

 console.log("create container with composite indexes");
  const containerDefWithCompositeIndexes = {
    id: "containerWithCompositeIndexingPolicy",
    indexingPolicy: {
      automatic: true,
      indexingMode: IndexingMode.consistent,
      includedPaths: [
        {
          path: "/*",
        },
      ],
      excludedPaths: [
        {
          path: '/"systemMetadata"/*',
        },
      ],
      compositeIndexes: [
        [
          { path: "/field", order: "ascending" },
          { path: "/key", order: "ascending" },
        ],
      ],
    },
  };
  const containerWithCompositeIndexes = (
    await database.containers.create(containerDefWithCompositeIndexes)
  ).container;

Use the Python SDK

containerPath = 'dbs/database/colls/collection'
container = client.ReadContainer(containerPath)

container['indexingPolicy']['indexingMode'] = 'consistent'

container["indexingPolicy"] = {

    "indexingMode":"consistent",
    "spatialIndexes":[
                {"path":"/location/*","types":["Point"]}
             ],
    "includedPaths":[{"path":"/age/*","indexes":[]}],
    "excludedPaths":[{"path":"/*"}]
}

container["indexingPolicy"] = {
    "indexingMode":"consistent",
    "includedPaths":[{"path":"/*","indexes":[]}],
    "excludedPaths":[{"path":"/name/*"}]
}

container['indexingPolicy']['compositeIndexes'] = [
                [
                    {
                        "path": "/name",
                        "order": "ascending"
                    },
                    {
                        "path": "/age",
                        "order": "descending"
                    }
                ]
                ]

response = client.ReplaceContainer(containerPath, container)

database_client = cosmos_client.get_database_client('database')
container_client = database_client.get_container_client('container')
container = container_client.read()

indexingPolicy = {
    'indexingMode': 'consistent'
}

indexingPolicy = {
    "indexingMode":"consistent",
    "spatialIndexes":[
        {"path":"/location/*","types":["Point"]}
    ],
    "includedPaths":[{"path":"/age/*","indexes":[]}],
    "excludedPaths":[{"path":"/*"}]
}

indexingPolicy = {
    "indexingMode":"consistent",
    "includedPaths":[{"path":"/*","indexes":[]}],
    "excludedPaths":[{"path":"/name/*"}]
}

indexingPolicy['compositeIndexes'] = [
    [
        {
            "path": "/name",
            "order": "ascending"
        },
        {
            "path": "/age",
            "order": "descending"
        }
    ]
]

response = database_client.replace_container(container_client, container['partitionKey'], indexingPolicy)

container_client.read(populate_quota_info = True,
                      response_hook = lambda h,p: print(h['x-ms-documentdb-collection-index-transformation-progress']))

Next steps

Read more about the indexing in the following articles:

• Indexing overview
• Indexing policy