Diagnose and troubleshoot Azure Cosmos DB .NET SDK request timeout exceptions

  • Article

APPLIES TO: NoSQL

The HTTP 408 error occurs if the SDK was unable to complete the request before the timeout limit occurred.

It is important to make sure the application design is following our guide for designing resilient applications with Azure Cosmos DB SDKs to make sure it correctly reacts to different network conditions. Your application should have retries in place for timeout errors as these are normally expected in a distributed system.

When evaluating the case for timeout errors:

  • What is the impact measured in volume of operations affected compared to the operations succeeding? Is it within the service SLAs?
  • Is the P99 latency / availability affected?
  • Are the failures affecting all your application instances or only a subset? When the issue is reduced to a subset of instances, it's commonly a problem related to those instances.

Customize the timeout on the Azure Cosmos DB .NET SDK

The SDK has two distinct alternatives to control timeouts, each with a different scope.

Request level timeouts

The CosmosClientOptions.RequestTimeout (or ConnectionPolicy.RequestTimeout for SDK v2) configuration allows you to set a timeout for the network request after the request left the SDK and is on the network, until a response is received.

The CosmosClientOptions.OpenTcpConnectionTimeout (or ConnectionPolicy.OpenTcpConnectionTimeout for SDK v2) configuration allows you to set a timeout for the time spent opening an initial connection. Once a connection is opened, subsequent requests will use the connection.

An operation started by a user can span multiple network requests, for example, retries. These two configurations are per-request, not end-to-end for an operation.

CancellationToken

All the async operations in the SDK have an optional CancellationToken parameter. This CancellationToken parameter is used throughout the entire operation, across all network requests and retries. In between network requests, the cancellation token might be checked and an operation canceled if the related token is expired. The cancellation token should be used to define an approximate expected timeout on the operation scope.

Note

The CancellationToken parameter is a mechanism where the library will check the cancellation when it won't cause an invalid state. The operation might not cancel exactly when the time defined in the cancellation is up. Instead, after the time is up, it cancels when it's safe to do so.

Troubleshooting steps

The following list contains known causes and solutions for request timeout exceptions.

CosmosOperationCanceledException

This type of exception is common when your application is passing CancellationTokens to the SDK operations. The SDK checks the state of the CancellationToken in-between retries and if the CancellationToken is canceled, it will abort the current operation with this exception.

The exception's Message / ToString() will also indicate the state of your CancellationToken through Cancellation Token has expired: true and it will also contain Diagnostics that contain the context of the cancellation for the involved requests.

These exceptions are safe to retry on and can be treated as timeouts from the retrying perspective.

Solution

Verify the configured time in your CancellationToken, make sure that it's greater than your RequestTimeout and the CosmosClientOptions.OpenTcpConnectionTimeout (if you're using Direct mode). If the available time in the CancellationToken is less than the configured timeouts, and the SDK is facing transient connectivity issues, the SDK won't be able to retry and will throw CosmosOperationCanceledException.

High CPU utilization

High CPU utilization is the most common case. For optimal latency, CPU usage should be roughly 40 percent. Use 10 seconds as the interval to monitor maximum (not average) CPU utilization. CPU spikes are more common with cross-partition queries where it might do multiple connections for a single query.

The timeouts will contain Diagnostics, which contain:

"systemHistory": [
{
"dateUtc": "2021-11-17T23:38:28.3115496Z",
"cpu": 16.731,
"memory": 9024120.000,
"threadInfo": {
"isThreadStarving": "False",
....
}

},
{
"dateUtc": "2021-11-17T23:38:28.3115496Z",
"cpu": 16.731,
"memory": 9024120.000,
"threadInfo": {
"isThreadStarving": "False",
....
}

},
...
]
  • If the cpu values are over 70%, the timeout is likely to be caused by CPU exhaustion. In this case, the solution is to investigate the source of the high CPU utilization and reduce it, or scale the machine to a larger resource size.
  • If the threadInfo/isThreadStarving nodes have True values, the cause is thread starvation. In this case the solution is to investigate the source/s of the thread starvation (potentially locked threads), or scale the machine/s to a larger resource size.
  • If the dateUtc time in-between measurements isn't approximately 10 seconds, it also would indicate contention on the thread pool. CPU is measured as an independent Task that is enqueued in the thread pool every 10 seconds, if the time in-between measurement is longer, it would indicate that the async Tasks aren't able to be processed in a timely fashion. Most common scenarios are when doing blocking calls over async code in the application code.

Solution

The client application that uses the SDK should be scaled up or out.

Socket or port availability might be low

When running in Azure, clients using the .NET SDK can hit Azure SNAT (PAT) port exhaustion.

Solution 1

If you're running on Azure VMs, follow the SNAT port exhaustion guide.

Solution 2

If you're running on Azure App Service, follow the connection errors troubleshooting guide and use App Service diagnostics.

Solution 3

If you're running on Azure Functions, verify you're following the Azure Functions recommendation of maintaining singleton or static clients for all of the involved services (including Azure Cosmos DB). Check the service limits based on the type and size of your Function App hosting.

Solution 4

If you use an HTTP proxy, make sure it can support the number of connections configured in the SDK ConnectionPolicy. Otherwise, you'll face connection issues.

Create multiple client instances

Creating multiple client instances might lead to connection contention and timeout issues. The Diagnostics contain two relevant properties:

{
    "NumberOfClientsCreated":X,
    "NumberOfActiveClients":Y,
}

NumberOfClientsCreated tracks the number of times a CosmosClient was created within the same AppDomain, and NumberOfActiveClients tracks the active clients (not disposed). The expectation is that if the singleton pattern is followed, X would match the number of accounts the application works with and that X is equal to Y.

If X is greater than Y, it means the application is creating and disposing client instances. This can lead to connection contention and/or CPU contention.

Solution

Follow the performance tips, and use a single CosmosClient instance per account across an entire process. Avoid creating and disposing clients.

Hot partition key

Azure Cosmos DB distributes the overall provisioned throughput evenly across physical partitions. When there's a hot partition, one or more logical partition keys on a physical partition are consuming all the physical partition's Request Units per second (RU/s). At the same time, the RU/s on other physical partitions are going unused. As a symptom, the total RU/s consumed will be less than the overall provisioned RU/s at the database or container, but you'll still see throttling (429s) on the requests against the hot logical partition key. Use the Normalized RU Consumption metric to see if the workload is encountering a hot partition.

Solution

Choose a good partition key that evenly distributes request volume and storage. Learn how to change your partition key.

High degree of concurrency

The application is doing a high level of concurrency, which can lead to contention on the channel.

Solution

The client application that uses the SDK should be scaled up or out.

Large requests or responses

Large requests or responses can lead to head-of-line blocking on the channel and exacerbate contention, even with a relatively low degree of concurrency.

Solution

The client application that uses the SDK should be scaled up or out.

Failure rate is within the Azure Cosmos DB SLA

The application should be able to handle transient failures and retry when necessary. Any 408 exceptions aren't retried because on create paths it's impossible to know if the service created the item or not. Sending the same item again for create will cause a conflict exception. User applications business logic might have custom logic to handle conflicts, which would break from the ambiguity of an existing item versus conflict from a create retry.

Failure rate violates the Azure Cosmos DB SLA

Contact Azure Support.

Next steps