Azure Cosmos DB SDK observability

APPLIES TO: NoSQL

The Azure Cosmos DB .NET and Java SDKs support distributed tracing to help you monitor your applications. Tracing the flow of requests is helpful in debugging, analyzing latency and performance, and gathering diagnostics. Instrument tracing for your applications using OpenTelemetry, which is vendor-neutral and has a set of semantic conventions to ensure a standardized data format regardless of your chosen exporter, or use the Application Insights SDK or Azure Monitor OpenTelemetry Distro.

Get started

Distributed tracing is available in the following SDKs:

Trace attributes

Azure Cosmos DB traces follow the OpenTelemetry database specification and also provide several custom attributes. You can see different attributes depending on the operation of your request, and these attributes are core attributes for all requests.

Gather diagnostics

If you configured logs in your trace provider, you can automatically get diagnostics for Azure Cosmos DB requests that failed or had high latency. These logs can help you diagnose failed and slow requests without requiring any custom code to capture them.

CosmosClientOptions options = new CosmosClientOptions()
{
    CosmosClientTelemetryOptions = new CosmosClientTelemetryOptions()
    {
        DisableDistributedTracing = false,
        CosmosThresholdOptions = new CosmosThresholdOptions()
        {
            PointOperationLatencyThreshold = TimeSpan.FromMilliseconds(100),
            NonPointOperationLatencyThreshold = TimeSpan.FromMilliseconds(500)
        }
    },
};

{ 
    "Logging": {​
        "LogLevel": {​
            "Azure-Cosmos-Operation-Request-Diagnostics": "Information"​
        }​
    }
}

    CosmosAsyncClient client = new CosmosClientBuilder()
            .endpoint("<Your account endpoint>")
            .key("<Your account key>")
            .clientTelemetryConfig(new CosmosClientTelemetryConfig()
                .diagnosticsThresholds(
                    new CosmosDiagnosticsThresholds()
                        .setPointOperationLatencyThreshold(Duration.ofMillis(100))
                        .setNonPointOperationLatencyThreshold(Duration.ofMillis(2000))
                        .setRequestChargeThreshold(100)))
            .buildAsyncClient();

Configure OpenTelemetry

To use OpenTelemetry with the Azure Cosmos DB SDKs, add the Azure.Cosmos.Operation source to your trace provider. OpenTelemetry is compatible with many exporters that can ingest your data. The following sample uses the Azure Monitor OpenTelemetry Exporter, but you can choose to configure any exporter you wish. Depending on your chosen exporter, you might see a delay ingesting data of up to a few minutes.

This sample shows how to configure OpenTelemetry for a .NET console app. See the complete sample on GitHub.

namespace Cosmos.Samples.OpenTelemetry
{
    using global::OpenTelemetry;
    using global::OpenTelemetry.Trace;
    using global::OpenTelemetry.Resources;
    using System;
    using System.Threading.Tasks;
    using Newtonsoft.Json;
    using Microsoft.Azure.Cosmos;
    using Microsoft.Extensions.Azure;
    using Microsoft.Extensions.Logging;
    using Microsoft.Extensions.Configuration;
    using Azure.Monitor.OpenTelemetry.Exporter;
    using System.Diagnostics;

    internal class Program
    {
        private static readonly string databaseName = "samples";
        private static readonly string containerName = "otel-sample";
        private static readonly string serviceName = "MySampleService";

        private static TracerProvider? _traceProvider;

        static async Task Main()
        {
            try
            {
                IConfigurationRoot configuration = new ConfigurationBuilder()
                                                            .AddJsonFile("AppSettings.json")
                                                            .Build();

                string endpoint = configuration["CosmosDBEndPointUrl"];
                if (string.IsNullOrEmpty(endpoint))
                {
                    throw new ArgumentNullException("Please specify a valid CosmosDBEndPointUrl in the appSettings.json");
                }

                string authKey = configuration["CosmosDBAuthorizationKey"];
                if (string.IsNullOrEmpty(authKey) || string.Equals(authKey, "Super secret key"))
                {
                    throw new ArgumentException("Please specify a valid CosmosDBAuthorizationKey in the appSettings.json");
                }

                string aiConnectionString = configuration["ApplicationInsightsConnectionString"];
                if (string.IsNullOrEmpty(authKey) || string.Equals(authKey, "Super secret connection string"))
                {
                    throw new ArgumentException("Please specify a valid ApplicationInsightsConnectionString in the appSettings.json");
                }

                // <SetUpOpenTelemetry>
                ResourceBuilder resource = ResourceBuilder.CreateDefault().AddService(
                            serviceName: serviceName,
                            serviceVersion: "1.0.0");

                // Set up logging to forward logs to chosen exporter
                using ILoggerFactory loggerFactory
                    = LoggerFactory.Create(builder => builder
                                                        .AddConfiguration(configuration.GetSection("Logging"))
                                                        .AddOpenTelemetry(options =>
                                                        {
                                                            options.IncludeFormattedMessage = true;
                                                            options.SetResourceBuilder(resource);
                                                            options.AddAzureMonitorLogExporter(o => o.ConnectionString = aiConnectionString); // Set up exporter of your choice
                                                        }));
                /*.AddFilter(level => level == LogLevel.Error) // Filter  is irrespective of event type or event name*/

                AzureEventSourceLogForwarder logforwader = new AzureEventSourceLogForwarder(loggerFactory);
                logforwader.Start();

                // Configure OpenTelemetry trace provider
                AppContext.SetSwitch("Azure.Experimental.EnableActivitySource", true);
                _traceProvider = Sdk.CreateTracerProviderBuilder()
                    .AddSource("Azure.Cosmos.Operation", // Cosmos DB source for operation level telemetry
                               "Sample.Application") 
                    .AddAzureMonitorTraceExporter(o => o.ConnectionString = aiConnectionString) // Set up exporter of your choice
                    .AddHttpClientInstrumentation() // Added to capture HTTP telemetry
                    .SetResourceBuilder(resource)
                    .Build();
                // </SetUpOpenTelemetry>

                ActivitySource source = new ActivitySource("Sample.Application");
                using (_ = source.StartActivity(".Net SDK : Azure Monitor : Open Telemetry Sample")) // Application level activity to track the entire execution of the application
                {
                    using (_ = source.StartActivity("GATEWAY MODE")) // Activity to track the execution of the gateway mode
                    {
                        await Program.RunCosmosDbOperation(ConnectionMode.Gateway, endpoint, authKey);
                    }
                    using (_ = source.StartActivity("DIRECT MODE")) // Activity to track the execution of the direct mode
                    {
                        await Program.RunCosmosDbOperation(ConnectionMode.Direct, endpoint, authKey);
                    }
                }
               
            }
            finally
            {
                _traceProvider?.Dispose();
                // Sleep is required for logging in console apps to ensure that telemetry is sent to the back-end even if application terminates.
                await Task.Delay(5000);

                Console.WriteLine("End of demo.");
            }
        }

        private static async Task RunCosmosDbOperation(ConnectionMode connMode, string endpoint, string authKey)
        {
            // <EnableDistributedTracing>
            CosmosClientOptions options = new CosmosClientOptions()
            {
                CosmosClientTelemetryOptions = new CosmosClientTelemetryOptions()
                {
                    DisableDistributedTracing = false
                },
                ConnectionMode = connMode
            };
            // </EnableDistributedTracing>

            using (CosmosClient client = new CosmosClient(endpoint, authKey, options))
            {
                Console.WriteLine($"Getting container reference for {containerName}.");

                ContainerProperties properties = new ContainerProperties(containerName, partitionKeyPath: "/id");

                await client.CreateDatabaseIfNotExistsAsync(databaseName);
                Container container = await client.GetDatabase(databaseName).CreateContainerIfNotExistsAsync(properties);

                await Program.RunCrudDemo(container);
            }
        }

        public static async Task RunCrudDemo(Container container)
        {
            // Any operations will automatically generate telemetry 

            for(int i = 1; i <= 5; i++)
            {
                await container.CreateItemAsync(new Item { Id = $"{i}", Status = "new" }, new PartitionKey($"{i}"));
                Console.WriteLine($"Created document with id: {i}");
            }

            for (int i = 1; i <= 5; i++)
            {
                await container.ReadItemAsync<Item>($"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Read document with id: {i}");
            }

            try
            {
                await container.ReadItemAsync<Item>($"random key", new PartitionKey($"random partition"));
            }
            catch(Exception)
            {
                Console.WriteLine("Generate exception by reading an invalid key");
            }
            
            for (int i = 1; i <= 5; i++)
            {
                await container.ReplaceItemAsync(new Item { Id = $"{i}", Status = "updated" }, $"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Updated document with id: {i}");
            }

            for (int i = 1; i <= 5; i++)
            {
                await container.DeleteItemAsync<Item>($"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Deleted document with id: {i}");
            }
        }
    }

    internal class Item
    {
        [JsonProperty("id")]
        public string Id { get; set; }

        public string Status { get; set; }
    }
}

Configure the Application Insights SDK

There are many different ways to configure Application Insights depending on the language your application is written in and your compute environment. For more information, see the Application Insights documentation. Ingestion of data into Application Insights can take up to a few minutes.

The following sample shows how to configure Application Insights for a .NET console app. See the complete sample on GitHub.

namespace Cosmos.Samples.ApplicationInsights
{
    using System;
    using System.Threading.Tasks;
    using Newtonsoft.Json;
    using Microsoft.Azure.Cosmos;
    using Microsoft.Extensions.Configuration;
    using Microsoft.Extensions.DependencyInjection;
    using Microsoft.ApplicationInsights;
    using Microsoft.ApplicationInsights.WorkerService;
    using Microsoft.ApplicationInsights.DataContracts;

    internal class Program
    {
        private static readonly string databaseName = "samples";
        private static readonly string containerName = "ai-sample";

        private static TelemetryClient telemetryClient;

        static async Task Main()
        {
            try
            {
                IConfigurationRoot configuration = new ConfigurationBuilder()
                            .AddJsonFile("AppSettings.json")
                            .Build();

                string endpoint = configuration["CosmosDBEndPointUrl"];
                if (string.IsNullOrEmpty(endpoint))
                {
                    throw new ArgumentNullException("Please specify a valid CosmosDBEndPointUrl in the appSettings.json");
                }

                string authKey = configuration["CosmosDBAuthorizationKey"];
                if (string.IsNullOrEmpty(authKey) || string.Equals(authKey, "Super secret key"))
                {
                    throw new ArgumentException("Please specify a valid CosmosDBAuthorizationKey in the appSettings.json");
                }

                string aiConnectionString = configuration["ApplicationInsightsConnectionString"];
                if (string.IsNullOrEmpty(authKey) || string.Equals(authKey, "Super secret connection string"))
                {
                    throw new ArgumentException("Please specify a valid ApplicationInsightsConnectionString in the appSettings.json");
                }

                // <SetUpApplicationInsights>
                IServiceCollection services = new ServiceCollection();
                services.AddApplicationInsightsTelemetryWorkerService((ApplicationInsightsServiceOptions options) => options.ConnectionString = aiConnectionString);

                IServiceProvider serviceProvider = services.BuildServiceProvider();
                telemetryClient = serviceProvider.GetRequiredService<TelemetryClient>();
                // </SetUpApplicationInsights>

                var infoOperation = telemetryClient.StartOperation<DependencyTelemetry>(".Net SDK : ApplicationInsights SDK"); // Application level activity to track the entire execution of the application

                var gops = telemetryClient.StartOperation<DependencyTelemetry>("GATEWAY MODE"); // Activity to track the execution of the gateway mode
                await Program.RunCosmosDbOperation(ConnectionMode.Gateway, endpoint, authKey);
                telemetryClient.StopOperation(gops);

                var dops = telemetryClient.StartOperation<DependencyTelemetry>("DIRECT MODE"); // Activity to track the execution of the direct mode
                await Program.RunCosmosDbOperation(ConnectionMode.Direct, endpoint, authKey); 
                telemetryClient.StopOperation(dops);

                telemetryClient.StopOperation(infoOperation);
            }
            finally
            {
                // Explicitly calling Flush() followed by sleep is required for Application Insights logging in console apps to ensure that telemetry is sent to the back-end even if application terminates.
                telemetryClient?.Flush();
                await Task.Delay(5000);

                Console.WriteLine("End of demo.");
            }
        }

        private static async Task RunCosmosDbOperation(ConnectionMode connMode, string endpoint, string authKey)
        {
            // <EnableDistributedTracing>
            CosmosClientOptions options = new CosmosClientOptions()
            {
                CosmosClientTelemetryOptions = new CosmosClientTelemetryOptions()
                {
                    DisableDistributedTracing = false
                },
                ConnectionMode = connMode
            };
            // </EnableDistributedTracing>

            using (CosmosClient client = new CosmosClient(endpoint, authKey, options))
            {
                Console.WriteLine($"Getting container reference for {containerName}.");

                ContainerProperties properties = new ContainerProperties(containerName, partitionKeyPath: "/id");

                await client.CreateDatabaseIfNotExistsAsync(databaseName);
                Container container = await client.GetDatabase(databaseName).CreateContainerIfNotExistsAsync(properties);

                await Program.RunCrudDemo(container);
            }
        }

        public static async Task RunCrudDemo(Container container)
        {
            // Any operations will automatically generate telemetry 

            for (int i = 1; i <= 5; i++)
            {
                await container.CreateItemAsync(new Item { Id = $"{i}", Status = "new" }, new PartitionKey($"{i}"));
                Console.WriteLine($"Created document with id: {i}");
            }

            for (int i = 1; i <= 5; i++)
            {
                await container.ReadItemAsync<Item>($"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Read document with id: {i}");
            }

            for (int i = 1; i <= 5; i++)
            {
                await container.ReplaceItemAsync(new Item { Id = $"{i}", Status = "updated" }, $"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Updated document with id: {i}");
            }

            for (int i = 1; i <= 5; i++)
            {
                await container.DeleteItemAsync<Item>($"{i}", new PartitionKey($"{i}"));
                Console.WriteLine($"Deleted document with id: {i}");
            }
        }
    }

    internal class Item
    {
        [JsonProperty("id")]
        public string Id { get; set; }

        public string Status { get; set; }
    }
}

Once trace data is ingested into Application Insights, you can visualize it in the Azure portal to understand the request flow in your application. Here's an example of trace data from a cross partition query in the transaction search in the left navigation of the Azure portal.

Next Steps

• Get started with the .NET SDK
• Get started with the Java SDK
• Monitor Azure Cosmos DB