Add and modify Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications

This guide provides instructions on integrating and customizing OpenTelemetry (OTel) instrumentation within Azure Monitor Application Insights.

To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

Automatic data collection

The distros automatically collect data by bundling OpenTelemetry instrumentation libraries.

Included instrumentation libraries

The Azure Monitor Exporter doesn't include any instrumentation libraries.

You can collect dependencies from the Azure Software Development Kits (SDKs) using the following code sample to manually subscribe to the source.

// Create an OpenTelemetry tracer provider builder.
// It is important to keep the TracerProvider instance active throughout the process lifetime.
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
	// The following line subscribes to dependencies emitted from Azure SDKs
    .AddSource("Azure.*")
    .AddAzureMonitorTraceExporter()
    .AddHttpClientInstrumentation(o => o.FilterHttpRequestMessage = (_) =>
	{
    	// Azure SDKs create their own client span before calling the service using HttpClient
		// In this case, we would see two spans corresponding to the same operation
		// 1) created by Azure SDK 2) created by HttpClient
		// To prevent this duplication we are filtering the span from HttpClient
		// as span from Azure SDK contains all relevant information needed.
		var parentActivity = Activity.Current?.Parent;
		if (parentActivity != null && parentActivity.Source.Name.Equals("Azure.Core.Http"))
		{
		    return false;
		}
		return true;
	})
    .Build();

To reduce or increase the number of logs sent to Azure Monitor, configure logging to set the appropriate log level or apply filters. For example, you can choose to send only Warning and Error logs to OpenTelemetry/Azure Monitor. OpenTelemetry doesn't control log routing or filtering - your ILogger configuration makes these decisions. For more information on configuring ILogger, see Configure logging.

Footnotes

  • ¹: Supports automatic reporting of unhandled/uncaught exceptions
  • ²: Supports OpenTelemetry Metrics

Note

The Azure Monitor OpenTelemetry Distros include custom mapping and logic to automatically emit Application Insights standard metrics.

Tip

All OpenTelemetry metrics whether automatically collected from instrumentation libraries or manually collected from custom coding are currently considered Application Insights "custom metrics" for billing purposes. Learn more.

Add a community instrumentation library

You can collect more data automatically when you include instrumentation libraries from the OpenTelemetry community.

Caution

We don't support or guarantee the quality of community instrumentation libraries. To suggest one for our distro, post or up-vote in our feedback community. Be aware, some are based on experimental OpenTelemetry specs and might introduce future breaking changes.

The following example demonstrates how the Runtime Instrumentation can be added to collect extra metrics:

// Create a new OpenTelemetry meter provider and add runtime instrumentation and the Azure Monitor metric exporter.
// It is important to keep the MetricsProvider instance active throughout the process lifetime.
var metricsProvider = Sdk.CreateMeterProviderBuilder()
    .AddRuntimeInstrumentation()
    .AddAzureMonitorMetricExporter();

Collect custom telemetry

This section explains how to collect custom telemetry from your application.

Depending on your language and signal type, there are different ways to collect custom telemetry, including:

  • OpenTelemetry API
  • Language-specific logging/metrics libraries
  • Application Insights Classic API

The following table represents the currently supported custom telemetry types:

Language Custom Events Custom Metrics Dependencies Exceptions Page Views Requests Traces
ASP.NET Core
   OpenTelemetry API Yes Yes Yes Yes
   ILogger API Yes
   AI Classic API
Java
   OpenTelemetry API Yes Yes Yes Yes
   Logback, Log4j, JUL Yes Yes
   Micrometer Metrics Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Node.js
   OpenTelemetry API Yes Yes Yes Yes
Python
   OpenTelemetry API Yes Yes Yes Yes
   Python Logging Module Yes
   Events Extension Yes Yes

Note

Application Insights Java 3.x and Application Insights Node.js 3.x collect telemetry from the Application Insights Classic API. This behavior simplifies upgrades and temporarily supports custom telemetry until the OpenTelemetry API includes all custom telemetry types.

Add custom metrics

In this context, the custom metrics term refers to manually instrumenting your code to collect extra metrics beyond what the OpenTelemetry Instrumentation Libraries automatically collect.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum, and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Other aggregation types aren't meaningful in most cases.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the Application Insights GetMetric Classic API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser effect on your application's performance.

Histogram example

public class Program
{
    // Create a static readonly Meter object named "OTel.AzureMonitor.Demo".
    // This meter will be used to track metrics about the application.
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        // Create a new MeterProvider object using the OpenTelemetry SDK.
        // The MeterProvider object is responsible for managing meters and sending
        // metric data to exporters.
        // It is important to keep the MetricsProvider instance active
        // throughout the process lifetime.
        //
        // The MeterProviderBuilder is configured to add a meter named
        // "OTel.AzureMonitor.Demo" and an Azure Monitor metric exporter.
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter()
            .Build();

        // Create a new Histogram metric named "FruitSalePrice".
        // This metric will track the distribution of fruit sale prices.
        Histogram<long> myFruitSalePrice = meter.CreateHistogram<long>("FruitSalePrice");

        // Create a new Random object. This object will be used to generate random sale prices.
        var rand = new Random();
        
        // Record a few random sale prices for apples and lemons, with different colors.
        // Each record includes a timestamp, a value, and a set of attributes.
        // The attributes can be used to filter and analyze the metric data.
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "green"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "apple"), new("color", "red"));
        myFruitSalePrice.Record(rand.Next(1, 1000), new("name", "lemon"), new("color", "yellow"));

        // Display a message to the user and wait for them to press Enter.
        // This allows the user to see the message and the console before the
        // application exits.
        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Counter example

public class Program
{
    // Create a static readonly Meter object named "OTel.AzureMonitor.Demo".
    // This meter will be used to track metrics about the application.
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        // Create a new MeterProvider object using the OpenTelemetry SDK.
        // The MeterProvider object is responsible for managing meters and sending
        // metric data to exporters.
        // It is important to keep the MetricsProvider instance active
        // throughout the process lifetime.
        //
        // The MeterProviderBuilder is configured to add a meter named
        // "OTel.AzureMonitor.Demo" and an Azure Monitor metric exporter.
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter()
            .Build();

        // Create a new counter metric named "MyFruitCounter".
        // This metric will track the number of fruits sold.
        Counter<long> myFruitCounter = meter.CreateCounter<long>("MyFruitCounter");

        // Record the number of fruits sold, grouped by name and color.
        myFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
        myFruitCounter.Add(2, new("name", "apple"), new("color", "green"));
        myFruitCounter.Add(5, new("name", "apple"), new("color", "red"));
        myFruitCounter.Add(4, new("name", "lemon"), new("color", "yellow"));

        // Display a message to the user and wait for them to press Enter.
        // This allows the user to see the message and the console before the
        // application exits.
        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
}

Gauge example

public class Program
{
    // Create a static readonly Meter object named "OTel.AzureMonitor.Demo".
    // This meter will be used to track metrics about the application.
    private static readonly Meter meter = new("OTel.AzureMonitor.Demo");

    public static void Main()
    {
        // Create a new MeterProvider object using the OpenTelemetry SDK.
        // The MeterProvider object is responsible for managing meters and sending
        // metric data to exporters. 
        // It is important to keep the MetricsProvider instance active
        // throughout the process lifetime.
        //
        // The MeterProviderBuilder is configured to add a meter named
        // "OTel.AzureMonitor.Demo" and an Azure Monitor metric exporter.
        using var meterProvider = Sdk.CreateMeterProviderBuilder()
            .AddMeter("OTel.AzureMonitor.Demo")
            .AddAzureMonitorMetricExporter()
            .Build();

        // Get the current process.
        var process = Process.GetCurrentProcess();
        
        // Create a new observable gauge metric named "Thread.State".
        // This metric will track the state of each thread in the current process.
        ObservableGauge<int> myObservableGauge = meter.CreateObservableGauge("Thread.State", () => GetThreadState(process));

        // Display a message to the user and wait for them to press Enter.
        // This allows the user to see the message and the console before the
        // application exits.
        System.Console.WriteLine("Press Enter key to exit.");
        System.Console.ReadLine();
    }
    
    private static IEnumerable<Measurement<int>> GetThreadState(Process process)
    {
        // Iterate over all threads in the current process.
        foreach (ProcessThread thread in process.Threads)
        {
            // Create a measurement for each thread, including the thread state, process ID, and thread ID.
            yield return new((int)thread.ThreadState, new("ProcessId", process.Id), new("ThreadId", thread.Id));
        }
    }
}

Add custom exceptions

Select instrumentation libraries automatically report exceptions to Application Insights. However, you might want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't ordinarily reported. You might wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

  • To log an Exception using an Activity:

    // Start a new activity named "ExceptionExample".
    using (var activity = activitySource.StartActivity("ExceptionExample"))
    {
        // Try to execute some code.
        try
        {
            throw new Exception("Test exception");
        }
        // If an exception is thrown, catch it and set the activity status to "Error".
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error);
            activity?.RecordException(ex);
        }
    }
    
  • To log an Exception using ILogger:

    // Create a logger using the logger factory. The logger category name is used to filter and route log messages.
    var logger = loggerFactory.CreateLogger("ExceptionExample");
    
    try
    {
        // Try to execute some code.
        throw new Exception("Test Exception");
    }
    catch (Exception ex)
    {
        // Log an error message with the exception. The log level is set to "Error" and the event ID is set to 0.
        // The log message includes a template and a parameter. The template will be replaced with the value of the parameter when the log message is written.
        logger.Log(
            logLevel: LogLevel.Error,
            eventId: 0,
            exception: ex,
            message: "Hello {name}.",
            args: new object[] { "World" });
    }
    

Add custom spans

You might want to add a custom span in two scenarios. First, when there's a dependency request not already collected by an instrumentation library. Second, when you wish to model an application process as a span on the end-to-end transaction view.

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. You create ActivitySource directly by using its constructor instead of by using TracerProvider. Each ActivitySource class must be explicitly connected to TracerProvider by using AddSource(). It's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

// Create an OpenTelemetry tracer provider builder.
// It is important to keep the TracerProvider instance active throughout the process lifetime.
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
        .AddSource("ActivitySourceName")
        .AddAzureMonitorTraceExporter()
        .Build();

// Create an activity source named "ActivitySourceName".
var activitySource = new ActivitySource("ActivitySourceName");

// Start a new activity named "CustomActivity". This activity will be traced and the trace data will be sent to Azure Monitor.
using (var activity = activitySource.StartActivity("CustomActivity"))
{
    // your code here
}

StartActivity defaults to ActivityKind.Internal, but you can provide any other ActivityKind. ActivityKind.Client, ActivityKind.Producer, and ActivityKind.Internal are mapped to Application Insights dependencies. ActivityKind.Server and ActivityKind.Consumer are mapped to Application Insights requests.

Send custom events

Application Insights stores custom events in the customEvents table. You can analyze, filter, and visualize them by using Azure Monitor tools.

Use custom events to track important actions that support business goals. Examples include selecting a button, submitting a form, and completing a purchase.

Combine custom events with authenticated user IDs and session context to enable:

  • Tracking behavior across sessions.
  • Analyzing conversion funnels based on user actions.
  • Segmenting users by how they interact with your app.

If you want to automate the collection of client-side interaction events, you can use the plugin in the JavaScript SDK.

Custom events are in Public Preview and use Azure.Monitor.OpenTelemetry.Exporter 1.4.0-beta.3.

Important

See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

To send a CustomEvent using ILogger, set the "microsoft.custom_event.name" attribute in the message template.

// Create a logger factory and configure OpenTelemetry with Azure Monitor
var loggerFactory = LoggerFactory.Create(builder =>
{
    builder
        .AddOpenTelemetry(options =>
        {
            options.AddAzureMonitorLogExporter();
        });
});

// Create a logger for the specified category
var logger = loggerFactory.CreateLogger(logCategoryName);

// Log a custom event with a custom name and additional attribute
// The 'microsoft.custom_event.name' value will be used as the name of the customEvent
logger.LogInformation("{microsoft.custom_event.name} {additional_attrs}", "test-event-name", "val1");

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Add a custom property to a Span

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests, dependencies, traces, or exceptions table.

To add span attributes, use either of the following two ways:

  • Use options provided by instrumentation libraries.
  • Add a custom span processor.

Tip

The advantage of using options provided by instrumentation libraries, when they're available, is that the entire context is available. As a result, users can select to add or filter more attributes. For example, the enrich option in the HttpClient instrumentation library gives users access to the httpRequestMessage itself. They can select anything from it and store it as an attribute.

  1. Many instrumentation libraries provide an enrich option. For guidance, see the readme files of individual instrumentation libraries:

  2. Use a custom processor:

    Tip

    Add the processor shown here before the Azure Monitor Exporter.

    // Create an OpenTelemetry tracer provider builder.
    // It is important to keep the TracerProvider instance active throughout the process lifetime.
    using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            // Add a source named "OTel.AzureMonitor.Demo".
            .AddSource("OTel.AzureMonitor.Demo") // Add a new processor named ActivityEnrichingProcessor.
            .AddProcessor(new ActivityEnrichingProcessor()) // Add the Azure Monitor trace exporter.
            .AddAzureMonitorTraceExporter() // Add the Azure Monitor trace exporter.
            .Build();
    

    Add ActivityEnrichingProcessor.cs to your project with the following code:

    public class ActivityEnrichingProcessor : BaseProcessor<Activity>
    {
        // The OnEnd method is called when an activity is finished. This is the ideal place to enrich the activity with additional data.
        public override void OnEnd(Activity activity)
        {
            // Update the activity's display name.
            // The updated activity will be available to all processors which are called after this processor.
            activity.DisplayName = "Updated-" + activity.DisplayName;
            // Set custom tags on the activity.
            activity.SetTag("CustomDimension1", "Value1");
            activity.SetTag("CustomDimension2", "Value2");
        }
    }
    

Set the user IP

You can populate the client_IP field for requests by setting an attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Use the custom property example, but replace the following lines of code in ActivityEnrichingProcessor.cs:

// Add the client IP address to the activity as a tag.
// only applicable in case of activity.Kind == Server
activity.SetTag("client.address", "<IP Address>");

Set the user ID or authenticated user ID

You can populate the user_Id or user_AuthenticatedId field for requests by using the following guidance. User ID is an anonymous user identifier. Authenticated User ID is a known user identifier.

Important

Consult applicable privacy laws before you set the Authenticated User ID.

Use the custom property example:

// Add the user ID to the activity as a tag, but only if the activity is not null.
activity?.SetTag("enduser.id", "<User Id>");

Add log attributes

OpenTelemetry uses .NET's ILogger. Attaching custom dimensions to logs can be accomplished using a message template.

Get the trace ID or span ID

You can obtain the Trace ID and Span ID of the currently active Span using following steps.

Note

The Activity and ActivitySource classes from the System.Diagnostics namespace represent the OpenTelemetry concepts of Span and Tracer, respectively. It's because parts of the OpenTelemetry tracing API are incorporated directly into the .NET runtime. To learn more, see Introduction to OpenTelemetry .NET Tracing API.

// Get the current activity.
Activity activity = Activity.Current;
// Get the trace ID of the activity.
string traceId = activity?.TraceId.ToHexString();
// Get the span ID of the activity.
string spanId = activity?.SpanId.ToHexString();

Next steps