Add and modify Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications

  • Article

This guide provides instructions on integrating and customizing OpenTelemetry (OTel) instrumentation within Azure Monitor Application Insights.

To learn more about OpenTelemetry concepts, see the OpenTelemetry overview or OpenTelemetry FAQ.

Automatic data collection

The distros automatically collect data by bundling OpenTelemetry instrumentation libraries.

Included instrumentation libraries

Requests

  • JMS consumers
  • Kafka consumers
  • Netty
  • Quartz
  • RabbitMQ
  • Servlets
  • Spring scheduling

Note

Servlet and Netty autoinstrumentation covers the majority of Java HTTP services, including Java EE, Jakarta EE, Spring Boot, Quarkus, and Micronaut.

Dependencies (plus downstream distributed trace propagation)

  • Apache HttpClient
  • Apache HttpAsyncClient
  • AsyncHttpClient
  • Google HttpClient
  • gRPC
  • java.net.HttpURLConnection
  • Java 11 HttpClient
  • JAX-RS client
  • Jetty HttpClient
  • JMS
  • Kafka
  • Netty client
  • OkHttp
  • RabbitMQ

Dependencies (without downstream distributed trace propagation)

  • Cassandra
  • JDBC
  • MongoDB (async and sync)
  • Redis (Lettuce and Jedis)

Metrics

  • Micrometer Metrics, including Spring Boot Actuator metrics
  • JMX Metrics

Logs

  • Logback (including MDC properties) ¹ ³
  • Log4j (including MDC/Thread Context properties) ¹ ³
  • JBoss Logging (including MDC properties) ¹ ³
  • java.util.logging ¹ ³

Default collection

Telemetry emitted by the following Azure SDKs is automatically collected by default:

[//]: # "Azure Cosmos DB 4.22.0+ due to https://github.com/Azure/azure-sdk-for-java/pull/25571"
[//]: # "the remaining above names and links scraped from https://azure.github.io/azure-sdk/releases/latest/java.html"
[//]: # "and version synched manually against the oldest version in maven central built on azure-core 1.14.0"
[//]: # ""
[//]: # "var table = document.querySelector('#tg-sb-content > div > table')"
[//]: # "var str = ''"
[//]: # "for (var i = 1, row; row = table.rows[i]; i++) {"
[//]: # "  var name = row.cells[0].getElementsByTagName('div')[0].textContent.trim()"
[//]: # "  var stableRow = row.cells[1]"
[//]: # "  var versionBadge = stableRow.querySelector('.badge')"
[//]: # "  if (!versionBadge) {"
[//]: # "    continue"
[//]: # "  }"
[//]: # "  var version = versionBadge.textContent.trim()"
[//]: # "  var link = stableRow.querySelectorAll('a')[2].href"
[//]: # "  str += '* [' + name + '](' + link + ') ' + version + '\n'"
[//]: # "}"
[//]: # "console.log(str)"

Footnotes

  • ¹: Supports automatic reporting of unhandled/uncaught exceptions
  • ²: Supports OpenTelemetry Metrics
  • ³: By default, logging is only collected at INFO level or higher. To change this setting, see the configuration options.
  • ⁴: By default, logging is only collected when that logging is performed at the WARNING level or higher.

Note

The Azure Monitor OpenTelemetry Distros include custom mapping and logic to automatically emit Application Insights standard metrics.

Tip

All OpenTelemetry metrics whether automatically collected from instrumentation libraries or manually collected from custom coding are currently considered Application Insights "custom metrics" for billing purposes. Learn more.

Add a community instrumentation library

You can collect more data automatically when you include instrumentation libraries from the OpenTelemetry community.

Caution

We don't support or guarantee the quality of community instrumentation libraries. To suggest one for our distro, post or up-vote in our feedback community. Be aware, some are based on experimental OpenTelemetry specs and might introduce future breaking changes.

You can't extend the Java Distro with community instrumentation libraries. To request that we include another instrumentation library, open an issue on our GitHub page. You can find a link to our GitHub page in Next Steps.

Collect custom telemetry

This section explains how to collect custom telemetry from your application.

Depending on your language and signal type, there are different ways to collect custom telemetry, including:

  • OpenTelemetry API
  • Language-specific logging/metrics libraries
  • Application Insights Classic API

The following table represents the currently supported custom telemetry types:

Language Custom Events Custom Metrics Dependencies Exceptions Page Views Requests Traces
ASP.NET Core
   OpenTelemetry API Yes Yes Yes Yes
   ILogger API Yes
   AI Classic API
Java
   OpenTelemetry API Yes Yes Yes Yes
   Logback, Log4j, JUL Yes Yes
   Micrometer Metrics Yes
   AI Classic API Yes Yes Yes Yes Yes Yes Yes
Node.js
   OpenTelemetry API Yes Yes Yes Yes
Python
   OpenTelemetry API Yes Yes Yes Yes
   Python Logging Module Yes
   Events Extension Yes Yes

Note

Application Insights Java 3.x listens for telemetry that's sent to the Application Insights Classic API. Similarly, Application Insights Node.js 3.x collects events created with the Application Insights Classic API. This makes upgrading easier and fills a gap in our custom telemetry support until all custom telemetry types are supported via the OpenTelemetry API.

Add custom metrics

In this context, the custom metrics term refers to manually instrumenting your code to collect additional metrics beyond what the OpenTelemetry Instrumentation Libraries automatically collect.

The OpenTelemetry API offers six metric "instruments" to cover various metric scenarios and you need to pick the correct "Aggregation Type" when visualizing metrics in Metrics Explorer. This requirement is true when using the OpenTelemetry Metric API to send metrics and when using an instrumentation library.

The following table shows the recommended aggregation types for each of the OpenTelemetry Metric Instruments.

OpenTelemetry Instrument Azure Monitor Aggregation Type
Counter Sum
Asynchronous Counter Sum
Histogram Min, Max, Average, Sum, and Count
Asynchronous Gauge Average
UpDownCounter Sum
Asynchronous UpDownCounter Sum

Caution

Aggregation types beyond what's shown in the table typically aren't meaningful.

The OpenTelemetry Specification describes the instruments and provides examples of when you might use each one.

Tip

The histogram is the most versatile and most closely equivalent to the Application Insights GetMetric Classic API. Azure Monitor currently flattens the histogram instrument into our five supported aggregation types, and support for percentiles is underway. Although less versatile, other OpenTelemetry instruments have a lesser impact on your application's performance.

Histogram example

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.metrics.Meter;

public class Program {

    public static void main(String[] args) {
        Meter meter = GlobalOpenTelemetry.getMeter("OTEL.AzureMonitor.Demo");
        DoubleHistogram histogram = meter.histogramBuilder("histogram").build();
        histogram.record(1.0);
        histogram.record(100.0);
        histogram.record(30.0);
    }
}

Counter example

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;

public class Program {

    public static void main(String[] args) {
        Meter meter = GlobalOpenTelemetry.getMeter("OTEL.AzureMonitor.Demo");

        LongCounter myFruitCounter = meter
                .counterBuilder("MyFruitCounter")
                .build();

        myFruitCounter.add(1, Attributes.of(AttributeKey.stringKey("name"), "apple", AttributeKey.stringKey("color"), "red"));
        myFruitCounter.add(2, Attributes.of(AttributeKey.stringKey("name"), "lemon", AttributeKey.stringKey("color"), "yellow"));
        myFruitCounter.add(1, Attributes.of(AttributeKey.stringKey("name"), "lemon", AttributeKey.stringKey("color"), "yellow"));
        myFruitCounter.add(2, Attributes.of(AttributeKey.stringKey("name"), "apple", AttributeKey.stringKey("color"), "green"));
        myFruitCounter.add(5, Attributes.of(AttributeKey.stringKey("name"), "apple", AttributeKey.stringKey("color"), "red"));
        myFruitCounter.add(4, Attributes.of(AttributeKey.stringKey("name"), "lemon", AttributeKey.stringKey("color"), "yellow"));
    }
}

Gauge example

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.Meter;

public class Program {

    public static void main(String[] args) {
        Meter meter = GlobalOpenTelemetry.getMeter("OTEL.AzureMonitor.Demo");

        meter.gaugeBuilder("gauge")
                .buildWithCallback(
                        observableMeasurement -> {
                            double randomNumber = Math.floor(Math.random() * 100);
                            observableMeasurement.record(randomNumber, Attributes.of(AttributeKey.stringKey("testKey"), "testValue"));
                        });
    }
}

Add custom exceptions

Select instrumentation libraries automatically report exceptions to Application Insights. However, you might want to manually report exceptions beyond what instrumentation libraries report. For instance, exceptions caught by your code aren't ordinarily reported. You might wish to report them to draw attention in relevant experiences including the failures section and end-to-end transaction views.

You can use opentelemetry-api to update the status of a span and record exceptions.

  1. Add opentelemetry-api-1.0.0.jar (or later) to your application:

    <dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-api</artifactId>
  <version>1.0.0</version>
</dependency>

  2. Set status to error and record an exception in your code:

     import io.opentelemetry.api.trace.Span;
 import io.opentelemetry.api.trace.StatusCode;

 Span span = Span.current();
 span.setStatus(StatusCode.ERROR, "errorMessage");
 span.recordException(e);

Add custom spans

You might want to add a custom span in two scenarios. First, when there's a dependency request not already collected by an instrumentation library. Second, when you wish to model an application process as a span on the end-to-end transaction view.

  • Use the OpenTelemetry annotation

    The simplest way to add your own spans is by using OpenTelemetry's @WithSpan annotation.

    Spans populate the requests and dependencies tables in Application Insights.

    1. Add opentelemetry-instrumentation-annotations-1.32.0.jar (or later) to your application:

      <dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-instrumentation-annotations</artifactId>
    <version>1.32.0</version>
</dependency>

    2. Use the @WithSpan annotation to emit a span each time your method is executed:

      import io.opentelemetry.instrumentation.annotations.WithSpan;

@WithSpan(value = "your span name")
public void yourMethod() {
}

    By default, the span ends up in the dependencies table with dependency type InProc.

    For methods representing a background job not captured by autoinstrumentation, we recommend applying the attribute kind = SpanKind.SERVER to the @WithSpan annotation to ensure they appear in the Application Insights requests table.

  • Use the OpenTelemetry API

    If the preceding OpenTelemetry @WithSpan annotation doesn't meet your needs, you can add your spans by using the OpenTelemetry API.

    1. Add opentelemetry-api-1.0.0.jar (or later) to your application:

      <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

    2. Use the GlobalOpenTelemetry class to create a Tracer:

      import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;

static final Tracer tracer = GlobalOpenTelemetry.getTracer("com.example");

    3. Create a span, make it current, and then end it:

      Span span = tracer.spanBuilder("my first span").startSpan();
try (Scope ignored = span.makeCurrent()) {
    // do stuff within the context of this 
} catch (Throwable t) {
    span.recordException(t);
} finally {
    span.end();
}

Send custom telemetry using the Application Insights Classic API

We recommend you use the OpenTelemetry APIs whenever possible, but there might be some scenarios when you have to use the Application Insights Classic API.

  1. Add applicationinsights-core to your application:

    <dependency>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>applicationinsights-core</artifactId>
  <version>3.4.18</version>
</dependency>

  2. Create a TelemetryClient instance:

    static final TelemetryClient telemetryClient = new TelemetryClient();

  3. Use the client to send custom telemetry:

    Events

    telemetryClient.trackEvent("WinGame");

    Logs

    telemetryClient.trackTrace(message, SeverityLevel.Warning, properties);

    Metrics

    telemetryClient.trackMetric("queueLength", 42.0);

    Dependencies

    boolean success = false;
long startTime = System.currentTimeMillis();
try {
    success = dependency.call();
} finally {
    long endTime = System.currentTimeMillis();
    RemoteDependencyTelemetry telemetry = new RemoteDependencyTelemetry();
    telemetry.setSuccess(success);
    telemetry.setTimestamp(new Date(startTime));
    telemetry.setDuration(new Duration(endTime - startTime));
    telemetryClient.trackDependency(telemetry);
}

    Exceptions

    try {
    ...
} catch (Exception e) {
    telemetryClient.trackException(e);
}

Modify telemetry

This section explains how to modify telemetry.

Add span attributes

These attributes might include adding a custom property to your telemetry. You might also use attributes to set optional fields in the Application Insights schema, like Client IP.

Add a custom property to a Span

Any attributes you add to spans are exported as custom properties. They populate the customDimensions field in the requests, dependencies, traces, or exceptions table.

You can use opentelemetry-api to add attributes to spans.

Adding one or more span attributes populates the customDimensions field in the requests, dependencies, traces, or exceptions table.

  1. Add opentelemetry-api-1.0.0.jar (or later) to your application:

    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

  2. Add custom dimensions in your code:

    import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.common.AttributeKey;

AttributeKey attributeKey = AttributeKey.stringKey("mycustomdimension");
Span.current().setAttribute(attributeKey, "myvalue1");

Set the user IP

You can populate the client_IP field for requests by setting an attribute on the span. Application Insights uses the IP address to generate user location attributes and then discards it by default.

Java automatically populates this field.

Set the user ID or authenticated user ID

You can populate the user_Id or user_AuthenticatedId field for requests by using the following guidance. User ID is an anonymous user identifier. Authenticated User ID is a known user identifier.

Important

Consult applicable privacy laws before you set the Authenticated User ID.

Populate the user ID field in the requests, dependencies, or exceptions table.

  1. Add opentelemetry-api-1.0.0.jar (or later) to your application:

    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

  2. Set user_Id in your code:

    import io.opentelemetry.api.trace.Span;

Span.current().setAttribute("enduser.id", "myuser");

Add log attributes

Logback, Log4j, and java.util.logging are autoinstrumented. Attaching custom dimensions to your logs can be accomplished in these ways:

Get the trace ID or span ID

You can obtain the Trace ID and Span ID of the currently active Span using following steps.

You can use opentelemetry-api to get the trace ID or span ID.

  1. Add opentelemetry-api-1.0.0.jar (or later) to your application:

    <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

  2. Get the request trace ID and the span ID in your code:

    import io.opentelemetry.api.trace.Span;

Span span = Span.current();
String traceId = span.getSpanContext().getTraceId();
String spanId = span.getSpanContext().getSpanId();

Next steps