Filter Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications

Use this guide to filter OpenTelemetry (OTel) data in Azure Monitor Application Insights. Filtering helps you exclude unnecessary telemetry and prevent collection of sensitive data to optimize performance and support compliance.

Reasons to filter out telemetry include:

  • Filtering out health check telemetry to reduce noise.
  • Ensuring personal data and credentials aren't collected.
  • Filtering out low-value telemetry to optimize performance.

To learn more about OpenTelemetry concepts, review the OpenTelemetry overview or OpenTelemetry FAQ.

Note

For Azure Function Apps, see Use OpenTelemetry with Azure Functions.

Filter OpenTelemetry using instrumentation libraries

For a list of all instrumentation libraries included with the Azure Monitor OpenTelemetry Distro, review Add and modify Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications.

Many instrumentation libraries provide a filter option. For guidance, review the corresponding readme files:

Note

Currently, the Azure Monitor OpenTelemetry Distro includes a copy of the SqlClient instrumentation source code to ensure stability while the upstream OpenTelemetry SqlClient library remains experimental.

Options such as SetDbStatementForStoredProcedure aren't usable in our distribution because the code is embedded and not referencing the external package.

Once the SqlClient instrumentation reaches a stable release, Azure Monitor switches to referencing the official package and customization via builder.AddSqlClientInstrumentation(options => { ... }) is available.

Filter telemetry using span processors

  1. Use a custom processor:

    Tip

    Add the processor shown here before adding Azure Monitor.

    // Create an ASP.NET Core application builder.
    var builder = WebApplication.CreateBuilder(args);
    
    // Configure the OpenTelemetry tracer provider to add a new processor named ActivityFilteringProcessor.
    builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddProcessor(new ActivityFilteringProcessor()));
    // Configure the OpenTelemetry tracer provider to add a new source named "ActivitySourceName".
    builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddSource("ActivitySourceName"));
    // Add the Azure Monitor telemetry service to the application. This service will collect and send telemetry data to Azure Monitor.
    builder.Services.AddOpenTelemetry().UseAzureMonitor();
    
    // Build the ASP.NET Core application.
    var app = builder.Build();
    
    // Start the ASP.NET Core application.
    app.Run();
    
  2. Add ActivityFilteringProcessor.cs to your project with the following code:

    public class ActivityFilteringProcessor : BaseProcessor<Activity>
    {
        // The OnStart method is called when an activity is started. This is the ideal place to filter activities.
        public override void OnStart(Activity activity)
        {
            // prevents all exporters from exporting internal activities
            if (activity.Kind == ActivityKind.Internal)
            {
                activity.IsAllDataRequested = false;
            }
        }
    }
    

If a particular source isn't explicitly added by using AddSource("ActivitySourceName"), then none of the activities created by using that source are exported.

Filter telemetry at ingestion using data collection rules

Reduce noise or standardize telemetry before Azure Monitor stores it in a Log Analytics workspace. To achieve that goal, use ingestion-time transformations in a data collection rule (DCR) to filter or modify telemetry after Azure Monitor receives it.

Transformations use a Kusto Query Language (KQL) statement that runs on each ingested record.

Use a workspace transformation DCR for Application Insights tables. A Log Analytics workspace supports one workspace transformation DCR. Put all transformations for that workspace in the same DCR.

Activation can take up to 60 minutes after an update.

Use these resources to learn more:

Map OpenTelemetry signals to Log Analytics tables

Application Insights stores common OpenTelemetry (OTel) signals in these tables:

  • AppTraces (logs)
  • AppRequests (incoming requests)
  • AppDependencies (outgoing dependencies)
  • AppExceptions (exceptions)
  • AppMetrics (metrics)

In a workspace transformation DCR, use the stream name format Microsoft-Table-<TableName> for each table. For example, use Microsoft-Table-AppRequests for the AppRequests table.

Note

A workspace transformation DCR applies to all data ingested into the selected tables in that Log Analytics workspace. If multiple applications share the same workspace, scope each transformation by using a filter such as AppRoleName or ResourceGUID.

Use workspace transformation DCR samples

The following samples show workspace transformation DCR JSON. Use dataFlows to define one transformation per table. Keep transformKql on one line in the DCR definition.

Create a workspace transformation DCR template

Use this template to create a workspace transformation DCR. Replace <workspace-location> and <workspace-resource-id> with values from your Log Analytics workspace.

{
  "kind": "WorkspaceTransforms",
  "location": "<workspace-location>",
  "properties": {
    "dataSources": {},
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "<workspace-resource-id>",
          "name": "laDest"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": ["Microsoft-Table-AppRequests"],
        "destinations": ["laDest"],
        "transformKql": "source"
      }
    ]
  }
}
Drop a telemetry type by dropping a table

Use this sample to block an entire telemetry type, such as all traces or all requests.

Add one data flow per table that you want to drop.

[
  {
    "streams": ["Microsoft-Table-AppTraces"],
    "destinations": ["laDest"],
    "transformKql": "source | where 1 == 0"
  },
  {
    "streams": ["Microsoft-Table-AppRequests"],
    "destinations": ["laDest"],
    "transformKql": "source | where 1 == 0"
  },
  {
    "streams": ["Microsoft-Table-AppDependencies"],
    "destinations": ["laDest"],
    "transformKql": "source | where 1 == 0"
  },
  {
    "streams": ["Microsoft-Table-AppExceptions"],
    "destinations": ["laDest"],
    "transformKql": "source | where 1 == 0"
  },
  {
    "streams": ["Microsoft-Table-AppMetrics"],
    "destinations": ["laDest"],
    "transformKql": "source | where 1 == 0"
  }
]
Drop health check requests

Use this sample to drop common health, readiness, and liveness endpoints.

{
  "streams": ["Microsoft-Table-AppRequests"],
  "destinations": ["laDest"],
  "transformKql": "source | extend url = tolower(tostring(Url)) | where not(url contains '/health' or url contains '/healthz' or url contains '/ready' or url contains '/readyz' or url contains '/live' or url contains '/livez') | project-away url"
}
Keep only failing health check requests

Use this sample to keep failing health checks and drop successful health checks.

{
  "streams": ["Microsoft-Table-AppRequests"],
  "destinations": ["laDest"],
  "transformKql": "source | extend url = tolower(tostring(Url)) | where not((url contains '/health' or url contains '/healthz' or url contains '/ready' or url contains '/readyz' or url contains '/live' or url contains '/livez') and Success == true) | project-away url"
}
Keep only failed or slow requests

Use this sample to keep requests that fail or exceed a latency threshold.

{
  "streams": ["Microsoft-Table-AppRequests"],
  "destinations": ["laDest"],
  "transformKql": "source | where Success == false or DurationMs >= 1000"
}
Keep only Warning and higher traces

Use this sample to keep trace records with SeverityLevel of Warning (2), Error (3), or Critical (4).

{
  "streams": ["Microsoft-Table-AppTraces"],
  "destinations": ["laDest"],
  "transformKql": "source | where SeverityLevel >= 2"
}
Keep only failed or slow dependencies

Use this sample to keep dependency calls that fail or exceed a latency threshold.

{
  "streams": ["Microsoft-Table-AppDependencies"],
  "destinations": ["laDest"],
  "transformKql": "source | where Success == false or DurationMs >= 500"
}
Remove SQL statements from dependency data

Use this sample to remove SQL statements stored in the Data column while keeping dependency timing and success information.

{
  "streams": ["Microsoft-Table-AppDependencies"],
  "destinations": ["laDest"],
  "transformKql": "source | extend dependencyType = tolower(tostring(DependencyType)) | extend Data = iff(dependencyType == 'sql', '', Data) | project-away dependencyType"
}
Drop synthetic traffic

Use this sample to drop records that include a SyntheticSource value.

{
  "streams": ["Microsoft-Table-AppRequests"],
  "destinations": ["laDest"],
  "transformKql": "source | where isempty(SyntheticSource)"
}
Drop cancellation exceptions

Use this sample to drop common cancellation exception types.

{
  "streams": ["Microsoft-Table-AppExceptions"],
  "destinations": ["laDest"],
  "transformKql": "source | where not(ExceptionType contains 'OperationCanceledException' or ExceptionType contains 'TaskCanceledException')"
}
Remove client IP addresses and query strings

Use this sample to remove client IP addresses and to store URLs without query strings.

[
  {
    "streams": ["Microsoft-Table-AppRequests"],
    "destinations": ["laDest"],
    "transformKql": "source | extend Url = tostring(split(tostring(Url), '?')[0]) | project-away ClientIP"
  },
  {
    "streams": ["Microsoft-Table-AppDependencies"],
    "destinations": ["laDest"],
    "transformKql": "source | extend Data = tostring(split(tostring(Data), '?')[0]) | project-away ClientIP"
  }
]
Scope a transformation to a single service

Use this sample to scope a transformation to a single service when multiple applications share a workspace.

{
  "streams": ["Microsoft-Table-AppRequests"],
  "destinations": ["laDest"],
  "transformKql": "source | where AppRoleName == '<app-role-name>' | where Success == false or DurationMs >= 1000"
}

Next steps