Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Use this guide to filter OpenTelemetry (OTel) data in Azure Monitor Application Insights. Filtering helps you exclude unnecessary telemetry and prevent collection of sensitive data to optimize performance and support compliance.
Reasons to filter out telemetry include:
- Filtering out health check telemetry to reduce noise.
- Ensuring personal data and credentials aren't collected.
- Filtering out low-value telemetry to optimize performance.
To learn more about OpenTelemetry concepts, review the OpenTelemetry overview or OpenTelemetry FAQ.
Note
For Azure Function Apps, see Use OpenTelemetry with Azure Functions.
Filter OpenTelemetry using instrumentation libraries
For a list of all instrumentation libraries included with the Azure Monitor OpenTelemetry Distro, review Add and modify Azure Monitor OpenTelemetry for .NET, Java, Node.js, and Python applications.
Many instrumentation libraries provide a filter option. For guidance, review the corresponding readme files:
Note
Currently, the Azure Monitor OpenTelemetry Distro includes a copy of the SqlClient instrumentation source code to ensure stability while the upstream OpenTelemetry SqlClient library remains experimental.
Options such as SetDbStatementForStoredProcedure aren't usable in our distribution because the code is embedded and not referencing the external package.
Once the SqlClient instrumentation reaches a stable release, Azure Monitor switches to referencing the official package and customization via builder.AddSqlClientInstrumentation(options => { ... }) is available.
Filter telemetry using span processors
Use a custom processor:
Tip
Add the processor shown here before adding Azure Monitor.
// Create an ASP.NET Core application builder. var builder = WebApplication.CreateBuilder(args); // Configure the OpenTelemetry tracer provider to add a new processor named ActivityFilteringProcessor. builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddProcessor(new ActivityFilteringProcessor())); // Configure the OpenTelemetry tracer provider to add a new source named "ActivitySourceName". builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) => builder.AddSource("ActivitySourceName")); // Add the Azure Monitor telemetry service to the application. This service will collect and send telemetry data to Azure Monitor. builder.Services.AddOpenTelemetry().UseAzureMonitor(); // Build the ASP.NET Core application. var app = builder.Build(); // Start the ASP.NET Core application. app.Run();Add
ActivityFilteringProcessor.csto your project with the following code:public class ActivityFilteringProcessor : BaseProcessor<Activity> { // The OnStart method is called when an activity is started. This is the ideal place to filter activities. public override void OnStart(Activity activity) { // prevents all exporters from exporting internal activities if (activity.Kind == ActivityKind.Internal) { activity.IsAllDataRequested = false; } } }
If a particular source isn't explicitly added by using AddSource("ActivitySourceName"), then none of the activities created by using that source are exported.
Filter telemetry at ingestion using data collection rules
Reduce noise or standardize telemetry before Azure Monitor stores it in a Log Analytics workspace. To achieve that goal, use ingestion-time transformations in a data collection rule (DCR) to filter or modify telemetry after Azure Monitor receives it.
Transformations use a Kusto Query Language (KQL) statement that runs on each ingested record.
Use a workspace transformation DCR for Application Insights tables. A Log Analytics workspace supports one workspace transformation DCR. Put all transformations for that workspace in the same DCR.
Activation can take up to 60 minutes after an update.
Use these resources to learn more:
- Review data collection rules (DCRs).
- Review transformations in Azure Monitor.
- Review the workspace transformation tutorial for the Azure portal.
- Review the workspace transformation tutorial for Resource Manager templates.
- Review Create a transformation in Azure Monitor.
- Review supported KQL features in transformations.
- Review the Application Insights tables that support ingestion-time transformations.
Map OpenTelemetry signals to Log Analytics tables
Application Insights stores common OpenTelemetry (OTel) signals in these tables:
AppTraces(logs)AppRequests(incoming requests)AppDependencies(outgoing dependencies)AppExceptions(exceptions)AppMetrics(metrics)
In a workspace transformation DCR, use the stream name format Microsoft-Table-<TableName> for each table. For example, use Microsoft-Table-AppRequests for the AppRequests table.
Note
A workspace transformation DCR applies to all data ingested into the selected tables in that Log Analytics workspace. If multiple applications share the same workspace, scope each transformation by using a filter such as AppRoleName or ResourceGUID.
Use workspace transformation DCR samples
The following samples show workspace transformation DCR JSON. Use dataFlows to define one transformation per table. Keep transformKql on one line in the DCR definition.
Create a workspace transformation DCR template
Use this template to create a workspace transformation DCR. Replace <workspace-location> and <workspace-resource-id> with values from your Log Analytics workspace.
{
"kind": "WorkspaceTransforms",
"location": "<workspace-location>",
"properties": {
"dataSources": {},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "<workspace-resource-id>",
"name": "laDest"
}
]
},
"dataFlows": [
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source"
}
]
}
}
Drop a telemetry type by dropping a table
Use this sample to block an entire telemetry type, such as all traces or all requests.
Add one data flow per table that you want to drop.
[
{
"streams": ["Microsoft-Table-AppTraces"],
"destinations": ["laDest"],
"transformKql": "source | where 1 == 0"
},
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | where 1 == 0"
},
{
"streams": ["Microsoft-Table-AppDependencies"],
"destinations": ["laDest"],
"transformKql": "source | where 1 == 0"
},
{
"streams": ["Microsoft-Table-AppExceptions"],
"destinations": ["laDest"],
"transformKql": "source | where 1 == 0"
},
{
"streams": ["Microsoft-Table-AppMetrics"],
"destinations": ["laDest"],
"transformKql": "source | where 1 == 0"
}
]
Drop health check requests
Use this sample to drop common health, readiness, and liveness endpoints.
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | extend url = tolower(tostring(Url)) | where not(url contains '/health' or url contains '/healthz' or url contains '/ready' or url contains '/readyz' or url contains '/live' or url contains '/livez') | project-away url"
}
Keep only failing health check requests
Use this sample to keep failing health checks and drop successful health checks.
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | extend url = tolower(tostring(Url)) | where not((url contains '/health' or url contains '/healthz' or url contains '/ready' or url contains '/readyz' or url contains '/live' or url contains '/livez') and Success == true) | project-away url"
}
Keep only failed or slow requests
Use this sample to keep requests that fail or exceed a latency threshold.
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | where Success == false or DurationMs >= 1000"
}
Keep only Warning and higher traces
Use this sample to keep trace records with SeverityLevel of Warning (2), Error (3), or Critical (4).
{
"streams": ["Microsoft-Table-AppTraces"],
"destinations": ["laDest"],
"transformKql": "source | where SeverityLevel >= 2"
}
Keep only failed or slow dependencies
Use this sample to keep dependency calls that fail or exceed a latency threshold.
{
"streams": ["Microsoft-Table-AppDependencies"],
"destinations": ["laDest"],
"transformKql": "source | where Success == false or DurationMs >= 500"
}
Remove SQL statements from dependency data
Use this sample to remove SQL statements stored in the Data column while keeping dependency timing and success information.
{
"streams": ["Microsoft-Table-AppDependencies"],
"destinations": ["laDest"],
"transformKql": "source | extend dependencyType = tolower(tostring(DependencyType)) | extend Data = iff(dependencyType == 'sql', '', Data) | project-away dependencyType"
}
Drop synthetic traffic
Use this sample to drop records that include a SyntheticSource value.
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | where isempty(SyntheticSource)"
}
Drop cancellation exceptions
Use this sample to drop common cancellation exception types.
{
"streams": ["Microsoft-Table-AppExceptions"],
"destinations": ["laDest"],
"transformKql": "source | where not(ExceptionType contains 'OperationCanceledException' or ExceptionType contains 'TaskCanceledException')"
}
Remove client IP addresses and query strings
Use this sample to remove client IP addresses and to store URLs without query strings.
[
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | extend Url = tostring(split(tostring(Url), '?')[0]) | project-away ClientIP"
},
{
"streams": ["Microsoft-Table-AppDependencies"],
"destinations": ["laDest"],
"transformKql": "source | extend Data = tostring(split(tostring(Data), '?')[0]) | project-away ClientIP"
}
]
Scope a transformation to a single service
Use this sample to scope a transformation to a single service when multiple applications share a workspace.
{
"streams": ["Microsoft-Table-AppRequests"],
"destinations": ["laDest"],
"transformKql": "source | where AppRoleName == '<app-role-name>' | where Success == false or DurationMs >= 1000"
}
Next steps
- To further configure the OpenTelemetry distro, review Azure Monitor OpenTelemetry configuration.
- To review the source code, review the Azure Monitor AspNetCore GitHub repository.
- To install the NuGet package, check for updates, or view release notes, use the Azure Monitor AspNetCore NuGet Package page.
- To become more familiar with Azure Monitor and OpenTelemetry, review the Azure Monitor Example Application.
- To learn more about OpenTelemetry and its community, review the OpenTelemetry .NET GitHub repository.
- To enable usage experiences, enable web or browser user monitoring.
- To review frequently asked questions, troubleshooting steps, support options, or to provide OpenTelemetry feedback, review OpenTelemetry help, support, and feedback for Azure Monitor Application Insights.