Log-based and preaggregated metrics in Application Insights
Note
The following documentation relies on the Application Insights classic API. The long-term plan for Application Insights is to collect data using OpenTelemetry. For more information, see Enable Azure Monitor OpenTelemetry for .NET, Node.js, Python and Java applications and our OpenTelemetry Roadmap. Migration guidance is available for .NET, Node.js, and Python.
This article explains the difference between "traditional" Application Insights metrics that are based on logs and preaggregated metrics. Both types of metrics are available to users of Application Insights. Each one brings a unique value in monitoring application health, diagnostics, and analytics. The developers who are instrumenting applications can decide which type of metric is best suited to a particular scenario. Decisions are based on the size of the application, expected volume of telemetry, and business requirements for metrics precision and alerting.
Log-based metrics
In the past, the application monitoring telemetry data model in Application Insights was solely based on a few predefined types of events, such as requests, exceptions, dependency calls, and page views. Developers can use the SDK to emit these events manually by writing code that explicitly invokes the SDK. Or they can rely on the automatic collection of events from autoinstrumentation. In either case, the Application Insights back end stores all collected events as logs. The Application Insights panes in the Azure portal act as an analytical and diagnostic tool for visualizing event-based data from logs.
Using logs to retain a complete set of events can bring great analytical and diagnostic value. For example, you can get an exact count of requests to a particular URL with the number of distinct users who made these calls. Or you can get detailed diagnostic traces, including exceptions and dependency calls for any user session. Having this type of information can improve visibility into the application health and usage. It can also cut down the time necessary to diagnose issues with an app.
At the same time, collecting a complete set of events might be impractical or even impossible for applications that generate a large volume of telemetry. For situations when the volume of events is too high, Application Insights implements several telemetry volume reduction techniques that reduce the number of collected and stored events. These techniques include sampling and filtering. Unfortunately, lowering the number of stored events also lowers the accuracy of the metrics that, behind the scenes, must perform query-time aggregations of the events stored in logs.
Note
In Application Insights, the metrics that are based on the query-time aggregation of events and measurements stored in logs are called log-based metrics. These metrics typically have many dimensions from the event properties, which makes them superior for analytics. The accuracy of these metrics is negatively affected by sampling and filtering.
Preaggregated metrics
In addition to log-based metrics, in late 2018, the Application Insights team shipped a public preview of metrics that are stored in a specialized repository that's optimized for time series. The new metrics are no longer kept as individual events with lots of properties. Instead, they're stored as preaggregated time series, and only with key dimensions. This change makes the new metrics superior at query time. Retrieving data happens faster and requires less compute power. As a result, new scenarios are enabled, such as near real time alerting on dimensions of metrics and more responsive dashboards.
Important
Both log-based and preaggregated metrics coexist in Application Insights. To differentiate the two, in the Application Insights user experience the preaggregated metrics are now called standard metrics. The traditional metrics from the events were renamed to log-based metrics.
The newer SDKs (Application Insights 2.7 SDK or later for .NET) preaggregate metrics during collection. This process applies to standard metrics sent by default, so the accuracy isn't affected by sampling or filtering. It also applies to custom metrics sent by using GetMetric, which results in less data ingestion and lower cost.
For the SDKs that don't implement preaggregation (that is, older versions of Application Insights SDKs or for browser instrumentation), the Application Insights back end still populates the new metrics by aggregating the events received by the Application Insights event collection endpoint. Although you don't benefit from the reduced volume of data transmitted over the wire, you can still use the preaggregated metrics and experience better performance and support of the near real time dimensional alerting with SDKs that don't preaggregate metrics during collection.
The collection endpoint preaggregates events before ingestion sampling. For this reason, ingestion sampling never affects the accuracy of preaggregated metrics, regardless of the SDK version you use with your application.
SDK supported preaggregated metrics table
| Current production SDKs | Standard metrics (SDK preaggregation) | Custom metrics (without SDK preaggregation) | Custom metrics (with SDK preaggregation) |
|---|---|---|---|
| .NET Core and .NET Framework | Supported (V2.13.1+) | Supported via TrackMetric | Supported (V2.7.2+) via GetMetric |
| Java | Not supported | Supported via TrackMetric | Not supported |
| Node.js | Supported (V2.0.0+) | Supported via TrackMetric | Not supported |
| Python | Not supported | Supported | Partially supported via OpenCensus.stats |
Note
The metrics implementation for Python by using OpenCensus.stats is different from GetMetric. For more information, see the Python documentation on metrics.
Codeless supported preaggregated metrics table
| Current production SDKs | Standard metrics (SDK preaggregation) | Custom metrics (without SDK preaggregation) | Custom metrics (with SDK preaggregation) |
|---|---|---|---|
| ASP.NET | Supported 1 | Not supported | Not supported |
| ASP.NET Core | Supported 2 | Not supported | Not supported |
| Java | Not supported | Not supported | Supported |
| Node.js | Not supported | Not supported | Not supported |
- ASP.NET autoinstrumentation on virtual machines/virtual machine scale sets and on-premises emits standard metrics without dimensions. The same is true for Azure App Service, but the collection level must be set to recommended. The SDK is required for all dimensions.
- ASP.NET Core autoinstrumentation on App Service emits standard metrics without dimensions. SDK is required for all dimensions.
Use preaggregation with Application Insights custom metrics
You can use preaggregation with custom metrics. The two main benefits are:
- Configure and alert on a dimension of a custom metric
- Reduce the volume of data sent from the SDK to the Application Insights collection endpoint
There are several ways of sending custom metrics from the Application Insights SDK. If your version of the SDK offers GetMetric and TrackValue, these methods are the preferred way of sending custom metrics. In this case, preaggregation happens inside the SDK. This approach reduces the volume of data stored in Azure and also the volume of data transmitted from the SDK to Application Insights. Otherwise, use the trackMetric method, which preaggregates metric events during data ingestion.
Custom metrics dimensions and preaggregation
All metrics that you send using OpenTelemetry, trackMetric, or GetMetric and TrackValue API calls are automatically stored in both logs and metrics stores. These metrics can be found in the customMetrics table in Application Insights and in Metrics Explorer under the Custom Metric Namespace called "azure.applicationinsights". Although the log-based version of your custom metric always retains all dimensions, the preaggregated version of the metric is stored by default with no dimensions. Retaining dimensions of custom metrics is a Preview feature that can be turned on from the Usage and estimated cost tab by selecting With dimensions under Send custom metrics to Azure Metric Store.
Why is collection of custom metrics dimensions turned off by default?
The collection of custom metrics dimensions is turned off by default because in the future storing custom metrics with dimensions will be billed separately from Application Insights. Storing the nondimensional custom metrics remain free (up to a quota). You can learn about the upcoming pricing model changes on our official pricing page.
Create charts and explore log-based and standard preaggregated metrics
Use Azure Monitor metrics explorer to plot charts from preaggregated and log-based metrics and to author dashboards with charts. After you select the Application Insights resource you want, use the namespace picker to switch between standard and log-based metrics. You can also select a custom metric namespace.
Pricing models for Application Insights metrics
Ingesting metrics into Application Insights, whether log-based or preaggregated, generates costs based on the size of the ingested data. For more information, see Azure Monitor Logs pricing details. Your custom metrics, including all its dimensions, are always stored in the Application Insights log store. Also, a preaggregated version of your custom metrics with no dimensions is forwarded to the metrics store by default.
Selecting the Enable alerting on custom metric dimensions option to store all dimensions of the preaggregated metrics in the metric store can generate extra costs based on custom metrics pricing.