Azure Monitor and Prometheus

Article
11/29/2024

Prometheus is a popular open-source monitoring and alerting solution that's widely used in the cloud-native ecosystem. Prometheus is used to monitor and alert on the performance of infrastructure and workloads and often used in Kubernetes environments.

Use Prometheus as an Azure managed service, or as a self managed service to collect metrics. Prometheus metrics can be collected from your Azure Kubernetes Service (AKS) clusters, Azure Arc-enabled Kubernetes clusters, virtual machines, and virtual machine scale sets.

Prometheus metrics are stored in an Azure Monitor workspace, where you can analyze and visualize the data using Metrics Explorer with PromQL and Azure Managed Grafana.

Important

Azure Monitor managed and hosted Prometheus is intended for storing information about service health of customer machines and applications. It is not intended for storing any data classified as Personal Identifiable Information (PII) or End User Identifiable Information (EUII). We strongly recommend that you do not send any sensitive information (usernames, credit card numbers etc.) into Azure Monitor hosted Prometheus fields like metric names, label names, or label values.

Azure Monitor Managed Service for Prometheus

Azure Monitor Managed Service for Prometheus is a component of Azure Monitor Metrics that provides a fully managed and scalable environment for running Prometheus. It simplifies the deployment, management, and scaling of Prometheus in an Azure Kubernetes Service, allowing you to focus on monitoring your applications and infrastructure.

As a fully managed service, Azure Monitor managed service for Prometheus automatically deploys Prometheus in AKS or ARC-enabled Kubernetes. The service provides high availability, SLA guarantees, and automatic software updates. The service provides a highly scalable metrics store, with data retention of up to 18 months.

Azure Monitor managed service for Prometheus provides preconfigured alerts, rules, and dashboards. With recommended dashboards from the Prometheus community and native Grafana integration, you can achieve comprehensive monitoring immediately. Natively integrates with Azure Managed Grafana, and also works with self-managed Grafana.

Pricing is based on ingestion and query with no additional storage cost. For more information, see the Metrics tab in Azure Monitor pricing.

Enable Azure Monitor managed service for Prometheus

Azure Monitor managed service for Prometheus collects data from Azure Kubernetes services:

Azure Kubernetes service (AKS)
Azure Arc-enabled Kubernetes

To enable Azure Monitor managed service for Prometheus, you must create an Azure Monitor workspace to store the metrics. Once the workspace is created, you can onboard services that collect Prometheus metrics.

To collect Prometheus metrics from your Kubernetes cluster, see Enable monitoring for Kubernetes clusters.
To configure remote-write to collect data from a self-managed Prometheus server, see Azure Monitor managed service for Prometheus remote write.

Azure hosted self-managed Prometheus

In addition to the managed service for Prometheus, you can install and manage your own Prometheus instance and use remote-write to store metrics in an Azure Monitor workspace.

Using remote-write, you can collect data from self-managed Prometheus servers running in the following environments:

Azure virtual machines
Azure virtual machine scale sets
Arc-enabled servers
Self manged Azure-hosted or arc-enabled Kubernetes clusters.

Self-managed Kubernetes services

Send metrics from self-managed Prometheus on Kubernetes clusters. For more information on remote-write to Azure Monitor workspaces for Kubernetes services, see the following articles:

Virtual Machines and Virtual Machine Scale sets

Send data from self-managed Prometheus on virtual machines and virtual machine scale sets. The virtual machines can be in an Azure-managed environment or on-premises. Fro more information, see Send Prometheus metrics from Virtual Machines to an Azure Monitor workspace.

Data storage

Prometheus metrics are stored in an Azure Monitor workspace. The data is stored in a time-series database that can be queried using Prometheus Query Language (PromQL). You can store data from several Prometheus data sources in a single Azure Monitor workspace. For more information, see Azure Monitor workspace overview.

Data is retained in Azure Monitor workspaces for 18 months.

Query and analyze Prometheus metrics

Prometheus data is retrieved using Prometheus Query Language (PromQL). You can write your own queries, use queries from the open source community, and use Grafana dashboards that include PromQL queries. See the Prometheus project.

The following Azure services support querying Prometheus metrics from an Azure Monitor workspace:

Azure Monitor Metrics explorer with PromQL

Metrics Explorer with PromQL allows you to analyze and visualize platform and Prometheus metrics. Metrics explorer supports PromQL for Prometheus metrics. Metrics Explorer with PromQL (preview) is available from the Metrics menu item of the Azure Monitor workspace where your Prometheus metrics are stored. For more information, see Metrics Explorer with PromQL.

Azure workbooks

Create charts and dashboards powered by Azure Monitor managed service for Prometheus using Azure Workbooks and PromQL queries. For more information, see Query Prometheus metrics using Azure workbooks

Grafana integration

VisualizE Prometheus metrics using Azure Managed Grafana. Connect your Azure Monitor workspace to a Grafana workspace so that it can be used as a data source in a Grafana dashboard. You then have access to multiple prebuilt dashboards that use Prometheus metrics and the ability to create any number of custom dashboards. For more information, see Link a Grafana workspace to an Azure Monitor workspace

Prometheus query API

Use PromQL via the REST API to query Prometheus metrics stored in an Azure Monitor workspace. For more information, see Prometheus query API.

Rules and alerts

Prometheus supports recording rules and alert rules using PromQL queries. Rules and alerts are automatically deployed Azure Monitor managed service for Prometheus. Metrics recorded by recording rules are stored in the Azure Monitor workspace and can be queried by dashboards or by other rules. For your AKS cluster, a set of recording rules is provided to allow easy quick start.

Alerts fired by alert rules can trigger actions or notifications, as defined in the action groups configured for the alert rule. You can also view fired and resolved Prometheus alerts in the Azure portal along with other alert types.

Service limits and quotas

Azure Monitor Managed service for Prometheus has default limits and quotas for ingestion. When you reach the ingestion limits throttling can occur. You can request an increase in these limits. For information on Prometheus metrics limits, see Azure Monitor service limits.

To monitor and alert on your ingestion metrics, see Monitor Azure Monitor workspace metrics ingestion.

Limitations/Known issues - Azure Monitor managed Service for Prometheus

The minimum frequency for scraping and storing metrics is 1 second.
During node updates, you might experience gaps lasting 1 to 2 minutes in some metric collections from our cluster level collector. This gap is due to a regular action from Azure Kubernetes Service to update the nodes in your cluster. This behavior is expected and occurs due to the node it runs on being updated. Recommended alert rules are not affected by this behavior.
Managed Prometheus for Windows nodes isn't automatically enabled. To enable monitoring for Windows nodes & pods in your clusters, see Monitor Windows nodes & pods in your clusters.

Case sensitivity

Azure managed Prometheus is a case insensitive system. It treats strings, such as metric names, label names, or label values, as the same time series if they differ from another time series only by the case of the string.

Note

This behavior is different from native open source Prometheus, which is a case sensitive system.
Self-managed Prometheus instances running in Azure VMs, VMSSs, or Azure Kubernetes Service (AKS) clusters are case sensitive systems.

In Azure managed Prometheus the following time series are considered the same:

diskSize(cluster="chinanorth3", node="node1", filesystem="usr_mnt")
diskSize(cluster="chinanorth3", node="node1", filesystem="usr_MNT")

The above examples are a single time series in a time series database.

Any samples ingested against them are stored as if they're scraped/ingested against a single time series.
If the preceding examples are ingested with the same timestamp, one of them is randomly dropped.
The casing that's stored in the time series database and returned by a query is unpredictable. Different casing may be returned at different times for the same time series.
Any metric name or label name/value matcher present in the query is retrieved from time series database by making a case-insensitive comparison. If there's a case sensitive matcher in a query, it's automatically treated as a case-insensitive matcher when making string comparisons.

It's best practice to ensure that a time series is produced or scraped using a single consistent case.

In open source Prometheus, the above time series are treated as two different time series. Any samples scraped/ingested against them are stored separately.

Prometheus references

Following are links to Prometheus documentation.