Migrate from self-hosted Prometheus to Azure Monitor managed service for Prometheus

This article provides guidance for organizations that are planning to migrate from self-managed Prometheus to Azure Monitor managed service for Prometheus. Prometheus is a widely adopted open-source monitoring solution known for its powerful capabilities in collecting, storing, and querying time-series data. You might start with self-managed Prometheus setups, but as your systems scale, the operational overhead of managing Prometheus environments can become significant. Azure Monitor managed service for Prometheus delivers the core benefits of Prometheus along with scalability and reduced maintenance efforts.

You can use Azure Monitor managed service for Prometheus to use Prometheus functionality while you benefit from Azure cloud-native, enterprise-grade capabilities. Key advantages are:

• A fully managed service:
  • Automatic upgrades and scaling.
  • Data retention for 18 months with no cost for storage.
  • Simple pricing based on ingestion and query.
• Monitoring and observability:
  • End-to-end, at-scale monitoring.
  • Out-of-the-box dashboards, alerts, and scrape configurations.
  • Native integration with key Azure Kubernetes Service (AKS) components, including Customer Control Plane.
  • Compliance with Azure Trust Center.
• Native integration with other Azure services, such as Azure Managed Grafana.

Use Azure Managed Prometheus in either of the following configurations:

• Fully managed service or drop-in replacement for self-managed Prometheus: In this case, a managed add-on within your AKS or Azure Arc-enabled Kubernetes cluster collects data. You can use custom resources (pod and service monitors) and the add-on ConfigMaps to configure data collection. The format of the pod/service monitors and ConfigMaps are the same as open-source Prometheus. In this way, you can use existing configs directly with Azure Managed Prometheus.
• Remote-write target: Use Prometheus remote write to send metrics from your existing Prometheus server running in Azure or non-Azure environments to send data to an Azure Monitor workspace. You can gradually migrate from self-hosted to the fully managed add-on.

Enabling the Azure Managed Prometheus add-on in an AKS cluster deploys the pod monitor and service monitor custom resource definitions (CRDs) so that you can create your own custom resources. Use pod monitors and service monitors to customize scraping targets, similar to the open-source software Prometheus Operator.

Prometheus metrics are stored in an Azure Monitor workspace, which is a unique environment for data that Azure Monitor collects. Each workspace has its own data repository, configuration, and permissions. Data is stored for 18 months.

Azure Managed Prometheus rule groups provide a managed and scalable way to create and update recording rules and alerts. The rule groups are following on Prometheus rules configuration. You can convert your existing recording rules and alerts to an Azure Managed Prometheus rule group. Prometheus alerts are integrated with other alerts in Azure Monitor.

Azure Managed Grafana is a data visualization platform built on top of the Grafana software by Grafana Labs. Azure operates and supports this fully managed Azure service. Whether you're using Azure Managed Grafana or self-hosted Grafana, you can query metrics from an Azure Monitor workspace. When you enable Managed Prometheus for your AKS or Azure Arc-enabled Kubernetes, we provide out-of-the-box dashboards that are the same as the ones used by open-source Prometheus Operator.

The following limitations apply to Azure Monitor managed service for Prometheus:

• The minimum frequency for scraping and storing metrics is one second. See Case sensitivity in Azure Managed Prometheus.
• Certain limitations apply to metric names, label names, and label values. See Metric names, label names, and label values.
• Certain limitations apply to the PromQL query API for Azure Managed Prometheus. See Query Prometheus metrics by using the API and PromQL.
• The PrometheusRule CRD isn't supported with Azure Managed Prometheus.

Before you begin the migration, review the following details of your current self-hosted Prometheus stack.

An Azure Monitor workspace is highly scalable and can support a large volume of metrics ingestion. You can increase the default limits as your scale requires it.

• For the managed add-on, the data volume of metrics depends on the size of the AKS cluster and how many workloads you plan to run. You can enable Azure Managed Prometheus on a few clusters to estimate the metrics volume.
• If you plan to use remote write before you fully migrate to the managed add-on agent, you can determine the metrics ingestion volume based on historical usage. You can also inspect the metric prometheus_remote_storage_samples_in_total to evaluate the metrics volume being sent through remote write.

The Prometheus version is required if you're using remote write to send data to an Azure Monitor workspace. See Supported versions.

Pricing is based on metrics ingestion and query volume. For more information on metrics pricing, see Azure Monitor pricing. You can use the Azure pricing calculator to estimate the cost.

Review the following configurations for your self-hosted Prometheus setup. This assessment helps to identify any customizations that require attention during migration.

• Alerting and recording rules configuration
• Active data sources and exporters
• Dashboards

There are two methods to configure Azure Managed Prometheus, as described on the following tabs.

There are two methods to configure metrics collection, as described on the following tabs.

Azure Managed Prometheus supports Prometheus alerting rules and recording rules with Prometheus rule groups. See Convert your existing rules to a Prometheus rule group Azure Resource Manager template.

With the managed add-on, recommended recording rules are automatically set up as you enable Managed Prometheus for your AKS or Azure Arc-enabled cluster. To review the list of automatically provisioned recording rules, see Default Prometheus metrics configuration in Azure Monitor. Prometheus community recommended alerts are also available, and you can create them out-of-the-box.

If you're using Grafana, connect Grafana to Azure Monitor Prometheus metrics. You can reuse existing dashboards by importing them to Grafana. If you're using the Azure Managed Grafana or Azure Monitor dashboards with Grafana, the default or recommended dashboards are automatically set up and provisioned so that you can visualize the metrics. To review the list of automatically provisioned dashboards, see Default Prometheus metrics configuration in Azure Monitor.

After your migration is finished, validate that your setup is working as expected:

• Verify that you can query metrics from an Azure Monitor workspace. You can query the metrics directly from the Metrics option for the workspace instance in the Azure portal or with a Grafana instance connected to the workspace.
• Verify that you can access the Prometheus interface for Azure Managed Prometheus to verify jobs and confirm that targets are scraped. See Access Prometheus interface for Azure Managed Prometheus.
• Verify that any alerting workflows trigger as expected.
• Verify that remote write is working as expected. See Verify remote-write deployment.
• For more troubleshooting guidance, see Troubleshoot collection of Prometheus metrics in Azure Monitor.

Azure Monitor workspaces have default limits and quotas for ingestion. You might experience throttling as you onboard more clusters and reach the ingestion limits. Monitor and alert on the workspace ingestion limits to ensure that you don't reach throttling limits.

• Review best practices for scaling an Azure Monitor workspace.
• Enable Managed Prometheus for AKS/Azure Arc-enabled clusters.