Setup of Network Observability for Azure Kubernetes Service (AKS) - BYO Prometheus and Grafana

  • Article

AKS Network Observability is used to collect the network traffic data of your AKS cluster. Network Observability enables a centralized platform for monitoring application and network health. Prometheus collects AKS Network Observability metrics, and Grafana visualizes them. Both Cilium and non-Cilium data plane are supported. In this article, learn how to enable the Network Observability add-on and use BYO Prometheus and Grafana to visualize the scraped metrics.

Note

Starting with Kubernetes version 1.29, the network observability feature no longer supports Bring Your Own (BYO) Prometheus and Grafana. However, you can still enable it using the Azure Managed Prometheus and Grafana offering

Important

AKS Network Observability is currently in PREVIEW. See the Supplemental Terms of Use for Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

For more information about AKS Network Observability, see What is Azure Kubernetes Service (AKS) Network Observability?.

Prerequisites

  • An Azure account with an active subscription. Create an account.

  • Installations of BYO Prometheus and Grafana.

  • If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

    • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.

    • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.

    • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

  • Minimum version of Azure CLI required for the steps in this article is 2.44.0. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.

Install the aks-preview Azure CLI extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

# Install the aks-preview extension
az extension add --name aks-preview

# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview

Register the NetworkObservabilityPreview feature flag

az feature register --namespace "Microsoft.ContainerService" --name "NetworkObservabilityPreview"

Use az feature show to check the registration status of the feature flag:

az feature show --namespace "Microsoft.ContainerService" --name "NetworkObservabilityPreview"

Wait for the feature to say Registered before preceding with the article.

{
  "id": "/subscriptions/23250d6d-28f0-41dd-9776-61fc80805b6e/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/NetworkObservabilityPreview",
  "name": "Microsoft.ContainerService/NetworkObservabilityPreview",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}

When the feature is registered, refresh the registration of the Microsoft.ContainerService resource provider with az provider register:

az provider register -n Microsoft.ContainerService

Create a resource group

A resource group is a logical container into which Azure resources are deployed and managed. Create a resource group with az group create command. The following example creates a resource group named myResourceGroup in the chinanorth location:

az group create \
    --name myResourceGroup \
    --location chinanorth

Create AKS cluster

Create an AKS cluster with az aks create command. The following example creates an AKS cluster named myAKSCluster in the myResourceGroup resource group:

Non-Cilium clusters support the enablement of Network Observability on an existing cluster or during the creation of a new cluster.

New cluster

Use az aks create in the following example to create an AKS cluster with Network Observability and non-Cilium.

az aks create \
    --name myAKSCluster \
    --resource-group myResourceGroup \
    --location chinanorth \
    --generate-ssh-keys \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --pod-cidr 192.168.0.0/16 \
    --enable-network-observability

Existing cluster

Use az aks update to enable Network Observability on an existing cluster.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --enable-network-observability

Get cluster credentials

az aks get-credentials --name myAKSCluster --resource-group myResourceGroup

Enable Visualization on Grafana

Use the following example to configure scrape jobs on Prometheus and enable visualization on Grafana for your AKS cluster.

Note

The following section requires installations of Prometheus and Grafana.

  1. Add the following scrape job to your existing Prometheus configuration and restart your Prometheus server:

    scrape_configs:
  - job_name: "network-obs-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep
        regex: kappie(.*)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        separator: ":"
        regex: ([^:]+)(?::\d+)?
        target_label: __address__
        replacement: ${1}:${2}
        action: replace
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: instance
    metric_relabel_configs:
      - source_labels: [__name__]
        action: keep
        regex: (.*)

  2. In Targets of Prometheus, verify the network-obs-pods are present.

  3. Sign in to Grafana and import Network Observability dashboard with ID 18814.

Clean up resources

If you're not going to continue to use this application, delete the AKS cluster and the other resources created in this article with the following example:

  az group delete \
    --name myResourceGroup

Next steps

In this how-to article, you learned how to install and enable AKS Network Observability for your AKS cluster.