Filter container log collection with ConfigMap

Kubernetes clusters generate a large amount of data that's collected by Azure Monitor. Since you're charged for the ingestion and retention of this data, you can significantly reduce your monitoring costs by filtering out data that you don't need.

ConfigMaps are a Kubernetes mechanism that allows you to store non-confidential data such as a configuration file or environment variables. Container insights looks for a ConfigMap on each cluster with particular settings that define data that it should collect.

Prerequisites

• The minimum agent version supported to collect stdout, stderr, and environmental variables from container workloads is ciprod06142019 or later.

Configure and deploy ConfigMap

Use the following procedure to configure and deploy your ConfigMap configuration file to your cluster:

1. If you don't already have a ConfigMap for Container insights, download the template ConfigMap YAML file and open it in an editor.

2. Edit the ConfigMap YAML file with your customizations. The template includes all valid settings with descriptions. To enable a setting, remove the comment character (#) and set its value.

3. Create a ConfigMap by running the following kubectl command:

   kubectl config set-context <cluster-name>
   kubectl apply -f <configmap_yaml_file.yaml>
   
   # Example: 
   kubectl config set-context my-cluster
   kubectl apply -f container-azm-ms-agentconfig.yaml
   

   The configuration change can take a few minutes to finish before taking effect. Then all Azure Monitor Agent pods in the cluster will restart. The restart is a rolling restart for all Azure Monitor Agent pods, so not all of them restart at the same time. When the restarts are finished, you'll receive a message similar to the following result:

   configmap "container-azm-ms-agentconfig" created`.
   

Verify configuration

To verify the configuration was successfully applied to a cluster, use the following command to review the logs from an agent pod.

kubectl logs ama-logs-fdf58 -n kube-system -c ama-logs

If there are configuration errors from the Azure Monitor Agent pods, the output will show errors similar to the following:

***************Start Config Processing******************** 
config::unsupported/missing config schema version - 'v21' , using defaults

Use the following options to perform more troubleshooting of configuration changes:

• Use the same kubectl logs command from an agent pod.

• Review live logs for errors similar to the following:

  config::error::Exception while parsing config map for log collection/env variable settings: \nparse error on value \"$\" ($end), using defaults, please check config map for errors
  

Data is sent to the KubeMonAgentEvents table in your Log Analytics workspace every hour with error severity for configuration errors. If there are no errors, the entry in the table will have data with severity info, which reports no errors. The Tags column contains more information about the pod and container ID on which the error occurred and also the first occurrence, last occurrence, and count in the last hour.

Verify schema version

Supported config schema versions are available as pod annotation (schema-versions) on the Azure Monitor Agent pod. You can see them with the following kubectl command.

kubectl describe pod ama-logs-fdf58 -n=kube-system.

Filter container logs

Container logs are stderr and stdout logs generated by containers in your Kubernetes cluster. These logs are stored in the ContainerLogV2 table in your Log Analytics workspace. By default all container logs are collected, but you can filter out logs from specific namespaces or disable collection of container logs entirely.

Using ConfigMap, you can configure the collection of stderr and stdout logs separately for the cluster, so you can choose to enable one and not the other. The following example shows the ConfigMap settings to collect stdout and stderr excluding the kube-system and gatekeeper-system namespaces.

[log_collection_settings]
    [log_collection_settings.stdout]
        enabled = true
        exclude_namespaces = ["kube-system","gatekeeper-system"]

    [log_collection_settings.stderr]
        enabled = true
        exclude_namespaces = ["kube-system","gatekeeper-system"]

    [log_collection_settings.enrich_container_logs]
        enabled = true

Platform log filtering (System Kubernetes namespaces)

By default, container logs from the system namespace are excluded from collection to minimize the Log Analytics cost. Container logs of system containers can be critical though in specific troubleshooting scenarios. This feature is restricted to the following system namespaces: kube-system, gatekeeper-system, calico-system, azure-arc, kube-public, and kube-node-lease.

Enable platform logs using ConfigMap with the collect_system_pod_logs setting. You must also ensure that the system namespace is not in the exclude_namespaces setting.

The following example shows the ConfigMap settings to collect stdout and stderr logs of coredns container in the kube-system namespace.

[log_collection_settings]
    [log_collection_settings.stdout]
        enabled = true
        exclude_namespaces = ["gatekeeper-system"]
        collect_system_pod_logs = ["kube-system:coredns"]

    [log_collection_settings.stderr]
        enabled = true
        exclude_namespaces = ["kube-system","gatekeeper-system"]
        collect_system_pod_logs = ["kube-system:coredns"]

Annotation based filtering for workloads

Annotation-based filtering enables you to exclude log collection for certain pods and containers by annotating the pod. This can reduce your logs ingestion cost significantly and allow you to focus on relevant information without sifting through noise.

Enable annotation based filtering using ConfigMap with the following settings.

[log_collection_settings.filter_using_annotations]
   enabled = true

You must also add the required annotations on your workload pod spec. The following table highlights different possible pod annotations.

Following is an example of fluentbit.io/exclude: "true" annotation in a Pod spec:

apiVersion: v1 
kind: Pod 
metadata: 
 name: apache-logs 
 labels: 
  app: apache-logs 
 annotations: 
  fluentbit.io/exclude: "true" 
spec: 
 containers: 
 - name: apache 
  image: edsiper/apache_logs 

Filter environment variables

Enable collection of environment variables across all pods and nodes in the cluster using ConfigMap with the following settings.

[log_collection_settings.env_var]
    enabled = true

If collection of environment variables is globally enabled, you can disable it for a specific container by setting the environment variable AZMON_COLLECT_ENV to False either with a Dockerfile setting or in the configuration file for the Pod under the env: section. If collection of environment variables is globally disabled, you can't enable collection for a specific container. The only override that can be applied at the container level is to disable collection when it's already enabled globally.

ConfigMap settings

The following table describes the settings you can configure to control data collection with ConfigMap.

Impact on visualizations and alerts

If you have any custom alerts or workbooks using Container insights data, then modifying your data collection settings might degrade those experiences. If you're excluding namespaces or reducing data collection frequency, review your existing alerts, dashboards, and workbooks using this data.

To scan for alerts that reference these tables, run the following Azure Resource Graph query:

resources
| where type in~ ('microsoft.insights/scheduledqueryrules') and ['kind'] !in~ ('LogToMetric')
| extend severity = strcat("Sev", properties["severity"])
| extend enabled = tobool(properties["enabled"])
| where enabled in~ ('true')
| where tolower(properties["targetResourceTypes"]) matches regex 'microsoft.operationalinsights/workspaces($|/.*)?' or tolower(properties["targetResourceType"]) matches regex 'microsoft.operationalinsights/workspaces($|/.*)?' or tolower(properties["scopes"]) matches regex 'providers/microsoft.operationalinsights/workspaces($|/.*)?'
| where properties contains "Perf" or properties  contains "InsightsMetrics" or properties  contains "ContainerInventory" or properties  contains "ContainerNodeInventory" or properties  contains "KubeNodeInventory" or properties  contains"KubePodInventory" or properties  contains "KubePVInventory" or properties  contains "KubeServices" or properties  contains "KubeEvents" 
| project id,name,type,properties,enabled,severity,subscriptionId
| order by tolower(name) asc

Next steps

• See Data transformations in Container insights to add transformations to the DCR that will further filter data based on detailed criteria.