在 Azure Monitor 中查询容器日志

Kubernetes 群集的容器日志从容器主机和容器收集性能指标、清单数据和运行状况状态信息。 数据存储在 Log Analytics 工作区中，在 Azure Monitor 中使用 Log Analytics 进行日志查询。

此数据适用于包括迁移规划、容量分析、发现和按需性能故障排除的方案。 使用此数据的日志查询有助于查找趋势、诊断瓶颈、预测或关联数据，以帮助确定当前群集配置是否以最佳方式执行。

有关使用这些查询的信息，请参阅在 Azure Monitor Log Analytics 中使用查询。 有关使用 Log Analytics 运行查询并处理其结果的完整教程，请参阅 Log Analytics 教程。

重要

如果修改默认数据收集设置，查询可能不会返回预期的结果。 最值得注意的是，如果不从群集收集性能数据，因为它是冗余的，则使用 Perf 表的任何查询都不会返回结果。

有关 DCR 配置（包括禁用性能数据的收集），请参阅 筛选和自定义 Kubernetes 群集的数据收集。 有关更多数据收集选项，请参阅 使用 ConfigMap 配置容器日志收集。

打开 Log Analytics

Log Analytics 会话的范围取决于您如何启动它。 若要访问工作区中的所有数据，请从 Azure 门户中的“监视”菜单中选择“日志”。 若要将数据限制为单个 Kubernetes 群集，请从相关群集的菜单中选择“日志”。

显示启动 Log Analytics 的屏幕截图。

现有的日志查询

无需了解如何编写日志查询也能使用 Log Analytics。 而是从多个预生成查询中进行选择。 无需修改即可运行查询，或者将其用作自定义查询的开始。 选择 Log Analytics 屏幕顶部的“查询”，并查看“资源类型”为“Kubernetes 服务”的查询。

显示针对 Kubernetes 的 Log Analytics 查询的屏幕截图。

容器表

有关容器见解所使用的表列表及其详细说明，请参阅 Azure Monitor 表格引用。 这些表都可用于日志查询。

示例日志查询

从一两个示例开始生成查询，然后修改它们以适应需求的做法通常很有用。 为了帮助你构建更高级的查询，请试验以下示例查询。

本文中的查询假设你的集群已启用 Container insights，并且数据正在流入你的 Log Analytics 工作区。 若要启用容器见解，请参阅 启用 Kubernetes 集群监控。

列出容器的所有生命周期信息

此查询返回每个容器的生命周期详细信息，包括其映像、状态以及创建和完成时间。

ContainerInventory
| project Computer, Name, Image, ImageTag, ContainerState, CreatedTime, StartedTime, FinishedTime
| render table

Kubernetes 事件

此查询从 KubeEvents 表中返回 Kubernetes 事件，该事件按最新事件排序。

注意

默认情况下，容器见解不会收集普通事件类型，因此在查询 KubeEvents 表时看不到它们，除非启用 collect_all_kube_events ConfigMap 设置。 如果需要收集正常事件，请在 container-azm-ms-agentconfig ConfigMap 中启用 collect_all_kube_events 设置。 有关如何配置 ConfigMap 的信息，请参阅使用 ConfigMap 配置容器日志收集 。

KubeEvents
| where not(isempty(Namespace))
| sort by TimeGenerated desc
| render table

每分钟请求数（按照自定义指标）

InsightsMetrics
| where Name == "requests_count"
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc
| project RequestsPerMinute = Val - prev(Val), TimeGenerated
| render barchart

按名称和命名空间划分的 Pod

此查询使用 ContainerLogV2 表返回与指定名称和命名空间匹配的 Pod 的日志输出。

let startTimestamp = ago(1h);
KubePodInventory
| where TimeGenerated > startTimestamp
| project ContainerID, PodName=Name, Namespace
| where PodName contains "name" and Namespace startswith "namespace"
| distinct ContainerID, PodName
| join
(
    ContainerLogV2
    | where TimeGenerated > startTimestamp
)
on ContainerID
// at this point before the next pipe, columns from both tables are available to be "projected".
| project TimeGenerated, PodName, LogMessage, LogSource
| summarize by TimeGenerated, LogMessage
| order by TimeGenerated desc

Pod 横向扩展 (HPA)

此查询返回每个部署中横向扩展的副本数。 它使用 HPA 中配置的最大副本数计算横向扩展百分比。

let _minthreshold = 70; // minimum threshold goes here if you want to setup as an alert
let _maxthreshold = 90; // maximum threshold goes here if you want to setup as an alert
let startDateTime = ago(60m);
KubePodInventory
| where TimeGenerated >= startDateTime
| where Namespace !in('default', 'kube-system') // List of non system namespace filter goes here.
| extend labels = todynamic(PodLabel)
| extend deployment_hpa = reverse(substring(reverse(ControllerName), indexof(reverse(ControllerName), "-") + 1))
| distinct tostring(deployment_hpa)
| join kind=inner (InsightsMetrics
    | where TimeGenerated > startDateTime
    | where Name == 'kube_hpa_status_current_replicas'
    | extend pTags = todynamic(Tags) //parse the tags for values
    | extend ns = todynamic(pTags.k8sNamespace) //parse namespace value from tags
    | extend deployment_hpa = todynamic(pTags.targetName) //parse HPA target name from tags
    | extend max_reps = todynamic(pTags.spec_max_replicas) // Parse maximum replica settings from HPA deployment
    | extend desired_reps = todynamic(pTags.status_desired_replicas) // Parse desired replica settings from HPA deployment
    | summarize arg_max(TimeGenerated, *) by tostring(ns), tostring(deployment_hpa), Cluster=toupper(tostring(split(_ResourceId, '/')[8])), toint(desired_reps), toint(max_reps), scale_out_percentage=(desired_reps * 100 / max_reps)
    //| where scale_out_percentage > _minthreshold and scale_out_percentage <= _maxthreshold
    )
    on deployment_hpa

节点池横向扩展

此查询返回每个节点池中的活动节点数。 它计算自动缩放程序设置中的可用活动节点数和最大节点配置，以确定横向扩展百分比。 请查看查询中的注释行，以将该查询用于 结果数 警报规则。

let nodepoolMaxnodeCount = 10; // the maximum number of nodes in your auto scale setting goes here.
let _minthreshold = 20;
let _maxthreshold = 90;
let startDateTime = 60m;
KubeNodeInventory
| where TimeGenerated >= ago(startDateTime)
| extend nodepoolType = todynamic(Labels) //Parse the labels to get the list of node pool types
| extend nodepoolName = todynamic(nodepoolType[0].agentpool) // parse the label to get the nodepool name or set the specific nodepool name (like nodepoolName = 'agentpool)'
| summarize nodeCount = count(Computer) by ClusterName, tostring(nodepoolName), TimeGenerated
//(Uncomment the below two lines to set this as a log search alert)
//| extend scaledpercent = iff(((nodeCount * 100 / nodepoolMaxnodeCount) >= _minthreshold and (nodeCount * 100 / nodepoolMaxnodeCount) < _maxthreshold), "warn", "normal")
//| where scaledpercent == 'warn'
| summarize arg_max(TimeGenerated, *) by nodeCount, ClusterName, tostring(nodepoolName)
| project ClusterName,
    TotalNodeCount= strcat("Total Node Count: ", nodeCount),
    ScaledOutPercentage = (nodeCount * 100 / nodepoolMaxnodeCount),
    TimeGenerated,
    nodepoolName

系统容器可用性

此查询返回给定控制器类型的系统容器，并报告不可用的百分比。 将 _controllerKind 参数设置为 ReplicaSet 或 DaemonSet，以指定任一控制器类型。 若要将此查询用于 多个结果 警报规则，请参阅查询中的注释行。

let startDateTime = 5m; // the minimum time interval goes here
let _minalertThreshold = 50; //Threshold for minimum and maximum unavailable or not running containers
let _maxalertThreshold = 70;
let _controllerKind = 'ReplicaSet'; // set to 'ReplicaSet' or 'DaemonSet' to target either controller type
KubePodInventory
| where TimeGenerated >= ago(startDateTime)
| distinct ClusterName, TimeGenerated
| summarize Clustersnapshot = count() by ClusterName
| join kind=inner (
    KubePodInventory
    | where TimeGenerated >= ago(startDateTime)
    | where Namespace in('default', 'kube-system') and ControllerKind == _controllerKind // the system namespace filter goes here
    | distinct ClusterName, Computer, PodUid, TimeGenerated, PodStatus, ServiceName, PodLabel, Namespace, ContainerStatus
    | summarize arg_max(TimeGenerated, *), TotalPODCount = count(), podCount = sumif(1, PodStatus == 'Running' or PodStatus != 'Running'), containerNotrunning = sumif(1, ContainerStatus != 'running')
        by ClusterName, TimeGenerated, ServiceName, PodLabel, Namespace
    )
    on ClusterName
| project ClusterName, ServiceName, podCount, containerNotrunning, containerNotrunningPercent = (containerNotrunning * 100 / podCount), TimeGenerated, PodStatus, PodLabel, Namespace, Environment = tostring(split(ClusterName, '-')[3]), Location = tostring(split(ClusterName, '-')[4]), ContainerStatus
//Uncomment the below line to set for automated alert
//| where PodStatus == "Running" and containerNotrunningPercent > _minalertThreshold and containerNotrunningPercent < _maxalertThreshold
| summarize arg_max(TimeGenerated, *), c_entry=count() by PodLabel, ServiceName, ClusterName
//Below lines are to parse the labels to identify the impacted service/component name
| extend parseLabel = replace(@'k8s-app', @'k8sapp', PodLabel)
| extend parseLabel = replace(@'app.kubernetes.io\\/component', @'appkubernetesiocomponent', parseLabel)
| extend parseLabel = replace(@'app.kubernetes.io\\/instance', @'appkubernetesioinstance', parseLabel)
| extend tags = todynamic(parseLabel)
| extend tag01 = todynamic(tags[0].app)
| extend tag02 = todynamic(tags[0].k8sapp)
| extend tag03 = todynamic(tags[0].appkubernetesiocomponent)
| extend tag04 = todynamic(tags[0].aadpodidbinding)
| extend tag05 = todynamic(tags[0].appkubernetesioinstance)
| extend tag06 = todynamic(tags[0].component)
| project ClusterName, TimeGenerated,
    ServiceName = strcat( ServiceName, tag01, tag02, tag03, tag04, tag05, tag06),
    ContainerUnavailable = strcat("Unavailable Percentage: ", containerNotrunningPercent),
    PodStatus = strcat("PodStatus: ", PodStatus),
    ContainerStatus = strcat("Container Status: ", ContainerStatus)

容器日志

容器见解会将其从 AKS 群集中收集的容器日志存储在你的 Log Analytics 工作区的 ContainerLogV2 表 中。 运行以下示例查询，查找来自目标 Pod、部署或命名空间的标准错误日志和标准输出日志。

特定 Pod、命名空间和容器的容器日志

ContainerLogV2
| where _ResourceId =~ "clusterResourceID" //update with resource ID
| where PodNamespace == "podNamespace" //update with target namespace
| where PodName == "podName" //update with target pod
| where ContainerName == "containerName" //update with target container
| project TimeGenerated, Computer, ContainerId, LogMessage, LogSource

特定部署的容器日志

let startDateTime = datetime('start time'); //start time format: YYYY-MM-DD HH:MM:SS
let endDateTime = datetime('end time'); //end time format: YYYY-MM-DD HH:MM:SS
let KubePodInv = KubePodInventory
| where _ResourceId =~ "clusterResourceID" //update with resource ID
| where Namespace == "deploymentNamespace" //update with target namespace
| where ControllerKind == "ReplicaSet"
| extend deployment = reverse(substring(reverse(ControllerName), indexof(reverse(ControllerName), "-") + 1))
| where deployment == "deploymentName" //update with target deployment
| extend ContainerId = ContainerID
| summarize arg_max(TimeGenerated, *)  by deployment, ContainerId, PodStatus, ContainerStatus
| project deployment, ContainerId, PodStatus, ContainerStatus;
KubePodInv
| join
(
    ContainerLogV2
  | where TimeGenerated >= startDateTime and TimeGenerated < endDateTime
  | where PodNamespace == "deploymentNamespace" //update with target namespace
  | where PodName startswith "deploymentName" //update with target deployment
) on ContainerId
| project TimeGenerated, deployment, PodName, PodStatus, ContainerName, ContainerId, ContainerStatus, LogMessage, LogSource

特定命名空间中任一失败的 Pod 的容器日志

let startDateTime = datetime('start time'); //start time format: YYYY-MM-DD HH:MM:SS
let endDateTime = datetime('end time'); //end time format: YYYY-MM-DD HH:MM:SS
let KubePodInv = KubePodInventory
| where TimeGenerated >= startDateTime and TimeGenerated < endDateTime
| where _ResourceId =~ "clusterResourceID" //update with resource ID
| where Namespace == "podNamespace" //update with target namespace
| where PodStatus == "Failed"
| extend ContainerId = ContainerID
| summarize arg_max(TimeGenerated, *)  by  ContainerId, PodStatus, ContainerStatus
| project ContainerId, PodStatus, ContainerStatus;
KubePodInv
| join
(
    ContainerLogV2
| where TimeGenerated >= startDateTime and TimeGenerated < endDateTime
| where PodNamespace == "podNamespace" //update with target namespace
) on ContainerId
| project TimeGenerated, PodName, PodStatus, ContainerName, ContainerId, ContainerStatus, LogMessage, LogSource

容器见解默认可视化查询语句

这些查询来自容器见解中的 默认可视化视图。 如果启用自定义 成本优化设置 而不是默认图表，请使用这些查询。

节点计数(按状态)

此图表所需的表包括 KubeNodeInventory。

 let trendBinSize = 5m;
 let maxListSize = 1000;
 let clusterId = 'clusterResourceID'; //update with resource ID
 let rawData = KubeNodeInventory
| where ClusterId =~ clusterId
| distinct ClusterId, TimeGenerated
| summarize ClusterSnapshotCount = count() by Timestamp = bin(TimeGenerated, trendBinSize), ClusterId
| join hint.strategy=broadcast ( KubeNodeInventory
| where ClusterId =~ clusterId
| summarize TotalCount = count(), ReadyCount = sumif(1, Status contains ('Ready')) by ClusterId, Timestamp = bin(TimeGenerated, trendBinSize)
| extend NotReadyCount = TotalCount - ReadyCount ) on ClusterId, Timestamp
| project ClusterId, Timestamp, TotalCount = todouble(TotalCount) / ClusterSnapshotCount, ReadyCount = todouble(ReadyCount) / ClusterSnapshotCount, NotReadyCount = todouble(NotReadyCount) / ClusterSnapshotCount;
 rawData
| order by Timestamp asc
| summarize makelist(Timestamp, maxListSize), makelist(TotalCount, maxListSize), makelist(ReadyCount, maxListSize), makelist(NotReadyCount, maxListSize) by ClusterId
| join ( rawData
| summarize Avg_TotalCount = avg(TotalCount), Avg_ReadyCount = avg(ReadyCount), Avg_NotReadyCount = avg(NotReadyCount) by ClusterId ) on ClusterId
| project ClusterId, Avg_TotalCount, Avg_ReadyCount, Avg_NotReadyCount, list_Timestamp, list_TotalCount, list_ReadyCount, list_NotReadyCount

按状态划分的 Pod 数量

此图表所需的表包括 KubePodInventory。

 let trendBinSize = 5m;
 let maxListSize = 1000;
 let clusterId = 'clusterResourceID'; //update with resource ID
 let rawData = KubePodInventory
| where ClusterId =~ clusterId
| distinct ClusterId, TimeGenerated
| summarize ClusterSnapshotCount = count() by bin(TimeGenerated, trendBinSize), ClusterId
| join hint.strategy=broadcast ( KubePodInventory
| where ClusterId =~ clusterId
| summarize PodStatus=any(PodStatus) by TimeGenerated, PodUid, ClusterId
| summarize TotalCount = count(), PendingCount = sumif(1, PodStatus =~ 'Pending'), RunningCount = sumif(1, PodStatus =~ 'Running'), SucceededCount = sumif(1, PodStatus =~ 'Succeeded'), FailedCount = sumif(1, PodStatus =~ 'Failed'), TerminatingCount = sumif(1, PodStatus =~ 'Terminating') by ClusterId, bin(TimeGenerated, trendBinSize) ) on ClusterId, TimeGenerated
| extend UnknownCount = TotalCount - PendingCount - RunningCount - SucceededCount - FailedCount - TerminatingCount
| project ClusterId, Timestamp = TimeGenerated, TotalCount = todouble(TotalCount) / ClusterSnapshotCount, PendingCount = todouble(PendingCount) / ClusterSnapshotCount, RunningCount = todouble(RunningCount) / ClusterSnapshotCount, SucceededCount = todouble(SucceededCount) / ClusterSnapshotCount, FailedCount = todouble(FailedCount) / ClusterSnapshotCount, TerminatingCount = todouble(TerminatingCount) / ClusterSnapshotCount, UnknownCount = todouble(UnknownCount) / ClusterSnapshotCount;
 let rawDataCached = rawData;
 rawDataCached
| order by Timestamp asc
| summarize makelist(Timestamp, maxListSize), makelist(TotalCount, maxListSize), makelist(PendingCount, maxListSize), makelist(RunningCount, maxListSize), makelist(SucceededCount, maxListSize), makelist(FailedCount, maxListSize), makelist(TerminatingCount, maxListSize), makelist(UnknownCount, maxListSize) by ClusterId
| join ( rawDataCached
| summarize Avg_TotalCount = avg(TotalCount), Avg_PendingCount = avg(PendingCount), Avg_RunningCount = avg(RunningCount), Avg_SucceededCount = avg(SucceededCount), Avg_FailedCount = avg(FailedCount), Avg_TerminatingCount = avg(TerminatingCount), Avg_UnknownCount = avg(UnknownCount) by ClusterId ) on ClusterId
| project ClusterId, Avg_TotalCount, Avg_PendingCount, Avg_RunningCount, Avg_SucceededCount, Avg_FailedCount, Avg_TerminatingCount, Avg_UnknownCount, list_Timestamp, list_TotalCount, list_PendingCount, list_RunningCount, list_SucceededCount, list_FailedCount, list_TerminatingCount, list_UnknownCount

容器列表(按状态)

此图表所需的表包括 KubePodInventory 和 Perf。

 let startDateTime = datetime('start time'); //start time format: YYYY-MM-DD HH:MM:SS
 let endDateTime = datetime('end time'); //end time format: YYYY-MM-DD HH:MM:SS
 let trendBinSize = 15m;
 let maxResultCount = 10000;
 let metricUsageCounterName = 'cpuUsageNanoCores';
 let metricLimitCounterName = 'cpuLimitNanoCores';
 let KubePodInventoryTable = KubePodInventory
| where TimeGenerated >= startDateTime
| where TimeGenerated < endDateTime
| where isnotempty(ClusterName)
| where isnotempty(Namespace)
| where isnotempty(Computer)
| project TimeGenerated, ClusterId, ClusterName, Namespace, ServiceName, ControllerName, Node = Computer, Pod = Name, ContainerInstance = ContainerName, ContainerID, ReadySinceNow = format_timespan(endDateTime - ContainerCreationTimeStamp , 'ddd.hh:mm:ss.fff'), Restarts = ContainerRestartCount, Status = ContainerStatus, ContainerStatusReason = columnifexists('ContainerStatusReason', ''), ControllerKind = ControllerKind, PodStatus;
 let startRestart = KubePodInventoryTable
| summarize arg_min(TimeGenerated, *) by Node, ContainerInstance
| where ClusterId =~ 'clusterResourceID' //update with resource ID
| project Node, ContainerInstance, InstanceName = strcat(ClusterId, '/', ContainerInstance), StartRestart = Restarts;
 let IdentityTable = KubePodInventoryTable
| summarize arg_max(TimeGenerated, *) by Node, ContainerInstance
| where ClusterId =~ 'clusterResourceID' //update with resource ID
| project ClusterName, Namespace, ServiceName, ControllerName, Node, Pod, ContainerInstance, InstanceName = strcat(ClusterId, '/', ContainerInstance), ContainerID, ReadySinceNow, Restarts, Status = iff(Status =~ 'running', 0, iff(Status=~'waiting', 1, iff(Status =~'terminated', 2, 3))), ContainerStatusReason, ControllerKind, Containers = 1, ContainerName = tostring(split(ContainerInstance, '/')[1]), PodStatus, LastPodInventoryTimeGenerated = TimeGenerated, ClusterId;
 let CachedIdentityTable = IdentityTable;
 let FilteredPerfTable = Perf
| where TimeGenerated >= startDateTime
| where TimeGenerated < endDateTime
| where ObjectName == 'K8SContainer'
| where InstanceName startswith 'clusterResourceID'
| project Node = Computer, TimeGenerated, CounterName, CounterValue, InstanceName ;
 let CachedFilteredPerfTable = FilteredPerfTable;
 let LimitsTable = CachedFilteredPerfTable
| where CounterName =~ metricLimitCounterName
| summarize arg_max(TimeGenerated, *) by Node, InstanceName
| project Node, InstanceName, LimitsValue = iff(CounterName =~ 'cpuLimitNanoCores', CounterValue/1000000, CounterValue), TimeGenerated;
 let MetaDataTable = CachedIdentityTable
| join kind=leftouter ( LimitsTable ) on Node, InstanceName
| join kind= leftouter ( startRestart ) on Node, InstanceName
| project ClusterName, Namespace, ServiceName, ControllerName, Node, Pod, InstanceName, ContainerID, ReadySinceNow, Restarts, LimitsValue, Status, ContainerStatusReason = columnifexists('ContainerStatusReason', ''), ControllerKind, Containers, ContainerName, ContainerInstance, StartRestart, PodStatus, LastPodInventoryTimeGenerated, ClusterId;
 let UsagePerfTable = CachedFilteredPerfTable
| where CounterName =~ metricUsageCounterName
| project TimeGenerated, Node, InstanceName, CounterValue = iff(CounterName =~ 'cpuUsageNanoCores', CounterValue/1000000, CounterValue);
 let LastRestartPerfTable = CachedFilteredPerfTable
| where CounterName =~ 'restartTimeEpoch'
| summarize arg_max(TimeGenerated, *) by Node, InstanceName
| project Node, InstanceName, UpTime = CounterValue, LastReported = TimeGenerated;
 let AggregationTable = UsagePerfTable
| summarize Aggregation = max(CounterValue) by Node, InstanceName
| project Node, InstanceName, Aggregation;
 let TrendTable = UsagePerfTable
| summarize TrendAggregation = max(CounterValue) by bin(TimeGenerated, trendBinSize), Node, InstanceName
| project TrendTimeGenerated = TimeGenerated, Node, InstanceName , TrendAggregation
| summarize TrendList = makelist(pack("timestamp", TrendTimeGenerated, "value", TrendAggregation)) by Node, InstanceName;
 let containerFinalTable = MetaDataTable
| join kind= leftouter( AggregationTable ) on Node, InstanceName
| join kind = leftouter (LastRestartPerfTable) on Node, InstanceName
| order by Aggregation desc, ContainerName
| join kind = leftouter ( TrendTable) on Node, InstanceName
| extend ContainerIdentity = strcat(ContainerName, ' ', Pod)
| project ContainerIdentity, Status, ContainerStatusReason = columnifexists('ContainerStatusReason', ''), Aggregation, Node, Restarts, ReadySinceNow, TrendList = iif(isempty(TrendList), parse_json('[]'), TrendList), LimitsValue, ControllerName, ControllerKind, ContainerID, Containers, UpTimeNow = datetime_diff('Millisecond', endDateTime, datetime_add('second', toint(UpTime), make_datetime(1970,1,1))), ContainerInstance, StartRestart, LastReportedDelta = datetime_diff('Millisecond', endDateTime, LastReported), PodStatus, InstanceName, Namespace, LastPodInventoryTimeGenerated, ClusterId;
containerFinalTable
| limit 200

控制器列表(按状态)

此图表所需的表包括 KubePodInventory 和 Perf。

 let startDateTime = datetime('start time'); //start time format: YYYY-MM-DD HH:MM:SS
 let endDateTime = datetime('end time'); //end time format: YYYY-MM-DD HH:MM:SS
 let trendBinSize = 15m;
 let metricLimitCounterName = 'cpuLimitNanoCores';
 let metricUsageCounterName = 'cpuUsageNanoCores';
 let primaryInventory = KubePodInventory
| where TimeGenerated >= startDateTime
| where TimeGenerated < endDateTime
| where isnotempty(ClusterName)
| where isnotempty(Namespace)
| extend Node = Computer
| where ClusterId =~ 'clusterResourceID' //update with resource ID
| project TimeGenerated, ClusterId, ClusterName, Namespace, ServiceName, Node = Computer, ControllerName, Pod = Name, ContainerInstance = ContainerName, ContainerID, InstanceName, PerfJoinKey = strcat(ClusterId, '/', ContainerName), ReadySinceNow = format_timespan(endDateTime - ContainerCreationTimeStamp, 'ddd.hh:mm:ss.fff'), Restarts = ContainerRestartCount, Status = ContainerStatus, ContainerStatusReason = columnifexists('ContainerStatusReason', ''), ControllerKind = ControllerKind, PodStatus, ControllerId = strcat(ClusterId, '/', Namespace, '/', ControllerName);
let podStatusRollup = primaryInventory
| summarize arg_max(TimeGenerated, *) by Pod
| project ControllerId, PodStatus, TimeGenerated
| summarize count() by ControllerId, PodStatus = iif(TimeGenerated < ago(30m), 'Unknown', PodStatus)
| summarize PodStatusList = makelist(pack('Status', PodStatus, 'Count', count_)) by ControllerId;
let latestContainersByController = primaryInventory
| where isnotempty(Node)
| summarize arg_max(TimeGenerated, *) by PerfJoinKey
| project ControllerId, PerfJoinKey;
let filteredPerformance = Perf
| where TimeGenerated >= startDateTime
| where TimeGenerated < endDateTime
| where ObjectName == 'K8SContainer'
| where InstanceName startswith 'clusterResourceID' //update with resource ID
| project TimeGenerated, CounterName, CounterValue, InstanceName, Node = Computer ;
let metricByController = filteredPerformance
| where CounterName =~ metricUsageCounterName
| extend PerfJoinKey = InstanceName
| summarize Value = percentile(CounterValue, 95) by PerfJoinKey, CounterName
| join (latestContainersByController) on PerfJoinKey
| summarize Value = sum(Value) by ControllerId, CounterName
| project ControllerId, CounterName, AggregationValue = iff(CounterName =~ 'cpuUsageNanoCores', Value/1000000, Value);
let containerCountByController = latestContainersByController
| summarize ContainerCount = count() by ControllerId;
let restartCountsByController = primaryInventory
| summarize Restarts = max(Restarts) by ControllerId;
let oldestRestart = primaryInventory
| summarize ReadySinceNow = min(ReadySinceNow) by ControllerId;
let trendLineByController = filteredPerformance
| where CounterName =~ metricUsageCounterName
| extend PerfJoinKey = InstanceName
| summarize Value = percentile(CounterValue, 95) by bin(TimeGenerated, trendBinSize), PerfJoinKey, CounterName
| order by TimeGenerated asc
| join kind=leftouter (latestContainersByController) on PerfJoinKey
| summarize Value=sum(Value) by ControllerId, TimeGenerated, CounterName
| project TimeGenerated, Value = iff(CounterName =~ 'cpuUsageNanoCores', Value/1000000, Value), ControllerId
| summarize TrendList = makelist(pack("timestamp", TimeGenerated, "value", Value)) by ControllerId;
let latestLimit = filteredPerformance
| where CounterName =~ metricLimitCounterName
| extend PerfJoinKey = InstanceName
| summarize arg_max(TimeGenerated, *) by PerfJoinKey
| join kind=leftouter (latestContainersByController) on PerfJoinKey
| summarize Value = sum(CounterValue) by ControllerId, CounterName
| project ControllerId, LimitValue = iff(CounterName =~ 'cpuLimitNanoCores', Value/1000000, Value);
let latestTimeGeneratedByController = primaryInventory
| summarize arg_max(TimeGenerated, *) by ControllerId
| project ControllerId, LastTimeGenerated = TimeGenerated;
primaryInventory
| distinct ControllerId, ControllerName, ControllerKind, Namespace
| join kind=leftouter (podStatusRollup) on ControllerId
| join kind=leftouter (metricByController) on ControllerId
| join kind=leftouter (containerCountByController) on ControllerId
| join kind=leftouter (restartCountsByController) on ControllerId
| join kind=leftouter (oldestRestart) on ControllerId
| join kind=leftouter (trendLineByController) on ControllerId
| join kind=leftouter (latestLimit) on ControllerId
| join kind=leftouter (latestTimeGeneratedByController) on ControllerId
| project ControllerId, ControllerName, ControllerKind, PodStatusList, AggregationValue, ContainerCount = iif(isempty(ContainerCount), 0, ContainerCount), Restarts, ReadySinceNow, Node = '-', TrendList, LimitValue, LastTimeGenerated, Namespace
| limit 250;

节点列表(按状态)

此图表所需的表包括 KubeNodeInventory、KubePodInventory 和 Perf。

 let startDateTime = datetime('start time'); //start time format: YYYY-MM-DD HH:MM:SS
 let endDateTime = datetime('end time'); //end time format: YYYY-MM-DD HH:MM:SS
 let binSize = 15m;
 let limitMetricName = 'cpuCapacityNanoCores';
 let usedMetricName = 'cpuUsageNanoCores';
 let materializedNodeInventory = KubeNodeInventory
| where TimeGenerated < endDateTime
| where TimeGenerated >= startDateTime
| project ClusterName, ClusterId, Node = Computer, TimeGenerated, Status, NodeName = Computer, NodeId = strcat(ClusterId, '/', Computer), Labels
| where ClusterId =~ 'clusterResourceID'; //update with resource ID
 let materializedPerf = Perf
| where TimeGenerated < endDateTime
| where TimeGenerated >= startDateTime
| where ObjectName == 'K8SNode'
| extend NodeId = InstanceName;
 let materializedPodInventory = KubePodInventory
| where TimeGenerated < endDateTime
| where TimeGenerated >= startDateTime
| where isnotempty(ClusterName)
| where isnotempty(Namespace)
| where ClusterId =~ 'clusterResourceID'; //update with resource ID
 let inventoryOfCluster = materializedNodeInventory
| summarize arg_max(TimeGenerated, Status) by ClusterName, ClusterId, NodeName, NodeId;
 let labelsByNode = materializedNodeInventory
| summarize arg_max(TimeGenerated, Labels) by ClusterName, ClusterId, NodeName, NodeId;
 let countainerCountByNode = materializedPodInventory
| project ContainerName, NodeId = strcat(ClusterId, '/', Computer)
| distinct NodeId, ContainerName
| summarize ContainerCount = count() by NodeId;
 let latestUptime = materializedPerf
| where CounterName == 'restartTimeEpoch'
| summarize arg_max(TimeGenerated, CounterValue) by NodeId
| extend UpTimeMs = datetime_diff('Millisecond', endDateTime, datetime_add('second', toint(CounterValue), make_datetime(1970,1,1)))
| project NodeId, UpTimeMs;
 let latestLimitOfNodes = materializedPerf
| where CounterName == limitMetricName
| summarize CounterValue = max(CounterValue) by NodeId
| project NodeId, LimitValue = CounterValue;
 let actualUsageAggregated = materializedPerf
| where CounterName == usedMetricName
| summarize Aggregation = percentile(CounterValue, 95) by NodeId //This line updates to the desired aggregation
| project NodeId, Aggregation;
 let aggregateTrendsOverTime = materializedPerf
| where CounterName == usedMetricName
| summarize TrendAggregation = percentile(CounterValue, 95) by NodeId, bin(TimeGenerated, binSize) //This line updates to the desired aggregation
| project NodeId, TrendAggregation, TrendDateTime = TimeGenerated;
 let unscheduledPods = materializedPodInventory
| where isempty(Computer)
| extend Node = Computer
| where isempty(ContainerStatus)
| where PodStatus == 'Pending'
| order by TimeGenerated desc
| take 1
| project ClusterName, NodeName = 'unscheduled', LastReceivedDateTime = TimeGenerated, Status = 'unscheduled', ContainerCount = 0, UpTimeMs = '0', Aggregation = '0', LimitValue = '0', ClusterId;
 let scheduledPods = inventoryOfCluster
| join kind=leftouter (aggregateTrendsOverTime) on NodeId
| extend TrendPoint = pack("TrendTime", TrendDateTime, "TrendAggregation", TrendAggregation)
| summarize make_list(TrendPoint) by NodeId, NodeName, Status
| join kind=leftouter (labelsByNode) on NodeId
| join kind=leftouter (countainerCountByNode) on NodeId
| join kind=leftouter (latestUptime) on NodeId
| join kind=leftouter (latestLimitOfNodes) on NodeId
| join kind=leftouter (actualUsageAggregated) on NodeId
| project ClusterName, NodeName, ClusterId, list_TrendPoint, LastReceivedDateTime = TimeGenerated, Status, ContainerCount, UpTimeMs, Aggregation, LimitValue, Labels
| limit 250;
 union (scheduledPods), (unscheduledPods)
| project ClusterName, NodeName, LastReceivedDateTime, Status, ContainerCount, UpTimeMs = UpTimeMs_long, Aggregation = Aggregation_real, LimitValue = LimitValue_real, list_TrendPoint, Labels, ClusterId

Prometheus 指标

重要

以下示例需要使用容器见解将 Prometheus 指标发送到 Log Analytics 工作区中所述的配置。 这是已由托管版 Prometheus 取代的旧配置。 有关查询 Prometheus 指标的指导，请参阅 PromQL 的 Azure Monitor 指标资源管理器 。

若要查看 Azure Monitor 抓取并按命名空间筛选的 Prometheus 指标，请指定“prometheus”。 下面是一个示例查询，演示如何从 default Kubernetes 命名空间查看 Prometheus 指标。

InsightsMetrics
| where Namespace contains "prometheus"
| extend tags=parse_json(Tags)
| summarize count() by Name

Prometheus 数据也可直接按名称查询。

InsightsMetrics
| where Namespace contains "prometheus"
| where Name contains "some_prometheus_metric"

为了确定每种指标大小每天的摄取量（以 GB 为单位），并判断其是否过高，提供以下查询。

InsightsMetrics
| where Namespace contains "prometheus"
| where TimeGenerated > ago(24h)
| summarize VolumeInGB = (sum(_BilledSize) / (1024 * 1024 * 1024)) by Name
| order by VolumeInGB desc
| render barchart

输出将显示类似于以下示例的结果。 显示数据引入量的日志查询结果的屏幕截图。

若要估算一个月内每项指标的数据量（以 GB 为单位），以了解工作区中接收并引入的数据量是否过高，可使用以下查询。

InsightsMetrics
| where Namespace contains "prometheus"
| where TimeGenerated > ago(24h)
| summarize EstimatedGBPer30dayMonth = (sum(_BilledSize) / (1024 * 1024 * 1024)) * 30 by Name
| order by EstimatedGBPer30dayMonth desc
| render barchart

输出将显示类似于以下示例的结果。 显示数据引入量的日志查询结果的屏幕截图。

配置或抓取错误

为了调查任何配置或抓取错误，下面的示例查询将返回 KubeMonAgentEvents 表中的信息性事件。

KubeMonAgentEvents | where Level != "Info"

输出显示类似于以下示例的结果：

显示通过代理记录信息性事件的查询结果的屏幕截图。

后续步骤

容器见解不包含预定义的警报集。 若要了解如何针对 CPU 和内存使用率过高的情况创建建议的警报，为 DevOps 或操作流程和过程提供支持，请查看使用容器见解创建性能警报。

  • Last updated on