如何从用于容器的 Azure Monitor 查询日志How to query logs from Azure Monitor for containers

适用于容器的 Azure Monitor 从容器主机和容器收集性能指标、清单数据和运行状况状态信息。Azure Monitor for containers collects performance metrics, inventory data, and health state information from container hosts and containers. 每三分钟收集一次数据,并将其转发到 Azure Monitor 中的 Log Analytics 工作区。The data is collected every three minutes and forwarded to the Log Analytics workspace in Azure Monitor. 此数据可用于 Azure Monitor 中的查询This data is available for query in Azure Monitor. 此数据可应用于包括迁移计划、容量分析、发现和按需性能故障排除在内的方案。You can apply this data to scenarios that include migration planning, capacity analysis, discovery, and on-demand performance troubleshooting.

容器记录Container records

下表提供了适用于容器的 Azure Monitor 收集的记录的详细信息。In the following table, details of the records collected by Azure Monitor for containers are provided.

数据Data 数据源Data source 数据类型Data type 字段Fields
主机和容器的性能Performance for hosts and containers 从 cAdvisor 获取使用指标,从 Kube api 获取限制Usage metrics are obtained from cAdvisor and limits from Kube api Perf 计算机、ObjectName、CounterName(处理器时间百分比、磁盘读取 MB、磁盘写入 MB、内存使用 MB、网络接收字节数、网络发送字节数、处理器使用秒数、网络)、CounterValue、TimeGenerated、CounterPath、SourceSystemComputer, ObjectName, CounterName (%Processor Time, Disk Reads MB, Disk Writes MB, Memory Usage MB, Network Receive Bytes, Network Send Bytes, Processor Usage sec, Network), CounterValue, TimeGenerated, CounterPath, SourceSystem
容器库存Container inventory DockerDocker ContainerInventory TimeGenerated、计算机、容器名称、ContainerHostname、映像、ImageTag、ContainerState、ExitCode、EnvironmentVar、命令、CreatedTime、StartedTime、FinishedTime、SourceSystem、ContainerID、ImageIDTimeGenerated, Computer, container name, ContainerHostname, Image, ImageTag, ContainerState, ExitCode, EnvironmentVar, Command, CreatedTime, StartedTime, FinishedTime, SourceSystem, ContainerID, ImageID
容器日志Container log DockerDocker ContainerLog TimeGenerated、计算机、映像 ID、容器名称、LogEntrySource、LogEntry、SourceSystem、ContainerIDTimeGenerated, Computer, image ID, container name, LogEntrySource, LogEntry, SourceSystem, ContainerID
容器节点清单Container node inventory Kube APIKube API ContainerNodeInventory TimeGenerated、计算机、ClassName_s、DockerVersion_s、OperatingSystem_s、Volume_s、Network_s、NodeRole_s、OrchestratorType_s、InstanceID_g、SourceSystemTimeGenerated, Computer, ClassName_s, DockerVersion_s, OperatingSystem_s, Volume_s, Network_s, NodeRole_s, OrchestratorType_s, InstanceID_g, SourceSystem
Kubernetes 群集中的 Pod 清单Inventory of pods in a Kubernetes cluster Kube APIKube API KubePodInventory TimeGenerated、计算机、ClusterId、ContainerCreationTimeStamp、PodUid、PodCreationTimeStamp、ContainerRestartCount、PodRestartCount、PodStartTime、ContainerStartTime、ServiceName、ControllerKind、ControllerName、ContainerStatus、ContainerStatusReason、ContainerID、ContainerName、Name、PodLabel、Namespace、PodStatus、ClusterName、PodIp、SourceSystemTimeGenerated, Computer, ClusterId, ContainerCreationTimeStamp, PodUid, PodCreationTimeStamp, ContainerRestartCount, PodRestartCount, PodStartTime, ContainerStartTime, ServiceName, ControllerKind, ControllerName, ContainerStatus, ContainerStatusReason, ContainerID, ContainerName, Name, PodLabel, Namespace, PodStatus, ClusterName, PodIp, SourceSystem
Kubernetes 群集节点部分清单Inventory of nodes part of a Kubernetes cluster Kube APIKube API KubeNodeInventory TimeGenerated, Computer, ClusterName, ClusterId, LastTransitionTimeReady, Labels, Status, KubeletVersion, KubeProxyVersion, CreationTimeStamp, SourceSystemTimeGenerated, Computer, ClusterName, ClusterId, LastTransitionTimeReady, Labels, Status, KubeletVersion, KubeProxyVersion, CreationTimeStamp, SourceSystem
Kubernetes 事件Kubernetes Events Kube APIKube API KubeEvents TimeGenerated, Computer, ClusterId_s, FirstSeen_t, LastSeen_t, Count_d, ObjectKind_s, Namespace_s, Name_s, Reason_s, Type_s, TimeGenerated_s, SourceComponent_s, ClusterName_s, Message, SourceSystemTimeGenerated, Computer, ClusterId_s, FirstSeen_t, LastSeen_t, Count_d, ObjectKind_s, Namespace_s, Name_s, Reason_s, Type_s, TimeGenerated_s, SourceComponent_s, ClusterName_s, Message, SourceSystem
Kubernetes 群集中的服务Services in the Kubernetes cluster Kube APIKube API KubeServices TimeGenerated, ServiceName_s, Namespace_s, SelectorLabels_s, ClusterId_s, ClusterName_s, ClusterIP_s, ServiceType_s, SourceSystemTimeGenerated, ServiceName_s, Namespace_s, SelectorLabels_s, ClusterId_s, ClusterName_s, ClusterIP_s, ServiceType_s, SourceSystem
Kubernetes 群集节点部分的性能指标Performance metrics for nodes part of the Kubernetes cluster Perf | where ObjectName == "K8SNode"Perf | where ObjectName == "K8SNode" Computer、ObjectName、CounterName(cpuAllocatableBytes、memoryAllocatableBytes、cpuCapacityNanoCores、memoryCapacityBytes、memoryRssBytes、cpuUsageNanoCores、memoryWorkingsetBytes、restartTimeEpoc)、CounterValue、TimeGenerated、CounterPath、SourceSystemComputer, ObjectName, CounterName (cpuAllocatableBytes, memoryAllocatableBytes, cpuCapacityNanoCores, memoryCapacityBytes, memoryRssBytes, cpuUsageNanoCores, memoryWorkingsetBytes, restartTimeEpoch), CounterValue, TimeGenerated, CounterPath, SourceSystem
Kubernetes 群集容器部分的性能指标Performance metrics for containers part of the Kubernetes cluster Perf | where ObjectName == "K8SContainer"Perf | where ObjectName == "K8SContainer" CounterName(cpuRequestNanoCores、memoryRequestBytes、cpuLimitNanoCores、memoryWorkingSetBytes、restartTimeEpoch、cpuUsageNanoCores、memoryRssBytes)、CounterValue、TimeGenerated、CounterPath、SourceSystemCounterName ( cpuRequestNanoCores, memoryRequestBytes, cpuLimitNanoCores, memoryWorkingSetBytes, restartTimeEpoch, cpuUsageNanoCores, memoryRssBytes), CounterValue, TimeGenerated, CounterPath, SourceSystem
自定义指标Custom Metrics InsightsMetrics Computer、Name、Namespace、Origin、SourceSystem、Tags1、TimeGenerated、Type、Va、_ResourceIdComputer, Name, Namespace, Origin, SourceSystem, Tags1, TimeGenerated, Type, Va, _ResourceId

1 Tags 属性表示对应指标的多个维度1 The Tags property represents multiple dimensions for the corresponding metric. 有关 InsightsMetrics 表中收集和存储的指标的详细信息以及记录属性的说明,请参阅 InsightsMetrics 概述For more information about the metrics collected and stored in the InsightsMetrics table and a description of the record properties, see InsightsMetrics overview.

搜索日志以分析数据Search logs to analyze data

Azure Monitor 日志有助于查找趋势、诊断瓶颈、预测或关联有助于确定是否最优执行当前群集配置的数据。Azure Monitor Logs can help you look for trends, diagnose bottlenecks, forecast, or correlate data that can help you determine whether the current cluster configuration is performing optimally. 提供预定义日志搜索,可直接使用,也可通过自定义来按自己想要的方式返回信息。Pre-defined log searches are provided for you to immediately start using or to customize to return the information the way you want.

通过从预览窗格的“在分析中查看”下拉列表中选择“查看 Kubernetes 事件日志”或“查看容器日志”选项,对工作区中的数据执行交互式分析 。You can perform interactive analysis of data in the workspace by selecting the View Kubernetes event logs or View container logs option in the preview pane from the View in analytics drop-down list. “日志搜索”页面在用户所处的 Azure 门户页面的右侧显示。The Log Search page appears to the right of the Azure portal page that you were on.

在 Log Analytics 中分析数据

转发到工作区的容器日志输出为 STDOUT 和 STDERR。The container logs output that's forwarded to your workspace are STDOUT and STDERR. 由于 Azure Monitor 正在监视 Azure 托管的 Kubernetes (AKS),目前因生成了大量数据而不收集 Kube-system。Because Azure Monitor is monitoring Azure-managed Kubernetes (AKS), Kube-system is not collected today because of the large volume of generated data.

日志搜索查询示例Example log search queries

从一两个示例开始生成查询,然后修改它们以适应需求的做法通常很有用。It's often useful to build queries that start with an example or two and then modify them to fit your requirements. 可使用以下示例查询进行试验,帮助生成更高级的查询:To help build more advanced queries, you can experiment with the following sample queries:

查询Query 说明Description
ContainerInventoryContainerInventory
| project Computer, Name, Image, ImageTag, ContainerState, CreatedTime, StartedTime, FinishedTime| project Computer, Name, Image, ImageTag, ContainerState, CreatedTime, StartedTime, FinishedTime
| render table| render table
列出容器的所有生命周期信息List all of a container's lifecycle information
KubeEvents_CLKubeEvents_CL
| where not(isempty(Namespace_s))| where not(isempty(Namespace_s))
| sort by TimeGenerated desc| sort by TimeGenerated desc
| render table| render table
Kubernetes 事件Kubernetes events
ContainerImageInventoryContainerImageInventory
| summarize AggregatedValue = count() by Image, ImageTag, Running| summarize AggregatedValue = count() by Image, ImageTag, Running
映像清单Image inventory
选择“折线图”显示选项:Select the Line chart display option:
性能Perf
| where ObjectName == "K8SContainer" and CounterName == "cpuUsageNanoCores" | summarize AvgCPUUsageNanoCores = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName| where ObjectName == "K8SContainer" and CounterName == "cpuUsageNanoCores" | summarize AvgCPUUsageNanoCores = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName
容器 CPUContainer CPU
选择“折线图”显示选项:Select the Line chart display option:
性能Perf
| where ObjectName == "K8SContainer" and CounterName == "memoryRssBytes" | summarize AvgUsedRssMemoryBytes = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName| where ObjectName == "K8SContainer" and CounterName == "memoryRssBytes" | summarize AvgUsedRssMemoryBytes = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName
容器内存Container memory
InsightsMetricsInsightsMetrics
| where Name == "requests_count"| where Name == "requests_count"
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc| sort by TimeGenerated asc
| project RequestsPerMinute = Val - prev(Val), TimeGenerated| project RequestsPerMinute = Val - prev(Val), TimeGenerated
| render barchart| render barchart
每分钟请求数(按照自定义指标)Requests Per Minute with Custom Metrics

查询 Prometheus 指标数据Query Prometheus metrics data

以下示例是一个 Prometheus 指标查询,显示每个节点每个磁盘每秒的磁盘读取次数。The following example is a Prometheus metrics query showing disk reads per second per disk per node.

InsightsMetrics
| where Namespace == 'container.azm.ms/diskio'
| where TimeGenerated > ago(1h)
| where Name == 'reads'
| extend Tags = todynamic(Tags)
| extend HostName = tostring(Tags.hostName), Device = Tags.name
| extend NodeDisk = strcat(Device, "/", HostName)
| order by NodeDisk asc, TimeGenerated asc
| serialize
| extend PrevVal = iif(prev(NodeDisk) != NodeDisk, 0.0, prev(Val)), PrevTimeGenerated = iif(prev(NodeDisk) != NodeDisk, datetime(null), prev(TimeGenerated))
| where isnotnull(PrevTimeGenerated) and PrevTimeGenerated != TimeGenerated
| extend Rate = iif(PrevVal > Val, Val / (datetime_diff('Second', TimeGenerated, PrevTimeGenerated) * 1), iif(PrevVal == Val, 0.0, (Val - PrevVal) / (datetime_diff('Second', TimeGenerated, PrevTimeGenerated) * 1)))
| where isnotnull(Rate)
| project TimeGenerated, NodeDisk, Rate
| render timechart

若要查看 Azure Monitor 擦除的按命名空间筛选的 Prometheus 指标,请指定“prometheus”。To view Prometheus metrics scraped by Azure Monitor filtered by Namespace, specify "prometheus". 下面是一个示例查询,演示如何从 default kubernetes 命名空间查看 Prometheus 指标。Here is a sample query to view Prometheus metrics from the default kubernetes namespace.

InsightsMetrics 
| where Namespace == "prometheus"
| extend tags=parse_json(Tags)
| summarize count() by Name

Prometheus 数据也可直接按名称查询。Prometheus data can also be directly queried by name.

InsightsMetrics 
| where Namespace == "prometheus"
| where Name contains "some_prometheus_metric"

查询配置或抓取错误Query config or scraping errors

为了调查任何配置或抓取错误,下面的示例查询将返回 KubeMonAgentEvents 表中的信息性事件。To investigate any configuration or scraping errors, the following example query returns informational events from the KubeMonAgentEvents table.

KubeMonAgentEvents | where Level != "Info" 

输出显示类似于以下示例的结果:The output shows results similar to the following example:

通过代理记录信息性事件的查询结果

后续步骤Next steps

用于容器的 Azure Monitor 不包含预定义的警报集。Azure Monitor for containers does not include a predefined set of alerts. 请查看使用用于容器的 Azure Monitor 创建性能警报,了解如何针对高 CPU 和内存利用率创建建议的警报以支持 DevOps 或操作流程和过程。Review the Create performance alerts with Azure Monitor for containers to learn how to create recommended alerts for high CPU and memory utilization to support your DevOps or operational processes and procedures.