分析 HDInsight 上 Apache Kafka 的日志Analyze logs for Apache Kafka on HDInsight

了解如何使用 Azure Monitor 日志分析 HDInsight 上由 Apache Kafka 生成的日志。Learn how to use Azure Monitor logs to analyze logs generated by Apache Kafka on HDInsight.

备注

本文最近已更新,从使用术语“Log Analytics”改为使用术语“Azure Monitor 日志”。This article was recently updated to use the term Azure Monitor logs instead of Log Analytics. 日志数据仍然存储在 Log Analytics 工作区中,并仍然由同一 Log Analytics 服务收集并分析。Log data is still stored in a Log Analytics workspace and is still collected and analyzed by the same Log Analytics service. 我们正在更新术语,以便更好地反映 Azure Monitor 中日志的角色。We are updating the terminology to better reflect the role of logs in Azure Monitor. 有关详细信息,请参阅 Azure Monitor 术语更改See Azure Monitor terminology changes for details.

日志位置Logs location

群集中的 Apache Kafka 日志位于 /var/log/kafkaApache Kafka logs in the cluster are located at /var/log/kafka. 不管是否使用托管磁盘,都不会在整个群集生命周期内保存或保留 Kafka 日志。Kafka logs aren't saved or persisted across cluster life cycles, regardless if managed disks are used. 下表显示可用的日志。The following table shows the available logs.

日志Log 说明Description
kafka.outkafka.out Kafka 进程的 stdout 和 stderr。stdout and stderr of the Kafka process. 你将在此文件中找到 Kafka 的启动日志和关闭日志。You'll find Kafka startup and shutdown logs in this file.
server.logserver.log 主 Kafka 服务器日志。The main Kafka server log. 所有 Kafka 代理日志均在此处结束。All Kafka broker logs end up here.
controller.logcontroller.log 控制器日志(如果代理充当控制器)。Controller logs if the broker is acting as controller.
statechange.logstatechange.log 代理的所有状态更改事件都记录在此文件中。All state change events to brokers are logged in this file.
kafka-gc.logkafka-gc.log Kafka 垃圾回收统计信息。Kafka Garbage Collection stats.

为 Apache Kafka 启用 Azure Monitor 日志Enable Azure Monitor logs for Apache Kafka

对于所有 HDInsight 群集,启用适用于 HDInsight 的 Azure Monitor 日志的步骤都是相同的。The steps to enable Azure Monitor logs for HDInsight are the same for all HDInsight clusters. 通过访问以下链接来了解如何创建和配置所需的服务:Use the following links to understand how to create and configure the required services:

  1. 创建 Log Analytics 工作区。Create a Log Analytics workspace. 有关详细信息,请参阅 Azure Monitor 中的日志文档。For more information, see the Logs in Azure Monitor document.

  2. 创建 Kafka on HDInsight 群集。Create a Kafka on HDInsight cluster. 有关详细信息,请参阅 Apache Kafka on HDInsight 入门文档。For more information, see the Start with Apache Kafka on HDInsight document.

  3. 配置 Kafka 群集以使用 Azure Monitor 日志。Configure the Kafka cluster to use Azure Monitor logs.

重要

在数据可用于 Azure Monitor 日志之前可能需要等待约 20 分钟。It may take around 20 minutes before data is available for Azure Monitor logs.

查询日志Query logs

  1. Azure 门户中,选择 Log Analytics 工作区。From the Azure portal, select your Log Analytics workspace.

  2. 从左侧菜单的“常规”下,选择“日志” 。From the left menu, under General, select Logs. 从此处可搜索从 Kafka 所收集的数据。From here, you can search the data collected from Kafka. 在查询窗口中输入查询,然后选择“运行”。Enter a query in the query window and then select Run. 一下是一些示例搜索:The following are some example searches:

  • 磁盘使用率:Disk usage:

    Perf
    | where ObjectName == "Logical Disk" and CounterName == "Free Megabytes" and InstanceName == "_Total" and ((Computer startswith_cs "hn" and Computer contains_cs "-") or (Computer startswith_cs "wn" and Computer contains_cs "-")) 
    | summarize AggregatedValue = avg(CounterValue) by Computer, bin(TimeGenerated, 1h)
    
  • CPU 使用率:CPU usage:

    Perf 
    | where CounterName == "% Processor Time" and InstanceName == "_Total" and ((Computer startswith_cs "hn" and Computer contains_cs "-") or (Computer startswith_cs "wn" and Computer contains_cs "-")) 
    | summarize AggregatedValue = avg(CounterValue) by Computer, bin(TimeGenerated, 1h)
    
  • 每秒传入消息数:(将 your_kafka_cluster_name 替换为群集名称。)Incoming messages per second: (Replace your_kafka_cluster_name with your cluster name.)

    metrics_kafka_CL 
    | where ClusterName_s == "your_kafka_cluster_name" and InstanceName_s == "kafka-BrokerTopicMetrics-MessagesInPerSec-Count" 
    | summarize AggregatedValue = avg(kafka_BrokerTopicMetrics_MessagesInPerSec_Count_value_d) by HostName_s, bin(TimeGenerated, 1h)
    
  • 每秒传入字节数:(将 wn0-kafka 替换为工作器节点主机名。)Incoming bytes per second: (Replace wn0-kafka with a worker node host name.)

    metrics_kafka_CL 
    | where HostName_s == "wn0-kafka" and InstanceName_s == "kafka-BrokerTopicMetrics-BytesInPerSec-Count" 
    | summarize AggregatedValue = avg(kafka_BrokerTopicMetrics_BytesInPerSec_Count_value_d) by bin(TimeGenerated, 1h)
    
  • 每秒传出字节数:(将 your_kafka_cluster_name 替换为群集名称。)Outgoing bytes per second: (Replace your_kafka_cluster_name with your cluster name.)

    metrics_kafka_CL 
    | where ClusterName_s == "your_kafka_cluster_name" and InstanceName_s == "kafka-BrokerTopicMetrics-BytesOutPerSec-Count" 
    | summarize AggregatedValue = avg(kafka_BrokerTopicMetrics_BytesOutPerSec_Count_value_d) by bin(TimeGenerated, 1h)
    

    也可输入 * 来搜索全部已记录的类型。You can also enter * to search all types logged. 当前有以下日志可用于查询:Currently the following logs are available for queries:

    日志类型Log type 说明Description
    log_kafkaserver_CLlog_kafkaserver_CL Kafka broker server.logKafka broker server.log
    log_kafkacontroller_CLlog_kafkacontroller_CL Kafka broker controller.logKafka broker controller.log
    metrics_kafka_CLmetrics_kafka_CL Kafka JMX metricsKafka JMX metrics

    Apache kafka 日志分析 CPU 使用率

后续步骤Next steps

有关使用 Apache Kafka 的详细信息,请参阅以下文档:For more information on working with Apache Kafka, see the following documents: