Apache Ambari heartbeat issues in Azure HDInsight

This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.

Scenario: High CPU utilization

Issue

Ambari agent has high CPU utilization, which results in alerts from Ambari UI that for some nodes the Ambari agent heartbeat is lost. The heartbeat lost alert is usually transient.

Cause

Due to various ambari-agent bugs, in rare cases, your ambari-agent can have high (close to 100) percentage CPU utilization.

Resolution

  1. Identify process ID PID of ambari-agent:

    ps -ef | grep ambari_agent

  2. Then run the following command to show CPU utilization:

    top -p <ambari-agent-pid>

  3. Restart ambari-agent to mitigate issue:

    service ambari-agent restart

  4. If restart does not work, kill the ambari-agent process and then start it up:

    kill -9 <ambari-agent-pid>
service ambari-agent start

Scenario: Ambari agent not started

Issue

Ambari agent hasn't started which results in alerts from Ambari UI that for some nodes the Ambari agent heartbeat is lost.

Cause

The alerts are caused by the Ambari agent not running.

Resolution

  1. Confirm status of ambari-agent:

    service ambari-agent status

  2. Confirm if failover controller services are running:

    ps -ef | grep failover

    If failover controller services aren't running, it's likely due to a problem prevent hdinsight-agent from starting failover controller. Check hdinsight-agent log from /var/log/hdinsight-agent/hdinsight-agent.out file.

Scenario: Heartbeat lost for Ambari

Issue

Ambari heartbeat agent was lost.

Cause

OMS logs are causing high CPU utilization.

Resolution

Next steps

If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:

  • If you need more help, you can submit a support request from the Azure portal. Select Support from the menu bar or open the Help + support hub.
  • Last updated on