使用 Azure HDInsight 对 Apache Hadoop YARN 进行故障排除Troubleshoot Apache Hadoop YARN by using Azure HDInsight

了解处理 Apache Ambari 中的 Apache Hadoop YARN 有效负载时的最常见问题及其解决方法。Learn about the top issues and their resolutions when working with Apache Hadoop YARN payloads in Apache Ambari.

如何在群集上创建新的 YARN 队列?How do I create a new YARN queue on a cluster?

解决步骤Resolution steps

在 Ambari 中使用以下步骤可以创建新的 YARN 队列,并在所有队列之间均衡容量分配。Use the following steps in Ambari to create a new YARN queue, and then balance the capacity allocation among all the queues.

在此示例中,两个现有队列(defaultthriftsvr)的容量都从 50 % 更改为 25%,因此,新队列 (Spark) 具有 50% 的容量。In this example, two existing queues (default and thriftsvr) both are changed from 50% capacity to 25% capacity, which gives the new queue (spark) 50% capacity.

队列Queue 容量Capacity 最大容量Maximum capacity
默认值default 25%25% 50%50%
thrftsvrthrftsvr 25%25% 50%50%
sparkspark 50%50% 50%50%
  1. 依次选择“Abari 视图”图标和网格模式。 Select the Ambari Views icon, and then select the grid pattern. 接下来,选择“YARN 队列管理器”。 Next, select YARN Queue Manager.

    选择“Ambari 视图”图标

  2. 选择 default 队列。Select the default queue.

    选择 default 队列

  3. default 队列的容量从 50% 更改为 25%。For the default queue, change the capacity from 50% to 25%. thriftsvr 队列的容量更改为 25%。For the thriftsvr queue, change the capacity to 25%.

    将 default 和 thriftsvr 队列的容量更改为 25%

  4. 若要创建新队列,请选择“添加队列”。 To create a new queue, select Add Queue.

    选择“添加队列”

  5. 为新队列命名。Name the new queue.

    将队列命名为 Spark

  6. 容量值保留为 50%,并选择“操作”按钮。 Leave the capacity values at 50%, and then select the Actions button.

    选择“操作”按钮

  7. 选择“保存并刷新队列”。 Select Save and Refresh Queues.

    选择“保存并刷新队列”

YARN 计划程序 UI 中会立即显示这些更改。These changes are visible immediately on the YARN Scheduler UI.

其他阅读材料Additional reading

如何从群集下载 YARN 日志?How do I download YARN logs from a cluster?

解决步骤Resolution steps

  1. 使用安全外壳 (SSH) 客户端连接到 HDInsight 群集。Connect to the HDInsight cluster by using a Secure Shell (SSH) client. 有关详细信息,请参阅其他阅读材料For more information, see Additional reading.

  2. 若要列出当前正在运行的 YARN 应用程序的所有应用程序 ID,请运行以下命令:To list all the application IDs of the YARN applications that are currently running, run the following command:

    yarn top
    

    ID 会列在 APPLICATIONID 列中。The IDs are listed in the APPLICATIONID column. 可从 APPLICATIONID 列下载日志。You can download logs from the APPLICATIONID column.

    YARN top - 18:00:07, up 19d, 0:14, 0 active users, queue(s): root
    NodeManager(s): 4 total, 4 active, 0 unhealthy, 0 decommissioned, 0 lost, 0 rebooted
    Queue(s) Applications: 2 running, 10 submitted, 0 pending, 8 completed, 0 killed, 0 failed
    Queue(s) Mem(GB): 97 available, 3 allocated, 0 pending, 0 reserved
    Queue(s) VCores: 58 available, 2 allocated, 0 pending, 0 reserved
    Queue(s) Containers: 2 allocated, 0 pending, 0 reserved
    
                      APPLICATIONID USER             TYPE      QUEUE   #CONT  #RCONT  VCORES RVCORES     MEM    RMEM  VCORESECS    MEMSECS %PROGR       TIME NAME
     application_1490377567345_0007 hive            spark  thriftsvr       1       0       1       0      1G      0G    1628407    2442611  10.00   18:20:20 Thrift JDBC/ODBC Server
     application_1490377567345_0006 hive            spark  thriftsvr       1       0       1       0      1G      0G    1628430    2442645  10.00   18:20:20 Thrift JDBC/ODBC Server
    
  3. 若要下载所有应用程序主机的 YARN 容器日志,请使用以下命令:To download YARN container logs for all application masters, use the following command:

    yarn logs -applicationIdn logs -applicationId <application_id> -am ALL > amlogs.txt
    

    此命令创建名为 amlogs.txt 的日志文件。This command creates a log file named amlogs.txt.

  4. 若只要下载最新应用程序主机的 YARN 容器日志,请使用以下命令:To download YARN container logs for only the latest application master, use the following command:

    yarn logs -applicationIdn logs -applicationId <application_id> -am -1 > latestamlogs.txt
    

    此命令创建名为 latestamlogs.txt 的日志文件。This command creates a log file named latestamlogs.txt.

  5. 若要下载前两个应用程序主机的 YARN 容器日志,请使用以下命令:To download YARN container logs for the first two application masters, use the following command:

    yarn logs -applicationIdn logs -applicationId <application_id> -am 1,2 > first2amlogs.txt 
    

    此命令创建名为 first2amlogs.txt 的日志文件。This command creates a log file named first2amlogs.txt.

  6. 若要下载所有 YARN 容器日志,请使用以下命令:To download all YARN container logs, use the following command:

    yarn logs -applicationIdn logs -applicationId <application_id> > logs.txt
    

    此命令创建名为 logs.txt 的日志文件。This command creates a log file named logs.txt.

  7. 若要下载特定容器的 YARN 容器日志,请使用以下命令:To download the YARN container log for a specific container, use the following command:

    yarn logs -applicationIdn logs -applicationId <application_id> -containerId <container_id> > containerlogs.txt 
    

    此命令创建名为 containerlogs.txt 的日志文件。This command creates a log file named containerlogs.txt.

其他阅读材料Additional reading

后续步骤Next steps

如果你的问题未在本文中列出,或者无法解决问题,请访问以下渠道以获取更多支持:If you didn't see your problem or are unable to solve your issue, visit the following channel for more support:

  • 如果需要更多帮助,可以从 Azure 门户提交支持请求。If you need more help, you can submit a support request from the Azure portal. 从菜单栏中选择“支持” ,或打开“帮助 + 支持” 中心。Select Support from the menu bar or open the Help + support hub.