Scenario: Unable to read Apache Yarn log in Azure HDInsight

This article describes troubleshooting steps and possible resolutions for issues when interacting with Azure HDInsight clusters.

Issue

The Apache Yarn logs found from storage account isn't human-readable. The file parser doesn't work and produces the following error message:

java.io.IOException: Not a valid BCFile.

Cause

The Apache Yarn log is aggregated into IndexFile format, which is not supported by the file parser.

Resolution

  1. From a web browser, navigate to https://CLUSTERNAME.azurehdinsight.cn, where CLUSTERNAME is the name of your cluster.

  2. From Ambari UI, navigate to YARN > Configs > Advanced > Advanced yarn-site.

  3. For WASB storage: The default value for yarn.log-aggregation.file-formats is IndexedFormat,TFile. Change the value to TFile.

  4. For ADLS storage: The default value for yarn.nodemanager.log-aggregation.compression-type is gz. Change the value to none.

  5. Save the change and restart all affected services.

Next steps

If you didn't see your problem or are unable to solve your issue, visit the following channel for more support:

  • If you need more help, you can submit a support request from the Azure portal. Select Support from the menu bar or open the Help + support hub. For more detailed information, review How to create an Azure support request. Access to Subscription Management and billing support is included with your Microsoft Azure subscription, and Technical Support is provided through one of the Azure Support Plans.
  • Last updated on