Scenario: Apache Spark Streaming application stops after executing for 24 days in Azure HDInsight

  • Article

This article describes troubleshooting steps and possible resolutions for issues when using Apache Spark components in Azure HDInsight clusters.

Issue

An Apache Spark Streaming application stops after executing for 24 days and there are no errors in the log files.

Cause

The livy.server.session.timeout value controls how long Apache Livy should wait for a session to complete. Once the session length reaches the session.timeout value, the Livy session and the application are automatically killed.

Resolution

For long running jobs, increase the value of livy.server.session.timeout using the Ambari UI. You can access the Livy configuration from the Ambari UI using the URL https://<yourclustername>.azurehdinsight.cn/#/main/services/LIVY/configs.

Replace <yourclustername> with the name of your HDInsight cluster as shown in the portal.

Next steps

If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:

  • If you need more help, you can submit a support request from the Azure portal. Select Support from the menu bar or open the Help + support hub.