方案:Azure HDInsight 中 Apache Spark 活动的 IllegalArgumentExceptionScenario: IllegalArgumentException for Apache Spark activity in Azure HDInsight

本文介绍在 Azure HDInsight 群集中使用 Apache Spark 组件时出现的问题的故障排除步骤和可能的解决方案。This article describes troubleshooting steps and possible resolutions for issues when using Apache Spark components in Azure HDInsight clusters.

问题Issue

尝试在 Azure 数据工厂管道中执行 Spark 活动时,收到以下异常:You receive the following exception when trying to execute a Spark activity in an Azure Data Factory pipeline:

Exception in thread "main" java.lang.IllegalArgumentException:
Wrong FS: wasbs://additional@xxx.blob.core.windows.net/spark-examples_2.11-2.1.0.jar, expected: wasbs://wasbsrepro-2017-11-07t00-59-42-722z@xxx.blob.core.chinacloudapi.cn

原因Cause

如果应用程序 jar 文件不在 Spark 群集的默认/主存储中,则 Spark 作业将失败。A Spark job will fail if the application jar file is not located in the Spark cluster’s default/primary storage.

这是在此 bug 中跟踪的 Spark 开源框架的一个已知问题:如果 fs.defaultFS 和应用程序 jar 是不同的 URL,Spark 作业将失败This is a known issue with the Spark open-source framework tracked in this bug: Spark job fails if fs.defaultFS and application jar are different url.

此问题已在 Spark 2.3.0 中解决。This issue has been resolved in Spark 2.3.0.

解决方法Resolution

请确保将应用程序 jar 存储在 HDInsight 群集的默认/主存储上。Make sure the application jar is stored on the default/primary storage for the HDInsight cluster.

后续步骤Next steps

如果你的问题未在本文中列出,或者无法解决问题,请访问以下渠道获取更多支持:If you didn't see your problem or are unable to solve your issue, visit the following channels for more support:

  • 如果需要更多帮助,可以从 Azure 门户提交支持请求。If you need more help, you can submit a support request from the Azure portal. 从菜单栏中选择“支持” ,或打开“帮助 + 支持” 中心。Select Support from the menu bar or open the Help + support hub.