Apache Spark 作业失败,显示 Failed to parse byte stringApache Spark job fails with Failed to parse byte string

问题Problem

Spark-submit 作业失败,出现“Failed to parse byte string: -1”错误消息。Spark-submit jobs fail with a Failed to parse byte string: -1 error message.

java.util.concurrent.ExecutionException: java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:182)
... 108 more
Caused by: java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m.
Failed to parse byte string: -1

原因Cause

spark.driver.maxResultSize 应用程序属性的值为负。The value of the spark.driver.maxResultSize application property is negative.

解决方案Solution

分配给 spark.driver.maxResultSize 的值定义了每个 Spark 操作的序列化结果的最大大小(以字节为单位)。The value assigned to spark.driver.maxResultSize defines the maximum size (in bytes) of the serialized results for each Spark action. 可以为 spark.driver.maxResultSize 属性指定一个正值,以定义特定大小。You can assign a positive value to the spark.driver.maxResultSize property to define a specific size. 也可以指定值 0 来定义无限制的最大大小。You can also assign a value of 0 to define an unlimited maximum size. 不能为此属性指定负值。You cannot assign a negative value to this property.

如果作业的总大小高于 spark.driver.maxResultSize 值,则作业将中止。If the total size of a job is above the spark.driver.maxResultSize value, the job is aborted.

spark.driver.maxResultSize 设置过高的(或无限制的)值时,应小心。You should be careful when setting an excessively high (or unlimited) value for spark.driver.maxResultSize. 限制过高可能会导致驱动程序出现内存不足错误(在 spark.driver.memory 属性设置得不够高的情况下)。A high limit can cause out-of-memory errors in the driver if the spark.driver.memory property is not set high enough.

有关更多详细信息,请参阅 Spark 配置应用程序属性See Spark Configuration Application Properties for more details.