添加配置设置会覆盖所有默认的 spark.executor.extraJavaOptions 设置Adding a configuration setting overwrites all default spark.executor.extraJavaOptions settings

问题Problem

当通过在 Apache“Spark 配置”文本区域中输入配置设置来添加配置设置时,新设置将替换现有设置,而不是进行追加。When you add a configuration setting by entering it in the Apache Spark Config text area, the new setting replaces existing settings instead of being appended.

VersionVersion

Databricks Runtime 5.1 及更低版本。Databricks Runtime 5.1 and below.

原因Cause

当群集重启时,群集将从“群集”UI 中创建的配置文件中读取设置,并覆盖默认设置。When the cluster restarts, the cluster reads settings from a configuration file that is created in the Clusters UI, and overwrites the default settings.

例如,将以下 extraJavaOptions 添加到“Spark 配置”文本区域:For example, when you add the following extraJavaOptions to the Spark Config text area:

spark.executor.extraJavaOptions -
javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus
_jmx_exporter/jmx_prometheus_javaagent.yml

然后,在 spark.executor.extraJavaOptions 下的“Spark UI”>“环境”>“Spark 属性”中,则仅显示新添加的配置设置:Then, in Spark UI > Environment > Spark Properties under spark.executor.extraJavaOptions, only the newly added configuration setting shows:

-javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus
_jmx_exporter/jmx_prometheus_javaagent.yml

删除任何现有设置。Any existing settings are removed.

作为参考,默认设置为:For reference, the default settings are:

-Djava.io.tmpdir=/local_disk0/tmp -XX:ReservedCodeCacheSize=256m -
XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-executor-1 -
Djava.security.properties=/databricks/spark/dbconf/java/extra.security -XX:+PrintFlagsFinal -
XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -
Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty
peFactoryImpl -
Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.Documen
tBuilderFactoryImpl -
Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFact
oryImpl -
Djavax.xml.validation.SchemaFactory=https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory -
Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -
Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMX
SImplementationSourceImpl

解决方案Solution

将新的配置设置添加到 spark.executor.extraJavaOptions,但不丢失默认设置:To add a new configuration setting to spark.executor.extraJavaOptions without losing the default settings:

  1. 在“Spark UI”>“环境”>“Spark 属性”中,选择并复制默认为 spark.executor.extraJavaOptions 设置的所有属性。In Spark UI > Environment > Spark Properties, select and copy all of the properties set by default for spark.executor.extraJavaOptions.
  2. 单击 “编辑”Click Edit.
  3. 在“Spark 配置”文本区域(“群集”>“群集名称”>“高级选项”>“Spark”)中,粘贴默认设置 。In the Spark Config text area (Clusters > cluster-name > Advanced Options > Spark), paste the default settings.
  4. 在默认设置下追加新的配置设置。Append the new configuration setting below the default settings.
  5. 单击文本区域外部,然后单击“确认”。Click outside the text area, then click Confirm.
  6. 重启群集。Restart the cluster.

例如,假设你将以下设置粘贴到“Spark 配置”文本区域中。For example, let’s say you paste the following settings into the Spark Config text area. 新配置设置将追加到默认设置。The new configuration setting is appended to the default settings.

spark.executor.extraJavaOptions = -Djava.io.tmpdir=/local_disk0/tmp -
XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-
executor-1 -Djava.security.properties=/databricks/spark/dbconf/java/extra.security -
XX:+PrintFlagsFinal -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -
Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty
peFactoryImpl -
Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentB
uilderFactoryImpl -
Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactor
yImpl -
Djavax.xml.validation.SchemaFactory:https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xer
ces.internal.jaxp.validation.XMLSchemaFactory -
Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -
Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMXSImplem
entationSourceImpl -
javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus_jm
x_exporter/jmx_prometheus_javaagent.yml

重启群集后,默认设置和新添加的配置设置会显示在“Spark UI”>“环境”>“Spark 属性”中。After you restart the cluster, the default settings and newly added configuration setting appear in Spark UI > Environment > Spark Properties.