任务抢占Task preemption

Azure Databricks 中的 Apache Spark 计划程序会自动抢占任务以强制进行公平共享。The Apache Spark scheduler in Azure Databricks automatically preempts tasks to enforce fair sharing. 对于具有许多并发运行作业的群集,这保证了它们的交互式响应时间。This guarantees interactive response times on clusters with many concurrently running jobs.

提示

当任务被计划程序抢占时,其终止原因将设置为 preempted by schedulerWhen tasks are preempted by the scheduler, their kill reason will be set to preempted by scheduler. 此原因在 Spark UI 中可见,可用于调试抢占行为。This reason is visible in the Spark UI and can be used to debug preemption behavior.

抢占选项Preemption options

默认情况下,抢占是保守的:在计划程序干预之前,作业可能会缺少资源长达 30 秒。By default, preemption is conservative: jobs can be starved of resources for up to 30 seconds before the scheduler intervenes. 可以通过在群集启动时设置以下 Spark 配置属性来调整抢占:You can tune preemption by setting the following Spark configuration properties at cluster launch time:

  • 是否应启用抢占。Whether preemption should be enabled.

    spark.databricks.preemption.enabled true
    
  • 用于保证每个作业的公平份额分数。The fair share fraction to guarantee per job. 设置为 1.0 时意味着计划程序将积极尝试确保完美的公平份额。Setting this to 1.0 means the scheduler will aggressively attempt to guarantee perfect fair sharing. 设置为 0.0 时可有效禁用抢占。Setting this to 0.0 effectively disables preemption. 默认设置为 0.5,这意味着在最坏的情况下,工作将得到其公平份额的一半。The default setting is 0.5, which means at worst a jobs will get half of its fair share.

    spark.databricks.preemption.threshold 0.5
    
  • 在进行抢占之前,作业必须保持耗尽的时间。How long a job must remain starved before preemption kicks in. 设置为较低值时可提供更长的交互式响应时间,但会降低群集效率。Setting this to lower values will provide more interactive response times, at the cost of cluster efficiency. 建议值为 1-100 秒。Recommended values are from 1-100 seconds.

    spark.databricks.preemption.timeout 30s
    
  • 计划程序检查任务抢占的频率。How often the scheduler will check for task preemption. 应设置为小于抢占超时的值。This should be set to less than the preemption timeout.

    spark.databricks.preemption.interval 5s
    

有关作业计划的详细信息,请参阅应用程序中的计划For further information on job scheduling, see Scheduling Within an Application.