达到了执行上下文数目上限或附加的笔记本数目上限Maximum execution context or notebook attachment limit reached

问题Problem

笔记本或作业执行将停止,并返回以下错误之一:Notebook or job execution stops and returns either of the following errors:

Run result unavailable: job failed with error message
Context ExecutionContextId(1731742567765160237) is disconnected.
Can’t attach this notebook because the cluster has reached the attached notebook limit. Detach a notebook and retry.

原因Cause

将笔记本附加到群集时,Azure Databricks 会创建执行上下文When you attach a notebook to a cluster, Azure Databricks creates an execution context. 如果附加到某个群集的笔记本过多或创建了太多的作业,则在某个时间点,该群集会达到最大阈值限制(145 个执行上下文),Azure Databricks 将返回错误。If there are too many notebooks attached to a cluster or too many jobs are created, at some point the cluster reaches its maximum threshold limit of 145 execution contexts, and Azure Databricks returns an error.

解决方案Solution

配置上下文自动逐出,这允许 Azure Databricks 删除(逐出)空闲的执行上下文。Configure context auto-eviction, which allows Azure Databricks to remove (evict) idle execution contexts. 此外,从管道和 ETL 设计的角度来看,你可以通过以下方法来避免此问题:Additionally, from the pipeline and ETL design perspective, you can avoid this issue by using:

  • 使用较少的笔记本以减少创建的执行上下文数目。Fewer notebooks to reduce the number of execution contexts that are created.
  • 使用作业群集而不是交互式群集。A job cluster instead of an interactive cluster. 如果用例允许,将笔记本或 jar 作为作业提交。If the use case permits, submit notebooks or jars as jobs.