本文介绍如何替代 Databricks 资产捆绑包中 Azure Databricks 群集的设置。 请参阅什么是 Databricks 资产捆绑包?。
在 Azure Databricks 捆绑配置文件中,可以将顶级 resources
映射中的群集设置与 targets
映射中的群集设置合并,如以下所示。
对于作业,在作业定义中使用 job_cluster_key
来识别顶层 resources
映射中的作业群集设置,以便与 targets
映射中的作业群集设置连接:
# ...
resources:
jobs:
<some-unique-programmatic-identifier-for-this-job>:
# ...
job_clusters:
- job_cluster_key: <some-unique-programmatic-identifier-for-this-key>
new_cluster:
# Cluster settings.
targets:
<some-unique-programmatic-identifier-for-this-target>:
resources:
jobs:
<the-matching-programmatic-identifier-for-this-job>:
# ...
job_clusters:
- job_cluster_key: <the-matching-programmatic-identifier-for-this-key>
# Any more cluster settings to join with the settings from the
# resources mapping for the matching top-level job_cluster_key.
# ...
如果在顶级 resources
映射和同一 targets
的 job_cluster_key
映射中都定义了任何群集设置,则 targets
映射中的设置优先于顶级 resources
映射中的设置。
对于 DLT 管道,请使用 label
管道定义的群集设置来标识顶级 resources
映射中的群集设置,以便与映射中的 targets
群集设置联接,例如:
# ...
resources:
pipelines:
<some-unique-programmatic-identifier-for-this-pipeline>:
# ...
clusters:
- label: default | maintenance
# Cluster settings.
targets:
<some-unique-programmatic-identifier-for-this-target>:
resources:
pipelines:
<the-matching-programmatic-identifier-for-this-pipeline>:
# ...
clusters:
- label: default | maintenance
# Any more cluster settings to join with the settings from the
# resources mapping for the matching top-level label.
# ...
如果在顶级 resources
映射和同一 targets
的 label
映射中都定义了任何群集设置,则 targets
映射中的设置优先于顶级 resources
映射中的设置。
在此示例中,顶级resources
映射中的spark_version
与node_type_id
和num_workers
在targets
中的resources
映射相结合,以定义名为my-cluster
的job_cluster_key
的设置。
# ...
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
spark_version: 13.3.x-scala2.12
targets:
development:
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
node_type_id: Standard_DS3_v2
num_workers: 1
# ...
为此示例运行 databricks bundle validate
时,生成的图形如下所示:
{
"...": "...",
"resources": {
"jobs": {
"my-job": {
"job_clusters": [
{
"job_cluster_key": "my-cluster",
"new_cluster": {
"node_type_id": "Standard_DS3_v2",
"num_workers": 1,
"spark_version": "13.3.x-scala2.12"
}
}
],
"...": "..."
}
}
}
}
在此示例中,spark_version
和 num_workers
在顶级 resources
映射和 resources
的 targets
映射中定义。 在此示例中,targets
中的resources
映射中的spark_version
和num_workers
优先于顶级resources
映射中的spark_version
和num_workers
,以定义名为my-cluster
的job_cluster_key
的设置。
# ...
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
spark_version: 13.3.x-scala2.12
node_type_id: Standard_DS3_v2
num_workers: 1
targets:
development:
resources:
jobs:
my-job:
name: my-job
job_clusters:
- job_cluster_key: my-cluster
new_cluster:
spark_version: 12.2.x-scala2.12
num_workers: 2
# ...
为此示例运行 databricks bundle validate
时,生成的图形如下所示:
{
"...": "...",
"resources": {
"jobs": {
"my-job": {
"job_clusters": [
{
"job_cluster_key": "my-cluster",
"new_cluster": {
"node_type_id": "Standard_DS3_v2",
"num_workers": 2,
"spark_version": "12.2.x-scala2.12"
}
}
],
"...": "..."
}
}
}
}
在此示例中,顶级resources
映射中的node_type_id
与targets
中的num_workers
映射和resources
映射结合使用,以定义名为default
的label
设置。
# ...
resources:
pipelines:
my-pipeline:
clusters:
- label: default
node_type_id: Standard_DS3_v2
targets:
development:
resources:
pipelines:
my-pipeline:
clusters:
- label: default
num_workers: 1
# ...
为此示例运行 databricks bundle validate
时,生成的图形如下所示:
{
"...": "...",
"resources": {
"pipelines": {
"my-pipeline": {
"clusters": [
{
"label": "default",
"node_type_id": "Standard_DS3_v2",
"num_workers": 1
}
],
"...": "..."
}
}
}
}
在此示例中,num_workers
在顶级 resources
映射和 resources
的 targets
映射中定义。
num_workers
在resources
映射targets
中优先于顶级resources
映射num_workers
,用于定义命名为default
的label
的设置:
# ...
resources:
pipelines:
my-pipeline:
clusters:
- label: default
node_type_id: Standard_DS3_v2
num_workers: 1
targets:
development:
resources:
pipelines:
my-pipeline:
clusters:
- label: default
num_workers: 2
# ...
为此示例运行 databricks bundle validate
时,生成的图形如下所示:
{
"...": "...",
"resources": {
"pipelines": {
"my-pipeline": {
"clusters": [
{
"label": "default",
"node_type_id": "Standard_DS3_v2",
"num_workers": 2
}
],
"...": "..."
}
}
}
}