Manage identities, permissions, and privileges for Databricks Jobs
This article contains recommendations and instructions for managing identities, permissions, and privileges for Databricks Jobs.
Note
Secrets are not redacted from a cluster's Spark driver log stdout
and stderr
streams. To protect sensitive data, by default, Spark driver logs are viewable only by users with CAN MANAGE permission on job, single user access mode, and shared access mode clusters. To allow users with CAN ATTACH TO or CAN RESTART permission to view the logs on these clusters, set the following Spark configuration property in the cluster configuration: spark.databricks.acl.needAdminPermissionToViewLogs false
.
On No Isolation Shared access mode clusters, the Spark driver logs can be viewed by users with CAN ATTACH TO or CAN MANAGE permission. To limit who can read the logs to only users with the CAN MANAGE permission, set spark.databricks.acl.needAdminPermissionToViewLogs
to true
.
See Spark configuration to learn how to add Spark properties to a cluster configuration.
Default privileges for jobs
Jobs have the following privileges set by default:
- The creator of the job is granted the IS OWNER permission.
- Workspace admins are granted the CAN MANAGE permission.
- The creator of the job is set for Run as.
Admin permissions for jobs
By default, workspace admins can change the job owner or Run as configuration to any user or service principal in the workspace. Account admins can configure the RestrictWorkspaceAdmins
setting to change this behavior. See Restrict workspace admins.
How do jobs interact with Unity Catalog permissions?
Jobs run as the identity of the user in the Run as setting. This identity is evaluated against permission grants for the following:
- Unity Catalog-managed assets, including tables, volumes, models, and views.
- Legacy table access control lists (ACLs) for assets registered in the legacy Hive metastore.
- ACLs for compute, notebooks, queries, and other workspace assets.
- Databricks secrets. See Secret management.
Note
Unity Catalog grants and legacy table ACLs require compatible compute access modes. See Configure compute for jobs.
SQL tasks and permissions
The file task is the only SQL task type to respect the Run as identity fully.
SQL queries, alerts, and legacy dashboard tasks respect configured sharing settings.
- Run as owner: Runs of the scheduled SQL task always use the identity of the owner of the configured SQL asset.
- Run as viewer: Runs of the scheduled SQL task always use the identity set in the job Run as field.
To learn more about query sharing settings, see Configure query permissions.
Example
The following scenario illustrates the interaction of SQL sharing settings and the job Run as setting:
- User A is the owner of the SQL query named
my_query
. - User A configures
my_query
with the sharing setting Run as owner. - User B schedules
my_query
as a task in a job namedmy_job
. - User B configures
my_job
to run with a service principle namedprod_sp
. - When
my_job
runs, it uses the identity for User A to runmy_query
.
Now assume that User B does not want this behavior. Starting from the existing configuration, the following occurs:
- User A changes the sharing setting for
my_query
to Run as viewer. - When
my_job
runs, it uses the identifyprod_sp
.
Configure identity for job runs
To change the Run as setting, you must have the CAN MANAGE or IS OWNER permission on the job.
You can set the Run as setting to yourself or any service principal in the workspace on which you have the Service Principal User entitlement.
To configure the Run as setting for a job in the workspace UI, select an existing job using the following steps:
- Click Workflows in the sidebar.
- In the Name column, click the job name.
- In the Job details side panel, click the pencil icon next to the Run as field.
- Search for and select a user or service principal.
- Click Save.
For more information on working with service principals, see the following:
- Manage service principals
- Roles for managing service principals
- List the service principals that you can use.
Best practices for jobs governance
Databricks recommends the following for all production jobs:
Assign job ownership to a service principal
If the user who owns a job leaves your organization, the job might fail. Use service principals to make jobs robust to employee churn.
By default, workspace admins can manage job permissions and reassign ownership if necessary.
Run production jobs using a service principal
Jobs run using the privileges of the job owner by default. If you assign ownership to a service principal, job runs use the permissions of the service principal.
Using service principals for production jobs allows you to restrict write permissions on production data. If you run jobs using a user's permissions, that user needs the same permissions to edit the production data required by the job.
Always use Unity Catalog-compatible compute configurations
Unity Catalog data governance requires that you use a supported compute configuration.
For jobs with classic compute, Databricks recommends shared access mode for supported workloads. Use single user access mode when required.
Delta Live Tables pipelines configured with Unity Catalog have some limitations. See Limitations.
Restrict permissions on production jobs
Users that trigger, stop, or restart job runs need the Can Manage Run permission.
Users that view the job configuration or monitor runs need the Can View permission.
Only grant Can Manage or Is Owner privileges to users trusted to modify production code.
Control access to a job
Job access control enables job owners and administrators to grant fine-grained permissions on jobs. The following permissions are available:
Note
Each permission includes the grants of permissions below it in the following table.
Permission | Grant |
---|---|
Is Owner | The identity used for Run as by default. |
Can Manage | Users can edit the job definition, including permissions. Users can pause and resume a schedule. |
Can Manage Run | Users can trigger and cancel job runs. |
Can View | Users can view job run results. |
For information on job permission levels, see Job ACLs.
Configure job permissions
To configure permissions for a job in the workspace UI, select an existing job using the following steps:
- Click Workflows in the sidebar.
- In the Name column, click the job name.
- In the Job details panel, click Edit permissions. The Permission Settings dialog appears.
- Click the Select User, Group or Service Principal… field and start typing a user, group, or service principal. The field searches all available identities in the workspace.
- Click Add.
- Click Save.
Manage the job owner
Only workspace admins can edit the job owner. Exactly one job owner must be assigned. Job owners can be users or service principals.