Customize containers with Databricks Container Service

Databricks Container Services lets you specify a Docker image when you create compute. Some example use cases include:

• Library customization: you have full control over the system libraries you want installed.
• Golden container environment: your Docker image is a locked down environment that will never change.
• Docker CI/CD integration: you can integrate Azure Databricks with your Docker CI/CD pipelines.

You can also use Docker images to create custom deep learning environments on compute with GPU devices. For additional information about using GPU compute with Databricks Container Services, see Databricks Container Services on GPU compute.

For tasks to be executed each time the container starts, use an init script.

Requirements

• Your Azure Databricks workspace must have Databricks Container Services enabled.
• Your machine must be running a recent Docker daemon (one that is tested and works with Client/Server Version 18.03.0-ce) and the docker command must be available on your PATH.

Limitations

• Databricks Container Services is not supported on compute using shared access mode.
• Databricks Runtime for Machine Learning does not support Databricks Container Services.
• To access Volumes on Databricks Container Services, add the following configuration to the compute's Spark config field: spark.databricks.unityCatalog.volumes.enabled true.

Step 1: Build your base

Databricks recommends that you build your Docker base from a base that Databricks has built and tested. It is also possible to build your Docker base from scratch. This section describes the two options.

Option 1. Use a base built by Databricks

This example uses the 9.x tag for an image that will target a compute with runtime version Databricks Runtime 9.1 LTS and above:

FROM databricksruntime/standard:9.x
...

To specify additional Python libraries, such as the latest version of pandas and urllib, use the container-specific version of pip. For the databricksruntime/standard:9.x container, include the following:

RUN /databricks/python3/bin/pip install pandas
RUN /databricks/python3/bin/pip install urllib3

For the databricksruntime/standard:8.x container or lower, include the following:

RUN /databricks/conda/envs/dcs-minimal/bin/pip install pandas
RUN /databricks/conda/envs/dcs-minimal/bin/pip install urllib3

Base images are hosted on Docker Hub at https://hub.docker.com/u/databricksruntime. The Dockerfiles used to generate these bases are at https://github.com/databricks/containers.

Option 2. Build your own Docker base

You can also build your Docker base from scratch. The Docker image must meet these requirements:

• JDK 8u191 as Java on the system PATH
• bash
• iproute2 (ubuntu iproute)
• coreutils (ubuntu coreutils)
• procps (ubuntu procps)
• sudo (ubuntu sudo)
• Ubuntu Linux

To build your own image from scratch, you must create the virtual environment. You must also include packages that are built into Databricks compute, such as Python and R. To get started, you can use the appropriate base image:

• For R: databricksruntime/rbase
• For Python: databricksruntime/python
• For the minimal image built by Databricks: databricksruntime/minimal

You can also refer to the example Dockerfiles in GitHub.

Step 2: Push your base image

Push your custom base image to a Docker registry. This process is supported with the following registries:

• Docker Hub with no auth or basic auth.
• Azure Container Registry with basic auth.

Other Docker registries that support no auth or basic auth are also expected to work.

Step 3: Launch your compute

You can launch your compute using the UI or the API.

Launch your compute using the UI

1. On the Create compute page, specify a Databricks Runtime Version that supports Databricks Container Services.

2. Under Advanced options, select the Docker tab.

3. Select Use your own Docker container.

4. In the Docker Image URL field, enter your custom Docker image.

   Docker image URL examples:

   Registry	Tag format
   Docker Hub	<organization>/<repository>:<tag> (for example: databricksruntime/standard:latest)
   Azure Container Registry	<your-registry-name>.azurecr.cn/<repository-name>:<tag>

5. Select the authentication type. You can use secrets to store username and password authentication values. See Use secrets for authentication.

Launch your compute using the API

1. Generate an API token.

2. Use the Databricks CLI to launch a compute with your custom Docker base.

   databricks clusters create \
   --cluster-name <cluster-name> \
   --node-type-id Standard_DS3_v2 \
   --json '{
     "num_workers": 0,
     "docker_image": {
       "url": "databricksruntime/standard:latest",
       "basic_auth": {
         "username": "<docker-registry-username>",
         "password": "<docker-registry-password>"
       }
     },
     "spark_version": "14.3.x-scala2.12"
   }'
   

   basic_auth requirements depend on your Docker image type:

   • For public Docker images, do not include the basic_auth field.
   • For private Docker images, you must include the basic_auth field, using a service principal ID and password as the username and password.
   • For Azure Container Registry, you must set the basic_auth field to the ID and password for a service principal. See Azure Container Registry service principal authentication documentation for information about creating the service principal.
   • You can also use a secret to store authentication information. See Use secrets for authentication.

Use an init script

Databricks Container Services enable customers to include init scripts in the Docker container. In most cases, you should avoid init scripts and instead make customizations through Docker directly (using the Dockerfile). However, certain tasks must be executed when the container starts, instead of when the container is built. Use an init script for these tasks.

For example, suppose you want to run a security daemon inside a custom container. Install and build the daemon in the Docker image through your image building pipeline. Then, add an init script that starts the daemon. In this example, the init script would include a line like systemctl start my-daemon.

In the API, you can specify init scripts as part of the compute spec as follows. For more information, see the Clusters API.

"init_scripts": [
    {
        "file": {
            "destination": "file:/my/local/file.sh"
        }
    }
]

For Databricks Container Services images, you can also store init scripts in cloud storage.

The following steps take place when you launch a compute that uses Databricks Container Services:

1. VMs are acquired from the cloud provider.
2. The custom Docker image is downloaded from your repo.
3. Azure Databricks creates a Docker container from the image.
4. Databricks Runtime code is copied into the Docker container.
5. The init scripts are executed. See What are init scripts?.

Azure Databricks ignores the Docker CMD and ENTRYPOINT primitives.

Use secrets for authentication

Databricks Container Service supports using secrets for authentication. When creating your compute resource, instead of entering your plain text username or password, enter your secret using the {{secrets/<scope-name>/<dcs-secret>}} format. For information on creating secrets, see Secrets.

Enable Container Services

To use custom containers on your compute, a workspace admin must enable Databricks Container Services.

Workspace admins can enable Databricks Container Service using the Databricks CLI. In a JSON request body, specify enableDcs to true, as in the following example:

databricks workspace-conf set-status \
--json '{"enableDcs": "true"}'