Workspace Model Registry webhooks

Webhooks enable you to listen for Workspace Model Registry events so your integrations can automatically trigger actions. You can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows. For example, you can trigger CI builds when a new model version is created or notify your team members through Slack each time a model transition to production is requested.

Webhooks are available through the Databricks REST API or the Python client databricks-registry-webhooks on PyPI.

You can specify a webhook to trigger upon one or more of these events:

• MODEL_VERSION_CREATED: A new model version was created for the associated model.
• MODEL_VERSION_TRANSITIONED_STAGE: A model version's stage was changed.
• TRANSITION_REQUEST_CREATED: A user requested a model version's stage be transitioned.
• COMMENT_CREATED: A user wrote a comment on a registered model.
• REGISTERED_MODEL_CREATED: A new registered model was created. This event type can only be specified for a registry-wide webhook, which can be created by not specifying a model name in the create request.
• MODEL_VERSION_TAG_SET: A user set a tag on the model version.
• MODEL_VERSION_TRANSITIONED_TO_STAGING: A model version was transitioned to staging.
• MODEL_VERSION_TRANSITIONED_TO_PRODUCTION: A model version was transitioned to production.
• MODEL_VERSION_TRANSITIONED_TO_ARCHIVED: A model version was archived.
• TRANSITION_REQUEST_TO_STAGING_CREATED: A user requested a model version be transitioned to staging.
• TRANSITION_REQUEST_TO_PRODUCTION_CREATED: A user requested a model version be transitioned to production.
• TRANSITION_REQUEST_TO_ARCHIVED_CREATED: A user requested a model version be archived.

There are two types of webhooks based on their trigger targets:

• Webhooks with HTTP endpoints (HTTP registry webhooks): Send triggers to an HTTP endpoint.
• Webhooks with job triggers (job registry webhooks): Trigger a job in an Azure Databricks workspace. If IP allowlisting is enabled in the job's workspace, you must allowlist the workspace IPs of the model registry. See IP allowlisting for job registry webhooks for more information.

There are also two types of webhooks based on their scope, with different access control requirements:

• Model-specific webhooks: The webhook applies to a specific registered model. You must have CAN MANAGE permissions on the registered model to create, modify, delete, or test model-specific webhooks.
• Registry-wide webhooks: The webhook is triggered by events on any registered model in the workspace, including the creation of a new registered model. To create a registry-wide webhook, omit the model_name field on creation. You must have workspace admin permissions to create, modify, delete, or test registry-wide webhooks.

Each event trigger has minimal fields included in the payload for the outgoing request to the webhook endpoint.

• Sensitive information like artifact path location is excluded. Users and principals with appropriate ACLs can use client or REST APIs to query the Model Registry for this information.
• Payloads are not encrypted. See Security for information on how to validate that Azure Databricks is the source of the webhook.
• The text field facilitates Slack integration. To send a Slack message, provide a Slack webhook endpoint as the webhook URL.

The payload for a job registry webhook depends on the type of job and is sent to the jobs/run-now endpoint in the target workspace.

Single-task jobs have one of three payloads based on the task type.

Notebook and Python wheel jobs have a JSON payload with a parameter dictionary that contains a field event_message.

{
  "job_id": 1234567890,
  "notebook_params": {
    "event_message": "<Webhook Payload>"
  }
}

Python, JAR, and Spark submit jobs have a JSON payload with a parameter list.

{
  "job_id": 1234567890,
  "python_params": ["<Webhook Payload>"]
}

All other types of jobs have a JSON payload with no parameters.

{
  "job_id": 1234567890
}

Multi-task jobs have a JSON payload with all parameters populated to account for different task types.

{
  "job_id": 1234567890,
  "notebook_params": {
    "event_message": "<Webhook Payload>"
  },
  "python_named_params": {
    "event_message": "<Webhook Payload>"
  },
  "jar_params": ["<Webhook Payload>"],
  "python_params": ["<Webhook Payload>"],
  "spark_submit_params": ["<Webhook Payload>"]
}

Response

POST
/your/endpoint/for/event/model-versions/stage-transition
--data {
  "event": "MODEL_VERSION_TRANSITIONED_STAGE",
  "webhook_id": "c5596721253c4b429368cf6f4341b88a",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "to_stage": "Production",
  "from_stage": "None",
  "text": "Registered model 'someModel' version 8 transitioned from None to Production."
}

Response

POST
/your/endpoint/for/event/model-versions/tag-set
--data {
  "event": "MODEL_VERSION_TAG_SET",
  "webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "tags": [{"key":"key1","value":"value1"},{"key":"key2","value":"value2"}],
  "text": "example@yourdomain.com set version tag(s) 'key1' => 'value1', 'key2' => 'value2' for registered model 'someModel' version 8."
}

Response

POST
/your/endpoint/for/event/comments/create
--data {
  "event": "COMMENT_CREATED",
  "webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "comment": "Raw text content of the comment",
  "text": "A user commented on registered model 'someModel' version 8."
}

For security, Azure Databricks includes the X-Databricks-Signature in the header computed from the payload and the shared secret key associated with the webhook using the HMAC with SHA-256 algorithm.

In addition, you can include a standard Authorization header in the outgoing request by specifying one in the HttpUrlSpec of the webhook.

If a shared secret is set, the payload recipient should verify the source of the HTTP request by using the shared secret to HMAC-encode the payload, and then comparing the encoded value with the X-Databricks-Signature from the header. This is particularly important if SSL certificate validation is disabled (that is, if the enable_ssl_verification field is set to false).

import hmac
import hashlib
import json

secret = shared_secret.encode('utf-8')
signature_key = 'X-Databricks-Signature'

def validate_signature(request):
  if not request.headers.has_key(signature_key):
    raise Exception('No X-Signature. Webhook not be trusted.')

  x_sig = request.headers.get(signature_key)
  body = request.body.encode('utf-8')
  h = hmac.new(secret, body, hashlib.sha256)
  computed_sig = h.hexdigest()

  if not hmac.compare_digest(computed_sig, x_sig.encode()):
    raise Exception('X-Signature mismatch. Webhook not be trusted.')

If an Authorization header is set, clients should verify the source of the HTTP request by verifying the bearer token or authorization credentials in the Authorization header.

To use a webhook that triggers job runs in a different workspace that has IP allowlisting enabled, you must allowlist the region NAT IP where the webhook is located to accept incoming requests.

If the webhook and the job are in the same workspace, you do not need to add any IPs to your allowlist.

If your job is located in an Azure multitenant region, see Azure Databricks control plane addresses. For all other regions, contact your account team to identify the IPs you need to allowlist.

If audit logging is enabled for your workspace, the following events are included in the audit logs:

• Create webhook
• Update webhook
• List webhook
• Delete webhook
• Test webhook
• Webhook trigger

For webhooks with HTTP endpoints, the HTTP request sent to the URL specified for the webhook along with the URL and enable_ssl_verification values are logged.

For webhooks with job triggers, the job_id and workspace_url values are logged.

This section includes:

• HTTP registry webhook workflow example.
• job registry webhook workflow example.
• list webhooks example.
• two example notebooks: one illustrating the REST API, and one illustrating the Python client.

When an HTTPS endpoint is ready to receive the webhook event request, you can create a webhook using the webhooks Databricks REST API. For example, the webhook's URL can point to Slack to post messages to a channel.

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"model_name": "<model-name>",
  "events": ["MODEL_VERSION_CREATED"],
  "description": "Slack notifications",
  "status": "TEST_MODE",
  "http_url_spec": {
    "url": "https://hooks.slack.com/services/...",
    "secret": "anyRandomString"
    "authorization": "Bearer AbcdEfg1294"}}' https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create

from databricks_registry_webhooks import RegistryWebhooksClient, HttpUrlSpec

http_url_spec = HttpUrlSpec(
  url="https://hooks.slack.com/services/...",
  secret="secret_string",
  authorization="Bearer AbcdEfg1294"
)
http_webhook = RegistryWebhooksClient().create_webhook(
  model_name="<model-name>",
  events=["MODEL_VERSION_CREATED"],
  http_url_spec=http_url_spec,
  description="Slack notifications",
  status="TEST_MODE"
)

Response

{"webhook": {
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status":"TEST_MODE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}}}

You can also create an HTTP registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.

The previous webhook was created in TEST_MODE, so a mock event can be triggered to send a request to the specified URL. However, the webhook does not trigger on a real event. The test endpoint returns the received status code and body from the specified URL.

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/test

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().test_webhook(
  id="1234567890"
)

Response

{
 "status":200,
 "body":"OK"
}

To enable the webhook for real events, set its status to ACTIVE through an update call, which can also be used to change any of its other properties.

$ curl -X PATCH -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890", "status": "ACTIVE"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/update

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().update_webhook(
  id="1234567890",
  status="ACTIVE"
)

Response

{"webhook": {
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status": "ACTIVE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}}}

To disable the webhook, set its status to DISABLED (using a similar update command as above), or delete it.

$ curl -X DELETE -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/delete

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().delete_webhook(
  id="1234567890"
)

Response

{}

The workflow for managing job registry webhooks is similar to HTTP registry webhooks, with the only difference being the job_spec field that replaces the http_url_spec field.

With webhooks, you can trigger jobs in the same workspace or in a different workspace. The workspace is specified using the optional parameter workspace_url. If no workspace_url is present, the default behavior is to trigger a job in the same workspace as the webhook.

• An existing job.
• A personal access token. Note that access tokens can only be read by the MLflow service and cannot be returned by Azure Databricks users in the Model Registry API.

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>",
  "events": ["TRANSITION_REQUEST_CREATED"],
  "description": "Job webhook trigger",
  "status": "TEST_MODE",
  "job_spec": {
    "job_id": "1",
    "workspace_url": "https://my-databricks-workspace.com",
    "access_token": "dapi12345..."}}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create

from databricks_registry_webhooks import RegistryWebhooksClient, JobSpec

job_spec = JobSpec(
  job_id="1",
  workspace_url="https://my-databricks-workspace.com",
  access_token="dapi12345..."
)
job_webhook = RegistryWebhooksClient().create_webhook(
  model_name="<model-name>",
  events=["TRANSITION_REQUEST_CREATED"],
  job_spec=job_spec,
  description="Job webhook trigger",
  status="TEST_MODE"
)

Response

{"webhook": {
   "id":"1234567891",
   "creation_timestamp":1591440826026,
   "last_updated_timestamp":1591440826026,
   "status":"TEST_MODE",
   "events":["TRANSITION_REQUEST_CREATED"],
   "job_spec": {
     "job_id": "1",
     "workspace_url": "https://my-databricks-workspace.com"
}}}

You can also create a job registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.

$ curl -X GET -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>"}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/list

from databricks_registry_webhooks import RegistryWebhooksClient

webhooks_list = RegistryWebhooksClient().list_webhooks(model_name="<model-name>")

Response

{"webhooks": [{
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status": "ACTIVE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}},
{
   "id":"1234567891",
   "creation_timestamp":1591440826026,
   "last_updated_timestamp":1591440826026,
   "status":"TEST_MODE",
   "events":["TRANSITION_REQUEST_CREATED"],
   "job_spec": {
     "job_id": "1",
     "workspace_url": "https://my-databricks-workspace.com"
}}]}

Get notebook

Get notebook