User-defined operators in Lakeflow Designer

Lakeflow Designer lets you create user-defined operators that appear directly in the canvas alongside built-in operators. Use them to extend Lakeflow Designer with your own business logic, calculations, or integrations.

User-defined operators come in following scope:

  • Shared: The operator is registered in the workspace, so you and other users can add it to any visual data prep file. Its logic is a Python function or a Unity Catalog function, described in YAML. Write it yourself. See How user-defined operators work.

Choose a shared operator type

Shared user-defined operators have three types:

  • python-run-function: A standalone YAML file with inline Python stored in the workspace. Best for DataFrame-level transformations and external integrations. Permissions are managed at the workspace file level.

  • uc-udf: Wraps a Unity Catalog scalar function. Best for column-level transformations. Access is governed by Unity Catalog permissions.

  • uc-udtf: Wraps a Unity Catalog table-valued function. Best for table-level transforms like ML clustering and aggregation. Access is governed by Unity Catalog permissions.

Feature python-run-function uc-udf uc-udtf
Example use case DataFrame transforms, API integrations, email notifications Column-level calculations (BMI, interest rates) ML clustering, aggregation across rows
Input DataFrames Single values Entire table, row by row
Output DataFrames Single value Table (multiple rows)
Requires Unity Catalog function No Yes Yes
Access governance Workspace file permissions Unity Catalog permissions (EXECUTE, USE SCHEMA) Unity Catalog permissions (EXECUTE, USE SCHEMA)
Supported languages Python only SQL or Python in a SQL wrapper SQL or Python in a SQL wrapper

How user-defined operators work

A shared user-defined operator consists of:

  • Operator logic: The code that runs when the operator executes. This is an inline Python run() function for python-run-function operators, or a Unity Catalog function for uc-udf and uc-udtf operators. See Write logic for Lakeflow Designer user-defined operators.

  • YAML configuration: Tells Lakeflow Designer how to present the operator in the UI, including the operator's name, description, input parameters, UI widgets, and ports. All operator types use the user-defined-operator-v0.1.0 schema. For a python-run-function operator, the YAML is a standalone workspace file that also holds the inline code. For a Unity Catalog function, the YAML is embedded in the function body. See User-defined operator YAML reference.

  • Registration file: An entry in .user_defined_operators.yaml that lets Lakeflow Designer discover the operator. See Register and deploy your operator to Lakeflow Designer.

Build a shared operator

To build a shared user-defined operator from start to finish:

  1. Choose an operator type. See Choose a shared operator type.

  2. Write and test the operator logic. See Write logic for Lakeflow Designer user-defined operators.

  3. Describe the operator in YAML: its name, configuration properties, UI widgets, and ports. See User-defined operator YAML reference.

  4. Register the operator in .user_defined_operators.yaml. See Register and deploy your operator to Lakeflow Designer.

  5. Grant permissions so other users can discover and run the operator. See Permissions.

  6. Use the operator. It appears in the Operators pane in Lakeflow Designer, alongside the built-in operators.

Each tutorial walks through these steps for one complete operator.

Register and deploy your operator to Lakeflow Designer

For your operator to appear in Lakeflow Designer, register it in a .user_defined_operators.yaml file:

  • Workspace level: Place the file in the root of your workspace to make the operator visible to all users.

  • User level: Place the file in your user home folder (/Workspace/Users/<user-name>/.user_defined_operators.yaml) to make operators visible only to you.

The operators: section supports file paths, Unity Catalog function references, and glob patterns. You can mix entry types:

operators:
  # File path (python-run-function operators)
  - /Workspace/Users/me/udos/my_operator.yaml
  # Glob pattern (registers all matching files)
  - /Workspace/Users/me/udos/transforms/*.yaml
  # UC function reference (uc-udf and uc-udtf operators)
  - catalog: my_catalog
    schema: my_schema
    functionName: my_function

Update or remove an operator

When you change an operator's code, refresh your user-defined operators to load the change. In the Operators tab of the menu, click Refresh icon..

  • If the operator keeps the same version, refreshing loads the updated code.

  • If the operator has a new version, the operator on the canvas prompts you to upgrade to it (or keep the current version) after you refresh.

To remove an operator from Lakeflow Designer, delete its entry from .user_defined_operators.yaml. For uc-udf and uc-udtf operators, you can also drop the underlying Unity Catalog function with DROP FUNCTION if you no longer need it.

Permissions

Permission Purpose
Read access to .user_defined_operators.yaml. Discover the operator.
Read access to the YAML file (python-run-function only). Load the operator definition.
EXECUTE on the Unity Catalog function (operators based on Unity Catalog functions only). Run the operator.
USE SCHEMA on the schema (operators based on Unity Catalog functions only). Access the schema where the function is created.
Other permissions Depending on your operator, users may require other permissions. For example, USE CONNECTION on a Unity Catalog connection for HTTP API calls.

Troubleshooting

Issue Solution
Operator doesn't appear in Lakeflow Designer. Check that .user_defined_operators.yaml exists and lists your function or file path. For python-run-function operators, verify the file path and that the YAML file is accessible.
Schema validation fails. Verify your YAML against the official schema at https://your-workspace.cloud.databricks.com/static/schemas/user-defined-operator-v0.1.0.json.
Permission denied. For operators based on Unity Catalog functions, verify users have EXECUTE on the function and USE SCHEMA on the schema. For python-run-function operators, verify users have read access to the YAML file.
python-run-function operator fails at runtime. Check that the run() function signature matches def run(config, inputs, spark). Verify that port names in the code match the YAML and that the return dictionary keys match output port name values.
UDTF returns wrong types. UDTF return types must be explicit; you can't reference input column types.

Additional resources

Explore the following tutorials:

Example Type Description
Compound interest calculator uc-udf Calculate future investment values using the compound interest formula.
K-means clustering uc-udtf Segment data into clusters using scikit-learn.
All UI widgets uc-udf Reference operator showcasing all available UI widgets.

To write the operator logic, see Write logic for Lakeflow Designer user-defined operators. For a complete reference to the YAML schema, see User-defined operator YAML reference.

  • Last updated on