Pipeline task for jobs

Lakeflow Jobs provides a procedural approach to defining relationships between tasks. Lakeflow pipelines provide a declarative approach to defining relationships between datasets and transformations. Schedule a triggered pipeline to run as a task in a job, using the Jobs UI, the Lakeflow pipelines UI, or SQL.

Configure a pipeline task with the Jobs UI

Lakeflow pipelines manage all configurations for source code and compute in the pipeline definition.

To add a pipeline to a job, complete the following steps:

1. Create and name a new task and select pipeline for the Type.

2. In the Pipeline drop-down menu, select an existing pipeline. The pipeline must be a triggered pipeline. Continuous pipelines are not supported as a job task.

3. You can optionally trigger a full refresh on the pipeline.

4. You can optionally set parameter overrides in the Parameters field. See Parameters.

5. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.

To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Database Table Sync pipeline

The Database Table Sync pipeline task is a pipeline task that runs the pipeline maintaining a Lakebase synced table . Use it to refresh a synced table on a schedule, or when the source Unity Catalog table changes, so that operational applications read current data from Lakebase Postgres.

In the task Type drop-down, Database Table Sync pipeline appears under Ingestion and Transformation. Selecting it configures a pipeline task. In the Pipeline field, choose the pipeline associated with the synced table you want to refresh.

Lakebase has two offerings. The synced table you want to refresh determines which set of instructions to follow:

• Lakebase Autoscaling is the offering for new instances.

• Lakebase Provisioned is the original offering.

Ingestion pipeline

The Ingestion pipeline task is a pipeline task that runs an ingestion pipeline. In the task Type drop-down, choosing Ingestion pipeline starts the Add data wizard, which creates a pipeline task for an ingestion pipeline.

The wizard's first page asks for the data source. The following pages depend on the source you select.

Schedule a pipeline with the pipeline UI

Adding a schedule to a pipeline creates a job with a single pipeline task. You can only configure time-based schedule triggers using this UI. For more advanced triggering options, see Configure a pipeline task with the Jobs UI.

Configure a pipeline task in a scheduled job using the pipeline UI by completing the following steps:

1. In your workspace, click Jobs & Pipelines in the sidebar.

2. Click on the pipeline Name. The pipeline UI appears.

3. Click Schedule.

   • If no schedule exists for the pipeline, the New schedule dialog appears.
   • If one or more schedules already exist, click Add schedule.

4. Enter a unique name for the job in the Job name field.

5. (Optional) Update the schedule frequency.

   • Select Advanced for more verbose options including cron syntax.

6. (Optional) Under More options, configure one or more email addresses to receive alerts on pipeline start, success, or failure.

7. Click Create.

   Note

   If the pipeline is included in one or more scheduled jobs, the Schedule button shows the number of existing schedules, for example, Schedule (5).

Add a schedule to a materialized view or streaming table in Databricks SQL

Materialized views and streaming tables defined in Databricks SQL support time-based scheduling specified in CREATE or ALTER commands.

For details, see the following articles:

• CREATE MATERIALIZED VIEW
• CREATE STREAMING TABLE
• ALTER MATERIALIZED VIEW
• ALTER STREAMING TABLE