Migrate from classic compute to serverless compute

Migrate your workloads from classic compute to serverless compute. Serverless compute handles provisioning, scaling, runtime upgrades, and optimization automatically.

Most classic workloads can migrate with minimal or no code changes. For example, workloads that use df.cache() and df.persist() mostly migrate without code changes. This page focuses on those workloads. Certain workloads that depend on R or Scala notebooks require classic compute and can't migrate to serverless. For a full list of current limitations, see Serverless compute limitations.

Migrate to serverless manually

To migrate your workloads from classic compute to serverless compute, follow these steps:

1. Check prerequisites: Verify that your workspace, networking, and cloud storage access meet the requirements. See Prerequisites.

2. Update code: Make any necessary code and configuration changes. See Update your code.

3. Test your workloads: Validate compatibility and correctness before cutting over. See Test your workloads.

4. Choose a performance mode: Select the performance mode that best matches your workload requirements. See Choose a performance mode.

5. Migrate in phases: Roll out serverless incrementally, starting with new and low-risk workloads. See Migrate in phases.

6. Monitor costs: Track serverless DBU consumption and set up alerts. See Monitor costs.

Prerequisites

Before you begin migrating, you might need to update some legacy configurations in your workspace.

Confirm your workspace is in a supported region.

Update your code

The following sections list the code and configuration changes required to make your workloads compatible with serverless.

Data access

Legacy data access patterns are not supported on serverless. Update your code to use Unity Catalog instead.

# Classic
df = spark.read.csv("dbfs:/mnt/datalake/data.csv", header=True)
df.write.parquet("dbfs:/mnt/output/results")
df = spark.table("my_database.my_table")

# Serverless
df = spark.read.csv("/Volumes/main/sales/raw_data/data.csv", header=True)
df.write.parquet("/Volumes/main/analytics/output/results")
df = spark.table("main.my_database.my_table")  # three-level namespace

APIs and code

Certain APIs and code patterns are not supported on serverless. Reference this table to see if your code needs to be updated.

from pyspark.sql import functions as F

# sc.parallelize + rdd.map
# Classic:  rdd = sc.parallelize([1, 2, 3]); rdd.map(lambda x: x * 2).collect()
df = spark.createDataFrame([(1,), (2,), (3,)], ["value"])
result = df.select((F.col("value") * 2).alias("value")).collect()

# rdd.flatMap
# Classic:  sc.parallelize(["hello world"]).flatMap(lambda l: l.split(" ")).collect()
df = spark.createDataFrame([("hello world",)], ["line"])
words = df.select(F.explode(F.split("line", " ")).alias("word")).collect()

# rdd.groupByKey
# Classic:  rdd.groupByKey().mapValues(list).collect()
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "value"])
grouped = df.groupBy("key").agg(F.collect_list("value").alias("values")).collect()

# rdd.mapPartitions → applyInPandas
import pandas as pd
def process_group(pdf: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({"total": [pdf["id"].sum()]})
result = (spark.range(100).repartition(4)
    .groupBy(F.spark_partition_id())
    .applyInPandas(process_group, schema="total long").collect())

# sc.textFile → spark.read.text
df = spark.read.text("/Volumes/catalog/schema/volume/file.txt")

from pyspark.sql.functions import broadcast

# sc.broadcast → broadcast join
result = main_df.join(broadcast(lookup_df), "key")

# sc.accumulator → DataFrame aggregation
total = df.agg(F.sum("amount")).collect()[0][0]

# sqlContext.sql → spark.sql
result = spark.sql("SELECT * FROM main.db.table")

SQL behavior differences

Serverless compute enables ANSI mode by default. Databricks Runtime enables ANSI mode by default in version 17.0 and above. If your classic compute runs an earlier version, some queries that succeed there raise runtime errors on serverless.

The migration agent adjusts code for ANSI-mode behavior automatically. If you migrate manually, review the following operations, which most often change behavior:

To return NULL instead of raising an error, replace the operation with its try_* equivalent, such as try_cast, try_add, try_divide, or try_parse_url. Databricks recommends this approach instead of disabling ANSI mode, because ANSI mode surfaces data quality issues that a silent NULL hides.

To opt out of ANSI mode for casts, arithmetic, and parsing, set spark.sql.ansi.enabled to false. Serverless compute applies ANSI rules to implicit casts on table inserts regardless of this setting, because spark.sql.storeAssignmentPolicy cannot be changed on serverless compute. For a full description of ANSI behavior, see ANSI compliance in Databricks Runtime.

Libraries and environments

You can manage libraries and environments at the workspace level using base environments and at the notebook level using the notebook's serverless environment.

Pin Python packages in requirements.txt for reproducible environments. See Specify Python package versions.

Streaming

Streaming workloads are supported on serverless, but certain triggers are not supported. Update your code to use the supported triggers.

For continuous streaming, migrate to Spark Declarative Pipelines in continuous mode or use continuous-schedule jobs with AvailableNow. For large sources, set maxFilesPerTrigger or maxBytesPerTrigger to prevent out-of-memory errors.

# Classic (not supported on serverless — default trigger is ProcessingTime)
query = df.writeStream.format("delta").outputMode("append").start()

# Serverless (explicit AvailableNow trigger)
query = (df.writeStream.format("delta").outputMode("append")
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .start(output_path))
query.awaitTermination()

# With OOM prevention for large sources
query = (spark.readStream.format("delta")
    .option("maxFilesPerTrigger", 100)
    .option("maxBytesPerTrigger", "10g")
    .load(input_path)
    .writeStream.format("delta")
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .start(output_path))

Test your workloads

1. Quick compatibility test: Run the workload on classic compute with Standard access mode and Databricks Runtime 14.3 or above. If the run succeeds, the workload can migrate to serverless without any code changes.

2. A/B comparison (recommended for production): Run the same workload on classic (control) and serverless (experiment). Diff output tables and verify correctness. Iterate until outputs match.

3. Temporary configs: You can temporarily set supported Spark configs during testing. Remove them once stable.

Choose a performance mode

Serverless jobs and pipelines support two performance modes: standard and performance-optimized. The performance mode you choose depends on your workload requirements.

Migrate in phases

1. New workloads: Start all new notebooks and jobs on serverless.

2. Low-risk workloads: Migrate PySpark/SQL workloads already on standard access mode and Databricks Runtime 14.3 or above.

3. Complex workloads: Migrate workloads needing code changes (RDD rewrites, DBFS updates, trigger fixes).

4. Remaining workloads: Review periodically as capabilities expand.

Monitor costs

Serverless billing is based on DBU consumption, not cluster uptime. Validate cost expectations with representative workloads before migrating at scale. For tools and strategies to monitor serverless costs, see Monitor the cost of serverless compute.

Additional resources

• Best practices for serverless compute: Optimization tips for serverless workloads

• Serverless compute limitations: Full list of current limitations and unsupported features

• Configure the serverless environment: Manage libraries and dependencies

• Supported Spark configurations: Spark configs available on serverless

• Spark Connect vs. classic Spark: Behavioral differences in serverless architecture

• Cache DataFrames on serverless compute : Cache DataFrames with df.cache() and df.persist()

• Serverless network security: NCCs, Private Link, and firewall configuration

• Serverless compute release notes: Track new capabilities as they ship

• Unity Catalog upgrade guide: Migrate from Hive Metastore to Unity Catalog

You can also refer to the following blog posts for more information:

• What is serverless computing?: Overview of serverless capabilities and customer results

• Evolution of data engineering: How serverless compute is transforming notebooks and Lakeflow jobs: How serverless powers Lakeflow Jobs and Pipelines