Azure Databricks data science and ML capabilities

Azure Databricks has a unified platform for the full data science (DS) and machine learning (ML) lifecycle, from raw data ingestion through feature engineering, model training, deployment, and production monitoring. Azure Databricks integrates with popular open-source ML frameworks, adding enterprise-grade governance, observability, and operational tooling, collectively known as MLOps.

This page lists major DS and ML capabilities, organized by workflow stage.

Exploratory data analysis

Azure Databricks simplifies exploratory data analysis (EDA) by providing interactive, collaborative, and AI-assisted tools for data scientists. Data scientists can explore data using natural language chat, UIs, or code, and they can collaborate using both real-time co-editing and Git-based code sharing.

Category	Features
User interface	Notebooks provide collaborative spaces for exploration, visualization, and documentation for EDA. Dashboards provide SQL and visualization-based EDA.
Collaboration	Notebooks, dashboards, and other workspace assets are all shareable and governed by workspace permissions. See for example Collaborate using Databricks notebooks. Notebooks and Git folders allow Git-based versioning and collaboration.

Prepare and serve features

Azure Databricks simplifies data for ML by unifying governance of data and ML workloads. With all data managed under Unity Catalog with fine-grained access controls, you can adjust data engineering and ML boundaries to fit your organization. Data can be prepared for ML using any data engineering tools such as Lakeflow Spark Declarative Pipelines. Features are managed in a Feature Store for both batch and real-time serving, with a single, governed source of truth for features.

Feature type	Features
Batch features	Feature tables in Unity Catalog store precomputed batch features with automatic lineage and governance. Teams discover and reuse existing features rather than rebuilding pipelines from scratch. Declarative features provide a new API for defining features which can then be used for batch or real-time feature computation.
Real-time features	When featurization inputs are only available at serving time, feature serving has on-demand feature computation to complement feature tables. Features are defined as functions, rather than precomputed. Declarative features provide a new API for defining features which can then be used for batch or real-time feature computation.

Train ML models

Azure Databricks has flexible tools for training ML and deep learning models. Pre-configured and customizable environments allow you to use custom ML libraries, and serverless CPU and GPU-accelerated compute resources allow scaling up and scaling out on demand.

Category	Features
Types of ML	Azure Databricks supports all types of ML, including: Classic ML: Supervised and unsupervised learning with scikit-learn, XGBoost, LightGBM, Apache Spark MLlib, and other ML frameworks Deep learning: Neural network training with PyTorch, TensorFlow, and Hugging Face Transformers, including distributed training across multiple GPUs Hyperparameter tuning: Automated search across algorithm and hyperparameter spaces using tools like Optuna and Ray
Compute	Serverless compute starts instantly for interactive notebooks and scheduled workflows, with automatic scaling and no cluster management. It supports both CPU and GPU-accelerated clusters. Classic compute has single-machine and cluster management, for both CPU and GPU workloads.
Environments and libraries	Serverless compute environments provide base environments that can be fully customized for ML. For classic compute, the Databricks Runtime for Machine Learning provides pre-configured cluster environments with major ML libraries pre-installed and tested together, for CPU clusters.

Track and manage experiments

Azure Databricks-managed MLflow provides the foundation for reproducible, auditable ML development. Its integrations with Unity Catalog and Git provide tracking and lineage for data and code assets. Each model version in the registry links back to the training run, dataset, environment, and git commit that produced it, providing a complete audit trail for any deployed model.

Category	Features
Experiment tracking	MLflow tracking logs parameters, metrics, and artifacts for every training run. Compare runs in the MLflow UI to identify the best-performing configuration.
Model registry	Models in Unity Catalog provides an MLflow model registry integrated with Unity Catalog. Versioned model artifacts are governed with lifecycle aliases (`Staging`, `Production`), access control, lineage, and cross-workspace sharing.
Reproducibility	Notebooks and code can be versioned using Databricks Git folders and integrated with any Git provider.

Evaluate and monitor

Azure Databricks provides flexible evaluation for training and continuous monitoring for production. Real-time serving logs to inference tables governed in Unity Catalog.

Category	Features
Evaluation	MLflow ML evaluation can be used to define metrics to log to MLflow, or MLflow tracking can log metrics computed using your custom framework.

MLOps and governance

Azure Databricks provides a full suite of tools for ML operations (MLOps) and governance. MLOps Stacks provides templates for enabling automated, repeatable promotion from development to production using infrastructure-as-code. Data, features, models, and endpoints are fully governed by Unity Catalog.

Category	Features
CI/CD for ML	MLOps Stacks, built on Declarative Automation Bundles, provides code-based management and deployment of ML infrastructure and workflows. This includes CI/CD templates for automating training, evaluation, and deployment.
Workflow orchestration	Lakeflow Jobs orchestrates multi-step ML workflows as scheduled or triggered pipelines.
Data and model asset governance	Unity Catalog provides unified governance for data, features, and registered models. Fine-grained access controls, lineage tracking, and audit logs apply to all assets.

Open source support

Azure Databricks provides full support for the open-source ML ecosystem.

You can use any open-source ML framework on Azure Databricks: scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, Hugging Face Transformers, Ray, and more. MLflow or your custom tools can store model artifacts in open formats that can be exported and run outside Azure Databricks.

MLflow is open-source, created by Azure Databricks and used by 10,000+ organizations. Your experiment tracking data, model artifacts, and pipeline definitions are stored in open formats.

Data and AI governance are built upon the open-source Unity Catalog APIs, and data storage is based upon the open Delta Lake format. Your feature data and training datasets remain in open, portable files.

Learn more

Last updated on 2026-06-12