Concepts: Data science and machine learning on Azure Databricks

Data science and machine learning (DS and ML) extract insight and build predictive models from data. DS and ML include both interactive exploration and modeling and automated production systems. Classic ML includes techniques like classification, regression, anomaly detection, forecasting, and recommendation.

Modern deep learning and generative AI (GenAI) methods are technically types of ML. This section covers deep learning.

The ML lifecycle

The ML lifecycle covers the end-to-end journey from raw data to a production model and back again through monitoring and retraining. Key stages include:

1. Scope the use case by defining the prediction target, success metrics, and production requirements.
2. Run exploratory data analysis (EDA) to understand data distributions, predictive signals, and data quality issues before modeling.
3. Prepare data and features, managed within a feature store.
4. Train models and track experiments, logging experiment metadata for analysis and for deployment.
5. Evaluate model quality against held-out data and stakeholder criteria.
6. Register, stage and test models before promoting to production.
7. Deploy to production in real-time endpoints or batch inference jobs.
8. Monitor and retrain to adapt models to changing data or user behavior.

See Machine learning lifecycle for a guide to each stage.

What is an ML platform?

An ML platform is the combined infrastructure, tooling, and governance layer that supports the full ML lifecycle, from raw data to production models. A well-designed ML platform connects data engineering, interactive data science and production ML in a single governed system.

Key components include:

• Data assets such as files, tables, processing pipelines, and feature stores
• Experimentation tools such as notebooks and visualizations, with simple collaboration and AI assistance
• Training infrastructure with customizable environments and flexible compute resources
• Deployment and monitoring infrastructure for batch and real-time serving, with production dashboards and alerts
• MLOps and governance tools for orchestration, CI/CD, lineage, access management and audit logging

Key governance capabilities include:

• Unified governance of data and ML assets. Learn more at What is Unity Catalog?.

• Unified security approach. Learn more at Databricks AI Security.

• Unified administration of data and ML tooling. Learn more at Administration.

Also see Azure Databricks data science and ML capabilities and Azure Databricks architecture.

Learn more

• Machine learning lifecycle - ML lifecycle stages and best practices
• Azure Databricks data science and ML capabilities - Azure Databricks ML capabilities by workflow stage
• AI on Databricks - Use cases, customers, and other resources