Parallelize Hyperopt hyperparameter tuning
Note
The open-source version of Hyperopt is no longer being maintained.
Hyperopt will no longer be pre-installed on Databricks Runtime ML 17.0 and above. Azure Databricks recommends using Optuna instead for a similar experience and access to more up-to-date hyperparameter tuning algorithms.
This notebook shows how to use Hyperopt to parallelize hyperparameter tuning calculations. It uses the SparkTrials class to automatically distribute calculations across the cluster workers. It also illustrates automated MLflow tracking of Hyperopt runs so you can save the results for later.
Parallelize hyperparameter tuning with automated MLflow tracking notebook
After you perform the actions in the last cell in the notebook, your MLflow UI should display:
