使用 Python 配置自动化 ML 试验Configure automated ML experiments in Python

本指南介绍如何通过 Azure 机器学习 SDK 定义各种自动机器学习试验的配置设置。In this guide, learn how to define various configuration settings of your automated machine learning experiments with the Azure Machine Learning SDK. 自动化机器学习将自动选择算法和超参数,并生成随时可用于部署的模型。Automated machine learning picks an algorithm and hyperparameters for you and generates a model ready for deployment. 可以使用多个选项来配置自动化机器学习试验。There are several options that you can use to configure automated machine learning experiments.

若要查看自动化机器学习试验的端到端示例,请参阅教程:使用自动化机器学习训练分类模型For an end to end example of an automated machine learning experiment, see Tutorial: Train a classification model with automated machine learning.

自动化机器学习提供的配置选项:Configuration options available in automated machine learning:

  • 选择试验类型:分类、回归或时序预测Select your experiment type: Classification, Regression, or Time Series Forecasting
  • 数据源、格式和提取数据Data source, formats, and fetch data
  • 选择计算目标:本地或远程Choose your compute target: local or remote
  • 自动化机器学习试验设置Automated machine learning experiment settings
  • 运行自动化机器学习试验Run an automated machine learning experiment
  • 探索模型指标Explore model metrics
  • 注册和部署模型Register and deploy model

如果你更喜欢无代码体验,还可以在 Azure 机器学习工作室中创建自动化学习试验If you prefer a no code experience, you can also Create your automated machine learning experiments in Azure Machine Learning studio.

先决条件Prerequisites

在本文中,你需要:For this article you need,

选择试验类型Select your experiment type

在开始试验之前,应确定要解决的机器学习问题类型。Before you begin your experiment, you should determine the kind of machine learning problem you are solving. 自动化机器学习支持 classificationregressionforecasting 任务类型。Automated machine learning supports task types of classification, regression, and forecasting. 详细了解任务类型Learn more about task types.

下面的代码使用 AutoMLConfig 构造函数中的 task 参数将试验类型指定为 classificationThe following code uses the task parameter in the AutoMLConfig constructor to specify the experiment type as classification.

from azureml.train.automl import AutoMLConfig

# task can be one of classification, regression, forecasting
automl_config = AutoMLConfig(task = "classification")

数据源和格式Data source and format

自动化机器学习支持驻留在本地桌面上或云中(例如 Azure Blob 存储)的数据。Automated machine learning supports data that resides on your local desktop or in the cloud such as Azure Blob Storage. 数据可以读入 Pandas 数据帧或 Azure 机器学习 TabularDataset 中 。The data can be read into a Pandas DataFrame or an Azure Machine Learning TabularDataset. 了解有关数据集的详细信息Learn more about datasets.

机器学习中对训练数据的要求:Requirements for training data in machine learning:

  • 数据必须为表格格式。Data must be in tabular form.
  • 要预测的值(目标列)必须位于数据中。The value to predict, target column, must be in the data.

对于远程试验,必须能够从远程计算访问训练数据。For remote experiments, training data must be accessible from the remote compute. AutoML 仅在处理远程计算时才接受 Azure 机器学习 TabularDatasetAutoML only accepts Azure Machine Learning TabularDatasets when working on a remote compute.

Azure 机器学习数据集公开的功能可以:Azure Machine Learning datasets expose functionality to:

  • 轻松地将数据从静态文件或 URL 源传输到工作区。Easily transfer data from static files or URL sources into your workspace.
  • 在云计算资源上运行时,使数据可用于训练脚本。Make your data available to training scripts when running on cloud compute resources. 有关使用 Dataset 类将数据装载到远程计算目标的示例,请参阅如何使用数据集进行训练See How to train with datasets for an example of using the Dataset class to mount data to your remote compute target.

下面的代码从一个 Web URL 创建 TabularDataset。The following code creates a TabularDataset from a web url. 有关从其他源(例如本地文件和数据存储)创建数据集的代码示例,请参阅创建 TabularDatasetSee Create a TabularDatasets for code examples on how to create datasets from other sources like local files and datastores.

from azureml.core.dataset import Dataset
data = "https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv"
dataset = Dataset.Tabular.from_delimited_files(data)

对于本地计算试验,我们建议使用 pandas 数据帧以提高处理速度。For local compute experiments, we recommend pandas dataframes for faster processing times.

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv("your-local-file.csv")
train_data, test_data = train_test_split(df, test_size=0.1, random_state=42)
label = "label-col-name"

训练、验证和测试数据Training, validation, and test data

可以直接在 AutoMLConfig 构造函数中指定单独的训练数据和验证数据集。You can specify separate training data and validation data sets directly in the AutoMLConfig constructor. 详细了解如何配置数据拆分和交叉验证(针对 AutoML 试验)。Learn more about how to configure data splits and cross validation for your AutoML experiments.

如果未显式指定 validation_datan_cross_validation 参数,则自动化 ML 将应用默认技术来决定如何执行验证。If you do not explicitly specify a validation_data or n_cross_validation parameter, automated ML applies default techniques to determine how validation is performed. 此决定依赖于分配给 training_data 参数的数据集中的行数。This determination depends on the number of rows in the dataset assigned to your training_data parameter.

训练数据大小Training data size 验证技术Validation technique
大于 20,000 行Larger than 20,000 rows 将应用训练/验证数据拆分。Train/validation data split is applied. 默认行为是将初始训练数据集的 10% 用作验证集。The default is to take 10% of the initial training data set as the validation set. 然后,该验证集将用于指标计算。In turn, that validation set is used for metrics calculation.
小于 20,000 行Smaller than 20,000 rows 将应用交叉验证方法。Cross-validation approach is applied. 默认折数取决于行数。The default number of folds depends on the number of rows.
如果数据集小于 1,000 行,则使用 10 折。If the dataset is less than 1,000 rows, 10 folds are used.
如果行数在 1,000 到 20,000 之间,则使用 3 折。If the rows are between 1,000 and 20,000, then three folds are used.

此时,你需要提供自己的 测试数据 来进行模型评估。At this time, you need to provide your own test data for model evaluation. 如果需要通过代码示例来演示如何引入你自己的测试数据进行模型评估,请参阅 此 Jupyter 笔记本Test 节。For a code example of bringing your own test data for model evaluation see the Test section of this Jupyter notebook.

用于运行试验的计算环境Compute to run experiment

接下来,确定要在何处训练模型。Next determine where the model will be trained. 自动化机器学习训练试验可根据以下计算选项运行。An automated machine learning training experiment can run on the following compute options. 了解本地和远程计算选项的优缺点Learn the pros and cons of local and remote compute options.

  • 本地 台式机或便携式计算机等本地计算机 – 如果数据集较小,并且你仍然处于探索阶段,则通常使用此选项。Your local machine such as a local desktop or laptop – Generally when you have a small dataset and you are still in the exploration stage. 有关本地计算示例,请参阅此笔记本See this notebook for a local compute example.

  • 云中的 远程 计算机 – Azure 机器学习托管计算是一个托管服务,可用于在 Azure 虚拟机的群集上训练机器学习模型。A remote machine in the cloud – Azure Machine Learning Managed Compute is a managed service that enables the ability to train machine learning models on clusters of Azure virtual machines.

    有关使用 Azure 机器学习托管计算的远程示例,请参阅此笔记本See this notebook for a remote example using Azure Machine Learning Managed Compute.

  • Azure 订阅中的 Azure Databricks 群集An Azure Databricks cluster in your Azure subscription. 有关更多详细信息,可参阅为自动化 ML 设置 Azure Databricks 群集You can find more details in Set up an Azure Databricks cluster for automated ML. 有关包含 Azure Databricks 的示例 Notebook,请参阅此 GitHub 站点See this GitHub site for examples of notebooks with Azure Databricks.

配置试验设置Configure your experiment settings

可以使用多个选项来配置自动化机器学习试验。There are several options that you can use to configure your automated machine learning experiment. 通过实例化 AutoMLConfig 对象来设置这些参数。These parameters are set by instantiating an AutoMLConfig object. 有关参数的完整列表,请参阅 AutoMLConfig 类See the AutoMLConfig class for a full list of parameters.

示例包括:Some examples include:

  1. 使用 AUC 作为主要指标加权的分类实验,其中实验超时分钟数设置为 30 分钟,且包含 2 折交叉验证。Classification experiment using AUC weighted as the primary metric with experiment timeout minutes set to 30 minutes and 2 cross-validation folds.

        automl_classifier=AutoMLConfig(task='classification',
                                       primary_metric='AUC_weighted',
                                       experiment_timeout_minutes=30,
                                       blocked_models=['XGBoostClassifier'],
                                       training_data=train_data,
                                       label_column_name=label,
                                       n_cross_validations=2)
    
  2. 下面是设置为 60 分钟后结束的回归试验示例,其中包含 5 折交叉验证。The following example is a regression experiment set to end after 60 minutes with five validation cross folds.

       automl_regressor = AutoMLConfig(task='regression',
                                       experiment_timeout_minutes=60,
                                       allowed_models=['KNN'],
                                       primary_metric='r2_score',
                                       training_data=train_data,
                                       label_column_name=label,
                                       n_cross_validations=5)
    
  3. 预测任务需要额外的设置,请参阅自动训练时序预测模型一文来了解更多详细信息。Forecasting tasks require extra setup, see the Autotrain a time-series forecast model article for more details.

    time_series_settings = {
        'time_column_name': time_column_name,
        'time_series_id_column_names': time_series_id_column_names,
        'forecast_horizon': n_test_periods
    }
    
    automl_config = AutoMLConfig(task = 'forecasting',
                                 debug_log='automl_oj_sales_errors.log',
                                 primary_metric='normalized_root_mean_squared_error',
                                 experiment_timeout_minutes=20,
                                 training_data=train_data,
                                 label_column_name=label,
                                 n_cross_validations=5,
                                 path=project_folder,
                                 verbosity=logging.INFO,
                                 **time_series_settings)
    

支持的模型Supported models

在自动化和优化过程中,自动化机器学习会尝试各种模型和算法。Automated machine learning tries different models and algorithms during the automation and tuning process. 用户不需要指定算法。As a user, there is no need for you to specify the algorithm.

这三个不同的 task 参数值确定要应用的算法或模型的列表。The three different task parameter values determine the list of algorithms, or models, to apply. 使用 allowed_modelsblocked_models 参数通过要包含或排除的可用模型来进一步修改迭代。Use the allowed_models or blocked_models parameters to further modify iterations with the available models to include or exclude.

下表按任务类型汇总了支持的模型。The following table summarizes the supported models by task type.

备注

如果你计划将自动化 ML 创建的模型导出为 ONNX 模型,只有标有 * 的算法才能转换为 ONNX 格式。If you plan to export your auto ML created models to an ONNX model, only those algorithms indicated with an * are able to be converted to the ONNX format. 详细了解如何将模型转换为 ONNXLearn more about converting models to ONNX.

另请注意,ONNX 目前只支持分类和回归任务。Also note, ONNX only supports classification and regression tasks at this time.

分类Classification 回归Regression 时序预测Time Series Forecasting
逻辑回归*Logistic Regression* 弹性网络*Elastic Net* AutoARIMAAutoARIMA
Light GBM*Light GBM* Light GBM*Light GBM* ProphetProphet
梯度提升*Gradient Boosting* 梯度提升*Gradient Boosting* 弹性网络Elastic Net
决策树*Decision Tree* 决策树*Decision Tree* Light GBMLight GBM
K 最近的邻域*K Nearest Neighbors* K 最近的邻域*K Nearest Neighbors* 渐进提升Gradient Boosting
线性 SVC*Linear SVC* LARS Lasso*LARS Lasso* 决策树Decision Tree
支持矢量分类 (SVC)*Support Vector Classification (SVC)* 随机梯度下降 (SGD)*Stochastic Gradient Descent (SGD)* 决策树Decision Tree
随机林*Random Forest* 随机林Random Forest LARS LassoLARS Lasso
极端随机树*Extremely Randomized Trees* 极端随机树*Extremely Randomized Trees* 随机梯度下降 (SGD)Stochastic Gradient Descent (SGD)
Xgboost*Xgboost* Xgboost*Xgboost* 随机林Random Forest
平均感知器分类器Averaged Perceptron Classifier 在线梯度下降回归量Online Gradient Descent Regressor XgboostXgboost
Naive Bayes*Naive Bayes* 快速线性回归量Fast Linear Regressor ForecastTCNForecastTCN
随机梯度下降 (SGD)*Stochastic Gradient Descent (SGD)* NaiveNaive
线性 SVM 分类器*Linear SVM Classifier* SeasonalNaiveSeasonalNaive
平均值Average
SeasonalAverageSeasonalAverage
ExponentialSmoothingExponentialSmoothing

主要指标Primary Metric

primary metric 参数决定了将在模型训练期间用于优化的指标。The primary metric parameter determines the metric to be used during model training for optimization. 你可选择的可用指标取决于所选择的任务类型,下表显示了每种任务类型的有效主要指标。The available metrics you can select is determined by the task type you choose, and the following table shows valid primary metrics for each task type.

选择自动化机器学习要优化的主要指标取决于许多因素。Choosing a primary metric for automated machine learning to optimize depends on many factors. 建议主要考虑选择最能体现业务需求的指标。We recommend your primary consideration be to choose a metric which best represents your business needs. 然后考虑指标是否适用于数据集配置文件(数据大小、范围、类分布等)。Then consider if the metric is suitable for your dataset profile (data size, range, class distribution, etc.).

如需了解上述指标的具体定义,请参阅了解自动化机器学习结果集Learn about the specific definitions of these metrics in Understand automated machine learning results.

分类Classification 回归Regression 时序预测Time Series Forecasting
accuracy spearman_correlation spearman_correlation
AUC_weighted normalized_root_mean_squared_error normalized_root_mean_squared_error
average_precision_score_weighted r2_score r2_score
norm_macro_recall normalized_mean_absolute_error normalized_mean_absolute_error
precision_score_weighted

分类方案的主要指标Primary metrics for classification scenarios

对于类别偏斜严重(类别不均衡)的小型数据集或预期的指标值非常接近 0.0 或 1.0 时,发布的阈值指标(如 accuracyaverage_precision_score_weightednorm_macro_recallprecision_score_weighted)可能也不是最优的。Post thresholded metrics, like accuracy, average_precision_score_weighted, norm_macro_recall, and precision_score_weighted may not optimize as well for datasets which are small, have very large class skew (class imbalance), or when the expected metric value is very close to 0.0 or 1.0. 在这些情况下,对于主要指标,AUC_weighted 可能是更好的选择。In those cases, AUC_weighted can be a better choice for the primary metric. 自动机器学习完成后,可以根据最能满足你业务需求的指标选择所需模型。After automated machine learning completes, you can choose the winning model based on the metric best suited to your business needs.

指标Metric 示例用例Example use case(s)
accuracy 图像分类、情绪分析、流失预测Image classification, Sentiment analysis, Churn prediction
AUC_weighted 欺诈检测、图像分类、异常检测/垃圾邮件检测Fraud detection, Image classification, Anomaly detection/spam detection
average_precision_score_weighted 情绪分析Sentiment analysis
norm_macro_recall 流失预测Churn prediction
precision_score_weighted

回归方案的主要指标Primary metrics for regression scenarios

当要预测的值的范围涵盖多个数量级时,r2_scorespearman_correlation 等指标可以更好地体现模型的质量。Metrics like r2_score and spearman_correlation can better represent the quality of model when the scale of the value-to-predict covers many orders of magnitude. 例如,在薪水估算中,许多人的薪水在 2 万至 10 万美元之间,但某些人的薪水在 1 亿美元的范围中,这极大地拉大了总量级范围。For instance salary estimation, where many people have a salary of $20k to $100k, but the scale goes very high with some salaries in the $100M range.

在这种情况下,normalized_mean_absolute_errornormalized_root_mean_squared_error 会认为 2 万美元的预测误差对于薪水为 3 万美元的工人和薪水为 2000 万美元的工人是没有区别的。normalized_mean_absolute_error and normalized_root_mean_squared_error would in this case treat a $20k prediction error the same for a worker with a $30k salary as a worker making $20M. 但实际上,在 2000 万美元薪水的基础上增减 2 万美元和增减前的预测应该非常接近(相对差值很小,增减占比 0.1%),而在 3 万美元薪水的基础上增减 2 万美元和增减前的预测应该非常大(相对差值很大,增减占比 67%)。While in reality, predicting only $20k off from a $20M salary is very close (a small 0.1% relative difference), whereas $20k off from $30k is not close (a large 67% relative difference). 当要预测的值在相似的量级范围内时,normalized_mean_absolute_errornormalized_root_mean_squared_error 会比较有用。normalized_mean_absolute_error and normalized_root_mean_squared_error are useful when the values to predict are in a similar scale.

指标Metric 示例用例Example use case(s)
spearman_correlation
normalized_root_mean_squared_error 价格预测(房屋/产品/小费),查看分数预测Price prediction (house/product/tip), Review score prediction
r2_score 航空延迟、薪金估算、Bug 解决时间Airline delay, Salary estimation, Bug resolution time
normalized_mean_absolute_error

时序预测方案的主要指标Primary metrics for time series forecasting scenarios

请参阅上面的回归注释。See regression notes, above.

指标Metric 示例用例Example use case(s)
spearman_correlation
normalized_root_mean_squared_error 价格预测(预测)、库存优化、需求预测Price prediction (forecasting), Inventory optimization, Demand forecasting
r2_score 价格预测(预测)、库存优化、需求预测Price prediction (forecasting), Inventory optimization, Demand forecasting
normalized_mean_absolute_error

数据特征化Data featurization

在每个自动化机器学习实验中,数据都是 自动缩放和规范化,以帮助对不同规模上的特征敏感的某些算法。In every automated machine learning experiment, your data is automatically scaled and normalized to help certain algorithms that are sensitive to features that are on different scales. 此缩放和规范化称为特征化。This scaling and normalization is referred to as featurization. 有关更多详细信息和代码示例,请参阅 AutoML 中的特征化See Featurization in AutoML for more detail and code examples.

AutoMLConfig 对象中配置试验时,可以启用/禁用设置 featurizationWhen configuring your experiments in your AutoMLConfig object, you can enable/disable the setting featurization. 下表列出了 AutoMLConfig 对象中的特征化的已接受设置。The following table shows the accepted settings for featurization in the AutoMLConfig object.

特征化配置Featurization Configuration 说明Description
"featurization": 'auto' 指示在处理过程中自动执行数据护栏和特征化步骤Indicates that as part of preprocessing, data guardrails and featurization steps are performed automatically. 默认设置。Default setting.
"featurization": 'off' 表示不应自动执行特征化步骤。Indicates featurization step shouldn't be done automatically.
"featurization": 'FeaturizationConfig' 指示应当使用自定义特征化步骤。Indicates customized featurization step should be used. 了解如何自定义特征化Learn how to customize featurization.

备注

自动化机器学习特征化步骤(特征规范化、处理缺失数据,将文本转换为数字等)成为了基础模型的一部分。Automated machine learning featurization steps (feature normalization, handling missing data, converting text to numeric, etc.) become part of the underlying model. 使用模型进行预测时,将自动向输入数据应用在训练期间应用的相同特征化步骤。When using the model for predictions, the same featurization steps applied during training are applied to your input data automatically.

集成配置Ensemble configuration

集成模型默认启用,在 AutoML 运行中显示为最终的运行迭代次数。Ensemble models are enabled by default, and appear as the final run iterations in an AutoML run. 目前支持 VotingEnsembleStackEnsembleCurrently VotingEnsemble and StackEnsemble are supported.

投票实现了使用加权平均值的软投票。Voting implements soft-voting which uses weighted averages. 堆栈实现使用一个两层实现,其中的第一层具有与投票集成相同的模型,第二层模型用于从第一层中查找模型的最佳组合。The stacking implementation uses a two layer implementation, where the first layer has the same models as the voting ensemble, and the second layer model is used to find the optimal combination of the models from the first layer.

如果使用 ONNX 模型,或启用了模型可解释性,则会禁用堆栈,仅使用投票。If you are using ONNX models, or have model-explainability enabled, stacking is disabled and only voting is utilized.

可以通过使用 enable_voting_ensembleenable_stack_ensemble 布尔参数来禁用集成训练。Ensemble training can be disabled by using the enable_voting_ensemble and enable_stack_ensemble boolean parameters.

automl_classifier = AutoMLConfig(
        task='classification',
        primary_metric='AUC_weighted',
        experiment_timeout_minutes=30,
        training_data=data_train,
        label_column_name=label,
        n_cross_validations=5,
        enable_voting_ensemble=False,
        enable_stack_ensemble=False
        )

若要更改默认集成行为,可以将多个默认参数作为 kwargsAutoMLConfig 对象中提供。To alter the default ensemble behavior, there are multiple default arguments that can be provided as kwargs in an AutoMLConfig object.

重要

以下参数不是 AutoMLConfig 类的显式参数。The following parameters aren't explicit parameters of the AutoMLConfig class.

  • ensemble_download_models_timeout_sec:在 VotingEnsemble 和 StackEnsemble 模型生成期间,会下载来自先前子运行的多个拟合模型。ensemble_download_models_timeout_sec: During VotingEnsemble and StackEnsemble model generation, multiple fitted models from the previous child runs are downloaded. 如果遇到此错误 AutoMLEnsembleException: Could not find any models for running ensembling,则可能需要为要下载的模型提供更多时间。If you encounter this error: AutoMLEnsembleException: Could not find any models for running ensembling, then you may need to provide more time for the models to be downloaded. 默认值为 300 秒并行下载这些模型,且没有最大超时限制。The default value is 300 seconds for downloading these models in parallel and there is no maximum timeout limit. 如果需要更多时间,请将此参数配置为大于 300 秒的值。Configure this parameter with a higher value than 300 secs, if more time is needed.

    备注

    如果已超时且下载了模型,则融合会使用它下载的多个模型继续执行。If the timeout is reached and there are models downloaded, then the ensembling proceeds with as many models it has downloaded. 并不需要下载所有模型才能在超时内完成。It's not required that all the models need to be downloaded to finish within that timeout.

以下参数只应用于 StackEnsemble 模型:The following parameters only apply to StackEnsemble models:

  • stack_meta_learner_type:元学习器是针对单个异类模型的输出而训练出来的模型。stack_meta_learner_type: the meta-learner is a model trained on the output of the individual heterogeneous models. 默认的元学习器是用于分类任务的 LogisticRegression(或为 LogisticRegressionCV,如果启用了交叉验证的话),以及用于回归/预测任务的 ElasticNet(或为 ElasticNetCV,如果启用了交叉验证的话)。Default meta-learners are LogisticRegression for classification tasks (or LogisticRegressionCV if cross-validation is enabled) and ElasticNet for regression/forecasting tasks (or ElasticNetCV if cross-validation is enabled). 此参数可以是下列字符串之一:LogisticRegressionLogisticRegressionCVLightGBMClassifierElasticNetElasticNetCVLightGBMRegressorLinearRegressionThis parameter can be one of the following strings: LogisticRegression, LogisticRegressionCV, LightGBMClassifier, ElasticNet, ElasticNetCV, LightGBMRegressor, or LinearRegression.

  • stack_meta_learner_train_percentage:指定为训练元学习器而保留的训练集的比例(选择训练的训练和验证类型时)。stack_meta_learner_train_percentage: specifies the proportion of the training set (when choosing train and validation type of training) to be reserved for training the meta-learner. 默认值为 0.2Default value is 0.2.

  • stack_meta_learner_kwargs:要传递给元学习器的初始值设定项的可选参数。stack_meta_learner_kwargs: optional parameters to pass to the initializer of the meta-learner. 这些参数和参数类型对来自相应模型构造函数的参数和参数类型进行镜像,然后再转发到模型构造函数。These parameters and parameter types mirror the parameters and parameter types from the corresponding model constructor, and are forwarded to the model constructor.

下面的代码示例展示了如何在 AutoMLConfig 对象中指定自定义融合行为。The following code shows an example of specifying custom ensemble behavior in an AutoMLConfig object.

ensemble_settings = {
    "ensemble_download_models_timeout_sec": 600
    "stack_meta_learner_type": "LogisticRegressionCV",
    "stack_meta_learner_train_percentage": 0.3,
    "stack_meta_learner_kwargs": {
        "refit": True,
        "fit_intercept": False,
        "class_weight": "balanced",
        "multi_class": "auto",
        "n_jobs": -1
    }
}

automl_classifier = AutoMLConfig(
        task='classification',
        primary_metric='AUC_weighted',
        experiment_timeout_minutes=30,
        training_data=train_data,
        label_column_name=label,
        n_cross_validations=5,
        **ensemble_settings
        )

退出条件Exit criteria

可以在 AutoMLConfig 中定义几个选项来结束试验。There are a few options you can define in your AutoMLConfig to end your experiment.

条件Criteria descriptiondescription
无条件No criteria 如果未定义任何退出参数,则试验将继续,直到主要指标不再需要执行其他步骤。If you do not define any exit parameters the experiment continues until no further progress is made on your primary metric.
在一段时间后After a length of time 在设置中使用 experiment_timeout_minutes 来定义试验应继续运行多长时间(以分钟为单位)。Use experiment_timeout_minutes in your settings to define how long, in minutes, your experiment should continue to run.

若要避免试验超时失败,最少需要 15 分钟,如果行数乘以列数的大小超过 10,000,000,则最少需要 60 分钟。To help avoid experiment time out failures, there is a minimum of 15 minutes, or 60 minutes if your row by column size exceeds 10 million.
达到某个分数A score has been reached 使用 experiment_exit_score 将在达到指定的主要指标分数后完成试验。Use experiment_exit_score completes the experiment after a specified primary metric score has been reached.

运行试验Run experiment

对于自动化 ML,可以创建 Experiment 对象,这是 Workspace 中用于运行实验的命名对象。For automated ML, you create an Experiment object, which is a named object in a Workspace used to run experiments.

from azureml.core.experiment import Experiment

ws = Workspace.from_config()

# Choose a name for the experiment and specify the project folder.
experiment_name = 'Tutorial-automl'
project_folder = './sample_projects/automl-classification'

experiment = Experiment(ws, experiment_name)

提交试验以运行和生成模型。Submit the experiment to run and generate a model. AutoMLConfig 传递给 submit 方法以生成模型。Pass the AutoMLConfig to the submit method to generate the model.

run = experiment.submit(automl_config, show_output=True)

备注

首先在新的计算机上安装依赖项。Dependencies are first installed on a new machine. 最长可能需要在 10 分钟后才会显示输出。It may take up to 10 minutes before output is shown. show_output 设置为 True 可在控制台上显示输出。Setting show_output to True results in output being shown on the console.

群集上的多个子运行Multiple child runs on clusters

自动化 ML 试验子运行可以在已经运行另一个试验的群集上执行。Automated ML experiment child runs can be performed on a cluster that is already running another experiment. 但是,计时取决于群集具有的节点数,以及这些节点是否可用于运行不同的试验。However, the timing depends on how many nodes the cluster has, and if those nodes are available to run a different experiment.

群集中的每个节点充当单个可以完成一次训练运行的虚拟机 (VM);对于自动化 ML,这意味着一个子运行。Each node in the cluster acts as an individual virtual machine (VM) that can accomplish a single training run; for automated ML this means a child run. 如果所有节点都处于忙状态,则新的试验将排队。If all the nodes are busy, the new experiment is queued. 但是,如果有空闲节点,新的试验将在可用节点/VM 中并行运行自动化 ML 子运行。But if there are free nodes, the new experiment will run automated ML child runs in parallel in the available nodes/VMs.

为了管理子运行及其执行时间,建议你为每个试验创建一个专用群集,使试验的 max_concurrent_iterations 数与群集中的节点数匹配。To help manage child runs and when they can be performed, we recommend you create a dedicated cluster per experiment, and match the number of max_concurrent_iterations of your experiment to the number of nodes in the cluster. 这样就可以同时使用群集的所有节点以及所需数量的并发子运行/迭代。This way, you use all the nodes of the cluster at the same time with the number of concurrent child runs/iterations you want.

AutoMLConfig 对象中配置 max_concurrent_iterationsConfigure max_concurrent_iterations in your AutoMLConfig object. 如果未进行配置,则默认情况下每个试验仅允许一个并发子运行/迭代。If it is not configured, then by default only one concurrent child run/iteration is allowed per experiment.

探索模型和指标Explore models and metrics

自动化 ML 提供用于监视和评估训练结果的选项。Automated ML offers options for you to monitor and evaluate your training results.

你可以使用以下自定义代码解决方案查看应用于特定自动化 ML 运行的超参数、缩放和规范化技术以及算法。You can view the hyperparameters, the scaling and normalization techniques, and algorithm applied to a specific automated ML run with the following custom code solution.

下面定义了自定义方法 print_model(),该方法打印自动化 ML 训练管道的每个步骤的超参数。The following defines the custom method, print_model(), which prints the hyperparameters of each step of the automated ML training pipeline.

from pprint import pprint

def print_model(model, prefix=""):
    for step in model.steps:
        print(prefix + step[0])
        if hasattr(step[1], 'estimators') and hasattr(step[1], 'weights'):
            pprint({'estimators': list(e[0] for e in step[1].estimators), 'weights': step[1].weights})
            print()
            for estimator in step[1].estimators:
                print_model(estimator[1], estimator[0]+ ' - ')
        elif hasattr(step[1], '_base_learners') and hasattr(step[1], '_meta_learner'):
            print("\nMeta Learner")
            pprint(step[1]._meta_learner)
            print()
            for estimator in step[1]._base_learners:
                print_model(estimator[1], estimator[0]+ ' - ')
        else:
            pprint(step[1].get_params())
            print()   

对于刚刚在同一个试验笔记本中提交和训练的本地或远程运行,可以使用 get_output() 方法传入最佳模型。For a local or remote run that was just submitted and trained from within the same experiment notebook, you can pass in the best model using the get_output() method.

best_run, fitted_model = run.get_output()
print(best_run)
         
print_model(fitted_model)

以下输出表明:The following output indicates that:

  • StandardScalerWrapper 技术用于在训练之前缩放和规范化数据。The StandardScalerWrapper technique was used to scale and normalize the data prior to training.

  • XGBoostClassifier 算法被识别为最佳运行,并显示了超参数值。The XGBoostClassifier algorithm was identified as the best run, and also shows the hyperparameter values.

StandardScalerWrapper
{'class_name': 'StandardScaler',
 'copy': True,
 'module_name': 'sklearn.preprocessing.data',
 'with_mean': False,
 'with_std': False}

XGBoostClassifier
{'base_score': 0.5,
 'booster': 'gbtree',
 'colsample_bylevel': 1,
 'colsample_bynode': 1,
 'colsample_bytree': 0.6,
 'eta': 0.4,
 'gamma': 0,
 'learning_rate': 0.1,
 'max_delta_step': 0,
 'max_depth': 8,
 'max_leaves': 0,
 'min_child_weight': 1,
 'missing': nan,
 'n_estimators': 400,
 'n_jobs': 1,
 'nthread': None,
 'objective': 'multi:softprob',
 'random_state': 0,
 'reg_alpha': 0,
 'reg_lambda': 1.6666666666666667,
 'scale_pos_weight': 1,
 'seed': None,
 'silent': None,
 'subsample': 0.8,
 'tree_method': 'auto',
 'verbose': -10,
 'verbosity': 1}

如果想了解工作区中其他试验的现有运行,请获取要浏览的特定运行 ID,并将其传递到 print_model() 方法中。For an existing run from a different experiment in your workspace, obtain the specific run ID you want to explore and pass that into the print_model() method.

from azureml.train.automl.run import AutoMLRun

ws = Workspace.from_config()
experiment = ws.experiments['automl-classification']
automl_run = AutoMLRun(experiment, run_id = 'AutoML_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx')

automl_run
best_run, model_from_aml = automl_run.get_output()

print_model(model_from_aml)

备注

自动化 ML 使用的算法本身具有随机性,这可能会导致建议的模型最终指标分数(如准确度)出现细微差异。The algorithms automated ML employs have inherent randomness that can cause slight variation in a recommended model's final metrics score, like accuracy. 自动化 ML 还可在必要时对数据执行操作,例如训练-测试拆分、训练-验证拆分或交叉验证。Automated ML also performs operations on data such as train-test split, train-validation split or cross-validation when necessary. 因此,如果多次使用相同的配置设置和主要指标运行一个试验,你可能会发现由于这些因素导致每个试验最终指标分数存在差异。So if you run an experiment with the same configuration settings and primary metric multiple times, you'll likely see variation in each experiments final metrics score due to these factors.

监视自动化机器学习运行Monitor automated machine learning runs

对于自动化机器学习运行,若要访问根据以前的运行生成的图表,请将 <<experiment_name>> 替换为相应的试验名称:For automated machine learning runs, to access the charts from a previous run, replace <<experiment_name>> with the appropriate experiment name:

from azureml.widgets import RunDetails
from azureml.core.run import Run

experiment = Experiment (workspace, <<experiment_name>>)
run_id = 'autoML_my_runID' #replace with run_ID
run = Run(experiment, run_id)
RunDetails(run).show()

自动化机器学习的 Jupyter Notebook 小组件

注册和部署模型Register and deploy models

可以注册模型,以便以后使用。You can register a model, so you can come back to it for later use.

若要从自动化 ML 运行注册模型,请使用 register_model() 方法。To register a model from an automated ML run, use the register_model() method.


best_run, fitted_model = run.get_output()
print(fitted_model.steps)

model_name = best_run.properties['model_name']
description = 'AutoML forecast example'
tags = None

model = remote_run.register_model(model_name = model_name, 
                                  description = description, 
                                  tags = tags)

有关如何创建部署配置以及将注册模型部署到 Web 服务的详细信息,请参阅如何部署模型以及在何处部署模型For details on how to create a deployment configuration and deploy a registered model to a web service, see how and where to deploy a model.

提示

对于已注册的模型,可通过 Azure 机器学习工作室获取一键式部署。For registered models, one-click deployment is available via the Azure Machine Learning studio. 请参阅如何从工作室部署已注册的模型See how to deploy registered models from the studio.

模型可解释性Model interpretability

模型可解释性让你可以了解模型进行预测的原因,以及基础特征重要性值。Model interpretability allows you to understand why your models made predictions, and the underlying feature importance values. SDK 包括各种包,这些包用于在训练和推理时间为本地和已部署的模型启用模型可解释性功能。The SDK includes various packages for enabling model interpretability features, both at training and inference time, for local and deployed models.

有关如何在自动化机器学习试验中启用可解释性功能的代码示例,请参阅操作方法See the how-to for code samples on how to enable interpretability features specifically within automated machine learning experiments.

有关如何在自动化机器学习之外的其他 SDK 区域中启用模型解释和特征重要性的基本信息,请参阅可解释性方面的概念文章。For general information on how model explanations and feature importance can be enabled in other areas of the SDK outside of automated machine learning, see the concept article on interpretability.

备注

解释客户端目前不支持 ForecastTCN 模型。The ForecastTCN model is not currently supported by the Explanation Client. 如果此模型作为最佳模型返回,则不会返回解释仪表板,并且不支持按需解释运行。This model will not return an explanation dashboard if it is returned as the best model, and does not support on-demand explanation runs.

后续步骤Next steps