使用 Azure 机器学习 CLI 和 Python SDK 为表格数据设置 AutoML 训练

若要使用 SDK 信息，请安装适用于 Python 的 Azure 机器学习 SDK v2。

可以通过两种方式安装 SDK：

• 创建一个计算实例，它已装有最新的 Azure 机器学习 Python SDK 并针对 ML 工作流进行了配置。 有关详细信息，请参阅创建 Azure 机器学习计算实例。
• 在本地计算机上安装 SDK。

若要使用 CLI 信息，请安装适用于机器学习的 Azure CLI 和扩展。

要连接到您的 Azure 机器学习工作区，请将工作区详细信息输入 MLClient 并从 azure.ai.ml 进行连接。

以下示例使用默认的 Azure 身份验证，并使用默认工作区配置或文件夹结构中的 config.json 文件的配置。 如果找不到 config.json，则在创建 MLClient 时，您需要手动提供订阅 ID、资源组和工作区。

from azure.identity import DefaultAzureCredential
from azure.ai.ml import MLClient

credential = DefaultAzureCredential()
ml_client = None
try:
    ml_client = MLClient.from_config(credential)
except Exception as ex:
    print(ex)
    # Enter details of your Azure Machine Learning workspace
    subscription_id = "<SUBSCRIPTION_ID>"
    resource_group = "<RESOURCE_GROUP>"
    workspace = "<AZUREML_WORKSPACE_NAME>"
    ml_client = MLClient(credential, subscription_id, resource_group, workspace)

在 CLI 中，首先登录到 Azure 帐户。 如果帐户与多个订阅相关联，则需要 设置订阅。

az login

还可以设置默认值以避免在每个 CLI 命令中键入这些值：

az configure --defaults group=<RESOURCE_GROUP> workspace=<AZUREML_WORKSPACE_NAME> location=<LOCATION>

有关详细信息，请参阅 CLI 设置。

可使用 mltable Python SDK 创建 MLTable，如以下示例所示：

import mltable

paths = [
    {'file': './train_data/bank_marketing_train_data.csv'}
]

train_table = mltable.from_delimited_files(paths)
train_table.save('./train_data')

此代码会创建一个新文件 ./train_data/MLTable，其中包含文件格式和加载说明。

以下 YAML 代码定义了一个 MLTable，该表格和数据文件可以放置在本地文件夹或云中的远程文件夹中。 数据文件是 .csv 或 Parquet 文件。 在这种情况下，将 YAML 文本写入本地文件 ./train_data/MLTable。

$schema: https://azuremlschemas.azureedge.net/latest/MLTable.schema.json

paths:
  - file: ./bank_marketing_train_data.csv
transformations:
  - read_delimited:
        delimiter: ','
        encoding: 'ascii'

from azure.ai.ml.constants import AssetTypes
from azure.ai.ml import automl, Input

# note that this is a code snippet -- you might have to modify the variable values to run it successfully

# make an Input object for the training data
my_training_data_input = Input(
    type=AssetTypes.MLTABLE, path="./data/training-mltable-folder"
)

# configure the classification job
classification_job = automl.classification(
    compute=my_compute_name,
    experiment_name=my_exp_name,
    training_data=my_training_data_input,
    target_column_name="y",
    primary_metric="accuracy",
    n_cross_validations=5,
    enable_model_explainability=True,
    tags={"my_custom_tag": "My custom value"}
)

# Limits are all optional
classification_job.set_limits(
    timeout_minutes=600, 
    trial_timeout_minutes=20, 
    max_trials=5,
    enable_early_termination=True,
)

# Training properties are optional
classification_job.set_training(
    blocked_training_algorithms=["logistic_regression"], 
    enable_onnx_compatible_models=True
)

$schema: https://azuremlsdk2.blob.core.chinacloudapi.cn/preview/0.0.1/autoMLJob.schema.json
type: automl

experiment_name: <my_exp_name>
description: A classification AutoML job
task: classification

training_data:
    path: "./train_data"
    type: mltable

compute: azureml:<my_compute_name>
primary_metric: accuracy  
target_column_name: y
n_cross_validations: 5
enable_model_explainability: True

tags:
    <my_custom_tag>: <My custom value>

limits:
    timeout_minutes: 600 
    trial_timeout_minutes: 20 
    max_trials: 5
    enable_early_termination: True

training:
    blocked_training_algorithms: ["logistic_regression"] 
    enable_onnx_compatible_models: True

from azure.ai.ml.automl import ColumnTransformer

transformer_params = {
    "imputer": [
        ColumnTransformer(fields=["CACH"], parameters={"strategy": "most_frequent"}),
        ColumnTransformer(fields=["PRP"], parameters={"strategy": "most_frequent"}),
    ],
}
regression_job.set_featurization(
    mode="custom",
    transformer_params=transformer_params,
    blocked_transformers=["LabelEncoding"],
    column_name_and_types={"CHMIN": "Categorical"},
)

$schema: https://azuremlsdk2.blob.core.chinacloudapi.cn/preview/0.0.1/autoMLJob.schema.json
type: automl

experiment_name: <my_exp_name>
description: A classification AutoML job
task: classification

training_data:
    path: "./train_data"
    type: mltable

compute: azureml:<my_compute_name>
primary_metric: accuracy  
target_column_name: y
n_cross_validations: 5
enable_model_explainability: True

featurization:
    mode: custom
    column_name_and_types:
        CHMIN: Categorical
    blocked_transformers: ["label_encoder"]
    transformer_params:
        imputer:
            - fields: ["CACH", "PRP"]
            parameters:
                strategy: most_frequent

limits:
    # limit settings

training:
    # training settings

# Submit the AutoML job
returned_job = ml_client.jobs.create_or_update(
    classification_job
)  # submit the job to the backend

print(f"Created job: {returned_job}")

# Get a URL for the status of the job
returned_job.services["Studio"].endpoint

在以下 CLI 命令中，作业 YAML 配置位于路径 ./automl-classification-job.yml：

run_id=$(az ml job create --file automl-classification-job.yml -w <Workspace> -g <Resource Group> --subscription <Subscription>)

你可以使用存储的运行 ID 返回有关作业的信息。 --web 参数打开 Azure 机器学习工作室 Web UI，你可在其中深入了解作业的详细信息：

az ml job show -n $run_id --web

# Define pipeline
@pipeline(
    description="AutoML Classification Pipeline",
    )
def automl_classification(
    classification_train_data,
    classification_validation_data
):
    # define the automl classification task with automl function
    classification_node = classification(
        training_data=classification_train_data,
        validation_data=classification_validation_data,
        target_column_name="y",
        primary_metric="accuracy",
        # currently need to specify outputs "mlflow_model" explictly to reference it in following nodes 
        outputs={"best_model": Output(type="mlflow_model")},
    )
    # set limits and training
    classification_node.set_limits(max_trials=1)
    classification_node.set_training(
        enable_stack_ensemble=False,
        enable_vote_ensemble=False
    )

    command_func = command(
        inputs=dict(
            automl_output=Input(type="mlflow_model")
        ),
        command="ls ${{inputs.automl_output}}",
        environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:latest"
    )
    show_output = command_func(automl_output=classification_node.outputs.best_model)


pipeline_job = automl_classification(
    classification_train_data=Input(path="./training-mltable-folder/", type="mltable"),
    classification_validation_data=Input(path="./validation-mltable-folder/", type="mltable"),
)

# set pipeline level compute
pipeline_job.settings.default_compute = compute_name

# submit the pipeline job
returned_pipeline_job = ml_client.jobs.create_or_update(
    pipeline_job,
    experiment_name=experiment_name
)
returned_pipeline_job

# ...
# Note that this is a snippet from the bankmarketing example you can find in our examples repo -> https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1h_automl_in_pipeline/automl-classification-bankmarketing-in-pipeline

有关如何在管道中包含 AutoML 的更多示例，请参阅 示例存储库。

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline

description: AutoML Classification Pipeline
experiment_name: <exp_name>

# set the default compute for the pipeline steps
settings:
    default_compute: azureml:<my_compute>

# pipeline inputs
inputs:
    classification_train_data:
        type: mltable
        path: "./train_data"
    classification_validation_data:
        type: mltable
        path: "./valid_data"

jobs:
    # Configure the automl training node of the pipeline 
    classification_node:
        type: automl
        task: classification
        primary_metric: accuracy
        target_column_name: y
        training_data: ${{parent.inputs.classification_train_data}}
        validation_data: ${{parent.inputs.classification_validation_data}}
        training:
            max_trials: 1
        limits:
            enable_stack_ensemble: False
            enable_vote_ensemble: False
        outputs:
            best_model:
                type: mlflow_model

    show_output:
        type: command
        inputs:
            automl_output: ${{parent.jobs.classification_node.outputs.best_model}}
        environment: "AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:latest"
        command: >-
            ls ${{inputs.automl_output}}
        

现在，使用以下命令启动管道运行。 管道配置位于路径 ./automl-classification-pipeline.yml。

> run_id=$(az ml job create --file automl-classification-pipeline.yml -w <Workspace> -g <Resource Group> --subscription <Subscription>)
> az ml job show -n $run_id --web

from azure.ai.ml.constants import TabularTrainingMode

# Set the training mode to distributed
classification_job.set_training(
    allowed_training_algorithms=["LightGBM"],
    training_mode=TabularTrainingMode.DISTRIBUTED
)

# Distribute training across 4 nodes for each trial
classification_job.set_limits(
    max_nodes=4,
    # other limit settings
)

# Set the training mode to distributed
training:
    allowed_training_algorithms: ["LightGBM"]
    training_mode: distributed

# Distribute training across 4 nodes for each trial
limits:
    max_nodes: 4

from azure.ai.ml.constants import TabularTrainingMode

# Set the training mode to distributed
forecasting_job.set_training(
    enable_dnn_training=True,
    allowed_training_algorithms=["TCNForecaster"],
    training_mode=TabularTrainingMode.DISTRIBUTED
)

# Distribute training across 4 nodes
# Train 2 trial models in parallel => 2 nodes per trial
forecasting_job.set_limits(
    max_concurrent_trials=2,
    max_nodes=4,
    # other limit settings
)

# Set the training mode to distributed
training:
    allowed_training_algorithms: ["TCNForecaster"]
    training_mode: distributed

# Distribute training across 4 nodes
# Train 2 trial models in parallel => 2 nodes per trial
limits:
    max_concurrent_trials: 2
    max_nodes: 4