Data featurization in automated machine learning

APPLIES TO: Python SDK azureml v1

Learn about the data featurization settings in Azure Machine Learning, and how to customize those features for automated machine learning experiments.

Feature engineering and featurization

Training data consists of rows and columns. Each row is an observation or record, and the columns of each row are the features that describe each record. Typically, the features that best characterize the patterns in the data are selected to create predictive models.

Although many of the raw data fields can be used directly to train a model, it's often necessary to create additional (engineered) features that provide information that better differentiates patterns in the data. This process is called feature engineering, where the use of domain knowledge of the data is leveraged to create features that, in turn, help machine learning algorithms to learn better.

In Azure Machine Learning, data-scaling and normalization techniques are applied to make feature engineering easier. Collectively, these techniques and this feature engineering are called featurization in automated ML experiments.

Prerequisites

This article assumes that you already know how to configure an automated ML experiment.

For information about configuration, see the following articles:

• For a code-first experience: Configure automated ML experiments by using the Azure Machine Learning SDK for Python.
• For a low-code or no-code experience: Create, review, and deploy automated machine learning models by using the Azure Machine Learning studio.

Configure featurization

In every automated machine learning experiment, automatic scaling and normalization techniques are applied to your data by default. These techniques are types of featurization that help certain algorithms that are sensitive to features on different scales. You can enable more featurization, such as missing-values imputation, encoding, and transforms.

For experiments that you configure with the Python SDK, you can enable or disable the featurization setting and further specify the featurization steps to be used for your experiment. If you're using the Azure Machine Learning studio, see the steps to enable featurization.

The following table shows the accepted settings for featurization in the AutoMLConfig class:

Featurization configuration	Description
"featurization": 'auto'	Specifies that, as part of preprocessing, data guardrails and featurization steps are to be done automatically. This setting is the default.
"featurization": 'off'	Specifies that featurization steps are not to be done automatically.
"featurization": 'FeaturizationConfig'	Specifies that customized featurization steps are to be used. Learn how to customize featurization.

Automatic featurization

The following table summarizes techniques that are automatically applied to your data. These techniques are applied for experiments that are configured by using the SDK or the studio UI. To disable this behavior, set "featurization": 'off' in your AutoMLConfig object.

Featurization steps	Description
Drop high cardinality or no variance features*	Drop these features from training and validation sets. Applies to features with all values missing, with the same value across all rows, or with high cardinality (for example, hashes, IDs, or GUIDs).
Impute missing values*	For numeric features, impute with the average of values in the column. For categorical features, impute with the most frequent value.
Generate more features*	For DateTime features: Year, Month, Day, Day of week, Day of year, Quarter, Week of the year, Hour, Minute, Second. For forecasting tasks, these additional DateTime features are created: ISO year, Half - half-year, Calendar month as string, Week, Day of week as string, Day of quarter, Day of year, AM/PM (0 if hour is before noon (12 pm), 1 otherwise), AM/PM as string, Hour of day (12-hr basis) For Text features: Term frequency based on unigrams, bigrams, and trigrams. Learn more about how this is done with BERT.
Transform and encode*	Transform numeric features that have few unique values into categorical features. One-hot encoding is used for low-cardinality categorical features. One-hot-hash encoding is used for high-cardinality categorical features.
Word embeddings	A text featurizer converts vectors of text tokens into sentence vectors by using a pre-trained model. Each word's embedding vector in a document is aggregated with the rest to produce a document feature vector.
Cluster Distance	Trains a k-means clustering model on all numeric columns. Produces k new features (one new numeric feature per cluster) that contain the distance of each sample to the centroid of each cluster.

In every automated machine learning experiment, your data is automatically scaled or normalized to help algorithms perform well. During model training, one of the following scaling or normalization techniques are applied to each model.

Scaling & processing	Description
StandardScaleWrapper	Standardize features by removing the mean and scaling to unit variance
MinMaxScalar	Transforms features by scaling each feature by that column's minimum and maximum
MaxAbsScaler	Scale each feature by its maximum absolute value
RobustScalar	Scales features by their quantile range
PCA	Linear dimensionality reduction using Singular Value Decomposition of the data to project it to a lower dimensional space
TruncatedSVDWrapper	This transformer performs linear dimensionality reduction by means of truncated singular value decomposition (SVD). Contrary to PCA, this estimator does not center the data before computing the singular value decomposition, which means it can work with scipy.sparse matrices efficiently
SparseNormalizer	Each sample (that is, each row of the data matrix) with at least one non-zero component is rescaled independently of other samples so that its norm (l1 or l2) equals one

Data guardrails

Data guardrails help you identify potential issues with your data (for example, missing values or class imbalance). They also help you take corrective actions for improved results.

Data guardrails are applied:

• For SDK experiments: When the parameters "featurization": 'auto' or validation=auto are specified in your AutoMLConfig object.
• For studio experiments: When automatic featurization is enabled.

You can review the data guardrails for your experiment:

• By setting show_output=True when you submit an experiment by using the SDK.

• In the studio, on the Data guardrails tab of your automated ML run.

Data guardrail states

Data guardrails display one of three states:

State	Description
Passed	No data problems were detected and no action is required by you.
Done	Changes were applied to your data. We encourage you to review the corrective actions that AutoML took, to ensure that the changes align with the expected results.
Alerted	A data issue was detected but couldn't be remedied. We encourage you to revise and fix the issue.

Supported data guardrails

The following table describes the data guardrails that are currently supported and the associated statuses that you might see when you submit your experiment:

Guardrail	Status	Condition for trigger
Missing feature values imputation	Passed Done	No missing feature values were detected in your training data. Learn more about missing-value imputation. Missing feature values were detected in your training data and were imputed.
High cardinality feature detection	Passed Done	Your inputs were analyzed, and no high-cardinality features were detected. High-cardinality features were detected in your inputs and were handled.
Validation split handling	Done	The validation configuration was set to 'auto' and the training data contained fewer than 20,000 rows. Each iteration of the trained model was validated by using cross-validation. Learn more about validation data. The validation configuration was set to 'auto', and the training data contained more than 20,000 rows. The input data has been split into a training dataset and a validation dataset for validation of the model.
Class balancing detection	Passed Alerted Done	Your inputs were analyzed, and all classes are balanced in your training data. A dataset is considered to be balanced if each class has good representation in the dataset, as measured by number and ratio of samples. Imbalanced classes were detected in your inputs. To fix model bias, fix the balancing problem. Learn more about imbalanced data. Imbalanced classes were detected in your inputs and the sweeping logic has determined to apply balancing.
Memory issues detection	Passed Done	The selected values (horizon, lag, rolling window) were analyzed, and no potential out-of-memory issues were detected. Learn more about time-series forecasting configurations. The selected values (horizon, lag, rolling window) were analyzed and will potentially cause your experiment to run out of memory. The lag or rolling-window configurations have been turned off.
Frequency detection	Passed Done	The time series was analyzed, and all data points are aligned with the detected frequency. The time series was analyzed, and data points that don't align with the detected frequency were detected. These data points were removed from the dataset.
Cross validation	Done	In order to accurately evaluate the model(s) trained by AutoML, we leverage a dataset that the model is not trained on. Hence, if the user doesn't provide an explicit validation dataset, a part of the training dataset is used to achieve this. For smaller datasets (fewer than 20,000 samples), cross-validation is leveraged, else a single hold-out set is split from the training data to serve as the validation dataset. Hence, for your input data we leverage cross-validation with 10 folds, if the number of training samples are fewer than 1000, and 3 folds in all other cases.
Train-Test data split	Done	In order to accurately evaluate the model(s) trained by AutoML, we leverage a dataset that the model is not trained on. Hence, if the user doesn't provide an explicit validation dataset, a part of the training dataset is used to achieve this. For smaller datasets (fewer than 20,000 samples), cross-validation is leveraged, else a single hold-out set is split from the training data to serve as the validation dataset. Hence, your input data has been split into a training dataset and a holdout validation dataset.
Time Series ID detection	Passed Fixed	The data set was analyzed, and no duplicate time index were detected. Multiple time series were found in the dataset, and the time series identifiers were automatically created for your dataset.
Time series aggregation	Passed Fixed	The dataset frequency is aligned with the user specified frequency. No aggregation was performed. The data was aggregated to comply with user provided frequency.
Short series handling	Passed Fixed	Automated ML detected enough data points for each series in the input data to continue with training. Automated ML detected that some series did not contain enough data points to train a model. To continue with training, these short series have been dropped or padded.

Customize featurization

You can customize your featurization settings to ensure that the data and features that are used to train your ML model result in relevant predictions.

To customize featurizations, specify "featurization": FeaturizationConfig in your AutoMLConfig object. If you're using the Azure Machine Learning studio for your experiment, see the how-to article. To customize featurization for forecastings task types, refer to the forecasting how-to.

Supported customizations include:

Customization	Definition
Column purpose update	Override the autodetected feature type for the specified column.
Transformer parameter update	Update the parameters for the specified transformer. Currently supports Imputer (mean, most frequent, and median) and HashOneHotEncoder.
Drop columns	Specifies columns to drop from being featurized.
Block transformers	Specifies block transformers to be used in the featurization process.

Create the FeaturizationConfig object by using API calls:

featurization_config = FeaturizationConfig()
featurization_config.blocked_transformers = ['LabelEncoder']
featurization_config.drop_columns = ['aspiration', 'stroke']
featurization_config.add_column_purpose('engine-size', 'Numeric')
featurization_config.add_column_purpose('body-style', 'CategoricalHash')
#default strategy mean, add transformer param for for 3 columns
featurization_config.add_transformer_params('Imputer', ['engine-size'], {"strategy": "median"})
featurization_config.add_transformer_params('Imputer', ['city-mpg'], {"strategy": "median"})
featurization_config.add_transformer_params('Imputer', ['bore'], {"strategy": "most_frequent"})
featurization_config.add_transformer_params('HashOneHotEncoder', [], {"number_of_bits": 3})

Featurization transparency

Every AutoML model has featurization automatically applied. Featurization includes automated feature engineering (when "featurization": 'auto') and scaling and normalization, which then impacts the selected algorithm and its hyperparameter values. AutoML supports different methods to ensure you have visibility into what was applied to your model.

Consider this forecasting example:

• There are four input features: A (Numeric), B (Numeric), C (Numeric), D (DateTime).
• Numeric feature C is dropped because it is an ID column with all unique values.
• Numeric features A and B have missing values and hence are imputed by the mean.
• DateTime feature D is featurized into 11 different engineered features.

To get this information, use the fitted_model output from your automated ML experiment run.

automl_config = AutoMLConfig(…)
automl_run = experiment.submit(automl_config …)
best_run, fitted_model = automl_run.get_output()

Automated feature engineering

The get_engineered_feature_names() returns a list of engineered feature names.

fitted_model.named_steps['timeseriestransformer']. get_engineered_feature_names ()

This list includes all engineered feature names.

['A', 'B', 'A_WASNULL', 'B_WASNULL', 'year', 'half', 'quarter', 'month', 'day', 'hour', 'am_pm', 'hour12', 'wday', 'qday', 'week']

The get_featurization_summary() gets a featurization summary of all the input features.

fitted_model.named_steps['timeseriestransformer'].get_featurization_summary()

Output

[{'RawFeatureName': 'A',
  'TypeDetected': 'Numeric',
  'Dropped': 'No',
  'EngineeredFeatureCount': 2,
  'Tranformations': ['MeanImputer', 'ImputationMarker']},
 {'RawFeatureName': 'B',
  'TypeDetected': 'Numeric',
  'Dropped': 'No',
  'EngineeredFeatureCount': 2,
  'Tranformations': ['MeanImputer', 'ImputationMarker']},
 {'RawFeatureName': 'C',
  'TypeDetected': 'Numeric',
  'Dropped': 'Yes',
  'EngineeredFeatureCount': 0,
  'Tranformations': []},
 {'RawFeatureName': 'D',
  'TypeDetected': 'DateTime',
  'Dropped': 'No',
  'EngineeredFeatureCount': 11,
  'Tranformations': ['DateTime','DateTime','DateTime','DateTime','DateTime','DateTime','DateTime','DateTime','DateTime','DateTime','DateTime']}]

Output	Definition
RawFeatureName	Input feature/column name from the dataset provided.
TypeDetected	Detected datatype of the input feature.
Dropped	Indicates if the input feature was dropped or used.
EngineeringFeatureCount	Number of features generated through automated feature engineering transforms.
Transformations	List of transformations applied to input features to generate engineered features.

Scaling and normalization

To understand the scaling/normalization and the selected algorithm with its hyperparameter values, use fitted_model.steps.

The following sample output is from running fitted_model.steps for a chosen run:

[('RobustScaler', 
  RobustScaler(copy=True, 
  quantile_range=[10, 90], 
  with_centering=True, 
  with_scaling=True)), 

  ('LogisticRegression', 
  LogisticRegression(C=0.18420699693267145, class_weight='balanced', 
  dual=False, 
  fit_intercept=True, 
  intercept_scaling=1, 
  max_iter=100, 
  multi_class='multinomial', 
  n_jobs=1, penalty='l2', 
  random_state=None, 
  solver='newton-cg', 
  tol=0.0001, 
  verbose=0, 
  warm_start=False))

To get more details, use this helper function:

from pprint import pprint

def print_model(model, prefix=""):
    for step in model.steps:
        print(prefix + step[0])
        if hasattr(step[1], 'estimators') and hasattr(step[1], 'weights'):
            pprint({'estimators': list(e[0] for e in step[1].estimators), 'weights': step[1].weights})
            print()
            for estimator in step[1].estimators:
                print_model(estimator[1], estimator[0]+ ' - ')
        elif hasattr(step[1], '_base_learners') and hasattr(step[1], '_meta_learner'):
            print("\nMeta Learner")
            pprint(step[1]._meta_learner)
            print()
            for estimator in step[1]._base_learners:
                print_model(estimator[1], estimator[0]+ ' - ')
        else:
            pprint(step[1].get_params())
            print()   

This helper function returns the following output for a particular run using LogisticRegression with RobustScalar as the specific algorithm.

RobustScaler
{'copy': True,
'quantile_range': [10, 90],
'with_centering': True,
'with_scaling': True}

LogisticRegression
{'C': 0.18420699693267145,
'class_weight': 'balanced',
'dual': False,
'fit_intercept': True,
'intercept_scaling': 1,
'max_iter': 100,
'multi_class': 'multinomial',
'n_jobs': 1,
'penalty': 'l2',
'random_state': None,
'solver': 'newton-cg',
'tol': 0.0001,
'verbose': 0,
'warm_start': False}

Predict class probability

Models produced using automated ML all have wrapper objects that mirror functionality from their open-source origin class. Most classification model wrapper objects returned by automated ML implement the predict_proba() function, which accepts an array-like or sparse matrix data sample of your features (X values), and returns an n-dimensional array of each sample and its respective class probability.

Assuming you have retrieved the best run and fitted model using the same calls from above, you can call predict_proba() directly from the fitted model, supplying an X_test sample in the appropriate format depending on the model type.

best_run, fitted_model = automl_run.get_output()
class_prob = fitted_model.predict_proba(X_test)

If the underlying model does not support the predict_proba() function or the format is incorrect, a model class-specific exception will be thrown. See the RandomForestClassifier and XGBoost reference docs for examples of how this function is implemented for different model types.

BERT integration in automated ML

BERT is used in the featurization layer of automated ML. In this layer, if a column contains free text or other types of data like timestamps or simple numbers, then featurization is applied accordingly.

For BERT, the model is fine-tuned and trained utilizing the user-provided labels. From here, document embeddings are output as features alongside others, like timestamp-based features, day of week.

Learn how to set up natural language processing (NLP) experiments that also use BERT with automated ML.

Steps to invoke BERT

In order to invoke BERT, set enable_dnn: True in your automl_settings and use a GPU compute (vm_size = "STANDARD_NC6" or a higher GPU). If a CPU compute is used, then instead of BERT, AutoML enables the BiLSTM DNN featurizer.

Automated ML takes the following steps for BERT.

1. Preprocessing and tokenization of all text columns. For example, the "StringCast" transformer can be found in the final model's featurization summary. An example of how to produce the model's featurization summary can be found in this notebook.

2. Concatenate all text columns into a single text column, hence the StringConcatTransformer in the final model.

   Our implementation of BERT limits total text length of a training sample to 128 tokens. That means, all text columns when concatenated, should ideally be at most 128 tokens in length. If multiple columns are present, each column should be pruned so this condition is satisfied. Otherwise, for concatenated columns of length >128 tokens BERT's tokenizer layer truncates this input to 128 tokens.

3. As part of feature sweeping, AutoML compares BERT against the baseline (bag of words features) on a sample of the data. This comparison determines if BERT would give accuracy improvements. If BERT performs better than the baseline, AutoML then uses BERT for text featurization for the whole data. In that case, you will see the PretrainedTextDNNTransformer in the final model.

BERT generally runs longer than other featurizers. For better performance, we recommend using "STANDARD_NC24r" or "STANDARD_NC24rs_V3" for their RDMA capabilities.

AutoML will distribute BERT training across multiple nodes if they are available (upto a max of eight nodes). This can be done in your AutoMLConfig object by setting the max_concurrent_iterations parameter to higher than 1.

Supported languages for BERT in AutoML

AutoML currently supports around 100 languages and depending on the dataset's language, AutoML chooses the appropriate BERT model. For German data, we use the German BERT model. For English, we use the English BERT model. For all other languages, we use the multilingual BERT model.

In the following code, the German BERT model is triggered, since the dataset language is specified to deu, the three letter language code for German according to ISO classification:

from azureml.automl.core.featurization import FeaturizationConfig

featurization_config = FeaturizationConfig(dataset_language='deu')

automl_settings = {
    "experiment_timeout_minutes": 120,
    "primary_metric": 'accuracy',
# All other settings you want to use
    "featurization": featurization_config,
    
    "enable_dnn": True, # This enables BERT DNN featurizer
    "enable_voting_ensemble": False,
    "enable_stack_ensemble": False
}

Next steps

• Learn how to set up your automated ML experiments:

  • For a code-first experience: Configure automated ML experiments by using the Azure Machine Learning SDK.
  • For a low-code or no-code experience: Create your automated ML experiments in the Azure Machine Learning studio.

• Learn more about how and where to deploy a model.

• Learn more about how to train by using automated machine learning on a remote resource.