Convert Word to Vector component

This article describes how to use the Convert Word to Vector component in Azure Machine Learning designer to do these tasks:

• Apply various Word2Vec models (Word2Vec, FastText, GloVe pretrained model) on the corpus of text that you specified as input.
• Generate a vocabulary with word embeddings.

This component uses the Gensim library. For more information about Gensim, see its official website, which includes tutorials and an explanation of algorithms.

More about converting words to vectors

Converting words to vectors, or word vectorization, is a natural language processing (NLP) process. The process uses language models to map words into vector space. A vector space represents each word by a vector of real numbers. It also allows words with similar meanings have similar representations.

Use word embeddings as initial input for NLP downstream tasks such as text classification and sentiment analysis.

Among various word embedding technologies, in this component, we implemented three widely used methods. Two, Word2Vec and FastText, are online-training models. The other is a pretrained model, glove-wiki-gigaword-100.

Online-training models are trained on your input data. Pretrained models are trained offline on a larger text corpus (for example, Wikipedia, Google News) that usually contains about 100 billion words. Word embedding then stays constant during word vectorization. Pretrained word models provide benefits such as reduced training time, better word vectors encoded, and improved overall performance.

Here's some information about the methods:

• Word2Vec is one of the most popular techniques to learn word embeddings by using a shallow neural network. The theory is discussed in this paper, available as a PDF download: Efficient Estimation of Word Representations in Vector Space. The implementation in this component is based on the Gensim library for Word2Vec.

• The FastText theory is explained in this paper, available as a PDF download: Enriching Word Vectors with Subword Information. The implementation in this component is based on the Gensim library for FastText.

• The GloVe pretrained model is glove-wiki-gigaword-100. It's a collection of pretrained vectors based on a Wikipedia text corpus, which contains 5.6 billion tokens and 400,000 uncased vocabulary words. A PDF download is available: GloVe: Global Vectors for Word Representation.

How to configure Convert Word to Vector

This component requires a dataset that contains a column of text. Preprocessed text is better.

1. Add the Convert Word to Vector component to your pipeline.

2. As input for the component, provide a dataset that contains one or more text columns.

3. For Target column, choose only one column that contains text to process.

   Because this component creates a vocabulary from text, the content of columns differs, which leads to different vocabulary contents. That's why the component accepts only one target column.

4. For Word2Vec strategy, choose from GloVe pretrained English Model, Gensim Word2Vec, and Gensim FastText.

5. If Word2Vec strategy is Gensim Word2Vec or Gensim FastText:

   • For Word2Vec Training Algorithm, choose from Skip_gram and CBOW. The difference is introduced in the original paper (PDF).

     The default method is Skip_gram.

   • For Length of word embedding, specify the dimensionality of the word vectors. This setting corresponds to the size parameter in Gensim.

     The default embedding size is 100.

   • For Context window size, specify the maximum distance between the word being predicted and the current word. This setting corresponds to the window parameter in Gensim.

     The default window size is 5.

   • For Number of epochs, specify the number of epochs (iterations) over the corpus. Corresponds to the iter parameter in Gensim.

     The default epoch number is 5.

6. For Maximum vocabulary size, specify the maximum number of the words in the generated vocabulary.

   If there are more unique words than the max size, prune the infrequent ones.

   The default vocabulary size is 10,000.

7. For Minimum word count, provide a minimum word count. The component will ignore all words that have a frequency lower than this value.

   The default value is 5.

8. Submit the pipeline.

Examples

The component has one output:

• Vocabulary with embeddings: Contains the generated vocabulary, together with each word's embedding. One dimension occupies one column.

The following example shows how the Convert Word to Vector component works. It uses Convert Word to Vector with default settings to the preprocessed Wikipedia SP 500 Dataset.

Source dataset

The dataset contains a category column, along with the full text fetched from Wikipedia. The following table shows a few representative examples.

Output vocabulary with embeddings

The following table contains the output of this component, taking the Wikipedia SP 500 dataset as input. The leftmost column indicates the vocabulary. Its embedding vector is represented by values of remaining columns in the same row.

In this example, we used the default Gensim Word2Vec for Word2Vec strategy, and Training Algorithm is Skip-gram. Length of word Embedding is 100, so we have 100 embedding columns.

Technical notes

This section contains tips and answers to frequently asked questions.

• Difference between online-training and pretrained model:

  In this Convert Word to Vector component, we provided three different strategies: two online-training models and one pretrained model. The online-training models use your input dataset as training data, and generate vocabulary and word vectors during training. The pretrained model is already trained by a much larger text corpus. The pretrained model is actually a collection of word/embedding pairs.

  The GloVe pre-trained model summarizes a vocabulary from the input dataset and generates an embedding vector for each word from the pretrained model. Without online training, the use of a pretrained model can save training time. It has better performance, especially when the input dataset size is relatively small.

• Embedding size:

  In general, the length of word embedding is set to a few hundred. For example, 100, 200, 300. A small embedding size means a small vector space, which could cause word embedding collisions.

  The length of word embeddings is fixed for pretrained models. In this example, the embedding size of glove-wiki-gigaword-100 is 100.

Next steps

See the set of components available to Azure Machine Learning.

For a list of errors specific to the designer components, see Machine Learning error codes.