Word Embedding NLP
In Natural Language Processing, word embeddings are a feature learning technique that maps vocabulary words to vectors of real numbers. Known as an embedding, they represent every word as a point in high-dimensional space. This method, which is a type of vector semantics, learns word meaning representations straight from the distribution of words in texts.
Word embeddings produce short, dense vectors in contrast to conventional, sparse vector representations such as Bag of Words (BoW) or TF-IDF (term frequency-inverse document frequency). Usually between 50 and 1000, the number of dimensions, represented by the letter “d,” is substantially smaller than the vocabulary size. The dimensions in these dense vectors lack a straightforward, unambiguous interpretation, in contrast to the dimensions in sparse models.
The distributional hypothesis, which holds that a word’s meaning can be deduced from its usage settings, is the foundation of word embeddings. These techniques learn representations in which comparable words typically have similar vectors by looking for co-occurrence patterns in big textual collections. This indicates that they record contextual hierarchy along with syntactic and semantic information and word relationships.
Word embedding techniques
There are various techniques for creating word embeddings, which can be broadly divided into:
- Count Vector (Bag of Words), TF-IDF Vector, and Co-Occurrence Vector are examples of frequency-based embeddings. These are determined by the distribution of word counts.
- Usually called word embeddings, prediction-based embeddings frequently make use of shallow neural networks. Among them are models such as Skip-Gram and CBOW (Continuous Bag of Words).
- Another way to differentiate word embeddings is by whether they generate a single vector for every word form or a separate vector based on the context:
- Static Embeddings: Word2Vec, GloVe, and FastText are examples of techniques that learn a single, fixed embedding for every distinct word in the lexicon.
- One well-known framework that trains word embeddings using shallow neural networks is Word2Vec (Mikolov et al., 2013). Its goal is to produce vectors that capture word context. The two primary varieties are CBOW, which predicts a target word from its context, and Skip-Gram, which predicts context words from a target word. In terms of word analogies and similarity, Word2Vec is said to do better than other approaches.
- Global Vectors, or GloVe (Pennington et al., 2014), is another popular static embedding paradigm. Its foundation lies in the use of ratios of probability from the word-word co-occurrence matrix to capture global corpus statistics.
- FastText is a Word2Vec addon (Bojanowski et al., 2017). By using subword models, which represent each word as itself plus a bag of constituent n-grams (similar to character n-grams), it tackles the problem of unknown words and word sparsity, particularly in morphologically rich languages. The total of the embeddings of the word’s constituent n-grams is the word embedding.
- Contextualized Embeddings: Transformer Network-based models and BERT (Bidirectional Encoder Representations from Transformers) models develop embeddings in which a word’s vector varies depending on the context in which it appears. This gives the representation the ability to capture the particular meaning (word sense) of the word instance in its textual context, which is a major benefit over static embeddings. In order to avoid functioning as a straightforward bag-of-words model, transformer networks integrate location information. The hidden states of deep models, such Transformers or bi-directional LSTMs, can yield contextualized embeddings.
Why Word Embeddings Are Essential in NLP?
The following reasons make word embeddings essential:
- Lower dimensionality in contrast to sparse vectors with high dimensions.
- Permit word semantic associations to be captured by models.
- Make it possible to employ dense, learnt representations, which are useful inputs for neural networks and other machine learning models.
Large amounts of unannotated text are usually used to learn word embeddings. Using pretrained word embeddings that have been trained on large corpora is standard procedure. For a variety of NLP tasks, these pretrained embeddings can be a helpful initialization or basis layer. Off-the-shelf embeddings can be optimized on a smaller target corpus if task-specific data is scarce.
Characteristics and Applications
Word embeddings have the following characteristics:
- Measuring Word Similarity: The cosine similarity between the embedding vectors of words is commonly used to quantify how similar they are. High cosine similarity is a sign of semantic or syntactic similarity.
- Word Analogies: Word analogies like “king – man + woman ≈ queen” are made possible by vector operations, which can disclose semantic links.
- Visualization: Embeddings can be shown via clustering, projecting high-dimensional vectors into 2D space using methods like t-SNE, or displaying the most similar words (closest in cosine).
Applications: Word embeddings are frequently employed in a range of NLP tasks, such as
- Text Categorization.
- Information Retrieval.
- Sentiment Analysis.
- WSD (word sense disambiguation), especially when contextual embeddings are used.
- Entity Linking.
- Coreference Resolution.
- Query Improvement/Expansion.
- Examining how semantics evolve throughout time.
- Acting as the foundational layer of additional supervised neural networks.
Because they combine all senses into a single vector, static embeddings suffer with polysemous words, or words with numerous meanings, even though they offer a basic representation for a word type. This is addressed with contextualized embeddings, which give each word token a unique vector based on its particular context. Another difficulty is handling unknown words, or words that are not in the training vocabulary; methods like as employing subword units (as in FastText), a designated ‘UNK’ symbol, or averaging pre-existing embeddings are used. For example, by retrofitting vectors to make synonyms closer and antonyms farther away, external lexical resources such as thesauruses (like WordNet) can also be utilized to enhance embeddings.
In conclusion, a significant departure from the explicit feature engineering used in traditional NLP pipelines, word embeddings offer rich, learnt vector representations of words that capture semantic and syntactic meaning and context. This allows for the development of more potent deep learning models for a variety of applications.