Generative and Discriminative Models Applications & Examples

The contrast between generative and discriminative models is important for NLP tasks like sequence prediction and parsing.

The two categories of models are broken out as follows:

Generative Model

• A combined probability distribution over inputs (x) and outputs (y) is defined by a generative model and is commonly written as P(x, y).
• Given the label (y), they model the conditional probability of the input (x), P(x|y).
• The objective is to comprehend how a class may produce the input data and to develop a model that forecasts the joint probability of a (input, label) pair.
• A generative model returns the class that is most likely to have produced an observation.
• The distinction between conditional (discriminative) and generative models is compared to a topographical map and a photograph of a skyline. Similar to a generative model, the topographical map may be used to address a greater range of concerns; however, it is far more challenging to create properly than a skyline image.

Applications & Examples of Generative Models

• Classification uses generative model Naive Bayes.
• Part-of-speech (POS) tagging often uses generative sequence classification methods like Hidden Markov Models (HMMs). There is a close relationship between visible Markov models (VMMs).
• Generative models are used in statistical parsing to model the joint probability P(x, y). They frequently combine a parsing model’s generating and evaluative elements. History-based models and lexicalised models, like Model 2, one of the three suggested by Collins (1997), are examples. Another generative model that draws inspiration from head-driven history-based models is the Dependency Model with Valence (DMV). It is also possible to use Probabilistic Context-Free Grammars (PCFGs) as models for generative parsing.
• Because generative models enable the derivation of P(y|x) and P(x) through conditionalization and marginalisation, they may be used to both language modelling and parsing.
• Generative techniques, like the phrase-based translation model or the generative models of transduction used for bittext alignment, are a part of statistical machine translation (SMT).
• Generative models are language models (LMs) that give word sequences probability. The most basic kind is the n-gram model.
• In models that gain from both generative and discriminative training, generative models can be used as sentence encoders.

Using a generative classification model has the drawback of potentially limiting the characteristics that may be classified, especially when the expectation-maximization technique is employed.

Discriminative Model

• The conditional probability of the output (y) given the input (x), P(y|x), is directly modelled by a discriminative model. This conditional probability is the only one they utilize.
• Their objective is to determine which input qualities are most helpful in differentiating between the several potential classes. They concentrate on characteristics that classify the variations among instances.
• To select the class with the highest conditional probability, the class conditional probability function P(y|x) may be thought of as a scoring function.
• They are frequently employed to carry out a statistical parsing model’s evaluative component.
• Given the input, discriminative models seek to maximize the likelihood of a right analysis. This may include maximizing the likelihood of a thorough analysis (global discriminative models) or local decisions (local discriminative models).

Applications & Examples of Discriminative Models

• One popular discriminative approach is the logistic regression model. A different way to formulate logistic regression is to use Maximum Entropy (MaxEnt).
• Regression, maximum entropy, boosting, and Support Vector Machines (SVMs) are examples of discriminative classification techniques. In the context of discriminative training techniques for HMMs, SVMs and the perceptron algorithm are also discussed.
• Conditional Random Fields (CRFs) are a discriminative method for sequence prediction/labeling. Decisions made by local discriminative models are dependent on pertinent portions of the input sentence (x) and the derivation history. The full label sequence is directly treated by global discriminative models for sequence prediction. For discriminative sequence labelling, structured SVMs and structured perceptrons are also employed.
• Discriminative models in statistical parsing might be either global or local models. Global discriminative models enable component scores to be set on the full study and can include features that represent global attributes, whereas local discriminative models are frequently conditional history-based models. In transition-based dependency parsing, discriminative techniques like SVMs, perceptron learning, and memory-based learning have gained popularity. Additionally, log-linear models have been employed. Collins developed a framework for discriminative parsing techniques known as the global linear model.
• The top candidates produced by a base parser (often generative) are reranked using a global discriminative model in discriminative reranking models, which frequently result in notable accuracy gains. In the global discriminative model, the log probability that the generative base parser assigns is usually a crucial component.
• In many speech and language processing tasks, discriminative models have been demonstrated to perform better than generative techniques and are frequently more accurate. They are able to take use of the possibility of possessing characteristics of arbitrary complexity. In statistical parsing, global discriminative models appear to have a minor empirical advantage, particularly those that incorporate global characteristics.
• Models based on discriminative features were employed in early work on semantic role labelling.
• Discriminative methods are also used in statistical machine translation.

Difference Between Generative and Discriminative Models

• The fundamental distinction is that discriminative models model P(y|x), whereas generative models model P(x, y) or P(x|y).
• The difference between the generating (which maps input to candidate analyses) and evaluative (which ranks analyses) components of a statistical parsing model is connected to, but separate from, the difference between generative and discriminative models in parsing. While discriminative models usually apply the evaluative component, generative models usually combine two elements.
• One way to think of local discriminative models is as discriminative iterations of generative models.
• The output of a generative model is frequently included into the feature representations of discriminative parsers.
• Discriminative models concentrate on the characteristics that aid in class distinction, but generative models could offer insights into the process of data generation (such as creating an image of a dog).
• Accuracy and efficiency are traded off in statistical parsing; models that consider more global structure have a higher capacity for disambiguation but become more computationally demanding. Although they may be slower, discriminative models that include global characteristics are typically more accurate.

In conclusion, discriminative models concentrate directly on the decision border between classes, learning to predict the class label from the input features, whereas generative models concentrate on the underlying distribution of the data and how inputs are created given a class. Both strategies are frequently used to different NLP problems, with discriminative models frequently demonstrating superior empirical performance in classification tasks.