Natural language processing question bank 01

Differentiate between generative and discriminative models

Question:

Difference between generative and discriminative models.

Answer:

Generative and discriminative models are probability models.

In generative model, we model the conditional probability of the input x given the label y. Generative model learns the joint probability distribution P(x, y) and uses Bayes’ theorem to find the conditional probability. 

In discriminative model, we directly model the conditional probability P(y|x).

• Generative models estimate joint distribution but discriminative models estimate conditional distribution.

• In generative models, dependence assumption has to be specified for P(w|t) and P(t) but in discriminative models, arbitrary features can be incorporated for modeling P(t|w).

• Generate models can be used in unsupervised learning but discriminative models require labeled data, suitable for (semi-) supervised learning.

*************************

Related questions:

• Why is it better to compute likelihood and probability values in log space?

• One common way probabilities are assigned to rules in a PCFG is to use a treebank. When will some rules have a probability of 0.0? What is the problem if some rule has a probability of 0.0? How can it be overcome?

• You are building an ngram model of a corpus. Should you stem the words and do the counts or leave them in the surface form? Give pros and cons and include what characteristics of the corpus might influence your decision.

• List down the features of PCFGs

• How many parameters are required to fully define a Hidden Markov Model?  

• Differentiate between generative and discriminative models.