Generative Model vs Discriminative Model

In this post, we are going to compare the two types of machine learning models-generative model and discriminative model-, whose underlying ideas are quite different. Also, a typical generative classification algorithm called Gaussian Discriminant Analysis will be introduced.

Discriminative Model

The most basic discriminative classifier in Machine Learning is Logistic Regression, and a decision boundary is learned to segregate points from two classes in training set. When a new point comes, the algorithm classifies it by checking which side of the boundary it falls. Logistic Regression do classification by calculating p(Y|X,θ), which is modeled by:

g is the sigmoid function. So this is the idea of Discriminative Algorithm: directly calculate p(Y|X,θ).

Generative Model

Generative Algorithm tries to achieve the same goal p(Y|X,θ) using another way: Bayes Rule, which is in the form of:

The key to solve this equation is to get p(X|Y), this need building seperate models for different Y.

When the input X is continuous random variable, we can then use the Gaussian Discriminant Analysis (GDA) model, which models p(x|y) using a multivariate normal distribution.

If we write out them:

then we can calculate the probability for a specific x using Bayes Rule. When X is discrete, we turn to Naive Bayes, which is also a Generative Algorithm.

In conclusion, the idea for Generative Model is to build a model for each class, and then use Bayes Rule to back up getting P(Y|X).

This picture shows the difference ideas of the two groups of algorithms.