Naive Bayes Classifier

Naive Bayes is often the first algorithm taught for text classification. It is built entirely on probability and counting, with no iterative "training" loop. Instead, it relies on Bayes' rule to update a prior belief based on new evidence.

The "naive" in its name comes from a massive assumption it makes: that every feature is perfectly independent of every other feature. While this is almost never true in the real world—the word "urgent" and the word "win" often appear together—pretending they are independent makes the math incredibly simple and surprisingly effective.

The Probability Chain

To classify a message, Naive Bayes starts with a Prior probability: how likely is a message to be Spam before we even look at its contents? If 50% of your dataset is Spam, your starting belief is 0.5.

Then, for every word in the message, it asks: How likely is it to see this word in a Spam message? This is the Likelihood. It simply multiplies the prior by the likelihood of each word.

$$P(Spam | \text{"urgent win"}) \propto P(Spam) \times P(\text{"urgent"} | Spam) \times P(\text{"win"} | Spam)$$

In the interactive visual above, you can see this chain of multiplications happening step-by-step. Each new piece of evidence scales the running product up or down.

The Zero-Frequency Problem

This multiplication reveals a fatal flaw. What happens if the word "meeting" has never appeared in a Spam message in your training data?

Its likelihood, $P(\text{"meeting"} | Spam)$, would be exactly 0. Because the probabilities are multiplied together, that single 0 destroys the entire product. Even if the message contains 100 other words that strongly indicate Spam, the final unnormalized score will be 0.

Laplace Smoothing

To fix this, we apply Laplace Smoothing (often called add-one smoothing, controlled by $\alpha$). We simply pretend we saw every possible word at least $\alpha$ times.

$$P(w | C) = \frac{\text{count}(w, C) + \alpha}{\text{total words}(C) + \alpha \times \text{vocabulary size}}$$

By adding a small, artificial count to every word, we guarantee that no probability is ever exactly zero. Try turning the smoothing down to 0 in the lab and watch how a single unexpected word can veto all other evidence.

Explore Next

• Classification Threshold
• k-Nearest Neighbors