Naive Bayes is a probabilistic classification algorithm.
It is based on Bayes’ Theorem and the naive independence assumption.
Suppose we have a feature vector $\mathbf{X} = (x_1, x_2, …, x_n)$ and a class $y$.
Bayes Theorem:
$$P(y|\mathbf{X}) = \frac{P(\mathbf{X}|y) \cdot P(y)}{P(\mathbf{X})}$$
Here:
- $P(y|\mathbf{X})$ = Posterior probability of class $y$ given features $\mathbf{X}$
- $P(\mathbf{X}|y)$ = Likelihood of features given class $y$
- $P(y)$ = Prior probability of class $y$
- $P(\mathbf{X})$ = Evidence (same for all classes, can be ignored)

Independence Assumption

Naive Bayes assumes that features are conditionally independent given the class:

$$P(\mathbf{X}|y) = P(x_1, x_2, …, x_n | y) \approx \prod_{i=1}^{n} P(x_i|y)$$

This makes computation efficient, especially for high-dimensional features. Posterior probability becomes:

$$P(y|\mathbf{X}) \propto P(y) \prod_{i=1}^{n} P(x_i|y)$$

Classification Rule

To classify a new instance $\mathbf{X}$:

$$\hat{y} = \arg\max_{y} P(y) \prod_{i=1}^{n} P(x_i|y)$$

Compute posterior probabilities for each class.
Predict the class $y$ with the highest posterior probability.

Pros and Cons

Advantages:

Simple and fast to train
Works well with small datasets
Effective for text classification (spam detection, sentiment analysis)

Disadvantages:

Assumes feature independence (rarely perfectly true)
Can perform poorly with correlated features

Worked Example: Weather Dataset (PlayTennis)

Features: Outlook ($x_1$), Temperature ($x_2$), Humidity ($x_3$), Wind ($x_4$)
Target: PlayTennis ($y$)

Day	Outlook	Temp	Humidity	Wind	PlayTennis
1	Sunny	Hot	High	Weak	No
2	Sunny	Hot	High	Strong	No
3	Overcast	Hot	High	Weak	Yes
4	Rain	Mild	High	Weak	Yes
5	Rain	Cool	Normal	Weak	Yes
6	Rain	Cool	Normal	Strong	No
7	Overcast	Cool	Normal	Strong	Yes
8	Sunny	Mild	High	Weak	No
9	Sunny	Cool	Normal	Weak	Yes
10	Rain	Mild	Normal	Weak	Yes
11	Sunny	Mild	Normal	Strong	Yes
12	Overcast	Mild	High	Strong	Yes
13	Overcast	Hot	Normal	Weak	Yes
14	Rain	Mild	High	Strong	No

Step 1: Compute Prior Probabilities

Count of Yes = 9, Count of No = 5, Total instances = 14

$$P(y=\text{Yes}) \approx 0.643, \quad P(y=\text{No}) \approx 0.357$$

Step 2: Compute Likelihoods

Outlook ($x_1$):

$$P(\text{Sunny}|Yes)=0.222, \quad P(\text{Sunny}|No)=0.6$$
$$P(\text{Overcast}|Yes)=0.444, \quad P(\text{Overcast}|No)=0$$
$$P(\text{Rain}|Yes)=0.333, \quad P(\text{Rain}|No)=0.4$$

Temperature ($x_2$):

$$P(\text{Hot}|Yes)=0.222, \; P(\text{Cool}|Yes)=0.333, \; P(\text{Mild}|Yes)=0.444$$
$$P(\text{Hot}|No)=0.4, \; P(\text{Cool}|No)=0.2, \; P(\text{Mild}|No)=0.4$$

Humidity ($x_3$):

$$P(\text{High}|Yes)=0.333, \; P(\text{Normal}|Yes)=0.667$$
$$P(\text{High}|No)=0.8, \; P(\text{Normal}|No)=0.2$$

Wind ($x_4$):

$$P(\text{Weak}|Yes)=0.667, \; P(\text{Strong}|Yes)=0.333$$
$$P(\text{Weak}|No)=0.4, \; P(\text{Strong}|No)=0.6$$

Step 3: Posterior Probability for Yes

New instance: $\mathbf{X}_{15} = (\text{Sunny, Cool, High, Strong})$

$$P(y=\text{Yes}|\mathbf{X}_{15}) \propto 0.643 \cdot 0.222 \cdot 0.333 \cdot 0.333 \cdot 0.333 \approx 0.00525$$

Step 4: Posterior Probability for No

$$P(y=\text{No}|\mathbf{X}_{15}) \propto 0.357 \cdot 0.6 \cdot 0.2 \cdot 0.8 \cdot 0.6 \approx 0.0206$$

Step 5: Classification Decision

Since $0.0206 > 0.00525$, the 15th instance is classified as:

$$\hat{y} = \text{No}$$

PDF

NaiveBaysWeatherDataSet

Video

Reach PostNetwork Academy

Website: www.postnetwork.co
YouTube Channel: PostNetwork Academy
Facebook Page: PostNetwork Academy
LinkedIn Page: PostNetwork Academy
GitHub Repos: PostNetwork Academy

Thank You!

Naive Bayes Classification Algorithm for Weather Dataset

Introduction to Naive Bayes Classifier