Support Vector Machine (SVM)
A Simple Numerical Example – Detailed Explanation
Author: Bindeshwar Singh Kushwaha
PostNetwork Academy
Introduction: Type and Purpose of SVM
- Type of Algorithm: Supervised Machine Learning Algorithm
- Used for Classification and Regression (SVR)
- Discriminative Model – finds decision boundaries
- Known as a Maximum-Margin Classifier
Purpose:
- Find the optimal hyperplane that separates classes
- Maximize margin between support vectors and boundary
- Handle non-linear data using kernel trick
Dataset Setup: Table of 8 Points
Two classes: \(+1\) (Blue), \(-1\) (Red)
Linearly separable by the hyperplane \(x_2 = 2\)
| Point | Coordinates $(x_1, x_2)$ | Class $y_i$ |
|---|---|---|
| $P_1$ | $(2,3)$ | $+1$ |
| $P_2$ | $(3,3)$ | $+1$ |
| $P_3$ | $(3,4)$ | $+1$ |
| $P_4$ | $(4,3)$ | $+1$ |
| $P_5$ | $(1,1)$ | $-1$ |
| $P_6$ | $(2,1)$ | $-1$ |
| $P_7$ | $(1,2)$ | $-1$ |
| $P_8$ | $(2,0.5)$ | $-1$ |
Step 1: Propose a Separating Hyperplane
Suppose \( w = \begin{bmatrix}0 \\ 1\end{bmatrix}, b = -2 \)
Decision boundary equation: \( w^T x + b = 0 \Rightarrow x_2 = 2 \)
Step 2: Verify Classification
Decision function: \( f(x) = w^T x + b = x_2 – 2 \)
Classification rule:
\[
\text{Predict }
\begin{cases}
+1, & f(x) > 0 \\
-1, & f(x) < 0
\end{cases}
\]
Check all points:
- Class +1: (2,3)→1, (3,3)→1, (3,4)→2, (4,3)→1
- Class −1: (1,1)→−1, (2,1)→−1, (1,2)→0, (2,0.5)→−1.5
All points correctly classified except $(1,2)$ exactly on the hyperplane.
SVM Constraint
\[
y_i(w^T x_i + b) \ge 1
\]
- If \(y_i = +1\), we require \(w^T x_i + b \ge 1\)
- If \(y_i = -1\), we require \(w^T x_i + b \le -1\)
- Points on the margin satisfy equality → support vectors
Step 3: Margin Calculation
\[
d = \frac{|w^T x_i + b|}{\|w\|}
\]
For $(2,3)$, $y=+1$ → \(d = 1\)
For $(1,1)$, $y=-1$ → \(d = 1\)
Margin width = 2
Step 4: Predicting a New Point
New point \(x_{new} = (2.5, 2.5)\)
\(f(x_{new}) = 2.5 – 2 = 0.5 > 0 \Rightarrow +1\)
Distance to hyperplane = 0.5
Summary
- Separating hyperplane: \(x_2 = 2\)
- Weight vector: \(w = [0, 1]^T, b = -2\)
- Margin width = 2, support vectors highlighted
- SVM constraint ensures all points outside margin
- New point $(2.5, 2.5)$ correctly classified as $+1$