K-Nearest Neighbors (KNN) Classifier and Imputation using KNN
Author: Bindeshwar Singh Kushwaha
What is K-Nearest Neighbors (KNN)?
- KNN is a supervised machine learning algorithm.
- It is easy to understand and does not involve complex math.
- Commonly used for classification tasks, especially with labeled data.
- We’ll use the Iris dataset, which has flower measurements.
Iris Dataset Sample Features
- The dataset includes four features:
- Sepal Length
- Sepal Width
- Petal Length
- Petal Width
- Target class is the Iris flower type: Setosa, Versicolor, or Virginica.
Sample Data Points
In the original dataset, there are 150 instances with four features and three classes. For simplicity, we use only 10 instances, 9 labeled and 1 unlabeled.
| S. No. | Sepal Len. | Sepal Width | Petal Len. | Petal Width | Label |
|---|---|---|---|---|---|
| 1 | 4.6 | 3.4 | 1.4 | 0.3 | Iris-setosa |
| 2 | 5.0 | 3.4 | 1.5 | 0.2 | Iris-setosa |
| 3 | 4.4 | 2.9 | 1.4 | 0.2 | Iris-setosa |
| 4 | 7.0 | 3.2 | 4.7 | 1.4 | Iris-versicolor |
| 5 | 6.4 | 3.2 | 4.5 | 1.5 | Iris-versicolor |
| 6 | 6.9 | 3.1 | 4.9 | 1.5 | Iris-versicolor |
| 7 | 6.3 | 3.3 | 6.0 | 2.5 | Iris-virginica |
| 8 | 5.8 | 2.7 | 5.1 | 1.9 | Iris-virginica |
| 9 | 7.1 | 3.0 | 5.9 | 2.1 | Iris-virginica |
| 10 | 5.0 | 3.3 | 1.4 | 0.2 | Unknown |
Understanding KNN with Distance Calculation
Distance from instance 10 to instance 1:
\( d(1,10) = \sqrt{(4.6 – 5.0)^2 + (3.4 – 3.3)^2 + (1.4 – 1.4)^2 + (0.3 – 0.2)^2} = 0.424 \)
Other Distances from Instance 10
d(2,10) = 0.141 d(3,10) = 0.721 d(4,10) = 4.042 d(5,10) = 3.643 d(6,10) = 4.194 d(7,10) = 5.305 d(8,10) = 4.193 d(9,10) = 5.325
Conclusion with k = 3
- Nearest: 0.141, 0.424, 0.721 (all Iris-setosa)
- Predicted class for instance 10: Iris-setosa
KNN Classifier (Python Steps)
from sklearn.neighbors import KNeighborsClassifier X_train = [...] y_train = [...] model = KNeighborsClassifier(n_neighbors=3, p=2) model.fit(X_train, y_train) X_pred = [[5, 3.3, 1.4, 0.2]] prediction = model.predict(X_pred)
What is KNN Imputation?
- Used to fill missing values based on nearest neighbors.
- Uses available features to compute distances.
- Impute with mean/mode from nearest rows.
Sample Dataset (with Missing Value)
| ID | Height (cm) | Weight (kg) | Age |
|---|---|---|---|
| 1 | 160 | 55 | 25 |
| 2 | 165 | 60 | 27 |
| 3 | NaN | 62 | 28 |
| 4 | 170 | 65 | 29 |
| 5 | 158 | 54 | 24 |
Imputation Steps (k = 2)
- Compare row 3 with others using Weight & Age only.
- Closest: Row 2 and 4
- Imputed height = \( \frac{165 + 170}{2} = 167.5 \)
Final Imputed Dataset
| ID | Height (cm) | Weight (kg) | Age |
|---|---|---|---|
| 1 | 160 | 55 | 25 |
| 2 | 165 | 60 | 27 |
| 3 | 167.5 | 62 | 28 |
| 4 | 170 | 65 | 29 |
| 5 | 158 | 54 | 24 |
Python Steps for KNN Imputation
from sklearn.impute import KNNImputer
import pandas as pd
import numpy as np
df = pd.DataFrame({...})
imputer = KNNImputer(n_neighbors=2, metric='nan_euclidean')
imputed = imputer.fit_transform(df.drop("ID", axis=1))
Video
Follow PostNetwork Academy
- Website: www.postnetwork.co
- YouTube: www.youtube.com/@postnetworkacademy
- Facebook: www.facebook.com/postnetworkacademy
- LinkedIn: www.linkedin.com/company/postnetworkacademy
- GitHub: www.github.com/postnetworkacademy