Introduction
Imagine you move to a new neighborhood and want to know which restaurant you might like.
You could ask people who live nearby:
"Which restaurants do you usually visit?"
If most people with tastes similar to yours recommend the same restaurant, you might choose it too.
This simple idea is surprisingly close to how one machine learning algorithm works.
It is called:
K-Nearest Neighbors
or simply:
KNN
Instead of building a complicated tree or combining hundreds of models, KNN looks at nearby examples in the data and uses them to make a prediction.
But what does "nearby" mean when data has many features?
How does KNN decide how many neighbors to use?
And why is the letter K so important?
Let's understand KNN step by step.
Quick Answer: What Is K-Nearest Neighbors?
K-Nearest Neighbors (KNN) is a supervised machine learning algorithm that makes predictions by finding the K training examples closest to a new data point.
For classification, KNN commonly uses the classes of those neighbors to make a vote.
For regression, it can use the values of the nearest neighbors to estimate a numerical result.
The basic idea is:
New Data Point ↓ Find Nearby Examples ↓ Choose K Neighbors ↓ Look at Their Results ↓ Make Prediction
1. What Does "K" Mean?
The K in KNN represents the number of neighbors the algorithm considers.
For example:
K = 3
means:
Look at the 3 nearest neighbors.
If:
K = 5
the algorithm considers:
5 nearest neighbors.
Scikit-learn's KNeighborsClassifier uses n_neighbors to specify this number, with a default of 5.
2. What Does "Nearest" Mean?
Nearest means:
Most similar according to a chosen distance measurement.
Imagine points on a graph.
A X B C
If X is the new data point, the algorithm searches for the existing points closest to X.
Those become its neighbors.
3. A Simple Real-Life Example
Imagine a school wants to predict whether a new student will enjoy a particular activity.
The system already has information about previous students.
Suppose the new student is most similar to:
- Student A
- Student B
- Student C
and all three enjoyed football.
KNN might predict:
The new student will probably enjoy football.
The model is using similarity between examples.
4. KNN Is Based on Similarity
The central idea is:
Similar data points tend to have similar outcomes.
This makes KNN different from the Decision Tree and Random Forest algorithms we studied earlier.
Decision Tree
Learns decision rules.
Random Forest
Combines many Decision Trees.
KNN
Looks at nearby examples.
This is why KNN is often called an instance-based or non-generalizing learning method: it keeps the training examples and uses them when making predictions rather than building a traditional fitted model in advance.
5. How Does KNN Make a Classification?
Let's imagine we want to classify a new fruit.
We already have examples of:
- Apples
- Oranges
- Bananas
The new fruit is represented by its features.
KNN searches for nearby examples.
Suppose:
Neighbor 1 → Apple Neighbor 2 → Apple Neighbor 3 → Orange Neighbor 4 → Apple Neighbor 5 → Orange
If:
K = 5
then the votes are:
Apple = 3
Orange = 2
The prediction becomes:
Apple
This is called majority voting.
6. Visualizing KNN
Imagine the following:
● Apple ● Apple X ← New Point ● Orange ● Apple
The algorithm asks:
Which existing points are closest to X?
Then it examines their labels.
If most nearby points are Apple, the new point may be classified as Apple.
7. What Happens When K = 1?
Suppose:
K = 1
The algorithm looks at only the closest example.
New Point ↓ Closest Neighbor ↓ Prediction
This can make the model very sensitive to individual examples and noise.
For example, if the closest training point happens to be unusual, the prediction can be affected strongly.
8. What Happens When K Is Larger?
Now imagine:
K = 15
The model considers 15 nearby examples instead of one.
This can make the prediction less sensitive to one unusual data point.
However, using a very large K can also make the model less sensitive to local patterns.
Scikit-learn notes that, in general, a larger K tends to suppress noise but can make classification boundaries less distinct.
9. Choosing the Right K
Choosing K is important.
There is no single K value that is best for every dataset.
You can test different values such as:
K = 1 K = 3 K = 5 K = 7 K = 9
Then evaluate the model.
The best value depends on the data and the problem.
10. Why Is an Odd K Sometimes Useful?
For a simple two-class classification problem, an odd K can reduce the chance of a tie.
For example:
K = 5
could produce:
Class A → 3 votes Class B → 2 votes
But:
K = 4
could produce:
Class A → 2 votes Class B → 2 votes
An odd K does not automatically make a model better, but it can make simple majority voting easier to resolve.
11. How Does KNN Measure Distance?
KNN needs a way to determine which points are close to one another.
One common measurement is:
Euclidean Distance
It is the familiar straight-line distance between points.
For two points:
A = (x₁, y₁) B = (x₂, y₂)
the Euclidean distance is:
√[(x₂ − x₁)² + (y₂ − y₁)²]
Scikit-learn's default Minkowski metric with p=2 corresponds to Euclidean distance.
12. Simple Distance Example
Suppose we have:
Point A = (2, 3) Point B = (5, 7)
The distance is:
√[(5 − 2)² + (7 − 3)²] = √(3² + 4²) = √25 = 5
So the distance between the two points is:
5
KNN uses distance calculations like this to determine which training examples are closest.
13. Why Feature Scaling Matters
This is an important concept for KNN.
Imagine your data contains:
- Age: 18–70
- Annual income: 20,000–200,000
Income has much larger numerical values.
Without appropriate scaling, a feature with a large numerical range can have a much stronger effect on distance calculations.
Therefore, feature scaling is often important when using distance-based algorithms such as KNN.
14. KNN Classification
KNN classification predicts a category.
Examples include:
- Spam / Not Spam
- Fraud / Normal
- Buy / Not Buy
- Cat / Dog
- Approved / Rejected
The process is:
New Data ↓ Find K Nearest Neighbors ↓ Check Their Classes ↓ Majority Vote ↓ Prediction
15. KNN Regression
KNN can also be used for regression.
Instead of predicting a category, it predicts a numerical value.
For example:
House price
Suppose the nearest houses have prices:
$280,000 $300,000 $310,000 $290,000 $320,000
A KNN regression model can combine the target values of nearby examples, commonly using their mean.
The result might be approximately:
$300,000
16. KNN Classification vs KNN Regression
| Feature | KNN Classification | KNN Regression |
|---|---|---|
| Prediction | Category | Number |
| Example | Spam / Not Spam | House price |
| Basic method | Neighbor voting | Neighbor values |
| Main parameter | K | K |
| Output | Class | Numerical value |
Both use the same fundamental idea:
Find nearby examples and use their information.
17. Distance-Weighted KNN
Not every neighbor has to have the same influence.
Imagine:
Neighbor A → Distance 1 Neighbor B → Distance 5
Neighbor A is much closer.
It may make sense to give Neighbor A more influence.
KNN can use distance weighting, where closer points receive greater influence. Scikit-learn supports both uniform weighting and distance-based weighting.
18. Uniform vs Distance Weighting
Uniform Weighting
Every neighbor gets equal influence.
Neighbor 1 → Equal Neighbor 2 → Equal Neighbor 3 → Equal
Distance Weighting
Closer neighbors receive more influence.
Very Close → Stronger Influence Farther Away → Weaker Influence
This can be useful when nearby examples are more representative of the new point.
19. KNN in Python
Scikit-learn provides KNeighborsClassifier for KNN classification.
A simple example:
from sklearn.neighbors import KNeighborsClassifier model = KNeighborsClassifier(n_neighbors=5) model.fit(X_train, y_train) prediction = model.predict(X_test)
Let's understand it.
Create the model
model = KNeighborsClassifier(n_neighbors=5)
This tells the model to use:
5 neighbors
Train
model.fit(X_train, y_train)
The training examples are stored for later neighbor searches.
Predict
prediction = model.predict(X_test)
The algorithm finds nearby training examples and uses their labels to produce predictions.
20. KNN Regression in Python
For regression, Scikit-learn provides KNeighborsRegressor.
Example:
from sklearn.neighbors import KNeighborsRegressor model = KNeighborsRegressor(n_neighbors=5) model.fit(X_train, y_train) prediction = model.predict(X_test)
This model uses the target values of nearby training examples to estimate the numerical output.
21. Advantages of KNN
Simple Concept
The basic idea is easy to understand:
Find similar examples and use them.
Easy to Implement
Libraries such as Scikit-learn make implementation straightforward.
Can Handle Classification
KNN can predict categories.
Can Handle Regression
KNN can also predict numerical values.
Flexible Distance Measures
Different distance metrics can be used depending on the problem. Scikit-learn supports several distance metrics through its neighbor algorithms.
22. Limitations of KNN
KNN also has important limitations.
Prediction Can Become Expensive
Because the algorithm searches for neighbors when making predictions, large datasets can make prediction more computationally demanding.
Sensitive to Feature Scaling
Features with very different scales can distort distance calculations.
Sensitive to K
A poor choice of K can hurt performance.
High-Dimensional Data Can Be Difficult
Nearest-neighbor methods become less effective in high-dimensional spaces because distances become less informative, a problem often described as the curse of dimensionality.
23. KNN vs Decision Tree
You have already learned Decision Trees.
Here is a simple comparison:
| Feature | KNN | Decision Tree |
|---|---|---|
| Main idea | Find similar examples | Learn decision rules |
| Structure | No traditional tree | Tree |
| Classification | Yes | Yes |
| Regression | Yes | Yes |
| Distance important | Yes | No |
| Feature scaling | Often important | Usually less important |
| Easy to understand | Yes | Yes |
| Prediction process | Find neighbors | Follow branches |
24. KNN vs Random Forest
Now compare KNN with Random Forest.
| Feature | KNN | Random Forest |
|---|---|---|
| Main idea | Similarity | Multiple Decision Trees |
| Model type | Instance-based | Ensemble |
| Classification | Yes | Yes |
| Regression | Yes | Yes |
| Distance | Important | Not the core idea |
| Interpretability | Conceptually simple | More complex |
| Scaling | Often important | Generally less important |
These algorithms solve problems in very different ways.
25. When Should You Use KNN?
KNN can be useful when:
- Similar examples are expected to have similar outcomes.
- The dataset is not extremely large.
- The number of features is manageable.
- You want a simple similarity-based method.
- You can choose and evaluate an appropriate distance metric.
It can be especially useful as a baseline model for certain classification and regression tasks.
26. When Might KNN Not Be a Good Choice?
KNN may be less suitable when:
- The dataset is extremely large.
- There are many irrelevant features.
- The feature scales are very different and have not been handled.
- The data has very high dimensionality.
- Prediction speed is critical.
- Similarity between examples is not meaningful.
The right choice always depends on the dataset.
27. A Complete KNN Workflow
Here is the complete process:
Collect Data ↓ Prepare Data ↓ Scale Features When Appropriate ↓ Choose Distance Metric ↓ Choose K ↓ Find Nearest Neighbors ↓ Use Their Labels or Values ↓ Make Prediction ↓ Evaluate Model ↓ Tune K and Other Settings
28. A Simple Example From Start to Finish
Suppose an online store wants to predict whether a customer will buy a product.
The data includes:
- Age
- Website visits
- Time spent on the website
- Previous purchases
A new customer arrives.
KNN compares this customer with previous customers.
It finds the most similar customers.
Suppose:
Customer 1 → Bought Customer 2 → Bought Customer 3 → Bought Customer 4 → Did Not Buy Customer 5 → Bought
With:
K = 5
the prediction becomes:
Bought
The algorithm did not create a complicated set of rules.
It simply looked for similar examples.
29. The Most Important Idea to Remember
If you remember only one thing about KNN, remember this:
"Similar Data Points Can Help Predict Each Other."
That is the heart of the algorithm.
New Example ↓ Find Similar Examples ↓ Look at Their Outcomes ↓ Predict New Outcome
30. Why KNN Matters in Your Machine Learning Journey
You have now seen several different ways that machine learning algorithms can learn.
Regression
Predict numerical values.
Logistic Regression
Predict categories using a mathematical model.
Decision Tree
Make decisions through learned splits.
Random Forest
Combine multiple Decision Trees.
KNN
Use nearby examples to make predictions.
This shows an important lesson:
There is no single way for a machine learning model to learn from data.
Different algorithms use different ideas.
Frequently Asked Questions
What is KNN in Machine Learning?
KNN, or K-Nearest Neighbors, is a supervised machine learning algorithm that makes predictions using the closest training examples.
What does K mean in KNN?
K represents the number of nearest neighbors considered when making a prediction.
How does KNN classify data?
It finds the K nearest training examples and commonly uses majority voting to determine the predicted class.
Can KNN be used for regression?
Yes. KNN regression uses the target values of nearby examples to estimate a numerical output.
What distance does KNN use?
KNN can use different distance metrics. Euclidean distance is a common choice, and Scikit-learn's default Minkowski configuration with p=2 corresponds to Euclidean distance.
Why is feature scaling important for KNN?
Because KNN relies on distance. A feature with a much larger numerical scale can dominate the distance calculation if the data is not appropriately scaled.
Is a higher K always better?
No. A larger K can reduce the effect of noise but can also make the model less sensitive to local patterns. The appropriate value is data-dependent.
Is KNN easy for beginners?
Yes. Its core idea is relatively simple: find nearby examples and use their information to make a prediction.
Conclusion
K-Nearest Neighbors is one of the easiest machine learning algorithms to understand because its central idea is based on something humans already do:
Compare something new with things that are similar.
KNN finds nearby training examples, examines their outcomes, and uses that information to make a prediction.
For classification, it can use voting.
For regression, it can combine nearby numerical values.
The most important factors to understand are:
K → Neighbors → Distance → Similarity → Prediction
But KNN also teaches an important lesson.
Sometimes the best prediction does not come from creating complicated rules.
Sometimes, the answer is already hidden in the examples around you.