Introduction
Imagine opening a spreadsheet containing information about thousands of people.
At first, it looks manageable.
Then you discover something:
The spreadsheet has 500 columns.
Each column represents a different feature.
Some columns may be useful.
Some may contain similar information.
Some may contribute very little.
And suddenly, understanding the dataset becomes much harder.
This is where an important machine learning technique can help:
Dimensionality Reduction
Dimensionality reduction attempts to represent complex data using fewer dimensions while preserving important information according to the chosen method.
Instead of working with hundreds of features, we may be able to represent the data using a much smaller number of dimensions.
For example:
500 Features ↓ Dimensionality Reduction ↓ 20 Features
Or for visualization:
100 Features ↓ 2 Dimensions ↓ 2D Graph
But how does this work?
What exactly is a "dimension"?
What is PCA?
And does reducing features mean losing important information?
Let's understand it step by step.
Quick Answer: What Is Dimensionality Reduction?
Dimensionality reduction is the process of representing data using fewer dimensions while attempting to preserve important information or structure.
A simplified process looks like:
High-Dimensional Data ↓ Analyze Important Structure ↓ Transform / Select Features ↓ Lower-Dimensional Representation ↓ Easier Analysis
Dimensionality reduction is commonly used for:
- Data visualization
- Noise reduction
- Feature compression
- Exploratory data analysis
- Machine learning preprocessing
1. What Does "Dimension" Mean?
In machine learning, a dimension can be thought of as a feature or variable used to represent data.
Suppose we have customer data containing:
- Age
- Income
- Number of purchases
- Website visits
We have:
4 features
So we can describe the dataset as having four dimensions.
For example:
Customer ↓ Age Income Purchases Website Visits
Each feature adds another dimension to the representation.
2. A Simple Example
Suppose you have a dataset with:
10 Features
You might represent it as:
X₁ X₂ X₃ X₄ X₅ X₆ X₇ X₈ X₉ X₁₀
Now imagine that many of these features contain overlapping information.
A dimensionality reduction method may create a smaller representation:
10 Dimensions ↓ 3 Dimensions
The three new dimensions may capture important patterns from the original data.
3. Why Can Too Many Features Be a Problem?
Having more features is not automatically bad.
But high-dimensional datasets can create several challenges.
For example:
- More computation
- More storage
- Harder visualization
- Potentially more noise
- More difficult analysis
- Potentially increased risk of overfitting in some models
The problem becomes particularly noticeable when datasets contain hundreds or thousands of features.
4. The Curse of Dimensionality
You may remember this term from the KNN article.
It is called:
The Curse of Dimensionality
As the number of dimensions increases, data can become increasingly sparse, and concepts such as distance and neighborhood can become less informative.
This can make some machine learning tasks more difficult.
Dimensionality reduction can sometimes help by creating a lower-dimensional representation of the data.
However, it is not a universal solution to every high-dimensional problem.
5. Does Dimensionality Reduction Simply Delete Features?
Not necessarily.
There are two broad ideas worth distinguishing.
Feature Selection
Choose a subset of the original features.
For example:
100 Features ↓ Select 20 ↓ 20 Original Features
Feature Extraction
Transform the original features into a new, smaller set of features.
For example:
100 Original Features ↓ Transformation ↓ 10 New Components
PCA is an example of a feature-extraction approach.
6. Feature Selection vs Feature Extraction
Let's make this difference very clear.
Feature Selection
Keeps some original features.
Age Income Purchases Website Visits
might become:
Age Income
Feature Extraction
Creates new representations from combinations of the original features.
Original Features ↓ Mathematical Transformation ↓ New Components
The new components may not have simple real-world names.
7. Why Reduce Dimensions?
There are several reasons.
Easier Visualization
Humans cannot directly visualize hundreds of dimensions.
Reducing data to two or three dimensions can make visualization possible.
Faster Computation
Fewer features can sometimes reduce computational requirements.
Remove Redundant Information
Some methods can capture shared structure using fewer components.
Reduce Noise
In some situations, removing less useful variation can improve downstream analysis.
Simplify Data
A smaller representation can make complex datasets easier to explore.
8. Visualizing High-Dimensional Data
Imagine your dataset has:
200 features.
You cannot easily draw a 200-dimensional graph.
But you could transform the data into:
200 Dimensions ↓ 2 Dimensions ↓ Scatter Plot
Now each observation can be represented as a point.
Conceptually:
● ● ● ● ● ○ ○ ○
This can help you visually explore whether groups or patterns exist.
9. What Is PCA?
One of the most famous dimensionality reduction methods is:
Principal Component Analysis
Usually shortened to:
PCA
PCA transforms a dataset into a new set of variables called principal components.
The first principal component captures the greatest amount of variance possible under the PCA formulation.
The next component captures the greatest remaining variance subject to being orthogonal to the previous component, and so on.
This allows a dataset to be represented using fewer components.
10. What Is a Principal Component?
A principal component is a new direction in the transformed feature space.
Imagine your data points form an elongated cloud:
● ● ● ● ●
Instead of describing the data using the original coordinate axes, PCA finds directions that capture important variation.
Conceptually:
Original Data ↓ Find Important Directions ↓ Principal Components
11. PCA Example
Imagine a dataset has:
5 Features
PCA might transform it into:
PC1 PC2 PC3
So:
5 Original Features ↓ PCA ↓ 3 Principal Components
The new components are combinations of the original features.
12. What Does PCA Try to Preserve?
PCA is commonly described as finding directions that maximize variance.
In simple terms:
It tries to preserve important variation in the data using fewer components.
Suppose:
100 Features
are reduced to:
10 Components
Those 10 components attempt to capture as much of the original variance as possible given the chosen number of components.
13. What Is Variance?
Variance describes how much values differ from their average.
Imagine:
10 11 10 11 10
The values are close together.
Now:
2 20 5 40 10
has much greater variation.
PCA uses variance as a central concept when determining its principal components.
14. Why Is Scaling Important Before PCA?
This is an important practical point.
Suppose your dataset contains:
Age → 20–70 Income → 20,000–200,000
Income has a much larger numerical scale.
Because PCA is sensitive to the scale of variables, a feature with a larger variance due simply to its measurement scale can dominate the components.
Therefore, standardizing features is often appropriate before PCA when the variables are measured on substantially different scales.
15. PCA Workflow
A simplified PCA workflow is:
Original Data ↓ Prepare Data ↓ Scale Features When Appropriate ↓ Calculate Principal Components ↓ Rank Components ↓ Choose Number of Components ↓ Transform Data ↓ Analyze Reduced Data
This is the basic idea.
16. How Many Components Should We Keep?
This is an important question.
Suppose PCA produces:
PC1 PC2 PC3 PC4 PC5 ... PC20
Do we need all 20?
Maybe not.
We can examine the amount of variance explained by each component.
For example:
PC1 → 40% PC2 → 25% PC3 → 15% PC4 → 8% ...
The first three components together explain:
80%
of the variance in this hypothetical example.
A practitioner may decide that retaining those components provides a useful trade-off between dimensionality and retained variance.
The appropriate choice depends on the task.
17. Explained Variance
One useful concept in PCA is:
Explained Variance Ratio
It indicates the proportion of the dataset's variance associated with each principal component.
For example:
PC1 → 50% PC2 → 20% PC3 → 10%
Together:
80%
So the first three components explain 80% of the variance in this example.
This can help guide how many components to retain.
18. PCA for Visualization
Suppose we have:
100 Features
We can use PCA to transform the data into:
2 Components
Then plot:
PC1 vs PC2
This can reveal possible structures.
For example:
● ● ● ● ● ● ○ ○ ○ ○ ○
You might visually notice two groups.
This does not prove that the groups are meaningful, but it can help with exploratory analysis.
19. PCA and Clustering
PCA and clustering can sometimes be used together.
For example:
High-Dimensional Data ↓ PCA ↓ Lower-Dimensional Data ↓ Clustering
Why?
Reducing dimensions may make the dataset easier to visualize or can sometimes improve the behavior of a downstream algorithm.
However, PCA should not automatically be applied before every clustering task.
The effect should be evaluated for the specific dataset.
20. PCA in Python
Scikit-learn provides a PCA implementation.
A simple example is:
from sklearn.decomposition import PCA pca = PCA(n_components=2) X_reduced = pca.fit_transform(X)
This asks PCA to transform the data into:
2 components
The result is stored in:
X_reduced
21. Checking Explained Variance
You can inspect:
pca.explained_variance_ratio_
This provides the proportion of variance explained by each selected principal component.
For example, the result could conceptually look like:
[0.52, 0.21]
This would mean:
PC1 → 52% PC2 → 21%
for that particular dataset.
22. PCA Is Not the Only Dimensionality Reduction Method
PCA is important, but it is not the only method.
Other approaches include:
Truncated SVD
Useful in certain high-dimensional sparse data settings.
t-SNE
Often used for nonlinear visualization of high-dimensional data.
UMAP
Another nonlinear dimensionality-reduction technique commonly used for visualization and exploratory analysis.
Autoencoders
Neural-network-based models that can learn lower-dimensional representations.
Each method has different goals and assumptions.
23. Linear vs Nonlinear Dimensionality Reduction
PCA is fundamentally a linear dimensionality reduction method.
This means it looks for linear combinations of the original features.
Other methods can model nonlinear relationships.
For example:
PCA ↓ Linear Transformation
while some nonlinear methods attempt to preserve more complex structures.
Choosing between them depends on the data and purpose.
24. Why t-SNE Is Different
t-SNE is often used to create two-dimensional or three-dimensional visualizations of high-dimensional data.
Its goal is different from simply maximizing global variance.
It focuses heavily on preserving local neighborhood relationships.
This can make it useful for visual exploration.
However, t-SNE visualizations require careful interpretation and are not always appropriate for general-purpose feature compression.
25. What About UMAP?
UMAP is another nonlinear dimensionality reduction method.
It is commonly used to visualize high-dimensional datasets.
Like t-SNE, it can help reveal local structures that may be difficult to see in the original feature space.
However, visualization results can depend on parameters and should not automatically be interpreted as proof of meaningful clusters.
26. Dimensionality Reduction and Machine Learning
Dimensionality reduction can sometimes be used before a machine learning model.
For example:
Original Features ↓ Dimensionality Reduction ↓ Reduced Features ↓ Machine Learning Model ↓ Prediction
This can sometimes reduce computation or noise.
But reducing dimensions can also remove information.
Therefore, it should be evaluated rather than assumed to improve every model.
27. Does Dimensionality Reduction Always Improve Accuracy?
No.
This is an important point.
Reducing dimensions can:
- Help
- Have little effect
- Or hurt performance
Why?
Because the removed information may contain useful predictive signals.
For example:
100 Features ↓ 10 Features
may make the model simpler.
But some useful information may have been discarded.
Therefore, dimensionality reduction should be tested using proper validation.
28. Advantages of Dimensionality Reduction
Easier Visualization
High-dimensional data can be represented in two or three dimensions.
Reduced Complexity
Fewer dimensions can make analysis easier.
Potentially Lower Computation
Some algorithms can become faster with fewer features.
Helps Explore Data
Reduced representations can reveal patterns that are difficult to inspect directly.
Can Reduce Redundancy
Some methods can represent correlated information more compactly.
29. Limitations of Dimensionality Reduction
Information Can Be Lost
A smaller representation cannot always preserve everything.
New Features May Be Hard to Interpret
PCA components are combinations of original features.
Results Depend on the Method
Different techniques preserve different properties of the data.
Scaling Matters
Methods such as PCA can be strongly influenced by feature scale.
Visualization Can Be Misleading
A two-dimensional representation is only an approximation of a high-dimensional dataset.
30. A Simple Real-World Example
Imagine a company has customer data with:
100 features
These include:
- Purchase history
- Website behavior
- Product interactions
- Spending patterns
- Engagement measurements
The company wants to visualize customer behavior.
It could use PCA:
100 Features ↓ PCA ↓ 2 Components ↓ Scatter Plot
The resulting visualization may reveal broad patterns or possible groups.
The company can then investigate those patterns further.
31. Dimensionality Reduction vs Feature Selection
Let's make the difference clear again.
Feature Selection
Choose existing features.
100 Features ↓ Choose 20 ↓ 20 Original Features
Dimensionality Reduction Through Feature Extraction
Create new features.
100 Features ↓ Transformation ↓ 20 New Components
This distinction is important.
32. Dimensionality Reduction vs Clustering
These two topics are related but different.
Clustering
Find groups.
Data ↓ Clusters
Dimensionality Reduction
Create a simpler representation.
Data ↓ Fewer Dimensions
They can also be used together:
Data ↓ Dimensionality Reduction ↓ Clustering
But they solve different problems.
33. Dimensionality Reduction vs Classification
Classification predicts known categories.
Features ↓ Classifier ↓ Spam / Not Spam
Dimensionality reduction transforms the representation of the data.
Features ↓ Dimensionality Reduction ↓ Reduced Representation
The reduced representation can potentially be used as input to a classifier.
34. Common Beginner Mistakes
Mistake 1: Thinking More Dimensions Are Always Better
More features can provide useful information, but they can also add redundancy or noise.
Mistake 2: Assuming PCA Simply Deletes Features
PCA creates new components from the original features.
Mistake 3: Ignoring Scaling
Feature scale can strongly influence PCA.
Mistake 4: Keeping Too Few Components
Too much reduction can remove useful information.
Mistake 5: Treating a Visualization as Proof
A two-dimensional plot is an approximation of the original high-dimensional structure.
35. The Most Important Terms
Remember these concepts:
Dimension
A feature or coordinate used to represent data.
Dimensionality Reduction
Representing data with fewer dimensions.
Feature Selection
Selecting a subset of original features.
Feature Extraction
Creating new features from existing ones.
PCA
Principal Component Analysis.
Principal Component
A new direction created by PCA.
Explained Variance
The proportion of variance represented by components.
36. A Complete Dimensionality Reduction Workflow
Collect Data ↓ Understand Features ↓ Clean Data ↓ Scale Features When Appropriate ↓ Choose Reduction Method ↓ Select Number of Dimensions ↓ Transform Data ↓ Evaluate Information Retained ↓ Visualize or Use Reduced Data
This is the basic workflow.
37. The Bigger Picture
You have now covered several important unsupervised learning ideas:
Unsupervised Learning ↓ Clustering ↓ K-Means ↓ Hierarchical Clustering ↓ Association Rules ↓ Dimensionality Reduction
Notice how each method answers a different question.
Clustering
Which observations belong together?
Association Rules
Which items occur together?
Dimensionality Reduction
How can complex data be represented more simply?
These are different tools for discovering and understanding data.
38. Connecting Everything You Have Learned
Your machine learning journey has now covered:
Regression
Predict numbers.
Classification
Predict categories.
Clustering
Discover groups.
Association Rules
Discover relationships.
Dimensionality Reduction
Simplify complex representations.
This gives you a much broader understanding of what machine learning can do.
Frequently Asked Questions
What is dimensionality reduction in machine learning?
It is the process of representing data using fewer dimensions while attempting to preserve important information or structure.
Why is dimensionality reduction used?
It can help with visualization, computational efficiency, exploratory analysis, redundancy, and sometimes noise reduction.
What is PCA?
PCA, or Principal Component Analysis, is a linear dimensionality reduction method that transforms data into principal components ordered according to the variance they capture.
Does PCA remove features?
PCA does not simply select and delete original features. It creates new components that are combinations of the original features.
Is PCA supervised or unsupervised?
PCA is generally considered an unsupervised dimensionality reduction technique because it does not require target labels.
How many PCA components should I keep?
There is no universal number. You can consider explained variance, downstream model performance, visualization needs, and the purpose of the analysis.
Why should data sometimes be standardized before PCA?
Because PCA is sensitive to feature scale. Variables measured on larger scales can otherwise have disproportionate influence.
Is dimensionality reduction always beneficial?
No. Reducing dimensions can remove useful information and may sometimes hurt downstream performance.
What is the difference between feature selection and dimensionality reduction?
Feature selection keeps a subset of original features, while feature extraction methods such as PCA create new representations from the original features.
Can dimensionality reduction be used with clustering?
Yes. It can sometimes be used before clustering for visualization or as preprocessing, although its effect should be evaluated for the particular dataset.
Conclusion
Dimensionality reduction helps us deal with one of the major challenges of modern machine learning:
Too much information.
When a dataset contains hundreds or thousands of features, it can become difficult to visualize, analyze, and process.
Dimensionality reduction provides ways to create a smaller representation.
One of the most important techniques is:
Principal Component Analysis — PCA
PCA transforms the original features into new components that capture important variation in the data.
The key ideas to remember are:
High Dimensions → Transformation → Fewer Dimensions → Easier Analysis
But remember:
Less data does not automatically mean better data.
Reducing dimensions can remove useful information, so the result should always be evaluated for the actual task.
With this article, we have completed the planned Unsupervised Learning foundation.
Now the learning path takes a major turn.
Until now, we have discussed models that learn from labeled data and methods that discover structure in existing data.
But what if an AI system learns by taking actions, receiving rewards or penalties, and improving its behavior through experience?
That brings us to one of the most fascinating areas of artificial intelligence.