table of contents
1. Dimensionality reduction
2. How does principal component analysis work?
3. Implementation in Python
4. Evaluation and interpretation
5. Conclusions and future steps
Many real-world problems in machine learning involve datasets with thousands or even millions of features. Training on such datasets can be computationally intensive, and the resulting solutions can be even more difficult to interpret.
As the number of features increases, the data points become more sparse, the distance between points becomes less clear, and it becomes harder to distinguish between close and far points, so the distance metric becomes less informative. This is because Curse of Dimensionality.
As data becomes sparse, models become harder to train and tend to overfit by capturing noise rather than underlying patterns, resulting in poor generalization to new, unseen data.
Dimensionality reduction is used in data science and machine learning to reduce the number of variables or features in a dataset while preserving as much of the original information as possible. This technique can simplify complex datasets, improve computational efficiency, and aid in data visualization.
One of the most commonly used techniques to mitigate the curse of dimensionality is Principal Component Analysis (PCA). PCA reduces the number of features in a dataset while preserving most of the useful information by finding the axes that account for the most variance in the dataset. These axes are Main components.
PCA aims not to make predictions but to find a low-dimensional representation of a dataset while preserving most of the variance, Unsupervised learning algorithm.
But why does preserving differences mean preserving important information?
Imagine you are analyzing a dataset about crime in a city. The data has a number of features, such as “crimes against people – with injury” and “crimes against people – without injury”. Surely, where there is a high proportion of the first example, there will also be a high proportion of the second example.
In other words, the two features in this example are highly correlated, so by reducing the redundancy in the data (whether the victim was injured or not), it is possible to reduce the dimensionality of the dataset.
The PCA algorithm is just a sophisticated way of doing it.
Now, let us explain in detail how the PCA algorithm works internally in the following steps:
Step 1: Center the data
Because PCA is sensitive to the scale of the data, we first subtract the mean of each feature in the dataset so that all features have a mean of zero.
Step 2: Calculate the covariance matrix
Now we need to calculate a covariance matrix to capture how each pair of features in the data varies. yeah Depending on the features, the resulting covariance matrix is yeah X yeah shape.
In the image below, more correlated features are closer to red, and of course each feature is highly correlated with each other.
Step 3: Eigenvalue decomposition
Next, we need to perform an eigenvalue decomposition of the covariance matrix. If you don't remember, given a covariance matrix Σ (a square matrix), eigenvalue decomposition is the process of finding a set of scalars (eigenvalues) and vectors (eigenvectors) such that:
where:
- Σ is the n×n covariance matrix.
- V These nonzero vectors are called eigenvectors.
- λ is a scalar, called an eigenvalue, associated with the eigenvector. V.
Eigenvectors It indicates the direction of maximum variance of the data (principal component), eigenvalue It quantifies the variance captured by each principal component.
In the case of a matrix a When decomposed into eigenvalues and eigenvectors, it can be expressed as follows:
where:
- question is a matrix whose columns are the eigenvectors of a.
- Λ is a diagonal matrix, whose diagonal elements are a.
Then you can use the same procedure to find the eigenvalues and eigenvectors of the covariance matrix.
In the image above, you can see that the first eigenvector points in the direction of the greatest variance in the data, and the second eigenvector points in the direction of the second greatest variance.
Step 4: Select principal components
As mentioned before, the eigenvalues quantify the variance of the data in the direction of the corresponding eigenvector, so we sort the eigenvalues in descending order and keep only the top n ones we want. Main components.
The image below shows the proportion of variance captured by each principal component in a two-dimensional PCA.
Step 5: Project the data
Finally, we need to project the original data onto the dimensions represented by the selected principal components. To do this, we need to multiply the centered data set by the matrix of eigenvectors found in the decomposition of the covariance matrix.
Now that you have a solid understanding of the key concepts of principal component analysis, it's time to write some code.
First, we need to set up the environment by importing the numpy package for mathematical computations and matplotlib for visualization.
import numpy as np
import matplotlib.pyplot as plt
Next, we'll encapsulate all the concepts discussed in the previous sections into a Python class with the following method:
A constructor method to initialize the algorithm's parameters: the number of required components, a matrix to store the component vector, and an array to store the explained variance for each selected dimension.
In the fitting method, the code implements the first four steps presented in the previous section and also calculates the explained variance for each component.
The transform method performs the last step described in the previous section: projecting the data onto the selected dimensions.
The final method is a helper function that plots the explained variance for each selected principal component as a bar chart.
Here is the complete code:
class PCA:
def __init__(self, n_components):
self.n_components = n_components
self.components = None
self.mean = None
self.explained_variance = Nonedef fit(self, X):
# Step 1: Standardize the data (subtract the mean)
self.mean = np.mean(X, axis=0)
X_centered = X - self.mean
# Step 2: Compute the covariance matrix
cov_matrix = np.cov(X_centered, rowvar=False)
# Step 3: Compute the eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
# Step 4: Sort the eigenvalues and corresponding eigenvectors
sorted_indices = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[sorted_indices]
eigenvectors = eigenvectors[:, sorted_indices]
# Step 5: Select the top n_components
self.components = eigenvectors[:, :self.n_components]
# Calculate explained variance
total_variance = np.sum(eigenvalues)
self.explained_variance = eigenvalues[:self.n_components] / total_variance
def transform(self, X):
# Step 6: Project the data onto the selected components
X_centered = X - self.mean
return np.dot(X_centered, self.components)
def plot_explained_variance(self):
# Create labels for each principal component
labels = [f'PCA{i+1}' for i in range(self.n_components)]
# Create a bar plot for explained variance
plt.figure(figsize=(8, 6))
plt.bar(range(1, self.n_components + 1), self.explained_variance, alpha=0.7, align='center', color='blue', tick_label=labels)
plt.xlabel('Principal Component')
plt.ylabel('Explained Variance Ratio')
plt.title('Explained Variance by Principal Components')
plt.show()
Now we use the implemented class on a simulated dataset created using the numpy package. The dataset has 10 features and 100 samples.
# create simulated data for analysis
np.random.seed(42)
# Generate a low-dimensional signal
low_dim_data = np.random.randn(100, 4)# Create a random projection matrix to project into higher dimensions
projection_matrix = np.random.randn(4, 10)
# Project the low-dimensional data to higher dimensions
high_dim_data = np.dot(low_dim_data, projection_matrix)
# Add some noise to the high-dimensional data
noise = np.random.normal(loc=0, scale=0.5, size=(100, 10))
data_with_noise = high_dim_data + noise
X = data_with_noise
Before performing PCA, one question remains. How to choose the correct or optimal number of dimensions• In general, you should look for a number of components that accounts for at least 95% of the explained variance of the dataset.
To do this, let's look at how each principal component contributes to the total variance of the dataset.
# Apply PCA
pca = PCA(n_components=10)
pca.fit(X)
X_transformed = pca.transform(X)print("Explained Variance:\n", pca.explained_variance)
>> Explained Variance (%):
[55.406, 25.223, 11.137, 5.298, 0.641, 0.626, 0.511, 0.441, 0.401, 0.317]
Then, plot the cumulative sum of variance and see which number of dimensions achieves the optimal value of 95% of the total variance.
As shown in the graph above, the optimal number of dimensions for our dataset is 4, with a total explained variance of 97.064%. In other words, we have transformed our dataset with 10 features into a dataset with only 3 dimensions, while preserving more than 97% of the original information.
This means that most of the original 10 features are highly correlated, and the algorithm transformed that high-dimensional data into uncorrelated principal components.
We created PCA classes using only the numpy package and were successful in reducing the dimensionality of the dataset from 10 features to 4 while preserving approximately 97% of the variance in the data.
We also looked at how to obtain the optimal number of principal components for a PCA analysis, which can be customized depending on the problem you are facing (for example, you may be interested in retaining only 90% of the variance).
This shows the potential of PCA analysis to address the curse of dimensionality discussed earlier, and also leaves us with some points for further investigation.
- Perform classification or regression tasks using other machine learning algorithms on the dataset reduced using the PCA algorithm and compare the performance of models trained on the original and PCA transformed datasets to evaluate the impact of dimensionality reduction.
- Visualizing data using PCA can make high-dimensional data more interpretable and help discover patterns that were not evident in the original feature space.
- Consider looking into other dimensionality reduction techniques such as: t-distributed stochastic neighborhood embedding (t-SNE) and Linear discriminant analysis (LDA).
The complete code is available here.
