Machine learning achieves high accuracy with 99% variance retention in hyperspectral imaging

Hyperspectral imaging provides detailed spectral information for analyzing environments and materials, but the sheer amount of data and strong correlations between features often hinders machine learning applications, especially in the presence of limited ground truth data. Parisa Parand of Allameh Tabatabai University and Mahmoud Samadpour of KN Toosi University of Technology, along with colleagues, tackled this challenge by investigating how principal component analysis (PCA) can improve machine learning performance using hyperspectral data. Their work shows that reducing the first 150 spectral bands to only two principal components preserves more than 99% of the data variance, simplifies data complexity, and crucially improves prediction accuracy. By training a machine learning model on this reduced dataset, the team achieved an extremely high coefficient of determination of 94.7%, establishing PCA as a powerful technique for efficient and accurate analysis in hyperspectral imaging workflows.

Hyperspectral imaging and soil moisture estimation with dimensionality reduction.

High dimensionality and strong feature correlations pose significant challenges to machine learning models, especially when ground truth datasets are limited. This study investigates a hyperspectral dataset consisting of 150 spectral bands with soil moisture as the target variable. As supported by analysis of the covariance matrix and eigenvalue distribution, the optimal number of principal components was determined to be 2, retaining more than 99% of the total variance.

Projecting the data onto these components improved visualization and interpretability compared to the original high-dimensional space, separated target values ​​more clearly, and reduced data complexity. A random forest regression model trained on PCA-transformed data achieved a coefficient of determination (R2) of 94.7%. This shows that PCA-based feature reduction can improve computational efficiency while maintaining strong predictive power in hyperspectral machine learning workflows. Optical imaging systems increasingly capture large amounts of high-dimensional data, enabling detailed characterization of materials and environmental conditions. Hyperspectral optical imaging provides dense spectral information by recording reflectance or radiance values ​​over tens to hundreds of continuous wavelength bands.

Although these multidimensional datasets provide rich feature representations, their size, redundancy, and strong correlation between bands pose computational and analytical challenges. Machine learning (ML) techniques have become essential tools for extracting meaningful information from high-dimensional optical datasets and supporting tasks such as materials identification, biological tissue analysis, food quality monitoring, environmental assessment, and precision agriculture. Although research has focused on hyperspectral classification, few studies have considered hyperspectral regression, where the goal is to estimate continuous physical or chemical parameters. Hyperspectral regression problems often suffer from the curse of dimensionality and require large datasets for reliable generalization.

Hyperspectral bands exhibit strong correlations and redundant information, so their inherent dimensionality is often lower than their spectral resolution. Dimensionality reduction is therefore essential for efficient and robust hyperspectral machine learning pipelines. Among the available techniques, PCA remains widely used due to its simplicity, interpretability, and computational efficiency. PCA orthogonally transforms the data into a new coordinate system in which the principal components capture the maximum variance. This study shows that PCA-based dimensionality reduction significantly improves machine learning performance in hyperspectral optical imaging.

By compressing the spectral bands into two principal components while retaining more than 99% of the variance, PCA reduced redundancy, improved computational efficiency, and enabled a clearer separation of soil moisture-related spectral patterns. Our results confirm that PCA is an effective strategy to simplify high-dimensional hyperspectral datasets without compromising prediction accuracy, supporting the development of more interpretable and scalable machine learning models for optical imaging applications. Future work should evaluate alternative feature extraction methods and test the framework across diverse imaging scenarios and sensor platforms.

Hyperspectral soil moisture prediction using machine learning

This work pioneers a methodology for using hyperspectral imaging data to improve machine learning performance, specifically addressing the challenges posed by high dimensionality and feature correlation. The scientists used a dataset consisting of 679 samples featuring continuous soil moisture values ​​and 125 spectral bands ranging from 450 nm to 950 nm, acquired using a Cubert UHD 285 hyperspectral snapshot camera during field activities in Germany. The system acquired images of 50 × 50 pixels with a spectral resolution of approximately 4 nm, and reference soil moisture values ​​were obtained using a TRIME-PICO time-domain reflectance measurement sensor. To ensure reproducibility, the research team implemented a controlled experimentation framework that uses a Python-based workflow to initialize a fixed random state.

Data preparation involves importing a dataset using the pandas library and decomposing it into a feature matrix and target vector. Prior to PCA, all spectral features were standardized to ensure that each band contributed equally to the analysis. The researchers then trained a random forest regression model on the PCA-transformed data and evaluated the impact of dimensionality reduction on predictive performance. This approach demonstrated a coefficient of determination (R2) of 94.7%. This shows that PCA-based feature reduction effectively maintains strong predictive ability while improving the efficiency of hyperspectral machine learning workflows. The researchers addressed the challenges posed by high dimensionality, specifically 125 spectral bands, by reducing the dataset to only two principal components while preserving more than 99% of the original variance. Analysis of the covariance matrix, eigenvalue distribution, and scree plot validated this dimensionality reduction and confirmed that the additional components contributed slightly to the overall data representation. The team measured soil moisture ranging from about 25% to 43%, but found that the majority of the samples were concentrated around 32%, with a reasonably narrow distribution over the measurement period.

Correlation heatmaps revealed strong interband correlations, especially at 642 nm and 742 nm, indicating spectral redundancy within the measured wavelengths. This redundancy supports the use of dimensionality reduction techniques to reduce computational complexity while preserving important information. Applying principal component analysis (PCA), the random forest regression model trained on the transformed data achieved a coefficient of determination (R²) of 0.947, demonstrating that approximately 94.7% of soil moisture variation can be explained by hyperspectral features.

Projecting the data onto the first two principal components showed clear clustering and effectively separated samples based on soil moisture level. These findings confirm that PCA is an effective approach to simplify high-dimensional hyperspectral datasets without compromising prediction performance. This supports the development of more interpretable and scalable machine learning models for optical imaging applications. In future work, we plan to consider alternative feature extraction methods and test our framework across different imaging scenarios.