Enhanced EEG signal classification in brain computer interfaces using hybrid deep learning models

This section delves into the methodological underpinnings of the study, encompassing a detailed exploration of the dataset, Pre-processing, the implementation of feature extraction techniques, and the application of classification algorithms. A comprehensive overview of the experimental design and procedures is provided, facilitating a thorough understanding of the research methodology and its execution.

Proposed algorithm

In this paper, the authors have developed an advanced architecture within a BCI system designed to improve the classification of motor imagery tasks by effectively interpreting the brain signals associated with imagined movements. As illustrated in Fig. 3, the proposed system processes EEG signals by first feeding them into a sophisticated hybrid deep learning model that combines CNN with LSTM networks. This combination is specifically chosen to capitalize on CNNs’ exceptional ability to extract spatial features from multi-channel EEG data and LSTMs’ strength in capturing temporal dependencies over time.

To execute the experiment, the present research initiated the development of this hybrid CNN-LSTM model due to the unique advantages it offers in analyzing complex, non-linear EEG signals. Traditional machine learning algorithms like RF and SVC have been extensively used in previous BCI research and provide a solid benchmark; however, they are often limited by their inability to fully exploit the intricate temporal and spatial patterns inherent in EEG data. In contrast, our CNN-LSTM model was developed to overcome these limitations, leveraging the CNN’s capability to detect spatial patterns across the EEG electrode array and the LSTM’s ability to model the temporal sequence of brain activity during motor imagery.

For model selection, this study thoroughly evaluated various deep learning architectures, our evaluation revealed that while standalone CNNs and LSTMs provide good accuracy, their performance is significantly enhanced when combined into a hybrid model. The hybrid CNN-LSTM architecture outperformed these individual models by effectively capturing the full scope of EEG signal characteristics – from spatial feature maps generated by the CNN to temporal sequences processed by the LSTM. This comprehensive feature extraction and sequence modelling have led to a notable improvement in classification accuracy, making the hybrid model particularly well-suited for BCI applications.

Furthermore, recognizing the importance of data diversity in model training, the research incorporated GANs to augment our dataset with synthetic EEG signals. This augmentation strategy addresses the challenge of limited training data, which is common in EEG studies, by providing additional examples that mimic the real EEG signal distribution. The inclusion of GANs ensures that our CNN-LSTM model is trained on a more robust dataset, leading to better generalization and improved performance in real-world scenarios.

The newly developed hybrid CNN-LSTM network architecture consists of two main components: the first is the CNN module, responsible for multi-domain feature extraction, and the second is the LSTM module, which captures essential temporal dynamics for accurate classification. The use of GANs for data augmentation further enhances the robustness and reliability of the proposed model, resulting in superior classification performance compared to both traditional machine learning models and individual deep learning architectures.

Dataset description

In this study, the present work utilized the EEG Motor Movement/Imagery Dataset (version 1.0.0), a publicly accessible dataset hosted on PhysioNet⁴⁴. The data were recorded using the BCI2000 system, which captured 64-channel EEG signals following the international 10–10 system, at a sampling rate of 160 Hz. The dataset initially included 109 participants; however, due to annotation errors, data from six participants (S038, S088, S089, S092, S100, and S104) were excluded, leaving a final sample of 103 participants.

The experimental protocol consisted of 14 runs per participant, with the first two runs serving as baseline measurements—one with eyes open and the other with eyes closed—each lasting one minute. The remaining 12 runs involved four different tasks, each repeated three times. These tasks were as follows:

• Task 1: The participant opened and closed the corresponding fist when a target appeared on the left or right side of the screen, then relaxed.

• Task 2: The participant imagined performing the actions described in Task 1.

• Task 3: The participant opened and closed both fists if the target appeared at the top of the screen, or moved both feet if the target appeared at the bottom, then relaxed.

• Task 4: The participant imagined performing the actions described in Task 3.

Each task was repeated across three runs, resulting in a comprehensive dataset that captured both real and imagined motor movements. Each of these runs was annotated with markers indicating the onset of movement (T1 for left or both hands, T2 for right hand or both feet) and rest periods (T0). The dataset provided a robust foundation for analyzing motor imagery and movement in EEG signals across a large and diverse participant pool.

The research focused on the imagined movement tasks within the EEG Motor Movement/Imagery Dataset, specifically utilizing task combinations (A0) and (B0) across experimental runs 4, 6, 8, 10, 12, and 14. These runs involved participants imagining specific motor actions, with each run comprising three tasks, each lasting 5 s. The original dataset encodes these tasks as follows: T0 denotes the baseline period, T1 corresponds to the imagined movement of the left fist in task combination (A0) and both fists in task combination (B0), and T2 corresponds to the imagined movement of the right fist in task combination (A0) and both feet in task combination (B0). Each experimental run follows a consistent timeline: 5 s of baseline (T0), 5 s of T1, another 5 s of baseline, and 5 s of T2, resulting in a total of 15 baseline segments, 8 T1 segments, and 7 T2 segments per run.

To facilitate effective classification for our study, the authors restructured the original task annotations into five distinct classes. Specifically, the authors assigned the label ‘E’ to T1 events related to the imagined left fist movement (task combination b), ‘F’ to T1 events related to the imagined movement of both fists (task combination d), ‘G’ to T2 events corresponding to the imagined right fist movement (task combination b), and ‘H’ to T2 events corresponding to the imagined movement of both feet (task combination d). Additionally, this study introduced a fifth class, ‘I,’ representing the baseline (T0) segments.

In Fig. 4. given our study’s focus on classifying imagined motor movements using advanced ML and DL techniques, this study further concentrated on analyzing EEG data from regions over the sensorimotor cortex, informed by previous research indicating the involvement of frontoparietal and central areas during motor imagery⁴⁵. Building on this, we extracted six regions of interest (ROIs) from the dataset, creating six separate subsets of data. Each data segment (a 5-second epoch) was then labelled with one of the five classes (E, F, G, G, I), resulting in a structured input matrix of size 640 × 2, where the two columns represent paired contralateral channels.

Our main objective was to leverage these datasets in training and evaluating various ML and DL models, with a particular emphasis on our proposed hybrid model that integrates CNNs and LSTM networks. This CNN + LSTM model is designed to enhance classification performance by exploiting CNNs’ capability to extract spatial features from EEG data and LSTMs’ ability to capture temporal dependencies. This hybrid approach provides a robust architecture for accurately classifying complex EEG signals associated with imagined motor tasks, advancing the field of BCIs and neuroimaging analysis.

Pre-processing

The quality of the raw EEG data significantly influences the performance of any BCI system. EEG signals are typically noisy and contain a variety of artifacts, such as eye blinks and muscle movements, which can obscure the neural signals relevant to motor imagery tasks. Therefore, the pre-processing stage in the algorithm is essential to ensure that the subsequent stages operate on clean and reliable data. The traditional methods for EEG data pre-processing and feature extraction have predominantly utilized techniques such as autoregressive (AR) modeling, STFT, WT, and DWT. In contrast, our approach employs a comprehensive pre-processing pipeline, including normalization, bandpass filtering, spatial filtering, and artifact removal to ensure clean and standardized EEG signals. For advanced feature extraction, the research utilizes Wavelet Transform and Riemannian Geometry to capture both time-frequency characteristics and the intrinsic geometric structure of the EEG data. Additionally, dimensionality reduction techniques like PCA and t-distributed Stochastic Neighbor Embedding (t-SNE) are applied to reduce computational complexity and enhance feature discrimination.

A key innovation in our approach is the incorporation of GANs, which are used to generate synthetic data, thereby augmenting the training set and improving the robustness and accuracy of the classification model. By integrating these advanced methods into the preprocessing, feature extraction, and data augmentation stages, our approach lays a robust foundation for accurate and efficient EEG signal classification^46,47,48.

• Normalization: By normalizing the EEG data, the algorithm ensures that each feature contributes equally to the learning process. This step is crucial in preventing features with larger scales from dominating the model’s decision-making process, which could lead to biased and inaccurate classifications.

  $$Xnormalized = \frac{{X – \mu }}{\sigma }$$

  (1)

  where X is the raw data, µ is the mean, and σ is the standard deviation.

• Bandpass filtering: EEG signals contain a wide range of frequencies, but not all of these frequencies are relevant for motor imagery tasks. By applying a bandpass filter (typically within the 0.5–50 Hz range), The algorithm isolates the frequency bands that are most informative for distinguishing between different motor imagery states. This step reduces the noise and irrelevant data, improving the signal-to-noise ratio, which is critical for accurate classification.

  $$\:H\left(s\right)=\:\frac{1}{\sqrt{1+{\left(\frac{s}{\omega\:c}\right)}^{2n}}}\:\:\:$$

  (2)

  where s is the Laplace transform variable, ω_c is the cutoff frequency, and n is the filter order.

• Spatial filtering: The application of spatial filtering, such as CSP, enhances the discriminability of EEG signals by focusing on spatial patterns that differentiate between motor imagery tasks. This technique maximizes the variance between classes, making it easier for classifiers to distinguish between different motor states.

  $$W = \arg \max \frac{{{W^\tau }{X_1}X_1^\tau W}}{{{W^\tau }{X_2}X_2^\tau W}}$$

  (3)

  where X₁ and X₂ and are the covariance matrices of two different classes.

• Artifact removal: EEG data often contain non-neural artifacts, such as those caused by eye blinks or muscle contractions. Techniques like ICA are employed to separate and remove these artifacts, ensuring that the classifiers focus on the neural signals that are truly indicative of motor imagery. This step is vital for improving the robustness and accuracy of the classification process.

  where X is the observed data, A is the mixing matrix, and S is the source signal matrix (artifacts).

Feature extraction

Feature extraction is a critical step that transforms the pre-processed EEG data into a more informative representation, which captures the underlying neural activity associated with motor imagery. The features extracted during this stage provide the classifiers with the information needed to differentiate between different motor imagery states effectively.

• Wavelet transform: EEG signals are non-stationary, meaning that their statistical properties can change over time. The Wavelet Transform is particularly useful for analyzing such signals because it provides both time and frequency information, allowing the algorithm to capture transient events that are crucial for distinguishing between different motor imagery tasks. By decomposing the EEG signals into various frequency bands, the Wavelet Transform helps capture subtle temporal dynamics that may be missed by traditional Fourier analysis⁴⁶.

  $$x\left( t \right) = \sum\limits_{i = 1}^J {\sum\limits_{k = 1}^k {{c_j},{}_k{\psi _j},{k^{\left( t \right)}}} }$$

  (5)

  where c_j, k are wavelet coefficients, _kψ_j,k(t) are the wavelets, j is the scale, and k is the position.

• Riemannian geometry: The use of Riemannian geometry in feature extraction leverages the intrinsic geometric properties of EEG data. By representing covariance matrices on a Riemannian manifold, the algorithm can better capture the relationships between different EEG channels. This approach is particularly effective in capturing the spatial structure of EEG signals, which is critical for distinguishing between different motor imagery tasks.

  $$\:C=\frac{1}{T}\sum\:_{t=1}^{T}x\left(t\right){x\left(t\right)}^{T}$$

  (6)

• Principal component analysis (PCA): PCA is employed to reduce the dimensionality of the EEG data while retaining the most important features. This step simplifies the data, making it easier for classifiers to process without losing significant information. By focusing on the principal components that account for the most variance in the data, PCA helps the algorithm to emphasize the most informative aspects of the EEG signals.

  where Z is the projected data, X is the mean-cantered data, and W contains the eigenvectors of the covariance matrix of X.

• t-SNE (t-distributed stochastic neighbor Embedding): While not typically used for direct classification, t-SNE is valuable for visualizing the high-dimensional EEG data in lower dimensions. This technique can reveal underlying structures in the data, such as clusters corresponding to different motor imagery states, providing insights that can guide the development and tuning of classifiers.

  Given a set of high-dimensional points X1, X2,. . ,Xn, t-SNE constructs a probability distribution Pij that represents the pairwise similarities between points:

  $${P_{ij}} = \frac{{\exp – \frac{{{{\left\| {{x_i} – {x_j}} \right\|}^2}}}{{2\sigma _i^2}}}}{{\sum\nolimits_{k = i} {\exp – \frac{{{{\left\| {{x_j} – {x_k}} \right\|}^2}}}{{2\sigma _i^2}}} }}$$

  (8)

  t-SNE then seeks a lower-dimensional representation y1,y2,. . ,yn that minimizes the divergence between P_{i j} and a similar distribution Q_{i j} defined in the low-dimensional space:

  $$KL(P\left\| Q \right.) = \sum\limits_{i = 1} {\sum\limits_{j = 1} {{P_{ij}}\log \frac{{{P_{ij}}}}{{{Q_{ij}}}}} }$$

  (9)

  Where:

  $${Q_{ij}} = \frac{{{{\left( {1 + {{\left\| {{y_i} – {y_j}} \right\|}^2}} \right)}^{ – 1}}}}{{\sum\nolimits_{k = 1} {{{\left( {1 + {{\left\| {{y_k} – {y_j}} \right\|}^2}} \right)}^{ – 1}}} }}$$

  (10)

Time domain features are statistical measures computed directly from the amplitude values of the EEG signal within predefined time windows as shown in Fig. 5. These features offer valuable insights into the central tendency and dispersion of the neural activity recorded by EEG electrodes. Figure 5 illustrates the extracted features in both the time and frequency domains. Subfigure (a) represents the time-domain features, which include statistical measures such as Mean, Median, Variance, and Standard Deviation (Std-dev). These features are derived directly from EEG signal amplitudes over predefined time windows. The Mean represents the average neural activity, while the Median provides a more robust measure against outliers. Variance quantifies the dispersion of EEG amplitudes, and Standard Deviation indicates the overall spread of values. Subfigure (b) visualizes the frequency-domain features, where power spectral density variations are analyzed to characterize signal behavior in the frequency spectrum. Key extracted features include Mean Frequency, Median Frequency, Variance of Frequency, and Standard Deviation of Frequency, which provide insights into how neural activity is distributed across different frequency bands. These frequency-domain features play a crucial role in understanding rhythmic neural patterns associated with motor imagery tasks. In Table 4. Among the time domain features, statistical measures such as Mean, Median Variance, and Standard Deviation (Std-dev) play a pivotal role in characterizing the temporal dynamics of EEG signals⁴⁷. Here’s a detailed exploration of these features:

Mean: The Mean represents the average amplitude of the EEG signal within a specified epoch or time window. It provides a measure of the central tendency of the signal, indicating the typical magnitude of neural activity during the recording period. A higher mean value suggests overall elevated levels of neural activation, while a lower mean value indicates reduced activity.

Where N is the number of observations, and x_i represents each individual observation.

Median: The Median is the middle value of the EEG signal when all amplitude values are arranged in ascending order. Unlike the mean, which is influenced by extreme values (outliers), the median is robust to outliers and provides a measure of the central tendency that is less affected by extreme fluctuations in the signal. It offers insights into the typical or representative amplitude value within the data, particularly in the presence of non-Gaussian or skewed distributions.

Variance: Variance quantifies the dispersion or spread of amplitude values around the mean of the EEG signal. It measures the average squared deviation of each amplitude value from the mean, providing information about the variability or scatter of the data points. A higher variance indicates greater variability in neural activity, reflecting fluctuations or changes in the underlying cognitive processes.

Where N is the number of data points in the population, n is the sample size, xi is each individual observation, µ is the population mean, and \(\bar{x}\) is the sample mean.

These time domain features are fundamental in EEG analysis, offering quantitative descriptors of the temporal structure of brain activity during motor imagery tasks. By computing and analyzing these statistical measures, imagery, paving the way for the development of effective brain-computer interface systems and rehabilitation protocols.

Why GANs??

In the context of EEG signal processing, GANs can be leveraged to address challenges such as data scarcity and to enhance feature learning. In this study, GANs are employed primarily for data augmentation and feature generation. The EEG dataset used in our research is limited in size, which poses challenges for training deep learning models effectively. By generating synthetic EEG signals, GANs help in creating a more diverse and representative training set, leading to improved generalization of the models²⁷. Additionally, the complex features generated by GANs can capture intricate patterns in EEG data, which are often missed by conventional feature extraction methods. The GAN architecture used in this research consists of a deep convolutional generator and a discriminator network. The generator is designed to learn the distribution of the EEG signals and generate realistic synthetic samples, while the discriminator aims to distinguish between real and generated EEG signals. The GAN was trained for 100 epochs with a batch size of 64, utilizing the Wasserstein GAN architecture with gradient penalty to ensure stable training and convergence. The quality of the synthetic EEG signals generated by the GAN was evaluated using the Fréchet Inception Distance (FID) score, which measures the distance between the real and synthetic data distributions. A lower FID score indicates that the generated EEG signals closely resemble the real signals, validating the effectiveness of the GAN in data augmentation. The incorporation of GAN-generated EEG signals in the training process led to a notable improvement in model performance. Specifically, the hybrid CNN-LSTM model demonstrated a 10% increase in accuracy and a significant reduction in overfitting when trained with the augmented dataset. This highlights the effectiveness of GANs in enhancing the generalization capabilities of deep learning models for EEG classification.

Classification

Classification is the process of categorizing data points into predefined classes. In this paper, a classifier is employed to distinguish between different types of movements: left hand, and right hand. Five ML classification algorithms and two Deep Learning algorithms and one proposed hybrid model were utilized for this purpose:

K-Nearest neighbors (KNN)

KNN is a simple, yet effective, non-parametric algorithm widely used in BCI research. It classifies a sample based on the majority label of its nearest neighbors. Despite its simplicity, KNN often suffers from high computational cost and sensitivity to noisy data.

Support vector classifier (SVC)

SVC is a supervised learning model used for classification. It works by finding the hyperplane that best divides a dataset into classes. The model seeks to maximize the margin between the classes while minimizing classification errors.

Random forest (RF)

RF is an ensemble learning method that constructs multiple decision trees during training and outputs the mode of the classes for classification. It is highly effective in handling large datasets and provides high accuracy. In Fig. 10 Confusion metric is illustrated.

Logistic regression (LR)

Logistic Regression is a linear model used for binary classification. It estimates the probability that a given input belongs to a particular class (e.g., 0 or 1) by fitting a logistic function to the data. The model outputs probabilities between 0 and 1, which can be thresholded to assign class labels.

Naïve Bayes (NB)

Naive Bayes is a probabilistic classifier based on Bayes’ theorem, assuming independence between predictors. Despite its simplicity and the unrealistic independence assumption, it performs well in various classification tasks, especially in text classification.

Convolutional neural networks (CNN)

CNNs are deep learning models designed for processing grid-like data such as images. They are particularly effective at extracting spatial features through convolutional layers, which apply filters to input data to capture patterns like edges, textures, and shapes.

The function f represents the activation function (like ReLU), W represents the filter weights, and b is the bias.

Long short-term memory (LSTM)

LSTM networks are a type of recurrent neural network (RNN) designed to learn long-term dependencies in sequential data. LSTMs overcome the vanishing gradient problem present in standard RNNs by using memory cells that can maintain information over time. The activation function σ controls what information is retained or discarded from the cell state.

Proposed hybrid model (CNN + LSTM)

In this paper, this study proposed a novel approach for enhancing the classification accuracy of motor imagery tasks in BCI systems by integrating CNNs and LSTM networks into a hybrid model. This hybrid model is designed to leverage the strengths of both CNNs and LSTMs, making it highly effective in processing and analyzing complex EEG signals associated with imagined motor movements.

As depicted in Fig. 6, the EEG signals are first pre-processed and segmented into time-series data, which are then fed into the CNN module of our hybrid architecture.

The CNN component is responsible for extracting spatial features from the EEG data. Given the ability of CNNs to identify patterns and structures in grid-like data, the present work designed the CNN module with multiple convolutional layers, each followed by a pooling layer to reduce the dimensionality and computational load while retaining the most important features. The filters in the convolutional layers are specifically tuned to capture the distinct patterns present in the EEG signals, which vary in both amplitude and frequency. The detailed architecture of the CNN-LSTM hybrid model is described in Table 5.

To further enhance the model’s ability to classify these features accurately, the extracted spatial features are passed to the LSTM module, which is adept at capturing temporal dependencies. The LSTM network processes the sequential data, learning the temporal relationships between the features extracted by the CNN. This is particularly important for EEG data, where the temporal dynamics play a crucial role in distinguishing between different motor imagery tasks.

In our architecture, the authors experimented with various configurations of convolutional filters and LSTM units to identify the optimal combination for EEG signal analysis. Inspired by previous work on deep learning models, we incorporated both small and large convolutional filters within the CNN to ensure that the model can effectively capture both fine-grained and broad patterns in the data. This dual-filter approach, as shown in Fig. 6, allows the model to achieve a balance between capturing localized details and understanding broader patterns in the EEG signals.

The output of the LSTM module is then passed through a fully connected layer, which serves as the classifier in our hybrid model. This layer aggregates the learned features and outputs the final classification decision, predicting the specific motor imagery task being performed. To optimize the classification performance, this study also experimented with different activation functions and regularization techniques, which helped in reducing overfitting and improving the model’s generalization capabilities. Furthermore, to validate the effectiveness of the proposed hybrid model, the current investigation compared its performance against other classical machine learning algorithms such as SVM, Neural Networks (NN), and RF. The results showed that our CNN + LSTM model consistently outperformed these traditional approaches, underscoring the advantage of combining spatial and temporal feature extraction in a single architecture. This hybrid approach provides a powerful tool for advancing BCI applications, enabling more accurate and reliable control of external devices based on motor imagery tasks.

Comparison experiments

To assess the efficacy of our proposed methods, additional experiments were conducted. The performance of the models was evaluated using several metrics: accuracy, precision, recall, F1-score, Matthew Correlation Coefficient (MCC), and response time are calculated as follows⁴⁸: