This section provides a detailed assessment of the EegenCoder model and demonstrates the classification capabilities in the BCI Competition IV 2A dataset.37. Comparing the performance of the model with a variety of established models, highlighting its effects in deciphering complex patterns inherent in the EEG signal of the motor image task. Subsequent subsections detail the performance metrics of the model, comparative analysis with other models, and explain the importance of these results regarding the progression of brain-computer interface technology.
Dataset
In our study, we utilized the BCI Competition IV dataset 2A (BCI-2A) primarily for training and evaluation of the EegenCoder model. The BCI-2A dataset consists of recordings from nine healthy subjects, each performing four different motor imaging (MI) tasks.
Each subject participated in two sessions recorded on different days. Each session consisted of six runs, each run included 48 attempts (12 attempts per MI task), with a total of 288 attempts per session. EEG signals were recorded using 22 AG/AGCL electrodes at a sampling rate of 250 Hz. The signal was filtered between 0.5 Hz and 100 Hz, and a 50 Hz notch filter was applied to reduce power line interference.
At the start of each session, a recording was made for approximately 5 minutes to estimate the impact of EOG and was divided into three blocks. Open your eyes for 2 minutes, close your eyes for 1 minute, and use your eyes for 1 minute. Due to technical issues, the EOG block of subject A04T only contains the state of eye movement.
During the experiment, subjects were seated in a comfortable armchair in front of a computer screen. Each exam began with a black screen with a staring cross and a short acoustic warning tone. After 2 seconds, an arrow-shaped cue (left, right, down, or up) appeared for 1.25 seconds, prompting subjects to perform the corresponding motor image task until the fixed intersection disappeared in 6 seconds.
In our study, one session was used for model training and the other session was reserved for evaluation testing. Raw Mi EEG signals from all bands and channels are \(c \ times t \) Two-dimensional matrix. Minimal preprocessing was applied to the raw data, normalizing the signal using standard scalers to zero mean and unit variance.
Our study focuses on the BCI-2A dataset, due to its increased complexity and the greater challenge it presents, and better demonstrates the performance capabilities of the model. The dataset is well documented and widely used in the BCI community, ensuring reliability and relevance for assessing EEG classification models.
Certainly, here is a revised section that includes information transfer rate (ITR) as a new metric.
Performance Metrics
To assess the performance of EegenCoder, we employ three important metrics: Accuracy, Cohen's Kappa and ITR. These metrics provide a comprehensive assessment of the model's classification capabilities.
The accuracy (ACC) is calculated as follows:
$$\begin{aligned}\text{acc}=\frac{\sum^n_{i=1}\frac{tp_i}{i_i}}{n}\end{aligned}$$
(8)
where n The number of categories. \(tp_i \) Represents the true positive count of a class Iand \(i_i \) The total number of samples for a class I. Accuracy measures the percentage of correctly classified samples and provides a simple assessment of the overall performance of the model.
Cohen's Kappa (Kappa) is calculated using the formula.
$$\begin{aligned}\text{kappa}=\frac{1}{n}\sum^n_{a=1}\frac{p_a -p_e}{1 -p_e}\end{aligned}$$
(9)
where \(p_a \) Shows the actual percentage of the contract \(p_e \) It represents the expected percentage of a contract by chance. Kappa is particularly important for this task as it is tailored to coincidental agreements, providing a more reliable measure of model performance, especially in scenarios with disproportionate class distributions.
Information Transfer Rate (ITR) is another important metric in the BCI field. This is to quantify the speed and efficiency of information transmission from the brain to the computer. The ITR is calculated using formulas.
$$\begin{aligned}\text{itr}=\frac{60}{t}\left[ \log _2 N + P \log _2 P + (1 – P) \log _2 \left( \frac{1 – P}{N – 1} \right) \right] \end {aligned} $$
(10)
where t Average time in seconds per trial, n The number of possible targets or classes, and p Classification accuracy. The ITR is measured in bits per minute (bits per minute) and provides insight into how efficiently the system can convert brain signals into executable commands.
Accuracy and Cohen Kappa are standard metrics for assessing the performance of classification tasks. Accuracy provides a direct measure of the model's ability to correctly classify brain wave segments, usually expressed as a percentage. However, in a dataset with unbalanced classes, it may not be sufficient to rely solely on accuracy. Cohen's Kappa addresses this issue with the possibility of a coincidence agreement and provides a more reliable rating metric. Kappa is expressed as a decimal number in the range 0 to 1.
ITR is particularly important in the BCI domain. This makes it a critical metric for practical applications where both accuracy and efficiency are important because it takes into account not only accuracy but communication speed. By incorporating ITR, we ensure that our assessment captures the real-world ease of use of EegenCoder in BCI applications.
This dual evaluation approach, currently augmented in ITR, ensures a comprehensive assessment of the validity, reliability and efficiency of the model in classifying MI-EEG signals. Furthermore, these metrics are commonly used in similar studies and are suitable for evaluation.
Training structure
The model is trained on a specific set of parameters as outlined in Table 1.
The Crossentropyloss function is used in label smoothing, which is set to a value of 0.1 to soften the target distribution, and may improve model generalization. To further normalize the training process and prevent overfitting, a dropout ratio of 0.3 is applied across the network, and a weight loss attenuation with a factor of 0.5 is applied to all MLP layers.
Results for BCI IV 2A dataset
The EegenCoder model was comprehensively evaluated using the BCI Competition IV dataset 2A. Performance was evaluated on three key metrics: accuracy, Cohen's kappa and ITR. We compared the EegenCoder with four cutting-edge models: ATCNET, TCNETFUSION, EEGTCNET, and D-ACTNET. To ensure the robustness and reliability of the results, we conducted experiments using five different random seeds. Each model was trained and tested in the same experimental setting, and the mean results from these five iterations were reported.
While reimplementing Atcnet, TcnetFusion, and Eegtcnet, Altaheri et al.32d-actnet source code was not available. As a result, we used the average performance metrics reported in the D-ACTNET paper as the basis for the comparison. Due to this limitation, we did not calculate the ITR for d-actnet. Classification results for all nine subjects, as well as detailed comparisons between the models, are shown in Table 2.
In terms of accuracy and Cohen's kappametrics, EegenCoder outperformed the comparison model of eight out of nine subjects, except for subject 4. This model demonstrated particularly important performance improvements in subjects 2 and 5, highlighting the enhanced ability to manage EEG signal variation observed in these individuals.
From an ITR comparison perspective, the EegenCoder model is superior to the other models. ITR mainly evaluates the prediction accuracy and speed of the model, and the benefits of EegenCoder are attributed to three main factors: First, EegenCoder's architecture significantly compresses continuous information on its base, allowing for a four-layer transformer without introducing excessive complexity. Second, Pytorch optimization accelerates the calculation of attention mechanisms and improves overall model efficiency. Finally, the use of a stable transformer that is faster and reduces memory compared to traditional transformers contributes to faster predictions. As a result, EegenCoder already shows superior speeds with impressive accuracy when compared to other models.
The results show that not only does Eegencoder perform well overall, but it also shows the resilience of subjects whose other models tend to decline. This resilience can be attributed to the architecture of the model. This may be adept at capturing the nuances of EEG signals across diverse cognitive tasks. However, further research is needed to review these findings and maximize the potential of EegenCoder in real BCI applications.
Ablation research
A series of ablation experiments were conducted to verify the effectiveness of various enhancements applied to EegenCoder. We began by integrating data from all nine subjects and integrating each training and test set into a single comprehensive data set. This approach has allowed us to more effectively evaluate the generalization of model improvement across a variety of subjects.
Here we present a selection of important experiments that will help us assess the impact of a particular modification. These experiments included removing transformer components from the DSTS block, using five shift windows instead of five dropout branches, varying the number of transformer layers, adjusting the amount of DSTS branches in the Eegencoder, and comparing the performance of the modified stable transformer with a vanilla transformer. Results were averaged over five iterations to ensure statistical significance of the results. Each was initialized with a different random seed. The summary results are shown in Table 3.
The data in Table 3 shows the impact of each change on EegenCoder performance. The removal of the trans component significantly reduced both the accuracy and Cohen kappa, highlighting its contribution to the effectiveness of the model. Adjusting the number of transformer layers showed that a balance is needed to optimize performance, as evidenced by a slight decrease in accuracy at 8 layers compared to the two. Similarly, we found that the number of DSTS branches was a factor. A single branch reduces performance, while 10 branches do not improve that much. Finally, a comparison of stable and vanilla transformer variants demonstrates the importance of corrections to achieve higher accuracy and Cohen kappa.
