A deep learning framework for gender sensitive speech emotion recognition based on MFCC feature selection and SHAP analysis

To validate the presented approach, all stages were developed and simulated in MATLAB. This simulation displays the results of an emotion recognition algorithm applied to the speaker’s speech. It also enables the comparison of these results to earlier research conducted in comparable settings. The wrapper method’s feature selection mechanism is tied to the class clause. To choose the best training and optimization strategy from the available options, all methodologies and classifications were tested and compared. The technique that produced the best results was then selected. Certain techniques needed more than 72 h of training and evaluation. The results of these tests are shown in Table 9. Each speaker may be represented by one or more Gaussian cones facing either direction. The number of these cones, commonly known as the method’s order, directly affects the algorithm’s accuracy and execution time. A larger number of cones increases accuracy and processing speed, but each model requires more data^41,42.

As seen in the table above, the bat algorithm performs the best, hence it was used for feature selection. The most successful optimization procedure is determined by the highest proportion of emotions detected and time efficiency. Table 10 presents the results of the suggested strategy and technique, split down by gender. Gender effects how emotions are recognized in the voice across all emotion categories. The proposed system’s accuracy and performance were compared to the results of speech recognition trials conducted at the University of Berlin, as well as similar algorithms used on a common database^{17,29,43,44,45}.

To further contextualize the effectiveness of our proposed model, we conducted a comparative evaluation with several state-of-the-art deep learning architectures commonly used in speech emotion recognition, including VGGNet, InceptionV3, ResNet-50, and a recent Transformer-based model. All models were trained and tested under the same 10-fold cross-validation protocol using the Berlin Emo-DB dataset. Performance metrics including accuracy, precision, recall, and F1-score were recorded for all models. As shown in Fig. 10; Table 11, our proposed model consistently outperformed the benchmark models, achieving the highest overall accuracy of 89.5%, compared to 85.4% (VGG), 84.7% (InceptionV3), 82.9% (ResNet-50), and 83.6% (Transformer). The proposed model also demonstrated faster training convergence and lower computational overhead due to its optimized feature selection and classification pipeline. These results confirm the efficacy and efficiency of our system for practical emotion recognition applications.

To evaluate the feasibility of deploying the model in real-time settings, we measured inference time and throughput on the NVIDIA Jetson Nano (quad-core ARM Cortex-A57, 4GB RAM). The model achieved an average inference time of 72 ms per sample, equivalent to 13.9 FPS, confirming its capability for near-real-time processing. Latency was measured using ONNX-runtime with quantized 8-bit weights, and end-to-end delays remained below 120 ms under moderate system load. These results validate the system’s applicability for edge-deployed use cases, such as emotion-aware dialogue agents or in-vehicle driver monitoring systems. To further validate the proposed model’s performance, we conducted a comparative analysis with recent SOTA deep learning architectures for speech emotion recognition, including 2D CNNs⁷, RNNs with attention mechanisms¹³, and Transformer-based models³. As shown in Table 9; Fig. 10, our framework achieves superior accuracy (89.5%) compared to ResNet-50 (82.9%), InceptionV3 (84.7%), and a Transformer baseline (83.6%). Notably, the proposed model’s integration of temporal (LSTM) and spectral (MFCC) features reduces misclassification between high-arousal emotions (e.g., anger vs. happiness) by 15% over pure CNN-based approaches⁷. While attention-based RNNs¹³ achieve comparable precision (87.2%), our gender-separated pipeline significantly improves female emotion detection (96.35% vs. 89.1% in³, addressing a key limitation in generalizability. However, the computational cost of our wrapper-based feature selection (Table 7) remains higher than end-to-end Transformer models, suggesting a trade-off between interpretability and scalability for real-time applications. Compared to state-of-the-art conversational models like CHAN⁴⁶ that achieve 86.2% accuracy on dyadic speech, our gender-specific approach demonstrates superior performance (89.5%) on monologue datasets while using 30% fewer parameters.

To provide a more rigorous and comprehensive assessment of the proposed model’s performance, we extended the evaluation beyond traditional accuracy and detection rate by incorporating additional metrics: precision, recall, F1-score, and the area under the receiver operating characteristic curve (AUC-ROC). Precision quantifies the proportion of correctly identified positive instances among all predicted positives, thereby addressing the issue of false positives. Recall, or sensitivity, measures the ability of the model to correctly identify all relevant instances, minimizing false negatives. The F1-score, defined as the harmonic mean of precision and recall, offers a balanced evaluation metric particularly useful in imbalanced classification scenarios. Moreover, we employed the AUC-ROC metric to evaluate the model’s discriminative capacity. The ROC curve illustrates the trade-off between the true positive rate (TPR) and false positive rate (FPR) across varying thresholds, with the AUC providing a single scalar value to summarize performance—where higher values indicate better separability between emotion classes. To ensure the robustness and reliability of our results, we implemented 10-fold cross-validation on the Berlin emotional speech database. The dataset was partitioned into ten non-overlapping subsets; in each fold, one subset was used for testing while the remaining nine were used for training. The final performance metrics were averaged across all folds to mitigate variance arising from data partitioning. Table 12 summarizes the Comprehensive metrics per emotion including precision, recall, F1-score, and AUC-ROC for each class. The inclusion of AUC-ROC addresses classification performance for low-frequency emotion classes such as fear and disgust, offering deeper insight into clinical and affective applications.

To rigorously evaluate the significance of the proposed model’s performance improvements over baseline methods, a series of statistical analyses were conducted based on 10-fold cross-validation results. A paired t-test comparing the model’s accuracy to the best-performing baseline (Bat Algorithm) revealed a statistically significant improvement (t(9) = 4.32, p < 0.01), confirming that the observed gains—approximately 10% over human-level accuracy and 15% over previous methods—are not due to random variation. Additionally, a one-way ANOVA assessing gender-based accuracy differences indicated a significant effect of gender on classification performance (F(1, 18) = 8.67, p = 0.008), with a post-hoc Tukey test confirming that female speakers achieved significantly higher accuracy (96.35%) than male speakers (87.18%) (p < 0.05). To further validate the reliability of the model, 95% confidence intervals were computed: overall accuracy was 89.5% ± 1.2% for the proposed model versus 83.1% ± 1.8% for the baseline, while gender-specific accuracies were 96.35% ± 0.9% for females and 87.18% ± 1.5% for males. These results collectively substantiate the model’s statistical robustness, its consistent improvement over baseline methods, and the relevance of gender-separated classification. To rigorously evaluate the significance of the proposed model’s performance improvements over baseline methods, a series of statistical analyses were conducted based on 10-fold cross-validation results. A paired t-test comparing the model’s accuracy to the best-performing baseline (Bat Algorithm) revealed a statistically significant improvement (t(9) = 4.32, p < 0.01), confirming that the observed gains—approximately 10% over human-level accuracy and 15% over previous methods—are not due to random variation. Additionally, a one-way ANOVA assessing gender-based accuracy differences indicated a significant effect of gender on classification performance (F(1, 18) = 8.67, p = 0.008), with a post-hoc Tukey test confirming that female speakers achieved significantly higher accuracy (96.35%) than male speakers (87.18%) (p < 0.05). To further validate the reliability of the model, 95% confidence intervals were computed: overall accuracy was 89.5% ± 1.2% for the proposed model versus 83.1% ± 1.8% for the baseline, while gender-specific accuracies were 96.35% ± 0.9% for females and 87.18% ± 1.5% for males. These results collectively substantiate the model’s statistical robustness, its consistent improvement over baseline methods, and the relevance of gender-separated classification. In other hand, to further demonstrate the effectiveness of the proposed model, we conducted additional experiments using baseline classifiers, including a simple CNN, Support Vector Machine (SVM), and Random Forest (RF). All models were trained on the same extracted features and evaluated using 10-fold cross-validation. The baseline CNN achieved an average accuracy of 78.2%, while the SVM and RF models obtained 74.1% and 76.3%, respectively. In contrast, our proposed model achieved 89.5%, clearly outperforming the baseline approaches. This confirms the substantial performance gain provided by our model’s hybrid structure, which combines deep feature extraction with temporal modeling and topological enhancement.

To further highlight the advantages of the proposed AI-based model, we compared it with traditional speech emotion recognition approaches based on hand-crafted acoustic features. These methods used time-domain features (e.g., pitch, energy) and frequency-domain features (e.g., MFCCs), classified using standard machine learning techniques such as k-Nearest Neighbors (k-NN) and Gaussian Naive Bayes (GNB). These traditional pipelines achieved average accuracies of 71.5% (k-NN) and 69.8% (GNB) on the same dataset and under the same 10-fold cross-validation protocol. In contrast, the proposed deep learning model achieved 89.5%, indicating a clear improvement in accuracy and generalization. Additionally, the AI-based method was more robust to inter-speaker variability and noise, and eliminated the need for manual feature engineering, thereby offering a more scalable and efficient solution for real-world emotion recognition tasks. To assess the individual contributions of core components in the proposed model, we conducted ablation studies by systematically removing key elements. Three variants were tested: Without MFCCs (replaced by raw waveform input): accuracy dropped to 82.4%. Without LSTM (CNN only): accuracy reduced to 84.1%, indicating loss of temporal modeling. Without both MFCCs and LSTM: accuracy declined further to 78.5%. These results confirm that MFCCs provide essential frequency-domain information, while the LSTM layer is crucial for capturing temporal patterns in emotional speech. The combination of both components yields the highest accuracy (89.5%) and supports the model’s robustness across different speaker conditions. Temporal analysis of LSTM attention weights demonstrated distinct emotion-dependent patterns. For fear and anger, the model prioritized mid-sentence frames, coinciding with peaks in vocal tension. Sadness and tiredness, however, exhibited uniformly distributed attention across utterances, consistent with their steadier prosodic profiles.

Figure 6 shows the results of testing the proposed strategy on the University of Berlin’s emotional database and comparing it to a similar approach. As seen by the graph and table, the proposed strategy is effective. While it is somewhat less good at detecting specific emotion patterns than similar algorithms, it excels at recognizing a broader range of emotions and has a higher average accuracy in emotion detection.

The suggested emotion identification system is largely dependent on speech signals; however, as part of a larger multimodal system, facial expressions might be included to increase detection accuracy. When images are used as auxiliary input, they are first scanned using face recognition software. If the Haar-Cascade conditions are fulfilled, human faces will appear. The selected picture window is pre-processed into a 64 × 64 pixel image for input into a convolutional neural network. Depending on the architecture utilized, the CNN phase extracts one of the seven facial emotions (such as rage, pleasure, or sadness) as the network’s output via convolution, pooling, ReLU activation, and fully connected layers¹⁷. In object identification applications, pseudo-Haar features like Haar wavelets are often utilized. These features, which were initially used in the Viola-Jones method for real-time face identification, compare pixel sums in white and black rectangles (Fig. 11). This approach recognizes certain patterns, such as facial traits, by placing these rectangles across the image. Although the major emphasis is on speech-based emotion identification, facial expression recognition might give additional information in multimodal systems. To advance this direction, we propose a concrete fusion strategy for multimodal SER. Specifically, we will compare early fusion (concatenating MFCC features with CNN-extracted facial features before feeding into LSTM) and late fusion (combining the softmax outputs from speech and vision models using weighted averaging or ensemble learning). Early fusion offers richer joint feature representations but requires tight temporal alignment; late fusion is more robust to asynchronous modalities. Prior studies such as^{13] and [11} demonstrate promising results with transformer-based late fusion in multimodal SER. Our framework will adopt a similar approach using synchronized facial and speech signals from datasets like RAVDESS and CREMA-D, allowing us to evaluate cross-modal complementarity in emotion classification.

Viola-Jones approach pulls characteristics from photos by placing many rectangles of varied sizes over the face, using a 24 × 24 window. These rectangles calculate the difference in total pixels between the white and black portions, which is then compared to specified training data values. If the calculated value exceeds a certain threshold, the desired attribute is discovered in that region. Calculating 160,000 attributes for each 24 × 24 window may be time-consuming and computationally expensive. Once the face recognition system has extracted the facial images, they are fed into a deep neural network, such as a Convolutional Neural Network (CNN) (Fig. 12). A CNN is a deep neural network that was specifically designed to analyze visual input. CNNs interpret input pictures using a series of convolutional layers in which learnable filters (kernels) identify edges, textures, and patterns. Each neuron in the network has a set of weights and biases that are adjusted during training to decrease prediction errors. The network incorporates non-linear activation functions, such as ReLU (Rectified Linear Unit), to provide non-linearity, enabling it to learn complex patterns. The CNN operates by assigning scores to each available category based on the features learned during training. These scores, which correspond to different classes or categories (in this example, the seven facial emotions), are calculated by routing image data through the network’s layers. After the network has processed the input, the class with the highest score is selected as the final output, which determines the facial expression or emotion in the image. This approach combines the Viola-Jones algorithm’s rapid feature extraction with deep neural networks’ exceptional learning capabilities, yielding a dependable system for identifying facial emotions.

In the last layer, known as the fully connected layer, convolutional neural networks (CNNs) use a cost function similar to Support Vector Machines (SVM) or Softmax. CNNs use the same concepts as regular neural networks. CNNs differ greatly from standard artificial neural networks in terms of architecture. CNNs are specifically designed to accept images as input, enabling unique characteristics to be directly included into the network architecture. This assumption enables a more efficient implementation of the forward function and, more importantly, significantly reduces the number of network parameters⁴³. Standard neural networks typically feature a simple list of neurons, while CNNs have a three-dimensional list. Following the convolution layer, the data is normalized using an activation function called ReLU. A pooling layer is often put between many convolution layers in a multilayer architecture. This is done to reduce computational complexity, restrict the number of parameters, and avoid overfitting. The pooling layer reduces image size (input) and spatial dimensions (width and height), hence lowering the network’s computational cost. The pooling approach, also known as max pooling, uses a size maximization function on each depth slice of the input mass. The method chooses the greatest value from a 2 × 2 rectangle inside each depth slice. As a result, the depth dimension remains constant while the spatial dimension shrinks. The pooling layer gets an input mass with dimensions W×H×D. Depending on the window size and network objectives, it generates an output that averages or selects the highest value from the provided range as shown in Fig. 13.

As a consequence, the output of all neurons (activations) may be approximated concurrently using matrix multiplication and bias. In reality, this is the system’s last and scoring layer for output (Fig. 14).

AlexNet, created by Geoff Hinton and Alex Krizhevsky, was the first neural network to get significant attention in the field of machine vision. AlexNet won the ILSVRC competition in 2012, outperforming all other systems by a wide margin. The network’s architecture was similar to that of LeNet, but it was deeper, larger, and had more convolution layers. In recent years, new deep convolutional neural network topologies have been proposed, which provide even more accurate results. Achieving such astounding results often requires tremendous hardware resources.

This research makes use of the Xception architecture developed by François Chollet at Google Brain. Xception is an extension of Google’s InceptionV3 architecture. When compared to frequently used architectures like ResNet, VGGNet, and AlexNet, Xception significantly reduces the number of parameters required for computation. This parameter reduction not only accelerates training, but also shortens processing time and improves output accuracy. The primary novelty of Xception is the lowering of convolutional network parameters via two concurrent and separable convolution operations: point-wise convolution and depth-wise convolution. This strategy maintains accuracy while significantly lowering the number of parameters. To fully understand the effect of these convolutions, we must consider both the computational complexity of the convolution layer and the number of parameters involved. This project is totally open source and was built using the Python programming language. The CPU used is an Intel Core i7 MQ4700, which has eight logical cores and can reach 4.2 GHz. Face recognition is performed using the OpenCV framework and the Viola-Jones approach. The TensorFlow-Keras framework is used to identify different emotional states in human faces. The FER2013 Kaggle dataset, consisting of 709,280 photographs for training and 58,930 images for testing, is utilized (Fig. 15). The collection features photographs of humans in seven different emotional states: furious, disgusted, terrified, thrilled, melancholy, surprised, and neutral. For training, 20% of the data is put aside for validation, with the remaining 18% used for the validation set. The network was trained for about 14 h using the stated hardware and design⁴⁷. Figure 16 demonstrates the intermixing of different emotional states after the training and evaluation periods are completed.

Although the proposed approach shows promising results on the Berlin emotional speech database, it is important to acknowledge its limitation in terms of generalizability. The Berlin database primarily contains recordings from a limited number of speakers with controlled emotional expressions in German, which may not represent the diversity of real-world speech patterns, accents, and emotional nuances. To enhance the robustness and applicability of the model, future work will involve testing the system on more diverse and multilingual datasets such as RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song), SAVEE (Surrey Audio-Visual Expressed Emotion), and CREMA-D (Crowd-Sourced Emotional Multimodal Actors Dataset). This will help validate the model’s performance across different demographics, languages, and recording conditions. In real-world deployments, especially in customer-facing applications such as feedback analytics and virtual assistants, ethical considerations must be prioritized. One major concern is bias arising from demographic imbalances, such as underrepresentation of minority groups, dialects, or emotional expression styles in training datasets. The Berlin Emo-DB, for example, contains only a small set of speakers from a homogeneous cultural background. To mitigate this, future work will employ balanced data augmentation strategies (e.g., oversampling underrepresented classes, pitch shifting for accent diversity) and explore adversarial domain adaptation techniques to reduce demographic bias. Transparent reporting of model fairness metrics (e.g., per-group accuracy) will be incorporated to ensure equitable performance across speaker demographics.

While the model achieves high classification performance, it is also important to ensure transparency in its decision-making process. The primary features used for classification—Mel-frequency cepstral coefficients (MFCCs)—are well-established in speech processing for capturing timbral and spectral characteristics. In emotional speech, MFCCs can reflect changes in vocal tension, pitch dynamics, and resonance, which are directly influenced by emotional states such as anger (high energy, pitch variability) or sadness (lower energy and slower speech). To further enhance the interpretability of the model, we employed SHapley Additive exPlanations (SHAP) values to analyze feature importance across emotion classes. SHAP provides a unified measure of each feature’s contribution to a specific prediction by assigning importance scores based on cooperative game theory. This analysis revealed that specific MFCC coefficients consistently held higher importance in distinguishing aroused emotions (e.g., anger and happiness) versus subdued states (e.g., tiredness and sadness). While a full integration of LIME (Local Interpretable Model-Agnostic Explanations) is left for future work, preliminary experiments showed that the model’s predictions were locally influenced by distinct patterns in the MFCC spectrum, supporting the interpretability of the extracted features. These explainability tools provide additional confidence in the system’s decision-making process and lay the foundation for its potential use in high-stakes or user-facing applications.

Although the current evaluation was performed on the Berlin Emo-DB dataset under controlled conditions, deploying a speech emotion recognition (SER) system in real-world settings introduces multiple challenges. These include variations in background noise, discrepancies in recording devices, speaker-specific factors (e.g., accents, prosody), and differing acoustic environments. To mitigate these issues, subsequent research will assess the model on additional datasets featuring diverse environmental conditions and speaker profiles (e.g., CREMA-D, RAVDESS, SAVEE). Moreover, data augmentation techniques—such as injecting noise, simulating reverberation, and applying frequency shifts—will be utilized to enhance the model’s resilience against real-world acoustic distortions. For practical implementations—including customer service analytics, emotion-aware virtual assistants, and driver monitoring systems—these measures are essential to ensure robustness and generalizability. Therefore, future deployments will integrate adaptive preprocessing modules, noise suppression techniques, and real-time calibration mechanisms to maintain accurate emotion detection across varying use cases. SHAP value analysis revealed consistent discriminative patterns in MFCC feature importance across emotion classes. MFCC coefficients 3, 5, 7, and 11 were critical for high-arousal emotions (e.g., anger and happiness), with MFCC-5 alone contributing 28% of the variance in SHAP scores for anger. Conversely, low-frequency MFCCs (1–3) showed higher importance for subdued emotions like sadness and tiredness, reflecting known vocal resonance shifts during these states. Ranking of MFCCs by mean SHAP values further confirmed these spectral-emotion associations.