Integrating skeleton based representations for robust yoga pose classification using deep learning models

We have implemented 9 distinct yoga pose classification techniques as mentioned in Table 10. All experiments were conducted on Kaggle’s cloud-based platform using consistent system configuration as shown in Table 11. The experiments were conducted in three input modalities – (i) Direct Image Input, (ii) Mediapipe Pose Skeleton Image Input and (iii) YOLOv8 Pose Skeleton Image Input, to evaluate and compare performance of skeleton based and non-skeleton based input scenarios. Under these input scenarios, we described the performance of deep learning models (VGG16, ResNet50 and Xception) below.

Performance assessment of deep learning models using direct image input

VGG16 achieves high accuracy 86.33% as shown in Table 12 and excels at identifying specific poses like Chair Pose, Goddess Pose, and Seated Forward Bend Pose, with high precision and recall. However, it struggles with poses that have subtle structural features, such as Fish Pose. The confusion matrix, loss, and accuracy curves, and ROC curve of VGG16 with direct image input are shown in Fig. S1, Fig. S2, and Fig. S3, respectively. ResNet50, with an overall accuracy of 66.41%, has lower precision and recall across many poses, particularly for Fish Pose and Warrior 2 Pose, indicating difficulty distinguishing similar poses. Despite these challenges, ResNet50 performs better with poses like Side Plank Pose and Warrior 3 Pose. The confusion matrix, loss, and accuracy curves, and ROC curve of ResNet50 with direct image input are shown in Fig. S4, Fig. S5, and Fig. S6, respectively. Xception, with an accuracy of 84.38%, performs well across a wide range of poses, excelling in poses like Dolphin Plank Pose, Downward Facing Dog Pose, and Warrior 3 Pose. The confusion matrix, loss, and accuracy curves, and ROC curve of Xception with direct image input are shown in Fig. S7, Fig. S8, and Fig. S9, respectively. Classification summaries for VGG16, ResNet50, and Xception are shown in Table S1, Table S2, and Table S3.

Performance of pretrained deep learning models on Mediapipe skeleton image input

MediaPipe skeleton extraction emphasizes key pose features, reducing background noise to enhance classification. VGG16 emerged as the best performer, achieving 96.09% accuracy with balanced precision, recall, and F1-score, supported by consistent training progress and robust sensitivity-specificity balance. The confusion matrix, loss and accuracy curves, and ROC curve of VGG16 with Mediapipe skeleton image input are shown in Fig. S10, Fig. S11, and Fig. S12, respectively. ResNet50, while achieving 88.28% accuracy, effectively captured skeletal details but faced occasional misclassifications, as reflected in its metrics and training curves. The confusion matrix, loss, and accuracy curves, and ROC curve of ResNet50 with Mediapipe skeleton image input are shown in Fig. S13, Fig. S14, and Fig. S15, respectively. Xception followed with 93.36% accuracy, leveraging depthwise separable convolutions for robust classification despite minor errors. The confusion matrix, loss, and accuracy curves, and ROC curve of Xception with Mediapipe skeleton image input are shown in Fig. S16, Fig. S17, and Fig. S18, respectively. Classification summaries for VGG16, ResNet50, and Xception are shown in Table 13, Table S4, and Table S5.

Performance of pretrained deep learning models on YOLOv8-pose skeleton image input

This section compares the performance of VGG16, ResNet50, and Xception on yoga pose classification using YOLOv8-Pose skeleton images, which emphasize key pose features by reducing background distractions. VGG16 achieved notable results with high accuracy, balanced sensitivity and specificity, and consistent training improvement. The confusion matrix, loss and accuracy curves, and ROC curve of Xception with YOLOv8 Pose skeleton image input are shown in Fig. S19, Fig. S20, and Fig. S21, respectively. ResNet50 demonstrated robust generalization with smooth training and a strong AUC, though it faced occasional misclassifications. The confusion matrix, loss and accuracy curves, and ROC curve of ResNet50 with YOLOv8 Pose skeleton image input are shown in Fig. S22, Fig. S23, and Fig. S24, respectively. Xception leveraged its efficient architecture to achieve high accuracy and low overfitting, despite minor challenges with similar classes. The confusion matrix, loss and accuracy curves, and ROC curve of Xception with YOLOv8 Pose skeleton image input are shown in Fig. S25, Fig. S26, and Fig. S27, respectively. These results underline the models’ effectiveness, with VGG16 and Xception excelling in this application. Classification summaries for VGG16, ResNet50, and Xception are shown in Table S6, Table S7, and Table S8. In comparison, VGG16 achieved 91.41%, ResNet50 achieved 75.00%, and Xception achieved 85.55% accuracy with YOLOv8 Pose skeleton image input as shown in Table 12.

Best model performance: VGG16 with Mediapipe skeleton image input

From the comparison discussed in the previous section, it is observed that the highest performing model on this evaluation is VGG16 when using Mediapipe skeleton image input. As shown in Table 12, we can observe that VGG16 performs better than the rest of the models, including ResNet50 and Xception, in all the evaluation metrics, with an accuracy of 96.09%, a precision of 96.27%, a recall of 96.09% and an F1-score of 96.10%.

This performance signifies that VGG16, when used with Mediapipe skeleton input, forms an excellent classier for the various poses in the dataset. The model’s high performance is reflected in greater detail within the classification report found in Table 13, in which the model achieves perfect recall and precision for many of the individual poses, such as the Downward Facing Dog Pose, Goddess Pose, and Warrior 2 Pose, to name but a few, with an overall accuracy of 96.09%. In all input types, the VGG16 model maintained the least number of trainable parameters: 541,712. At the same time, ResNet50 and Xception have a higher number of parameters: 2,114,576. The ability of VGG16 to consistently deliver better performance across a variety of metrics confirms it as the most reliable model in this work for skeleton-based pose classification using Mediapipe images.

Hyperparameter tuning for the best model: VGG16 with Mediapipe skeleton image input

A hyperparameter tuning process was carried out to improve the model’s performance further and reduce the risk of overfitting. This involved systematically adjusting key hyperparameters such as learning rate, batch size, number of epochs, and the optimizer type to find the optimal combination that would enhance generalization. Additionally, early stopping was employed to monitor the model’s performance on a validation set, halting training once the validation loss showed no improvement for a specified number of epochs. These measures helped ensure that the VGG16 model remained robust and generalizable, achieving excellent results without overfitting. VGG16’s ability to consistently deliver better performance across a variety of metrics confirms it as the most reliable model for skeleton-based pose classification using Mediapipe images.

The performance of the VGG16 model for Mediapipe skeleton-based pose classification was enhanced through hyperparameter tuning, as summarized in Table 14. Experiments showed that a \(3 \times 3\) filter size consistently outperformed \(5 \times 5\), likely because smaller filters are better at capturing fine-grained skeletal details, which are essential for distinguishing between similar poses. A pooling size of \(2\times 2\) yielded higher accuracy and F1-scores compared to \(3 \times 3\), as it preserves more spatial information, maintaining the relative positions of keypoints critical for accurate classification. Models trained with a batch size of 32 performed better than those with a batch size of 64, as smaller batches allow for more frequent and precise weight updates, improving the stability of gradient estimates and helping the model generalize better. Among optimizers, Adam consistently outperformed RMSProp due to its adaptive learning rate, which adjusts each parameter individually and allows the model to converge faster and more reliably. Furthermore, experiments with dense layer sizes revealed that a fully connected layer of 1024 neurons achieved better performance than smaller configurations such as 512 or 256. The larger dense layer provided higher representational capacity, enabling the network to model complex relationships between skeletal keypoints and capture subtle distinctions between challenging pose classes, while still avoiding overfitting due to proper regularization. The best configuration, achieved in Experiment 1, included a \(3 \times 3\) filter size, \(2\times 2\) pooling size, batch size of 32, dense layer of 1024 neurons, and Adam optimizer, resulting in the highest accuracy (96.09%) and F1-score (96.13%). Overall, these hyperparameter choices helped the VGG16 model focus on subtle skeletal features, train efficiently, and generalize well, leading to superior performance in Mediapipe skeleton-based pose classification.

Evaluation of generalization for our best model: VGG16 with Mediapipe skeleton image input

To validate the robustness of the Mediapipe+VGG16 model on a limited dataset, we applied 5-fold cross-validation. The results (\(93.55 \pm 0.94\)) summarized in Table 15 were slightly lower than the base model accuracy of 96.09%, as expected due to evaluation across multiple, diverse data splits. The low standard deviation demonstrates consistent performance across folds, confirming the model’s reliability and strong generalization capability.

To comprehensively evaluate the generalization capability of the Mediapipe+VGG16 model, we conducted experiments using a custom dataset³⁶ specifically designed to simulate real-world scenarios. This dataset was constructed by capturing 20 images per yoga pose class collected from YouTube videos, which introduced significant diversity in pose presentation and background elements. These variations were intended to test the model’s ability to perform accurately under conditions that differ substantially from the controlled environment of the original dataset.

The results demonstrated the model’s effectiveness in generalizing to previously unseen data, achieving an accuracy of 93.75%, a precision of 94.41%, a recall of 93.75%, and an F1-score of 93.78%. These metrics highlight the model’s robustness, as it maintained high performance despite the added complexity and variability in the test set. However, a slight performance drop was observed when compared to the scores obtained on the original dataset, as detailed in Table 16. Despite this, the model’s scores remained strong, indicating that it can adapt effectively to data drawn from real-world sources. Overall, This capability to generalize well beyond the training conditions enhances the model’s reliability and broadens its applicability in real-world settings.

Comparison between state-of-the-art approaches for yoga pose classification

The comparison of the results by the existing state-of-the-art approaches is summarized in Table 17. The proposed approach comprehensively addresses key challenges such as low-resolution datasets, class imbalance, overlapping class features, and evaluation using both skeletonized and non-skeletonized images. It offers a robust evaluation framework and demonstrates scalability by effectively classifying 16 different yoga poses.

By contrast, previous works are only partially solved with respect to the mentioned challenges: for example¹⁹, and²⁴ solve overlapping class features and use non-skeletonized images but fail to address other significant challenges such as low-resolution datasets or class imbalance. Moreover¹⁹, supports more classes – 82-while²⁴ only handles six classes. Similarly²⁵, and³² provide balanced solutions by addressing low-resolution datasets, class imbalance, and overlapping features, yet they do not incorporate skeletonized images into their methodologies.

Works like²⁰ and²⁷ lean more towards skeletonized image processing but fail to overcome challenges such as low-resolution datasets and overlapping class features. Also²⁸, and³⁰ have partial solutions for some of the above challenges- low-resolution datasets or overlap in features–but lack a holistic evaluation approach.

The proposed approach outperforms existing methods by addressing key challenges comprehensively and integrating skeletonized and non-skeletonized representations. This dual strategy ensures robust, accurate yoga pose classification, offering a scalable and reliable solution for real-world applications.

Critical analysis and feature visualization of the best performing model

Critical insights into the feature extraction process of the VGG16 model when classifying poses using Mediapipe skeleton input are brought forth by Grad-CAM visualization in Fig. 11. Grad-CAM, which stands for Gradient-weighted Class Activation Mapping, is a technique for visualizing what parts of an input image are most responsible for a model’s classification decisions. In this case, each subfigure represents a distinct yoga pose class, with the highlighted regions (in warmer colors) representing the key areas that the model looks at when making classification decisions.

MediaPipe-based skeletons likely outperform YOLOv8-Pose because BlazePose detects 33 keypoints with higher stability and consistency across joint locations, providing more precise skeletal representations that improve VGG16’s ability to discriminate subtle pose variations. Beyond accuracy, MediaPipe offers several advantages: it is lightweight and optimized for real-time inference, making it highly suitable for mobile and embedded devices; it provides anatomically consistent landmark detection across a wide range of body orientations and occlusions; and it includes built-in smoothing and temporal filtering mechanisms, which reduce jitter and noise in skeleton keypoints during dynamic poses. These strengths not only enhance pose classification but also increase robustness in real-world applications such as digital fitness or rehabilitation systems.

For very different poses like Downward Facing Dog Pose and Warrior II Pose, the model puts strong emphasis on alignments of limbs and angles of joints. The highlighted regions are the locations of important skeletal features, for example, the torso and extended arms that are critical to distinguish between these poses. In the Goddess Pose, the model has realized the symmetrical position of the arms and legs, which is unique to this pose, so it gives itself a confident classification. The visualizations show that the model consistently attends to the most relevant skeletal features for each pose. For example, poses involving intricate configurations of the legs or arms, like the Tree Pose and Lord of the Dance Pose, show focused attention around the corresponding joints and limb configurations. The accuracy of the spotlighted regions suggests that the Mediapipe skeleton input does indeed eliminate noise to a large extent, allowing the VGG16 model to focus on the discriminative features.

The Grad-CAM maps illustrate misclassification prevention, showing the robustness of the model in handling difficult poses like the Side Plank Pose and the Wide-Angle Seated Forward Bend Pose, where overlapping limbs would be problematic for a less skilled model. Through focusing on unique skeletal patterns, the VGG16 model maintains high levels of precision and recall while drastically lowering errors.

The correctly classified poses shown in Fig. 12 highlight the model’s ability in accurately identifying unique skeletal features associated with poses such as the Goddess Pose and the Lord of the Dance Pose. These types of poses are characterized by marked skeletal structures that are very unique, hence allowing the VGG16 model to distinguish one from another based on these clear contrasts. Furthermore, the incorporation of Mediapipe’s skeletal input presumably facilitated feature extraction, thereby augmenting the model’s resilience in these instances.

On the other side, the misclassifications portrayed in Fig. 13 are owed to the plain similarities of skeletal structures happening between actual and predicted poses. For example, the misclassification of the Dolphin Plank Pose as the Wide-Angle Seated Forward Bend Pose may indicate the model was misled by either overlapping skeletal keypoints or ambiguous joint angles in its input representation. It can also be that there is not enough discrimination in training data between the two poses–Dolphin Plank and Fish–or such discrimination is not learned, hence the model is ambiguous. These errors underline the difficulty of discrimination of poses with subtle or overlapping skeletal features, which suggests that further refinement of the dataset or model architecture is necessary to improve the sensitivity specific to each pose.