We chose IPAT based on its reliability and clear potential for automation. The IPAT assesses six anatomical features: head and neck alignment, trunk alignment, arm position, leg position, global symmetry, and leg anatomical position, each with a maximum score of 12 points from 0 (poor) to 2 (optimal). A score of 0 to 8 indicates suboptimal positioning requiring intervention, 9 to 11 indicates acceptable positioning, and 12 indicates ideal positioning. Additionally, five different body alignments are classified (supine, prone, right lateral, left lateral, and semilateral (approximately 45° between supine and lateral)). To be able to assess detailed body part relationships, a custom body model was developed with 23 anatomical key points marked for each image.
A total of six annotators, including clinical experts and trained non-experts, participated in image scoring and keypoint labeling. Annotators underwent standardized training consisting of on-site structured instructional sessions, practice with reference subsets, consistent application of scoring criteria, and supervised practice rounds to ensure inter-rater reliability. Final interrater agreement after rater training and evaluation in a 156-case cohort resulted in a Fleiss rating. κ 0.69 for suboptimal positioning and 0.79 for body alignment. The total training load was approximately 5-8 hours per annotator over 4-5 sessions. Based on the observed annotation agreement, the distribution of standing cases, and the average class imbalance ratio across the IPAT scores of the test cohort of approximately 1:4, we estimated the minimum sample required if the reliability metric varied by 5% or less. n>1340.10 The annotated cohort data were integrated into the study dataset.
Study design and setting
This retrospective observational study was conducted from June 2023 to February 2025 in two physically separated level III/IV NICUs within the same medical institution at Semmelweis University, Hungary. The recording and study were approved by the Semmelweis University Research Ethics Board (SE-RKEB: 265/2022). Continuous audio/video recording was performed using a proprietary camera system. The system consists of an off-the-shelf camera and single-board computer components in a custom enclosure designed for use in closed incubators and features active infrared illumination for 24-hour recording in a variety of lighting conditions, including complete darkness. Videos were recorded at a resolution of 1280 × 720 pixels and 25 frames/s. All procedures were performed in accordance with the Declaration of Helsinki. Data processing complied with the requirements of the EU General Data Protection Regulation.
Eligibility and consent
Preterm infants (gestational age at birth less than 37 weeks) admitted to participating NICUs were included. Parents/legal guardians provided written informed consent for video recording and data analysis prior to camera installation. Exclusions were applied if mechanical ventilation, hemodynamic/vasopressor support, centrally acting drugs (opioid analgesics, sedatives), or presence of active infection were required. Infants with a gestational age of 37 weeks or older at birth or those with insufficient clinical data for analysis were also excluded.
Selection procedure
A total of 8,400 hours of video were captured from the 88 infants enrolled during their NICU stay. Recordings were made continuously while the infant was housed in a closed incubator receiving routine clinical care. From the image pool, two complementary datasets were constructed.
-
1.
Random surveillance dataset: still images systematically extracted at 2-hour intervals from all available video recordings across enrolled infants.
-
2.
Longitudinal follow-up dataset: Four infants representing major gestational age categories (GA 24, 27, 30, and 33 weeks at birth) were purposefully selected for a 24-hour intensive observation period with image extraction every 10 minutes. Selection was made based on GA variability and length of available continuous recordings, with minimal exclusions.
Individual still images were systematically extracted from the video recordings. Images were retained for analysis only if they met all of the following technical criteria:
-
Infants in active, quiet, or intermediate sleep stages (not awake or crying) recorded to assess long-term posture.
-
More than 70% of key points on the body can be seen without obstruction
-
Infants placed in closed incubators (images during open incubator care or radiant warmer use are excluded)
-
Adequate image quality determined by automatic brightness quality score threshold
-
No ongoing nursing care activities
Images with technical recording failures, empty bed frames, poor lighting/resolution, and incomplete body visualization were excluded.
Adaptive sampling enhancements
Adaptive resampling was employed to adequately represent the full range of IPAT cumulative scores. Preliminary analysis of a random surveillance dataset identified underrepresented clinical cases (0 ≤ IPAT cumulative score ≤ 7). For infants and time periods that presented these rare cases, additional high-frequency sampling (1-10 minute intervals) was performed within ±30 minutes of the identified event.
Desired results and variables
The model was trained to predict IPAT score and posture using body keypoint coordinates as input data. The model output was evaluated for posture classification (5 categories), IPAT category score (6 features × 3 scores), cumulative score (0 to 12), and repositioning requirement (binary yes/no). All output types were one-hot encoded as categorical data. No additional patient demographic or clinical data was included in model training.
Generating body key points and features
Specifications for key points of the body were derived from Ref. 17including fingers and toes similar to the AggPose model.18 The resulting body model consisted of 23 independent keypoints. Although the inclusion of toes is justified by the IPAT scale’s anatomical leg position assessment requirements, the use of fingers allows for more accurate hand position analysis, such as detecting the position of the arm, such as a hand touching the face. A list of key points and an example body model and positioning class are shown in Figure 1.
be Illustration of the location of key points on the body used in the study. b–e Example view with body pose overlay. Body alignment is on the right side (b), prone (c), supine (d), and hemilateral (e), respectively.
Coordinates of body keypoints were normalized using the length of the largest side of the bounding box across the body.19 The feature vector is derived from the normalized coordinates as a scalar input to the classifier model. The feature vector consists of normalized coordinates and derived scalars: Lengths of body parts, distances between unconnected joints, joint angles, and angles between segments (such as the angle between two thighs). The evaluation process is preceded by feature engineering, which first incorporates a comprehensive set of features and then removes redundant and low-impact features.
predictive model
To predict pose and positioning quality from keypoints, four machine learning algorithms were selected to cover different inductive biases: Random Forest, Extreme Gradient Boost (XGBoost), Support Vector Machine (SVM), and CatBoost. These algorithms represent complementary approaches. Random Forest uses parallel ensemble learning with independent decision trees. XGBoost employs sequential boosting, where each tree corrects the errors of its predecessors. SVM uses hyperplanes in high-dimensional space to optimize margin-based separation. Tree-based methods effectively capture geometric relationships and decision thresholds in pose data. Comparing them tests whether performance on this task is method dependent or converges to an upper bound imposed by annotation noise.
The model was trained and evaluated using Python 3 and Google Compute Engine. Leave-One-Infant-Out cross-validation was used during training and validation. The cycle ran through all infants. In each cycle, the model was trained on the remaining images and its predictive performance was evaluated on selected infant images. The results of each run were finally combined using macro averaging for unbalanced multiclass classification.
Model interpretability
To understand the black-box decision-making process of the classifier, we adopted the SHapley Additive exPlanations (SHAP) method.20 We generated Shapley bee importance data to interpret the magnitude of each feature’s contribution to model output, revealing the relative importance and influence patterns of skeletal keypoint features. To increase interpretability, we projected the derived feature set onto the original anatomical keypoints of the infant body model. We illustrated their spatial distribution on a body schema, thereby relating the model to specific postural landmarks. The importance map also highlights the weight of missing key points in the decision.
performance analysis
In accordance with Clinical AI Research recommendations,twenty one The model was evaluated using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), Fleiss’s kappa, precision, recall, F1 score, and confusion matrix.
During the classification of decisions that require relocation, we observed performance differences between the following two prediction strategies: (a) Predict fine-grained IPAT component scores individually and then compare their cumulative sum to a threshold. (b) Directly predict relocation instructions as binary results. We present results for both cases to explore the impact of these architectural choices on model performance.
body model
We investigated the utility of widely used models that rely on different keypoint definitions as inputs for prediction. The first model we evaluated was the infant pose estimator AGMA-PESS, which was reported as the best performing model in a recent evaluation.twenty two The model was originally trained to perform pose estimation in a controlled laboratory setting with standard human movements, rather than the difficult incubator environment characterized by neonatal proportions, variable infrared lighting, medical device occlusions, and closed incubator reflections. The second model we evaluated was ViTPose (a giant variant of ViTPose).twenty three A vision transformer-based architecture pre-trained on a common human pose dataset. This model is the current state-of-the-art general-purpose model on the COCO dataset and is also the best-performing model for infant body modeling.14,22 The model covers fewer key points than required for evaluation, so expansion is required. Using a new output generation head of 23 keypoints, we successfully fine-tuned a pre-trained ViTPose model on our dataset, achieving an average keypoint similarity of 0.73. To fit the ViTPose model to a richer set of keypoints, we integrated these additional keypoints into the model head and fine-tuned the extraction of keypoints from the input image preprocessed by Backbone. The added coordinates include key points on the fingers and feet (corresponding to points 9, 14, 18, and 22 in Figure 1), upper chest and navel key points (points 10, 23) that complement the points on the shoulders and hips, and finally the crown and neck points. The evaluation process is shown in Figure 2.
be Input images are manually annotated with IPAT scores, posture, and body key points. b Keypoint annotations are used to enhance and fine-tune the ViTPose pose estimator. c Multiple classifier methods are trained and evaluated based on manually annotated and estimated keypoints to provide body alignment classification and repositioning instructions. The suboptimal positioning that requires intervention decisions is derived in two different ways: using threshold regression estimated cumulative IPAT scores and using binary prediction.
Called an end-to-end solution, this ViTPose model generates keypoints directly from raw incubator images. These keypoints are fed into the CatBoost classifier for training, which outputs the next best positioning metric.
In the next section, we evaluated the manually annotated model across classifier types and compared it to the fine-tuned ViTPose model to separate the contribution of the pose estimator from that of downstream classifiers.
