Fatigue and stamina prediction of athletic person on track using thermal facial biomarkers and optimized machine learning algorithm

Image acquisition

A team of 5 members are engaged in collecting facial image of sports persons both on the ground and resting time using thermal camera. The participants (Sports players) are briefed about our research work and the images are collected after getting appropriate ethical consent from the sports players. The running track length is about 400 m. The sports persons are at age group ranging between 20 to 30 years. The field study was carried out by the sports co-coordinator of “Jaya Sakthi Engineering College, Chennai-602024” and all the experiments were done according to the relevant guidelines and protocols.

The extraction of facial features of sportsperson is challenging due to the introduction of annoying noise signals, environmental lighting conditions and temperature fluctuations. However, thermal cameras are capable of capturing the sportsperson’s face irrespective of environmental lighting condition. The facial images are acquired using HIKMICRO Mini2 USB Thermal Camera with 256 × 192 IR resolution, 25 Hz reframe rate with 50°wide angle. This thermal camera is capable of capturing image with temperature range of − 4°F to 622°F. The thermal camera has flexible measurement setting such as 3-Dimensions presets with Automatic Center Spot, Hot Spot and Cold Spot recognition. This thermal camera has flexible 15 color palettes for predicting even the minute facial temperature variations. In this proposed model, iron bow palette is selected for image acquisition task and its justification is shown in Fig. 1a.

During the outdoor acquisition, HIKMICRO Mini2 USB Thermal Camera was mounted on a stabilized tripod at a fixed height (1.4 m) and downward tilted angle (10°-15°) ensuring the standardized view point (Fig. 1b). The distance between the mounted camera and the players were kept at a distance of 0.8 to 1.2 m for optimal field of view. This controlled setup facilitates the accurate ROI extraction and tracking for classifying fatigue, stamina and pain. The proposed Thermal Facial Sports Person (TFSP) dataset consist of 500 thermal images of female sports persons and 500 thermal images of male sports person. Among the 500 images, 250 facial images are captured at running state and remaining 250 images at rest state for both male and female players. Nearly 70% of the images are used for training and remaining 30% is used for testing the proposed classification model.

The overall methodology of the proposed architecture is shown in Fig. 1c respectively. After the image acquisition process, the images are pre-processed using proposed HEOP algorithm for improving its pixel clarity. The facial biomarkers (cheeks, lips, eyes, tongue, jaws and mouth) of the sports person are analyzed and segmented for classification. The proposed ECOC-MCSVM classification model predicts whether the player’s condition is fatigue, pain or stamina. The model is further optimized using the proposed BO-MPR model.

HEOP—preprocessing of facial thermal images

The proposed Histogram Equalized Order statistics and Power law function (HEOP) pre-processing method enhances the performance of proposed ECOC-MCSVM model by minimizing noise, generalizing unseen data and avoids over fitting. HEOP improves the contrast of the image and redistributes the pixel intensities equally. The proposed HEOP based preprocessing method consist of cascaded filters such as CLAHE, order static filter and Power law transform. The intensities of each pixel of thermal image are calculated as in Eq. (1).

Where, \(H(i)\) is the number of pixels with intensity \((i)\), intensity level of pixel co-ordinates \(I\left(x,y\right)\) of thermal image and Kronecker delta function \(\delta\) is the constant value ‘0’ or ‘1’. The acquired thermal image has temporal variations due to the weather conditions such as sunlight exposure, rain, and technical artifacts (sensor and calibration noise). HEOP removes uneven illumination in the image and preserves the local details. The output for HEOP pre-processed images at each stage is tabulated in Table 2 below. Its performance is validated by Peak to Signal Noise Ratio (PSNR) and Mean Squared Error (MSE). The proposed HEOP algorithm is compared with traditional Mean filter. It is witnessed that the proposed system removes the noisy pixels effectively from thermal image, when comparing the traditional mean filters. The workflow of the proposed HEOP pre-processing method is explained with its pseudo code in Table 3.

Histogram Equalization (HE) redistributes the pixel intensity of thermal image and enhances its clarity. The order statistics filter removes the noisy pixels and results in the smooth image with fine details. The kernel size (k) is chosen 3 × 3 as it reduces noise without degrading the edges and its empirical validation is given in Table 4. The power law function (\(\upgamma )\) is used to control the brightness of the thermal image and suppresses the background information. The gamma (\(\upgamma )\) value is tuned (0.4–1) to find the optimal value for enhancing the image quality. From the observation, the gamma (\(\upgamma )\) is chosen as ‘0.8’ because it highlights the darker region without degrading its pixel quality.

The iron bar palette of HIKMICRO thermal camera has the capacity to function at both day and night environment irrespective environmental hindrance. On hot weather, the facial temperature of thermal image is uniformly warm sometimes and makes it difficult to distinguish thermal facial biomarkers. However, this problem is addressed by the proposed HEOP model that enhances the pixel intensities of thermal image through cascaded filters. This filter removes the salt and pepper noise from the acquired thermal image without losing its finer details. The darker regions of thermal images are enhanced without over exposing using the power law transform method. The gamma (\(\upgamma )\) value adjusts the brightness and contrast of thermal image and suppress the background highlighting the facial features of sports person. This proposed methodology prioritizes the enhancement of both global and local contrast of the facial thermal image and its empirical validation is shown in Table 5.

Block processing based temperature detection (BPTD) for facial biomarkers

In the Block Processing based Temperature Detection (BPTD) algorithm, HEOP based preprocessed thermal image (256 × 192 pixels) is divided into small 12 blocks (64 × 64 pixels) as shown in Fig. 2. In this method, image matrix is in form of square blocks and corresponding operations are performed by traversing each individual blocks. These blocks cover the entire thermal image without any overlaps. The temperature of facial features is analyzed in each block of image using corresponding biomarkers. Sportsperson’s eyes, lips, teeth, jaws and nose images are selected as the facial biomarker parameters and corresponding temperature is measured. The temperature for eyes approximately lies between the range 30–38 °C during normal condition and corresponding histogram is generated. Similarly the temperature ranges of all the biomarkers are analyzed for predicting the sportsperson’s stamina, fatigue and pain condition. The BPTD method isolates the facial ROI by segmenting the thermal image into 12 thermal blocks. Each block is of resolution 64 × 64 pixels and it used for analyzing the temperature distribution patterns in facial regions. This BPTD examines the temperature variations in each blocks rather than focusing the entire image. Therefore it distinguished the minute temperature variations in thermal facial biomarkers such as eyes, lips, teeth, jaws, tongue and nose. Outdoor images are often affected by the sunlight reflections and it results in the noisy image. This problem is addressed by the proposed HEOP pre-processing model and BPTD as it focuses only on the blocks with human temperature range avoiding the irrelevant regions. The traditional methods using RGB images rely on the visual features of the images like contours and edges. The BPTD model resides on the thermal gradients instead of making it robust to occlusion and lighting conditions. The inter-subject variability of the sportspersons is handled by both the hardware setup and normalization techniques. The thermal facial biomarkers are identified using their respective temperature gradients. The thermal face is geometrically aligned by predicting the regions with consistent thermal intensity patterns. The corresponding ROI of facial thermal images are transformed into common spatial co-ordinate by subjecting to affine transformation process such as rotation, scaling and translation. Then the extracted ROI is resized into fixed input size of 64 × 64 using bilinear interpolation for classification.

The facial temperature of sportsperson is monitored, while playing in the field and resting time irrespective of lighting conditions. The players from three different sports such as running, cricket and hockey are analyzed for stamina, fatigue and pain conditions. The facial temperature variations of three different sport persons at two conditions (playing and resting time) are depicted in Fig. 3.

The facial biomarkers are selected based on the corresponding protective sports equipment (PSE) and show in Table 6. In cricket, players wear helmet, so thermal image block comprising eyes and tongue are selected for stamina, fatigue and pain measurement. Similarly, for hockey player, the jaws, teeth and tongue are selected because hockey players will be running continuously, so the eyes and cheeks can’t be visualized properly. For athletes, eyes, nose, teeth, jaws and mouth thermal image blocks are considered for stamina, fatigue and pain measurement. During running, the temperature is high in facial regions and fatigue is measured from the eyes, nose, teeth, jaws, and mouth blocks with average temperature of 38 °C. Similarly, pain is measured from mouth, eyes, and jaws blocks, with average temperature of 36 °C. Stamina is measured from cheek, lip and jaw blocks, with average temperature of 33 °C. It is observed that the persons on track during running have higher temperature, when comparing the players at resting time. While considering the cricket players, the temperature of fatigue (37 °C) is as high as pain (36.5 °C).The cricket players with stamina has moderate temperature at resting time (34 °C). During playing cricket, the facial temperature of player is at range of 37 °C. For hockey player, temperature range is high during stamina, fatigue and pain condition, because the hockey players are running entire game. The temperature for the players, off the field is somewhat low when comparing on field for all players such as cricket, hockey and running. From Fig. 3, it is witnessed that irrespective of sports, the persons with stamina has facial temperature of range (31–36 °C), fatigue (32–37 °C) and pain (33–38 °C) respectively. The above measured temperature values are fed to the proposed ECOC-MCSVM algorithm for classifying stamina, fatigue, and pain of sportspersons.

Proposed ECOC-MCSVM based classification

The BPTD extracted temperature features are trained in the proposed Error-Correcting Output Codes based Multi-class Support Vector Machine (ECOC-MCSVM) model for classification of stamina, fatigue, and pain. This hybrid model encodes the unique arbitrary value for each class to distinguish it from other classes. The proposed method classifies three health parameter (stamina, fatigue, and pain) based on facial expressions of sportsperson. The proposed MCSVM is multiclass classifier, yet it handles the multiclass problems into set of binary SVM classifiers. The detailed flow of the proposed ECOC- MCSVM classification model is shown in Fig. 4. In the proposed ECOC- MCSVM model, 70% of 500 thermal images are used for training the model, while 30% of images are used for testing. Initially the raw thermal images are pre-processed using HEOP technique, and then BPTD is applied for temperature features extraction.

The extracted features are trained in the proposed model for classification with sufficient data so that it doesn’t falls into the overfitting or underfitting. A good classification model should have balance between its bias and variation so the training and testing datasets are divided in proper ratio. In this proposed ECOC- MCSVM model, K fold cross validation is used to divide dataset into k-folds, where ‘k-1’ folds are used for training and ‘1’ fold for validation. In this proposed ECOC- MCSVM model the value for k is assigned as ‘5’ and their performance is measured in terms of accuracy. Table 7 compares the athletic person’s pain using proposed facial biomarkers across the manual measurement of pain using Graphic Rating Scale (GRS). The pain of sportsperson is measured based on the athlete’s feedback and the rating scale is between zeros to eight. Likewise, Table 8 compares the athletic person’s stamina using Stamina model. Similarly, Table 9 compares the fatigue condition of sportsperson across the proposed and manual measurement using Fatigue Assessment Scale (FAS). Bayesian Optimized- Multiple Polynomial Regression analysis (BO-MPR) predicts the relationship between the stamina, fatigue and pain conditions of the sportsperson using the linear equations as in Eq. 2.

Here,\(Y\) is the dependent variable, \({X}_{1},{X}_{2},{\dots ,X}_{n}\) are the independent variables, \({\beta }_{0},{\beta }_{1,}{\beta }_{2},{\beta }_{n}\) are the coefficients and \(\in\) is the error term. This proposed BO-MPR optimized model predicts the health status of the sportsperson using the polynomial equations. The corresponding facial biomarkers are selected as the independent variables to calculate the dependent variables such as stamina, fatigue and pain. The proposed BO-MPR polynomial model analyzes the facial features of the sportsperson and predicts the stamina, fatigue and pain condition as in Table 10. The significance of the co-efficient in polynomial regression is estimated using the parameters such as ’s-statistics’ and ‘p- statistics’ values . An effective classification model has low p-value and high t-statistics value to reduce the error rate. The proposed model has higher t-statistics value and low p-value (< 0.05). Hence, it is witnessed that the proposed model identifies the stamina, fatigue and pain of the sportsperson precisely. Based on the generated coefficients (\({\beta }_{0},{\beta }_{1,}{\beta }_{2},{\beta }_{n}\)) the corresponding histograms and fitting plots for pain is shown in Fig. 5, stamina is shown in Fig. 6 and fatigue is shown in Fig. 7.

The player’s facial features such as eyes, nose, lips and tongue are identified and analyzed using the histogram and normal probability distribution plot. The features of the players suffering from pain condition are observed and its corresponding residuals are shown in Fig. 5a and b. The proposed ECOC-MCSVM and BO-MPR model compare the actual trained features and the predicted features to evaluate the performance of the proposed model. Similarly, stamina and fatigue condition of the players are also identified using the histograms and normal probability distribution as shown in Figs. 6a,b, 7a and b respectively. The red diagonal line in Figs. 5b, 6b and 7b is ideal normal distribution and the blue points represent the predicted residuals. If the blue points fall near the red line then it shows that the proposed models classify the player’s condition efficiently. In case, if it deviates from the red line then it indicates the presents of outliers and the model needs more training and pre-processing. Since our proposed model has effective pre-processing and normalization techniques the predicted residuals lies close to the red line. From the histogram and normal probability plot analysis, it is witnessed that the overall prediction rate of the proposed BO-MPR model is good for stamina, fatigue and pain condition. The data points are scattered near the fitting line, which is a sign of a good trained model with higher accuracy rate.