Clinical decision support for vestibular diagnosis: Large-scale machine learning coaching with living experience

As shown in Figure 4, the overall development process involves several important stages of data processing, functional evaluation, ML model development, and performance evaluation. All processes were developed using Python (Python Software Foundation, Delaware).

The feature selection process consisted of determining the most relevant areas of the clinical record questionnaire. The 145 field analysis was performed in two parts. First, two feature ranking algorithms were implemented to assess the importance of features in terms of diagnostic results. Second, two vestibular experts selected features from clinical record surveys that they considered most relevant when making a diagnosis.

Algorithm Analysis

Two feature selection algorithms were used for data analysis. Select recursive elimination using a Support Vector Machine (RFE-SVM) to capture complex feature interactions, and Select K Best (SKB) scores to independently assess the importance of individual features. This hybrid approach combines the nonlinear relationships of RFE-SVMs with the computational efficiency of SKBs and the ability to identify univariate correlations.^21,22. The RFE-SVM method is a feature selection technique that recursively eliminates features and uses the remaining features to build a model. By analyzing the performance of the model, we assess the importance of each feature and identify the most important features for predicting target attributes.^{twenty one}. In RFE-SVM, the procedure starts by training a Support Vector Machine (SVM) classifier with all available features and calculates the ranking criteria for each feature. This criterion is usually derived from the weight vector of the SVM. The process is repeated until the minimum ranking criteria are removed and the predefined number of features remain. RFE-SVM ranking criteria are determined by solving the weight vector optimization problem w and bias bsubject to the condition that all data points are correctly classified as expressed in the following formula:

where w It's a weight vector, x_I is the input vector of I-th example of training, y_I Corresponding labels n Represents the total number of training examples^{twenty three}. RFE-SVM has successfully analyzed medical data, including distinguishing between different neurological conditions and diagnosing vestibular disorders.^24,25.

Meanwhile, SKB scores, also known as chi-square tests, are another feature selection technique that evaluates the dependencies of features and target variables. Calculate chi-square statistics for each feature and target, and help to identify which features are most beneficial in relation to the outcome variable^26,27. Specifically, chi-square statistics for functions \(f \) and target variables \(y \) It is calculated as:

where \({o} _{{ij}} \) The frequency of observation of feature f with a value \(I\) Target variable Y with value \(j \)and \({e} _{{ij}} \) The expected frequency of the feature \(f \) With value \(I\) and target variables \(y \) With value \(j \) Under the hypothesis of wasteful independence.

Expected frequency \({e} _{{ij}} \) It is calculated as:

where \({count} \left(f = i \right)\) The number of instances with the feature \(f \) It's worth it \(I\), \({count}(y = j)\) The number of instances with target variables \(y \) It's worth it \(j \)n is the total number of instances.

SKB has proven valuable in identifying the most relevant features, especially in text classification.²⁸ and biomedical data analysis²⁹the ability to capture important relationships between variables improves classification performance.

After applying these two algorithms, we selected the top 30 fields common to both methods, allowing us to identify the most important and relevant features of the classification task. This combined approach allows for the strength of both RFE-SVM and SKB to enhance the function selection process.

Selection of experience-based hybrid features

In addition to selecting algorithmic features, two vestibular experts identified 20 additional features that are considered important in making accurate diagnosis. Their choice was informed by both their clinical experience and established frameworks such as ICVD³. ICVD is currently the most widely recognized system for classifying vestibular disorders and offers a standardized approach to diagnosis and management. Important features were selected within the standardized ICVD based on living experience. These 20 features served as minimal clinical guidance for teaching ML models, and prevented the exclusion of important features in the actual settings. For example, the ML model may consider a patient's response to a question about dizziness frequency as just one feature with weights equal to the other frequencies. However, experienced vestibular experts recognize it as an important feature in making a correct diagnosis, as it provides important insight into the severity and progression of the disorder.³⁰. This highlights the need to integrate domain expertise into functional selection, ensuring that clinically important questionnaire-based features are not underestimated or overlooked by purely algorithmic approaches.

ML model development

For the ML model, the complete dataset consisting of 3349 records was split into training and test sessions. The training session used stratification and 5x cross-validation during model development to account for 80% of the data, ensuring a balanced class percentage and a robust performance assessment of the overall validation times. The remaining 20% of the data was assigned to the test session and served as invisible test data to assess the performance of the model.

Training sessions were utilized for training and optimization of three different ML models: CatBoost, Random Forest, and Xgboost. These models were selected for their proven effectiveness in a variety of classification tasks and their ability to process complex, higher dimensions of data.³¹.

CatBoost is a gradient boost algorithm that utilizes ordering and categorical feature encoding to improve performance³². Minimize the following objective functions:

where \(l\left({y}_{i}, f\left({x}_{i}\right)\right)\) is the loss function of IInstances, and \(\ omega(f)\) It is a normalization term.

Random Forest is an ensemble learning method that builds multiple decision trees and combines predictions to final classify.³³. Each decision tree is constructed using a random subset of features and instances to help reduce overfitting and improve generalization. The final prediction is obtained by aggregating the predictions of all decision trees.

where \({repair)\) It is a prediction of IThe decision tree, and \(n \) The total number of forests.

XgBoost is another gradient boost algorithm that optimizes the following objective functions:

where \(l\left({y}_{i}, \widehat {{y}_{i}}\right)\) is the loss function of IInstance, \(\widehat {{y} _{i}} \) It's the predicted value \(\omega \left({f}_{i}\right)\) This is the normalization term for ITH decision tree³⁴.

An optimization algorithm was implemented to find the best hyperparameters for each model. Methods such as grid search, random search, and Bayesian optimization are adopted to efficiently explore hyperparameter spaces and identify the most promising configurations.³⁵.

These models were trained using selected features and curated datasets, and performance was optimized using validation sets for hyperparameter tuning. The validation set allowed us to fine-tune the hyperparameters of the model and assess the performance of invisible data during the training process.

To assess the effectiveness of the vestibular disorder classification, results were divided into three groups: “correct”, “partially correct” and “fraud.” The “correct” category indicates that the initial predictions from the developed model match the first impressions of vestibular experts. The “partially correct” category indicates when the second prediction from the developed model corresponds to the first impression of a vestibular expert, or when the second impression of a vestibular expert matches either the first or second prediction from the developed model. This classification was operated through a systematic rule-based algorithm that was consistently applied to all test cases without subjective clinical judgment, ensuring a reproducible assessment of results and eliminating potential biases. Finally, the “wrong” category indicates when the algorithm does not match the first and second diagnosis of the vestibular expert. A similar approach has been used in previous studies assessing multiple categories of outcomes.^36,37.

Another ongoing pilot usability survey has been conducted to assess the practical feasibility of conducting diagnostic questionnaires in a clinical setting.

metric

Several metrics were utilized to assess the performance of the ML model, including confusion matrix, accuracy, specificity, and sensitivity.^38,39. The confusion matrix provides a tabular summary of model classification performance and displays counts of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions³⁹. In our assessment, both “correct” and “partially correct” predictions are considered true positives (TPs) as they are perfectly or partially consistent with the physician's impression. A “false” prediction is considered false negative (FN) or false positive (FP) depending on the actual diagnosis. Accuracy measures the overall accuracy of the model's predictions and is calculated as follows:

The singularity, also known as the true negative rate, quantifies the ability of the model to correctly identify negative instances and is calculated as follows:

Sensitivity, or true positive rate, evaluates the ability of the model to correctly identify positive instances and is calculated as follows:

These metrics collectively provide a comprehensive assessment of the performance of a model, allowing informed decisions about model selection and improvements⁴⁰.