Proposed architecture
The proposed architecture consists of four steps. 1) Data Aggregation, 2) “Human-like Classifier”, 3) “Classifier”, 4) “Deep Classifier”, detailed in the next section.
Data aggregation
The dataset is divided into three subsets: training, validation and testing. In a training set, data points belonging to intermediate classes are aggregated into one class. That is, the order result is converted from N-class output to three-class output, as follows:
Class “L” (low): A data point with a result that originally belongs to class 1.
Class “I” (intermediate): A data point with results that originally belong to classes 2 to N-1.
Class “H” (high): A data point with a result that originally belongs to class n.
For example, a five class output data set would be three class output. Class L for class 1 points is class I points in class 1, points in class 2, 3, and 4, and class 5 points.
Classifier 1: “Human-like classifier”
As mentioned earlier, most classification errors occur at points belonging to the middle class, so the first step is to separate these difficult points from other “easy to classify” points. This was done using “classifier 1” built and trained using a new training set with three classes of aggregate results. This classifier uses a human-like approach by learning how to group individuals into three categories: “L” (low probability), “I” (intermediate probability), and “H” (high probability). Additionally, most prediction errors contain misclassifications within intermediate classes, all aggregated into one class, so this classifier can be accurately categorized into one of three classes L, I, or H.
At the end of this step, this decision was considered the final of the points classified into classes L and H. For points classified as class I, more classifiers were needed to determine which original intermediate class they belonged to.
Classifier 2: “Classifier Separation”
Following a final confirmation of the determination of points belonging to classes L and H, the goal of all subsequent steps was to determine that the points predicted to belong to class I were the exact original intermediate classes that belonged prior to aggregation. To that end, the second classifier was constructed using the “One vs. Rest” strategy and trained at points belonging to the intermediate class from the original training set. By using this classifier, you remove the first and last classes, reduce the number of classes in the output, and simplify the classification task by reducing the mathematical complexity of the model.
Once classifier 2 was trained and ready for implementation, all points predicted by classifier 1 as belonging to class I was passed to classifier 2, predicting the original intermediate class. To avoid data leakage, classifier 2 performance was measured using a validation set instead of a test set.
Deep Classifier
Given that intermediate classes are often the most challenging task in these types of problems, these classes became the subject of deeper layers of classifiers. This should reduce the class of output and give new classifiers the opportunity to become “experts” in some specific classes. Therefore, to determine which classes to delete, we measured the performance of classifier 2 for each class in the output of the validation set, and for the two classes with the highest accuracy, the decision was considered final and therefore the class was deleted. In this way, classifier 3 (the first deep classifier) can be constructed and trained using the remaining classes. This process is repeated until the intermediate class remains. If the number of classes in the output is odd, you need to delete one class when three classes are still there to make sure that the final classifier has two classes left for training. The total number of deep classifiers obtained is \(\frac {n} {2} -2 \) when \(n \) It is even number, \(\frac {n+1} {2} -2 \) when \(n \) It's strange \(n \) The number of classes in the output. For example, if you use classifier 2 to separate classes 2, 3, 4, and 5 and their performance (measured using a validation set), these classes have 70%, 65%, 75%, and 80% accuracy, and the decisions for the two classes are the best accuracy (in this example, classes 4 and 5) are constructed using the final class.
Overall Architecture Overview
To actually understand the entire architecture, here is an example of how to predict the output of five classes. First, the data set is split into training (80%), validation (10%), and test (10%) sets. Next, in the training set, all intermediate classes (classes 2, 3, and 4) are aggregated into one class (i). At the end of this step, we get three classes: 3 (originally class 1), I (originally class 2, 3, and 4), and H (originally class 5). Next, the “human-like classifier” classifies the points into one of three new classes. (1) If the point is classified as class L or H, the decision is considered final. (2) If a point is classified as Class I, this point “determines exactly to separate classifier or classifier 2 to determine exactly which class it belongs to, and to measure the performance of each class's “separate classifier”. They were trained in the remaining two classes, and their decisions are considered final, and the classification task illustrates these steps.
Flowchart showing the proposed architecture
Testing the proposed architecture
To test the effectiveness of the proposed architecture, we used public health datasets collected through a global online survey of maternal and neonatal health providers [7]. The results of the proposed architecture were compared with results from classic architectures (single 5 class classifiers) that were constructed and trained using the same dataset.
Dataset Description
This paper uses secondary data from the Global Online Survey distributed between March 2020 and July 2020 to target maternal and neonatal healthcare providers. The study is available in 12 languages and was published online using Kobotoolbox. This consisted of a mix of closure and open-ended questions about healthcare provider background information, the characteristics of the facilities they worked for, the level of preparation for the Covid-19 pandemic, and experience throughout the early stages of the pandemic. The complete survey is available on the MATCO Study website [8]and a more detailed description of the respondents and various features used are available in previously published works. [9]. The model's output (predicted dependent variable or variable) was the perception of respondents who felt protected from COVID-19 infection at work at the time of survey. This was collected on a 5-point Likert scale. 1) Not protected at all, 2) Minimum protection, 3) Some protection, 4) Fully protected, 5) Fully protected. This dataset was selected because it matches the problem the proposed architecture is trying to solve. Likert scale naturally creates ordered results in multiple classes, and the results (feeling of protection) reflect real-world scenarios that are difficult to quantify and distinguish between subtle differences in subjective emotions, thereby providing a robust testbed to the proposed architecture.
Machine Learning Models
To address class imbalances (different percentages of classes of output in the training set), training data were augmented using the synthetic minority oversampling technique (SMOTE) of the imbalanced learn library. [10]. The feature was then properly encoded based on its type: 1 Hot Encoding [11] For order encoding of category features and order features. Numerical features were standardized using the SCIKIT-LEARN standard scaler (functions centered around the mean with unit standard deviation) to allow for faster convergence of the model. A 10x cross-validation was used to ensure internal validity of the results. Hyperparameter tuning was performed using grid search. Seven algorithms were used in addition to traditional statistical methods (logistic regression) to construct the model. The algorithms used were SVM, K-Nearest Neighbor (KNN), Decision Tree (DT), Random Forest (RF), XGBoost, CatBoost, and Artificial Neural Networks (ANN). These algorithms were chosen for their high predictive capabilities in general health care environments and in particular public health and mental health applications. [9, 12,13,14,15,16]. Depending on the type, these architectures are popular for medical applications for several reasons. For example, some algorithms, such as decision trees, are easy to interpret and visualize, allowing you to gain insight into the decision process. Other algorithms such as SVM and ANN are designed to handle complex, nonlinear, high-dimensional relationships between predictors, making them suitable for medical tasks. Finally, ensemble methods such as Random Forest and boost methods such as Xgboost and CatBoost are known for their scalability for large data sets. This is important for medical research that includes extensive patient data. [17].
The model was first constructed and trained using traditional architectures (a single five-class classifier), and then compared the results between the two architectures using the new proposed architecture. All models were developed using Spyder, an open source integrated development environment (IDE) for Python, and trained using Apple's M3 Pro chip with an 11-core CPU, 14-core GPU, 18GB integrated memory, and a 512 TB SSD.
Test the generalizability of the model
Two additional validation experiments were conducted to ensure that the optimization was not tailored to previous issues only. In the first experiment, we used the same data set, but with different outputs. This is respondents' perceptions of feelings rated by the community during the Covid-19 pandemic. The output was a 5-point Likert scale output ranging from “Nothing” (Class 1) to “High Value” (Class 5). In the second experiment, the architecture was tested on a completely different dataset: “Wine Quality Dataset” [18]. In this dataset, 11 physical and chemical properties of approximately 1600 wine samples were collected, and sample quality was assessed on a scale of 1 to 10. The task here was to build a model that could accurately predict wine quality based on physicochemical properties.
The same pre-processing procedure used in the previous experiment was applied to two validation experiments, and the same ML algorithm was implemented. For each dataset, the model was constructed and trained using traditional architectures (single N class classifiers) and trained using newly proposed architectures. As a final step, we compared the results for each dataset to assess the performance of the new architecture.
