Estimating the strength of soil stabilized with cement and lime at optimal compaction using ensemble-based multiple machine learning

Preliminary statistics of the collected database

A total of 190 records were collected from literature for UCS test results of cohesive soil stabilized with cement and lime. Each record contains the following data¹: Cement weight ratio (C), %, Lime weight ratio (Li), %, Liquid limit (LL), %, Plasticity index (PI), %, Optimum moisture content (OMC), %, Maximum dry density (MDD), g/cm³, and Unconfined compressive strength (UCS), MPa. Data-preprocessing through data cleaning and dimensionality reduction methods was conducted to reduce the complexities in the models run. The collected records were divided into training set (140 records) and validation set (50 records). Tables 1 and 2 summarize their statistical characteristics and the Pearson correlation matrix. Finally, Fig. 1 shows the histograms for both inputs and outputs and Fig. 2 shows the relations between the inputs and the outputs.

Table 1 presents the statistical characteristics for both training and validation sets in terms of mean, range, standard deviation and variance of each parameter. The main goal of this table is to assure that both sets have the same statistical characteristics and hence, validation set could be used to monitor the learning process and stop the training before over-fitting. While Table 2 presents the correlation between each parameter, there are two aims for this table, the first is to insure that all considered parameters are independent (correlation factor < 0.9), the second is to rank the inputs according to their correlation (importance) to the output.

Figure 1, shows the histogram of each parameter, it used to insure that there is no “gaps” in the distribution of any parameters because predictive model should be trained using the considered full range of each parameter. Gaps in the distribution may lead to inaccurate predictions in these zones. The histograms in Fig. 1 showed that there is no bias in (C, LL, OMC & MDD) and although there is lift shifting in (PI & Li) but the output (UCS) still unbiased.

Finally, Fig. 2 presents the relations between the output and each input, it is another graphical alternative to the correlation matrix presented in Table 2. The fitting line in red gives a general idea about the relation (forward or revers) while the (R²) evaluates the strength of this relation.

Research program

Eight different ML classification techniques were used to predict the (UCS) of stabilized soil using the collected database. These techniques are “Gradient Boosting (GB)”, “CN2 Rule Induction (CN2)”, “Naive Bayes (NB)”, “Support vector machine (SVM), “Stochastic Gradient Descent (SGD)”, “K-Nearest Neighbors (KNN)”, “Tree Decision (Tree)” and “Random Forest (RF)”. All the eight developed models were created using used to predict (UCS) using (C), (Li), (LL), (PI), (OMC) and (MDD). All the developed models were created using “Orange Data Mining” software version 3.36. The considered data flow diagram is shown in Fig. 3. The flow began with reading the database and store it in a table, then the validation set is splatted in a separate table. Copies from the training table is introduced to the training modules of the considered techniques, graphical presentations are generated for the developed models that have ones such as CN2, Tree & RF.

Both training and validation sets are evaluated for all techniques in the node called “Test and score” the predicted values for both training and validation sets for all techniques are stored in table called “All outputs” which feeds the presentation generators. For each technique, the related outputs are selected from the “All outputs” table and stored in a separate table and the relation between predicted and experimental values is plotted.

Decision-based classifier and ensemble machine learning techniques

Gradient boosting (GB)

Gradient boosting is a popular machine learning technique used for both regression and classification tasks¹⁵. The GB framework is presented in Fig. 4. It is an ensemble method that combines multiple weak predictive models, typically decision trees, to create a strong predictive model. The basic idea behind gradient boosting is to iteratively add weak models to the ensemble, with each subsequent model aiming to correct the mistakes made by the previous models⁶. This is done by minimizing a loss function, which measures the difference between the predicted and actual values¹⁵. Here’s a step-by-step overview of how gradient boosting works: Initialize the ensemble model by creating the first weak model, often a shallow decision tree. Fit the weak model to the training data and calculate the residuals, which are the differences between the predicted and actual values. Train a new weak model on the residuals. This model is designed to predict the residuals based on the input features³. Add the new model to the ensemble by combining it with the previous models, weighted by a learning rate. The learning rate determines the contribution of each model to the final prediction. Repeat steps 2 to 4 for a specified number of iterations or until a stopping criterion is met. In each iteration, the new weak model is trained to predict the negative gradient of the loss function with respect to the ensemble’s predictions². Finally, the ensemble model is formed by combining the predictions of all the weak models, weighted by their respective learning rates. Gradient boosting has several advantages: It can handle different types of data, including categorical and numerical features. It can capture complex interactions between variables and automatically handle feature interactions. It typically produces highly accurate predictions, as the ensemble model improves with each iteration. It is less prone to overfitting compared to other algorithms like decision trees. However, gradient boosting also has some considerations: It can be computationally expensive, especially when dealing with large datasets or complex models¹⁵. It requires careful tuning of hyperparameters such as the number of iterations, learning rate, and tree depth. It may be sensitive to noisy or irrelevant features, which can lead to overfitting. Overall, gradient boosting is a powerful technique that has been successfully applied in various domains, including finance, healthcare, and natural language processing. It is implemented in popular machine learning libraries such as XGBoost, LightGBM, and CatBoost.

CN2 rule induction (CN2)

CN2 (Conceptual Clustering on Numerical Data) is a rule induction algorithm used for classification tasks¹⁵. The CN2 framework is presented in Fig. 5. It is designed to discover compact and accurate sets of classification rules from numerical data. The CN2 algorithm follows a top-down, greedy search approach to build a set of rules that collectively describe the data. Here’s a step-by-step overview of how CN2 works: Initialize an empty set of rules. Select a target class and find the best rule for that class². The best rule is the one that maximizes a quality measure, such as information gain or accuracy, when applied to the current dataset. Prune the instances covered by the best rule from the dataset. Repeat steps 2 and 3 until a stopping criterion is met. This could be a maximum rule length, a minimum number of instances covered by each rule, or a predefined number of rules. Optionally, post-process the generated rules to remove redundant or conflicting rules. The resulting set of rules can be used to classify new instances by sequentially applying each rule and assigning the instance to the target class of the first matching rule. CN2 has some key characteristics and advantages: It is well-suited for datasets with numerical attributes, as it handles them naturally. It can discover interpretable rules that describe the underlying patterns in the data. CN2 is a deterministic algorithm that produces consistent results²¹. It is relatively efficient compared to other rule induction algorithms. However, there are also some considerations with CN2: CN2 is a top-down algorithm, which means it may miss some rules that could be discovered by a bottom-up approach. It assumes that each instance can be classified by a single rule, which may not always be the case in complex datasets⁶. The performance of CN2 heavily depends on the quality of the quality measure used to select the best rule. Choosing an appropriate quality measure is crucial. The CN2 algorithm has been used successfully in various domains, including medicine, finance, and customer relationship management. It provides a transparent and interpretable way to extract knowledge from numerical data.

Naive Bayes (NB)

Naive Bayes is a popular machine learning algorithm for classification tasks. The NB framework is presented in Fig. 6. It is based on Bayes’ theorem and makes the assumption of attribute independence, which is why it is called “naive”. Despite this simplifying assumption, Naive Bayes has been proven to be effective in many real-world applications and is known for its simplicity, speed, and ability to handle large feature spaces¹⁵. Here’s an overview of how the Naive Bayes algorithm works: Data Preparation: Prepare a labeled dataset where each instance consists of a set of features (attributes) and a corresponding class label. The features can be categorical or numerical. Probability Estimation: Calculate the prior probabilities of each class, which represent the probability of encountering each class in the dataset. This is done by counting the occurrences of each class in the training set. Feature Probability Estimation: For each feature, estimate the conditional probability of that feature given each class. This is typically done differently depending on whether the feature is categorical or numerical⁶. Categorical Features: Calculate the probability of each category occurring within each class by counting the occurrences of each category in the training set. Numerical Features: For each class, estimate the mean and standard deviation of the feature values. Assuming a Gaussian distribution, use these statistics to calculate the probability density function of the feature value given the class. Classification: Given a new instance with unknown class, calculate the posterior probability of each class given the instance’s features using Bayes’ theorem. The class with the highest posterior probability is assigned as the predicted class for the instance. Naive Bayes has several advantages: It is computationally efficient and can handle large datasets and high-dimensional feature spaces. The assumption of attribute independence makes it robust and requires fewer training instances compared to other algorithms. It can handle both categorical and numerical features¹⁵. Naive Bayes is relatively less prone to overfitting, as long as the independence assumption holds reasonably well. However, there are some considerations with Naive Bayes: The assumption of attribute independence may not hold in some cases, leading to suboptimal results. It can be sensitive to the presence of irrelevant or redundant features. Naive Bayes tends to underestimate the probability of rare events due to the assumption of attribute independence. Despite these limitations, Naive Bayes is widely used in various domains, including text classification, spam filtering, sentiment analysis, and recommendation systems. It serves as a simple and effective baseline algorithm for many classification tasks.

Support vector machine (SVM)

Support Vector Machines (SVMs) are a powerful and versatile supervised learning algorithm used for classification and regression tasks²¹. The SVM framework is presented in Fig. 7. They are particularly effective in handling complex, high-dimensional datasets. The main idea behind SVMs is to find an optimal hyperplane that separates different classes in the feature space. The hyperplane is selected in such a way that it maximizes the margin, which is the distance between the hyperplane and the nearest data points of each class. The data points closest to the hyperplane are called support vectors ³. Here’s an overview of how the Support Vector Machine algorithm works for binary classification: Data Preparation: Prepare a labeled dataset where each instance consists of a set of features (attributes) and a corresponding class label. Feature Mapping: Transform the input features into a higher-dimensional space using a kernel function. This allows SVMs to find nonlinear decision boundaries in the original feature space. Hyperplane Selection: Find the optimal hyperplane that maximizes the margin between the classes. This can be done by solving an optimization problem that involves minimizing the classification error and maximizing the margin. Soft Margin Classification: In cases where the data is not perfectly separable, SVMs can allow some misclassifications by introducing a slack variable²¹. This leads to a soft margin that allows for a trade-off between maximizing the margin and minimizing the misclassification errors. Kernel Trick: SVMs leverage the kernel trick, which avoids the explicit computation of the high-dimensional feature space. Instead, it uses a kernel function to calculate the inner products between the transformed feature vectors, making the computations efficient. Classification: To classify new instances, the algorithm computes the distances between the new instances and the hyperplane. The predicted class is determined by the side of the hyperplane on which the new instance falls. SVMs have several advantages: They can handle high-dimensional feature spaces effectively. SVMs are robust against overfitting, especially when using a proper regularization parameter. The kernel trick allows SVMs to capture complex nonlinear relationships in the data. SVMs have a solid theoretical foundation and are well-studied³. However, there are some considerations with SVMs: SVMs can be computationally expensive, especially for large datasets. SVMs require proper tuning of hyperparameters, such as the choice of kernel function and regularization parameter. SVMs can be sensitive to outliers in the data. SVMs have been successfully applied in various domains, including text classification, image recognition, and bioinformatics.

Stochastic gradient descent (SGD)

Stochastic Gradient Descent (SGD) is an optimization algorithm commonly used for training machine learning models, particularly in large-scale settings⁶. The SGD framework is presented in Fig. 8. It is an iterative algorithm that updates the model parameters based on the gradients of the loss function estimated from a randomly sampled subset of the training data. Here’s an overview of how Stochastic Gradient Descent works: Data Preparation: Prepare a labeled dataset with the input features and corresponding target labels⁶. Model Initialization: Initialize the model’s parameters, such as the weights and biases, with random values or predefined values. Random Shuffling: Shuffle the training data randomly. This step is typically performed in each epoch to ensure a different ordering of the data in each iteration. Iterative Update: Iterate through the training data in small batches, known as mini-batches, rather than using the entire dataset at once. This is the “stochastic” part of SGD. Gradient Calculation: For each mini-batch, calculate the gradients of the loss function with respect to the model parameters. These gradients indicate the direction and magnitude of the parameter updates. Parameter Update: Update the model parameters using the calculated gradients¹⁵. The update is performed in the opposite direction of the gradients, scaled by a learning rate. The learning rate determines the step size taken in the parameter space. Repeat Steps 4 to 6: Iterate through the mini-batches for a specified number of epochs or until a convergence criterion is met. Each iteration updates the model parameters based on a different subset of the training data. SGD has some advantages: It is computationally efficient and memory-friendly, as it processes the data in mini-batches rather than the entire dataset. SGD can handle large-scale datasets, making it suitable for scenarios with a massive amount of training data. It can continually update the model parameters as new data becomes available, making it suitable for online learning scenarios²¹. However, there are also considerations with SGD: SGD is sensitive to the learning rate. Choosing an appropriate learning rate can be crucial for convergence and model performance. The algorithm may get stuck in suboptimal solutions or saddle points, particularly for non-convex loss functions. The random sampling of mini-batches can introduce noise, which can affect the convergence and stability of the optimization process. Variations of SGD, such as mini-batch SGD and momentum-based SGD, have been proposed to address some of the limitations and improve convergence speed. SGD is widely used as the optimization algorithm for training various machine learning models.

K-nearest neighbors (K-NN)

K-nearest neighbors (K-NN) is a popular supervised learning algorithm used for both classification and regression tasks¹⁵. The K-NN framework is presented in Fig. 9. It is a non-parametric algorithm that makes predictions based on the similarity of the training instances to the new, unseen instances. Here’s an overview of how the K-nearest neighbors algorithm works for classification: Data Preparation: Prepare a labeled dataset where each instance consists of a set of features (attributes) and a corresponding class label³. Select the Value of K: Choose a value for K, which represents the number of nearest neighbors to consider when making predictions. The value of K is typically determined through experimentation or cross-validation. Distance Calculation: Calculate the distance between the new instance and each instance in the training set. The most commonly used distance metric is Euclidean distance, but other metrics like Manhattan distance or cosine similarity can also be used. Nearest Neighbor Selection: Select the K instances with the smallest distances to the new instance. These instances are called the K nearest neighbors. Majority Voting: For classification, determine the class label of the new instance based on the class labels of its K nearest neighbors. The class label with the highest frequency among the neighbors is assigned as the predicted class label. K-NN has several characteristics: K-NN is a lazy learner, meaning it does not build an explicit model during training. Instead, it memorizes the training instances and performs calculations at prediction time. It can handle both categorical and numerical features. K-NN can capture complex decision boundaries and is capable of nonlinear classification. The choice of K affects the algorithm’s performance. Smaller values of K can lead to more flexible decision boundaries, but they are more sensitive to noise and outliers. Larger values of K provide smoother decision boundaries but may lead to over-smoothing. K-NN does not make any assumptions about the underlying data distribution, making it versatile and applicable to various domains⁶. However, there are considerations with K-NN: K-NN can be computationally expensive, especially when dealing with large datasets or high-dimensional feature spaces. Efficient data structures, such as KD-trees or ball trees, are often used to speed up the search for nearest neighbors. The algorithm is sensitive to the choice of distance metric. Different metrics may perform better or worse depending on the data characteristics. K-NN may struggle with imbalanced datasets, as the majority class can dominate the prediction for nearby instances. K-NN is widely used in various domains, such as recommendation systems, image recognition, and anomaly detection.

Tree decision (tree)

Decision trees are a popular machine learning algorithm used for both classification and regression tasks². The Tree framework is presented in Fig. 10. They are simple yet powerful models that make predictions by recursively partitioning the feature space based on the values of input features. Here’s an overview of how decision trees work: Data Preparation: Prepare a labeled dataset where each instance consists of a set of features (attributes) and a corresponding class label or target value. Tree Construction: The decision tree algorithm starts by selecting the best feature to split the dataset. This is typically done using a measure of impurity, such as Gini impurity or entropy. The selected feature is used as the root node of the tree. Splitting: The dataset is split into subsets based on the possible values of the selected feature. Each subset corresponds to a branch of the tree, and the splitting process is repeated recursively for each subset. Stopping Criteria: The splitting process continues until a stopping criterion is met⁶. This could be reaching a maximum tree depth, having a minimum number of instances in a leaf node, or not achieving a significant improvement in impurity reduction. Leaf Node Assignment: Once the stopping criterion is met, the leaf nodes of the tree are assigned with the majority class label for classification tasks or the mean or median value for regression tasks. Prediction: To make predictions for new instances, the algorithm traverses the decision tree from the root node to a leaf node, following the splits based on the feature values. The prediction is based on the class label or target value associated with the reached leaf node. Decision trees have several advantages: They are easy to interpret and visualize, as the decision-making process is represented by a tree structure. Decision trees can handle both categorical and numerical features. They can capture nonlinear relationships between features. Decision trees can handle missing values and outliers in the data. However, there are considerations with decision trees: Decision trees are prone to overfitting, especially when the tree grows too deep or when the dataset contains noisy or irrelevant features. Techniques like pruning or using regularization parameters can help mitigate overfitting. Decision trees can be sensitive to small changes in the training data, which can lead to different tree structures and predictions. The algorithm may create complex and highly specific decision boundaries, which may not generalize well to unseen data. Decision trees may struggle with imbalanced datasets, as they can be biased towards the majority class. Despite these limitations, decision trees are widely used in various domains such as finance, healthcare, and customer relationship.

Random forest (RF)

Random forest is an ensemble learning algorithm that combines multiple decision trees to create a more robust and accurate model¹⁵. The RF framework is presented in Fig. 11. It is a popular and powerful machine learning algorithm that is used for classification, regression, and other tasks. Here’s an overview of how Random Forest works: Data Preparation: Prepare a labeled dataset where each instance consists of a set of features (attributes) and a corresponding class label or target value. Tree Construction: The Random Forest algorithm creates multiple decision trees using a subsample of the training data and a subset of the features. The subsample is typically generated by random sampling with replacement (i.e., bootstrapping), and the subset of features is randomly selected for each split. Tree Training: Each decision tree is trained on the subsample and the subset of features, using a similar process to the standard decision tree algorithm. Voting: To make predictions for new instances, the algorithm aggregates the predictions of all the decision trees. For classification tasks, the majority vote of the decision trees is taken as the predicted class label. For regression tasks, the mean or median value of the decision trees’ predictions is taken as the predicted target value. Random Forest has several advantages: It is highly accurate and robust, as it combines the predictions of multiple decision trees, which reduces overfitting and improves generalization performance. Random Forest can handle nonlinear relationships between features. It can handle missing values and outliers in the data. Random Forest can be used for feature selection, as it provides an estimate of feature importance based on their contribution to the overall prediction. However, there are considerations with Random Forest: The algorithm can be computationally expensive, especially when dealing with large datasets or high-dimensional feature spaces. The interpretability of Random Forest is lower than that of a single decision tree, as the prediction process involves multiple decision trees. Random Forest may not perform as well as other algorithms, such as neural networks or gradient boosting, in certain scenarios. Overall, Random Forest is a powerful and versatile algorithm that has been successfully applied in various domains, including finance, healthcare, and natural language processing. It is implemented in popular machine learning libraries such as scikit-learn and TensorFlow.

Symbolic regressions

Artificial neural network (ANN)

Artificial neural networks (ANNs) are computational models inspired by the structure and functionality of biological neural networks in the human brain¹⁹. The ANN framework is presented in Fig. 12. They are a subset of machine learning algorithms that have gained significant attention and success in various fields, including computer vision, natural language processing, speech recognition, and many others. An ANN consists of interconnected artificial nodes, called artificial neurons or simply “neurons”. These neurons are organized into layers, typically divided into an input layer, one or more hidden layers, and an output layer. Each neuron receives input signals, processes them, and produces an output signal based on an activation function. The connections between neurons are represented by weights, which determine the strength and influence of one neuron on another. During the training process, these weights are adjusted iteratively based on a loss function, which measures the difference between predicted outputs and the desired outputs. This process is typically carried out using optimization algorithms like gradient descent, which aim to minimize the loss function. The most common type of ANN is the feedforward neural network, where signals flow only in one direction, from the input layer through the hidden layers to the output layer¹⁹. Another type is the recurrent neural network (RNN), which has connections that form loops, allowing information to persist and flow in cycles. RNNs are particularly suited for tasks involving sequential or temporal data, such as speech recognition or language translation. Deep neural networks (DNNs) are ANNs with multiple hidden layers. They have been particularly successful in recent years due to advances in computational power, availability of large datasets, and improvements in training algorithms. Deep learning, which refers to the use of DNNs, has achieved remarkable results in areas like image classification, object detection, text generation, and more. Overall, ANNs are powerful tools for pattern recognition, prediction, and decision-making tasks. They can automatically learn complex relationships and extract meaningful features from raw data, making them valuable tools in various domains. However, they also require careful design, training, and validation to ensure optimal performance and avoid overfitting or other issues.

Response surface methodology (RSM)

Response Surface Methodology (RSM) is a statistical and mathematical technique used for optimizing and modeling complex processes²¹. The RSM framework is presented in Fig. 13. It is particularly useful when the relationship between input variables and output responses is nonlinear and cannot be easily understood through simple linear models²¹. RSM is commonly employed in fields such as engineering, chemistry, manufacturing, and experimental design. The primary goal of RSM is to find the optimal settings of input variables that maximize or minimize a response of interest²². The process involves conducting a series of experiments or simulations to collect data on the response variable at different combinations of input variables²³. These data points are then used to fit a mathematical model that describes the relationship between the input variables and the response²⁴. The model is typically constructed using regression techniques such as polynomial regression. The response surface, as the name suggests, is a visual representation of the relationship between the input variables and the response²⁵. It is a three-dimensional plot where the axes represent the input variables, and the surface represents the response. By analyzing the response surface, researchers can identify the optimal values of the input variables that lead to the desired response²⁶. RSM allows for the optimization of processes by finding the combination of input variables that maximizes a desired output response or minimizes undesired responses. It helps in reducing the number of experiments needed to determine the optimal settings and provides insights into the interactions between variables²⁷. RSM can also be used for sensitivity analysis to identify the most critical factors affecting the response. In summary, Response Surface Methodology is a powerful statistical technique used for optimizing complex processes by modeling the relationship between input variables and output responses. It helps researchers and engineers make informed decisions and improve the efficiency and effectiveness of their processes.

Sensitivity analysis

In the context of modeling and analysis, sensitivity analysis refers to the study of how the uncertainty in the output of a model or system can be attributed to different sources of uncertainty in its input variables²⁸. Sensitivity analysis helps identify which input variables have the most significant impact on the output and how changes in input variables affect the output of interest²⁹. There are various methods for conducting sensitivity analysis, including: One-at-a-Time (OAT) Sensitivity Analysis: This method involves varying one input variable at a time while keeping all other variables constant at their base values³⁰. It helps assess the individual effect of each input variable on the output but does not capture potential interactions between variables. Local Sensitivity Analysis: Local sensitivity analysis calculates the sensitivity of the output to changes in the input variables around a specific point or scenario³¹. It provides insights into how small perturbations in input variables near the current operating point affect the output. Global Sensitivity Analysis (GSA): GSA considers the entire input parameter space and evaluates the sensitivity of the output to variations in input variables across the entire range³¹. It helps identify the most influential input variables and interactions between variables, even in high-dimensional and nonlinear systems. Common GSA methods include: Variance-Based Sensitivity Analysis (e.g., Sobol’ indices): Decomposes the variance of the output into contributions from individual input variables or combinations of variables. Screening Methods (e.g., Morris method): Identifies influential input variables using a limited number of model evaluations. Response Surface-Based Methods (e.g., polynomial chaos expansion): Approximates the model response using a surrogate model and evaluates the sensitivity of the surrogate model to input variations³². Graphical Methods: Graphical techniques such as tornado diagrams, scatter plots, and sensitivity curves provide visual representations of the sensitivity of the output to changes in input variables. They offer intuitive insights into the relative importance of input variables and their relationships³³. Monte Carlo Simulation: Monte Carlo simulation involves sampling input variables from their probability distributions and propagating them through the model to estimate the distribution of the output³⁴. Sensitivity analysis can be performed by examining how changes in input distributions affect the output distribution. Sensitivity analysis is crucial for decision-making, risk assessment, and model validation in various fields such as engineering, finance, environmental science, and healthcare. It helps stakeholders understand the uncertainties associated with model predictions and identify key drivers of variability in the system under study³⁵. A preliminary sensitivity analysis was carried out on the collected database to estimate the impact of each input on the (Fc) values. “Single variable per time” technique is used to determine the “Sensitivity Index” (SI) for each input using Hoffman and Gardener formula³⁵ as follows:

Performance evaluation indices

The following section discusses the results of each model. The accuracies of developed models were evaluated by comparing SSE, MAE, MSE, RMSE, Error %, Accuracy % and R² between predicted and calculated shear strength parameters values. The definition of each used measurement is presented in Eqs. (2)–(7)³⁷.

where N is the total number of entries, \({y}_{i}\) is the ith entries, \(\widehat{y}\) is the modal value, \(\overline{y }\) is the average value.

Consent to participate

The authors have the consent to participate in this publication of the research paper.