Hello friends, In this sixth installment of our Machine Learning series, we will go into more theoretical detail about the use of tree-based models, which we briefly mentioned before.
Tree-based models are a powerful and flexible method used to split data sets into branches to solve classification and regression problems.
In this chapter;
- It explains fundamental concepts such as entropy and the Gini index and provides a detailed explanation of how decision trees are created and used.
- We also provide examples of how to apply decision trees to classification and regression problems.
Decision Trees for Classification
Given a labeled dataset, a classification tree extracts labels by learning a series of if-else questions about individual features. Unlike linear models, trees can capture nonlinear relationships between features and labels. Furthermore, trees do not require that features be on the same scale, for example through standardization.
When we say classification tree, we actually mean a decision tree trained for a classification task.
I imported the dataset used in this study, the Wisconsin Breast Cancer Dataset. You can now import it directly into your project from the UC Irvine ML repository referenced in this article series and other beginner-level resources on artificial intelligence, without having to download it in .csv format.
Click on this link and press the “Import in Python” button to see a sample usage.
Let's remember again how easy it is to use decision trees in this way.
Why decision trees?
- If there is a nonlinear relationship in your data and you want to capture that relationship,
- If model interpretability is important to you and you want to visualize the decision-making process,
- If you are working with categorical data and do not want to convert this data to numerical data,
- If you want to select features to highlight more important features,
- If you are working on multiple classification problems, decision trees may have an advantage over logistic regression.
A numerical comparison will show the difference, but let's first look at it visually to get a better understanding of how it works. You can visualize the comparison using the following function:
Now, we will build a logistic regression model and draw a comparison chart.
Because the data set is so small, the number of unpredictable data is very small. However, it is important to note that decision trees have better boundary drawing ability when capturing nonlinear relationships.
How do decision trees learn to classify?
Let's start with some definitions of terminology: A decision tree is a data structure made up of individual units arranged in a hierarchy. node.a node These are the points that include questions and predictions.
of root The node where a decision tree begins to grow. It has no parent node and contains a question that leads to two child nodes via two branches. Internal Nodes is a node that has a parent. It also contains a question that leads to two child nodes. Finally, a node that has no children is called a leaf. A leaf has one parent node and no questions. This is where predictions are made. Recall that when a classification tree is trained on a labeled dataset, the tree learns patterns from the features in a way that produces the purest leaves. In other words, the tree is trained to have one class label dominate each leaf.
Visually, you can look at the parts of a decision tree like this: the root, branches, internal nodes, and leaves form the parts of a decision tree.
So what mathematics does a decision tree follow when creating these leaves?
The important thing to note here is that when a regression tree is trained on a dataset, the purity of the nodes is Mean Square Error of the target for that node.
That is, a regression tree attempts to find a split that produces a leaf such that the target value for each leaf is as close as possible to the specified mean for that leaf.
Node Mean Squared Error (MSE)
where N is the number of instances in the node.
Yi is the actual target value for the ith sample.
yˉ is the average target value of all samples in the node.
Average of the target value within the node
When a regression tree is trained, it tries to reduce the MSE at each fold, so at each fold, it aims to reduce the total MSE by ensuring that the target value for a leaf is as close as possible to the average target value for that leaf.
Combining these concepts, a regression tree seeks to minimize the sum of the MSE of the left and right child nodes at each fold. This minimization is Total purity loss.
Total purity loss
The MSE parent is the MSE of the parent node.
MSE Left and MSE Right are the MSE of the left and right child nodes.
N total is the total number of instances in the master node.
N left and N right are the number of samples in the left and right subnodes.
The regression tree algorithm tries to maximize the purity decay (ΔMSE) at each split. Information Acquisition To measure this concept of reduced purity.
What is information gain?
Information gain measures how much uncertainty or impurity is reduced by splitting a data set based on a particular feature. This measure is used when building decision trees and selecting which features to use at each split step. The goal is to split the data set in a way that maximizes information gain.
Entropy and Information Gain
Information gain is based on the concept of entropy, which is a measure of the uncertainty or randomness in a data set. Higher entropy indicates a more complex or random data set, while lower entropy indicates a more orderly data set.
Entropy calculation
Entropy is used to measure the uncertainty in a data set. The entropy equation is:
In this formula,
- H(D) is the entropy of the data set (D).
- (k) is the number of classes.
- pi is the probability of the ith class.
Information Gain Calculation
Information gain is how much entropy is reduced when splitting a dataset based on a particular property.
In this formula,
- IG(D, A) is the information gain of a feature (A) on a dataset (D).
- (H(D)) is the entropy of the dataset (before splitting).
- Value(A) is the set of different values that feature(A) can take.
- P is the subset of the dataset for which feature (A) takes value (v).
- |D| is the total number of samples in the dataset.
- |𝐷𝑣| is the number of samples in the subset that have value (v).
- H(𝐷𝑣) is the entropy of the subset.
summary:
- Entropy measures the uncertainty in a data set.
- Information gain measures the reduction in entropy when splitting a dataset based on a particular feature.
- Splits with high information gain result in a purer (less ambiguous) dataset and are therefore preferred when building decision trees.
- Information gain is used to select the optimal split at each node of the decision tree, creating a more accurate and effective model.
This information gain criterion can use the Gini index and the concepts of entropy. standard The values you use may vary from project to project, depending on the size of your dataset and hyperparameters.
Decision Trees for Regression Analysis
In regression analysis, the target variable is ConsecutiveIn other words, the output of the model is RealAgain, using decision trees allows the model to learn the dataset better. Linear regression.
In this study, we use the Auto MPG dataset from the UC Irvine Machine Learning Repository.
Regression trees were shown to slightly outperform linear regression (slide to the end of the cell above to see the results). Note that these values may vary depending on the size of your dataset and your hyperparameters.
Finally, let's visualize this comparison (slide again to the end of the cells below to see the plot):
This concludes our first detailed look at decision trees. In the next section, Bias-variance tradeoff, Bag packing, Random Forest, Boost ve Tuning the Model concept.
See you in the next article.
