In the previous article, we introduced the core mechanics of Gradient Boosting. gradient boosted linear regression.
This example was intentionally simple. The goal was understanding, not performance.
By using a linear model, we were able to do all the steps explicitly, including residuals, updates, and the additive nature of the model. I also linked to gradient descent It's very clear.
In this article, we'll move on to the settings where Gradient Boosting is actually useful. Decision tree regressor.
We reuse the same conceptual framework as before, but the behavior of the algorithm changes in important ways. Unlike linear models, decision trees are nonlinear and piecewise constant. Combining these through gradient boosting prevents them from collapsing into a single model. Instead, each new tree adds structure and improves the predictions of previous trees.
For this reason, we will only briefly summarize general gradient boosting mechanisms and instead focus on those specific to gradient boosting mechanisms. Gradient-boosted decision tree: How the tree is trained with residuals, how the ensemble evolves, and why this approach is so powerful.
1. Machine learning in three steps
To keep the explanation consistent and intuitive, we will use the same three-step framework again.
1. Base model
Use a decision tree regressor as the base model.
Decision trees are nonlinear in their structure. Divide the feature space into regions and assign a constant prediction to each region.
The important point is that adding trees does not collapse into a single tree.
Each new tree introduces additional structure to the model.
This is where Gradient Boosting becomes especially powerful.
1 screw. ensemble model
Gradient boosting is a mechanism used to: totalling Combine these base models into a single predictive model.
2. Model fitting
For clarity, use: stump of decisionmeans a tree with a depth of 1 and a single split.
Each tree is residual of the previous model.
2 screws. ensemble learning
The ensemble itself is constructed using Gradient descent in function space.
Here, the object being optimized is not a parameter, but functionand those functions are decision trees.
3. Model tuning
Decision trees have several hyperparameters:
- maximum depth
- Minimum number of samples required for splitting
- Minimum number of samples per leaf
In this article, we fix the tree depth to 1.
At the ensemble level, two additional hyperparameters are essential:
- of learning rate
- of Boosting iteration count
These parameters control the training speed and complexity of the model.
2. Gradient boosting algorithm
Gradient boosting algorithms follow a simple, iterative structure.
2.1 Algorithm overview
The main steps of the gradient boosting algorithm are:
- Initialization
Start with a constant model. For regression with squared loss, this is the mean value of the target. - Residual calculation
Compute the residual between the current prediction and the observed value. - suitable for weak learners
Train a decision tree regressor to predict these residuals. - Update model
Add a new tree to the existing model, scaled according to the learning rate. - repeat
Repeat until the selected number of boosting steps is reached or the error stabilizes.
2.2 Dataset
To illustrate how gradient-boosted trees work, I'll use several types of datasets that I've generated.
- piecewise linear datathe relationship changes for each segment.
- nonlinear datacurve pattern etc.
- binary targetfor classification tasks
For classification purposes, let's start first. squared loss To make it easier. This allows us to reuse the same mechanism as regression. The loss function can later be replaced by an alternative function suitable for classification, such as logistic loss or exponential loss.
These different datasets help highlight how Gradient Boosting adapts to different data structures and loss functions while relying on the same underlying algorithm.
2.3 Initialization
The gradient boosting process starts with a constant model.
For regression with squared loss, this initial prediction is simply Mean value of target variable.
This average value represents the best initial prediction before learning structure from features.
It's also a good time to remember: Almost all regression models can be considered an improvement over the global average..
- k-NN Find similar observations and use the mean value of the neighborhood to make predictions.
- Decision tree regressor Split the dataset into regions, calculate the average value within each leaf, and predict new observations that fall into this leaf.
- weight-based model Adjust or update feature weights to balance the grand average for a given new observation.
Again, for gradient boosting we start from the average value. And we'll see how it gradually corrects itself.
2.4 First tree
The first decision tree is then residual of this early model.
After initialization, the residual is just the difference between the observed value and the mean value.
To build this first tree, use the exact same steps as in the next article. Decision tree regressor.
The only difference is the target. Instead of predicting the original value, the tree predicts the residual.
This first tree provides an initial correction to the constant model and sets the direction of the boosting process.
2.5 Model updates
Once the first tree is trained on the residuals, the first improved predictions can be computed.
The updated model is obtained by combining the initial prediction and the modification of the first tree.
f1(x) = f0 + learning rate * h1(x)
where:
- f0 is the initial prediction and is equal to the mean value of the target
- h1(x) is the prediction of the first tree trained on the residuals.
- learning_rate controls how much this correction is applied
This update step is the core mechanism of Gradient Boosting.
Each tree does not replace the current predictions, but slightly adjusts them, allowing the model to gradually improve and remain stable.
2.6 Repeating the process
The same steps are repeated after the first update is applied.
At each iteration, new residuals are computed using the current predictions, and a new decision tree is trained to predict these residuals. This tree is added to the model using the learning rate.
To make this process easier in Excel, you can write formulas in a completely automated way. Once this is done, simply copy the expressions for the second tree and all subsequent trees to the right.
As each iteration progresses, all predictions of the residual model are grouped together. This makes the structure of the final model very clear.
Finally, predictions can be written in compact form.
f(x) = f0 + eta * (h1(x) + h2(x) + h3(x) + …)
This expression emphasizes an important idea. In other words, the final model is simply the initial prediction plus a weighted sum of the residual predictions.
It also opens the door to expansion possibilities. For example, the learning rate does not have to be constant. It can decay through an iterative process and decrease over time.
This is the same idea for decay in gradient descent or stochastic gradient descent.
3. Understand the final model
3.1 How models evolve through iterations
Start with a partitioned dataset. In the visualization below you can see all the intermediate models generated during the gradient boosting process.
first, Initial constant predictionequal to the mean value of the target.
then come f1obtained after adding the first tree with a single split.
Next, f2after adding the second tree, etc.
Local fixes are introduced for each new tree. As more trees are added, the model gradually adapts to the structure of the data.
The same behavior appears when: curve dataset. Even though the individual trees are piecewise constant, their additive combination results in a smooth curve that follows the underlying pattern.
When applied to binary targetthe algorithm will still work, but some predictions may be negative or greater than 1. This is expected if you treat the problem as a regression and use a squared error loss that does not limit the output range.
If you want a probability-like output, you should use a classification-oriented loss function such as logistic loss instead.
In conclusion, Gradient Boosting can be applied to: Different types of datasetsincluding piecewise, nonlinear, and binary cases. Regardless of the dataset, the final model remains piecewise constant by constructionbecause it is constructed as a sum of decision trees.
However, by accumulating many small corrections, the overall prediction can approximate a complex pattern.
3.2 Comparison with a single decision tree
Viewing these plots naturally raises questions such as:
Doesn't gradient boosting end up creating a tree, similar to a decision tree regressor?
This impression is understandable, especially when dealing with small datasets. Visually, the final predictions look similar, making it difficult to distinguish between the two approaches at first glance.
But when you look at it, the difference is obvious How to calculate the split.
A single decision tree regressor series of divisions. Each split divides the available data into smaller subsets. As the tree grows, new decisions are made based on fewer and fewer observations, which can make the model more susceptible to noise.
Once segmentation occurs, data points that fall into different regions are no longer associated. Each region is treated independently and early decisions cannot be changed.
Gradient boost trees work in a completely different way.
Each tree in the boosting process is Entire dataset. Observations are never removed from the learning process. In every iteration, every data point contributes through the residual.
This fundamentally changes the behavior of the model.
What a single tree makes a difficult, irreversible decision. Gradient boosting, on the other hand, allows later trees to: correct a mistake Made by someone before.
Rather than committing to one rigid partition of the feature space, the model gradually refines its predictions through a series of small adjustments.
This ability to revise and improve previous decisions Gradient boosted trees are actually robust and powerful.
3.3 Comprehensive comparison with other models
compared to single decision treeGradient-boosted trees produce smoother predictions, reduce overfitting, and improve generalization.
compared to linear modelnaturally captures nonlinear patterns, automatically models feature interactions, and does not require manual feature engineering.
compared to Nonlinear weight-based modelGradient Boosted Trees, such as kernel methods and neural networks, offer a different set of tradeoffs. They rely on simple, interpretable constructs, are less sensitive to feature scaling, and require fewer assumptions about the structure of the data. In many real-world situations, training is also faster and requires fewer adjustments.
These combined characteristics explain why Gradient-boosted decision tree regressor It performs very well in a wide range of real-world applications.
conclusion
This article showed how Gradient Boosting builds powerful models by combining simple decision trees trained on residuals. Starting from a fixed prediction, the model is gradually refined through small local modifications.
We found that this approach naturally adapts to different types of datasets, and the choice of loss function is essential, especially for classification tasks.
By combining the flexibility of trees with the stability of boosting, gradient-boosted decision trees deliver powerful performance in practice while remaining conceptually simple and easy to interpret.
