Lessons Learned from My ML Journey: Data Partitioning and Data Leakage

Statistical Modeling and Machine Learning

So you might be wondering why I'm starting by looking back at my journey instead of diving right into the meat of the story. The reason is simple: many people claim to be building ML models, but in reality they're just creating statistical models. I was one of them. I'm not saying one is better than the other, but I think it's important to recognize the subtle differences between statistical modeling and ML before we get into the technical details.

The purpose of Statistical model teeth To make inferencesThe main purpose of Machine Learning teeth For predictionSimply put, ML models leverage statistics and mathematics to generate predictions that can be applied to real-world scenarios, and this is where data partitioning and data leakage become an issue, especially in the context of supervised machine learning.

Initially, I thought that my understanding of statistical analysis would be sufficient for my forecasting tasks, but I quickly realized that without knowledge of data preparation techniques such as proper data partitioning, and awareness of potential pitfalls such as data leakage, even the most sophisticated statistical models will result in poor predictive performance.

So let’s get started!

What is data partitioning?

Data splitting essentially means splitting a dataset into several parts to optimize the predictive performance of a model.

Consider the simple concept of OLS regression, which many of us are familiar with. We have all heard about this concept in business/statistics/finance, economics, or engineering courses. It is a fundamental ML technique.

Suppose you have a dataset of house prices and factors that may affect house prices.

in Traditional statistical analysis, We hire The entire dataset Since our goal is simply to understand the factors that influence house prices, we need to develop a regression model. In other words, a regression model can explain how the degree of change in prices is associated with the predictors.

But in ML, while the statistics remain the same, splitting the data becomes important. Let me explain why: Assuming you train a model on the entire set, how do you know the predictive performance of the model on unknown data?

For this very reason, we usually split a dataset into two sets: training and testing. The goal is to train a model on one set and evaluate its performance on the other. Essentially, the testing set should serve as the real data, which means that the model should not have any access to the testing data during the entire training phase.

Here comes a pitfall I hadn't noticed before: splitting the data into two sets is not inherently wrong, but it does run the risk of creating an unreliable model. Imagine you were to fine-tune a model by training it on the training set, validating its accuracy on the test set, and repeating the process. This would lead to model selection bias, defeating the purpose of “unseen data” altogether, since the test data was seen multiple times during model development. This would undermine the model's ability to truly predict unseen data, and would result in overfitting issues.

Prevention methods:

Ideally, the dataset should be split into two blocks (three different splits).

(training set + validation set) → 1st block
Test set → 2nd block

The model can be trained and validated on the first block. The second block (the test set) does not participate in the model training process at all. Think of the test set as a danger zone.

How you split your data depends on the size of your dataset. The industry standard is 60%-80% for the training set (first block) and 20%-40% for the test set. The validation set is usually curved away from the first block, so that the actual training set is 70%-90% of the first block, and the rest is for the validation set.

The best way to understand this concept is visually.

Leave-One-Out (LOOV) method (Image by author)

Besides LOOV (Figure), there are several other data partitioning techniques.

K-fold cross-validation divides the data into multiple 'K' folds and iterates the training process accordingly.
Rolling Window Cross-Validation (for Time Series Data)
Blocked Cross-Validation (for time series data)
Stratified Sampling Split for Imbalanced Classes

Note: Time series data requires special care when splitting the data because there is a temporal order to it. Splitting the dataset randomly can result in a mess of time order. (I learned that the hard way)

Most importantly, regardless of the technique used, the “test set” is kept separate and unchanged until model selection.

“In machine learning, Data leaks “This is a mistake made by machine learning model creators who accidentally share information between their test and training datasets.” — Analytics Vidhya

This relates to the first point about test data being contaminated by training data, which is an example of data leakage, but a validation set alone cannot prevent data leakage.

To prevent data leakage, you need to be careful with your data handling process, from Exploratory Data Analysis (EDA) to feature engineering: any step that allows training and testing data to interact can lead to leakage.

There are two main types of leaks:

Trains, tests, contamination

A common mistake I see people make is to apply a standardization/preprocessing step to the whole set before splitting the data, for example using mean imputation to handle missing values/outliers across the whole dataset, which will incorporate information from the test data into the training data, resulting in a higher accuracy of the model compared to its actual performance.

2. Target Leak

Target leakage can occur when features (predictors) have some dependency on the variables they predict (targets), or when feature data is not available at the time of prediction.

Let's take as an example a piece of data I worked with, where we were trying to predict sales performance based on an advertising campaign. We tried to include conversion rate, but overlooked the fact that conversion rate is only known after the campaign, meaning we don't have this information at the time of prediction. Moreover, since conversion rate is tied to sales data, we would have a typical target leakage. Including conversion rate would mean that the model would learn from data it would not normally have access to, leading to overly optimistic predictions.

Sample (fictional) dataset (image by author)

How to prevent data leaks:

In summary, to address the issue of data leakage, keep the following points in mind:

Proper Data Preprocessing
Careful Cross-Validation
Careful feature selection

Conclusion

That's it, thanks for reading! I hope this article cleared up some common misconceptions about data splitting and shed light on best practices for building efficient ML models.

This is not only to document my learning process but also for mutual learning, so if you spot any gaps in my technical knowledge or have any insights you'd like to share, feel free to message me.