The Definitive Guide to Overfitting and Underfitting in Machine Learning

Overfitting and underfitting are two important concepts in machine learning and are the main causes of poor performance of machine learning models. In this tutorial, we will explain overfitting and underfitting in machine learning and through practical demonstrations, help you understand how to avoid them.

What is overfitting?

When a model performs very well on training data but poorly on test data (new data), this is called overfitting. In this case, the machine learning model learns the details and noise in the training data, which negatively impacts the model's performance on the test data. Overfitting can occur when the bias is low and the variance is high.

Overfitting in ML

Reasons for overfitting

The data used for training was not cleaned and contains noise (garbage values)
This model has a high variance
The training dataset used is of insufficient size
The model is too complex

How to deal with overfitting

Using K-fold cross-validation
Use regularization techniques such as Lasso and Ridge
Training models with sufficient data
Adoption of ensemble techniques

What is underfitting?

When a model has not learned the patterns in the training data well enough to generalize well to new data, it is called underfitting. An underfitting model will perform poorly on the training data and produce unreliable predictions. Underfitting occurs because of high bias and low variance.

ML underfitting

Reasons for underfitting

The data used for training was not cleaned and contains noise (garbage values)
This model is highly biased
The training dataset used is of insufficient size
The model is too simple

How to deal with underfitting

Increasing the number of features in the dataset
Increasing the complexity of the model
Reducing noise in your data
Extend the training period of the data

Now that we understand what overfitting and underfitting are, let's see what is a well-fitted model in this tutorial on overfitting and underfitting in machine learning.

What is good for machine learning?

To find the right model, you need to look at how your machine learning model performs over time using your training data. As the algorithm learns over time, the model's error on the training data and the error on the test dataset will decrease. If you train your model for too long, it may learn unnecessary details and noise in the training set, leading to overfitting. To achieve a good fit, you should stop training when the error starts to increase.

The fit model is good.

Demo – Goodness of fit analysis of the IRIS dataset

Importing libraries

Loading the dataset

Here, we evaluate the generalization efficiency of the model using K-fold cross-validation with 20 folds (K=20). Within each fold, we estimate the training and testing errors using the training and testing datasets, respectively.

K-fold cross-validation

Plot the mean absolute error (MAE) of the training phase versus the MAE of the test phase

Calculating MAE

Plot Error

By using K-fold cross-validation, we were able to significantly reduce the error on the test dataset.

Conclusion

Overfitting and underfitting are two important concepts related to the bias-variance tradeoff in machine learning. In this tutorial, we learned the basics of overfitting and underfitting in machine learning and how to avoid them. We also looked at the various reasons why they occur.

If you want to learn the fundamentals of machine learning and gain a comprehensive understanding that you can immediately use in your work, Simplilearn's AI ML course, in partnership with Purdue and in collaboration with IBM, is the perfect fit. This 12-month bootcamp program provides comprehensive applied training in key concepts such as machine learning, deep learning with Keras and Tensorflow, advanced deep learning and computer vision, and natural language processing.

Do you have any questions about this tutorial on overfitting and underfitting in machine learning? If you have any questions, write them in the comments section and we will answer them. To learn more, watch this video: Overfitting and Underfitting.

Happy learning!

Source link