The input variables that you give to a machine learning model are called features. Each column in your dataset constitutes a feature. To train the best model, you want to make sure that you only use important features. With too many features, your model may capture unimportant patterns and learn from noise. The technique of selecting the important parameters for your data is called feature selection.
In this article titled “Everything You Need to Know About Feature Selection”, we will explain everything you need to know about feature selection.
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
Why feature selection?
Machine learning models follow a simple rule: whatever you put into them, you get out: if you put garbage into the model, you can expect the output to be garbage, where garbage is the noise in your data.
To train a model, we collect huge amounts of data to improve machine learning. Usually, most of the collected data is noise, and some columns in the dataset may not contribute significantly to the model's performance. Moreover, a large amount of data can slow down the training process, making the model slow. The model may also learn from this irrelevant data and become inaccurate.
Feature selection is what separates good data scientists from the rest. With the same model and computational facilities, why do some people win the race with faster and more accurate models? The answer is feature selection. Not only do you have to choose the right model for your data, but you also have to choose the right data to feed into your model.
Consider a table containing information about old cars, and the model determines which cars need to be crushed to get spare parts.
Figure 1: Old Cars Dataset
From the above table, we can see that the car model, year of manufacture, and mileage are very important in determining whether the car is old enough to be crushed. However, the name of the car's previous owner does not determine whether the car should be crushed or not. Moreover, it may confuse the algorithm that finds patterns between names and other features. Hence, we can remove the column.
Figure 2: Column removal for feature selection
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
What is feature selection?
Feature selection is a method of reducing the input variables to a model by using only the relevant data and removing noise in the data.
It is the process of automatically selecting relevant features for your machine learning model based on the type of problem you are trying to solve. It does this by including or excluding important features without altering them. This reduces the noise in your data and reduces the size of your input data.
Figure 3: Feature selection
Feature Selection Models
There are two types of feature selection models:
- Supervised Model: Supervised feature selection refers to a technique in which the output label class is used for feature selection. The target variable is used to identify variables that can improve the efficiency of the model.
- Unsupervised Models: Unsupervised feature selection refers to techniques that do not require an output label class for feature selection. Use for unlabeled data.
Figure 4: Feature selection model
Supervised models can be further divided into three categories:
1. Filter methodIn this method, features are related to the output, i.e. how Correlation On the output. Use correlation to see if the features are positively or negatively correlated with the output label and remove the features accordingly. Ex: Information Gain, Chi-squared test, Fisher score, etc.
Figure 5: Flowchart of the filter method
2. Wrapper method: Split the data into subsets and use this to train the model. Based on the model output, add and remove features and train the model again. Use a greedy approach to form subsets and evaluate the accuracy of all possible combinations of features. Ex: forward selection, backward elimination, etc.
Figure 6: Flowchart of the wrapper method
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
3. Intrinsic Method: This method combines the characteristics of both filter and wrapper methods to create optimal subsets.
Figure 7: Flowchart of the unique model
This method handles the iterative process of machine training while minimizing computational costs. Example: Lasso and Ridge regression.
How to choose a feature selection model?
How do we know which feature selection model is suitable for our model? The process is relatively simple, and the model depends on the types of input and output variables.
There are two main types of variables:
- Numeric variables: Includes integers, floating point numbers, and numeric values.
- Categorical variables: Includes labels, strings, Boolean variables, etc.
Based on whether you have numerical or categorical variables as input and output, you can choose a feature selection model as follows:
|
Input variables |
Output Variables |
Feature Selection Models |
|
Number |
Number |
|
|
Number |
category |
|
|
category |
Number |
|
|
category |
category |
|
Table 1: Search for feature selection models
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
Feature Selection with Python
Get hands-on experience with feature selection by working with the Kobe Bryant dataset, which analyzes shots taken by Kobe from different areas of the court to determine which shots went in.
The dataset looks like this:
Figure 8: Kobe Bryant Dataset
As you can see, the dataset has 25 different columns, not all of which are needed.
First, start by loading the required modules:
Figure 9: Importing a module
First, look at the loc_x and loc_y columns, which likely represent longitude and latitude.
Figure 10: Plotting the latitude and longitude columns of a dataset
The diagram looks like this:
Figure 11: Latitude and longitude plot
From the diagram above, you can see that it resembles two “D's” on a basketball court. Instead of having two separate columns, we can change the coordinates into polar format to have one column: [‘angle’].
Figure 12: Converting latitude and longitude into polar format
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
You can combine the minutes and seconds columns into a single time column.
Figure 13: Combining two columns
Let's look at the unique values in the “team_id” and “team_name” columns.
Figure 14: Unique values for “team_id” and “team_name”
The entire column contains only one value and can be removed. Let's look at the “match_up” and “opponent” columns.
Figure 15: “match_up” and “opponent” columns
They contain the same information. Let's plot the values of the 'dist' and 'shot_distance' columns on the same graph to see the difference.
Figure 16: Plot of “dist” and “shot_distance” columns
They contain the exact same information. Look at the shot_zone_area, shot_zone_basic, and shot_zone_range columns.
Figure 17: Plotting columns for different shot zones
The figure below shows the plot.
Figure 18: Different shot zones
You can see that includes the different parts of the court where the shots were taken, this information is already present in the angle and distance columns.
Now, let's remove all the columns we don't need.
Figure 19: Deleting a column
After combining columns and removing unnecessary columns, we have a dataset with only 11 significant columns.
Figure 20: Final dataset
Learn the fundamentals of object-oriented programming, web development with Django, and more in Caltech's AI and Machine Learning graduate program. Enroll now.
AI/ML careers are just around the corner!
AI Engineer Master's ProgramSee Program
Conclusion
In this article titled “All You Need to Know About Feature Selection”, we discussed how important it is to select the best features for your machine learning model. We then looked at what feature selection is and some feature selection models. We then moved on to a simple way to select the right feature selection model based on the input and output values. Finally, we saw how to implement feature selection in Python with a demo. If you want to learn more about feature selection and related basic functions of Python, Simplielarn's Python Certification Course is the perfect choice. This Python Certification Course covers the fundamentals of Python, including data manipulation, conditional statements, shell scripting, Django, and more, preparing you for a rewarding career as a professional Python programmer.
Did you find this article on feature selection helpful? Do you have any doubts or queries? Please leave a comment in the comments section of this article and our experts will get back to you as soon as possible.
