Outlier Detection with Scikit-Learn and Matplotlib: a Practical Guide | by Riccardo Andreoni | Oct, 2023

Learn how visualizations, algorithms, and statistics help you to identify anomalies for your machine learning tasks.

Imagine a room filled with colorful balloons, each symbolizing a data point in a dataset. Due to their different features, the balloons float at different heights. Now, picture some helium-filled balloons that unexpectedly soar far above the rest. Just as these exceptional balloons disrupt the uniformity of the room, outliers disrupt the pattern in a dataset.

Returning from this colorful analogy to pure statistic, outliers are defined as anomalies, or better, data points that deviate significantly from the rest of the dataset.

Consider a Machine Learning algorithm developed to diagnose diseases based on patient data. In this real-world example, outliers could be extremely high values in laboratory results or physiological parameters. While their origin may consist in various reasons like data collection errors, measurement inaccuracies, or genuine rare events, their presence can lead the algorithm to make incorrect diagnoses.

This is the reason why we, Machine Learning or Data Science practitioners, must always treat outliers with care.

In this short post, I will discuss several methods to efficiently identify and remove outliers from your data.

One of them is SVM, which I explored in this post.

Outliers are nonrepresentative data points in a dataset, or better, data points that deviate significantly from the rest. Despite their simple definition, detecting these anomalies is not always straightforward but first, let’s answer the following basic question.

Why do we want to detect outliers in a dataset?

There exist two answers to this question. The first reason for detecting outliers is that these…