During my early days building e-commerce platforms, I closely monitored site performance metrics like a hawk: conversion rates, average cart values, and daily visits. One morning, I spotted something strange—a handful of customers with massive shopping carts who never checked out. At first, I brushed it off as normal abandonment, but curiosity got the better of me. By quickly running a simple outlier check (using the Interquartile Range for cart values), I realized that these few “weird” data points weren’t just random. It turned out there was a glitch in the checkout process when people selected certain shipping options. Fixing the issue not only recovered potential sales but also retained some of the platform’s most enthusiastic buyers, preventing churn.
That experience taught me that outliers can be more than statistical oddities; sometimes they’re the biggest clues about why customers leave. Whether it’s a high-value cart that never converts or a customer who suddenly stops visiting after months of regular purchases, those anomalies can reveal critical pain points or overlooked opportunities.
Table of Contents
How to Find Outliers (with Python Examples)
What Are Outliers?
An outlier is any data point that lies outside the general “pattern” of the dataset. In e-commerce, think of:
- A high-volume buyer who abruptly stops purchasing.
- A customer who places one gigantic order, dwarfing typical cart values.
- A small group of accounts with repeated login issues while everyone else is fine.
Univariate vs. Multivariate Outliers
- Univariate Outlier: Abnormal behavior in one feature (e.g., “Purchase Amount” far outside the normal range).
- Multivariate Outlier: A data point that looks normal in one dimension but stands out when you consider multiple features together (e.g., “High Browsing Time + Low Purchase Amount”).
Why should I remove outliers in these cases?
You may ask “these observations might be important even if they are outliers, why should I remove them?” and you would be right. In fact, in many cases, these outliers are the meat of the analysis.
That said, if you’re trying to derive insights from the majority of your data, outliers can skew your statistical measures, potentially leading to a significantly different outcome than you’d reach without them.
- Mean: Even a small number of outliers can pull the mean towards extremely high or low values.
- Variance and Standard Deviation: Outliers inflate the variance and standard deviation, potentially obscuring genuine trends.
- Correlation: A single outlier can heavily influence the correlation coefficient between two variables.
Because of these effects, data scientists often explore robust statistics like the median or the interquartile range (IQR) to mitigate the impact of outliers.
If you’re using your data to produce machine learning models, the existence of these outliers may also distort your models. Most machine learning algorithms, particularly those based on distance (e.g., k-Nearest Neighbors) or gradient descent (e.g., linear or logistic regression), can be sensitive to outliers. A few extreme data points can distort the decision boundaries or regression lines, leading to suboptimal models with high variance or incorrect predictions.
How to Find Outliers (with Python Examples)
Let’s consider a customer churn analysis as a practical example. When analyzing churn, you want to identify patterns that might signal declining customer engagement. Below are some practical methods you can implement in Python.
1. Statistical Methods
These approaches rely on numerical metrics (like mean, standard deviation, and percentiles) to identify data points that deviate significantly from the general distribution.
1.1. Z-Score
The Z-score measures how many standard deviations a data point is from the mean. Data points with Z-scores beyond a certain threshold (e.g., ±3) are often flagged as outliers.
import pandas as pd
import numpy as np
# Example dataset: Purchase amounts for each customer
# Notice a clear outlier in the fourth item
purchase_data = pd.DataFrame({
'customer_id': range(1, 11),
'purchase_amount': [50, 60, 55, 700, 58, 62, 55, 57, 59, 52]
})
# Calculate mean and std
mean_val = purchase_data['purchase_amount'].mean()
std_val = purchase_data['purchase_amount'].std()
# Compute Z-scores
purchase_data['z_score'] = (purchase_data['purchase_amount'] - mean_val) / std_val
# Flag outliers based on Z-score threshold
threshold = 3
purchase_data['is_outlier'] = purchase_data['z_score'].abs() > threshold
print(purchase_data)Pros: Simple, fast, and intuitive.
Cons: Assumes the data follows a roughly normal distribution, which isn’t always true (especially in e-commerce).
1.2 Interquartile Range (IQR)
The IQR is calculated as Q3 – Q1, where Q1 is the 25th percentile and Q3 is the 75th percentile of the data. Outliers are often defined as points lying below Q1 – 1.5*IQR or above Q3 + 1.5*IQR.
Q1 = purchase_data['purchase_amount'].quantile(0.25)
Q3 = purchase_data['purchase_amount'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
purchase_data['is_outlier_iqr'] = (
(purchase_data['purchase_amount'] < lower_bound) |
(purchase_data['purchase_amount'] > upper_bound)
)
print(purchase_data[['customer_id', 'purchase_amount', 'is_outlier_iqr']])
Pros: More robust to skewed distributions.
Cons: The 1.5 multiplier is somewhat arbitrary.
2. Visualization Techniques
These methods use graphical representations (such as box plots or scatter plots) to quickly spot anomalies in the data by visually highlighting unusual patterns or clusters.
- Box Plot: Quickly shows the median, quartiles, and potential outliers.
- Scatter Plot: Especially helpful for multivariate anomalies (e.g., “Time on Site” vs. “Amount Spent”).
import matplotlib.pyplot as plt
plt.boxplot(purchase_data['purchase_amount'])
plt.title("Box Plot of Purchase Amounts")
plt.ylabel("Amount")
plt.show()
For the above dataset, the boxplot may look something like this:
You can clearly see there is an outlier at the very top end of the box plot.
A box plot can highlight unusual spending patterns that might correlate with impending churn. For instance, if a once-consistent customer’s purchase amount skyrockets or plummets, they might be at risk.
3. Algorithmic Approaches
These advanced models (like Isolation Forest, DBSCAN, or Local Outlier Factor) automate the process of detecting outliers by analyzing density, distance, or isolation measures to flag points that differ from the majority.
Let’s use Isolation Forest as an example. This algorithm isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. Outliers are easier to isolate compared to normal points because they require fewer random splits to be isolated. Ultimately, the algorithm produces:
- A label: +1 for inliers, -1 for outliers.
- A decision function score: A continuous value indicating the degree of “outlierness.” The lower (more negative) the score, the more anomalous the data point.
Sample code:
# Initialize Isolation Forest
# 'contamination' is an estimate of the proportion of outliers in the data
iso_forest = IsolationForest(contamination=0.1, random_state=42)
# Fit the model to the 'purchase_amount' column and get predictions (+1 for inlier, -1 for outlier)
purchase_data['if_label'] = iso_forest.fit_predict(purchase_data[['purchase_amount']])
# Get the outlier scores (the lower the score, the more anomalous the point)
purchase_data['if_score'] = iso_forest.decision_function(purchase_data[['purchase_amount']])
print(purchase_data)
Assume you get a DataFrame that looks something like this (your exact results may vary):
customer_id purchase_amount if_label if_score
0 1 50 1 0.146351
1 2 60 1 0.113784
2 3 55 1 0.130894
3 4 700 -1 -0.543597
4 5 58 1 0.121675
5 6 62 1 0.108972
6 7 55 1 0.130894
7 8 57 1 0.125098
8 9 59 1 0.117879
9 10 52 1 0.138312
In this hypothetical example, the row with if_label = -1 (in this case, customer 4 with a purchase_amount of 700) is flagged as an outlier.
Tools for Detecting Outliers
Below is a concise overview of common outlier detection tools, plus tips on when each one is your best choice:
- Python: A flexible option for large-scale or automated outlier detection, perfect if you need to integrate machine learning and handle big data pipelines.
- R: Ideal for specialized statistical models and advanced visualizations—use R when your analysis hinges on deep statistical techniques or cutting-edge research packages.
- Good ol’ Excel: Already in most people’s toolkits, Excel is handy for smaller datasets, simple charts, and quick, no-code checks on any suspicious data points.
Handling Outliers
So you’ve found these outliers. Great! Now, what can you do with them?
1. Removal vs. Retention
Remove Them: If they’re genuine errors, removing them can reduce data skew.
Retain Them: In churn analysis like what I had earlier, those “strange” data points might be exactly the warning signs you need.
2. Modify (Trimming or Winsorizing)
You might transform outliers by capping them at a certain percentile. This is a middle ground between removing valuable but extreme observations and keeping them fully intact.
3. Investigate
Outliers aren’t always bad data. Sometimes they expose crucial friction points—like a broken payment method or a missed marketing opportunity. For example, if a regular buyer abruptly drops off, that’s a cause for immediate follow-up.
Continue your journey
Outliers can distort your metrics, but they can also offer the most direct insight into issues in your data, or even the entire system. By using simple statistical checks like Z-scores or IQR—and optionally leveling up with advanced algorithms—you can quickly spot those “sore thumb” data points. From there, you can investigate underlying issues, refine your workflows or models, and ensure you’re not missing critical signals that could influence the success of your data-driven strategies.
In my experience, shining a spotlight on anomalies has helped me detect hidden bugs, streamline workflows, and uncover strategic insights that might otherwise remain invisible. If you’re ready to elevate your data analysis skills and turn these insights into tangible results, consider exploring Udacity’s data analysis-related Nanodegree Programs—you’ll master advanced techniques in statistical modeling, data visualization, and analytics, paving the way for your data-driven projects to become measurable success stories. Here are some programs that might be of interest (ordered from foundational to advanced):
- Business Analytics Nanodegree program: In this program, you’ll master data fundamentals—spreadsheets, statistical analysis, financial modeling, and dashboards using Excel, SQL, and Tableau—to drive informed business decisions, culminating in real-world projects like NYSE S&P analysis and digital music store optimization.
- Data Analyst Nanodegree program: In this intermediate-level program, you’ll clean, explore, and visualize data using Jupyter Notebook, NumPy, pandas, and Matplotlib, while practicing iterative analysis, data imputation, and appropriate encodings on real-world projects.
- Data Scientist Nanodegree program: In this program, you’ll learn to solve real-world data science challenges—like recommending movies, forecasting housing prices, or predicting healthcare outcomes—by building projects that mirror industry work and integrate the latest AI tools.
