EDA workflow

Chapter 22 11 code blocks 3537 words

Every previous chapter taught a tool. This one teaches the order you use them in.

Exploratory Data Analysis is the part of the job nobody writes a ticket for and everybody skips when they are in a hurry — and it is where almost every serious mistake gets caught. A model trained on a column that is 60% missing, a report averaging a value that has two populations inside it, a "record month" that is really a duplicated import: all of these are found in twenty minutes of EDA and all of them survive to production without it.

What EDA is for

EDA has three jobs, and they are worth separating because students tend to only do the third.

The six-step checklist

Give students a fixed order. Under pressure, a checklist beats intuition.

The order is not arbitrary. Step 2 changes how you read step 3; step 3 tells you which pairs in step 4 are worth looking at; step 6 comes last because you cannot tell an outlier from a typo until you know what normal looks like.

Step 1 — shape and types

import sqlite3
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_theme(style="whitegrid")
penguins = sns.load_dataset("penguins")

print(penguins.shape)          # (344, 7) — rows, columns
penguins.head()                # what does one row actually look like?
penguins.info()                # dtypes + non-null counts, in one view
penguins.describe()            # numeric summary
penguins.describe(include="object")   # categorical summary

(344, 7)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 344 entries, 0 to 343
Data columns (total 7 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   species            344 non-null    object 
 1   island             344 non-null    object 
 2   bill_length_mm     342 non-null    float64
 3   bill_depth_mm      342 non-null    float64
 4   flipper_length_mm  342 non-null    float64
 5   body_mass_g        342 non-null    float64
 6   sex                333 non-null    object 
dtypes: float64(4), object(3)
memory usage: 18.9+ KB

info() is the highest-value single call in pandas: it gives you the row count, every column name, every dtype and the non-null count all at once. Read it before anything else.

Step 2 — missing data

# how much is missing, per column, as a count and a percentage
miss = pd.DataFrame({
    "missing": penguins.isna().sum(),
    "pct":     (penguins.isna().mean() * 100).round(1)
}).sort_values("missing", ascending=False)
print(miss)

# which rows are incomplete?
penguins[penguins.isna().any(axis=1)]

                   missing  pct
sex                     11  3.2
bill_depth_mm            2  0.6
bill_length_mm           2  0.6
flipper_length_mm        2  0.6
body_mass_g              2  0.6
island                   0  0.0
species                  0  0.0

Then ask the question that matters more than the count: is it missing at random? If the missing rows are concentrated in one group, dropping them silently biases everything downstream.

# are the missing values spread evenly across species? penguins.assign(missing_sex=penguins.sex.isna()) \ .groupby("species")["missing_sex"].mean().round(3)

Step 3 — one variable at a time

Before any relationship, look at each column alone. For a numeric column you want the shape, the centre, the spread and the tails:

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.histplot(data=penguins, x="body_mass_g", kde=True, ax=axes[0])
sns.boxplot(data=penguins,  y="body_mass_g", ax=axes[1])
sns.ecdfplot(data=penguins, x="body_mass_g", ax=axes[2])
plt.tight_layout()

For a categorical column you want the levels and their balance:

penguins["species"].value_counts()
penguins["species"].value_counts(normalize=True).round(3)   # as proportions

sns.countplot(data=penguins, x="species", hue="species", legend=False)

Four things to write down for every numeric column: is it symmetric or skewed, is it unimodal or multimodal, is the range plausible, and how many distinct values does it have (a "numeric" column with 3 distinct values is really a category).

Step 4 — two variables

There are only three combinations, and each has a standard chart.

# numeric x numeric
sns.scatterplot(data=penguins, x="flipper_length_mm", y="body_mass_g", hue="species")

# numeric x categorical
sns.boxplot(data=penguins, x="species", y="body_mass_g", hue="species", legend=False)

# categorical x categorical — proportions, not raw counts
ct = pd.crosstab(penguins["island"], penguins["species"], normalize="index")
sns.heatmap(ct, annot=True, fmt=".2f", cmap="Blues")

Step 5 — many variables at once

# correlation matrix — numeric_only is required on pandas 2+
corr = penguins.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0, vmin=-1, vmax=1)

# every numeric pair at once, coloured by group
sns.pairplot(penguins, hue="species", diag_kind="kde", corner=True)

A pairplot with hue is often the single highest-information chart in an entire analysis. On the penguins data it shows in one glance that the three species separate almost perfectly on bill and flipper measurements — which is the whole finding.

Step 6 — outliers and data quality

An outlier is not automatically an error. Sort them into three buckets:

# the standard 1.5 x IQR rule — the same rule the boxplot whiskers use
q1, q3 = penguins["body_mass_g"].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
outliers = penguins[(penguins.body_mass_g < lo) | (penguins.body_mass_g > hi)]
print(f"{len(outliers)} outliers outside [{lo:.0f}, {hi:.0f}]")

# sentinel hunt: suspiciously round or repeated values
penguins["body_mass_g"].value_counts().head(10)

0 outliers outside [1750, 6550]

The rest of the data-quality sweep is quick and worth doing every time:

df.duplicated().sum() # exact duplicate rows df.duplicated(subset=["student_id", "course_id"]).sum() # business-key duplicates df["city"].str.strip().str.lower().nunique() # 'Tashkent ' vs 'tashkent' vs 'Tashkent' df.nunique() # constant columns (1) or ID-like columns (== len(df))

Which plot for which question The table to put on the classroom wall.

A full worked EDA

Twenty minutes on the titanic dataset, following the checklist end to end. This is the version to run live in class.

Steps 1–2: shape, types, missingness

titanic = sns.load_dataset("titanic")
print(titanic.shape)            # (891, 15) — one row is one passenger
titanic.info()

pd.DataFrame({"missing": titanic.isna().sum(),
              "pct": (titanic.isna().mean()*100).round(1)}) \
  .query("missing > 0").sort_values("missing", ascending=False)

(891, 15)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 15 columns):
 #   Column       Non-Null Count  Dtype   
---  ------       --------------  -----   
 0   survived     891 non-null    int64   
 1   pclass       891 non-null    int64   
 2   sex          891 non-null    object  
 3   age          714 non-null    float64 
 4   sibsp        891 non-null    int64   
 5   parch        891 non-null    int64   
 6   fare         891 non-null    float64 
 7   embarked     889 non-null    object  
 8   class        891 non-null    category
 9   who          891 non-null    object  
 10  adult_male   891 non-null    bool    
 11  deck         203 non-null    category
 12  embark_town  889 non-null    object  
 13  alive        891 non-null    object  
 14  alone        891 non-null    bool    
dtypes: bool(2), category(2), float64(2), int64(4), object(5)
memory usage: 80.7+ KB

columnmissingpctwhat to do

deck68877.2Too sparse to use as-is — drop it, or treat "unknown" as its own level age17719.9Substantial. Do not drop the rows — impute, or analyse with and without embarked / embark_town20.2Negligible — safe to drop those two rows

Note immediately that 891 − 177 = 714: any statement about age is based on 714 passengers, not 891. That sentence belongs in the report.

Step 3: one variable at a time

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.histplot(data=titanic, x="age", bins=30, kde=True, ax=axes[0])
sns.countplot(data=titanic, x="pclass", hue="pclass", legend=False, ax=axes[1])
sns.histplot(data=titanic, x="fare", bins=40, ax=axes[2])
axes[2].set_title("fare is heavily right-skewed")
plt.tight_layout()

titanic["survived"].value_counts(normalize=True).round(3)   # ~38% survived

Age is roughly symmetric with a bump of young children. Fare is severely right-skewed — a few very expensive tickets — so its mean is far above its median, and any "average fare" needs care.

Step 4: two variables — the actual question

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.barplot(data=titanic, x="sex",    y="survived", hue="sex",    legend=False, ax=axes[0])
sns.barplot(data=titanic, x="pclass", y="survived", hue="pclass", legend=False, ax=axes[1])
sns.boxplot(data=titanic, x="survived", y="age", hue="survived", legend=False, ax=axes[2])
axes[0].set_title("survival rate by sex")
axes[1].set_title("survival rate by class")
plt.tight_layout()

Step 5: many variables, and the interaction

# does class change the effect of sex? (it does — a lot)
sns.catplot(data=titanic, x="pclass", y="survived", hue="sex",
            kind="bar", height=4, aspect=1.3)

# the same thing as a grid of numbers
pd.crosstab(titanic["sex"], titanic["pclass"], values=titanic["survived"],
            aggfunc="mean").round(2)

The finding: survival is dominated by sex, and modulated by class. Women in first class survived at a far higher rate than women in third; men in third class fared worst of all. A single-variable summary would have missed the interaction entirely — which is the whole reason step 5 exists.

Step 6: outliers and quality

titanic[titanic["fare"] > 300][["pclass", "fare", "survived"]] # real, not errors (titanic["fare"] == 0).sum() # 15 zero fares — crew or staff? investigate titanic.duplicated().sum() The £512 fares are genuine first-class tickets, so they stay. The zero fares are the interesting anomaly — worth a footnote, and exactly the kind of thing that only surfaces because you looked.

EDA anti-patterns

Summary

• EDA has three jobs: understand the data, check the data, find the signal. Most people only do the third.
• Work the checklist in order: shape & types → missing → one variable → two variables → many variables → outliers.
• Always answer "what is one row?" first. After a JOIN the answer may have changed — that is fan-out from Chapter 5.
• info() is the single highest-value call in pandas: rows, dtypes and non-null counts at once.
• Missingness is a finding, not a chore. Measure how much, check whether it is random, and treat dropna()/fillna() as decisions you document.
• Report the median when a column is skewed, and never a mean without its n.
• Always colour by the obvious group — Simpson's paradox can reverse a pooled correlation completely.
• Sort outliers into impossible, sentinel and genuinely extreme. Only the first two get removed, and only with a reason.
• A barplot of a 0/1 column is a proportion chart — one of the most useful tricks here.
• The deliverable is the findings list, not the charts.

Next: Chapter 23 collects every Seaborn gotcha into one revision catalogue and closes Part II.