Heatmap & correlation

Chapter 18 34 code blocks 8525 words

Every plot in the previous chapters took a long (tidy) DataFrame and let seaborn do the grouping. This chapter breaks that pattern. sns.heatmap() and sns.clustermap() are seaborn's two matrix functions: they do not group anything, they do not accept x=, y= or hue=, and they will not read a tidy table. They take a rectangle of numbers and paint each cell with a colour.

That single structural fact is where almost every beginner gets stuck, so we will spend real time on it: how to reshape ordinary data into matrix form with pivot, pivot_table, crosstab and corr. Then we cover the most common matrix of all — the correlation matrix — and, just as importantly, how to read one without fooling yourself.

Heatmap basics and the data shape it needs

A heatmap encodes a numeric value as colour. The row index becomes the y-axis, the column index becomes the x-axis, and each cell's value becomes a colour drawn from a colormap. That is the entire idea.

The minimal call

import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

sns.set_theme(style="white")

# flights is long/tidy: one row per (year, month) with a passenger count
flights = sns.load_dataset("flights")
print(flights.head())
#    year month  passengers
# 0  1949   Jan         112
# 1  1949   Feb         118

# reshape long -> wide: rows = month, columns = year, cells = passengers
matrix = flights.pivot(index="month", columns="year", values="passengers")

sns.heatmap(matrix)
plt.show()

   year month  passengers
0  1949   Jan         112
1  1949   Feb         118
2  1949   Mar         132
3  1949   Apr         129
4  1949   May         121

The canonical heatmap: flights pivoted to month × year. Two patterns jump out at once — traffic grows every year, and every summer peaks.

The canonical flights heatmap shows two things instantly that a table of 144 numbers hides: air traffic grows steadily from 1949 to 1960 (colour brightens left to right) and every year has a summer peak in July–August (a bright horizontal band). Pattern detection by eye is what heatmaps are for.

What "matrix form" means precisely

If you pass a plain NumPy array you lose the row and column names, because a NumPy array has no index. Prefer a DataFrame so the tick labels are meaningful.

Accepted arguments, at a glance

sns.heatmap(
    data,              # 2-D DataFrame or ndarray - the ONLY positional arg
    annot=False,       # write the numbers inside the cells
    fmt=".2g",         # format string used for the annotations
    cmap=None,         # colormap name or object
    center=None,       # value that maps to the middle of a diverging map
    vmin=None, vmax=None,   # clamp the colour scale
    robust=False,      # use 2nd/98th percentiles instead of min/max
    linewidths=0, linecolor="white",
    cbar=True, cbar_kws=None,
    square=False,      # force cells to be square
    xticklabels="auto", yticklabels="auto",
    mask=None,         # boolean array; True cells are not drawn
    ax=None,
)

Key options: annot, fmt, cmap, vmin/vmax/center, linewidths, cbar

annot and fmt — printing the numbers

annot=True writes each cell's value on top of the colour. This turns the heatmap into a coloured table, which is often what a report actually needs. The catch is legibility: text shrinks as the matrix grows.

flights = sns.load_dataset("flights")
matrix = flights.pivot(index="month", columns="year", values="passengers")

fig, ax = plt.subplots(figsize=(10, 6))
sns.heatmap(
    matrix,
    annot=True,
    fmt="d",              # integers, no decimal point
    cmap="viridis",
    linewidths=0.5,
    ax=ax,
)
plt.show()

fmt is a standard Python format spec applied to each number. Get it wrong and you get a ValueError — most commonly fmt="d" on floats.

Annotation text is styled through annot_kws:

sns.heatmap(matrix, annot=True, fmt="d",
            annot_kws={"size": 8, "weight": "bold"})

You can also annotate with different values from the ones you colour by — pass a second array to annot. This is how you colour by p-value and label with the effect size, for instance:

# colour by correlation, but print the sample size in each cell
sns.heatmap(corr, annot=counts, fmt="d", cmap="coolwarm", center=0)

cmap — choosing a colormap that does not lie

Colormap choice is not decoration; it determines what the reader concludes. Three families matter.

Perceptual uniformity. A colormap is perceptually uniform when equal steps in the data produce equal-looking steps in colour. viridis, rocket and mako are engineered this way. The old jet/rainbow map is not: it has a bright cyan-yellow band that the eye reads as a sharp edge even when the data changes smoothly, and a dark blue-to-cyan stretch that compresses real differences into invisibility. Rainbow maps invent boundaries that are not in the data. Do not use them.

Colour-blind safety. Roughly 1 in 12 men has some form of red–green colour vision deficiency. viridis, mako and cividis survive that; a red–green diverging map does not. For diverging data prefer RdBu_r or seaborn's vlag (blue–red) over RdYlGn. A sequential map that also varies in lightness stays readable in greyscale and in photocopies.

Append _r to any matplotlib colormap name to reverse it: "RdBu" runs red→blue, "RdBu_r" runs blue→red, which is what you want for correlations so that positive is warm.

vmin, vmax, center — controlling the scale

By default the colour scale stretches from the data's minimum to its maximum. That is convenient and dangerous.

# correlations: always pin the scale so charts are comparable
sns.heatmap(corr, cmap="coolwarm", center=0, vmin=-1, vmax=1)

Three related knobs:

• vmin / vmax — hard floor and ceiling of the colour scale. Values outside are clipped to the end colours.
• center — the value that maps to the middle colour of a diverging map. For correlations that is 0; for a percent-change matrix it is also 0; for a ratio matrix it might be 1. Setting center alone also switches seaborn's default colormap to a diverging one.
• robust=True — sets the limits from the 2nd and 98th percentiles instead of min/max, so one extreme outlier does not flatten the whole picture into a single colour. Useful for exploration; state it in the caption, because it hides the extremes.

linewidths, square, cbar

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(
    matrix,
    cmap="mako",
    linewidths=0.5,          # thin gutters between cells
    linecolor="white",
    square=True,             # cells are squares, not rectangles
    cbar=True,
    cbar_kws={"label": "passengers", "shrink": 0.8},
    ax=ax,
)
ax.set_title("Air passengers by month and year")
plt.tight_layout()
plt.show()

linewidths — small gutters (0.5) separate cells and make a small matrix easier to read. On a 200×200 matrix set it to 0: the lines would consume more pixels than the data. square=True — enforces a 1:1 aspect ratio. Right for correlation matrices (which are square in meaning as well as shape); wrong when you have 12 rows and 60 columns. cbar=False — drop the colour bar. Sensible when annot=True already prints every number, or when several heatmaps in a grid should share one bar. cbar_kws — passed to plt.colorbar: {"label": ...}, {"shrink": 0.5}, {"orientation": "horizontal"}, {"ticks": [-1, 0, 1]}. xticklabels / yticklabels — False hides them, an integer n shows every n-th label, a list supplies your own. Essential for large matrices.

The correlation matrix

The overwhelmingly most common heatmap in practice is a correlation matrix: every numeric column correlated against every other, drawn as a square grid. It is the standard first look at a new dataset.

Building it

penguins = sns.load_dataset("penguins")

# penguins has species, island, sex - all non-numeric
# numeric_only=True is REQUIRED in pandas 2.x
corr = penguins.corr(numeric_only=True)
print(corr.round(2))

                   bill_length_mm  ...  body_mass_g
bill_length_mm               1.00  ...         0.60
bill_depth_mm               -0.24  ...        -0.47
flipper_length_mm            0.66  ...         0.87
body_mass_g                  0.60  ...         1.00

[4 rows x 4 columns]

A correlation matrix with center=0 and a diverging colormap. Red is positive, blue negative, pale is near zero. The diagonal is always 1.00 and carries no information.

The result for penguins (Pearson, rounded to two decimals):

Read a few cells. Flipper length and body mass correlate at 0.87 — longer flippers, heavier bird, strongly. Bill depth correlates negatively with both, at −0.58 and −0.47, which is the Simpson's-paradox trap discussed below. The diagonal is 1.00 everywhere, because every variable correlates perfectly with itself, and the matrix is a mirror image across that diagonal.

The standard plot

corr = penguins.corr(numeric_only=True)

fig, ax = plt.subplots(figsize=(7, 6))
sns.heatmap(
    corr,
    annot=True,
    fmt=".2f",
    cmap="coolwarm",     # diverging
    center=0,            # white sits exactly at r = 0
    vmin=-1, vmax=1,     # fixed scale, comparable across charts
    square=True,
    linewidths=0.5,
    cbar_kws={"label": "Pearson r", "shrink": 0.8},
    ax=ax,
)
ax.set_title("Penguin measurements: correlation matrix")
plt.tight_layout()
plt.show()

Choosing the method

corr_pearson = penguins.corr(numeric_only=True) # default corr_spearman = penguins.corr(method="spearman", numeric_only=True) corr_kendall = penguins.corr(method="kendall", numeric_only=True)

DataFrame.corr handles missing values pairwise: for each pair of columns it drops rows where either is NaN. Different cells can therefore be computed from different numbers of rows, which is worth knowing when your data has heavy, uneven missingness.

Correlating against one target

In a modelling context you often care about one column only. Sort it and plot it as a single-column heatmap:

corr = penguins.corr(numeric_only=True)
target = corr[["body_mass_g"]].drop(index="body_mass_g").sort_values(
    "body_mass_g", ascending=False
)

fig, ax = plt.subplots(figsize=(3, 4))
sns.heatmap(target, annot=True, fmt=".2f", cmap="coolwarm",
            center=0, vmin=-1, vmax=1, cbar=False, ax=ax)
ax.set_title("Correlation with body mass")
plt.tight_layout()
plt.show()

Note the double brackets: corr[["body_mass_g"]] returns a one-column DataFrame (2-D, works), whereas corr["body_mass_g"] returns a Series (1-D, raises an error).

Including categorical variables

corr ignores object and category columns entirely. If you need them, encode them first — and encode honestly.

titanic = sns.load_dataset("titanic")

# binary categories -> 0/1 is legitimate; the resulting r is a point-biserial correlation
df = titanic[["survived", "pclass", "age", "fare", "sibsp", "parch"]].copy()
df["is_male"] = (titanic["sex"] == "male").astype(int)

corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm",
            center=0, vmin=-1, vmax=1, square=True)
plt.tight_layout()
plt.show()

Reading a correlation matrix correctly

A correlation coefficient r is one number between −1 and +1 summarising how two variables move together. Compressing a scatterplot into one number throws away a great deal, and knowing exactly what was thrown away is the difference between analysis and superstition.

Pearson r measures LINEAR association only

Pearson's r is the covariance of two variables divided by the product of their standard deviations. It answers one question: how well does a straight line describe the relationship? Anything a straight line cannot capture is invisible to it.

x = np.linspace(-3, 3, 200)
y = x ** 2                       # a perfect parabola - zero randomness
print(np.corrcoef(x, y)[0, 1])   # ~ 0.0

-1.9203276205777008e-16

Here y is a deterministic function of x: knowing x tells you y exactly. Yet r ≈ 0, because the best-fitting straight line through a symmetric parabola is flat. A correlation of zero means "no linear relationship", never "no relationship".

Correlation is not causation

If A and B correlate, at least five explanations remain open: A causes B; B causes A; a third variable C causes both (confounding); the association is an artefact of how the sample was selected (collider bias); or it is chance, which becomes likely once you test many pairs at once. A 10×10 matrix contains 45 distinct correlations, so a few sizeable ones will appear in pure noise.

The penguins matrix illustrates confounding directly. Bill depth correlates −0.58 with flipper length across the whole dataset. Split by species and the sign flips: within every species, deeper bills go with longer flippers. The aggregate negative sign exists only because Gentoo penguins are large-bodied with shallow bills and dominate one end of the range. This is Simpson's paradox, and no heatmap can warn you about it — only conditioning on the grouping variable can.

for sp, g in penguins.groupby("species"):
    r = g["bill_depth_mm"].corr(g["flipper_length_mm"])
    print(sp, round(r, 2))
# Adelie     0.31
# Chinstrap  0.58
# Gentoo     0.71
# ... yet overall: -0.58

Adelie 0.31
Chinstrap 0.58
Gentoo 0.71

The diagonal is noise

Every cell on the main diagonal is exactly 1.00 by definition. It carries zero information, yet on a vmin=-1, vmax=1 scale it is the most saturated colour on the chart and drags the eye straight to it. Masking it (see the next section) removes a distraction and lets the remaining colours use more of the scale.

The matrix is symmetric — half is redundant

corr(A, B) = corr(B, A), so the upper triangle mirrors the lower. A k-variable matrix shows k² cells but contains only k(k−1)/2 distinct facts. For k = 10 that is 100 cells carrying 45 pieces of information — 55% visual redundancy. This is the whole motivation for masking one triangle.

Pearson vs Spearman vs Kendall

Spearman is Pearson applied to the ranks. Because ranking destroys distances, a single extreme outlier can no longer dominate, and any monotonic relationship — including y = x³ or y = ex — comes out as exactly 1.0.

x = np.arange(1, 101)
y = np.exp(x / 20)                      # monotonic but strongly curved
s = pd.Series(x); t = pd.Series(y)
print(s.corr(t))                        # pearson  ~ 0.79
print(s.corr(t, method="spearman"))     # spearman = 1.00

0.8581420519497872
0.9999999999999999

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
for ax, method in zip(axes, ["pearson", "spearman"]):
    c = penguins.corr(method=method, numeric_only=True)
    sns.heatmap(c, annot=True, fmt=".2f", cmap="coolwarm",
                center=0, vmin=-1, vmax=1, square=True,
                cbar=False, ax=ax)
    ax.set_title(method.capitalize())
plt.tight_layout()
plt.show()

How big is big? A rough magnitude table

|r|Conventional labelr² (variance explained)

0.0 – 0.3weak / negligible0 – 9% 0.3 – 0.7moderate9 – 49% 0.7 – 0.9strong49 – 81% 0.9 – 1.0very strong (check for leakage or duplication)81 – 100%

Squaring r is the more honest summary: r² is the share of variance in one variable linearly explained by the other. An r of 0.5 sounds like "half", but explains only 25% of the variance.

Sign and magnitude are independent. −0.85 is a stronger relationship than +0.30. Students routinely rank correlations by signed value rather than absolute value; when sorting, use corr.abs().

Very high correlations are a warning, not a prize

An |r| above roughly 0.95 between two features usually means one of three things: the columns are duplicates or unit conversions of each other (height in cm and in inches); one leaks the target (titanic["alive"] vs titanic["survived"] — the same fact twice); or they are near-collinear, which destabilises linear-model coefficients. Correlation heatmaps are used in feature selection precisely to find and drop these.

Masking the upper triangle

mask takes a boolean array the same shape as the data. Wherever the mask is True, the cell is left blank. Since half the correlation matrix is redundant, masking one triangle roughly doubles the information density.

corr = penguins.corr(numeric_only=True)

# np.triu = upper triangle; k=1 keeps the diagonal visible
mask = np.triu(np.ones_like(corr, dtype=bool), k=1)

fig, ax = plt.subplots(figsize=(7, 6))
sns.heatmap(
    corr, mask=mask,
    annot=True, fmt=".2f",
    cmap="coolwarm", center=0, vmin=-1, vmax=1,
    square=True, linewidths=0.5,
    cbar_kws={"shrink": 0.75},
    ax=ax,
)
plt.tight_layout()
plt.show()

A correlation matrix is symmetric, so half of it is duplicated ink. Masking the upper triangle removes the repetition and the useless diagonal.

The k offset controls whether the diagonal survives:

np.ones_like(corr, dtype=bool) builds a same-shaped array of True; np.triu zeroes out everything below the chosen diagonal, leaving True only where you want blanks. Building the mask from corr itself guarantees the shapes match.

The mask is a general mechanism, not a triangle-only trick. Any boolean condition works — for example, hide correlations too weak to care about:

# show only |r| >= 0.3, and only the lower triangle
weak = corr.abs() < 0.3
upper = np.triu(np.ones_like(corr, dtype=bool), k=0)
sns.heatmap(corr, mask=(weak | upper), annot=True, fmt=".2f",
            cmap="coolwarm", center=0, vmin=-1, vmax=1, square=True)

It is also the standard way to render missing data: mask=df.isna() leaves NaN cells blank rather than letting them adopt an accidental colour.

Reshaping data into matrix form

This is the section to reread when a heatmap refuses to work. Four tools cover essentially every case.

1. df.pivot() — reshape only, no aggregation

flights = sns.load_dataset("flights")

matrix = flights.pivot(index="month", columns="year", values="passengers")
print(matrix.shape)   # (12, 12)

(12, 12)

Read it as: index becomes the rows (y-axis), columns becomes the columns (x-axis), values fills the cells. pivot performs no arithmetic — it only rearranges. It therefore requires each (index, columns) pair to appear exactly once. If a pair repeats you get ValueError: Index contains duplicate entries, cannot reshape, which is pandas telling you it does not know which of the duplicate values to put in the cell. That is your signal to switch to pivot_table.

2. pd.pivot_table() — reshape and aggregate

When several rows land in the same cell you must say how to combine them. That is aggfunc.

titanic = sns.load_dataset("titanic")

# mean survival rate by class and sex
rates = pd.pivot_table(
    titanic,
    index="class",
    columns="sex",
    values="survived",
    aggfunc="mean",         # default is "mean"
)
print(rates.round(3))
#         female   male
# First    0.968  0.369
# Second   0.921  0.157
# Third    0.500  0.135

sns.heatmap(rates, annot=True, fmt=".1%", cmap="RdYlBu",
            vmin=0, vmax=1, linewidths=0.5,
            cbar_kws={"label": "survival rate"})
plt.tight_layout()
plt.show()

sex     female   male
class                
First    0.968  0.369
Second   0.921  0.157
Third    0.500  0.135

Useful pivot_table arguments:

# tips: average bill by day and party size, missing combos as 0
tips = sns.load_dataset("tips")
m = pd.pivot_table(tips, index="day", columns="size",
                   values="total_bill", aggfunc="mean", fill_value=0)
sns.heatmap(m, annot=True, fmt=".1f", cmap="rocket_r", linewidths=0.5)
plt.tight_layout()
plt.show()

3. pd.crosstab() — counts and proportions of two categoricals

crosstab takes two Series (not a DataFrame plus column names) and returns a contingency table of counts.

# raw counts
counts = pd.crosstab(titanic["class"], titanic["embark_town"])
print(counts)
#         Cherbourg  Queenstown  Southampton
# First          85           2          127
# Second         17           3          164
# Third          66          72          353

sns.heatmap(counts, annot=True, fmt="d", cmap="Blues", linewidths=0.5)
plt.tight_layout()
plt.show()

embark_town  Cherbourg  Queenstown  Southampton
class                                          
First               85           2          127
Second              17           3          164
Third               66          72          353

normalize converts counts to proportions and is where most of the analytical value lives:

props = pd.crosstab(titanic["class"], titanic["embark_town"], normalize="index")
sns.heatmap(props, annot=True, fmt=".1%", cmap="Blues",
            vmin=0, vmax=1, linewidths=0.5)
plt.tight_layout()
plt.show()

crosstab also aggregates a third variable, which makes it a compact alternative to pivot_table:

pd.crosstab(titanic["class"], titanic["sex"],
            values=titanic["age"], aggfunc="mean").round(1)

1. df.corr() — the special case

A correlation matrix is already square and already labelled on both axes, so no reshaping is required. It is a matrix-producing function like the other three.

The reverse direction: melt and stack

Going from matrix back to long is occasionally needed — to feed a scatterplot, or to build a ranked table of the strongest pairs.

long_again = matrix.reset_index().melt(
    id_vars="month", var_name="year", value_name="passengers"
)

# top correlated pairs, excluding self-pairs and duplicates
corr = penguins.corr(numeric_only=True)
mask = np.triu(np.ones_like(corr, dtype=bool), k=0)
pairs = corr.mask(mask).stack().sort_values(key=abs, ascending=False)
print(pairs.head())

body_mass_g        flipper_length_mm    0.871202
flipper_length_mm  bill_length_mm       0.656181
body_mass_g        bill_length_mm       0.595110
flipper_length_mm  bill_depth_mm       -0.583851
body_mass_g        bill_depth_mm       -0.471916
dtype: float64

clustermap

sns.clustermap() is a heatmap plus hierarchical clustering. It computes distances between rows and between columns, builds dendrograms, and then reorders the matrix so that similar rows sit next to each other and similar columns sit next to each other. Block structure that was scattered across an arbitrary alphabetical ordering becomes visible as contiguous patches.

corr = penguins.corr(numeric_only=True)

g = sns.clustermap(
    corr,
    cmap="coolwarm", center=0, vmin=-1, vmax=1,
    annot=True, fmt=".2f",
    figsize=(8, 8),
    linewidths=0.5,
)
plt.show()

clustermap reorders rows and columns so similar variables sit together, and draws the dendrogram that produced the order. Note the axis order is not your input order.

Two structural differences from heatmap:

• It is figure-level. It creates and owns its own figure, so ax= is not accepted and it cannot be placed inside a plt.subplots() grid. It returns a ClusterGrid whose parts you reach through g.ax_heatmap, g.ax_row_dendrogram, g.ax_col_dendrogram and g.ax_cbar.
• It cannot handle NaN. The clustering step raises ValueError: The condensed distance matrix must contain only finite values. Drop or impute first.

Important arguments

A complete example with row colours

penguins = sns.load_dataset("penguins").dropna()

# the numeric measurements, one row per bird
X = penguins[["bill_length_mm", "bill_depth_mm",
              "flipper_length_mm", "body_mass_g"]]

# map species to a colour stripe so we can check the clusters
species = penguins["species"]
lut = dict(zip(species.unique(), sns.color_palette("Set2", 3)))
row_colors = species.map(lut)

g = sns.clustermap(
    X,
    z_score=1,              # standardise each column - units differ wildly
    cmap="vlag", center=0,
    method="ward", metric="euclidean",
    row_colors=row_colors,
    yticklabels=False,      # 333 rows - labels would be unreadable
    figsize=(8, 10),
    dendrogram_ratio=(0.12, 0.15),
)
g.ax_heatmap.set_xlabel("")
plt.show()

Without z_score=1 this plot would be meaningless: body mass is measured in thousands of grams while bill depth is around 15 mm, so Euclidean distance would be almost entirely body mass. Standardising puts every column on the same footing. The species colour stripe then answers the real question — do the measurement clusters recover the species labels? Largely yes, with Adelie and Chinstrap overlapping.

order = g.dendrogram_row.reordered_ind
print(X.index[order][:10])          # the labels as drawn, top to bottom

# cut the tree into k groups
from scipy.cluster.hierarchy import fcluster
labels = fcluster(g.dendrogram_row.linkage, t=3, criterion="maxclust")

Index([226, 312, 238, 270, 252, 342, 308, 338, 241, 314], dtype='int64')

When to use which

Use heatmap whenUse clustermap when

The axis order is meaningful (months, years, ordered classes)The axis order is arbitrary and you want structure discovered You need the plot inside a subplot gridA standalone figure is fine The data contains NaNThe data is complete You want maximum control over layoutYou want grouping and dendrograms for free

Clustering the flights matrix would be a mistake: months and years have a natural order, and shuffling them to put similar months together destroys the seasonal reading that makes the chart worth drawing.

Gotchas

1. df.corr() without numeric_only in pandas 2.x

penguins = sns.load_dataset("penguins")

# corr = penguins.corr()                    # ValueError in pandas 2.x
corr = penguins.corr(numeric_only=True)     # correct

In pandas 1.x, corr() silently dropped non-numeric columns. Pandas 1.5 deprecated that with a FutureWarning; pandas 2.0 changed the default to numeric_only=False, so penguins.corr() now tries to correlate the string column species and raises ValueError: could not convert string to float: 'Adelie'. The same applies to .mean(), .sum(), .median() and friends on mixed DataFrames. Any tutorial written before 2023 will hit this.

The explicit alternative, which is arguably better practice because it documents what you meant:

corr = penguins.select_dtypes(include="number").corr()

1. Forgetting center=0 on a diverging map

A diverging colormap promises "the middle colour is the neutral value". If your correlations run from −0.1 to +0.9 and you omit center=0, seaborn puts white at +0.4, so a genuinely positive 0.2 renders in the blue "negative" range. The chart contradicts the data. With correlations, always set center=0 and preferably vmin=-1, vmax=1 as well.

3. annot=True on a large matrix

Twelve variables give 144 cells and the text is already cramped; thirty variables give 900 unreadable smudges. Rules of thumb: annotate up to roughly 15×15, drop annotations above that and let colour do the work, and use fmt=".2f" rather than the default ".2g" so that all the numbers have the same width and the grid stays visually aligned.

# large matrix: no annotation, thinner labels, no cell borders
fig, ax = plt.subplots(figsize=(12, 10))
sns.heatmap(big_corr, cmap="coolwarm", center=0, vmin=-1, vmax=1,
            annot=False, linewidths=0, square=True,
            xticklabels=2, yticklabels=2, ax=ax)   # every 2nd label
plt.tight_layout()

1. Missing vmin/vmax across multiple charts

Already covered under the options, and worth repeating because it survives code review so often: without fixed limits, each subplot invents its own scale and the shared colour bar becomes a lie. Any faceted or before/after comparison must pin vmin and vmax.

5. Correlation of a constant column is NaN

df = pd.DataFrame({"a": [1, 2, 3, 4], "b": [5, 5, 5, 5], "c": [2, 4, 6, 8]})
print(df.corr())
#      a   b    c
# a  1.0 NaN  1.0
# b  NaN NaN  NaN
# c  1.0 NaN  1.0

     a   b    c
a  1.0 NaN  1.0
b  NaN NaN  NaN
c  1.0 NaN  1.0

Pearson's r divides by each variable's standard deviation. A constant column has sd = 0, so the calculation is 0/0. The result is a NaN row and column that render as blank stripes across your heatmap. The fix is to drop zero-variance columns before correlating — they carry no information for any model either:

num = df.select_dtypes("number") num = num.loc[:, num.std(numeric_only=True) > 0] corr = num.corr()

The same appears with near-constant columns (one distinct value plus a handful of exceptions), where r is defined but wildly unstable.

6. Counts vs normalised proportions

Restating the crosstab warning as a general rule: a heatmap of raw counts shows where the volume is, not where the effect is. Rare categories can carry the strongest signal and still look invisible. Decide deliberately whether the reader needs "how many" or "what share", and label the colour bar accordingly.

7. Clustermap reorders — the axis order is not your order

Covered above; listed here because it belongs on the checklist. If you annotate a clustermap by joining external labels positionally, you will attach the wrong labels to the wrong rows.

8. Top and bottom rows cut in half

Matplotlib 3.1.0 and 3.1.1 shipped a regression that clipped the first and last rows of a heatmap to half height. It is the single most-asked seaborn question of that era. Modern matplotlib is fixed, but the workaround still appears everywhere:

ax = sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0)

# only needed on matplotlib 3.1.0 / 3.1.1
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)

The related, still-current problem is labels running off the canvas. Always finish with plt.tight_layout(), and rotate long tick labels:

ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha="right")
ax.set_yticklabels(ax.get_yticklabels(), rotation=0)
plt.tight_layout()

1. sns.load_dataset() needs internet

load_dataset downloads a CSV from the seaborn-data GitHub repository on every call unless it is cached. Offline — in an exam room, on a locked-down machine, behind a proxy — it raises URLError or a timeout. Two workarounds:

# A) point seaborn at a local folder of the CSVs
#    clone https://github.com/mwaskom/seaborn-data once, then:
flights = sns.load_dataset("flights", data_home="C:/data/seaborn-data", cache=True)

# B) skip seaborn entirely and read the CSV yourself
flights = pd.read_csv("C:/data/seaborn-data/flights.csv")

By default seaborn caches downloads under ~/seaborn-data, so once a dataset has been fetched on a machine it keeps working offline. Downloading the handful of teaching datasets in advance is worth the two minutes.

10. Category ordering is alphabetical unless you say otherwise

After a pivot, the index and columns are sorted — which is why flights.pivot(...) happens to come out in calendar order (its month column is an ordered Categorical) while a plain string column would come out April, August, December. Reindex explicitly when order matters:

order = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] matrix = matrix.reindex(order)

1. Cells you cannot see are not zero

An empty combination in a pivot becomes NaN and renders as blank (or as whatever facecolor the axes has). A reader may well interpret blank as zero. Say so in the caption, or use fill_value deliberately when zero really is the right answer.

Practice set

# ---------------------------------------------------------------
# Chapter 18 practice - heatmap, correlation, clustermap
# Load with sns.load_dataset(...). Answer the question in a comment
# under each plot: what does the chart actually show?
# ---------------------------------------------------------------

# 1. flights: pivot to month x year and draw the canonical heatmap with
#    cmap="viridis" and a labelled colour bar. Then draw it again with
#    annot=True, fmt="d". Which version is easier to read, and why?

# 2. penguins: build the Pearson correlation matrix with
#    numeric_only=True. Plot it with annot=True, fmt=".2f",
#    cmap="coolwarm", center=0, vmin=-1, vmax=1, square=True.
#    Name the strongest positive pair and the strongest negative pair.

# 3. Repeat exercise 2 with the upper triangle AND the diagonal masked
#    (np.triu with k=0). Count how many distinct correlations remain
#    and check that it equals k*(k-1)/2.

# 4. penguins: draw Pearson and Spearman matrices as two panels of one
#    figure with a shared vmin/vmax. Which pair changes the most, and
#    what does the scatterplot of that pair look like?

# 5. titanic: use pd.pivot_table to get mean survival by class and sex,
#    then heatmap it with fmt=".1%", vmin=0, vmax=1. Write one sentence
#    stating what the chart shows about class and sex.

# 6. titanic: pd.crosstab of class vs embark_town. Draw three heatmaps -
#    raw counts, normalize="index", normalize="columns". Explain in a
#    comment what different question each one answers.

# 7. tips: pivot_table of mean tip percentage (tip / total_bill) by day
#    and time. Handle the empty cells explicitly and justify your choice
#    of fill_value=0 versus leaving NaN.

# 8. penguins: clustermap of the four numeric measurements with
#    z_score=1, method="ward", and a row_colors stripe for species.
#    Then print g.dendrogram_row.reordered_ind and confirm that the
#    drawn row order differs from the DataFrame order.

# Bonus: build x = linspace(-3, 3, 200), y = x**2, print the Pearson and
#        Spearman correlations, then scatterplot the pair. Explain why
#        both coefficients miss an exact deterministic relationship.

Interview questions

Q: Why can't you pass a tidy DataFrame straight to sns.heatmap()?

Because heatmap is a matrix function, not a statistical one. It has no x, y or hue parameters and does no grouping. It reads df.index as the y-axis, df.columns as the x-axis and df.values as the colours, so the data must already be a 2-D grid. You get there with pivot, pivot_table, crosstab or corr.

Q: What is the difference between df.pivot() and pd.pivot_table()?

pivot only reshapes: it requires exactly one value per (index, columns) pair and raises ValueError: Index contains duplicate entries otherwise. pivot_table aggregates duplicates using aggfunc (mean by default) and additionally supports fill_value, margins and multiple aggregation functions. If pivot errors on duplicates, that error is telling you that you need to decide how to combine them.

Q: A correlation of 0 — does that mean the two variables are unrelated?

No. Pearson's r measures linear association only. y = x² over a symmetric range gives r ≈ 0 even though y is a deterministic function of x. Zero correlation means no linear relationship; it says nothing about curved or conditional ones. Scatterplot the pair before concluding anything, and consider Spearman if the relationship might be monotonic-but-curved.

Q: When would you use Spearman instead of Pearson?

When the relationship is monotonic but not straight, when the data are skewed or outlier-heavy, or when a variable is ordinal rather than interval. Spearman correlates the ranks, so it is invariant to any monotonic transformation and far less sensitive to a single extreme point. The trade-off is that it discards the magnitude of the gaps between values. Comparing Pearson and Spearman matrices is a cheap way to locate the pairs affected by outliers or curvature.

Q: Why mask the upper triangle of a correlation heatmap?

Because corr(A, B) = corr(B, A), so the matrix is symmetric: a k-variable matrix draws k² cells but contains only k(k−1)/2 distinct numbers. The diagonal adds another k cells of pure noise — always 1.00, always the most saturated colour on the chart. Masking with np.triu(np.ones_like(corr, dtype=bool), k=0) removes the duplication and the distraction.

Q: What colormap would you pick for a correlation matrix, and why not jet?

A diverging map — coolwarm, RdBu_r or vlag — with center=0 and vmin=-1, vmax=1, so that white marks true zero, the two directions are visually distinct and the scale is comparable across charts. jet and other rainbow maps are not perceptually uniform: they create bright bands the eye reads as boundaries that do not exist in the data, and they compress real differences in the dark blue region. They also fail badly for colour-blind readers and in greyscale.

Q: In pandas 2.x, why does df.corr() raise an error on a real dataset?

Because the default changed to numeric_only=False, so pandas now tries to correlate string columns and raises ValueError: could not convert string to float. Pandas 1.x silently dropped them; 1.5 deprecated that behaviour. The fix is df.corr(numeric_only=True) or df.select_dtypes("number").corr(). The same change affects mean, sum and other reductions on mixed DataFrames.

Q: How does clustermap differ from heatmap, and what is the main trap?

clustermap runs hierarchical clustering on rows and columns, draws dendrograms and reorders the matrix so similar rows and columns sit together. It is figure-level (no ax=), it cannot handle NaN, and it usually needs z_score or standard_scale when the columns are in different units. The main trap is that the displayed order is not the input order — recover it with g.dendrogram_row.reordered_ind before joining any external labels positionally. Use plain heatmap whenever the axis order is meaningful, as with months and years.

Q: You see r = 0.98 between two features. Good news?

Almost never. It usually means duplicated columns or unit conversions, target leakage (the same fact recorded twice), or near-collinearity that will destabilise a linear model's coefficients. Investigate the pair, then drop or combine one of them. Very high correlations are a data-quality signal first and a finding second.

Summary

• heatmap and clustermap are matrix functions: they take a 2-D grid, not tidy data. Rows become the y-axis, columns the x-axis, values the colours. There is no x=, y= or hue=.
• Reach matrix form with df.pivot() (one value per cell), pd.pivot_table(aggfunc=...) (many values per cell), pd.crosstab() (two categoricals) or df.corr() (numeric columns). Since pandas 2.0, pivot takes keyword arguments only.
• The correlation recipe: corr = df.corr(numeric_only=True), then annot=True, fmt=".2f", cmap="coolwarm", center=0, vmin=-1, vmax=1, square=True. numeric_only=True is mandatory in pandas 2.x.
• Pearson's r measures linear association only — a perfect parabola scores ≈ 0. Correlation is never causation; confounding, selection effects and multiple comparisons all manufacture it. Always scatterplot the interesting pairs.
• The diagonal is always 1.0 and carries no information; the matrix is symmetric, so half of it is redundant. Mask one triangle with np.triu(np.ones_like(corr, dtype=bool), k=0).
• Use method="spearman" for monotonic, ranked, skewed or outlier-heavy data. Magnitude conventions (weak < 0.3, moderate 0.3–0.7, strong > 0.7) are field-dependent rules of thumb, not thresholds of truth — report n, and remember r² is the honest summary.
• Colormaps: diverging with center=0 for signed data, sequential (viridis, rocket, mako) for magnitudes, never rainbow. Pin vmin/vmax whenever two charts must be comparable, and prefer colour-blind-safe maps.
• Common failures: annot=True unreadable past about 15×15; constant columns producing NaN stripes; raw counts hiding effects that proportions reveal; blank cells being read as zeros; clipped top and bottom rows on matplotlib 3.1.x, fixed by plt.tight_layout() or an explicit ylim nudge.
• clustermap reorders rows and columns and cannot accept NaN or ax=; standardise with z_score when units differ, and use plain heatmap when the existing axis order is itself meaningful.

Next: Chapter 19 covers regression plots — regplot, lmplot and residplot.