What ML is, and when you need it

Chapter 2 15 code blocks 2954 words

The code in this chapter uses these filesrun it on your own machine

The scene: the rules got out of hand

You're a developer at a rental agency. The director asks: "Can you tell in advance which flats will rent quickly?"

You think about it. Easy, I'll write a rule.

python2 lines

if price < 500:
    return "goes fast"

It works, but not well. You add furniture:

python2 lines

if price < 500 and furnished:
    return "goes fast"

Then the district. Then the distance to the metro. Then the floor. Two weeks later your code is one long chain of ifs that nobody understands, and when a new district appears you have to rewrite all of it.

That is the point where you need ML.

Traditional programming vs ML: the difference is one thing

python5 lines

import pandas as pd
show(pd.DataFrame([
    {"": "Traditional programming", "You provide": "Data + RULES", "The computer gives back": "Answers"},
    {"": "Machine Learning", "You provide": "Data + ANSWERS", "The computer gives back": "RULES"},
]))

	You provide	The computer gives back
Traditional programming	Data + RULES	Answers
Machine Learning	Data + ANSWERS	RULES

The whole of ML is in those two rows.

In traditional programming you write the rule. In ML you give the computer thousands of examples ("this flat went fast, this one didn't") and it finds the rule itself.

python18 lines

diagramma(
    {
        "Data\n(flat features)": (0, 0, 3.2, 1.1, "manba"),
        "RULE\n(you write it: if/else)": (0, 1.6, 3.2, 1.1, "qayta"),
        "Answer\n(goes fast / doesn't)": (0, 3.2, 3.2, 1.1, "saqlash"),
        "Data\n(flat features) ": (5.2, 0, 3.2, 1.1, "manba"),
        "Answers\n(from history: went / didn't)": (5.2, 1.6, 3.2, 1.1, "manba"),
        "RULE\n(the model finds it)": (5.2, 3.2, 3.2, 1.1, "xizmat"),
    },
    [
        ("Data\n(flat features)", "RULE\n(you write it: if/else)"),
        ("RULE\n(you write it: if/else)", "Answer\n(goes fast / doesn't)"),
        ("Data\n(flat features) ", "RULE\n(the model finds it)"),
        ("Answers\n(from history: went / didn't)", "RULE\n(the model finds it)"),
    ],
    nom="an_vs_ml.png",
    izoh="Left: traditional programming. Right: machine learning. The rule and the answer have swapped places.",
)

Left: traditional programming. Right: machine learning. The rule and the answer have swapped places.

Left: traditional programming. Right: machine learning. The rule and the answer have swapped places.

The single idea behind every algorithm

You'll see dozens of algorithms in this book. All of them do one thing:

Y = f(X)

X is what you know (number of rooms, area, district, distance to the metro)
Y is what you want to know (will it go fast)
f is the rule that connects them

Linear regression, decision trees, neural networks: they differ only in what shape of f they search for. The task is the same.

Column names

The data is synthetic Tashkent rental listings, and its column names are Uzbek. xonalar rooms, maydon_m2 area in m², metro_km distance to the metro, mebel furnished (0/1), narx_usd monthly rent in USD. The target ijaraga_berildi_30k means rented within 30 days.

python8 lines

df = pd.read_csv(DATA / "uy_ijara.csv")
X_example = df[["xonalar", "maydon_m2", "metro_km", "mebel"]].head(5)
y_example = df["ijaraga_berildi_30k"].head(5)
print("X is what the model SEES:")
print(X_example.to_string(index=False))
print()
print("y is the answer the model has to FIND:")
print(y_example.to_string(index=False))

Output

X is what the model SEES:
 xonalar  maydon_m2  metro_km  mebel
       2       40.6      1.43      1
       2       55.5      4.18      1
       5      114.0      3.84      1
       3       72.1      2.99      1
       2       46.4      4.58      0

y is the answer the model has to FIND:
0
0
0
0
1

When you need ML, and when you DON'T

This is the question people skip most. Many projects die right here.

python14 lines

show(pd.DataFrame([
    {"Situation": "The rule is simple and doesn't change", "Need ML?": "NO",
     "Why": "`if age < 18`: doing this with a model is silly"},
    {"Situation": "A rule exists but is very complex", "Need ML?": "YES",
     "Why": "Recognising faces, understanding text: you can't write it by hand"},
    {"Situation": "The rule changes over time", "Need ML?": "YES",
     "Why": "Market prices, fashion, user behaviour"},
    {"Situation": "There is NO data, or very little", "Need ML?": "NO",
     "Why": "You can't build a model from 100 rows; collect data first"},
    {"Situation": "Mistakes are very costly, 100% accuracy needed", "Need ML?": "CAREFUL",
     "Why": "ML never gives 100%. You need human oversight"},
    {"Situation": "You must explain why it answered that way", "Need ML?": "CAREFUL",
     "Why": "Bank loans, medicine. Choose a simple model (Chapter 26, SHAP)"},
]))

Situation	Need ML?	Why
The rule is simple and doesn't change	NO	`if age < 18`: doing this with a model is silly
A rule exists but is very complex	YES	Recognising faces, understanding text: you can't write it by hand
The rule changes over time	YES	Market prices, fashion, user behaviour
There is NO data, or very little	NO	You can't build a model from 100 rows; collect data first
Mistakes are very costly, 100% accuracy needed	CAREFUL	ML never gives 100%. You need human oversight
You must explain why it answered that way	CAREFUL	Bank loans, medicine. Choose a simple model (Chapter 26, SHAP)

The first question is not "which algorithm"

On a new project the first question is: "can a simple rule solve this?"

If yes, write the rule. It's faster, cheaper, understandable, and it doesn't break.

In Chapter 7 we measure exactly that: how far a simple rule falls behind ML.

Types of ML

python8 lines

show(pd.DataFrame([
    {"Type": "Supervised", "Data": "X and y: the answers EXIST",
     "Example": "Will the flat go fast; what's the price", "In this book": "Chapters 5-22"},
    {"Type": "Unsupervised", "Data": "Only X: NO answers",
     "Example": "Splitting customers into groups", "In this book": "Chapters 27-29"},
    {"Type": "Reinforcement", "Data": "Actions and rewards",
     "Example": "Games, robot control", "In this book": "no (a separate field)"},
]))

Type	Data	Example	In this book
Supervised	X and y: the answers EXIST	Will the flat go fast; what's the price	Chapters 5-22
Unsupervised	Only X: NO answers	Splitting customers into groups	Chapters 27-29
Reinforcement	Actions and rewards	Games, robot control	no (a separate field)

Most of this book is supervised, because most practical work is.

Classification and regression

Supervised learning splits in two. The difference is only the type of y.

python7 lines

print("Our file has TWO targets, so two different tasks:")
print()
print(f"ijaraga_berildi_30k : {sorted(df.ijaraga_berildi_30k.unique().tolist())}  -> {df.ijaraga_berildi_30k.nunique()} distinct values")
print("  => CLASSIFICATION (predict a label)")
print()
print(f"narx_usd            : {df.narx_usd.min()} .. {df.narx_usd.max()}  -> {df.narx_usd.nunique()} distinct values")
print("  => REGRESSION (predict a number)")

Output

Our file has TWO targets, so two different tasks:

ijaraga_berildi_30k : [0, 1]  -> 2 distinct values
  => CLASSIFICATION (predict a label)

narx_usd            : 225 .. 1515  -> 207 distinct values
  => REGRESSION (predict a number)

python6 lines

show(pd.DataFrame([
    {"": "Classification", "What y is": "A label (category)", "Example": "spam/not spam, 3 kinds of flower",
     "Metric": "accuracy, F1, recall", "In this book": "Chapters 8-10"},
    {"": "Regression", "What y is": "A continuous number", "Example": "price, temperature, time",
     "Metric": "RMSE, MAE, R2", "In this book": "Chapter 15"},
]))

	What y is	Example	Metric	In this book
Classification	A label (category)	spam/not spam, 3 kinds of flower	accuracy, F1, recall	Chapters 8-10
Regression	A continuous number	price, temperature, time	RMSE, MAE, R2	Chapter 15

"A 1-5 star rating": classification or regression?

It can be either, and it's your decision.

As classification, the model sees 3 and 4 as "two different categories" and doesn't know they are close. As regression, the model may output 3.7, and it understands the distance between 1 and 5.

In practice regression often works better for ratings. But you have to try it: don't guess, measure.

The three commands of ML work

Every model in sklearn understands the same three commands. Swap the algorithm and these three stay the same.

python16 lines

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

num = ["xonalar", "maydon_m2", "metro_km", "mebel"]
X, y = df[num], df["ijaraga_berildi_30k"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

model = DecisionTreeClassifier(max_depth=4, random_state=42)

model.fit(X_tr, y_tr)                       # 1. LEARN
pred = model.predict(X_te)                  # 2. PREDICT
score = model.score(X_te, y_te)             # 3. EVALUATE

print(f"fit()     -> the model learned, from {len(X_tr)} rows")
print(f"predict() -> {len(pred)} predictions, for example: {pred[:8].tolist()}")
print(f"score()   -> {score:.4f}")

Output

fit()     -> the model learned, from 840 rows
predict() -> 360 predictions, for example: [0, 1, 0, 0, 1, 1, 1, 1]
score()   -> 0.7417

We swap the model, and only one line changes:

python10 lines

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

for name, m in [("DecisionTree", DecisionTreeClassifier(max_depth=4, random_state=42)),
                ("LogisticRegression", LogisticRegression(max_iter=2000)),
                ("RandomForest", RandomForestClassifier(n_estimators=100, random_state=42)),
                ("KNN", KNeighborsClassifier())]:
    m.fit(X_tr, y_tr)
    print(f"  {name:20s} score = {m.score(X_te, y_te):.4f}")

Output

  DecisionTree         score = 0.7417
  LogisticRegression   score = 0.7639
  RandomForest         score = 0.6806
  KNN                  score = 0.6917

The key to reading this book

fit / predict / score: these three words are the same for 20 algorithms. That's why learning a new algorithm is easy: you only look for answers to how does it work and when does it fit. The code pattern doesn't change.

Overfitting: memorising vs understanding

In the table above the DecisionTree has max_depth=4. Why did I set a limit? What happens if I remove it?

python7 lines

print(f"{'max_depth':>10} {'train score':>12} {'test score':>11} {'gap':>8}")
print("-" * 44)
for d in [2, 4, 8, 15, None]:
    m = DecisionTreeClassifier(max_depth=d, random_state=42).fit(X_tr, y_tr)
    tr_s, te_s = m.score(X_tr, y_tr), m.score(X_te, y_te)
    flag = "   <-- MEMORISED" if tr_s - te_s > 0.15 else ""
    print(f"{str(d):>10} {tr_s:>12.4f} {te_s:>11.4f} {tr_s-te_s:>8.4f}{flag}")

Output

 max_depth  train score  test score      gap
--------------------------------------------
         2       0.6952      0.7611  -0.0659
         4       0.7048      0.7417  -0.0369
         8       0.7869      0.7139   0.0730
        15       0.9143      0.6778   0.2365   <-- MEMORISED
      None       1.0000      0.6583   0.3417   <-- MEMORISED

With the limit removed, the tree becomes almost perfect on train, and gets worse on test.

That is overfitting: the model didn't learn the rule, it memorised the answers.

python8 lines

show(pd.DataFrame([
    {"Case": "Underfitting", "Train score": "LOW", "Test score": "LOW",
     "Cause": "The model is too simple", "Fix": "A stronger model, more features"},
    {"Case": "Good", "Train score": "Good", "Test score": "Good, and close",
     "Cause": "A balance was found", "Fix": "This is where you stop"},
    {"Case": "Overfitting", "Train score": "VERY HIGH", "Test score": "LOW",
     "Cause": "The model memorised", "Fix": "Simplify, regularise, get more data"},
]))

Case	Train score	Test score	Cause	Fix
Underfitting	LOW	LOW	The model is too simple	A stronger model, more features
Good	Good	Good, and close	A balance was found	This is where you stop
Overfitting	VERY HIGH	LOW	The model memorised	Simplify, regularise, get more data

This is the bias-variance tradeoff, and Chapters 12-14 open it up mathematically. For now one thing is enough: if the train score is much higher than the test score, don't trust it.

The exam analogy

A student prepares for an exam. There are two ways:

Understand the topic -> can answer a new question too
Memorise the answers -> can answer exactly those questions, nothing else

The second student scores 100% on the homework problems (high train score) and fails the exam (low test score).

A model is exactly the same. That's why we test it on data it has not seen (Chapter 6).

Key terms

python10 lines

show(pd.DataFrame([
    {"Term": "Feature", "Meaning": "An input column: X", "In our data": "xonalar, maydon_m2, tuman"},
    {"Term": "Target", "Meaning": "The column being predicted: y", "In our data": "ijaraga_berildi_30k"},
    {"Term": "Sample", "Meaning": "One row", "In our data": "One listing"},
    {"Term": "Label", "Meaning": "A target value in classification", "In our data": "0 or 1"},
    {"Term": "Train", "Meaning": "The part the model learns from", "In our data": "840 rows"},
    {"Term": "Test", "Meaning": "The part the model hasn't seen", "In our data": "360 rows"},
    {"Term": "Hyperparameter", "Meaning": "A setting YOU choose", "In our data": "max_depth=4"},
    {"Term": "Parameter (weight)", "Meaning": "A number the model learns ITSELF", "In our data": "coefficients"},
]))

Term	Meaning	In our data
Feature	An input column: X	xonalar, maydon_m2, tuman
Target	The column being predicted: y	ijaraga_berildi_30k
Sample	One row	One listing
Label	A target value in classification	0 or 1
Train	The part the model learns from	840 rows
Test	The part the model hasn't seen	360 rows
Hyperparameter	A setting YOU choose	max_depth=4
Parameter (weight)	A number the model learns ITSELF	coefficients

Hyperparameter vs parameter: don't mix them up

A parameter is a number the model finds itself during fit() (a line's coefficient, a tree's split threshold).

A hyperparameter is a setting you choose BEFORE fit() (max_depth, n_estimators, C).

One of the most common pairs in interviews. Chapter 24 is about choosing hyperparameters.

What we learned today

python8 lines

show(pd.DataFrame([
    {"Concept": "The essence of ML", "In one sentence": "You don't write the rule; the model finds it in the data"},
    {"Concept": "Y = f(X)", "In one sentence": "Every algorithm does this, it just searches for f differently"},
    {"Concept": "When ML isn't needed", "In one sentence": "When the rule is simple or the data is scarce"},
    {"Concept": "Classification / regression", "In one sentence": "The only difference is the type of y: a label or a number"},
    {"Concept": "fit / predict / score", "In one sentence": "The same three commands for 20 algorithms"},
    {"Concept": "Overfitting", "In one sentence": "Train high, test low: the model memorised"},
]))

Concept	In one sentence
The essence of ML	You don't write the rule; the model finds it in the data
Y = f(X)	Every algorithm does this, it just searches for f differently
When ML isn't needed	When the rule is simple or the data is scarce
Classification / regression	The only difference is the type of y: a label or a number
fit / predict / score	The same three commands for 20 algorithms
Overfitting	Train high, test low: the model memorised

Next chapter: data usually isn't in a CSV, it's in a database. How do we get it out?

previous1. A new dataset: 12 steps next3. SQL: getting data out of a database

Questions about this chapter?

The course, homework review and the other five books are in Uzbek. Questions in English are welcome on Telegram.

Write on Telegram Contents