EDA workflow

22-bob 11 kod bloki 3461 so'z

Oldingi har bir bo'lim bitta tool ni o'rgatdi. Bu bo'lim esa ularni qanday tartibda ishlatishni o'rgatadi.

Exploratory Data Analysis — bu ishning shunday qismiki, unga hech kim alohida ticket yozmaydi va shoshib qolganda hamma uni tashlab ketadi — vaholanki deyarli har qanday jiddiy xato aynan shu yerda ushlanadi. 60% i missing bo'lgan column da o'qitilgan model, ichida ikkita alohida population yashiringan qiymatning o'rtachasini chiqaradigan report, aslida takroran import qilingan "rekord oy" — bularning barchasi yigirma daqiqalik EDA da topiladi va EDA bo'lmasa, barchasi production gacha bemalol yetib boradi.

EDA nima uchun kerak

EDA ning uchta vazifasi bor va ularni ajratib ko'rsatish foydali, chunki o'quvchilar odatda faqat uchinchisini bajarishadi.

Olti bosqichli checklist

O'quvchilarga qat'iy tartib bering. Vaqt tig'iz bo'lganda checklist intuitsiyadan ustun keladi.

Tartib bejiz tanlanmagan. 2-bosqich 3-bosqichni qanday o'qishingizni o'zgartiradi; 3-bosqich 4-bosqichdagi qaysi juftliklarga qarash arziydiganini aytadi; 6-bosqich esa oxirida turadi, chunki normal holat qanday ekanini bilmaguningizcha outlier ni oddiy terish xatosidan ajrata olmaysiz.

1-bosqich — shape va types

import sqlite3
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_theme(style="whitegrid")
penguins = sns.load_dataset("penguins")

print(penguins.shape)          # (344, 7) — row lar, column lar
penguins.head()                # bitta row aslida qanday ko'rinadi?
penguins.info()                # dtype lar + non-null count lar, bitta ko'rinishda
penguins.describe()            # numeric xulosa
penguins.describe(include="object")   # categorical xulosa

(344, 7)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 344 entries, 0 to 343
Data columns (total 7 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   species            344 non-null    object 
 1   island             344 non-null    object 
 2   bill_length_mm     342 non-null    float64
 3   bill_depth_mm      342 non-null    float64
 4   flipper_length_mm  342 non-null    float64
 5   body_mass_g        342 non-null    float64
 6   sex                333 non-null    object 
dtypes: float64(4), object(3)
memory usage: 18.9+ KB

info() — pandas dagi eng qimmatli bitta chaqiruv: u sizga row soni, har bir column nomi, har bir dtype va non-null count ni bir vaqtning o'zida beradi. Uni hamma narsadan oldin o'qing.

2-bosqich — missing data

# har bir column da qancha missing bor: son va foiz sifatida
miss = pd.DataFrame({
    "missing": penguins.isna().sum(),
    "pct":     (penguins.isna().mean() * 100).round(1)
}).sort_values("missing", ascending=False)
print(miss)

# qaysi row lar to'liq emas?
penguins[penguins.isna().any(axis=1)]

                   missing  pct
sex                     11  3.2
bill_depth_mm            2  0.6
bill_length_mm           2  0.6
flipper_length_mm        2  0.6
body_mass_g              2  0.6
island                   0  0.0
species                  0  0.0

Keyin sanoqdan ham muhimroq savolni bering: u tasodifiy tarzda missing bo'lganmi? Agar missing row lar bitta guruhda to'plangan bo'lsa, ularni tashlab yuborish keyingi barcha hisob-kitoblarni sezdirmasdan buzadi.

# missing qiymatlar species bo'yicha bir tekis tarqalganmi? penguins.assign(missing_sex=penguins.sex.isna()) \ .groupby("species")["missing_sex"].mean().round(3)

3-bosqich — bir vaqtda bitta o'zgaruvchi

Har qanday bog'liqlikdan oldin har bir column ga alohida qarang. Numeric column da sizga shakl, markaz, tarqoqlik va dumlar kerak:

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.histplot(data=penguins, x="body_mass_g", kde=True, ax=axes[0])
sns.boxplot(data=penguins,  y="body_mass_g", ax=axes[1])
sns.ecdfplot(data=penguins, x="body_mass_g", ax=axes[2])
plt.tight_layout()

Categorical column da esa sizga darajalar va ularning muvozanati kerak:

penguins["species"].value_counts()
penguins["species"].value_counts(normalize=True).round(3)   # proportion sifatida

sns.countplot(data=penguins, x="species", hue="species", legend=False)

Har bir numeric column uchun yozib qo'yiladigan to'rt narsa: u simmetrikmi yoki skew ga egami, u bitta cho'qqilimi yoki ko'p cho'qqilimi, qiymatlar oralig'i ishonarlimi va unda nechta turli qiymat bor (3 ta turli qiymatga ega "numeric" column aslida category dir).

4-bosqich — ikkita o'zgaruvchi

Bor-yo'g'i uchta birikma mavjud va har birining o'z standart chart i bor.

# numeric x numeric
sns.scatterplot(data=penguins, x="flipper_length_mm", y="body_mass_g", hue="species")

# numeric x categorical
sns.boxplot(data=penguins, x="species", y="body_mass_g", hue="species", legend=False)

# categorical x categorical — xom count emas, proportion lar
ct = pd.crosstab(penguins["island"], penguins["species"], normalize="index")
sns.heatmap(ct, annot=True, fmt=".2f", cmap="Blues")

5-bosqich — bir vaqtda ko'p o'zgaruvchi

# correlation matrix — pandas 2+ da numeric_only majburiy
corr = penguins.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0, vmin=-1, vmax=1)

# barcha numeric juftliklar birdaniga, guruh bo'yicha ranglangan
sns.pairplot(penguins, hue="species", diag_kind="kde", corner=True)

hue bilan chizilgan pairplot ko'pincha butun tahlildagi eng ko'p ma'lumot beradigan yagona chart bo'ladi. Penguins data sida u bir qarashda uchta species ning bill va flipper o'lchovlari bo'yicha deyarli mukammal ajralishini ko'rsatadi — bu esa butun natijaning o'zi.

6-bosqich — outlier lar va data sifati

Outlier avtomatik ravishda xato degani emas. Ularni uchta guruhga ajrating:

# standart 1.5 x IQR qoidasi — boxplot mo'ylovlari ishlatadigan aynan shu qoida
q1, q3 = penguins["body_mass_g"].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
outliers = penguins[(penguins.body_mass_g < lo) | (penguins.body_mass_g > hi)]
print(f"{len(outliers)} outliers outside [{lo:.0f}, {hi:.0f}]")

# sentinel ovi: shubhali darajada butun yoki takrorlanuvchi qiymatlar
penguins["body_mass_g"].value_counts().head(10)

0 outliers outside [1750, 6550]

Data sifatini tekshirishning qolgan qismi tez bajariladi va har safar qilishga arziydi:

df.duplicated().sum() # aynan bir xil duplicate row lar df.duplicated(subset=["student_id", "course_id"]).sum() # business-key duplicate lari df["city"].str.strip().str.lower().nunique() # 'Tashkent ' va 'tashkent' va 'Tashkent' df.nunique() # o'zgarmas column lar (1) yoki ID ga o'xshash column lar (== len(df))

Qaysi savolga qaysi plot Sinf devoriga osib qo'yiladigan jadval.

To'liq ishlangan EDA

titanic dataset ida yigirma daqiqa, checklist ni boshdan oxirigacha bajarib. Bu — darsda jonli ravishda ishga tushiriladigan variant.

1–2-bosqichlar: shape, types, missing data

titanic = sns.load_dataset("titanic")
print(titanic.shape)            # (891, 15) — bitta row bitta yo'lovchi
titanic.info()

pd.DataFrame({"missing": titanic.isna().sum(),
              "pct": (titanic.isna().mean()*100).round(1)}) \
  .query("missing > 0").sort_values("missing", ascending=False)

(891, 15)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 15 columns):
 #   Column       Non-Null Count  Dtype   
---  ------       --------------  -----   
 0   survived     891 non-null    int64   
 1   pclass       891 non-null    int64   
 2   sex          891 non-null    object  
 3   age          714 non-null    float64 
 4   sibsp        891 non-null    int64   
 5   parch        891 non-null    int64   
 6   fare         891 non-null    float64 
 7   embarked     889 non-null    object  
 8   class        891 non-null    category
 9   who          891 non-null    object  
 10  adult_male   891 non-null    bool    
 11  deck         203 non-null    category
 12  embark_town  889 non-null    object  
 13  alive        891 non-null    object  
 14  alone        891 non-null    bool    
dtypes: bool(2), category(2), float64(2), int64(4), object(5)
memory usage: 80.7+ KB

columnmissingpctnima qilish kerak

deck68877.2Shu holicha ishlatish uchun juda siyrak — tashlab yuboring yoki "unknown" ni alohida daraja sifatida qarang age17719.9Sezilarli miqdor. Row larni o'chirmang — impute qiling yoki ular bilan va ularsiz tahlil qiling embarked / embark_town20.2Ahamiyatsiz — bu ikki row ni xavfsiz tashlab yuborsa bo'ladi

Darhol e'tibor bering: 891 − 177 = 714. Ya'ni yosh haqidagi har qanday xulosa 891 emas, 714 yo'lovchiga asoslangan. Bu jumla report ga kirishi kerak.

3-bosqich: bir vaqtda bitta o'zgaruvchi

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.histplot(data=titanic, x="age", bins=30, kde=True, ax=axes[0])
sns.countplot(data=titanic, x="pclass", hue="pclass", legend=False, ax=axes[1])
sns.histplot(data=titanic, x="fare", bins=40, ax=axes[2])
axes[2].set_title("fare is heavily right-skewed")
plt.tight_layout()

titanic["survived"].value_counts(normalize=True).round(3)   # ~38% omon qolgan

Age taxminan simmetrik, yosh bolalar tomonida kichik do'nglik bilan. Fare esa kuchli o'ngga skew qilgan — bir nechta juda qimmat chipta bor — shuning uchun uning mean i median idan ancha yuqori va har qanday "o'rtacha fare" ehtiyotkorlikni talab qiladi.

4-bosqich: ikkita o'zgaruvchi — asosiy savol

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
sns.barplot(data=titanic, x="sex",    y="survived", hue="sex",    legend=False, ax=axes[0])
sns.barplot(data=titanic, x="pclass", y="survived", hue="pclass", legend=False, ax=axes[1])
sns.boxplot(data=titanic, x="survived", y="age", hue="survived", legend=False, ax=axes[2])
axes[0].set_title("survival rate by sex")
axes[1].set_title("survival rate by class")
plt.tight_layout()

5-bosqich: ko'p o'zgaruvchi va o'zaro ta'sir

# class sex ning ta'sirini o'zgartiradimi? (ha — va juda kuchli)
sns.catplot(data=titanic, x="pclass", y="survived", hue="sex",
            kind="bar", height=4, aspect=1.3)

# xuddi shu narsa raqamlar to'ri ko'rinishida
pd.crosstab(titanic["sex"], titanic["pclass"], values=titanic["survived"],
            aggfunc="mean").round(2)

Natija: omon qolish asosan sex bilan belgilanadi va class uni kuchaytiradi yoki susaytiradi. Birinchi class dagi ayollar uchinchi class dagilarga qaraganda ancha yuqori darajada omon qolishgan; uchinchi class dagi erkaklar esa eng yomon ahvolda bo'lgan. Bitta o'zgaruvchili xulosa bu o'zaro ta'sirni butunlay o'tkazib yuborar edi — 5-bosqich aynan shuning uchun mavjud.

6-bosqich: outlier lar va sifat

titanic[titanic["fare"] > 300][["pclass", "fare", "survived"]] # haqiqiy, xato emas (titanic["fare"] == 0).sum() # 15 ta nol fare — ekipajmi yoki xodimlarmi? tekshiring titanic.duplicated().sum() £512 lik fare lar haqiqiy birinchi class chiptalari, shuning uchun ular joyida qoladi. Nol fare lar esa qiziqarli anomaliya — izohga arziydi va aynan shunday narsa faqat siz qarab chiqqaningiz uchun yuzaga chiqadi.

EDA anti-pattern lari

Xulosa

• EDA ning uchta vazifasi bor: data ni tushunish, data ni tekshirish, signal ni topish. Ko'pchilik faqat uchinchisini bajaradi.
• Checklist ni tartib bilan bajaring: shape & types → missing → bitta o'zgaruvchi → ikkita o'zgaruvchi → ko'p o'zgaruvchi → outlier lar.
• Doim avval "bitta row nima?" degan savolga javob bering. JOIN dan keyin javob o'zgargan bo'lishi mumkin — bu 5-bo'limdagi fan-out.
• info() — pandas dagi eng qimmatli yagona chaqiruv: row lar, dtype lar va non-null count lar birdaniga.
• Missing data — bu natija, zerikarli yumush emas. Qanchaligini o'lchang, tasodifiymi yoki yo'qligini tekshiring va dropna()/fillna() ni hujjatlashtiriladigan qarorlar sifatida qarang.
• Column skew ga ega bo'lsa median ni e'lon qiling va n siz hech qachon mean bermang.
• Doim eng aniq guruh bo'yicha rang bering — Simpson paradoksi birlashtirilgan correlation ni butunlay teskarisiga aylantirishi mumkin.
• Outlier larni imkonsiz, sentinel va haqiqatan ekstremal turlarga ajrating. Faqat dastlabki ikkitasi olib tashlanadi va faqat asos bilan.
• 0/1 column ning barplot i — proportion chart i, bu yerdagi eng foydali usullardan biri.
• Yakuniy mahsulot — chart lar emas, natijalar ro'yxati.

Keyingisi: 23-bo'lim barcha Seaborn gotcha larini bitta takrorlash katalogiga jamlaydi va II qismni yakunlaydi.