Xatolar jurnali
Bitta fayl
Tez loyihasida bitta fayl alohida o'rinda turadi: 12-mistakes-and-lessons.md — xatolar va saboqlar jurnali.
Uning birinchi qatorlari:
jurnal = (TEZ.parent.parent / "12-mistakes-and-lessons.md").read_text(encoding="utf-8") print("\n".join(jurnal.splitlines()[:4]))
# 12. Mistakes and Lessons Every mistake made while building Tez, what it cost, and the rule that prevents it next time. Read this file before every action. Add every new mistake the same day.
Ikki qoida: har bir harakatdan oldin shu faylni o'qish, va har bir yangi xatoni o'sha kuni yozish. Bu kitobdagi ko'p boblar shu jurnaldagi qatorlarga havola qiladi. Bu bob — jurnalning o'zi haqida: unda nima bor, u qanday ishlaydi va nega u loyihaning eng foydali fayllaridan biri.
Jurnaldagi hamma xatolar
Jurnal — jadval: raqam, sana, xato, uning narxi (o'lchangan) va undan chiqqan qoida. Uni fayldan o'qib, qisqartirilgan holda chiqaramiz:
import pandas as pd qatorlar = [] for qator in jurnal.split("## Mistakes")[1].split("## Incidents")[0].splitlines(): katak = [k.strip() for k in qator.strip().strip("|").split("|")] if len(katak) == 5 and katak[0].isdigit(): qatorlar.append({"#": int(katak[0]), "sana": katak[1], "xato": katak[2][:95], "qoida": katak[4][:80]}) xatolar = pd.DataFrame(qatorlar).sort_values("#") show(xatolar, 30) print(f"jami: {len(xatolar)} ta xato") print("sana bo'yicha:", xatolar["sana"].value_counts().sort_index().to_dict())
jami: 25 ta xato
sana bo'yicha: {'2026-10-31': 6, '2026-11-01': 8, '2026-11-02': 10, '2026-11-03': 1}
| # | sana | xato | qoida |
|---|---|---|---|
| 1 | 2026-11-01 | Made group_by optional in tools.json for whole-table totals without checking the grammar. lla | Before every action, item 3. Every key is required inside its own schema so llam |
| 2 | 2026-11-01 | Compared tez9 (new grammar) with tez8 results recorded under the old grammar. | Before every action, item 4. |
| 3 | 2026-11-01 | Added 4 new tables but forgot their domain_data entries in training/settings.json. | Before every action, item 5. |
| 4 | 2026-11-01 | Wrote the llama.cpp source path without the llama_src/ folder in my notes. | Before every action, item 6. |
| 5 | 2026-11-01 | Said the tez9 benchmark would end at 19:25 using 3.6 s per answer only. | Before every action, item 7. |
| 6 | 2026-11-01 | batch12 and B1 added almost only long requests (75% over 11 words, 2% of 4 to 5 words). | Before every action, item 8. |
| 7 | 2026-11-01 | After batch13 the gate still missed by 0.1 point (20.50% vs 20.60%); needed rows were not compu | Compute the needed row count with the gate formula before choosing batch size. |
| 8 | 2026-11-01 | Generator lets count_distinct group by the same column it counts; the number check covers onl | Closed 2026-11-02: group_and_total returns no plan when the counted column is |
| 9 | 2026-10-31 | Whole-table totals over 0 rows return 1 row, so keep_plan divided by zero and let bad plans thr | Every new tool shape gets a test with an empty result. |
| 10 | 2026-10-31 | Trusted the B1 judge without measuring it. | Measure every judge or checker on a small labelled sample first. |
| 11 | 2026-10-31 | First coded-column requests gave the codes themselves ("jins 2"). | Check a sample of every new data kind by hand before scaling. |
| 12 | 2026-10-31 | compare_models counted tez6 results with stale verdicts. | Rescore every saved answer with the current engine before comparing. |
| 13 | 2026-10-31 | A waiting loop matched its own command line and never ended. | Wait on a file or an exact process ID, never on a pattern that matches the waite |
| 14 | 2026-10-31 | test_cannot searched for "{" which also appears in tool lines. | Tests check exact named placeholders, not single characters. |
| 15 | 2026-11-02 | Counted asked-on-clear with a script that assumed asks are written as {"ask": ...} in `expect | Count outcomes from the saved reason field of accuracy/run.py, never re-derive |
| 16 | 2026-11-02 | Trained tez9 with new data and a schema change together, so one run could not show which change | Change one thing per trained model where possible, or rerun the current model on |
| 17 | 2026-11-02 | Spike 1b transfer check (spike1_excel_read/postgres/transfer.py, fingerprint + same) comp | A transfer check for money compares exact values (DECIMAL or row-by-row hash on |
| 18 | 2026-11-02 | Explained the slow first answer of every benchmark suite (14 to 32 s) as model loading without | Explain a timing only from the log or the timing code, never from a guess. |
| 19 | 2026-11-02 | In 13-why-the-model-falls-short.md, described the training column names from the domain table.j | Any claim about what the model saw is measured on the training rows and the rend |
| 20 | 2026-11-02 | Wrote in 13 and 14 that tez9 memorised "women = SEX 2" from the hr_staff headers, without check | Before naming a cause, check a model or a run that lacks it; if it shows the sam |
| 21 | 2026-11-02 | Called tez10 "tez8's data plus 858 refusal rows" (plan 14, book chapter 29) without diffing the | Before training, diff the new training file against the current model's file fie |
| 22 | 2026-11-02 | The 858 refusal rows went into tez10 without the length check from mistake 6: none is shorter t | Every added row set, refusals included, passes the coverage gate on its own befo |
| 23 | 2026-11-02 | Set the clean ship bar (310 of 320) without measuring how much a plain retrain changes. Older c | Measure retrain churn with a control run before setting a regression bar, and se |
| 24 | 2026-11-02 | Told the user continuation training (tez8 adapter, lr 5e-5, 858 refusal rows plus 3,432 replay | Never promise a training method's side effects before measuring them on a pilot; |
| 25 | 2026-11-03 | Reported the stack as 88.8% precision, "1-2 questions from 90%", on the 105 real-table question | Keep a frozen held-out set from the start; every fix is chosen on one set and th |
Yigirma beshta xato, to'rt kun ichida. Ko'p tuyulishi mumkin — lekin bu jurnal to'liq bo'lgani uchun ko'p. Har bir xato, hatto "papka yo'lini noto'g'ri yozdim" (4-raqam) ham, yozilgan.
Turlari
Xatolarni guruhlaymiz. Bu tasnif — bizning bahomiz, jurnalning o'zida yo'q:
| Tur | Raqamlar | Umumiy saboq |
|---|---|---|
| Grammar va solishtirish | 1, 2 | Grammar o'zgarsa — haqiqiy grammar ni yasab tekshir, ikkala modelni qayta ishga tushir (9, 24-boblar) |
| Ma'lumot tarkibi | 6, 7, 8, 11, 21, 22 | Yangi ma'lumotni yasashdan oldin testga solishtir; har yangi turdan namunani qo'lda o'qi (16-bob) |
| Tekshiruvchilar | 9, 10, 17 | Har qanday tekshiruvchini avval buzilgan nusxa bilan sina — u xatoni ushlayaptimi (5, 13, 17-boblar) |
| Natijani sanash | 12, 15 | Hukmni qaytadan hisoblama — saqlangan reason dan ol; eski hukmlarni yangi engine bilan qayta bahola (24-bob) |
| Sabab haqida taxmin | 18, 19, 20 | Sababni aytishdan oldin log, haqiqiy prompt yoki shu sababi yo'q holatni tekshir |
| O'lchanmagan da'vo | 23, 24, 25 | Chegara, va'da yoki foizni aytishdan oldin uni ko'rmagan ma'lumotda o'lcha (22, 29-boblar) |
| Jarayon | 3, 4, 5, 13, 14, 16 | Yo'llarni tekshir, vaqtni o'lchangan ishdan ol, bitta modelda bitta o'zgarish |
Eng xavfli ikki guruh — sabab haqida taxmin va o'lchanmagan da'vo. Bu xatolar kod xatosi emas, fikrlash xatosi: kuzatuv to'g'ri, lekin uning izohi yoki undan chiqqan va'da tekshirilmagan. Va ular eng ishonchli ko'rinadi, chunki izoh mantiqan chiroyli. 25-xato bunga eng yaxshi misol: 88.8 foiz haqiqiy o'lchov edi, lekin sozlash uchun ishlatilgan savollarda — yangi savollarda u 73.1 foiz bo'lib chiqdi (29-bob).
Kitob ham tekshiruvchi bo'ldi
17 dan 20 gacha bo'lgan qatorlarga qarang:
for q in xatolar[xatolar["#"].between(17, 20)].itertuples(): print(f"{q._1}. {q.xato}")
17. Spike 1b transfer check (`spike1_excel_read/postgres/transfer.py`, `fingerprint` + `same`) comp 18. Explained the slow first answer of every benchmark suite (14 to 32 s) as model loading without 19. In 13-why-the-model-falls-short.md, described the training column names from the domain table.j 20. Wrote in 13 and 14 that tez9 memorised "women = SEX 2" from the hr_staff headers, without check
To'rttasi ham shu kitob yozilayotgan kuni topilgan. 17 — 5-bobni yozayotganda: spike dagi "pass" tekshiruvi buzilgan nusxani ushlay olmasligi ko'rindi. 18 — 6-bobda: birinchi javobning sekinligi "model yuklash" deb tushuntirilgan edi, server logi boshqacha ko'rsatdi. 19 va 20 — modelning zaif joylarini tahlil qilganda chiqarilgan ikki xulosa tekshiruvdan o'tmadi.
Bundan tashqari, boblarni yozish paytida jurnalga kirmagan, lekin planga yozilgan bir nechta kamchilik topildi: Excel loaderi tasodifan oddiy qatorni oraliq jami deb chetga oladi (11-bob); training ma'lumotida inglizcha so'z o'zbek gapida qolib, kirillga ma'nosiz o'girilgan (12-bob); sanalar har doim "kun/oy/yil" deb o'qiladi (10-bob).
Nega kitob yozish xato topadi? Chunki har bir bobda har bir da'vo kod bilan qayta o'lchanadi. "Bu tekshiruv xatoni ushlaydi" deb yozish uchun uni buzilgan nusxada ishlatib ko'rish kerak. "Bu sekinlik — model yuklash" deb yozish uchun logni ochish kerak. Va o'lchov ba'zan da'voni rad etadi.
Jurnal qanday ishlaydi
Jurnalning qiymati xatolar ro'yxatida emas — undagi qoidalarda. Har bir xato jadvalning oxirgi ustunida qoidaga aylanadi, va eng muhim qoidalar faylning boshidagi "har bir harakatdan oldin" ro'yxatiga ko'tariladi:
oldin = jurnal.split("## Before every action")[1].split("## Mistakes")[0] print(oldin.strip())
1. Read this file. 2. Say in one line what will change and why. Wait for approval before code. 3. Changing tools.json, the schema or the grammar: generate the real llama.cpp grammar and check key order against the training data order. Rerun the current model before training a new one. 4. Comparing models: same engine, same grammar, same questions, same machine. If the grammar changed, rerun both models; rescoring old answers is not enough. 5. New table, tool or setting: run the whole pipeline on one plan (plans, phrase, check, second check, export) before a long batch. 6. Long command chains: check every path exists with `ls` first. 7. Time estimates: take them from the measured time of the last run of the same job, never from a guess. 8. New training data: measure its length mix and tool mix against the test suites before generating, not after. 9. Never call a model better until the pass rules are measured. 10. Building a train file by hand: soup takes the last `val_split` share of the file as validation (`soup_cli/data/loader.py`, `_split_val`), so never append new rows at the end. Adding a pair file to export reshuffles every row (one seed), so it changes old rows too.
Bu ro'yxat — o'n bandlik nazorat ro'yxati. Har bir band ortida bitta yoki bir nechta qimmat xato bor. Masalan, 7-band ("vaqtni o'lchangan ishdan ol") 5-xatodan: tez9 benchmarki qachon tugashi faqat bitta javob vaqtidan hisoblangan, va foydalanuvchi kutib, oxiri o'zi so'rashga majbur bo'lgan. 10-band ("faylning oxirgi 5 foizi") — tez10 tayyorlanayotganda topilgan tuzoqdan (15-bob).
Ro'yxatning foydasini o'lchash qiyin — oldi olingan xatoni sanab bo'lmaydi. Bitta kuzatuv bor: tez10 tayyorlanayotganda ikkita tuzoq — bitta seed qayta aralashtirishi va tekshirish qismi — trainingdan oldin topildi: har bir qadamdan keyin natija o'lchab ko'rilgani uchun. Ular jurnal tufayli oldindan ma'lum emas edi, lekin topilgan kuniyoq jurnalga 10-band bo'lib yozildi — va keyingi har bir model uchun endi oldindan ma'lum.
Hodisalar
Jurnalda yana bitta kichik jadval bor — xato bo'lmagan, lekin yozib qo'yilgan hodisalar:
print(jurnal.split("## Incidents")[1].strip())
(not caused by a mistake, kept for the record) | Date | What happened | Response | |---|---|---| | 2026-11-01 | batch13 toolshort check: one connection to the local Gemma server timed out (WinError 10060) while the server log showed no error. Cause unknown. | Reran only the check stage from saved phrases with run2.cmd. | | 2026-11-01 | tez9 recheck: llama-server hung about 6 minutes loading the model from /mnt/d (process in disk wait on `p9_client_rpc`, the WSL to Windows file bridge); D: read at 49 MB/s afterwards. Cause unknown. | Waited; it recovered. For long CPU benchmarks, consider a copy of the model on the WSL disk so loading does not depend on the bridge. | | 2026-11-03 | tez8s live run: at 09:21 the Windows drive (/mnt/c) returned Input/output error on results/tez8s_cpu_real_banks.md and accuracy/run.py; real_banks and real_industry failed to start, the other 18 suites finished. Cause unknown (WSL file bridge). | Reran only the two failed suites once the drive read again. |
Uchalasining ham sababi noma'lum. Ular baribir yozilgan: agar yana takrorlansa, birinchi marta qanday hal qilingani ma'lum bo'ladi.
Nega bunday jurnal kerak
Uch sabab.
Birinchi — xotira. Loyiha haftalar davom etadi, va har bir qaror ortidagi sabab tez unutiladi. "Nega
contains faqat ko'p so'zli qiymat bilan?" — javob jurnalda va kodning test nomida.
Ikkinchi — halollik. Xato yozilgan joyda uni yashirish qiyin. Bu kitobning har bir bobi xatolarni ochiq ko'rsata oldi, chunki ular allaqachon jurnalda edi — o'lchangan narxi bilan.
Uchinchi — tezlik. Bu g'alati tuyulishi mumkin, lekin jurnal ishni tezlashtiradi. Har bir harakatdan oldin bir daqiqa o'qish — olti soatlik trainingni behuda sarflashdan arzon.
Xulosa
- Xatolar jurnali — bitta fayl: har bir xato, uning o'lchangan narxi va undan chiqqan qoida.
- To'rt kunda yigirma beshta xato yozilgan; eng xavfli guruhlar — sabab haqida tekshirilmagan taxminlar va o'lchanmagan da'volar.
- 17–20-xatolar kitob yozish paytida topildi: har bir da'voni kod bilan qayta o'lchash xatoni ochadi.
- Eng muhim qoidalar faylning boshidagi o'n bandlik "har bir harakatdan oldin" ro'yxatiga ko'tarilgan.
- Jurnal xotira, halollik va tezlik beradi.
Oltinchi qism tugadi. Yettinchi qismda Tez real hayotga chiqadi: kuchsiz kompyuter, maxfiylik, yangilash sikli va keyingi qadamlar.
Bu bobni bepul o'qidingiz.
Kitob o'rgatadi, lekin sizning kodingiz nega ishlamaganini ayta olmaydi. Kursda uy vazifangiz tekshiriladi.