Agent dasturi: tez12 va tez13 nimaga yetmadi

31-bob 8 kod bloki 3654 so'z

Maqsad

30-bobdagi ilova tez8 bilan ishlaydi. Tahlilchi esa ko'proq narsa so'radi: ikki ustundan yangi ustun hisoblash ("kreditni depozitga nisbati"), sonli ustunni songa ko'paytirish yoki bo'lish, yangi grafik turlari (treemap, qatlamli, issiqlik xaritasi, 3D) va natijani tanlangan joyga saqlash ("ish stoliga Hisobotlar papkasiga saqla"). Buning uchun asboblar ro'yxatiga yangi tool lar qo'shildi va ikki model o'qitildi: tez12-agent va tez13-agent. Ikkalasi ham qabul qilinmadi. Bu bob — nega shunday bo'lgani haqida, raqamlar bilan.

Hamma modellar bir xil engine bilan qayta baholanadi: saqlangan javob qayta ishga tushiriladi va natijasi kutilgan natija bilan solishtiriladi. Eski verdikt ishlatilmaydi, chunki engine o'zgargan (24-bob).

import json
from collections import Counter
from pathlib import Path

from accuracy.run import open_data, expected_results, judge, load_json
from runtime.client import parse_answer

JADVALLAR = Path("/mnt/d/Tez_spikes/data")
NATIJA = TEZ / "results"
SOZLAMA = load_json(TEZ / "accuracy" / "settings.json")
QATOR = {"loans": "loans", "deposits": "deposits", "cash_desk": "cash_desk", "transactions": "bank",
         "realistic_loans": "loans", "realistic_deposits": "deposits", "realistic_cash": "cash_desk"}
REAL = ("real_regions", "real_banks", "real_industry", "real_fx_rates", "real_credit_clients", "real_retail")
ESKI = {**QATOR, **{f"noisy_{k}": v for k, v in QATOR.items()}, **{t: t for t in REAL}}
AGENT = {s.name: s.name.split("_", 2)[2] for s in sorted((TEZ / "accuracy" / "suites").glob("agent_*"))}
MODELLAR = {"tez8": ("tez8g", "tez8"), "churn8": ("churn8g", "churn8"), "tez12-agent": ("tez12a", "tez12a"), "tez13-agent": ("tez13a", "tez13a")}


def xavfsiz(con, savol, chiqish, kutilgan):
    try:
        return judge(con, savol, parse_answer(chiqish), kutilgan, SOZLAMA)[0]
    except ValueError:
        return False


def baholash(toplam, savollar_fayli, nom):
    natija = {}
    for suite, jadval in toplam.items():
        savollar = load_json(TEZ / "accuracy" / "suites" / suite / f"{savollar_fayli}.json")
        idlar = {s["id"]: s for s in savollar}
        con = open_data(str(JADVALLAR / f"{jadval}.csv"))
        kutilgan = expected_results(con, savollar, SOZLAMA)
        for model, (asosiy, fresh) in MODELLAR.items():
            fayl = NATIJA / nom.format(fresh if savollar_fayli == "fresh2" else asosiy, suite)
            if not fayl.exists():
                continue
            for qator in fayl.read_text(encoding="utf-8").splitlines():
                yozuv = json.loads(qator)
                natija[(model, suite, yozuv["id"])] = (xavfsiz(con, idlar[yozuv["id"]], yozuv["output"], kutilgan), yozuv)
    return natija


eski = baholash(ESKI, "questions", "{}_cpu_{}.jsonl")
fresh = baholash({t: t for t in REAL}, "fresh2", "fresh2{}_cpu_{}.jsonl")
agent = baholash(AGENT, "questions", "{}_cpu_{}.jsonl")


def jami(natija, model, boshi=""):
    qiymat = [v[0] for (m, s, _), v in natija.items() if m == model and s.startswith(boshi)]
    return f"{sum(qiymat)}/{len(qiymat)}" if qiymat else "–"


print(f"{'model':13s} {'eski 745':>9s} {'fresh2 92':>10s} {'ask':>6s} {'compute':>8s} {'charts':>7s} {'save':>6s}")
for model in MODELLAR:
    print(f"{model:13s} {jami(eski, model):>9s} {jami(fresh, model):>10s}", *[f"{jami(agent, model, 'agent_' + k):>7s}" for k in ("ask", "compute", "charts", "save")])

model          eski 745  fresh2 92    ask  compute  charts   save
tez8            707/745      59/92       –       –       –       –
churn8          700/745      61/92       –       –       –       –
tez12-agent     689/745      62/92   20/20   17/22   12/22    8/20
tez13-agent     702/745      56/92   20/20   17/22   17/22   11/20

Jadvalni o'qish: tez8 — hozirgi mahsulot; churn8 — tez8 ning o'zi, faqat seed boshqa (shovqin o'lchovi). Yangi tool larni bilmagani uchun tez8 va churn8 agent to'plamlariga yugurtirilmagan. tez13-agent eski savollarda tez12-agent dan ancha yaxshi va tez8 ga yaqin, grafik va saqlashda ham o'sgan. Lekin fresh2 da — ko'rilmagan, odam yozgan savollarda — tez8 dan pastga tushgan. Qaror qoidasi (14-bobdagi, o'qitishdan oldin yozilgan) aynan shu o'rinda modelni rad etadi.

Bitta raqam yetmaydi: qaysi savollar buzilgan va qaysilari tuzalgan, buni tez8 ga nisbatan sanaymiz:

def buzilgan(natija, model):
    buz = tuz = 0
    for (m, s, i), (togri, _) in natija.items():
        if m != model:
            continue
        asl = natija[("tez8", s, i)][0]
        buz += asl and not togri
        tuz += togri and not asl
    return buz, tuz


for model in ("churn8", "tez12-agent", "tez13-agent"):
    (b1, t1), (b2, t2) = buzilgan(eski, model), buzilgan(fresh, model)
    print(f"{model:13s} eski: {b1:2d} buzildi, {t1:2d} tuzaldi   fresh2: {b2:2d} buzildi, {t2:2d} tuzaldi")

churn8        eski: 15 buzildi,  8 tuzaldi   fresh2:  4 buzildi,  6 tuzaldi
tez12-agent   eski: 35 buzildi, 17 tuzaldi   fresh2:  3 buzildi,  6 tuzaldi
tez13-agent   eski: 19 buzildi, 14 tuzaldi   fresh2: 11 buzildi,  8 tuzaldi

churn8 — ya'ni hech narsa o'zgarmasdan, faqat tasodif — eski savollarda 15 tasini buzadi. Demak 15 atrofidagi buzilish shovqin. tez12-agent ikki baravar ko'p buzgan; tez13-agent shovqin chegarasiga qaytgan. fresh2 da esa tez13-agent 11 tasini buzgan, shovqin esa 4 ta.

tez12-agent: aytilmagan joy

tez12-agent ni buzgan asosiy sabab o'qitish qatorlarida edi. So'rovlarni teacher model yozadi, keyin checker model (o'sha oiladan) so'rovni qayta o'qib reja yozadi; ikki reja bir xil natija bersa, qator qabul qilinadi. Teacher "Excelga saqla" deb yozib joyni tushirib qoldirsa, checker ham o'zicha "Downloads" ni qo'yadi — ikkalasi bir xil taxmin qilgani uchun noto'g'ri qator o'tib ketadi. Model shundan "joy aytilmasa ham, joy qo'y" degan qoidani o'rgandi.

Endi bu model-siz qoida bilan tekshiriladi: so'rovda nomlanmagan ixtiyoriy argument (joy, papka, grafik turi) rejada bo'lsa, qator rad etiladi yoki o'sha argument rejadan olib tashlanadi. Xuddi shu qoidani uch o'qitish fayliga qo'llaymiz:

QATORLAR = [json.loads(q) for q in (DATA / "agent" / "oqitish_qatorlari.jsonl").read_text(encoding="utf-8").splitlines()]
FAYLLAR = ("tez8", "tez12-agent", "tez13-agent")
print(f"{'fayl':13s} {'qatorlar':>8s} {'saqlash':>8s} {'joy aytilmagan':>15s}")
for fayl in FAYLLAR:
    q = [r for r in QATORLAR if r["fayl"] == fayl]
    saqlash = [r for r in q if "joy" in r]
    aytilmagan = sum(any(a.endswith(" place") for a in r["aytilmagan"]) for r in saqlash)
    print(f"{fayl:13s} {len(q):8d} {len(saqlash):8d} {aytilmagan:15d}")

fayl          qatorlar  saqlash  joy aytilmagan
tez8             19886     1100               0
tez12-agent      27127     3042             643
tez13-agent      32377     5287              13

tez12-agent faylida yuzlab saqlash qatori joyni so'rovda aytmasdan qo'yadi. tez13-agent faylida bunday qator deyarli yo'q: qolganlarining aksariyati eksportda ataylab qo'shilgan yozuv xatolari ("Destkop", "Dowlnoads"), qoida ularni o'qiy olmaydi; faqat bittasida joy haqiqatan aytilmagan. Natija yuqoridagi jadvalda ko'rinadi: eski savollardagi buzilish shovqin darajasiga qaytdi.

Talablar so'rov matnida ham bor

tez13-agent ning saqlashdagi xatolarini birma-bir ko'rib chiqqanda, sabab rejada emas, so'rov matnida chiqdi. Odamlar joyni o'z tilida aytadi ("ish stoliga", "yuklab olinganlarga"), Excel uchun ko'pincha shunchaki "saqla" deydi va papka nomini kirillda yozadi. Teacher esa "Desktop" va "Excel" deb yozardi. O'qitish fayllari va test to'plamlarini solishtiramiz:

import re

KIRILL = re.compile("[а-яёўқғҳА-ЯЁЎҚҒҲ]")
NOMI = {"Desktop": ["desktop", "десктоп"], "Documents": ["documents", "документс"], "Downloads": ["downloads", "даунлоадс"]}
FORMAT = ["excel", "эксел", "xlsx", "хлсх", "html", "хтмл", "ҳтмл"]


def ulush(qism, butun):
    return f"{qism / butun:4.0%} ({qism}/{butun})" if butun else "–"


testlar = []
for suite in (TEZ / "accuracy" / "suites").iterdir():
    for savol in load_json(suite / "questions.json"):
        if savol["expect"] == "ask":
            continue
        for qadam in savol["expect"][0]["steps"]:
            if qadam["tool"].startswith("export_"):
                testlar.append((savol["text"].lower(), qadam))
joy = [(t, q) for t, q in testlar if "place" in q["args"]]
ona = sum(not any(s in t for s in NOMI[q["args"]["place"]]) for t, q in joy)
excel = [(t, q) for t, q in testlar if q["tool"] == "export_xlsx"]
sozsiz = sum(not any(s in t for s in FORMAT) for t, _ in excel)
papka = [q for _, q in testlar if "folder" in q["args"]]
kirill = sum(bool(KIRILL.search(q["args"]["folder"])) for q in papka)
print(f"{'':13s} {'joy ona tilida':>16s} {'oddiy saqla':>16s} {'kirill papka':>16s}")
print(f"{'testlar':13s} {ulush(ona, len(joy)):>16s} {ulush(sozsiz, len(excel)):>16s} {ulush(kirill, len(papka)):>16s}")
for fayl in FAYLLAR:
    q = [r for r in QATORLAR if r["fayl"] == fayl and "joy" in r]
    j = [r for r in q if r["joy"] and not any(a.endswith(" place") for a in r["aytilmagan"])]
    x = [r for r in q if "format_sozsiz" in r and "export_xlsx" in r["asboblar"]]
    p = [r for r in q if r["papka_kirill"] is not None]
    print(f"{fayl:13s} {ulush(sum(r['joy'] == 'ona_tili' for r in j), len(j)):>16s} {ulush(sum(r['format_sozsiz'] for r in x), len(x)):>16s} {ulush(sum(r['papka_kirill'] for r in p), len(p)):>16s}")

                joy ona tilida      oddiy saqla     kirill papka
testlar            69% (11/16)      33% (10/30)        38% (3/8)
tez8                         –     6% (61/1100)                –
tez12-agent     12% (151/1299)    6% (131/2170)     12% (89/759)
tez13-agent     23% (479/2054)    8% (253/3301)   18% (179/1012)

tez13-agent fayli uchala ko'rsatkichda o'sgan — teacher prompt iga joylarning o'zbekcha va ruscha nomlari qo'shildi, Excel so'zini aytish majburiyati olib tashlandi, generator ga kirill papka nomlari qo'shildi. Lekin ulushlar hali ham testlardagidan ancha past: yangi qatorlar ko'p, eski qatorlarning "Desktop" va "Excel" i undan ham ko'p. 4B model bir necha yuz qatorli xulqni o'rgandi, 30 tadan kam qatorli xulqni o'rganmadi.

tez13-agent nega fresh2 da tushdi: so'rashni unutdi

fresh2 dagi 92 savoldan 13 tasida to'g'ri javob — aniqlashtirib so'rash (savol noaniq). Har bir model shu 13 ta savolda nima qilganini sanaymiz:

FRESH2 = {(s, q["id"]): q for s in REAL for q in load_json(TEZ / "accuracy" / "suites" / s / "fresh2.json")}


def soradi(yozuv):
    javob = parse_answer(yozuv["output"])
    return isinstance(javob, dict) and "ask" in javob


soraladi = [(s, i) for (m, s, i) in fresh if m == "tez8" and FRESH2[(s, i)]["expect"] == "ask"]
for model in MODELLAR:
    n = sum(soradi(fresh[(model, s, i)][1]) for s, i in soraladi)
    print(f"{model:13s} {len(soraladi)} ta noaniq savoldan {n} tasida so'radi")
for fayl in FAYLLAR:
    q = [r for r in QATORLAR if r["fayl"] == fayl]
    print(f"{fayl:13s} o'qitish faylida so'rash qatorlari: {ulush(sum(r['savol'] for r in q), len(q))}")

tez8          13 ta noaniq savoldan 3 tasida so'radi
churn8        13 ta noaniq savoldan 3 tasida so'radi
tez12-agent   13 ta noaniq savoldan 4 tasida so'radi
tez13-agent   13 ta noaniq savoldan 0 tasida so'radi
tez8          o'qitish faylida so'rash qatorlari:   5% (980/19886)
tez12-agent   o'qitish faylida so'rash qatorlari:   4% (980/27127)
tez13-agent   o'qitish faylida so'rash qatorlari:   3% (980/32377)

Har bir yangi o'qitish qatori modelga "bajar" deb o'rgatadi; yangi tool lar uchun bitta ham so'rash qatori qo'shilmadi. Natijada so'rash qatorlarining ulushi har o'qitishda kamaydi va tez13-agent noaniq savolda umuman so'ramay qo'ydi. Bu xavf oldindan aytilgan edi (tashqi tanqidda ham), o'lchangan edi (yangi qatorlarda 0 ta so'rash qatori), lekin keyinga qoldirildi. 12-fayldagi 38-xato — shu.

Ishonch darvozasi bu safar yordam bermadi

tez8 mahsulotda confidence gate bilan ishlaydi: javobning token ehtimolliklari geometrik o'rtachasi chegaradan past bo'lsa, javob o'rniga savol qaytadi. Chegara har model uchun alohida, faqat validation qatorlarida tanlanadi. tez13-agent ning 1 619 validation qatori bo'yicha tanlov:

qatorlar = [json.loads(q) for q in (NATIJA / "gate13_validation.jsonl").read_text(encoding="utf-8").splitlines()]


def togri_soni(chegara):
    soni = 0
    for r in qatorlar:
        ushlandi = not r["asked"] and r["confidence"] < chegara
        soni += (r["expect_ask"] and (r["asked"] or ushlandi)) or (not r["expect_ask"] and not ushlandi and not r["asked"] and r["match"])
    return soni


print(f"validation qatorlari: {len(qatorlar)}, so'rash kutilgan: {sum(r['expect_ask'] for r in qatorlar)}")
for chegara in (0.0, 0.90, 0.94, 0.97, 0.99):
    print(f"chegara {chegara:.2f}: {togri_soni(chegara)} to'g'ri")
guruh = Counter()
for (m, s, i), (togri, y) in fresh.items():
    if m != "tez13-agent" or soradi(y) or y.get("confidence") is None:
        continue
    nom = "to'g'ri" if togri else ("so'rashi kerak edi" if FRESH2[(s, i)]["expect"] == "ask" else "noto'g'ri bajardi")
    guruh[(nom, y["confidence"] < 0.97)] += 1
for nom in ("to'g'ri", "noto'g'ri bajardi", "so'rashi kerak edi"):
    print(f"fresh2, {nom:20s}: {guruh[(nom, True)]:2d} tasi 0,97 dan past, {guruh[(nom, False)]:2d} tasi yuqori")

validation qatorlari: 1619, so'rash kutilgan: 68
chegara 0.00: 1556 to'g'ri
chegara 0.90: 1556 to'g'ri
chegara 0.94: 1556 to'g'ri
chegara 0.97: 1552 to'g'ri
chegara 0.99: 1520 to'g'ri
fresh2, to'g'ri             :  4 tasi 0,97 dan past, 52 tasi yuqori
fresh2, noto'g'ri bajardi   :  9 tasi 0,97 dan past, 14 tasi yuqori
fresh2, so'rashi kerak edi  : 11 tasi 0,97 dan past,  2 tasi yuqori

Validation qatorlarida darvoza hech narsa bermaydi: model u yerda deyarli doim to'g'ri va so'rash kerak bo'lgan joyda doim so'raydi, chunki bu qatorlar o'qitish qatorlari bilan bir xil teacher tomonidan yozilgan. Qoida bo'yicha eng past teng chegara olinadi, ya'ni darvoza amalda o'chiq. fresh2 da esa ishonch farq qiladi: so'rashi kerak bo'lgan javoblarning ko'pchiligi past ishonchda. Lekin bu chegarani test to'plamidan o'qib bo'lmaydi — u holda raqam o'zini aldaydi (24-bob). Darvoza uchun test bo'lmagan, lekin odam yozgan so'rovlar to'plami kerak.

Million qator

Professor so'radi: "5 000 qatorli fayl ochiladi deyilyapti, 1 million qatorda muammo bo'lmaydimi?" Model qatorlarni o'qimaydi: u faqat ustun nomlari va bir nechta namuna qiymatni ko'radi va qisqa reja yozadi. Rejani DuckDB barcha qatorlar ustida bajaradi. Shuning uchun fayl hajmi modelning ishini deyarli o'zgartirmaydi; o'zgaradigani — faylni ochish vaqti va xotira. Ilovaning o'z kodi bilan 1 000 000 qatorli sintetik bank jadvalida o'lchandi (o'lchov build ichida emas, 2026-11-07 da alohida bajarilgan; natija faylda saqlangan):

print((TEZ / "results" / "million_rows_wsl.txt").read_text(encoding="utf-8"))

2026-11-07, WSL on the development PC (8 cores, 15 GB), DuckDB 2.0.0, synthetic bank table from spike1_excel_read/fixture/spec.json, 1,000,000 rows, seed 7.
Generated: mln.csv 143.0 MB, mln.xlsx 93.2 MB. Opened with app.table.open_table (read, describe, find problems), then four plans with accuracy.execute.run_plan.
mln.csv: opened in 0.9 s, peak 727 MiB, rows 1,000,000
  sum by branch                         0.01 s, 13 rows back
  filter + top 10                       0.07 s, 10 rows back
  count distinct clients per channel    0.01 s, 4 rows back
  remove duplicates                     1.08 s, 997018 rows back
  peak after queries 1445 MiB
mln.xlsx: opened in 17.4 s, peak 1978 MiB, rows 1,000,000
  sum by branch                         0.01 s, 12 rows back
  filter + top 10                       0.05 s, 10 rows back
  count distinct clients per channel    0.01 s, 4 rows back
  remove duplicates                     1.07 s, 997018 rows back
  peak after queries 1978 MiB

CSV bir soniyadan kam vaqtda ochiladi, Excel fayl esa 17 soniya atrofida: sekin qismi — so'rovlar emas, XLSX ni o'qish. Cheklovlar ham bor: oynada 200 qator ko'rinadi (hisob hamma qatorda), Excelga saqlash 1 048 575 qator bilan cheklangan (bu Excel varag'ining o'z chegarasi), o'lchov kuchli kompyuterda qilingan, 1 milliondan katta fayl va 8 GB xotirali kompyuter hali sinalmagan. Model uchun yana taxminan 4,5 GB kerak (26-bob).

Uch tashqi tanqid

Ish davomida foydalanuvchi uchta AI (Gemini, Grok, ChatGPT) dan pipeline haqida fikr so'radi. Har bir da'vo fayllar bilan solishtirildi (16-faylning 13–15-bo'limlari). Foydali chiqqanlari: oxiridagi tinish belgisi (testlarda 1%, o'qitishda 41% — lekin tez8 ham shu ulush bilan yaxshi ishlaydi, hal qiluvchi emas), checker solishtira olmaydigan argument — fayl nomi (36-xato), "different result" larni rejaning qaysi qismi buzganiga ko'ra bo'lish. Rad etilganlari: rejalarni aynan bir xil bo'lishini talab qilish (bir xil natijali ikki xil reja modelga teng deb o'rgatilishi kerak), ikkinchi checker ni olib tashlash (u aynan eng qisqa, odamga o'xshash so'rovlarni qutqaradi). Eng muhim ogohlantirish — so'rash qatorlari — to'g'ri edi va keyinga qoldirilgani qimmatga tushdi.

Xatolar jurnalidagi yangi qatorlar

jurnal = (TEZ.parents[1] / "12-mistakes-and-lessons.md").read_text(encoding="utf-8").splitlines()
for satr in jurnal:
    qism = [q.strip() for q in satr.split("|")]
    if len(qism) > 4 and qism[1].isdigit() and 34 <= int(qism[1]) <= 38:
        print(f"{qism[1]}. {qism[2]}: {qism[3][:150]}...")

34. 2026-11-06: The tez12-agent training rows were measured for tool mix and length mix, never for what each argument needs from the request: 639 of 1,942 save rows (...
35. 2026-11-06: Designed the data pipeline (teacher Gemma, checker Gemma, execution-only verification) and three fixes for tez12-agent without reading what the litera...
36. 2026-11-06: The named-argument rule (mistake 34) covered place, folder and chart kind but not the file name of a save, although `file_name` is required and the ap...
37. 2026-11-06: After the 35B trial failed, I stopped the engine process I could see (qwen36.exe) and not its parent, the colibri Python wrapper that held the port. T...
38. 2026-11-06: Deferred ask rows for the new tools (an outside critique raised it; I measured 0 ask rows in 6,366 new agent rows and wrote it as proposal g) with the...

Keyingi qadam: tez14

tez14 uchun reja faqat o'lchangan kamchiliklardan iborat, boshqa hech narsa o'zgarmaydi:

1. Har bir yangi tool uchun so'rash qatorlari: qo'llab-quvvatlanmaydigan joy ("C diskka saqla"), chizadigan narsasi yo'q grafik, bitta ustunli hisob. So'rash ulushi testlardagiga yaqin bo'lishi kerak, va gate endi bu ulushni har bir tool uchun alohida tekshiradi.
2. Ustun nomida sana bor jadval ("01122024ulushifoizda"): "2024 yil dekabr" so'zi sana filtri emas, ustun tanlash ekanini o'rgatadigan bitta yangi domen.
3. Saqlash konventsiyalari uchun qo'shimcha qatorlar: "yuklab olinganlar", ">" bilan yozilgan ichma-ich papka, o'zbek kirillidagi papka nomi.
4. Gate so'rov matnidagi konventsiyalarni ham o'lchaydi (joy qanday aytilgan, so'rash ulushi), faqat reja shaklini emas.
5. Darvoza uchun odam yozgan, test bo'lmagan so'rovlar to'plami.

Bu bobdagi saboq kitob boshidagi saboq bilan bir xil: model deyarli hech qachon asosiy muammo bo'lmagan. Muammo — ma'lumot odamlar aslida qanday so'rashiga o'xshamasligida. Ko'proq ma'lumot yordam bermadi; haqiqiy so'rovlarga o'xshagan va har bir xulqni o'lchaydigan ma'lumot yordam beradi.