tables.py — jadvallar

4-bob 6 kod bloki 659 so'z

Endi eng muhim qaror: ma'lumotni qanday saqlaymiz?

To'rtta jadval

python7 qator

documents      <- butun hujjat (PDF, sayt sahifasi)
   |
   | bitta hujjat -> ko'p bo'lak
   v
chunks         <- matn bo'lagi + VEKTOR
settings       <- ilova sozlamalari
chat_history   <- savol-javob tarixi

Nega ikkita jadval, bitta emas?

Chunki qidiruv bo'lak darajasida bo'ladi, hujjat darajasida emas. 50 sahifali PDF ni bitta vektorga aylantirsangiz — u vektor hech narsani anglatmaydi. "Bu hujjat hamma narsa haqida" degani bo'ladi.

Lekin javob berganda foydalanuvchiga manba ko'rsatish kerak: "bu javob falon hujjatdan olindi". Shuning uchun bog'lanish saqlanadi.

Kod

python60 qator

import uuid
from sqlalchemy import (Column, String, Text, Integer, DateTime,
                        ForeignKey, Index, func)
from sqlalchemy.dialects.postgresql import UUID, JSONB
from sqlalchemy.orm import relationship
from pgvector.sqlalchemy import Vector

from app.database import Base
from app.config import settings

_MODEL_DIM = {
    "bge-m3": 1024,
    "nomic-embed-text": 768,
    "mxbai-embed-large": 1024,
    "all-minilm": 384,
}


def _local_dim() -> int:
    nom = settings.ollama_embed_model.split(":")[0]
    return _MODEL_DIM.get(nom, 1024)


LOCAL_DIM = _local_dim()
CLOUD_DIM = 1536


class Document(Base):
    __tablename__ = "documents"
    __table_args__ = {"schema": "rag"}

    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    title = Column(String(500), nullable=False)
    source_type = Column(String(50), nullable=False)
    source_url = Column(Text, nullable=True)
    language = Column(String(10), default="uz")
    created_at = Column(DateTime(timezone=True), server_default=func.now())

    chunks = relationship("Chunk", back_populates="document",
                          cascade="all, delete-orphan")


class Chunk(Base):
    __tablename__ = "chunks"
    __table_args__ = (
        Index("idx_chunks_document_id", "document_id"),
        {"schema": "rag"},
    )

    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    document_id = Column(UUID(as_uuid=True),
                         ForeignKey("rag.documents.id", ondelete="CASCADE"))
    chunk_text = Column(Text, nullable=False)
    chunk_index = Column(Integer, nullable=False)
    language = Column(String(10), default="uz")

    embedding_local = Column(Vector(LOCAL_DIM), nullable=True)
    embedding_cloud = Column(Vector(CLOUD_DIM), nullable=True)

    document = relationship("Document", back_populates="chunks")

Eng muhim qator

python1 qator

embedding_local = Column(Vector(LOCAL_DIM), nullable=True)

Vector(1024) — bu pgvector bergan maxsus tur. Oddiy PostgreSQL'da bunday ustun yo'q.

O'lcham ANIQ mos kelishi shart. bge-m3 1024 ta son qaytaradi. Agar ustun Vector(768) bo'lsa, saqlashda xato chiqadi:

matn1 qator

expected 768 dimensions, not 1024

Shuning uchun o'lcham modeldan hisoblanadi, qo'lda yozilmaydi:

python1 qator

LOCAL_DIM = _local_dim()

Modelni .env da o'zgartirsangiz — ustun o'lchami avtomatik moslashadi. Agar buni qo'lda 1024 deb yozib qo'ysangiz, model almashtirganda sabab topolmay yarim kun yo'qotasiz.

nullable=True nega?

Chunk avval matn sifatida saqlanadi, vektor keyin qo'shiladi. Sabablar:

Embedding sekin — 1000 ta chunk uchun bir necha daqiqa
Ollama ishlamayotgan bo'lishi mumkin
Model almashtirsangiz — hammasini qayta hisoblash kerak

Shuning uchun jarayon ikki bosqichli: avval matn, keyin vektor.

cascade — bolalarni o'chirish

python2 qator

chunks = relationship("Chunk", back_populates="document",
                      cascade="all, delete-orphan")

Hujjat o'chirilsa — uning barcha chunk'lari ham o'chadi. Busiz bazada yetim chunk'lar qoladi: matn bor, lekin qaysi hujjatdan kelgani noma'lum. Qidiruv ularni topadi, manba ko'rsata olmaydi.

ondelete="CASCADE" — bu SQL darajasida, cascade= esa Python darajasida. Ikkalasi ham kerak: birinchisi baza to'g'ridan-to'g'ri o'zgartirilsa ishlaydi, ikkinchisi SQLAlchemy orqali.

schema="rag" nega?

Jadvallar public emas, rag schema'sida. Bu papka kabi: bazada boshqa loyihalar bo'lsa, aralashib ketmaydi. To'liq nomi: rag.documents.

oldingi3. database.py — bazaga ulanish keyingi5. bootstrap.py — birinchi G'ALABA

Bu bobni bepul o'qidingiz.

Kitob o'rgatadi, lekin sizning kodingiz nega ishlamaganini ayta olmaydi. Kursda uy vazifangiz tekshiriladi.

Telegramda yozish Narxlar