Vector Database (Vektorli Baza)

6-bob 20 kod bloki 2733 so'z

Vector database nima?

Oddiy ma'lumotlar bazasi (PostgreSQL, MySQL) matn, raqamlar va sanalarni saqlaydi. Siz WHERE name = 'Nihol' deb aniq qidiruv qilasiz. Lekin RAG tizimida savol boshqacha: "Bu savolga mazmun jihatdan eng yaqin matn qaysi?"

Vector database -- bu vektorlarni (sonlar ro'yxatini) saqlash va ular orasida o'xshashlik bo'yicha qidiruv qilish imkonini beradigan tizim. Har bir matn chunk embedding model orqali vektorga aylantiriladi, so'ngra qidiruv vaqtida savol ham vektorga aylantiriladi va eng yaqin vektorlar topiladi.

Analogiya: oddiy bazada siz kutubxonadan kitob nomini aniq yozib qidirasiz. Vector database da esa "yog'och uylar haqida kitob" desangiz, mazmunan mos kitoblarni topadi -- garchi nomida "yog'och" so'zi bo'lmasa ham.

Nima uchun oddiy PostgreSQL yetarli emas?

Oddiy PostgreSQL da: - vector turi yo'q -- 4096 ta sonni qayerga saqlaysiz? TEXT ustuniga JSON sifatida? Sekin. - <=> (cosine distance) operatori yo'q -- o'xshashlik hisoblash uchun SQL funksiya yozish kerak. - Vector index yo'q -- 100K vektor orasida har safar hammani tekshirish (sequential scan) juda sekin.

pgvector bu muammolarning barchasini hal qiladi -- PostgreSQL ni vector database ga aylantiradi.

pgvector batafsil

O'rnatish

-- Extension ni yoqish (bir marta)
CREATE EXTENSION IF NOT EXISTS vector;

Nihol loyihasida schema faylida aynan shu qator mavjud:

-- 20260218000000_consolidated_schema.sql dan
CREATE EXTENSION IF NOT EXISTS vector;

Ustun turi

pgvector vector(N) turini qo'shadi, bu yerda N -- vektor o'lchami:

-- 1536 o'lchamli vektor (OpenAI text-embedding-3-small)
embedding_cloud vector(1536)

-- 4096 o'lchamli vektor (qwen3-embedding:8b)
embedding_local vector(4096)

O'lcham embedding model ga bog'liq: - OpenAI text-embedding-3-small -- 1536 - OpenAI text-embedding-3-large -- 3072 - qwen3-embedding:8b -- 4096 - Cohere embed-multilingual-v3 -- 1024

O'lcham nima uchun muhim? Kattaroq o'lcham = ko'proq ma'lumot saqlanadi = aniqroq qidiruv. Lekin ko'proq xotira va sekinroq hisoblash. 1536 ko'p holatlarda yetarli.

Distance operatorlari

pgvector uchta masofani hisoblash operatorini taqdim etadi:

Cosine distance nima uchun matn uchun eng mos? Chunki u vektorlarning yo'nalishini taqqoslaydi, hajmini emas. Ikki gap bir xil mavzuda bo'lsa, ularning vektorlari bir xil yo'nalishda bo'ladi -- garchi biri uzun, ikkinchisi qisqa bo'lsa ham. Cosine distance buni to'g'ri baholaydi.

Amaliy misol: "Nihol bank loyihalari" va "Nihol banking projects" -- turli tillarda, lekin cosine distance past bo'ladi, chunki mazmunan yaqin.

Bizning SQL so'rovimiz

Nihol loyihasida qidiruv quyidagicha amalga oshiriladi:

-- chat.py dan olingan haqiqiy SQL so'rov
SELECT
    ch.id,
    ch.content_id,
    ch.chunk_text,
    c.title,
    c.url,
    1 - (ch.embedding_local <=> :vec) AS similarity
FROM wde.lv_content_chunks ch
JOIN wde.lv_extracted_content c ON c.id = ch.content_id
WHERE ch.embedding_local IS NOT NULL
ORDER BY ch.embedding_local <=> :vec
LIMIT :lim

Bu so'rov nima qiladi: 1. ch.embedding_local <=> :vec -- har bir chunk ning embedding vektori bilan so'rov vektori orasidagi cosine distance ni hisoblaydi. 2. 1 - distance = similarity. Distance 0 ga yaqin = juda o'xshash. Similarity 1 ga yaqin = juda o'xshash. 3. ORDER BY distance -- eng yaqinlarini birinchi qo'yadi. 4. LIMIT :lim -- faqat eng yaxshi N ta natijani qaytaradi (bizda RAG_TOP_K = 10).

Til bo'yicha filtrlash ham mavjud:

-- Til filtri bilan variant
SELECT
    ch.id, ch.content_id, ch.chunk_text, c.title, c.url,
    1 - (ch.embedding_local <=> :vec) AS similarity
FROM wde.lv_content_chunks ch
JOIN wde.lv_extracted_content c ON c.id = ch.content_id
JOIN wde.lv_target_pages p ON p.id = c.page_id
JOIN wde.lv_languages l ON l.id = p.language_id
WHERE ch.embedding_local IS NOT NULL
  AND l.code = :lang
ORDER BY ch.embedding_local <=> :vec
LIMIT :lim

Bu variant faqat belgilangan tildagi chunklar orasidan qidiradi. Agar foydalanuvchi o'zbek tilida savol bersa, faqat o'zbek tilidagi chunklar qaytariladi.

Similarity threshold

Natijalarni qaytarishdan oldin threshold (chegara) tekshiriladi:

# chat.py dan olingan haqiqiy kod
return [
    {
        "chunk_id": r[0],
        "content_id": r[1],
        "chunk_text": r[2],
        "title": r[3],
        "url": r[4],
        "similarity": float(r[5]),
    }
    for r in rows
    if float(r[5]) >= threshold  # RAG_SIMILARITY_THRESHOLD (default 0.0, admin paneldan sozlanadi)
]

Agar chegara qo'yilgan bo'lsa, undan past similarity li chunk qaytarilmaydi. Joriy sozlamada chegara 0.0 (filtrsiz) -- top_k va heuristik tanlash shovqinni o'zi kamaytiradi, lekin kerak bo'lsa admin paneldan ko'tarish mumkin. Masalan, foydalanuvchi "ob-havo" haqida so'rasa, bizning bazada bu haqda ma'lumot yo'q -- barcha similarity lar past bo'ladi va bo'sh natija qaytariladi.

IVFFlat index

Katta hajmli bazalar uchun (10K+ vektor) har safar barcha vektorlarni tekshirish (sequential scan) juda sekin. IVFFlat index bu muammoni hal qiladi:

CREATE INDEX idx_chunks_embedding ON chunks
    USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

IVFFlat qanday ishlaydi? Vektorlarni klasterlarga bo'ladi (bu misolda 100 ta klaster). Qidiruv vaqtida avval eng yaqin klasterlarni topadi, keyin faqat o'sha klasterlar ichida qidiradi. Tezlik 10-50x oshadi, lekin 100% aniqlik kafolatlanmaydi (approximate search).

lists parametri -- klasterlar soni. Qoida: lists = sqrt(total_vectors). 100K vektor uchun lists = 316. 1M vektor uchun lists = 1000.

Nihol loyihasining cloud versiyasida (Supabase) IVFFlat index ishlatiladi:

-- consolidated_schema.sql dan
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON chunks
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

HNSW index

IVFFlat dan aniqroq, lekin ko'proq xotira ishlatadi:

CREATE INDEX idx_embedding_hnsw ON chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

HNSW (Hierarchical Navigable Small World) -- graf asosidagi algoritm. Har bir vektor bir nechta boshqa vektorga bog'langan. Qidiruv grafda "yurish" orqali amalga oshadi.

• m = 16 -- har bir node ning maksimal bog'lanishlar soni. Kattaroq = aniqroq, lekin ko'proq xotira.
• ef_construction = 64 -- index qurilayotganda qidiruv chuqurligi. Kattaroq = sifatliroq index, lekin sekinroq quriladi.

Qachon index kerak?

Nihol loyihasida ~377 ta chunk bor. Index kerak emas -- sequential scan millisekundlarda ishlaydi.

Boshqa vector database lar

Pinecone

Managed cloud vector database. Serverless rejimi mavjud:

# DIQQAT: pinecone.init() 3.0 (2024) da OLIB TASHLANGAN.
# Eski kod: pinecone.init(api_key=..., environment=...)
from pinecone import Pinecone

pc = Pinecone(api_key="xxx")
index = pc.Index("nihol-kb")

# Vektor qo'shish
index.upsert(vectors=[
    ("chunk-1", [0.1, 0.2, ...], {"title": "Bank loyihasi", "lang": "uz"})
])

# Qidiruv
results = index.query(vector=[0.1, 0.2, ...], top_k=5, include_metadata=True)

Afzalligi: hech narsa o'rnatish kerak emas, scaling avtomatik. Kamchiligi: vendor lock-in, narxi oshib ketishi mumkin, ma'lumotlar chet elda.

Weaviate

Open source, GraphQL API, multi-modal (matn + rasm):

import weaviate

client = weaviate.Client("http://localhost:8080")

# Collection yaratish
client.schema.create_class({
    "class": "Chunk",
    "vectorizer": "text2vec-openai",
    "properties": [
        {"name": "content", "dataType": ["text"]},
        {"name": "source", "dataType": ["string"]},
    ]
})

Chroma

Eng yengil variant, prototyping uchun ideal:

import chromadb

client = chromadb.Client()
collection = client.create_collection("nihol")

collection.add(
    documents=["Nihol bank loyihasi", "Nihol telekom loyihasi"],
    ids=["chunk-1", "chunk-2"],
)

results = collection.query(query_texts=["bank ishlari"], n_results=2)

Afzalligi: pip install, hech qanday server kerak emas, in-memory. Kamchiligi: production uchun mos emas, katta hajmga chidamaydi.

Qdrant

Rust da yozilgan, juda tez:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient("localhost", port=6333)

client.create_collection(
    collection_name="nihol",
    vectors_config=VectorParams(size=4096, distance=Distance.COSINE),
)

Afzalligi: payload filtering (metadata bo'yicha filtrlash) juda kuchli. Kamchiligi: alohida server.

FAISS

Facebook ning kutubxonasi. Database emas, in-memory library:

import faiss
import numpy as np

dimension = 4096
index = faiss.IndexFlatIP(dimension)  # Inner product

vectors = np.random.rand(377, dimension).astype("float32")
faiss.normalize_L2(vectors)  # Cosine uchun normalize qilish
index.add(vectors)

query = np.random.rand(1, dimension).astype("float32")
faiss.normalize_L2(query)
distances, indices = index.search(query, k=5)

Afzalligi: eng tez batch qidiruv. Kamchiligi: persistence yo'q, metadata yo'q, faqat vektorlar bilan ishlaydi.

Milvus

Distributed tizim, milliardlab vektorlar uchun:

from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType

connections.connect("default", host="localhost", port="19530")
# ... schema va collection yaratish

Afzalligi: horizontal scaling, milliardlab vektor. Kamchiligi: murakkab o'rnatish (etcd + MinIO + Milvus), kichik loyihalar uchun overkill.

Katta taqqoslash jadvali

Nima uchun biz pgvector tanladik?

Nihol loyihasida pgvector tanlashning sabablari:

1. PostgreSQL allaqachon ishlatilmoqda. Sahifalar, kontentlar, chunklar -- hammasi PostgreSQL da. Alohida vector database qo'shish = yana bir server, yana bir ulanish, yana bir nuqta buzilishi mumkin.

2. Bitta SQL so'rovda hammasi. Vector search + metadata filter + JOIN -- barchasini bitta SQL da yozish mumkin. Pinecone da avval vector search, keyin alohida so'rov bilan metadata olish kerak.

3. ~377 ta chunk uchun yetarli. pgvector millionlab vektorlarni ham ishlata oladi, lekin bizning scale da hech qanday cheklov yo'q.

4. SQL tanish. Jamoa PostgreSQL va SQL ni yaxshi biladi. Yangi query language o'rganish kerak emas.

5. Ikki deployment uchun biri. Local versiyada oddiy PostgreSQL + pgvector. Cloud versiyada Supabase (PostgreSQL based) + pgvector. Bir xil schema, bir xil so'rovlar.

Bizning dual-column yondashuvimiz

Nihol loyihasida bitta jadvalda ikkita embedding ustuni mavjud:

# tables.py dan olingan haqiqiy kod
class LvContentChunks(Base):
    __tablename__ = "lv_content_chunks"
    __table_args__ = {"schema": "wde"}

    id = Column(Integer, primary_key=True)
    content_id = Column(Integer, ForeignKey("wde.lv_extracted_content.id"), nullable=False)
    chunk_text = Column(Text, nullable=False)
    content_size_bytes = Column(Integer)
    embedding_local = Column(Text)   # qwen3-embedding (4096 o'lcham)
    embedding_cloud = Column(Text)   # OpenAI embedding (1536 o'lcham)
    created_on = Column(DateTime(timezone=True), nullable=False)

Nima uchun ikkita ustun?

• embedding_local -- Ollama orqali qwen3-embedding:8b bilan yaratiladi. 4096 o'lcham. Bepul, internet kerak emas, lekin GPU kerak.
• embedding_cloud -- OpenAI API orqali text-embedding-3-small bilan yaratiladi. 1536 o'lcham. Pullik, internet kerak, lekin tez va GPU kerak emas.

Local versiyada embedding_local ustunidan qidiriladi. Cloud versiyada embedding_cloud ustunidan. Bir xil chunk matn, turli embedding lar.

Amaliy: jadval yaratish

pgvector bilan jadval yaratish va ma'lumot qo'shish:

-- 1. Extension yoqish
CREATE EXTENSION IF NOT EXISTS vector;

-- 2. Jadval yaratish
CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    metadata JSONB DEFAULT '{}',
    embedding vector(1536)
);

-- 3. Vektor qo'shish
INSERT INTO chunks (content, metadata, embedding)
VALUES (
    'Nihol kompaniyasi 1997 yilda tashkil topgan',
    '{"source": "nihol.uz", "lang": "uz"}',
    '[0.023, -0.041, 0.089, ...]'  -- 1536 ta son
);

-- 4. Cosine similarity bo'yicha qidiruv
SELECT
    id,
    content,
    1 - (embedding <=> '[0.019, -0.038, 0.091, ...]') AS similarity
FROM chunks
WHERE embedding IS NOT NULL
ORDER BY embedding <=> '[0.019, -0.038, 0.091, ...]'
LIMIT 5;

-- 5. Index yaratish (10K+ vektor uchun)
CREATE INDEX idx_chunks_embedding
    ON chunks USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

-- 6. Metadata bilan filtrlash
SELECT
    id,
    content,
    1 - (embedding <=> '[0.019, ...]') AS similarity
FROM chunks
WHERE embedding IS NOT NULL
  AND metadata->>'lang' = 'uz'
ORDER BY embedding <=> '[0.019, ...]'
LIMIT 5;

Hybrid search

Faqat vector search yetarli emasmi? Ko'pincha ha, lekin ba'zi holatlarda keyword search ham kerak. Masalan, "HUMO" so'zini vector search topa olmasa (noyob atama), keyword search topadi.

Nihol loyihasining joriy versiyasida hybrid search ishlatilmaydi -- sof vector search yetarli natija bermoqda. Lekin loyihaning oldingi (cloud, Supabase) iteratsiyasida hybrid search sinab ko'rilgan edi -- vector search va keyword search natijalarini RRF (Reciprocal Rank Fusion) orqali birlashtirish. Qanday ishlashini bilish foydali:

-- consolidated_schema.sql dan olingan haqiqiy funksiya (qisqartirilgan)
CREATE OR REPLACE FUNCTION hybrid_search_chunks(
    query_text TEXT,
    query_embedding vector(1536),
    match_threshold float,
    match_count int,
    final_count int,
    p_user_id uuid
) RETURNS TABLE (...) AS $$
BEGIN
    RETURN QUERY
    WITH vector_search AS (
        SELECT c.id, c.content,
            1 - (c.embedding <=> query_embedding) AS similarity,
            ROW_NUMBER() OVER (ORDER BY c.embedding <=> query_embedding) AS rank
        FROM chunks c
        WHERE 1 - (c.embedding <=> query_embedding) > match_threshold
        LIMIT match_count
    ),
    keyword_search AS (
        SELECT c.id, c.content,
            ts_rank(to_tsvector('russian', c.content),
                    plainto_tsquery('russian', query_text)) AS rank_score,
            ROW_NUMBER() OVER (ORDER BY rank_score DESC) AS rank
        FROM chunks c
        WHERE to_tsvector('russian', c.content) @@ plainto_tsquery('russian', query_text)
        LIMIT match_count
    ),
    rrf_scores AS (
        SELECT
            COALESCE(v.id, k.id) AS id,
            COALESCE(v.content, k.content) AS content,
            (COALESCE(1.0 / (60 + v.rank), 0.0)
           + COALESCE(1.0 / (60 + k.rank), 0.0)) AS rrf_score
        FROM vector_search v
        FULL OUTER JOIN keyword_search k ON v.id = k.id
    )
    SELECT * FROM rrf_scores ORDER BY rrf_score DESC LIMIT final_count;
END;
$$;

RRF formulasi: score = 1/(k + rank_vector) + 1/(k + rank_keyword). Bu yerda k = 60 (standart qiymat). Agar chunk ikkala qidiruvda ham yuqori o'rinda bo'lsa, uning RRF ball yuqori bo'ladi.

Xulosa

Vector database tanlashda asosiy savollar:

1. Allaqachon PostgreSQL ishlatilmoqdami? Ha -- pgvector. Yangi baza qo'shmang.
2. Necha vektor bo'ladi? 100K dan kam -- pgvector yetarli. Millionlab -- Qdrant yoki Milvus.
3. Managed xizmat kerakmi? Ha -- Pinecone yoki Supabase (pgvector). Yo'q -- self-hosted.
4. Murakkab filtrlash kerakmi? Ha -- Qdrant yoki pgvector (SQL). Yo'q -- Chroma yoki FAISS.
5. Budget? Cheklangan -- open source (pgvector, Qdrant, Milvus). Cheklanmagan -- Pinecone.

Nihol loyihasi uchun pgvector to'g'ri tanlov: sodda, ishonchli, allaqachon mavjud infratuzilma ichida.

class LvContentChunks(Base):
    __tablename__ = "lv_content_chunks"
    __table_args__ = {"schema": SCHEMA}

    id = Column(Integer, lv_chunk_id_seq, primary_key=True, server_default=lv_chunk_id_seq.next_value())
    content_id = Column(Integer, ForeignKey("wde.lv_extracted_content.id", ondelete="CASCADE"), nullable=False)
    chunk_text = Column(Text, nullable=False)
    content_size_bytes = Column(Integer)
    embedding_local = Column(Text)
    embedding_cloud = Column(Text)
    created_on = Column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc))

    content = relationship("LvExtractedContent", back_populates="chunks")

def retrieve_chunks(
    session: Session,
    query_embedding: list[float],
    top_k: int | None = None,
    threshold: float | None = None,
    lang_code: str | None = None,
    rubrics: list[str] | None = None,