How Vector Databases Work — distiliaa Learn

In the last lesson you built semantic search with a brute-force loop: compare the query to every document, sort, done. That works beautifully — right up until you have ten million vectors and your "search" takes seconds per query. Vector databases exist to make nearest-neighbor search fast at scale, and the trick they use is delightfully pragmatic: they stop being exact.

Why does this exist?

Comparing a query against N vectors costs O(N) — every extra document makes every search slower. At millions of vectors and thousands of queries per second, exact search melts your servers. Vector databases trade a tiny bit of accuracy (maybe missing the 47th-best match) for orders-of-magnitude speedups using approximate nearest neighbor (ANN) indexes. Nobody notices the missing match; everybody notices a 3-second search.

The scale problem, in numbers

Say each vector has 1536 dimensions stored as 4-byte floats — about 6 KB per vector. With 10 million documents:

Storage: ~60 GB of raw vectors.
One brute-force query: 10 million distance computations, each touching 1536 floats. Even at billions of FLOPS, that's tens to hundreds of milliseconds of pure math — per query, per core.

You can't cache your way out (every query vector is different), so you need a smarter data structure.

Approximate nearest neighbors

The key insight: users don't need the mathematically exact top-10. They need 10 highly relevant results, fast. ANN indexes exploit this by organizing vectors so the search only visits a small, promising fraction of the data.

Two families dominate:

1. IVF — inverted file index

Cluster all vectors into, say, 1,000 groups (using k-means). At query time, find the few clusters whose centers are closest to the query, and search only inside those clusters. If you probe 10 of 1,000 clusters, you've cut the work by ~99%.

The knob is nprobe: probe more clusters → better recall, slower search.

2. HNSW — hierarchical navigable small world graphs

Build a graph where each vector links to its near neighbors, with a hierarchy of "express lanes": sparse top layers for long jumps, dense bottom layers for precision. Search starts at the top, greedily hops toward the query, and descends. It's like navigating flights → highways → streets to find an address.

HNSW is the default in most modern vector DBs (Qdrant, Weaviate, pgvector's HNSW index) because it has excellent speed/recall trade-offs — at the cost of more memory and slower index builds.

The universal trade-off triangle

Every ANN index balances three things — you get to pick two:

Recall — what fraction of the true nearest neighbors you actually find (e.g. 95%).
Latency — how fast each query returns.
Memory / build cost — how much RAM the index eats and how long it takes to build or update.

Tuning parameters (nprobe for IVF, ef_search for HNSW) slide you along the recall/latency curve at runtime. There's no free lunch, only well-chosen compromises.

Measure recall on YOUR data

Index benchmarks use standard datasets that may look nothing like your embeddings. Before shipping, take 1,000 real queries, compute exact top-10 with brute force, and check what fraction your index returns. If recall@10 is above ~0.95, users will never notice the approximation.

What a vector database adds beyond the index

An ANN library (like FAISS) is just the math. A vector database wraps it with the boring-but-essential stuff:

Metadata filtering — "nearest neighbors where lang = 'en' and date > 2024". Harder than it sounds: filtering after ANN search can return too few results, so good DBs filter during traversal.
CRUD — inserting and deleting vectors without rebuilding the whole index.
Persistence, replication, backups — it's your data, treat it like data.
Hybrid search — combining vector scores with keyword (BM25) scores, which you met in the RAG module.

# Typical vector DB usage (Qdrant-style pseudocode)
client.create_collection(
    "docs",
    vectors_config={"size": 1536, "distance": "Cosine"},
)

client.upsert("docs", points=[
    {"id": 1, "vector": embed(text), "payload": {"lang": "en", "source": "faq.md"}},
])

hits = client.search(
    "docs",
    query_vector=embed("how do I reset my password?"),
    query_filter={"must": [{"key": "lang", "match": {"value": "en"}}]},
    limit=5,
)

Do you even need one?

Honest sizing guide:

Under ~100k vectors: NumPy in memory, or pgvector in the Postgres you already run. Brute force is fine.
100k – 10M: pgvector with an HNSW index, or a managed vector DB. Pick whatever fits your ops story.
10M+ or strict latency SLOs: dedicated vector database with tuned HNSW/IVF, sharding, and replicas.

The most common production mistake is adopting a new database for 50,000 vectors that Postgres would happily handle.

Build it yourself

Generate 100,000 random 384-dim vectors and time a brute-force top-10 query.
Install FAISS or pgvector, build an HNSW index over the same vectors, and time it again.
Compute recall@10 against your brute-force results for 100 queries.
Turn ef_search down until recall drops below 0.9 — feel the trade-off with your own hands.

Summary

Exact nearest-neighbor search scales linearly with data — too slow past a few hundred thousand vectors.
ANN indexes (IVF clusters, HNSW graphs) search a small promising subset, trading recall for speed and memory.
Tune the recall/latency trade-off with nprobe / ef_search, and measure recall on your own data.
A vector database adds filtering, CRUD, persistence, and hybrid search on top of the raw index.
Don't over-provision: small datasets are happiest in the database you already run.