Chunking

Before a document can be searched, it has to be split into pieces — chunks. This sounds like a boring preprocessing detail, and that's exactly why it causes so many production RAG failures: nobody thinks about it until retrieval quality is mysteriously bad. Chunking decisions quietly determine what your system can and cannot find.

The problem: units of retrieval

When the retriever runs, it returns the top few chunks — whatever those are — and pastes them into the prompt. That gives chunks two jobs at once:

1. Be findable. The chunk's embedding must sharply represent its content, so the right question lands on it.
2. Be usable. Once retrieved, the chunk must contain enough surrounding information for the model to actually answer.

These jobs pull in opposite directions. Small chunks are precise to find but may cut off the context needed to answer ("Refunds take 5-7 business days" — refunds of what, under which plan?). Large chunks carry full context but embed mushily and waste prompt tokens. Every chunking strategy is a negotiation between those two forces.

Play with it

Drag the sliders below and watch the chunks re-form live. Try chunk size 100 with 0% overlap, then 800 with 30%. Switch documents — notice that technical docs and narrative prose break differently.

Things worth noticing while you play:

• Fixed-size mode cuts mid-sentence. At small sizes it even cuts mid-word. A chunk that ends "Refunds are processed to the orig" embeds badly and reads worse.
• Overlap is insurance for boundaries. The double-shaded regions appear in two chunks. If a key fact straddles a boundary, overlap ensures at least one chunk contains it whole.
• Sentence-aware mode respects meaning. Boundaries land on punctuation, so every chunk is made of complete thoughts. Chunk sizes vary more, but each chunk is coherent.
• More overlap = more chunks. Overlap isn't free: it inflates storage, embedding cost, and the odds of retrieving near-duplicate chunks.

The main strategies

Fixed-size chunking splits every N characters (or tokens), usually with 10-20% overlap. It's trivial to implement and works surprisingly well as a baseline. Its weakness is exactly what you saw: boundaries ignore meaning.

Sentence/recursive chunking splits on natural boundaries — paragraphs first, then sentences, then words — only cutting deeper when a piece is still too big. This is what popular libraries' "recursive character splitter" does, and it's the sane default for prose.

Structure-aware chunking uses the document's own structure: Markdown headings, HTML sections, code functions. A chunk that maps to "## Refund policy" is a beautiful retrieval unit. When your documents have structure, use it.

Semantic chunking embeds each sentence and starts a new chunk when the topic (embedding similarity) shifts. More expensive, sometimes better, often not worth it until you've exhausted the simpler options.

Here's a minimal fixed-size splitter so nothing is magic:

def chunk_text(text: str, size: int = 500, overlap: int = 75) -> list[str]:
    chunks = []
    stride = size - overlap
    for start in range(0, len(text), stride):
        chunks.append(text[start:start + size])
        if start + size >= len(text):
            break
    return chunks

That's genuinely the entire algorithm behind "fixed-size with overlap." The playground above runs the same logic.

Picking numbers

There is no universal best chunk size — but there are sane starting points:

• Prose / docs: 300-800 characters (roughly 75-200 tokens), 10-20% overlap, sentence-aware boundaries.
• Dense reference material (API docs, policies): smaller chunks, since answers are localized.
• Narrative or argumentative text: larger chunks, since meaning spreads across paragraphs.

Then — and this is the part teams skip — evaluate. Collect 20 real questions, check whether the correct passage is retrieved at each setting, and tune. Chunking is an empirical knob, not a matter of taste.

Build it yourself

1. Implement chunk_text above in your language of choice.
2. Extend it to be sentence-aware: split the text on ., !, ? first, then greedily pack sentences into chunks up to the size limit.
3. Run both on a real document you know well and print the chunks. Find one example where fixed-size splits a key fact across a boundary — then confirm that 15% overlap rescues it.

Summary

• Chunks are the unit of retrieval: they must be small enough to embed sharply and large enough to answer from.
• Fixed-size with overlap is the trivial baseline; sentence-aware/recursive splitting is the sane default; structure-aware wins when documents have headings.
• Overlap protects facts at boundaries, at the cost of extra storage and duplication.
• Start around 300-800 characters with 10-20% overlap — then measure with real questions instead of guessing.
• Store metadata (source, section, position) with every chunk for citations and debugging.