Chunking Strategies for RAG Systems That Actually Get Used

Chunking is the boring foundation nobody wants to talk about. It also decides more about your RAG quality than embedding-model choice, reranker choice, or prompt tuning combined. Get it wrong once during ingestion and every retrieval afterwards inherits the mistake.

This is a working guide to chunking strategies for RAG systems — how teams that ship real RAG choose chunk size, overlap, and boundaries for the corpora they actually have.

Why chunking matters

Embeddings compress a passage of text into a fixed-size vector. Compress too much text and the vector represents nothing in particular. Compress too little and the vector is missing the context that makes the passage findable.

The chunk boundary also decides what the model sees at generation time. If the answer straddles two chunks and only one is retrieved, the model has half the story. That's a hallucination waiting to happen.

The default that's usually wrong

Almost every tutorial ships with 512-token chunks and 50-token overlap. It's a fine starting point for prose-heavy documents. It's a terrible starting point for code, tables, transcripts, or anything structured.

The right chunk size depends on what your documents look like. A knowledge base of short FAQ entries wants 100–200 token chunks. A repository of long PDFs wants 500–1000. Legal contracts want much more overlap than product docs. There is no single default.

Where teams get chunking wrong

Splitting on character count

Character-based chunkers cut mid-sentence, mid-word, and mid-table. The resulting chunks embed poorly because they're missing the linguistic structure the embedding model was trained on. Use token boundaries, and prefer natural breakpoints — paragraphs, headings, sections.

Ignoring structure

A markdown document has structure. HTML has structure. Even PDFs, if parsed well, have structure. Throwing that structure away and chunking as plain text loses the strongest retrieval signal you have.

One common pattern we see: include the section heading with every chunk from that section. Suddenly retrieval improves noticeably, especially on queries that mention the section name.

No overlap

Zero-overlap chunking makes the boundary between chunks a hard line. Answers that cross the boundary are lost. Some overlap — usually 10–20% of the chunk — restores continuity without inflating the corpus much.

Too much overlap

The opposite mistake. 50% overlap doubles the corpus size and doesn't help retrieval much beyond a point. Every retrieval returns near-duplicate chunks and the prompt fills up with redundant content.

Chunking strategies that hold up

Fixed-size with structural respect

Target a chunk size (say 500 tokens), but split at natural boundaries — paragraph, section, sentence — that get closest to the target. Add 10–15% overlap. This is the workhorse strategy for prose.

Header-aware chunking

For markdown, HTML, or well-structured PDFs: prepend the section header (and optionally parent headers) to each chunk. This gives the embedding contextual anchoring that pure content lacks.

Semantic chunking

Split at semantic boundaries instead of size boundaries. Compute embeddings for adjacent sentences, split where the distance jumps. Good on continuous prose (essays, reports). Not worth the complexity for structured content.

The overhead is real. Semantic chunking is 5–10× slower than fixed-size and needs careful tuning. Use it when fixed-size hits a ceiling, not before.

Element-based chunking

For tables, code blocks, and lists — chunk the element as one unit. Splitting a table mid-row destroys it. A 30-row table might be one big chunk with a summary embedded, or a small chunk per row with the header row prepended.

Sliding window on transcripts

For call transcripts, meeting notes, or podcast content: sliding-window chunks of 400 tokens with 100-token overlap. Timestamps make it possible to link back to the source moment. Speaker labels stay with the chunk.

Metadata is half the game

Every chunk should carry metadata: source document, section, page number, tenant, timestamp, whatever your queries filter on. Metadata isn't part of the embedding — it's the filter that narrows retrieval before the vector search runs.

Without good metadata, you're relying entirely on vector similarity, and vector similarity is imperfect. With good metadata, retrieval can hard-filter to the right tenant, product, or date range before doing similarity search.

Chunking for tables and code

Tables lose meaning when split. Code loses meaning when split. Both deserve special handling.

For tables: extract the table as a whole, and either embed the full table (with a size cap) or embed a natural-language summary alongside the raw table. Some teams generate a "what this table shows" caption with an LLM and embed that.

For code: chunk by function, method, or class where the parser can identify boundaries. Chunk by file when it can't. Never chunk by line count for code — the results are useless.

Testing chunking

Build a small eval set. Twenty to fifty query/answer pairs where you know which document contains the answer. Try three chunking strategies. Measure recall at top 3 for each.

You'll usually find one strategy clearly wins. The one that wins depends on your corpus, which is why "just use 512 tokens" is not the answer. Our RAG debugging post covers how to run this measurement.

When to re-chunk

Re-chunking a corpus is expensive but not catastrophic. Do it when:

Retrieval quality is stuck and you've ruled out embedding and reranking. Documents have gotten structurally different (you added a new document type). You've moved to an embedding model with a different context window or better structural sensitivity.

Version your chunker settings. Log which version produced each chunk. This makes rollbacks and A/B tests possible instead of scary.

Mistakes teams make

Treating chunking as a one-time decision. Chunking is a parameter to tune, not a config to set once and forget.

Ignoring the source format. The same chunker for PDFs, HTML pages, and Slack exports will underperform on all three.

Forgetting metadata. Chunks without metadata are chunks that can't be filtered, updated cleanly, or removed reliably.

Optimizing chunking before fixing ingestion. If your PDFs are ingesting as blobs of unreadable text, no chunker will save you.

Trade-offs

Smaller chunks retrieve more precisely, ground the model less. Bigger chunks ground better, retrieve messier. More overlap catches boundary-straddling answers, inflates the corpus. Semantic chunking is better for prose, worse for latency and complexity.

Pick trade-offs deliberately and measure them. Your corpus will disagree with generic advice more often than not.

Common misconceptions

"There's a best chunk size." There isn't. There's one that works best for your data.

"Bigger context windows make chunking irrelevant." They don't. Retrieval quality still depends on chunk boundaries even when the model could fit everything.

"Semantic chunking always beats fixed-size." It often doesn't. On structured content, header-aware fixed-size wins.

Frequently asked questions

What's a good starting chunk size?

400–600 tokens with 10–15% overlap for general prose. Adjust based on your corpus and your eval set, not tutorials.

Should I use semantic chunking?

Only if fixed-size with structural respect isn't enough. Semantic chunking is slower and finicky.

How much overlap should I use?

10–20%. Zero misses boundary-straddling answers. 50% inflates the corpus for little gain.

How do I chunk PDFs?

Extract text with a structure-aware parser, respect headings and sections, and treat tables as whole units. If the parser is bad, no chunker fixes it.

Do I need to re-chunk when I upgrade my embedding model?

Not always. Try the new embeddings on your existing chunks first. Re-chunk only if quality doesn't move.

Conclusion

Chunking is the layer between your documents and everything downstream in RAG. Fixed-size with structural respect is the workhorse. Metadata is half the win. Special cases — tables, code, transcripts — deserve special handling.

Teams that build reliable RAG treat chunking as an ongoing decision instrumented with an eval set. Not as a one-line config value copied from a tutorial.

If retrieval quality is holding your product back and you're not sure whether chunking is the issue, the AI Audit is a structured way to find out.