Debugging RAG Retrieval Quality in Production

When RAG stops working, teams usually blame the model. Almost every time we've been called in to fix a RAG pipeline, the problem was upstream — chunking, embeddings, or the retrieval query. The model is doing its best with what it was handed.

This post is a working checklist for debugging RAG retrieval quality in production. Not theory. What to look at, in what order, when your RAG feature starts feeling worse than it did last month.

The mental model

RAG has four stages: ingestion, chunking, retrieval, and generation. A failure at any stage looks like a bad answer, so debugging means isolating which stage broke.

The order matters. Debugging generation before you've verified retrieval is like debugging a rendering bug before you've checked the API response. You'll spin your wheels.

Start at retrieval, not generation

The first thing to check when a RAG answer is wrong: what chunks came back? Log the retrieved chunks alongside the response. Every request. You'll find that most "bad answers" are actually cases where the right chunk wasn't retrieved.

If the right chunk was retrieved and the answer is still wrong, that's a generation problem — and now you know. If the right chunk wasn't retrieved, retrieval is the fix, and touching the prompt won't help.

The one query test

Pick a query where you know the correct answer and where you know which chunk should be retrieved. Run it. Check that the chunk is in the top three results.

If it isn't, you have a retrieval problem. Now the question is: which part of retrieval?

Retrieval failure modes

The chunk doesn't exist

The ingestion step never processed the source document. Or it processed it and the content was empty because the parser choked on the format. Check your document store directly. If the text isn't there in a searchable form, nothing downstream will find it.

This is embarrassingly common with PDFs. Scanned PDFs have no text layer. HTML with heavy JavaScript rendering often ingests as empty. Fix ingestion, not retrieval.

The chunk is too small or too big

If your chunks are 100 tokens, an answer that spans two paragraphs gets split across chunks and neither one alone is enough. If your chunks are 2,000 tokens, embeddings become mush because the vector is trying to represent too many concepts at once.

Most working RAG systems land at 300–800 tokens per chunk with some overlap. Our post on chunking strategies goes deeper. If you have big documents and small chunks, or vice versa, this is often the fix.

Query and content are semantically mismatched

The user asked "how do I reset my password?" and the doc says "credential recovery procedure." Embeddings should bridge that, but sometimes they don't — especially with domain-specific jargon your embedding model wasn't trained on.

Two fixes. Rewrite the query first (a small LLM call that expands it with synonyms and related terms). Or, ingest the docs with a hypothetical-question layer — for each chunk, generate the questions a user might ask, and embed those alongside the content.

Metadata filters are wrong

If you're filtering by tenant, workspace, product, or date, a wrong filter will silently drop the right chunk. Log the filters used at retrieval time. Every request.

A pattern we see often: the tenant ID is off by one because of a naming mismatch between the ingestion pipeline and the retrieval query. The system looks broken, but the fix is renaming one field.

Look at the vectors

If retrieval is failing and the chunk exists and the filters are right, look at the actual embedding distance. Compute the cosine similarity between the query embedding and the target chunk embedding. If it's low — say, under 0.3 — the embedding model is the problem.

Try a different embedding model. This is usually cheaper than it sounds because you can re-embed a corpus in under an hour for most datasets. Our post on choosing an embedding model covers what to try.

Reranking as a debug tool

If you're not reranking, add it. Retrieve the top 20 candidates by vector similarity, then use a cross-encoder or a small LLM to rerank them by relevance to the query. This usually catches cases where the right chunk is in the top 20 but not the top 3.

Reranking also helps you diagnose. If reranking pulls the right chunk from position 15 to position 1, your embedding is under-performing but salvageable. If reranking can't find the right chunk in the top 20, you have a deeper retrieval problem.

Build a small evaluation set

You can't fix retrieval quality if you can't measure it. Take 30–50 queries where you know the correct answer and the correct source chunk. Run retrieval on them. Measure how often the correct chunk is in the top 1, top 3, and top 10.

Those three numbers become your dashboard. Every change to the pipeline — new chunker, new embeddings, new reranker — gets measured against them. Without this, you're changing things and hoping.

The generation side

If retrieval looks right and the answer is still wrong, the prompt is the next place to look. Common issues:

The model is ignoring the context. If your prompt is 500 tokens of instructions and 300 tokens of context, the model reads the instructions harder than the context. Move context above instructions and be explicit: "Answer using only the context below."

The model is answering from world knowledge. If the user asks something the context doesn't cover, the model fills in from training. Add the "if the context doesn't contain the answer, say so" instruction and enforce it in verification.

The model is combining chunks incorrectly. Two chunks about different customers get merged into a Frankenstein answer. Include chunk-level source metadata in the prompt and ask the model to attribute claims to specific chunks.

Mistakes teams make

Changing the model first. Switching from one LLM to another rarely fixes RAG. The problem is upstream.

Not logging chunks. Without the retrieved chunks in the logs, debugging RAG is guessing.

Chasing embedding models. Embedding models matter, but only after you've verified chunking, ingestion, and filters. Fix the boring stuff first.

No eval set. Every RAG improvement should be measured. "It feels better" is not a metric.

Trade-offs

Better retrieval usually costs latency (reranking adds a step) or cost (more candidates means larger inference calls). Bigger chunks retrieve worse but ground answers better. Smaller chunks retrieve better but leave the model less to work with.

The right point on the curve depends on your product. A support bot that must be right is different from a discovery tool that just needs to be useful.

Common misconceptions

"RAG is a solved problem." It isn't. RAG is a stack of things that all have to work.

"Bigger context windows kill RAG." They don't. Context windows help but stuffing 200 chunks into a prompt still degrades output quality. Retrieval quality matters regardless.

"Vector search alone is enough." Rarely. Hybrid retrieval — combining vector search with keyword search — outperforms pure vector on most real corpora, especially with proper nouns and technical terms.

Frequently asked questions

What's the first thing to check when RAG is wrong?

Log the retrieved chunks. Verify the right chunk came back. Half the time it didn't, and that's your bug.

Should I use hybrid retrieval?

Almost always yes. Vector plus keyword outperforms pure vector on real data, especially with jargon and identifiers.

How many chunks should I retrieve?

Retrieve 20, rerank, keep the top 3–5 for the prompt. Sending the model too many chunks degrades output.

When should I re-embed my corpus?

When you change chunker settings, upgrade embedding models, or discover that retrieval quality on new documents lags older ones.

Is fine-tuning worth it for RAG?

Rarely. Fix retrieval and prompting first. Fine-tuning is a last resort and doesn't fix hallucinations from bad context.

Conclusion

Debugging RAG retrieval quality is unglamorous work. Log the chunks. Build the eval set. Check ingestion. Verify chunking. Try reranking. Look at the vectors. Change one thing at a time.

The teams that ship reliable RAG treat retrieval like a data pipeline — instrumented, measured, versioned. Not like a black box that either works or doesn't.

If your RAG feature has quietly gotten worse and you can't tell why, that's exactly the shape of problem the AI Audit is designed to unpack.