RAG Retrieval Debugging: Why Results Come Back Wrong

“The RAG system gives wrong answers” is not a diagnosis. It describes at least six unrelated faults with unrelated fixes, and the usual response, tweaking the prompt and shrinking the chunk size, addresses none of them. This is a stage-by-stage checklist for locating the failure before changing anything.

The governing rule: never debug a RAG pipeline by reading generated answers. The generator improvises over failed retrieval and ignores good retrieval, so end-to-end output is the noisiest possible signal. Instrument the retrieval stage and look at what came back.

Step 0: make the failure reproducible

Collect 20 to 50 real failing questions and, for each, the chunk that should have answered it. Support tickets and search logs are the right source; questions written by the team drift toward the documentation’s own vocabulary and hide the vocabulary-mismatch failures that dominate real traffic.

For every failing question, log the top 20 retrieved chunks with scores. That single log answers the first and most important question.

Step 1: is the answer chunk in the corpus at all?

Search the raw source documents for the answer text directly, using grep or a plain database query, not the retriever.

If the text is not there, retrieval is not the problem. The usual causes are an ingestion filter silently skipping file types, a crawl depth limit, a permissions boundary in the source system, or a document that failed extraction and indexed as an empty string. Count indexed chunks per source document and look at the zero and near-zero rows; a 400-page manual that produced three chunks did not extract.

If the text is there, continue.

Step 2: is the answer chunk in the index, intact?

Fetch the chunk that contains the answer by document ID and read it as stored.

Three failures show up here:

• The answer is split across two chunks. A boundary landed mid-explanation, so neither chunk is a complete answer and neither embeds near the question. This is what overlap exists to prevent.
• The chunk is context-free. Text like “This limit does not apply to enterprise tenants” is a fine sentence and a useless retrieval unit, because the vector carries no signal about which limit. The fix is prepending the heading path into the embedded text, not merely storing it as metadata.
• The chunk is a wall of unrelated topics. A 2,000-token chunk covering installation, licensing and troubleshooting embeds as an average of all three and lands near none of them.

All three are chunking faults. The sizes and splitting strategy that avoid them are in the chunking strategy guide.

Step 3: does the chunk match its own question?

Embed the question, embed the known-good chunk, and compare their similarity directly, outside the index.

If similarity is high but the chunk did not rank, the index or the filter is at fault. Skip to steps 5 and 6.

If similarity is low, the embedding stage is at fault. Check, in order:

1. Metric mismatch. Cosine and dot product are equivalent only for unit-normalised vectors. If the collection was created with the wrong metric for the model, or normalisation is applied inconsistently between indexing and querying, every score is subtly wrong. This is the single most common silent misconfiguration.
2. Query and documents embedded differently. Several embedding models expect an asymmetric setup, with a distinct instruction or prefix for queries versus passages. Using the same call for both quietly degrades every result.
3. Silent truncation. Embedding APIs cap input length. Some client libraries and ingestion frameworks truncate to fit rather than erroring, so indexing appears to succeed while the tail of every long chunk was never embedded. Find out whether anything in your pipeline truncates for you.
4. Domain mismatch. Public leaderboard rankings come from general retrieval sets. Heavy jargon, non-English content, and source code reshuffle them, which is the point BEIR made about zero-shot generalisation across heterogeneous domains. If the corpus is specialised, evaluate two or three candidate models on your own questions.

Model changes are expensive: vectors from different models are not comparable, so switching means re-embedding and re-indexing everything. Rule out 1 through 3 first, since they are configuration bugs with free fixes.

Step 4: is the query the problem?

Some failures are on the query side and no amount of index tuning touches them.

• Vocabulary mismatch. Users write “it keeps logging me out” and the documentation says “session expiry”. Dense retrieval handles some of this and not all of it.
• Exact-token queries. Error codes, part numbers, function names and SKUs are where dense retrieval is weakest and where users expect a precise hit. This is the case for hybrid search: a lexical or sparse index alongside the dense one, with the two result lists fused. Reciprocal rank fusion is the usual merge because it needs no score normalisation between systems, summing 1 / (k + rank) across lists with k = 60 conventionally.
• Multi-hop questions. “Does the enterprise plan include the feature added in v3?” needs two chunks that no single vector is close to. Query decomposition, not chunk tuning, is the fix.
• Conversational fragments. “What about the other one?” carries no retrievable content. Rewrite follow-up turns into standalone queries using the conversation history before embedding.

A quick discriminator: if a failing question works when you paste in a sentence from the target document, the retrieval stack is sound and the problem is query-side.

Step 5: is a filter eating the results?

Filtered queries fail in a way that looks exactly like poor retrieval.

Run the failing question with all filters removed. If the right chunk appears, the filter is the fault. Common causes are a metadata field missing on older documents so a filter on it excludes them, a type mismatch such as a version stored as a string and filtered as a number, and a default filter injected by the application layer that nobody remembers adding.

Then check where the filter runs. Post-filtering, which searches first and discards non-matching results afterwards, means a selective filter can return almost nothing while the index is perfectly healthy, because rejected chunks consumed the top-k slots. The filter has to be applied during search, not over its output. Stores differ substantially here, which is one of the main axes in Qdrant vs Milvus vs Pinecone.

Step 6: is the approximate index losing recall?

Approximate search trades recall for speed, and the trade is adjustable.

Re-run the failing queries against an exact brute-force scan. If the right chunk ranks well under exact search and poorly under the approximate index, the index parameters are wrong, not the embeddings. Raise the query-time ef first: it buys recall at the cost of latency without rebuilding anything. If that fixes it, the build parameters need attention too.

Aggressive quantisation shows up the same way. Binary quantisation in particular needs a rescoring pass over full-precision vectors; without one, recall drops in a way that looks like an embedding problem.

Index memory is worth checking here as well, since a store swapping or evicting under memory pressure produces erratic results that no parameter change fixes. The chunk and index sizing calculator sizes a collection the way Qdrant’s capacity planning guide does, at 1.5 times raw vector bytes, so the estimate includes index structures rather than vectors alone.

Step 7: it retrieves correctly and the answer is still wrong

Once the right chunk is reliably in the top 5, the remaining faults are downstream.

• Ranked too low. The right chunk is at position 18 and only the top 5 are passed on. This is what a cross-encoder reranker fixes: retrieve 50 to 100, rerank, pass 5 to 10. Reranking cannot fix a candidate set that never contained the answer, which is why it belongs here and not at step 1.
• Buried in the context window. Models attend unevenly across long prompts, and material in the middle is used less reliably than material at either end, the effect documented as “lost in the middle”. With a small reranked set, place the strongest evidence first and last rather than in rank order.
• Contradicted by a stale chunk. Two versions of the same document both rank, and the model averages them. Deduplicate at index time and filter by version at query time.
• Ignored. The context is right and the model answers from its own parameters anyway. Instruct it to answer only from the provided context and to say when the context does not contain the answer. Generation-side evaluation frameworks such as Ragas score exactly this, through faithfulness and context-utilisation metrics.

Order matters more than any single fix

The stages are listed in this order because each one’s diagnosis is only valid if the stages above it are sound. Tuning a reranker over a candidate set that never contained the answer, or swapping the embedding model when the real fault is a metric mismatch, wastes the effort and leaves the bug in place.

Two habits prevent most of this. Keep a golden set of 50 to 200 labelled questions and track recall@k on it after every change, one variable at a time. And build the pipeline in the order that makes each stage measurable, as set out in the RAG pipeline build order.

See also

• RAG pipeline architecture: components and build order
• RAG chunking strategy: picking chunk size and overlap
• Qdrant vs Milvus vs Pinecone: vector database comparison
• Chunk and index sizing calculator
• LLM observability and tracing
• Production LLM telemetry and ops