Epistemic Noise
All notes

Note · 2 min

RAG failure modes

RAG failures become easier to debug once “RAG quality” stops being treated as one thing.

First ask whether the information exists in the corpus at all. A retriever cannot recover a document that was never ingested, was parsed incorrectly, was removed by a metadata filter, or is simply stale.

Then ask whether retrieval found the right evidence. Chunking may have separated a fact from the context that gives it meaning. Dense retrieval can prefer a semantically similar distractor. Sparse search can miss a paraphrase. Query rewriting can rescue an ambiguous query or damage one that was already precise. A reranker helps only if the useful evidence entered the candidate set in the first place.

Then inspect what actually reached the model. Good retrieval can still become bad context through duplication, ordering, oversized chunks or several passages that disagree.

Only after those stages pass do I start blaming generation.

If the correct evidence was available and the model ignored it, then grounding, prompting, citation behavior, context handling or the model itself becomes interesting.

This ordering sounds obvious, but it changes debugging completely. Without it, teams can spend days changing prompts and models for an answer that never entered the context window.