Why RAG Systems Fail: Build, Debug, and Evaluate Retrieval in Python

Category: AI & Machine Learning · Reading level: intermediate · Prerequisites: Python, basic vector-space similarity · Experiment: offline, repeatable, no paid API key

A retrieval-augmented generation (RAG) demo often looks convincing: upload a PDF, ask a question, and receive a fluent answer. But how do you know the answer came from the correct passage? And what happens when the query is phrased differently from the source document? The harder problem is not getting a model to produce text. It is building a pipeline whose failures you can identify and measure.

In this tutorial we construct a small, auditable retrieval benchmark based on company policy documents. We calculate evidence recall and reciprocal rank, separate retrieval errors from generation errors, examine chunking and abstention, and design a realistic evaluation plan. The accompanying evaluate_rag.py actually runs offline using scikit-learn. It measures retrieval only: it does not pretend that a keyword retriever is a complete production LLM-powered RAG system.

1. The problem: a confident answer can cite the wrong policy

Imagine an internal knowledge assistant serving questions about invoices, refunds, backups, and account deletion. A user asks: “How long must customer invoices be retained?” The answer exists in a retention policy. However, a search system may rank a generic financial report higher because it repeats “invoice” more frequently. A language model can then confidently hallucinate a retention period.

A useful failure taxonomy is retrieval miss (correct passage never reached the prompt), ranking miss (correct passage exists but is buried), context dilution (irrelevant passages overwhelm the answer), generation miss (relevant evidence exists but the response misinterprets it), and unsupported certainty (no answer exists and the system invents one). These are different engineering problems, so “accuracy” alone is a poor debugging metric.

2. Retrieval as a measurable ranking problem

Suppose document chunks are indexed as vectors $d_1,\dots,d_N$, and a question is transformed into a vector $q$. A typical similarity score is cosine similarity:

$$\operatorname{sim}(q,d_i)=\frac{q^\top d_i}{\|q\|_2\|d_i\|_2}.$$

Vector similarity does not measure truth. It measures closeness under an embedding or text representation. Our exercise uses TF–IDF with unigrams and bigrams, deliberately avoiding network dependencies. Dense embeddings may better handle paraphrases, but they also need evaluation; an expensive model does not make bad chunk boundaries disappear.

For a question $j$, let $G_j$ be the set of annotated relevant evidence IDs, and $R_j(k)$ the top-$k$ retrieved set. Then:

$$\operatorname{Recall@k}_j = \frac{|G_j\cap R_j(k)|}{|G_j|}.$$

With a single gold document, this is 1 if it appears in the top $k$, otherwise 0. Mean reciprocal rank (MRR) rewards earlier placement: $\operatorname{RR}=1/r$, where $r$ is the first relevant rank. The corpus must have human-checked evidence labels; generating both the test questions and the ranking judgments with the same LLM creates correlated measurement errors.

3. A tiny test collection that actually runs

The download contains eight short policy passages and seven annotated questions. The dataset is intentionally small enough that readers can inspect each mistake. Run:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python evaluate_rag.py --k 3 --out retrieval_results.json

The core ranking operation is:

vec = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True,
                      stop_words="english")
doc_vectors = vec.fit_transform([doc.text for doc in docs])
question_vector = vec.transform([question])
scores = cosine_similarity(question_vector, doc_vectors)[0]
ranked = np.argsort(-scores, kind="stable")[:3]

The script prints recall@3, MRR@3 and the evidence IDs for each example. With this authored miniature collection, scores may be perfect: that proves the harness executes on these examples, not that it generalizes. Add at least 50–100 independently authored questions before using the scores to compare production designs.

4. Break the system deliberately

Failure A — chunk size. A chunk that stops before “except where legal retention applies” can turn a qualified policy into an incorrect categorical answer. A chunk so large that it includes three policies can reduce ranking precision. Evaluate chunk sizes using the same document version and fixed question set; record chunk overlap and token counts.

Failure B — paraphrase. “What is the invoice preservation window?” may share fewer surface words with “invoices are retained for seven years.” Our TF–IDF baseline may miss it. Compare a dense embedding model only after you have the exact paraphrased evaluation set, and use lexical retrieval as a credible baseline.

Failure C — contradictory or superseded documents. A 2023 retention policy and a 2026 amendment should not be treated equally. Add document version, effective date, ownership, and access scope as metadata; then test retrieval filters separately from similarity ranking.

Failure D — permission leakage. Ranking alone is not enough. A production assistant must filter documents by authorization before exposing their content to a generator. A correct answer obtained from an unauthorized source is still a security failure.

Failure E — no answer exists. Add deliberately unanswerable questions. For example, if none of the policy passages identifies the CEO, the model should abstain, not fabricate a name. Retrieval scores must be accompanied by answerability and abstention evaluations.

5. Separate retrieval from generation evaluation

Inspect a pipeline as two stages:

A high recall@5 does not imply a grounded answer, because the generator may cite the wrong passage or combine two policies incorrectly. Conversely, an apparently correct answer might come from the model’s memorized knowledge rather than the retrieved evidence. To test grounding, deliberately change a controlled fact in the supplied passages and check whether the answer follows that change.

6. An evaluation protocol for a real product

Start with a frozen set of documents (hash each original file). Have domain reviewers write representative questions, including paraphrases, multi-hop requests, ambiguous language, and negatives. For each question, record an answerability flag, relevant passage IDs, expected factual answer, and permission context. Make a development set for tuning and a separate held-out set for final comparison.

Run a matrix of configurations: lexical vs dense retrieval, chunk sizes, overlap, optional reranker, and top-k. Keep generation settings constant when comparing retrieval. Save individual failures, not only averages. A configuration that improves overall Recall@5 but fails all security-related questions is not acceptable. Include latency, embedding cost and context size alongside quality.

For generation, store the exact prompt, model identifier, temperature, retrieved text, answer and citations. A rubric-based LLM judge can help triage cases, but calibrate it against blinded human labels; judges have their own position bias, verbosity bias and knowledge gaps.

7. Common implementation mistakes

Avoid fitting retrieval vocabulary on hidden evaluation questions; this contaminates the test. Avoid treating LLM citation markers as proof of support. Avoid averaging Recall@k across examples with no gold evidence without a documented convention. Avoid reporting “faithfulness 95%” without defining how claims and sources were annotated. And avoid shipping the public evaluation questions in the assistant’s system prompt: that turns a held-out benchmark into training data.

8. What should readers build next?

Replace the TF–IDF baseline with a dense embedding index, then add an optional reranker. Preserve the evaluation harness. Introduce an actual answer generator only after retrieval quality is measurable. Next, build the claim-level evaluation described in How to Evaluate LLM Applications, and compare RAG with other solutions in RAG vs Fine-Tuning vs Prompting.

Takeaway: a successful RAG engineering workflow treats retrieval quality, answer correctness, grounding, permissions, and costs as separate testable properties. A fluent demo is not a reliability guarantee. If you want to assess your fundamentals in retrieval and model evaluation, explore the Algonique AI Skill Assessment.

Frequently asked questions

What is a good Recall@5 for RAG? There is no universal cutoff; the answer depends on source completeness, question difficulty, corpus size, and the cost of a missed document. Establish a human-annotated baseline and a task-specific acceptance threshold.

Are embeddings better than TF–IDF? Sometimes, especially with paraphrases, but lexical matching can excel at identifiers and exact terminology. Measure both, and consider hybrid retrieval.

Does RAG eliminate hallucinations? No. Good evidence retrieval reduces one failure mode but does not guarantee that the generator interprets or cites evidence correctly.

Should I use an LLM to grade itself? An LLM judge can help scale evaluation, but it needs calibration against human reference judgments and should never be your sole quality signal.

Sources and further reading

• scikit-learn TF–IDF documentation
• Information retrieval evaluation overview, Stanford IR book
• NIST TREC evaluation resources