How to Evaluate LLM Applications in Python: Build a Reproducible Testing Harness

Category: AI & Machine Learning · Level: intermediate/advanced · Experiment: offline test harness with explicit gold examples · Tools: Python standard library

An LLM application can appear reliable after ten impressive demos and still fail during deployment. The failures often happen at the edges: the model answers a question not covered by the documents, invents a citation, confuses two entities or gives the right answer for the wrong reason. Before tuning prompts or changing models, you need a repeatable evaluation contract.

This guide builds a small evaluation harness that reads saved application outputs, matches them to frozen gold questions, computes transparent metrics and records per-example failures. The downloadable llm_eval.py runs offline without a model API: the included example predictions demonstrate the evaluation mechanics, while a real application can save its own outputs in the same JSON format. We deliberately avoid presenting hand-authored fixture scores as evidence of an LLM’s performance.

1. Define exactly what success means

Suppose we build a policy Q&A assistant. The application retrieves document fragments and produces a natural-language answer. It must do at least four things correctly: retrieve the relevant passage, answer the question accurately, cite the evidence, and abstain when the documents contain no answer.

These are not interchangeable. An answer can be lexically similar to a reference while citing nonexistent evidence; it can retrieve the right passage but misread a number; it can provide a sensible general-world answer that violates the organization’s document-only constraints.

A useful evaluation starts by recording input question, evidence IDs, answerability, expected answer, predicted answer, citations, retrieved IDs and model/prompt version for each example. This row-level dataset is your debugging instrument.

2. Gold questions are test cases, not prompt examples

A gold record should include a stable ID, question, verified expected answer, permitted evidence IDs and an abstention expectation. A human reviewer familiar with the policy should confirm correctness. For this educational case, the sample records are authored examples such as:

{
  "id": "q1",
  "question": "How long are invoices retained?",
  "answer": "seven years",
  "evidence_ids": ["retention"],
  "should_abstain": false
}

Keep development and held-out sets separate. It is fine to tune a prompt on development cases. It is not fine to inspect every failing held-out example, change the prompt repeatedly and still call that same set an unbiased final test. When production data shifts, refresh your benchmark with a new version, while preserving historical snapshots for regression comparison.

3. Retrieval quality: Recall@k

Let $G_i$ be the relevant evidence IDs for question $i$ , and $R_i(k)$ the retrieved IDs within the first $k$ results. Then:

$\operatorname{Recall@k}_i=\frac{|G_i\cap R_i(k)|}{|G_i|}.$

We aggregate over answerable questions with at least one annotated evidence ID. An unanswerable question has an empty gold set, so it should be evaluated with separate abstention metrics instead of arbitrarily assigning Recall@k=1. Our script implements precisely that convention.

Recall@k measures whether supporting context was available. It does not mean the model used it. A system can return all the correct passages and still produce a wrong answer.

4. Answer similarity is not faithfulness

A convenient automatic metric is token-level F1, based on overlap between prediction and reference. With token precision $P$ and token recall $R$ :

$F_1=\frac{2PR}{P+R}.$

This is useful for recognizing mechanical regressions, but not for judging semantic or factual accuracy. “Seven years” and “7 years” may score imperfectly without being different claims. “No refund is allowed” and “refund is allowed” share many tokens while contradicting each other. Therefore the harness labels its output lexical_answer_f1 and never calls it factual accuracy or faithfulness.

For production use, add structured field checks for dates, prices and IDs, and sampled human adjudication of factual claims. If you use an LLM judge, blind it to system variants where possible, shuffle comparison order and measure agreement against human labels. Do not use one generated answer as both the test’s target and its independent evaluator.

5. Citation support: what exactly did we measure?

Our baseline metric checks whether citation IDs were among retrieved IDs. This is a useful structural sanity check: citing a passage that wasn’t in the context is suspect. But it is not a claim-level grounding metric. A model can cite the correct file and still state a number that appears nowhere in that file.

For actual faithfulness, decompose the answer into atomic claims. Have a reviewer label each claim as supported, contradicted or not enough evidence using the cited passage text. Report a supported-claim fraction only over an explicitly annotated sample, and preserve counterexamples. Separately track whether citations point to unauthorized or outdated sources.

6. Abstention and negative questions

If no provided document identifies the company’s CEO, answering with a plausible name is a failure. The desired output is a clearly worded abstention. Include negative questions in the test set and compute the fraction that are correctly refused. Also test the opposite error: overly conservative systems may refuse perfectly answerable questions.

A single “abstention accuracy” score can hide which direction the errors run. Maintain a confusion matrix for answerable/unanswerable cases; report false-answer rate on unanswerable items and false-refusal rate on answerable items. The demo harness prints a simple correctness proportion as a starting point.

7. Run the offline evaluation harness

Save the predictions as a JSON array of records with id, answer, retrieved, citations, and abstained. Run:

# No pip dependencies required
python llm_eval.py --out evaluation.json
# Use your actual saved model outputs:
python llm_eval.py --predictions my_predictions.json --out result.json

The script refuses mismatched or duplicate IDs, because silently dropping failed questions inflates results. It writes aggregate metrics and every individual scored row, so you can compare versions and inspect regression failures.

predictions = {row["id"]: row for row in outputs}
if set(predictions) != {row["id"] for row in gold}:
    raise ValueError("Outputs and gold IDs must match exactly")
# Evaluate retrieval, structural citations, abstention, lexical answer F1.

On the handcrafted three-question fixture, retrieval recall, structural citation validity and abstention correctness are all 1.0; lexical answer F1 is lower because the output contains explanatory words. These figures simply validate example wiring. They are not a performance claim about any particular model or RAG implementation.

8. Turn the harness into a trustworthy regression suite

Start with at least four strata: straightforward factual questions, ambiguous questions, multi-document questions and negatives. Freeze document versions and review answers. Then add production-like noise: different languages, misspellings, misleading adjacent passages, long-context truncation, missing permissions and outdated information.

For each prompt or model release, evaluate on the same frozen benchmark. Retain the exact model ID, prompts, retrieval settings, dataset hashes, timestamp, latency, token usage, and cost. Use bootstrapping at the question level to estimate uncertainty when sample sizes permit. If a change improves F1 by 0.01 on 20 questions, you have little reason to infer meaningful improvement without looking at individual errors and confidence intervals.

Add operational thresholds: unacceptable factual errors, maximum false-answer rate, median and tail latency, and price per successful task. A slower system with better correctness may be preferable for financial or compliance content, while a low-risk FAQ assistant might optimize throughput. State those trade-offs before comparing models.

9. What not to conclude from this tutorial

The harness does not call a language model. Its current evidence and answer labels are authored by the tutorial writer, not a representative production sample. Token F1 is not a truth detector. Citation ID membership is not faithful attribution. The test suite should grow with real domain-expert judgments, adversarial cases and traces from permitted user interactions before it is used in business decisions.

Takeaway: make quality observable before optimizing it. Extend the pipeline by measuring RAG Retrieval Quality, then review evaluation-related knowledge through the Algonique AI Skill Assessment.

Frequently asked questions

Is LLM-as-a-judge enough? No. It can help triage and scale assessment, but must be checked against human labels and monitored for bias.

What is the difference between correctness and faithfulness? Correctness asks whether the answer is factually right; faithfulness asks whether its claims are supported by the supplied evidence.

How many evaluation examples do I need? There is no universal number. Start with a representative stratified suite, measure uncertainty, and expand where failures are concentrated.

Does exact match work for LLM evaluation? It is useful for structured outputs and strict IDs, but insufficient for natural-language equivalence.