Evaluation ·
How to evaluate a RAG system before you trust it in production
Build the test set from questions people actually asked, score retrieval separately from generation, and gate every deploy on the result. Then accept that this bounds your risk rather than removing it.
Evaluate a retrieval system by building a test set from questions people actually asked, scoring retrieval separately from generation, and refusing to ship any change that moves the numbers the wrong way. The metric definitions below are the standard ones — context recall, context precision, faithfulness, answer relevance and the ranking measures — and each one carries the failure it does not catch, which is the part the tool documentation leaves out. Then accept the limit of what you have built: Barnett et al. concluded, from three deployed systems, that validation of a retrieval system is only feasible during operation and that robustness evolves rather than being designed in at the start (arXiv 2401.05856, January 2024). A pre-production harness bounds your risk. It does not remove it, and a vendor who says otherwise is selling you a demo.
The golden set is the whole project, and it is not a spreadsheet from an afternoon
A usable test set is drawn from real questions, has the answering passage recorded alongside the answer, and deliberately includes the cases the team argues about. Most teams have a list of questions someone invented, which measures nothing except that person's imagination.
Source from logs. Support tickets, chat transcripts, analyst queries, the questions people ask in the internal channel where they give up on the search tool. These carry the phrasing your users actually use, which is almost never the phrasing your documentation uses, and that mismatch is itself one of the things you need to measure.
Record for each case the question, the passage that genuinely answers it — document, version and location, not just a summary — the expected answer, and a label for the category it belongs to. The passage is the part teams skip and the part that makes retrieval measurable. Without it you can score the final answer and nothing else, which leaves you unable to tell a ranking problem from a generation problem.
Size: a few hundred adjudicated pairs is enough to detect the changes worth acting on, and it is small enough that subject experts will actually complete it. Adjudication by experts, not by the engineering team. And treat expert disagreement as data rather than noise — where two reviewers who know the domain disagree about the correct answer, you have found a question the system cannot reasonably be expected to get right, and it should be scored as ambiguous rather than counted as a failure.
When we take on a RAG development and LLM integration engagement this set is built before anything is tuned, because tuning against an unmeasured system is guesswork with an invoice attached. The schema below is the one we use. Copy it.
- Questions mined from real logs, in the users' own phrasing
- The answering passage recorded with document, version and location
- Adjudicated by domain experts, with disagreements marked rather than resolved by vote
- Stratified by category so you can report per stratum instead of one average
[
{
"id": "gs-0142",
"question": "Can a customer withdraw consent for marketing messages after opting in?",
"category": "consent",
"expected_behaviour": "answer",
"reference_answer": "Yes. Consent can be withdrawn at any time and the withdrawal takes effect for future messages, not retrospectively.",
"reference_passages": [
{
"doc_id": "policy-privacy",
"doc_version": "2026-03-11",
"locator": "section 7.2",
"chunk_ids": ["policy-privacy@2026-03-11#c41"]
}
],
"corpus_snapshot": "2026-04-01",
"source": "support-ticket-88213",
"adjudication": { "reviewers": ["ap", "sd"], "agreed": true, "note": "" },
"added": "2026-04-02"
}
]Why each field is there
Nothing in that schema is decoration. Every field exists because a harness without it produces a number you cannot act on, and most of them were added after a specific argument about what a failing case meant.
- id — stable across re-adjudication, so a case that regressed can be discussed by name rather than by row number.
- question — the user's phrasing, verbatim from the log. Rewriting it into documentation vocabulary destroys the mismatch you are trying to measure.
- category — the stratum. Scores are reported per category, because one collapsed category averages away against nine healthy ones and it is always the category somebody cares about.
- expected_behaviour — one of answer, refuse, clarify or route. This is what makes unanswerable and out-of-scope cases scoreable instead of noise.
- reference_answer — what a competent human would say. Used by the generation metrics, and for the omission check that faithfulness alone cannot do.
- reference_passages — document, version, human-readable locator and the chunk ids. The chunk ids are what make retrieval measurable at all; the version and locator are what let a human re-check the annotation a year later.
- corpus_snapshot — the corpus this case was adjudicated against. Without it, corpus drift silently converts correct answers into failures.
- source — the ticket, transcript or thread the question came from. Provenance is what stops a set drifting back towards questions somebody invented.
- adjudication — who reviewed it and whether they agreed. Disagreement is recorded rather than resolved by vote, because a case two domain experts read differently is a case the system cannot fairly be scored on.
Include the cases that have no answer
A test set made only of answerable questions measures the happy path and rewards a system that never refuses. Unanswerable, out-of-scope and adversarial cases belong in the set from the beginning, because refusal is a behaviour you are shipping.
Four categories are worth building deliberately. Questions whose answer is genuinely absent, where the correct behaviour is to say so. Questions that are out of scope — a customer asking a policy assistant for investment advice — where the correct behaviour is to decline and route. Questions that are ambiguous without a clarifying turn. And adversarial cases: attempts to extract another customer's data, documents containing instructions, questions phrased to elicit advice you may not give.
Score refusal correctness as its own metric. A system can improve on answer accuracy while quietly losing the ability to refuse, and a blended score hides that completely. In regulated deployments refusal behaviour is often what the compliance function cares about most, and the property least likely to be measured.
Put the injection attempts inside the documents rather than in the user's message, because that is the realistic attack: a supplier sends a PDF, it is ingested, and the instruction inside it is read as though it came from you.
Score retrieval separately from generation
One number for the whole system tells you it got worse and nothing about where. Retrieval and generation are different components with different failure modes and different fixes, and they need separate metrics reported side by side.
On the retrieval side, recall at k is the primary measure: over your golden set, how often the annotated answering passage appears in the top k retrieved. Report it at two depths — a small k matching what you actually pass to the model, and a larger one. The gap between them is the size of the prize available from reranking, and it converts an argument about whether to add a reranker into an estimate. Mean reciprocal rank and nDCG are useful supplements where position within the retrieved set matters.
On the generation side, four measures carry most of the weight. Groundedness: is every claim in the answer supported by a retrieved span. Citation precision: do the cited spans actually support the sentences that cite them. Answer relevance: does the answer address the question asked. Refusal correctness, as above. Keep these apart. An intervention that improves relevance while degrading groundedness has made the system more persuasive and less trustworthy, which is the worst direction available.
Cost and latency belong on the same dashboard. Tokens per query, cost per query, and p50 and p95 latency. Quality and cost trade against each other continuously and the trade is a business decision — it cannot be made by a team that sees only one of the two numbers.
- Recall at k, at two depths, as the primary retrieval measure
- Groundedness, citation precision, answer relevance and refusal correctness, reported separately
- Tokens, cost and p95 latency alongside quality, never on a different page
- No single blended score, ever
Retrieval metrics: context recall, context precision, MRR and nDCG
Retrieval metrics score the shortlist, not the answer. Four are worth reporting, and each is blind to something one of the others catches — which is the reason to report four rather than pick a favourite.
The definitions below are the ones in general use, so a reader who has been through the Ragas or Qdrant documentation will recognise them. What is added here is the last line of each: the failure the metric does not catch. That line is where evaluation programmes go wrong, because a metric that looks healthy is read as a component that works.
Context recall
What it measures: whether the passage that actually answers the question was retrieved at all.
How it is computed: in the classic form, recall at k over the golden set — the annotated answering passage appears in the top k, averaged across cases. In the framework form, Ragas breaks the reference answer into claims and scores the number of claims in the reference supported by the retrieved context, divided by the total number of claims in the reference (Ragas documentation). Both forms need a reference. Recall is the one metric you cannot compute without annotation, which is why the golden set comes first.
What a bad score means for the product: the answer was never in the room. No prompt change, no larger model and no reranker can recover a passage that was not retrieved. The fix is upstream — chunking, embeddings, hybrid search, or the discovery that the document is not in the corpus at all.
What it does not catch: how much rubbish arrived with it. A retriever that returns fifty chunks per query scores well on recall and hands the model a context window in which the answering passage is outnumbered. It also cannot see that a passage came from a superseded version of a document — matching text, wrong effective date, and the case passes.
Context precision
What it measures: whether the relevant chunks are ranked above the irrelevant ones.
How it is computed: the mean of precision at each rank position, weighted by whether the chunk at that position is relevant, over the retrieved set. It is order-sensitive by construction — an irrelevant chunk at position one costs far more than the same chunk at position five (Ragas documentation).
What a bad score means for the product: your context window is mostly filler. That is paid for three times, in tokens, in latency, and in dilution — every irrelevant chunk is another passage the model can ground a sentence in that is merely adjacent to the answer.
What it does not catch: near-duplicates. Five copies of the same clause, across two document versions and a PDF export, are all scored relevant, so precision looks excellent while the context carries one fact. It is also blind to what you failed to retrieve — perfect precision over a single relevant chunk is entirely compatible with a half-answer. Precision and recall are read together or not at all.
Mean reciprocal rank
What it measures: how far down the list the first correct passage sits.
How it is computed: one divided by the rank of the first relevant result, averaged over the set. A passage found at position one scores 1, at position four scores 0.25, and a miss scores nothing.
What a bad score means for the product: the passage exists in the index but sits below the cut you pass to the model. This is the reranking case, and reading MRR next to recall at two depths turns an argument about whether to add a reranker into an estimate of what it would buy.
What it does not catch: everything after the first hit. A question that can only be answered by combining two passages is scored as solved when one of them is found. Where answers require synthesis — most policy and contract questions do — MRR flatters the system and coverage at k is the number to report.
Normalised discounted cumulative gain
What it measures: ranking quality where relevance is graded rather than binary, with results devalued the further down they appear.
How it is computed: each result contributes its relevance grade, discounted by the logarithm of its rank position, cumulated down the list and then normalised against the best ordering possible for that query. Järvelin and Kekäläinen introduced the discount to devaluate late-retrieved documents, on the argument that the greater the ranked position of a relevant document, the less valuable it is to the user (ACM Transactions on Information Systems 20(4), 2002).
What a bad score means for the product: the right documents are being found and put in the wrong order, which for a fixed context budget is the same as not finding them.
What it does not catch: the quality of your own grades. nDCG is exactly as good as the graded labels behind it, and grading is the expensive part that gets delegated. Normalisation also hides difficulty — every query is scored against its own ideal, so a query whose ideal ordering is one mediocre passage can score a perfect 1. Report the distribution and the worst stratum, never the mean on its own.
Generation metrics: faithfulness, citation precision, answer relevance, refusal
Generation metrics score the answer against the context it was given. That framing is the thing to hold on to, because it is also the source of the blind spot every one of them shares: none of them knows whether the context was right.
Report these four separately and never blend them. An intervention that improves relevance while degrading faithfulness has made the system more persuasive and less trustworthy, and a single composite score is precisely the instrument that would hide it.
Faithfulness, also called groundedness
What it measures: whether every claim in the answer is supported by the retrieved context.
How it is computed: break the response into individual claims, check each one against the retrieved context, and score the number of supported claims divided by the total number of claims in the response (Ragas documentation). An answer with two claims, one of them unsupported, scores 0.5.
What a bad score means for the product: the model is filling gaps from its own weights. This is what users mean when they say the system hallucinates, and it is the failure that ends deployments, because it is invisible to the person least equipped to catch it — the one who asked the question because they did not know the answer.
What it does not catch: faithfulness is measured against the retrieved context, not against the world. Retrieve last year's policy, answer it perfectly, and the score is 1.0 while the customer is told a rule that no longer applies. Nor does it catch omission: an answer that is entirely grounded and quietly leaves out the exception is fully faithful and materially wrong. It is only interpretable alongside context recall, which is the metric that knows whether the right passage was there.
Citation precision
What it measures: whether the spans an answer cites actually support the sentences that cite them.
How it is computed: for each cited sentence, check the cited span against the claim; precision is the share of citations that hold. It is close kin to faithfulness and worth keeping separate, because a system can be grounded in the context as a whole and still attach the wrong reference to the sentence a reader will click.
What a bad score means for the product: a citation that does not support its claim is worse than no citation, because it is checkable and eventually someone checks it. We treat this as a data integrity bug rather than a wording problem, and it is the first thing instrumented in our RAG development work.
What it does not catch: citation recall — how many claims carry no citation at all. A system that cites one uncontroversial sentence perfectly and leaves the other four bare scores 1.0 on precision. Report both numbers or the metric can be gamed by citing less.
Answer relevance
What it measures: whether the answer addresses the question that was asked.
How it is computed: Ragas generates a set of questions from the answer, embeds them, and averages the cosine similarity between each generated question and the original one. The documentation is explicit that it assesses how well the answer matches the intent of the question without evaluating factual accuracy.
What a bad score means for the product: the system is answering a neighbouring question. This is common when retrieval finds the right document and the wrong section, and it reads to a user as evasiveness rather than error.
What it does not catch: anything at all about truth. A fluent, confident, wholly invented answer to exactly the right question scores well here. On its own, answer relevance rewards the single failure mode you most need to catch, which is why it is only ever read next to faithfulness.
Refusal correctness
What it measures: on cases where the correct behaviour is to decline, whether the system declined — and, in the other direction, whether it answered the cases it should have answered.
How it is computed: compare the expected_behaviour field against what the system did, and report both directions as two numbers. Wrong refusals and missed refusals are different product failures and averaging them together makes both invisible.
What a bad score means for the product: either the system is answering questions that have no answer, or it has retreated into declining everything and is no longer worth the licence fee. In a regulated deployment this is often the number the compliance function cares about most, and the one least likely to be on the dashboard.
What it does not catch: why it refused. A refusal caused by retrieval returning nothing looks identical to a refusal caused by judgement. Join refusals to the retrieval trace before celebrating them — refused with an empty shortlist is a retrieval bug with good manners.
Calibrate the judge before you trust the judge
Using a language model to score outputs is fine, and it is only fine once you have checked that its judgements agree with your reviewers on the cases where the two could reasonably differ. An uncalibrated judge is a random number generator with good manners.
The procedure is short. Take a sample from the golden set, have humans score it, have the judge score the same sample, and measure agreement — a chance-corrected statistic rather than raw percentage agreement, since a judge that always says pass looks excellent on a set that is mostly passes. Then read the disagreements, because they show what the judge is systematically missing. The recurring gap in our experience is that judges reward fluency: a confidently written wrong answer scores better than a hesitant correct one.
Pin the judge. Model, version, prompt and temperature belong under version control, because the judge is itself a model and it drifts when the provider updates it. A silent judge change makes every historical score incomparable, and you will not notice until you are explaining why the numbers moved in a week when nobody deployed anything. Keep a small human-scored set permanently so the calibration can be re-run whenever the judge changes.
Make it a gate, not a report
An evaluation that produces a number nobody is obliged to act on becomes a chart in a monthly deck. Wire it into continuous integration and let it block deploys, or do not bother building it.
Run the suite on every change to prompts, chunking, embeddings, retrieval parameters, reranker or model. Fix what must not move underneath you: a corpus snapshot, a pinned judge, a fixed seed where the stack allows one. Otherwise a change in the numbers is ambiguous between your change and the environment's, and an ambiguous signal is quickly ignored.
Define acceptance thresholds with the business before you measure anything. Measure first and the number you happen to get becomes the number you accept, and the conversation about whether it is good enough never happens.
Hold out a second set. After thirty tuning iterations against a golden set you have overfitted to it and the scores no longer generalise. Keep a release set that is run rarely and never tuned against; when the two diverge, the difference is the size of your overfitting.
- Corpus snapshot, pinned judge version, fixed seed
- Thresholds agreed with the business before the first measurement
- A held-out release set that nobody tunes against
- A failing gate blocks the deploy — including when it is inconvenient
"""Retrieval gate. Standard library only.
python eval_retrieval.py golden.json --retriever app.retrieval:search \
--k 5 --min-coverage 0.9 --min-mrr 0.7
Exits non-zero when any stratum is below threshold, so CI stops the deploy.
There are no default thresholds here: a default is a number nobody agreed to.
"""
import argparse, importlib, json, statistics, sys
from collections import defaultdict
def coverage_at_k(retrieved, reference, k):
"""Fraction of the annotated answering passages that reached the top k."""
top = set(retrieved[:k])
return sum(1 for chunk_id in reference if chunk_id in top) / len(reference)
def reciprocal_rank(retrieved, reference):
"""1/rank of the first annotated passage; 0 when none of them appear."""
for rank, chunk_id in enumerate(retrieved, start=1):
if chunk_id in reference:
return 1.0 / rank
return 0.0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("golden", help="JSON array of golden-set cases")
ap.add_argument("--retriever", required=True,
help="module:function, called as f(question, k) -> ranked chunk ids")
ap.add_argument("--k", type=int, required=True, help="the k you really pass to the model")
ap.add_argument("--min-coverage", type=float, required=True)
ap.add_argument("--min-mrr", type=float, required=True)
args = ap.parse_args()
module_name, function_name = args.retriever.split(":")
search = getattr(importlib.import_module(module_name), function_name)
with open(args.golden, encoding="utf-8") as fh:
cases = json.load(fh)
scores = defaultdict(list)
for case in cases:
# Cases whose correct behaviour is refusal have no answering passage to
# find. They belong to the generation harness, not to this one.
if case["expected_behaviour"] != "answer":
continue
reference = [c for p in case["reference_passages"] for c in p["chunk_ids"]]
retrieved = search(case["question"], args.k)
scores[case["category"]].append((
coverage_at_k(retrieved, reference, args.k),
reciprocal_rank(retrieved, reference),
))
failed = False
print(f"{'stratum':28} {'n':>4} {'coverage@k':>11} {'mrr':>6}")
for category in sorted(scores) + ["ALL"]:
rows = [r for c in scores for r in scores[c]] if category == "ALL" else scores[category]
coverage = statistics.mean(r[0] for r in rows)
mrr = statistics.mean(r[1] for r in rows)
# Gate every stratum, not only the total. One collapsed category
# averages away against nine healthy ones, and it is always the
# category somebody cares about.
bad = coverage < args.min_coverage or mrr < args.min_mrr
failed = failed or bad
print(f"{category:28} {len(rows):>4} {coverage:>11.3f} {mrr:>6.3f} {'FAIL' if bad else ''}")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())What that script does and what it deliberately leaves out
It loads the golden set, calls your retriever once per answerable case, scores coverage at k and reciprocal rank, reports both per stratum and in total, and returns a non-zero exit code when any stratum is under threshold. That exit code is the whole point: it is what a CI job reads, and it is why the same numbers stop being a chart in a monthly deck.
The thresholds are required arguments with no defaults. A default would be a number nobody in your business agreed to, and the moment it exists it becomes the number you accept. Measure a baseline, agree the thresholds against the cost of a wrong answer in your context, then pass them in.
This half runs in seconds and needs no judge, which is the argument for splitting it out: it can run on every commit while the generation half — faithfulness, citation precision, answer relevance, refusal correctness, all of which need a model in the loop — runs on the merge. Generation scoring is the same loop with the judge substituted for the metric functions, and the same rule about per-stratum thresholds applies.
The ways an evaluation harness quietly stops being valid
Harnesses decay. The four failure modes we see repeatedly are corpus drift, leakage, overfitting and judge drift, and none of them announce themselves.
Corpus drift is the commonest. Your golden set was annotated against a corpus that has since changed: the policy was updated, so the answer that was correct in March is now wrong, and the harness marks a correct answer as a failure. Version the corpus alongside the golden set and re-adjudicate on a schedule.
Leakage happens when golden set examples end up in the few-shot prompt or in a fine-tuning run. The scores improve and mean nothing. Keep the two sets physically separate and check before every training job.
Add one more to the list: the harness measures what it was built to measure, and your users have since started asking a different kind of question. Refresh the golden set from recent logs, and compare the distribution of question types in the harness against production. When those diverge, your excellent scores describe a system nobody uses that way any more. The diagnostic order for the failures the harness surfaces is in why your RAG system gives wrong answers.
Evaluation after launch is the part that decides the outcome
Pre-production evaluation tells you whether to launch. Production evaluation tells you whether it is still working, and Barnett et al. are right that some classes of failure only appear once real users arrive.
Four things to run continuously. Sample live traffic and score it with the same judge and the same metrics, so pre-production and production numbers are comparable. Log retrieval score distributions and alert when they shift, since a change in the distribution is the earliest signal of a stale index, a failed ingestion or embedding drift. Triage negative feedback weekly against the taxonomy rather than reading it as sentiment. And have a human review a fixed number of answers a week, chosen at random rather than by complaint, because complaints are biased towards the failures users noticed.
That last one is worth defending. The dangerous failures in a retrieval system are the confident wrong answers nobody questions. Those never generate a thumbs-down, so a monitoring strategy built on user feedback is structurally blind to exactly the class of error that will eventually cause the incident.
All of this depends on a trace per query: the question, the retrieved documents with scores, the assembled context, the output, the citations, the model and prompt versions, latency and cost. Without it, evaluation in production is not possible and neither is answering the question that always arrives eventually, which is why the system said what it said. The same record does duty for regulatory purposes, as set out in what the DPDP Act actually requires of an AI system.
- Score sampled live traffic with the same metrics as pre-production
- Alert on shifts in retrieval score distribution
- Weekly human review of a random sample, not only of complaints
- A full trace per query, versioned and retained deliberately
Sources
Every regulatory or measured claim above is attributed here. Where we could not find a source we trust, the sentence says so instead of guessing.
- Seven Failure Points When Engineering a Retrieval Augmented Generation System — Barnett, Kurniawan, Thudumu, Brannelly, Abdelrazek — arXiv 2401.05856, January 2024 · validation is only feasible during operation; robustness evolves rather than being designed in
- RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es, James, Espinosa-Anke, Schockaert — arXiv 2309.15217, September 2023 · reference-free metrics separating retrieval quality from generation faithfulness
- Ragas metric definitions: context precision, context recall, faithfulness, response relevancy — Ragas documentation · the formulas quoted in the metric sections above, including that response relevancy does not evaluate factual accuracy
- Cumulated gain-based evaluation of IR techniques — Järvelin & Kekäläinen — ACM Transactions on Information Systems 20(4), 2002, pp. 422–446 · the rank discount behind nDCG: later results are devalued because position lowers their value to the user
Questions people ask about this
How large does the golden set need to be?
A few hundred adjudicated pairs is enough to detect changes worth acting on, and small enough that domain experts will actually finish it. Coverage of the categories that matter beats raw size — fifty well-chosen adversarial and unanswerable cases will tell you more than another two hundred routine ones.
Can we use an LLM to generate the test questions?
For coverage of a corpus, partly. Generated questions tend to mirror the document's own phrasing, which is precisely the mismatch you need to measure, so a set built that way will flatter your retrieval. Use generation to fill gaps in coverage, and take the core of the set from real user logs.
What accuracy should we expect before going live?
There is no general answer, and any figure quoted before someone has seen your corpus and your question distribution is invented. What we do is measure a baseline first, then agree thresholds with the business against that baseline and the cost of a wrong answer in your context. A wrong answer in an internal helpdesk and a wrong answer in a lending decision are not the same event.
Which RAG metrics should we actually report?
Context recall and one ranking measure on the retrieval side, faithfulness and answer relevance on the generation side, refusal correctness alongside them, and cost and latency on the same page. Report every one of them per category rather than as a single average, and never combine them into one composite score — the composite is what hides an intervention that improved relevance by degrading faithfulness.
Is RAGAS or a similar framework enough on its own?
It gives you metric implementations and saves real work, which is worth having. It does not give you a golden set drawn from your users, adjudication by your experts, thresholds agreed with your business, or the discipline of gating deploys. Those four are the parts that determine whether the evaluation changes any decision.
Related
Where this connects
RAG & LLM Integration
Retrieval that cites its sources, measured with a real evaluation harness.
AI Consulting & Readiness Assessment
A paid diagnostic that tells you what is worth building and what is not.
Data Engineering for AI
The pipelines and quality layer every AI project stalls on.
Read next: Why your RAG system gives wrong answers, in order of likelihood · The gap between an AI demo and something that runs unattended
More notes on the blog index, and the practical questions about cost, IP and data residency are on the FAQ.
We build the harness before we tune anything
Untested tuning is guesswork with a bill attached. Bring us a system you cannot measure and the first phase will be making it measurable, with a written baseline you keep.