How to evaluate a RAG system before you trust it in production

Source from logs. Support tickets, chat transcripts, analyst queries, the questions people ask in the internal channel where they give up on the search tool. These carry the phrasing your users actually use, which is almost never the phrasing your documentation uses, and that mismatch is itself one of the things you need to measure.

Record for each case the question, the passage that genuinely answers it — document, version and location, not just a summary — the expected answer, and a label for the category it belongs to. The passage is the part teams skip and the part that makes retrieval measurable. Without it you can score the final answer and nothing else, which leaves you unable to tell a ranking problem from a generation problem.

Size: a few hundred adjudicated pairs is enough to detect the changes worth acting on, and it is small enough that subject experts will actually complete it. Adjudication by experts, not by the engineering team. And treat expert disagreement as data rather than noise — where two reviewers who know the domain disagree about the correct answer, you have found a question the system cannot reasonably be expected to get right, and it should be scored as ambiguous rather than counted as a failure.

• Questions mined from real logs, in the users' own phrasing
• The answering passage recorded with document, version and location
• Adjudicated by domain experts, with disagreements marked rather than resolved by vote
• Stratified by category so you can report per stratum instead of one average

On the retrieval side, recall at k is the primary measure: over your golden set, how often the annotated answering passage appears in the top k retrieved. Report it at two depths — a small k matching what you actually pass to the model, and a larger one. The gap between them is the size of the prize available from reranking, and it converts an argument about whether to add a reranker into an estimate. Mean reciprocal rank and nDCG are useful supplements where position within the retrieved set matters.

On the generation side, four measures carry most of the weight. Groundedness: is every claim in the answer supported by a retrieved span. Citation precision: do the cited spans actually support the sentences that cite them. Answer relevance: does the answer address the question asked. Refusal correctness, as above. Keep these apart. An intervention that improves relevance while degrading groundedness has made the system more persuasive and less trustworthy, which is the worst direction available.

Cost and latency belong on the same dashboard. Tokens per query, cost per query, and p50 and p95 latency. Quality and cost trade against each other continuously and the trade is a business decision — it cannot be made by a team that sees only one of the two numbers.

• Recall at k, at two depths, as the primary retrieval measure
• Groundedness, citation precision, answer relevance and refusal correctness, reported separately
• Tokens, cost and p95 latency alongside quality, never on a different page
• No single blended score, ever

Run the suite on every change to prompts, chunking, embeddings, retrieval parameters, reranker or model. Fix what must not move underneath you: a corpus snapshot, a pinned judge, a fixed seed where the stack allows one. Otherwise a change in the numbers is ambiguous between your change and the environment's, and an ambiguous signal is quickly ignored.

Define acceptance thresholds with the business before you measure anything. Measure first and the number you happen to get becomes the number you accept, and the conversation about whether it is good enough never happens.

Hold out a second set. After thirty tuning iterations against a golden set you have overfitted to it and the scores no longer generalise. Keep a release set that is run rarely and never tuned against; when the two diverge, the difference is the size of your overfitting.

• Corpus snapshot, pinned judge version, fixed seed
• Thresholds agreed with the business before the first measurement
• A held-out release set that nobody tunes against
• A failing gate blocks the deploy — including when it is inconvenient

Four things to run continuously. Sample live traffic and score it with the same judge and the same metrics, so pre-production and production numbers are comparable. Log retrieval score distributions and alert when they shift, since a change in the distribution is the earliest signal of a stale index, a failed ingestion or embedding drift. Triage negative feedback weekly against the taxonomy rather than reading it as sentiment. And have a human review a fixed number of answers a week, chosen at random rather than by complaint, because complaints are biased towards the failures users noticed.

That last one is worth defending. The dangerous failures in a retrieval system are the confident wrong answers nobody questions. Those never generate a thumbs-down, so a monitoring strategy built on user feedback is structurally blind to exactly the class of error that will eventually cause the incident.

All of this depends on a trace per query: the question, the retrieved documents with scores, the assembled context, the output, the citations, the model and prompt versions, latency and cost. Without it, evaluation in production is not possible and neither is answering the question that always arrives eventually, which is why the system said what it said. The same record does duty for regulatory purposes, as set out in what the DPDP Act actually requires of an AI system.

• Score sampled live traffic with the same metrics as pre-production
• Alert on shifts in retrieval score distribution
• Weekly human review of a random sample, not only of complaints
• A full trace per query, versioned and retained deliberately

How large does the golden set need to be?

A few hundred adjudicated pairs is enough to detect changes worth acting on, and small enough that domain experts will actually finish it. Coverage of the categories that matter beats raw size — fifty well-chosen adversarial and unanswerable cases will tell you more than another two hundred routine ones.

Can we use an LLM to generate the test questions?

For coverage of a corpus, partly. Generated questions tend to mirror the document's own phrasing, which is precisely the mismatch you need to measure, so a set built that way will flatter your retrieval. Use generation to fill gaps in coverage, and take the core of the set from real user logs.

What accuracy should we expect before going live?

There is no general answer, and any figure quoted before someone has seen your corpus and your question distribution is invented. What we do is measure a baseline first, then agree thresholds with the business against that baseline and the cost of a wrong answer in your context. A wrong answer in an internal helpdesk and a wrong answer in a lending decision are not the same event.

Is RAGAS or a similar framework enough on its own?

It gives you metric implementations and saves real work, which is worth having. It does not give you a golden set drawn from your users, adjudication by your experts, thresholds agreed with your business, or the discipline of gating deploys. Those four are the parts that determine whether the evaluation changes any decision.