Why your RAG system gives wrong answers, in order of likelihood

Ingestion pipelines drop things without complaining. A PDF extractor returns empty strings for scanned pages and the job reports success. A file type nobody handled is skipped by a glob pattern. An access filter excluded a whole folder. A crawl hit a rate limit on the third of five sources and moved on. In every case the index looks healthy and the answer was never there.

The diagnostic is direct: search the raw ingested text — not the vector store, the raw text — for a distinctive phrase from the document that should hold the answer. If it is absent, stop. Nothing downstream recovers content that was never indexed. Make the check permanent: record document, page, character and failure counts per source on every run and alert on a change. Inherited corpora almost always contain a source that stopped updating months earlier.

• Search raw extracted text first, not the index
• Log per-source document, page, character and failure counts on every run
• Alert on a drop, not only on an error — silent partial success is the normal failure mode

The diagnostic that separates them is recall at two depths. Compute recall at five and recall at fifty over questions with known answer passages. If recall at fifty is high and recall at five is poor, retrieval works and ranking does not — add a reranker and most of the problem disappears. If recall at fifty is also poor, the embedding or the query is the issue and a reranker will not save you.

The usual cause is vocabulary mismatch. The user asks about a cancellation charge; the document says pre-closure fee. Dense embeddings handle paraphrase well and handle acronyms, product codes and company jargon badly, because those tokens carry meaning only inside your organisation. Hybrid retrieval fixes much of it: lexical search alongside dense search, fused, then reranked with a cross-encoder. Query rewriting helps where phrasing is short or pronoun-laden, which in a chat interface is most of the time.

Three mechanisms account for most of it. Truncation, where the assembled context exceeded a limit and the tail that held the answer was cut. Compression or deduplication steps that discarded the passage as redundant. And position effects: Liu et al., in Lost in the Middle (Transactions of the ACL, volume 12, 2024), found performance is highest when the relevant information sits at the beginning or the end of the input and degrades significantly in the middle of a long context.

The consequence is direct. Stuffing forty chunks into a long window is not a substitute for ranking. Rerank a shortlist, pass fewer and better passages, position them deliberately. Log the final assembled context, not only the retrieved set — the two diverge more often than teams expect, invisibly unless you record both.

It is common with internal documentation nobody owns. A wiki page written four years ago by someone who has left; a policy PDF superseded by an email; a rate table a department maintains privately. Retrieval systems surface all of it, because they read everything and remember none of the organisational context that told humans which document to trust.

The only durable fix is ownership: a named owner and a review date per source, with content past its review date refreshed or removed. When a wrong answer traces to a stale source, report it to the owner as a document defect rather than a model bug, and the incentive lands in the right place.

Easy to miss: some questions are not retrieval questions. How many contracts expire next quarter, what the exposure to a counterparty is, which customers hold both products — these are aggregation queries over structured data, and similarity search over chunks cannot answer them reliably. Route them to a query engine and let retrieval decline. That routing is data engineering work more than model work.

Our answers got worse after we upgraded the embedding model. Why?

Almost certainly because the index was not rebuilt, so queries are embedded with the new model and documents with the old one. The two vector spaces are not comparable and similarity scores become meaningless while still looking plausible. Treat a re-embed as a versioned deployment, and keep the old index until the new one has been evaluated.

Would a larger context window remove the need for good retrieval?

No. Liu et al. found that performance depends significantly on where the relevant information sits within a long context, with the middle handled worst. Longer windows let you be lazier about how much you retrieve, not about how well you rank, and they cost more per query for the privilege.

How do we stop the system answering when it does not know?

Give it an explicit refusal path and make refusal a scored outcome rather than a failure. Check groundedness after generation: match each claim back to the span it cites, and if support falls below a threshold, refuse or escalate. A compliance team will take a refusal over an invented clause every time, provided the refusal rate is monitored.

How many wrong answers do we need before this triage is worth running?

Fifty is enough to see the shape of the distribution, and you can usually collect fifty from a week of logs. The point is not statistical precision. It is finding out whether your failures are ninety per cent retrieval or fifty-fifty, because those two situations call for entirely different work.