Retrieval-Augmented Generation: Why Sources Can Still Mislead
Retrieval-augmented generation (RAG) can cite real documents and still mislead. Explore retrieval failures, evidence gaps and five practical verification checks.
DeepTech analysis · 5 November 2026. This article interprets published research; DeepTech has not independently tested the systems discussed.
Retrieval-augmented generation (RAG) connects AI-generated answers with retrieved external information and is often described as a way to give AI answers reliable sources. That is useful, but it creates a dangerous misunderstanding: a source-backed answer is not automatically a verified answer. The difference becomes obvious when the document is genuine but outdated, the relevant paragraph is missing, or the model draws a conclusion the document never made.
Instead of asking whether RAG can attach citations, a better question is: can we trace each important claim to sufficient, current and correctly interpreted evidence? This article explains how to make that distinction and what an organisation should measure before relying on RAG.
What retrieval-augmented generation actually does
RAG combines retrieval of external information with language generation. A user asks a question, a search component selects relevant passages from a collection of material, and a language model uses the material as context when producing its response. The foundational research by Patrick Lewis and colleagues explored this combination for knowledge-intensive language tasks in 2020.
Different products implement retrieval and generation differently. Therefore, an answer that mentions RAG tells you little about its actual reliability. It does not reveal whether the source collection is complete, whether citations match the claims or whether access to restricted documents is controlled.
A real-looking citation can support the wrong conclusion
Consider this fictional example, created solely to explain the problem. An employee asks an assistant whether a product is covered by a three-year warranty. The retrieval system finds an authentic company document that refers to a three-year warranty, and the assistant replies yes. But the document applies only to a discontinued product range. The citation is real; the answer is wrong for the employee’s product.
A second fictional case is a question about a law or regulation. The system retrieves a previous version of the rules but misses a later amendment. The response may accurately describe an older document while still misleading the reader about what applies today. More confident wording, longer answers or additional links do not fix this problem.
Three different failures that look similar to readers
- Retrieval failure: the correct evidence exists in the collection, but the search component does not return it.
- Evidence failure: the returned material is genuine but incomplete, obsolete, contradictory or not relevant to the specific question.
- Interpretation failure: the necessary evidence is available, but the generated answer misreads it, omits an exception or makes an unsupported leap.
These failures require different remedies. Improving search may address the first. Better document governance, dates and version control are needed for the second. The third requires testing how faithfully the system uses the retrieved evidence and whether it declines to answer when support is insufficient.
What the published research establishes
The 2020 RAG paper by Lewis and colleagues reported improvements over comparison approaches on several knowledge-intensive tasks in the conditions studied. Its contribution was to explore models combining generated language with retrieved external information, not to prove that citations make every output factually correct.
A 2025 Google Research study examined whether retrieved context actually contained enough information to answer a question. The researchers found that models could still provide incorrect answers when the available context was insufficient, and investigated approaches that encourage abstention. Those findings concern the evaluated models, datasets and procedures, rather than every RAG deployment.
An original editorial test: five questions before trusting an answer
A practical way to review a RAG response is to separate the claim, the cited material and the context of use. The following checklist is a DeepTech editorial framework for readers and reviewers; it is not a standard or a validated benchmark.
- Identity: does the cited document describe the same product, patient population, system or regulation the question is about?
- Time: what is the date of the evidence, and could a newer version change the answer?
- Support: does the document explicitly justify the claim, including limits and exceptions?
- Completeness: might a missing page, appendix, policy or document reverse the interpretation?
- Outcome: what should the assistant do when the evidence is insufficient — explain uncertainty, request clarification or decline to answer?
Why two RAG systems can behave differently on the same documents
Imagine two assistants searching the same internal manual. One retrieves only the paragraph that matches a keyword; the other also retrieves the neighbouring eligibility conditions. The first answer may appear decisive but miss the exception. The second has a better chance of identifying the boundary — yet can still misinterpret what it reads.
That example is hypothetical; it is not the result of a comparative experiment. Its purpose is to show why the number of retrieved pages or citations is not, by itself, a useful quality metric. Coverage and interpretation both matter.
How to evaluate RAG without being misled by a single score
Start with a collection of questions for which there are known, independently checked answers. Include ordinary cases, ambiguous questions, outdated documents and questions that the document collection genuinely cannot answer. Keep a frozen test set separate from examples used to refine the system.
Then report at least three separate outcomes: whether the necessary evidence was retrieved, whether the final answer was supported by that evidence, and whether the system responded appropriately when information was missing. A high answer score on familiar questions should not hide unsupported statements on unfamiliar ones.
Where a system handles sensitive information, evaluate access controls separately. Accurate retrieval is not a success if private records are exposed to a user who should not receive them. For consequential decisions, qualified human review and an incident-correction process remain necessary.
When RAG is useful — and when to be cautious
RAG can be valuable for searching lengthy manuals, finding relevant passages in internal knowledge bases and incorporating frequently updated information without relying entirely on a model’s stored knowledge. However, it deserves greater caution when the source material is incomplete, conclusions depend on legal or clinical context, or an error could cause physical, financial or personal harm.
A sensible publication or procurement claim is therefore not ‘RAG eliminates hallucinations’. A stronger, testable claim would be that a particular implementation achieved specified performance on documented questions and evidence collections, with reported failure cases and limits.
DeepTech takeaway: evidence is more important than the appearance of evidence
Source links can help readers investigate a claim. But the publisher or system operator must still check that the material actually supports the sentence beside it. This is why transparent evaluation matters more than decorative citations or a high automatic SEO score.
Further reading from DeepTech: how independent AI testing should be evaluated and the accuracy and accountability challenges of generative AI.
