RAG invents answers without evidence: how do you test refusal and requests for missing material?
When a knowledge base answers without evidence, check whether retrieval supplied irrelevant material and whether generation treated similar text as support. Build clearly answerable and unanswerable questions, requiring explicit limits and requests for specific missing material.
Start troubleshooting here
When a knowledge base answers without evidence, check whether retrieval supplied irrelevant material and whether generation treated similar text as support. Build clearly answerable and unanswerable questions, requiring explicit limits and requests for specific missing material. Raising a similarity threshold alone does not establish that fabrication has been eliminated.
Pair questions with and without sufficient evidence
This applies when an unrecorded product parameter receives an exact number or a cited document lacks the asserted fact. Use a small, bounded collection and annotate the facts it supports. Unanswerable samples must truly exceed its scope; do not demand refusal merely because wording changes when the source still permits a reasonable answer.
Check whether retrieved material actually supports the claim
Read the actual passages for the required object and conditions instead of relying on scores. Inspect instructions for insufficient-evidence behavior and map citations to individual claims. Record complete retrieval failure separately from partial missing information. If only one field is unknown, supported portions of the answer need not necessarily be refused.
Specify missing-answer behavior and the material needed
Ask the answer to distinguish supported information from missing information, leave unknown fields unknown and identify the material needed. Calibrate any retrieval threshold on your own questions and check whether scores are comparable across query types. Supporting excerpts or points can aid verification, but the mere presence of a citation does not establish factual support.
Track both unsupported answers and incorrect refusals
Test answerable, unanswerable and partly answerable cases, counting unsupported answers and refusals despite adequate evidence. Requests should identify specific material, such as a product version document, rather than vaguely ask for more information. Retest previously correct questions after changes; refusing everything can create a misleadingly low fabrication rate.
This article ran no model evaluation and reports no accuracy gain. Execute the proposed design in the actual system. Scores and refusal thresholds depend on the retriever and cannot simply transfer between systems. Decide in advance whether public general knowledge is allowed; if the task requires knowledge-base-only answers, model memory cannot replace missing evidence.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.