Retrieval hits are high but answers are untrustworthy: how do you evaluate recall and groundedness separately?
When retrieval hits are high but answers are unreliable, evaluate evidence retrieval separately from whether actual context supports answer claims. A relevant document in candidates does not prove use of the correct passage, and citations do not prove conclusions.
Start troubleshooting here
When retrieval hits are high but answers are unreliable, evaluate evidence retrieval separately from whether actual context supports answer claims. A relevant document in candidates does not prove use of the correct passage, and citations do not prove conclusions. Save retrieval and generation inputs and inspect passages and claims instead of masking distinct failures in one overall score.
Preserve candidates and passages actually used for generation
This applies to invented numbers, lost exceptions and wrong-section citations despite correct material in results. Define annotation granularity, acceptable evidence and allowed outside knowledge. Document-level hits can be too broad: an answer in the file does not establish successful retrieval and generation if the used passage lacks its key sentence.
Define evidence recall separately from answer support
Locate evidence in candidates and final context, then split answers into checkable claims labeled supported, conflicting or unknown. Groundedness in supplied material differs from real-world truth if sources are stale. Judge models need rubrics and spot checks rather than assuming citation formatting proves every claim.
Check each claim and omitted conditions against the context
Keep separate records for passage relevance and answer support, including limitations and effective dates. Fix candidate generation for recall gaps; adjust generation and refusal when complete evidence is supplied but exceeded. Change the relevant stage and inspect unnecessary refusals. Retrieval scores are not direct probabilities of answer truth.
Count retrieval and generation failures separately
Count missing evidence, context-assembly loss and unsupported generation separately and review examples. Include unanswerable and partly answerable questions to detect related-but-insufficient evidence used to fill conclusions. Acceptance needs recall and supported claims. High document hits, fluency and average scores do not replace either check.
This article ran no retrieval or judge model and reports no gains. Evaluators define hits and support differently, requiring explicit inputs and annotations. Retain sample counts and types rather than treat a few manual observations as population metrics. Critical facts still need direct source checks instead of only automatic scores.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.