What is retrieval evaluation? Checking coverage and ranking separately from generation
Understand retrieval evaluation through manual questions, including queries, corpora, relevance labels, Recall@k, ranking metrics, and missed-evidence diagnosis.
Understand the concept
Retrieval evaluation checks whether required evidence is found and where it ranks. It usually needs queries, a corpus, and relevance judgments rather than fluent final answers. Good retrieval does not guarantee good generation, and a correct answer may lack reliable evidence. Evaluate both stages and the full task separately.
Identify the source passages needed to answer
If a manual question needs a rule and exception, mark both rather than only an easy sentence. Tie labels to versions and review them after changes. Resolve unclear relevance definitions before averaging scores.
Sentence Transformers uses queries, corpora, and relevant IDs; Microsoft distinguishes retrieval and answer evaluation. Align label granularity with returned chunks. This describes preparation without building a real dataset or reporting scores.
Coverage and early ranking are different measures
Recall@k measures how much labeled relevant content appears in the first k results; ranking measures additionally consider positions. Late evidence differs from absent evidence. Check denominators and handling of unanswerable queries.
Inspect single-passage and multi-evidence questions separately. An average can hide cross-passage failures. More returned results change coverage and noise; report irrelevant content and context usage alongside recall.
Compare under fixed conditions and inspect final answers
Compare chunking or model changes with the same queries and corpus, retaining per-query hits. Fix ingestion if evidence is absent from the index. If retrieval is complete but generation omits exceptions, inspect generation inputs and instructions.
Model-based judges can help but need source-checked sampling of important failures. No universal passing threshold is prescribed. Suitability depends on tasks and error costs; record tested scope and uncovered formats.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.