Search results are wrong: how do you distinguish poor recall from incorrect reranking?
For poor retrieval, first check whether the correct passage appeared among candidates before reranking. Reranking cannot invent a candidate that was never retrieved.
Start troubleshooting here
For poor retrieval, first check whether the correct passage appeared among candidates before reranking. Reranking cannot invent a candidate that was never retrieved. If it was present but ranked poorly, inspect reranker inputs and fields. Measure recall omissions separately from ranking errors instead of changing the reranker while the actual evidence remains missing.
Establish whether initial retrieval ever contained the right passage
This applies when relevant files rank too low or answers cite related passages that do not support the question. Annotate the correct passage and acceptable alternatives for each failure and retain current filters. Similar titles do not establish relevant content. Document-level labels alone can count a wrong section as a successful retrieval and hide the problem.
Save candidate lists separately from reranked results
Save candidate IDs and their reranked order and confirm that reranking received body text rather than an empty or truncated title. Inspect query wording and field configuration, including whether long passages lose key sentences. Filters excluding the correct material create a candidate-generation or access-scope issue; increasing unrelated passage scores does not repair it.
Adjust recall or ranking at the stage that lost the evidence
For poor recall, compare keyword, vector and hybrid retrieval and appropriate candidate ranges. Compare reranking strategies only after candidates are adequate. Change one condition at a time and record latency and final passages. The number passed to generation must respect context limits; retrieving more does not mean placing every candidate into the final prompt.
Compare inclusion and position on a fixed question set
On a fixed labeled set, measure whether supporting evidence is retrieved and whether it falls within the range actually used. Record recall and ordering changes separately, then inspect final answer support. Include identifiers and paraphrases to catch regressions across query types. Do not report percentage gains without actual measurements.
This article ran no retrieval or reranking service and specifies no universal candidate count. Services impose particular reranker input limits; consult the relevant version. Labels may admit several reasonable sources, so evaluation should not demand a single ordering. A higher reranker score alone does not establish factual accuracy or correct access filtering.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.