The same questions are reused for prompt tuning: how do you detect evaluation overfitting?

Mark questions used in debugging and examples

This applies to high scores on a small fixed set but poor paraphrases and answers embedded directly in demonstrations. Record sources, related questions and prompt history and distinguish development from final evaluation. Public benchmarks can have unknown training overlap. Renaming files does not create genuinely new questions.

Separate development data from a held-out test set

Check overlap with demonstrations, training and fine-tuning, including paraphrases and questions derived from shared documents. Repeated candidate selection using test results also reduces independence. Split by real task units; randomly separating related rows can leak common information, and different row numbers do not establish independent data.

Fix scoring before comparing candidates

Iterate on development data and reserve tests unused in selection, grouping by time or source where suitable. Fix questions, rubrics and versions for comparison and add representative new samples as needed. If tests have been repeatedly used, label them honestly as development results and establish fresh acceptance rather than delete failures to raise averages.

Report independent test results and data overlap

Report development and independent sample counts, types and outcomes and check remaining near-duplicates. Test unseen wording and type-specific regressions. Acceptance requires evidence from independent material. Near-perfect results on a reused small set or the best single run do not establish general capability gains.

This article ran no evaluation and confirmed no training-data leak; it describes evaluation organization. Even held-out data may not represent real work and needs versioned expansion. Data confidentiality and quality matter; do not obtain unauthorized private material merely to expand tests. Mark unknown training overlap uncertain rather than proven.