LLM judges versus human evaluation: calibrate the scoring standard first
Compare model grading with human review, checking judge bias, references and disagreement resolution.
Choose an approach
Consider model judges for repeated grading under clear criteria and human calibration for unclear standards or complex facts. Combine automated candidate scores with human disagreement review. A numeric model score is not inherently objective; calibrate against checked examples and consistent task criteria.
Judge models can be influenced by presentation
The MT-Bench and Chatbot Arena paper discusses position and verbosity biases. OpenAI recommends calibrating automated grading against human feedback. If swapping answer order changes a verdict, presentation may be driving it. Humans also disagree, so define shared criteria. Facts need evidence rather than confidence, and a judge explanation is not independent verification.
Inspect scores and disagreements on one answer set
Use human-checked correct and incorrect samples with identical requirements and references. Keep item-level decisions and inspect clustered disagreements. Swap pairwise order to detect position effects. A judge without sources cannot verify factual grounding. Reserve samples not used to tune grading prompts. No judge was run and no human-agreement rate is claimed.
Maintain judge versions and human standards
Replay disagreements after task or judge changes to detect score drift. Keep human review for important errors; scores are not automatic publication permission. Include labeling, calls and disagreement analysis. Models can flag candidates for review under predefined sampling rules. Report sample scope and omissions instead of extending limited agreement to every task.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.