LLM judge scores fluctuate: how do you test rubrics and evaluation bias?

Preserve rubrics, answer order and judge versions

This applies to repeated scores changing, longer answers always winning and swapped positions reversing preferences. Preserve full inputs and outputs and check omitted evidence or differing rules. Absolute scoring and pairwise preference address different questions; a win table is not automatically a dimension-specific factual-accuracy measure.

Separate ambiguous criteria from position and length bias

Inspect observable criteria and necessary references or evidence and check judgments for irrelevant style preferences. Blinding candidate sources may reduce some biases without guaranteeing neutrality. Judge models can themselves fail calculations and facts. Verify important values directly rather than assume requesting a scoring explanation solves reliability.

Calibrate with anchors and swap candidate order

Provide anchors for scores or pass conditions, bound the rubric and require evidence. Randomize and swap paired positions and preserve disagreement rather than select convenient results. Keep a human-checked calibration set and establish alignment before scaling assistance. Reevaluate older samples after rubric changes instead of mixing scores from different versions.

Compare repeated scores with human judgments

Repeat judgments on the same samples and compare with human labels across order swaps, length changes and task types. Record disagreement and disputed cases for review. Acceptance needs both stability and validity. Valid JSON is not sufficient, and consistently favoring wrong answers remains an invalid evaluation.

This article called no judge model or reported agreement rates. Research establishes possible bias types, not identical magnitudes for every model. Human labels can disagree and need rules and reasons. Multiple judges may assist but cannot automatically eliminate shared mistakes. Judge critical results against task evidence.