How do you compare models fairly with matching prompts and sampling settings?

Fix task inputs and evaluation goals first

This applies to rankings where one model receives more evidence or more output space. Preserve system instructions, user input and tools and verify the same task. Model-specific adaptation can be legitimate, but define beforehand whether comparing identical inputs or individually optimized systems. Do not combine those experiments without distinction.

Check shared parameters and model-specific constraints

Inspect effective temperature, top-p and maximum output and whether unsupported settings are ignored. Compare truncation, retries and safety limits, including one model receiving automatic retries while another failure counts directly. A shared seed value does not establish equivalent randomness across models or devices.

Preserve every output and complete run conditions

Use predefined questions and scoring and retain every output and error, not only successes. Repeat stochastic tasks where appropriate and record settings and versions. Blind evaluators to source and randomize order where needed. Separate quality from speed and do not change weights afterward to obtain a favored conclusion.

Compare task quality and state the valid comparison scope

Compare paired task outcomes and typical failures, then inspect variation by question type and repeated run. Report sample counts, conditions and differences that could not be controlled. Success means traceable fair comparison, not a particular winner. Few questions or a single run cannot establish superiority across all languages and tasks.

This article called no model and supplies no ranking, price or performance result. Sampling support changes and needs current verification. Identical inputs and individually optimized settings address different questions; both can be studied with explicit labels. Disclose noncomparable conditions rather than conceal them behind matching parameter names.