Offline evaluation and online A/B: separate fixed samples from real behavior
Compare fixed-dataset quality checks with randomized product experiments, assessing coverage, assignment and stopping rules.
Choose an approach
Use offline examples to catch defined errors before release and appropriate online validation for user behavior. They complement each other. Higher offline scores may miss real inputs and more clicks do not establish accurate answers. Define changes and criteria before fixing datasets or assignment rules rather than changing success after seeing outcomes.
Fixed-input quality differs from real user response
OpenAI evaluates task examples; Microsoft’s experimentation work studies controlled online comparisons. Offline inputs support repeatable error diagnosis. Online randomization compares behavior when conditions are appropriate. Changing both model and layout prevents attributing behavior solely to answer quality. Task completion and factual accuracy are distinct, so keep content checks and product behavior separate.
Fix datasets and assignment for one change
Replay normal and boundary examples on both versions before defining a permitted online experiment. Specify assignment units and observation windows and check users do not switch inconsistently between variants. Record errors and completion and define recovery for serious regressions. Do not retune samples or declare winners from short fluctuations. No live traffic or experiment was changed here and no statistical result is reported.
Preserve explainable evaluation records over time
Add emerging failures to offline data while retaining a stable baseline. Track concurrent product changes online to avoid attributing seasonal traffic to models. Include labeling, analysis and recovery. Results support the evaluated conditions; new languages or audiences need validation. Expand exposure according to predefined evidence and report verified conditions and uncertainty.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.