A model upgrade breaks a working task: how do you preserve evidence and verify a rollback?

Capture the model identifier at the time of failure

Use this process when formatting becomes unstable, citation quality drops or a previously working tool choice fails after an upgrade. Record the time and available model identifier. If a service exposes only a dynamic alias, note that the underlying version cannot be pinned; the same alias does not prove an identical model was used on every request.

Separate model changes from other configuration changes

Compare the actual requests before and after the upgrade, including system instructions and tool schemas, and check whether the service changed defaults such as temperature. Distinguish application-cache hits from fresh generation. Start with a minimal failing example rather than attributing every difference after a multi-component release to the model upgrade.

Compare versions on the same task set

Where the previous version remains available, run the same evaluation set in isolation with identical data and tool results. Categorize failures as format errors, factual errors or unfinished tasks and retain raw outputs. Use an existing rollback only if the service actually permits the old version and its API remains compatible; otherwise make evidence-backed compatibility changes.

Retest previously successful tasks after rollback

After rollback or adjustment, rerun the original failures and sample previously successful tasks to ensure a formatting fix did not break other work. Record the actual version and each result, including successful tool completion. State which tasks the fix covers rather than reporting only an average score.

Some hosted products cannot restore older models; retaining an application prompt cannot recreate a withdrawn underlying version. This page promises no provider retention period and performs no production switch. When pinning is unavailable, preserve the failure time and request identifier for the service maintainer to investigate.