A prompt change worsens results: how do you compare template versions and keep a recovery record?
When a template change worsens results, associate outputs with specific versions and compare on the same test set rather than judging the latest answer alone. Save the recoverable old template and current diff and ensure the model and evidence did not change simultaneously.
Start troubleshooting here
When a template change worsens results, associate outputs with specific versions and compare on the same test set rather than judging the latest answer alone. Save the recoverable old template and current diff and ensure the model and evidence did not change simultaneously. Adopt a new template only after the original failure is fixed and previously working tasks still pass.
Link requests to complete template versions
This applies when several people edit a prompt without tracing failures to a change or restoring earlier behavior. Version templates explicitly and record the version used by each request. If runtime variables and multiple files assemble the prompt, committing one fragment does not preserve the complete request.
Separate template differences from other component changes
Compare actual old and new text for overridden requirements and examples not updated with field definitions. Check model and retrieval versions separately. Preserve raw example outputs, not just manual scores whose basis cannot later be checked. Where one request uses several template fragments, record each version rather than only the top-level filename.
Keep changes and recovery tied to a specific failure
Use version control for templates and assembly instructions, referencing sensitive settings securely rather than publishing them. Explain the failure each change addresses and compare a fixed evaluation set. For regressions, restore the verified template combination instead of appending several untested modifications whose effects cannot be distinguished.
Verify with unseen examples and a recovery exercise
Release records should connect requests to template versions and comparison results. Retest original failures and held-out inputs rather than adapting only to known examples. Recovery should retrieve the actual old template and dependency instructions; a commit identifier without runtime assembly rules is not enough for reproducibility.
Pinning a prompt does not pin a hosted model behind a dynamic service or freeze a knowledge base. This page changes no production template or deployment. Exclude private data and real credentials from logs; publish redacted diagnostic examples rather than committing entire conversations unchanged.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.