Prompt caching versus answer caching: reuse computation or reuse a result
Distinguish provider-side prompt caching from saved application answers, and explain how prefix matching, changing material and access conditions affect reuse and validation.
Choose an approach
Prompt caching reuses processing for part of the input while the model still generates an answer. An answer cache returns a previously saved result. For shared fixed instructions, check the provider’s prefix-cache conditions; for identical reusable queries, consider result caching. They can be combined, but hit conditions and invalidation rules must be explicit.
An identical prefix is different from an identical question
OpenAI’s prompt-caching guide concerns reusable input prefixes; actual hits follow service rules. The same question with changed preceding instructions should not be assumed to hit. A cache hit does not remove generation: output computation and results still belong to the current request.
Application answer caches save results and select them using a cache key. RFC 9111 covers response freshness and reuse conditions; model applications also need material and task versions. A key containing only question text can return an old conclusion after the source changes.
Track the two kinds of hits under matched conditions
Use requests sharing fixed material but asking different questions, inspect reported cache usage, then change the prefix and repeat. For answer caching, test identical questions and changed source versions separately. Record hits separately instead of attributing every shorter wait to one mechanism.
If users can access different material, confirm whether results may be shared between them. Use public samples to test access scope without including real user content in examples. A cached answer should retain the source version that produced it, rather than only final text.
Invalidation determines whether reuse remains useful
After fixed instructions change, recheck prompt caching. When material is withdrawn or rules change, invalidate affected answers. Save generation-version identifiers and define an explainable expiry period instead of indefinitely reusing all responses. Retention should reflect how the business material changes.
For costs and waiting time, measure hits and misses separately. Caching reduces repeated work; it cannot improve unsupported facts. This article checks caching mechanisms and response-reuse principles without measuring savings or guaranteeing any request will hit.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.