Prompt caching: when it saves AI latency and cost

What prompt caching does

Prompt caching lets a provider reuse work from a repeated prompt prefix instead of processing the same long beginning from scratch every time. OpenAI's prompt caching guide explains prefix reuse, cache-related usage fields and supported-model behavior. Caching reuses prior input processing; the model still generates a new response. If every request starts with the same long policy, schema or tool description, caching may help. If every request is short or completely different, there is little to reuse.

Design for stable prefixes

Put stable content first: system instructions, developer instructions, durable policy text, schema definitions and fixed examples. Put variable content later: user question, retrieved snippets, current date, random IDs and conversation-specific notes. Even tiny changes near the beginning can reduce reuse. If your app injects a timestamp into the first line, you may accidentally make every prefix unique. Use stable version labels instead, such as policy_version: 2026-10-04, and put per-request facts after the cacheable block.

Good candidates

• Structured extraction: a long schema and instructions stay the same while documents change.
• Customer support drafting: policy rules stay stable while ticket text changes.
• Code review assistant: repository rules and rubric stay stable while diffs change.
• Long style guide: editorial guidance stays stable while the topic changes.

OpenAI's pricing page separates pricing categories, so cost planning should check whether cached input has different economics for the selected model. Avoid copying exact rates into code. Read the current page when building or reviewing estimates.

Bad candidates

Prompt caching is weak when requests are brief, the application rewrites the prompt every time, or retrieval always appears before stable instructions. It also will not fix a bloated prompt. If the stable prefix contains irrelevant examples, outdated policy or too many tool descriptions, caching may make waste cheaper but still harms quality. First remove unused context. Then arrange what remains for reuse.

Concrete example

A support tool has a 2,000-token policy block, a 600-token output schema and a 400-token ticket. Bad order: ticket first, current timestamp second, policy third. Good order: stable system instruction, policy block, schema, then ticket. If the policy changes weekly, version it and expect cache behavior to reset around the change. If the ticket includes retrieved docs, keep them after the stable prefix. Track input tokens, cached input tokens, latency and final answer quality during testing. A cache hit that produces poor answers is not a win.

Prompt caching versus RAG

Prompt caching and RAG solve different problems. RAG keeps prompts smaller by retrieving relevant documents. Prompt caching makes repeated prompt prefixes cheaper or faster when supported. A handbook assistant might use both: cached instructions and answer format first, retrieved handbook chunks later. If the handbook itself is huge and changes often, do not paste all of it just because caching exists. Retrieval and good chunking are still the cleaner architecture.

Problems to check

• Unstable prefix: dates, UUIDs or user-specific text appear before reusable content.
• Hidden prompt builder changes: code reorders fields and cache effectiveness drops.
• Quality ignored: teams celebrate lower cost while stale instructions degrade answers.
• Throughput surprise: cached tokens may still matter for rate limits, depending on provider rules.
• No measurement: nobody records cached-token usage, so savings remain guessed.

Next steps

Inspect one production prompt. Mark each paragraph stable or variable. Move stable content to the front, remove anything unused and add a version label for major prompt blocks. Then run a small load test with repeated requests and record token usage fields, latency and quality. Prompt caching is most useful when it rewards an already tidy prompt design.

Architecture note

If repeated context is actually a large knowledge base, consider retrieval first. Anthropic's contextual retrieval article is a useful reminder that retrieval exists to choose relevant information at runtime. Cache stable instructions; retrieve changing evidence. That split keeps both quality and cost easier to reason about.

Measure before assuming

Cache behavior should appear in usage data, not just architecture diagrams. Add a dashboard or log sample that shows total input, cached input, latency and model. If a deployment changes prompt order, the dashboard should make the regression visible quickly.

Check the reported reuse

For OpenAI Responses requests, inspect usage.input_tokens_details.cached_tokens, as documented in the prompt cache diagnostics guide. Compare repeated requests with the same prefix against a deliberate prefix change. Record total input, reused input and time to first output. Check the selected model's minimum cacheable length, supported settings and billing categories before expecting savings. A stable prompt is a candidate for reuse, not a guarantee of a hit on every request. Keep cache writes and cache reads separate in the cost worksheet when the model prices them differently.

Sources and further reading

Documentation checked October 4, 2026. Product features and availability can change.

• OpenAI prompt caching guide
• OpenAI API pricing
• Anthropic contextual retrieval
• OpenAI prompt cache diagnostics