Production systems · reviewed · reviewed Aug 31, 2026 · 4 min

How does prompt caching work?

A provider can reuse internal computation for an identical eligible prompt prefix from a recent request. Stable tools, policy, examples, or documents go first; changing conversation or user data goes later. The model still receives the logical input and generates a new response.

Prompt caching reuses recent prefix computation to reduce input cost or latency; it does not remember facts or reuse answers.

Interactive note 17

Move the first changed segment

Request B differs from request A starting at:

reusedTool definitions900 toy tokens
reusedSystem policy800 toy tokens
reusedStable examples1,200 toy tokens
recomputedNew user task350 toy tokens

2,900 reused · 350 recomputed

Fixed token counts illustrate prefix invalidation only. Actual eligibility, matching, retention, prices, and latency are provider- and version-specific.

Reuse the prefix, generate again

Long-running applications often send the same initial material on many requests: tool definitions, product policy, examples, a large document, or early conversation history. Processing that identical prefix repeatedly costs compute before the model can handle the new suffix.

A prompt cache lets the serving system reuse eligible internal work for a recent matching prefix.

Two request streams sharing a long blue prefix and diverging at short gray suffixes, with the second stream reusing stored prefix work.

The stable prefix can reuse serving computation; each request still supplies its own suffix and produces a new generation.

flowchart LR
  A[Stable tools + policy + examples] --> C[Cached prefix state]
  D1[Dynamic request A] --> G1[New generation A]
  C --> G1
  D2[Dynamic request B] --> G2[New generation B]
  C --> G2

The logical prompt is not shortened. The model still conditions on the prefix and the new suffix. Only part of serving computation and billing may be treated as cached under the provider's current contract.

Prefix order is the interface

Caching normally depends on exact or provider-defined matching from the beginning of the request to a cache boundary. A change near the front invalidates reuse for everything after it. Put stable, shared content first and rapidly changing content last.

A useful order for an agent request is often:

stable tool definitions and system policy;
stable examples or reference material;
growing but reusable conversation history;
the newest tool result, user request, time-sensitive fact, or run state.

Do not reorder content blindly for cache hits. Instruction precedence, recency, provider message rules, and task quality still matter. Measure the complete workload rather than optimizing a token counter in isolation.

Small changes can break a prefix: timestamps, randomly ordered tools, generated IDs, whitespace, serialization differences, or tool schemas whose descriptions change each request. Log prompt-version hashes and provider usage fields so a falling hit rate is diagnosable without storing sensitive prompt text unnecessarily.

Prompt cache, KV cache, memory, and response cache

These mechanisms answer different questions:

Prompt cache: can a later API request reuse computation for a matching input prefix?
Decode KV cache: can one generation reuse attention keys and values from its earlier token positions while producing the next token?
Agent memory: which durable external records should be stored and selected into a future context?
Response cache: can the application return a previously generated answer for an equivalent request without running the model again?

Prompt caching does not make the model learn a fact, persist canonical state, or return the same output. Sampling and model behaviour still produce a new completion. A response cache has a much stronger semantic invalidation problem because it reuses the answer itself.

Provider contracts are versioned

Providers differ in automatic versus explicit caching, minimum prompt size, eligible content, cache boundaries, lookback rules, retention, data controls, write and read prices, supported models, and reported usage fields. These details change.

Treat cache configuration as infrastructure tied to a provider, model, and API version. Do not hard-code one provider's thresholds or discount into a vendor-neutral architecture. Confirm current documentation before using cost estimates or retention claims in a production decision.

The first request may pay a cache-write premium or receive no benefit. A cache entry can expire before reuse. A high cache-hit ratio can still lose money when traffic is sparse, prefixes are small, or the application continually writes dynamic material into the cached region.

Measure task-level benefit

Track at least:

uncached, cache-write, and cache-read input tokens;
hit rate by prompt and tool-set version;
time to first token and end-to-end latency distributions;
total cost per request and per successful task;
output quality and task success under the same logical prompt;
invalidation causes and retention-related misses.

Test cold, warm, expired, partially matching, and concurrent traffic. Include growing conversations and tool outputs that change every turn. A recent cross-provider study found consistent cost reductions in its long-horizon workloads but more variable latency effects, including cases where naive dynamic caching added overhead. Treat those results as evidence for measurement, not a guaranteed percentage for another system.

The practical rule is: make the stable prefix genuinely stable, inspect provider-reported cached usage, and keep caching invisible to correctness. If turning the cache off changes the intended answer or permission behaviour, the application has coupled an optimization to its product contract.

Sources