What is a KV cache?

Prefill once, decode step by step

Autoregressive generation has two visibly different phases. Prefill processes the input tokens and creates the initial attention state. Decode produces one new token at a time.

In causal self-attention, a new position may read earlier positions, but an earlier position cannot change after a later token appears. Each layer can therefore keep the earlier key and value vectors. On the next step it computes the new query, key, and value, appends the new key and value to the cache, and attends over the stored sequence.

flowchart LR
  P[Prompt tokens] --> F[Prefill all layers]
  F --> C[Keys and values per layer]
  C --> Q[New-token query]
  Q --> A[Attend to cached keys and values]
  A --> T[Choose next token]
  T --> K[Append its key and value]
  K --> C

Without that cache, generating token 1,001 after token 1,000 would repeat much of the work already done for positions 1 through 999. With it, the model still reads a growing set of cached values, but it avoids recreating them at every decode step.

The memory trade-off

The cache stores keys and values for every retained position, at every cached layer, for every active sequence. Its memory therefore grows with sequence length, batch size, layer count, vector width, and numeric precision. Long prompts and long outputs can make cache capacity—not model weights—the serving bottleneck.

Serving systems use different strategies around that pressure:

• dynamic caches allocate as sequences grow;
• static caches reserve a known maximum and can be easier to compile efficiently;
• quantized or offloaded caches reduce device memory at possible latency or accuracy cost;
• paged allocation reduces fragmentation when many differently sized requests share a server;
• prefix caching reuses the state for an identical trusted prefix across compatible requests.

Batch scheduling now matters. One request with a long output keeps cache pages and compute resources alive while other requests arrive and finish. Continuous batching can improve throughput, but latency, fairness, cancellation, and memory pressure must remain visible separately.

FlashAttention solves a related but different problem: it reorganizes exact attention computation to reduce expensive movement through the GPU memory hierarchy. A faster attention kernel is not the same thing as retaining past keys and values between decode steps, although serving stacks often use both.

Cache identity is part of correctness

A reusable prefix is more than matching text. Correct reuse depends on the exact model and adaptation, tokenizer and token IDs, positional scheme, attention implementation, relevant decoding context, and any other input that changes the cached tensors. A stale or mismatched cache can produce incorrect output without an obvious exception.

Cross-request reuse also creates an isolation boundary. Cache keys must include every identity and configuration dimension required by the product, and cached tensors from one private request must never become readable by another user through timing, lookup, or accidental handle reuse. Cancellation and failure must release or account for occupied pages.

A cache is not memory

A KV cache is not agent memory, conversation history, a vector database, or learned model knowledge. It is ephemeral numeric inference state derived from exact prior tokens. It does not make the context window larger; it makes repeated use of positions already inside that window cheaper.

It is also not a free speed switch. Caching consumes memory, introduces position and mask bookkeeping, complicates batching, and may be inappropriate during training. Prefix reuse cannot be assumed across model versions or differently tokenized text.

Verify equivalence, isolation, and invalidation

For model correctness, compare logits and generated tokens with caching enabled and disabled under deterministic decoding. Cover single-token and multi-token prefill, empty and maximum-length inputs, padding sides, attention masks, position IDs, sliding windows, cache truncation, and supported numeric precisions. Small numerical differences need declared tolerances; changed token choices need investigation.

For serving, test mixed prompt and output lengths under sustained concurrency. Measure time to first token, time per output token, throughput, peak cache memory, allocation failures, queue time, and fairness. Cancel requests at every phase and assert that their pages are reclaimed without affecting another sequence.

For prefix reuse, change one identity dimension at a time: token, system instruction, model revision, adapter, tenant, and authorization scope. Prove misses when reuse is unsafe and equivalent output when it is allowed. Inject evictions, stale handles, worker restart, and memory pressure. Finally, check observability: an operator should be able to distinguish prefill cost, decode cost, cache hit, cache miss, eviction, and capacity rejection without logging private prompt content.