Lesson 08 · Generation repeats
Training target revealed
The animal did not cross the street because it was too tired
01 · Smallest useful mechanism
Generation has two phases. During prefill, the model processes the prompt and creates attention keys and values for every prompt position and layer. During decode, it produces one token at a time. A new token still needs to attend to earlier context, but those earlier keys and values do not need to be recalculated. The KV cache keeps them available and appends one new pair after each generated token. This speeds up repeated generation, but the cache grows with active context and consumes memory for every request. It is a temporary working shelf, not learned knowledge. Some serving systems can reuse a compatible cached prefix across requests, but that is an additional optimization—not the model permanently remembering a conversation.
The KV cache stores temporary attention projections for the active sequence.
02 · Experiment
Calculate by hand · architecture calculator
Cache shapes and byte counts are calculated from declared architecture parameters; no hidden runtime tensors are invented.
03 · Production-minded practice
Commit to a written claim and evidence plan. The Lab stores the draft locally and never sends it to a server.
Your learning artifact
Stored only in this browser. No account required; course reset does not delete it.
04 · Check your understanding
Next: Inference now makes sense end to end. The remaining question is how training made “tired” receive a high score.
Start with the attention calculation
When a new token arrives, earlier keys and values are unchanged and reusable. Only the new position needs a query to read from that history.
Configuration
Memory per sequence
32 KV heads · Every query head stores its own K and V.
4.00 GB
100.0% of MHA
1 KV head · All query heads share one K/V head.
128 MB
3.1% of MHA
8 KV heads · Groups of query heads share K/V heads.
1.00 GB
25.0% of MHA
bytes = 2 × tokens × layers × KV heads × head dimension × bytes per element
The factor 2 stores both keys and values. Batch size is 1, cache metadata is excluded, and head dimension equals model dimension ÷ query heads.
