What is a context window?

The input packet

A model API call receives a bounded sequence of tokens. In a chat or agent product, application code may assemble that sequence from system instructions, user messages, conversation history, tool schemas, retrieved documents, repository files, tool observations, and formatting markers.

flowchart LR
  I[Instructions] --> P[Context assembly]
  H[History] --> P
  T[Tool schemas and results] --> P
  R[Retrieved evidence] --> P
  P --> B{Token budget}
  B --> M[One model call]

The advertised context length is a capacity limit, usually covering input plus generated output according to the provider contract. The harness must reserve room for the response and decide what to omit, truncate, summarize, or retrieve again.

Available does not mean used well

Putting a fact inside the context makes it available to the computation; it does not guarantee that the final representation will use it correctly. Position, distractors, conflicting instructions, document order, input length, and task complexity can change behaviour. Long-context evaluations have found that models may retrieve a simple hidden item yet degrade on multi-step use or information placed in less favourable positions.

More context can therefore hurt: irrelevant material consumes tokens, increases latency, expands the injection surface, and can dilute important evidence. Good context engineering is selective and testable rather than maximal.

Compaction and durable memory

An agent run can outgrow the window. A harness may drop old turns, summarize them, keep structured state outside the transcript, or retrieve selected records later. Each technique is lossy unless the original evidence remains available.

A key–value cache speeds autoregressive inference by reusing internal attention computations for tokens already processed. It is not durable memory and does not decide which facts should survive a long task. Conversation history is also not memory unless the application stores and reintroduces it on a later call.

Context is neither memory nor truth

The context window is not the model's training corpus, parameter memory, database, authorization boundary, or source of truth. A document in context may be stale, malicious, irrelevant, or invisible to the current user under product policy.

A larger numeric limit is not proof of better long-context reasoning. Token counts are tokenizer-specific, and a context that fits technically may still fail behaviourally.

Evaluate context assembly

Record the exact assembled context or a privacy-safe structural trace: source, order, token count, truncation, trust class, and version for each segment. Test that protected data is filtered before context assembly and that lower-trust content cannot silently override higher-priority instructions.

Create cases that move the same evidence to the beginning, middle, and end; add realistic distractors; vary length; reorder documents; introduce conflicts; and require aggregation across multiple passages. Measure task success, citation correctness, latency, cost, and truncation—not only whether a hidden string can be repeated.

For compaction, freeze long task traces and verify which obligations, decisions, identifiers, permissions, and unresolved failures survive. A summary that reads well but drops one safety constraint is a failed state transition.