Lesson 06 · Updates accumulate
Unfinished prompt
The animal did not cross the street because it was too …
01 · Smallest useful mechanism
Imagine every token position carrying a working page through the transformer. The page begins with that token’s initial vector. Each attention layer can add a note gathered from other positions, and each MLP can add a locally computed note. The page is the shared channel through which components communicate; it is not readable prose. Its size stays the same while its contents become increasingly contextual. For next-token prediction, the final page at the last prompt position goes to the language-model head. We can project earlier pages into vocabulary space to watch candidates rise and fall, but that projection is an interpretability probe. It helps form hypotheses about how a prediction develops; it is not a recording of private thoughts.
Every block adds updates to temporary token states that keep the same shape; those updates are not automatically evidence in the factual sense.
02 · Experiment
Calculate by hand · exact residual updates
The walkthrough is illustrative and internally consistent. Model-backed hidden-state inspection remains future research.
03 · Production-minded practice
Commit to a written claim and evidence plan. The Lab stores the draft locally and never sends it to a server.
Your learning artifact
Stored only in this browser. No account required; course reset does not delete it.
04 · Check your understanding
Next: The final state at “too” is ready to be compared with every vocabulary token.
Prompt · final position selected
Residual stream snapshot
Vocabulary lookup
layer 0 / 6
48 visible teaching dimensions stand in for thousands of coordinates. Position and color are visual encodings, not named human concepts.
Teaching logit lens
The last row still mostly represents the input token “too”.
An intermediate projection is a probe. It does not prove that a layer “believes” a word or that each coordinate has a stable human meaning.