Lesson 02 · Text becomes tokens
Unfinished prompt
The animal did not cross the street because it was too …
01 · Smallest useful mechanism
Before a model can process text, a tokenizer turns it into vocabulary entries called tokens. A token may be a whole word, part of a word, punctuation, whitespace, or part of an emoji. Each token receives a numeric ID used as a lookup address; the number itself carries no meaning. Boundaries come from the tokenizer vocabulary and rules, not grammar. A familiar word may fit in one token while an unusual name, code identifier, or underrepresented language needs several. Tokenization happens before transformer computation. It determines the sequence the model actually sees and affects how much text fits in the context window, how usage is counted, and which pieces later layers must combine.
The model receives reusable token pieces and IDs—not words.
02 · Experiment
Calculate by hand · compare with local AI
Guided rules are illustrative. The optional pinned Qwen3.6 tokenizer is the vocabulary used by Bonsai; it is not the GPT-2 course trace.
03 · Production-minded practice
Commit to a written claim and evidence plan. The Lab stores the draft locally and never sends it to a server.
Your learning artifact
Stored only in this browser. No account required; course reset does not delete it.
04 · Check your understanding
Next: IDs are still only addresses. The next step looks up a learned vector for each address and adds position.
Tokenizer output
Guided teaching tokenizer
Illustrative subword rules—not Bonsai output
Selected token
Decoded text
The
Token ID
5636
Characters / bytes
3 / 3
Model vocabulary form
The
UTF-8 84 · 104 · 101
Unicode U+0054 · U+0068 · U+0065
Why it matters
The model never receives words directly. It receives IDs, and every ID consumes one position in the context window.
