What are tokens and tokenization?

The encoding boundary

A language model does not receive words directly. A tokenizer applies a defined segmentation procedure and whatever normalization its contract declares—possibly none—then looks up the resulting pieces in a fixed vocabulary and emits integer IDs. The model uses each ID to select an embedding vector.

Token pieces have unequal width because the vocabulary reuses frequent strings and decomposes unfamiliar ones; the sequence order remains explicit.

flowchart LR
  T[Raw Unicode text] --> N[Normalization rules]
  N --> S[Subword segmentation]
  S --> P[Token pieces]
  P --> I[Vocabulary IDs]
  I --> E[Embedding vectors]

A token may be a whole common word, part of a word, punctuation, whitespace combined with text, a byte sequence, or a special control symbol. Token boundaries are therefore not linguistic truths. They are decisions made by a particular tokenizer trained or constructed for a particular vocabulary.

Why models use subwords

A vocabulary containing every possible word would be enormous and would still fail on new names, spelling variants, code, and productive word formation. A character-only vocabulary avoids unknown words but creates much longer sequences. Subword methods occupy the middle: frequent strings can become one token, while rare strings are composed from smaller pieces.

Byte-pair encoding repeatedly merges frequent adjacent units. Unigram tokenization begins with many candidate pieces and selects segmentations under a probabilistic model. Byte fallback can represent otherwise unknown text as bytes. Different methods and training corpora produce different boundaries even for the same visible string.

The trade-off is visible in sequence length. A common English phrase may use few tokens, while an uncommon language, identifier, or whitespace-heavy code sample may use many. Since attention and context limits operate on tokens, not characters, this affects capacity and latency as well as billed usage in token-priced APIs.

The tokenizer is part of the model version

Token IDs have meaning only relative to one vocabulary. ID 42 from one tokenizer is not interchangeable with ID 42 from another. Changing normalization, merge rules, special tokens, or vocabulary ordering changes the input seen by the model.

Training and inference must use compatible tokenization. A model trained with one mapping and served with another receives the wrong embedding rows even if the decoded pieces look plausible. Store the tokenizer artifact, configuration, special-token policy, and hash alongside the model checkpoint.

Special tokens need explicit handling. Beginning, end, padding, role, separator, image, and tool markers may be inserted by the application or a chat template. User text that resembles a special marker should not silently gain control authority.

Observable consequences

Tokenization can split a number, URL, emoji, accented string, or source-code identifier in surprising ways. Small edits can change several downstream IDs. Unicode normalization can make visually similar strings equivalent or distinct. Leading spaces often affect the selected piece.

These facts explain why character counts are unreliable proxies for context usage. They also create security and evaluation cases: invisible characters, mixed scripts, unusual whitespace, very long byte sequences, and language-specific fragmentation can change cost or bypass naive string checks.

Tokens are not words

A token is not always a word, syllable, character, or byte. A tokenizer is not the same as the model: it encodes and decodes symbols but does not predict the next token. A large vocabulary does not automatically produce better language understanding; it trades vocabulary size against sequence length and statistical coverage.

Decoding is not guaranteed to reconstruct an arbitrary pre-normalized string byte-for-byte unless the tokenizer contract promises it. Some pipelines deliberately normalize text before segmentation.

Validate tokenizer compatibility and budgets

Freeze a tokenizer version and build golden cases that assert both pieces and IDs for ordinary prose, every supported language, whitespace, punctuation, numbers, URLs, emoji, mixed scripts, invalid or unusual bytes, code, and all special tokens. Test encode–decode properties according to the declared normalization contract.

Measure token counts by language and important input type, not only on English averages. Set bounds for maximum input expansion and reject or truncate at an application-owned boundary. Verify that truncation preserves required instructions and does not split application protocols incorrectly.

At model packaging time, prove that the checkpoint and tokenizer hashes are compatible. At API boundaries, test the exact chat or tool template that inserts control tokens. In security tests, include confusable characters, invisible separators, repeated whitespace, and user text resembling control syntax.