How does a language model choose the next token?

From scores to one token

For the current context, the model produces one score—often called a logit—for every token in its vocabulary. Softmax converts those scores into probabilities that sum to one. A decoder then applies a selection policy.

flowchart LR
  C[Current token context] --> M[Model logits]
  M --> S[Scale and softmax]
  S --> F[Optional top-k or top-p filter]
  F --> D[Sample or choose maximum]
  D --> T[Append selected token]
  T --> C

Greedy decoding always chooses the highest-probability token at the current step. Sampling draws according to the distribution, so a lower-probability candidate can sometimes be selected. That one choice changes the context for every later step.

Temperature, top-k, and top-p

Temperature rescales logits before softmax. Lower values concentrate probability on the strongest candidates; higher values flatten the distribution. Temperature does not add knowledge or “creativity” to the model. It changes how aggressively the decoder explores alternatives already present in the distribution.

Top-k keeps only the k highest-scoring tokens. Top-p, or nucleus sampling, keeps the smallest set whose cumulative probability reaches a threshold such as 0.9. Both remove a tail of candidates before drawing. Their effects depend on the model, prompt, step, and implementation order, so copied settings are not universal quality presets.

An output also needs stopping rules: end-of-sequence tokens, application stop strings, maximum new tokens, wall-clock budgets, or protocol-specific terminal states. Streaming changes when partial text becomes visible, not how the model learned.

Reproducibility has boundaries

A random seed can make sampling repeatable only when the entire execution path is compatible: model weights, tokenizer, prompt bytes, decoding implementation, numerical kernels, hardware, batching, and concurrency can all matter. A temperature of zero is often treated as deterministic, but serving systems may still produce differences when ties, floating-point order, or backend versions change.

Store the complete generation configuration with evaluation evidence. A model name without its decoder and stop policy is not a complete system version.

Sampling controls are not reasoning controls

Sampling is not reasoning, factual verification, or confidence calibration. The probability of a token is conditional on preceding tokens under the model; it is not the probability that the resulting claim is true.

Greedy output is not automatically safer or more correct, and a higher temperature is not automatically more creative. Beam search, penalties, constrained decoding, and rejection or reranking procedures optimize different objectives and can introduce their own failure modes.

Tune decoding against product outcomes

Test generation as a versioned component. Freeze prompts, model and tokenizer versions, decoding settings, stop rules, maximum length, and runtime. For sampled behaviour, run multiple seeds and report distributions rather than one attractive example.

Assert deterministic protocol properties directly: valid schemas, allowed tokens, termination, maximum output, UTF-8 handling, tool-call structure, and streaming assembly. Include cases where a stop string spans chunks or token boundaries and where the model emits an end token immediately.

For task quality, compare plausible decoding configurations on a held-out set using outcome metrics appropriate to the task. Track repetition, truncation, refusal, unsupported claims, latency, and token cost. When a decoder changes, keep the model fixed first so its effect remains identifiable.