What happens after pretraining?

Pretraining creates a base model

During language-model pretraining, a model learns to predict tokens across a broad dataset. That objective can produce extensive capabilities, but it does not directly specify how a helpful assistant should interpret an instruction, refuse a request, format a tool call, or choose between two plausible answers.

Post-training applies additional learning signals:

flowchart LR
  P[Pretrained base model] --> S[Supervised demonstrations]
  S --> I[Instruction-tuned model]
  I --> C[Preference comparisons]
  C --> O[Preference optimization]
  O --> R[Candidate release model]

Supervised fine-tuning trains on input–desired-output examples. Preference methods train from comparisons, rankings, critiques, rewards, or other signals about which response is better under a rubric. Different projects use different stage orderings and algorithms.

The classic RLHF explanation uses three broad stages: begin with a pretrained model, train a reward model from ranked outputs, then optimize the language model against that learned reward while limiting excessive drift. This is one influential pipeline, not a definition of all modern post-training; direct preference and other optimization methods can use different components.

Full updates and adapters

Full fine-tuning updates all or most model parameters. Parameter-efficient methods update a smaller set. LoRA, for example, freezes base weights and trains low-rank matrices that modify selected layer computations. This reduces training and storage cost but does not make data quality, evaluation, or version compatibility optional.

An adapter belongs to a specific base model architecture and checkpoint. Merging, stacking, quantizing, or serving adapters changes the release artifact and must be validated as such.

Preference data encodes a specification

A preferred response reflects the annotator instructions, examples, available candidates, user population, language, and product values. No dataset represents universal human preference. Disagreement and ambiguity are important evidence, not merely label noise to hide.

Optimization can exploit gaps in the preference signal. A model may learn verbosity, confident style, rubric keywords, or safe-looking refusals without improving the intended underlying capability. Behaviour can also regress on tasks not represented in the adaptation data.

Weight changes do not replace system controls

Fine-tuning is not a reliable way to load frequently changing facts; retrieval or ordinary data systems are usually easier to update and cite. It is not the same as prompting, which changes current context without modifying weights.

Post-training does not guarantee truthfulness, harmlessness, obedience, or product safety. It changes a statistical policy under measured conditions. Permissions, validation, evidence, and consequence management remain application responsibilities.

Evaluate adaptation and regressions

Freeze the base checkpoint, tokenizer, training data, sampling process, code, optimizer, seeds, and resulting artifact. Validate example formatting, masks, label construction, preference-pair direction, data leakage, duplication, and train/evaluation separation.

Compare the adapted model with the base on both target behaviour and regression suites. Slice by task, language, user group, safety category, input length, and domain. Use blinded human evaluation where judgement is essential and measure annotator agreement rather than collapsing it silently.

Probe for reward or rubric exploitation: unnecessary length, style mimicry, sycophancy, over-refusal, fabricated confidence, and brittle trigger phrases. A successful post-training run improves declared behaviours within known limits; it does not erase the need to evaluate the complete deployed system.