What happens to data before model training?

A dataset is the output of a pipeline

Raw data may begin as web pages, books, code repositories, licensed collections, product events, conversations, images, or expert demonstrations. Training normally consumes none of those sources directly. A sequence of programs and decisions turns them into model-ready examples.

flowchart LR
  S[Versioned sources] --> R[Rights and provenance]
  R --> P[Parse and normalize]
  P --> F[Filter and deduplicate]
  F --> E[Construct examples]
  E --> D[Split and decontaminate]
  D --> M[Sample a mixture]
  M --> T[Tokenize and shard]
  T --> X[Frozen training manifest]

Each arrow changes the effective dataset. Removing boilerplate changes which language remains. Deduplicating repeated code changes how often that pattern affects gradients. A quality filter may remove spam while also suppressing dialects or minority topics. The model learns from the resulting distribution, not from an abstract idea of “the internet.”

Provenance begins before parsing

For each source, record where it came from, when it was captured, which version was used, who supplied it, what licence or product policy applies, and which deletion or retention rules follow it. A URL alone is weak provenance: pages change, repositories move, and generated archives can contain material from several origins.

Keep source identifiers through transformations. If a record is normalized, split into chunks, filtered, or joined with metadata, lineage should still answer which original records contributed to the final example. That enables audits, correction, removal, and analysis of unexpected behaviour.

Access to data does not establish permission to train on it. Legal rights, user expectations, privacy, security, and product policy are separate questions that must be resolved for the intended use and jurisdiction.

Cleaning is a distribution decision

Parsing extracts the useful representation from source formats. Normalization may repair encoding, standardize whitespace, remove markup, identify languages, or separate code from generated files. Filtering may remove low-quality, unsafe, private, corrupted, or policy-disallowed material.

Deduplication operates at several levels:

• exact duplicates such as repeated files or mirrored pages;
• near duplicates with small formatting or version differences;
• template-heavy records whose boilerplate overwhelms their unique content;
• semantic overlap that is harder to define and more expensive to detect.

Duplicate removal is not just storage optimization. Frequent examples receive more optimization weight. Leaving one benchmark answer copied across thousands of pages can distort both training and later evaluation.

Every filter has false positives and false negatives. Record its version, thresholds, reasons, and before/after counts. Review removal rates by source, language, domain, time period, and other relevant slices so a global “quality” score does not silently erase useful coverage.

Examples and mixtures define the learning signal

A model trains on examples, not documents. A language-model example may be a bounded token sequence assembled from one or more documents. Supervised fine-tuning may pair an instruction with a desired response. Preference training may join a prompt, candidate outputs, a ranking, and an annotation rubric.

The data mixture determines how frequently categories appear during optimization. Teams can sample sources in proportion to raw size, cap dominant sources, upweight scarce domains, or schedule different mixtures over time. Those choices change what the loss rewards even when every underlying record stays the same.

Tokenization adds another contract. Maximum sequence length, truncation direction, special tokens, document boundaries, packing, and loss masks determine which parts of an example contribute to learning. A correct source record can become a broken training example if an answer is truncated or the mask trains on padding and hidden metadata.

Splits protect the question you want to answer

Training, validation, and evaluation splits must be separated by the unit that could leak. Random rows are insufficient when rows from the same user, document, repository, conversation, or time period are near duplicates.

For a claim about future traffic, use a temporal boundary. For a claim about unseen repositories, group by repository. For a claim about generalising to new users, separate users. Search the training candidate pool for exact and approximate overlap with protected benchmarks before training, and keep an exposure history for later audits.

Contamination is not only a copied answer. A benchmark may leak through paraphrases, solution explanations, test fixtures, metadata, or repeatedly discussed public results. The correct response is to document likely exposure and interpret the score accordingly, not to pretend that one detector proves purity.

The manifest makes a run reproducible

A durable training manifest should identify source snapshots, transformation code, filter versions, exclusion lists, deduplication configuration, split rules, mixture weights, tokenizer, sequence construction, shard checksums, and aggregate counts. It should also name known limitations and data that could not be inspected fully.

When a source must be removed, lineage should reveal affected examples, shards, checkpoints, indexes, and derived datasets. Deleting the raw file alone does not undo copies already embedded in later artifacts, and removing an example from a future run does not remove its influence from an existing checkpoint.

The practical teaching boundary is simple: data curation defines the experience from which optimization learns; model training defines how that experience changes parameters. Neither is meaningful without the other.