How do we know the model learned a pattern instead of the training set?

examplepredictionlossgradientchanged parameters

01 · Smallest useful mechanism

Learning succeeds only when a fitted rule transfers to relevant unseen data.

A simple model may underfit and miss real structure. A flexible model may fit every training example, including noise. Held-out data tests the claim that the learned parameters capture something reusable. Modern over-parameterized networks complicate the classical U-shaped story, so capacity alone cannot certify or condemn a model.

Training error usually falls with model capacity; validation error reveals whether the model found a transferable pattern, an oversimplification, or a memorized shortcut.

02 · Experiment

Test the prediction

Illustrative risk curves · compare claims, not benchmark scores

The curves are deliberately shaped teaching data for underfitting, balance, and memorization. They are not measured results from a named architecture.

Capacity experiment

Fit more. Then check what transfers.

Model complexity4/10Training noise35%

Best held-out evidence

This region currently balances fit and transfer under the illustrative split.

Illustrative risk curves

Same split · increasing capacity

— training— validation

Training error: 17.8%
Validation error: 12.4%

These curves are teaching data. A real claim requires measured results, a versioned split, uncertainty, and checks that match the intended deployment.

03 · Reason from evidence

A model scores 99% on validation and 73% after release. An audit also finds near-duplicate records on both sides of the original split.

Your brief: Rewrite the original performance claim and propose an evaluation that separates leakage from genuine deployment shift.

Evidence your answer should use

A split grouped by the real unit of independence
Duplicate and provenance analysis across all partitions
Metrics with uncertainty on a frozen, deployment-matched test set
Subgroup and time-window results tied to the model and data version

There is no reveal button here. A strong response names the claim, the evidence that would test it, and what the evidence cannot establish.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

Capstone decision rubric

Is the learning claim ready to leave the notebook?

Ready

The response links the intended deployment claim to an independent split, uncertainty, subgroup or shift analysis, and a named acceptance threshold.

Incomplete

The response reports training fit or one aggregate validation score without checking leakage, independence, calibration, or deployment match.

Unsafe

The response treats a high score as proof of safety or authorizes deployment without failure costs, monitoring, and a rollback condition.

04 · Check your understanding

What does zero training error establish by itself?

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Continue into the LLM course to watch these same operations scale into embeddings, attention, next-token loss, and transformer training.