Every lesson moves through a concise mechanism, experiment, written evidence brief, and checkpoint. Drafts stay in your browser and can be exported as Markdown; the complete theory chapter remains available afterward as optional reference.
10
lesson experiments
10
exportable worksheets
10
reference chapters
10
knowledge checks
Implementation field guide
Choose the smallest stack that makes cases, traces, graders, and decisions reproducible. The durable artifact is the evaluation design and its evidence—not a vendor dashboard.
Trace-linked datasets and experiment runs
Connect dataset items, runs, observations, and evaluator scores so a failed case remains inspectable.
Config-driven regression and red-team jobs
Run repeatable eval and adversarial suites locally or in CI, with targets and attack strategies under version control.
Reports, pass/fail suites, and production monitoring
Apply declared metrics and tests to offline batches or monitored production windows, then track their results over time.
Test cases, goldens, traces, and span-level checks
Express end-to-end cases and component checks in a pytest-style workflow, including metrics attached to traces and spans.
Model–dataset–metric benchmarks
Run model evaluation suites, retain sample-level results, and use supported bootstrap estimates for uncertainty.
Versioned scorers and trace feedback loops
Keep deterministic scorers and model judges attached to runs and traces, with explicit judge versions and feedback.
Canonical learner path
Each route keeps the mandatory learning loop concise, saves its worksheet independently, and loads the long-form chapter only when the learner opens the deep dive.
