Is the apparent improvement real, useful, and safe?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Report the evaluation unit, variation, and failure severity.

Choose whether the unit is a turn, conversation, case, or final state. Compare candidates on the same cases, repeat stochastic runs, report counts and uncertainty, inspect slices, and preserve severe failure categories. Latency and cost remain separate decision dimensions.

“Nine out of ten passed” is unsafe evidence when the tenth run crossed an authority boundary.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The report uncertainty workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Paired evaluation run

Trigger the run. Inspect the failures. Defend the verdict.

This browser-only simulation follows a production eval lifecycle with pinned cases and captured outputs. The calculations are real and reproducible; the model responses are teaching fixtures.

1
Load dataset
8 versioned cases
2
Run systems
16 paired outputs
3
Apply evaluators
code + model rubrics
4
Aggregate
intervals + slices
5
Apply gates
critical first

The result does not exist yet.

Review the manifest, choose a candidate, and trigger the same eight paired cases against both versions.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
State the candidate improvement and the conditions under which it is expected.
Evaluation design
Use paired cases, repeated trials, intervals, slices, severity, latency, and cost.
Release evidence
Export raw counts, configuration, code version, case IDs, exclusions, and all critical failures.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What must accompany a 94% success rate?

Theory notes and sources

What the experiment does not prove

Opaque composite scores can improve even while a critical slice regresses.

Common mistake: Statistical significance implies practical or safety significance. An interval describes uncertainty under assumptions; product value and unacceptable risk still require explicit thresholds and severity reasoning.

Transfer exercise: Compare two versions whose mean quality differs slightly but whose critical-failure counts differ from zero to one.

Better than AveragePrimary research showing why aggregate NLP results can obscure meaningful subgroup behavior.Confidence intervals in NLP evaluationPrimary guidance on uncertainty reporting for NLP evaluation results.Holistic Evaluation of Language ModelsA primary research framework connecting scenarios, adaptations, metrics, and transparent raw results.Hugging Face Evaluate — EvaluatorOfficial documentation for model–dataset–metric evaluation, multi-task suites, and bootstrap estimates for supported tasks.Hugging Face — LightEvalOfficial toolkit guidance for reproducible model benchmarks across backends with custom tasks, metrics, and sample-level results.Langfuse — Datasets and experimentsOfficial implementation documentation for linking dataset items, experiment runs, traces, observations, and scores; useful infrastructure, not independent validity evidence.Promptfoo — Results export and analysisOfficial implementation documentation for per-case results and machine-readable artifacts. It warns that config sanitization is best-effort and exports can retain environment or sensitive evaluation data, so review and redact before sharing.MLflow — GenAI evaluation quickstartOfficial implementation workflow connecting a dataset, prediction function, scorers, aggregate metrics, per-record results, and run comparison.Evidently — Evaluation testsOfficial implementation guidance for dataset-level pass/fail conditions, reference comparisons, custom thresholds, and critical versus warning results.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Offline evidence freezes one environment; the next lesson defines how launch stages, monitoring, feedback, and change triggers keep the claim alive.