Which cases deserve space in a limited evaluation budget?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Represent the use case and its failure surface.

A useful case set combines expected traffic with deliberately oversampled boundaries and severe risks. It separates development cases from held-out cases, records provenance, and reports which populations, languages, tasks, and environment failures remain uncovered.

More examples do not repair a test set that omits the conditions where errors matter most.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The design representative cases workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose the evidence that makes an eight-case pilot set decision-useful rather than merely large.

Fixture northstar-cases-v1

Oversample critical slicesInclude privacy, authorization, and stale-policy cases even when rare.Include both supported localesExercise English and Polish cases with equivalent acceptance rules.Inject an environment failureInclude a retrieval timeout or unknown tool outcome.Reserve held-out casesDo not tune prompts or graders on the final release slice.Duplicate the easiest caseUse repeated golden-path examples to increase the sample count.Copy production tickets verbatimUse raw customer data because realism is more important than consent and minimization.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
Name the user and risk conditions the case set represents.
Evaluation design
Create a capability × slice × test-type matrix and a held-out split.
Release evidence
Attach case IDs, provenance, severity, locale, expected evidence, and known omissions.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What should a case manifest disclose?

Theory notes and sources

What the experiment does not prove

Aggregate success can look excellent while multilingual, ambiguous, or authorization-sensitive slices fail systematically.

Common mistake: A large synthetic golden-path set is representative by default. Representativeness depends on the intended population and risk model; synthetic data also needs provenance and realism checks.

Transfer exercise: Spend eight case slots across common billing questions, Polish input, ambiguity, stale policy, authorization failure, and tool timeout.

CheckListA primary paper on capability-oriented behavioral testing for NLP systems.Datasheets for DatasetsA primary proposal for documenting dataset motivation, composition, collection, and recommended use.Holistic Evaluation of Language ModelsA primary research framework connecting scenarios, adaptations, metrics, and transparent raw results.Langfuse — Datasets and experimentsOfficial implementation documentation for linking dataset items, experiment runs, traces, observations, and scores; useful infrastructure, not independent validity evidence.DeepEval — Evaluation datasetsOfficial documentation for versionable single-turn and conversational goldens, local dataset files, and repeatable evaluation inputs.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Once cases are chosen, we need oracles and traces that say what success means without treating fluent prose as hidden proof.