What decision should this evaluation support?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

Prior knowledge: Basic AI product and testing concepts help. The agent course is useful context, but no statistics beyond rates and intervals is assumed.
Study time: 3–4 hours guided · about 9 hours of optional deep dives
By the end: Design an evaluation that supports a real release decision, exposes severe failures, calibrates subjective judges, and remains valid after launch.

01 · Smallest useful mechanism

Name the system, conditions, and decision first.

The same model can behave differently when retrieval, tools, prompts, approvals, or users change. A useful charter identifies the complete system under test, the people affected, the intended conditions, prohibited outcomes, resource limits, and who owns the decision.

A benchmark score is evidence about a defined setup; it is not a release decision for an undefined product.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The define the decision workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Select the statements that belong in the release contract. Commit before the reference classification appears.

Fixture northstar-contract-v1

Evaluate the complete copilotPin model, prompt, retrieval snapshot, tools, approval flow, and support-agent role.Zero cross-customer disclosureAny disclosure of another customer’s data blocks release.No unapproved mutationRefunds, cancellations, and address changes require exact current approval.Set latency and cost budgetsMeasure p95 latency and cost per resolved case without weakening safety gates.Let tone offset a privacy failureA sufficiently friendly response can compensate for one cross-account disclosure.Evaluate every support taskClaim fitness for all possible requests, including tasks outside intended use.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
State the concrete release or design decision and the exact system version it covers.
Evaluation design
Define intended use, affected users, system boundary, non-goals, benefit claims, hard gates, budgets, and decision owner.
Release evidence
Show how every gate maps to an observable trace, final state, human judgment, or production indicator.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Which statement is a defensible release gate?

Theory notes and sources

What the experiment does not prove

Without a predeclared contract, teams can move thresholds after seeing results or optimize a component that is not the product bottleneck.

Common mistake: A strong public benchmark score proves the support copilot is ready. Public benchmarks can inform model selection, but product fitness requires scenario-specific system evidence and explicit risk gates.

Transfer exercise: Rewrite “the copilot should be good” as one measurable benefit claim, one hard safety gate, and one non-goal.

NIST AI Risk Management Framework 1.0A system-lifecycle framework for mapping context, measuring trustworthiness, and managing AI risk.Holistic Evaluation of Language ModelsA primary research framework connecting scenarios, adaptations, metrics, and transparent raw results.Model Cards for Model ReportingA primary source for intended-use documentation and disaggregated performance reporting.Evidently — LLM evaluation guideVendor-authored practical guidance distinguishing model benchmarks from product evaluation and offline experiments from ongoing monitoring.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: A precise contract tells us which cases the evaluation set must represent—and which high-severity failures cannot be sampled away.