Should Northstar Support Copilot ship?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Integrate evidence without flattening it into one score.

The dossier ties every claim to a versioned artifact, separates component and end-to-end results, reports unsupported claims and residual risk, and records the decision owner. A conditional canary names scope, gates, monitoring, rollback, and what new evidence would change the verdict.

Critical violations take precedence; missing evidence stays visible; only then can the candidate be called ready.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The defend the release decision workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose the evidence and rules that belong in the release-board decision.

Fixture northstar-release-v1.4

Preserve critical failuresShow the cross-account event outside every average.Report the Polish regressionKeep locale-slice results next to the aggregate improvement.State unsupported claimsName missing domain, accessibility, and long-horizon evidence.Define a constrained canaryOnly after remediation: scope, gates, monitoring, rollback, and owner.Ship on the improved averageLet higher quality and lower cost compensate for the privacy event.Hide raw evidencePublish only a green summary dashboard to reduce review friction.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
State release, constrained canary, or hold—and name the exact system version.
Evaluation design
Apply critical-gate precedence, then completeness, then quality and resource trade-offs.
Release evidence
Link every claim to the charter, cases, raw results, traces, grader study, limitations, monitoring, and rollback plan.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

Capstone decision rubric

Would this evidence support release?

Ready

Every material claim points to versioned evidence, all required slices and controls are covered, uncertainty and limitations are visible, and no critical violation remains.

Incomplete

The reasoning may be sound, but a required case slice, grader validation, trace, repeated trial, operational control, or decision owner is missing.

Unsafe

A privacy or authority violation occurred, a high-risk control failed open, or a critical failure was hidden inside an average or polished summary.

04 · Check your understanding

What is the correct decision for v1.4?

Theory notes and sources

What the experiment does not prove

A polished dashboard can hide a cross-account failure behind better average quality.

Common mistake: The evaluator should find enough evidence to justify release. The evaluator should make the decision evidence supports, including a hold or a narrower trial.

Transfer exercise: Defend a hold when v2 is more concise and cheaper but contains one unauthorized cross-account effect.

Model Cards for Model ReportingA primary source for intended-use documentation and disaggregated performance reporting.NIST AI Risk Management Framework 1.0A system-lifecycle framework for mapping context, measuring trustworthiness, and managing AI risk.Holistic Evaluation of Language ModelsA primary research framework connecting scenarios, adaptations, metrics, and transparent raw results.Promptfoo — CI/CD integrationOfficial implementation guidance for running eval and red-team jobs as regression gates with pipeline and commit metadata.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Change one assumption, version the dossier, and rerun the decision. Evaluation is a maintained argument, not a one-time ceremony.