What evidence keeps a release claim valid after launch?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Treat monitoring as continuing evaluation, not a drift alarm.

Shadow and canary stages limit exposure while testing real task mix. Monitor system versions, policy changes, human edits, overrides, escalations, outcome proxies, latency, cost, incidents, and slices. Define rollback and reevaluation triggers before launch.

Production signals are selected and delayed; they need sampling, denominators, privacy controls, and explicit response rules.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The monitor after launch workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose the signals and controls needed for a constrained canary.

Fixture northstar-monitoring-v1

Log system versionsPin prompt, model, retrieval, tools, policies, and rollout cohort.Sample outcomes and effectsMeasure edits, overrides, escalations, incidents, and final states.Predeclare rollback triggersCritical privacy or authority events stop the canary.Minimize and govern logsUse privacy-safe sampling, retention, access, and redaction.Use thumbs-up aloneTreat voluntary positive feedback as a representative outcome sample.Change prompts without reevaluationCall prompt edits operational details outside the evaluation boundary.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
State what the canary is allowed to establish and which residual risks remain.
Evaluation design
Define cohorts, indicators, sampling, privacy controls, alerts, appeals, incident flow, and rollback.
Release evidence
Attach denominators, versions, change log, alert rationale, and the owner for every response.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Which change should trigger reevaluation?

Theory notes and sources

What the experiment does not prove

Thumbs-up feedback excludes silent failures, abandoned conversations, and users who cannot report harm.

Common mistake: Offline accuracy remains valid until a drift detector fires. Behavior can change through prompts, tools, policies, users, or workflow even when input statistics look stable.

Transfer exercise: Define one rollback trigger for privacy, one for quality, and one change that forces full reevaluation.

NIST AI RMF Playbook — MeasureGuidance on repeatable evaluation, production monitoring, documentation, and measurement limits.MandolinePrimary research on evaluating model performance under distribution shift using user-specified slices.Model Cards for Model ReportingA primary source for intended-use documentation and disaggregated performance reporting.Evidently — Production monitoring overviewOfficial implementation documentation for batch and production evaluation pipelines; sampling and response rules still require product-specific design.Langfuse — Code evaluatorsOfficial guidance for applying deterministic evaluators to controlled experiments or selected production observations.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: The final lesson assembles every claim, result, limitation, and operational control into a release decision that another reviewer can audit.