Did the system retrieve and use authoritative evidence?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Grade retrieval and generation as separate components.

A groundedness evaluation starts with gold evidence or clearly defined authority rules. It measures whether the right evidence was retrievable, whether the system selected it, and whether each material claim follows from it. Unsupported questions should produce calibrated abstention or escalation.

Citation presence is not evidence quality; authority, freshness, relevance, and entailment all matter.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The audit grounded claims workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose the checks needed before calling a policy answer grounded.

Fixture northstar-grounding-v1

Check document authorityConfirm the source is the canonical policy for this case.Check policy versionReject superseded or future-dated policy snapshots.Link claims to exact spansVerify each material claim follows from the cited passage.Test unsupported questionsRequire uncertainty or escalation when evidence is missing.Score citation count onlyPass any answer containing at least two links.Reuse another customer’s invoiceTreat a numerically similar invoice as acceptable evidence.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
Separate the retrieval-quality claim from the answer-faithfulness claim.
Evaluation design
Build a pack with gold passages, stale distractors, conflicts, and unsupported questions.
Release evidence
Export a claim ledger with exact spans, versions, dispositions, and required repairs.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Why can a faithful answer still be wrong?

Theory notes and sources

What the experiment does not prove

A perfectly faithful answer to an obsolete refund policy is still wrong.

Common mistake: If every sentence has a citation, the response is grounded. A citation can be stale, irrelevant, from another customer, or fail to entail the claim.

Transfer exercise: Label each material claim supported, contradicted, stale, irrelevant, or unverifiable.

RAGASA primary paper on reference-free evaluation signals for retrieval-augmented generation.ARESA primary paper on automated evaluation of retrieval-augmented systems.NIST AI RMF Playbook — MeasureGuidance on repeatable evaluation, production monitoring, documentation, and measurement limits.NIST TREC — Relevance judgments and trec_evalAuthoritative test-collection guidance on matching topics, document collections, qrels, and retrieval measures; it also documents the limits of pooled relevance judgments.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Evidence-backed words are only half the product; the next lesson evaluates whether tool proposals and real state changes obey policy.