Which evidence shows that a RAG change is genuinely better?

Versioned evidence corpus · deterministic retrieval fixtures

Lighthouse Policy Evidence Assistant. A fictional support assistant answers questions from a versioned product-policy corpus. It must retrieve current, authorized evidence, cite exact source spans, and abstain when evidence is missing or conflicting.

01 · Smallest useful mechanism

Evaluate every boundary and preserve sample-level evidence.

Retrieval metrics answer whether expected evidence was found and ranked. Context metrics inspect what generation actually received. Grounding and answer metrics test support and usefulness. Slices and critical cases keep aggregate gains honest.

Measure candidate recall before judging prose produced from missing evidence.

02 · Experiment

Test the prediction

Deterministic retrieval fixture

This workbench uses inspectable ranking and evidence fixtures and injected failures. It does not make live calls or execute external effects.

Retrieval workbench · lighthouse-eval-v6

Choose the release analysis for a candidate with better averages and one severe retrieval leak.

Injected failure

Recall@5 rises from 0.78 to 0.86, but one cross-tenant chunk is selected and Polish citation correctness falls by 18 points.

Choose the implementation

Release on aggregate improvementAverage retrieval and answer scores are higher.Ask one judge for the winnerGive final answers to a model judge and select the higher score.Hold on critical gates and inspect stagesPreserve paired cases, slices, component metrics, and severe-failure precedence.

The trace has not run.

Commit to an implementation, then inject the failure and inspect the actual state transitions.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Build a paired run manifest with relevance judgments, hand-checked ranking metrics, context decisions, claim support, slices, critical gates, latency, and cost.
Break it on purpose: Improve averages while introducing one ACL leak, one zero-result query, one judge false pass, and one language-specific regression.
Completion evidence: The report identifies each responsible stage, preserves raw cases and missingness, calculates metrics reproducibly, and holds release on the critical failure.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What should happen when expected evidence is absent from the candidate set?

Primary sources · verified 2026-08-02

What the fixture does not prove

A stronger generator can mask worse retrieval on easy questions; an automated judge can reward fluent unsupported prose; average Recall@k can hide complete failure for one language, tenant, or policy type.

Common mistake: RAGAS, nDCG, or one model judge can be the release gate by itself. Each metric embodies a construct, labels, thresholds, and blind spots. Release evidence needs component metrics, calibrated judgments, critical invariants, and raw cases.

Transfer exercise: Compare baseline and candidate on eight paired queries; inspect Recall@5, nDCG@5, context precision, claim support, abstention, latency, and critical failures.

Thakur et al. — BEIR benchmarkPrimary benchmark comparing lexical, sparse, dense, late-interaction, and reranking systems across heterogeneous retrieval tasks.Es et al. — RAGASPrimary research separating retrieval relevance, answer faithfulness, and answer quality; automated metrics still require product-specific validation.Muennighoff et al. — MTEBPrimary benchmark showing that embedding performance varies by task and dataset; similarity performance alone does not establish retrieval fitness.Liu et al. — Lost in the MiddlePrimary evidence that relevant information position and excess context can materially affect long-context question-answering performance.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Evaluation establishes the release argument; production security and operations must keep corpus, index, permissions, and evidence coherent after launch.