Would this RAG system be safe and useful to pilot?

Versioned evidence corpus · deterministic retrieval fixtures

Lighthouse Policy Evidence Assistant. A fictional support assistant answers questions from a versioned product-policy corpus. It must retrieve current, authorized evidence, cite exact source spans, and abstain when evidence is missing or conflicting.

01 · Smallest useful mechanism

Defend one coherent system from source ingestion to release decision.

The capstone asks for an architecture and release dossier that link every answer claim to corpus governance, retrieval evidence, context selection, generation validation, evaluation, monitoring, and recovery.

A vector search demo becomes a product only when evidence, authority, failure, and operation form one contract.

02 · Experiment

Test the prediction

Deterministic retrieval fixture

This workbench uses inspectable ranking and evidence fixtures and injected failures. It does not make live calls or execute external effects.

Retrieval workbench · lighthouse-capstone-v8

Choose the pilot proposal that remains coherent under compound retrieval and corpus failures.

Injected failure

The candidate index improves average recall while one replica serves deleted content, a poisoned duplicate ranks first, and reranking times out.

Choose the implementation

Pilot the best demo configurationUse the higher recall index, add more context, and monitor user complaints.Ship retrieval improvements onlyKeep the new index and defer governance, grounding, and rollback work.Run a constrained evidence-backed pilotGate versions, prefilter authority, validate claims, preserve traces, and rehearse rollback.

The trace has not run.

Commit to an implementation, then inject the failure and inspect the actual state transitions.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Produce the full architecture, source and state inventories, trace, threat model, evaluation charter, critical gates, SLOs, rollout, and rollback dossier.
Break it on purpose: Combine stale and poisoned sources, ACL drift, empty retrieval, reranker timeout, unsupported generation, deletion lag, load shedding, and version skew.
Completion evidence: The system fails closed on authority, answers only supported claims, identifies each failed stage, preserves evidence, rolls back coherently, and limits the pilot to declared residual risk.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

Capstone decision rubric

Would this RAG system be safe and useful to pilot?

Ready

The retrieval contract, corpus provenance, authorization filters, ranking stages, claim-level support, evaluation evidence, monitoring, and rollback plan remain coherent under injected failures.

Incomplete

The happy path works, but freshness, permissions, retrieval quality, abstention, citation support, or an operational recovery rule remains implicit or untested.

Unsafe

Forbidden, stale, or poisoned content can reach generation; unsupported claims can appear authoritative; or aggregate metrics hide failures that matter to users.

04 · Check your understanding

What is the strongest evidence that a RAG system is ready to pilot?

Primary sources · verified 2026-08-02

What the fixture does not prove

Teams often optimize components independently: better embeddings, a stronger reranker, a longer prompt. Compound failures reveal whether those improvements preserve the product’s authority and truthfulness.

Common mistake: Passing an end-to-end demo is enough to pilot. A pilot needs representative evaluation, critical safety gates, rollback, observability, governed data, and explicit residual risk.

Transfer exercise: Review Lighthouse under stale policy, missing evidence, cross-tenant near-match, reranker timeout, indirect injection, deletion lag, and a fluent unsupported answer.

Lewis et al. — Retrieval-Augmented GenerationThe original RAG paper formalizing generation with parametric memory plus retrieved non-parametric evidence and explicit provenance goals.Thakur et al. — BEIR benchmarkPrimary benchmark comparing lexical, sparse, dense, late-interaction, and reranking systems across heterogeneous retrieval tasks.Es et al. — RAGASPrimary research separating retrieval relevance, answer faithfulness, and answer quality; automated metrics still require product-specific validation.OWASP GenAI — Vector and Embedding WeaknessesCurrent retrieval-specific threat guidance for cross-context leakage, poisoning, access control, inversion, provenance, and monitoring.OpenAI — Retrieval guideOfficial implementation contract for vector stores, semantic and hybrid search, attribute filters, rankers, thresholds, chunking, and result inspection.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Return to the evaluation course to calibrate the judges, uncertainty, and production monitoring used in the RAG release dossier.