How should a query produce a broad but authorized candidate set?

Versioned evidence corpus · deterministic retrieval fixtures

Lighthouse Policy Evidence Assistant. A fictional support assistant answers questions from a versioned product-policy corpus. It must retrieve current, authorized evidence, cite exact source spans, and abstain when evidence is missing or conflicting.

01 · Smallest useful mechanism

Treat retrieval as candidate generation with independent signals.

Lexical retrieval excels at exact identifiers and rare terms. Dense retrieval can connect paraphrases but depends on model behavior and domain fit. Hybrid retrieval preserves both result lists and combines their ranks.

Fuse ranks or calibrated evidence; do not average incomparable raw scores.

02 · Experiment

Test the prediction

Deterministic retrieval fixture

This workbench uses inspectable ranking and evidence fixtures and injected failures. It does not make live calls or execute external effects.

Retrieval workbench · lighthouse-hybrid-v3

Question to consider

Which candidate strategy finds both a paraphrased rule and an exact product identifier?

Complication

Dense search ranks another tenant’s similar policy first; BM25 ranks a stale TrailCam 4 clause first.

What would you do?

Choose the approach you would be prepared to defend in a design review.

Dense top fiveTrust semantic similarity and filter the final prompt afterward.Average BM25 and cosine scoresNormalize neither channel and sort their raw arithmetic mean.Filtered lexical + dense RRFApply trusted filters per channel, fuse ranks, and retain channel evidence.

Consider the trade-offs first.

Choose an answer, then run the scenario. You’ll see what the system checks, what happens under the complication, and where the control boundary holds or breaks.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Implement filtered BM25 and dense retrieval, preserve per-channel ranks, fuse by stable chunk ID, and emit a candidate trace.
Break it on purpose: Use an exact stale identifier, a paraphrase, a cross-tenant near-match, and an embedding-model version mismatch.
Completion evidence: Eligible relevant evidence appears within candidate k, forbidden chunks never enter either list, and the trace reconstructs every fused rank.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Why is reciprocal rank fusion useful for hybrid retrieval?

Primary sources · verified 2026-08-02

What the fixture does not prove

A semantic-only demo can miss product codes, negation, dates, or low-resource language. A lexical-only system can miss paraphrases and concept-level matches. Filters applied too late can leak forbidden content.

Common mistake: Cosine 0.82 is universally better than BM25 12.4. The scores come from different functions and distributions. Combine ranks or calibrate scores on representative relevance judgments.

Transfer exercise: Search “activated camera return window” and “TrailCam 4 §4.2” through lexical, dense, and reciprocal-rank-fused fixtures.

Karpukhin et al. — Dense Passage RetrievalPrimary research on dual-encoder dense retrieval, passage indexing, training examples, and top-k retrieval accuracy for open-domain QA.Thakur et al. — BEIR benchmarkPrimary benchmark comparing lexical, sparse, dense, late-interaction, and reranking systems across heterogeneous retrieval tasks.OpenAI — Retrieval guideOfficial implementation contract for vector stores, semantic and hybrid search, attribute filters, rankers, thresholds, chunking, and result inspection.Muennighoff et al. — MTEBPrimary benchmark showing that embedding performance varies by task and dataset; similarity performance alone does not establish retrieval fitness.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Candidate retrieval optimizes coverage; reranking and context assembly must decide which small evidence set deserves the generation budget.