How should candidates become a compact, ordered evidence packet?

Versioned evidence corpus · deterministic retrieval fixtures

Lighthouse Policy Evidence Assistant. A fictional support assistant answers questions from a versioned product-policy corpus. It must retrieve current, authorized evidence, cite exact source spans, and abstain when evidence is missing or conflicting.

01 · Smallest useful mechanism

Spend generation context on evidence, not retrieval leftovers.

A reranker can score query-document interaction more precisely over a small candidate set. Context assembly then removes duplicates, preserves necessary parent context, handles conflicts, and orders spans within a token budget.

Candidate recall and final context precision are different optimization problems.

02 · Experiment

Test the prediction

Deterministic retrieval fixture

This workbench uses inspectable ranking and evidence fixtures and injected failures. It does not make live calls or execute external effects.

Retrieval workbench · lighthouse-context-v4

Choose the context packet that preserves the current rule and its exception within 1,500 tokens.

Injected failure

Five overlapping FAQ chunks outrank the exception and consume 1,300 tokens.

Choose the implementation

Pack candidates until fullAppend fused results in order and truncate at the token limit.Rerank top 30Use a stronger scorer but keep duplicate spans and ignore conflicts.Rerank, dedupe, and pair evidenceCollapse overlap, keep rule with exception, label conflict, and record selection.

The trace has not run.

Commit to an implementation, then inject the failure and inspect the actual state transitions.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Create a reranking and assembly ledger with candidate ranks, reranker scores, dedupe decisions, conflicts, selection reasons, order, and tokens.
Break it on purpose: Add five overlap duplicates, bury the exception, time out the reranker, and permute the final evidence order.
Completion evidence: The rule and exception remain selected within budget, fallback is explicit, duplicates cannot dominate, and order-sensitivity tests stay within the acceptance threshold.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What is the primary job of candidate retrieval versus context assembly?

Primary sources · verified 2026-08-02

What the fixture does not prove

Blind top-k packing repeats overlapping chunks, buries decisive evidence, and increases latency and prompt-injection surface. Long context capacity does not guarantee uniform use of every position.

Common mistake: If context fits in the window, adding it cannot hurt. Distractors, contradictions, redundancy, and evidence position can change model behavior even when every token technically fits.

Transfer exercise: Compare fused-rank order, cross-encoder reranking, and a budgeted evidence packet when three chunks repeat the same rule.

Khattab and Zaharia — ColBERTPrimary research on late interaction: independently encoded query and document tokens combined with fine-grained MaxSim scoring.Liu et al. — Lost in the MiddlePrimary evidence that relevant information position and excess context can materially affect long-context question-answering performance.Thakur et al. — BEIR benchmarkPrimary benchmark comparing lexical, sparse, dense, late-interaction, and reranking systems across heterogeneous retrieval tasks.Anthropic — Contextual RetrievalVendor-authored experiments combining chunk context, embeddings, BM25, and reranking. The measured gains are useful hypotheses to reproduce on the target corpus.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: A precise evidence packet still needs a generation contract that maps claims to supporting spans and declines unsupported answers.