Awesome Testing
Request training offer
AI Learning Lab
0/8 complete

RAG lesson 04 · Rerank and assemble context

How should candidates become a compact, ordered evidence packet?

Versioned evidence corpus · deterministic retrieval fixtures

Lighthouse Policy Evidence Assistant. A fictional support assistant answers questions from a versioned product-policy corpus. It must retrieve current, authorized evidence, cite exact source spans, and abstain when evidence is missing or conflicting.

01 · Smallest useful mechanism

Spend generation context on evidence, not retrieval leftovers.

A reranker can score query-document interaction more precisely over a small candidate set. Context assembly then removes duplicates, preserves necessary parent context, handles conflicts, and orders spans within a token budget.

Candidate recall and final context precision are different optimization problems.

02 · Experiment

Test the prediction

Deterministic retrieval fixture

This workbench uses inspectable ranking and evidence fixtures and injected failures. It does not make live calls or execute external effects.

Retrieval workbench · lighthouse-context-v4

Question to consider

Which context packet preserves the current rule and its exception within 1,500 tokens?

Complication

Five overlapping FAQ chunks outrank the exception and consume 1,300 tokens.

What would you do?

Choose the approach you would be prepared to defend in a design review.

Consider the trade-offs first.

Choose an answer, then run the scenario. You’ll see what the system checks, what happens under the complication, and where the control boundary holds or breaks.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement
Create a reranking and assembly ledger with candidate ranks, reranker scores, dedupe decisions, conflicts, selection reasons, order, and tokens.
Break it on purpose
Add five overlap duplicates, bury the exception, time out the reranker, and permute the final evidence order.
Completion evidence
The rule and exception remain selected within budget, fallback is explicit, duplicates cannot dominate, and order-sensitivity tests stay within the acceptance threshold.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

04 · Check your understanding

What is the primary job of candidate retrieval versus context assembly?

Primary sources · verified 2026-08-02

What the fixture does not prove

Blind top-k packing repeats overlapping chunks, buries decisive evidence, and increases latency and prompt-injection surface. Long context capacity does not guarantee uniform use of every position.

Common mistake: If context fits in the window, adding it cannot hurt. Distractors, contradictions, redundancy, and evidence position can change model behavior even when every token technically fits.

Transfer exercise: Compare fused-rank order, cross-encoder reranking, and a budgeted evidence packet when three chunks repeat the same rule.

Khattab and Zaharia — ColBERTPrimary research on late interaction: independently encoded query and document tokens combined with fine-grained MaxSim scoring.Liu et al. — Lost in the MiddlePrimary evidence that relevant information position and excess context can materially affect long-context question-answering performance.Thakur et al. — BEIR benchmarkPrimary benchmark comparing lexical, sparse, dense, late-interaction, and reranking systems across heterogeneous retrieval tasks.Anthropic — Contextual RetrievalVendor-authored experiments combining chunk context, embeddings, BM25, and reranking. The measured gains are useful hypotheses to reproduce on the target corpus.
Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.

Next: A precise evidence packet still needs a generation contract that maps claims to supporting spans and declines unsupported answers.