What is retrieval-augmented generation?

The evidence pipeline

RAG keeps knowledge in an external corpus instead of relying only on model parameters. At request time, the system converts a user need into one or more searches, retrieves candidate passages, ranks and filters them, assembles a context packet, and asks a model to answer from that packet. The answer can carry citations back to exact source spans.

Retrieval selects evidence from a larger governed corpus into a finite request context; it does not write that evidence into model memory.

flowchart LR
  D[Governed documents] --> I[Parse, chunk, index]
  Q[User question] --> R[Retrieve candidates]
  I --> R
  R --> K[Rerank and authorize]
  K --> C[Context with provenance]
  C --> G[Generate or abstain]
  G --> V[Verify claims and citations]

This architecture is useful for current, private, domain-specific, or frequently changing information. Updating an index can be much cheaper and more controllable than retraining a model. Retrieval also gives the application an observable place to enforce access and attach provenance.

Retrieval has several models

Lexical retrieval matches terms and is often strong for exact identifiers, names, error codes, and rare phrases. Dense retrieval maps queries and passages into vectors and can match similar meaning expressed with different words. Hybrid retrieval combines both. A reranker then applies a more expensive relevance model to a smaller candidate set.

Chunking is part of retrieval behaviour. Pieces that are too small may lose context; pieces that are too large may dilute the relevant evidence and consume the model's context budget. Store document ID, version, location, access metadata, and transformation lineage with every chunk so a citation can return to an authoritative source.

The best candidate is not automatically permitted evidence. Authorization should filter at or before retrieval and be checked again before the context is sent to the model. Post-generation redaction cannot reliably undo exposure of private content already placed in model input.

Generation must remain grounded

Retrieved text is evidence, not an instruction with higher authority than the user or system policy. Documents can contain mistakes, obsolete facts, conflicts, or prompt injection. Label provenance and authority, delimit content, and require the generation step to distinguish supported claims from synthesis or uncertainty.

A citation is useful only if it supports the nearby material claim. Merely attaching a relevant document to an unsupported answer is citation decoration. When evidence is absent, stale, conflicting, or below a relevance threshold, an explicit abstention or clarification is often the correct result.

Failures are compositional

A wrong answer can begin in ingestion, parsing, chunking, metadata, indexing, query rewriting, retrieval, authorization, reranking, context ordering, generation, citation attachment, or freshness. End-to-end correctness alone cannot identify the broken component.

RAG can also fail silently when the model ignores good evidence or follows its parameter memory instead. Conversely, a well-grounded answer can still be unhelpful or incorrect if the corpus itself is wrong. Version the corpus and index alongside prompts, model, retriever, and evaluation cases.

Retrieval does not make an answer grounded

RAG is not a synonym for a vector database, embeddings, or “put more documents in the prompt.” It does not guarantee truth, eliminate hallucination, or make untrusted documents safe. It is not automatically better than direct search, a database query, a deterministic report, or a longer static context.

RAG also does not train the model on retrieved documents. The evidence affects the current inference context while the model parameters normally remain unchanged.

Evaluate retrieval and grounded use separately

Evaluate the pipeline by component and end to end. Build cases with a versioned corpus snapshot, question, authorized evidence set, expected source spans, acceptable answer claims, required citations, forbidden disclosures, and abstention conditions.

For retrieval, measure recall at a candidate depth, ranking metrics, latency, and performance across domains, languages, time ranges, and exact-identifier queries. Include lexical, dense, and hybrid baselines. For context assembly, test deduplication, ordering, token budgets, provenance, authorization, and stale or conflicting sources.

For generation, grade answer relevance, claim-level faithfulness to retrieved evidence, citation correctness, completeness, uncertainty, and refusal when evidence is insufficient. Calibrate any model judge against human-labelled examples and keep raw passages and outputs available for review.

Inject missing documents, stale index versions, poisoned chunks, prompt injection, access revocation, duplicate passages, contradictory sources, retrieval timeout, and an answerable question whose correct evidence ranks just below the cutoff. The release decision should report which component failed instead of hiding all failures inside one average score.