Production systems · reviewed · reviewed Aug 31, 2026 · 3 min

Should you use prompting, RAG, tools, or fine-tuning?

Change the layer that owns the gap: prompts for instructions, RAG for evidence, tools for authoritative reads or effects, code for exact policy, and fine-tuning for repeated behavioural patterns worth encoding in weights. Evaluate each addition against a simple baseline.

Prompting, retrieval, tools, code, and fine-tuning solve different failure classes and often work together.

Interactive note 16

Change the layer that owns the gap

First try: RAG or direct authoritative lookup
What changes: The evidence selected for this request
Boundary: Do not encode volatile facts into weights when they need updates or citations.

Real systems combine mechanisms. Add one layer at a time and keep it only when representative evaluations show a useful gain.

Diagnose the missing resource

Start with a failing case and ask what the system lacked at the moment of failure. The choices are not maturity levels and “fine-tuned” is not automatically more advanced.

Missing resource	First mechanism to try	What changes
Clear task, constraints, or output contract	Prompting or structured output	Current request context
Current, private, or citable evidence	RAG or direct lookup	Evidence in current context
Authoritative state or an external action	Tool or ordinary API	What the system can observe or affect
Exact invariant, permission, calculation, or workflow	Deterministic code	Enforced application behaviour
Repeated style, classification, or task behaviour at scale	Fine-tuning	Model parameters

Use the smallest mechanism that fixes the measured gap without moving an authoritative responsibility into probabilistic text.

Prompting defines the current task

Prompting is appropriate when the model has enough capability and evidence but needs a clearer goal, constraints, examples, rubric, or output shape. It is fast to change and easy to compare with a baseline.

Prompts are poor stores for frequently changing facts and poor enforcement for permissions. Extremely long instruction stacks compete for context and can become inconsistent. If a requirement can be expressed as a schema or deterministic check, move that part into code and keep the prompt focused on judgement.

Retrieval supplies evidence

Use RAG when answers depend on a body of current, private, domain-specific, or citable material too large or volatile to place permanently in the prompt. Retrieval lets the application update sources without changing model weights and preserves a route back to provenance.

RAG adds ingestion, chunking, indexing, ranking, authorization, freshness, and grounding failures. For an exact account balance, order state, or permission decision, query the authoritative service directly instead of hoping semantic search returns the right paragraph.

Tools connect state and effects

Use tools when the system must calculate deterministically, inspect live state, search a service, run code, or cause an external effect. A model can select a tool and propose arguments; the application still owns authentication, authorization, validation, execution, reconciliation, and result formatting.

A tool is not automatically better than retrieval. Searching a document corpus is a tool-backed retrieval operation. Reading one record by stable ID is ordinary data access. Choose an interface that exposes the needed capability with the smallest safe surface.

Deterministic code owns hard rules

Use ordinary software for arithmetic, state transitions, permissions, quotas, data validation, routing that must be exact, and irreversible effect control. A model can help interpret ambiguous intent, but it should not be the only place where “never transfer more than the approved amount” exists.

This option is often omitted from prompting-versus-RAG comparisons, yet it is the correct baseline for many product requirements. Not every natural-language input needs a natural-language implementation.

Fine-tuning changes repeated behaviour

Consider fine-tuning when representative evaluations show a stable behavioural gap that prompt and system design cannot solve economically or reliably: specialised classification, consistent format or tone, domain patterns, tool-selection behaviour, or performance with a smaller model.

Fine-tuning requires curated examples, protected evaluation data, training operations, checkpoint versioning, and regression analysis. It is slower to update than retrieval and harder to inspect than a prompt. It does not load live facts, enforce permissions, or guarantee truth.

Start with an evaluated prompt baseline. Collect failures and high-quality target examples from the real distribution. Fine-tune only when the expected quality, latency, or cost gain justifies a new model artifact and lifecycle.

Compose mechanisms deliberately

A production system may use all five layers:

flowchart LR
  P[Prompt: goal and contract] --> M[Adapted model]
  R[Retrieved evidence] --> M
  M --> C[Structured proposal]
  C --> V[Deterministic policy]
  V --> T[Tool reads or effects]
  T --> M

Add one layer at a time and rerun the same cases. Track task success, critical failures, latency, tokens, calls, and operating cost. Complexity is justified only when it produces a measured improvement the simpler system could not provide.

Sources