How does an AI evaluation loop work?

Start with the decision, not the metric

An evaluation should answer a practical question: can this prompt change ship, is the new model safer on this workflow, did retrieval improve supported answers, or does the agent recover from tool failure within budget?

Without a decision, a score becomes decoration. State the system configuration being compared, the users and tasks represented, unacceptable failures, required evidence, and the threshold or review rule that changes an engineering action.

flowchart LR
  D[Decision and risk] --> C[Cases and criteria]
  C --> R[Versioned trials]
  R --> G[Deterministic gates and graders]
  G --> A[Aggregate and slice]
  A --> F[Inspect failures and traces]
  F --> X[Bounded system change]
  X --> R
  F --> N[New regression case]
  N --> C

A case is more than a prompt

A useful case specifies the conditions under which behaviour is judged:

• input or user goal;
• relevant initial state and evidence;
• expected properties or acceptable outcomes;
• prohibited outputs or effects;
• model, prompt, tools, retrieval, policy, and decoder configuration;
• budgets for latency, tokens, calls, or money;
• the grader and evidence it may inspect.

For a classifier, a labelled input may be enough. For a RAG system, preserve the document snapshot and expected citations. For an agent, preserve environment state, permissions, tool behaviour, allowed effects, and terminal state.

Separate cases, trials, and graders

A case defines one scenario. A trial is one execution of one configuration on that case. A grader converts trial evidence into structured results.

The distinction matters because generation can vary. Ten trials of one case are not ten independent requirements; they reveal a success distribution and possible severe trajectories. Likewise, changing a grader changes the measurement system even when the model output is identical.

Version all three. Store enough manifest data to reproduce which model snapshot, prompt, tool schema, dataset revision, environment, and grader produced a result.

Grade the strongest available evidence first

Prefer deterministic checks when the requirement is deterministic:

• exact schema and parser validity;
• program exit status and test results;
• database or filesystem end state;
• cited source identifiers;
• permission and approval events;
• latency, token, and call budgets.

Use rubrics, expert review, or model judges for qualities that cannot be reduced safely: relevance, clarity, explanation quality, or whether a trajectory was unnecessarily repetitive. Calibrate qualitative graders against human-labelled examples and keep the evidence they cite.

A single judge score should not override a failed authorization gate or a wrong application state.

Aggregate without hiding failures

The mean pass rate is not the whole result. Report sample count, repeated-trial success, uncertainty, severe failure count, latency and cost distributions, and performance slices that represent important conditions.

Useful slices may include language, input length, tenant, tool availability, document freshness, task difficulty, or permission mode. A candidate can improve overall while regressing badly on one small but important group.

Define critical failures separately. One unauthorized effect, fabricated citation, or silent data-loss event may block a release even when hundreds of ordinary cases pass.

Failure analysis is where learning happens

Inspect traces, not only final text. Locate the first observable divergence:

• wrong or missing context;
• ambiguous instruction;
• retrieval miss;
• invalid tool choice or arguments;
• policy denial or approval failure;
• tool error or unknown side effect;
• model ignored valid evidence;
• grader misclassified the result.

Assign the failure to the layer that can change it. Adding prompt prose will not repair a cross-tenant authorization bug. Replacing a model will not repair a stale retrieval index. Tightening a grader will not improve the product behaviour it merely measures.

Change one bounded part, then rerun

Keep the baseline. Change the prompt, context selector, model, tool description, retrieval configuration, harness policy, or decoder intentionally. Rerun the same representative cases under comparable conditions and include held-out cases that were not used to tune the change.

When a real failure is confirmed, add the smallest regression case that preserves the violated contract. Do not fill the suite with near-duplicate transcripts. Keep a separate challenge set so repeated repair does not overfit the visible evaluation.

Offline and production evidence complete each other

Offline evaluations are controlled and repeatable. Production monitoring reveals real task distributions, integrations, user behaviour, and failures the original cases did not anticipate. Neither replaces the other.

Sample production failures with privacy and access controls, triage them, reproduce defensible cases offline, and add them to the regression or held-out pool. When the model, prompt, context pipeline, tool set, policy, grader, or environment changes, decide which evaluations must run again.

The loop is healthy when it produces a release decision, an explainable failure taxonomy, and better future cases—not merely a larger dashboard.