How do AI graders work?

Match the grader to the claim

An evaluation task defines inputs and success criteria. A trial produces an output, trace, artifacts, and perhaps real state changes. A grader measures one declared property of that evidence.

flowchart LR
  T[Trial evidence] --> C[Code checks]
  T --> M[Model grader]
  T --> H[Human review]
  C --> D[Decision record]
  M --> D
  H --> D
  C -->|hard invariant| G[Release gate]
  D --> G

Use code for facts a program can observe directly: schema validity, tests, calculations, file state, permissions, prohibited effects, latency, token count, and exact references. These checks are cheap, repeatable, and explainable.

Use trained human reviewers when the criterion requires domain judgement, context, or a value decision that cannot be reduced to a stable programmatic contract. Give them examples, a rubric, a disagreement process, and enough evidence to reproduce the judgement.

Use a model grader when a clear rubric can be applied to many outputs and imperfect automation is acceptable as one signal. Typical uses include relevance, completeness, style, groundedness, or trajectory review. The judge input should contain the criterion and evidence it needs—not hidden information unavailable to the system being evaluated.

A model judge is another model system

A model grader has its own provider, model revision, prompt, decoding settings, context construction, and output parser. Change any of them and the measurement instrument changed. Freeze and version that manifest beside the candidate system.

Prefer narrow assertions over “score this answer from 1 to 10.” Ask for one observable dimension, evidence citations, a verdict from a small defined set, and an abstention when evidence is missing. Pairwise comparison can be easier than assigning an absolute score, but it introduces ordering effects and still needs calibration.

Do not send a raw agent trace to a judge and assume it can infer every policy. Deterministic effect and authorization checks should run separately. The judge may assess whether evidence supports a claim; it should not overrule proof that a forbidden operation occurred.

Calibrate before scaling

Create a calibration set labelled independently by qualified humans. Include obvious passes and failures, boundary cases, important slices, disagreements, adversarial formatting, and severe risks. Compare the grader with those labels using raw counts and a confusion matrix, not only correlation or average score.

Choose thresholds from the cost of errors. A false pass on unauthorized data access is not equivalent to a false fail on writing style. Route low-confidence, conflicting, novel, and high-impact cases to human review. Keep a held-out set so repeated prompt tuning does not merely fit the calibration examples.

Known model-judge probes include:

• swap candidate order in a pairwise comparison;
• blind model and vendor names;
• add irrelevant length, confident tone, headings, or citations;
• compare human-written and model-written text of equal quality;
• paraphrase without changing meaning;
• place instructions to the judge inside the candidate answer;
• run the same case repeatedly and across judge revisions.

A robust verdict should not flip for transformations the rubric declares irrelevant.

Combine imperfect layers deliberately

Several graders are useful when they fail differently. A code check can verify citations exist, a retrieval check can verify the cited passages, a model grader can assess whether they support the claim, and a human can review disagreements. This is defense in depth, not permission to average incompatible signals into one opaque number.

Preserve each assertion and its evidence. Report pass, fail, abstain, grader error, and unavailable separately. A missing judge response is not a pass, and disagreement is information rather than noise to discard.

A judge score is not ground truth

A model judge is not ground truth, an independent expert, or automatically impartial because its prompt says so. Research has observed position, verbosity, familiarity, and self-preference biases. Agreement with human labels on one dataset does not transfer automatically to a different domain, language, risk, or response format.

Human review is not infallible either. Reviewers need training, blinding where appropriate, fatigue controls, and adjudication. Deterministic code is only as correct as its oracle and environment. Every grader type needs validation.

Validate the grader before trusting it

Unit-test code graders with known passes, failures, malformed evidence, and boundary values. For model graders, freeze the manifest and run the calibration, invariance, bias, injection, missing-evidence, and repetition probes above. Assert the output parser fails closed on unknown labels and truncated or malformed responses.

Measure false-pass and false-fail counts by important slice and severity. Inspect every severe false pass. Compare judge revisions on the same frozen evidence before upgrading, and keep a rollback path. Sample production verdicts for independent human review and watch disagreement rates rather than only average scores.

Finally, test the evaluation pipeline itself: candidate identity must remain blinded where declared, traces must map to the correct trial, graders must receive the intended evidence, and one failed grader must not silently remove a case from the denominator. A release report should make it possible to trace every aggregate back to task, trial, assertion, grader version, verdict, and evidence.