Which metrics make an AI evaluation useful?

Begin with the claim

“Better” is not measurable until the product claim is explicit. A parser may need 99.9% schema-valid output. A research assistant may need every material claim supported by an authorized source. A coding agent may need to finish a task and leave the repository in a passing state without touching forbidden paths.

The metric follows the claim and evidence. Do not choose BLEU, a model judge, or a benchmark pass rate merely because the tool exposes it.

flowchart LR
  C[Product claim] --> E[Strongest observable evidence]
  E --> M[Metric and threshold]
  M --> S[Aggregate and important slices]
  S --> D[Release or investigation decision]

Match the metric to the evidence

Use the strongest available measurement:

• Exact and property checks: schema validity, enum membership, program output, test results, database state, permissions, and budgets.
• Classification metrics: precision, recall, false-positive and false-negative rates when cases have defensible labels.
• Retrieval metrics: recall at a candidate depth, reciprocal rank, ranking gain, and access-filter correctness.
• Generated-answer metrics: claim support, citation correctness, relevance, completeness, abstention, and human preference.
• Agent metrics: task success, forbidden effects, tool and turn counts, recovery, terminal-state evidence, and trajectory quality.
• Operational metrics: latency percentiles, input and output tokens, cache usage, calls, retries, and money per completed task.

Several metrics may be necessary because a final answer can be correct after an unsafe or extremely expensive trajectory. Conversely, an efficient trajectory that changes the wrong state is still a failure.

Reliability depends on how attempts are used

Suppose one independent trial succeeds with probability p and the system can make k attempts.

pass@k = 1 − (1 − p)^k estimates the chance that at least one attempt succeeds. It is useful when the product can generate several candidates and a reliable selector or verifier chooses a good one.

pass^k = p^k estimates the chance that all attempts succeed. It is useful when repeated use must be consistently safe or correct.

With p = 0.8 and k = 3, pass@3 is 99.2%, while pass^3 is 51.2%. The same base success rate tells opposite stories depending on the product requirement. Independence is also an assumption: repeated runs can share the same prompt, retrieval defect, tool outage, or model bias.

Report the number of cases and trials, the sampling configuration, and uncertainty around estimates. A change from 82% to 84% across twenty cases is different evidence from the same difference across ten thousand independent cases.

Aggregates need slices and gates

A mean hides tails and minority conditions. Slice results by factors that can change behaviour: language, task type, input length, difficulty, time period, document freshness, tool availability, permission mode, user group, model route, or environment.

Predeclare critical failures separately. One cross-tenant disclosure, unauthorized payment, destructive write, or fabricated high-stakes citation may block a release even when the average quality score improves. Count both frequency and severity; do not average a catastrophic effect into a harmless formatting miss.

Use distributions for latency and cost. Median latency says little about a user waiting at the 99th percentile. Cost per API call can improve while cost per successful task worsens because the agent makes more calls or retries.

A grader has metrics too

Human and model graders are measurement instruments. Compare them against adjudicated examples and report agreement, class-specific errors, sensitivity to response order, verbosity, style, and evidence placement.

For a binary gate, inspect precision and recall relative to the risk. A low-precision safety grader interrupts many valid tasks. A low-recall one lets dangerous cases through. For scores, examine consistency and whether score differences correspond to decisions humans actually care about.

Do not use the same model output as both the product answer and unchallenged proof of its correctness. Give graders the evidence they need, keep their prompts and versions fixed during a comparison, and route deterministic failures around subjective scoring.

A useful report preserves the denominator

Every headline metric should carry enough context to interpret it: cases, trials, excluded or invalid runs, configuration, date, dataset revision, grader, confidence or variability, slices, and critical-failure counts.

Pair the number with examples from important failures and the first divergent trace step. Metrics decide where to look and whether a threshold is met; diagnosis explains what to change.