How do you test an AI agent?

Define the whole system under test

An agent result depends on more than a model. Freeze the model, prompts, harness version, tool schemas, credentials and scopes, environment snapshot, policies, budgets, case data, and graders. If any of these changes, the system under test changed.

Each case needs an initial state, user goal, permitted and prohibited effects, success criteria, resource budget, and observable terminal state. “Produce a good answer” is insufficient when the task can search private data, modify a record, or create a file.

Grade outcomes and trajectories

A final response can look correct after an unacceptable run. Evaluate at least four layers:

Agent reliability is multi-dimensional: a plausible final artifact cannot erase a bad trajectory or an unauthorized effect.

• Outcome: Was the requested artifact or state produced correctly?
• Trajectory: Were tool choices, arguments, evidence, ordering, and recovery acceptable?
• Effects: Did any forbidden, duplicate, cross-tenant, or unapproved operation occur?
• Resources: How many turns, tokens, retries, seconds, and external calls were consumed?

flowchart TB
  T[Versioned task and environment] --> R[Repeated agent trials]
  R --> O[Outcome grader]
  R --> J[Trajectory grader]
  R --> P[Policy and effect gates]
  R --> B[Budget measurements]
  O --> D[Release evidence]
  J --> D
  P --> D
  B --> D

Deterministic state and policy checks should take precedence over subjective quality scores. A helpful report does not compensate for an unauthorized purchase attempt.

Inject failures deliberately

Happy-path cases rarely expose the hardest runtime errors. Test:

• timeouts before and after a tool may have performed an effect;
• malformed, stale, contradictory, or injected evidence;
• unavailable authorization and policy services;
• permission revocation during a run;
• context truncation and missing observations;
• duplicate completion events and replay;
• tool results from the wrong tenant or resource;
• budget exhaustion and cancellation.

The agent should classify known failure, explicit denial, cancellation, incomplete work, and unknown outcome differently. Unknown must not be silently converted into “nothing happened.”

Repeat trials

The same case can follow different trajectories because model generation and external systems vary. Repeat trials under identical declared conditions. Report success counts, severe failures, variation, latency, and cost. Keep individual trace IDs so an aggregate can be audited.

Transcripts are not evaluation evidence

Agent testing is not a collection of attractive transcripts. It is not enough to ask another model whether the final answer looks good. A judge that cannot see tool calls, approvals, state, or effects cannot grade them.

It is also not enough to test the model and harness independently. Interface failures emerge when context reaches the model, proposals cross policy, tools return partial results, and traces feed graders.

Build a decision-grade agent evaluation

Build the smallest complete evaluation loop:

1. Create versioned cases in a sandbox with inspectable initial and expected states.
2. Capture a structured trace for every proposal, decision, operation, observation, and stop reason.
3. Add deterministic hard gates for permissions, prohibited effects, and terminal state.
4. Add outcome and trajectory rubrics with cited evidence.
5. Run a baseline and candidate on the same cases with repeated trials.
6. Review every critical event and a sample of passes near thresholds.
7. Convert confirmed failures into regression cases while preserving fresh held-out evidence.

The evaluation is complete only when it supports a decision, exposes what remains unknown, and names the conditions that would trigger rollback or reevaluation.