Where do AI agents fail?

Start before the final answer

The final response may be wrong because retrieval missed a document, policy denied a needed action, a tool timed out, state failed to persist, or the grader expected the wrong thing. It may also be correct despite an unsafe intermediate read, redundant calls, or an effect the final prose never mentions.

Debug from the trace and environment state. Preserve the user goal, principal, prompt and tool versions, model configuration, context sources, proposals, policy decisions, approvals, operations, results, state revisions, costs, timings, and terminal reason.

flowchart LR
  G[Goal] --> C[Context]
  C --> M[Model proposal]
  M --> P[Protocol parse]
  P --> A[Policy and approval]
  A --> T[Tool and environment]
  T --> S[Persisted state]
  S --> D[Done gate]
  D --> R[Grader]

The first step whose observed state differs from the contract is the first divergence. Later mistakes may be consequences.

Failure layers

Goal and context: the system misunderstood scope, omitted an instruction, retrieved stale evidence, exceeded the context budget, mixed tenants, or treated untrusted content as policy.

Model proposal: the model chose the wrong tool, produced invalid arguments, ignored evidence, repeated an action, or claimed completion too early.

Protocol: streaming chunks were assembled incorrectly, tool-call IDs were mismatched, a refusal was parsed as data, or a truncated response reached business logic.

Policy and approval: the gate allowed too much, denied a legitimate scoped action, evaluated the wrong principal, reused stale approval, or approved one payload and executed another.

Tool and environment: a command failed, a dependency changed, the sandbox lacked a resource, output was truncated, the network returned partial data, or the operation produced an unknown outcome.

State: two workers advanced one run, an observation was lost, a failed action became a successful memory, a retry duplicated an effect, or cancellation arrived after dispatch.

Completion and grading: the harness accepted a model stop without evidence, or a grader misread a correct artifact, rewarded style, or ignored a severe intermediate effect.

Preserve outcome certainty

Do not collapse every tool result into success or failure. A useful effect state includes:

• rejected before dispatch;
• known failure with no effect;
• known success with observed effect;
• unknown outcome after possible dispatch;
• later reconciled success or failure.

Unknown is a real state. Repeating a timed-out payment, deployment, email, or file mutation can duplicate the effect. Reconcile using an operation ID or inspect the authoritative target before another attempt.

Likewise, a model API retry differs from an effect retry. A failed generation request normally has not changed business state; a lost response from a side-effecting tool may have.

Separate capability from reliability

A model may be capable of solving a task in one sampled trajectory while the product is unreliable across repeated trials. Report where variation occurs: search strategy, tool choice, argument construction, recovery, or stopping.

Distinguish:

• systematic defects, which repeat under the same condition;
• stochastic misses, which vary across comparable trials;
• environment defects, which come from unavailable or drifting dependencies;
• measurement defects, where the product behaved correctly but the grader failed.

Increasing retries can hide a low first-attempt success rate while increasing cost and exposure to side effects. A better model can reduce proposal errors while leaving the same authorization flaw untouched.

Repair the owning layer

Map evidence to the smallest responsible boundary:

• missing current facts → ingestion, retrieval, or context selection;
• ambiguous task contract → product or prompt design;
• malformed call → schema, model, or decoding contract;
• unauthorized effect → policy and execution gate;
• repeated effect → idempotency and state machine;
• tool outage → timeout, fallback, or availability design;
• false failure label → grader calibration;
• premature completion → terminal-state evidence.

Change one bounded component, replay deterministic trace fixtures where possible, then rerun representative end-to-end cases. Add the failure as a regression case at the layer that could have prevented it.

The strongest diagnosis says more than “the agent hallucinated.” It names the earliest broken contract, the evidence that proves it, the downstream effects, and the control that should make the same failure observable or impossible next time.