What is agentic testing?

A tester with a loop

A test generator maps an input to proposed test cases or code. An agentic tester can observe a system, choose a tool, perform an action, inspect the result, update its working model, and continue until a stopping condition.

flowchart LR
  C[Charter and risk] --> O[Observe]
  O --> H[Form hypothesis]
  H --> A[Choose bounded action]
  A --> E[Inspect evidence and effects]
  E -->|new question| O
  E -->|stop condition| R[Report with trace]

This supports exploratory behaviour, but autonomy does not create testing skill automatically. The harness must supply useful observations, tools, state, budgets, safety policy, and a way to distinguish evidence from speculation.

White-box and black-box modes

In white-box testing, the agent can inspect code, tests, configuration, history, logs, and runtime state. It may connect a risky code path to an executable scenario, generate a focused test, or investigate a failure. Repository access improves context but also exposes secrets and can bias the agent toward implementation assumptions.

In black-box testing, the agent works through public interfaces such as HTTP, browser, mobile UI, or command line. It can explore workflows as a user would, but needs reliable state setup, identities, cleanup, and oracles. A browser accessibility snapshot can provide semantic controls; screenshots add visual evidence.

Hybrid work connects both: discover a symptom through the product, inspect code or telemetry, form a cause hypothesis, then reproduce it through the external interface.

Give exploration a charter

An unconstrained instruction such as “test this site” produces an attractive transcript with unclear coverage. Supply a charter: target, risks, users, data boundaries, time or action budget, allowed tools, forbidden effects, and reporting format.

Keep a lightweight coverage model. Track features, states, data types, roles, platforms, and failure conditions visited. The agent can choose the next experiment, while the harness records what remains untouched. Novelty is useful only when observations can be reproduced and tied to a risk.

Every finding needs evidence: starting state, actions, actual result, expected basis, impact, and reproducibility. An agent’s confidence score is not an oracle. Requirements, deterministic state, contracts, telemetry, comparison systems, and human judgement provide different forms of evidence.

Constrain effects and cost

Exploration can send messages, create accounts, delete data, trigger payments, or expose private information. Use disposable environments, scoped identities, allowlisted targets, effect budgets, and approval for consequential actions. Record every proposal, policy decision, tool result, and state change.

Stop on goal completion, exhausted coverage budget, repeated no-progress, unsafe ambiguity, or an environment that can no longer be trusted. “Keep testing” is not a measurable terminal condition.

Where exploratory agents do not fit

Agentic testing is not a replacement for deterministic regression suites, specialist review, or production monitoring. It is not automatically superior to a well-designed script for a stable requirement.

A long trace is not evidence of coverage, and a plausible bug report is not a confirmed defect. The field and terminology are evolving; compare concrete capabilities rather than product labels.

Evaluate the testing agent

Evaluate the testing agent using seeded defects, known-risk charters, clean environment snapshots, and hidden outcomes. Repeat runs because exploration paths vary. Grade defects found, false reports, severity, coverage, reproduction quality, forbidden effects, time, tokens, and tool calls.

Test failure handling: stale pages, misleading logs, inaccessible controls, flaky services, expired credentials, duplicate actions, injected content, and partial cleanup. Confirm that the agent marks uncertainty instead of manufacturing a conclusion.

Compare it with manual exploration and existing automation on the same charter. The useful question is not whether the agent can click and write reports, but whether it adds reproducible information at an acceptable cost and risk.