How do browser agents use accessibility trees?

The browser already computes a semantic view

Browsers derive an accessibility tree from HTML, CSS, native control semantics, and ARIA. It represents concepts such as button, textbox, accessible name, checked state, heading level, and relationships. Assistive technologies use this interface; automation can also use a serialized snapshot as structured evidence.

flowchart LR
  DOM[DOM + styles + ARIA] --> B[Browser computation]
  B --> TREE[Accessibility snapshot]
  TREE --> H[Agent harness]
  H --> M[Model proposes action]
  M --> T[Playwright tool]
  T --> B

A harness can attach a reference to each relevant node. The model sees something like button “Save” [ref=e17] and proposes a click on e17. The tool resolves that reference against current page state and returns a new observation. This is still an agent loop: the model proposes; the harness validates and executes.

Why this can be better than pixels

Text snapshots are often smaller than screenshots and make labels and states explicit. A semantic reference can be more stable than coordinates when layout shifts. The trace is also easier to read: “click button Save” says more than “click at 842, 611.”

Screenshots remain necessary for visual defects, canvas applications, charts, spatial relationships, or elements missing from the accessibility tree. DOM inspection may be necessary for implementation details. Strong browser harnesses select evidence based on the task rather than declaring one representation universally sufficient.

Semantics are an interface

Native HTML usually supplies the correct role, keyboard behaviour, state, and accessible-name computation. ARIA can add missing semantics but does not recreate native behaviour automatically. A clickable div styled as a button may look correct while remaining a generic node with incomplete keyboard support.

That is primarily an accessibility defect. It also makes semantic automation weaker: the agent receives less intent, may need brittle selectors, or may fail to distinguish controls. Browser-agent reliability can therefore reveal—but does not replace—accessibility testing.

State changes between observations

A reference belongs to a particular snapshot. Navigation, rerendering, a modal, or another actor can make it stale. Before a consequential action, the harness should resolve the current target and relevant state rather than trusting a remembered coordinate or label.

Browser sessions can expose cookies, private pages, downloads, clipboard data, and destructive controls. Constrain origins, credentials, downloads, uploads, and side effects. A visible button is not proof that the agent is authorized to press it.

What the tree cannot show

The accessibility tree is not the DOM, a screenshot, or a complete accessibility audit. It does not prove colour contrast, visual order, focus visibility, zoom behaviour, or understandable copy.

A browser tool is not an autonomous tester. Exploration requires goals, observations, coverage strategy, oracles, state control, and a record of what changed.

Evaluate semantic browser operation

Test pages with native controls, custom widgets, duplicate labels, hidden elements, iframes, shadow DOM, dynamic lists, modals, stale references, and navigation. Include correct and deliberately broken semantics. Assert the operation’s end state, not only that a click returned success.

Run keyboard and assistive-technology-oriented accessibility checks independently. Compare snapshot-driven and screenshot-driven behaviour on tasks where visual evidence matters. Measure completion, incorrect actions, stale-reference recovery, token and image cost, latency, and forbidden effects.

For destructive flows, seed a disposable environment and make policy a hard gate. Repeating a browser trial against shared mutable state makes the result difficult to interpret and potentially unsafe.