In my previous post about AI coding agents in 2026, I described what I called another revolution: models are becoming capable of discovering things for themselves. We could say that they have gained even more autonomy and intelligence, and can now operate at a higher level of abstraction.
Today, I would like to show one practical example of this change: self-healing tests. I will demonstrate how we can implement a workflow in which an AI agent diagnoses failing end-to-end tests and, in the right circumstances, proposes and verifies a repair.
This post will be practical and somewhat technical, and it comes with a complete example repository. As usual, I will use my Awesome LocalStack as the system under test.
Before We Heal Anything
Before moving on to the practical part, it is worth organising some of the theory. The subject of self-healing tests is controversial. When Microsoft introduced Playwright Agents, perhaps it presented the Healer agent a little too quickly, without explaining the broader context, building a complete narrative, or adding engineering's famous it depends. The official description is certainly attractive: the Healer executes a failing test, inspects the current UI, suggests a patch, and reruns the test. The problem is that a failing test is not necessarily a broken test.
I would like to return to this subject because the increasing intelligence of models makes this kind of workflow appear realistic today. In my opinion, it can also be effective, as I will demonstrate using my example. Of course, the demo should be treated with a slight pinch of salt because its failures are relatively simple. Nevertheless, I am confident that the same workflow can deal with more complex problems as well.
Most importantly, this workflow scales. It improves together with Claude Code, Codex, Copilot, Cursor, and the models behind them. Claude Code is the agent used in my implementation, but it is not a fundamental requirement. It could be replaced by Codex or another capable coding agent. This is the same broader idea I discussed in Agentic Testing: modern coding agents already know how to inspect repositories, execute tools, observe results, and iterate towards a goal.
Isolated Tests
In an older (and now rather outdated) article about isolated Cypress UI tests, I explained the subtle difference between an isolated and a non-isolated test. This is a good opportunity to revisit that distinction.
An isolated test exercises one product or project while mocking every external dependency that could make the test fail. When testing a frontend in isolation, for example, we can mock its backend at the HTTP layer. Playwright provides page.route() for intercepting and mocking network requests, just as Cypress provides its own network-interception mechanism.
In general, these tests should not need to heal themselves. They could, but there is not much value in doing so. Isolated tests should run in the pipeline after every commit and act as a release gate. Their result should be binary: the build either passes or it does not.
When we think of isolated tests as blocking tests and as a quality gate before release, we can state their expected condition very clearly: they should always pass. If they fail, the developer whose change caused the failure should fix them immediately. We should not allow those failures to accumulate in a queue that somebody may investigate later.
We generally want as many isolated tests as practical. They are lightweight, run reliably, require relatively little environmental setup or investment, and can be developed and maintained alongside the production code. Following the logic of the test pyramid, they should form the majority of the tests in a project.
Non-Isolated Tests
On the other side, we have tests with external dependencies. These might be large-scale functional (or performance) tests executed once per day, week, or month. They might exercise an entire system composed of dozens of microservices. They may integrate with test environments or sandboxes supplied by payment providers such as BLIK, card processors, Stripe, or PayPal, or with external systems such as Salesforce.
The purpose of keeping some of those dependencies real is to detect integration defects. When we replace an external service with a mock in an isolated test, we always carry the risk that our model of that service is incomplete or has drifted away from reality. Isolation gives us speed and determinism, but it cannot prove that every real integration still works.
These non-isolated, large-scale checks are often the classic end-to-end tests maintained by testers or SREs. They can provide substantial business value because they exercise the actual product from beginning to end. Customers and stakeholders also value them because they provide evidence that the system works as a whole.
At the same time, poorly designed end-to-end tests can quickly become a burden. Instead of providing confidence, they generate cost, noise, and frustration for development teams. That is the familiar trade-off: the tests that provide the broadest business confidence are usually also the slowest, most expensive, and most difficult to maintain.
Because these tests usually require a great deal of babysitting and maintenance effort, it is worth looking for ways to reduce the manual work around them. One way to do that is to implement a self-healing loop. This is the loop I will describe in the rest of this article.
Self-Healing Starts with Triage
The idea of self-healing tests is not new. We have already seen several proposed implementations of it. Healenium, for example, can repair Selenium tests when a locator stops matching after an element ID or class changes. This is useful, but it represents a relatively narrow interpretation of healing: a test fails, so the mechanism tries to repair the test.
In my Agentic Testing article, I showed that many tasks performed on a developer's computer (or on a virtual machine used as a CI runner) can now be delegated to modern agent harnesses such as Claude Code or Codex. In this post, I will show how to implement a self-healing workflow using Claude Code.
Before discussing its implementation, however, we need to establish how the agent should approach healing. Its first responsibility should not be to change code. It should perform triage.
The agent must use the available evidence to determine whether a failed test indicates:
- a defect in the application;
- a misunderstanding or defect in the requirements;
- a defect in the test itself;
- a CI or test-environment problem;
- a failure in an external dependency;
- or another cause that cannot yet be established.
This distinction is essential, and it was the part I found missing from earlier self-healing implementations and most discussions of the subject. They concentrated almost exclusively on repairing tests. That approach begins with a dangerous assumption: if a test failed, the test must be wrong.
When maintaining real end-to-end suites, a test defect is only one of several possible explanations. There may be an actual application bug. The environment may be misconfigured or temporarily unavailable. Tests may be competing for state in a shared environment. An external service may have changed its behaviour. The runner may have exhausted a resource, or an API may have started returning 429 Too Many Requests because a rate limit was reached.
The number of possible causes grows very quickly once a test crosses process, repository, network, and organisational boundaries. A self-healing design must take this uncertainty into account. Otherwise, it risks making the pipeline green by changing the test that correctly detected a product or integration problem.
So, spoiler alert: the first task I will give Claude Code is not “fix this test.” I will ask it to investigate the failure and decide whether it represents a product defect, a test defect, a CI or environment problem, or an inconclusive state. Only one of those classifications should allow it to propose a repair to the test repository.
Demo Architecture
I described this environment recently in From One Local Stack to Three Training Profiles, so I will not repeat its complete architecture here. This experiment targets aitesters.byst.re, a disposable sibling of awesome.byst.re backed by a seeded H2 sandbox that is reset regularly.
The initial CI profile contains five Playwright smoke tests: four credential-free authentication checks and one authenticated catalog-assistant scenario. The latter asks about available iPhones, expects two backend tool calls, and verifies that the final answer remains consistent with the returned catalog snapshot.
flowchart TD
Tests["Playwright smoke tests"] --> Gateway["aitesters.byst.re gateway"]
Gateway --> Frontend["React frontend"]
Gateway --> Backend["Spring Boot backend"]
Backend --> Database["Seeded H2 database"]
Backend <--> Model["Deterministic<br/>Ollama mock"]
Despite its small size, the scenario crosses several repositories: self-healing-tests-demo owns the tests, vite-react-frontend the UI, test-secure-backend the API and tool loop, ollama-mock the deterministic model responses, and awesome-localstack the deployment.
The same assertion can therefore fail because of the test, product code, model fixture, live data, or environment. The Playwright error alone is not enough: Claude needs evidence, expected behaviour, and the relevant source repositories before it can assign ownership confidently.
The Failure-Only CI Workflow
The most important workflow rule is simple: when the Playwright tests pass, Claude Code never starts. AI is not part of ordinary test execution. It is a diagnostic and repair stage triggered only by a failure.
flowchart TD
E2E["Run Playwright tests"] --> Passed{"Passed?"}
Passed -->|yes| Done["Finish"]
Passed -->|no| Evidence["Collect evidence<br/>and source context"]
Evidence --> Claude["Claude Code triage"]
Claude --> Gate{"TEST_DEFECT<br/>confidence >= 0.90?"}
Gate -->|no| Report["Report diagnosis<br/>keep run red"]
Gate -->|yes| Patch["Validate patch<br/>open draft PR"]
Patch --> Verify["Run Playwright<br/>on the PR"]
Verify --> Green["Green repair PR"]
The E2E step uses continue-on-error so that GitHub Actions can upload the Playwright report and start triage after the test command exits with an error. This does not turn the build green. A final step deliberately restores the failure after the evidence and diagnosis have been published.
The workflow also guards same-repository events and forked pull requests, retains evidence, produces structured reports, and bundles eligible repairs. I invoke a pinned Claude Code CLI through a small Python module instead of embedding all of that logic in YAML. The module controls the prompt, JSON schema, turn and cost limits, and the sanitized live event stream.
The complete GitHub Actions implementation remains available in the example repository. The expandable version below keeps only the control flow relevant to this article.
🔧 Essential GitHub Actions control flow (click to expand)
This abridged version omits standard checkout, Node.js, dependency, and browser setup. The repository workflow contains the complete implementation.
The Failure-Triage Prompt
The YAML starts Claude, but the prompt defines the investigation. It does not say “make the tests pass.” It gives Claude an evidence order, requires a focused reproduction and browser investigation, and permits an edit only after a high-confidence TEST_DEFECT classification. The prompt is combined with CLAUDE.md, which loads the product overview and triage contract, and with a JSON schema that makes the final result machine-readable.
🧠 Claude Code failure-triage prompt (click to expand)
The project-level CLAUDE.md loads the permanent repository instructions
and domain context:
The failure-specific prompt then controls the investigation:
Giving Claude Enough Context
A test error alone is rarely enough to classify an end-to-end failure. A timeout tells us where the test stopped, but not whether the cause was a stale locator, an unavailable API, incorrect product behaviour, bad test data, or a service further down the dependency chain. If we give Claude only the final stack trace, we should expect a shallow diagnosis.
The workflow therefore preserves the complete Playwright evidence set before Claude starts. This includes the console output, JSON results, HTML report, screenshots, videos, traces, and the error-context snapshot generated for the failed scenario. Claude reads the failed test and its page object as well, so it can compare the intended assertion with what the browser actually observed.
Execution evidence is only one half of the context. The repository also contains a product overview describing the environments, service topology, important user flows, and ownership boundaries. It records business expectations that cannot be inferred safely from the implementation alone. For example, the catalog assistant's final answer must remain consistent with the product data returned by its tools. Without that rule, an agent might see contradictory values and “repair” the assertion instead of recognising a product defect.
The triage contract defines the available classifications and the evidence required before changing anything. A separate repository registry tells the workflow which source repositories may explain a failure. These repositories are cloned only after the E2E tests fail and are placed under .triage-sources, where Claude can inspect them without receiving write access.
Claude is not expected to read every repository from beginning to end. It starts with the failed assertion and browser artifacts, forms an initial ownership hypothesis, and then inspects only the most likely source checkout. It also reruns the failed scenario without retries and performs one focused live-browser investigation using Playwright CLI. I described why this terminal-oriented browser interface works particularly well for agents in Playwright CLI, Skills and Isolated Agentic Testing.
This is an important part of the design. The model is not magically discovering the architecture from a failed locator. We deliberately provide two kinds of information: evidence describing what happened and product context describing what should have happened. The quality of the triage depends on both.
The Triage Contract
The investigation must end with one of four explicit classifications:
PRODUCT_BUG: the test still expresses valid behaviour, but the product violates it;TEST_DEFECT: the product behaves correctly, but the test, setup, locator, assertion, or data is wrong;CI_ENVIRONMENT: the runner, browser, network, target environment, or another operational dependency prevented meaningful evaluation;INCONCLUSIVE: the evidence is missing, contradictory, intermittent, or insufficient.
These four outputs are deliberately broader than the list of possible causes discussed earlier. A rate limit, unavailable external sandbox, or shared-environment collision will usually become CI_ENVIRONMENT. A requirements disagreement remains INCONCLUSIVE until the available product context supports a stronger conclusion.
Claude returns its decision as structured JSON validated against the repository's triage-result schema. The result includes not only a label, but also confidence, concrete evidence, reproduction details, the likely owning repository, the smallest proposed change, and verification performed after a repair.
Claude's JSON output is only a recommendation. Deterministic Python policy permits a draft pull request only for a TEST_DEFECT with confidence of at least 0.90, targeting the test repository, and when the reported files exactly match an allow-listed TypeScript or TSX diff. Every other result remains read-only.
This keeps reasoning probabilistic but control deterministic: Claude investigates and proposes; code decides whether a repair may become a draft pull request.
Case One: A Product Bug Claude Refused to Heal
The first experiment tested restraint. In the Awesome LocalStack catalog-assistant flow, I added a business-level assertion that compares the product snapshot returned by a tool with the assistant's final answer:
const productSnapshot =
await toolsPage.readProductSnapshot();
await expect(
toolsPage.assistantMessageContents.last()
).toContainText(
`${productSnapshot.stockQuantity} unit(s) in stock`
);
The value was not hardcoded in the test: it came from the tool output produced during the same interaction. The workflow failed because that output reported 50 units, while the assistant's final answer reported only 5.
Claude traced the incorrect value to a deterministic fixture in ollama-mock, classified the failure as PRODUCT_BUG with 0.95 confidence, and changed nothing.
The publishing job was skipped and the workflow remained red. This is the safety property I care about: self-healing must not turn broken product behaviour into a green pipeline by weakening a valid test.
Case Two: A Test Defect Claude Repaired
For the second experiment, I introduced a defect in the test data. I replaced the catalog prompt supported by the deterministic model mock with a new, perfectly reasonable sentence. The intent stayed the same, but ollama-mock recognises this scenario through an exact prompt match. It therefore returned its unsupported-prompt response and produced no catalog tool calls.
In the failed Claude triage job, Claude first confirmed that the page, tool configuration, and other smoke tests were healthy. It then compared the browser snapshot with the triggering commit and found that the rejected prompt differed from the allow-listed prompt shown by the application. Finally, it inspected the cloned ollama-mock repository and confirmed the exact string used to select the catalog scenario.
This time the result was TEST_DEFECT with 0.96 confidence. Claude restored the supported prompt in tests/smoke/catalog-tools.smoke.spec.ts, reran the failed scenario without retries, and then ran the complete E2E suite. The focused test passed in 8.1 seconds and the full suite passed five out of five tests.
Claude classified the failure as a test defect, applied a one-file fix, and verified both the focused scenario and the complete E2E suite.
The deterministic policy accepted the one-file patch, and the separate publishing job created draft pull request #4. The PR's own Playwright pipeline is green, but Claude still cannot merge it. A human can review the diagnosis, diff, and verification before deciding whether the repair belongs on main.
This is the complete self-healing loop: detect a failure, gather evidence, classify it, make the smallest eligible change, verify it, and propose it for review. The first case stopped after diagnosis because the product was wrong. The second continued to a pull request because the evidence showed that the test was wrong.
Cost, Latency, and the Real Trade-off
This workflow requires judgement, not only code generation. For that reason, I would rather perform the triage with one of the most capable models (such as Fable, Opus, or GPT-5.6 Sol) than optimise immediately for the cheapest possible invocation. That level of intelligence inevitably comes with a cost.
The two successful investigations shown above were relatively inexpensive. The product-bug triage cost $0.8664, while the test-defect diagnosis and repair cost $0.8136. Each required approximately two minutes of Claude work. The complete experiment was more expensive, however. While preparing this article, I tried different failures, prompts, permission models, turn limits, and publishing strategies. I used Opus for those iterations and generated approximately $15 in API usage.
The dashboard shows the wider experimentation cost, not the cost of one production triage. Prompt caching reduced the bill, but repeated experiments still accumulated roughly $15 of usage.
Fifteen dollars may sound like a lot for a blog demo. On the other hand, the result was a working workflow that collected evidence, reproduced failures, inspected several repositories, distinguished a product bug from a test defect, repaired the eligible failure, verified the change, and opened a reviewable pull request. Looked at from the perspective of convenience and engineering time saved, the amount may not seem particularly large.
The individual and company perspectives will always be different. An individual may compare $15 with the value of learning and completing a personal project. A company must multiply the same workflow across teams, repositories, and recurring failures. That is why this job runs only after an E2E failure and why it has explicit turn, time, and cost limits. I will leave the final judgement to the reader. For a broader discussion of model pricing, prompt caching, learning budgets, and organisational cost control, see AI Coding Agents in 2026: Discovery, Skills, Costs, and What Comes Next.
Conclusion
There is no doubt that the development of language models is opening significant new possibilities. We can create intelligent cron jobs (loops, routines) and add an assessment step to our delivery pipelines. As this demo shows, that assessment can diagnose a failure and repair a test. It could also report a product bug to the appropriate team or, with a different permission model, prepare a fix in the production repository. The space of possible workflows is enormous.
The implementation of such ideas also becomes easier with every generation of models and harnesses. I would therefore encourage everyone, once again, to spend time experimenting with them. The progress will not stop. Month by month, workflows that previously required a custom agent platform become possible through a repository, a prompt, a coding-agent harness, and a few deterministic safety checks.
You can try this yourself. Give this article and the demo repository to Codex or Claude Code and ask it to build a similar workflow for your project. It will probably produce something useful—and it may produce something better than my implementation, because mine was deliberately built as a demo while yours can be designed around your actual systems, failure modes, and engineering policies.
I recently saw another example of this progress in Awesome LocalStack. A few days ago, I was asked to add multi-factor authentication to the application. The resulting standards-based TOTP functionality can now be enabled from the user profile. To be honest, I was surprised that the first end-to-end implementation worked. A feature spanning backend security, frontend flows, authenticator enrollment, recovery codes, tests, and deployment would previously have required much more manual coordination.
That does not mean we should give an agent unlimited access and trust every change. The lesson from this experiment is almost the opposite: give the model enough context and freedom to investigate, but surround its decisions with observable evidence, explicit classifications, deterministic policy, and human review. Self-healing should not mean making every pipeline green. It should mean understanding the failure first, changing only what the evidence permits, and making the next human decision substantially easier.
Comments
Loading comments...