How should AI-generated tests and test data be used?

Generate candidates, not confidence

A model can turn a requirement, schema, example, or code path into candidate partitions, boundary values, fixtures, API calls, assertions, mocks, or browser steps. It is especially helpful for broadening a list and translating known intent into repetitive test code.

The model can also reproduce the implementation’s mistake, invent an API, assert a meaningless status code, mock away the behaviour, or create data that violates the real domain. A test that passes is only executable text. Its value depends on the fault it can reveal.

Start from the test model

Before asking for code, state the behaviour and partitions:

• valid, invalid, empty, extreme, and malformed inputs;
• state transitions and forbidden transitions;
• identity, tenant, role, and permission boundaries;
• time, retries, partial failure, and duplicate effects;
• compatibility with existing fixtures, helpers, and conventions;
• the observable result and independent oracle.

Give the generator representative project tests and the actual interface. Ask it to reuse fixtures and utilities rather than build a parallel testing architecture. Remove secrets and personal data from prompts and outputs.

For synthetic data, preserve the constraints relevant to the test while avoiding claims that generated rows reproduce the real distribution. Label provenance and seed generation where repeatability matters.

flowchart LR
  R[Requirement and risks] --> G[Generate candidate tests]
  G --> X[Execute and inspect coverage]
  X --> M[Apply fixed realistic mutants]
  M -->|survivor| T[Add or repair a contract test]
  M -->|killed| E[Record evidence]
  T --> X

Close the loop with execution

Research systems that generate stronger tests commonly iterate: inspect code, propose a test, run it, observe failure or coverage, repair it, and try uncovered behaviour. Execution eliminates syntax errors and false assumptions, but coverage alone can reward tests that touch code without checking it.

Mutation testing asks a sharper question: if the implementation contained a plausible defect, would the suite fail? Framework-generated mutants make small code changes. Semantic mutants encode requirement misunderstandings such as reversed authorization or incorrect retry ownership. Both expose weak assertions that line coverage cannot.

Never repair a frozen mutant after seeing which test passed. A mutant that does not compile is invalid, not killed. A surviving mutant may be equivalent, unobservable, or evidence of a missing contract test; classify it before changing the suite.

Keep an independent oracle

Generating implementation and tests in the same conversation creates correlated errors. Improve independence by grounding expectations in requirements, schemas, examples, a reference implementation, deterministic state checks, or human review. Ask a different process to challenge the suite, but do not treat a second model as ground truth.

Generated browser tests often overfit temporary text or selectors. Prefer user-visible roles and outcomes, preserve the project’s page and fixture patterns, and confirm that the test fails when the intended behaviour is broken.

Generated volume is not test strength

Generated test volume is not coverage of risk. High line coverage is not strong assertions. A realistic-looking synthetic dataset is not proof of statistical, privacy, or domain fidelity.

AI-generated tests do not make requirements, test design, or maintenance optional. If nobody can explain what a test protects, its future failures will be difficult to interpret.

Challenge the generated suite

First run generated tests against the unchanged system and inspect failures. Then verify that each test has a clear reason, controls its state, and observes the contract rather than implementation trivia.

Measure compile rate, executable rate, flaky rate, behaviour and branch coverage, meaningful mutants killed, unique defects found, review effort, and maintenance cost. Keep generated and human-written provenance only if it supports learning; acceptance criteria should be the same.

Seed benchmark tasks with known faults and hidden edge cases. Compare one-shot generation with an iterative workflow that receives execution and coverage feedback. For security and policy tests, require deterministic negative assertions and isolated data rather than a model judge alone.