How do you know whether AI assistance is worth it?

The denominator is an accepted outcome

AI can produce more test cases, code, or text while making the complete job slower. Generated volume counts activity. Productivity asks whether a defined outcome was completed to the required quality with less scarce effort or acceptable additional value.

Choose a unit such as:

• one reviewed test-design task with required risk coverage;
• one merged change that passes the full acceptance suite;
• one reproducible exploratory finding accepted by a tester;
• one support case resolved without a harmful escalation.

Then define “accepted” before comparing workflows. If AI-assisted code needs later cleanup, introduces a regression, or fails review, that work belongs to the same outcome cost.

Count the whole workflow

A useful accounting model is:

total task cost = human active work
                + waiting and coordination
                + review and correction
                + model and tool charges
                + infrastructure and maintenance
                + expected failure impact

Expected failure impact combines likelihood with consequence. It will be approximate, but omitting it makes a cheap unreliable workflow look artificially attractive. For low-consequence experiments, rollback time may be enough. For sensitive data or production effects, security, compliance, incident, and recovery costs matter.

Track elapsed time and human attention separately. A background agent can take longer wall-clock time while freeing an engineer for other work. Conversely, rapid suggestions can consume continuous review attention and fragment work.

Productivity evidence does not transfer automatically

Controlled research has reported very different outcomes. One experiment found a large completion-time improvement for a bounded JavaScript HTTP-server task, while explicitly leaving code-quality effects unmeasured. A later randomized study of experienced maintainers working in mature repositories found that its AI-allowed tasks took longer, despite participants believing AI had saved time.

The studies do not cancel each other. They describe different people, tools, models, repositories, task shapes, and outcome measures. Together they show why “AI makes developers 30% faster” is not a portable constant.

Experience can change both directions. A novice may benefit from examples and explanations but lack the judgement to catch a plausible defect. An expert may review accurately yet receive little value on a familiar task. A model may help with repetitive translation but struggle with implicit architecture or unusual domain constraints.

Run a task-level comparison

Build a representative task bank before deciding which workflow wins. Stratify it by activity—test design, data creation, code generation, debugging, review, exploration, documentation—and by difficulty, repository familiarity, consequence, and available oracle.

Compare at least:

1. the current baseline workflow;
2. the same workflow with a declared AI tool and usage policy;
3. optionally, a more autonomous agent workflow.

Freeze acceptance criteria and starting state before assigning the workflow. Measure completion, quality gates, severe failures, elapsed time, human active time, review cycles, model and tool cost, and later maintenance. If people select AI only for easy tasks, a raw comparison will confuse task selection with productivity.

Use paired tasks or random assignment where practical. For personal teaching or team learning, a smaller repeated crossover can still expose where the tool helps: perform comparable tasks with and without assistance, preserve traces, and review the result after enough cases to overcome one lucky session.

Segment before averaging

Report the distribution and important slices. A positive mean can hide slower expert work, weaker security tasks, or expensive failures. A negative mean can hide a highly valuable niche such as unfamiliar API discovery or mechanical fixture creation.

Useful slices include task type, novelty, user experience, repository size, context quality, model and harness version, language, number of tool calls, review burden, and consequence level. Keep the sample count beside every percentage.

Do not use acceptance rate, tokens, lines of code, prompts sent, or subjective satisfaction as the sole outcome. Each can explain the workflow, but none proves useful completed work. Perceived speed is especially weak when prompting and reviewing feel easier than writing while still increasing total duration.

Turn the result into a delegation policy

A good productivity study ends with a scoped operating rule, not a universal verdict. For example: use AI to propose boundary cases and draft repetitive fixtures; require a human-owned oracle and mutation challenge; avoid it for small changes in deeply familiar code unless the engineer chooses otherwise.

Re-evaluate when the model, harness, pricing, repository, team experience, or acceptance process changes. The question is never simply whether AI is expensive. It is whether this versioned workflow produces more accepted value from the constrained time, attention, money, and risk budget available.