How do you make an AI-assisted session reproducible?

Preserve the conditions, not only the chat

A copied prompt is rarely enough to reproduce an AI-assisted task. The result may also depend on hidden application instructions, selected files, retrieved passages, conversation history, model revision, decoding settings, tool descriptions, permissions, current database state, and failures observed during the run.

A transcript records what happened. A reproducibility bundle records both what happened and the conditions needed to interpret or rerun it.

flowchart LR
  M[Versioned session manifest] --> R1[Original trial]
  M --> R2[Clean replay]
  R1 --> E1[Trace, artifact, effects]
  R2 --> E2[Trace, artifact, effects]
  E1 --> C[Compare contract-level outcomes]
  E2 --> C

Identical transcripts are not the target. Two valid runs can use different wording or tool paths. Compare the required artifact, assertions, external effects, policy decisions, latency, and cost.

The minimum session bundle

Preserve identifiers or hashes for:

• the user goal, acceptance criteria, input files, attachments, and authoritative evidence;
• provider, requested model, returned model revision when available, and API or client version;
• system, developer, and user prompts in their original order;
• temperature, sampling controls, seed when supported, context limits, and stop conditions;
• harness version, tool names and schemas, MCP server versions, policy and approval mode;
• initial environment: repository commit and diff, fixtures, account roles, feature flags, data snapshot, clock, locale, and network assumptions;
• every model response, tool proposal, validation decision, approval, execution result, retry, and terminal reason;
• final files, reports, test results, external effects, token usage, timings, and charges.

Use stable trace and operation identifiers to connect these records. Telemetry conventions can standardize fields such as provider, requested and returned model, response ID, token usage, and tool operation, but a generic tracing schema cannot know your task contract or repository state.

Canonical state and derived records

The chat is not the canonical copy of a changed file, database row, sent message, or deployed application. Preserve the actual artifact and observed effect in its owning system, then link it to the run.

Summaries, compacted context, screenshots, and model explanations are derived evidence. Keep them when useful, but do not let them replace the exact prompt asset, source document, diff, command status, or environment snapshot they describe.

For side effects, record operation identity and outcome certainty. A timed-out tool call may have succeeded even if its result never reached the transcript. Replaying it without reconciliation can duplicate a message, payment, or deployment.

Replay at three levels

Protocol replay feeds recorded model and tool responses into the harness. It tests deterministic parsing, policy, state transitions, and rendering without depending on a live model.

System replay starts from a clean environment and calls the declared model and tools again. It tests the integrated workflow but may produce another valid trajectory.

Evaluation replay runs the same versioned cases for several trials and compares outcome distributions. It is the correct level for claims such as “this agent succeeds reliably” or “this prompt reduced failures.”

Stub volatile services when testing failure handling. Use real integrations when their behaviour is part of the claim. Label the difference; a mocked browser result does not reproduce a production browser session.

Reproducibility has a privacy budget

Complete prompts and tool results can contain personal data, credentials, proprietary code, or customer records. Do not log everything by default. Separate low-risk metadata from sensitive content, redact before export, restrict access, set retention, and document when an exact replay is impossible because evidence cannot be retained.

Hashes can prove that an artifact changed or remained identical, but they cannot recover deleted evidence or show that its meaning was correct. Store the minimum content justified by debugging, audit, or evaluation needs.

Test the reproduction claim

Choose a completed session and rebuild its initial conditions from the manifest. Confirm that missing versions or fixtures fail visibly rather than silently using current defaults. Run several trials and compare contract-level outcomes, important trajectory constraints, and effects.

Finally, change one dependency—the prompt, model alias, tool schema, permission, retrieved document, or repository commit—and verify that the manifest and comparison report expose the change. A reproducible workflow is one in which another engineer can explain why two runs are comparable and where they differ, even when the model does not emit the same tokens.