Awesome Testing

Agent lesson 08 · Defend the whole system

Can you prove that the research agent is useful without giving it purchase authority?

Shared teaching scenario · fictional product data

Research three laptops under €900, verify current evidence, and write laptop-comparison.md. Do not purchase anything or contact a vendor.

01 · Smallest useful mechanism

Design the whole agent as an evidence-producing control system.

The capstone combines the course into one architecture review. You will specify the laptop-research agent without granting purchase authority, trace a representative run, identify failure modes, choose deterministic boundaries, and define the evidence required before calling the task complete.

A deployable design makes goals, state, proposals, policy decisions, effects, observations, stopping rules, and evaluation evidence explicit.

02 · Experiment

Test the prediction

Deterministic browser simulation

The defend the whole system experiment uses inspectable, repeatable teaching data. It does not claim to be a live agent trajectory.

Architecture review

Defend every transition before you reveal the reference design.

Research three laptops under €900, verify current evidence, and write laptop-comparison.md. Do not purchase anything or contact a vendor.

Your review must prove

useful outcome · current evidence · bounded effects · observable trace · repeatable evaluation

  1. 01Write a measurable task and terminal-state contract.
  2. 02Draw the model–harness–tool ownership boundary.
  3. 03List each tool, effect class, permission, approval, and timeout.
  4. 04Specify context sources, authority, provenance, and expiry.
  5. 05Trace one success, one stale-evidence failure, and one denied action.
  6. 06Define graders for outcome, trajectory, forbidden effects, reliability, and cost.

03 · Production-minded practice

Prove the mechanism under failure

Use your own stack or a contained mock environment. Do not point failure drills at real accounts, people, or irreversible services.

  1. Build

    Produce an architecture review containing the task contract, ownership map, context manifest, tool matrix, representative trace, threat model, terminal-state validator, and evaluation plan.

  2. Break it on purpose

    Run one stale-evidence case and one attempted external mutation after the user’s permission has been revoked.

  3. Evidence of completion

    The stale claim is refreshed or disclosed, the mutation is denied under current policy, the report remains useful, and every acceptance claim points to inspectable evidence.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Capstone decision rubric

Would this agent be safe to trial?

Ready

Every effect crosses current identity, policy, approval, and outcome checks; the trace supports recovery; evaluations cover success, forbidden effects, variance, cost, and stop conditions.

Incomplete

The design has a useful loop but leaves ownership, freshness, retry semantics, terminal validation, or evidence-backed evaluation implicit.

Unsafe

The design relies on prompt obedience, cached authorization, unrestricted tools, blind retries, or a good final answer as proof that no prohibited effect occurred.

04 · Check your understanding

What should the harness consult before every effectful call, including a retry?

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.

Next: Re-run the design with a different task, compare its risk surface, or return to the LLM course to inspect the policy inside each model turn.