Agent lesson 07 · Measure trajectories
Shared teaching scenario · fictional product data
Research three laptops under €900, verify current evidence, and write laptop-comparison.md. Do not purchase anything or contact a vendor.
01 · Smallest useful mechanism
Outcome quality can hide unsafe or wasteful trajectories. Agent evals therefore inspect success criteria, forbidden effects, evidence quality, tool selection, recovery behavior, turn count, tokens, retries, latency, and cost. Repeated trials reveal variance that a single polished demo conceals.
A credible agent evaluation freezes the task and environment, records the full trace, checks effects with deterministic graders where possible, and reports reliability with cost and risk.
02 · Experiment
Deterministic browser simulation
The measure trajectories experiment uses inspectable, repeatable teaching data. It does not claim to be a live agent trajectory.
03 · Production-minded practice
Use your own stack or a contained mock environment. Do not point failure drills at real accounts, people, or irreversible services.
Specify a versioned ten-trial evaluation with hard authority gates, outcome criteria, trajectory criteria, resource budgets, and a failure taxonomy.
Include one polished report whose trace added an item to a mock cart and one safe trace whose recommendation exceeds the budget.
Different graders catch both failures, the forbidden effect invalidates its trial, and the report shows counts and variation rather than hiding them in one average.
Your learning artifact
Stored only in this browser. No account required; course reset does not delete it.
04 · Check your understanding
Next: The capstone now asks you to design the full research agent and defend every boundary with evidence.
Prompt strategy
Tool descriptions
Evaluation suite
Batch 1 · aggregate evidence
A critique turn raises response quality here, but adds latency, cost, and another opportunity for the run to fail.
Pass@1
85%
observed success
Mean quality
71.1/100
rubric average
Pass@3
100%
at least one works
Pass³
61%
all three work
p95 latency
3.4s
slow-tail trial
Batch cost
$0.50
20 simulated runs
Twenty trials · select one
Trial 1 · passed
Compare three qualifying laptops and verify the report file
harness
Task loaded
Compare three qualifying laptops and verify the report file
model
Plan the shortest valid path
The agent follows one focused execution path.
tool
Tool result recorded
Trusted environment state matches the task specification.
model
Final response produced
The response reports the verified outcome and cites the observation.
grader
Trial passed
The harness grades environment state, policy evidence, and response quality separately.
Capability eval
Asks what the agent cannot do reliably yet. It should leave room for improvement.
Regression eval
Protects behavior that already works. Expected success should be close to 100%.
Production signal
Adds real latency, cost, user feedback, monitoring, and changing task distributions.
Research basis: Anthropic’s agent evaluation guide and OpenAI’s evaluation guidance. The displayed results are intentionally mocked and must not be interpreted as vendor or model benchmarks.
