Which attacks and misuse paths could violate the system contract?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Derive adversarial tests from assets, actors, trust boundaries, and effects.

The red-team plan covers direct and indirect prompt injection, malicious retrieved text, cross-customer access, social engineering, tool overreach, secret exfiltration, encoding tricks, and control unavailability. Each case names the expected containment and evidence.

A refusal rate is not a safety score: the system must block prohibited effects while preserving legitimate work.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The probe severe failures workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose controls and evidence for the contained adversarial suite.

Fixture northstar-red-team-v1

Label source authorityRetrieved and user text remain untrusted data.Minimize tool scopeExpose only tenant-scoped, task-required operations.Use synthetic canariesDetect attempted cross-account disclosure without real PII.Test control failureVerify high-risk operations fail closed when authorization is unavailable.Disable all legitimate supportCall the system safe because it refuses every request.Use real customer secretsPlace production PII in attack prompts to improve realism.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
State the assets, actors, trust boundaries, and prohibited effects in scope.
Evaluation design
Create contained injection, disclosure, overreach, encoding, and control-failure cases.
Release evidence
Record expected containment, observed effects, control decisions, and residual risk for every attack.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Why is a 100% refusal rate not sufficient safety evidence?

Theory notes and sources

What the experiment does not prove

A system can refuse obvious attacks and still obey hostile text retrieved from its own knowledge base.

Common mistake: Passing benign cases demonstrates safety. Benign capability evidence and adversarial containment evidence support different claims.

Transfer exercise: Insert an instruction into a policy article and trace whether it remains untrusted data at every boundary.

NIST Generative AI ProfileAuthoritative risk guidance for generative-AI systems and their lifecycle controls.OWASP Prompt InjectionApplication-security guidance on direct and indirect prompt-injection risks.NIST AI Risk Management Framework 1.0A system-lifecycle framework for mapping context, measuring trustworthiness, and managing AI risk.Promptfoo — Red-team configurationOfficial configuration reference for versionable targets, plugins, strategies, languages, and run metadata in automated adversarial testing.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Adversarial cases reveal severity; the next lesson combines repeated trials, paired comparisons, slices, and uncertainty without averaging critical failures away.