How do you red-team an AI system?

Start with a threat model

Name the system and decision boundary before inventing attacks. List assets such as private data, credentials, model weights, prompts, indexes, code, money, user trust, and service capacity. Draw trust boundaries around users, model providers, retrieval sources, tools, sandboxes, memory, graders, and operators.

For each adversary, state:

• their goal and the harm or property they want to cause;
• what identity, access, knowledge, time, budget, and automation they possess;
• which inputs, data, dependencies, or interfaces they can influence;
• whether they observe scores, errors, traces, model output, or real state;
• which controls and monitoring they know or can probe.

A public user, malicious tenant, compromised document author, supply-chain attacker, insider, and external tool do not have the same capabilities. A result found with direct database access does not demonstrate an exploit available through a chat box.

flowchart LR
  A[Asset and harm] --> T[Threat actor and capability]
  T --> S[Attack surface and scenario]
  S --> X[Controlled adversarial trial]
  X --> E[Trace, effects, and detection evidence]
  E --> C[Control or accepted residual risk]
  C --> R[Regression and monitoring]

Test the whole attack path

The language model is one component. Relevant attack surfaces include:

• training, fine-tuning, evaluation, and feedback data poisoning;
• model, package, prompt, skill, and tool supply chains;
• direct prompts and indirect instructions inside web pages, files, messages, images, or retrieved text;
• retrieval manipulation, embedding-store access, source spoofing, and citation laundering;
• tool schema abuse, argument injection, confused-deputy access, excessive permissions, and duplicate effects;
• cross-session memory poisoning and cross-tenant data leakage;
• unsafe rendering or execution of generated HTML, SQL, shell, code, URLs, and files;
• resource exhaustion through context, output, loops, fan-out, retries, or expensive tools;
• model extraction, membership or private-data inference, and sensitive output;
• evaluation and monitor evasion, grader injection, and audit-log tampering;
• human approval fatigue, misleading previews, and social engineering around the agent.

Trace the chain from adversary-controlled input to the protected asset or effect. “The model repeated an instruction” is not automatically a security impact. “An untrusted document caused the application to read another tenant's secret and place it in an outbound request” is a concrete path with boundaries and observable harm.

Write bounded scenarios

Each scenario needs an identifier, hypothesis, target version, preconditions, attacker capability, initial state, allowed test actions, prohibited real effects, success criteria, instrumentation, cleanup, and stop conditions. Use synthetic identities, data, credentials, and destinations. Put agents in a sandbox whose limits match production closely enough for the claimed result.

Use black-box tests to represent an external attacker, gray-box tests with limited design knowledge, and white-box review to inspect policy and data flow. These approaches answer different questions. Automation can mutate payloads and explore combinations, while a skilled human can notice new composition failures; neither replaces the other.

Begin with known taxonomies rather than only memorable prompts. NIST's adversarial-machine-learning taxonomy distinguishes attacker goals, lifecycle stage, knowledge, and capabilities across predictive and generative systems. MITRE ATLAS maps tactics and techniques. OWASP's current GenAI risks provide application-level review areas. Tailor them to the actual system and add abuse cases from domain experts.

Expect layered controls

Prompt instructions are useful product context but are not a security boundary. Robust controls sit at the boundary they protect:

• isolate untrusted content from trusted instructions and preserve provenance;
• minimize tool credentials, scopes, network access, and data visibility;
• authorize the exact resolved action against current identity and resource;
• validate and safely render model output before another interpreter sees it;
• require meaningful approval for consequential, novel, or irreversible effects;
• cap iterations, fan-out, time, tokens, cost, storage, and retries;
• isolate tenants, sessions, sandboxes, indexes, caches, and logs;
• record proposal, policy decision, effect, and outcome as separate events;
• detect suspicious sequences and provide rapid revocation, containment, and rollback.

A successful red-team finding can expose prevention, detection, response, or recovery weakness. Do not declare it fixed merely because one payload stopped working. Remove the underlying authority or data-flow flaw, then rerun variants and adjacent paths.

Operate the exercise safely

Obtain explicit authorization for targets, methods, dates, data, and operators. Coordinate with service owners and incident response. Define who can halt the exercise. Avoid production users and irreversible effects unless a separately approved, tightly controlled test requires them.

Protect findings, traces, exploit code, credentials, and affected data according to their sensitivity. Do not paste live secrets or personal data into third-party models as test material. Clean up accounts, tokens, files, jobs, and poisoned content, then independently verify cleanup.

For external discoveries, preserve evidence and use the provider's disclosure process. A public demonstration that exposes users or a currently exploitable path is not justified by calling it research.

Turn findings into durable evidence

A useful finding contains the violated security objective, exact system manifest, attacker assumptions, reproducible steps, trace and terminal state, impact, existing controls and why they failed, detection result, recommended owner, and remediation status. Separate theoretical, feasible in a lab, demonstrated in realistic conditions, and observed-in-the-wild evidence.

After remediation, freeze the smallest safe regression case that would have detected the control failure. Add variations that change irrelevant wording, encoding, position, language, and content source. Keep some strategies protected so the system is not optimized only against published strings. Monitor the production event pattern that would reveal recurrence.

Red teaming is not a jailbreak contest

Red teaming is not ordinary quality assurance with adversarial vocabulary, a leaderboard of jailbreak success, or permission to attack systems outside the approved scope. A model policy bypass without a defined asset and impact may be a product finding, but it is not automatically a system compromise.

It is also not proof of security. Failure to find an exploit depends on attacker capability, coverage, tools, time, observability, and environment fidelity. Report those limits and the residual risk rather than issuing a universal “safe” verdict.

Run bounded exercises and close findings

First test the red-team harness. Seed known vulnerable and known blocked scenarios and confirm that setup, identities, instrumentation, graders, effects, cleanup, and reports distinguish them. Verify that a grader cannot be instructed by the candidate output to ignore an attack and that missing traces or policy decisions fail the trial rather than disappearing.

Exercise direct and indirect injection, encoded and multilingual variants, nested retrieved content, poisoned memory, cross-tenant references, malformed tool calls, permission changes, time-of-check/time-of-use races, duplicate writes, sandbox escape attempts, egress, resource exhaustion, and monitor evasion. Combine attacks because real failures often cross several individually reasonable components.

Measure counts by threat objective and system boundary: attempted, reached, blocked, detected, contained, recovered, and unresolved. Track time to detection and containment, severe false negatives, unnecessary blocks, resource cost, and environment differences. Do not collapse them into one “robustness score.”

Finally, rerun the original frozen scenario, semantic variants, adjacent interfaces, and the normal product suite after every fix. A remediation is acceptable only when it closes the underlying path without quietly breaking legitimate work or moving the same authority flaw to another tool.