What is an AI system?

Draw the boundary before naming the failure

A model maps input to output. A product must decide where that input comes from, who may provide it, how output becomes a decision or effect, what data is stored, and what happens when a component is unavailable or wrong.

flowchart LR
  U[User and environment] --> A[Application and policy]
  D[Data and retrieved evidence] --> A
  A --> M[Model inference]
  M --> A
  A --> T[Tools or product action]
  T --> O[Observable outcome]
  O --> E[Evaluation and monitoring]
  E --> A

This system boundary includes the model and tokenizer, prompts, retrieval, tool adapters, permissions, UI, storage, infrastructure, human review, and operational controls. The exact list depends on the product. A search assistant and a medical device can use a similar model while forming very different systems with very different obligations.

Behaviour emerges between components

The same model can perform differently when the prompt template, retrieved evidence, decoder, tool interface, or context order changes. A correct model response can become a harmful product outcome if application code applies it without validation. A weaker model can sometimes produce a better system result when the surrounding workflow supplies strong evidence and deterministic checks.

Version the deployable combination. “We tested model X” is incomplete if production uses a different tokenizer, system prompt, retrieval index, harness policy, or runtime.

Claims need owners and evidence

Start with an intended-use claim: who uses the system, for what decision, under which conditions, and with what fallback. Map each critical claim to evidence and an owner. Model metrics, integration tests, security reviews, human factors, monitoring, and incident response answer different questions.

Risk is contextual. A wording error in a brainstorming tool and the same error in an automated account action have different consequences. Permissions and human intervention should follow the possible effect, not the confidence of a fluent response.

Keep the model inside the system boundary

An AI system is not synonymous with its largest model. A model benchmark is not an end-to-end product evaluation, and a prompt is not a reliable security boundary. “Human in the loop” is not a control unless the person has time, information, authority, and a usable way to reject the action.

The system is also not static. Providers, data, user behaviour, connected services, threats, and costs change. A launch evaluation cannot replace ongoing observation and change control.

Evaluate the complete system

Create a component and data-flow map, then enumerate outcomes and effects at the outer boundary. Test individual contracts—schemas, authorization, preprocessing, model behaviour, retrieval, tools—and the interactions between them. Include unavailable services, stale data, partial failures, retries, duplicated actions, and rollback.

Use production-like tasks and realistic users. Evaluate important slices and prohibited outcomes, not only averages. Capture enough version and trace evidence to reproduce a failure without storing unnecessary sensitive content.

Finally, test the operating model: monitoring detects the declared failures, an owner receives an actionable signal, a safe fallback exists, and a rollback or disable path works under time pressure.