How do you monitor an AI system in production?

Observe one request as a system

A production answer is the end of a path, not a single model call. Record enough structured evidence to connect the request, application revision, prompt or policy revision, model and parameters, retrieved material, tool proposals and decisions, external operations, final response, latency, usage, and outcome.

flowchart LR
  R[Request] --> C[Context and retrieval]
  C --> M[Model calls]
  M --> T[Tool and policy events]
  T --> O[Response and effects]
  O --> D[Delayed outcome]
  R -. trace ID .-> D

Use a trace ID to preserve this relationship across components and retries. Keep the event vocabulary stable and versioned: operation names, outcome classes, model identity, data-source identity, stop reasons, grader revisions, and error categories should mean the same thing across releases. Emerging conventions such as OpenTelemetry's generative-AI attributes can reduce private naming, but their stability level must be checked before treating them as a frozen contract.

Content is different from metadata. Prompts, outputs, retrieved passages, tool arguments, and results can contain credentials, personal data, proprietary code, or adversarial text. Default to structured metadata, explicit redaction, access controls, short retention, and sampled content only when a declared diagnostic or evaluation need justifies it. Observability must not quietly become a second ungoverned memory system.

Measure four layers

Service health asks whether the software is available: request volume, error class, timeout and retry counts, queue time, model and tool latency, token use, cache behaviour, and cost. These signals are necessary, but a fast incorrect answer is still incorrect.

System behaviour asks what path the system took: retrieval misses, malformed model output, denied tool calls, approval requests, repeated operations, budget exhaustion, cancellation, fallback use, handoff, and stop reason. For agents, count real effects separately from model-proposed actions.

Product outcomes ask whether the system helped: accepted suggestions, corrected fields, resolved tasks, repeated attempts, escalation, abandonment, rollback, support reports, or domain-specific results. Instrument the nearest trustworthy outcome rather than substituting response length or user clicks for quality.

Sampled quality and risk apply versioned graders and human review to representative traces. Include important user, language, task, tool, and risk slices. Preserve pass, fail, abstain, unavailable, and disagreement separately. Averages can hide a severe regression in a small group.

Each alert needs an owner and an action. Page on conditions that require timely intervention, such as forbidden effects, cross-tenant access, or a sustained inability to complete a critical workflow. Put diagnostic trends on dashboards or review queues. If nobody can say what a threshold changes, it is not yet a useful alert.

Drift names a symptom, not a cause

“The model drifted” is usually too vague to debug. At least five different changes can alter production results:

• traffic drift: requests, languages, users, or task difficulty changed;
• data drift: documents, indexes, schemas, features, or labels changed;
• dependency drift: a model alias, API, tool, policy service, or external site changed;
• system drift: prompts, harness code, permissions, retrieval, or graders changed;
• outcome drift: the relationship between observed inputs and the desired result changed.

A distribution change is not automatically harm, and stable input statistics do not prove quality. Localize the first changed boundary. Compare manifests, slices, trace events, retrieved evidence, and delayed outcomes. Replay a privacy-safe sample against the previous system when possible. Roll back or constrain the affected component when the evidence warrants it; do not retrain a model merely because a dashboard moved.

Feedback can also change the traffic you later observe. Recommendations affect what users select, agents change files and records that become future context, and human overrides may enter later training sets. Keep the original proposal, policy decision, applied effect, user correction, and eventual outcome distinct so that the system does not learn from its own unlabelled behaviour as if it were truth.

Turn operations into evaluation evidence

Production monitoring and offline evaluation form a loop:

1. detect a reproducible failure, disagreement, or new task slice;
2. preserve the smallest authorized evidence needed to understand it;
3. remove or transform sensitive data according to policy;
4. write a case with initial state, expected invariants, and graders;
5. verify that the case fails on the affected version;
6. fix the system and run both the focused case and broader held-out suites;
7. deploy gradually and watch the same production signal.

Do not add every production transcript directly to a benchmark. Repeatedly tuning against visible incidents creates a regression suite, not an independent estimate of generalization. Maintain held-out and time-split evidence, and record case provenance so training or prompt examples cannot silently overlap evaluation data.

A dashboard is not a response system

AI observability is not a dashboard of token counts, a warehouse of every prompt, or a single model-judge score. Logs explain what the instrumented system recorded; they do not prove unobserved real-world outcomes. A trace can be complete and still use a wrong grader, omit an external effect, or expose sensitive information.

Drift is not synonymous with model-weight change. Hosted weights can remain fixed while the prompt, context, traffic, tools, or provider-side serving stack changes. Conversely, a new model can preserve product quality. Version manifests and outcome evidence are more useful than attributing every movement to the model.

Prove detection, diagnosis, and rollback

Before release, send synthetic canary requests through the complete telemetry path. Assert that trace relationships survive streaming, retries, fallbacks, tool calls, cancellation, and asynchronous outcomes. Verify every manifest field, unit, clock, outcome class, and sampling decision. Unknown model revisions and missing terminal events should be visible, not coerced into a normal success.

Test privacy controls with secrets, personal data, proprietary text, tool arguments, retrieved passages, and prompt injection. Confirm redaction happens before export, access is least-privileged, retention expires, deletion propagates, and low-volume sensitive fields cannot be reconstructed through high-cardinality labels. Test that disabling content capture leaves operational telemetry useful.

Backtest alerts on known incidents and normal variation. Inject model timeouts, retrieval staleness, tool failures, malformed output, permission denials, judge outages, duplicate effects, traffic shifts, and delayed labels. Check detection latency, false alarms, routing, runbooks, rollback authority, and whether a monitoring failure is itself monitored.

Finally, run a shadow or canary comparison for material releases. Break results down by declared slices and severity, inspect disagreements, and keep an explicit rollback condition. The monitoring system is working when an engineer can move from an alert to the exact affected requests, system manifest, evidence, decision, and safe response—not merely when a chart looks unusual.