What evidence should an LLM API emit—and what must never enter routine telemetry?

Shared system · deterministic provider fixtures

Harbor Support Drafting API. One bounded request moves through typed items, tools, state, budgets, and evidence without live provider calls.

01 · Smallest useful mechanism

Join least privilege, data governance, and semantic traces.

Production telemetry should connect user-visible outcomes to request versions, provider events, tools, policy decisions, usage, latency, and evaluations while minimizing sensitive content.

Log the decision structure and references; sample sensitive payloads only under an explicit policy.

02 · Experiment

Test the prediction

Deterministic protocol fixture

This workbench uses inspectable provider-event fixtures and injected failures. It does not make live calls or execute external effects.

Protocol workbench · harbor-trace-v1

Choose the trace that supports an incident review without copying customer content.

Injected failure

The draft cites the wrong policy and the debug logger captured full messages.

Choose the implementation

Log every request and responseKeep full prompts forever for maximum debuggability.Log only HTTP statusAvoid all semantic metadata.Trace decisions and referencesStore versions, categories, outcomes, IDs, timing, and controlled evidence references.

The trace has not run.

Commit to an implementation, then inject the failure and inspect the actual state transitions.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Define a trace schema, data-classification policy, redaction tests, access controls, and retention.
Break it on purpose: Enable full debug logging, mix tenant IDs, remove one policy span, and attempt trace access without incident role.
Completion evidence: Secrets and content are absent, tenant binding holds, missing evidence is detected, and access is auditable.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What belongs in routine semantic telemetry?

Primary sources · verified 2026-08-02

What the fixture does not prove

Without semantic traces incidents are guesses; with indiscriminate prompt logging the observability system becomes a high-value copy of private data and credentials.

Common mistake: Redacting API keys is enough to make full prompt logging safe. Prompts and tool results can contain personal, confidential, regulated, or cross-tenant data even without credentials.

Transfer exercise: Design a trace that proves which policy and tool result supported a draft without storing the customer’s message body.

OpenAI — Production best practicesOfficial operational guidance for API keys, staging, usage, limits, and production readiness.OpenTelemetry — Trace APIVendor-neutral specification for spans, context propagation, links, events, status, and trace lifecycle.Anthropic — How tool use worksOfficial explanation of client-executed and server-executed tools, call/result correlation, and the application-owned loop.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: The capstone combines the item contract, streaming, schemas, tools, state, budgets, and semantic evidence into one provider-portable service.