How should an application call an LLM API?

Define the request manifest

An SDK method can make a model call look local, but it crosses a network into a separately deployed service. Record the contract that produced each result:

• provider, endpoint, API and SDK revision;
• exact model snapshot when the provider exposes one, or the declared moving alias;
• developer instructions, input items, ordering, and context-construction revision;
• generation controls, reasoning or compute setting, stop conditions, and token limits;
• tool definitions, structured-output schema, and parallel-call policy;
• request deadline, retry policy, service tier, and data-handling mode;
• application trace ID, provider request ID, tenant, and safe diagnostic metadata.

Do not store sensitive content merely to make the manifest complete. Hash or version stable prompt assets, redact exported telemetry, and retain content only under a declared access and retention policy.

Moving aliases are convenient for development but can change behaviour without an application deploy. Pinned snapshots make comparison and rollback easier, while still leaving other provider-side dependencies outside your control. Whichever you use, expose it in evaluation and production evidence.

Model the terminal states

“Did not return text” is not one error. The client needs distinct states for:

• complete response with a declared finish or stop reason;
• valid but incomplete response because of a token, content, or time limit;
• provider rejection or policy refusal;
• invalid request, authentication, or permission failure;
• throttling or temporary capacity failure;
• provider or transport error before a usable response;
• caller cancellation or local deadline;
• malformed event or output that violates the application contract;
• lost connection after some response or job state may exist.

Preserve the provider's request ID, status, structured error, usage, and incomplete reason. Do not turn an unknown code into an empty successful answer. The harness can then decide whether to repair input, wait, retry, fall back, ask the user, or stop.

flowchart TD
  Q[Versioned request] --> A[One API attempt]
  A --> C{Classified result}
  C -->|complete| V[Validate output]
  C -->|permanent| F[Fail or repair request]
  C -->|transient and budget remains| B[Backoff plus jitter]
  B --> A
  C -->|ambiguous| R[Reconcile by request or job ID]
  C -->|deadline or cancellation| S[Stop safely]

Streaming changes delivery, not correctness

Streaming reduces time to first visible output by delivering typed events before the response is complete. It does not make total generation instantaneous or guarantee that an early fragment is valid.

Parse the provider's event protocol rather than splitting arbitrary bytes or lines. Accumulate text and tool arguments according to event type. A JSON prefix is not JSON, and a partial tool call must never execute. Commit a structured object only after the schema-valid terminal event. Preserve the final stop reason and usage event when available.

The user interface may render provisional text, but it should distinguish it from a completed artifact. If moderation, citation checks, or other policy applies to the complete result, choose whether to buffer, delay effects, or visibly retract. Cancellation stops the client's interest; it does not prove the provider stopped computation immediately.

If a stream disconnects, replaying the same prompt can produce a different continuation and another billable request. Decide whether a fresh answer is acceptable, whether a background response can be retrieved by ID, or whether the operation should end as incomplete.

Retry a classified operation at one layer

Retries are useful for transient transport failures, throttling, and some server errors. They are harmful for invalid schemas, unsupported parameters, denied authorization, exhausted quota that will not recover within the deadline, or prompts that repeatedly violate a policy.

Put the retry policy at one owning layer. SDK, service, queue worker, agent harness, and reverse proxy retries can multiply into a storm. Bound attempts and total elapsed time, use exponential backoff with random jitter, respect an explicit server retry delay, and preserve budget for the caller to recover. A timeout must include connection, response, and stream-read behaviour rather than relying on an unbounded default.

HTTP idempotency describes whether repeating the intended request has the same intended effect. A model-generation request usually does not mutate the user's business data, but repeating it can consume cost, allocate provider jobs, create stored responses, and yield different output. Use a provider-supported idempotency key or job identity where available; otherwise make duplication an explicit part of the contract.

Most importantly, separate model generation from tool execution. Retrying a model call may create another proposal. It must not automatically replay a purchase, message, file write, or deployment. External effects need their own idempotency key, authorization decision, effect record, and reconciliation path.

Validate beyond the schema

Structured output constrains shape; it does not establish truth, authorization, freshness, or domain correctness. Validate JSON syntax and schema first, then semantic constraints, references, permissions, and policy. Unknown enum values and missing required fields should fail closed rather than being silently defaulted.

Budget input before the call. Count or estimate tokens with the correct tokenizer when available, but leave headroom for provider framing, tools, images, and output. Decide explicitly whether oversize context is rejected, summarized, retrieved selectively, or truncated. Silent truncation can remove instructions or evidence while leaving a plausible answer.

Rate limits can apply to requests, input and output tokens, concurrent work, or batch queues. Admission control and bounded queues are more reliable than accepting unlimited work and hoping retries clear it. Treat cost and latency as per-request budgets and aggregate service constraints.

Transport success is not task success

An LLM API is not a deterministic function call. A successful HTTP status does not make the answer correct, a valid schema does not make it safe, and a provider request ID does not replace the application's trace and outcome record.

Streaming is not a queue, retry is not recovery, and temperature zero is not an exactly-once guarantee. Provider model names, capabilities, prices, and limits are operational inputs that can change; current documentation must be checked rather than copied forever into application assumptions.

Exercise the API contract under failure

Use a fake server or transport to emit every terminal state and event sequence: connection failure, delayed headers, mid-stream disconnect, duplicate events, malformed frames, partial UTF-8, missing terminal event, incomplete output, throttling with and without retry hints, permanent errors, cancellation, and a response arriving just after the local deadline.

Assert the exact attempt count, total deadline, backoff range, jitter injection, retry owner, request identity, and final application state. Freeze randomness and time in unit tests. In integration tests, verify SDK defaults rather than assuming them, and confirm sensitive content is absent from logs and metrics.

Contract-test structured responses with valid, invalid, truncated, oversized, and semantically impossible objects. Test unknown model IDs, moving-alias changes, context overflow, tool-schema revisions, and output limits. A fallback model must pass the same output and safety contract; “smaller” or “available” does not imply compatible.

Finally, simulate a model proposal followed by a timed-out external write. The system should reconcile the effect by its own idempotency key before any retry. This single test distinguishes a resilient agent harness from one that converts ordinary network ambiguity into duplicate real-world actions.