How do you keep an LLM API responsive without multiplying load or effects?

Shared system · deterministic provider fixtures

Harbor Support Drafting API. One bounded request moves through typed items, tools, state, budgets, and evidence without live provider calls.

01 · Smallest useful mechanism

Design one end-to-end budget and spend it deliberately.

Production clients need admission control, concurrency limits, deadlines, error classification, exponential backoff with jitter, idempotency, caching, and observability of tokens and tool work.

A retry that starts after the user deadline is load, not reliability.

02 · Experiment

Test the prediction

Deterministic protocol fixture

This workbench uses inspectable provider-event fixtures and injected failures. It does not make live calls or execute external effects.

Protocol workbench · harbor-budget-v1

Choose the policy for a 429 burst with three seconds left in the deadline.

Injected failure

Ten clients receive the same retry-after value while one effectful call has an unknown outcome.

Choose the implementation

Retry everything immediatelyEach layer makes three attempts independently.Wait a fixed two secondsRetry every operation together once.Admit and retry by budgetJitter safe reads, reconcile the write, and shed work that cannot finish.

The trace has not run.

Commit to an implementation, then inject the failure and inspect the actual state transitions.

03 · Implementation brief

Build it, break it, prove the outcome.

Implement: Add one shared limiter, end-to-end deadline, retry classifier, and usage ledger.
Break it on purpose: Trigger synchronized 429s, timeout after dispatch, queue saturation, and an oversized prompt.
Completion evidence: Retries remain bounded, unsafe effects reconcile, low-priority work sheds, and metrics show amplification and cost.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

When is an automatic retry defensible?

Primary sources · verified 2026-08-02

What the fixture does not prove

Independent retries at several layers create storms, inflate cost, duplicate effects, and make tail latency worse precisely when a provider is degraded.

Common mistake: More retries always improve success rate. Retries help only transient, safely repeatable operations within the remaining deadline and capacity budget.

Transfer exercise: Run twenty requests into a five-slot queue and compare immediate retry, jittered retry, and admission control during a simulated 429 burst.

OpenAI — Rate limitsOfficial limit model and retry guidance. Product-level deadlines, idempotency, and admission control remain application concerns.OpenAI — Prompt cachingOfficial description of cache eligibility and usage reporting for repeated prompt prefixes.OpenAI — Production best practicesOfficial operational guidance for API keys, staging, usage, limits, and production readiness.RFC 9110 — HTTP SemanticsStandards-track definitions for methods, status codes, retry-relevant semantics, and idempotency.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: Budgets keep the service alive; security and observability make its behavior reviewable without leaking the data it handles.