How should an AI system handle uncertainty?

Separate the signals

A language model predicts a distribution over the next token given its context. A high probability for one token means the model strongly preferred that continuation at that position. It does not directly mean that the completed answer is factually correct, that cited evidence supports it, or that an external operation will succeed.

Several things called “confidence” need separate names:

• token probability: preference among possible next token IDs;
• sequence score: an aggregate over token probabilities, strongly affected by wording and length;
• semantic consistency: whether repeated samples converge on the same meaning;
• self-report: text such as “I am 90% confident,” generated through the same language mechanism;
• grader score: a separate instrument applying a rubric to visible evidence;
• empirical calibration: how often a defined prediction was correct among comparable cases assigned a given score;
• system confidence: an application decision combining evidence, dependencies, policy, and cost of error.

Fluency is not on the list. A polished answer can be unsupported, while a hesitant style can accompany a correct one. The harness should not infer reliability from tone.

Calibration is a population property

Suppose a classifier emits 0.7 for many comparable cases. It is calibrated at that level if roughly 70% of those cases are correct under the declared label and data distribution. Calibration does not say which individual case is correct, and it does not transfer automatically to a new language, user group, task, model revision, or traffic shift.

flowchart LR
  S[Raw signal] --> C[Calibration on held-out labels]
  C --> P[Estimated risk for a declared slice]
  P --> D{Policy threshold}
  D -->|low risk| A[Answer or act]
  D -->|unclear| V[Retrieve, verify, or ask]
  D -->|high risk| H[Abstain or escalate]

For fixed-label tasks, reliability diagrams compare predicted score bands with observed correctness. Proper scoring rules such as log loss or the Brier score reward useful probabilities and penalize confident errors. A single expected-calibration-error number can hide small severe slices, so keep the underlying counts and plots.

Free-form generation is harder. Many token sequences express the same answer, and longer sequences naturally accumulate lower probability. Sampling several answers and grouping them by meaning can reveal semantic disagreement; research on semantic uncertainty shows this can detect some confabulations. It still does not detect every error—several samples can agree on the same false claim.

Locate what is uncertain

An application may face different uncertainties:

• the request is ambiguous or missing constraints;
• relevant knowledge is absent, stale, or outside the model;
• retrieved sources conflict or do not support the claim;
• a tool result is missing, malformed, or from an unknown state;
• the model produces several incompatible interpretations;
• the request differs from the calibrated evaluation population;
• a grader is unavailable or disagrees with deterministic evidence;
• the real-world effect is irreversible or has an unknown outcome.

These conditions call for different responses. Ask a clarifying question for ambiguity. Retrieve current evidence for missing knowledge. Re-run an idempotent read when a tool result is unavailable. Require approval for consequential effects. Escalate a high-risk domain decision. Preserve “unknown outcome” after an ambiguous write rather than retrying blindly.

Abstention belongs to the system contract

Abstention means deliberately not making a claim or effect under specified conditions. It can be a refusal, a request for missing information, a limited answer with named uncertainty, a handoff to a person, or a safe no-op. Define which form applies and what the user can do next.

Thresholds should reflect consequences. A slightly uncertain movie recommendation and an uncertain medication dose cannot share one policy. Compare at least three rates by slice: correct actions, harmful or incorrect actions, and abstentions or escalations. Increasing abstention can raise accuracy among answered cases while making the product useless; reducing it can improve completion while increasing severe false actions.

Evaluation metrics influence behaviour. A benchmark that scores only exact correct answers and treats abstention exactly like a wrong answer rewards guessing whenever there is any chance of success. Score confident errors, useful abstentions, and unnecessary refusals in a way that matches the actual decision cost.

Do not delegate the final decision to a prompt saying “answer only if confident.” The model can contribute a signal, but the harness owns evidence checks, calibrated thresholds, permissions, routing, and the allowed terminal states.

Communicate without false precision

Show users what matters: the evidence used, its date and scope, unresolved conflict, assumptions, actions taken, and a clear next step. A naked “87% confidence” suggests a universal frequency unless it was calibrated for that exact kind of claim and population.

Prefer statements such as “The retrieved policy does not cover contractors, so I need the employment type” or “The write timed out after submission; its outcome is unknown, so I will check the transaction before retrying.” These expose the missing evidence and safe response rather than imitating a feeling.

Confidence is not permission

Uncertainty is not one scalar stored inside a language model. Entropy is not the probability that a sentence is false. Repeated agreement is not independent verification. A model saying “I don't know” is not automatically calibrated, and a citation is not evidence until the cited material is retrieved and shown to support the claim.

Calibration is also not permanent certification. It depends on labels, scoring rules, slices, thresholds, and the system version. Change the model, prompt, tools, context construction, or population and the evidence must be checked again.

Evaluate abstention as a decision policy

Build labelled cases spanning easy, hard, ambiguous, unanswerable, conflicting, stale, out-of-distribution, and adversarial inputs. Include missing tools, malformed results, insufficient permissions, and unknown write outcomes. Define acceptable answer, verification, clarification, abstention, and escalation states before running the system.

For numeric scores, calculate reliability by score band and important slice; report sample counts, Brier or log loss, false-confidence events, and coverage-versus-risk curves. Choose thresholds on protected evidence, then evaluate once on a separate held-out set. Recheck after material version or traffic changes.

Probe superficial invariance. Paraphrase questions, alter tone, reorder irrelevant details, translate equivalent cases, and repeat generations. Confidence should not change merely because an answer is longer or more assertive. Insert false citations and confident model-generated rationales to confirm that grounding checks remain independent.

Finally, test the policy outcomes. Every high-impact confident error deserves review. So does every unnecessary refusal in an essential workflow. Simulate unavailable uncertainty services and malformed scores; the application should fall back to a declared safe state rather than treating missing confidence as permission to proceed.