How should open-ended replies be judged?

Shared evaluation case · deterministic fictional data

Northstar Support Copilot. A fictional ecommerce support copilot retrieves versioned policies and synthetic customer records, drafts replies, and proposes sandboxed account actions. It cannot send or mutate anything without current authorization and exact human approval.

01 · Smallest useful mechanism

Turn preferences into observable criteria and calibrate the grader.

A defensible study randomizes answer order, blinds system identity, defines rubric criteria, trains annotators, records disagreement, and uses adjudication. Automated judges are compared with human labels on representative cases, including position, verbosity, and self-preference checks.

Human labels contain disagreement; model-judge labels contain systematic biases. Both need measurement.

02 · Experiment

Test the prediction

Deterministic fictional fixture

The calibrate subjective judges workbench uses versioned, inspectable teaching data. It does not claim to reproduce live customer traffic or model behavior.

Evidence workbench

Build the evaluation argument

Choose the controls required before using a model judge in a release scorecard.

Fixture northstar-judges-v1

Use observable rubric criteriaSeparate correctness, policy, uncertainty, empathy, and concision.Blind and randomizeHide system identity and randomize response order.Measure agreementReport rater and judge agreement with adjudicated examples.Run bias probesSwap position and vary irrelevant verbosity.Delegate safety to the judgeLet one model judge decide cross-account privacy compliance.Discard disagreementsRemove cases where humans and the model judge differ.

03 · Decision artifact

Write the evaluation argument another reviewer can audit

Use synthetic or properly governed data. Keep critical failures visible instead of compressing them into one score.

Claim
Name which qualities require judgment and which safety rules remain deterministic.
Evaluation design
Write a blinded pairwise protocol, annotation guide, adjudication rule, and bias probes.
Release evidence
Report label counts, agreement, disagreements, limitations, and model-judge false passes and failures.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

What is the strongest reason to swap answer order in a judge study?

Theory notes and sources

What the experiment does not prove

A judge can reward polished verbosity while missing a policy violation.

Common mistake: An LLM judge is scalable ground truth. It is another measurement instrument whose validity, reliability, and failure modes must be established for the task.

Transfer exercise: Swap answer order and shorten the better answer. Does the judge preserve its decision for the right reasons?

Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaPrimary research on human agreement and known position, verbosity, and self-enhancement biases.NIST GenAI Pilot Evaluation ReportAuthoritative resources on combining expert and human evaluation evidence.NIST AI RMF Playbook — MeasureGuidance on repeatable evaluation, production monitoring, documentation, and measurement limits.DeepEval — G-EvalOfficial documentation for a configurable model-judge metric; its thresholds and outputs still need task-specific human validation.MLflow — LLM judges and scorersOfficial implementation guidance separating code scorers from model judges and aligning versioned judges with domain-expert feedback.Artstein and Poesio — Inter-Coder Agreement for Computational LinguisticsA peer-reviewed survey of agreement coefficients, their assumptions, and the limits of interpreting human annotation reliability.

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: A calibrated judge covers known quality criteria; red-team evaluation now probes where the system and its controls break under pressure.