How do you build an evaluation dataset?

A case is more than a prompt

An evaluation case describes the condition under which a claim will be checked. For a simple classifier that may be an input, expected label, and slice. For an agent it can include initial files or records, user goal, available tools, identities and permissions, permitted and prohibited effects, resource budget, expected terminal state, and grader rubric.

Store enough provenance to answer:

• where the case and labels came from, under what permission;
• when the underlying event occurred and when the case was authored;
• which transformations, translations, redactions, or synthetic generators were used;
• which people or systems labelled and adjudicated it;
• what population, task, severity, and other declared slices it represents;
• where the case has been published, retrieved, trained on, or shown during development.

Without that record, an exact score may be impossible to interpret or reproduce.

Keep different datasets for different jobs

flowchart LR
  P[Product claims and risks] --> D[Development set]
  P --> H[Protected held-out set]
  P --> F[Fresh and time-split set]
  I[Confirmed incidents] --> R[Regression suite]
  D --> T[Fast iteration]
  R --> T
  H --> E[Release estimate]
  F --> E

A development set gives fast feedback while prompts, tools, and code change. Engineers are expected to see it, so improvement on it eventually says less about unseen cases.

A regression suite freezes confirmed failures and contractual examples. It answers whether known behaviour stayed fixed. It grows useful institutional memory, but it is not a fresh estimate of general capability.

A protected held-out set is hidden from the people and automated loops optimizing the system. It supports a release comparison only while access and reuse remain controlled.

A fresh or time-split set uses evidence created after the training or tuning boundary when possible. It is especially useful when public benchmarks, changing traffic, or temporal knowledge make exposure likely.

The same case must not quietly play all four roles. Record promotion and access events. If a held-out failure becomes a debugging example, move it into the regression suite and replace the lost independent evidence.

Sample from claims, not convenience

Begin with the decisions and failures that matter. Include representative traffic, but do not let common easy requests erase rare severe risks. Define slices before looking at candidate scores: language, region, user group, task, input length, ambiguity, data source, tool, permission level, risk, and any product-specific condition that could change performance.

Random splitting is often unsafe when near-duplicates, users, documents, repositories, conversations, or time periods cross the boundary. Group related examples before splitting. Use time-based separation where future traffic matters. Deduplicate semantically as well as exactly, and inspect cluster boundaries rather than trusting one similarity threshold.

Synthetic cases can fill sparse boundaries and inject controlled failures. They do not prove production representativeness. Preserve the generator model, prompt, seed or sampling settings, filtering, and human review; keep synthetic and observed results visible as separate slices.

Labels also have uncertainty. Use an explicit rubric, examples, qualified reviewers, blinding where relevant, and adjudication for disagreements. Keep the original labels and disagreement rather than rewriting history after seeing model output. A broken reference answer is a dataset failure, not a model failure.

Contamination has several paths

Evaluation contamination means information from the evaluation can influence the system being measured. Exact test examples in pretraining data are only one path. Exposure can come through supervised or preference training, prompt examples, retrieval indexes, tool-accessible websites, fine-tuning after an incident, model-judge prompts, generated solutions, public leaderboards, or repeated human optimization against a fixed set.

Contamination does not prove that every answer was memorized, and the absence of an exact string match does not prove independence. Paraphrases, explanations, answer keys, code solutions, and task templates can transfer useful information. Closed model training data may make certainty impossible, so report exposure risk and detection limits rather than claiming “clean” without evidence.

Useful controls include training-data and retrieval-corpus overlap checks, grouped deduplication, access logs, private test items, post-cutoff or time-split cases, canary strings, and comparison with matched fresh tasks. Behavioural probes such as reconstructing unusual benchmark text can raise suspicion, but they are not a universal contamination detector.

Report the denominator and conditions

Publish counts, not only percentages: total cases, attempted trials, grader errors, exclusions, abstentions, failures, and severe events. Report slice sizes and uncertainty. For variable generation, use repeated paired trials on the same cases and show how often each system wins, fails, or disagrees—not just a difference between two rounded averages.

Freeze the evaluation manifest: dataset version and content hash, split, exclusions, case renderer, prompt and few-shot examples, model and decoding settings, tool environment, grader versions, trial count, and code revision. A named benchmark without these conditions is not a reproducible experiment.

A benchmark is not the product

An evaluation dataset is not a bag of impressive prompts, a public benchmark score, or a copy of recent production logs. More cases do not repair biased sampling, wrong labels, hidden duplicates, missing system state, or a grader that cannot observe the declared outcome.

“Held out” does not mean never seen by anyone. It means protected from the specific optimization process whose generalization you want to estimate. Once engineers, agents, prompts, training pipelines, or retrieval tools can use the answers, that protection has changed and must be recorded.

Audit the evaluation set

Test the dataset pipeline like production data code. Validate required fields, types, identities, permissions, temporal boundaries, label ranges, and referential integrity. Hash versioned case payloads. Fail on duplicated case IDs, unexpected split movement, missing provenance, and unreviewed transformations.

Run exact, normalized, and semantic-neighbour searches within and across splits, training examples, prompt fixtures, retrieval corpora, regression suites, and any public source you are authorized to inspect. Review the nearest matches manually. Seed known duplicates and paraphrases to measure whether the checks can find them.

Audit the labels. Blind-review random cases and every severe or disputed case. Measure agreement by slice, inspect systematic disagreement, execute expected code or tool outcomes where possible, and test graders on known passes, failures, malformed evidence, and abstentions.

Finally, rehearse the lifecycle: grant and revoke access, promote one held-out failure into the regression suite, replace it with fresh evidence, delete a case whose source permission changes, and reproduce an older report from its manifest. The dataset is trustworthy when you can explain exactly what it measures, who could have learned from it, and what remains unknown.