What Is a Golden Dataset?
A golden dataset is a fixed, curated set of inputs paired with reviewed statements of what a correct response to each one looks like, held under version control and re-run against every candidate change. It is a regression suite for a system whose outputs are open-ended, and the word carrying the weight is reviewed: the expectations are ones somebody entitled to say what correct means has actually read and signed off on.
What a row holds, and why the expectation is rarely a string
Exact-match comparison, which is what a golden file gives in ordinary software testing, mostly does not survive contact with generated text. Two responses can differ in every word and be equally correct, and a response can match a reference closely while being wrong in the one clause that mattered. So a row's expectation is usually a criterion rather than an output: facts the response must contain, content it must not contain, a reference answer to be compared for equivalence rather than for identity, a schema it has to parse against, or a short rubric applied by a reviewer or a judging pass. Where the output genuinely is closed — a classification label, an extracted field, a tool call and its arguments — exact matching returns, and those rows are the cheapest and the most trustworthy in the set.
The rows that earn their place carry two further things that are easy to omit and expensive to reconstruct. One is provenance: where the input came from, and who decided what the expectation should be, because six months later the only way to settle an argument about a row is to ask the person who wrote it. The other is the reason the row exists at all. Most rows in a healthy set were added because something failed once in production, and a row that records which failure it guards against can be retired deliberately when that failure stops being reachable. A row with no stated reason cannot be retired at all, only accumulated.
Size is a design choice rather than a target, and it is set from the opposite end to a benchmark's. The set has to stay small enough that a person can read all of it and still believe every expectation in it, because from the moment nobody has read the whole thing the suite is asserting things nobody is prepared to defend. Coverage is bought by choosing rows that sit on distinct behaviours, not by adding more rows of a shape already present.
| Field | What it is there for |
|---|---|
| The input | The request exactly as the system will receive it, including whatever context the real path would have supplied. |
| The expectation | A criterion for a correct response: required facts, forbidden content, a reference to compare against, a schema, or a rubric. |
| The grading method | How the expectation is applied — exact match, a schema check, a judging pass, or a person. |
| Provenance | Where the input came from and who approved the expectation, so that a disputed row has an owner. |
| The reason | What this row guards against, which is what makes it possible to retire the row rather than keep it forever. |
| Last review | When a person last confirmed that the expectation still describes the behaviour the product wants. |
The last two fields are the first to be dropped and the ones most missed. Without them a suite has no mechanism for shrinking, so it grows until it is slow enough that people stop running it on every change — at which point it has quietly become documentation.
The four ways a golden dataset stops being true
It goes stale by product change. An expectation is a statement about what the system should do, and a product's intentions move. A suite unreviewed since a behaviour was deliberately changed reports failures for the new and correct behaviour, and the quickest way to make that stop is to loosen the expectation — which is the first step of a suite becoming one that passes everything.
It leaks. The moment a row's input or its expected answer is pasted into a prompt as a worked example, added to the retrieval store, or used to tune the thing being tested, that row has stopped measuring generalisation and started measuring recall. It is the contamination problem a public benchmark has, arriving by a much shorter route, and it is invisible in the results: a contaminated suite looks exactly like a suite that is passing.
It drifts toward the easy. Rows that fail are uncomfortable and get discussed; rows that pass are quiet. Across enough revisions a set curated by people under delivery pressure loses its hardest cases first, through a sequence of individually reasonable decisions about rows that were not representative. The countermeasure is procedural rather than technical: retiring a row is a decision that gets recorded and reviewed like any other, rather than an edit.
And it is not a benchmark, which is the confusion that does the most damage once a number leaves the room it was produced in. A golden dataset is small, private, chosen to reflect one product's own traffic, and graded by criteria that product's team wrote. A score on it compares two versions of that product and nothing else. It does not rank configurations against one another, it does not transfer to another team's system, and it cannot support a claim about anything beyond the change it was run on.
How the Registry classifies a regression evaluation task
Assembling a fixed set of inputs with reviewed expected outputs and re-running it against each candidate change is filed as data processing in this registry's classification. Nothing is authored in any meaningful sense — inputs are collected and expectations are decided — and what a run produces is a comparison between two states of the same system.
The Registry's canonical brief for this entry is filed as:
Assemble a fixed set of inputs with reviewed expected outputs and re-run it against each candidate change.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
c604db46234735ad…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about golden dataset
What belongs in a golden dataset, how a response is graded against a row, and when an expectation ought to be rewritten are decisions belonging to whoever owns the product under test, and the canonical brief above encodes none of them. This registry curates no evaluation set for anyone else's system, grades no third party's outputs, and publishes no golden dataset of its own. It holds none against which the assessments it issues could be scored either, since it does not check them against what afterwards happened.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about golden dataset
- What is a golden dataset?
- A golden dataset is a fixed set of inputs paired with reviewed expectations of a correct response, held under version control and re-run against every candidate change. It is the regression suite for a system whose outputs are open-ended rather than exactly comparable.
- How is a golden dataset different from a benchmark?
- A benchmark is public, standardised, and built to compare different systems. A golden dataset is private, small, and built to compare two versions of one system against expectations its own team wrote. A score on it can support a release decision; it supports no claim about anybody else.
- How are expected outputs written when the output is free text?
- Usually as criteria rather than as strings: required facts, forbidden content, a schema the response must satisfy, a reference answer compared for equivalence, or a rubric applied by a reviewer or a judging pass. Exact matching is kept for the rows where the output really is closed, such as a label or a tool call.
- How large should a golden dataset be?
- Small enough that a person can read every row and still believe every expectation in it. There is no correct number and this registry publishes none; the useful constraint is that an unread suite asserts things nobody is prepared to defend.
- How does a golden dataset go stale?
- Four ways, and they compound. The product changes and the expectations no longer describe what is wanted. Rows leak into prompts, retrieval stores or tuning data and stop measuring generalisation. The hardest rows are retired one reasonable decision at a time. And the reviewers who knew why a row exists move on, leaving expectations that nobody can explain or safely delete.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.