What Is an LLM Benchmark?
A benchmark is a fixed dataset paired with a fixed scoring rule, applied under identical conditions. A benchmark score is therefore a measurement of one thing under one procedure, and it describes what that procedure measured and nothing wider.
The parts of a benchmark, and where results diverge
The dataset fixes only the items. Everything else that decides the number belongs to the protocol wrapped around it: the prompt each item is presented with, the way an answer is extracted from the response, and the rule that scores what was extracted. Two evaluations of the same configuration on the same dataset routinely disagree, and the disagreement is almost never in the items.
Extraction is the least discussed and most consequential. Whether a response is scored by exact string match, by parsing its final line, by a regular expression, or by another system reading it changes the number without anything about the configuration changing. So do the decoding settings used to produce the responses in the first place, which is why reported settings are part of a result rather than a footnote to it.
Two further parts of the protocol are routinely omitted from the headline. The first is how many worked examples were placed in the prompt: a run with several examples in context and a run with none are two different experiments carried out on the same items. The second is how many times each item was attempted. Where responses are sampled rather than taken greedily, one pass is one draw, and a comparison between two single passes cannot separate a difference in behaviour from a difference between draws. A result reported without a spread is a result whose precision is unstated.
| Component | Why it moves the number |
|---|---|
| The item set | What is being asked, and whether it resembles the intended use. |
| The prompt template | The same question phrased two ways is two measurements. |
| Answer extraction | How a free-form response is reduced to something scorable. |
| The scoring rule | Exact match, partial credit, or a judgement made by another system. |
| Decoding settings | Sampling parameters, which change the response before it is ever scored. |
| Examples in context | How many worked examples preceded the item. A run with them and a run without are separate experiments. |
| Attempts per item | Whether the item was tried once or repeatedly, which decides whether a spread can be reported at all. |
A result published without these is not reproducible, which is a statement about the publication rather than about the configuration it describes.
Contamination, judges, and the gap between a score and a job
The structural problem with a public benchmark is that it is public. Once a dataset is on the open web it can appear in a later training corpus, and a score on a contaminated item measures recall of the item rather than the ability the item was chosen to probe. Held-out sets, private splits and rotating items are all responses to this. None of them settles it permanently, because the response is published too.
A further dependence arrives whenever the scoring rule is itself a language system reading the answer. That arrangement is what makes free-form output scorable at all, since no exact-match rule can mark an essay, and it moves the result from a property of one configuration to a joint property of two: the one under test and the one doing the marking. Replacing the marker moves every score it produced, and how the marker itself behaves on the marking task is a separate question the benchmark does not ask.
The last gap is the ordinary one between any test and any job. A benchmark is a proxy chosen because it is measurable, and its relationship to a specific production task is an assumption made by whoever cites it. An aggregate over a mixed item set is also an average across sub-populations, so two results that meet at the top line can disagree on every group inside it. The dependable way to know how a configuration behaves on your task is an evaluation set built from your task.
How the Registry classifies a benchmark run
Running a fixed dataset against a candidate configuration and recording every score in a results table is filed as data processing in this registry's classification: material is consumed, quantities are derived, and a table is what the task produces.
The Registry's canonical brief for this entry is filed as:
Run a fixed benchmark dataset against a candidate configuration and record every score in a results table.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
945809740dcc7a8c…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about llm benchmark
This registry publishes no benchmark, cites none, and scores nothing. It does not run the configurations in its own list against any dataset, and the reason is stated openly at /method: the method has no evaluation step in it.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about llm benchmark
- What is an LLM benchmark?
- A benchmark is a fixed dataset and a fixed scoring rule applied under identical conditions, so that results can be compared. It specifies the items, the prompt template, how an answer is extracted from a response, and how that answer is scored.
- What is benchmark contamination?
- Contamination is the presence of benchmark items in a training corpus. A score on a contaminated item reflects recall of that item rather than the ability the item was chosen to test, and because public datasets are public, it accumulates over time rather than being fixed once.
- Why do published scores for the same configuration differ?
- Because the score depends on the prompt template, the answer-extraction rule, the scoring rule and the decoding settings, none of which are fixed by the dataset alone. Two evaluations that differ in any of them are measuring two different procedures.
- What does it mean when another language system does the scoring?
- It means the marking is done by reading rather than by matching, which is the only way free-form answers can be scored at scale. It also means the result depends on two configurations rather than one: the one being tested and the one marking it. Changing the marker changes every score it produced, and the marker’s own behaviour on the marking task has to be established separately.
- Should I build my own evaluation set?
- Where a decision depends on how something behaves on your task, yes: a public benchmark is a proxy chosen for measurability, and the link between it and your workload is an assumption. A small set of your own cases with a scoring rule fixed in advance answers the question a public score only gestures at.
Related entries
Fine-tuning · Chain-of-thought prompting · Prompt engineering
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.