What Is Top-K Sampling?
Top-k sampling truncates the next-token distribution to its k most probable candidates, rescales those to sum to one, and draws from them. The cutoff is a count, and it is the same count at every step — whether the step is one the distribution is nearly certain about, or one where the probability is spread across hundreds of plausible continuations.
A count, applied to a shape it never looks at
The rule runs after the distribution has been computed and it changes nothing about the ordering: the candidates are already ranked, and top-k keeps a prefix of that ranking. Because the prefix length is fixed, the amount of probability the surviving set carries is not, and that is the whole of the rule's behaviour. On a step where one continuation dominates, the k-th surviving candidate may carry so little that retaining it is indistinguishable from retaining noise — and unlike the leading candidate, it will occasionally be the one drawn. On a step where the distribution is genuinely undecided, the same k discards candidates that were in real contention, and does so at exactly the position in the text where the choice mattered most.
Both ends of the parameter are well defined and both are useful landmarks. At a k of one the rule keeps only the leading candidate, which is greedy decoding under another name: the same input tends to return the same output, and the sampler has stopped sampling. At a k as large as the vocabulary the rule has no effect at all, and the draw is made from the full distribution including the long tail of entries carrying almost nothing. Everything practical lies between those two, and where between them it lies is a property of a deployment rather than a number that transfers from one to another.
A second thing the parameter quietly buys is a bound on work. A cap on the candidate count bounds the size of the set that has to be selected out, rescaled and drawn from at every step, which is a smaller consideration than output quality but not nothing at high request volume. It is one reason a fixed cap has survived in samplers that also offer the adaptive rule, rather than being replaced by it.
| Situation | What each rule keeps |
|---|---|
| One candidate carries nearly all the probability | Top-k keeps k candidates regardless, so the surplus is tail. Top-p meets its threshold immediately and keeps very few. |
| The probability is spread across many candidates | Top-k keeps k of them and discards the rest wherever the boundary happens to fall. Top-p widens the set until the threshold is covered. |
| The cutoff is set to its minimum | Top-k at one candidate is greedy decoding. Top-p approaching zero reduces to the same thing, since one candidate is always retained. |
| The cutoff is set to its maximum | Top-k at the vocabulary size has no effect. Top-p at its ceiling has no effect either. |
| Both rules are enabled at once | A count cap and a mass threshold are applied in sequence, and the survivors are the intersection: whichever restriction is tighter decides that step. |
The first two rows are the whole argument for having both rules available. A fixed count is wrong in one direction on a confident step and wrong in the other direction on an uncertain one, and the two errors are not symmetrical — the first admits candidates nobody would defend, while the second removes candidates that were genuinely available.
Where the cap sits in a sampler, and what else is acting on the same numbers
Truncation is one operation among several applied to a single distribution between its computation and the draw, and the operations do not commute. Temperature rescales it; penalties intended to discourage repeating a token or a phrase adjust individual entries; a count cap truncates by rank; a mass threshold truncates by cumulative probability; and whatever survives is renormalised before the draw. Implementations differ on the order and on which of these are enabled by default, which is the practical reason a set of values copied from one runtime into another rarely reproduces the behaviour it produced there. The values were never describing the same pipeline.
The consequence that matters while debugging is that a symptom does not identify its parameter. Output that repeats itself is consistent with a cap tight enough that only one continuation survives, and equally consistent with a repetition penalty that is switched off, and equally consistent with a prompt that has left nothing to decide. Output that wanders is consistent with a cap wide enough to admit the tail, with a high temperature, and with a context that stopped constraining the subject some distance back. Changing one setting and reading the output is a weak instrument, and it is the instrument almost always used.
Finally, the cap is a filter on candidates and not a control on meaning. It cannot make a continuation more likely than the distribution already made it, and it cannot introduce a continuation that was not in the ranking. Whatever a particular k does to the character of the output, it does by removing options from the bottom of an ordering that had already been decided before the parameter was consulted at all.
How the Registry classifies a capped-pool generation run
Generating a batch of replies with the candidate pool capped at a fixed size and recording the variation across runs is filed as data processing in this registry's classification. Such a task consumes a configuration and produces measurements of it; nothing is authored for a reader, and the replies are the instrument rather than the deliverable.
The Registry's canonical brief for this entry is filed as:
Generate a batch of replies with the candidate pool capped at a fixed size and record the variation across runs.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
37c5cbfdab9c9c35…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about top-k sampling
Which value of k suits a deployment depends on that deployment's distributions, on what else in its sampler is enabled, and on the order those operations run in — none of which the canonical brief above fixes. This registry recommends no value, has compared no two settings, and operates no sampler of its own.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about top-k sampling
- What is top-k sampling?
- Top-k sampling truncates the next-token distribution to the k most probable candidates, rescales them, and samples one. The cutoff is a fixed count applied at every step, independent of how concentrated or how spread the distribution at that step happens to be.
- What does a k of one do?
- It keeps only the leading candidate, which is greedy decoding: the same input tends to return the same output and no sampling takes place. Variation remaining after that comes from elsewhere — batch composition, kernels, library versions — and not from the sampler.
- Why are top-k and top-p used together?
- Because they fail in opposite directions. A fixed count admits negligible candidates on a confident step and cuts genuine ones on an uncertain step; a mass threshold adapts, but can open very wide where the distribution is flat. Enabling both applies a ceiling on count and a floor on covered probability to the same set, and the tighter of the two governs each step.
- Does top-k affect determinism?
- Only at its minimum, where it removes the choice entirely. At any larger value a draw still takes place among the survivors, so repeated runs vary; a smaller k narrows how far they vary without making them identical.
- Is top-k applied before or after temperature?
- That depends on the implementation and is worth checking rather than assuming. Temperature commonly rescales the distribution first and truncation is applied to the result, in which case the two settings interact directly; some samplers order the operations differently, and a set of values does not carry between runtimes that disagree about the order.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.