What Is Speculative Decoding?
Speculative decoding is a serving technique that settles several token positions per pass over a large network instead of one. A cheap drafting step proposes a short run of candidate tokens; the large network then scores every one of those positions in a single forward pass; a verification rule keeps the longest leading run that agrees with what the large network would have produced by itself, discards the remainder, and the next round starts from the end of what was kept.
Why one pass can carry more than one token
The trick exists because a decoding step is limited by memory movement rather than by arithmetic. Producing a single token requires reading every weight of the network out of memory and into the compute units, and the arithmetic that follows occupies those units for a fraction of the time the read took. Adding further positions to the same pass costs very little on top: the weights are read once and applied to all of them together. A speculative round therefore buys its tokens where they are cheap — inside one pass — and pays for them somewhere cheap as well, in a drafting step whose cost is a fraction of the large network's.
What does the drafting is a deployment choice, and three arrangements are in general use. It can be a much smaller network of the same family, sharing the vocabulary so that its proposals are token IDs the large network reads directly. It can be extra prediction heads attached to the large network itself and trained to guess a few positions ahead, which removes the second set of weights entirely. Or it can involve no network at all: where the output is expected to copy heavily from the input — editing a document, answering out of a supplied passage, rewriting a function — a proposal can be lifted from the prompt by matching the recent suffix against it, which costs a lookup. All three share the property the technique needs, which is that a proposal is cheap and a rejection is survivable.
Verification is what earns the technique the right to be invisible. The large network is handed the drafted run and returns, for each of those positions, the distribution it would have produced there on its own. The drafted tokens are then tested in order: each is accepted with a probability derived from how far the drafter's confidence in it exceeded or fell short of the large network's, and at the first rejection the token is replaced by one drawn from an adjusted distribution assembled from the difference between the two. The rule is constructed so that what comes out is distributed exactly as a sample from the large network alone would have been. If every drafted token survives, one further token can be taken at no extra cost from the distribution already computed at the final position, so a round of k proposals can return as many as k plus one. Deployments that decode greedily use a simpler test with the same character: keep the drafted tokens while they match the large network's own first choice, and stop at the first place they diverge.
| Quantity | Effect of speculation |
|---|---|
| Sequential passes over the large weights | Fewer. One pass can settle several positions where the plain path settles one. |
| Total arithmetic performed | More. Rejected drafts are computed and thrown away, and the drafting step is additional work on top of that. |
| The distribution the output is drawn from | Unchanged under the standard acceptance rule. This is the property the rule is built to provide. |
| Memory held per sequence | Slightly more. The drafter carries state of its own, and drafted positions occupy cache entries until they are accepted or rolled back. |
| Tokens returned per round | Variable, from one up to the number drafted plus one, decided by where the first rejection falls. |
The second and third rows are the whole bargain. Arithmetic is spent to buy sequential passes, which is a good trade only while there is arithmetic to spare; and the third row is what makes the trade acceptable to anyone who cares what comes out, since a scheme that shifted the distribution would be a quality change wearing the costume of a latency optimisation.
Where the saving shrinks, and where it turns negative
Everything rests on the length of the accepted run. A drafter that agrees with the large network for a long stretch collapses several sequential passes into one; a drafter rejected at its first position has spent a drafting pass and a verification pass to deliver the single token the plain path would have delivered anyway. Agreement is not a constant, and it varies within one reply rather than between workloads: boilerplate, quoted material, closing punctuation and code that follows an obvious continuation are drafted well, while the place where a reply genuinely turns is precisely where a small drafter and a large one part company. That is why draft length is a tuned parameter and increasingly an adaptive one — drafting further ahead raises the ceiling on a good round and raises the waste on a bad one, and the right depth is a property of the traffic.
The second limit is that the spare capacity being spent is not always spare. The memory-bound argument above describes a server generating for one sequence or a few. A server already running a large batch has filled its compute units with other requests, and speculation now competes with them instead of using idle silicon, so the same configuration can shorten the latency of a single request while lowering the total throughput of the machine. Which of those two the deployment is being judged on is a product question rather than a serving one, and it is the question that decides whether the feature should be enabled at all.
Two smaller conditions are easy to trip over and worth stating plainly. The drafter and the large network have to agree about the vocabulary, because proposals travel as token IDs and an ID from a different tokenizer names different text; where the two vocabularies differ, something has to translate between them and the translation is not free. And exact reproduction of the non-speculative path is not among the technique's promises. The acceptance rule preserves the distribution, not the particular sequence any one run would have emitted — and bitwise equality across batch shapes, kernels and library versions was already not on offer before speculation was considered.
How the Registry classifies a draft-and-verify decoding task
Serving a batch of requests through a draft-and-verify decoding path and recording the change in latency per request is filed as data processing in this registry's classification. Nothing is authored by such a task: an existing workload is run again under an altered serving configuration, and what it yields is a set of measurements about the machine that ran it.
The Registry's canonical brief for this entry is filed as:
Serve a batch of requests through a draft-and-verify decoding path and record the change in latency per request.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
70eaafd5a79b7e00…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about speculative decoding
How many tokens a round of speculation actually keeps depends on the drafter, the large network, the draft depth and the content being generated, and not one of those four is written into the canonical brief above. This registry operates no serving stack, drafts no tokens and has timed nothing. It publishes no acceptance rate and no latency figure, for the sufficient reason that it has measured neither.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about speculative decoding
- What is speculative decoding?
- Speculative decoding is a serving technique in which a cheap drafting step proposes several tokens ahead, a large network verifies all of them in a single forward pass, and the longest leading run that agrees is kept. It reduces the number of sequential passes needed per token without changing the distribution the output is drawn from.
- Does speculative decoding change the output?
- Under the standard acceptance rule it does not change the distribution the output is drawn from, and that is the property the rule is constructed to give. It does not promise that a run reproduces the exact token sequence a non-speculative run would have produced; bitwise equality across batch shapes, kernels and library versions was never on offer in any case.
- What proposes the draft tokens?
- Commonly a much smaller network of the same family, sharing the vocabulary. Also used: extra prediction heads attached to the large network and trained to guess a few positions ahead, and, where the output copies heavily from the input, a lookup that proposes continuations found in the prompt itself.
- Can speculative decoding make serving slower?
- Yes, in two situations. If the drafter is rejected early and often, the drafting and verification work is spent to obtain tokens that would have arrived anyway. And on a server already saturated by a large batch, the extra arithmetic competes with other requests rather than filling idle capacity, so single-request latency can improve while total throughput falls.
- Is this the same as just serving a smaller network?
- No, and the difference is the point. Serving a smaller network is a substitution: what it produces is what the caller receives. In speculative decoding a proposal is accepted only where the large network’s own distribution supports it, and every rejected proposal is replaced from that distribution. The drafter influences speed rather than content.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.