What Is a KV Cache?
A KV cache is the store of key and value tensors a transformer has already computed for the positions of a sequence, kept so that generating the next token does not require computing them again. It holds intermediate activations — not text, and not weights. It exists because of one property of causal attention: the keys and values of earlier positions do not change when a new token is appended to the end.
Why anything is cached at all
Generation is autoregressive. One token is produced, appended to the sequence, and the sequence is processed again to produce the next. At each attention layer every position contributes a key and a value vector, and the position being generated computes a query that is scored against all the keys before it. Because the mask is causal — a position may attend only to positions at or before its own — nothing that happens later alters the key and value already computed for an earlier position. They are computed once and retained, and the retention is the cache.
Without it, producing each new token would mean recomputing keys and values across the entire prefix, so the total work of generating a long reply would grow with the square of its length. With it, a decoding step computes one query, one new key and one new value per layer, appends the latter two, and attends over what is already stored. That is what makes the two phases of serving so different in character: the prefill pass processes an entire prompt at once and fills the cache, and every decode step afterwards adds exactly one position per layer to it.
| Item | Held in the cache? |
|---|---|
| Key tensors | Yes — one per position, per attention layer, per key/value head. |
| Value tensors | Yes, on the same terms as the keys and allocated alongside them. |
| Query vectors | No. A query belongs to the position being generated and is not reused once its step is over. |
| Model weights | No. Weights are loaded once and shared by every request; a cache belongs to one sequence. |
| The tokens themselves | No. The text is held by the serving layer; the cache holds activations derived from it. |
| Anything from another request | No, unless the runtime has been asked to share a common prefix between them. |
The third row is what gets the mechanism its name. Queries are not kept because a query belongs to the position doing the attending, which is a new position at every step, while keys and values belong to the positions being attended to, which are not.
The cache is memory, and memory is the constraint
A KV cache grows with the length of the sequence, and the total a server holds grows with the number of sequences it is serving at once. For a given architecture its size is proportional to the number of layers, the number of key/value heads, the size of each head, the number of positions retained, and the width of the numeric type it is stored in — doubled, because keys and values are both kept. None of those factors is small, and the product competes for the same accelerator memory the weights already occupy. This is why a serving configuration that is comfortable at short context can run out of room at long context with no change to the workload other than length.
Three families of response follow directly from that arithmetic and all three are visible in current architectures. The first reduces the number of key/value heads, sharing them across query heads so that fewer keys and values are stored per position. The second restricts which positions have to be retained at all, which is what a windowed attention pattern does. The third stores the tensors in a narrower numeric type than the one the network computes in. Each trades something for the memory it saves, and which trade a deployment has made is a fact about that deployment rather than about the mechanism.
Allocation is a separate problem and is the one most often overlooked. A sequence’s final length is not known when it starts, so a runtime that reserves a contiguous block large enough for the longest sequence it might see wastes most of that reservation, while one that allocates fixed-size blocks as the sequence grows packs more sequences into the same memory at the cost of a level of indirection. Block management, eviction under pressure, and reuse of a shared prefix across requests are all decisions of the serving runtime, and they are the reason two deployments of identical weights can differ in how many concurrent requests they hold.
How the Registry classifies a serving-memory measurement task
Measuring the memory a served configuration holds for cached key and value tensors across a batch of concurrent requests is filed as data processing in this registry’s classification. What the task consumes is telemetry that already exists, and what it produces is a derived quantity — a measurement of a running system rather than a change to it.
The Registry's canonical brief for this entry is filed as:
Measure the memory a served configuration holds for cached key and value tensors across a batch of concurrent requests.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
005af7aaee002f96…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about kv cache
How much memory any particular cache occupies follows from that deployment’s layer count, head geometry, numeric type and served lengths, and the canonical brief above fixes none of the four. This registry serves no models, allocates no cache and has measured no deployment’s memory. The brief names an operation an operator might run; every number it would produce belongs to their hardware and not to this page.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about kv cache
- What is a KV cache?
- A KV cache is the store of key and value tensors already computed for the positions of a sequence, retained across decoding steps so that generating each new token does not require recomputing them. It holds intermediate activations for one sequence — not the text, and not the weights.
- Why are queries not cached?
- Because a query belongs to the position doing the attending, and that is a new position at every decoding step. Keys and values belong to the positions being attended to, and under a causal mask those do not change when a token is appended, which is what makes them worth keeping.
- What makes a KV cache large?
- Its size is proportional to the number of layers, the number of key/value heads, the head dimension, the number of positions retained and the width of the numeric type, with keys and values both stored. Serving several sequences at once multiplies it again, which is why long context and high concurrency compete for the same memory.
- Is a KV cache the same as prompt caching?
- They are related and not the same. A KV cache is the per-sequence store used while one reply is generated. Prompt caching is a serving feature that keeps the state for a shared leading portion of a request beyond a single generation, so a later request beginning with the same bytes need not process it again. The second is built on the first.
- Does discarding the KV cache change the output?
- Recomputing what was cached recomputes the same quantities, so a discarded cache costs work rather than correctness. Exact numerical equality across different batch shapes, kernels and hardware is a separate matter and is not guaranteed by anything about caching; that is a property of floating-point execution.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.