What Is Prompt Caching?
Prompt caching is a serving feature that keeps the computation already performed for the leading portion of a request, so that a later request beginning with exactly the same bytes does not have to process that portion again. What is kept is internal state — the key and value tensors for the cached positions — and not the reply. The cached part is skipped in processing and is still fully present in the request.
The prefix is matched from the first byte
A cached prefix is a leading run of a request, matched from its start. If a request opens with the same instruction, the same tool definitions and the same reference document as one already processed, the work of processing that run can be reused, and only the remainder has to be processed from scratch. The match is on content and it is strictly a prefix match rather than a search: the first byte that differs ends the reusable portion, and everything after it is treated as new whether or not it is identical to something seen before.
That one property dictates the entire engineering practice around the feature. Stable material goes at the front — the instruction, the tool schemas, the long fixed document, the worked examples. Volatile material goes at the back: the current turn, the passages that differ per query, anything carrying a timestamp, a request identifier or a name. A single variable placed near the front invalidates every byte after it, including the twenty pages of unchanged document sitting below, and the request that results is correct, complete, and reuses nothing.
| Change | Effect on reuse |
|---|---|
| A byte changed anywhere inside the prefix | Reuse ends at that byte. Everything after it is processed again, changed or not. |
| Anything inserted before the stable material | Reuse ends at the insertion, which is at the front, so nothing at all is reused. |
| A tool definition added, removed or reordered | Reuse ends where the tool list begins, if the list sits inside the prefix. |
| A timestamp or identifier in the instruction | Reuse ends on every request, because the value differs on every request. |
| Expiry or eviction of the entry | The entry is gone. The next request processes the prefix in full and may repopulate it. |
| A change after the prefix | None. Everything before the change is still reusable, which is the whole arrangement. |
The fourth row is the expensive one and the hardest to see, because nothing about the request looks wrong: one changing value at the front of an otherwise fixed instruction turns a carefully ordered prompt into an uncached one, the replies are unaffected, and the only symptom is a bill and a latency profile that never improved.
What prompt caching is not
It is not a response cache. A response cache stores an answer and returns it again for a matching question, so a second caller receives the first caller’s output. Prompt caching stores intermediate computation and then generates as usual, so the reply is produced afresh every time and sampling behaves exactly as the request’s configuration says it should. The distinction matters whenever somebody asks whether caching is why two replies came back identical: this feature is not the reason they would.
It does not reduce the room a request occupies. A cached prefix is still in the request, still counted against the limit, and still material the reply is generated with. What is saved is the work of processing it, not the space it takes — so caching is no help in fitting more into a window, and a request that is too long remains too long however much of it was reused.
And it is not free of platform-specific parameters. How long an entry survives, how large a leading run must be before it is eligible, how a cache is scoped between accounts and projects, and how the reused and freshly processed parts of a request are billed all differ between implementations and change on their own release schedules. Those are facts about a particular platform, published in that platform’s own documentation, and none of them is stated on this page — precisely because they would go stale here with nobody editing the file.
How the Registry classifies a prefix reuse task
Reusing a stable instruction prefix across a batch of requests and recording the change in cost per request is filed as data processing in this registry’s classification. The task produces a measurement taken over requests that were going to be sent in any case: the caching changes how the work is performed rather than what the work is.
The Registry's canonical brief for this entry is filed as:
Reuse a stable instruction prefix across a batch of requests and record the change in cost per request.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
86be8282aa58fd27…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about prompt caching
Whether a given prefix is eligible for reuse, how long it survives, and what the arrangement saves are properties of the platform serving the requests, and the canonical brief above names no platform. This registry operates no cache, sends no requests on an operator’s behalf, and has measured no cost before or after such a change. It sets out a mechanism and leaves the parameters to whoever publishes them.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about prompt caching
- What is prompt caching?
- Prompt caching is a serving feature that keeps the computation performed for the leading portion of a request, so a later request starting with exactly the same bytes can skip processing it. What is kept is internal state for those positions, not the reply.
- What invalidates a cached prefix?
- Any change inside it, matched from the first byte. A reworded instruction, a reordered tool list, or a timestamp inserted near the front each end reuse at the point where they occur, and everything after that point is processed again whether or not it changed.
- How should a request be ordered for caching?
- Stable material first, volatile material last: instruction, tool definitions and long fixed documents at the front; the current turn, per-query passages and anything carrying a time or an identifier at the end. The rule follows from prefix matching itself rather than from any preference of a particular platform.
- Does prompt caching change the answer?
- It is not meant to. The reused portion is computation that would otherwise have been repeated, and generation proceeds normally afterwards. It does not make a reply repeatable, and it is not a response cache: no previous answer is returned.
- Does a cached prefix still count against the context limit?
- Yes. The prefix is present in the request and occupies exactly the room it would occupy uncached. Caching saves the work of processing it, not the space, so it does nothing for a request that is too long.
- Is prompt caching the same as a KV cache?
- They are related. A KV cache is the per-sequence store of key and value tensors used while one reply is being generated. Prompt caching keeps that state for a shared leading run beyond a single generation so a later request can begin from it. One is the mechanism; the other is a serving feature built on it.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.