What Is Sliding Window Attention?
Sliding window attention restricts each position to attending over a fixed span of neighbouring positions instead of the whole preceding sequence. The window moves with the position: the thousandth token sees the span immediately behind it, the two-thousandth sees the span behind that. Nothing else about the operation changes — the same queries, keys, values and normalisation, over fewer pairs.
A mask, not a different operation
The difference between full and windowed attention lies entirely in which pairs of positions are permitted. Full causal attention lets a position see everything at or before itself; a sliding window lets it see only a fixed number of positions immediately behind. The scoring, the scaling and the weighted sum are identical in both. That is worth holding onto when reasoning about the design, because a windowed layer is not an approximation in the sense of a different formula. It is the same formula over a restricted set of pairs, and what it gives up is exactly the pairs it excluded — no more, and no less.
The cost consequence is the point of the arrangement. Under full attention the number of scored pairs grows with the square of the input length; under a fixed window each position scores a constant number of pairs, so the total grows in proportion to the length instead. The retained key and value tensors follow the same rule: a layer that cannot look back further than its span has no use for the keys and values of positions beyond it, so the memory one sequence occupies stops growing once its length passes the window.
| Property | How the two differ |
|---|---|
| Positions scored | Full: everything at or before the current position. Windowed: a fixed span immediately behind it. |
| Cost as input grows | Full: the pair count grows with the square of the length. Windowed: it grows in proportion to it. |
| Keys and values retained | Full: all of them. Windowed: those inside the span, so per-sequence memory stops growing. |
| Direct reach | Full: any earlier position, within a single layer. Windowed: only inside the span, within a single layer. |
| Indirect reach | Full: not needed. Windowed: further back through depth, roughly one window per layer, via the positions between. |
| The operation itself | Identical. Scoring, scaling and the weighted sum are unchanged; only the mask differs. |
The fifth row is where the design lives or dies and it is the least visible of the six. A restriction with no way around it would make anything further back than the span unreachable, which would be a severe limitation rather than an optimisation; the way around it is not a second attention pattern but the depth of the stack, and it is set out below.
How material travels further than the window
One windowed layer cannot connect two positions further apart than its span. A stack of them can, because each position’s output already summarises the span behind it, and the next layer’s window covers positions whose outputs summarised the spans behind them. Reach therefore accumulates with depth: after a number of layers a position can be influenced by material roughly that many windows back. It is the same argument as the receptive field of a stack of convolutions, and it has the same character — the influence is real, it is indirect, and it passes through every intermediate representation along the way.
That is also the honest limit of the design, and it is better stated than implied. An indirect path is not a direct one. Material that arrived through six intermediate summaries has been compressed six times, where full attention would have scored it against the current position without an intermediary. Whether the difference matters is a question about a specific task on a specific architecture, answered by measurement, and no such measurement is published here.
Two arrangements are commonly combined with the window and both are worth recognising when reading an architecture description. Some designs alternate windowed layers with full-attention layers, so that most layers are cheap while a few retain unrestricted reach within a single hop. Others keep a small number of the earliest positions permanently visible in addition to the sliding span. Which arrangement a given architecture uses, and how wide its window is, are stated in that architecture’s own documentation and are not restated here.
How the Registry classifies a windowed processing task
Processing an input longer than the attention window by attending within a fixed span at each position is filed as data processing in this registry’s classification. Existing material is consumed and transformed; the constraint that forced the arrangement is a property of the machinery, and machinery does not move a task out of the class its operation puts it in.
The Registry's canonical brief for this entry is filed as:
Process an input longer than the attention window by attending within a fixed span at each position.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
bcf20605ae78f397…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about sliding window attention
How wide a particular window is, how many layers stack above it, and whether an architecture mixes windowed layers with full ones are facts about that architecture, and the canonical brief above names none of them. This registry has run no architecture, measured no effective reach, and compared no windowed configuration against a full one. It classifies an operation and computes from the wording of a submitted task.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about sliding window attention
- What is sliding window attention?
- Sliding window attention restricts each position to attending over a fixed span of immediately preceding positions rather than the whole sequence. The window moves with the position, and the attention operation is otherwise unchanged — only the set of permitted pairs is smaller.
- Why use a sliding window?
- For cost. Full causal attention scores a number of pairs that grows with the square of the input length, while a fixed window scores a constant number per position, so the total grows in proportion to the length. The keys and values a sequence has to retain are bounded by the same span, which bounds serving memory as well.
- Can information travel further than the window?
- Yes, through depth. Each layer’s output at a position already summarises the span behind it, so the next layer reaches further back indirectly, and reach accumulates at roughly one window per layer. The path is real, but it is compressed at every hop, which is not the same as being scored against directly.
- Is sliding window attention the same as a context window?
- No, and the shared word causes real confusion. A context window is the total number of tokens one request may occupy, a limit of the deployment. A sliding window is an attention pattern inside the network, restricting which positions each position may score against. A configuration can have a long context limit and windowed attention at the same time.
- Does a sliding window reduce memory?
- It bounds the key and value tensors a sequence has to keep, because positions outside the span of a windowed layer are of no further use to it. Whether a deployment realises the saving depends on its serving runtime and on whether some of its layers use full attention, which would need their own retained tensors regardless.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.