What Is an Attention Mechanism?
An attention mechanism is the operation by which each position in a sequence builds its representation out of the other positions, weighted by how relevant each of them is to it. Every position emits a query, a key and a value; a query is scored against the keys, the scores are normalised into weights, and the output is the weighted sum of the values. It is the component that lets a representation at one position depend on material at a distant one without the two being adjacent.
Query, key, value, and the score between them
The three names are three projections of the same input. Each position’s representation is multiplied by three learned matrices to give a query, a key and a value. The query of a position is scored against the key of every position it is permitted to see, by dot product, divided by the square root of the key dimension so that the scale of the scores does not grow with that dimension. The scores across the visible positions are normalised with a softmax, giving weights that are non-negative and sum to one, and the output for that position is those weights applied to the corresponding values.
Two variations cover most of what the term is used for. Self-attention draws queries, keys and values from the same sequence, so positions attend to one another; cross-attention draws queries from one sequence and keys and values from another, which is how a decoder reads an encoded input. And the operation is run several times in parallel over different projections — multiple heads — whose outputs are concatenated and projected back to the working width, so different heads can carry different relationships instead of one set of weights having to represent all of them at once.
| Part | What it is |
|---|---|
| Query | A projection of the position doing the attending: what this position is looking for. |
| Key | A projection of each position attended to: what that position offers to be matched against. |
| Value | A second projection of the same position: what is carried forward once that position is selected. |
| Score | The dot product of a query with a key, divided by the square root of the key dimension. |
| Mask | The rule stating which positions a query may see — in a decoder, none later than its own. |
| Output | The values, weighted by the normalised scores, summed into one vector for that position. |
Keys and values are separate projections of the same position on purpose, and the separation is the part most often lost in summary. Being found and being used are different jobs with different parameters, so a position can be matched on one criterion and contribute something else entirely once the match has been made.
What it costs, and what the weights do not tell you
Every query is scored against every visible key, so the number of scores in one attention operation grows with the square of the sequence length — in one head, in one layer. Time and memory traffic follow it. That single fact is the origin of nearly every architectural variation downstream of it: windowed and sparse patterns, which reduce which pairs are scored at all, and tiled exact implementations, which avoid holding the whole matrix of scores in memory at once. The first kind changes the result. The second does not, and the difference is worth keeping straight when reading a claim about a long-context architecture.
Attention on its own has no notion of order. Nothing in the operation distinguishes the third position from the tenth; shuffle the inputs and the outputs shuffle with them, otherwise unchanged. Order enters through positional information added to or applied within the representations, and that is a separate mechanism with its own design choices. It is also why the question of how a network behaves beyond the lengths it was trained on is a question about positional encoding rather than about attention.
A note on interpretation, because attention weights are the most reached-for explanation in this field and they do not carry the load placed on them. A weight states how much one position’s value contributed to another position’s output, at one head, in one layer, in one forward pass. It does not state why, it is one contribution among many heads whose outputs are then mixed, and many layers follow before anything is emitted. A high weight is a fact about a computation. Treating it as a reason is an interpretation, and this registry offers none.
How the Registry classifies an attention cost comparison task
Comparing the compute cost of a full attention pass against a windowed one over a fixed input length is filed as data processing in this registry’s classification: the inputs are a configuration and a length, the output is a derived pair of quantities, and no part of the operation produces material that did not already exist.
The Registry's canonical brief for this entry is filed as:
Compare the compute cost of a full attention pass against a windowed one over a fixed input length.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
2d675cd338d8ca9d…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about attention mechanism
What either pass costs on real hardware depends on the implementation, the kernel, the memory hierarchy and the batch shape, and the canonical brief above fixes none of them. This registry runs no attention operation, holds no weights and has inspected no attention pattern of any model in its list. It files an operation somebody else would run.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about attention mechanism
- What is an attention mechanism?
- An attention mechanism computes, for each position in a sequence, a weighted sum of value vectors drawn from the positions it is permitted to see. The weights come from scoring that position’s query against those positions’ keys and normalising the result. It is what allows a representation at one position to depend on material at a distant one.
- What are queries, keys and values?
- Three learned projections of the same representations. The query is what a position is looking for, the key is what a position offers as a match, and the value is what is carried forward once a match is made. Keys and values are kept separate so that finding a position and using it are governed by different parameters.
- What is multi-head attention?
- Several attention operations run in parallel over different projections of the same input, with their outputs concatenated and projected back to the working width. Different heads can then carry different relationships between positions, rather than one set of projections having to represent all of them at once.
- Why is attention expensive on long inputs?
- Because each query is scored against every visible key, the number of scores grows with the square of the sequence length in each head of each layer. Sparse and windowed patterns reduce which pairs are scored; tiled exact implementations reduce memory traffic without changing which pairs are scored or what comes out.
- Is attention the same as the context window?
- No. Attention is an operation inside the network. A context window is a limit on how many tokens one request may occupy, set by the deployment. The two are connected — the cost of attention is one of the pressures on how long a request is allowed to be — but one is a mechanism and the other is a quota.
- Do attention weights explain a decision?
- They show how much each visible position contributed at one head, in one layer, in one forward pass. That is a measurement of a computation rather than an account of a reason, and it is one contribution among many heads and many layers. Attention weights are widely used as an illustration; they are not an explanation.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.