What Is a Token in an LLM?
A token is the unit of text a language model actually reads and emits. Text is converted into a sequence of integer token IDs before anything else happens, and the field's limits, prices and context measurements are all stated in those units rather than in words or characters.
How text becomes tokens
Tokenization sits between the text and the network. A tokenizer holds a fixed vocabulary, typically tens of thousands of entries, built by a training procedure that repeatedly merges frequently co-occurring byte sequences into single entries. Common words usually survive as one token. Rare words, names and unusual spellings are split into several pieces. Whitespace is generally carried on the token that follows it, which is why adding a leading space can change the segmentation of everything after it.
Because the vocabulary was built from a corpus, a token count is a property of that corpus as much as of the text being counted. Ordinary English prose written in a widely used vocabulary lands at somewhat more tokens than it has words, because the frequent words survive whole and only the rest are broken up. Code, markup, non-Latin scripts and long digit strings all push the ratio further, sometimes considerably. There is no conversion constant, and any figure offered as one is a rule of thumb about one vocabulary rather than a fact about text — which is why this registry publishes none.
| Input | Typical tokenization |
|---|---|
| A short common word | One token, usually including the space in front of it. |
| A long or rare word | Several tokens, split at the vocabulary’s merge boundaries. |
| A proper name | Often several tokens, depending on how often it appeared in the tokenizer’s training corpus. |
| A long number | Frequently more than one token; how digits are grouped varies by vocabulary. |
| Text in a non-Latin script | Commonly more tokens per character than Latin text in the same vocabulary. |
These are directional statements about how tokenizers behave, not measurements. The only reliable count for a given piece of text is the one produced by the tokenizer that will actually read it.
Where the count is misread
Three things are measured in tokens: the context limit, the price, and the latency budget. All three count the input and the generated output together in the usual arrangement, so a long prompt is not free merely because the reply is short.
The common misreading is to treat a token as a word. A limit budgeted in words leaves an unpredictable margin, because the ratio depends on the specific text — a document of tables and identifiers can consume several times the tokens of the same length of prose. Where a limit has to be respected exactly, the count is taken with the tokenizer rather than estimated from it.
That margin is also what makes the resulting bug hard to reproduce. An input measured in characters is not shortened for the sender when it turns out to be a little too long: the request is refused, or some part of it is dropped, and which of the two happens is decided by the surrounding software rather than by the count. So code that budgets in characters does not fail on every document. It fails on the documents whose ratio happens to be unfavourable — the ones full of identifiers, tables or a second script — and passes on the plain prose it was tested against.
How the Registry classifies a token-counting task
Counting the tokens in a batch of transcripts and re-splitting each one to fit a fixed request budget is filed as data processing in this registry's classification: existing material is measured, and the quantities that come out are derived rather than looked up.
The Registry's canonical brief for this entry is filed as:
Count the tokens in a batch of support transcripts and split each one to fit a fixed request budget.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
37595cae10f15ba5…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about token
Which vocabulary does the counting, and therefore what number comes out, is not something the canonical brief above specifies, and no count on this page is offered as one.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about token
- What is a token in an LLM?
- A token is the unit of text a language model reads and emits: a word, a fragment of a word, or a punctuation mark, mapped to an integer ID by a tokenizer with a fixed vocabulary. Context limits and prices are stated in tokens rather than in words.
- How many tokens is a word?
- There is no fixed conversion, and this registry offers none. Ordinary English prose comes out at somewhat more tokens than words, since common words survive as single entries and rarer ones are split; code, markup, identifiers and non-Latin scripts differ substantially from that and from each other. The count for a specific text is obtained by running that text through the tokenizer that will read it.
- Do input and output tokens count against the same limit?
- In the usual arrangement, yes. A request’s input tokens and the tokens generated in reply are both charged against the context limit, so a long input reduces the room left for an answer. Pricing frequently separates the two rates while the limit still applies to their sum.
- Why does adding a space change the token count?
- Because most vocabularies encode a leading space as part of the token that follows it. A word with a space in front of it and the same word without one are distinct vocabulary entries, so where whitespace falls can change how the surrounding sequence is segmented.
- Why is the token the unit for pricing rather than the word?
- Because the work a request costs scales with the sequence the network actually processes, and that sequence is measured in tokens. Rates are therefore quoted per token — commonly at one rate for input and a higher one for generated output — and a price stated per thousand or per million tokens is a rate against the count that one specific tokenizer produces, not against the words a reader would count.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.