What Is a Tokenizer?
A tokenizer is the component that converts text into the integer IDs a language model reads, and converts IDs back into text afterwards. It is a fixed, deterministic program with a fixed vocabulary, trained once and shipped alongside the network it belongs to.
Byte-pair encoding and the vocabulary
Most current tokenizers are built by some variant of byte-pair encoding. Training starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, until the vocabulary reaches its target size. What ships is an ordered list of those merges plus the vocabulary they produced. Encoding a new string applies the merges in order — nothing is decided at request time, and the same string always encodes the same way.
Working from bytes rather than characters is what lets a fixed vocabulary encode any input at all, including scripts that were rare in the training corpus and byte sequences that are not valid text in any encoding. Such input is not rejected; it is simply encoded as more tokens, which is why an unusual document can consume its budget far faster than its length suggests.
Byte-pair encoding is not the only scheme in service, and the alternatives differ in how the vocabulary is chosen rather than in what it is for. WordPiece scores a candidate merge by its effect on the likelihood of the training corpus instead of by raw frequency. Unigram methods run in the opposite direction, starting from an oversized candidate vocabulary and pruning entries, keeping the segmentation the surviving vocabulary makes most probable. All of them ship the same two artefacts — a fixed vocabulary and a fixed procedure for applying it — and all of them make the same guarantee, which is the one that matters downstream: encoding is settled before the request, not during it.
| Component | Purpose |
|---|---|
| Vocabulary | The fixed set of entries, each mapped to an integer ID. |
| Merge list | The ordered pair merges applied when encoding, which make encoding deterministic. |
| Special tokens | Reserved IDs marking boundaries such as the start of a sequence or the end of a turn. |
| Normalisation rules | Any transformation applied before encoding, such as Unicode normalisation. |
| Decoder | The inverse mapping, from IDs back to bytes. |
A network and its tokenizer are a matched pair. IDs produced against one vocabulary are meaningless to a network trained against another, which is why a tokenizer is not a component that can be upgraded on its own.
Why two counts of the same text disagree
A count is only defined relative to a vocabulary. The same paragraph measured against two vocabularies yields two numbers and neither is wrong. This is the mechanism behind a recurring surprise: an estimate taken from one provider's counting tool can misstate both the cost and the fit of a request sent somewhere else.
Two further sources of disagreement are worth naming. Whether the counter includes the special tokens a chat format adds — role markers and turn boundaries — changes the total by a fixed amount per message, which matters most when there are many short messages. And whether it counts the input alone or the input plus the expected reply changes what the number is a budget for.
How the Registry classifies a tokenization task
Tokenizing a corpus of messages and reporting the distribution of counts across them is filed as data processing here: existing material is transformed, and summary quantities are computed from the result.
The Registry's canonical brief for this entry is filed as:
Tokenize a corpus of customer emails and report the distribution of token counts per message.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
200a306c775c5496…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about tokenizer
Which vocabulary performs the tokenization is the variable that decides every number such a task produces, and the canonical brief above does not name one. This registry has run no tokenizer over any corpus, holds no vocabulary of its own, and offers no ratio by which a count in one vocabulary may be converted into a count in another.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about tokenizer
- What is a tokenizer?
- A tokenizer converts text into the integer token IDs a language model reads, and converts them back into text afterwards. It holds a fixed vocabulary and a fixed set of merge rules, both established when it was trained.
- What is byte-pair encoding?
- Byte-pair encoding builds a vocabulary by starting from individual bytes and repeatedly merging the most frequent adjacent pair into a new entry, until the vocabulary reaches its target size. Encoding then applies those merges in the order they were learned.
- Why do token counts differ between providers?
- Because each provider ships its own vocabulary, and a count is defined relative to a vocabulary. The same text encoded against two different vocabularies produces two different counts, and both are correct for their own tokenizer.
- Can a tokenizer be swapped after training?
- Not without retraining. The token IDs a tokenizer emits are the input space the weights were learned against, so replacing the vocabulary changes what every ID means. Swapping a tokenizer is a training decision, not a configuration one.
- Is tokenizing the same as splitting text on spaces?
- No. Splitting on whitespace produces words. A tokenizer produces vocabulary entries, which can be shorter than a word where a rare word is broken up, can carry the preceding space as part of the entry, and are defined by a merge procedure rather than by punctuation. The two agree often enough on simple English to be confused with one another, and disagree on precisely the inputs — code, identifiers, non-Latin scripts — where an exact count is being relied on.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.