What Is Model Distillation?
Distillation trains a smaller network, the student, on the recorded outputs of a larger one, the teacher, rather than on the original labels alone. What transfers is the teacher's behaviour on the inputs it was run against — which is a narrower thing than the teacher's knowledge, and the distinction is where most disappointment with the method originates.
Teacher, student, and what is copied
The procedure is: run a set of inputs through the teacher, record what came out, and train the student to reproduce it. The recorded signal can be the full probability distribution over the vocabulary at each step, the single answer the teacher settled on, or a longer generated response including any intermediate reasoning it produced. Distributions carry more per example than final answers, because the relative weight the teacher placed on the alternatives it rejected is itself information.
Where the full distribution is used, it is usually softened before the comparison is made: the scores behind it are divided by a value greater than one, which flattens the distribution and raises the weight carried by the alternatives the teacher ranked below its first choice. The same operation is applied to the student's distribution so the two are compared on the same footing. Softening is the mechanism by which the teacher's relative preferences among rejected answers survive into the student at all — a hard label discards them completely, because it records only which answer won.
The student is a smaller architecture chosen in advance, and the ceiling on what it can absorb is set by that choice. Distillation redistributes behaviour into a smaller network as far as the smaller network's capacity allows. It is not lossless compression, and where the loss lands is decided by the inputs the teacher was run on rather than by the method. What the objective rewards, throughout, is agreement with the teacher — so a teacher that answered an input wrongly supplies a target the student is trained toward, and the procedure has no separate notion of the answer being correct.
| Signal used | What the student is trained on |
|---|---|
| Output distribution | The teacher’s probabilities across the vocabulary at each step, so rejected alternatives carry information too. |
| Final answers | Only what the teacher settled on, discarding the distribution behind it. |
| Generated sequences | Longer teacher responses, including intermediate reasoning where the teacher produced it. |
| Internal representations | The teacher’s activations, matched layer by layer where the architectures allow it. |
Every variant is bounded by the inputs the teacher was run on. A behaviour the teacher was never asked to display is absent from the recorded data and is therefore not transferred by any of them.
Distillation, quantization and pruning are three different operations
All three produce a cheaper artefact and they are routinely conflated. Distillation trains a new, separate network. Quantization stores the same network's existing weights at lower numeric precision. Pruning removes parameters from the existing network. Only the first involves training a different network at all.
The practical consequence is about what has to be re-tested. A distilled student is a new configuration with its own behaviour, and an evaluation of the teacher says little about it. A quantized copy is a modified form of the same network, and the question there is how much the approximation moved it. Treating the two as interchangeable is how a change ships with the wrong evidence attached.
How the Registry classifies a distillation run
Training a student on the recorded outputs of a teacher and recording the size reduction is filed as data processing in this registry's classification: recorded material is consumed as input, and what comes out is a derived artefact with derived properties.
The Registry's canonical brief for this entry is filed as:
Train a smaller student network on the recorded outputs of a larger teacher network and record the size reduction.
Submitted for assessment it is
classified as Data processing, and its wording is hashed once — to
d562c6c3b900800d…, the first sixteen of sixty-four hexadecimal
characters — with the wording itself never stored. The hash is what the derivation
reads.
That class's own page is /tasks/analysis.
Classification is one of three inputs. The other two are the configuration submitted with the task, and the permanent chart derived from that configuration — fixed by the model name, the training cutoff and the temperature alone, and never reading the task at all. The same ascendant, ruling planet and harmony therefore appear on every assessment a given configuration receives, whatever it was asked to do. The derivation is published in full at /method.
What this page does not claim about distillation
Which inputs the teacher was run on, and therefore which behaviours were available to transfer, is not encoded in the canonical brief above and is the variable that decides the outcome. No teacher has ever been run at this registry and no student network exists here. The Registry compares nothing, scores nothing, and would not be able to tell a successful distillation from an unsuccessful one.
The Registry does not run this task, does not inspect any system's output for it, and validates no assessment it issues against what afterwards happens. What it does is compute — from a published method, for one submitted task and one submitted configuration — a verdict and a recommended execution window. It computes neither on this page.
Questions about distillation
- What is model distillation?
- Distillation trains a smaller student network to reproduce the recorded outputs of a larger teacher network. The training signal comes from the teacher’s behaviour on a set of inputs rather than from the original labels alone.
- What is the difference between a teacher and a student?
- The teacher is the existing, larger network whose outputs are recorded. The student is the smaller network trained on those recordings. They are separate artefacts, and the student’s capacity is fixed by its own architecture rather than by the teacher’s.
- Is distillation the same as quantization?
- No. Distillation trains a new, smaller network on a larger one’s outputs. Quantization leaves the network intact and stores its existing weights at lower numeric precision. One produces a different network; the other produces a cheaper representation of the same one.
- Does a distilled student behave like its teacher?
- It is trained to reproduce the teacher’s behaviour on the inputs the teacher was run against. Whether it does so on inputs outside that set is not established by the procedure and has to be measured separately, on the cases that matter.
- Does a student inherit the teacher’s mistakes?
- The objective rewards agreement with the recorded outputs, so where the teacher answered an input wrongly, that answer is the target the student is trained toward. The procedure has no separate notion of correctness, which is why distillation datasets are often filtered or checked before they are used rather than taken as produced.
Related entries
Order an assessment for data processing tasks
The Registry issues a permanent, numbered task risk assessment for one submitted task and one submitted configuration. EUR 1.90 Standard, EUR 4.90 Extended, EUR 14.90 Full Chart, which adds the permanent chart. Assessments from EUR 1.90; machine-readable at /pricing.json.