What is knowledge distillation? How a smaller model learns from a teacher
Learn about teacher and student models, soft targets, and generative distillation through support classification, including differences from quantization and checks for inherited errors.
Understand the concept
Knowledge distillation trains a student to imitate a teacher, often transferring some capabilities from a larger model to a smaller one. Signals can include output probability distributions rather than only correct labels. It requires training and independent evaluation, does not preserve every capability, and is distinct from compressing files or reducing numerical precision.
A teacher can provide more than a class label
For support messages classified as refunds or delivery, a hard label names one class while a teacher distribution also expresses how plausible the other is. The student can learn these relationships. The original paper discusses soft targets and temperature, but distillation methods vary by task.
Generative models can learn teacher texts or token distributions. Hugging Face’s GKD implementation uses teacher feedback on student-generated sequences, so distillation is not always copying a fixed answer set. Check the implementation’s required data and loss.
Distinguish distillation from quantization and fine-tuning
Quantization changes numerical representations of weights or computation; distillation trains student parameters. They can be combined but need separate checks. Converting a large model to lower precision does not establish distillation, and smaller size does not establish preserved behavior.
Supervised fine-tuning learns target answers, while distillation emphasizes teacher-derived signals. Teachers can be wrong or weak in a language. More teacher samples do not automatically correct these biases; independent evidence and coverage checks remain necessary.
Confirm deployment benefits with matched tasks
Use held-out support messages to compare teacher and student classification errors, then measure latency and memory on the target device. This is an evaluation design, not a training run or speed result. Fewer parameters do not guarantee faster execution on every device.
Check rare cases the teacher handled and fabrication when information is missing. Verify licenses and service terms before producing training data. Keep the student version and data provenance rather than treating an opaque model name as proof of a reliable substitute.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.