Knowledge Distillation

A training technique that transfers behavior from a teacher model, or an ensemble, into a student model better suited for deployment. The student learns from expected labels and from the teacher's probability distribution, often softened with temperature, because probabilities across classes expose relationships that a discrete label does not capture.

A common goal is to reduce cost and latency while preserving part of the quality, although the student may also use a different architecture. Distillation requires its own data, loss design, and evaluations, and it can inherit teacher errors and biases. It differs from quantization, which lowers the numerical precision of a representation, and from generic fine-tuning, whose training signal need not come from a teacher.