Knowledge Distillation
A model compression technique where a small 'student' model is trained to mimic the outputs and probability distributions of a large 'teacher' model.
Think of It Like This
Like a master chef passing down their refined recipes to an apprentice so the apprentice can cook just as well but much faster.
Instead of training on raw hard labels, the student learns from the teacher's 'soft labels,' which contain rich information about the relationships between classes. This allows the smaller model to achieve performance comparable to the massive teacher model but with vastly reduced latency and memory footprint. It is widely used to deploy LLMs on edge devices.