A training technique in which a smaller student model learns to imitate a larger teacher model, keeping much of its skill at lower cost.
Knowledge distillation was popularized in a 2015 paper by Geoffrey Hinton and colleagues. Instead of training a small model only on raw data, developers train it on the outputs of a larger, more capable model. The student learns not just the right answers but the teacher's patterns of confidence across possible answers.
The result is a model that is faster and cheaper to run while retaining a useful share of the teacher's ability on targeted tasks. For example, a company might use a large model to label thousands of customer messages, then distill that skill into a small model that runs on its own servers. Distilled models usually lose some accuracy, and model licences sometimes restrict using one provider's outputs to train another model.
Further training a pre-trained AI model on a smaller, specific dataset so it performs better on a particular task or style.
QuantizationReducing the numerical precision of an AI model's weights, for example from 16-bit to 4-bit numbers, to make it smaller and faster.
Small Language ModelA compact language model, typically with a few hundred million to around ten billion parameters, built to run efficiently on modest hardware.
Small language models now run on laptops and phones. We explain how they work, when they beat giant cloud models, and how to pick and deploy one for your product.