Table of Contents
- Introduction
- Hidden Knowledge
- Distillation Procedure
- Experiment on MNIST
- Additional Experiment on MNIST
1. Introduction
In this article we will explore the process of knowledge distillation in artificial intelligence: how it works, its importance, and the reasons for using it.
How can we compress and transfer knowledge from a large model or an ensemble of models trained on extensive datasets to a single small model without losing much performance?
We train large models because we seek to extract patterns from large volumes of data using techniques such as dropout or data augmentation. However, during the prediction phase, our goal is to obtain results quickly, which makes using a large model costly.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean introduced a method called distillation to transfer knowledge to smaller models efficiently.
2. Hidden Knowledge
In machine learning, a model typically learns to differentiate between multiple classes by maximizing the probability of the correct outcome. However, it also generates probabilities for incorrect answers, which provides valuable information about the model's generalization.
For example, in MNIST classification, a digit 2 might have a minimal probability of being classified as 3 or 7, but these probabilities, although low, reveal similarities between the data. This information, known as hidden knowledge, allows small models to also learn to generalize.
3. Distillation Procedure
To transfer the generalization ability of the large model to the small one, we use class probabilities as softened targets or soft targets. The process consists of:
- Generating predictions from the large model on the same training set.
- Applying a temperature T to the softmax function to broaden the probability distribution and highlight hidden similarities.
- Training the small model with these soft targets in combination with the actual labels.
This procedure reduces the need for large models without losing prediction capability.
4. Experiment on MNIST
Tests were performed on the MNIST database with two models:
- A small model (784 ? 800 ? 800 ? 10), which without regularization obtained 146 errors in tests.
- A large model (784 ? 1200 ? 1200 ? 10) with advanced training techniques, achieving only 67 errors.
Applying distillation, the small model was able to reduce errors to 74, demonstrating that the technique allows transferring the generalization ability of the large model to the small model.
5. Additional Experiment on MNIST
In an additional experiment, the digit 3 was removed from the training set of the small model. Even without ever having seen a 3, the distilled model managed to correctly predict in 133 out of 1010 test cases. This confirmed that distillation effectively transfers knowledge.
At Q2BSTUDIO, a company specialized in development and technology services, we apply innovations such as knowledge distillation to optimize artificial intelligence models. Our team works on implementing efficient solutions that allow our clients to leverage the latest advances in AI without compromising performance.
