Can I read Neural Collapse Inspired Knowledge Distillation on EtoBox?

Neural Collapse Inspired Knowledge Distillation by Zhang, Shuoxi; Song, Zijian; He, Kun is a scholarly article available to read on EtoBox.

What is Neural Collapse Inspired Knowledge Distillation about?

Existing knowledge distillation (KD) methods have demonstrated their ability in achieving student network performance on par with their teachers. However, the knowledge gap between the teacher and student remains significant and may hinder the effectiveness of the distillation process. In this work, we introduce the structure of Neural Collapse (NC) into the KD framework. NC typically occurs in the final phase of training, resulting in a graceful geometric structure where the last-layer features form a simplex equiangular tight frame. Such phenomenon has improved the generalization of deep network training. We hypothesize that NC can also alleviate the knowledge gap in distillation, thereby enhancing student performance. This paper begins with an empirical analysis to bridge the connection between knowledge distillation and neural collapse. Through this analysis, we establish that transferring the teacher's NC structure to the student benefits the distillation process. Therefore, instead of merely transferring instance-level logits or features, as done by existing distillation methods, we encourage students to learn the teacher's NC structure. Thereby, we propose a new distillation

Author

Zhang, Shuoxi; Song, Zijian; He, Kun

Published

2024

Language

EN