Can I read MOMA:Distill from Self-Supervised Teachers on EtoBox?

MOMA:Distill from Self-Supervised Teachers by Yao, Yuchong; Desai, Nandakishor; Palaniswami, Marimuthu is a scholarly article available to read on EtoBox.

What is MOMA:Distill from Self-Supervised Teachers about?

Contrastive Learning and Masked Image Modelling have demonstrated exceptional performance on self-supervised representation learning, where Momentum Contrast (i.e., MoCo) and Masked AutoEncoder (i.e., MAE) are the state-of-the-art, respectively. In this work, we propose MOMA to distill from pre-trained MoCo and MAE in a self-supervised manner to collaborate the knowledge from both paradigms. We introduce three different mechanisms of knowledge transfer in the propsoed MOMA framework. : (1) Distill pre-trained MoCo to MAE. (2) Distill pre-trained MAE to MoCo (3) Distill pre-trained MoCo and MAE to a random initialized student. During the distillation, the teacher and the student are fed with original inputs and masked inputs, respectively. The learning is enabled by aligning the normalized representations from the teacher and the projected representations from the student. This simple design leads to efficient computation with extremely high mask ratio and dramatically reduced training epochs, and does not require extra considerations on the distillation target. The experiments show MOMA delivers compact student models with comparable performance to existing state-of-the-art methods

Author

Yao, Yuchong; Desai, Nandakishor; Palaniswami, Marimuthu

Published

2023

Language

EN