Can I read Direct Distillation between Different Domains on EtoBox?

Direct Distillation between Different Domains by Tang, Jialiang; Chen, Shuo; Niu, Gang; Zhu, Hongyuan; Zhou, Joey Tianyi; Gong, Chen; Sugiyama, Masashi is a scholarly article available to read on EtoBox.

What is Direct Distillation between Different Domains about?

Knowledge Distillation (KD) aims to learn a compact student network using knowledge from a large pre-trained teacher network, where both networks are trained on data from the same distribution. However, in practical applications, the student network may be required to perform in a new scenario (i.e., the target domain), which usually exhibits significant differences from the known scenario of the teacher network (i.e., the source domain). The traditional domain adaptation techniques can be integrated with KD in a two-stage process to bridge the domain gap, but the ultimate reliability of two-stage approaches tends to be limited due to the high computational consumption and the additional errors accumulated from both stages. To solve this problem, we propose a new one-stage method dubbed ``Direct Distillation between Different Domains" (4Ds). We first design a learnable adapter based on the Fourier transform to separate the domain-invariant knowledge from the domain-specific knowledge. Then, we build a fusion-activation mechanism to transfer the valuable domain-invariant knowledge to the student network, while simultaneously encouraging the adapter within the teacher network to lear

Author

Tang, Jialiang; Chen, Shuo; Niu, Gang; Zhu, Hongyuan; Zhou, Joey Tianyi; Gong, Chen; Sugiyama, Masashi

Published

2024

Language

EN