Can I read Test-Time Distribution Normalization for Contrastively Learned Vision-language Models on EtoBox?

Test-Time Distribution Normalization for Contrastively Learned Vision-language Models by Zhou, Yifei; Ren, Juntao; Li, Fengyu; Zabih, Ramin; Lim, Ser-Nam is a scholarly article available to read on EtoBox.

What is Test-Time Distribution Normalization for Contrastively Learned Vision-language Models about?

Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations. One of the most representative approaches proposed recently known as CLIP has garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. To this end, we propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE

Author

Zhou, Yifei; Ren, Juntao; Li, Fengyu; Zabih, Ramin; Lim, Ser-Nam

Published

2023

Language

EN