Can I read ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens on EtoBox?

ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens by Guo, Yangyang; Zhang, Haoyu; Wong, Yongkang; Nie, Liqiang; Kankanhalli, Mohan is a scholarly article available to read on EtoBox.

What is ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens about?

Learning a versatile language-image model is computationally prohibitive under a limited computing budget. This paper delves into the \emph{efficient language-image pre-training}, an area that has received relatively little attention despite its importance in reducing computational cost and footprint. To that end, we propose a vision token pruning and merging method ELIP, to remove less influential tokens based on the supervision of language outputs. Our method is designed with several strengths, such as being computation-efficient, memory-efficient, and trainable-parameter-free, and is distinguished from previous vision-only token pruning approaches by its alignment with task objectives. We implement this method in a progressively pruning manner using several sequential blocks. To evaluate its generalization performance, we apply ELIP to three commonly used language-image pre-training models and utilize public image-caption pairs with 4M images for pre-training. Our experiments demonstrate that with the removal of ~30$\%$ vision tokens across 12 ViT layers, ELIP maintains significantly comparable performance with baselines ($\sim$0.32 accuracy drop on average) over various downstr

Author

Guo, Yangyang; Zhang, Haoyu; Wong, Yongkang; Nie, Liqiang; Kankanhalli, Mohan

Published

2023

Language

EN