Can I read ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models on EtoBox?

ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models by Ye, Xubing; Gan, Yukang; Ge, Yixiao; Zhang, Xiao-Ping; Tang, Yansong is a scholarly article available to read on EtoBox.

What is ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models about?

Large Vision Language Models (LVLMs) have achieved significant success across multi-modal tasks. However, the computational cost of processing long visual tokens can be prohibitively expensive on resource-limited devices. Previous methods have identified redundancy in visual tokens within the Large Language Model (LLM) decoder layers and have mitigated this by pruning tokens using a pre-defined or fixed ratio, thereby reducing computational overhead. Nonetheless, we observe that the impact of pruning ratio varies across different LLM layers and instances (image-prompt pairs). Therefore, it is essential to develop a layer-wise and instance-wise vision token pruning strategy to balance computational cost and model performance effectively. We propose ATP-LLaVA, a novel approach that adaptively determines instance-specific token pruning ratios for each LLM layer. Specifically, we introduce an Adaptive Token Pruning (ATP) module, which computes the importance score and pruning threshold based on input instance adaptively. The ATP module can be seamlessly integrated between any two LLM layers with negligible computational overhead. Additionally, we develop a Spatial Augmented Pruning (SA

Author

Ye, Xubing; Gan, Yukang; Ge, Yixiao; Zhang, Xiao-Ping; Tang, Yansong

Published

2024

Language

EN