Can I read On the Perception Bottleneck of VLMs for Chart Understanding on EtoBox?

On the Perception Bottleneck of VLMs for Chart Understanding by Liu, Junteng; Zeng, Weihao; Zhang, Xiwen; Wang, Yijun; Shan, Zifei; He, Junxian is a scholarly article available to read on EtoBox.

What is On the Perception Bottleneck of VLMs for Chart Understanding about?

Chart understanding requires models to effectively analyze and reason about numerical data, textual elements, and complex visual components. Our observations reveal that the perception capabilities of existing large vision-language models (LVLMs) constitute a critical bottleneck in this process. In this study, we delve into this perception bottleneck by decomposing it into two components: the vision encoder bottleneck, where the visual representation may fail to encapsulate the correct information, and the extraction bottleneck, where the language model struggles to extract the necessary information from the provided visual representations. Through comprehensive experiments, we find that (1) the information embedded within visual representations is substantially richer than what is typically captured by linear extractors, such as the widely used retrieval accuracy metric; (2) While instruction tuning effectively enhances the extraction capability of LVLMs, the vision encoder remains a critical bottleneck, demanding focused attention and improvement. Therefore, we further enhance the visual encoder to mitigate the vision encoder bottleneck under a contrastive learning framework. Emp

Author

Liu, Junteng; Zeng, Weihao; Zhang, Xiwen; Wang, Yijun; Shan, Zifei; He, Junxian

Published

2025

Language

EN