Can I read PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance on EtoBox?

PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance by Liu, Ruyang; Tang, Haoran; Liu, Haibo; Ge, Yixiao; Shan, Ying; Li, Chen; Yang, Jiankun is a scholarly article available to read on EtoBox.

What is PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance about?

The past year has witnessed the significant advancement of video-based large language models. However, the challenge of developing a unified model for both short and long video understanding remains unresolved. Most existing video LLMs cannot handle hour-long videos, while methods custom for long videos tend to be ineffective for shorter videos and images. In this paper, we identify the key issue as the redundant content in videos. To address this, we propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three core components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user's instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue. Moreover, our codebase also integrates the most advanced video Direct Preference Optimization (DPO) and visual interleave training. Extensive experiments have validated the per

Author

Liu, Ruyang; Tang, Haoran; Liu, Haibo; Ge, Yixiao; Shan, Ying; Li, Chen; Yang, Jiankun

Published

2024

Language

EN