Can I read SEAL: Semantic Attention Learning for Long Video Representation on EtoBox?

SEAL: Semantic Attention Learning for Long Video Representation by Wang, Lan; Chen, Yujia; Tran, Du; Boddeti, Vishnu Naresh; Chu, Wen-Sheng is a scholarly article available to read on EtoBox.

What is SEAL: Semantic Attention Learning for Long Video Representation about?

Long video understanding presents challenges due to the inherent high computational complexity and redundant temporal information. An effective representation for long videos must process such redundancy efficiently while preserving essential contents for downstream tasks. This paper introduces SEmantic Attention Learning (SEAL), a novel unified representation for long videos. To reduce computational complexity, long videos are decomposed into three distinct types of semantic entities: scenes, objects, and actions, allowing models to operate on a handful of entities rather than a large number of frames or pixels. To further address redundancy, we propose an attention learning module that balances token relevance with diversity formulated as a subset selection optimization problem. Our representation is versatile, enabling applications across various long video understanding tasks. Extensive experiments show that SEAL significantly outperforms state-of-the-art methods in video question answering and temporal grounding tasks and benchmarks including LVBench, MovieChat-1K, and Ego4D.

Author

Wang, Lan; Chen, Yujia; Tran, Du; Boddeti, Vishnu Naresh; Chu, Wen-Sheng

Published

2024

Language

EN