Can I read Long-range Multimodal Pretraining for Movie Understanding on EtoBox?

Long-range Multimodal Pretraining for Movie Understanding by Argaw, Dawit Mureja; Lee, Joon-Young; Woodson, Markus; Kweon, In So; Heilbron, Fabian Caba is a scholarly article available to read on EtoBox.

What is Long-range Multimodal Pretraining for Movie Understanding about?

Learning computer vision models from (and for) movies has a long-standing history. While great progress has been attained, there is still a need for a pretrained multimodal model that can perform well in the ever-growing set of movie understanding tasks the community has been establishing. In this work, we introduce Long-range Multimodal Pretraining, a strategy, and a model that leverages movie data to train transferable multimodal and cross-modal encoders. Our key idea is to learn from all modalities in a movie by observing and extracting relationships over a long-range. After pretraining, we run ablation studies on the LVU benchmark and validate our modeling choices and the importance of learning from long-range time spans. Our model achieves state-of-the-art on several LVU tasks while being much more data efficient than previous works. Finally, we evaluate our model's transferability by setting a new state-of-the-art in five different benchmarks.

Author

Argaw, Dawit Mureja; Lee, Joon-Young; Woodson, Markus; Kweon, In So; Heilbron, Fabian Caba

Published

2023

Language

EN