Can I read HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training on EtoBox?

HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training by Ye, Qinghao; Xu, Guohai; Yan, Ming; Xu, Haiyang; Qian, Qi; Zhang, Ji; Huang, Fei is a scholarly article available to read on EtoBox.

What is HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training about?

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SS

Author

Ye, Qinghao; Xu, Guohai; Yan, Ming; Xu, Haiyang; Qian, Qi; Zhang, Ji; Huang, Fei

Published

2022

Language

EN