Can I read Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment on EtoBox?

Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment by Fei, Hao; Wu, Shengqiong; Zhang, Meishan; Zhang, Min; Chua, Tat-Seng; Yan, Shuicheng is a scholarly article available to read on EtoBox.

What is Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment about?

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal align

Author

Fei, Hao; Wu, Shengqiong; Zhang, Meishan; Zhang, Min; Chua, Tat-Seng; Yan, Shuicheng

Published

2024

Language

EN