Can I read End-to-End Multimodal Representation Learning for Video Dialog on EtoBox?

End-to-End Multimodal Representation Learning for Video Dialog by Alamri, Huda; Bilic, Anthony; Hu, Michael; Beedu, Apoorva; Essa, Irfan is a scholarly article available to read on EtoBox.

What is End-to-End Multimodal Representation Learning for Video Dialog about?

Video-based dialog task is a challenging multimodal learning task that has received increasing attention over the past few years with state-of-the-art obtaining new performance records. This progress is largely powered by the adaptation of the more powerful transformer-based language encoders. Despite this progress, existing approaches do not effectively utilize visual features to help solve tasks. Recent studies show that state-of-the-art models are biased toward textual information rather than visual cues. In order to better leverage the available visual information, this study proposes a new framework that combines 3D-CNN network and transformer-based networks into a single visual encoder to extract more robust semantic representations from videos. The visual encoder is jointly trained end-to-end with other input modalities such as text and audio. Experiments on the AVSD task show significant improvement over baselines in both generative and retrieval tasks.

Author

Alamri, Huda; Bilic, Anthony; Hu, Michael; Beedu, Apoorva; Essa, Irfan

Published

2022

Language

EN