Can I read DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention on EtoBox?

DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention by Liu, Fenglin (author);Wu, Xian (author);Ge, Shen (author);Ren, Xuancheng (author);Fan, Wei (author);Sun, Xu (author);Zou, Yuexian (author) is a Computer Science article available to read on EtoBox.

What is DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention about?

Vision-and-language (V-L) tasks require the system to understand both vision content and natural language, thus learning fine-grained joint representations of vision and language (a.k.a. V-L representations) is of paramount importance. Recently, various pre-trained V-L models are proposed to learn V-L representations and achieve improved results in many tasks. However, the mainstream models process both vision and language inputs with the same set of attention matrices. As a result, the generated V-L representations are entangled in one common latent space . To tackle this problem, we propose DiMBERT (short for Di sentangled M ultimodal-Attention BERT ), which is a novel framework that applies separated attention spaces for vision and language, and the representations of multi-modalities can thus be disentangled explicitly. To enhance the correlation between vision and language in disentangled spaces, we introduce the visual concepts to DiMBERT which represent visual information in textual format. In this manner, visual concepts help to bridge the gap between the two modalities. We pre-train DiMBERT on a large amount of image–sentence pairs on two tasks: bidirectional language mode

Who reads DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention?

Author

Liu, Fenglin (author);Wu, Xian (author);Ge, Shen (author);Ren, Xuancheng (author);Fan, Wei (author);Sun, Xu (author);Zou, Yuexian (author)

Publisher

Association for Computing Machinery (ACM)

Published

2022

Language

EN

Field

Computer Science (Physical Sciences)