Can I read Multimodal Grounding for Sequence-to-Sequence Speech Recognition on EtoBox?

Multimodal Grounding for Sequence-to-Sequence Speech Recognition by Caglayan, Ozan; Sanabria, Ramon; Palaskar, Shruti; Barrault, Loïc; Metze, Florian is a scholarly article available to read on EtoBox.

What is Multimodal Grounding for Sequence-to-Sequence Speech Recognition about?

Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/or acoustic context that helps us to resolve ambiguities or to recall named entities. Motivated by this, there have been many works studying the integration of visual information into the speech recognition pipeline. Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system. This approach, however, is not end-to-end as it requires fine-tuning the whole model with an adaptation layer. In this paper, we propose novel end-to-end multimodal ASR systems and compare them to the adaptive approach by using a range of visual representations obtained from state-of-the-art convolutional neural networks. We show that adaptive training is effective for S2S models leading to an absolute improvement of 1.4% in word error rate. As for the end-to-end systems, although they perform better than baseline, the improvements are slightly less than adaptive training, 0.8 absolute WER reduction in single-best

Author

Caglayan, Ozan; Sanabria, Ramon; Palaskar, Shruti; Barrault, Loïc; Metze, Florian

Published

2018

Language

EN