Can I read Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data on EtoBox?

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data by Kang, Yu; Liu, Tianqiao; Li, Hang; Hao, Yang; Ding, Wenbiao is a scholarly article available to read on EtoBox.

What is Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data about?

Multimodal pre-training for audio-and-text has recently been proved to be effective and has significantly improved the performance of many downstream speech understanding tasks. However, these state-of-the-art pre-training audio-text models work well only when provided with large amount of parallel audio-and-text data, which brings challenges on many languages that are rich in unimodal corpora but scarce of parallel cross-modal corpus. In this paper, we investigate whether it is possible to pre-train an audio-text multimodal model with extremely low-resource parallel data and extra non-parallel unimodal data. Our pre-training framework consists of the following components: (1) Intra-modal Denoising Auto-Encoding (IDAE), which is able to reconstruct input text (audio) representations from a noisy version of itself. (2) Cross-modal Denoising Auto-Encoding (CDAE), which is pre-trained to reconstruct the input text (audio), given both a noisy version of the input text (audio) and the corresponding translated noisy audio features (text embeddings). (3) Iterative Denoising Process (IDP), which iteratively translates raw audio (text) and the corresponding text embeddings (audio features)

Author

Kang, Yu; Liu, Tianqiao; Li, Hang; Hao, Yang; Ding, Wenbiao

Published

2022

Language

EN