Can I read Contrastive Learning for Task-Independent SpeechLLM-Pretraining on EtoBox?

Contrastive Learning for Task-Independent SpeechLLM-Pretraining by Züfle, Maike; Niehues, Jan is a scholarly article available to read on EtoBox.

What is Contrastive Learning for Task-Independent SpeechLLM-Pretraining about?

Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.

Author

Züfle, Maike; Niehues, Jan

Published

2024

Language

EN