Can I read SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies on EtoBox?

SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies by Lian, Jiachen; Zhou, Xuanru; Ezzes, Zoe; Vonk, Jet; Morin, Brittany; Baquirin, David; Mille, Zachary; Tempini, Maria Luisa Gorno; Anumanchipalli, Gopala Krishna is a scholarly article available to read on EtoBox.

What is SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies about?

Speech is a hierarchical collection of text, prosody, emotions, dysfluencies, etc. Automatic transcription of speech that goes beyond text (words) is an underexplored problem. We focus on transcribing speech along with non-fluencies (dysfluencies). The current state-of-the-art pipeline SSDM suffers from complex architecture design, training complexity, and significant shortcomings in the local sequence aligner, and it does not explore in-context learning capacity. In this work, we propose SSDM 2.0, which tackles those shortcomings via four main contributions: (1) We propose a novel \textit{neural articulatory flow} to derive highly scalable speech representations. (2) We developed a \textit{full-stack connectionist subsequence aligner} that captures all types of dysfluencies. (3) We introduced a mispronunciation prompt pipeline and consistency learning module into LLM to leverage dysfluency \textit{in-context pronunciation learning} abilities. (4) We curated Libri-Dys and open-sourced the current largest-scale co-dysfluency corpus, \textit{Libri-Co-Dys}, for future research endeavors. In clinical experiments on pathological speech transcription, we tested SSDM 2.0 using nfvPPA corp

Author

Lian, Jiachen; Zhou, Xuanru; Ezzes, Zoe; Vonk, Jet; Morin, Brittany; Baquirin, David; Mille, Zachary; Tempini, Maria Luisa Gorno; Anumanchipalli, Gopala Krishna

Published

2024

Language

EN