Applying 80% modality dropout to audio and video features during training reversed negative co-learning on IEMOCAP, lifting unimodal test accuracy from 27% to 47% and beating the unimodal baseline.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This work presents a large-scale audio-visual speech recognition system based on a recurrent neural network transducer (RNN-T) architecture. To support the development of such a system, we built a large audio-visual (A/V) dataset of segmented utterances extracted from YouTube public videos, leading to 31k hours of audio-visual training content. The performance of an audio-only, visual-only, and audio-visual system are compared on two large-vocabulary test sets: a set of utterance segments from public YouTube videos called YTDEV18 and the publicly available LRS3-TED set. To highlight the contribution of the visual modality, we also evaluated the performance of our system on the YTDEV18 set artificially corrupted with background noise and overlapping speech. To the best of our knowledge, our system significantly improves the state-of-the-art on the LRS3-TED set.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Negative to Positive Co-learning with Aggressive Modality Dropout
Applying 80% modality dropout to audio and video features during training reversed negative co-learning on IEMOCAP, lifting unimodal test accuracy from 27% to 47% and beating the unimodal baseline.