Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that mWhisper-Flamingo, a late-fusion audio-visual model that trains Whisper's decoder on video-only and audio-visual inputs, beats audio-only Whisper on all nine MuAViC languages in noisy conditions and sets a new state…

desk verdict Solid multilingual AVSR extension with a genuinely useful dropout trick, but the abstract overstates the 'all languages' noise-robustness claim; send to review with requested revisions. read the letter →

arxiv 2502.01547 v3 pith:AC3QYYT5 submitted 2025-02-03 eess.AS cs.CVcs.SD

Andrew Rouditchenko , Samuel Thomas , Hilde Kuehne , Rogerio Feris , James Glass This is my paper · ORCID
classification eess.AScs.CVcs.SD
keywords audio-visualspeechrecognitionmultilingualnoiserobustnessdecodermodalitydropoutWhisperAV-HuBERTMuAViClipreading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that multilingual audio-visual speech recognition can be built by attaching a pre-trained lip-reading video encoder to Whisper rather than training from scratch. The proposed model, mWhisper-Flamingo, combines Whisper's multilingual audio encoder and decoder with a multilingual AV-HuBERT visual encoder, and adds decoder modality dropout: during training, either both modalities or only the video stream is fed to the decoder, with the missing modality's features zeroed out. The authors claim this makes the decoder actually use lip information, and report that the audio-visual model consistently beats audio-only Whisper on all nine MuAViC languages in noisy conditions, with roughly a 10% relative improvement in average non-English WER at 0-dB babble. If true, this offers a practical recipe for making speech recognition robust to noise in languages that lack large audio-visual training sets.

What carries the argument

The load-bearing mechanism is decoder modality dropout applied in a late-fusion decoder. Whisper's decoder already contains separate cross-attention layers for audio and video features; during training, the model randomly replaces one modality's embedding sequence by a zero vector, so the decoder must learn to transcribe from video-only input as well as from audio-visual input. The decisive configuration is to drop audio half the time ($p_V=0.5$) and never drop video ($p_A=0$), because Whisper was already fine-tuned on audio and the bottleneck is teaching the decoder to exploit the lip stream. The visual features come from a multilingual AV-HuBERT encoder pre-trained on multilingual videos, whose parameters are fine-tuned during the audio-visual stage, unlike the frozen encoder in the original Whisper-Flamingo.

What would settle it

Take the trained mWhisper-Flamingo model and decode each MuAViC language with the audio stream zeroed at test time, leaving video-only input. If on any lower-resource language, such as Arabic, German, Greek, or Russian, the video-only WER is close to the chance level for that language, or no better than the audio-only model's output, then the assumption that the multilingual lip encoder supports those languages is false and the claimed mechanism would not hold there.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a late-fusion Whisper decoder can be taught to integrate visual speech by randomly zeroing out one modality's features during training. mWhisper-Flamingo replaces Whisper-Flamingo's English-only visual encoder with a multilingual AV-HuBERT, fine-tunes that encoder, and trains with probabilities $p_{AV}=0.5$, $p_A=0$, $p_V=0.5$, meaning half the steps present audio-visual inputs and half present video-only inputs. The ablation shows that without this dropout the average non-English WER on a five-language noisy subset is 44.6%, and with it the WER drops to 36.6%. At 0-dB babble on the full nine-language MuAViC benchmark, mWhisper-Flamingo small reaches 50.4% average non-English WER versus 55.3% for fine-tuned audio-only Whisper small, and the medium model reaches 43.7% versus 48.0%.

Load-bearing premise

The method works only if the multilingual AV-HuBERT lip encoder provides genuinely useful visual features for all nine MuAViC languages, including the four low-resource languages with as little as 10 hours of video; if the lip stream is uninformative for those languages, training the decoder half the time from video alone would hurt rather than help.

Editorial extensions

If this is right

  • At 0-dB babble, mWhisper-Flamingo small cuts average non-English WER from 55.3% to 50.4% relative to fine-tuned audio-only Whisper small; the medium model cuts 48.0% to 43.7%.
  • The audio-visual model beats audio-only Whisper across six noise types and five SNR levels on Spanish, French, Italian, and Portuguese, with the largest gains on babble and overlapping speech.
  • A smaller audio-visual model (651M parameters) outperforms a larger audio-only model (769M parameters) on the higher-resource languages under 0-dB babble, 37.4% versus 38.1% WER.
  • On clean MuAViC audio, mWhisper-Flamingo medium achieves a non-English average WER of 20.4%, matching fine-tuned audio-only Whisper (20.1%) and beating all prior audio-visual models trained on the same 1,141 hours.
  • Ablations identify video-only dropout as the critical ingredient: removing it raises noisy non-English WER from 36.6% to 44.6%, while replacing it with audio-only dropout gives no improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same late-fusion dropout recipe could be applied to other multimodal systems, such as audio-visual translation or video question answering, whenever one modality is reliable and the other is informative but hard to learn from.
  • The result suggests a cheap route to noise-robust ASR for additional languages: keep Whisper's audio backbone and swap in a lip-reading encoder trained on that language's video, using video-heavy modality dropout, rather than collecting tens of thousands of hours of transcribed video.
  • Because video-only training mattered more than audio-only training, a testable prediction follows: mWhisper-Flamingo's advantage over audio-only Whisper should grow as SNR drops, and the model should decode from silent video alone with WER far below chance on every MuAViC language.
  • The fact that fine-tuning the visual encoder mattered more when dropout was enabled (36.6 versus 40.6) suggests the dropout itself creates the gradient pressure that makes visual fine-tuning useful; varying $p_V$ while keeping the encoder frozen would test this directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes mWhisper-Flamingo, a multilingual audio-visual speech recognition model that combines Whisper with a multilingual AV-HuBERT visual encoder, using Flamingo-style gated cross-attention layers in the Whisper decoder. The key methodological addition is decoder modality dropout, which randomly zeroes either the audio or video embedding sequence at the decoder cross-attention input so the model trains on paired audio-visual, audio-only, and video-only inputs. Experiments on the 9-language MuAViC benchmark report clean-condition audio-visual SOTA results, a 0-dB babble average non-English WER improvement of about 10.4% relative for small and 10.6% for medium models over fine-tuned audio-only Whisper, multi-noise results for four higher-resource languages, and an ablation showing the dropout configuration and fine-tunable visual encoder are important. The paper releases code and models.

Significance. If the claims are scoped appropriately, the contribution is solid: it extends the Whisper-Flamingo approach to multilingual input, demonstrates a simple and transferable decoder-side modality dropout technique, and shows consistent average gains on the higher-resource MuAViC languages in challenging babble noise. The 0-dB babble results for Es, Fr, It, and Pt are internally consistent, and the ablation in Table III cleanly attributes the gain to the combination of a fine-tunable multilingual visual encoder and decoder modality dropout. The code and model release is a concrete reproducibility strength. The main weakness is that the abstract and introduction overclaim language coverage: the paper's own Table II shows no meaningful improvement for Arabic at 0-dB babble, so the 'all languages' noise-robustness statement is not supported as written. This is a correctness-of-claims issue rather than a methodological flaw, and it is fixable by re-scoping.

major comments (3)
  1. [Abstract and Section I] The abstract states that audio-visual mWhisper-Flamingo 'consistently outperforms audio-only Whisper on all languages in noisy conditions,' but Table II (0-dB babble, small models) shows Arabic WER of 101 for both fine-tuned Whisper small and mWhisper-Flamingo small, a relative change of -0.8%; for the medium model the Arabic gain is only 96.4 to 95.3, i.e., 1.1% relative. The only 9-language noisy result is the single babble condition in Table II, while the multi-noise, multi-SNR results in Figure 2 and Tables A1/A2 cover only Es, Fr, It, and Pt. The claim should be revised to state that the improvement is on average over non-English languages, or specifically for higher-resource languages, and the Arabic result should be discussed explicitly rather than hidden in an average.
  2. [Section III-B and Table I] The clean-condition results do not support the unqualified 'state-of-the-art WER on MuAViC' statement. In Table I, fine-tuned audio-only Whisper medium achieves 20.1% average non-English WER, while mWhisper-Flamingo medium achieves 20.4%; for small models the numbers are 23.5% versus 23.8%. The video stream therefore provides no measurable clean-audio benefit in these experiments. The SOTA claim should be restricted to audio-visual models fine-tuned on MuAViC, and the paper should explicitly acknowledge that the reported video benefit appears only in noisy conditions.
  3. [Section III-C and Figure 2] The multi-noise, multi-SNR evidence is limited to four higher-resource languages (Es, Fr, It, Pt), but the conclusion claims the model 'outperforms audio-only Whisper in diverse noise settings' without the same language scope. Given the Arabic counterexample in Table II, the generalization from four languages to 'diverse noise settings' overall is not established. The authors should either report multi-noise results for low-resource languages or explicitly scope the conclusion to the four evaluated languages.
minor comments (6)
  1. [Section III-A] The text says 'other dataloading details and hyperparameters closely follow Whisper-Flamingo [27]', but reference [27] is AV-CPL, not Whisper-Flamingo [17]. Either the citation is incorrect or it should cite [17].
  2. [Section III-D] There is a typo: 'the modal trains only on audio-visual inputs' should read 'the model trains only on audio-visual inputs'.
  3. [Section III-B] The definition of 'Avg non-En' is implicit. State explicitly that it is the arithmetic mean of the per-language WERs excluding English, and similarly for the H.R. and L.R. averages, to avoid ambiguity.
  4. [Figure 2] The caption states the WER is averaged over four languages and five SNR levels, but the aggregation is not fully specified. Clarify whether this is an average of per-language WERs or a pooled error rate, and add error bars if multiple evaluation runs are available.
  5. [Table III] The ablation is run on only five of the nine languages (En, Es, Fr, It, Pt), but the text refers to the result as 'the average non-En WER.' State the language subset explicitly in the table caption or in the surrounding text.
  6. [Section III-A] The paper does not specify how the MuAViC train, validation, and test splits are used, or that the validation set is used for checkpoint selection on the noisy condition. Adding this detail would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out benchmark results with externally pre-trained components; no fitted parameter is renamed as a prediction.

full rationale

This is an empirical systems paper, not a derivation chain: the reported WERs are held-out test numbers on the external MuAViC benchmark (Tables I, II, A1, A2), and the central comparison against fine-tuned audio-only Whisper is an independent benchmark outcome rather than a consequence of the training objective. The only tuned hyperparameters, the decoder modality dropout probabilities (pAV=0.5, pA=0, pV=0.5), are selected on the noisy validation set via token prediction accuracy (Section III-A) and are not renamed as predictions; the test WERs are measured after model selection. The visual encoder is an externally pre-trained multilingual AV-HuBERT [19], not a component whose evaluation criterion is the same as the paper's target, and the paper's ablation (Table III) isolates decoder modality dropout as an empirically verified cause of improvement against fixed baselines. The one self-citation to the authors' prior Whisper-Flamingo [17] serves as an architectural starting point and as a baseline, not as load-bearing evidence for the central noise-robustness claim. A non-circular internal-consistency issue exists: the abstract's 'all languages in noisy conditions' claim is contradicted by Table II for Arabic (small-model relative improvement -0.8%, with 101 WER for both audio-only and audio-visual), and Figure 2's multi-noise evaluation covers only four high-resource languages. That overbroad scoping is a correctness/claim-precision issue, not circularity, because the Arabic and Figure 2 numbers are independently measured held-out results rather than quantities derived from the claim itself.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the choice of dropout probabilities, which are fitted on the validation set, and on the transferability of pre-trained models. No new physical or conceptual entities are introduced.

free parameters (1)
  • decoder modality dropout probabilities = pAV=0.5, pA=0, pV=0.5
    Selected via ablation study (Table III) on the MuAViC validation set; the paper reports this setting works best for noisy multilingual WER.
assumptions (3)
  • domain assumption The pre-trained Whisper and AV-HuBERT models provide useful initializations for the target task.
    The method relies on features from models pre-trained on large external datasets (Section II, III-A). If these features were not transferable, fine-tuning on 1,141 hours of MuAViC would not suffice.
  • domain assumption MuAViC video frames are correctly lip-cropped and time-aligned with audio.
    The visual encoder consumes grayscale lip crops aligned to a reference face (Section III-A); misalignment would make the visual modality uninformative.
  • ad hoc to paper Zeroing a modality's embedding sequence at the cross-attention input is a valid proxy for a missing modality.
    The authors acknowledge that cross-attention with a zero vector yields a zero vector but retains biases (Section II). This is a core assumption of decoder modality dropout.

how reviews work

0 comments
Cite this review

Pith. "Pith review of mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition." pith.science (2026). https://pith.science/paper/AC3QYYT5

@misc{pith2026250201547,
  author       = {Pith},
  title        = {Pith review of: mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AC3QYYT5}},
  note         = {Machine review of arXiv:2502.01547}
}
read the original abstract

Audio-Visual Speech Recognition (AVSR) combines lip-based video with audio and can improve performance in noise, but most methods are trained only on English data. One limitation is the lack of large-scale multilingual video data, which makes it hard to train models from scratch. In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT). To enable better multi-modal integration and improve the noisy multilingual performance, we introduce decoder modality dropout where the model is trained both on paired audio-visual inputs and separate audio/visual inputs. mWhisper-Flamingo achieves state-of-the-art WER on MuAViC, an AVSR dataset of 9 languages. Audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions.

Figures

Figures reproduced from arXiv: 2502.01547 by the authors.

Figure 1
Figure 1. In mWhisper-Flamingo, the AV-HuBERT and Whisper encoders [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Multilingual WER (↓ is better) for different noise types averaged over 4 languages (Es, Fr, It, Pt) and 5 SNR levels {−10, −5, 0, 5, 10}. indicating that more video training data is helpful. The relative improvements for English are much better at 48.4% and 39.8% for the small and medium models, which we attribute to having more English training data. mWhisper-Flamingo medium achieves the best multilingual WER (43.7… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

    cs.SD 2025-05 conditional novelty 5.0 of 10

    CoGenAV learns audio-visual speech representations that achieve 1.27% WER on LRS2 AVSR and 20.5% WER on LRS2 VSR using 223 hours of labeled data.

Reference graph

Works this paper leans on

51 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” in ICML, 2023

    2023

  2. [2]

    Less is more: Accurate speech recognition & translation without web-scale data,

    K. C. Puvvada, P. ˙Zelasko, H. Huang, O. Hrinchuk, N. R. Koluguri, K. Dhawan, S. Majumdar, E. Rastorgueva, Z. Chen, V . Lavrukhin, J. Balam, and B. Ginsburg, “Less is more: Accurate speech recognition & translation without web-scale data,” in Interspeech, 2024

    work page 2024

  3. [3]

    Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre- Training for Adaptation to Unseen Languages,

    A. Rouditchenko, S. Khurana, S. Thomas, R. Feris, L. Karlinsky, H. Kuehne, D. Harwath, B. Kingsbury, and J. Glass, “Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre- Training for Adaptation to Unseen Languages,” in Interspeech, 2023

    work page 2023

  4. [4]

    Ml-superb: Multilingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen, E.-P. Hu, W.-P. Huang, H.-L. Chung, X. Chang, S.-W. Li, A. Mohamed, H. yi Lee, and S. Watanabe, “Ml-superb: Multilingual speech universal performance benchmark,” in Interspeech, 2023

    work page 2023

  5. [5]

    Whisper-AT: Noise- Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers,

    Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-AT: Noise- Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers,” in Interspeech, 2023

    work page 2023

  6. [6]

    Large language models are efficient learners of noise-robust speech recognition,

    Y . Hu, C. CHEN, C.-H. H. Yang, R. Li, C. Zhang, P.-Y . Chen, and E. Chng, “Large language models are efficient learners of noise-robust speech recognition,” in ICLR, 2024

    work page 2024

  7. [7]

    Deep audio-visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” IEEE TPAMI, 2018

    work page 2018

  8. [8]

    Audio-visual speech recognition with a hybrid ctc/attention architec- ture,

    S. Petridis, T. Stafylakis, P. Ma, G. Tzimiropoulos, and M. Pantic, “Audio-visual speech recognition with a hybrid ctc/attention architec- ture,” in SLT, 2018

    work page 2018

Show all 51 references
  1. [9]

    End-to-end audiovisual speech recognition,

    S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic, “End-to-end audiovisual speech recognition,” in ICASSP, 2018

    2018

  2. [10]

    Discriminative multi-modality speech recognition,

    B. Xu, C. Lu, Y . Guo, and J. Wang, “Discriminative multi-modality speech recognition,” in CVPR, 2020

    2020

  3. [11]

    End-to-end audio-visual speech recognition with conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in ICASSP, 2021

    2021

  4. [12]

    Transformer-Based Video Front- Ends for Audio-Visual Speech Recognition for Single and Muti-Person Video,

    D. Serdyuk, O. Braga, and O. Siohan, “Transformer-Based Video Front- Ends for Audio-Visual Speech Recognition for Single and Muti-Person Video,” in Interspeech, 2022

    2022

  5. [13]

    Robust Self-Supervised Audio- Visual Speech Recognition,

    B. Shi, W.-N. Hsu, and A. Mohamed, “Robust Self-Supervised Audio- Visual Speech Recognition,” in Interspeech, 2022

    2022

  6. [14]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” in ICASSP, 2023

    2023

  7. [15]

    Audio-visual efficient conformer for robust speech recognition,

    M. Burchi and R. Timofte, “Audio-visual efficient conformer for robust speech recognition,” in WACV, 2023

    2023

  8. [16]

    Large language models are strong audio- visual speech recognition learners,

    U. Cappellazzo, M. Kim, H. Chen, P. Ma, S. Petridis, D. Falavigna, A. Brutti, and M. Pantic, “Large language models are strong audio- visual speech recognition learners,” arXiv preprint arXiv:2409.12319 , 2024

    2024 arXiv

  9. [17]

    Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation,

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation,” in Interspeech, 2024

    2024

  10. [18]

    Learning audio- visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio- visual speech representation by masked multimodal cluster prediction,” in ICLR, 2022

    2022

  11. [19]

    Efficient training for multilingual visual speech recognition: Pre-training with discretized visual speech representation,

    M. Kim, J. Yeo, S. J. Park, H. Rha, and Y . M. Ro, “Efficient training for multilingual visual speech recognition: Pre-training with discretized visual speech representation,” in MM, 2024

    2024

  12. [20]

    Muavic: A multilingual audio-visual corpus for robust speech recogni- tion and robust speech-to-text translation,

    M. Anwar, B. Shi, V . Goswami, W.-N. Hsu, J. Pino, and C. Wang, “Muavic: A multilingual audio-visual corpus for robust speech recogni- tion and robust speech-to-text translation,” in Interspeech, 2023

    2023

  13. [21]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” NeurIPS, 2022

    2022

  14. [22]

    Improving neural networks by preventing co-adaptation of feature detectors,

    G. Hinton, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012

    2012 arXiv

  15. [23]

    Moddrop: adaptive multi-modal gesture recognition,

    N. Neverova, C. Wolf, G. Taylor, and F. Nebout, “Moddrop: adaptive multi-modal gesture recognition,” TPAMI, 2015

    2015

  16. [24]

    Recurrent neural network transducer for audio-visual speech recognition,

    T. Makino, H. Liao, Y . Assael, B. Shillingford, B. Garcia, O. Braga, and O. Siohan, “Recurrent neural network transducer for audio-visual speech recognition,” in ASRU, 2019

    2019

  17. [25]

    u-hubert: Unified mixed-modal speech pretrain- ing and zero-shot transfer to unlabeled modality,

    W.-N. Hsu and B. Shi, “u-hubert: Unified mixed-modal speech pretrain- ing and zero-shot transfer to unlabeled modality,” NeurIPS, 2022

    2022

  18. [26]

    Av-data2vec: Self- supervised learning of audio-visual speech representations with contex- tualized target representations,

    J. Lian, A. Baevski, W.-N. Hsu, and M. Auli, “Av-data2vec: Self- supervised learning of audio-visual speech representations with contex- tualized target representations,” ASRU, 2023

    2023

  19. [27]

    Av-cpl: Continu- ous pseudo-labeling for audio-visual speech recognition,

    A. Rouditchenko, R. Collobert, and T. Likhomanenko, “Av-cpl: Continu- ous pseudo-labeling for audio-visual speech recognition,” arXiv preprint, 2023

    2023

  20. [28]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017

    2017

  21. [29]

    Reducing transformer depth on demand with structured dropout,

    A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” in ICLR, 2020

    2020

  22. [30]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint, 2018

    2018

  23. [31]

    The multilingual tedx corpus for speech recognition and translation,

    E. Salesky, M. Wiesner, J. Bremerman, R. Cattoni, M. Negri, M. Turchi, D. W. Oard, and M. Post, “The multilingual tedx corpus for speech recognition and translation,” in Interspeech, 2021

    2021

  24. [32]

    Jointly learning visual and auditory speech representations from raw data,

    A. Haliassos, P. Ma, R. Mira, S. Petridis, and M. Pantic, “Jointly learning visual and auditory speech representations from raw data,” in ICLR, 2023

    2023

  25. [33]

    Braven: Improving self-supervised pre-training for visual and auditory speech recognition,

    A. Haliassos, A. Zinonos, R. Mira, S. Petridis, and M. Pantic, “Braven: Improving self-supervised pre-training for visual and auditory speech recognition,” in ICASSP, 2024

    2024

  26. [34]

    Unified speech recognition: A single model for auditory, visual, and audiovisual inputs,

    A. Haliassos, R. Mira, H. Chen, Z. Landgraf, S. Petridis, and M. Pantic, “Unified speech recognition: A single model for auditory, visual, and audiovisual inputs,” in NeurIPS, 2024

    2024

  27. [35]

    Visual speech recognition for multiple languages in the wild,

    P. Ma, S. Petridis, and M. Pantic, “Visual speech recognition for multiple languages in the wild,” Nature Machine Intelligence , 2022

    2022

  28. [36]

    Learning cross-lingual visual speech representations,

    A. Zinonos, A. Haliassos, P. Ma, S. Petridis, and M. Pantic, “Learning cross-lingual visual speech representations,” in ICASSP, 2023

    2023

  29. [37]

    Lip reading for low-resource languages by learning and combining general speech knowledge and language-specific knowledge,

    M. Kim, J. H. Yeo, J. Choi, and Y . M. Ro, “Lip reading for low-resource languages by learning and combining general speech knowledge and language-specific knowledge,” in ICCV, 2023

    2023

  30. [38]

    Visual speech recognition for languages with limited labeled data using automatic labels from whisper,

    J. H. Yeo, M. Kim, S. Watanabe, and Y . M. Ro, “Visual speech recognition for languages with limited labeled data using automatic labels from whisper,” in ICASSP, 2024

    2024

  31. [39]

    Dlib-ml: A machine learning toolkit,

    D. E. King, “Dlib-ml: A machine learning toolkit,” The Journal of Machine Learning Research , 2009

    2009

  32. [40]

    Lipreading using temporal convolutional networks,

    B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” in ICASSP, 2020

    2020

  33. [41]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke et al. , “Pytorch: An imperative style, high-performance deep learning library,” NeurIPS, 2019

    2019

  34. [42]

    PyTorch Lightning,

    W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,” 2023

    2023

  35. [43]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, 2019

    2019

  36. [44]

    Musan: A music, speech, and noise corpus,

    D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” arXiv preprint, 2015

    2015

  37. [45]

    Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech perception,

    H. Han, M. Anwar, J. Pino, W.-N. Hsu, M. Carpuat, B. Shi, and C. Wang, “Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech perception,” ACL, 2024

    2024

  38. [46]

    Intuitive multilingual audio-visual speech recognition with a single-trained model,

    J. Hong, S. Park, and Y . Ro, “Intuitive multilingual audio-visual speech recognition with a single-trained model,” in Findings of EMNLP , 2023

    2023

  39. [47]

    Multilingual audio-visual speech recognition with hybrid ctc/rnn-t fast conformer,

    M. Burchi et al. , “Multilingual audio-visual speech recognition with hybrid ctc/rnn-t fast conformer,” in ICASSP, 2024

    2024

  40. [48]

    Parameter-efficient cross-language transfer learning for a language- modular audiovisual speech recognition,

    Z. Li, T. Graave, J. Liu, T. Lohrenz, S. Kunzmann, and T. Fingscheidt, “Parameter-efficient cross-language transfer learning for a language- modular audiovisual speech recognition,” in ASRU, 2023

    2023

  41. [49]

    Tailored design of audio-visual speech recognition models using branchformers,

    D. Gimeno-G ´omez and C.-D. Mart ´ınez-Hinarejos, “Tailored design of audio-visual speech recognition models using branchformers,” arXiv preprint arXiv:2407.06606, 2024

    2024 arXiv

  42. [50]

    Interleaved audio/audiovisual transfer learning for av- asr in low-resourced languages,

    Z. Li, P. Blumenberg, J. Liu, T. Graave, T. Lohrenz, S. Kunzmann, and T. Fingscheidt, “Interleaved audio/audiovisual transfer learning for av- asr in low-resourced languages,” in Interspeech, 2024

    2024

  43. [51]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. V on Platen, Y . Saraf, J. Pinoet al. , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021. V. A PPENDIX A. Full Noisy Results Table A1 a...

    2021 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.