Pith. sign in

REVIEW 4 major objections 4 minor 51 references

CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoGenAV claims a frozen Whisper ASR head can decode lip-movement features, reaching 1.27% WER on LRS2 AVSR.

desk verdict Solid AV-speech representation paper with real-looking gains, but the headline numbers are not cleanly separated from AV-HuBERT pretraining. read the letter →

arxiv 2505.03186 v2 pith:7ZKXMR65 submitted 2025-05-06 cs.SD cs.CVeess.AS

Detao Bai , Zhiheng Ma , Xihan Wei , Liefeng Bo This is my paper · ORCID
classification cs.SDcs.CVeess.AS
keywords audio-visualspeechrecognitionvisualcontrastivelearninggenerativetextpredictionfrozenASRheadlipreadingnoiserobustnessseparation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoGenAV learns audio-visual speech representations from synchronized audio, video, and text, trained on 223 hours of LRS2. Its central claim is that combining a sequence-level contrastive loss, which aligns audio and visual features frame by frame, with a generative loss, which forces those features to be decodable by a frozen pre-trained Whisper ASR model, yields representations that transfer across many speech tasks. The paper reports a word error rate (WER) of 1.27% on LRS2 audio-visual speech recognition, 20.5% WER on visual-only speech recognition, and a reduction of more than 70% in WER over a fine-tuned audio-only Whisper baseline under 0 dB noise. The same frozen features improve speech enhancement and separation and perform competitively on active speaker detection. The result matters because it suggests a small labeled dataset plus a frozen ASR decoder can provide much of the robustness that audio-visual models usually require large-scale training to achieve.

What carries the argument

The machinery is a dual objective trained through a frozen ASR head. The contrastive part is a sequence-to-sequence contrastive loss: frame-wise cosine similarity between audio and video features, ReLU-clipped and averaged over time, trained with binary cross-entropy against synchrony labels. The generative part is a log-likelihood loss computed by a frozen pre-trained Whisper model serving as a speech-recognition head; before features reach Whisper they pass through a Feature Adaptation Module made of a $\Delta$ Upsampler, which doubles temporal resolution using temporal-difference convolution, and a GatedFFN-MHA layer, which uses multi-head attention and a gated feed-forward network to close modality mismatch. The combination lets the model learn features that are simultaneously time-aligned across modalities and semantically decodable as text.

What would settle it

Train the same CoGenAV architecture on the same LRS2 data but replace Whisper as the frozen SR head with a randomly initialized ASR encoder-decoder of matched size: if visual-only WER stays near 20.5%, the frozen-ASR assumption is not load-bearing; if it collapses, the result depends on Whisper's pretrained latent space. Alternatively, if an independent replication with the released code cannot reproduce the 1.27% clean AVSR WER on LRS2, the central claim is falsified.

Watch

Extended reading notes

Core claim

CoGenAV establishes a tri-modal alignment among audio, visual, and text streams by jointly training a visual/audio encoder with two complementary objectives. The contrastive objective computes temporally averaged, ReLU-clipped frame-wise cosine similarities between audio-only and video-only features and trains the model to tell synchronized pairs from misaligned or cross-speaker pairs. The generative objective feeds adapted features, sometimes visual-only, sometimes audio-only, sometimes both, into a frozen Whisper encoder-decoder and maximizes the log-likelihood of the ground-truth transcription. Because Whisper's weights stay frozen, all learning pressure is on the CoGenAV backbone and a small Feature Adaptation Module, which upsamples 25 fps features to 50 fps and applies gated attention before decoding. The paper's core discovery is that this contrastive-generative synchronization captures both fine-grained temporal correspondence and linguistic content, producing a single visual representation that reaches 1.27% WER on LRS2 AVSR and 20.5% WER on VSR, and that remains effective as a frozen visual feature for enhancement, separation, and active speaker detection.

Load-bearing premise

The load-bearing premise is that a frozen pre-trained ASR model like Whisper defines a fixed acoustic-to-text latent space, and that lip-movement features can be mapped into that space by a small learned adapter; if that mapping cannot be learned, the generative loss stops teaching the visual encoder and the visual and noisy AVSR results collapse.

Editorial extensions

If this is right

  • Large ASR models can be reused as frozen decoders for lip-reading rather than fine-tuned, avoiding catastrophic forgetting and heavy compute.
  • Visual-only and audio-visual speech recognition are obtained from the same checkpoint, since the same visual encoder feeds either input stream to the frozen Whisper head.
  • Noisy speech recognition gains a new recipe: align noisy audio with visual features during training, then rely on the visual stream at test time.
  • The representation is not tied to Whisper's fine-tuning data, so the same pretrained features should transfer to datasets beyond LRS2.
  • Frozen CoGenAV visual features can serve as drop-in replacements for task-specific visual encoders in separation, enhancement, and active speaker detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 70% noisy-AVSR improvement likely comes from the contrastive loss teaching the visual stream to carry the information that noisy audio loses; the ablations show removing the contrastive term hurts noisy AVSR more than clean VSR.
  • A testable extension is to swap the frozen Whisper head for a different large ASR model; if visual-only WER stays low, the adaptation generalizes across ASR latent spaces, while a collapse would show the result is Whisper-specific.
  • Because the visual encoder is not limited by Whisper's training data, scaling CoGenAV to multilingual or unlabeled video could be done by pairing it with a multilingual frozen ASR head.
  • The paper's limitation statement names LRS3 as the next evaluation target; if the gains hold there, the 223-hour data-efficiency claim becomes a general property rather than an LRS2 artifact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CoGenAV, an audio-visual representation learning method trained on 223 hours of LRS2 with a dual objective: a sequence-to-sequence contrastive alignment loss (LCo) between audio and visual features, and a generative text-prediction loss (LGen) through a frozen Whisper ASR head, using a lightweight Feature Adaptation Module. The authors report a VSR WER of 20.5 on LRS2, a clean AVSR WER of 1.27 with a fine-tuned Whisper head, a noisy AVSR WER of 2.6 at 0 dB SNR, and improvements on AVSS, AVSE, and ASD. The central claim is that the contrastive-generative synchronization objective learns versatile, data-efficient audio-visual representations that transfer across tasks.

Significance. If the central attribution is validated, the paper would make a useful contribution: it demonstrates a relatively simple recipe for adapting a frozen ASR model to visual and audio-visual inputs, and the open-source release would support reproducibility. The reported versatility across recognition, reconstruction, and synchronization tasks is attractive. However, the current evidence does not isolate the effect of the proposed objectives from the AV-HuBERT initialization, and the headline clean-AVSR SOTA margin is small and unreplicated. The significance therefore depends on additional ablations that are missing; the contribution is potentially valuable but not yet established at the level claimed.

major comments (4)
  1. [Section 3.3 and Tables 1-3] The visual encoder and transformer encoder are initialized from AV-HuBERT weights, a model pretrained on substantially more audio-visual data. The paper never ablates this initialization: no experiment compares CoGenAV to the same downstream pipeline (same Feature Adaptation Module, same frozen Whisper head) using frozen AV-HuBERT features or randomly initialized encoders. Because AV-HuBERT already provides strong audio-visual representations, the reported VSR (20.5), noisy AVSR (2.6), and clean AVSR (1.27) numbers could be substantially inherited from pretraining rather than produced by LCo and LGen. This also undermines the data-efficiency framing, which does not count the pretraining corpus.
  2. [Section 5.1, Table 7] The ablation for the contrastive loss confounds lambda with the modality dropping schedule. The lambda=0 row uses modality dropping ratios (AV:0.2, V:0.6, A:0.2) while the lambda=1 rows use (AV:0.4, V:0.4, A:0.2). Therefore the improvement from 22.5 to 20.4 WER cannot be attributed solely to LCo; the claim that the dual objective provides a 'significant benefit' is not supported by this table.
  3. [Section 4.2.1, Table 2] The over-70% noise-robustness claim compares CoGenAV+Whisper0 (frozen Whisper, audio-visual input) against Whisper* (fine-tuned audio-only Whisper). Modality and fine-tuning status differ simultaneously, so the comparison does not isolate the effect of visual information. A matched audio-only CoGenAV baseline, or an AV-HuBERT-initialized audio-visual baseline with the same frozen Whisper head, is needed to support the claim that the contrastive-generative synchronization specifically produces the noise robustness.
  4. [Section 4.2.2, Table 3] The clean-AVSR state-of-the-art claim rests on a 1.27 WER that is only 0.23 WER below the Whisper* audio-only baseline of 1.5. No error bars, confidence intervals, or multiple-seed results are reported. Since the same fine-tuned Whisper* is used as the SR head, the improvement may be within run-to-run variation. The comparison to AutoAVSR (1.5 WER, 3,448 hours) is also not apples-to-apples because CoGenAV uses a LRS2-fine-tuned Whisper head while AutoAVSR is a single end-to-end model.
minor comments (4)
  1. [Throughout] There are several typos and formatting inconsistencies, including 'Resluts', 'CoGeneA V', 'Talel.7', 'Deta Upsampler', and the inconsistent use of 'Gate FFN' versus 'GatedFFN-MHA'.
  2. [Section 5.2] The limitation section states that more comprehensive testing could be performed 'once LRS3 becomes publicly accessible,' but LRS3-TED has been publicly available for research for several years. This statement should be corrected.
  3. [Figure 2] The cross-modal alignment heatmap is qualitative only; adding a quantitative metric (e.g., average diagonal similarity or contrastive accuracy) would strengthen the claim that LCo improves temporal alignment.
  4. [Section 2.2.2] The sentence 'we a lightweight Feature Adaptation Module' is missing a verb; it should read 'we design a lightweight Feature Adaptation Module.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the reported numbers are direct supervised evaluations of the proposed training objectives, though the AV-HuBERT initialization is not ablated.

full rationale

CoGenAV is an empirical representation-learning paper. The claimed results (VSR 20.5, noisy AVSR 2.6, clean AVSR 1.27, SDRi 16.0 dB, mAP 96.3) are test-set evaluations of models trained with the stated losses (Eq. 5) on LRS2; they are not derived from fitted constants, nor do they reduce by construction to the model's inputs. The generative objective (Eq. 4) maximizes the log-likelihood of ground-truth text through a frozen Whisper head, and the contrastive objective (Eqs. 1–2) is a binary cross-entropy loss on frame-wise cosine similarity; neither of these equals the reported WER or SI-SNRi metrics by definition. The paper contains no self-citations by the present authors, and no uniqueness theorem or cited ansatz is used to force the architecture. The one substantive concern—initializing the visual encoder and transformer from AV-HuBERT weights (Sec. 3.3) and not ablating that initialization in VSR/AVSR—weakens the causal attribution of the gains and the '223 hours only' data-efficiency framing, but this is an experimental-control limitation, not circularity: the evaluations would still distinguish among methods even if some performance were inherited from pretraining. Section 5.2 also explicitly acknowledges the LRS2-only evaluation scope. No step in the derivation chain equates a prediction to its input by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on domain assumptions about frozen Whisper as a teacher, LRS2 as a sufficient training signal, and the contrastive similarity as a valid synchronization target. It introduces no new physical or conceptual entities; the Delta Upsampler and GatedFFN-MHA are architectural components adapted from prior work (TDN and GLU). There are two hand-tuned hyperparameters (lambda and modality dropping ratios) that affect the reported numbers.

free parameters (2)
  • lambda (LCo weight) = 1
    The balancing coefficient for the contrastive loss in L = LGen + lambda * LCo; set to 1 based on ablation in Table 7. VSR WER at lambda=1 is 20.4, at lambda=0 is 22.5, at lambda=2 is 21.2.
  • modal dropping ratios = audio 0.2, visual 0.4, audio-visual 0.4
    Proportions of pure audio, pure visual, and audio-visual inputs during training; chosen via experiments in Table 7 to balance VSR (20.5) and noisy AVSR (2.6) performance.
assumptions (3)
  • domain assumption Frozen Whisper's representations are an appropriate latent target for visual speech features
    Section 2.2.2 uses Whisper as the SR head and assumes visual features can be aligned to its acoustic-textual space; if false, the generative loss cannot train the visual encoder.
  • domain assumption LRS2's 223 hours of synchronized audio-video-text provide enough signal to learn transferable audio-visual representations
    The whole method trains on LRS2 only; the claim of data efficiency depends on this dataset being sufficient.
  • domain assumption Frame-wise cosine similarity with ReLU and temporal mean (Eq.1) is a valid training signal for synchronization
    The Seq2Seq contrastive loss uses this formulation to push audio and video features into alignment; if the similarity measure is not sensitive to true synchronization, the contrastive objective would not add value.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization." pith.science (2026). https://pith.science/paper/7ZKXMR65

@misc{pith2026250503186,
  author       = {Pith},
  title        = {Pith review of: CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZKXMR65}},
  note         = {Machine review of arXiv:2505.03186}
}
read the original abstract

The inherent synchronization between a speaker's lip movements, voice, and the underlying linguistic content offers a rich source of information for improving speech processing tasks, especially in challenging conditions where traditional audio-only systems falter. We introduce CoGenAV, a powerful and data-efficient model designed to learn versatile audio-visual representations applicable across a wide range of speech and audio-visual tasks. CoGenAV is trained by optimizing a dual objective derived from natural audio-visual synchrony, contrastive feature alignment and generative text prediction, using only 223 hours of labeled data from the LRS2 dataset. This contrastive-generative synchronization strategy effectively captures fundamental cross-modal correlations. We showcase the effectiveness and versatility of the learned CoGenAV representations on multiple benchmarks. When utilized for Audio-Visual Speech Recognition (AVSR) on LRS2, these representations contribute to achieving a state-of-the-art Word Error Rate (WER) of 1.27. They also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.5 on LRS2, and significantly improve performance in noisy environments by over 70%. Furthermore, CoGenAV representations benefit speech reconstruction tasks, boosting performance in Speech Enhancement and Separation, and achieve competitive results in audio-visual synchronization tasks like Active Speaker Detection (ASD). Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.

Figures

Figures reproduced from arXiv: 2505.03186 by the authors.

Figure 1
Figure 1. Our method. Left: Audio-Visual Representation framework and the Contrastive-Generative [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Cross-Modal Alignment Heatmap. The brighter the color, the higher the similarity between [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 34 canonical work pages

  1. [1]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492–28518. PMLR, 2023

    2023

  2. [2]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051, 2024

    arXiv 2024

  3. [3]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023

    2023

  4. [4]

    Qwen2-audio technical report, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024

    2024

  5. [5]

    Assessment for automatic speech recognition: Ii

    Andrew Varga and Herman JM Steeneken. Assessment for automatic speech recognition: Ii. noisex-92: A database and an experiment to study the effect of additive noise on speech recognition systems. Speech communication, 12(3):247–251, 1993

    1993

  6. [6]

    Auto-avsr: Audio-visual speech recognition with automatic labels

    Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, and Maja Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2023

    work page 2023

  7. [7]

    mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

    Andrew Rouditchenko, Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, and James Glass. mwhisper-flamingo for multilingual audio-visual noise-robust speech recognition. arXiv preprint arXiv:2502.01547, 2025

    work page Pith review arXiv 2025

  8. [8]

    Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech perception, 2024

    HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi, and Chang- han Wang. Xlavs-r: Cross-lingual audio-visual speech representation learning for noise-robust speech perception, 2024

    work page 2024

Show all 51 references
  1. [9]

    Learning audio-visual speech representation by masked multimodal cluster prediction

    Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201.02184, 2022

    2022 arXiv

  2. [10]

    Unified speech recognition: A single model for auditory, visual, and audiovisual inputs, 2024

    Alexandros Haliassos, Rodrigo Mira, Honglie Chen, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Unified speech recognition: A single model for auditory, visual, and audiovisual inputs, 2024

    2024

  3. [11]

    Speech recognition models are strong lip- readers

    KR Prajwal, Triantafyllos Afouras, and Andrew Zisserman. Speech recognition models are strong lip- readers. In Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 2425–2429, 2024

    2024

  4. [12]

    Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation

    Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass. Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation. arXiv preprint arXiv:2406.10082, 2024

    2024 arXiv

  5. [13]

    van de Ven, Nicholas Soures, and Dhireesha Kudithipudi

    Gido M. van de Ven, Nicholas Soures, and Dhireesha Kudithipudi. Continual learning and catastrophic forgetting, 2024

    2024

  6. [14]

    Lip reading sentences in the wild

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, July 2017

    2017

  7. [15]

    Maas: Multi-modal assignation for active speaker detection

    Juan León Alcázar, Fabian Caba, Ali K Thabet, and Bernard Ghanem. Maas: Multi-modal assignation for active speaker detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 265–274, 2021

    2021

  8. [16]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    2016

  9. [17]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  10. [18]

    Latentsync: Audio conditioned latent diffusion models for lip sync

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsync: Audio conditioned latent diffusion models for lip sync. arXiv preprint arXiv:2412.09262, 2024. 12

    2024 arXiv

  11. [19]

    Asr is all you need: cross-modal distillation for lip reading, 2020

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Asr is all you need: cross-modal distillation for lip reading, 2020

    2020

  12. [20]

    Schuller, and Maja Pantic

    Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Björn W. Schuller, and Maja Pantic. Lira: Learning visual speech representations from audio through self-supervision, 2021

    2021

  13. [21]

    u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality, 2022

    Wei-Ning Hsu and Bowen Shi. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality, 2022

    2022

  14. [22]

    Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations, 2024

    Jiachen Lian, Alexei Baevski, Wei-Ning Hsu, and Michael Auli. Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations, 2024

    2024

  15. [23]

    Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning

    Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, and Furu Wei. Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning. IEEE Transactions on Multimedia, 26:1055–1064, 2024

    2024

  16. [24]

    Conformer: Convolution-augmented transformer for speech recognition, 2020

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition, 2020

    2020

  17. [25]

    Multilingual audio-visual speech recognition with hybrid ctc/rnn-t fast conformer

    Maxime Burchi, Krishna C Puvvada, Jagadeesh Balam, Boris Ginsburg, and Radu Timofte. Multilingual audio-visual speech recognition with hybrid ctc/rnn-t fast conformer. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP) , pag...

    2024

  18. [26]

    Lrs3-ted: a large-scale dataset for visual speech recognition, 2018

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: a large-scale dataset for visual speech recognition, 2018

    2018

  19. [27]

    Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation

    Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei-Ning Hsu, Juan Pino, and Changhan Wang. Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation. arXiv preprint arXiv:2303.00628, 2023

    2023 arXiv

  20. [28]

    Braven: Improving self-supervised pre-training for visual and auditory speech recognition, 2024

    Alexandros Haliassos, Andreas Zinonos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Braven: Improving self-supervised pre-training for visual and auditory speech recognition, 2024

    2024

  21. [29]

    Large language models are strong audio-visual speech recognition learners, 2025

    Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, and Maja Pantic. Large language models are strong audio-visual speech recognition learners, 2025

    2025

  22. [30]

    Visualvoice: Audio-visual speech separation with cross-modal consistency, 2021

    Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency, 2021

    2021

  23. [31]

    Ctcnet: A cnn- transformer cooperation network for face image super-resolution

    Guangwei Gao, Zixiang Xu, Juncheng Li, Jian Yang, Tieyong Zeng, and Guo-Jun Qi. Ctcnet: A cnn- transformer cooperation network for face image super-resolution. IEEE Transactions on Image Processing, 32:1978–1991, 2023

    1978

  24. [32]

    Muse: Multi-modal target speaker extraction with visual cues, 2021

    Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li. Muse: Multi-modal target speaker extraction with visual cues, 2021

    2021

  25. [33]

    Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction, 2023

    Jiuxin Lin, Xinyu Cai, Heinrich Dinkel, Jun Chen, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Zhiyong Wu, Yujun Wang, and Helen Meng. Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction, 2023

    2023

  26. [34]

    Attention is all you need in speech separation, 2021

    Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation, 2021

    2021

  27. [35]

    Separate in the speech chain: Cross-modal conditional audio-visual target speech extraction, 2024

    Zhaoxi Mu and Xinyu Yang. Separate in the speech chain: Cross-modal conditional audio-visual target speech extraction, 2024

    2024

  28. [36]

    A light weight model for active speaker detection, 2023

    Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, and Liangyin Chen. A light weight model for active speaker detection, 2023

    2023

  29. [37]

    End-to-end active speaker detection, 2022

    Juan Leon Alcazar, Moritz Cordes, Chen Zhao, and Bernard Ghanem. End-to-end active speaker detection, 2022

    2022

  30. [38]

    Loconet: Long-short context network for active speaker detection, 2024

    Xizi Wang, Feng Cheng, Gedas Bertasius, and David Crandall. Loconet: Long-short context network for active speaker detection, 2024. 13

    2024

  31. [39]

    Lr-asd: Lightweight and robust network for active speaker detection

    Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, Liangyin Chen, and Yanru Chen. Lr-asd: Lightweight and robust network for active speaker detection. International Journal of Computer Vision, pages 1–21, 2025

    2025

  32. [40]

    Tdn: Temporal difference networks for efficient action recognition, 2021

    Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition, 2021

    2021

  33. [41]

    Dauphin, Angela Fan, Michael Auli, and David Grangier

    Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, 2017

    2017

  34. [42]

    Musan: A music, speech, and noise corpus, 2015

    David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus, 2015

    2015

  35. [43]

    Audio- visual speech recognition with a hybrid ctc/attention architecture, 2018

    Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. Audio- visual speech recognition with a hybrid ctc/attention architecture, 2018

    2018

  36. [44]

    Es3: Evolving self-supervised learning of robust audio-visual speech representations

    Yuanhang Zhang, Shuang Yang, Shiguang Shan, and Xilin Chen. Es3: Evolving self-supervised learning of robust audio-visual speech representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27069–27079, June 2024

    2024

  37. [45]

    Syncvsr: Data-efficient visual speech recognition with end-to-end crossmodal audio token synchronization, 2024

    Young Jin Ahn, Jungwoo Park, Sangha Park, Jonghyun Choi, and Kee-Eung Kim. Syncvsr: Data-efficient visual speech recognition with end-to-end crossmodal audio token synchronization, 2024

    2024

  38. [46]

    Sub-word level lip reading with visual attention, 2021

    K R Prajwal, Triantafyllos Afouras, and Andrew Zisserman. Sub-word level lip reading with visual attention, 2021

    2021

  39. [47]

    Deep audio-visual speech recognition

    Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):8717–8727, December 2022

    2022

  40. [48]

    Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition, 2022

    Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, and Zhouhan Lin. Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition, 2022

    2022

  41. [49]

    Audio-visual efficient conformer for robust speech recognition, 2023

    Maxime Burchi and Radu Timofte. Audio-visual efficient conformer for robust speech recognition, 2023

    2023

  42. [50]

    Audio-visual speech enhancement and separation by utilizing multi-modal self-supervised embeddings, 2023

    I-Chun Chern, Kuo-Hsuan Hung, Yi-Ting Chen, Tassadaq Hussain, Mandar Gogate, Amir Hussain, Yu Tsao, and Jen-Cheng Hou. Audio-visual speech enhancement and separation by utilizing multi-modal self-supervised embeddings, 2023

    2023

  43. [51]

    Time domain audio visual speech separation, 2019

    Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-Wu Chen, Meng Yu, Lei Xie, and Dong Yu. Time domain audio visual speech separation, 2019. 14

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.