{"id":"a468088a-1f3d-4e1c-9946-04ce30390666","arxiv_id":"2505.03186","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CoGenAV learns audio-visual speech representations that achieve 1.27% WER on LRS2 AVSR and 20.5% WER on LRS2 VSR using 223 hours of labeled data.","lead":"This paper introduces CoGenAV, a model trained on 223 hours of LRS2 to align audio, lip movements, and text via a contrastive loss and a generative text-prediction loss through a frozen Whisper head. It reports a state-of-the-art 1.27% WER on LRS2 audio-visual speech recognition and strong lip-reading and noise-robustness results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains are not isolated from AV-HuBERT initialization; no same-pipeline AV-HuBERT baseline appears in VSR or AVSR.","rationale":"The reader identified the frozen-Whisper modality-gap assumption as the weakest point, but the paper's VSR results with the full Feature Adaptation Module provide direct evidence that visual features can be made decodable by Whisper, so I do not consider that assumption the most dangerous. The unablated AV-HuBERT initialization is more load-bearing because it threatens the causal claim that the proposed contrastive-generative training creates the representations responsible for the reported gains. If the gains are mostly inherited from AV-HuBERT, the data-efficiency claim and the contribution of LCo and LGen are both unsupported. The paper's own AVSS table demonstrates that AV-HuBERT is a strong baseline, making its absence from the AVSR and VSR comparisons particularly salient. This concern is addressable with a controlled baseline and an initialization ablation, so it supports the existing CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":12532,"tokens_out":9809,"duration_ms":95002,"concrete_test":"Run a controlled baseline for Tables 1-3: freeze the AV-HuBERT Large encoder, feed its visual features through the same Delta Upsampler/GatedFFN-MHA and the same Whisper head, with the same fine-tuning protocol as CoGenAV, and evaluate VSR, noisy AVSR, and clean AVSR. In parallel, train CoGenAV Large from random initialization (no AV-HuBERT weights) on the same 223 hours with the same losses. If the AV-HuBERT baseline matches CoGenAV or random-init training collapses, the reported advantages are not attributable to the proposed objectives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that the contrastive-generative synchronization objectives produce the strong VSR, noisy-AVSR, and clean-AVSR numbers. Section 3.3 states that the visual encoder and transformer encoder are initialized from AV-HuBERT weights, a model pretrained on substantially more audio-visual data. The paper never ablates this initialization, and Tables 1-3 compare against published systems or audio-only Whisper rather than against the same downstream pipeline with frozen AV-HuBERT features and the same Whisper head plus Feature Adaptation Module. AV-HuBERT is already a strong audio-visual representation: the paper's own Table 4 uses AV-HuBERT as the reference for AVSS and shows CoGenAV only 1.6 dB better. Therefore the reported VSR 20.5, noisy-AVSR 2.6, and clean-AVSR 1.27 could be substantially inherited from pretraining rather than produced by LCo and LGen. The data-efficiency framing ('only 223 hours') is especially affected because the pretraining corpus is not counted. Section 5.2's stated limitation to LRS2 compounds this: there is no cross-dataset evidence that the learned representations transfer. This does not disprove the method, but it makes the central attribution and data-efficiency claims unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoGenAV, an audio-visual representation learning method trained on 223 hours of LRS2 with a dual objective: a sequence-to-sequence contrastive alignment loss (LCo) between audio and visual features, and a generative text-prediction loss (LGen) through a frozen Whisper ASR head, using a lightweight Feature Adaptation Module. The authors report a VSR WER of 20.5 on LRS2, a clean AVSR WER of 1.27 with a fine-tuned Whisper head, a noisy AVSR WER of 2.6 at 0 dB SNR, and improvements on AVSS, AVSE, and ASD. The central claim is that the contrastive-generative synchronization objective learns versatile, data-efficient audio-visual representations that transfer across tasks.","tokens_in":12782,"tokens_out":3708,"duration_ms":37386,"significance":"If the central attribution is validated, the paper would make a useful contribution: it demonstrates a relatively simple recipe for adapting a frozen ASR model to visual and audio-visual inputs, and the open-source release would support reproducibility. The reported versatility across recognition, reconstruction, and synchronization tasks is attractive. However, the current evidence does not isolate the effect of the proposed objectives from the AV-HuBERT initialization, and the headline clean-AVSR SOTA margin is small and unreplicated. The significance therefore depends on additional ablations that are missing; the contribution is potentially valuable but not yet established at the level claimed.","major_comments":[{"comment":"The visual encoder and transformer encoder are initialized from AV-HuBERT weights, a model pretrained on substantially more audio-visual data. The paper never ablates this initialization: no experiment compares CoGenAV to the same downstream pipeline (same Feature Adaptation Module, same frozen Whisper head) using frozen AV-HuBERT features or randomly initialized encoders. Because AV-HuBERT already provides strong audio-visual representations, the reported VSR (20.5), noisy AVSR (2.6), and clean AVSR (1.27) numbers could be substantially inherited from pretraining rather than produced by LCo and LGen. This also undermines the data-efficiency framing, which does not count the pretraining corpus.","section":"Section 3.3 and Tables 1-3"},{"comment":"The ablation for the contrastive loss confounds lambda with the modality dropping schedule. The lambda=0 row uses modality dropping ratios (AV:0.2, V:0.6, A:0.2) while the lambda=1 rows use (AV:0.4, V:0.4, A:0.2). Therefore the improvement from 22.5 to 20.4 WER cannot be attributed solely to LCo; the claim that the dual objective provides a 'significant benefit' is not supported by this table.","section":"Section 5.1, Table 7"},{"comment":"The over-70% noise-robustness claim compares CoGenAV+Whisper0 (frozen Whisper, audio-visual input) against Whisper* (fine-tuned audio-only Whisper). Modality and fine-tuning status differ simultaneously, so the comparison does not isolate the effect of visual information. A matched audio-only CoGenAV baseline, or an AV-HuBERT-initialized audio-visual baseline with the same frozen Whisper head, is needed to support the claim that the contrastive-generative synchronization specifically produces the noise robustness.","section":"Section 4.2.1, Table 2"},{"comment":"The clean-AVSR state-of-the-art claim rests on a 1.27 WER that is only 0.23 WER below the Whisper* audio-only baseline of 1.5. No error bars, confidence intervals, or multiple-seed results are reported. Since the same fine-tuned Whisper* is used as the SR head, the improvement may be within run-to-run variation. The comparison to AutoAVSR (1.5 WER, 3,448 hours) is also not apples-to-apples because CoGenAV uses a LRS2-fine-tuned Whisper head while AutoAVSR is a single end-to-end model.","section":"Section 4.2.2, Table 3"}],"minor_comments":[{"comment":"There are several typos and formatting inconsistencies, including 'Resluts', 'CoGeneA V', 'Talel.7', 'Deta Upsampler', and the inconsistent use of 'Gate FFN' versus 'GatedFFN-MHA'.","section":"Throughout"},{"comment":"The limitation section states that more comprehensive testing could be performed 'once LRS3 becomes publicly accessible,' but LRS3-TED has been publicly available for research for several years. This statement should be corrected.","section":"Section 5.2"},{"comment":"The cross-modal alignment heatmap is qualitative only; adding a quantitative metric (e.g., average diagonal similarity or contrastive accuracy) would strengthen the claim that LCo improves temporal alignment.","section":"Figure 2"},{"comment":"The sentence 'we a lightweight Feature Adaptation Module' is missing a verb; it should read 'we design a lightweight Feature Adaptation Module.'","section":"Section 2.2.2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is attribution: the paper's strongest claims (data-efficient SOTA AVSR, VSR, noise robustness) are not isolated from the AV-HuBERT initialization or from the fine-tuned Whisper head. If the authors can provide same-pipeline baselines with AV-HuBERT features and random initialization, the contribution may be solid. Without those, the manuscript currently overstates what the proposed objective alone contributes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"To be direct: this is a solid, workmanlike AV-speech representation paper, and the numbers are promising. It combines a SyncNet-style contrastive loss with generative text prediction through a frozen Whisper head, keeps the temporal dimension in the contrastive objective, and tests the features across VSR, AVSR, enhancement, separation, and ASD. The LRS2 VSR 20.5 and noisy-AVSR 2.6 are genuinely good; the clean AVSR 1.27 with fine-tuned Whisper is competitive with 'SOTA' but is a single-seed number with no error bars.\n\nWhat is actually new is small but real: the seq2seq contrastive formulation plus the gated adaptation module, and the demonstration that these components work as a package. The ablations inside the pipeline are decent—removing LCo hurts VSR by about 2 WER, removing the gated FFN or Delta Upsampler hurts, and Table 7 shows the right direction. The writing is clear and the code link is a plus.\n\nThe soft spots are in proportion. The stress-test concern is mostly fair: Section 3.3 says all models start from AV-HuBERT weights, and there is no ablation that replaces the trained CoGenAV encoder with frozen AV-HuBERT features in the same downstream pipeline. But not wholly fair: the Whisperer Large row in Table 1 (26.3) is a same-Whisper-head AV-HuBERT comparison reproduced by the authors, so the VSR gain is not completely unexplained. What is missing is a clean AVSR ablation with AV-HuBERT initialization but no CoGenAV training, and error bars on the 1.27 claim. The 'data-efficient, only 223 hours' framing is overstated because the pretraining corpus that produced the initialization is not counted and no same-pipeline no-pretrain baseline is shown. The noise-robustness claim compares only against Whisper variants, not other AVSR systems, so 'over 70%' is real relative to that baseline but not a general robustness claim.\n\nNone of this invalidates the method. The central hypothesis—that contrastive-generative synchronization helps—gains support; it is just not isolated. The citation pattern looks normal. This paper deserves a serious referee. I would send it to a conference or workshop with a request for an initialization ablation and repeated-seed reporting, and I would not cite the SOTA claim in my own work until that ablation exists.","headline":"Solid AV-speech representation paper with real-looking gains, but the headline numbers are not cleanly separated from AV-HuBERT pretraining.","tokens_in":13321,"tokens_out":3578,"would_cite":false,"duration_ms":34603,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoGenAV claims a frozen Whisper ASR head can decode lip-movement features, reaching 1.27% WER on LRS2 AVSR.","keywords":["audio-visual speech recognition","visual speech recognition","contrastive learning","generative text prediction","frozen ASR head","lip reading","noise robustness","speech separation"],"falsifier":"Train the same CoGenAV architecture on the same LRS2 data but replace Whisper as the frozen SR head with a randomly initialized ASR encoder-decoder of matched size: if visual-only WER stays near 20.5%, the frozen-ASR assumption is not load-bearing; if it collapses, the result depends on Whisper's pretrained latent space. Alternatively, if an independent replication with the released code cannot reproduce the 1.27% clean AVSR WER on LRS2, the central claim is falsified.","tokens_in":12345,"feed_emoji":"🎙️","tokens_out":5495,"duration_ms":51849,"temperature":0.7,"pith_summary":"CoGenAV learns audio-visual speech representations from synchronized audio, video, and text, trained on 223 hours of LRS2. Its central claim is that combining a sequence-level contrastive loss, which aligns audio and visual features frame by frame, with a generative loss, which forces those features to be decodable by a frozen pre-trained Whisper ASR model, yields representations that transfer across many speech tasks. The paper reports a word error rate (WER) of 1.27% on LRS2 audio-visual speech recognition, 20.5% WER on visual-only speech recognition, and a reduction of more than 70% in WER over a fine-tuned audio-only Whisper baseline under 0 dB noise. The same frozen features improve speech enhancement and separation and perform competitively on active speaker detection. The result matters because it suggests a small labeled dataset plus a frozen ASR decoder can provide much of the robustness that audio-visual models usually require large-scale training to achieve.","feed_headline":"Frozen Whisper plus lip video hits 1.27% word error rate","feed_subtitle":"Trained on just 223 hours, the same contrastive-generative features cut noisy AVSR errors by over 70%.","key_machinery":"The machinery is a dual objective trained through a frozen ASR head. The contrastive part is a sequence-to-sequence contrastive loss: frame-wise cosine similarity between audio and video features, ReLU-clipped and averaged over time, trained with binary cross-entropy against synchrony labels. The generative part is a log-likelihood loss computed by a frozen pre-trained Whisper model serving as a speech-recognition head; before features reach Whisper they pass through a Feature Adaptation Module made of a $\\Delta$ Upsampler, which doubles temporal resolution using temporal-difference convolution, and a GatedFFN-MHA layer, which uses multi-head attention and a gated feed-forward network to close modality mismatch. The combination lets the model learn features that are simultaneously time-aligned across modalities and semantically decodable as text.","core_discovery":"CoGenAV establishes a tri-modal alignment among audio, visual, and text streams by jointly training a visual/audio encoder with two complementary objectives. The contrastive objective computes temporally averaged, ReLU-clipped frame-wise cosine similarities between audio-only and video-only features and trains the model to tell synchronized pairs from misaligned or cross-speaker pairs. The generative objective feeds adapted features, sometimes visual-only, sometimes audio-only, sometimes both, into a frozen Whisper encoder-decoder and maximizes the log-likelihood of the ground-truth transcription. Because Whisper's weights stay frozen, all learning pressure is on the CoGenAV backbone and a small Feature Adaptation Module, which upsamples 25 fps features to 50 fps and applies gated attention before decoding. The paper's core discovery is that this contrastive-generative synchronization captures both fine-grained temporal correspondence and linguistic content, producing a single visual representation that reaches 1.27% WER on LRS2 AVSR and 20.5% WER on VSR, and that remains effective as a frozen visual feature for enhancement, separation, and active speaker detection.","pith_inferences":["The 70% noisy-AVSR improvement likely comes from the contrastive loss teaching the visual stream to carry the information that noisy audio loses; the ablations show removing the contrastive term hurts noisy AVSR more than clean VSR.","A testable extension is to swap the frozen Whisper head for a different large ASR model; if visual-only WER stays low, the adaptation generalizes across ASR latent spaces, while a collapse would show the result is Whisper-specific.","Because the visual encoder is not limited by Whisper's training data, scaling CoGenAV to multilingual or unlabeled video could be done by pairing it with a multilingual frozen ASR head.","The paper's limitation statement names LRS3 as the next evaluation target; if the gains hold there, the 223-hour data-efficiency claim becomes a general property rather than an LRS2 artifact."],"forward_implications":["Large ASR models can be reused as frozen decoders for lip-reading rather than fine-tuned, avoiding catastrophic forgetting and heavy compute.","Visual-only and audio-visual speech recognition are obtained from the same checkpoint, since the same visual encoder feeds either input stream to the frozen Whisper head.","Noisy speech recognition gains a new recipe: align noisy audio with visual features during training, then rely on the visual stream at test time.","The representation is not tied to Whisper's fine-tuning data, so the same pretrained features should transfer to datasets beyond LRS2.","Frozen CoGenAV visual features can serve as drop-in replacements for task-specific visual encoders in separation, enhancement, and active speaker detection."],"supporting_citations":[{"why":"Supplies the frozen Whisper ASR model used as the speech-recognition head and defines the audio preprocessing scheme.","marker":"[1]"},{"why":"Provides the AutoAVSR baseline and lip-region preprocessing code that CoGenAV builds on and compares against.","marker":"[6]"},{"why":"Supplies the AV-HuBERT visual encoder and transformer initialization as well as a strong VSR comparison point.","marker":"[9]"},{"why":"Provides the USR clean-AVSR baseline that CoGenAV outperforms with substantially less training data.","marker":"[10]"},{"why":"Previous work integrating visual features into Whisper; serves as a comparison for clean AVSR and a motivation for the frozen-head design.","marker":"[12]"},{"why":"LRS2 is the dataset used for all core training and for the VSR, AVSR, separation, and enhancement evaluations.","marker":"[14]"},{"why":"AV-SepFormer is the separation head whose visual encoder is replaced with frozen CoGenAV features.","marker":"[33]"},{"why":"LRASD is the active-speaker-detection architecture whose encoders are replaced with CoGenAV representations.","marker":"[39]"},{"why":"MUSAN supplies the natural, music, and babble noise used in robustness training and noisy evaluations.","marker":"[42]"}],"fun_headline_variants":["Contrastive-generative lip sync hits 1.27% word error rate","Tri-modal sync model: 223 hours, 70% noise error cut","CoGenAV: one backbone for ASR, VSR, and denoising","Lip-video encoder beats benchmark with frozen Whisper","Audio-visual learning: 1.27% WER, 70% noise cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen pre-trained ASR model like Whisper defines a fixed acoustic-to-text latent space, and that lip-movement features can be mapped into that space by a small learned adapter; if that mapping cannot be learned, the generative loss stops teaching the visual encoder and the visual and noisy AVSR results collapse.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive-generative lip sync hits 1.27% word error rate","Tri-modal sync model: 223 hours, 70% noise error cut","CoGenAV: one backbone for ASR, VSR, and denoising","Lip-video encoder beats benchmark with frozen Whisper","Audio-visual learning: 1.27% WER, 70% noise cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1382,"prompt_tokens":1038,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":654,"tokens_out":344,"duration_ms":3514,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:56.531135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same CoGenAV architecture on the same LRS2 data but replace Whisper as the frozen SR head with a randomly initialized ASR encoder-decoder of matched size: if visual-only WER stays near 20.5%, the frozen-ASR assumption is not load-bearing; if it collapses, the result depends on Whisper's pretrained latent space. Alternatively, if an independent replication with the released code cannot reproduce the 1.27% clean AVSR WER on LRS2, the central claim is falsified.","supporting_citations":[{"cited_title":"Auto-avsr: Audio-visual speech recognition with automatic labels","cited_arxiv_id":null,"evidence_quote":"Provides the AutoAVSR baseline and lip-region preprocessing code that CoGenAV builds on and compares against."},{"cited_title":"Unified speech recognition: A single model for auditory, visual, and audiovisual inputs, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the USR clean-AVSR baseline that CoGenAV outperforms with substantially less training data."},{"cited_title":"Lip reading sentences in the wild","cited_arxiv_id":null,"evidence_quote":"LRS2 is the dataset used for all core training and for the VSR, AVSR, separation, and enhancement evaluations."},{"cited_title":"Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction, 2023","cited_arxiv_id":null,"evidence_quote":"AV-SepFormer is the separation head whose visual encoder is replaced with frozen CoGenAV features."},{"cited_title":"Lr-asd: Lightweight and robust network for active speaker detection","cited_arxiv_id":null,"evidence_quote":"LRASD is the active-speaker-detection architecture whose encoders are replaced with CoGenAV representations."}],"review_version":1}