{"id":"116e4cef-d84d-412d-91f3-13f4a7c2d8c0","arxiv_id":"2508.07608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AD-AVSR combines dual-stream audio encoding, audio-guided visual refinement, visual-guided noise suppression, and thresholded audio-visual pair selection to improve audio-visual speech recognition word error rates under noise.","lead":"This paper proposes AD-AVSR, an audio-visual speech recognition system that runs two separate audio encoders and two cross-modal enhancement modules to improve recognition in noisy conditions. The authors report strong word error rates on the LRS2 and LRS3 benchmarks, but the claim that it consistently beats all prior methods is not fully supported by the paper's own table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the 'consistently surpasses SOTA' claim: AVEC has a lower LRS2 average and wins at SNR 5 and 0; on LRS3 AVEC wins at SNR 5. The central claim is falsified by the paper's own numbers.","rationale":"The reader's verdict is CONDITIONAL, and I agree. My stress-test pass concentrates on the strongest claim in the paper: consistent SOTA superiority. Reading Table 1 carefully shows this claim is not merely unverified but contradicted by the authors' own reported numbers. The AVEC baseline, which is the strongest competitor, beats AD-AVSR on LRS2 average WER (4.7 vs 4.8) and at two of five SNR levels; on LRS3 it wins at SNR=5. AV-Hubert ties on two conditions. Therefore the abstract's 'consistently surpasses' is false under any reasonable reading of 'consistently.' This is a correctness risk that doesn't depend on external assumptions about how baselines were trained; it is internal to the paper. The reader's weakest_assumption (baseline comparability) is a legitimate external concern, but the internal contradiction is more decisive. The threshold sensitivity is also a real issue—TBSM accounts for ~10 WER points in Table 4 and tau is taken without a sweep—but it is secondary because even with perfect hyperparameters the SOTA claim would still be contradicted by Table 1. The appropriate fix is to soften the claim to 'competitive with SOTA and better at low SNR,' which would make the contribution credible but incremental. No change in verdict: CONDITIONAL remains appropriate, with the condition being the correction of the overclaim and the release of code/variances.","tokens_in":14688,"tokens_out":4755,"duration_ms":44855,"concrete_test":"Recompute pairwise comparisons from Table 1. For each baseline and each SNR condition, determine whether AD-AVSR's WER is strictly lower. Specifically, verify: LRS2 avg AVEC = (2.6+2.8+3.4+5.0+9.7)/5 = 4.7 vs AD-AVSR = (2.4+2.8+3.6+6.0+9.4)/5 = 4.84 ≈ 4.8; at SNR=5 AVEC 3.4 < 3.6, at SNR=0 AVEC 5.0 < 6.0. LRS3: at SNR=5 AVEC 3.1 < 3.2; clean AV-Hubert 2.0 = 2.0. If any baseline is strictly better or tied in any condition, the word 'consistently surpasses' is unsupported. The test is a table audit, no new experiments required.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not baseline reproducibility but an internal contradiction in the paper's central claim. The abstract states AD-AVSR 'consistently surpasses SOTA methods in both performance and noise robustness,' and Section 1 claims 'state-of-the-art results on LRS2 and LRS3.' Yet in Table 1, on LRS2 the AVEC baseline (row 6) has average WER 4.7 vs AD-AVSR's 4.8, and AVEC is strictly better at SNR=5 (3.4 vs 3.6) and SNR=0 (5.0 vs 6.0), with a tie at SNR=10 (2.8 both). On LRS3, AVEC is better at SNR=5 (3.1 vs 3.2) and AV-Hubert matches AD-AVSR on clean and SNR=10 (2.0 vs 2.0 and 2.1 vs 2.1). Thus AD-AVSR does not 'consistently surpass' either method across the reported conditions. The claim to SOTA status is therefore falsified by the paper's own evidence, independent of any external comparability issues. This is load-bearing because the abstract and introduction tout consistent superiority as the main contribution; if the claim is merely 'competitive and better at low SNR,' the contribution is still plausible but incremental, exactly as the reader notes. The threshold tau sensitivity is a secondary issue, but the internal contradiction is the primary correctness risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AD-AVSR, an audio-visual speech recognition system built around an asymmetric dual-stream audio encoding. A time-domain audio stream (1D-CNN + ResNet18) feeds a visual-enhanced audio branch, while a frequency-domain stream (mel-spectrogram + 1D-CNN, with 25-frame averaging/repetition) feeds an audio-enhanced visual branch. Two modules, the Audio-aware Visual Refinement Module (AVRM) and the Cross-modal Noise Suppression Masking Module (CMNSM), perform bidirectional cross-modal enhancement, followed by a Threshold-based Selection Mechanism (TBSM) that prunes weakly correlated audio-visual pairs. The fused features are processed by a Conformer encoder and Transformer decoder with combined CTC/attention loss. Experiments on LRS2 and LRS3 report word error rates under clean audio, various SNR conditions, and visual corruption, together with ablations and parameter/FLOP comparisons. The central claim is that AD-AVSR consistently surpasses state-of-the-art AVSR methods in both performance and noise robustness.","tokens_in":15053,"tokens_out":6072,"duration_ms":64985,"significance":"The proposed direction—deliberately creating asymmetric information density between modalities during bidirectional fusion—is well motivated, and the ablations show a substantial improvement on LRS2 at SNR=-5 dB (WER from 24.9% to 9.4%) with only a small FLOP increase. The evaluation uses standard public benchmarks and includes both audio noise and visual corruption, which is appropriate for the robustness claim. However, the empirical basis for the headline SOTA claim is not established by the paper's own numbers: Table 1 shows several conditions where AVEC or AV-Hubert match or beat AD-AVSR, and no uncertainty quantification is provided. If the claims are revised to present the method as competitive with specific advantages at severe noise, and if baseline comparability and threshold sensitivity are addressed, the contribution would be a useful addition to the AVSR literature.","major_comments":[{"comment":"The abstract claims AD-AVSR 'consistently surpasses SOTA methods' and §1 claims 'state-of-the-art results on the LRS2 and LRS3 datasets', but Table 1 contradicts this. On LRS2, AVEC has a lower average WER (4.7 vs 4.8), is strictly better at SNR=5 (3.4 vs 3.6) and SNR=0 (5.0 vs 6.0), and ties at SNR=10 (2.8). On LRS3, AVEC is better at SNR=5 (3.1 vs 3.2) and AV-Hubert ties on clean and SNR=10. The paper's own §4.2 wording ('consistently performs well in most cases') is weaker than the abstract. Either provide statistical evidence that the residual differences are within noise and rephrase the claims as 'competitive, with gains at severe noise', or the headline contribution is not supported by the reported results.","section":"Abstract, §1, Table 1"},{"comment":"Baseline comparability and experimental uncertainty are not established. The paper states that it follows the visual front-end of [13], but it does not state whether the baseline numbers for AVEC, AV-Hubert, A+VH, V-CAFE, etc. were obtained by retraining under identical preprocessing/augmentation/evaluation or are copied from the original papers. This matters because Table 2 reports different clean WERs from Table 1 for the same methods (e.g., Conformer clean 4.6 vs 4.9/4.8; AD-AVSR 2.4 vs 2.5/2.8). No error bars, seeds, or significance tests are reported. Differences of 0.1-0.2 WER cannot be interpreted without uncertainty quantification. Please specify the evaluation protocol per baseline and report multiple runs or confidence intervals.","section":"§4.1, Tables 1-2"},{"comment":"The TBSM threshold tau=0.095 is imported directly from [48], an audio-visual event localization paper, yet the ablation attributes the largest single gain to TBSM (WER from 19.3 to 9.4 at SNR=-5). Because this is a free hyperparameter that is load-bearing for the performance claim, the paper should include a sensitivity analysis of WER as a function of tau and justify the chosen value for AVSR. Similarly, the visual region count k=9 in AVRM (Eq. (8), §4.1) and the CTC/attention loss weight lambda=0.9 (§4.1, Eq. (12)) should be justified with ablations or explicit references.","section":"§3.4, Eq. (10), Tables 3-4"},{"comment":"The simultaneous audio-visual corruption comparison is incomplete. Table 2 compares only Conformer, V-CAFE, and AV-RelScore, omitting AVEC and AV-Hubert, which are the strongest comparators in Table 1. Without these baselines under the same dual-corruption protocol, the claim of robustness under audio-visual corruption is not established against the current SOTA. Please add the missing baselines or restrict the claim to the methods actually compared.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The displayed equations for AVRM contain notation that is difficult to parse (e.g., the exact role of the all-ones vector and the flattening operation). Please rewrite these equations with clear tensor dimensions and define each term explicitly.","section":"§3.3, Eqs. (6)-(8)"},{"comment":"The curriculum learning strategy is mentioned as adopted from [48], but no schedule or details are given. Please specify how the curriculum is constructed, as it may affect reproducibility.","section":"§4.1"},{"comment":"The '+TBSM' row shows a 31.6% parameter increase but only a 2.3% FLOP increase. Please briefly explain where the parameter increase comes from and why it does not translate to FLOPs.","section":"Table 3"},{"comment":"The AVEC baseline [6] is not defined as an acronym in the text; please spell it out at first use. Also, the paper should state more clearly which baseline numbers are reproduced from the original papers and which are newly generated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I see no sign of fabrication or misattribution; [48] is clearly credited for TBSM. The main issue is that the abstract and introduction overstate the empirical contribution relative to Table 1. This is correctable within the scope of a revision by softening the claims, adding uncertainty quantification, and addressing baseline comparability. Given the workshop context, I would not reject outright, but the revised version must substantiate or carefully limit the SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent AVSR system paper that assembles known cross-modal tricks into a bidirectional dual-stream pipeline. The abstract's SOTA claim doesn't survive contact with its own Table 1.\n\nThe two genuinely interesting pieces are the audio dual-stream encoding (time-domain for the visual-enhances-audio branch, frequency-domain for the audio-enhances-visual branch) and the closed-loop design of AVRM and CMNSM. The ablation in Table 4 shows each module helps, and Table 5 shows dual-stream encoding beats either single stream by a wide margin (WER 2.41 vs 3.63/4.04). That's real evidence.\n\nBut the headline claim is false as stated. On LRS2, AVEC has a lower average WER (4.7 vs 4.8) and wins at SNR 5 and 0. On LRS3, AVEC wins at SNR 5, and AV-Hubert ties at clean and SNR 10. So 'consistently surpasses SOTA' is contradicted by the numbers. The text itself softens to 'consistently performs well in most cases,' which is accurate but should be the abstract. Without error bars, no code, no seeds, and no sensitivity analysis for the TBSM threshold tau (which is imported from a different task), the robustness gains can't be separated from tuning. Also note that Table 4 shows +TBSM alone gives a bigger WER drop (24.9→16.3) than +AVRM+CMNSM (24.9→19.3), yet the full system is 9.4; that's a large interaction, worth unpacking. The parameter count jumps 31% for TBSM while FLOPs only +2.3%; that's plausible but warrants detail.\n\nNone of this is fatal to the architecture. The core idea—explicit asymmetry via dual-stream encoding, and bidirectional enhancement with pruning—is plausible and the ablations support the components. What's missing is honest framing and enough empirical hygiene.\n\nThis deserves a serious referee if the authors fix the SOTA claim, release code, and add error bars plus tau sensitivity. As is, I'd treat the results as 'competitive and better at low SNR,' not SOTA. I'd accept it for peer review with the expectation of major revisions.","headline":"Sensible architecture, but the paper's own Table 1 refutes the 'consistently surpasses SOTA' claim; needs code, error bars, and threshold sensitivity before the empirical story holds.","tokens_in":15581,"tokens_out":2990,"would_cite":false,"duration_ms":30529,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AD-AVSR claims that bidirectional, asymmetric audio-visual enhancement with pair pruning achieves lower word error rates than prior systems on LRS2 and LRS3.","keywords":["audio-visual speech recognition","cross-modal enhancement","dual-stream encoding","noise suppression masking","threshold-based selection","LRS2","LRS3","word error rate"],"falsifier":"Retrain the reported pipeline and every baseline under one shared protocol, then sweep the pruning threshold around 0.095: if the clean-set word-error-rate advantage over AVEC and AV-RelScore disappears, or if the ablation gain from threshold-based selection at SNR -5 dB on LRS2 vanishes, the paper's central claim is not settled.","tokens_in":14524,"feed_emoji":"🎙️","tokens_out":8630,"duration_ms":80576,"temperature":0.7,"pith_summary":"The paper argues that audio-visual speech recognition is held back by unidirectional or symmetric fusion, which cannot handle the inherent asymmetry between audio frames, visual frames, and noise. It proposes AD-AVSR, a dual-stream architecture in which each modality enhances the other in a closed loop: audio is encoded twice (time-domain and frequency-domain) so the enhancing modality carries denser information, an audio-aware visual refinement module reweights visual regions from audio cues, and a cross-modal noise suppression masking module uses visual context to mask noisy audio time steps. A threshold-based selection mechanism then prunes weakly correlated audio-visual frame pairs before fusion. On LRS2 and LRS3 the model reports the lowest word error rates in the paper's comparison under clean, audio-noisy, and visually corrupted conditions.","feed_headline":"Two-way audio-video cleanup cuts AVSR errors below prior systems","feed_subtitle":"New dual-stream model prunes weak audio-video pairs and reports lowest word error rates on LRS2 and LRS3.","key_machinery":"The load-bearing mechanism is bidirectional modality enhancement built on an intentional information-density asymmetry. Audio is encoded in two streams: a time-domain stream (waveform through 1D convolution and ResNet18) that feeds the visual-enhanced audio branch, and a frequency-domain stream (mel-spectrogram with 25-frame averaging and repetition) that feeds the audio-enhanced visual branch. The Audio-aware Visual Refinement Module divides each frame into nine regions and reweights them with audio-derived attention; the Cross-modal Noise Suppression Masking Module applies audio-query visual-key attention and then generates a per-time-step mask in [0,1] that suppresses audio noise. The Thr","core_discovery":"The central claim is that deliberately asymmetric bidirectional enhancement is better than unidirectional or symmetric fusion for audio-visual speech recognition. The paper's delivered result is AD-AVSR, which couples a dual-stream audio encoder with two cooperating cross-modal modules and a pair-pruning stage. In the paper's experiments, the complete model reduces word error rate at SNR -5 dB on LRS2 from 24.9% for the baseline to 9.4%, a 62.2% relative reduction, and prints the best average word error rate on both datasets across clean and noisy conditions. The claim that matters is stated by the authors as state-of-the-art performance and noise robustness on the LRS2 and LRS3 benchmarks.","pith_inferences":["A natural extension the paper does not test is sensitivity of the pruning threshold: the value 0.095 is imported from audio-visual event localization, and a sweep over thresholds would show whether the large TBSM gain is robust or peaked.","The asymmetric-density principle may transfer to other audio-visual tasks where one modality is temporally denser, such as sound source localization or audio-visual event classification, since the same pair-pruning idea already appears there.","Because the gains are reported with near-constant FLOPs, the architecture is plausibly a practical upgrade for latency-sensitive AVSR systems, but the paper does not measure real-time latency, memory, or on-device inference.","The state-of-the-art claim is entangled with a single visual frontend and one noise-augmentation recipe; the architecture's generality across other visual frontends and corruption models is untested."],"forward_implications":["If the claim holds, AD-AVSR sets a new printed word-error-rate bar on LRS2 and LRS3 under clean, audio-noisy, and audio-visual-corrupted evaluations.","The closed-loop design implies that audio-guided visual refinement and visual-guided audio denoising reinforce each other, so robustness to one corrupted modality need not come at the cost of the other.","The ablation attributes a large share of the total gain to pair pruning: at SNR -5 dB on LRS2, adding the threshold-based selection mechanism after the two refinement modules drops word error rate from 19.3% to 9.4%.","The dual-stream audio encoding matters: combining the time-domain and frequency-domain streams (A3) improves clean LRS2 word error rate to 2.41% from 3.63% for either single stream.","The reported cost of the full model is modest: about 16 million extra parameters and a 2.3% increase in FLOPs over the baseline."],"supporting_citations":[{"why":"Supplies the LRS2 dataset used for training and evaluation.","marker":"[32]"},{"why":"Supplies the LRS3 dataset used for the larger-scale evaluation.","marker":"[2]"},{"why":"Supplies the visual frontend, the AV-RelScore baseline, and the visual corruption evaluation protocol.","marker":"[13]"},{"why":"Is the V-CAFE baseline that enhances audio with visual cues, a key comparison for the enhancement modules.","marker":"[14]"},{"why":"Is the AVEC baseline, the strongest prior Conformer-based audio-visual system in the comparison.","marker":"[6]"},{"why":"Is the Conformer AVSR baseline and the source of the hybrid CTC/attention training recipe.","marker":"[23]"},{"why":"Is the AV-Hubert self-supervised baseline used for LRS3 comparison.","marker":"[31]"},{"why":"Is the A+VH visual-hallucination baseline used for comparison.","marker":"[46]"},{"why":"Supplies the threshold-based selection mechanism, its default threshold, and the curriculum learning strategy.","marker":"[48]"},{"why":"Supplies the CTC loss used for end-to-end training.","marker":"[11]"}],"fun_headline_variants":["Asymmetric dual-stream model cuts noisy AVSR errors by 62%","Bidirectional dual-stream audio-video model tops AVSR benchmarks","Asymmetric audio-video fusion beats symmetric AVSR in noise","Pruned audio-video pairs + dual streams lift AVSR to SOTA"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The state-of-the-art claim assumes every competing method in the tables was trained and evaluated under the same visual frontend, noise corpus, augmentation, and test protocol as AD-AVSR, but the paper does not report reproduction details for those baselines.","fun_headline_variants_meta":{"raw":{"variants":["Asymmetric dual-stream model cuts noisy AVSR errors by 62%","Bidirectional dual-stream audio-video model tops AVSR benchmarks","Asymmetric audio-video fusion beats symmetric AVSR in noise","Pruned audio-video pairs + dual streams lift AVSR to SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":3872,"prompt_tokens":744,"completion_tokens":3128,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":3061}},"tokens_in":488,"tokens_out":3128,"duration_ms":27068,"temperature":1.0,"reasoning_tokens":3061,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:59:31.220317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the reported pipeline and every baseline under one shared protocol, then sweep the pruning threshold around 0.095: if the clean-set word-error-rate advantage over AVEC and AV-RelScore disappears, or if the ablation gain from threshold-based selection at SNR -5 dB on LRS2 vanishes, the paper's central claim is not settled.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LRS2 dataset used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the visual frontend, the AV-RelScore baseline, and the visual corruption evaluation protocol."},{"cited_title":"Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition","cited_arxiv_id":"2207.06020","evidence_quote":"Is the V-CAFE baseline that enhances audio with visual cues, a key comparison for the enhancement modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the AVEC baseline, the strongest prior Conformer-based audio-visual system in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the Conformer AVSR baseline and the source of the hybrid CTC/attention training recipe."},{"cited_title":"Robust Self-Supervised Audio-Visual Speech Recognition","cited_arxiv_id":"2201.01763","evidence_quote":"Is the AV-Hubert self-supervised baseline used for LRS3 comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the A+VH visual-hallucination baseline used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the threshold-based selection mechanism, its default threshold, and the curriculum learning strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CTC loss used for end-to-end training."}],"review_version":1}