Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

AD-AVSR claims that bidirectional, asymmetric audio-visual enhancement with pair pruning achieves lower word error rates than prior systems on LRS2 and LRS3.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AD-AVSR combines dual-stream audio encoding, audio-guided visual refinement, visual-guided noise suppression, and thresholded audio-visual pair selection to improve audio-visual speech recognition word error rates under noise.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Sensible architecture, but the paper's own Table 1 refutes the 'consistently surpasses SOTA' claim; needs code, error bars, and threshold sensitivity before the empirical story holds. the 4 major comments →

arxiv 2508.07608 v1 pith:MWDZA6VZ submitted 2025-08-11 cs.MM cs.CVcs.SDeess.AS

AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition

Junxiao Xue , Xiaozhen Liu , Xuecheng Wu , Xinyi Yin , Danlei Huang , Fei Yu This is my paper
classification cs.MM cs.CVcs.SDeess.AS
keywords audio-visual speech recognitioncross-modal enhancementdual-stream encodingnoise suppression maskingthreshold-based selectionLRS2LRS3word error rate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that audio-visual speech recognition is held back by unidirectional or symmetric fusion, which cannot handle the inherent asymmetry between audio frames, visual frames, and noise. It proposes AD-AVSR, a dual-stream architecture in which each modality enhances the other in a closed loop: audio is encoded twice (time-domain and frequency-domain) so the enhancing modality carries denser information, an audio-aware visual refinement module reweights visual regions from audio cues, and a cross-modal noise suppression masking module uses visual context to mask noisy audio time steps. A threshold-based selection mechanism then prunes weakly correlated audio-visual frame pairs before fusion. On LRS2 and LRS3 the model reports the lowest word error rates in the paper's comparison under clean, audio-noisy, and visually corrupted conditions.

Core claim

The central claim is that deliberately asymmetric bidirectional enhancement is better than unidirectional or symmetric fusion for audio-visual speech recognition. The paper's delivered result is AD-AVSR, which couples a dual-stream audio encoder with two cooperating cross-modal modules and a pair-pruning stage. In the paper's experiments, the complete model reduces word error rate at SNR -5 dB on LRS2 from 24.9% for the baseline to 9.4%, a 62.2% relative reduction, and prints the best average word error rate on both datasets across clean and noisy conditions. The claim that matters is stated by the authors as state-of-the-art performance and noise robustness on the LRS2 and LRS3 benchmarks.

What carries the argument

The load-bearing mechanism is bidirectional modality enhancement built on an intentional information-density asymmetry. Audio is encoded in two streams: a time-domain stream (waveform through 1D convolution and ResNet18) that feeds the visual-enhanced audio branch, and a frequency-domain stream (mel-spectrogram with 25-frame averaging and repetition) that feeds the audio-enhanced visual branch. The Audio-aware Visual Refinement Module divides each frame into nine regions and reweights them with audio-derived attention; the Cross-modal Noise Suppression Masking Module applies audio-query visual-key attention and then generates a per-time-step mask in [0,1] that suppresses audio noise. The Thr

Load-bearing premise

The state-of-the-art claim assumes every competing method in the tables was trained and evaluated under the same visual frontend, noise corpus, augmentation, and test protocol as AD-AVSR, but the paper does not report reproduction details for those baselines.

What would settle it

Retrain the reported pipeline and every baseline under one shared protocol, then sweep the pruning threshold around 0.095: if the clean-set word-error-rate advantage over AVEC and AV-RelScore disappears, or if the ablation gain from threshold-based selection at SNR -5 dB on LRS2 vanishes, the paper's central claim is not settled.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, AD-AVSR sets a new printed word-error-rate bar on LRS2 and LRS3 under clean, audio-noisy, and audio-visual-corrupted evaluations.
  • The closed-loop design implies that audio-guided visual refinement and visual-guided audio denoising reinforce each other, so robustness to one corrupted modality need not come at the cost of the other.
  • The ablation attributes a large share of the total gain to pair pruning: at SNR -5 dB on LRS2, adding the threshold-based selection mechanism after the two refinement modules drops word error rate from 19.3% to 9.4%.
  • The dual-stream audio encoding matters: combining the time-domain and frequency-domain streams (A3) improves clean LRS2 word error rate to 2.41% from 3.63% for either single stream.
  • The reported cost of the full model is modest: about 16 million extra parameters and a 2.3% increase in FLOPs over the baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is sensitivity of the pruning threshold: the value 0.095 is imported from audio-visual event localization, and a sweep over thresholds would show whether the large TBSM gain is robust or peaked.
  • The asymmetric-density principle may transfer to other audio-visual tasks where one modality is temporally denser, such as sound source localization or audio-visual event classification, since the same pair-pruning idea already appears there.
  • Because the gains are reported with near-constant FLOPs, the architecture is plausibly a practical upgrade for latency-sensitive AVSR systems, but the paper does not measure real-time latency, memory, or on-device inference.
  • The state-of-the-art claim is entangled with a single visual frontend and one noise-augmentation recipe; the architecture's generality across other visual frontends and corruption models is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AD-AVSR, an audio-visual speech recognition system built around an asymmetric dual-stream audio encoding. A time-domain audio stream (1D-CNN + ResNet18) feeds a visual-enhanced audio branch, while a frequency-domain stream (mel-spectrogram + 1D-CNN, with 25-frame averaging/repetition) feeds an audio-enhanced visual branch. Two modules, the Audio-aware Visual Refinement Module (AVRM) and the Cross-modal Noise Suppression Masking Module (CMNSM), perform bidirectional cross-modal enhancement, followed by a Threshold-based Selection Mechanism (TBSM) that prunes weakly correlated audio-visual pairs. The fused features are processed by a Conformer encoder and Transformer decoder with combined CTC/attention loss. Experiments on LRS2 and LRS3 report word error rates under clean audio, various SNR conditions, and visual corruption, together with ablations and parameter/FLOP comparisons. The central claim is that AD-AVSR consistently surpasses state-of-the-art AVSR methods in both performance and noise robustness.

Significance. The proposed direction—deliberately creating asymmetric information density between modalities during bidirectional fusion—is well motivated, and the ablations show a substantial improvement on LRS2 at SNR=-5 dB (WER from 24.9% to 9.4%) with only a small FLOP increase. The evaluation uses standard public benchmarks and includes both audio noise and visual corruption, which is appropriate for the robustness claim. However, the empirical basis for the headline SOTA claim is not established by the paper's own numbers: Table 1 shows several conditions where AVEC or AV-Hubert match or beat AD-AVSR, and no uncertainty quantification is provided. If the claims are revised to present the method as competitive with specific advantages at severe noise, and if baseline comparability and threshold sensitivity are addressed, the contribution would be a useful addition to the AVSR literature.

major comments (4)
  1. [Abstract, §1, Table 1] The abstract claims AD-AVSR 'consistently surpasses SOTA methods' and §1 claims 'state-of-the-art results on the LRS2 and LRS3 datasets', but Table 1 contradicts this. On LRS2, AVEC has a lower average WER (4.7 vs 4.8), is strictly better at SNR=5 (3.4 vs 3.6) and SNR=0 (5.0 vs 6.0), and ties at SNR=10 (2.8). On LRS3, AVEC is better at SNR=5 (3.1 vs 3.2) and AV-Hubert ties on clean and SNR=10. The paper's own §4.2 wording ('consistently performs well in most cases') is weaker than the abstract. Either provide statistical evidence that the residual differences are within noise and rephrase the claims as 'competitive, with gains at severe noise', or the headline contribution is not supported by the reported results.
  2. [§4.1, Tables 1-2] Baseline comparability and experimental uncertainty are not established. The paper states that it follows the visual front-end of [13], but it does not state whether the baseline numbers for AVEC, AV-Hubert, A+VH, V-CAFE, etc. were obtained by retraining under identical preprocessing/augmentation/evaluation or are copied from the original papers. This matters because Table 2 reports different clean WERs from Table 1 for the same methods (e.g., Conformer clean 4.6 vs 4.9/4.8; AD-AVSR 2.4 vs 2.5/2.8). No error bars, seeds, or significance tests are reported. Differences of 0.1-0.2 WER cannot be interpreted without uncertainty quantification. Please specify the evaluation protocol per baseline and report multiple runs or confidence intervals.
  3. [§3.4, Eq. (10), Tables 3-4] The TBSM threshold tau=0.095 is imported directly from [48], an audio-visual event localization paper, yet the ablation attributes the largest single gain to TBSM (WER from 19.3 to 9.4 at SNR=-5). Because this is a free hyperparameter that is load-bearing for the performance claim, the paper should include a sensitivity analysis of WER as a function of tau and justify the chosen value for AVSR. Similarly, the visual region count k=9 in AVRM (Eq. (8), §4.1) and the CTC/attention loss weight lambda=0.9 (§4.1, Eq. (12)) should be justified with ablations or explicit references.
  4. [§4.2, Table 2] The simultaneous audio-visual corruption comparison is incomplete. Table 2 compares only Conformer, V-CAFE, and AV-RelScore, omitting AVEC and AV-Hubert, which are the strongest comparators in Table 1. Without these baselines under the same dual-corruption protocol, the claim of robustness under audio-visual corruption is not established against the current SOTA. Please add the missing baselines or restrict the claim to the methods actually compared.
minor comments (4)
  1. [§3.3, Eqs. (6)-(8)] The displayed equations for AVRM contain notation that is difficult to parse (e.g., the exact role of the all-ones vector and the flattening operation). Please rewrite these equations with clear tensor dimensions and define each term explicitly.
  2. [§4.1] The curriculum learning strategy is mentioned as adopted from [48], but no schedule or details are given. Please specify how the curriculum is constructed, as it may affect reproducibility.
  3. [Table 3] The '+TBSM' row shows a 31.6% parameter increase but only a 2.3% FLOP increase. Please briefly explain where the parameter increase comes from and why it does not translate to FLOPs.
  4. [References] The AVEC baseline [6] is not defined as an acronym in the text; please spell it out at first use. Also, the paper should state more clearly which baseline numbers are reproduced from the original papers and which are newly generated.

Circularity Check

0 steps flagged

No circular derivation: modular architecture evaluated on external benchmarks; no self-citation chain.

full rationale

The paper's derivation chain is not circular. The proposed AD-AVSR consists of explicitly specified modules—audio dual-stream encoding, AVRM, CMNSM, and TBSM—each defined by its own equations operating on audio and visual inputs. The central results are measured word error rates on the external LRS2 and LRS3 benchmarks, and the ablations (Tables 4 and 5) compare configurations of the proposed modules, not re-predictions of fitted quantities. The threshold τ = 0.095 is imported from reference [48], and the visual frontend follows reference [13]; neither is a self-citation, and neither threshold is fitted to the test set and then reported as a prediction. The claimed 'consistently surpasses SOTA' statement is contradicted by the paper's own Table 1 in places (e.g., AVEC has lower LRS2 average WER and wins at SNR 5 and 0), but this is a correctness/validity issue about the strength of the empirical claim, not a circularity reduction where an output is equivalent to an input by construction. No equation or module definition reduces to the quantity it is said to predict. Therefore, the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper makes empirical claims, so the ledger is dominated by experimental assumptions and hand-set hyperparameters rather than mathematical axioms or invented physical entities. The two most consequential entries are the comparability of baselines and the transferred TBSM threshold.

free parameters (4)
  • TBSM pruning threshold tau = 0.095
    Imported from [48]; controls how many audio-visual pairs are kept. TBSM is the largest single ablation gain, but no sensitivity analysis is reported.
  • AVRM visual region count k = 9
    Chosen for the audio-aware visual refinement module; no ablation over k is provided.
  • CTC/attention loss weight lambda = 0.9
    Hand-set loss balancing weight in Eq. (12); no sensitivity analysis is reported.
  • Audio frames averaged/repeated per visual frame in A2 = 25
    The frequency-domain branch averages 25 audio frames and repeats the result to create information asymmetry; this is a design choice only indirectly tested through the A1/A2/A3 comparison.
axioms (4)
  • domain assumption Baseline numbers in Table 1 are directly comparable to AD-AVSR under the same data splits, preprocessing, augmentation, and evaluation protocol.
    Load-bearing for the SOTA claim; the paper does not describe baseline reproduction or release code.
  • domain assumption BiLSTM-processed feature similarity in Eq. (9) is a valid proxy for true audio-visual correspondence, and threshold pruning retains only positive pairs.
    Central to TBSM; no ground-truth alignment is used and tau is assumed to transfer from [48].
  • ad hoc to paper Giving the enhancing modality higher per-frame information density via 25-frame averaging and repetition improves bidirectional fusion.
    Core motivation for the dual-stream encoding strategy; it is not derived and is only indirectly tested in Table 5.
  • standard math CTC conditional independence and the hybrid CTC/attention objective are appropriate for AVSR.
    Standard practice carried over from [23] and [41].

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/MWDZA6VZ

@misc{pith2026250807608,
  author       = {Pith},
  title        = {Pith review of: AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MWDZA6VZ}},
  note         = {Machine review of arXiv:2508.07608}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Audio-visual speech recognition (AVSR) combines audio-visual modalities to improve speech recognition, especially in noisy environments. However, most existing methods deploy the unidirectional enhancement or symmetric fusion manner, which limits their capability to capture heterogeneous and complementary correlations of audio-visual data-especially under asymmetric information conditions. To tackle these gaps, we introduce a new AVSR framework termed AD-AVSR based on bidirectional modality enhancement. Specifically, we first introduce the audio dual-stream encoding strategy to enrich audio representations from multiple perspectives and intentionally establish asymmetry to support subsequent cross-modal interactions. The enhancement process involves two key components, Audio-aware Visual Refinement Module for enhanced visual representations under audio guidance, and Cross-modal Noise Suppression Masking Module which refines audio representations using visual cues, collaboratively leading to the closed-loop and bidirectional information flow. To further enhance correlation robustness, we adopt a threshold-based selection mechanism to filter out irrelevant or weakly correlated audio-visual pairs. Extensive experimental results on the LRS2 and LRS3 datasets indicate that our AD-AVSR consistently surpasses SOTA methods in both performance and noise robustness, highlighting the effectiveness of our model design.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Existence of the longest arcs for left-invariant three-dimensional contact sub-Lorentzian structures

    math.DG 2026-03 unverdicted novelty 5.0

    Sufficient conditions are given for existence of longest arcs in left-invariant three-dimensional contact sub-Lorentzian structures on solvable Lie groups and the universal cover of SL(2,R).

Reference graph

Works this paper leans on

54 extracted references · 42 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and An- drew Zisserman. 2018. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence 44, 12 (2018), 8717–8727

    work page 2018

  2. [2]

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. LRS3-TED: a large-scale dataset for visual speech recognition.arXiv preprint arXiv:1809.00496 (2018)

    Pith/arXiv arXiv 2018

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  4. [4]

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. 41–48

    work page 2009

  5. [5]

    Vivek Bhardwaj, Mohamed Tahar Ben Othman, Vinay Kukreja, Youcef Belkhier, Mohit Bajaj, B Srikanth Goud, Ateeq Ur Rehman, Muhammad Shafiq, and Habib Hamam. 2022. Automatic speech recognition (asr) systems for children: A sys- tematic literature review. Applied Sciences 12, 9 (2022), 4419

    work page 2022

  6. [6]

    Maxime Burchi and Radu Timofte. 2023. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision . 2258–2267

    work page 2023

  7. [7]

    Dongjie Fu, Xize Cheng, Xiaoda Yang, Wang Hanting, Zhou Zhao, and Tao Jin

    work page

  8. [8]

    William W Gaver. 1993. What in the world do we hear?: An ecological approach to auditory event perception. Ecological psychology 5, 1 (1993), 1–29

    work page 1993

  9. [9]

    Xiaobao Guo, Nithish Muthuchamy Selvaraj, Zitong Yu, Adams Wai-Kin Kong, Bingquan Shen, and Alex Kot. 2023. Audio-visual deception detection: Do- los dataset and parameter-efficient crossmodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 22135–22145

    work page 2023

  10. [10]

    Xiaobao Guo, Zitong Yu, Nithish Muthuchamy Selvaraj, Bingquan Shen, Adams Wai-Kin Kong, and Alex C Kot. 2024. Benchmarking Cross-Domain Audio-Visual Deception Detection. arXiv preprint arXiv:2405.06995 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al

    work page

  12. [12]

    John Hershey and Javier Movellan. 1999. Audio vision: Using audio-visual synchrony to locate sounds. Advances in neural information processing systems 12 (1999)

    work page 1999

  13. [13]

    Joanna Hong, Minsu Kim, Jeongsoo Choi, and Yong Man Ro. 2023. Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18783–18794

    work page 2023

  14. [14]

    Joanna Hong, Minsu Kim, Daehun Yoo, and Yong Man Ro. 2022. Visual context- driven audio feature enhancement for robust end-to-end audio-visual speech recognition. arXiv preprint arXiv:2207.06020 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. 2023. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18795–18805

    work page 2023

  16. [16]

    Yuchen Hu, Ruizhe Li, Chen Chen, Heqing Zou, Qiushi Zhu, and Eng Siong Chng

    work page

  17. [17]

    Denis Ivanko, Dmitry Ryumin, and Alexey Karpov. 2023. A review of recent advances on deep learning methods for audio-visual speech recognition. Mathe- matics 11, 12 (2023), 2665

    work page 2023

  18. [18]

    Mahdie Karbasi and Dorothea Kolossa. 2022. ASR-based speech intelligibility prediction: A review. Hearing Research 426 (2022), 108606

    work page 2022

  19. [19]

    Minsu Kim, Jeong Hun Yeo, and Yong Man Ro. 2022. Distinguishing homophenes using multi-head visual-audio memory for lip reading. InProceedings of the AAAI conference on artificial intelligence , Vol. 36. 1174–1182

    work page 2022

  20. [20]

    Sungnyun Kim, Kangwook Jang, Sangmin Bae, Hoirin Kim, and Se-Young Yun

    work page

  21. [21]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980 (2014)

    Pith/arXiv arXiv 2014

  22. [22]

    Pengfei Ma, Wenyu Liao, Ying Zhuo, Hongyan Ma, Yanping Zhu, and Genda Chen. 2024. Characterization of alkali-silica reaction (ASR) products and CSH using SWIR spectroscopy for nondestructive detection of ASR. Construction and Building Materials 416 (2024), 135207

    work page 2024

  23. [23]

    Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2021. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 7613–7617

    work page 2021

  24. [24]

    Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2020. Lipreading using temporal convolutional networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 6319–6323

    work page 2020

  25. [25]

    In 2024 IEEE Spoken Language Technology Workshop (SLT)

    Learning video temporal dynamics with cross-modal attention for ro- bust audio-visual speech recognition. In 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 447–454

    work page 2024

  26. [26]

    Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)

    Pith/arXiv arXiv 2019

  27. [27]

    Andrew Rouditchenko, Yuan Gong, Samuel Thomas, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, and James Glass. 2024. Whisper-flamingo: Integrating visual features into whisper for audio-visual speech recognition and translation. arXiv preprint arXiv:2406.10082 (2024)

    Pith/arXiv arXiv 2024

  28. [28]

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al

  29. [29]

    Dmitry Ryumin, Alexandr Axyonov, Elena Ryumina, Denis Ivanko, Alexey Ka- shevnik, and Alexey Karpov. 2024. Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systems. Expert Systems with Applications 252 (2024), 124159

    work page 2024

  30. [30]

    Huaxin Pang, Shikui Wei, Gangjian Zhang, Shiyin Zhang, Shuang Qiu, and Yao Zhao. 2022. Heterogeneous feature alignment and fusion in cross-modal augmented space for composed image retrieval. IEEE Transactions on Multimedia 25 (2022), 6446–6457

    work page 2022

  31. [31]

    Bowen Shi, Wei-Ning Hsu, and Abdelrahman Mohamed. 2022. Robust self- supervised audio-visual speech recognition. arXiv preprint arXiv:2201.01763 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  32. [32]

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2017. Lip reading sentences in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6447–6456

    work page 2017

  33. [33]

    Ke Sun, Chun Yu, Weinan Shi, et al. 2018. Lip-interact: Improving mobile device interaction with silent speech commands. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology . 581–593

    work page 2018

  34. [34]

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition . 1–9

    2015

  35. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  36. [36]

    Changchong Sheng, Gangyao Kuang, Liang Bai, Chenping Hou, Yulan Guo, Xin Xu, Matti Pietikäinen, and Li Liu. 2024. Deep learning for visual speech analysis: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    work page 2024

  37. [37]

    Jiadong Wang, Zexu Pan, Malu Zhang, Robby T Tan, and Haizhou Li. 2024. Restoring speaking lips from occlusion for audio-visual speech recognition. In Proceedings of the AAAI conference on artificial intelligence , Vol. 38. 19144–19152

    work page 2024

  38. [38]

    Jiadong Wang, Xinyuan Qian, and Haizhou Li. 2024. Predict-and-update network: Audio-visual speech recognition inspired by human speech perception.IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    work page 2024

  39. [39]

    Xinyu Wang, Haotian Jiang, Haolin Huang, Yu Fang, Mengjie Xu, and Qian Wang. 2025. DCIM-AVSR: Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 1–5

    work page 2025

  40. [40]

    Xingmei Wang, Jiachen Mi, Boquan Li, Yixu Zhao, and Jiaxiang Meng. 2024. CAT- Net: Cross-modal fusion for audio–visual speech recognition. Pattern Recognition Letters 178 (2024), 216–222

    work page 2024

  41. [41]

    Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid CTC/attention architecture for end-to-end speech recogni- tion. IEEE Journal of Selected Topics in Signal Processing 11, 8 (2017), 1240–1253

    work page 2017

  42. [42]

    He Wang, Pengcheng Guo, Pan Zhou, and Lei Xie. 2024. Mlca-avsr: Multi-layer cross attention fusion based audio-visual speech recognition. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 8150–8154

    work page 2024

  43. [43]

    Karren Yang, Dejan Marković, Steven Krenn, Vasu Agrawal, and Alexander Richard. 2022. Audio-visual speech codecs: Rethinking audio-visual speech enhancement by re-synthesis. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition . 8227–8237

    work page 2022

  44. [44]

    Shih-Wei Yang, Li-Hsiang Shen, Hong-Han Shuai, and Kai-Ten Feng. 2024. CMAF: Cross-Modal Augmentation via Fusion for Underwater Acoustic Image Recogni- tion. ACM Transactions on Multimedia Computing, Communications and Applica- tions 20, 5 (2024), 1–25

    work page 2024

  45. [45]

    Fan Yu, Haoxu Wang, Ziyang Ma, and Shiliang Zhang. 2024. Hourglass-avsr: Down-up sampling-based computational efficiency model for audio-visual speech recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 7940–7944

    work page 2024

  46. [46]

    Fang Zhang, Yongxin Zhu, Xiangxiang Wang, Huang Chen, Xing Sun, and Linli Xu. 2024. Visual hallucination elevates speech recognition. In Proceedings of the �������� ���������� ����������� ����������� ��� ������ ������������ ������ ����������� ��� ���� ������� ������ ����� ������� ������� AAAI Conference on Artificial Intelligence , Vol. 38. 19542–19550

    work page 2024

  47. [47]

    Zi-Qiang Zhang, Jie Zhang, Jian-Shu Zhang, Ming-Hui Wu, Xin Fang, and Li- Rong Dai. 2022. Learning contextually fused audio-visual representations for audio-visual speech recognition. In 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 1346–1350

    work page 2022

  48. [48]

    Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. 2020. Discriminative multi- modality speech recognition. In Proceedings of the IEEE/CVF conference on Com- puter Vision and Pattern Recognition . 14433–14442

    work page 2020

  49. [54]

    Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. 2021. Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 8436–8444

    work page 2021

  50. [2014]

    arXiv preprint arXiv:1412.5567 (2014)

    Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 (2014)

    Pith/arXiv arXiv 2014

  51. [2015]

    International journal of computer vision 115 (2015), 211–252

    Imagenet large scale visual recognition challenge. International journal of computer vision 115 (2015), 211–252

    work page 2015

  52. [2022]

    Advances in neural information processing systems 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

    2022

  53. [2023]

    arXiv preprint arXiv:2305.09212 (2023)

    Cross-modal global interaction and local alignment for audio-visual speech recognition. arXiv preprint arXiv:2305.09212 (2023)

    Pith/arXiv arXiv 2023

  54. [2024]

    In Proceedings of the 32nd ACM International Conference on Multimedia

    Boosting Speech Recognition Robustness to Modality-Distortion with Contrast-Augmented Prompts. In Proceedings of the 32nd ACM International Conference on Multimedia. 3838–3847

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.