pith. sign in

arxiv: 2606.31225 · v2 · pith:L4JQQ7NInew · submitted 2026-06-30 · 💻 cs.MM

A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR

Lin Chen , Jingping Fang , Hairui Liu , Chenyang Xu , Junhao Chen , Xiaorui Li , Weidong Cai , Xiaoming Chen This is my paper

Pith reviewed 2026-07-02 01:21 UTC · model grok-4.3

classification 💻 cs.MM
keywords visual speech recognitionevent streamsneuromorphicflow matchingmulti-speakerword error ratelip reading
0
0 comments X

The pith

LipsFlow converts RGB video to event streams and applies OT-CFM to reach 22.3% WER at 240 ms latency for multi-speaker visual speech recognition.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents LipsFlow as a framework that turns ordinary RGB videos into high-temporal-resolution event streams to capture rapid lip movements that standard cameras miss. It adds speaker tracking and detection steps to break multi-person scenes into single-speaker segments, then models the resulting features with Optimal Transport Conditional Flow Matching. This setup is meant to deliver robustness against motion blur and occlusions while keeping inference fast. A sympathetic reader would care because it points toward practical systems that could run lip reading on ordinary video in crowded or fast-moving settings without heavy compute.

Core claim

LipsFlow converts RGB videos into neuromorphic event streams, uses ByteTrack tracking and TalkNet active speaker detection to segment multi-speaker scenes into single-speaker clips, and introduces Optimal Transport Conditional Flow Matching to enforce deterministic straight-line trajectories in latent space, reducing inference to two ODE steps while adding dual-level BERT-based semantic supervision; the method reports a state-of-the-art 22.3% word error rate at 240 ms latency on competitive benchmarks.

What carries the argument

Optimal Transport Conditional Flow Matching (OT-CFM), which produces deterministic straight-line paths in semantic latent space to enable low-latency generation from dense event-based features.

If this is right

  • Multi-speaker scenes become tractable by first isolating individual speakers through tracking and detection before per-speaker analysis.
  • Inference drops to two ODE steps while preserving robustness to visual degradation such as blur and occlusion.
  • Homophene ambiguities are addressed by combining token-level BERT weight tying with sentence-level priors.
  • Event-based representations become a practical route for efficient visual speech recognition pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The low-latency design could support live applications such as real-time captioning in group conversations if the event conversion holds up under varied lighting.
  • Further tests on datasets with heavier speaker overlap would show whether the isolation step remains the limiting factor.
  • Pairing the event stream input with audio features might reduce remaining errors in acoustically noisy environments.

Load-bearing premise

Converting RGB video into event streams captures the necessary microsecond-level articulatory details without critical loss of information, and that ByteTrack plus TalkNet can reliably isolate single-speaker clips amid occlusions and motion.

What would settle it

Replace the event-stream conversion step with standard frame sampling on the same benchmarks and measure whether word error rate rises above 22.3% or latency exceeds 240 ms while keeping all other pipeline components fixed.

Figures

Figures reproduced from arXiv: 2606.31225 by Chenyang Xu, Hairui Liu, Jingping Fang, Junhao Chen, Lin Chen, Weidong Cai, Xiaoming Chen, Xiaorui Li.

Figure 1
Figure 1. Figure 1: Our work simultaneously focuses on the following two challenges: (a) Tradi￾tional VSR methods are only applicable to single-speaker scenarios, while we are able to adapt to multi-speaker scenarios; (b) Compared with video frames, event streams provide more subtle information gain for lip reading, thus compensating for the loss of fine-grained motion between consecutive frames. current mainstream research [… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of LipsFlow Framework. Event-based visual features are ex￾tracted via hierarchical interpolation and Conformer encoding, then conditioned on speaker embeddings through OT-CFM to generate semantic token representations. The BERT [12] weight tying is utilized alongside multi-level supervision signals during training (token-level LCE, sentence-level Lsem, and auxiliary CTC loss LCTC for align￾ment), … view at source ↗
Figure 3
Figure 3. Figure 3: Robustness of Identity Tracking in Multi-Speaker. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Stabilization and Extraction of Features. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Multi-Speaker Interference with Active Speaker Detection. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
read the original abstract

Visual Speech Recognition (VSR) tasks in complex multi-speaker scenarios are severely hindered by rapid head motions, occlusions, and subtle lip articulations. Traditional RGB-based methods struggle here due to low rates and motion blur of frames. To overcome these, we propose LipsFlow, a neuromorphic-inspired VSR framework that converts RGB videos into high-temporal-resolution event streams. For multi-speaker, we employ ByteTrack tracking and TalkNet active speaker detection to temporally segment scenes into single-speaker clips, enabling focused per-speaker analysis. By explicitly capturing microsecond-level articulatory dynamics via learnable event-based representations, LipsFlow achieves inherent robustness against visual degradation. To efficiently model these dense event-based features and adapt to speaker-specific articulatory patterns, we introduce Optimal Transport Conditional Flow Matching (OT-CFM). It enforces deterministic, straight-line trajectory generation in a semantic latent space, slashing inference latency to just two Ordinary Differential Equation (ODE) steps. Furthermore, we design a Dual-Level Semantic Supervision mechanism combining token-level BERT weight tying and sentence-level priors to resolve homophene ambiguities. Validated on competitive benchmarks, LipsFlow achieves a state-of-the-art WER of 22.3\% at 240 ms latency, establishing a highly robust and efficient paradigm for event-based VSR.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper proposes LipsFlow, a neuromorphic-inspired framework for multi-speaker visual speech recognition (VSR). It converts RGB video to high-temporal-resolution event streams, uses ByteTrack and TalkNet to segment multi-speaker scenes into single-speaker clips, models the event features with Optimal Transport Conditional Flow Matching (OT-CFM) to enable straight-line trajectories in latent space with only two ODE steps, and applies Dual-Level Semantic Supervision (BERT token tying plus sentence priors) to resolve homophenes. The central claim is a state-of-the-art word error rate (WER) of 22.3% at 240 ms latency on competitive benchmarks, establishing robustness to motion blur and low frame rates via event-based microsecond dynamics.

Significance. If the empirical claims hold after verification, the work would be significant as one of the first explorations of event-based neuromorphic methods combined with conditional flow matching for VSR. The reported latency reduction to two ODE steps and the multi-speaker handling via tracking/detection would address practical deployment constraints in noisy visual environments. However, the absence of any equations, ablations, or implementation details currently prevents assessment of whether these gains are attributable to the proposed components or to unstated preprocessing choices.

major comments (3)
  1. [Abstract / framework pipeline] Abstract and framework pipeline description: The headline result (22.3% WER at 240 ms) is presented as arising from event streams that capture microsecond-level articulatory dynamics unavailable to RGB. No description is given of the RGB-to-event simulator, its parameters, temporal resolution, or any ablation comparing it to frame interpolation or upsampling of the original RGB input. Without this, the claimed robustness to motion blur and low frame rate cannot be isolated from the conversion step itself.
  2. [Abstract] Abstract: The OT-CFM component is described as enforcing deterministic straight-line trajectories in semantic latent space and reducing inference to two ODE steps, yet no equations, loss formulation, conditioning mechanism, or dependence on learned parameters are supplied. This makes it impossible to determine whether the reported efficiency is a property of the method or an artifact of unspecified implementation choices.
  3. [Abstract] Abstract: Performance numbers are stated without dataset details, train/test splits, error bars, baseline comparisons, or ablation studies isolating the contribution of event conversion, OT-CFM, or Dual-Level Semantic Supervision. The soundness of the SOTA claim therefore cannot be evaluated from the supplied information.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the need for greater technical detail. We will revise the manuscript to address these points by adding the requested descriptions, formulations, and studies.

read point-by-point responses

  1. Referee: [Abstract / framework pipeline] Abstract and framework pipeline description: The headline result (22.3% WER at 240 ms) is presented as arising from event streams that capture microsecond-level articulatory dynamics unavailable to RGB. No description is given of the RGB-to-event simulator, its parameters, temporal resolution, or any ablation comparing it to frame interpolation or upsampling of the original RGB input. Without this, the claimed robustness to motion blur and low frame rate cannot be isolated from the conversion step itself.

    Authors: We agree that the event conversion process requires explicit description and validation. The revised manuscript will add a dedicated subsection detailing the RGB-to-event simulator, its parameters, and temporal resolution, along with an ablation comparing event streams to RGB frame interpolation and upsampling to isolate the contribution to robustness. revision: yes

  2. Referee: [Abstract] Abstract: The OT-CFM component is described as enforcing deterministic straight-line trajectories in semantic latent space and reducing inference to two ODE steps, yet no equations, loss formulation, conditioning mechanism, or dependence on learned parameters are supplied. This makes it impossible to determine whether the reported efficiency is a property of the method or an artifact of unspecified implementation choices.

    Authors: We acknowledge that the abstract omits the mathematical details. The revision will expand the methods section with the full OT-CFM equations, loss formulation, conditioning mechanism, and learned parameter dependencies to show that the two-ODE-step efficiency arises directly from the straight-line trajectory property of the approach. revision: yes

  3. Referee: [Abstract] Abstract: Performance numbers are stated without dataset details, train/test splits, error bars, baseline comparisons, or ablation studies isolating the contribution of event conversion, OT-CFM, or Dual-Level Semantic Supervision. The soundness of the SOTA claim therefore cannot be evaluated from the supplied information.

    Authors: We will update the manuscript to report the specific datasets, train/test splits, error bars across runs, full baseline comparisons, and ablations that isolate the contributions of event conversion, OT-CFM, and Dual-Level Semantic Supervision, enabling proper assessment of the 22.3% WER result. revision: yes

Circularity Check

0 steps flagged

No circularity detected; empirical claims rest on external benchmarks without internal reductions

full rationale

The manuscript reports an empirical WER result on competitive benchmarks and describes a pipeline involving RGB-to-event conversion, ByteTrack, TalkNet, and OT-CFM without providing equations, training objectives, or parameter-fitting procedures. No load-bearing step reduces by construction to its own inputs, self-citations, or fitted quantities; the central performance claim is therefore self-contained against external validation rather than internally forced.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities beyond the high-level framework name; all performance claims rest on unstated modeling choices and data-processing steps.

pith-pipeline@v0.9.1-grok · 5781 in / 1261 out tokens · 31005 ms · 2026-07-02T01:21:01.233038+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · 4 internal anchors

  1. [1]

    arXiv preprint arXiv:1912.02671 (2021)

    Arriandiaga, A., Morrone, G., Pasa, L., Badino, L., Bartolozzi, C.: Audio-visual target speaker enhancement on multi-talker environment using event-driven cam- eras. arXiv preprint arXiv:1912.02671 (2021)

    work page arXiv 1912

  2. [2]

    LipNet: End-to-End Sentence-level Lipreading

    Assael, Y.M., Shillingford, B., Whiteson, S., Freitas, N.D.: Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv:1611.01599 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

    Bulzomi, H., Schweiker, M., Gruel, A., Martinet, J.: End-to-end neuromorphic lip- reading. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 4101–4108. IEEE (2023)

    work page 2023

  4. [4]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Cao, J., Pang, J., Weng, X., Khirodkar, R., Kitani, K.: Observation-centric sort: Rethinking sort for robust multi-object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9686–9696 (2023)

    work page 2023

  5. [5]

    In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Cappellazzo, U., Kim, M., Chen, H., Ma, P., Petridis, S., Falavigna, D., Brutti, A., Pantic, M.: Large language models are strong audio-visual speech recognition learners. In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2025)

    work page 2025

  6. [6]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2018) 16 Lin Chen et al

    Chen, R.T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural Ordinary Differential Equations. In: Advances in Neural Information Processing Systems (NeurIPS) (2018) 16 Lin Chen et al

    work page 2018

  7. [7]

    In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2025)

    Choi, J., Kim, J.H., Li, J., Chung, J.S., Liu, S.: V2sflow: Video-to-speech genera- tion with speech decomposition and rectified flow. In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2025)

    work page 2025

  8. [8]

    IEEE Micro41(2), 29–35 (2021)

    Choquette, J., Gandhi, W., Giroux, O., Stam, N., Krashinsky, R.: NVIDIA A100 Tensor Core GPU: Performance and Innovation. IEEE Micro41(2), 29–35 (2021)

    work page 2021

  9. [9]

    In: Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)

    Chung, J.S., Nagrani, A., Zisserman, A.: VoxCeleb2: Deep Speaker Recognition. In: Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech). pp. 1086–1090 (2018)

    work page 2018

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

    Dampfhoffer, M., Mesquida, T.: Neuromorphic lip-reading with signed spiking gated recurrent units. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 2141–2151 (2024)

    work page 2024

  11. [11]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition (CVPR)

    Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S.: Retinaface: Single-shot multi-level face localisation in the wild. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition (CVPR). pp. 5203–5212 (2020)

    work page 2020

  12. [12]

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (NAACL). pp. 4171–4186 (2019)

    work page 2019

  13. [13]

    IEEE Transactions on Multimedia25, 8725–8737 (2023)

    Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H.: StrongSORT: Make DeepSORT Great Again. IEEE Transactions on Multimedia25, 8725–8737 (2023)

    work page 2023

  14. [14]

    In: International Conference on Learning Representations (ICLR) (2020)

    Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned step size quantization. In: International Conference on Learning Representations (ICLR) (2020)

    work page 2020

  15. [15]

    IEEE Transactions on Pattern Analysis and Machine In- telligence44(1), 154–180 (2022)

    Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A.J., Conradt, J., Daniilidis, K., Scaramuzza, D.: Event- based vision: A survey. IEEE Transactions on Pattern Analysis and Machine In- telligence44(1), 154–180 (2022)

    work page 2022

  16. [16]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR)

    Gehrig, D., Gehrig, M., Hidalgo-Carrió, J., Scaramuzza, D.: Video to events: Recy- cling video datasets for event cameras. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR). pp. 3586–3595 (2020)

    work page 2020

  17. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Gehrig, M., Scaramuzza, D.: Recurrent vision transformers for object detection with event cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13884–13893 (2023)

    work page 2023

  18. [18]

    In: Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP)

    Ghazvininejad, M., Levy, O., Liu, Y., Zettlemoyer, L.: Mask-predict: Parallel de- coding of conditional masked language models. In: Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP). pp. 6112– 6121 (2019)

    work page 2019

  19. [19]

    In: Proceedings of the 23rd International Conference on Machine Learning (ICML)

    Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural net- works. In: Proceedings of the 23rd International Conference on Machine Learning (ICML). pp. 369–376 (2006)

    work page 2006

  20. [20]

    In: Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)

    Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented transformer for speech recognition. In: Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech). pp. 5036–5040 (2020)

    work page 2020

  21. [21]

    In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2024) A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR 17

    Guo, Y., Du, C., Ma, Z., Chen, X., Yu, K.: VoiceFlow: Efficient text-to-speech with rectified flow matching. In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2024) A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR 17

    work page 2024

  22. [22]

    arXiv preprint arXiv:2212.06246 (2022)

    Haliassos, A., Ma, P., Mira, R., Petridis, S., Pantic, M.: Jointly learning visual and auditory speech representations from raw data. arXiv preprint arXiv:2212.06246 (2022)

    work page arXiv 2022

  23. [23]

    In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2025)

    Hao, B., Zhou, D., et al.: Lipgen: Viseme-guided lip video generation for enhancing visual speech recognition. In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2025)

    work page 2025

  24. [24]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    work page 2016

  25. [25]

    In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) (2017)

    Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., An- dreetto, M., Adam, H.: MobileNets: Efficient convolutional neural networks for mobile vision applications. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) (2017)

    work page 2017

  26. [26]

    In: Proceedings of the European Conference on Computer Vision (ECCV)

    Huang, Z., Zhang, T., Heng, W., Shi, B., Zhou, S.: Rife: Real-time intermediate flow estimation for video frame interpolation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 624–642 (2022)

    work page 2022

  27. [27]

    In: International Conference on Learning Representations (ICLR) (2017)

    Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-Softmax. In: International Conference on Learning Representations (ICLR) (2017)

    work page 2017

  28. [28]

    A Study of BFLOAT16 for Deep Learning Training

    Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D.T., Jammalamadaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., Dubey, P.: A Study of BFLOAT16 for Deep Learning Training. arXiv preprint arXiv:1905.12322 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  29. [29]

    Journal of Machine Learning Research10, 1755–1758 (2009)

    King, D.E.: Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research10, 1755–1758 (2009)

    work page 2009

  30. [30]

    In: Advances in Neural Informa- tion Processing Systems (NeurIPS) (2024)

    Kornilov, N., Mokrov, P., Gasnikov, A., Korotin, A.: Optimal Flow Matching: Learning Straight Trajectories in Just One Step. In: Advances in Neural Informa- tion Processing Systems (NeurIPS) (2024)

    work page 2024

  31. [31]

    In: International Conference on Learning Representations (ICLR) (2023)

    Lipman, Y., Chen, R.T.Q., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. In: International Conference on Learning Representations (ICLR) (2023)

    work page 2023

  32. [32]

    In: The Twelfth International Conference on Learning Representations (ICLR) (2024)

    Liu, A.H., Le, M., Vyas, A., Shi, B., Tjandra, A., Hsu, W.N.: Generative pre- training for speech with flow matching. In: The Twelfth International Conference on Learning Representations (ICLR) (2024)

    work page 2024

  33. [33]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  34. [34]

    In: International Conference on Learning Representations (ICLR) (2024)

    Liu, X., Zhang, X., Ma, J., Peng, J., Liu, Q.: Instaflow: One step is enough for high- quality diffusion-based text-to-image generation. In: International Conference on Learning Representations (ICLR) (2024)

    work page 2024

  35. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Liu, X., Lakomkin, E., Vougioukas, K., Ma, P., Chen, H., Xie, R., Doulaty, M., Moritz, N., Kolář, J., Petridis, S., Pantic, M., Fuegen, C.: Synthvsr: Scaling up visual speech recognition with synthetic supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18806–18815 (2023)

    work page 2023

  36. [36]

    In: Interna- tional Conference on Learning Representations (ICLR) (2019)

    Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. In: Interna- tional Conference on Learning Representations (ICLR) (2019)

    work page 2019

  37. [37]

    Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., Pantic, M.: Auto-avsr:Audio-visualspeechrecognitionwithautomaticlabels.In:ICASSP2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). p. 1–5 (2023) 18 Lin Chen et al

    work page 2023

  38. [38]

    In: ICASSP 2024 - 2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP) (2024)

    Mehta, S., Tu, R., Beskow, J., Székely, É., Henter, G.E.: Matcha-tts: A fast tts architecture with conditional flow matching. In: ICASSP 2024 - 2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP) (2024)

    work page 2024

  39. [39]

    Representation Learning with Contrastive Predictive Coding

    van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  40. [40]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4195– 4205 (2023)

    work page 2023

  41. [41]

    Qin, Z., Zhou, S., Wang, L., Duan, J., Hua, G., Tang, W.: MotionTrack: Learning RobustShort-termandLong-termMotionsforMulti-ObjectTracking.In:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17939–17948 (2023)

    work page 2023

  42. [42]

    In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP)

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP). pp. 3982–3992 (2019)

    work page 2019

  43. [43]

    In: International Conference on Learning Representations (ICLR) (2021)

    Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., Liu, T.Y.: FastSpeech 2: Fast and high-quality end-to-end text to speech. In: International Conference on Learning Representations (ICLR) (2021)

    work page 2021

  44. [44]

    Roth, J., Chaudhuri, S., Klejch, O., Marvin, R., Gallagher, A., Kaver, L., Ramaswamy, S., Stopczynski, A., Schmid, C., Xi, Z., Pantofaru, C.: AVA- ActiveSpeaker:AnAudio-VisualDatasetforActiveSpeakerDetection.In:ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). pp. 4492–4496. IEEE (2020)

    work page 2020

  45. [45]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Shen, X., Wang, X., Shen, L., Zhang, K., Yu, X.: Cross-view isolated sign language recognition via view synthesis and feature disentanglement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 20647– 20657. IEEE (2025)

    work page 2025

  46. [46]

    In: International Conference on Learning Representations (ICLR) (2022)

    Shi, B., Hsu, W.N., Lakhotia, K., Mohamed, A.: Learning audio-visual speech rep- resentation by masked multimodal cluster prediction. In: International Conference on Learning Representations (ICLR) (2022)

    work page 2022

  47. [47]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Tan, G., Wang, Y., Han, H., Cao, Y., Wu, F., Zha, Z.J.: Multi-Grained Spatio- Temporal Features Perceived Network for Event-Based Lip-Reading. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 20062–20071 (2022)

    work page 2022

  48. [48]

    In: Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)

    Tao, R., Pan, Z., Das, R.K., Qian, X., Shou, M.Z., Li, H.: Is someone speaking?: Exploring long-term temporal features for audio-visual active speaker detection. In: Proceedings of the 29th ACM International Conference on Multimedia (ACM MM). p. 3927–3935 (2021)

    work page 2021

  49. [49]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 30 (2017)

    work page 2017

  50. [50]

    IEEE Transactions on Acoustics, Speech, and Signal Processing37(3), 328–339 (1989)

    Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.J.: Phoneme recog- nition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing37(3), 328–339 (1989)

    work page 1989

  51. [51]

    Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., Qiao, Y.: VideoMAEV2:Scalingvideomaskedautoencoderswithdualmasking.In:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14549–14560 (2023) A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR 19

    work page 2023

  52. [52]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6023–6032 (2019)

    work page 2019

  53. [53]

    In: International Conference on Learning Representations (ICLR) (2018)

    Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (ICLR) (2018)

    work page 2018

  54. [54]

    In: Proceed- ings of the AAAI Conference on Artificial Intelligence (AAAI) (2025)

    Zhang, W., Wang, J., Luo, Y., Yu, L., Yu, W., He, Z., Shen, J.: Mtga: Multi-view temporal granularity aligned aggregation for event-based lip-reading. In: Proceed- ings of the AAAI Conference on Artificial Intelligence (AAAI) (2025)

    work page 2025

  55. [55]

    In: Proceedings of the European Conference on Computer Vision (ECCV)

    Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., Wang, X.: ByteTrack: Multi-Object Tracking by Associating Every Detection Box. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 1–21. Springer (2022)

    work page 2022

  56. [56]

    IEEE Transactions on Multimedia26, 1–13 (2023)

    Zhu, Q., Zhou, L., Zhang, Z., Liu, S., Jiao, B., Zhang, J., Dai, L., Jiang, D., Li, J., Wei, F.: Vatlm: Visual-audio-text pre-training with unified masked prediction for speech representation learning. IEEE Transactions on Multimedia26, 1–13 (2023)

    work page 2023