Pith. sign in

REVIEW 3 major objections 6 minor 29 references

For Romanian lip reading, human transcripts beat ASR labels at equal scale, but doubling the pseudo-labeled data closes most of the gap and transfers to word recognition.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 12:52 UTC pith:LMJB5RN2

load-bearing objection Solid dual-labeled Romanian VSR resource with clean scaling curves; the human-vs-pseudo comparison is real and the LRRo transfer is strong, even if the domain is narrow and only one architecture is used. the 3 major comments →

arxiv 2607.08112 v1 pith:LMJB5RN2 submitted 2026-07-09 cs.CV

VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness

Iulia-Maria Udrea , Alexandra Diaconu , Bogdan Alexe This is my paper
classification cs.CV
keywords visual speech recognitionlip readingRomanianpseudo-labelssupervision qualityaudio-visual speech recognitiondomain shiftlow-resource languages
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces VSRo-200, the first large Romanian sentence-level lip-reading corpus: 200 hours of podcast video, all labeled by a fine-tuned Romanian speech recognizer, with 100 of those hours also transcribed by people. The shared data let the authors hold the underlying videos fixed and ask how much label quality matters when training visual speech models. At matched sizes, human labels give lower word-error rates; once the cheaper automatic labels are scaled to the full 200 hours, performance continues to improve and approaches the human-supervised results. The same visual encoder, when frozen and topped with a light classifier, also lifts accuracy on the earlier LRRo isolated-word benchmarks well above prior numbers. The authors further show that audio-visual fusion helps under acoustic noise and that domain shifts (vlogs, technical talk, archival black-and-white footage) still hurt, driven by vocabulary gaps and visual degradation.

Core claim

In a controlled low-resource Romanian setting, human annotations outperform ASR pseudo-labels at fixed data scales, yet scaling the pseudo-labeled set from 100 h to 200 h further reduces word-error rate (e.g., 53.6 % to 48.8 % on unseen speakers) and largely closes the gap, while the learned visual representations transfer to isolated-word recognition and substantially surpass previously reported LRRo accuracies.

What carries the argument

VSRo-200: a 200-hour Romanian podcast corpus in which every clip carries an ASR pseudo-label and a deliberately overlapping 100-hour subset also carries human transcripts, enabling direct, distribution-matched comparison of supervision quality.

Load-bearing premise

The 100 hours chosen for human transcription are visually cleaner and more carefully filtered than the rest of the data, so measured gains may partly reflect easier videos rather than pure label quality.

What would settle it

Retrain the same architecture on a random 100-hour human-annotated sample drawn without the “visually clean” filter; if the human-versus-pseudo gap disappears or reverses at matched scale, the supervision-quality claim is undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VSRo-200, a 200-hour Romanian sentence-level visual speech recognition dataset built from YouTube podcasts. All clips carry ASR pseudo-labels from a fine-tuned Whisper model; a 100-hour subset also has human transcripts, enabling matched-scale comparison of supervision quality. Using a MultiVSR-style transformer, the authors report scaling curves for human vs. pseudo labels on seen and unseen speakers, OOD robustness across curated YouTube categories, confidence-aware late fusion for AVSR under Gaussian and babble noise, and transfer of frozen visual features to the LRRo isolated-word benchmark, where they substantially exceed prior Romanian results.

Significance. If the empirical claims hold, VSRo-200 fills a clear gap: Romanian VSR previously lacked large sentence-level data with both weak and full supervision under a shared distribution. The dual-annotation design is a genuine contribution for studying quality-vs-scale trade-offs in low-resource VSR, beyond purely pseudo-labeled multilingual corpora. The scaling curves (with three-shuffle variance at 100 h), OOD breakdown with OOV statistics, noise-robustness results, and strong LRRo transfer (e.g., Wild Acc@1 72.7% vs. prior 51.6%) are concrete, falsifiable findings that other groups can build on. Metadata-only release under CC BY-NC-SA and an explicit right-to-be-forgotten process are responsible practices for a face-video resource.

major comments (3)
  1. [Section 3 / 4.3] Section 3 / 4.3: The central supervision-quality claim is incomplete without reporting the ASR word error rate of the pseudo-labels against the human transcripts on the overlapping 100-hour subset (and ideally CER / insertion-deletion breakdown). Without this noise magnitude, readers cannot judge whether the small matched-scale gap (e.g., 53.3% vs. 53.6% WER on unseen speakers at 100 h) reflects high-quality pseudo-labels, residual visual difficulty, or both. Adding this single statistic would make the quality-vs-scale trade-off fully interpretable.
  2. [Section 4.4] Section 4.4 and Eq. (1)–(2): The AVSR experiments use late, confidence-aware fusion of two independently trained models (Whisper + MultiVSR) rather than a jointly trained audio-visual encoder. This is a legitimate robustness study, but the manuscript should state more explicitly that the gains are for post-hoc fusion, not end-to-end AVSR; joint training could change the relative contribution of vision and the entropy-based α_t rule. A short clarification (or one joint-training baseline if feasible) would prevent over-generalization of the multimodal claims.
  3. [Table 2] Table 2: Each OOD category is only ~15 minutes. Point estimates alone (e.g., black-and-white 87.97% WER) leave open whether differences are stable. Bootstrap or multi-seed intervals, or at least clip-level variance, would strengthen the domain-shift conclusions that currently rest on small samples.
minor comments (6)
  1. [Abstract / Section 4] Throughout: the acronym is inconsistently spaced as “A VSR” vs. “AVSR”; standardize to AVSR.
  2. [Figure 1] Figure 1: the two panels share the same y-axis range and legend; a single shared legend and slightly larger markers would improve readability in print.
  3. [Section 4.2] Section 4.2: state the tokenizer vocabulary size and whether it is shared with the Romanian Whisper model used for pseudo-labels; this affects the fusion setup in 4.4.
  4. [Table 1] Table 1: MultiVSR is listed as ~12000 hours; confirm the exact figure against the cited paper for consistency.
  5. [Section 3.1] Section 3.1: briefly note how many annotators and whether any inter-annotator agreement was measured on a double-transcribed sample.
  6. [A.3] Supplementary Table 4: the large seen-speaker gender gap under female-only training (67.26% vs. 51.20% WER) is interesting; a one-sentence pointer in the main text would help readers find it.

Circularity Check

0 steps flagged

No significant circularity: empirical dataset paper with held-out WER measurements and independent label sources

full rationale

VSRo-200 is an empirical dataset-and-benchmark paper, not a theoretical derivation. All central claims (human vs. pseudo-label scaling curves in Fig. 1, noise-robustness curves in Fig. 2, OOD WER in Table 2, LRRo transfer accuracies in Table 3) are measured on held-out speakers or external benchmarks. Pseudo-labels are produced by an independently fine-tuned ASR model; human labels are independent transcriptions. The only self-citation ([28]) supplies the Romanian ASR used for labeling and is not load-bearing for the VSR supervision or robustness conclusions. The fusion weight α_t is an entropy ratio evaluated at inference time (Eqs. 1–2), not a parameter fitted to the reported metrics. No equation equates a claimed prediction to a fitted free parameter, no uniqueness theorem is imported, and no known result is merely renamed. The derivation chain is therefore self-contained against external evaluation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

As an empirical dataset-and-benchmark paper the work rests on standard computer-vision and speech-processing assumptions plus a handful of concrete filtering and training choices. No new physical entities or free parameters are fitted to produce the central scaling claim; the free parameters listed are engineering thresholds and hyperparameters that affect data composition and optimization but are not tuned to manufacture the reported WER gaps.

free parameters (4)
  • face-embedding cosine threshold = 0.5
    Host-guest separation discards shots whose face embedding cosine similarity to a host bank exceeds 0.5; the numeric cut-off is chosen by the authors and directly controls which clips enter the final set.
  • SyncNet offset tolerance = ±10 frames
    Clips whose audio-visual offset exceeds ±10 frames are discarded; the window size is an author-chosen filter that affects both quantity and quality of retained data.
  • Adam learning rate = 1e-4
    All models are trained with learning rate 10^{-4} reduced on plateau; the value is a conventional but free hyper-parameter that influences final WER.
  • beam size and n-gram constraint = K=5, N=5
    Decoding uses beam size K=5 and an n-gram repetition mask of length N=5; both are free decoding choices that affect reported WER.
axioms (4)
  • domain assumption A Whisper-large model fine-tuned on Romanian produces pseudo-labels whose residual noise is the dominant difference from human transcripts when the underlying video is held fixed.
    Invoked throughout Section 3 and 4.3; without it the human-versus-pseudo comparison would be confounded by label-generation artifacts.
  • domain assumption Podcast videos with frontal, well-lit faces constitute a valid low-resource continuous-speech domain for studying VSR supervision and robustness.
    Stated in the data-collection design (Section 3) and later acknowledged as a limitation; the entire benchmark rests on this domain choice.
  • ad hoc to paper Entropy of the next-token distribution is a reliable proxy for modality confidence in the audio-visual fusion rule.
    Equations (1)–(2) in Section 4.4 define α_t from entropy; the mapping is motivated but not independently validated against other uncertainty measures.
  • domain assumption Standard face-detection, tracking and diarization tools (InsightFace, S3FD, Pyannote, SyncNet) introduce negligible systematic bias relative to the supervision-quality signal being measured.
    The entire preprocessing pipeline (Section 3) relies on these off-the-shelf components without ablation of their error rates.

pith-pipeline@v1.1.0-grok45 · 17140 in / 3160 out tokens · 40764 ms · 2026-07-10T12:52:26.507628+00:00 · methodology

0 comments
read the original abstract

We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast videos. All samples are annotated with pseudo-labels generated by a fine-tuned Romanian ASR model, while a subset of 100 hours is additionally transcribed by humans, enabling controlled analysis of supervision quality under a unified framework. Building on this dataset, we establish a benchmark for visual speech recognition in low-resource settings. We systematically study the impact of supervision quality, showing that while human annotations provide better performance at fixed data scales, pseudo-labels enable continued improvements through scalability. We further evaluate robustness under domain shift using curated out-of-distribution (OOD) test sets, and analyze audio-visual speech recognition (AVSR) under noisy conditions, where multimodal fusion significantly improves robustness compared to audio-only models. Finally, we demonstrate that representations learned on VSRo-200 transfer effectively to the LRRo benchmark for isolated word recognition, substantially outperforming previously reported results. Overall, VSRo-200 provides a new testbed for studying supervision, domain generalization, and multimodal fusion in low-resource visual speech recognition.

Figures

Figures reproduced from arXiv: 2607.08112 by Alexandra Diaconu, Bogdan Alexe, Iulia-Maria Udrea.

Figure 1
Figure 1. Figure 1: Impact of supervision quality and training data scale on VSR performance under different [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Robustness to acoustic noise under different conditions. (a) Gaussian noise with zero-shot [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the proposed confidence-aware audio-visual fusion. At each decoding step, [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 3 internal anchors

  1. [1]

    LipNet: End-to-End Sentence-level Lipreading

    Y . M. Assael, B. Shillingford, S. Whiteson, and N. De Freitas, “Lipnet: End-to-end sentence- level lipreading,”arXiv preprint arXiv:1611.01599, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    Lip reading in the wild,

    J. S. Chung and A. Zisserman, “Lip reading in the wild,” inAsian conference on computer vision. Springer, 2016, pp. 87–103

    work page 2016

  3. [3]

    Lip reading sentences in the wild

    J. S. Chung, A. W. Senior, O. Vinyals, A. Zissermanet al., “Lip reading sentences in the wild.” inCvpr, vol. 1, no. 2, 2017, p. 7

    work page 2017

  4. [4]

    LRS3-TED: a large-scale dataset for visual speech recognition

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,”arXiv preprint arXiv:1809.00496, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  5. [5]

    End-to-end audio- visual speech recognition,

    S. Petridis, T. Stafylakis, P. Ma, F. Cai, G. Tzimiropoulos, and M. Pantic, “End-to-end audio- visual speech recognition,” in2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 6548–6552

    work page 2018

  6. [6]

    Lipreading using temporal convolutional networks,

    B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading using temporal convolutional networks,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6319–6323

    work page 2020

  7. [7]

    Training strategies for improved lip- reading,

    P. Ma, Y . Wang, S. Petridis, J. Shen, and M. Pantic, “Training strategies for improved lip- reading,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 8472–8476

    work page 2022

  8. [8]

    A multimodal german dataset for automatic lip reading systems and transfer learning,

    G. Schwiebert, C. Weber, L. Qu, H. Siqueira, and S. Wermter, “A multimodal german dataset for automatic lip reading systems and transfer learning,” inProceedings of the Thirteenth Language Resources and Evaluation Conference, 2022, pp. 6829–6836

    work page 2022

  9. [9]

    Visual Speech Recognition for Multiple Languages in the Wild,

    P. Ma, S. Petridis, and M. Pantic, “Visual Speech Recognition for Multiple Languages in the Wild,”Nature Machine Intelligence, vol. 4, pp. 930–939, 2022. [Online]. Available: https://doi.org/10.1038/s42256-022-00550-z

    work page doi:10.1038/s42256-022-00550-z 2022

  10. [10]

    Scaling multilingual visual speech recognition,

    K. Prajwal, S. Hegde, and A. Zisserman, “Scaling multilingual visual speech recognition,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    work page 2025

  11. [11]

    Visual speech recognition for languages with limited labeled data using automatic labels from whisper,

    J. H. Yeo, M. Kim, S. Watanabe, and Y . M. Ro, “Visual speech recognition for languages with limited labeled data using automatic labels from whisper,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 471–10 475

    work page 2024

  12. [12]

    Lrro: a lip reading data set for the under-resourced romanian language,

    A. C. Jitaru, ¸ S. Abdulamit, and B. Ionescu, “Lrro: a lip reading data set for the under-resourced romanian language,” inProceedings of the 11th ACM multimedia systems conference, 2020, pp. 267–272

    work page 2020

  13. [13]

    Toward language-independent lip reading: A transfer learning approach,

    A.-C. Jitaru, L.-D. ¸ Stefan, and B. Ionescu, “Toward language-independent lip reading: A transfer learning approach,” in2021 International Symposium on Signals, Circuits and Systems (ISSCS). IEEE, 2021, pp. 1–4

    work page 2021

  14. [14]

    End-to-end lip reading in romanian with cross-lingual domain adaptation and lateral inhibition,

    E.-C. M˘anescu, R.-A. Sm˘adu, A.-M. Avram, D.-C. Cercel, and F. Pop, “End-to-end lip reading in romanian with cross-lingual domain adaptation and lateral inhibition,” in2023 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). IEEE, 2023, pp. 287–293

    work page 2023

  15. [15]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inProceedings of the 40th International Confer- ence on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 23–29 Jul 2023, pp. 28 492–28 518

    work page 2023

  16. [16]

    An audio-visual corpus for speech percep- tion and automatic speech recognition,

    M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech percep- tion and automatic speech recognition,”The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421–2424, 2006. 10

    work page 2006

  17. [17]

    Lrw-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,

    S. Yang, Y . Zhang, D. Feng, M. Yang, C. Wang, J. Xiao, K. Long, S. Shan, and X. Chen, “Lrw-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,” in2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, 2019, pp. 1–8

    work page 2019

  18. [18]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

    work page 2023

  19. [19]

    The Multilingual TEDx Corpus for Speech Recognition and Translation,

    E. Salesky, M. Wiesner, J. Bremerman, R. Cattoni, M. Negri, M. Turchi, D. W. Oard, and M. Post, “The Multilingual TEDx Corpus for Speech Recognition and Translation,” inInterspeech 2021, 2021, pp. 3655–3659

    work page 2021

  20. [20]

    V oxCeleb2: Deep Speaker Recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep Speaker Recognition,” in Interspeech 2018, 2018, pp. 1086–1090

    work page 2018

  21. [21]

    Looking to listen at the cocktail party: A speaker- independent audio-visual model for speech separation,

    A. Ephrat, T. Halperin, and S. Peleg, “Looking to listen at the cocktail party: A speaker- independent audio-visual model for speech separation,”ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 112:1–112:11, 2018

    work page 2018

  22. [22]

    PySceneDetect: Python and OpenCV-based scene cut/transition detection program & library,

    B. Castellano, “PySceneDetect: Python and OpenCV-based scene cut/transition detection program & library,” 2024. [Online]. Available: https://github.com/Breakthrough/PySceneDetect/

    work page 2024

  23. [23]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, X. Niannan, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” inCVPR, 2019

    work page 2019

  24. [24]

    Sample and computation redistribution for efficient face detection,

    J. Guo, J. Deng, A. Lattas, and S. Zafeiriou, “Sample and computation redistribution for efficient face detection,” inInternational Conference on Learning Representations, 2022

    work page 2022

  25. [25]

    Pyannote. audio: neural building blocks for speaker diarization,

    H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “Pyannote. audio: neural building blocks for speaker diarization,” in ICASSP 2020-2020 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2020, pp. 7124–7128

    work page 2020

  26. [26]

    S3fd: Single shot scale-invariant face detector,

    S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3fd: Single shot scale-invariant face detector,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 192–201

    work page 2017

  27. [27]

    Out of time: automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” inAsian conference on computer vision. Springer, 2016, pp. 251–263

    work page 2016

  28. [28]

    Ro-n3ws: Enhancing generalization in low-resource asr with diverse romanian speech benchmarks,

    A. Diaconu, M. Vînaga, and B. Alexe, “Ro-n3ws: Enhancing generalization in low-resource asr with diverse romanian speech benchmarks,”arXiv preprint arXiv:2603.02368, 2026

    work page arXiv 2026

  29. [29]

    MUSAN: A Music, Speech, and Noise Corpus

    D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,”arXiv preprint arXiv:1510.08484, 2015. A Technical appendices and supplementary material A.1 Ethical statement We share VSRo-200 under the Creative Commons Attribution Non-Commercial Share-Alike 4.0 (CC BY-NC-SA 4.0) license, aiming for open and responsible research. The dataset ...

    work page internal anchor Pith review Pith/arXiv arXiv 2015