{"id":"74c86c2a-5de2-4926-af1c-1cb8cefa5e5a","arxiv_id":"2502.00464","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"An end-to-end Spanish lipreading system based on the CTC/Attention architecture achieves state-of-the-art WER of 24.8% on VLRF and 34.5%/59.5% on LIP-RTVE speaker-dependent/independent partitions, with an ablation study and a new four-corpus benchmark.","lead":"This paper adapts a hybrid CTC/Attention lipreading model to Spanish and reports large word error rate improvements on two Spanish video corpora, plus a consolidated benchmark across four datasets. A generalist reader might care because lipreading models for low-resource languages are rare, and the paper shows that transfer learning from English pretraining can drive Spanish visual speech recognition.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: the qualitative SOTA claim is robust despite single-seed runs and acknowledged transfer dependence.","rationale":"I read the full manuscript and checked the reader's weakest_assumption against the actual tables. The single-run issue is real but is not load-bearing for the strongest claim because the SOTA margins are 25-45 absolute WER points. Additionally, Table 2 contains multiple ablated configurations that all beat the previous bests, so the conclusion does not hinge on one lucky seed. The transfer concern is based on the 'no fine-tuning' rows, but those rows demonstrate exactly what the paper claims: the pretrained model must be fine-tuned. There is no evidence of test-set leakage: the LM fine-tuning text is described as domain text, and the without-LM rows already exceed prior SOTA. The paper is transparent about GPU-memory limits and provides code/model links. I therefore cannot identify a concern that would invalidate the central claim; the CONDITIONAL verdict is a reasonable request for reproducibility, and I would keep it.","tokens_in":17815,"tokens_out":15136,"duration_ms":134865,"concrete_test":"Run the released recipe (https://github.com/david-gimeno/evaluating-end2end-spanish-lipreading) on VLRF and LIP-RTVE SD/SI with 5 random seeds and verify the mean WER remains below roughly 35% (VLRF), 45% (SD), and 70% (SI); also verify that the sign of the data-augmentation delta is stable. If these fall outside those ranges, the precise margins need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The central claim—large WER improvements over prior results on VLRF and LIP-RTVE—survives scrutiny. The most plausible threat, single-seed training, would have to produce >20-point WER swings to erase the margins (24.8 vs 59.7; 34.5 vs ~80; 59.5 vs ~95), and even the '− Language Model' rows in Table 2 (35.9, 41.4, 64.8) remain far below previous bests. The dependence on Ma et al. (2022) pretrained weights is explicitly tested by the '− Fine-Tuning' rows, which show poor transfer without target-domain fine-tuning; this is a stated experimental design, not a hidden assumption. The only genuine limitations are (i) point estimates are single runs, so the precise margins and small ablation deltas (e.g., −0.9 to −2.2 for data augmentation) are not statistically quantified, and (ii) the LIP-RTVE LM is fine-tuned on in-domain TV-newscast text, which could inflate the LM contribution; however, removing the LM still leaves results far above prior SOTA. Neither limitation undermines the qualitative central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an end-to-end continuous Spanish lipreading system based on the hybrid CTC/Attention architecture of Ma et al. (2022), pre-trained on more than 1500 hours of English data and fine-tuned on Spanish corpora. It reports word error rates of 24.8% on VLRF (speaker-dependent), 34.5% on LIP-RTVE (speaker-dependent), and 59.5% on LIP-RTVE (speaker-independent), compared with previous bests of approximately 59.7%, 80%, and 95%, respectively. The paper also presents an ablation study covering data augmentation, language-model fine-tuning, language-model removal, and the CTC and attention decoder branches; an error analysis with WER histograms, Zipf-law statistics, and vocabulary-coverage measures; and a proposed four-corpus Spanish benchmark that includes CMU-MOSEAS es and MuA ViC es.","tokens_in":17998,"tokens_out":6894,"duration_ms":64297,"significance":"If the reported results are reproducible, this is a substantial advance for Spanish visual speech recognition, with large absolute WER reductions on two corpora of very different recording conditions. The paper is commendably transparent: it reports bootstrap confidence intervals, states GPU-related limitations, releases code and trained models, and clearly marks results taken from Ma et al. (2022) rather than re-run. The qualitative state-of-the-art claim is robust because the reported margins (24.8 vs. 59.7 on VLRF; 34.5 vs. ~80 and 59.5 vs. ~95 on LIP-RTVE) are far larger than plausible training-stochasticity effects. The main weakness is that the ablation study, which is a stated contribution, is based on a single training run per configuration and several ablation deltas fall within the reported confidence intervals; this affects component-level conclusions but not the headline results.","major_comments":[{"comment":"The ablation conclusions in Table 2 are not statistically supported because each configuration is a single training run and the bootstrap intervals quantify only test-set sampling noise, not training stochasticity. For example, removing data augmentation changes WER from 24.8±3.4 to 23.9±3.4 on VLRF and from 34.5±1.2 to 32.3±1.2 on LIP-RTVE SD, both deltas being smaller than the reported interval half-widths. The statements that data augmentation \"slightly hinders\" learning and the relative ordering of the LM fine-tuning effects therefore cannot be distinguished from random seed variation. Please provide repeated-seed results or explicitly reframe these component-level conclusions as preliminary observations rather than definite findings.","section":"Section 6, Table 2"},{"comment":"All hyperparameters (α=0.1, λ=0.1, β=0.4, learning rates, epoch counts, beam size, and random crop size) are inherited from Ma et al. (2022) and are not tuned on VLRF or LIP-RTVE. Consequently, the phrase \"best setting along our experiments\" in Section 6 is too strong: the configuration is the best among the tested ablations, not a tuned optimum. The large SOTA margins are probably insensitive to this, but the ablation deltas and the ranking of configurations could change under a targeted hyperparameter search. The manuscript should state explicitly that no hyperparameter search was performed on the target datasets.","section":"Section 5.1 and Section 6"}],"minor_comments":[{"comment":"The row \"− Fine-Tuning >100†\" is used to argue that transfer without fine-tuning fails, but the footnote only says \"due to a peculiarity of the WER metric.\" Please explain concretely, for instance that insertion-heavy hypotheses can push WER above 100%, so that readers can interpret the magnitude.","section":"Table 2, footnote"},{"comment":"The text \"one the first multi-lingual datasets\" is missing \"of,\" and the corpus names are inconsistent (LIP-RTVE versus LIP-R TVE, MuA ViC versus MuA ViCes). Please standardize the names throughout.","section":"Section 4"},{"comment":"It is unclear whether the \"nearly 300k sentences\" used for VLRF LM fine-tuning are the VLRF transcripts themselves or an external text corpus; please clarify the source and its relationship to the VLRF test set.","section":"Section 5.1"},{"comment":"Since the paper cites Yeo et al. (2024) as surpassing the state of the art on MuA ViC with around 46% WER, the benchmark table should either include that result or explicitly note that only Ma et al. (2022) results are listed for comparability.","section":"Table 5"},{"comment":"The caption states that both axes are logarithmic, but the axis labels use powers of two without axis titles; adding axis titles and units would improve readability.","section":"Figure 5"},{"comment":"The term \"rigorous\" for the error analysis is somewhat strong given the absence of statistical tests on the Zipf-law and vocabulary-coverage comparisons; \"detailed\" would be more measured.","section":"Abstract and Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a language-resources and speech-processing venue. I do not see a circularity or novelty concern: the SOTA numbers are independent held-out measurements, and the dependence on Ma et al. (2022) pretrained weights is explicitly tested by the \"− Fine-Tuning\" rows. The main revision request concerns the statistical support for the ablation component of the paper; the headline SOTA claim is robust. The self-citations are to the databases and prior baselines being improved, which is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ping. This paper is a solid empirical contribution to Spanish VSR. The headline result is real: fine-tuning Ma et al.'s CTC/Attention Conformer on VLRF and LIP-RTVE drops WER from 59.7% to 24.8% on VLRF, and from roughly 80/95% to 34.5/59.5% on LIP-RTVE SD/SI. Even the lowest-performing full-model variant in the ablation, with no LM, sits far below previous bests. That is a genuine improvement, not a fiddling gain.\n\nWhat's actually new is not the architecture—it's borrowed from Ma et al. (2022)—but the fine-tuning recipes, the consolidated four-corpus Spanish benchmark (controlled, TV, vlog, TED), the component ablation, and a Zipf-law-based error analysis. The paper is honest about its constraints: single GPU, batch size 1, no auxiliary-task pre-training, and it flags these in the limitations section. That transparency earns credit.\n\nSoft spots, in proportion. First, every configuration is a single training run. The bootstrap CIs only capture test-set sampling, not seed variance. For the big SOTA claims this doesn't matter—you'd need implausibly large run-to-run variance to erase a 20-45 point gap. But the smaller ablation deltas (e.g., data augmentation seems to hurt by 0.9-2.2 points) are within noise and should be read as suggestive. Second, the LM for LIP-RTVE is fine-tuned on contemporary TV-newscast text, which is in-domain; that likely inflates the LM contribution, though the LM-free rows still beat prior SOTA. Third, the Zipf analysis is descriptive. The coverage statistics correlate with WER across corpora, but the authors don't claim causation, and the comparison of VLRF vs LIP-RTVE SD shows other factors dominate. Fine.\n\nWho should read it: anyone working on lipreading for low-resource languages, or on Spanish VSR specifically. It's a useful baseline consolidator and an honest ablation. If this crossed my desk as an editor, I'd send it to review—the empirical margins are important even if the novelty is incremental. I'd ask the authors for either multiple seeds or a clear statement that the reported deltas are single-run point estimates, and to double-check LM fine-tuning leakage. But those are revision-level requests, not grounds for rejection.\n\nMy take: if you're in this area, cite it and use the benchmark. Not a game-changer, but a solid, trustworthy step forward.","headline":"Solid empirical step for Spanish lipreading: fine-tuning an existing Conformer gives huge WER gains and a new four-corpus benchmark, with honest limitations; single-seed runs are a caveat, not a fatal flaw.","tokens_in":18644,"tokens_out":2349,"would_cite":true,"duration_ms":21109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A visual-only system reaches 24.8% WER on Spanish studio lipreading and 34.5% on TV newscasts.","keywords":["Visual Speech Recognition","lipreading","Spanish","CTC/Attention","Conformer","benchmark","word error rate"],"falsifier":"Retrain the VLRF and LIP-RTVE configurations multiple times with fresh random seeds and from random initialization without English pretraining. If the seed-to-seed spread in WER approaches the claimed margins over the previous state of the art, or if removing the English pretraining keeps error rates above the old baselines, the conclusion that the architecture itself is responsible for the gains would be weakened.","tokens_in":17518,"feed_emoji":"👄","tokens_out":6943,"duration_ms":61625,"temperature":0.7,"pith_summary":"The paper aims to show that an end-to-end visual speech recognition system, built on the hybrid CTC/Attention architecture and fine-tuned from English-pretrained weights, brings continuous Spanish lipreading from a feasibility exercise to usable accuracy. On the controlled VLRF corpus the best configuration reaches 24.8% word error rate, against a prior best of 59.7%; on LIP-RTVE television newscasts it reaches 34.5% in the speaker-dependent setting and 59.5% in the speaker-independent setting, where previous systems sat near 80% and 95%. The authors also consolidate a four-corpus Spanish lipreading benchmark spanning studio recordings, TV newscasts, YouTube vlogs, and TED talks, with ablations showing which components carry the accuracy. If the results hold, silent, visual-only Spanish speech recognition is no longer limited to laboratory conditions.","feed_headline":"Spanish lipreader halves error rates on studio and TV data","feed_subtitle":"Visual-only system reports 24.8% and 34.5% WER on two Spanish corpora, roughly halving prior best results.","key_machinery":"The load-bearing mechanism is the hybrid CTC/Attention training and inference scheme, in which a CTC branch imposes speech-like local alignment while an attention decoder handles global text conditioning, trained jointly and combined with a character-level Transformer language model during beam search. Around it sits a visual front end that replaces the first layer of a ResNet with a 3D convolution to capture lip motion across five frames, feeding a twelve-layer Conformer encoder that mixes local convolutional and global attention features. The paper further depends on transfer: pretrained English visual speech weights and a general Spanish language model are fine-tuned on each target corpus, and the no-fine-tuning rows in the ablation show that without this initialization the system collapses on VLRF.","core_discovery":"The central claim is that a single end-to-end architecture, consisting of a 3D-convolution front end, a Conformer encoder, a CTC/Attention decoder, and a character-level Transformer language model integrated by shallow fusion, can be fine-tuned to outperform every previously published continuous Spanish lipreading system when it is initialized from English-pretrained visual speech weights. The paper reports 24.8% WER on VLRF, 34.5% on the speaker-dependent LIP-RTVE partition, and 59.5% on the speaker-independent partition, with the ablation study attributing the largest degradation to removing the CTC branch and the language model. The authors present this as evidence that the architecture adapts to different data conditions, and they add CMU-MOSEAS and MuAViC Spanish results from the pretrained model to form a four-way benchmark for future work.","pith_inferences":["The same fine-tuning recipe may transfer to other low-resource languages, but the authors do not test this; the large no-fine-tuning gap suggests the English-pretrained visual encoder is supplying most of the skill.","The Zipf and vocabulary-coverage analysis points to a cheap data-selection heuristic for future corpora: prioritize utterances whose words fall in the high-frequency head of the target domain, since test coverage by the top 1000 words tracks task difficulty.","Because each configuration was run once, the ablation order of importance is provisional; a repeated-seed study could plausibly reorder the CTC and language-model contributions.","In practical terms, 24.8% and 34.5% WER make visual-only Spanish lipreading usable for constrained tasks such as keyword spotting or studio dictation, while 59.5% on open-domain speaker-independent newscasts remains too error-prone for unconstrained use."],"forward_implications":["Visual-only Spanish lipreading is workable in controlled and broadcast settings, with the best reported WERs of 24.8% on VLRF and 34.5% on speaker-dependent LIP-RTVE.","The CTC branch is the most important decoder component: dropping it raises WER by 15.9 to 35.6 points, while dropping the attention decoder raises it by 9.1 to 18.6 points.","The language model matters substantially: removing it costs 5.3 to 11.1 points, and fine-tuning it to the domain pays off most on TV newscasts.","The proposed four-corpus benchmark gives later Spanish lipreading work fixed points of comparison across controlled recordings, TV newscasts, YouTube vlogs, and TED talks."],"supporting_citations":[{"why":"Supplies the hybrid CTC/Attention Conformer architecture, the English pretrained weights, the Spanish fine-tuning recipe, the character tokenizer, and the previous Spanish results on CMU-MOSEAS and MuAViC that the benchmark builds on.","marker":"(Ma et al., 2022)"},{"why":"Defines the VLRF corpus, the controlled-recording Spanish database whose previous best result this paper beats.","marker":"(Fernandez-Lopez et al., 2017)"},{"why":"Defines LIP-RTVE, the TV newscast database, its speaker-dependent and speaker-independent partitions, and the baseline around 80% and 95% WER.","marker":"(Gimeno-Gómez & Martínez-Hinarejos, 2022)"},{"why":"Sets the prior state of the art on VLRF at 59.7% WER that this paper improves to 24.8%.","marker":"(Gimeno-Gómez & Martínez-Hinarejos, 2024)"},{"why":"Introduces the hybrid CTC/Attention training and shallow-fusion decoding paradigm that is the paper's core architecture.","marker":"(Watanabe et al., 2017)"},{"why":"Provides the Spanish YouTube monologue corpus (CMU-MOSEAS) used as one benchmark domain and a fine-tuning dataset.","marker":"(Zadeh et al., 2020)"},{"why":"Provides the Multilingual-TEDx Spanish split on which MuAViC Spanish is built, giving the largest fine-tuning corpus.","marker":"(Salesky et al., 2021)"},{"why":"Provides MuAViC, the multilingual TED-talk corpus whose Spanish partition completes the benchmark.","marker":"(Anwar et al., 2023)"},{"why":"Reports an earlier end-to-end VLRF result around 72% WER and supplies the external sentences used to fine-tune the language model for VLRF.","marker":"(Fernandez-Lopez & Sukno, 2022)"}],"fun_headline_variants":["Spanish lipreading halves WER on VLRF and LIP-RTVE","Single end-to-end model halves Spanish lipreading errors","Spanish lipreading: error rates cut in half on two corpora","One model, two corpora, half the errors: Spanish lipreading","Spanish lipreading: new state-of-the-art on two databases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains rest on the assumption that one training run per configuration is representative, and that the English-pretrained front end transfers to Spanish lips; the paper reports no repeated seeds, and its own no-fine-tuning rows show the pretrained weights are doing most of the work.","fun_headline_variants_meta":{"raw":{"variants":["Spanish lipreading halves WER on VLRF and LIP-RTVE","Single end-to-end model halves Spanish lipreading errors","Spanish lipreading: error rates cut in half on two corpora","One model, two corpora, half the errors: Spanish lipreading","Spanish lipreading: new state-of-the-art on two databases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00119,"raw_usage":{"total_tokens":4907,"prompt_tokens":937,"completion_tokens":3970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3879}},"tokens_in":553,"tokens_out":3970,"duration_ms":26952,"temperature":1.0,"reasoning_tokens":3879,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:53:00.169488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the VLRF and LIP-RTVE configurations multiple times with fresh random seeds and from random initialization without English pretraining. If the seed-to-seed spread in WER approaches the claimed margins over the previous state of the art, or if removing the English pretraining keeps error rates above the old baselines, the conclusion that the architecture itself is responsible for the gains would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the VLRF corpus, the controlled-recording Spanish database whose previous best result this paper beats."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MuAViC, the multilingual TED-talk corpus whose Spanish partition completes the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports an earlier end-to-end VLRF result around 72% WER and supplies the external sentences used to fine-tune the language model for VLRF."}],"review_version":1}