{"id":"5257d6b2-8a99-4fcf-8a67-be57e6f9dde8","arxiv_id":"2504.15792","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A CNN-GRU lip-reading model reaches 87% accuracy on known speakers and 63% on unseen speakers for 18 German words, the first such system for German.","lead":"Researchers built and tested deep learning models that recognize 18 spoken German words from lip movements alone, using videos of 32 speakers. It is the first reported AI lip-reading system for German and could support hearing-impaired users and silent interfaces.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 87.3%/62.61% numbers rest on Vosk-generated word boundaries that are not manually verified; train and test share the same segmentation bias, so accuracy may reflect timestamp artifacts rather than lip-reading.","rationale":"The paper's strongest contribution is empirical: a first German lip-reading model with reported accuracies. For that claim to hold, the evaluation must measure lip-reading, not artifacts of the automatic pipeline. The Vosk-generated segmentation is the least secure link: the authors manually filtered validation/test clips for wrong words, but they did not report any check of timestamp accuracy or of the effect of forcing all clips to 28 frames. Since the same segmentation pipeline is used for training and test, a systematic Vosk bias is shared and cannot be detected from the reported accuracy alone. This is not an attack on Vosk; it is a specific, testable risk. The reader identified the same issue, and I agree with the conditional verdict. I did not select the small unseen-speaker set (only four speakers in Dataset C) as the primary attack because the known-speaker accuracy also depends on the segmentation pipeline, and the label/timestamp concern is upstream of both numbers. The comparison to English systems is also imperfect, but it is secondary. A manual timestamp re-annotation experiment would settle the point; the authors can run it without releasing the dataset. If the accuracy survives, the central claim is substantially strengthened; if not, the paper should report accuracy after corrected segmentation and avoid claiming parity with English benchmarks.","tokens_in":9330,"tokens_out":4106,"duration_ms":39591,"concrete_test":"Ask the authors to draw a stratified random sample of ~100 clips per word class from the Dataset B test set (1,800 clips), manually annotate the true lip-movement interval for the spoken word in each clip (using audio and visual inspection), compare these intervals with the Vosk timestamps, and re-run the final GRUConv model on clips re-extracted with the corrected intervals. Report per-class accuracy before and after. If accuracy changes by more than 3–5 percentage points or drops below the claimed 87.3%, the Vosk segmentation is a learned artifact and the headline numbers need qualification; if accuracy is unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 states that all word labels and timestamps were generated with the Vosk speech recognizer and that only validation/test clips were manually reviewed, with 'videos containing incorrect words' removed. Section 2.2 then forces every clip to 28 frames, by shortening or extending. This leaves two unverified links in the accuracy chain. First, manual review checks word identity, not whether the Vosk time interval actually contains the full articulated word; if Vosk has systematic onset/offset bias for particular words, those biased clips are used both for training and for the reported test evaluation. The network can then achieve high accuracy by learning the segmentation artifact (e.g., a consistent portion of the preceding/following phoneme) rather than the lip movement of the word. Second, the 28-frame fixed length interacts with this: words longer than 28 frames are either truncated or temporally compressed, and if truncation/compression correlates with Vosk's timing errors, the model has an additional non-linguistic cue. Because the same pipeline produced train and test clips, the 87.3% (Dataset B) and 62.61% (Dataset C) accuracies do not, by themselves, establish performance on cleanly segmented German lip-reading input. The reader's weakest assumption is therefore the right one; it is load-bearing because both headline numbers would be inflated if the artifact is present.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a deep-learning system for recognizing 18 German words from silent lip movements, based on a newly assembled dataset of 38,391 video clips from 32 YouTube speakers. The authors compare three architectures (Conv3D, GRU, GRUConv), two crops (face vs. lips), and six color spaces, selecting hyperparameters by validation accuracy. They report 70.29% validation accuracy for the Conv3D model with lip crops on Dataset A, 77.59% for GRUConv on the same validation split, and, after retraining on the larger Dataset B, 87.3% test accuracy on known speakers and 62.61% on an unseen-speaker Dataset C. The paper claims this is the first automatic lip-reading system for German and that its accuracy is comparable to English-language systems.","tokens_in":9603,"tokens_out":8512,"duration_ms":73070,"significance":"If the reported numbers are robust, this is a useful contribution: it introduces the first German-language lip-reading dataset and demonstrates that a CNN-GRU hybrid can generalize to unseen speakers at 62.61% accuracy on 18 words. The experimental design has real strengths: speaker-disjoint datasets, a held-out test set, validation-based model selection, and explicit comparisons of preprocessing choices. The paper is also transparent about its main limitations, including automatic speech-recognition-based labeling and fixed-length video processing. However, the headline accuracies depend on the automatic word boundaries produced by Vosk, and the manuscript does not provide evidence that these boundaries are reliable enough to rule out segmentation artifacts. The lack of repeated runs and confidence intervals further limits the strength of the claims, especially for the small unseen-speaker set.","major_comments":[{"comment":"The training data are labeled entirely by the Vosk speech recognizer; manual review is applied only to the validation and test sets and only to remove clips with incorrect words, not to check word boundaries. If Vosk has systematic timing biases (e.g., a consistent lead or lag for particular words), those biased clips are used in both training and testing, and the network could learn to exploit the segmentation artifact rather than the lip movement. This is load-bearing for the headline numbers (87.3% on Dataset B, 62.61% on Dataset C). Please quantify the reliability of the automatic boundaries, for example by manually verifying a stratified sample of training clips or by re-evaluating the final model on a test set with manually corrected boundaries, and report the resulting accuracy change.","section":"2.1"},{"comment":"All clips are forced to 28 frames by 'shortened or extended ... by repeating the last frame'. This means words longer than about one second are truncated, and the truncation point is determined by Vosk's timestamps. The combination of automatic boundaries and fixed-length truncation can create a non-linguistic cue (e.g., the model may see the same consistent portion of an adjacent phoneme). Please report the per-class duration distribution, the fraction of clips truncated or padded, and either adopt a variable-length model (CTC is mentioned as future work) or provide an analysis showing that truncation behavior does not differ systematically across classes.","section":"2.2"},{"comment":"All accuracy values are single runs with best-epoch selection on a validation set; no confidence intervals, repeated runs, or significance tests are reported. The conclusion that color spaces 'did not reveal any relevant different correct classification rates' is based on one-run differences of up to about 4 percentage points (LAB 73.47% vs. HSV 69.23%), which could easily be noise. Please report repeated-seed means and standard deviations for the key comparisons (crop, color space, model, final test) and use a paired test (e.g., McNemar) for the final evaluations.","section":"3"},{"comment":"Speaker-independent generalization is evaluated on Dataset C with only 4 speakers and 3,950 clips, yet only the aggregate accuracy of 62.61% is reported. With such a small number of speakers, the aggregate could be dominated by one or two speakers. Please report per-speaker accuracies and clip counts, and provide a confidence interval for the 62.61% estimate.","section":"3"}],"minor_comments":[{"comment":"The test split for Dataset B is not listed in the table, although the text and the 87.3% result imply that such a split exists; please state the number of test clips used.","section":"Table 1"},{"comment":"The word 'Valdiation' in Table 1 should be 'Validation'.","section":"Table 1"},{"comment":"The word 'bedeudet' in Table 2 appears to be a typo for 'bedeutet' (the German word for 'means').","section":"Table 2"},{"comment":"The abstract says 1,806 video clips were 'split into word segments'; please clarify that these are 1,806 source videos from which 38,391 word-segment clips were generated by automatic speech recognition.","section":"Abstract and 2.1"},{"comment":"The confusion matrices described in the text are not visible in this version of the manuscript; please include the figure or provide a table of per-class accuracies so the claimed confusions can be examined.","section":"Figure 5"},{"comment":"The Discussion contains a typo: 'decrreased' should be 'decreased'.","section":"4"},{"comment":"The comparison with English-language systems (word accuracies of 76%-98%) is not controlled, because the word sets, number of classes, and datasets differ; please qualify the statement that the German system is 'comparable to English-language algorithms'.","section":"4"},{"comment":"The claim that 'dataset size impacts accuracy more than model choice' is not directly supported, since Datasets A and B differ not only in size but also in speakers and in the exact training procedure; please soften the claim or add a controlled comparison.","section":"4"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is an English translation of a paper published in HNO in 2022. The scientific content appears unchanged; the arXiv version may be more appropriate as a technical report than as a new archival submission. The lack of a released dataset or code limits reproducibility, though the paper is transparent about its pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the resource: a 38k-clip, 32-speaker, 18-word German lipreading dataset and the first benchmark for German visual word recognition. That alone makes the paper worth engaging with. The architecture work is standard—Conv3D, GRU, GRUConv—but the contribution was never supposed to be architectural. The paper also earns credit for a clean experimental pipeline: speaker-disjoint splits, lip versus face cropping, six color spaces, validation-based model selection, and an honest discussion of the limitations. The face-crop result (34% vs 70% for lips) and the near-null color-space result are useful negative findings that the field should have on record.\n\nNow the soft spots, in proportion. The load-bearing issue is the segmentation. All word boundaries and labels came from Vosk, and only validation/test clips were manually checked for word identity, not for whether the timestamp actually contained the full articulation. The fixed 28-frame window then shortens or extends every clip. The stress-test worry is real: if Vosk has systematic onset/offset bias, train and test share that same bias, and the network can learn the segmentation artifact rather than the lip movement. That would inflate both the 87.3% known-speaker and 62.61% unseen-speaker numbers. The paper's own discussion acknowledges \"potential errors in dataset creation\" but does not quantify them.\n\nSecondary issues: no code or data release, so nothing is independently checkable; every reported number is a single run with no error bars or significance tests; Dataset C is small and imbalanced (one class is 24% of it); and the comparison to English-language systems is not apples-to-apples, since those use much larger vocabularies and different evaluation protocols. None of these are fatal for a resource-creation paper, but together they mean the headline accuracies should be read as provisional.\n\nWho gets value from this: visual speech recognition researchers, particularly those working on German or on low-resource lipreading, and the hearing-aid community that wants a German-language feasibility data point. It deserves a serious referee and a request for revision—mainly to release the dataset and code, verify or re-segment at least the test set by hand, and report multiple runs. If the authors can show the accuracies survive a clean re-segmentation, this becomes a solid benchmark paper.\n\nRecommendation: send it to peer review, but make artifact release and label verification conditions of acceptance.","headline":"First German lipreading dataset and benchmark, but the headline accuracies rest on unverified ASR timestamps and no released artifacts, so treat the numbers as provisional.","tokens_in":10127,"tokens_out":1174,"would_cite":true,"duration_ms":12891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a deep-learning system that recognizes 18 German words from lip movements, reaching 87.3% accuracy on known speakers and 62.61% on unseen speakers.","keywords":["lip reading","visual speech recognition","German language","deep learning","gated recurrent unit","3D convolutional neural network","word classification","GRUConv"],"falsifier":"Hand-correct the word labels and timestamps of a random sample of the training segments, retrain GRUConv from scratch on the corrected subset, and evaluate on datasets B and C; a large change in accuracy or confusion pattern would show that the reported numbers depend on the automatic labeling. In parallel, test the model on a manually verified set of German words longer than 28 frames; if accuracy drops sharply on those, the fixed frame window is truncating information.","tokens_in":9107,"feed_emoji":"👄","tokens_out":6290,"duration_ms":55027,"temperature":0.7,"pith_summary":"This paper sets out to build the first automatic deep-learning lip-reading system for German. It claims that a hybrid network called GRUConv, combining 3D convolutions with gated recurrent units, can classify 18 visually distinguishable German words from short lip-movement videos, reaching 87.3% accuracy on speakers it has seen and 62.61% on speakers it has not. The result matters because no comparable German-language system existed, and because it suggests German lip-reading can reach the accuracy level reported for English systems. The authors also show that cropping to the mouth region matters far more than color space: lip crops give about 70% validation accuracy versus 34% for face crops, while color spaces differ by only a few percent.","feed_headline":"German lip-reading AI hits 87% on known speakers","feed_subtitle":"First deep-learning German word recognizer from lip movements, with 63% accuracy on unseen speakers.","key_machinery":"The load-bearing component is the GRUConv model, a hybrid in which three 3D-convolutional blocks (with batch normalization, max pooling, and dropout) extract spatial appearance at each frame, two bidirectional GRU layers track how the mouth region changes over the 28-frame input, and a multilayer perceptron maps the resulting representation to 18 word classes via softmax. The mouth-region crop, resized to 150×100 pixels and converted to LAB color space, supplies the input; the fixed 28-frame window is padded by repeating the last frame. This architecture is what the comparison experiments identify as the highest-accuracy configuration, reaching 77.59% validation accuracy on dataset A and then 87.3% on dataset B's test set.","core_discovery":"On the authors' own terms, the central discovery is that a GRUConv architecture—3D convolutional layers for spatial features followed by two bidirectional gated recurrent unit layers for temporal features, then a softmax classifier—can recognize German words from lip movements at accuracy comparable to English-language lip-reading systems. Trained on 38,391 automatically segmented clips of 18 words from 32 speakers, the model reaches 87.3% correct classification on a test set from known speakers and 62.61% on a test set of unseen speakers. The paper further reports that the largest accuracy gains come from the choice of input region rather than color space or model family, and that expanding the training data improved accuracy more than changing architecture.","pith_inferences":["Because only validation and test clips were manually cleaned, a fully hand-cleaned training set might shift the reported accuracies; the direction and size of that shift is unknown from the paper alone.","The fixed 28-frame input suggests that longer words are compressed or truncated; variable-length inputs with connectionist temporal classification, which the authors mention as future work, would likely improve unseen-speaker accuracy if word-length variation is the bottleneck.","The claim that dataset size matters more than model choice is a testable statement: training GRUConv on progressively larger random subsets of Dataset B should show a smooth accuracy curve that flattens, revealing the marginal value of more German lip-reading data.","Because the videos come from natural, in-the-wild German speech, the method may transfer to real-world captioning or silent-dictation applications, but measuring that transfer requires testing on non-YouTube recordings with different lighting and camera angles."],"forward_implications":["A German-language machine lip-reading system is feasible at word level, with accuracy on known speakers around 87% and on unseen speakers around 63%.","The mouth-region crop, not the color space, is the decisive preprocessing choice; face crops lose roughly half the accuracy.","Adding training data appears to improve accuracy more than changing model architecture, so dataset construction is the main scaling lever.","The same pipeline could extend to more German word classes and, with future changes such as connectionist temporal classification, to variable-length words or sentences.","If the speaker-independent result holds, the system could support automated subtitles for hearing-impaired viewers and silent dictation in noisy environments."],"supporting_citations":[{"why":"The open-source speech recognizer used to generate word labels and timestamps for all training clips, so the dataset correctness depends on it.","marker":"[24]"},{"why":"LipNet, cited as the English sentence-level lip-reading benchmark that the German system is compared against.","marker":"[8]"},{"why":"The large-scale English visual speech recognition system, the other English-language reference point for comparability.","marker":"[9]"},{"why":"A CNN-RNN hybrid that reports English word-level accuracy of 76% and sentence accuracy of 47%, providing the numeric comparison for German word accuracy.","marker":"[20]"},{"why":"Survey of lip-reading technology giving the 76%-98% range of English word recognition rates that supports the claim that German results are comparable.","marker":"[7]"},{"why":"Study of color spaces for image classification, cited for the expectation that color space choice has limited effect and that LAB often performs best.","marker":"[25]"}],"fun_headline_variants":["German lip-reading AI reaches 87% accuracy","First German lip-reading AI: 87% known, 63% unseen","Input region key for German lip-reading AI","Neural net reads German lips at 87% accuracy","GRUConv model tops German lip-reading at 87%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels and word timestamps come from an automatic speech recognizer and were not manually verified; if they mislabel or mistime words, the network learns from noisy targets and the reported accuracies no longer describe clean German lip-reading.","fun_headline_variants_meta":{"raw":{"variants":["German lip-reading AI reaches 87% accuracy","First German lip-reading AI: 87% known, 63% unseen","Input region key for German lip-reading AI","Neural net reads German lips at 87% accuracy","GRUConv model tops German lip-reading at 87%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1883,"prompt_tokens":958,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":844}},"tokens_in":574,"tokens_out":925,"duration_ms":7103,"temperature":1.0,"reasoning_tokens":844,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:17:34.259918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-correct the word labels and timestamps of a random sample of the training segments, retrain GRUConv from scratch on the corrected subset, and evaluate on datasets B and C; a large change in accuracy or confusion pattern would show that the reported numbers depend on the automatic labeling. In parallel, test the model on a manually verified set of German words longer than 28 frames; if accuracy drops sharply on those, the fixed frame window is truncating information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The open-source speech recognizer used to generate word labels and timestamps for all training clips, so the dataset correctness depends on it."},{"cited_title":"Lip reading sentences in the wild","cited_arxiv_id":null,"evidence_quote":"A CNN-RNN hybrid that reports English word-level accuracy of 76% and sentence accuracy of 47%, providing the numeric comparison for German word accuracy."},{"cited_title":"A survey of research on lipreading technology","cited_arxiv_id":null,"evidence_quote":"Survey of lip-reading technology giving the 76%-98% range of English word recognition rates that supports the claim that German results are comparable."},{"cited_title":"Gowda and Chun Yuan","cited_arxiv_id":null,"evidence_quote":"Study of color spaces for image classification, cited for the expectation that color space choice has limited effect and that LAB often performs best."}],"review_version":1}