{"id":"dc6e161b-90ca-4c91-9aec-87870403fa48","arxiv_id":"2505.13784","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Joint training on lipreading and German Sign Language mouthing videos improves mouthing word recognition over a mouthing-only model, though the gains are small and lack statistical validation.","lead":"This paper tries to recognize the silent mouth movements that accompany German Sign Language, matching each video clip to the German word being mouthed. It finds that training the model together with lipreading videos gives small accuracy gains, and that mouthing and lipreading are related but distinct tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central MTL improvement over the mouthing baseline is within clip-level noise; the paper reports no error bars and the M test set is randomly sampled from the same signers as training.","rationale":"The reader identified exactly the same load-bearing weakness: the random clip-level split allows signer identity leakage, and the small reported accuracy gaps are not accompanied by error bars or significance tests. My independent reading of Table I confirms this: the MTL-vs-baseline differences on M are between -1.07 and +2.53 percentage points, while the per-class test set has only 50 clips, making within-run variance likely on the order of 2 percentage points per class and clip-level identity correlation a serious confound. The conclusion and abstract generalize beyond what the evidence supports, because the strongest gains appear on the perturbed set and on VSR datasets rather than consistently on the clean mouthing test set. The reader's CONDITIONAL verdict is appropriate: the direction is plausible and the paper provides a useful dataset and reproducible code, but the central quantitative claim needs robustness checks (repeated runs, error bars, signer-independent evaluation) before it can be accepted at face value. No dishonesty or methodological fraud is suggested; the issue is standard experimental rigor for small datasets.","tokens_in":8421,"tokens_out":1461,"duration_ms":12208,"concrete_test":"Retrain the baseline and the best MTL configuration (M & GLipsR, and M & GLipsM) at least 5 times with different seeds and jointly with a signer-disjoint split of the mouthing dataset (e.g., group signers into training/validation/test so no signer overlaps splits), then report mean and standard deviation of top-1 accuracy on M and M. If the MTL advantage over the baseline disappears or falls within one standard error under the signer-independent split, the central claim should be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Table I reports single-run top-1 accuracies with no error bars, significance tests, or signer-independent partition. Training/validation/test splits are obtained by randomly sampling video clips from the Public DGS Corpus (Section III-A1), so clips of the same signer appear in all three splits. Under these conditions, the claimed central result that 'MTL improves mouthing recognition' rests on differences such as baseline M 44.00% vs. MTL M&GLipsR 46.53% on M, and 34.67% vs. 41.07% on the perturbed M. With 750 test clips (15 classes x 50), the standard error of a 44% accuracy is roughly 1.8 percentage points, so a 2.53-point gap is about 1.4 standard errors and is not significant without repeated runs. Moreover, without a signer-independent split, part of the apparent improvement may reflect signer-specific appearance or articulation patterns memorized from training clips. The abstract's stronger claim that multi-task learning improves 'both mouthing recognition and VSR accuracy as well as model robustness' is therefore not established by the reported numbers. The paper itself acknowledges (Section IV) that fine-tuning gains are 'little benefit on the M test set' and that DANN 'does not outperform the baseline for M', and it relies on MTL as the main positive result; that result needs variance estimates and a signer-conditioned evaluation to support the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a mouthing recognition task for German Sign Language (DGS) in which mouthing videos are classified into the corresponding spoken German words. To address the scarcity of mouthing annotations, the authors construct a mouthing dataset from the Public DGS Corpus and compare three transfer-learning strategies — fine-tuning, domain-adversarial training (DANN), and multi-task learning (MTL) — using three visual speech recognition (VSR) datasets: LRW (English), GLipsM (German words matching the mouthing vocabulary), and GLipsR (German words unrelated to the mouthing vocabulary). The central claim, stated in the abstract and conclusion, is that multi-task learning improves both mouthing recognition and VSR accuracy as well as model robustness, suggesting that mouthing recognition should be treated as a distinct but related task to VSR.","tokens_in":8847,"tokens_out":5311,"duration_ms":47552,"significance":"If the reported improvements are real, the paper makes a useful and timely contribution to sign language recognition by demonstrating that comparatively abundant VSR data can benefit a much scarcer mouthing recognition task, and it is one of the first works to use spoken words as labels for mouthing recognition. The experimental design is thoughtful in its manipulation of task relatedness (same vocabulary, different vocabulary, and different language), and the public release of the code is a concrete asset for reproducibility. However, the strength of the conclusions is currently limited by the lack of variance estimates, the use of a single run per condition, and the absence of a signer-independent evaluation, all of which are load-bearing for the headline claims.","major_comments":[{"comment":"The M dataset is split into training, validation, and test sets by randomly sampling video clips in an 8:1:1 ratio, with no signer-independent partition. Since the Public DGS Corpus contains videos from many signers across Germany, clips from the same signer almost certainly appear in both training and test sets. This creates a risk that the reported accuracies reflect signer identity or recording conditions rather than mouthing recognition quality. A signer-disjoint split, or at least a detailed report of the number of signers per split and per class, is necessary to support the abstract's and conclusion's claim that MTL improves mouthing recognition for new signers. Without such evidence, the central result may not transfer to signers unseen during training.","section":"Section III-A1"},{"comment":"All accuracy numbers in Table I are single-run results with no error bars, no repeated seeds, and no significance tests. With 750 test clips (15 classes × 50), the standard error of the 44.00% baseline accuracy is roughly 1.8 percentage points, so the best MTL gain on M (46.53% for MTL M&GLipsR) is only about 1.4 standard errors above the baseline. The statement in Section IV that MTL \"improves the performance and robustness for mouthing recognition significantly\" is therefore not statistically supported. Similarly, the robustness claim rests on a single perturbed test set (M) generated with one combination of Gaussian noise and histogram equalization, without quantifying the variance of that perturbation. The paper should report multiple seeds, confidence intervals, or a significance test before making this claim.","section":"Table I and Section IV"},{"comment":"The claim that MTL significantly improves German VSR accuracy is not consistent with Table I. For GLipsM, the baseline is 38.18%, while MTL M&GLipsM reaches 37.92% (lower than baseline) and MTL M&GLipsM&LRW reaches 40.05%. For GLipsR, MTL M&GLipsR reaches 41.60% versus a 41.47% baseline, a difference of 0.13 points, and only the three-task model M&GLipsM&GLipsR shows a clearer gain (43.20% versus 41.47%). Given these mixed results, the conclusion that MTL \"significantly improves both mouthing recognition and German lipreading\" overstates the evidence. The paper should either temper the claim or provide statistical evidence of consistent improvement across tasks.","section":"Section IV and Conclusion"},{"comment":"The statement that \"task relatedness does not greatly impact the transfer learning benefits in this context\" is based on a comparison of only three VSR datasets that differ in language, word overlap, and video quality. The huge discrepancy in baseline difficulty (LRW 83.87% versus GLipsM 38.18% and GLipsR 41.47%) is itself a confound: any transfer effects are entangled with the absolute difficulty of each source task. The paper should either analyze task relatedness with more controlled conditions (e.g., same-language datasets with different word overlap) or soften this claim to reflect the limited evidence.","section":"Section IV"}],"minor_comments":[{"comment":"The column header \"M M\" is ambiguous; the second column appears to refer to the perturbed test set M, but the bar is missing. Use a distinct label such as \"M (perturbed)\" to avoid confusion.","section":"Table I"},{"comment":"The text says GLips is already split into training, validation, and test sets, but it does not state how many instances remain per class after randomly selecting the 15 related and 15 unrelated word classes. Clarify that the subsequent removal step yields 397 training instances per class in all datasets.","section":"Section III-A2"},{"comment":"The description of DANN states that each batch consists of samples from both domains in equal proportions, but it does not specify how the gradient reversal loss is weighted relative to the classification loss. Since equal weighting is later mentioned for MTL and DANN, state this explicitly in the method section.","section":"Section III-B2"},{"comment":"The early-stopping rule (validation accuracy for M does not improve for over 100 epochs after surpassing 1000 epochs) is unusual and not justified. A standard early-stopping criterion based on the M validation set would be simpler; if this particular rule is intended to reduce overfitting, explain the rationale.","section":"Section III-C"},{"comment":"In the related work, the authors mention that prior work follows either approach (a) recognizing words/phrases or (b) recognizing mouth shapes. It would help to explicitly state that the present work follows approach (a), which clarifies the novelty claim.","section":"Section II"},{"comment":"The statement acknowledges that the vast majority of signers are visibly white adults, but it does not cite the corpus documentation for this observation. Adding a citation to the Public DGS Corpus documentation would strengthen this acknowledgment.","section":"Ethical Impact Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a conference-style contribution (indeed, it is published at FG 2025), and the core idea — transferring from VSR to mouthing recognition via MTL — is reasonable and timely. The main obstacles to acceptance in a journal are the missing statistical rigor (single runs, no error bars) and the absence of a signer-conditioned evaluation, both of which are addressable with additional experiments. I do not see an irreparable flaw, only claims that currently outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for two concrete reasons: it formulates mouthing recognition as word-level classification with spoken-language labels, and it gives the first comparison of VSR-to-mouthing transfer that I know of. The authors also release code, and the architecture and transfer methods are standard enough that the comparison is fair.\n\nWhat the paper does well: the dataset construction from Public DGS Corpus is careful, with balanced classes and equal-size splits across the four datasets. The experimental design, comparing fine-tuning, DANN, and MTL across three VSR sources with varying relatedness, is sensible. The perturbed-test-set robustness check is a nice addition. And the authors are honest in the discussion: they note that fine-tuning gives little benefit on M, that DANN does not beat the baseline, and that only MTL shows gains.\n\nWhere the soft spots are: the central claim, that MTL improves both mouthing and VSR accuracy as well as robustness, is not established by the numbers. Table I shows single-run top-1 accuracies with no error bars or significance tests. The best MTL gap over the mouthing baseline is 46.53% vs. 44.00% on M, which is about 1.4 standard errors with 750 test clips. On the perturbed test set the gap is larger (41.07 vs. 34.67), but that is also a single run, and the perturbed set is a synthetic variant of the same clips. More importantly, the train/validation/test split is done by randomly sampling clips, so the same signers appear in all splits. That means part of the MTL advantage could come from memorizing signer-specific appearance. The abstract's string claim — \"improves both mouthing recognition and VSR accuracy as well as model robustness\" — is not consistent with Table I, where several MTL runs hurt VSR accuracy (e.g., M & GLipsM & GLipsR drops GLipsR from 41.47 to 41.60? actually baseline GLipsR 41.47, MTL gives 43.20; but LRW drops from 83.87 to 80-81). So the conclusion should be softened.\n\nThat said, these are fixable problems. The task formulation and dataset selection are a real contribution, and the direction of the work is plausible, even if the headline effect is not yet proven. The absence of a released dataset is a minor but real limitation, since the authors only provide code.\n\nRecommendation: send it to a serious referee. The referee should push for repeated seeds, error bars, and a signer-independent split (or at least a signer-disjoint subset analysis). If the MTL advantage survives that, the paper becomes solid. If it does not, the dataset and task definition still justify the paper as a preliminary study. Worth reading for anyone working on sign language recognition or transfer learning for non-manual signals.","headline":"A useful new dataset and task formulation, but the paper's central claim that MTL improves mouthing recognition is not statistically supported by the single-run results on a randomly split, non-signer-independent test set.","tokens_in":9251,"tokens_out":1883,"would_cite":false,"duration_ms":18857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-task learning with visual speech recognition improves both mouthing recognition in German Sign Language and German lipreading; mouthing should be treated as a related but distinct task.","keywords":["sign language recognition","mouthing recognition","visual speech recognition","lipreading","transfer learning","multi-task learning","German Sign Language","domain adaptation"],"falsifier":"Re-run the comparison with a signer-disjoint split of the mouthing corpus—holding out all clips of a set of signers for testing—and check whether the multi-task gains over the mouthing-only baseline persist; if they shrink to noise, the central transfer claim does not generalize to unseen signers.","tokens_in":8204,"feed_emoji":"👄","tokens_out":8449,"duration_ms":75021,"temperature":0.7,"pith_summary":"This paper tries to establish that German Sign Language mouthing can be recognized as word-level classification of short mouth videos, and that the chronic shortage of mouthing annotations can be alleviated by borrowing data from visual speech recognition (lipreading). The authors build a 15-class mouthing dataset from the Public DGS Corpus and pair it with three lipreading datasets: English, German with unrelated words, and German with the same target words. Comparing fine-tuning, domain-adversarial training, and multi-task learning, they find that multi-task learning—treating mouthing and lipreading as distinct but related tasks—improves mouthing accuracy and resilience to unseen perturbations, and also helps German lipreading. If the finding holds, existing lipreading corpora could substitute for expensive sign-language mouthing annotation in other data-scarce settings.","feed_headline":"Lipreading data boosts sign-language mouthing recognition","feed_subtitle":"Joint training with lipreading lifts mouthing accuracy and stability, even when the spoken words do not match.","key_machinery":"The load-bearing mechanism is hard parameter sharing in multi-task learning: a single 3D-convolutional plus bidirectional-GRU encoder processes 30-frame, 96×96 mouth crops, and separate task-specific classifiers predict the word for each dataset, with all losses summed with equal weight. Each dataset is treated as one task, so mouthing and lipreading are forced to share a visual representation while keeping their own output layers. The same encoder architecture, in a domain-adversarial variant with a gradient-reversal layer, provides the contrast case that spoken articulation and mouthing are not the same task under a domain shift.","core_discovery":"On its own terms, the paper's discovery is that mouthing recognition and visual speech recognition are better modelled as separate tasks sharing an encoder than as one task viewed through different domains. A baseline trained only on the mouthing dataset reaches 44.00% test accuracy, and 34.67% on a perturbed version of that test set. Multi-task models trained jointly on mouthing plus a German lipreading dataset reach up to 46.53% on the clean test set and 41.07% on the perturbed set, with five of seven multi-task configurations beating the mouthing-only baseline. Treating mouthing and lipreading as the same task through domain-adversarial training fails to beat the baseline. Notably, the German lipreading dataset with words unrelated to the mouthing labels produced the largest mouthing gains, while the English lipreading source produced smaller gains and the German same-word source fell in between, which the authors read as evidence that task relatedness is not the main driver of transfer.","pith_inferences":["The reported gains are measured on a random split of video clips, not on signer-disjoint splits; a signer-holdout test is the direct way to check whether the improvements reflect mouthing generalization rather than signer identity.","If the transfer effect is driven by low-level mouth dynamics, then lipreading data from other languages or even synthetic mouth animations might serve as source tasks, and the English result in this paper may understate that possibility because of differences in video quality.","The datasets were artificially matched in size (397 clips per class), which is not the regime where transfer usually pays off most; testing with a much larger lipreading source or a much smaller mouthing target could show larger gains than reported here.","The same multi-task recipe could be applied to other non-manual markers in sign languages, as long as they can be assigned discrete labels aligned with spoken or signed units."],"forward_implications":["Mouthing recognition can be framed as word classification, giving it a label space that lipreading datasets already populate.","Languages with scarce sign-language mouthing annotations can recruit lipreading corpora through multi-task learning instead of waiting for new manual annotation.","Because unrelated German words transferred as well as or better than matching words, shared mouth dynamics may matter more than shared vocabulary.","Transfer improves resilience to unseen perturbations even where clean-test gains are small, so auxiliary lipreading data may buy stability before it buys accuracy.","Stacking too many transfer tasks hurts performance: joint training on all four datasets produced the lowest mouthing accuracy, indicating task conflicts."],"supporting_citations":[{"why":"Supplies the Public DGS Corpus mouthing videos and timestamp annotations from which the 15-class mouthing dataset is built.","marker":"[19]"},{"why":"Provides the LRW English lipreading dataset used as the least-related transfer source task.","marker":"[7]"},{"why":"Provides the German Lips dataset used for both the same-word and unrelated-word German lipreading source tasks.","marker":"[25]"},{"why":"Describes the domain-adversarial network (DANN) used to test whether mouthing and lipreading form one task under domain shift.","marker":"[13]"},{"why":"Gives the mouth-region cropping implementation that standardizes all videos to 96×96 mouth crops as model input.","marker":"[20]"},{"why":"Supplies the Conv3D plus recurrent architecture that the baseline and transfer models are built upon.","marker":"[3]"},{"why":"Motivates the hard parameter sharing multi-task learning design with shared layers and task-specific layers.","marker":"[28]"}],"fun_headline_variants":["Mouthing recognition gets a lipreading boost","Sign language mouthing improves via lipreading multi-task","Unrelated lipreading words still lift mouthing accuracy","Treat mouthing and lipreading as separate tasks, share encoder","Lipreading aids mouthing even when words don't match"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mouthing dataset is split into training, validation, and test sets by randomly sampling video clips, so the same signer can appear on both sides of the split; the accuracy comparisons therefore assume the models are learning mouthing patterns, not signer identity.","fun_headline_variants_meta":{"raw":{"variants":["Mouthing recognition gets a lipreading boost","Sign language mouthing improves via lipreading multi-task","Unrelated lipreading words still lift mouthing accuracy","Treat mouthing and lipreading as separate tasks, share encoder","Lipreading aids mouthing even when words don't match"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2252,"prompt_tokens":895,"completion_tokens":1357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1279}},"tokens_in":511,"tokens_out":1357,"duration_ms":8781,"temperature":1.0,"reasoning_tokens":1279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:28.517231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with a signer-disjoint split of the mouthing corpus—holding out all clips of a set of signers for testing—and check whether the multi-task gains over the mouthing-only baseline persist; if they shrink to noise, the central transfer claim does not generalize to unseen signers.","supporting_citations":[{"cited_title":"Konrad, T","cited_arxiv_id":null,"evidence_quote":"Supplies the Public DGS Corpus mouthing videos and timestamp annotations from which the 15-class mouthing dataset is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LRW English lipreading dataset used as the least-related transfer source task."},{"cited_title":"Schwiebert, C","cited_arxiv_id":null,"evidence_quote":"Provides the German Lips dataset used for both the same-word and unrelated-word German lipreading source tasks."},{"cited_title":"Ganin, E","cited_arxiv_id":null,"evidence_quote":"Describes the domain-adversarial network (DANN) used to test whether mouthing and lipreading form one task under domain shift."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the mouth-region cropping implementation that standardizes all videos to 96×96 mouth crops as model input."},{"cited_title":"Zhang and Q","cited_arxiv_id":null,"evidence_quote":"Motivates the hard parameter sharing multi-task learning design with shared layers and task-specific layers."}],"review_version":1}