{"id":"afc8e253-4a36-4db4-8f3a-78e3103147d5","arxiv_id":"2501.13996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A custom 7-word Persian lip-reading dataset is used to train an LSTM that reports 89% accuracy and is deployed on the Surena-V humanoid robot for real-time command recognition.","lead":"This paper builds a small Persian lip-reading dataset of 2,800 video clips from 20 speakers saying 7 words, trains CNN and LSTM models, and reports 89% accuracy with an LSTM, which is then installed on the Surena-V humanoid robot for real-time word recognition. It is a niche application paper: useful for Persian human-robot interaction, but with limited evaluation, no public data or code, and some unresolved methodological gaps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 89% LSTM accuracy is not established until the train/test split is shown to be speaker-independent; with only 20 speakers, a clip-level random split likely lets the model exploit speaker identity rather than lip-reading.","rationale":"The reader's weakest assumption identifies exactly the same vulnerability, and I agree it is the load-bearing one. The paper is otherwise a plausible application of standard CNN/LSTM pipelines to a new small dataset, and the dataset description (20 speakers, 7 words, 20 repetitions, 2,800 clips) is internally consistent with the counts. However, the central quantitative claim—89% test accuracy—cannot be interpreted without knowing the split. In speaker-dependent lip reading, identity cues are famously strong, especially when the input is a face crop; with only 20 speakers, a random clip split is not a minor methodological detail but a potential source of most of the apparent performance. The absence of released dataset/code means the only way to resolve this is for the authors to report a speaker-independent evaluation or demonstrate that their split was already speaker-disjoint. No fraud or internal contradiction is alleged; the paper simply has not supplied the necessary evidence. I therefore keep the reader's CONDITIONAL verdict unchanged, with the condition being that the speaker-independent result must be reported. Additional weaknesses (no baselines, no error bars, no quantitative robot evaluation) are secondary and reinforce the same recommendation rather than changing it.","tokens_in":6804,"tokens_out":4904,"duration_ms":44084,"concrete_test":"Ask the authors to rerun the exact LSTM pipeline with a speaker-disjoint split: group clips by speaker (20 groups) and evaluate with leave-one-speaker-out or GroupShuffleSplit (e.g., train 16 speakers, validate 2, test 2 unseen speakers), reporting mean and standard deviation of test accuracy across folds. Also report, for the original '89%' split, a per-speaker contingency table showing how many speakers appear in both training and test. If held-out-speaker accuracy is substantially below 89% or near chance, the original result is a speaker-leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the central accuracy claim is that training and test clips are independent samples of the word-to-lip-movement mapping. The manuscript never establishes this. In Methodology, II. Direct Method, B. LSTM Approach, the only split description is: 'The dataset was divided into three subsets—training, validation, and test data ... A random state was set during the data splitting process.' No grouping by speaker is mentioned. With 20 speakers and 2,800 clips (7 words × 20 repetitions × 20 speakers), a clip-level random split almost certainly places each speaker in both training and test sets. Because the direct pipeline feeds 300×300 face crops into the CNN/LSTM, the model can memorize speaker-specific appearance, lighting, and recording session rather than generalizable lip dynamics. This would inflate test accuracy and invalidate the claim that Surena-V recognizes commands from new users. The text/table mismatch worsens the problem: the text reports 89% test accuracy, but Table III reports only Val-Acc/Train-Acc (LSTM 95/93) on the last epoch, so the reported number is not cleanly tied to a held-out partition. The conclusion's own call for 'a wider variety of speakers' in future work concedes the generalization risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a Persian lip-reading system developed for the Surena-V humanoid robot. The authors collected a new dataset of 2,800 video clips from 20 speakers pronouncing seven Persian words, recorded with the robot's RGB-D camera at 20 fps. Two methods are compared: an indirect method based on facial landmark tracking (dlib points 48–68) fed into CNN classifiers, and a direct method operating on raw video frames using a CNN and an LSTM. The LSTM is reported to achieve 89% test accuracy and is implemented on the robot for real-time command recognition. The paper also discusses the dataset construction, preprocessing, training curves, and a qualitative real-world implementation using YOLOv5 for object detection.","tokens_in":7057,"tokens_out":2088,"duration_ms":20469,"significance":"If the reported accuracy holds under proper external validation, the work would be a useful contribution to Persian lip reading and to human-robot interaction in noisy or hearing-impaired settings. The authors provide a new resource (the dataset), compare two complementary pipelines, and demonstrate integration on a physical humanoid robot, which is a practical step beyond offline recognition benchmarks. The central weakness is that the evaluation does not establish speaker independence, and the reported 89% test accuracy is not directly backed by the results table, which lists only training and validation metrics. Because the claims are reproducible and the dataset is a concrete deliverable, the paper merits revision rather than rejection, but the accuracy claim currently lacks the statistical grounding needed for a strong acceptance.","major_comments":[{"comment":"The train/validation/test split is not described as speaker-independent. With 20 speakers and 2,800 clips, a random clip-level split almost certainly places the same speaker in both training and test sets. Under such a split, the model can exploit speaker-specific appearance, lighting, and recording session rather than generalizable lip dynamics. The central claim of 89% test accuracy therefore cannot be interpreted as the robot recognizing words from unseen users. The authors must either explicitly state that the split groups clips by speaker or re-run the evaluation with a leave-speaker-out protocol; the conclusion's own call for \"a wider variety of speakers\" in future work confirms this concern.","section":"II.B (LSTM Approach) and Dataset section"},{"comment":"The text reports an LSTM test accuracy of 89%, but Table III only lists Val-Acc/Train-Acc for the last epoch (LSTM: 95/93; CNN: 75/76). The reported 89% is thus not tied to a clearly defined held-out partition, and no test-set accuracy, precision, recall, F1, confusion matrix, or error bars are provided anywhere. This mismatch makes it impossible to verify the paper's headline number and its relation to the validation accuracy. The authors should report consistent metrics for a fixed, disclosed test partition, preferably with per-class results and confidence intervals.","section":"IV.B (Direct Method Results) and Table III"},{"comment":"The indirect method's best result is reported as 52% accuracy (Table III), but the paper does not state a chance level or compare this result to any baseline. For a seven-class problem, random classification is 14.3%, so 52% is above chance, yet it is far below the direct method and no statistical significance test is provided. More importantly, the description of normalization and preprocessing is vague ('normalization techniques are employed'), and the claimed robustness to lighting and head movement is not empirically demonstrated. The authors should clarify what baseline this 52% is measured against and whether the landmark-based pipeline is evaluated under the same split protocol as the direct method.","section":"II.A (Indirect Method)"}],"minor_comments":[{"comment":"The LSTM architecture description does not specify key hyperparameters (number of LSTM units per layer, dropout rate, learning rate, batch size, number of epochs, optimizer variant), despite the text mentioning 'standard optimization techniques.' These details are necessary for reproducibility.","section":"II.B"},{"comment":"The phrase 'the MobileNet model is trained to recognize patterns' appears after a discussion of VGG19 and ResNet; clarify whether MobileNet is the only model trained or whether all three are compared, and report results for VGG19 and ResNet if they were evaluated.","section":"II.A"},{"comment":"The paper states that performance metrics such as precision, recall, and F1 were reported, but no such metrics appear in the manuscript; either include them or remove the statement.","section":"III.B (Results, Direct Method)"},{"comment":"There is a typo: 'the sturctue of both approachs' should read 'the structure of both approaches.' Also, the sentence beginning 'also Table II provides a summary' is a grammatical fragment.","section":"Methodology (direct method descriptions)"},{"comment":"Figure 3 shows subplots labeled (a) and (b), but the caption does not specify which subplot corresponds to the CNN and which to the LSTM. Please clarify in the caption or text.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a conference-style systems paper with a useful new dataset and a physical robot demonstration. The main risk is that the headline accuracy is not validated under speaker-independent conditions, which is a standard expectation for lip-reading evaluations. The authors are not misrepresenting their data, but the evaluation protocol needs to be corrected before publication. I do not see evidence of inappropriate citation practices, though the self-citations are concentrated in the introduction. The manuscript's fit with the venue is adequate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a real, if modest, engineering contribution: a new Persian lip-reading dataset of 2,800 clips (7 command words, 20 speakers) recorded with the Surena-V's own camera, plus a working real-time deployment of an LSTM on the robot. That integration appears to be the first for Persian, and the dataset fills a genuine gap. The authors also do a fair job situating the work against prior Persian and other-language lip-reading datasets, and the methods, while standard, are described clearly enough to be reproduced if the data were released.\n\nThe soft spots are real and one is load-bearing. The paper never states that the train/validation/test split is speaker-independent, and with only 20 speakers a clip-level random split almost certainly puts the same speaker in both training and test sets. The reported 89% LSTM accuracy could then reflect the model memorizing speaker-specific appearance rather than learning generalizable lip dynamics. The conclusion's own call for \"a wider variety of speakers\" in future work concedes exactly this risk. The stress-test concern is correct, and the reader's conditional verdict is the right one. There is also a concrete text/table mismatch: the text says the LSTM hit 89% on the test set, but Table III reports only validation and training accuracy (95/93) and no test column at all. That needs fixing before the claim can be evaluated. Error bars and baselines would help, and releasing the data and code would turn this from an interesting demonstration into a reusable resource.\n\nThe indirect method's 52% is poor but above chance (7 classes), and the authors should say more about why it underperforms. Minor typos and grammar issues are cosmetic.\n\nWho benefits: researchers working on lip-reading for low-resource languages, especially Persian, and HRI groups wanting to add visual speech to humanoid robots. It deserves serious peer review because the contribution is new and the flaws are addressable. I would not desk-reject it, but I would send it back with a clear request: show speaker-independent results, reconcile the 89% figure with the table, and ideally share the dataset.","headline":"A genuinely new Persian lip-reading dataset and robot deployment, but the 89% accuracy claim is not trustworthy until the authors show the train/test split is speaker-independent.","tokens_in":7628,"tokens_out":2206,"would_cite":true,"duration_ms":20163,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a Persian lip-reading dataset and reports that an LSTM trained on it recognizes seven words with 89% accuracy, integrated into the Surena-V humanoid robot for real-time human-robot interaction.","keywords":["Persian lip reading","humanoid robot","Surena-V","LSTM","CNN","human-robot interaction","word-level lip-reading dataset"],"falsifier":"Train the same LSTM on clips from some speakers and test only on clips from speakers never used in training. If accuracy falls near the 14.3% chance level for seven words, the reported 89% was mostly speaker identity; if it stays high, the lip-reading claim is confirmed.","tokens_in":6597,"feed_emoji":"👄","tokens_out":10246,"duration_ms":83491,"temperature":0.7,"pith_summary":"The paper claims that a Persian lip-reading system built from a newly collected 2,800-clip dataset can recognize seven command words from silent video, and that an LSTM version reaches 89% test accuracy when mounted on the Surena-V humanoid robot. The paper argues this matters because robots in caregiving, customer service, or noisy settings need a visual channel for speech when audio is unreliable. It compares an indirect landmark-tracking pipeline against direct CNN and LSTM classifiers on raw frames, and reports that the direct LSTM wins by a wide margin. The demonstrated payoff is a real-time human-robot interaction loop in which the robot recognizes the command and responds.","feed_headline":"Persian lip-reading LSTM hits 89% on humanoid robot Surena-V","feed_subtitle":"A new 2,800-clip dataset lets Surena-V understand seven spoken Persian commands from silent video.","key_machinery":"The central object is the newly built Persian lip-reading dataset, and the mechanism that carries the top result is the direct LSTM architecture. In the indirect method, dlib landmarks 48–68 are tracked frame-by-frame into a (x, y, frame) array and classified by MobileNet, yielding 52%. In the direct method, raw face-cropped frames are fed as spatiotemporal sequences: a CNN branch with Conv2D, batch normalization, and pooling is paired with stacked LSTM layers through TimeDistributed flattening, producing a seven-class output. The LSTM's ability to model the temporal ordering of lip motion across the 20-frame, one-second window is what the paper credits for the 89% accuracy.","core_discovery":"On the paper's own terms, the discovery is that end-to-end deep learning on raw face-cropped video is the right route to Persian lip reading for humanoid command control. The LSTM, with time-distributed layers over 20-frame sequences, is reported to classify the seven words at 89% accuracy on the test set, outperforming the direct CNN (75%) and the indirect MobileNet-on-landmarks pipeline (52%). The paper also presents the dataset itself as a contribution: 20 speakers, 7 robot-relevant Persian words, 20 repetitions each, cropped to 300×300 and recorded at 20 fps with the robot's RGB-D camera. The real-world claim is that this model, once embedded in Surena-V, recognizes commands such as 'begir' (take) and 'benevis' (write) in real time and triggers the robot's predefined response.","pith_inferences":["A speaker-independent evaluation, holding out entire speakers rather than random clips, would reveal how much of the reported 89% is true lip reading versus speaker appearance.","The paper reports 89% test accuracy while its results table lists 95% validation accuracy on the final epoch; a reproducible release should state exactly which split each number comes from.","The same direct LSTM pipeline could be extended from seven isolated words to phrases or continuous commands by replacing the word classifier with a sequence-to-sequence or CTC head.","Because the dataset was recorded with the robot's own RGB-D camera, fine-tuning on live, varied head poses and distances is a cheap way to test whether the demo generalizes outside the laboratory."],"forward_implications":["If the 89% accuracy transfers to new speakers, humanoid robots can take spoken commands in crowded or noisy rooms without using audio.","The 2,800-clip dataset can serve as a benchmark for word-level Persian lip reading, giving later work a fixed vocabulary and recording setup to compare against.","The ranking of methods (LSTM 89%, CNN 75%, landmark-MobileNet 52%) points future work toward end-to-end spatiotemporal models rather than hand-tracked landmarks.","The Surena-V integration demonstrates a complete perception-to-action loop: camera capture, face cropping, LSTM command recognition, and robot response."],"supporting_citations":[{"why":"Prior Persian word-level lip-reading dataset; gives the 21.43% AV-Hubert baseline this work aims to beat.","marker":"[8]"},{"why":"Earlier Persian audiovisual dataset with HMM at 47.73%; defines the prior state for Persian lip reading.","marker":"[9]"},{"why":"Turkish lip-reading dataset evaluated with CNN, LSTM, and BGRU; supplies a cross-language comparison for the architecture choice.","marker":"[10]"},{"why":"Indonesian lip-reading work reaching 95% with LRCN; provides a strong small-vocabulary benchmark in another low-resource language.","marker":"[11]"},{"why":"Urdu lip-reading dataset with 62% LSTM accuracy; another low-resource example that motivates the direct LSTM design.","marker":"[12]"},{"why":"Arabic word-level lip-reading system with up to 82.86% accuracy; sets a close accuracy target the paper compares itself against.","marker":"[13]"},{"why":"Transfer learning for image classification; underlies the choice of VGG19 and ResNet in the indirect landmark pipeline.","marker":"[15]"}],"fun_headline_variants":["Surena-V lip-reads Persian commands at 89% accuracy","Persian lip-reading LSTM: 89% on Surena-V robot","Silent Persian commands now understood by Surena-V robot","Surena-V humanoid lip-reads 7 Persian words at 89%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the test clips show speakers the model did not already see during training; the paper does not state that its random split kept each speaker on only one side.","fun_headline_variants_meta":{"raw":{"variants":["Surena-V lip-reads Persian commands at 89% accuracy","Persian lip-reading LSTM: 89% on Surena-V robot","Silent Persian commands now understood by Surena-V robot","Surena-V humanoid lip-reads 7 Persian words at 89%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2515,"prompt_tokens":892,"completion_tokens":1623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1545}},"tokens_in":508,"tokens_out":1623,"duration_ms":10376,"temperature":1.0,"reasoning_tokens":1545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:49:12.854878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LSTM on clips from some speakers and test only on clips from speakers never used in training. If accuracy falls near the 14.3% chance level for seven words, the reported 89% was mostly speaker identity; if it stays high, the lip-reading claim is confirmed.","supporting_citations":[{"cited_title":"Word-level Persian Lipreading Dataset","cited_arxiv_id":"2304.04068","evidence_quote":"Prior Persian word-level lip-reading dataset; gives the 21.43% AV-Hubert baseline this work aims to beat."},{"cited_title":"”Persian Audi Visual Database of CV syllables” (2021)","cited_arxiv_id":null,"evidence_quote":"Earlier Persian audiovisual dataset with HMM at 47.73%; defines the prior state for Persian lip reading."},{"cited_title":"”Lip Reading Using Various Deep Learning Models with Visual Turkish Data” (2024)","cited_arxiv_id":null,"evidence_quote":"Turkish lip-reading dataset evaluated with CNN, LSTM, and BGRU; supplies a cross-language comparison for the architecture choice."},{"cited_title":"”Indonesian Lip-Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long-Term Recurrent Convolutional Network” (2024)","cited_arxiv_id":null,"evidence_quote":"Indonesian lip-reading work reaching 95% with LRCN; provides a strong small-vocabulary benchmark in another low-resource language."},{"cited_title":"Deep Learning for Lip Reading using Audio-Visual Information for Urdu Language","cited_arxiv_id":"1802.05521","evidence_quote":"Urdu lip-reading dataset with 62% LSTM accuracy; another low-resource example that motivates the direct LSTM design."},{"cited_title":"”Read my lips: Artificial intelligence word-level arabic lipreading system” (2022)","cited_arxiv_id":null,"evidence_quote":"Arabic word-level lip-reading system with up to 82.86% accuracy; sets a close accuracy target the paper compares itself against."},{"cited_title":"”Transfer learning for image classification.” 2018 second international conference on electronics, communication and aerospace technology (ICECA)","cited_arxiv_id":null,"evidence_quote":"Transfer learning for image classification; underlies the choice of VGG19 and ResNet in the indirect landmark pipeline."}],"review_version":1}