{"id":"500ab592-72b4-4a60-ad87-a2ca45f477fd","arxiv_id":"2505.17055","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A Vision Transformer fine-tuned on a new 41-class Palestinian Sign Language mathematical gesture dataset reportedly reaches 97.59% accuracy, but the dataset split and evaluation are contradictory.","lead":"Researchers built a new video dataset of 41 Palestinian Sign Language gestures for math words and fine-tuned a Vision Transformer to recognize them, reporting 97.59% accuracy. The work targets deaf and hard-of-hearing students in Palestine, but the evaluation methodology is internally inconsistent and the real-time claim is unverified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 97.59% test accuracy is arithmetically incompatible with the described 24/9/8 video split, since test videos' classes are absent from training; the central claim is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing flaw: the described data split cannot support the reported accuracy unless frames leak across the train/test boundary. My independent reading of Section 2.1 confirms this: 41 videos, each a distinct class, split into 24/9/8, means 8 test classes are unseen during training. This is a logical, not merely empirical, inconsistency. The model cannot classify never-seen classes at 97.59%. The only escape is frame-level leakage, which invalidates the metric equally. The central claim of the paper therefore collapses, and the verdict should remain REJECT. I am not raising a disagreement with consensus or a stylistic issue; this is an internal inconsistency in the reported evaluation. The dataset itself may be a real contribution, but the paper as written does not support its headline performance claim, and no amount of reframing as a dataset paper fixes the unsupported accuracy figure without redoing the evaluation. The concrete test would settle the matter by checking the actual split on the hosted dataset; if the dataset is unavailable or the split is undocumented, that further supports rejection because the claim cannot be verified.","tokens_in":6830,"tokens_out":1536,"duration_ms":17291,"concrete_test":"Download the Hugging Face dataset, extract the split assignment for every video/frame, and check whether any source video contributes frames to both train and test. Then retrain the ViT with a strict per-video split (no frame leakage) and evaluate only on test classes that were held out during training; report per-class accuracy and a confusion matrix. If the original split is confirmed to be video-level with disjoint classes, the reported 97.59% accuracy cannot be reproduced and the central claim is invalid.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is the 97.59% accuracy on the test set, but Section 2.1 describes a split that makes that number impossible. The dataset has 41 gesture videos, each representing a distinct class; these are split into 24 training videos, 9 validation videos, and 8 test videos. If each video is a distinct class, the 8 test classes are entirely absent from the training set. A ViT fine-tuned on only 24 classes cannot classify 8 unseen classes at 97.59% accuracy; expected performance on those classes would be near chance. If instead the split is frame-level, then frames from the same video appear in both training and test, which is a leakage that invalidates the reported metric. Either reading makes the accuracy unsupported. Additional internal inconsistencies (batch size 23 in the text vs 32 in Table 1; frame counts summing to 2895 vs the stated 2896) corroborate that the evaluation pipeline is not reported reliably. The paper also provides no per-class accuracy, no confusion matrix, and no released code, so the reported aggregate cannot be independently verified. The dataset may be a useful resource, but the headline performance claim fails under the described methodology.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a custom dataset of 41 Palestinian Sign Language (PSL) mathematical gesture classes, fine-tunes a ViT-base-patch16-224 model on it, and reports a classification accuracy of 97.59% along with similarly high precision, recall, and F1 scores. The authors claim this supports real-time mathematics-learning support for hard-of-hearing students and describe the dataset as the first domain-specific PSL mathematical-gesture resource. The methodology section describes data collection from 21 volunteers, a 24/9/8 video split into training, validation, and test sets, and a standard ViT fine-tuning pipeline.","tokens_in":7036,"tokens_out":3226,"duration_ms":34780,"significance":"If the empirical claims were sound, this paper would provide a useful first dataset for mathematical gestures in Palestinian Sign Language and demonstrate a practical classifier for an under-resourced sign language. The dataset idea and the focus on STEM education for deaf and hard-of-hearing students are genuinely valuable. However, the central performance claim is not supported by the described methodology: the data split as stated makes the reported accuracy impossible, and the manuscript provides no per-class results, no confusion matrix, and no released code. The contribution is therefore currently a dataset description with an invalid evaluation rather than a verified recognition result.","major_comments":[{"comment":"The data split described in §2.1 is internally contradictory and invalidates the headline accuracy. The paper states that the dataset consists of 41 gesture videos, each representing a distinct class, and that the split is 24 training videos, 9 validation videos, and 8 test videos. If this is a class-level split, the 8 test classes are entirely absent from the training set, so a model trained on only 24 classes cannot correctly classify the 8 unseen test classes, and a reported 97.59% accuracy is arithmetically impossible. If the split is instead frame-level within videos, then frames from the same video appear in both training and test sets, which is data leakage and makes the reported metric an in-sample fit rather than an out-of-sample measurement. Either reading leaves the central accuracy claim unsupported.","section":"§2.1, Table 2"},{"comment":"There are unexplained numerical inconsistencies in the reporting of the experimental setup. The text in §2.2 states a batch size of 23, while Table 1 lists a batch size of 32. The frame counts in §2.1 sum to 2316 + 289 + 290 = 2895, not the stated total of 2896. These discrepancies, together with the absence of released code, checkpoints, or per-class results, mean that the evaluation pipeline cannot be verified or reproduced from the manuscript.","section":"§2.1 and Table 1"},{"comment":"The paper reports only aggregate precision, recall, and F1 values and does not report the accuracy value in Table 2, despite the abstract claiming 97.59% accuracy. The conclusion in §4 explicitly lists confusion matrices and per-class accuracy as future work. Given the class-level split contradiction, the aggregate metrics alone cannot support the claimed effectiveness of the model on all 41 gesture classes.","section":"Table 2 and §4"},{"comment":"The algorithm treats the model as processing frames independently: it loops over frames, assigns features from each frame, and then classifies using only the features after the loop. No temporal aggregation or sequence modeling is specified, which contradicts the paper's claim that the model captures temporal dependencies of dynamic PSL gestures. This is a conceptual gap for a system described as recognizing sign-language movement rather than isolated static frames.","section":"Algorithm 1 and §2.2"}],"minor_comments":[{"comment":"The typeset equations for accuracy, precision, recall, and F1 are missing their denominators, and the definition of TN repeats the definition of TP ('correctly predicted positive instances') instead of stating negative instances.","section":"Equations (1)–(5)"},{"comment":"The abstract contains the phrase 'inclusive frosting digital integration,' which appears to be a typographical error; it should likely be 'inclusive digital integration.'","section":"Abstract"},{"comment":"The heading 'Compassion of sign language datasets' should read 'Comparison of sign language datasets.'","section":"Table 3 heading"},{"comment":"There are minor typographical errors, including 'board compatibility' for 'broad compatibility,' 'tracnsformer_encoder' for 'transformer_encoder' in Algorithm 1, and 'Tabel 2' for 'Table 2.'","section":"§2.2"},{"comment":"The text describes the volunteers as '21 proficient volunteers in PSL' but the significance of the number of videos per class and the relationship between volunteers and gesture classes is not explained; providing per-class participant counts would clarify dataset diversity.","section":"§2.1"}],"recommendation":"reject","confidential_remarks":"The central evaluation claim is internally inconsistent as described, and the numerical discrepancies make it impossible to treat the reported 97.59% accuracy as a reliable out-of-sample result. The dataset itself may be a useful contribution, but in its current form the paper's main performance claim cannot be accepted. Should the authors resubmit, they would need a properly described split, per-class metrics, released evaluation code or clear experimental artifacts, and a corrected presentation of the model's temporal handling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the genuine contribution: a new dataset of 41 mathematical gesture classes in Palestinian Sign Language, recorded by 21 volunteers and grounded in the PSL dictionary. That fills a real gap; prior work on PSL is scarce and no mathematical-gesture PSL dataset exists. The paper also makes a sensible qualitative comparison with alphabet-based datasets and frames the contribution as a domain-specific resource for STEM education.\n\nThe problem is the central accuracy claim. The paper says the 41 videos are each a distinct class, then splits into 24 train, 9 validation, and 8 test videos. If each video is a distinct class, the 8 test classes were never in training, so a classifier cannot reach 97.59% on them. If the split is frame-level, frames from the same video leak into both train and test, inflating the metric. Either way the reported number is not a valid out-of-sample result. The internal inconsistencies support this: batch size 23 in the text vs 32 in Table 1; frame counts 2316+289+290=2895 vs stated 2896; and the conclusion says future work will add per-class accuracy and confusion matrices, which belong in this paper.\n\nBecause the accuracy is the paper's main evidence, the claims about real-time promise and classroom readiness are not supported. The dataset itself may still be useful, but the paper does not document it with enough rigor (video lengths, resolution, per-class frame counts, participant details) to serve as a reliable resource reference yet.\n\nOn citation pattern: the paper cites relevant SLR literature and self-cites prior PSL linguistic work appropriately. No red flags there, though references are a bit stale at times.\n\nMy recommendation: this deserves a serious referee only if the evaluation is redone and the split is clarified. As written, I would not accept it for publication. The best path forward is major revision with a corrected evaluation protocol, per-class results, and code release. A short rebuttal pointing to the accessibility motivation does not fix an invalid test metric.","headline":"The new PSL math-gesture dataset is a genuine resource, but the paper's headline 97.59% accuracy is unsupported by the described split and should not be taken at face value.","tokens_in":7566,"tokens_out":2209,"would_cite":false,"duration_ms":21820,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the first mathematical-gesture dataset for Palestinian Sign Language and reports a fine-tuned Vision Transformer at 97.59% accuracy.","keywords":["Palestinian Sign Language","mathematical gestures","Vision Transformer","sign language recognition","assistive technology","low-resource language","STEM education","computer vision"],"falsifier":"Download the released 41-video dataset, reproduce the train/validation/test partition, and recompute accuracy after removing any frame whose source video appears in more than one split; if the 8 test classes are truly absent from training, a 41-class classifier cannot score above chance on them, so checking the class-to-video mapping and per-class predictions directly settles whether 97.59% is reproducible.","tokens_in":6628,"feed_emoji":"🤟","tokens_out":9457,"duration_ms":95189,"temperature":0.7,"pith_summary":"This paper tries to establish that a small, curated dataset can bring an under-served sign language into AI-driven education. It introduces 41 classes of mathematical gestures in Palestinian Sign Language, covering numbers, arithmetic operations, and geometric shapes, recorded by fluent signers and standardised against the official PSL dictionary. It then fine-tunes a Vision Transformer on frames from those videos and reports 97.59% accuracy, with precision and recall at the same level and F1-score at 97.58%. The sympathetic reading is that domain-specific datasets plus pretrained transformer backbones are enough to make real-time PSL math recognition feasible, reducing dependence on scarce human interpreters in Palestinian schools.","feed_headline":"A vision transformer reads Palestinian math signs at 97.59% accuracy","feed_subtitle":"A new 41-class dataset and fine-tuned model may bring real-time math tutoring to deaf students.","key_machinery":"The load-bearing mechanism is the Vision Transformer (ViT-base-patch16-224), which splits each 224x224 frame into 16x16 patches, adds positional embeddings, runs self-attention across patches, and finishes with an MLP head that outputs one of 41 gesture classes. It is fine-tuned from pretrained weights on 2,896 frames with horizontal flipping, rotation, color jitter, resizing, and normalization as augmentation. The dataset is the companion mechanism: 41 classes spanning numbers, operations, and shapes, each class represented by one video, with frames extracted to capture full sign movement; the paper states that PSL grammar, especially non-manual markers, guided the choice of which gestures to include.","core_discovery":"The central claim is that a fine-tuned Vision Transformer can classify 41 mathematical signs of Palestinian Sign Language with a reported accuracy of 97.59%, precision of 97.59%, recall of 97.59%, and F1-score of 97.58%, using a new custom dataset of 2,896 frames. The discovery is the pairing of resource and model: the dataset is, by the authors' account, the first domain-specific PSL collection for mathematics, built from a standardised dictionary and recorded by 21 proficient signers, and the ViT is a general image-classification backbone rather than a bespoke sign-language architecture. The authors' claim is that this pairing closes a concrete gap, since PSL has been left out of the sign-language-recognition literature that focuses on ASL, ISL, and ArSL, and that the high metrics point to deployable AI tutoring for hard-of-hearing students in STEM classrooms.","pith_inferences":["The paper stops at classification; a natural extension the authors do not carry out is an end-to-end classroom test that measures latency, acceptance by deaf students, and whether real-time feedback improves math learning.","A static-frame ViT is a deliberate simplification; if temporal dynamics turn out to matter for PSL math signs, adding a recurrent or video-transformer layer should raise accuracy further, which is a testable comparison on the same dataset.","A signer-independent split, training and testing on disjoint sets of people, would be a stronger generalization test than the current video-level split, and is a concrete way to check whether the 97.59% accuracy holds beyond the 21 volunteers who recorded the data.","The paper's own future-work list, confusion matrices and per-class accuracy, is the minimal diagnostic for identifying confusable signs such as similar number hand shapes; until such numbers are reported, the aggregate 97.59% says little about which signs drive the result."],"forward_implications":["Palestinian Sign Language gains a first domain-specific mathematical dataset and a reported classification baseline, giving later work something concrete to extend or contest.","The reported metrics support the feasibility of an AI layer for math education that responds to signed numbers, operations, and shapes in real time.","The fine-tuning recipe, pretrained transformer, small frame set, and standard augmentation, is light enough to be repeated for other low-resource sign languages with similar domain vocabularies.","If the dataset is widely used, future systems can combine this classifier with text-to-speech or on-screen math symbols, effectively translating PSL math signs into the written notation that hearing teachers and standard curricula expect."],"supporting_citations":[{"why":"Supplies the PSL-specific grammatical and non-manual marker patterns that guided which gestures to include in the dataset.","marker":"[26]"},{"why":"Provides the prior ViT-based sign-language result and accuracy benchmark that the paper extends to PSL.","marker":"[21]"},{"why":"Shows vision transformers outperforming CNNs on Arabic sign language, justifying the ViT choice for a similar under-resourced language.","marker":"[22]"},{"why":"Gives the video-transformer baseline for isolated sign recognition that the paper compares conceptually with its per-frame ViT approach.","marker":"[23]"},{"why":"Documents the low enrolment of deaf and hard-of-hearing students in Palestinian public schools, motivating the educational application.","marker":"[5]"}],"fun_headline_variants":["97.59%: ViT classifies 41 Palestinian math signs from new dataset","New PSL math dataset lifts ViT to 97.59% accuracy on 41 signs","ViT recognizes 41 Palestinian sign language math gestures at 97.59%","AI math tutor for deaf students: ViT reads PSL at 97.59%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 97.59% figure stands on the assumption that the test videos are genuine held-out samples of the 41 trained classes, with no frames from test videos leaking into training or validation, and that the reported split actually allows a 41-class classifier to be evaluated.","fun_headline_variants_meta":{"raw":{"variants":["97.59%: ViT classifies 41 Palestinian math signs from new dataset","New PSL math dataset lifts ViT to 97.59% accuracy on 41 signs","ViT recognizes 41 Palestinian sign language math gestures at 97.59%","AI math tutor for deaf students: ViT reads PSL at 97.59%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2175,"prompt_tokens":909,"completion_tokens":1266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1173}},"tokens_in":525,"tokens_out":1266,"duration_ms":10547,"temperature":1.0,"reasoning_tokens":1173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:50:20.517235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download the released 41-video dataset, reproduce the train/validation/test partition, and recompute accuracy after removing any frame whose source video appears in more than one split; if the 8 test classes are truly absent from training, a 41-class classifier cannot score above chance on them, so checking the class-to-video mapping and per-class predictions directly settles whether 97.59% is reproducible.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PSL-specific grammatical and non-manual marker patterns that guided which gestures to include in the dataset."},{"cited_title":"R., Bhatt, C","cited_arxiv_id":null,"evidence_quote":"Provides the prior ViT-based sign-language result and accuracy benchmark that the paper extends to PSL."},{"cited_title":"F., & Maghari, A","cited_arxiv_id":null,"evidence_quote":"Shows vision transformers outperforming CNNs on Arabic sign language, justifying the ViT choice for a similar under-resourced language."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the low enrolment of deaf and hard-of-hearing students in Palestinian public schools, motivating the educational application."}],"review_version":1}