{"id":"7bb0cf8f-308e-4bfd-8171-e584c33eb010","arxiv_id":"2505.07890","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A compact transformer using Mediapipe skeletal landmarks reports 90.67% accuracy on the 226-word AUTSL Turkish Sign Language recognition benchmark.","lead":"TSLFormer is a compact transformer that recognizes Turkish Sign Language words using only hand and body joint positions, without processing video pixels. It reports 90.67% accuracy on the AUTSL word benchmark, suggesting that a lightweight landmark-based model could power real-time assistive tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 90.67% accuracy may reflect signer identity rather than signer-independent generalization; the split is never stated to be signer-disjoint.","rationale":"The reader's weakest assumption—that the 80/20 split may not be signer-independent—is exactly the load-bearing issue for this paper. The manuscript describes the split only in terms of class distribution, never signer identity, and the central conclusion is about feasibility for real-world deployment where users are unseen. The paper's own citation of the AUTSL baseline dropping from about 95.95% to about 62% when tested on unseen signers makes the distinction concrete and shows the authors are aware of it. The reported 90.67% is plausible for a signer-dependent random split, but it does not establish the claimed generalization unless the test signers are disjoint from training signers. The frame-sampling inconsistency (16 frames in the introduction and discussion versus 15 frames per second in Section III.C) is a secondary reproducibility issue but does not affect the logical validity of the central claim as directly. I agree with the reader's conditional assessment: keep the verdict conditional until a signer-independent evaluation is provided or the current split is shown to be signer-disjoint.","tokens_in":10936,"tokens_out":3715,"duration_ms":38847,"concrete_test":"Retrain and evaluate TSLFormer using an official AUTSL signer-independent protocol, e.g., disjoint sets of signers for training/validation/test as used in the ChaLearn challenge (reference [26]), keeping the architecture, Mediapipe preprocessing, frame sampling, and hyperparameters identical. If accuracy on held-out signers falls substantially below 90.67% (toward the roughly 62% range cited in Section III.A), the headline claim must be qualified as signer-dependent; if it remains near 90.67%, the concern is resolved. A cheaper preliminary check is to compute the overlap of signer identities between the current training and test splits and to report per-signer test accuracy restricted to signers absent from training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that 90.67% test accuracy shows sign language recognition can be carried out using only joint-based features—depends on the test set measuring generalization to unseen signers. The paper never states that it does. Section III.C describes only an 80/20 split with \"similar distribution of classes\" and 4-fold cross-validation on the training data; signer identity is not mentioned as a split criterion. Because AUTSL contains multiple videos per signer, a random class-stratified split can place videos from the same signer in both training and test sets. The model could then exploit signer-specific motion and appearance patterns, so 90.67% would measure familiar signer identity rather than signer-independent recognition. The paper itself notes in Section III.A that the AUTSL baseline reaches about 95.95% under favorable conditions but drops to around 62% for users not in the training set, demonstrating that this distinction is large. The deployment scenario invoked throughout the paper—real-time mobile assistive communication for hearing-impaired individuals—requires handling new signers. The comparison in Table III is also only meaningful if all methods are evaluated under the same signer-independent protocol, such as the ChaLearn/AUTSL challenge referenced in [26]. Without a signer-disjoint evaluation, the reported accuracy does not support the stated generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TSLFormer, a lightweight transformer-based model for isolated Turkish Sign Language (TİD) word recognition, using only 3D skeletal landmarks extracted with Mediapipe (hand and upper-body joints) instead of raw RGB or depth video. On the AUTSL dataset, the authors report a 4-fold cross-validation accuracy of 92.85% and a final held-out test accuracy of 90.67%, and compare this with prior AUTSL results, arguing that the landmark-only input offers a favorable accuracy/efficiency trade-off for real-time and mobile assistive systems. The manuscript also describes a real-time webcam demonstration application.","tokens_in":11178,"tokens_out":2576,"duration_ms":28672,"significance":"If the 90.67% accuracy genuinely reflects signer-independent generalization, the paper would provide a useful lightweight baseline for isolated sign recognition, and its explicit comparison of landmark-only input against multi-modal RGB/skeleton/face systems is a valuable design point. The paper is transparent about its preprocessing pipeline, reports cross-validation results, includes a confusion matrix, and acknowledges the limitations of skeletal landmarks (e.g., difficulty with fine finger articulation). However, the central empirical claim is currently not supported because the evaluation protocol—-specifically, whether the test split is signer-disjoint—-is not specified, and the comparison in Table III appears to mix evaluation protocols. The reproducibility-oriented choices (public dataset, standard architecture, documented hyperparameters) are strengths that would allow the evaluation issue to be corrected in a revision.","major_comments":[{"comment":"The description of the 80/20 train/test split states only that both subsets have a similar class distribution; it never states that the split is signer-independent. Since AUTSL contains multiple videos per signer (43 signers, over 36,000 videos), a random class-stratified split will almost certainly place videos from the same signer in both training and test sets, allowing the model to exploit signer-specific motion and appearance patterns. The paper itself notes in Section III.A that the AUTSL baseline drops from about 95.95% to about 62% when tested on users not in the training set, which demonstrates how large this effect can be. Therefore, the central claim in Section VI that 90.67% test accuracy shows successful sign recognition using joint-based features is not yet supported. The authors should re-run the evaluation with a signer-disjoint split (e.g., following the ChaLearn/AUTSL protocol referenced in [26]) and report the signer-independent accuracy.","section":"III.C (two-stage validation)"},{"comment":"The comparison in Table III mixes incompatible evaluation protocols. The cited methods SAM-SLR [23], S3D [24], TD-SL [25], USTC-SLR [31], Jalba [30], and VLE-Transformer [27] were evaluated under the ChaLearn 2021 signer-independent challenge protocol, whereas the proposed TSLFormer result is, based on the description in Section III.C, obtained on a non-signer-disjoint random split. Under these conditions, the table does not support the statement that TSLFormer achieves 'competitive performance' or that it is 'a valid and effective alternative' to those methods. The authors should either evaluate TSLFormer under the same signer-independent protocol or clearly label the protocols and restrict the comparison to methods evaluated under the same conditions.","section":"Table III (comparison with existing work)"},{"comment":"The final test performance is reported as a single point estimate (90.67% accuracy, recall, F1) with no variance, no per-fold test results, no confidence intervals, and no class-wise statistics. Because the test set is a single random split, the reported numbers could vary substantially with the split, and the equality of accuracy, recall, and F1 suggests that the averaging scheme is not explained. The authors should report the standard deviation across multiple seeds or splits, and clarify how recall and F1 are aggregated (micro vs. macro) given the 226-class setting.","section":"IV.A (final test results)"}],"minor_comments":[{"comment":"The abstract states '227 different words' while the main text (Sections I and III.A) consistently says '226 different words'; this inconsistency should be corrected.","section":"Abstract and I (dataset statistics)"},{"comment":"The introduction says '16 evenly sampled frames' and Section V says '16 anchor frames,' but Section III.C says frames were sampled at '15 frames per second.' These descriptions are inconsistent regarding the number vs. rate of sampled frames; the authors should clarify the actual temporal sampling procedure.","section":"I and V (temporal sampling)"},{"comment":"The paper alternates between 'TSL' and 'TİD' for Turkish Sign Language, and between 'Turkish Sign Language' and 'Turkish Sign Language (TİD)'; a single consistent abbreviation should be used.","section":"Throughout (terminology)"},{"comment":"The real-time demo is described qualitatively without quantitative latency or accuracy measurements on live input; adding even basic metrics would strengthen the claim of suitability for real-time use.","section":"IV.B (demo application)"},{"comment":"The statement 'Since there are no loops or recurrent structures in RNNs or LSTMs' is imprecise: RNNs and LSTMs do have recurrent loops. The sentence appears to mean that transformers have no recurrent structure, and should be reworded.","section":"III.C (model description)"}],"recommendation":"major_revision","confidential_remarks":"The core problem is the undefined test protocol: the paper does not establish signer-independent evaluation, which is essential for the stated deployment scenario and for the comparison in Table III. The authors are likely able to address this within a revision by re-running experiments with a signer-disjoint split, but without that the reported 90.67% should not be compared with ChaLearn-protocol results. I would not recommend rejection, as the architecture and preprocessing pipeline are clearly presented and reproducible, but the revision must contain a corrected evaluation to be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TSLFormer is a straightforward engineering contribution, not a breakthrough. The specific combination—Mediapipe hand/pose landmarks, a compact two-layer transformer, and the AUTSL benchmark—is new, and the reported 90.67% test accuracy is plausible if the evaluation is what it seems. The paper is clearly written, the pipeline is easy to reproduce, and the authors are honest about the loss of finger-level detail. The webcam demo is a nice addition.\n\nThe load-bearing problem is the evaluation protocol. The paper never states that the 80/20 split is signer-independent. AUTSL has 43 signers with many videos each, so a random class-stratified split will typically put the same signers in both training and test. The authors themselves note that the AUTSL baseline drops from 95.95% to about 62% on unseen signers. Without a signer-disjoint test set, 90.67% likely reflects familiar signer identity, not the generalization to new users that the introduction and abstract promise. The phrase “never seen during training” is ambiguous: it excludes the exact videos, but not necessarily the signers. Table III compounds the problem because most cited numbers come from the signer-independent ChaLearn challenge; comparing those to a possibly signer-dependent split is misleading.\n\nThe smaller issues are real but minor. The number of sampled frames is 16 in the abstract, 15 in Section III.C, and 16 again in the Discussion. No error bars or per-fold results are given. No code or data is provided. The evaluation would be much stronger with one sentence confirming signer disjointness (or reporting the overlap) and with a rerun under the same protocol as the comparison methods. Those fixes are straightforward.\n\nWho will get value: people building lightweight SLR systems, especially on AUTSL, and anyone comparing landmark-only methods. It’s a useful data point, not a result that changes the field.\n\nI’d send this to peer review, not desk reject. The engineering is sound and the presentation is honest, but the main claim needs to be revalidated under a signer-independent protocol before it can support the real-time assistive-use conclusion. With that fix it would be a solid applied paper.","headline":"A modest but honest engineering result on lightweight Turkish sign language recognition, with the main caveat that the test split may not be signer-independent.","tokens_in":11741,"tokens_out":3389,"would_cite":false,"duration_ms":33351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hand-joint landmarks alone hit 90.67% on Turkish signs","keywords":["Turkish Sign Language recognition","transformer","skeletal landmarks","Mediapipe","AUTSL dataset","self-attention","lightweight model","gesture classification"],"falsifier":"Hold out all videos of a specific subset of AUTSL signers during training and test only on those signers; if TSLFormer's accuracy falls well below 90.67%—for instance toward the roughly 62% signer-independent level reported for AUTSL—the claim that joint-only features make practical sign recognition feasible for unseen users would be contradicted.","tokens_in":10747,"feed_emoji":"🤟","tokens_out":6737,"duration_ms":57913,"temperature":0.7,"pith_summary":"TSLFormer is a lightweight transformer that recognizes isolated Turkish Sign Language words using only 3D skeletal joint landmarks extracted by Mediapipe, rather than raw RGB or depth video. Trained and tested on the AUTSL dataset's 226-word, 43-signer collection of over 36,000 videos, the model reports 90.67% test accuracy with a 4-fold cross-validated average of 92.85%. The paper argues this demonstrates that joint-based features alone are sufficient for word-level sign recognition, cutting per-frame input from hundreds of thousands of pixels to 144 coordinates. That matters because it makes real-time, mobile, and embedded assistive communication systems for hearing-impaired users computationally feasible.","feed_headline":"Hand-joint landmarks alone hit 90.67% on Turkish signs","feed_subtitle":"A tiny transformer on 3D skeletal data reads 226 Turkish signs without raw video, enabling real-time assistive devices.","key_machinery":"The central mechanism is the transformer encoder's multi-head self-attention over the frame sequence, which lets every frame attend to every other frame and thereby captures the temporal co-occurrence of joint positions as a sign unfolds. Each frame is a 144-dimensional vector of XYZ coordinates from Mediapipe's Pose and Hands models; an embedding layer projects it to 512 dimensions, positional encoding adds order information, two encoder layers with layer normalization and dropout process the sequence, mean pooling summarizes it, and a linear layer outputs one of 226 word classes. The pipeline treats the joint-coordinate sequence as a sentence-like linguistic unit, with an EOS token marking the end of the gesture.","core_discovery":"On its own terms, the paper establishes that a two-layer transformer encoder with 512 hidden units and four attention heads, fed 16 evenly sampled frames of 48 upper-body and hand landmarks (144 XYZ features per frame) with zero padding and an EOS token, classifies 226 Turkish sign words with 90.67% accuracy and 90.67% F1. The central claim is that this accuracy shows sign language recognition can be carried out successfully using only joint-based features, making the efficiency/accuracy trade-off attractive for real-time and low-resource settings. The paper positions TSLFormer as a simpler alternative to multi-modal systems that exceed 98% accuracy by combining RGB, depth, skeleton, and face cues, while needing far less computation.","pith_inferences":["A direct signer-independent split of AUTSL would reveal how much of the reported 90.67% depends on signer identity; if accuracy drops toward the roughly 62% signer-independent baseline reported for AUTSL, the real-world generalization claim would need tempering.","The same landmark-plus-transformer recipe could plausibly transfer to other sign languages with comparable vocabulary sizes, since the method is dataset-agnostic, but that transfer remains untested.","Adding lightweight hand-shape or finger-spread descriptors to the 144-dimensional input might recover some of the finger-articulation detail the paper identifies as lost, potentially closing the gap to multi-modal models without full RGB processing.","The paper's observation that the model is confidently wrong on some errors suggests that a calibration or uncertainty-estimation layer would make the demo more trustworthy for assistive use."],"forward_implications":["Per-frame input drops from about 786,432 RGB pixel values to 144 landmark coordinates, making inference cheap enough for real-time webcam-based recognition.","The near-diagonal confusion matrix shows most misclassifications occur between visually similar signs, indicating the joint representation retains discriminative motion cues.","Because the model needs only Mediapipe landmarks, a working demo can run on a laptop webcam with motion-triggered recording and automatic segmentation.","The reported efficiency/accuracy balance positions TSLFormer as a candidate for embedded and mobile assistive devices, trading a few points of accuracy against much lower computational demand.","The paper's stated future work includes sentence-level recognition and evaluation across signer groups and datasets, which would test how well the approach transfers to new signers."],"supporting_citations":[{"why":"Supplies the 226-word, 43-signer AUTSL dataset and its 49.22% RGB baseline, against which TSLFormer's 90.67% is measured.","marker":"[2]"},{"why":"Supplies the Mediapipe Pose and Hands landmark extraction that reduces each frame to 144 XYZ joint features.","marker":"[3]"},{"why":"Supplies the transformer encoder with multi-head self-attention that the model uses to model temporal co-occurrence of joints.","marker":"[18]"},{"why":"Supplies the 4-fold cross-validation procedure used to estimate the 92.85% validation accuracy.","marker":"[22]"}],"fun_headline_variants":["90.67% sign accuracy with joints only, no video","Skeleton-only transformer reads 226 Turkish signs at 90.67%","Lightweight TSLFormer: skeletal landmarks hit 90.67%","No RGB, just joints: 90.67% Turkish sign recognition","TSLFormer: joint points alone deliver 90.67% on Turkish signs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 90.67% assumes the 80/20 split tests generalization to signers the model has never seen, but the paper does not state whether signers appear in both training and test sets.","fun_headline_variants_meta":{"raw":{"variants":["90.67% sign accuracy with joints only, no video","Skeleton-only transformer reads 226 Turkish signs at 90.67%","Lightweight TSLFormer: skeletal landmarks hit 90.67%","No RGB, just joints: 90.67% Turkish sign recognition","TSLFormer: joint points alone deliver 90.67% on Turkish signs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1652,"prompt_tokens":882,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":673}},"tokens_in":498,"tokens_out":770,"duration_ms":6917,"temperature":1.0,"reasoning_tokens":673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:27:29.122704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out all videos of a specific subset of AUTSL signers during training and test only on those signers; if TSLFormer's accuracy falls well below 90.67%—for instance toward the roughly 62% signer-independent level reported for AUTSL—the claim that joint-only features make practical sign recognition feasible for unseen users would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 4-fold cross-validation procedure used to estimate the 92.85% validation accuracy."}],"review_version":1}