REVIEW 3 major objections 5 minor 32 references
TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Hand-joint landmarks alone hit 90.67% on Turkish signs
desk verdict A modest but honest engineering result on lightweight Turkish sign language recognition, with the main caveat that the test split may not be signer-independent. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the transformer encoder's multi-head self-attention over the frame sequence, which lets every frame attend to every other frame and thereby captures the temporal co-occurrence of joint positions as a sign unfolds. Each frame is a 144-dimensional vector of XYZ coordinates from Mediapipe's Pose and Hands models; an embedding layer projects it to 512 dimensions, positional encoding adds order information, two encoder layers with layer normalization and dropout process the sequence, mean pooling summarizes it, and a linear layer outputs one of 226 word classes. The pipeline treats the joint-coordinate sequence as a sentence-like linguistic unit, with an EOS token marking the end of the gesture.
What would settle it
Hold out all videos of a specific subset of AUTSL signers during training and test only on those signers; if TSLFormer's accuracy falls well below 90.67%—for instance toward the roughly 62% signer-independent level reported for AUTSL—the claim that joint-only features make practical sign recognition feasible for unseen users would be contradicted.
Extended reading notes
Core claim
On its own terms, the paper establishes that a two-layer transformer encoder with 512 hidden units and four attention heads, fed 16 evenly sampled frames of 48 upper-body and hand landmarks (144 XYZ features per frame) with zero padding and an EOS token, classifies 226 Turkish sign words with 90.67% accuracy and 90.67% F1. The central claim is that this accuracy shows sign language recognition can be carried out successfully using only joint-based features, making the efficiency/accuracy trade-off attractive for real-time and low-resource settings. The paper positions TSLFormer as a simpler alternative to multi-modal systems that exceed 98% accuracy by combining RGB, depth, skeleton, and face cues, while needing far less computation.
Load-bearing premise
The reported 90.67% assumes the 80/20 split tests generalization to signers the model has never seen, but the paper does not state whether signers appear in both training and test sets.
Editorial extensions
If this is right
- Per-frame input drops from about 786,432 RGB pixel values to 144 landmark coordinates, making inference cheap enough for real-time webcam-based recognition.
- The near-diagonal confusion matrix shows most misclassifications occur between visually similar signs, indicating the joint representation retains discriminative motion cues.
- Because the model needs only Mediapipe landmarks, a working demo can run on a laptop webcam with motion-triggered recording and automatic segmentation.
- The reported efficiency/accuracy balance positions TSLFormer as a candidate for embedded and mobile assistive devices, trading a few points of accuracy against much lower computational demand.
- The paper's stated future work includes sentence-level recognition and evaluation across signer groups and datasets, which would test how well the approach transfers to new signers.
Reading between the lines
- A direct signer-independent split of AUTSL would reveal how much of the reported 90.67% depends on signer identity; if accuracy drops toward the roughly 62% signer-independent baseline reported for AUTSL, the real-world generalization claim would need tempering.
- The same landmark-plus-transformer recipe could plausibly transfer to other sign languages with comparable vocabulary sizes, since the method is dataset-agnostic, but that transfer remains untested.
- Adding lightweight hand-shape or finger-spread descriptors to the 144-dimensional input might recover some of the finger-articulation detail the paper identifies as lost, potentially closing the gap to multi-modal models without full RGB processing.
- The paper's observation that the model is confidently wrong on some errors suggests that a calibration or uncertainty-estimation layer would make the demo more trustworthy for assistive use.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TSLFormer, a lightweight transformer-based model for isolated Turkish Sign Language (TİD) word recognition, using only 3D skeletal landmarks extracted with Mediapipe (hand and upper-body joints) instead of raw RGB or depth video. On the AUTSL dataset, the authors report a 4-fold cross-validation accuracy of 92.85% and a final held-out test accuracy of 90.67%, and compare this with prior AUTSL results, arguing that the landmark-only input offers a favorable accuracy/efficiency trade-off for real-time and mobile assistive systems. The manuscript also describes a real-time webcam demonstration application.
Significance. If the 90.67% accuracy genuinely reflects signer-independent generalization, the paper would provide a useful lightweight baseline for isolated sign recognition, and its explicit comparison of landmark-only input against multi-modal RGB/skeleton/face systems is a valuable design point. The paper is transparent about its preprocessing pipeline, reports cross-validation results, includes a confusion matrix, and acknowledges the limitations of skeletal landmarks (e.g., difficulty with fine finger articulation). However, the central empirical claim is currently not supported because the evaluation protocol—-specifically, whether the test split is signer-disjoint—-is not specified, and the comparison in Table III appears to mix evaluation protocols. The reproducibility-oriented choices (public dataset, standard architecture, documented hyperparameters) are strengths that would allow the evaluation issue to be corrected in a revision.
major comments (3)
- [III.C (two-stage validation)] The description of the 80/20 train/test split states only that both subsets have a similar class distribution; it never states that the split is signer-independent. Since AUTSL contains multiple videos per signer (43 signers, over 36,000 videos), a random class-stratified split will almost certainly place videos from the same signer in both training and test sets, allowing the model to exploit signer-specific motion and appearance patterns. The paper itself notes in Section III.A that the AUTSL baseline drops from about 95.95% to about 62% when tested on users not in the training set, which demonstrates how large this effect can be. Therefore, the central claim in Section VI that 90.67% test accuracy shows successful sign recognition using joint-based features is not yet supported. The authors should re-run the evaluation with a signer-disjoint split (e.g., following the ChaLearn/AUTSL protocol referenced in [26]) and report the signer-independent accuracy.
- [Table III (comparison with existing work)] The comparison in Table III mixes incompatible evaluation protocols. The cited methods SAM-SLR [23], S3D [24], TD-SL [25], USTC-SLR [31], Jalba [30], and VLE-Transformer [27] were evaluated under the ChaLearn 2021 signer-independent challenge protocol, whereas the proposed TSLFormer result is, based on the description in Section III.C, obtained on a non-signer-disjoint random split. Under these conditions, the table does not support the statement that TSLFormer achieves 'competitive performance' or that it is 'a valid and effective alternative' to those methods. The authors should either evaluate TSLFormer under the same signer-independent protocol or clearly label the protocols and restrict the comparison to methods evaluated under the same conditions.
- [IV.A (final test results)] The final test performance is reported as a single point estimate (90.67% accuracy, recall, F1) with no variance, no per-fold test results, no confidence intervals, and no class-wise statistics. Because the test set is a single random split, the reported numbers could vary substantially with the split, and the equality of accuracy, recall, and F1 suggests that the averaging scheme is not explained. The authors should report the standard deviation across multiple seeds or splits, and clarify how recall and F1 are aggregated (micro vs. macro) given the 226-class setting.
minor comments (5)
- [Abstract and I (dataset statistics)] The abstract states '227 different words' while the main text (Sections I and III.A) consistently says '226 different words'; this inconsistency should be corrected.
- [I and V (temporal sampling)] The introduction says '16 evenly sampled frames' and Section V says '16 anchor frames,' but Section III.C says frames were sampled at '15 frames per second.' These descriptions are inconsistent regarding the number vs. rate of sampled frames; the authors should clarify the actual temporal sampling procedure.
- [Throughout (terminology)] The paper alternates between 'TSL' and 'TİD' for Turkish Sign Language, and between 'Turkish Sign Language' and 'Turkish Sign Language (TİD)'; a single consistent abbreviation should be used.
- [IV.B (demo application)] The real-time demo is described qualitatively without quantitative latency or accuracy measurements on live input; adding even basic metrics would strengthen the claim of suitability for real-time use.
- [III.C (model description)] The statement 'Since there are no loops or recurrent structures in RNNs or LSTMs' is imprecise: RNNs and LSTMs do have recurrent loops. The sentence appears to mean that transformers have no recurrent structure, and should be reworded.
Circularity Check
No circular derivation; the reported accuracy is an empirical holdout result, and the main residual concern (signer identity in the split) is a generalization-risk issue, not a circularity.
full rationale
TSLFormer is an empirical machine-learning study rather than a derivation chain. Mediapipe landmarks are produced by an external library, a transformer classifier is trained with cross-entropy on an 80% training split, and the reported 90.67% accuracy is measured on a separate 20% test set, with additional 4-fold cross-validation on the training data. No equation or construction step makes the predicted quantity equal to a fitted input: the class distribution is stratified, but the accuracy itself is not defined by any training-set statistic, and no parameter fitted on the test set is later relabeled as a prediction. The paper's comparisons in Table III are external baselines, not evidence that TSLFormer's result is forced by self-citation or by definition. The paper's own caveats, such as the landmark-resolution trade-off in Section III.B and the miscalibration observation in Section IV.B, are empirical limitations rather than definitional reductions. The most serious concern is that the 80/20 split is described only as having 'a similar distribution of classes' and signer identity is never stated as a split criterion, so the 90.67% figure may partly reflect familiar signer identity instead of signer-independent generalization; the paper even notes that the AUTSL baseline drops to around 62% for users not in the training set. That concern, however, is an evaluation-protocol and external-validity risk, not circularity: the accuracy is not equivalent by construction to the training labels or to a self-citation. No circular step meets the required evidence bar of quoting a specific reduction, so the honest finding is no significant circularity (score 0).
Assumptions & free parameters
free parameters (6)
- number of sampled frames per video =
inconsistent: 16 in introduction, 15 per second in method
- embedding dimension =
512
- number of encoder layers =
2
- number of attention heads =
4
- dropout rate =
0.2
- learning rate =
1e-4 with scheduler
assumptions (3)
- domain assumption Mediapipe landmark coordinates are a sufficiently rich representation of the 226 TSL words to support the reported accuracy.
- domain assumption The 80/20 train/test split reflects deployment conditions, including generalization to new signers.
- domain assumption AUTSL dataset labels and video contents are correct and complete.
Cite this review
Pith. "Pith review of TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks." pith.science (2026). https://pith.science/paper/5EMQNNEV
@misc{pith2026250507890,
author = {Pith},
title = {Pith review of: TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks},
year = {2026},
howpublished = {\url{https://pith.science/paper/5EMQNNEV}},
note = {Machine review of arXiv:2505.07890}
}
read the original abstract
This study presents TSLFormer, a light and robust word-level Turkish Sign Language (TSL) recognition model that treats sign gestures as ordered, string-like language. Instead of using raw RGB or depth videos, our method only works with 3D joint positions - articulation points - extracted using Google's Mediapipe library, which focuses on the hand and torso skeletal locations. This creates efficient input dimensionality reduction while preserving important semantic gesture information. Our approach revisits sign language recognition as sequence-to-sequence translation, inspired by the linguistic nature of sign languages and the success of transformers in natural language processing. Since TSLFormer uses the self-attention mechanism, it effectively captures temporal co-occurrence within gesture sequences and highlights meaningful motion patterns as words unfold. Evaluated on the AUTSL dataset with over 36,000 samples and 227 different words, TSLFormer achieves competitive performance with minimal computational cost. These results show that joint-based input is sufficient for enabling real-time, mobile, and assistive communication systems for hearing-impaired individuals.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[26]
Ozge Mercanoglu Sincan and Julio C. S. Jacques Junior and Sergio Escalera and Hacer Yalim Keles, ”ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research,” arXiv preprint arXiv:2105.05066, 2021. [Online]. Available: https://arxiv.org/abs/2105.05066
work page Pith review arXiv 2021
- [23]
-
[25]
Sarhan, Noha and Wilms, Christian and Closius, Vanessa and Brefeld, Ulf and Frintrop, Simone, ”Hands in Focus: Sign Lan- guage Recognition Via Top-Down Attention,” in2023 IEEE Interna- tional Conference on Image Processing (ICIP), 2023 pp. 2555-2559. doi:10.1109/ICIP49359.2023.10222729
arXiv 2023
- [31]
-
[30]
Alba, Juan and Docio, Laura and Banga, Eva Rueda and Vazquez, Marta and Torres, Sergio and Perez, Antonio, ”Fact Sheet: CVPR 2021 ChaLearn Looking at People Large Scale Signer Independent Isolated SLR Challenge,” 2021. [Online]. Available: https://tinyurl.com/ jalba2021fact Accessed: March 2025
work page 2021
- [1]
-
[2]
Sincan, Ozge Mercanoglu and Keles, Hacer Yalim, ”AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Base- line Methods,”IEEE Access, vol. 8, pp. 181340–181355, 2020. doi:10.1109/access.2020.3028072
arXiv 2020
-
[3]
Camillo Lugaresi and Jiuqiang Tang and Hadon Nash and Chris Mc- Clanahan and Esha Uboweja and Michael Hays and Fan Zhang and Chuo-Ling Chang and Ming Guang Yong and Juhyun Lee and Wan- Teh Chang and Wei Hua and Manfred Georg and Matthias Grundmann, ”MediaPipe: A Framework for Building Perception Pipelines,” arXiv preprint arXiv:1906.08172, 2019. [Online]....
arXiv 1906
Show all 32 references
-
[4]
[Online]
Anurag Arnab and Mostafa Dehghani and Georg Heigold and Chen Sun and Mario Lu ˇci´c and Cordelia Schmid, ”ViViT: A Video Vision Transformer,” arXiv preprint arXiv:2103.15691, 2021. [Online]. Avail- able: https://arxiv.org/abs/2103.15691
2021 arXiv
-
[5]
and Juang, B., ”An introduction to hidden Markov models,”IEEE ASSP Magazine, vol
Rabiner, L. and Juang, B., ”An introduction to hidden Markov models,”IEEE ASSP Magazine, vol. 3, no. 1, pp. 4-16, 1986. doi:10.1109/MASSP.1986.1165342
1986
-
[6]
Zhang, Jihai and Zhou, Wengang and Xie, Chao and Pu, Junfu and Li, Houqiang, ”Chinese sign language recognition with adaptive HMM,” in 2016 IEEE International Conference on Multimedia and Expo (ICME), 2016 pp. 1-6. doi:10.1109/ICME.2016.7552950
2016
-
[7]
Starner, Thad and Group, Massachusetts, ”Visual Recognition of Amer- ican Sign Language Using Hidden Markov Models,” 1995
1995
-
[8]
and Bottou, L
Lecun, Y . and Bottou, L. and Bengio, Y . and Haffner, P., ”Gradient- based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998. doi:10.1109/5.726791
1998 doi
-
[9]
[Online]
Karen Simonyan and Andrew Zisserman, ”Very Deep Convolu- tional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556, 2015. [Online]. Available: https://arxiv.org/abs/1409. 1556
2015 arXiv
-
[10]
[Online]
Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun, ”Deep Residual Learning for Image Recognition,” arXiv preprint arXiv:1512.03385, 2015. [Online]. Available: https://arxiv.org/abs/1512. 03385
2015 arXiv
-
[11]
doi:10.1007/978-3-319-16178-5 40
Pigou, Lionel and Dieleman, Sander and Kindermans, Pieter-Jan and Schrauwen, Benjamin, ”Sign Language Recognition Using Convolu- tional Neural Networks,” 2015. doi:10.1007/978-3-319-16178-5 40
2015 doi
-
[12]
[Online]
Du Tran and Lubomir Bourdev and Rob Fergus and Lorenzo Torresani and Manohar Paluri, ”Learning Spatiotemporal Features with 3D Con- volutional Networks,” arXiv preprint arXiv:1412.0767, 2015. [Online]. Available: https://arxiv.org/abs/1412.0767
2015 arXiv
-
[13]
43, 2012
Rautaray, Siddharth and Agrawal, Anupam, ”Vision based Hand Gesture Recognition for Human Computer Interaction: A Survey,”Artificial Intelligence Review, vol. 43, 2012. doi:10.1007/s10462-012-9356-9
2012 doi
-
[14]
Camgoz, Necati and Hadfield, Simon and Koller, Oscar and Ney, Hermann and Bowden, Richard, ”Neural Sign Language Translation,”
-
[15]
[Online]
Ali Ghadami and Alireza Taheri and Ali Meghdari, ”A Transformer- Based Multi-Stream Approach for Isolated Iranian Sign Language Recognition,” arXiv preprint arXiv:2407.09544, 2024. [Online]. Avail- able: https://arxiv.org/abs/2407.09544
2024 arXiv
-
[16]
3154-3160
Hara, Kensho and Kataoka, Hirokatsu and Satoh, Yutaka, ”Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition,” in2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017 pp. 3154-3160. doi:10.1109/ICCVW.2017.373
2017 doi
-
[17]
1250-1253
Buckley, Neil and Sherrett, Lewis and Lindo Secco, Emanuele, ”A CNN sign language recognition system with single & double- handed gestures,” in2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), 2021 pp. 1250-1253. doi:10.1109/COMPSAC51774.2021.00173
2021
-
[18]
Gomez and Lukasz Kaiser and Illia Polosukhin, ”Attention Is All You Need,” arXiv preprint arXiv:1706.03762, 2023
Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and Lukasz Kaiser and Illia Polosukhin, ”Attention Is All You Need,” arXiv preprint arXiv:1706.03762, 2023. [Online]. Available: https://arxiv.org/abs/1706. 03762
2023 arXiv
-
[19]
Hinton, ”Layer Normalization,” arXiv preprint arXiv:1607.06450, 2016
Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E. Hinton, ”Layer Normalization,” arXiv preprint arXiv:1607.06450, 2016. [Online]. Avail- able: https://arxiv.org/abs/1607.06450
2016 arXiv
-
[20]
Nitish Srivastava and Geoffrey Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov, ”Dropout: A Simple Way to Pre- vent Neural Networks from Overfitting,”Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: http://...
1929
-
[21]
Kingma and Jimmy Ba, ”Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2017
Diederik P. Kingma and Jimmy Ba, ”Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2017. [Online]. Avail- able: https://arxiv.org/abs/1412.6980
2017 arXiv
-
[22]
Kohavi, Ron, ”A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection,” 2001
2001
-
[28]
Mercanoglu Sincan, Ozge and Keles, Hacer Yalim, ”Using Motion History Images With 3D Convolutional Networks in Isolated Sign Language Recognition,”IEEE Access, vol. 10, pp. 18608–18618, 2022. doi:10.1109/access.2022.3151362
2022
-
[29]
3436-3445
De Coster, Mathieu and Van Herreweghe, Mieke and Dambre, Joni, ”Isolated Sign Recognition from RGB Video using Pose Flow and Self-Attention,” in2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021 pp. 3436-3445. doi:10.1109/CVPRW53098.2021.00383
2021
-
[32]
[Online]
Keiron O’Shea and Ryan Nash, ”An Introduction to Convolutional Neural Networks,” arXiv preprint arXiv:1511.08458, 2015. [Online]. Available: https://arxiv.org/abs/1511.08458
2015 arXiv
-
[33]
Rumelhart and Geoffrey E
David E. Rumelhart and Geoffrey E. Hinton and Ronald J. Williams, ”Learning internal representations by error propagation,” 1986. [Online]. Available: https://api.semanticscholar.org/CorpusID:62245742
1986
-
[2018]
doi:10.1109/CVPR.2018.00812
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.