{"id":"729fbfb4-1f2a-4057-86fc-43205c3a90b3","arxiv_id":"2412.18187","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On a newly collected 162-video sign language dataset, a 3D convolutional network outperformed CNN-LSTM, CNN-RNN-LSTM, and time-distributed CNN in classification accuracy.","lead":"The authors tested four standard neural network designs on sign language videos from a new Trinidad and Tobago Sign Language dataset and an American Sign Language dataset. The 3DCNN model performed best, reaching 91% accuracy on TTSL and 83% accuracy on ASL.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frame-level data split likely leaks the same recorded videos into train and test, inflating the reported 91%/83% accuracies and undermining the 3DCNN-best ranking.","rationale":"The decisive issue is not merely that the dataset is small or single-signer; it is that the evaluation protocol appears to violate independence between training and test data. If frames from the same video appear on both sides, every reported accuracy is a within-recording memorization score, and the model ranking is not evidence about sign recognition. This is a concrete, fixable flaw, and the paper contains enough internal evidence to make the concern credible. The reader's weakest assumption was about representativeness of 10-20 videos per sign; our concern is a more specific data-split defect, so agreement is partial. Because the central claim is supported only by the questionable accuracies and the authors do not release code/data for verification, the current manuscript should not be accepted. I would recommend REJECT for this version, with a clear path: re-run using video-level splitting and release the split metadata; if 3DCNN remains best, the central claim may then be reconsidered.","tokens_in":12751,"tokens_out":6554,"duration_ms":60767,"concrete_test":"Obtain the data/code or split metadata from the authors. Re-run the four models with an identical hyperparameter protocol but split at the video level: hold out all frames from entire videos (e.g., leave-one-video-out or stratified video-level 80/20), and evaluate either per-frame with a majority vote per test video or directly per video. Also run at least 5 seeds to get confidence intervals. If 3DCNN remains the top model and its accuracy on held-out videos is within a small margin of 91%/83%, the concern is resolved; if the accuracy drops substantially or another model wins, the central claim should be withdrawn.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—3DCNN is the best model, at 91% on TTSL and 83% on ASL—depends on how the 162-video dataset is divided. Section III.C says the capture application 'split the videos into frames' and then 'the dataset which is referring to the captured frames was divided by training, validation or testing' (80/20). Section III.D likewise says the videos were 'broken up' and then 'frames were extracted... then each dataset was parsed through each model.' This reads as a frame-level random split, not a video-level split. Because adjacent frames from the same 35-frame clip are near-duplicates, random frame-level splitting lets frames from the same recording appear in both training and test sets. The model can then achieve high accuracy by memorizing the specific signer/environment rather than learning the sign. That would inflate all four models' numbers and could change the ordering. The paper's own Section IV.E supports this: when the saved CNN-TD model was tested on a new webcam video of the same signer, it recognized only 'Doubles' despite reporting high recall on TTSL; 3DCNN recognized three of four signs. So the headline accuracies are not a trustworthy basis for ranking until the split respects video boundaries.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four neural architectures (CNN-LSTM, 3DCNN, CNN-RNN-LSTM, and CNN-TD) on a self-captured dataset of 162 videos covering 10 signs from American Sign Language and Trinidad and Tobago Sign Language. The central claim is that 3DCNN is the best-performing model, with 91% accuracy on the TTSL subset and 83% on the ASL subset. The paper also reports a small 'new sign' webcam evaluation in which 3DCNN correctly identified 3 of 4 TTSL signs while CNN-TD identified only Doubles, and concludes that 3DCNN is the most suitable basis for a sign-language tuition system.","tokens_in":12959,"tokens_out":3101,"duration_ms":28670,"significance":"If the comparative ranking were reliable, the paper would provide a useful empirical data point on a previously unexplored sign language (TTSL) and a reasonable baseline for future work on Caribbean sign languages. The authors are transparent about the small dataset and the same-signer/same-environment limitation, and the new-sign experiment is an honest attempt to probe generalization. However, the load-bearing issue is that the reported accuracies and the resulting ranking rest on an evaluation protocol whose validity is not established: the text in Sections III.C and III.D indicates a frame-level random split, which can leak near-duplicate frames from the same video into both training and test sets. The paper's own Section IV.E then shows poor transfer to a new video, which is consistent with inflated test results. The central claim therefore needs substantial rework before the comparison can be trusted.","major_comments":[{"comment":"The data-splitting procedure is described at the frame level, not the video level: Section III.C states that the capture application 'split the videos into frames' and then 'the dataset which is referring to the captured frames was divided by training, validation or testing, with training being 80% and testing and validation 20%', and Section III.D similarly says videos were 'broken up' before frames were extracted and parsed through each model. Because each video is a 35-frame clip, adjacent frames are near-duplicates, so a random frame-level split places frames from the same recording in both training and test sets. This violates the independence assumption and can inflate all reported accuracies, potentially changing the model ranking. The evaluation should be redone with a video-level split (e.g., group frames by source video and assign whole videos to train/validation/test), and the paper should report per-video or per-clip evaluation metrics.","section":"III.C and III.D"},{"comment":"The new-sign experiment directly undermines the external validity of the reported classification accuracies. The text states that both 3DCNN and CNN-TD had 87.5% recall on the TTSL dataset, yet when tested on a newly recorded webcam video of the same signer in the same environment, CNN-TD correctly identified only 'Doubles' and 3DCNN correctly identified 3 of 4 signs. The authors attribute this to the dataset being 'extremely similar in terms of the signs being completed by the same person in the same environment.' This is evidence that the high test-set accuracies reflect memorization of the recording conditions rather than robust sign recognition. The claim that 3DCNN is the best model for a real-time tuition system therefore needs support from an evaluation that controls for signer and environment, such as leave-one-signer-out or cross-environment testing.","section":"IV.E"},{"comment":"The paper reports a single 80/20 train/test split with no repeated runs, no error bars, and no statistical comparison. With roughly 162 videos total, the test set is on the order of tens of videos spread across 10 classes, so per-class support is very small. The reported differences between models (e.g., 91% vs. 87.5% on TTSL) may not be statistically distinguishable. The comparison should be repeated over multiple random splits (with video-level grouping) and reported as mean ± standard deviation, and the authors should state the per-class test support so readers can judge the reliability of the reported precision and recall values.","section":"IV (all subsections)"}],"minor_comments":[{"comment":"The title uses 'CCN TD' while the body consistently refers to 'CNN-TD'; this should be corrected.","section":"Title"},{"comment":"There is a typo: 'For 3CDNN processing' should read 'For 3DCNN processing'.","section":"III.B"},{"comment":"The heading 'ACCURACY AND LOSS VALUES INTERPRETION' contains a typo; it should be 'Interpretation'.","section":"IV (Table IV)"},{"comment":"Figure 6 is a generic illustration of a confusion matrix, but the per-model confusion matrices described in the text are not shown. Either include the actual confusion matrices or point to supplementary material.","section":"IV and figures"},{"comment":"Table XIII is referenced as presenting the new-sign results, but the table is not visible in the manuscript; the table should be included or the results should be described in full in the text.","section":"IV.E and Table XIII"},{"comment":"The discussion cites reference [28] to support the claim that other models outperform CNN-TD on video spatiotemporal datasets, but it does not explain how that prior work relates to the present architectures, hyperparameters, or dataset; please clarify the relevance.","section":"V.Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is more in the style of a workshop or short conference paper than a full journal article; the experiments need to be redone with a proper video-level split before the comparative claim can be accepted. The new TTSL dataset is a potentially valuable contribution, and the authors should be encouraged to release it (with consent) and to provide the exact split procedure and code. If the frame-level split is indeed what was done, the central ranking claim is not supported; if the authors instead used a video-level split, then the manuscript needs to state that explicitly and describe the grouping mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe stress-test note is right on the money, and it lands on a load-bearing wall. This paper compares four standard architectures on a newly collected Trinidad and Tobago Sign Language dataset (162 videos, 10 signs, one signer). The dataset is the only new thing here, and it is genuinely new—no prior TTSL recognition work is cited. That part deserves credit.\n\nWhat the paper does well: the data collection is described in enough detail to see what happened, the results include classification reports and confusion matrices, and the authors explicitly list small dataset size as a limitation in Section III.E. The new-sign experiment in Section IV.E is the most valuable part. When the saved models are tested on a fresh webcam video of the same signer, CNN-TD only recognizes Doubles and 3DCNN gets 3 of 4. The authors note this is interesting but don't connect the dots.\n\nThe dots connect to the split. Section III.C says the capture app split the videos into frames, and then \"the dataset which is referring to the captured frames was divided by training, validation or testing\" at 80/20. That's a frame-level random split. With 35-frame clips, random frame splits put near-duplicate frames from the same recording in both train and test. The network can memorize the signer and background. That inflates all four accuracy numbers and can change the ranking. The 91% for 3DCNN is therefore not a trustworthy basis for the paper's central claim.\n\nOther problems: no code or data released, no repeated runs, no error bars, and a test set of a few dozen videos. The paper would need a video-level split and ideally multiple signers before the ranking is meaningful.\n\nWho should read this: people working on low-resource sign language recognition, and anyone teaching evaluation methodology. The dataset, if released, could be a useful starting point. As it stands, the paper is a cautionary example of train/test leakage more than a reliable benchmark.\n\nI would not cite the accuracy numbers. I would send it to peer review only with a clear request: the authors must release the data and code and re-run with a video-level split. Without that, it's a desk reject.","headline":"The paper's central 3DCNN-best claim is undermined by a likely frame-level train/test split that leaks the same videos into both sides, and the new-sign test confirms poor generalization.","tokens_in":13495,"tokens_out":4253,"would_cite":false,"duration_ms":38167,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3DCNN that learns spatial and temporal features directly from video outperforms the three other compared architectures on both a new Trinidad and Tobago Sign Language dataset (91%) and American Sign Language (83%).","keywords":["sign language recognition","3DCNN","spatiotemporal features","Trinidad and Tobago Sign Language","American Sign Language","neural network comparison","video classification","CNN-LSTM"],"falsifier":"A concrete check: record a larger TTSL test set with multiple signers, different backgrounds, and varied lighting, then re-run the four architectures; if 3DCNN's margin over CNN-LSTM and CNN-TD shrinks or reverses on those held-out videos, the paper's central ranking is an artifact of the single-signer dataset rather than a property of the models.","tokens_in":12510,"feed_emoji":"🤟","tokens_out":6041,"duration_ms":49196,"temperature":0.7,"pith_summary":"The paper sets out to identify a neural network that can watch a video of someone signing, translate the sign, and grade whether the learner's form is correct, which is the core of a real-time sign-language tuition app. It compares four modern video-classification architectures (CNN-LSTM, 3DCNN, CNN-RNN-LSTM, and CNN-TD) on a newly assembled Trinidad and Tobago Sign Language dataset plus an American Sign Language dataset. The authors report that 3DCNN performs best on both, reaching 91% accuracy on TTSL and 83% accuracy on ASL. They conclude that 3DCNN, because it learns spatial and temporal features together from stacked frames, is the best foundation for building a translation and grading system for Caribbean sign languages.","feed_headline":"3DCNN tops rival sign-language models at 91% accuracy","feed_subtitle":"Architecture comparison on a new TTSL dataset favors 3DCNN for real-time sign translation and grading.","key_machinery":"The load-bearing mechanism is the three-dimensional convolutional layer, which slides learnable 3D filters across a volume made by stacking video frames, so the same operation captures both spatial structure in each frame and motion across frames. The paper contrasts this with two-frame CNN-plus-recurrent designs (CNN-LSTM and CNN-RNN-LSTM) and a time-distributed wrapper that applies a 2D CNN independently to each timestamp. The 3DCNN's direct spatiotemporal feature learning is what the paper credits for the higher accuracy and faster prediction on newly submitted signs.","core_discovery":"On the paper's own terms, the discovery is a comparative ranking: when the same ten-word, ten-to-twenty-videos-per-word dataset (six ASL signs, four TTSL signs) is run through four architectures under shared preprocessing, loss, and optimizer settings, the 3DCNN is the best classifier. Its classification reports show 91% accuracy on the TTSL subset and 83% on the ASL subset, with the highest recall among all models; it also used the least RAM and GPU during training and made correct predictions on three of four new TTSL signs in a saved-model webcam simulation, whereas the runner-up CNN-TD recognized only one. The paper therefore claims that 3DCNN's ability to learn spatiotemporal features directly from stacked video frames makes it the best available base for a sign-language learning and assessment application, at least for these datasets.","pith_inferences":["If 3DCNN's advantage holds on a larger multi-signer corpus, the same architecture could serve as a generic starting point for other under-resourced sign languages in the Caribbean, with fine-tuning on a small local vocabulary.","The finding that CNN-TD scored well on the split but failed on new signs suggests that single-signer accuracy may be inflated by background and clothing cues; a controlled experiment that removes the signer from the frame, for example by training on pose keypoints instead of raw pixels, would test whether the models are learning the sign or the person.","The paper's grading idea could be tested directly against human raters: compare the model's per-sign certainty score with a teacher's pass or fail judgment on the same videos to calibrate the certainty output as a usable grade."],"forward_implications":["A 3DCNN-based sign-language tuition application can translate a learner's recorded sign and give a correctness grade within seconds; in the paper's simulation, the 3DCNN returned predictions on new webcam signs in 3 to 10 seconds after the first run.","The same training pipeline extends to a sign language for which no prior recognition dataset exists, since the authors built and used a new TTSL dataset from scratch.","For small sign datasets, two-stage CNN-plus-recurrent designs can be worse than a single spatiotemporal model; CNN-RNN-LSTM underperformed because its setup was too complex for the data.","Time-distributed CNN is a viable competitor on the held-out TTSL split (87.5% recall) but fails to generalize to newly recorded signs, so model choice should be validated on fresh captures rather than only on the test split.","Resource use matters for deployment: 3DCNN had the highest parameter count but the lowest RAM and GPU usage during training, making it practical for a classroom-app use case."],"supporting_citations":[{"why":"Supplies the 3DCNN base architecture the paper adapts: learning spatiotemporal features with 3D convolutions.","marker":"[39]"},{"why":"Earlier 3D convolutional recurrent model that motivates the use of 3DCNNs for spatiotemporal sequence data.","marker":"[12]"},{"why":"Applies 3DCNN to hand gesture recognition for sign language, supporting the choice of 3D convolution for sign videos.","marker":"[3]"},{"why":"Defines the long-term recurrent convolutional network lineage that the CNN-RNN-LSTM baseline is built on.","marker":"[14]"},{"why":"Provides the CNN-LSTM architecture used as one of the four compared models.","marker":"[16]"},{"why":"Supplies the integrated CNN-RNN-LSTM setup the paper compares against 3DCNN.","marker":"[18]"},{"why":"Introduces convolutional LSTM, the basis of the ConvLSTM2D model used in the CNN-LSTM arm.","marker":"[35]"},{"why":"Describes the time-distributed spatiotemporal feature learning approach used in the CNN-TD arm.","marker":"[33]"},{"why":"Reported that other models outperform standalone CNN-TD on video spatiotemporal datasets, the comparison the paper's discussion engages with.","marker":"[28]"}],"fun_headline_variants":["3DCNN beats CNN-LSTM and 3DCNN hybrids at 91% accuracy","Sign language AI: 3DCNN achieves best accuracy of 91% in test","3DCNN wins sign language recognition race with 91% top accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that 10 to 20 videos per sign, all produced by one signer in one setting, represent the sign well enough that accuracy differences among models on this dataset will survive contact with real learners and new environments.","fun_headline_variants_meta":{"raw":{"variants":["3DCNN beats CNN-LSTM and 3DCNN hybrids at 91% accuracy","Sign language AI: 3DCNN achieves best accuracy of 91% in test","3DCNN wins sign language recognition race with 91% top accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1671,"prompt_tokens":908,"completion_tokens":763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":694}},"tokens_in":524,"tokens_out":763,"duration_ms":6737,"temperature":1.0,"reasoning_tokens":694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:55:42.551404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: record a larger TTSL test set with multiple signers, different backgrounds, and varied lighting, then re-run the four architectures; if 3DCNN's margin over CNN-LSTM and CNN-TD shrinks or reverses on those held-out videos, the paper's central ranking is an artifact of the single-signer dataset rather than a property of the models.","supporting_citations":[{"cited_title":"and Paluri, M., 2015","cited_arxiv_id":null,"evidence_quote":"Supplies the 3DCNN base architecture the paper adapts: learning spatiotemporal features with 3D convolutions."},{"cited_title":"and Zhang, H., 2018","cited_arxiv_id":null,"evidence_quote":"Earlier 3D convolutional recurrent model that motivates the use of 3DCNNs for spatiotemporal sequence data."},{"cited_title":"and Mekhtiche, M.A., 2020","cited_arxiv_id":null,"evidence_quote":"Applies 3DCNN to hand gesture recognition for sign language, supporting the choice of 3D convolution for sign videos."},{"cited_title":"and Darrell, T., 2015","cited_arxiv_id":null,"evidence_quote":"Defines the long-term recurrent convolutional network lineage that the CNN-RNN-LSTM baseline is built on."},{"cited_title":"and Hellinckx, P.,","cited_arxiv_id":null,"evidence_quote":"Provides the CNN-LSTM architecture used as one of the four compared models."},{"cited_title":"and Khehra, B.S., 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the integrated CNN-RNN-LSTM setup the paper compares against 3DCNN."},{"cited_title":"and Woo, W.C., 2015","cited_arxiv_id":null,"evidence_quote":"Introduces convolutional LSTM, the basis of the ConvLSTM2D model used in the CNN-LSTM arm."},{"cited_title":"and Zhang, L., 2018","cited_arxiv_id":null,"evidence_quote":"Describes the time-distributed spatiotemporal feature learning approach used in the CNN-TD arm."},{"cited_title":"and Fu, Z., 2020","cited_arxiv_id":null,"evidence_quote":"Reported that other models outperform standalone CNN-TD on video spatiotemporal datasets, the comparison the paper's discussion engages with."}],"review_version":1}