{"id":"90ec6b5a-1b24-4a33-8ca8-a07b7ec60fbc","arxiv_id":"2411.15200","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A GCN plus LSTM plus attention model trained on OpenPose skeleton data from 50 videos claims 85% accuracy at distinguishing pediatric dystonia from chorea.","lead":"This study trains a neural network to tell apart two childhood movement disorders, dystonia and chorea, from video of children holding up their arms. The model reports 85% accuracy, but the small dataset and likely test-leakage make the result a proof of concept rather than a validated diagnostic.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 85% accuracy likely reflects within-patient leakage: clips from the same children appear in both training and test, so the evaluation does not establish generalization to new patients.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the evaluation split is at clip level, not patient level, so clips from the same child appear on both sides of the train/test boundary. The paper's own numbers corroborate this: Section 3.2 says 45 patients are in the training set, and Table 3.1 shows 318 training clips and 88 test clips from 50 videos. With only 5 patients potentially absent from training, most test clips must come from patients already seen in training. This makes the headline accuracy a measure of within-patient memorization rather than cross-patient generalization. The proposed patient-level re-split is a direct, feasible test because the code and pose data are released. If accuracy drops, the rejection stands; if it holds, the concern is mitigated but external validation would still be needed. The paper's transparency about limitations and its attention-map analysis are commendable but do not compensate for the invalidated evaluation.","tokens_in":26764,"tokens_out":5520,"duration_ms":55315,"concrete_test":"Using the public GitHub repository, reconstruct the 15 fps clip set and assign each clip to its source patient (e.g., via video filename). Perform a strictly patient-level split: train on clips from 40 patients, validate on 5, and test on the remaining 5, stratified by class; repeat five times. Report the average held-out patient-level accuracy (one majority-vote prediction per patient) alongside clip-level metrics. If patient-level accuracy falls below the reported 85% CI, the original metric is inflated by within-patient leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4.2 says the dataset was split 80/10/10 using stratified sampling, and Table 3.1 lists 318 training clips and 88 test clips derived from 50 videos. Nowhere does the paper state that clips from the same child are restricted to a single split. Section 3.2 then reports '45 patients identified in the training set' out of 50 videos, implying at least 45 patients contribute clips to training. Since the test set has 88 clips and only 5 patients could be entirely absent from training, most test clips almost certainly come from patients already represented in training. The model can therefore exploit patient-specific static cues (body proportions, background, camera framing, and smoothed OpenPose tracking errors) that are shared across clips of the same video, memorizing identity rather than learning a generalizable movement signature. The reported 85% accuracy, 95% CI (0.797–0.906), and p-values are derived from clip-level bootstrap, treating clips as independent despite nesting within patients. The central claim that the model generalizes to unseen patients is not supported by this evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a proof-of-concept deep learning model for differentiating dystonia from chorea in children using video-based pose data. The model combines a GCN-based spatial branch, a bidirectional LSTM temporal branch, and attention mechanisms, and is evaluated on 50 videos using OpenPose keypoints. The authors report 85% accuracy, 81% sensitivity, 88% specificity, and an F1 score of 0.811 at 15 fps, alongside qualitative attention-map analyses. The paper also makes its code and data publicly available.","tokens_in":26966,"tokens_out":8213,"duration_ms":76726,"significance":"Automated classification of pediatric hyperkinetic movement disorders is clinically relevant, and the proposed architecture is a reasonable approach for skeleton-based spatiotemporal classification. The explicit attention-map analysis is a valuable step toward interpretability, and the public repository strengthens reproducibility. However, the quantitative results are not reliable because the train/test split appears to be at the clip level rather than the patient level, and the statistical inference ignores patient clustering. Given the small single-center dataset and single-rater labels (acknowledged in Section 4.4), the current evidence does not establish that the model generalizes to new patients. If the evaluation is corrected, the work could be of interest, but the present manuscript's central claim is not supported.","major_comments":[{"comment":"The evaluation split is described at the level of clips, not patients. Section 3.2 states that 45 patients were identified in the training set, while Table 3.1 reports 318 training clips and 88 test clips from a total of 50 videos. Unless the test set was deliberately restricted to the remaining 5 patients (which the paper never states), clips from the same patient appear in both training and test. Because videos of the same child share static cues such as body proportions, background, and OpenPose error patterns, the model can memorize patient-specific signatures rather than learn a generalizable movement signature. The reported 85% accuracy therefore does not support the claim of classification performance on unseen patients.","section":"Section 2.4.2, Section 2.5.3, Table 3.1, Section 3.2"},{"comment":"The evaluation protocol is internally inconsistent. Section 2.4.2 describes an 80/10/10 split, whereas Section 2.5.3 describes 5-fold cross-validation followed by fine-tuning and a final independent test set. Table 3.2 is captioned 'Cross-Validation Metrics' even though the text reports these as test-set metrics. Additionally, the 15 fps frame rate was selected after comparing test-set performance across frame rates, so the reported metrics are the best of multiple evaluations and the p-values do not account for this selection. The authors should specify the exact protocol, use a patient-disjoint holdout, and treat frame-rate selection as model selection.","section":"Section 2.5.3, Table 3.2"},{"comment":"The bootstrap confidence intervals and one-sample t-tests treat individual clips as independent units. Clips are nested within patients, and the effective sample size at the patient level is at most 50. This clustering violates the independence assumption and produces overconfident CIs and p-values. Inference should be performed at the patient level (e.g., cluster bootstrap, permutation test by patient, or mixed-effects model), and the baseline for the p-value should reflect the class imbalance and the number of distinct patients.","section":"Section 2.5.3, Table 3.2"},{"comment":"The class labeling is contradictory and makes sensitivity/specificity uninterpretable. Section 2.1.1 defines chorea as the negative class and dystonia as the positive class. Section 3.1 refers to 'dystonia-predominant HMDs (minority class)' but later states 'the minority class (chorea) compared to the majority class (dystonia)'. Table 3.2 reports Sensitivity=0.810 and Specificity=0.878, but without a consistent definition of the positive class, readers cannot tell which class these correspond to. The authors must define the reference class and report per-class performance.","section":"Sections 2.1.1, 3.1, Table 3.2"},{"comment":"The hyperparameter selection procedure is not fully specified and may bias the reported performance. Table 2.1 states that hyperparameters were 'tuned based on classification accuracy for the training dataset,' and Section 2.4.4 says the final model was evaluated on the test set after selecting the best hyperparameters. If the grid search used the test set (or the training set without a validation set), the reported accuracy is optimistically biased. Please clarify which data were used for model selection and whether a separate validation set was used.","section":"Section 2.4.4, Table 2.1"}],"minor_comments":[{"comment":"The qualitative analysis of attention maps is based on a small number of selected frames, and the claim that attention patterns are 'clinically meaningful' is not validated against independent clinician judgments or quantitative metrics. Please temper the claim or provide supporting evidence.","section":"Section 4.2"},{"comment":"The confidence threshold of 5% for keypoint detection is unusually low; please justify this choice, as low-confidence keypoints may degrade the input data.","section":"Section 2.1.2"},{"comment":"The section heading contains a typo: 'STatistical Ananlysis' should be 'Statistical Analysis'.","section":"Section 2.5.3"},{"comment":"The text says 'Figures 1-6 show frames' but the figures are numbered 3.4 through 3.7; please renumber or correct the references.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The reader's stress-test concern is valid: the paper's quantitative evaluation is undermined by the clip-level split and clustering issues. In my view, the central claim is not supportable as written. The paper could be resubmitted after a patient-level analysis, but with only 50 videos the results may change substantially. The GitHub repository and attention map analysis are strengths that could form the basis for a revised study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the headline accuracy of 85% is not a trustworthy estimate of generalization to new patients. Section 2.4.2 describes a stratified 80/10/10 split, but that appears to be at the clip level, not the patient level. Section 3.2 then says 45 of the 50 patients are 'identified in the training set,' which means most of the 88 test clips are from patients the model already saw in training. The model can exploit patient-specific static cues — body proportions, background, camera framing, and smoothed OpenPose tracking errors — rather than learning a generalized movement signature. The bootstrap CIs and p-values treat clips as independent, which compounds the problem.\n\nThat said, this is not a careless paper. The clinical task is real and underexplored, and this is likely the first attempt to separate pediatric dystonia-predominant from chorea-predominant presentations using pose data. The authors are unusually candid about limitations: single tertiary center, one consultant doing all the labels, class imbalance, selection bias, and they explicitly say 85% is below clinical acceptability. They also released the code and the OpenPose keypoint data on GitHub, which is a genuine contribution. The attention-map analysis is exploratory but shows the model sometimes attends to clinically sensible regions, such as legs during involuntary kicks and arms during posturing.\n\nThe fix is straightforward in principle: re-split at the patient level, report per-patient metrics, and ideally validate on an external cohort. Without that, the central claim fails. Smaller issues: no inter-rater reliability for the labels, no ablation study to justify the complex GCN+LSTM+attention architecture, and the description of the five-fold cross-validation plus a final test set is muddled.\n\nThis paper deserves a serious referee, but not as-is. A good reviewer could help the authors turn a plausible proof-of-concept into a credible evaluation. I would not cite the numbers as they stand, but I would point to the dataset and the honest limitations section.\n\nRecommendation: send to peer review with a strong request for patient-level split and external validation. It should not be desk-rejected.","headline":"The reported 85% accuracy is inflated by clip-level splitting that leaks patient identity, but the task, the released data, and the honest limitations write-up make this worth a serious revision rather than a desk reject.","tokens_in":27534,"tokens_out":2823,"would_cite":false,"duration_ms":28547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-LSTM network separates childhood dystonia from chorea in video with 85% accuracy.","keywords":["hyperkinetic movement disorders","pediatric dystonia","chorea","video-based diagnosis","pose estimation","graph convolutional network","long short-term memory","attention mechanism"],"falsifier":"Take the released code and data, rebuild the same model, and hold out all clips of entire patients—not individual clips—when testing. If test accuracy drops toward chance (50%) or substantially below the reported 85%, the paper's generalization claim would be refuted; if accuracy stays near 85%, the claim survives.","tokens_in":26525,"feed_emoji":"🩺","tokens_out":10732,"duration_ms":102083,"temperature":0.7,"pith_summary":"Two child movement disorders—dystonia's twisting postures and chorea's irregular flowing jerks—look similar on examination yet call for different treatments, and affected families typically wait years for a correct diagnosis. This proof-of-concept dissertation tries to make the distinction from ordinary video: children hold their arms outstretched, a pose-estimation tool records 25 body keypoints per frame, and a neural network decides whether the clip is chorea-predominant or dystonia-predominant. The network combines a graph convolutional network for spatial body-part relationships, a bidirectional LSTM for motion over time, and attention layers that reveal which body parts and frames drove each decision. The paper's central claim is that at 15 frames per second the model reaches 85% accuracy, 81% sensitivity, and 88% specificity, with an F1 score of 81%, on 50 clinical videos. If true, this would give clinicians a video-based second opinion that is faster and more reproducible than expert observation alone, and could shorten diagnostic delay.","feed_headline":"Video model tells childhood dystonia from chorea at 85%","feed_subtitle":"A graph-and-LSTM network reads body-keypoint videos of raised-hand tasks, tracing its own reasoning with attention maps.","key_machinery":"The machinery is a two-stream spatial-temporal network with an interpretability layer. Its input is a matrix of $x,y$ coordinates for 25 body keypoints, extracted by OpenPose, over 75 consecutive frames, ordered as a travelling sequence so joints that move together are adjacent. A spatial stream uses fully connected layers to encode five body parts, self-attention to weight the parts, and a graph convolutional network to model relationships between non-adjacent parts such as arm-leg coordination. A temporal stream uses four bidirectional LSTM layers to capture how joint positions evolve, with self-attention over frames. The two streams are fused by a multi-head attention layer before fully connected layers and a SoftMax output. Focal Loss with $\\alpha=0.1$ and bootstrapped oversampling compensate for class imbalance, and the learned attention scores are visualised as colour-coded body parts plus a temporal attention bar.","core_discovery":"On the paper's own terms, the discovery is a new application: automated, interpretable classification of pediatric hyperkinetic movement disorders from pose-extracted video. Using 50 baseline recordings (31 chorea-predominant, 19 dystonia-predominant) of a standardized outstretched-arm task, the authors convert OpenPose keypoint coordinates into a spatiotemporal input of 25 joints over 75 frames. The spatial branch encodes five body parts with shared-weight fully connected layers and a graph convolutional network; the temporal branch stacks four bidirectional LSTM layers; self-attention and multi-head attention weight parts, frames, and the merged representation. On the best configuration, sampled at 15 fps, the model achieves 85% accuracy, 81% F1, 81% sensitivity, and 88% specificity, with the paper's bootstrap confidence intervals and p-values indicating these are significantly above chance. The attention maps are offered as evidence that the model learned clinically meaningful movement patterns: sustained arm posturing in dystonia, diffuse multi-body attention in chorea, and misclassifications tied to occlusions, subtle or mixed presentations, and voluntary movements.","pith_inferences":["Beyond the paper, the decisive next validation is a patient-level split: if clips from the same child appear in both training and test sets, part of the 85% could reflect memorized individual movement signatures, so rerunning with each child confined to one fold would quantify the true generalization.","Beyond the paper, a natural stress test is to add healthy controls and mildly affected children, since the current cohort is drawn from severely affected patients referred for deep brain stimulation; the paper itself notes the model is not ready for a general population.","Beyond the paper, the attention maps could be scored against independent clinician judgment—do the body parts the model highlights match the features a neurologist would cite for the same video?—turning visual plausibility into a measurable claim.","Beyond the paper, because the pose tracker was trained on able-bodied movement, its keypoint errors in children with abnormal postures are a hidden variable; comparing model predictions against manual joint annotations on a subset of frames would show how sensitive the 85% is to pose-estimation quality."],"forward_implications":["A clinical deployment of this pipeline would turn a short, simple motor task into a quantitative readout, letting non-specialist centers flag children who need a movement-disorder workup instead of waiting years for expert review.","The attention maps provide a checkable rationale: each automated diagnosis could be reviewed by looking at which frames and body parts the model considered decisive, making the tool more acceptable in clinical workflows than a black box.","The 15 fps optimum gives future data collection a concrete protocol: record or downsample at 15 fps to save storage and compute while keeping the information the network needs.","Because the spatial and temporal pathways are modular, the same architecture can be retrained for other hyperkinetic disorders or for tracking severity over time, both directions the paper names as future work."],"supporting_citations":[{"why":"Supplies OpenPose, the pose-estimation tool whose 25 whole-body keypoints are the network's raw input.","marker":"Cao et al., 2019"},{"why":"Introduces spatial-temporal graph convolution for skeleton data, the modeling strategy the spatial branch of this network extends.","marker":"Yan et al., 2018"},{"why":"Provides the hierarchical part-level and body-level spatial reasoning design with LSTM clip processing that the architecture adapts.","marker":"Si et al., 2020"},{"why":"Supplies the self-attention and multi-head attention operations used to weight body parts, frames, and the fused spatial-temporal features.","marker":"Vaswani et al., 2023"},{"why":"Focal Loss is the training objective chosen to counter class imbalance and down-weight easy examples.","marker":"Lin et al., 2018"},{"why":"The closest prior video-based classifier of dystonia versus choreoathetosis, used as the comparison the paper claims to outperform.","marker":"Haberfehlner et al., 2024"},{"why":"Defines the LSTM cell that forms the temporal stream for capturing movement dynamics across frames.","marker":"Hochreiter and Schmidhuber, 1997"}],"fun_headline_variants":["AI video model separates dystonia from chorea in kids","85% accurate AI separates dystonia from chorea in kids","85% accuracy: deep learning reads kids' movement videos","Neural net classifies childhood movement disorders from video","Deep learning picks out dystonia vs chorea in children"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the train/validation/test split keeps every video clip from the same child in the same side of the split; if clips from one child appear in both training and test, the 85% figure could reflect patient-specific signatures rather than generalization to new patients.","fun_headline_variants_meta":{"raw":{"variants":["AI video model separates dystonia from chorea in kids","85% accurate AI separates dystonia from chorea in kids","85% accuracy: deep learning reads kids' movement videos","Neural net classifies childhood movement disorders from video","Deep learning picks out dystonia vs chorea in children"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3715,"prompt_tokens":1058,"completion_tokens":2657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2576}},"tokens_in":674,"tokens_out":2657,"duration_ms":16288,"temperature":1.0,"reasoning_tokens":2576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:04:55.741797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released code and data, rebuild the same model, and hold out all clips of entire patients—not individual clips—when testing. If test accuracy drops toward chance (50%) or substantially below the reported 85%, the paper's generalization claim would be refuted; if accuracy stays near 85%, the claim survives.","supporting_citations":[],"review_version":1}