{"id":"7766a74d-393c-4b45-9230-0958178898d6","arxiv_id":"2411.11548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A BiLSTM using 30-frame windows of joint angles plus raw coordinates reports 99% test accuracy on four exercises, with 95% and 88% accuracy on two additional real-world test sets.","lead":"The paper builds a webcam exercise classifier that combines joint angles with raw landmark coordinates and feeds 30-frame windows into a bidirectional LSTM, reporting about 99% accuracy on four exercises. It matters because it targets real-time fitness tracking that runs in a web browser without requiring the user to select the exercise manually.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99% test-set claim rests on an unspecified 70-15-15 split unit; if the split is made on 30-frame windows rather than on whole videos, the test set contains near-duplicate frames from training videos and the headline accuracy is inflated.","rationale":"The central claim in the abstract is 'achieving an accuracy of over 99% on the test set' and 'the proposed model is the best-performing one.' For that claim to be true, the test set must consist of sequences the model has not been trained on, drawn from videos not used in training. The manuscript never establishes this. The 70-15-15 split is described in Section 3.3.4 after the windowing step in Section 3.2, and no video_id grouping is mentioned. Because the windows are 30 consecutive frames from single-exercise videos, overlapping windows from the same video are near-duplicates; if any of these appear in both train and test, the model has effectively seen the test content. This is the kind of leakage that can produce 99% accuracy on a four-class balanced task regardless of generalization. The external 'Final My Test Video' and 'Final Test Gym Video' sets are separate and mitigate this for the generalizability claim, but they do not validate the headline 99% figure. The best-performing claim is independently contradicted by Table 20 (CNN with soft voting achieves 0.8856 vs 0.8791 on Final Test Gym), which makes the abstract's unconditional wording too strong. Because the issue is checkable by inspecting the code and re-running one split, the correct response is to require that evidence before accepting the headline; this matches the reader's conditional verdict. I do not see a reason to move the verdict further: the paper is transparent about many limitations and provides additional external test sets, and the concern can be resolved with a focused re-analysis.","tokens_in":14147,"tokens_out":5492,"duration_ms":48881,"concrete_test":"Inspect the released training script on GitHub: locate the split call described in Section 3.3.4. If it is a plain train_test_split on the windowed arrays, or does not use GroupShuffleSplit or GroupKFold keyed by video_id, leakage is confirmed. Then rerun the full pipeline with a strict video-level split (all windows of a video in one fold) and report test-set accuracy. If the accuracy drops materially (more than about one percentage point) or the ranking in Table 20 changes, the 99% headline and the 'best-performing' claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3.4 says only that 'the dataset is split into training, validation and, test sets (70-15-15)' and never states whether the split is video-level or window-level. Section 3.2 builds model inputs by grouping frames into 30-frame windows from the same video, so each video yields many highly overlapping windows. If those windows are assigned to folds before splitting (e.g., a plain train_test_split on the windowed samples), then the same recording, same person, same background, and nearly identical landmark sequences appear in both training and test. A BiLSTM can memorize this near-duplicate structure and report 99% on 'unseen' data without learning to generalize. The paper also does not verify that the 'additional videos from various free online sources' are disjoint from the Kaggle and InfiniteRep training videos, compounding the risk. The external home/gym test sets are genuinely separate, but they do not rescue the specific 99% test-set claim in the abstract; they are smaller and less central. A second, independent problem is that the paper's own Table 20 shows the CNN+soft-voting baseline (0.8856) outperforming the proposed BiLSTM (0.8791) on Final Test Gym, so the abstract's unconditional 'best-performing' claim is not supported even by the paper's data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a real-time exercise classifier based on BlazePose landmarks and a Bidirectional LSTM. The model consumes 30-frame windows of joint coordinates plus derived angles to classify four exercises (barbell bicep curl, push-up, squat, shoulder press). Training data combines the Kaggle Workout/Exercises Video Dataset, the InfiniteRep synthetic dataset, and additional online videos; the paper reports 99% accuracy on a 70-15-15 split and additional results on two external test sets (home and gym). The authors also reimplement two prior baselines, present a comparison table, and describe a Streamlit web application. The abstract claims both 'over 99%' test accuracy and that the proposed model is 'the best-performing one.'","tokens_in":14430,"tokens_out":5424,"duration_ms":53071,"significance":"The paper has several praiseworthy features: two external test sets recorded in home and gym conditions, ablations over feature types (raw coordinates, angles, invariant features), reimplementation of two literature baselines on the same data, and a public GitHub repository. If the data-splitting issue is resolved, the contribution is a useful practical system, though the methodological novelty is incremental: BiLSTM sequence classification over pose features is a standard recipe, and the main specific choices are the angle-plus-coordinate feature set and the 30-frame window. The external accuracies (about 95% on the home set and 88% on the gym set) are more credible and policy-relevant than the 99% headline and should be foregrounded.","major_comments":[{"comment":"The description 'the dataset is split into training, validation and, test sets (70-15-15)' does not specify whether the split is performed on whole videos or on the 30-frame windows constructed in Section 3.2. Because adjacent windows from the same video overlap by 29 frames, a window-level split would place near-duplicate landmark sequences in both training and test, which can produce an inflated test accuracy of 0.9924. The support of 791 in Tables 5 and 6 is consistent with window-level counting rather than video-level counting. Please state the split unit, use a video-level split (or otherwise guarantee that no window from a training video appears in the test set), report the number of train/validation/test videos, and verify that the 'additional videos from various free online sources' are disjoint from the Kaggle and InfiniteRep videos. If the split was window-level, re-run the evaluation and revise the abstract's 'over 99%' claim.","section":"Section 3.3.4"},{"comment":"The abstract's unconditional statement that 'the proposed model is the best-performing one' is not supported by the paper's own results. Table 20 reports Final Test Gym accuracy of 0.8856 for CNN with Soft Voting versus 0.8791 for the proposed BiLSTM with mixed features, while the proposed model is best on the held-out split and My Test Video. Please either restrict the 'best-performing' claim to the conditions where it holds (e.g., the held-out split and the home test set) or explicitly discuss the gym-setting result; as written, the claim overstates the evidence.","section":"Abstract and Section 4.7 (Table 20)"},{"comment":"The comparison with [5] is skipped, with the justification that earlier comparisons already demonstrate superiority of BiLSTM over LSTM and of mixed features over raw coordinates. Those earlier comparisons are against the authors' own LSTM and BiLSTM variants, not against the exact architecture, feature set, and 8-frame window of [5]. Consequently, the 'best-performing' claim is not established relative to all three prior approaches discussed in Section 2.2. Please either implement [5] or clearly state that the comparison covers only [4] and [6], and adjust the corresponding claims.","section":"Section 4.6"}],"minor_comments":[{"comment":"In the Sequence-Level block, the Shoulder Press row reports precision 1.00, recall 1.00, but F1-score 0; this is presumably a typo and should be 1.00.","section":"Table 16"},{"comment":"The coordinate landmark list contains apparent typos: 'LEFT HOOT' should likely be 'LEFT FOOT', and 'RIGHT THUMB, RIGHT THUMB' appears to duplicate the same landmark instead of listing two distinct entries.","section":"Table 2"},{"comment":"The support counts for 'Final Test Gym' are inconsistent across tables: Table 17 sums to 312 (90+76+58+88) while Tables 9 and 10 sum to 306 (90+70+58+88) for the same named set. Please reconcile the counts and make the evaluation protocol for each comparison explicit.","section":"Section 4.4 / Tables 17 and 19"},{"comment":"The column labeled 'Test Set Accuracy' is ambiguous because Section 3.3.4 does not define whether the underlying split is video-level or window-level; after resolving the split unit, rename this column accordingly (e.g., 'Held-out Video Test' or 'Held-out Window Test').","section":"Table 20"},{"comment":"The external test sets contain 202 and 306 window-level samples, but these windows are not independent; reporting per-video accuracy or confidence intervals would make the generalization claims more robust.","section":"Sections 4.3-4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads in places like a thesis excerpt: the chatbot and web-app sections are not central to the claimed contribution, and the comparison section omits one of the three cited baselines. The central issue is the unspecified split unit; if the split turns out to be window-level, the 99% accuracy in the abstract is not reliable, and a video-level re-evaluation would be required. The paper's practical value is real, but the claims need to be aligned with the evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nYou should know this paper before citing its headline numbers: the 99% test accuracy is probably inflated by an unspecified split unit, and the abstract's \"best-performing\" claim is at odds with the paper's own Table 20. That said, it is not a sloppy or dishonest paper. It does several things right, and the flaws are fixable.\n\nWhat is new: a BiLSTM on 30-frame windows using BlazePose angles plus raw coordinates for four exercises (squat, push-up, shoulder press, bicep curl). The ingredients are standard — BiLSTM, MediaPipe, angle features — but the combination, plus the integration into a Streamlit app with automatic exercise switching, is a reasonable applied contribution. The paper also provides two genuinely external test sets (home and gym videos), ablations over feature types, and reimplementations of prior baselines. That is more than most applied ML preprints do. It also explicitly refuses to evaluate the repetition counting logic, which is the right call given that it is rule-based angle thresholds, not learned.\n\nThe soft spots are real. First, Section 3.3.4 says only that the data was split 70-15-15. Since the model inputs are 30-frame windows from the same video, a window-level split would put near-duplicate frames of the same recording into training and test. The paper never states the split unit. If it is window-level, the 99% number is meaningless as a generalization estimate. The external sets are smaller and show 95% and 88% accuracy, which are more believable. Second, the abstract says the proposed model is the best-performing, but Table 20 shows the CNN+soft-voting baseline at 0.8856 on the gym test set versus the BiLSTM's 0.8791. The paper's own data contradict the claim. The author should qualify it. Third, the baseline reimplementations are approximate: for [4] they infer the angle features, for [6] they swap OpenPose for MediaPipe and substitute a landmark. That is acceptable for a rough comparison but should be stated more explicitly as approximate. Fourth, the additional online videos are not checked for overlap with the Kaggle and InfiniteRep sources, compounding the leakage risk.\n\nIn proportion: the external test sets do support the general claim that the model works in real-world conditions, just at lower accuracy. The central methodological contribution is a configuration, not a mechanism, so the paper's value is mostly as a prototype and a cautionary tale about split hygiene.\n\nI would send it to peer review, but with a major-revision request that the split unit be clarified and re-run at video level, and that the \"best-performing\" claim be corrected. The author seems capable of making those fixes. I would not cite it for the 99% number until then.","headline":"A practical BiLSTM exercise classifier with genuinely external test sets, but the 99% headline rests on an unspecified split unit and the 'best-performing' claim is undercut by the paper's own gym results.","tokens_in":14933,"tokens_out":3506,"would_cite":false,"duration_ms":29108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A bidirectional LSTM that reads 30-frame windows of body landmarks, with joint angles added to raw coordinates, classifies four exercises with over 99% test accuracy and runs in real time from a webcam.","keywords":["exercise classification","BiLSTM","pose estimation","BlazePose landmarks","joint angles","temporal sequence classification","repetition counting","real-time video analysis"],"falsifier":"Group every window by its source video ID, check that no video contributes windows to both training and test, then retrain from scratch with a video-level split and report test accuracy; if it drops well below 99%, the original claim did not measure generalization to unseen footage.","tokens_in":13934,"feed_emoji":"🏋️","tokens_out":6860,"duration_ms":64407,"temperature":0.7,"pith_summary":"The paper's claim is that exercise recognition should treat movement as a sequence, not as a bag of frames. A bidirectional LSTM that reads 30-frame windows of body landmarks, with 12 joint angles added to the raw (x, y, z) coordinates, can classify squats, push-ups, shoulder presses, and bicep curls in real time without the user manually selecting the exercise. The authors report 99.24% accuracy on their main test set, 95.05% on newly recorded home videos, and 87.91% on gym videos with harder angles and occlusions. They also report that the mixed angle-plus-coordinate input generalizes better than raw coordinates alone or angles alone, and that their model outperforms reimplemented prior approaches on the main and home test sets.","feed_headline":"BiLSTM exercise classifier tops 99% test accuracy","feed_subtitle":"Joint angles plus raw landmarks let a webcam app classify squats, presses, curls, and push-ups in real time.","key_machinery":"The load-bearing mechanism is the 30-frame sliding window of pose features fed into a two-layer Bidirectional LSTM. Each window contains the (x, y, z) coordinates of 22 relevant BlazePose landmarks plus 12 joint angles, including hip-shoulder-elbow, shoulder-elbow-wrist, hip-knee-ankle, shoulder-hip-knee, knee-ankle-heel, and ankle-heel-foot-index angles on both sides. The bidirectional pass lets the model read each moment in light of both earlier and later frames, so exercises that look similar at a single instant, like the start of a squat and a shoulder press, are separated by how the angles evolve over time. The same window is used for classification and feeds the rule-based angle thresholds that count repetitions once the exercise is known.","core_discovery":"The central discovery is that a BiLSTM fed 30-frame sequences of BlazePose landmark data, combining the 12 joint angles that are invariant to camera distance and body proportions with raw coordinates that help when landmarks are occluded, can recognize an exercise from its motion pattern alone. On the combined synthetic-and-real training set the model reaches over 99% accuracy on the held-out test split, and on two additional test sets that mimic real app use it reaches about 95% accuracy at home and about 88% in a gym. The paper presents this as evidence that temporal context plus perspective-invariant features solves the generalization problem that limits frame-wise and raw-coordinate classifiers. In the comparison experiments the mixed-feature BiLSTM scores highest on the main and home test sets; the paper concludes it is the best-performing approach.","pith_inferences":["The 99% test figure is only trustworthy if the 70-15-15 split separates whole videos, not individual 30-frame windows; the paper does not say which, so the number should be re-checked with a video-level split.","The drop from 99% to 88% on gym video hints that occluded joints break the angle features; a natural extension is to mask missing landmarks explicitly or train with random landmark dropout.","Because repetition counting is hand-coded angle thresholds while classification is learned, the two modules can disagree; an end-to-end counter trained from the same windows would remove that manual layer.","A direct transfer test on videos from sources never seen in training, such as new online uploads, would show whether the model generalizes beyond the particular clips used to build the dataset."],"forward_implications":["A real-time fitness app can recognize which of four exercises a user is doing without any manual selection, using only a laptop or phone webcam.","Combining viewpoint-invariant joint angles with raw coordinates is a better default feature set for pose-sequence classifiers than either feature type alone.","Sequence models that read 30 frames beat frame-wise or short-window alternatives on home-style video, so temporal context is the main driver of the improvement.","The same architecture can be extended to more exercises simply by adding labeled videos and retraining, reusing the existing angle-plus-coordinate pipeline."],"supporting_citations":[{"why":"Defines BlazePose, the 33-landmark pose estimator that produces every coordinate and angle in the input features.","marker":"[2]"},{"why":"Supplies the landmark-extraction pipeline that turns raw video frames into the pose inputs used for training and inference.","marker":"[3]"},{"why":"Describes the CNN-with-soft-voting baseline that the paper reimplements, and whose 30-frame windows and feature choices motivate the comparison.","marker":"[4]"},{"why":"Describes the raw-coordinate LSTM approach with 8-frame windows that the paper argues is too short and too coordinate-sensitive.","marker":"[5]"},{"why":"Describes the single-frame DNN with majority voting that the paper reimplements as a sequence-level baseline.","marker":"[6]"},{"why":"Supplies the InfiniteRep synthetic avatar videos, 100 per exercise class, used to enlarge the training set.","marker":"[7]"},{"why":"Supplies the real-world workout video dataset from which four exercise classes are drawn for training.","marker":"[10]"}],"fun_headline_variants":["BiLSTM plus angles hits 99% in exercise ID","Real-time rep counter: BiLSTM scores 99% accuracy","Webcam app: BiLSTM classifies 4 moves, 99% accurate","Angles and coordinates boost BiLSTM to 99% test score","Temporal BiLSTM for fitness: 99% accuracy, live demo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire 99% accuracy figure assumes the 70-15-15 split keeps whole videos apart; if 30-frame windows from the same video land in both training and test, the test set is not independent and the accuracy is inflated.","fun_headline_variants_meta":{"raw":{"variants":["BiLSTM plus angles hits 99% in exercise ID","Real-time rep counter: BiLSTM scores 99% accuracy","Webcam app: BiLSTM classifies 4 moves, 99% accurate","Angles and coordinates boost BiLSTM to 99% test score","Temporal BiLSTM for fitness: 99% accuracy, live demo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1433,"prompt_tokens":1031,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":647,"tokens_out":402,"duration_ms":4905,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:23:20.749542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Group every window by its source video ID, check that no video contributes windows to both training and test, then retrain from scratch with a video-level split and report test accuracy; if it drops well below 99%, the original claim did not measure generalization to unseen footage.","supporting_citations":[{"cited_title":"S., & Park, S","cited_arxiv_id":null,"evidence_quote":"Describes the CNN-with-soft-voting baseline that the paper reimplements, and whose 30-frame windows and feature choices motivate the comparison."},{"cited_title":"Muscle Vision: Real Time Keypoint Based Pose Classification of Physical Exercises","cited_arxiv_id":"2203.12111","evidence_quote":"Describes the raw-coordinate LSTM approach with 8-frame windows that the paper argues is too short and too coordinate-sensitive."},{"cited_title":"Recognizing Exercises and Counting Repetitions in Real Time","cited_arxiv_id":"2005.03194","evidence_quote":"Describes the single-frame DNN with majority voting that the paper reimplements as a sequence-level baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the InfiniteRep synthetic avatar videos, 100 per exercise class, used to enlarge the training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the real-world workout video dataset from which four exercise classes are drawn for training."}],"review_version":1}