REVIEW 3 major objections 6 minor 1 cited by
ExeChecker: Where Did I Go Wrong?
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read ExeChecker claims that a triplet-trained graph attention transformer can identify which joints caused an incorrect rehabilitation exercise, by reading the model's cross-spacetime attention scores.
desk verdict Useful dataset, but the JoA metric only measures recall, so the paper's central quantitative claim about joint localization is not supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the cross-spacetime attention map of a spatial-temporal graph attention transformer. At each timestamp, every joint attends not only to all joints in the current pose but also to all joints in $\tau=3$ neighbouring poses, giving a $\tau \times N \times N$ attention structure that is averaged over $H=8$ heads; for feedback, the map is read at the center frame and each joint's received attention is summed into a score. The other essential mechanism is the triplet ratio loss with in-triplet hard-negative mining and anchor swapping, which keeps the embedding training stable and forces the network to attend to the differences that separate correct from incorrect movement.
What would settle it
Take a correct exercise sequence and synthetically corrupt only one joint, such as the left knee, to create incorrect sequences; train ExeChecker and check whether the top-scoring joint is the corrupted one across many trials. Alternatively, mask the joints with the highest attention scores by replacing them with their correct positions and see whether the embedding still classifies the sequence as incorrect; if it does, the attention is not pointing at the cause.
Extended reading notes
Core claim
The paper's central claim is that the cross-spacetime attention learned by a spatial-temporal graph attention transformer can localize the joints that cause an exercise to be incorrect, not merely separate correct from incorrect sequences. ExeChecker builds an embedding with a triplet ranking loss over correctly executed anchor and positive sequences and incorrectly executed negative sequences; because the same network that produces the embedding also emits attention over joints and neighbouring frames, the attention values can be read as per-joint importance. The authors define joints of attention for each exercise from therapist knowledge, average the eight attention heads, take the center frame of the sequence, and sum the attention each joint receives to obtain raw joint scores. On the collected ExeCheck dataset and on UI-PRMD, these scores match the therapist-defined joints of attention better than hop-adjusted canonical time warping on most exercises, and the qualitative examples show the model catching subtle trunk and shoulder errors that alignment-based scores miss.
Load-bearing premise
The joint-level feedback stands on the assumption that the attention values, averaged over eight heads and read only from the center frame, identify the joints that caused the error rather than merely joints that differ between correct and incorrect videos.
Editorial extensions
If this is right
- A home-rehabilitation system could give joint-specific feedback from a single RGB-D camera, using only paired correct/incorrect recordings and no per-joint labels at training time.
- The joint-of-attention scoring metric gives a quantitative way to compare how well different methods localize movement errors, not just how well they classify them.
- On both ExeCheck and UI-PRMD, attention-based scores beat hop-adjusted canonical time warping for most exercises, so attention interpretability is a usable alternative to pairwise temporal alignment.
- The reprocessed UI-PRMD skeletons and new joint-of-attention annotations extend an existing public dataset to joint-level evaluation.
- The method's sensitivity to subtle errors such as hunching shoulders or a wobbling trunk suggests it can flag the small, therapist-relevant mistakes that large pose differences do not expose.
Reading between the lines
- One extension the paper leaves implicit: the per-frame attention could be aggregated over the time window where the error actually occurs instead of the single center frame, which may improve localization for exercises whose mistake happens early or late in the movement.
- A direct stress test of the attention-as-explanation assumption would be to synthesize incorrect sequences by perturbing only one known joint; if ExeChecker's top-scoring joint tracks the perturbed joint across many such sequences, the attention map is causal rather than merely discriminative.
- The same triplet recipe could be applied to other paired movement data such as sports technique or dance wherever a coach can record one correct and several flawed executions, turning expert demonstrations into weak supervision for joint-level feedback.
- Because the embedding separates correct from incorrect sequences, the attention scores could also be used for continuous error grading, estimating how far a joint's motion is from the correct trajectory, which the paper does not attempt.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ExeChecker, a triplet-network framework built on a spatial-temporal graph attention transformer (STGAT) for rehabilitation exercise analysis. Given paired correct and incorrect exercise recordings, the model learns embeddings that separate the two classes and, at inference, uses the cross-spacetime attention map to score joints and highlight those most likely responsible for the error. The authors contribute the ExeCheck dataset of paired correct/incorrect RGB-D recordings for ten exercises with therapist-defined joint-of-attention (JoA) annotations, a JoA scoring metric (Eq. 5), and comparisons against a hop-adjusted canonical time warping (CTW) baseline on ExeCheck and UI-PRMD. They report that ExeChecker outperforms CTW on 9 of 10 ExeCheck exercises and on all but one UI-PRMD exercise.
Significance. If the quantitative claims held, ExeChecker would be a useful step toward interpretable, joint-level feedback in home rehabilitation, and the released dataset and annotations would be a valuable community resource. The design has real merits: JoA labels are not used during training, evaluation is performed on held-out subjects and on a second dataset, and the method is compared with a nontrivial alignment baseline. The paper also openly states limitations (restricted mistake types, planned user study). However, the evaluation metric and the experimental protocol currently do not support the central claim of superior joint localization; the significance is therefore conditional on a corrected evaluation.
major comments (3)
- [Section 3.3, Eq. (5)] The JoA score in Eq. (5) computes the average predicted score over only the expert-defined JoA joints; non-JoA joints enter the sum only with s_i=0, so false positives are never penalized. After min-max normalization, a model that assigns its maximum score to a single true JoA joint and arbitrarily high scores to all other joints receives S_JoA=1, regardless of how many irrelevant joints were highlighted. Tables 2 and 3 therefore measure recall of the JoA set rather than localization accuracy, and the abstract's claim that ExeChecker 'outperformed the baseline ... in identifying joints of physical relevance' is not supported by these numbers. Please re-evaluate with a metric that penalizes false positives (e.g., precision/recall at top-k or joint-level IoU) and report both components.
- [Section 5.2, Tables 2 and 3] All quantitative results are computed on a single held-out subject (Subject 7 for ExeCheck, Subject 10 for UI-PRMD) with one model run per configuration. Tables 2-3 report point estimates with no variance or significance tests, and several differences are small (e.g., ExeCheck 2-HKR 0.402 vs 0.356; UI-PRMD m02 0.454 vs 0.479, where CTW actually wins). Because the central claim is comparative, please report results across multiple train/test subject splits or repeated training runs, with confidence intervals or paired tests.
- [Section 5.2, attention-to-joint mapping] The paper asserts without validation that the center-frame, head-averaged cross-spacetime attention map faithfully indicates which joints caused the incorrectness. If the attention map mainly captures general discriminative differences between the correct and incorrect sequences, the highlighted joints need not be the cause of the error, especially because the paired incorrect performances were instructed to differ at exactly the JoA joints. Please provide evidence for the faithfulness of the attention proxy, for example an ablation against another attribution method or a comparison of attention with kinematic error on held-out error types.
minor comments (6)
- [Abstract and Section 4] Typos should be corrected: 'interpretablity' in the abstract and 'weaing' in Section 4.
- [Section 3.1, Eq. (1)] The notation around Eq. (1) is unclear: please define sigma, W_h, X_tau, and the exact shape of A_tau, and fix the sentence 'A t τ is τ×N×N in dimension.'
- [Section 3.3] The min-max normalization is not defined for the case where all raw scores are equal; please specify how ties and zero-denominator cases are handled.
- [Table 1] The entry for 5-SA appears to read 'neck lShoulder' without a clear separator; please reformat the table so each joint is a distinct entry.
- [Section 5.4] Figures 3 and 4 are referenced but not included in the text, and exercises 6-9 are deferred to supplementary; please include all qualitative results or clearly mark them as supplementary-only.
- [Section 6] The concluding limitation statement is welcome, but the evaluation section should also discuss the construction confound: because participants were instructed to err at specific joints, the JoA labels coincide with the kinematic differences intentionally introduced.
Circularity Check
No circularity: the contrastive model never sees JoA labels and is evaluated on held-out and external data; the JoA metric weakness is a validity issue, not a circular reduction.
full rationale
ExeChecker's derivation chain is not circular. The model is trained with a triplet contrastive loss on correct-versus-incorrect exercise sequences; the joint-level JoA labels are never used as training targets. At inference, per-joint scores come from the STGAT cross-spacetime attention map, averaged over heads and read at the center frame. These scores are then compared with expert-defined JoA sets through Eq. (5). No parameter is fitted to the JoA labels, and no equation defines the predicted scores in terms of the ground-truth s_i. Evaluation on a held-out subject (subject 7 of ExeCheck) and on the external UI-PRMD dataset, with JoAs derived from the UI-PRMD paper's error descriptions, provides independent grounding. The S_JoA metric in Eq. (5) is a recall-style measure that does not penalize false positives, so the quantitative claim of better localization is weakened as a validity matter, but this is not a circularity. Similarly, the fact that incorrect performances were generated by purposely making the listed mistakes means the discriminative signal is correlated with the JoA labels; that is a data-design limitation, not an equation-level reduction. Self-citations to the authors' prior ExerciseCheck platform appear only in motivation and future-work context and are not load-bearing. Therefore no circular step is present.
Assumptions & free parameters
free parameters (4)
- CTW hop-adjustment exponent =
1/(h_i + 1)
- center frame for attention extraction =
center frame of the input sequence
- attention aggregation over heads =
average of 8 heads, sum over joints
- top_k visualization threshold =
5
assumptions (5)
- domain assumption STGAT attention weights are interpretable as joint importance
- domain assumption Expert JoA annotations are ground truth
- domain assumption Paired correct/incorrect recordings isolate the error joints
- domain assumption Triplet ratio loss yields a discriminative embedding for correct vs incorrect movement
- domain assumption PoseFormerV2 poses on UI-PRMD are accurate enough
invented entities (1)
-
Joints of Attention (JoA)
Cite this review
Pith. "Pith review of ExeChecker: Where Did I Go Wrong?." pith.science (2026). https://pith.science/paper/P7LIHOEG
@misc{pith2026241210573,
author = {Pith},
title = {Pith review of: ExeChecker: Where Did I Go Wrong?},
year = {2026},
howpublished = {\url{https://pith.science/paper/P7LIHOEG}},
note = {Machine review of arXiv:2412.10573}
}
read the original abstract
In this paper, we present a contrastive learning based framework, ExeChecker, for the interpretation of rehabilitation exercises. Our work builds upon state-of-the-art advances in the area of human pose estimation, graph-attention neural networks, and transformer interpretablity. The downstream task is to assist rehabilitation by providing informative feedback to users while they are performing prescribed exercises. We utilize a contrastive learning strategy during training. Given a tuple of correctly and incorrectly executed exercises, our model is able to identify and highlight those joints that are involved in an incorrect movement and thus require the user's attention. We collected an in-house dataset, ExeCheck, with paired recordings of both correct and incorrect execution of exercises. In our experiments, we tested our method on this dataset as well as the UI-PRMD dataset and found ExeCheck outperformed the baseline method using pairwise sequence alignment in identifying joints of physical relevance in rehabilitation exercises.
Figures
Forward citations
Cited by 1 Pith paper
-
A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions
A systematic review of Action Quality Assessment organizes the past decade of research into 7 trends, 9 dataset domains, and performance comparisons across 195 papers.
Reference graph
Works this paper leans on
-
[1]
In: British Machine Vision Conference (BMVC) (2016).https://doi.org/10.5244/C.30.119, 11 pages
Balntas,V.,Riba,E.,Ponsa,D.,Mikolajczyk,K.:Learninglocalfeaturedescriptors with triplets and shallow convolutional neural networks. In: British Machine Vision Conference (BMVC) (2016).https://doi.org/10.5244/C.30.119, 11 pages
doi:10.5244/c.30.119 2016
-
[2]
IEEE Transactions on Neural Systems and Rehabilitation Engineering27(7), 1436–1448 (July 2019)
Capecci, M., Ceravolo, M.G., Ferracuti, F., Iarlori, S., Monteriù, A., Romeo, L., Verdini, F.: The KIMORE Dataset: KInematic Assessment of MOvement and Clin- ical Scores for Remote Monitoring of Physical REhabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering27(7), 1436–1448 (July 2019). https://doi.org/10.1109/TNSRE.2019.2923060
arXiv 2019
-
[3]
Deb, S., Islam, M.F., Rahman, S., Rahman, S.: Graph convolutional networks for assessment of physical rehabilitation exercises. IEEE Transactions on Neural Systems and Rehabilitation Engineering30, 410–419 (2022).https://doi.org/ 10.1109/TNSRE.2022.3150392 ExeChecker 15
arXiv 2022
-
[4]
D’Innocente, A., Garg, N., Zhang, Y., Bazzani, L., Donoser, M.: Localized triplet loss for fine-grained fashion image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 3910–3915 (June 2021)
work page 2021
-
[5]
Nature Communications14(8294), 1–11 (2023).https: //doi.org/10.1038/s41467-023-44141-x
Gao,Q.,Yao,S.,Tian,Y.,Zhang,C.,Zhao,T.,Wu,D.,Yu,G.,Lu,H.:Automating General Movements Assessment with quantitative deep learning to facilitate early screening of cerebral palsy. Nature Communications14(8294), 1–11 (2023).https: //doi.org/10.1038/s41467-023-44141-x
-
[6]
Gu, Y., Pandit, S., Saraee, E., Nordahl, T., Ellis, T., Betke, M.: Home-based phys- ical therapy with an interactive computer vision system. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (Oct 2019).https://doi.org/10.1109/ICCVW.2019.00320
-
[7]
arXiv preprint arXiv:1703.07737 (2017).https://doi.org/10
Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re- identification. arXiv preprint arXiv:1703.07737 (2017).https://doi.org/10. 48550/arXiv.1703.07737
-
[8]
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: Similarity- based pattern recognition: Third International Workshop, SIMBAD 2015, Copen- hagen, Denmark, October 12-14, 2015. Proceedings 3. pp. 84–92. Springer (2015). https://doi.org/10.1007/978-3-319-24261-3_7
Show all 31 references
-
[9]
Expert Systems with Applications232, 120683 (2023).https://doi.org/10.1016/j.eswa.2023.120683
Hu, L., Liu, S., Feng, W.: Skeleton-based action recognition with local dynamic spatial–temporal aggregation. Expert Systems with Applications232, 120683 (2023).https://doi.org/10.1016/j.eswa.2023.120683
2023
-
[10]
IEEE Transactions on Neural Systems and Rehabilita- tion Engineering28(2), 468–477 (2020).https://doi.org/10.1109/TNSRE.2020
Liao, Y., Vakanski, A., Xian, M.: A deep learning framework for assessing physical rehabilitation exercises. IEEE Transactions on Neural Systems and Rehabilita- tion Engineering28(2), 468–477 (2020).https://doi.org/10.1109/TNSRE.2020. 2966249
2020 doi
-
[11]
Microsoft Azure Kinect,https://learn.microsoft.com/en-us/azure/kinect- dk/, accessed July 2024
2024
-
[12]
Com- puters in Biology and Medicine165, 107420 (2023).https://doi.org/10.1016/ j.compbiomed.2023.107420
Mourchid, Y., Slama, R.: D-stgcnt: A dense spatio-temporal graph conv-gru net- work based on transformer for assessment of patient physical rehabilitation. Com- puters in Biology and Medicine165, 107420 (2023).https://doi.org/10.1016/ j.compbiomed.2023.107420
2023
-
[13]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (June 2019)
Ogata, R., Simo-Serra, E., Iizuka, S., Ishikawa, H.: Temporal distance matrices for squat classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (June 2019)
2019
-
[14]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Pan, J.H., Gao, J., Zheng, W.S.: Action assessment by joint relation graphs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6331–6340 (2019).https://doi.org/10.1109/ICCV.2019.00643
2019
-
[15]
In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments
Pandit, S., Tran, S., Gu, Y., Saraee, E., Jansen, F., Singh, S., Cao, S., Sadeghi, A., Shandelman, E., Ellis, T., et al.: ExerciseCheck: A scalable platform for remote physical therapy deployed as a hybrid desktop and web application. In: Proceedings of the 12th ACM Internatio...
2019 doi
-
[16]
In: European Conference on Com- puter Vision (ECCV)
Parmar, P., Gharat, A., Rhodin, H.: Domain knowledge-informed self-supervised representations for workout form assessment. In: European Conference on Com- puter Vision (ECCV). pp. 105–123. Springer (2022).https://doi.org/10.1007/ 978-3-031-19839-7_7
2022
-
[17]
In: Proceedings of the IEEE/CVF Confer- 16 Y
Parmar, P., Morris, B.T.: What and how well you performed? A multitask learning approach to action quality assessment. In: Proceedings of the IEEE/CVF Confer- 16 Y. Guet al. ence on Computer Vision and Pattern Recognition (CVPR). pp. 304–313 (2019). https://doi.org/10.1109/CVP...
2019
-
[18]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Parmar, P., Tran Morris, B.: Learning to Score Olympic Events. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 20–28 (2017).https://doi.org/10.48550/arXiv.1611.05125
-
[19]
Pattern Recognition53, 130–147 (2016).https://doi.org/10.1016/j.patcog
Presti,L.L.,LaCascia,M.:3dskeleton-basedhumanactionclassification:Asurvey. Pattern Recognition53, 130–147 (2016).https://doi.org/10.1016/j.patcog. 2015.11.019
2016 doi
-
[20]
In: 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)
Réby, K., Dulau, I., Dubrasquet, G., Aimar, M.B.: Graph transformer for phys- ical rehabilitation evaluation. In: 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG). pp. 1–8. IEEE (2023).https: //doi.org/10.1109/FG57933.2023.10042778
2023
-
[21]
In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments
Saraee, E., Gu, Y., Pandit, S., Tran, S., Shandelman, E., Singh, S., Nordahl, T.J., Ellis, T., Betke, M.: ExerciseCheck: Data analytics for a remote monitoring and evaluation platform for home-based physical therapy. In: Proceedings of the 12th ACM International Conference on ...
2019
-
[22]
Sensors20(18), 5258 (2020).https: //doi.org/10.3390/s20185258
Sardari, F., Paiement, A., Hannuna, S., Mirmehdi, M.: Vi-net—view-invariant quality of human movement assessment. Sensors20(18), 5258 (2020).https: //doi.org/10.3390/s20185258
2020 doi
-
[23]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 815–823 (2015)
2015
-
[24]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Tang, Y., Ni, Z., Zhou, J., Zhang, D., Lu, J., Wu, Y., Zhou, J.: Uncertainty-aware score distribution learning for action quality assessment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9839–9848 (2020).https://doi.org/10.1...
2020
-
[25]
Data3(1) (2018).https://doi.org/10.3390/ data3010002, 15 pages
Vakanski, A., Jun, H.P., Paul, D., Baker, R.: A data set of human body movements for physical rehabilitation exercises. Data3(1) (2018).https://doi.org/10.3390/ data3010002, 15 pages
2018
-
[26]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2014)
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., Wu, Y.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2014)
2014
-
[27]
In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV)
Wang, X., Li, J., Hu, H.: Skeleton-based action quality assessment via partially connected lstm with triplet losses. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV). pp. 220–232. Springer (2022).https://doi.org/ 10.1007/978-3-031-18913-5_17
2022 doi
-
[28]
Neurocomputing537, 164– 186 (2023).https://doi.org/10.1016/j.neucom.2023.03.001
Xin, W., Liu, R., Liu, Y., Chen, Y., Yu, W., Miao, Q.: Transformer for skeleton- based action recognition: A review of recent advances. Neurocomputing537, 164– 186 (2023).https://doi.org/10.1016/j.neucom.2023.03.001
2023 doi
-
[29]
IEEE Transactions on Multimedia23, 1640–1653 (2020).https://doi.org/10.1109/TMM.2020.3001527
Yao, X., She, D., Zhang, H., Yang, J., Cheng, M.M., Wang, L.: Adaptive deep met- ric learning for affective image retrieval and classification. IEEE Transactions on Multimedia23, 1640–1653 (2020).https://doi.org/10.1109/TMM.2020.3001527
2020
-
[30]
In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhao, Q., Zheng, C., Liu, M., Wang, P., Chen, C.: PoseFormerV2: Exploring Fre- quency Domain for Efficient and Robust 3D Human Pose Estimation. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8877–8886 (2023).https://doi.org/10.4855...
-
[31]
Advances in Neural Information Processing Systems (NIPS)22(2009), 9 pages
Zhou, F., de la Torre, F.: Canonical time warping for alignment of human behavior. Advances in Neural Information Processing Systems (NIPS)22(2009), 9 pages
2009
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.