REVIEW 3 major objections 5 minor 28 references
Representing and Detecting Label Ambiguity in IMU-Based Exercise Evaluation
T0 review · 3 major / 5 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read Training IMU exercise classifiers on full label distributions flags borderline repetitions without hurting classification accuracy.
desk verdict Solid classification parity with a useful AGLD construction, but the headline ambiguity-detection gains are partly circular because AGLD is both train target and eval reference. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Automatically Generated Label Distributions (AGLD): one hundred simulated raters that share the same physiotherapist-defined criteria but draw independent thresholds from normals around fitted means, so relative class frequencies become a soft target that encodes borderline ambiguity without a large human rater pool.
What would settle it
Gather a multi-rater exercise set in which each rater also marks every repetition as ambiguous or clear; if AGLD entropy and the network's predicted entropy then fail to rank the marked-ambiguous cases above the clear ones, the claim that AGLD captures real ambiguity fails.
Extended reading notes
Core claim
A network trained with a Kullback-Leibler objective to reproduce automatically generated label distributions matches or exceeds a one-hot cross-entropy baseline on macro-F1 for the leading class across four IMU exercise datasets, and recovers ambiguous repetitions and their two competing classes more reliably from the predicted output distribution.
Load-bearing premise
The method assumes that sampling independent criterion thresholds from normal distributions with fixed heuristic widths produces label distributions whose entropy and top classes faithfully match the ambiguity human raters would actually perceive.
Editorial extensions
If this is right
- Feedback systems can flag borderline repetitions and name both competing classes instead of forcing a single hard assignment.
- Label-distribution training adds usable ambiguity information at no cost to classification of the leading class.
- Predicted entropy can serve as a practical detector of ambiguous executions once a low threshold is chosen.
- The same AGLD construction applies to both nominal fault categories and ordinal FMS scores.
Reading between the lines
- Simulated-rater distributions could transfer to other criteria-based clinical scores that suffer boundary disagreement without needing multi-annotator pools.
- Calibrating the threshold widths against real multi-rater data would turn AGLD entropy into a drop-in proxy for interrater reliability on new exercises.
- Soft targets of this form may reduce the apparent arbitrariness patients feel when nearly identical movements receive opposite hard scores.
- Extending evaluation past the top-two classes is needed before multi-way ambiguity is fully represented.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Automatically Generated Label Distributions (AGLD) for IMU-based exercise evaluation: a rule-based labeling pipeline is run with criterion thresholds sampled from N(μc, σc²) to produce a class distribution per repetition without a large human rater pool. A CNN is trained with Kullback–Leibler divergence to match the full AGLD (ambiguity approach) and compared to a one-hot cross-entropy baseline that collapses AGLD to its mode. On four datasets (RD, RGS, DS, HS), the ambiguity approach matches or exceeds baseline macro-F1 for the most relevant class (Table 1) and yields higher Famb1 and Ftop21 across intermediate entropy thresholds (Figs. 3–4), with hierarchical outcome trees at τ = 0.47 bit. The authors conclude that representing label distributions in the training target adds ambiguity information at no cost to classification.
Significance. If the AGLD is accepted as a useful proxy for borderline executions, the work is practically relevant for home-based physiotherapy feedback systems: it offers a concrete way to flag repetitions between categories and report competing classes rather than forcing a hard label. Strengths include evaluation on four real IMU datasets, participant/class/entropy-stratified five-fold CV, metrics reported across the full τ range rather than a single threshold, and an explicit hierarchical breakdown of detection errors. Classification parity (or better) under distributional training is a clean, useful empirical result. The main scientific contribution is the AGLD construction plus the demonstration that KLD training recovers distributional structure without harming mode classification; the clinical claim about human-perceived ambiguity is more provisional.
major comments (3)
- [§2.4–2.6, Eqs. (1)–(2); abstract/conclusion] Sections 2.4, 2.6 and Eqs. (1)–(2): Famb1 and Ftop21 treat AGLD entropy H(p) and top2(p) as ground truth, while the ambiguity model is trained with DKL(p ∥ q) to reproduce that same p. The baseline collapses p to one-hot. Higher Famb1/Ftop21 is therefore largely expected once the network approximates p; it does not independently show recovery of clinical borderline cases. The abstract and conclusion should state this dependency explicitly (e.g., “more reliably recovers the AGLD-defined ambiguity”) or add an evaluation that does not use AGLD as both target and reference.
- [§3.5, Figure 6; §4.4] Section 3.5 and Figure 6 provide the only external check of AGLD entropy against human rater disagreement (DS/HS only). Precision stays ~0.3–0.4 across the informative τ range, and raters never labeled ambiguity directly—only scores. This is too weak to support the claim that the system “detects ambiguity … more reliably” in the clinical sense. Either collect direct ambiguous/unambiguous rater labels on a subset, or substantially qualify the claim and treat Figure 6 as a consistency check rather than validation.
- [§2.2, §4.5, Figure 7] Section 2.2 and Limitations 4.5: σc is fixed heuristically (5°, 5 cm, 0.5 m/s) with independent draws per criterion, and no sensitivity analysis is reported. Because AGLD entropy (and thus Famb1/Ftop21 and the τ = 0.47 split) is a direct function of σc (illustrated in Fig. 7), the ambiguity-detection results are not shown to be robust to this free parameter. A modest sensitivity study (e.g., scale σc by 0.5× and 2× and re-report Famb1/Ftop21 and classification) is needed for the load-bearing half of the main claim.
minor comments (5)
- [Abstract; Table 1; §4.1] HS FMS 1 has only eight repetitions (Fig. 1); the large Fcls1 gain (0.32→0.63 in Table 1) is correctly flagged as unreliable in §4.1 but still appears in the abstract’s unqualified “matched or exceeded … on all four datasets.” Soften the abstract wording for HS.
- [§2.4, §3.4] The entropy threshold τ = 0.47 bit is motivated by a 0.9/0.1 two-class split (§2.4), but for K = 3 or 4 the same entropy can arise from other mass allocations. A short note on this would help readers interpret the outcome trees.
- [§1; §2.4] Related work on multi-annotator and noisy-label methods is discussed, but soft-label / label-smoothing baselines (training with a fixed soft target not derived from AGLD) are not compared. Even a brief discussion would clarify what is specific to AGLD versus any soft target.
- [Figure 5; Appendix A] Figure 5 and Appendix A trees are informative but dense; stating absolute counts of true positives for top-2 in the main text (not only percentages of parents) would aid comparison across datasets.
- [Figure 5; Appendix A] Minor: “ambivalence” appears in the outcome-tree node labels (Fig. 5, A1–A3) while the paper elsewhere uses “ambiguity”; unify terminology.
Circularity Check
Famb1/Ftop21 gains largely show the KLD network can reproduce its own synthetic AGLD target rather than independently recover clinical ambiguity; classification parity is non-circular.
-
fitted input called prediction
[Sections 2.4 (Eq. 2), 2.6 (Famb1 / Ftop21 definitions), Results 3.2–3.3]
"The ambiguity approach retains the full label distribution and trains the network to reproduce it. It minimizes the Kullback-Leibler divergence from the AGLD to the network output, L(n)KLD = DKL(p(n) ∥ q(n)) ... Ambiguity detection measures how well an approach recovers from its softmax output whether a repetition is ambiguous. ... the reference is positive when H(p(n)) > τ and the prediction is positive when H(q(n)) > τ, and report the F1-score ... denoted Famb1. ... A reference repetition is positive when H(p(n)) > τ ... and a prediction counts as a true positive only when ... top2(q(n)) equ"
The KLD objective forces q toward p. Famb1 and Ftop21 then score exactly the distributional properties (entropy and ordered top-2) of that same p. Any network that successfully approximates the training target will therefore outperform a one-hot baseline that never sees the distribution; the reported gains are statistically forced once approximation succeeds and do not constitute an independent external test of clinical ambiguity recovery.
-
self definitional
[Section 2.2 (AGLD generation) + 2.5–2.6 (AGLD as sole reference for all metrics)]
"Each instance assigns a single class to a repetition, and the relative frequencies of these assignments across the 100 instances define its automatically generated label distribution (AGLD). ... The AGLD is the reference against which the trained network is evaluated in Section 2.6. ... We take the highest-probability class of the AGLD, arg max k p(n)k , as the reference ... a repetition is ambiguous when the entropy of its distribution exceeds a threshold τ. We apply this criterion to both distributions..."
AGLD is constructed by the authors' own rule-based procedure (thresholds fitted in prior self-work [23], then sampled with heuristic σc). That same synthetic object is then used both as the KLD training target and as the sole ground-truth reference for Fcls1, Famb1 and Ftop21. There is no independent external label of 'ambiguous' for the primary metrics; the detection claim therefore reduces to 'the model recovers the synthetic distribution we defined and trained it on.'
full rationale
The paper's strongest claim has two halves. Classification of the AGLD mode (Fcls1, Table 1) is a fair head-to-head: both the one-hot baseline and the KLD model are scored against the same argmax(p), and the ambiguity approach matching or exceeding the baseline is a genuine empirical result. The second half—detecting ambiguity and the relevant classes more reliably—is not. Sections 2.4 and 2.6 define the ambiguity approach as minimizing DKL(p || q) so that the network output q approximates the full AGLD p, then define Famb1 and Ftop21 by treating H(p) and top2(p) as ground truth and H(q)/top2(q) as the prediction. Once the network can approximate p, superior recovery of entropy and top-2 is expected by construction; the one-hot baseline is deliberately denied the distributional target. The only external check (Section 3.5 / Figure 6, rater disagreement on DS/HS) shows low precision (~0.3–0.4) and is not used for the main Famb1/Ftop21 claims. AGLD construction itself (thresholds fitted in prior self-work then sampled with heuristic σc) is a methodological contribution, not a circular derivation, but it becomes the sole reference for the detection claim. Score 6 reflects partial circularity confined to the ambiguity-detection half of the central claim; the paper is otherwise self-contained and does not smuggle uniqueness theorems or rename known results.
Assumptions & free parameters
free parameters (6)
- σc (angular criteria) =
5°
- σc (distance criteria) =
5 cm
- σc (velocity criteria) =
0.5 m/s
- entropy threshold τ for unambiguous/ambiguous split =
0.47 bit
- μc (best-fit criterion thresholds) =
per-criterion, from prior random search
- network hyperparameters (lr, weight decay, dropout, batch size, patience) =
as listed in §2.4
assumptions (4)
- ad hoc to paper Human rater disagreement on criteria-based exercise scores is adequately modeled by independent Gaussian noise on per-criterion kinematic thresholds.
- domain assumption Shannon entropy of the label distribution is a graded measure of clinical ambiguity, and a fixed τ separates ambiguous from unambiguous repetitions.
- domain assumption The most frequent class of the AGLD is a valid single-label reference for classification F1.
- domain assumption OpenSim inverse-kinematics from the given IMU setups yields kinematic quantities accurate enough for criterion evaluation.
invented entities (1)
-
Automatically Generated Label Distribution (AGLD)
Cite this review
Pith. "Pith review of Representing and Detecting Label Ambiguity in IMU-Based Exercise Evaluation." pith.science (2026). https://pith.science/paper/CWUBIYJC
@misc{pith2026260704842,
author = {Pith},
title = {Pith review of: Representing and Detecting Label Ambiguity in IMU-Based Exercise Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/CWUBIYJC}},
note = {Machine review of arXiv:2607.04842}
}
read the original abstract
Home-based physiotherapy is performed without supervision, which leads to incorrect execution and motivates systems that assess movement automatically from inertial measurement units (IMUs). Such systems assign each repetition to a category, yet a relevant share of repetitions falls near a class boundary, where even trained raters disagree. Classifiers trained with one-hot labels collapse these borderline repetitions onto a single class and discard this ambiguity. We address this with a method that automatically generates a label distribution per repetition without a large rater pool. We train a network to reproduce the full distribution with a Kullback-Leibler objective, the ambiguity approach, and compare it against a one-hot cross-entropy baseline on four IMU exercise datasets. From the network output we further determine whether a repetition is ambiguous and which classes are relevant to it. The ambiguity approach matched or exceeded the baseline classification on all four datasets, and detected ambiguity and the relevant classes more reliably. Representing the label distribution in the training target therefore adds information about ambiguity at no cost to classification.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Ashari, A.; Hamid, T.A.; Hussain, M.R.; Hill, K.D. Effectiveness of Individualized Home-Based Exercise on Turning and Balance Performance Among Adults Older than 50 yrs. American Journal of Physical Medicine & Rehabilitation 2016, 95, 355–365. https://doi.org/10.1097/phm.0000000000000388
-
[2]
Latham, N.K.; Harris, B.A.; Bean, J.F.; Heeren, T.; Goodyear, C.; Zawacki, S.; Heislein, D.M.; Mustafa, J.; Pardasaney , P .; Giorgetti, M.; et al. Effect of a Home-Based Exercise Program on Functional Recovery Following Rehabilitation After Hip Fracture: A Randomized Clinical Trial. JAMA 2014, 311, 700–708. https://doi.org/10.1001/jama.2014.469
-
[3]
Gelaw, A.Y .; Janakiraman, B.; Gebremeskel, B.F.; Ravichandran, H. Effectiveness of Home-based rehabil- itation in improving physical function of persons with Stroke and other physical disability: A systematic review of randomized controlled trials. Journal of Stroke and Cerebrovascular Diseases: The Official Journal of National Stroke Association 2020, 29...
work page doi:10.1016/j.jstrokecerebrovasdis.2020.104800 2020
-
[4]
Flynn, A.; Allen, N.E.; Dennis, S.; Canning, C.G.; Preston, E. Home-based prescribed exercise improves balance-related activities in people with Parkinson’s disease and has benefits similar to centre-based exer- cise: a systematic review. Journal of Physiotherapy 2019, 65, 189–199. https://doi.org/10.1016/j.jphys.2019 .08.003
-
[5]
Argent, R.; Daly , A.; Caulfield, B. Patient Involvement With Home-Based Exercise Programs: Can Con- nected Health Interventions Influence Adherence? JMIR mHealth and uHealth 2018, 6, e47. https: //doi.org/10.2196/mhealth.8518
-
[6]
Faber, M.; Andersen, M.H.; Sevel, C.; Thorborg, K.; Bandholm, T.; Rathleff, M. The majority are not per- forming home-exercises correctly two weeks after their initial instructionan assessor-blinded study . PeerJ 2015, 3, e1102. https://doi.org/10.7717/peerj.1102
-
[7]
Lang, S.; McLelland, C.; MacDonald, D.; Hamilton, D.F. Do digital interventions increase adherence to home exercise rehabilitation? A systematic review of randomised controlled trials. Archives of Physiotherapy 2022, 12, 24. https://doi.org/10.1186/s40945-022-00148-z
-
[8]
Automatic Assessment of Functional Movement Screening Exercises with Deep Learn- ing Architectures
Spilz, A.; Munz, M. Automatic Assessment of Functional Movement Screening Exercises with Deep Learn- ing Architectures. Sensors 2023, 23, 5. https://doi.org/10.3390/s23010005
Show all 28 references
-
[9]
Functional movement screening: the use of fundamen- tal movements as an assessment of function - part 1
Cook, G.; Burton, L.; Hoogenboom, B.J.; V oight, M. Functional movement screening: the use of fundamen- tal movements as an assessment of function - part 1. International journal of sports physical therapy 2014, 9, 396–409
2014
-
[10]
Functional movement screening: the use of funda- mental movements as an assessment of function-part 2
Cook, G.; Burton, L.; Hoogenboom, B.J.; V oight, M. Functional movement screening: the use of funda- mental movements as an assessment of function-part 2. International journal of sports physical therapy 2014, 9, 549–63. 21 of 21
2014
-
[11]
Test-Retest and Interrater Reliability of the Functional Movement Screen
Shultz, R.; Anderson, S.C.; Matheson, G.O.; Marcello, B.; Besier, T. Test-Retest and Interrater Reliability of the Functional Movement Screen. Journal of Athletic T raining 2013, 48, 331–336. https://doi.org/10.4085/ 1062-6050-48.2.11
2013
-
[12]
A systematic review of interrater reliability of pressure ulcer classification systems
Kottner, J.; Raeder, K.; Halfens, R.; Dassen, T. A systematic review of interrater reliability of pressure ulcer classification systems. Journal of Clinical Nursing 2009, 18, 315–336. https://doi.org/10.1111/j.1365-2702.20 08.02569.x
2009 doi
-
[13]
Histologic Grading of Breast Cancer: Linkage of Patient Outcome with Level of Pathologist Agreement
Dalton, L.W.; Pinder, S.E.; Elston, C.E.; Ellis, I.O.; Page, D.L.; Dupont, W.D.; Blamey , R.W. Histologic Grading of Breast Cancer: Linkage of Patient Outcome with Level of Pathologist Agreement. Modern Pathology 2000, 13, 730–735. https://doi.org/10.1038/modpathol.3880126
-
[14]
Label Distribution Learning
Geng, X. Label Distribution Learning. IEEE T ransactions on Knowledge and Data Engineering 2016, 28, 1734–
2016
-
[15]
https://doi.org/10.1109/TKDE.2016.2545658
2016 doi
-
[16]
Deep Label Distribution Learning With Label Ambiguity
Gao, B.B.; Xing, C.; Xie, C.W.; Wu, J.; Geng, X. Deep Label Distribution Learning With Label Ambiguity. IEEE T ransactions on Image Processing 2017, 26, 2825–2838. https://doi.org/10.1109/TIP .2017.2689998
2017 doi
-
[17]
Label Enhancement for Label Distribution Learning via Prior Knowledge
Gao, Y .; Zhang, Y .; Geng, X. Label Enhancement for Label Distribution Learning via Prior Knowledge. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20), 2020, pp. 3223–3229. https://doi.org/10.24963/ijcai.2020/446
2020 doi
-
[18]
Mitigating label noise through data ambiguation
Lienen, J.; Hüllermeier, E. Mitigating label noise through data ambiguation. In Proceedings of the Thirty- Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational ...
2024 doi
-
[19]
Beyond confusion matrix: learning from multiple annotators with awareness of instance features
Li, J.; Sun, H.; Li, J. Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Machine Learning 2023, 112, 1053–1075. https://doi.org/10.1007/s10994-022-06211-x
2023 doi
-
[20]
Using Ranking-CNN for Age Estimation
Chen, S.; Zhang, C.; Dong, M.; Le, J.; Rao, M. Using Ranking-CNN for Age Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5183–5192
2017
-
[21]
Automatic Evaluation of Functional Movement Screening Based on Attention Mechanism and Score Distribution Prediction
Lin, X.; Huang, T.; Ruan, Z.; Yang, X.; Chen, Z.; Zheng, G.; Feng, C. Automatic Evaluation of Functional Movement Screening Based on Attention Mechanism and Score Distribution Prediction. Mathematics 2023, 11, 4936. https://doi.org/10.3390/math11244936
2023 doi
-
[22]
Automatic Evaluation Method for Functional Movement Screening Based on a Dual-Stream Network and Feature Fusion
Lin, X.; Chen, R.; Feng, C.; Chen, Z.; Yang, X.; Cui, H. Automatic Evaluation Method for Functional Movement Screening Based on a Dual-Stream Network and Feature Fusion. Mathematics 2024, 12, 1162. https://doi.org/10.3390/math12081162
2024 doi
-
[23]
Automatic Evaluation Method for Functional Move- ment Screening Based on Multi-Scale Lightweight 3D Convolution and an EncoderDecoder
Lin, X.; Liu, Y .; Feng, C.; Chen, Z.; Yang, X.; Cui, H. Automatic Evaluation Method for Functional Move- ment Screening Based on Multi-Scale Lightweight 3D Convolution and an EncoderDecoder. Electronics 2024, 13, 1813. https://doi.org/10.3390/electronics13101813
2024 doi
-
[24]
Boosting Automatic Exercise Evaluation Through Musculoskeletal Simulation-Based IMU Data Augmentation, 2025
Spilz, A.; Oppel, H.; Munz, M. Boosting Automatic Exercise Evaluation Through Musculoskeletal Simulation-Based IMU Data Augmentation, 2025. arXiv:2505.24415 [cs], https://doi.org/10.48550/arXiv . 2505.24415
-
[25]
GAITEX: Human motion dataset of impaired gait and rehabilitation exercises using inertial and optical sensors
Spilz, A.; Oppel, H.; Werner, J.; Stucke-Straub, K.; Capanni, F.; Munz, M. GAITEX: Human motion dataset of impaired gait and rehabilitation exercises using inertial and optical sensors. Scientific Data 2025, 13, 11. https://doi.org/10.1038/s41597-025-06439-x
2025 doi
-
[26]
Xsens MVN: Full 6DOF human motion tracking using miniature inertial sensors
Roetenberg, D.; Luinge, H.; Slycke, P . Xsens MVN: Full 6DOF human motion tracking using miniature inertial sensors. Xsens Motion T echnol. BV T ech. Rep.2009, 3
2009
-
[27]
OpenSim: open-source software to create and analyze dynamic simulations of movement
Delp, S.L.; Anderson, F.C.; Arnold, A.S.; Loan, P .; Habib, A.; John, C.T.; Guendelman, E.; Thelen, D.G. OpenSim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on bio-medical engineering 2007, 54, 1940–1950. https://doi.org/10.11...
2007 doi
-
[28]
Animating rotation with quaternion curves
Shoemake, K. Animating rotation with quaternion curves. SIGGRAPH Comput. Graph. 1985, 19, 245–254. https://doi.org/10.1145/325165.325242. Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) a...
1985 doi
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.