REVIEW 3 major objections 4 minor 57 references
FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Current multimodal language models cannot reliably assess the quality of bodyweight exercises, and the paper argues that visual perception — not domain reasoning — is the limiting factor.
desk verdict A carefully built fitness-AQA benchmark whose headline bottleneck result is undercut by a circular perception-oracle experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the paired perception and judgement question. For each quality aspect, a perception question asks which neutral visual manifestation is present, with option A defined as the correct-execution manifestation, and the matched judgement question asks whether that aspect is correct, with Yes derived from option A. The paper varies what information the model receives before answering the judgement question — no extra input, the perception question alone, the model's own perception answer, or the ground-truth perception answer — and the jump in judgement performance under ground-truth perception is what identifies perception as the bottleneck. The unified form-error taxonomy, with 38 errors across alignment, symmetry, stability, coordination, tempo, and completeness, is the shared label space that makes this cross-exercise decomposition possible.
What would settle it
Build judgement questions whose correct answer is not determined by the perception option — for example, where correctness depends on a threshold or combination that no single option encodes — and run the condition that gives the perception answer without the video. If judgement F1 still reaches the 94-96% range reported in Table 2, the perception-bottleneck conclusion is an artifact; the existing no-video row already approaches that range with no visual input at all.
Extended reading notes
Core claim
The paper claims that current MLLMs still struggle to assess exercise quality comprehensively and to localize form errors precisely, with visual perception as the key bottleneck. On FitAQA, the best model reaches 54.3% question-macro accuracy on perception, 72.1% F1 on judgement, and 27.0% recall at an IoU threshold of 0.7 on temporal grounding; supplying ground-truth perception answers raises the best judgement F1 to 94.4%. The paper also reports that open-source models are at or below the uniform-random baseline in judgement F1, that models tend to overpredict correct execution, and that temporal grounding degrades sharply under stricter overlap thresholds. These results are presented as evidence that capabilities demonstrated on general video understanding benchmarks do not transfer to recognizing and localizing quality-relevant posture and motion evidence.
Load-bearing premise
The bottleneck conclusion assumes that giving the model the correct perception answer provides only visual evidence and not the judgement answer; but because the correct-execution option was always labeled A and maps directly to 'Yes', the supplied answer may already encode the judgement, so the improvement may come from that leak rather than from better perception.
Editorial extensions
If this is right
- If the central finding holds, no evaluated MLLM is ready for unsupervised fitness coaching: each model falls at or below simple baselines on at least one of the three tasks.
- Because perception is identified as the bottleneck, improving fine-grained visual evidence recognition — joint angles, motion trajectories, postural deviations — should matter more for fitness AQA than improving reasoning.
- General video understanding benchmarks cannot be taken as evidence of quality-sensitive assessment; recognizing an action is different from judging how well it is performed.
- Temporal grounding of form errors requires more than coarse action semantics; models locate the exercise but not the specific repetitions that deviate from correct form.
Reading between the lines
- Editorial inference: the perception-oracle experiment may be confounded, because option A was defined as the correct-execution manifestation and maps directly to the judgement answer 'Yes'; the GT-perception-no-video row in Table 2, which reaches 95.8% F1 without any video, suggests the supplied perception answer already contains the judgement answer.
- Editorial inference: a cleaner test of the bottleneck would decouple the perception question from the judgement label, for example by asking for a neutral kinematic quantity (such as knee angle in degrees) and providing the acceptable range only in the judgement stage.
- Editorial inference: the per-dimension results suggest a diagnostic route for model improvement: symmetry and load-distribution errors are consistently hard, so targeted training data for those aspects may yield the largest gains.
- Editorial inference: the observed over-localization tendency (median predicted duration above ground truth for five of six models) implies that strict IoU thresholds may be punishing systematic temporal over-coverage as much as true localization error, so a calibration-based metric could give a complementary view.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FitAQA, a benchmark for evaluating multimodal large language models (MLLMs) on fitness action quality assessment. It contains 2,219 videos and 5,512 QA instances across 30 bodyweight exercises, with a unified taxonomy of 38 form errors in six quality dimensions. The benchmark defines three tasks: perception, judgement, and temporal grounding. The authors evaluate a range of open-source and closed-source MLLMs and report that current models perform weakly, especially on perception and precise temporal localization. A controlled perception-oracle experiment is used to argue that visual perception is a key bottleneck because judgement improves substantially when ground-truth perceptual evidence is supplied.
Significance. If the results stand, FitAQA would be a valuable resource for the MLLM and sports-AI communities. The dataset construction is careful: expert-reviewed annotations, a blinded 10% cross-check with 98.4% exact-answer agreement for perception and 97.3% mIoU for temporal grounding, shuffled option order, and multiple simple baselines (always-correct, always-error, whole-video). The main finding that current MLLMs are weak at fitness AQA and temporal grounding is supported by the reported numbers. However, the more novel bottleneck claim is not supported: the perception-oracle experiment leaks the judgement label through the fixed option-to-answer mapping, and the paper's own no-video rows (Table 7) already show that strong priors can produce high judgement F1 without visual input. The benchmark contribution can be salvaged, but the controlled-experiment conclusion needs to be either fixed with a decoupling control or substantially reframed.
major comments (3)
- [§Controlled Perception-Oracle Evaluation (Table 2) and Appendix 'Answer Inference'] The central bottleneck conclusion rests on comparing baseline judgement with the GT-perception setting. However, during dataset construction the judgement answer is deterministically derived from the perception answer (option A maps to Yes, non-A options map to No; Appendix 'Answer Inference'). The ground-truth perception answer is therefore logically equivalent to the judgement label, and the 'GT-perception, no video' row in Table 2, which reaches 95.8% F1 for GPT-5.5, shows that the model can recover the judgement answer without any video. This means the comparison between baseline and GT-perception does not isolate visual perception; it at least partly measures whether the model exploits the option-to-label mapping. The stress-test concern is valid: the claim that 'visual perception is a key bottleneck' is not supported by this experiment as designed. A decoupling control is needed, for example a perception oracle whose evidence does not uniquely determine the judgement answer, or a human-subject check that the perception options are not semantically transparent about correctness. The paper should also explicitly discuss the no-video GT-perception row as a limitation.
- [§Appendix 'Textual Bias Analysis' (Table 7)] The no-video controls in Table 7 show that GPT-5.5 achieves a judgement F1 of 68.3% without any video, which is higher than its video-based baseline F1 of 57.8%. The paper attributes this to a prediction bias toward the error class. This prior also contaminates the oracle comparison: in the GT-perception no-video setting, the model can combine its prior with the leaked perception answer and reach 95.8% F1. Before interpreting Table 2 as evidence of a perception bottleneck, the paper must analyze how this strong no-video prior interacts with the oracle manipulation, rather than treating the no-video controls only as a sanity check for perception and grounding tasks.
- [§Controlled Perception-Oracle Evaluation (Figure 5)] The monotonic relationship in Figure 5 between perception-oracle accuracy and judgement accuracy is presented as evidence of a positive dependence of judgement on perception quality. Because the oracle answers are generated from the same fixed option-to-label mapping, increasing oracle accuracy means the model is increasingly often told the correct judgement label; the monotonicity is therefore partly tautological. A more informative control would vary the correctness of perception evidence that is not label-equivalent, for example by providing a corrupted visual-state description rather than an option letter.
minor comments (4)
- [§Controlled Perception-Oracle Evaluation] The sentence 'supplying the perception question and its answer only provides perception evidence, leaving the model to reason with domain knowledge' is contradicted by the dataset-construction mapping and should be removed or replaced with a carefully qualified statement.
- [Table 1] In Table 1, the Qwen3.5-27B row contains the values '37.463.447.1' without separators; the formatting should be corrected to match the other rows.
- [§Dataset Statistics] The main text states that 95 temporal-grounding instances require localizing multiple intervals, but it does not give the denominator or the distribution across the 388 instances; please add this breakdown.
- [§Experiments, Evaluated Models] The paper evaluates closed-source models at 2 fps and UCF101 at its native 1 fps, while Table 9 studies frame-rate effects only for Qwen3.5-27B; a sentence noting this limitation would help readers interpret the cross-model comparisons.
Circularity Check
The GT-perception oracle is the judgement label under a fixed construction mapping, so the controlled experiment's perception-bottleneck conclusion is circular.
-
self definitional
[Section 'Question and Answer Generation' and Appendix 'Answer Inference'; evaluated in 'Controlled Perception-Oracle Evaluation' (Table 2, Prompt 6)]
"We then automatically derive the judgement answer from the perception answer: option A maps to Yes, non-A options map to No, and abstained perception answers remain abstained."
The controlled experiment injects the ground-truth perception answer as 'perception evidence' and measures judgement. But by construction the judgement label is a deterministic function of that answer: the correct-execution option maps to Yes and every other option maps to No. The paper's caveat that the options are 'neutral' does not remove this reduction, because experts revised the options so that each option has an unambiguous semantic mapping to the matched judgement answer. Thus GT-perception is not independent perceptual evidence; it is an encoding of the judgement label.
full rationale
FitAQA itself is an independent benchmark: the perception, judgement, and temporal-grounding questions are constructed from expert video annotations, and the main results showing that current MLLMs perform near or below simple baselines do not depend on the contested oracle experiment. The circularity is localized to the 'Controlled Perception-Oracle Evaluation' and the abstract/conclusion claim that visual perception is a key bottleneck. There, the paper explicitly derives the judgement answer from the perception answer during construction, then asserts that supplying the perception question and its answer 'only provides perception evidence.' That assertion is false by the paper's own reduction: the ground-truth perception answer determines the judgement label through the option-to-answer mapping, so the GT-perception setting is equivalent to giving the model the answer key. The no-video control in Table 2 (GPT-5.5 F1=95.8 without video) demonstrates the label is recoverable from the oracle alone. Thus the headline bottleneck conclusion is supported only by a self-definitional experiment. The benchmark results that MLLMs struggle on FitAQA remain valid, but the perceptual-bottleneck claim is not independently established. No load-bearing self-citation or uniqueness-theorem issue is present; the only circularity is this constructional label leakage.
Assumptions & free parameters
assumptions (4)
- domain assumption Expert sports-science annotations are treated as ground truth for form errors and temporal intervals.
- domain assumption The unified 38-error taxonomy is a complete and accurate representation of recurring bodyweight-exercise form errors.
- ad hoc to paper Perception question option A corresponds exactly to correct execution, making the perception-answer to judgement-answer mapping valid.
- domain assumption 10% blinded cross-check agreement generalizes to the full dataset.
Cite this review
Pith. "Pith review of FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/AX5SBPTZ
@misc{pith2026260808736,
author = {Pith},
title = {Pith review of: FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/AX5SBPTZ}},
note = {Machine review of arXiv:2608.08736}
}
read the original abstract
Fitness Action Quality Assessment (AQA) is important for intelligent sports training, yet the capabilities of Multimodal Large Language Models (MLLMs) in this setting remain underexplored. Existing benchmarks rely on action-specific annotation schemes and focus primarily on final assessment outputs, offering limited insight into how models assess exercise quality. We introduce FitAQA, a systematic benchmark for evaluating MLLMs in fitness AQA, containing 2,219 videos and 5,512 QA instances across 30 bodyweight exercises. In collaboration with experts in sports science, we develop a unified form error taxonomy that defines 38 recurring form errors within six complementary quality dimensions: alignment, symmetry, stability, coordination, tempo, and completeness. This taxonomy provides a shared assessment framework across different exercises. FitAQA further formulates three evaluation tasks: perception for recognizing relevant visual evidence, judgement for combining that evidence with domain knowledge to assess execution correctness, and temporal grounding for localizing form errors over time. Extensive evaluation shows that current MLLMs still struggle to assess exercise quality comprehensively and localize form errors precisely. Controlled experiments further indicate that visual perception is a key bottleneck, as judgement performance improves substantially when ground-truth perceptual evidence is provided. The dataset and evaluation code will be made publicly available.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=
Quo vadis, action recognition? a new model and the kinetics dataset , author=. proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=
-
[2]
Salehi, Mohammadreza and Park, Jae S and Yadav, Tanush and Kusupati, Aditya and Krishna, Ranjay and Choi, Yejin and Hajishirzi, Hannaneh and Farhadi, Ali , journal=
-
[3]
Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages=
Learning to score olympic events , author=. Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages=
-
[4]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
What and how well you performed? a multitask learning approach to action quality assessment , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[5]
The influence of movement tempo during resistance training on muscular strength and hypertrophy responses: a review , author=. Sports medicine , volume=. 2021 , publisher=
work page 2021
-
[6]
International journal of environmental research and public health , volume=
Exercise selection and common injuries in fitness centers: a systematic integrative review and practical recommendations , author=. International journal of environmental research and public health , volume=. 2022 , publisher=
work page 2022
-
[7]
Hurst, Aaron and Lerer, Adam and Goucher, Adam P and Perelman, Adam and Ramesh, Aditya and Clark, Aidan and Ostrow, AJ and Welihinda, Akila and Hayes, Alan and Radford, Alec and others , journal=
-
[8]
Advances in neural information processing systems , volume=
Visual instruction tuning , author=. Advances in neural information processing systems , volume=
Show all 57 references
-
[9]
Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad , booktitle=
-
[10]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Group-aware contrastive regression for action quality assessment , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[11]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Uncertainty-aware score distribution learning for action quality assessment , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[12]
Computers & Graphics , volume=
Classification of motor errors to provide real-time feedback for sports coaching in virtual reality—A case study in squats and Tai Chi pushes , author=. Computers & Graphics , volume=. 2018 , publisher=
2018
-
[13]
arXiv preprint arXiv:2506.11774 , year=
Real-Time Feedback and Benchmark Dataset for Isometric Pose Evaluation , author=. arXiv preprint arXiv:2506.11774 , year=
-
[14]
Li, Kunchang and Wang, Yali and He, Yinan and Li, Yizhuo and Wang, Yi and Liu, Yi and Wang, Zun and Xu, Jilan and Chen, Guo and Luo, Ping and others , booktitle=
-
[15]
Fu, Chaoyou and Dai, Yuhan and Luo, Yongdong and Li, Lei and Ren, Shuhuai and Zhang, Renrui and Wang, Zihan and Zhou, Chenyu and Shen, Yunhang and Zhang, Mengdan and others , booktitle=
-
[16]
Cai, Yuxuan and Zhang, Jiangning and Gan, Zhenye and He, Qingdong and Hu, Xiaobin and Zhu, Junwei and Wang, Yabiao and Wang, Chengjie and Xue, Zhucun and Fu, Chaoyou and others , journal=
-
[17]
2019 IEEE winter conference on applications of computer vision (WACV) , pages=
Action quality assessment across multiple actions , author=. 2019 IEEE winter conference on applications of computer vision (WACV) , pages=. 2019 , organization=
2019
-
[18]
, author=
Analysis of the Squat Exercise from Visual Data. , author=. ICINCO , pages=
-
[19]
2024 , organization=
Li, Yuan-Ming and Huang, Wei-Jin and Wang, An-Lan and Zeng, Ling-An and Meng, Jing-Ke and Zheng, Wei-Shi , booktitle=. 2024 , organization=
2024
-
[20]
IEEE transactions on circuits and systems for video technology , volume=
Learning to score figure skating sport videos , author=. IEEE transactions on circuits and systems for video technology , volume=. 2019 , publisher=
2019
-
[21]
Zhang, Yongji and Li, Siqi and Gao, Yue and Jiang, Yu , journal=
-
[22]
Zhang, Shiyi and Dai, Wenxun and Wang, Sujia and Shen, Xiangwei and Lu, Jiwen and Zhou, Jie and Tang, Yansong , booktitle=
-
[23]
Shao, Dian and Zhao, Yue and Dai, Bo and Lin, Dahua , booktitle=
-
[24]
Dong, Linfeng and Wang, Wei and Qiao, Yu and Sun, Xiao , journal=
-
[25]
IEEE Transactions on Neural Systems and Rehabilitation Engineering , volume=
A deep learning framework for assessing physical rehabilitation exercises , author=. IEEE Transactions on Neural Systems and Rehabilitation Engineering , volume=. 2020 , publisher=
2020
-
[26]
European conference on computer vision , pages=
Domain knowledge-informed self-supervised representations for workout form assessment , author=. European conference on computer vision , pages=. 2022 , organization=
2022
-
[27]
2025 International Conference On Rehabilitation Robotics (ICORR) , pages=
Skeleton-based transformer for classification of errors and better feedback in low back pain physical rehabilitation exercises , author=. 2025 International Conference On Rehabilitation Robotics (ICORR) , pages=. 2025 , organization=
2025
-
[28]
Zhao, Ziyi and Kiciroglu, Sena and Vinzant, Hugues and Cheng, Yuan and Katircioglu, Isinsu and Salzmann, Mathieu and Fua, Pascal , booktitle=
-
[29]
IEEE Transactions on Neural Systems and Rehabilitation Engineering , volume=
A skeleton-based rehabilitation exercise assessment system with rotation invariance , author=. IEEE Transactions on Neural Systems and Rehabilitation Engineering , volume=. 2023 , publisher=
2023
-
[30]
Fieraru, Mihai and Zanfir, Mihai and Pirlea, Silviu Cristian and Olaru, Vlad and Sminchisescu, Cristian , booktitle=
-
[31]
PloS one , volume=
LLM-FMS: A fine-grained dataset for functional movement screen action quality assessment , author=. PloS one , volume=. 2025 , publisher=
2025
-
[32]
Engineering Applications of Artificial Intelligence , volume=
Exercise quality assessment in monocular video streaming , author=. Engineering Applications of Artificial Intelligence , volume=. 2026 , publisher=
2026
-
[33]
Yin, Hao and Gu, Lijun and Parmar, Paritosh and Xu, Lin and Guo, Tianxiao and Liu, Xiujin and Fu, Weiwei and Zhang, Yang and Zheng, Tianyou , journal=
-
[34]
arXiv preprint arXiv:2512.15153 , year=
Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning , author=. arXiv preprint arXiv:2512.15153 , year=
-
[35]
Advances in Neural Information Processing Systems , volume=
What to say and when to say it: Live fitness coaching as a testbed for situated interaction , author=. Advances in Neural Information Processing Systems , volume=
-
[36]
Ji, Yuyang and Shen, Yixuan and Zhu, Shengjie and Kong, Yu and Liu, Feng , journal=. From
-
[37]
2026 , eprint=
Can Vision Language Models Judge Action Quality? An Empirical Evaluation , author=. 2026 , eprint=
2026
-
[38]
Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization , pages=
Fine-Tuning Large Multimodal Models for Fitness Action Quality Assessment , author=. Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization , pages=
-
[39]
arXiv preprint arXiv:1907.06987 , year=
A short note on the kinetics-700 human action dataset , author=. arXiv preprint arXiv:1907.06987 , year=
1907 arXiv
-
[40]
Soomro, Khurram and Zamir, Amir Roshan and Shah, Mubarak , journal=
-
[41]
Bai, Shuai and Cai, Yuxuan and Chen, Ruizhe and Chen, Keqin and Chen, Xionghui and Cheng, Zesen and Deng, Lianghao and Ding, Wei and Gao, Chang and Ge, Chunjiang and others , journal=
-
[42]
Singh, Aaditya and Fry, Adam and Perelman, Adam and Tart, Adam and Ganesh, Adi and El-Kishky, Ahmed and McLaughlin, Aidan and Low, Aiden and Ostrow, AJ and Ananthram, Akhila and others , journal=
-
[43]
LLaVA-OneVision Contributors , booktitle=
-
[44]
An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan...
-
[45]
Tang, Feilong and An, Xiang and Yan, Yunyao and Xie, Yin and Qin, Bin and Yang, Kaicheng and Shen, Yifei and Zhang, Yuanhan and Li, Chunyuan and Feng, Shikun and Chen, Changrui and Tan, Huajie and Hu, Ming and Zhang, Manyuan and Li, Bo and Feng, Ziyong and Liu, Ziwei and Ge, Z...
-
[46]
Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan , journal=
-
[47]
Zhang, Boqiang and Li, Kehan and Cheng, Zesen and Hu, Zhiqiang and Yuan, Yuqian and Chen, Guanzheng and Leng, Sicong and Jiang, Yuming and Zhang, Hang and Li, Xin and others , journal=
-
[48]
Qwen Team , journal=
-
[49]
Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others , journal=
-
[50]
Clement Farabet and Olivier Lacombe , journal =
-
[51]
Gemini Team , journal =
-
[52]
Wang, Haibo and Xu, Zhiyang and Cheng, Yu and Diao, Shizhe and Zhou, Yufan and Cao, Yixin and Wang, Qifan and Ge, Weifeng and Huang, Lifu , journal=
-
[53]
Liu, Ye and Qinghong Lin, Kevin and Chen, Chang Wen and Shou, Mike Zheng , journal=
-
[54]
Zhang, Jun and Wang, Teng and Ge, Yuying and Ge, Yixiao and Li, Xinhao and Shan, Ying and Wang, Limin , journal=
-
[55]
Gonzalez and Hao Zhang and Ion Stoica , booktitle=
Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica , booktitle=. Efficient Memory Management for Large Language Model Serving with
-
[56]
Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=
Transformers: State-of-the-art natural language processing , author=. Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=
2020
-
[57]
Proceedings of the 27th ACM International Conference on Multimedia , series=
Dutta, Abhishek and Zisserman, Andrew , title=. Proceedings of the 27th ACM International Conference on Multimedia , series=. 2019 , location=. doi:10.1145/3343031.3350535 , publisher=
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.