REVIEW 3 major objections 4 minor 27 references
PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Replacing a linear projector with a two-layer MLP raises language-guided pose estimation from 77.4 to 77.8 AP on COCO while keeping zero-shot transfer.
desk verdict A clean, honest incremental paper: the MLP connector swap is real and reproducible-ish, but the +0.4 AP claim is not yet causally established because the baseline is a published number, not a matched retrain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the nonlinear vision-language connector, a two-layer MLP with GELU nonlinearity: $Z = \mathrm{GELU}(I W_1 + b_1)$, $V = Z W_2 + b_2$, where $I$ is the sequence of 1024-dimensional patch features from DINOv2 and $V$ is the 4096-dimensional token sequence fed to the LLM. It replaces LocLLM's single linear projector and is the only architectural change evaluated. The MLP is fully trained while the vision encoder and LLM are adapted with LoRA, so the experiment isolates the effect of connector nonlinearity.
What would settle it
Train the exact PoseLLM setup (12 epochs, batch size 32, learning rate 5e-4, LoRA on DINOv2 and Vicuna, 224x224 input) with the connector replaced by a single linear layer and run several seeds; if the linear version matches or exceeds 77.8 AP, the MLP is not the causal factor.
Extended reading notes
Core claim
The paper's central claim is that the connector between the visual encoder and the language model is a bottleneck for LLM-based keypoint localization, and that a nonlinear connector removes that bottleneck. Concretely, PoseLLM replaces LocLLM's single linear projection with a two-layer MLP using GELU activation, mapping DINOv2 patch features into Vicuna-7B's input space. On the COCO validation set, the paper reports 77.8 AP versus LocLLM's 77.4 AP, with similar or slightly better cross-dataset numbers on Human-Art and MPII under zero-shot evaluation. The authors present this as establishing a stronger baseline for language-guided pose estimation and as evidence that lightweight nonlinear alignment helps fine-grained spatial tasks.
Load-bearing premise
The claim rests on comparing PoseLLM's 77.8 AP with LocLLM's published 77.4 AP, assuming both used the same training schedule, resolution, LoRA setup, and evaluation protocol, because the paper does not retrain the linear-projection baseline under its own configuration.
Editorial extensions
If this is right
- PoseLLM becomes the new numerical baseline for LLM-based pose estimation on COCO, ahead of the language-based methods in Table 2.
- The parity of cross-dataset scores on Human-Art and MPII indicates that the added nonlinearity does not come at the cost of zero-shot flexibility.
- The improvement is achieved without scaling the LLM or the training data, pointing to connector design as a low-cost axis for spatial accuracy.
- Because the connector adds only a few parameters, the same replacement can be applied to other LocLLM-style models to test whether the gain transfers.
Reading between the lines
- An immediate test is per-keypoint AP: if nonlinearity matters, the gain should concentrate on joints with strong spatial-textual ambiguity such as elbow, wrist, and ankle rather than nose and eyes, and the paper does not report that breakdown.
- The same MLP connector could be dropped into other VLM spatial tasks such as object keypoint detection or dense correspondence, where the linear-projector bottleneck is likely similar; the paper does not test these tasks.
- Under the assumption that the +0.4 AP is causal, wider or deeper connectors might buy further precision, but saturation is plausible because the MLP is trained from scratch while the backbones are LoRA-constrained.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PoseLLM, a language-guided human pose estimation model that replaces LocLLM's linear vision-language projector with a two-layer MLP with GELU activation. The model is trained on COCO only and reports 77.8 AP on the COCO validation set, out-performing LocLLM by 0.4 AP, while claiming comparable zero-shot generalization on Human-Art and MPII. The method section describes the architecture (DINOv2 ViT-L/14 encoder, Vicuna-7B LLM, LoRA fine-tuning, two-layer MLP connector), and the experiments compare against heatmap-based, regression-based, and language-based baselines on COCO, Human-Art, and MPII. The central claim is that the nonlinear connector improves localization accuracy without sacrificing cross-dataset generalization.
Significance. If the claimed causal effect were established, the paper would provide a simple, reproducible architectural change—replacing a linear projector with a two-layer MLP—that improves an existing LLM-based keypoint localization baseline. The idea is plausible and consistent with practices in general vision-language models such as LLaVA-1.5. A strength is that the code is promised to be released, which would facilitate verification. However, the empirical evidence as presented is insufficient to support the central attribution: the LocLLM comparison is not a controlled ablation, no variance is reported, and the reported gains (0.4 AP on COCO, at most 0.2 on cross-dataset metrics) are small relative to typical run-to-run and protocol variation. The paper's contribution would be more convincing if the authors retrained a linear-projector baseline under their exact training recipe and reported multiple seeds.
major comments (3)
- [§4.2, Table 2] The +0.4 AP improvement over LocLLM is not a controlled comparison. Table 2's caption states that all results were obtained using official model weights, so the LocLLM row comes from a model trained with a different pipeline, not under PoseLLM's training configuration (12 epochs, batch size 32, learning rate 5e-4, LoRA on DINOv2 and Vicuna-7B, 224×224 inputs). Any of these differences—LoRA rank and alpha, learning schedule, resolution, or evaluation details—could account for the gap. To support the claim that the nonlinear connector causes the improvement, the authors must retrain a linear-projector baseline under the identical PoseLLM recipe and compare it against the MLP connector under identical conditions.
- [§4.2, Tables 2 and 3] Only a single run is reported, with no error bars or seed-to-seed variance. The COCO improvement is 0.4 AP and the cross-dataset differences in Table 3 are at most about 0.1–0.2, which is within plausible variation for a single training run on COCO val. Please report results across at least three seeds (mean and standard deviation) and state whether the differences are statistically distinguishable. Without this, the central causal claim is not yet established.
- [§4.2, Table 3] The text states that PoseLLM shows 'slight improvements' on Human-Art and MPII, but several columns in Table 3 show decreases: Human-Art AP50 drops from 88.6 to 87.7, and MPII Elbow drops from 98.2 to 98.1. Given the absence of variance estimates, the data support only parity with LocLLM on cross-dataset generalization, not improvement. Please adjust the wording to match the numbers and provide matched comparisons before claiming that generalization is 'maintained' or 'slightly improved.'
minor comments (4)
- [§2.1] The symbol for V* appears as 'V ∗ [20]' with a formatting artifact; it should be rendered as 'V*' consistently.
- [§3.2] The section heading contains a typo: 'T ext Encoding' should be 'Text Encoding'.
- [Figure 1] The figure text contains 'Vison-Language Connector' twice; this should be 'Vision-Language Connector'.
- [Table 3] The MPII column labeled 'Mean0.1' should be labeled 'PCKh@0.1' for clarity and consistency with the evaluation metrics described in Section 4.
Circularity Check
No circularity: the MLP-alignment claim is an empirical comparison, not a fit-derived prediction; the only self-citations are non-load-bearing context.
full rationale
The paper contains no derivation chain in which a predicted quantity is identical to an input by construction. Its central claim is an empirical comparison: replacing LocLLM's linear projector with the two-layer MLP of Eqs. (3)-(4) and training on COCO yields 77.8 AP versus 77.4 AP for LocLLM (Table 2). The LocLLM row is obtained from official model weights, so the comparison is not a matched ablation under PoseLLM's exact training recipe; that is a potential threat to causal attribution, but it is an experimental-design concern, not a case where a fitted parameter is renamed as a prediction or where the outcome is entailed by definition. The paper does not fit anything to the COCO validation metric and then report that same fitted quantity as a prediction. The only self-citations (refs. 25 and 26) appear in the related-work discussion of keypoint-integrated instruction data and are not load-bearing evidence for PoseLLM's reported results. No uniqueness theorem, ansatz, or external result is imported from the authors' own prior papers to force the MLP choice. Consequently, no circular step can be exhibited with a specific equation-to-equation reduction or a fitted-input-called-prediction mechanism, and an honest non-finding is appropriate.
Assumptions & free parameters
free parameters (3)
- Training epochs =
12
- Learning rate =
5e-4
- LoRA rank and alpha =
not reported
assumptions (3)
- domain assumption DINOv2 patch features contain sufficient spatial detail for keypoint localization through a two-layer MLP.
- domain assumption The keypoint descriptions and prompt format used here are identical to LocLLM's.
- domain assumption Cross-entropy training on coordinate tokens with teacher forcing is a valid supervisory signal for pose regression.
Cite this review
Pith. "Pith review of PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment." pith.science (2026). https://pith.science/paper/Z2KZ3NSH
@misc{pith2026250709139,
author = {Pith},
title = {Pith review of: PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z2KZ3NSH}},
note = {Machine review of arXiv:2507.09139}
}
read the original abstract
Human pose estimation traditionally relies on architectures that encode keypoint priors, limiting their generalization to novel poses or unseen keypoints. Recent language-guided approaches like LocLLM reformulate keypoint localization as a vision-language task, enabling zero-shot generalization through textual descriptions. However, LocLLM's linear projector fails to capture complex spatial-textual interactions critical for high-precision localization. To address this, we propose PoseLLM, the first Large Language Model (LLM)-based pose estimation framework that replaces the linear projector with a nonlinear MLP vision-language connector. This lightweight two-layer MLP with GELU activation enables hierarchical cross-modal feature transformation, enhancing the fusion of visual patches and textual keypoint descriptions. Trained exclusively on COCO data, PoseLLM achieves 77.8 AP on the COCO validation set, outperforming LocLLM by +0.4 AP, while maintaining strong zero-shot generalization on Human-Art and MPII. Our work demonstrates that a simple yet powerful nonlinear connector significantly boosts localization accuracy without sacrificing generalization, advancing the state-of-the-art in language-guided pose estimation. Code is available at https://github.com/Ody-trek/PoseLLM.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in neural information processing systems 35, 23716–23736 (2022)
Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, 23716–23736 (2022)
work page 2022
-
[2]
In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. pp. 3686–3693 (2014)
work page 2014
-
[3]
Frontiers in Computer Science5, 1153160 (2023)
Avogaro, A., Cunico, F., Rosenhahn, B., Setti, F.: Markerless human pose estimation for biomedical appli- cations: a survey. Frontiers in Computer Science5, 1153160 (2023)
work page 2023
-
[4]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Chen, J., Hu, J., Wang, G., Jiang, Z., Zhou, T., Chen, Z., Lv, C.: Taoavatar: Real-time lifelike full-body talking avatars for augmented reality via 3d gaussian splatting. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 10723–10734 (2025)
work page 2025
-
[5]
Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (March 2023), https://lmsys.org/blog/2023-03-30-vicuna/
work page 2023
-
[6]
arXiv preprint arXiv:1606.08415 (2016)
Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
arXiv 2016
-
[7]
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)
work page 2022
-
[8]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Ju, X., Zeng, A., Wang, J., Xu, Q., Zhang, L.: Human-art: A versatile human-centric dataset bridging natural and artificial scenes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 618–629 (2023)
work page 2023
Show all 27 references
-
[9]
In: Proceedings of the IEEE/CVF international conference on computer vision
Li, J., Bian, S., Zeng, A., Wang, C., Pang, B., Liu, W., Lu, C.: Human pose regression with residual log- likelihood estimation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11025–11034 (2021)
2021
-
[10]
In: International conference on machine learning
Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023) 7
2023
-
[11]
In: European conference on computer vision
Li, Y., Yang, S., Liu, P., Zhang, S., Wang, Y., Wang, Z., Yang, W., Xia, S.T.: Simcc: A simple coordinate classification perspective for human pose estimation. In: European conference on computer vision. pp. 89–106. Springer (2022)
2022
-
[12]
In: Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755....
2014
-
[13]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26296–26306 (2024)
2024
-
[14]
Advances in neural information processing systems 36, 34892–34916 (2023)
Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023)
2023
-
[15]
arXiv preprint arXiv:2304.07193 (2023)
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
2023 arXiv
-
[16]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Soni, S., Dudhane, A., Debary, H., Fiaz, M., Munir, M.A., Danish, M.S., Fraccaro, P., Watson, C.D., Klein, L.J., Khan, F.S., et al.: Earthdial: Turning multi-sensory earth observations to interactive dialogues. In: Proceedings of the Computer Vision and Pattern Recognition Con...
2025
-
[17]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5693–5703 (2019)
2019
-
[18]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Toshev, A., Szegedy, C.: Deeppose: Human pose estimation via deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1653–1660 (2014)
2014
-
[19]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
Wang, D., Xuan, S., Zhang, S.: Locllm: Exploiting generalizable human keypoint localization via large lan- guage model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
2024
-
[20]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wu, P., Xie, S.: V?: Guided visual search as a core mechanism in multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13084–13094 (2024)
2024
-
[21]
arXiv preprint arXiv:2412.10302 (2024)
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al.: Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024)
2024 arXiv
-
[22]
Alexandria Engineering Journal 109, 144–156 (2024)
Xi, X., Zhang, C., Jia, W., Jiang, R.: Enhancing human pose estimation in sports training: Integrating spa- tiotemporal transformer for improved accuracy and real-time performance. Alexandria Engineering Journal 109, 144–156 (2024)
2024
-
[23]
In: Proceedings of the European conference on computer vision (ECCV)
Xiao, B., Wu, H., Wei, Y.: Simple baselines for human pose estimation and tracking. In: Proceedings of the European conference on computer vision (ECCV). pp. 466–481 (2018)
2018
-
[24]
Advances in neural information processing systems35, 38571–38584 (2022)
Xu,Y.,Zhang,J.,Zhang,Q.,Tao,D.:Vitpose:Simplevisiontransformerbaselinesforhumanposeestimation. Advances in neural information processing systems35, 38571–38584 (2022)
2022
-
[25]
Zhang, D., An, W., Shouno, H.: Keypoint-integrated instruction-following data generation for enhanced human pose and action understanding in multimodal models (2025),https://arxiv.org/abs/2409.09306
2025 arXiv
-
[26]
Zhang, D., Hussain, T., An, W., Shouno, H.: Llava-pose: Enhancing human pose and action understanding via keypoint-integrated instruction tuning (2025),https://arxiv.org/abs/2506.21317
2025 arXiv
-
[27]
ACM Computing Surveys56(1), 1–37 (2023) 8
Zheng, C., Wu, W., Chen, C., Yang, T., Zhu, S., Shen, J., Kehtarnavaz, N., Shah, M.: Deep learning-based human pose estimation: A survey. ACM Computing Surveys56(1), 1–37 (2023) 8
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.