REVIEW 3 major objections 6 minor 27 references
Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Dense correspondence supervision can replace paired in-the-wild 3D annotations and still reach 92 percent of the full-3D model's accuracy.
desk verdict A useful systematic comparison of annotation types for 3D human recovery, but the headline 92% claim compares against the wrong baseline and is inflated by about 5 points. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is a mapping φ from a DensePose (I,U,V) coordinate to the indices of the three SMPL vertices of the triangle containing that surface point and their barycentric weights. Because each SMPL vertex carries a fixed (I,U,V), the mapping lets the network project predicted SMPL vertices to 2D, take a barycentric weighted average to obtain a predicted dense keypoint, and supervise it against dense keypoints sampled from IUV maps with an L1 loss. A recursive refinement step, which compares a 3×3 neighborhood around each sparse 2D keypoint against the IUV body-part label and erases inconsistent regions, cleans noise before sampling. This conversion is what turns DensePose's cheap 2D surface annotations into a differentiable 3D supervisory signal.
What would settle it
Evaluate the same dense-correspondence versus full-3D comparison on 3DPW, whose ground-truth SMPL meshes come from IMU sensors and 3D scans rather than SMPLify; if the dense-correspondence model does not reach roughly 92 percent of the full-3D model's per-vertex error there, then the headline ratio is specific to UP-3D's fitting bias.
Extended reading notes
Core claim
On the UP-3D test set, keeping the input as image only, the model supervised by sampled dense keypoints and sparse 2D keypoints reaches a per-vertex error (PVE) of 137.7 mm versus 127.4 mm for the model supervised by full paired 3D annotations and sparse 2D keypoints, which corresponds to the 92 percent figure; with image plus IUV input, the corresponding numbers are 133.8 mm versus 125.5 mm. Adding dense correspondence to full 3D supervision improves PVE by 2.9 percent, and a hybrid of 20 percent 3D annotations plus dense correspondence matches full-3D performance. The paper interprets these results as showing that dense correspondence is an effective and cheaper substitute for in-the-wild 3D annotations, and that the two signals are complementary.
Load-bearing premise
The study treats the SMPL fits in UP-3D, produced by an optimization from 2D keypoints and silhouettes, as ground-truth in-the-wild 3D annotations; if those fits carry a systematic bias, the 92 percent ratio and absolute PVE values measure agreement with that fitting protocol rather than with true 3D body shape.
Editorial extensions
If this is right
- Training with dense correspondence plus sparse 2D keypoints reaches 92 percent of full-3D performance, so expensive in-the-wild 3D annotation is not a strict requirement.
- Combining only 20 percent paired in-the-wild 3D annotations with dense correspondence matches full-3D training, suggesting annotation budgets can be cut by four-fifths.
- Adding dense correspondence to full 3D supervision improves per-vertex error by 2.9 percent, so the two supervision types are complementary, not redundant.
- Sparse 2D keypoints alone are far weaker than dense correspondence (about 60 percent worse under the image-only setting), so 2D pose annotation is not an adequate substitute for geometric supervision.
- On UP-3D and 3DPW, the dense-correspondence-only model is comparable to several previous methods trained with paired 3D data, making it a viable strong baseline.
Reading between the lines
- The 92 percent figure is measured against UP-3D, whose ground truth is SMPLify fits; a fair implication is that the ratio could shift on datasets with sensor-based 3D ground truth such as IMU-captured sequences.
- Because DensePose-style IUV annotation is far cheaper than marker-based 3D capture, the result suggests a practical recipe: for a new in-the-wild domain, annotate or predict dense correspondence, clean it with 2D keypoints, and train without waiting for 3D fits.
- The dependence of the refinement step on accurate sparse 2D keypoints implies that improving 2D pose estimation should directly translate into better dense supervision and thus better 3D recovery.
- The same mapping-plus-barycentric-loss recipe could transfer to other parametric surface models (hands, animals, clothed bodies) whenever a dense 2D-to-surface correspondence annotation exists.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a systematic study of the cost-effectiveness trade-off among five annotation types for single-image 3D human recovery: sparse 2D keypoints, dense labeling (body part segmentation), dense correspondence (DensePose IUV), constrained 3D, and in-the-wild 3D annotations. Using a unified framework with a ResNet encoder and SMPL parameter regressor, the authors compare various input combinations and supervision signals on UP-3D, 3DPW, and COCO-DensePose. The main claims are (1) 3D annotations are highly efficient, (2) sparse 2D keypoints and dense labeling alone are insufficient, and (3) dense correspondence can substitute for in-the-wild 3D annotations, achieving 92% of the performance of a model trained with full paired in-the-wild 3D data, and matching the performance of a model trained with 60% of paired 3D annotations. The paper also compares against prior methods and analyzes the robustness of dense correspondence to noise and keypoint density.
Significance. The study is practically significant because it addresses the high cost of collecting 3D annotations in-the-wild, a bottleneck for 3D human recovery. The unified framework and the consistent ordering of annotation effectiveness across five input configurations lend credibility to the qualitative conclusions. The paper also provides code and models, and its analyses of noisy and sparse dense keypoints give actionable guidance for reducing annotation cost. If the quantitative claims were fully supported, this would be an important contribution; however, the headline percentage claims are currently overstated relative to the reported results.
major comments (3)
- [Abstract, Section 1, Table 3] The headline claim that dense correspondence achieves '92% of the performance of an upper-bound model' is not supported by the paper's own Table 3. The 92% figure compares the 'Dense & Sparse 2D' column with the '3D & Sparse 2D' column (e.g., 127.4/137.7 = 92.5% for Image Only and 125.5/133.8 = 93.8% for Image & IUV). However, the first column '3D & Dense & Sparse 2D' achieves 120.0 and 122.4 mm for the same input settings, respectively, outperforming the '3D & Sparse 2D' model by 5.8% and 2.5%. Thus '3D & Sparse 2D' is not an upper bound under the paper's own results. If dense-only is compared with the best supervision combination actually reported, the ratios are 120.0/137.7 = 87.1% for Image Only and 122.4/133.8 = 91.5% for Image & IUV. The 'upper-bound' wording and the 92% figure should be revised, or the specific baseline must be stated precisely.
- [Section 1, Table 6] The claim in Section 1 that models trained with dense correspondence 'can still achieve the same performance as the models trained with 60% paired in-the-wild 3D annotations' is contradicted by Table 6. For the Image Only input, the 60% 3D entry has PVE 132.2 mm, while the dense-only model (Table 3, 'Dense & Sparse 2D' with Image Only) has PVE 137.7 mm. The dense-only model is much closer to the 20% 3D entry (136.0 mm), and the difference to 60% (5.5 mm) is over three times the difference to 20% (1.7 mm). This quantitative claim should be corrected or removed.
- [Tables 3-7, Figures 4-6] All comparisons are based on single training runs with no reported variance, error bars, or repeated seeds. The differences underlying the headline claims are small (e.g., 133.8 vs 137.7 mm for Image & IUV, a 2.8% difference; 127.5 vs 127.4 mm for the 'comparable to full 3D' claim). Without a noise estimate, the quantitative percentages (92%, 60%, 2.9%, 1.5%) cannot be assessed for statistical reliability. The authors should provide at least a small number of repeated runs or confidence intervals for the key tables.
minor comments (6)
- [Appendix A] The word 'chech' should be 'check' in the sentence 'Then we chech whether the surrounding IUV is consistent with the 2D keypoint.'
- [Abstract] The phrase 'of an upper-bound models' should be 'of an upper-bound model.'
- [Section 4.1] The sentence 'can achieve the 92% performance of the model trained with full set 3D annotations' should read 'can achieve 92% of the performance of the model trained with a full set of 3D annotations.'
- [Appendix B, Table 7 caption] The phrase 'improves the performance of models bys 30%' should be 'improves the performance of models by 30%.'
- [References] Several reference names contain typographical errors: '[1] Rza Alp Gler' should be 'Riza Alp Güler', and '[26] V on et al.' should be 'von Marcard et al.'
- [Section 4, UP-3D description] The evaluation uses SMPLify fits as the in-the-wild 3D ground truth. This is a known limitation; the paper should explicitly acknowledge that the '92%' figure is measured against these pseudo-ground-truth fits, so the absolute ratio may not reflect true 3D accuracy in the wild, even though relative rankings remain meaningful.
Circularity Check
No circularity: the 92% claim is an empirical comparison between independently trained models, not a derivation from its own inputs.
full rationale
The paper's central claim is empirical: the model supervised with dense correspondence (sampled from DensePose IUV maps) achieves 92% of the PVE performance of a model supervised with paired in-the-wild 3D annotations, as reported in Table 3. These are two separate training configurations compared on the same test set; neither model's output is defined in terms of the other. The dense correspondence supervision is constructed through the mapping in Eqs. (3)-(4) from IUV coordinates to SMPL vertices and barycentric weights, which is a geometric correspondence, not a quantity derived from the PVE metric or from the 3D annotations used for the upper-bound model. The evaluation uses UP-3D SMPL fits as ground truth, but every compared model shares the same ground truth; a shared benchmark does not make any particular model's prediction equivalent to its own training input. The skeptical observation that the 3D-only model is not a true upper bound because the 3D+dense model is better is a legitimate concern about the strength of the paper's claim, but it is an empirical interpretation issue rather than circularity. There is no self-citation chain that supplies a load-bearing premise, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The paper is a controlled experimental study whose conclusions are measured performance comparisons, so no step in its derivation chain reduces to its own inputs.
Assumptions & free parameters
free parameters (2)
- Loss weights lambda_3D, lambda_dense, lambda_2D =
10, 1, 10 when all three losses are used; 10, 10 when two losses are used
- IUV refinement grid size =
3 by 3
assumptions (5)
- domain assumption SMPL is an adequate low-dimensional parametric model of the 3D human body.
- domain assumption DensePose IUV maps provide reliable dense correspondence for in-the-wild images.
- domain assumption UP-3D SMPL fits are valid in-the-wild ground truth.
- domain assumption Per-vertex Euclidean error on SMPL meshes captures pose and shape quality.
- domain assumption The annotation cost ordering in Figure 1 is qualitatively correct.
Cite this review
Pith. "Pith review of Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild." pith.science (2026). https://pith.science/paper/LDTU3GQ5
@misc{pith2026190806442,
author = {Pith},
title = {Pith review of: Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild},
year = {2026},
howpublished = {\url{https://pith.science/paper/LDTU3GQ5}},
note = {Machine review of arXiv:1908.06442}
}
read the original abstract
Though much progress has been achieved in single-image 3D human recovery, estimating 3D model for in-the-wild images remains a formidable challenge. The reason lies in the fact that obtaining high-quality 3D annotations for in-the-wild images is an extremely hard task that consumes enormous amount of resources and manpower. To tackle this problem, previous methods adopt a hybrid training strategy that exploits multiple heterogeneous types of annotations including 3D and 2D while leaving the efficacy of each annotation not thoroughly investigated. In this work, we aim to perform a comprehensive study on cost and effectiveness trade-off between different annotations. Specifically, we focus on the challenging task of in-the-wild 3D human recovery from single images when paired 3D annotations are not fully available. Through extensive experiments, we obtain several observations: 1) 3D annotations are efficient, whereas traditional 2D annotations such as 2D keypoints and body part segmentation are less competent in guiding 3D human recovery. 2) Dense Correspondence such as DensePose is effective. When there are no paired in-the-wild 3D annotations available, the model exploiting dense correspondence can achieve 92% of the performance compared to a model trained with paired 3D data. We show that incorporating dense correspondence into in-the-wild 3D human recovery is promising and competitive due to its high efficiency and relatively low annotating cost. Our model trained with dense correspondence can serve as a strong reference for future research.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Densepose: Dense human pose estimation in the wild
Rza Alp Gler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018. 1, 2, 4, 5, 6, 8, 10
work page 2018
-
[2]
2d human pose estimation: New benchmark and state of the art analysis
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014. 5
work page 2014
-
[3]
Keep it smpl: Automatic estimation of 3d human pose and shape from a single image
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In ECCV, 2016. 2, 5
work page 2016
-
[4]
Realtime multi-person 2d pose estimation using part affinity fields
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017. 6
work page 2017
-
[5]
Human pose estimation with iterative er- ror feedback
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative er- ror feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4733–4742,
-
[6]
Body parts dependent joint regressors for human pose estimation in still images
Matthias Dantone, Juergen Gall, Christian Leistner, and Luc Van Gool. Body parts dependent joint regressors for human pose estimation in still images. TPAMI, 36(11):2131–2143,
-
[7]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672–2680, 2014. 2
work page 2014
-
[8]
Holopose: Holistic 3d human reconstruction in-the-wild
Riza Alp Guler and Iasonas Kokkinos. Holopose: Holistic 3d human reconstruction in-the-wild. In CVPR, pages 10884– 10894, 2019. 4
work page 2019
Show all 27 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,
-
[10]
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and pre- dictive methods for 3d human sensing in natural environ- ments. TPAMI, 36(7):1325–1339, 2014. 1, 5, 10
2014
-
[11]
Clustered pose and nonlinear appearance models for human pose estimation
Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010. 5
2010
-
[12]
Learning effective hu- man pose estimation from inaccurate annotation
Sam Johnson and Mark Everingham. Learning effective hu- man pose estimation from inaccurate annotation. In CVPR,
-
[13]
End-to-end recovery of human shape and pose
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018. 1, 2, 3, 5, 7, 8
2018
-
[14]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 10
2015
-
[15]
Unite the peo- ple: Closing the loop between 3d and 2d human representa- tions
Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler. Unite the peo- ple: Closing the loop between 3d and 2d human representa- tions. In CVPR, 2017. 1, 2, 5, 7, 8, 10
2017
-
[16]
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 1, 5, 7
2014
-
[17]
Mosh: Motion and shape capture from sparse markers.TOG, 33(6):220, 2014
Matthew Loper, Naureen Mahmood, and Michael J Black. Mosh: Motion and shape capture from sparse markers.TOG, 33(6):220, 2014. 2, 5
2014
-
[18]
Smpl: A skinned multi- person linear model
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. TOG, 34(6):248, 2015. 1, 2, 3, 4
2015
-
[19]
Slim densepose: Thrifty learning from sparse annotations and motion cues
Natalia Neverova, James Thewlis, Riza Alp Guler, Iasonas Kokkinos, and Andrea Vedaldi. Slim densepose: Thrifty learning from sparse annotations and motion cues. In CVPR, pages 10915–10923, 2019. 1, 4, 6
2019
-
[20]
Training a feedback loop for hand pose estimation
Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. Training a feedback loop for hand pose estimation. In ICCV, pages 3316–3324, 2015. 3
2015
-
[21]
Neural body fitting: Unifying deep learning and model based human pose and shape esti- mation
Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Pe- ter Gehler, and Bernt Schiele. Neural body fitting: Unifying deep learning and model based human pose and shape esti- mation. In 3DV, 2018. 1, 2, 7, 8
2018
-
[22]
Learning to estimate 3d human pose and shape from a single color image
Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. Learning to estimate 3d human pose and shape from a single color image. In CVPR, 2018. 1, 2, 5, 7, 8
2018
-
[23]
Im2avatar: Colorful 3d reconstruction from a single image
Yongbin Sun, Ziwei Liu, Yue Wang, and Sanjay E Sarma. Im2avatar: Colorful 3d reconstruction from a single image. arXiv preprint arXiv:1804.06375, 2018. 1
2018 arXiv
-
[24]
Indirect deep structured learning for 3d human body shape and pose pre- diction
J Tan, Ignas Budvytis, and Roberto Cipolla. Indirect deep structured learning for 3d human body shape and pose pre- diction. In BMVC, 2017. 1, 2
2017
-
[25]
Bodynet: V olu- metric inference of 3d human body shapes
Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid. Bodynet: V olu- metric inference of 3d human body shapes. In ECCV, pages 20–36, 2018. 2
2018
-
[26]
Recovering accurate 3d human pose in the wild using imus and a moving camera
Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018. 2, 5, 8, 10
2018
-
[27]
right ankle
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018. 6 9 A. Sampling Dense Keypoints Since dense keypoint annotations are only available in COCO-DensePose dataset and training models purely us- ing sparse 2D keypoints wi...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.