REVIEW 3 major objections 5 minor 27 references
A two-stage perception-and-action pipeline lets a humanoid robot learn construction actions from RGB worker demonstrations, reproducing eight actions with an average mean-per-joint position error of 82.45 mm.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A humanoid robot learned eight construction motions from human video demonstrations via pose retargeting and reinforcement learning, with an average 82.45 mm joint tracking error.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection The integration is real and the dataset is useful, but the headline claim of reliable real-world execution leans on a sim-to-sim number. the 3 major comments →
Perception-and-action system for humanoid robot task execution in construction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's own terms, the central discovery is that a humanoid robot can learn a useful set of construction motions from human demonstrations through a vision-based perception-and-action (VPA) system, and that the learned motions transfer from simulation to the physical robot. The system's Humanoid-PoseNet reconstructs 3D human keypoints from RGB video and maps them through a shared latent space into a 14-joint humanoid pose representation, reporting a human-to-humanoid retargeting error of 48.46 mm MPJPE on held-out data. Humanoid-ActionNet then trains a teacher policy with PPO and distills it into a student policy that executes the retargeted poses on a 23-DoF humanoid while preserving
What carries the argument
The load-bearing piece is the pair of learned modules. Humanoid-PoseNet is a human-to-humanoid retargeting network: two encoders, one for human poses and one for robot poses, map poses into a shared latent space, and a decoder produces humanoid pose targets. A triplet loss built on bone-direction angular error aligns similar human and robot poses, while reconstruction and latent-consistency losses keep outputs inside the robot's feasible configuration. Humanoid-ActionNet is a teacher-student reinforcement-learning controller: a teacher policy trained with PPO on privileged simulation observations and physics-aware rewards—motion tracking, balance, root velocity, body orientation, foot contac
Load-bearing premise
The pipeline assumes the manually retargeted humanoid poses used as ground truth for both training and evaluation are correct, since the reported retargeting and execution errors are measured against those labels.
What would settle it
Take the manually retargeted ground-truth humanoid poses, run them through the physical robot's inverse kinematics and a balance/feasibility check, and count how many fall outside joint limits, violate foot-contact constraints, or would pitch the robot over. If a substantial share are infeasible, the 48.46 mm retargeting error and the 82.45 mm execution error are measured against targets the robot cannot physically follow, and the central claim fails.
If this is right
- If the reported numbers are representative, construction-specific humanoid skills can be initialized from ordinary video of human workers, so no specialized teleoperation rig or task-specific controller is needed to capture the motions.
- The retargeting module is not tied to one robot: the paper shows it also produces coherent poses for a different humanoid configuration, suggesting the same learned latent space can be reused for other humanoids with compatible kinematics.
- Because the off-the-shelf 3D pose estimators compared in the paper give near-identical downstream retargeting quality, the system's front end can be swapped as pose-estimation models improve.
- The sim-to-sim evaluation in a second physics engine acts as a safety gate: actions that do not survive transfer are not deployed on hardware, so the physical-robot results are the filtered subset that already passed that robustness check.
- The whole pipeline is an early path toward autonomous humanoid collaborators on construction sites, where the near-term expectation is worker-like whole-body motion, not dexterous tool use.
Where Pith is reading between the lines
- Editorial inference: The reported 82.45 mm MPJPE measures how closely the robot's joints track reference poses, not whether a brick was placed or a pipe delivered; task success is a separate quantity the paper does not measure.
- Editorial inference: The manual retargeting step that produced ground-truth labels is the pipeline's hidden dependency; unless those labels are checked against inverse kinematics or balance feasibility, both the retargeting error and the execution error are relative to labels that may themselves be imperfect.
- Editorial inference: Because physical deployment tested only the actions that passed the cross-simulator filter, the system's real-world generality is conditioned on that filter; actions that fail in one simulator remain untested on hardware.
- Editorial inference: The latent-space retargeting suggests a route to cross-robot transfer: the same human demonstration could be decoded into pose targets for different humanoids without re-collecting demonstrations, provided the target robot's kinematics are represented in the same way.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Vision-based Perception-and-Action (VPA), a two-module pipeline for teaching a Unitree G1 humanoid robot construction-related actions from RGB videos of human workers. Humanoid-PoseNet combines an off-the-shelf 3D pose estimator with a learned human-to-humanoid retargeting network (encoders/decoder with triplet, reconstruction, and latent-consistency losses). Humanoid-ActionNet is a teacher-student PPO architecture that tracks the retargeted pose trajectories under physics-aware rewards, with a student policy distilled for deployable proprioceptive observations. The authors report retargeting MPJPE of 48.46 mm, sim-to-sim (IsaacGym-to-MuJoCo) action-tracking MPJPE of 82.45 mm across eight actions, and qualitative real-robot deployment on a physical G1. The abstract and conclusion claim that the robot 'reliably executed eight construction-related actions' and achieved 'a reliable task success rate.'
Significance. If substantiated, the system would be a meaningful early step toward learning construction tasks from human demonstration on a full humanoid platform, connecting vision-based pose extraction, retargeting, and RL whole-body control. The paper has several strengths that deserve explicit credit: the architecture is clearly described; the data-collection protocol (five subjects, 30 actions, 450 trials) is documented in detail; the retargeting module is ablated; and a comparison against four whole-body control baselines (ExBody, ExBody+AMP, OmniH2O, ExBody2) is included. However, the headline quantitative claim is not supported as written: the only action-tracking number (82.45 mm MPJPE) comes from sim-to-sim transfer, not from the physical robot, and the retargeting ground truth is a manual process that is never independently validated. The paper's internal-consistency metrics are plausible as engineering-development evidence, but they do not yet establish reliable physical task execution.
major comments (3)
- [§5.2, Table 4, Abstract] The abstract states that 'the humanoid robot reliably executed eight construction-related actions, achieving an average motion-tracking error of 82.45 mm MPJPE,' but that number is measured in simulation. Section 5.2 reports Table 4 after describing the IsaacGym-to-MuJoCo sim-to-sim evaluation; the following paragraph on physical deployment says only that the G1 'successfully reproduced all eight tested actions' and that performance was 'consistent with the sim-to-sim evaluation trends,' with no real-robot MPJPE, per-action numbers, trial counts, or success criteria. Section 7 then asserts a 'reliable task success rate' without reporting any rate. The single quantitative piece of evidence for the central claim therefore does not concern physical execution at all. This is a direct evidence gap that must be fixed by either adding real-robot quantitative tracking results or substantially qu
- [§4.2, Table 3, Eq. (5)] The retargeting ground truth is produced by a manual process: 'the authors manually extracted and aligned the T-poses of the human and humanoid, and adjusted the body scale.' The reported retargeting MPJPE (48.46 mm, Table 3) is computed against these manual labels, and the ActionNet MPJPE (82.45 mm) compares executed poses to reference trajectories generated by Humanoid-PoseNet, which was trained to reproduce the same manual labels. Thus the primary performance measures are internal-consistency scores. The paper never independently verifies the manual labels via inverse kinematics, physical feasibility checks, or task outcomes. If the labels are not mechanically correct or task-faithful, the entire reference-trajectory chain is invalid. Please provide an external validation of the retargeting ground truth (e.g., IK-based joint-angle feasibility, contact/balance checks, or human evaluati
- [§5.2, Table 4, Fig. 14] The eight actions are selected after a sim-to-sim screening procedure, but the paper does not report how many actions were attempted, how many failed, or the selection criteria. Figure 14 shows representative failure cases in MuJoCo (loss of balance, unstable recovery, tracking breakdown), and Section 7 acknowledges 'non-negligible failure cases across the eight actions,' yet Table 4 reports only average per-joint MPJPE with no variance, no trial counts, and no success definition. The claim 'reliably executed' is not supported without per-action success rates and repeated-trial statistics. Please report the number of trials per action, standard deviations, and a defined success threshold (e.g., MPJPE below a threshold or posture-stability criterion), and discuss how the failure cases affect the reported averages.
minor comments (5)
- [§3.2, Eq. (1)] Minor notation: 'Where' should be lowercase 'where'; also the set B is used for matched bones but the superscripts h and r on u are not defined until the following sentence. Consider defining the skeleton notation before the equation.
- [§3.3, Table 1] The 'Regularization rewards' row refers to prior work (Cheng et al., 2024; He et al., 2024; Ji et al., 2025) without specifying the actual reward expressions. This makes the reward design not fully reproducible from the paper alone; please include the explicit formulas or a supplementary appendix.
- [§4.2] The data-splitting description says '80% of the data for model development (training and evaluation), with performance evaluated via 5-fold cross-validation. The remaining 20% was used as a test set.' It would be clearer to state whether the 5-fold cross-validation is on the 80% split and what the test-set size is, especially since Table 3 reports test-set MPJPE.
- [§5.2, Table 8] The comparison against baselines reports only point estimates of MPJPE with no variance or statistical significance. Adding standard deviations and, if feasible, paired tests across the same evaluation trials would strengthen the comparison.
- [§7] The Conclusion says 'a reliable task success rate' but no rate is given anywhere. Please either report a measured success rate or remove the phrase.
Circularity Check
No significant circularity: the reported MPJPE values are standard supervised/RL eval metrics, with cross-simulator transfer and external baselines providing independent content.
full rationale
The VPA pipeline's derivation chain is: RGB video → 3D human pose (PoseNet) → retargeted humanoid pose (Humanoid-PoseNet) → RL controller (Humanoid-ActionNet) → executed poses. Each stage is trained against a well-defined target and evaluated on held-out data or transfer settings. The retargeting ground truth is a manually retargeted humanoid pose set; the network is trained with an 80/20 split and evaluated on the test set, so the reported 48.46 mm MPJPE is a standard supervised learning result, not an identity. The ActionNet MPJPE (82.45 mm) compares executed poses to reference trajectories generated by Humanoid-PoseNet, but this number is obtained in MuJoCo after training in IsaacGym—a cross-simulator transfer that provides non-tautological evidence of tracking generalization. The reward functions do include keypoint-tracking terms, but evaluation on an independent simulator and comparison against external baselines (ExBody, ExBody2, OmniH2O) show the metric is not forced solely by construction. Self-citations are present (e.g., 3D-PoseNet) but are used as a comparison baseline, not as a load-bearing justification; no uniqueness theorem or ansatz is imported from the authors' prior work to forbid alternatives. The paper's real-robot claim of a 'reliable task success rate' is asserted without quantitative success criteria, and the abstract's headline MPJPE is a simulation result—this is an evidence gap about physical deployment, not circularity. The authors also acknowledge limitations (sim-to-sim failures, limited subjects, short-horizon tasks), which further indicates the derivation is not being presented as a closed self-justifying loop. Overall, I find no circular step that reduces a claimed prediction to its own input or to a self-citation chain.
Axiom & Free-Parameter Ledger
free parameters (12)
- Lambda1 loss weight (Eq.5) =
9.3
- Lambda2 loss weight (Eq.5) =
4.7
- Triplet margin c (Eq.2) =
0.3
- Latent dimension d =
14
- Proprioceptive window size w =
10
- PPO discount gamma =
0.998
- GAE lambda =
0.95
- PPO clip epsilon =
0.2
- PPO learning rate =
1e-4
- Reward weights in Table 1 =
5.0, 3.9, 6.0, 6.0, 20.0, 2.0, 0.81, -0.002
- PD gains Kp and Kd (Eq.16) =
Kp 200/300/40 Nm/rad; Kd 5/10/6/2 Nm s/rad
- Robot body scale in manual retargeting =
0.01
axioms (6)
- ad hoc to paper Manual retargeting labels are correct and task-faithful.
- domain assumption IsaacGym and MuJoCo dynamics adequately model the Unitree G1 and contact behavior.
- domain assumption The 14-joint representation captures all task-relevant humanoid motion and can be mapped to the 23 DoF G1.
- domain assumption Reward functions in Table 1 are sufficient to produce stable, task-faithful behavior.
- domain assumption PoseNet fine-tuned on 9000 frames gives human 3D poses accurate enough for retargeting.
- domain assumption Five student subjects' lab demonstrations generalize to real construction workers and sites.
Cite this review
Pith. "Pith review of Perception-and-action system for humanoid robot task execution in construction." pith.science (2026). https://pith.science/paper/QQCCA7TI
@misc{pith2026260801600,
author = {Pith},
title = {Pith review of: Perception-and-action system for humanoid robot task execution in construction},
year = {2026},
howpublished = {\url{https://pith.science/paper/QQCCA7TI}},
note = {Machine review of arXiv:2608.01600}
}
read the original abstract
Humanoid robots, with their human-like shape and multi-tasking capabilities, are well-aligned with human-dominated workplaces, like those in civil and construction engineering, where they could collaborate with human workers or autonomously perform physically demanding and hazardous tasks. Despite this promise, limited research has explored how to endow these robots with the practical capabilities needed to perform construction tasks. To this end, this study proposes a novel perception-and-action system that enables humanoid robots to learn and perform construction tasks from worker demonstrations. This system contains two deep networks: Humanoid-PoseNet, which extracts human postures and translates them into mechanically feasible poses for a humanoid robot; and Humanoid-ActionNet, which learns robot-executable actions based on these translated poses. Experimental results demonstrate that the humanoid robot reliably executed eight construction-related actions, achieving an average motion-tracking error of 82.45 mm MPJPE (Mean Per Joint Position Error). This work provides an early step toward deploying humanoid collaborators in construction.
Reference graph
Works this paper leans on
-
[1]
Aberman, K., Li, P., Lischinski, D., Sorkine-Hornung, O., Cohen-Or, D., & Chen, B. (2020). Skeleton-aware networks for deep motion retargeting. ACM Transactions on Graphics, 39 (4). https://doi.org/10.1145/3386569.3392462 Acosta, B., Yang, W., & Posa, M. (2022). Validating robotics simulators on real-world impacts. IEEE Robotics and Automation Letters, 7 ...
arXiv 2020
-
[14]
S., Demir, M., S ¸ eker, C., & Yıldırım, M
https://doi.org/10.1139/cjce-2025-0094 ¨ Ozbaltan, M., ¨ Ozbaltan, N., Bıçakcı Yes ¸ ilkaya, H. S., Demir, M., S ¸ eker, C., & Yıldırım, M. (2025). Task scheduling of multiple humanoid robot manipulators by using symbolic control. Biomimetics, 10 (6),
-
[16]
https://doi.org/10.1145/2816795.2818013 Martinez, J., Hossain, R., Romero, J., & Little, J. J. (2017). A simple yet effective baseline for 3d human pose estimation. In 2017 IEEE International Conference on Computer Vision (ICCV) (pp. 2659 – 2668). https://doi.org/10.1109/ICCV.2017.288 McCrory, S., Orsolino, R., Thanki, D., Penco, L., & Griffin, R. (2025)....
arXiv 2017
-
[18]
https://doi.org/10.1145/3197517.3201311 Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. Radosavovic, I., Xiao, T., Zhang, B., Darrell, T., Malik, J., & Sreenath, K. (2024). Real-...
-
[20]
Bin, Abbeel, P., Levine, S., & van de Panne, M
https://doi.org/10.1145/3450626.3459670 Peng, X. Bin, Abbeel, P., Levine, S., & van de Panne, M. (2018). DeepMimic. ACM Transactions on Graphics, 37 (4), 1 –
arXiv 2018
-
[178]
https://doi.org/ 10.1016/j.autcon.2013.08.002 Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016). Convolutional Pose Machines. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724 –
-
[186]
https://doi.org/10.1109/ TPAMI.2019.2929257 Chen, Z., Chen, K., Song, C., Zhang, X., Cheng, J. C., & Li, D. (2022). Global path planning based on BIM and physics engine for UGVs in indoor environments. Automation in Construction, 139 , Article 104263. https://doi.org/10.1016/j. autcon.2022.104263 Chen, Z., Ji, M., Cheng, X., Peng, Xuanbin, Peng, Xue Bin, ...
arXiv 2019
-
[264]
https://doi.org/ 10.1016/j.autcon.2018.05.002 Ross, S., Gordon, G., & Bagnell, D. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (pp. 627 –
-
[267]
International Labour Organization
https://doi.org/10.1016/j.shaw.2025.0 5.004 . International Labour Organization. (2015). Construction: A hazardous work. International Labour Organization . https://www.ilo.org/resource/construction-hazar dous-work . Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014). Human3.6M: Large scale datasets and predictive methods for 3D Human sensing i...
-
[306]
Tian, Y., Li, H., Cui, H., & Chen, J. (2022). Construction motion data library: An integrated motion dataset for on-site activity recognition. Scientific data, 9 (1),
work page 2022
-
[328]
https://doi.org/10.1109/JAS.2023.124140 Unitree Robotics. (2024). Unitree RL GYM. GitHub Repository Documentation . https://gi thub.com/unitreerobotics/unitree_rl_gym/tree/main . U.S. Bureau of Labor Statistics. (2025). Construction and extraction occupations. Occupational Outlook Handbook . https://www.bls.gov/ooh/construction-and-extract ion/ . V ¨ah ¨a...
-
[339]
https://doi.org/10.1061/9780784485248.040 Liu, Y., Jokar, D., Zheng, D., Habibnezhad, M., & Liu, Y. (2026). Advances in robotic perception for construction automation: A systematic literature review. In Proceedings of the 2026 Construction Research Congress (CRC) . Lu, C., Cheng, X., Li, J., Yang, S., Ji, M., Yuan, C., Yang, G., Yi, S., & Wang, X. (2025)....
arXiv 2026
-
[346]
https://doi.org/10.3390/biomimetics10060346 Pavlakos, G., Zhu, L., Zhou, X., & Daniilidis, K. (2018). Learning to estimate 3D Human pose and shape from a single color image. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 459 – 468). https://doi.org/10.1109/ CVPR.2018.00055 Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., ...
arXiv 2018
-
[533]
https://doi.org/10.1038/nature14236 Moon, G., Chang, J. Y., & Lee, K. M. (2019). Camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 10132 – 10141). https://doi. org/10.1109/ICCV.2019.01023 Myers, N., Kown, O., Yamsani, S., & Kim, J. (...
-
[630]
https://doi.org/10.1061/9780784483961.065 Liu, Y., & Jebelli, H. (2024). Intention-aware robot motion planning for safe worker – robot collaboration. Computer-Aided Civil and Infrastructure Engineering, 39 (15), 2242 –
-
[635]
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional continuous control using generalized advantage estimation . CoRR . Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. Sheng, Q., Zhou, Z., Li, J., Mi, X., Xiang, P., Chen, Z., Xu, H., Jia, S., Wu, X., Cui,...
2015
-
[726]
https://doi.org/10.1038/s41597-022-01841-1 Tong, Y., Liu, H., & Zhang, Z. (2024). Advancements in humanoid robots: A comprehensive review and future prospects. IEEE/CAA Journal of Automatica Sinica, 11 (2), 301 –
-
[729]
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2016). Fast and accurate deep network learning by Exponential linear units (ELUs). Construction Robotics. (2018). MULE 135 owner ’ s manual (Rev. A). Construction Robotics . https://www.construction-robotics.com/wp-content/uploads/2019/02/M ULE-135-MANUAL.pdf . Dao, J., Duan, H., & Fern, A. (2024). Sim-to...
arXiv 2016
-
[1165]
https://doi.org/10.1111/mice.13394 Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016). Benchmarking deep reinforcement learning for continuous control. Proceedings of the 33rd International Conference on International Conference on Machine Learning, 48 , 1329 – 1338 . Erez, T., Tassa, Y., & Todorov, E. (2015). Simulation tools for model-...
-
[1339]
https://doi.org/10.1109/TPAMI.2013.248 Ji, M., Peng, X., Liu, F., Li, J., Yang, G., Cheng, X., & Wang, X. (2025). ExBody2: Advanced expressive humanoid whole-body control. ArXiv Preprint ArXiv: 2412.13196. Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018). End-to-End recovery of Human shape and pose. In 2018 IEEE/CVF Conference on Computer Vis...
Pith/arXiv arXiv 2013
-
[2024]
Soto, D., Garcia, B., & Skibniewski, M. (2020). Future of robotics and automation in construction. Construction 4.0, 289 –
work page 2020
-
[2269]
https://doi.org/10.1111/mice.13129 Liu, Y., Ojha, A., & Jebelli, H. (2024). Vision-based ergonomic risk assessment of back- support exoskeleton for construction workers in material handling tasks. Computing in Civil Engineering, 2023 , 331 –
-
[4732]
Wetzel, E., Liu, J., Leathem, T., & Sattineni, A
https://doi.org/10.1109/CVPR.2016.511 . Wetzel, E., Liu, J., Leathem, T., & Sattineni, A. (2022). The use of Boston Dynamics SPOT in support of LiDAR scanning on active construction sites. In 2022 Proceedings of the 39th ISARC (pp. 86 – 92). https://doi.org/10.22260/ISARC2022/0014 Xiong, R., & Tang, P. (2021). Pose guided anchoring for detecting proper us...
-
[5263]
Kolotouros, N., Pavlakos, G., Black, M., & Daniilidis, K. (2019). Learning to reconstruct 3D Human pose and shape via model-fitting in the loop. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 2252 – 2261). https://doi.org/ 10.1109/ICCV.2019.00234 Li, Z., Yu, Y., Zeng, N., Tian, F., Zhang, S., Sun, H., & Li, Q. (2025). Human-centr...
arXiv 2019
-
[5468]
https://doi.org/10.3390/s21165468 Zhang, J., Wang, D., An, X., Lv, M., Chen, D., & Sun, A. (2025). A Voxel-based 3D reconstruction and action recognition method for construction workers. Advanced Engineering Informatics, 65, Article 103203. https://doi.org/10.1016/j. aei.2025.103203 Zhao, X., Liu, Y., & Jebelli, H. (2026). A Soft Pneumatic UAV Landing and...
-
[6478]
https://doi.org/ 10.1109/LRA.2022.3174367 Annabi, L., Ma, Z., & Nguyen, S. M. (2024). Unsupervised motion retargeting for human- robot imitation. In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (pp. 204 – 208). https://doi.org/10.1145/3610978.3640588 Asadi, K., Haritsa, V. R., Han, K., & Ore, J. P. (2021). Automated o...
-
[7173]
https:// doi.org/10.1109/TPAMI.2022.3222784 Ferrari, P., Rossini, L., Ruscelli, F., Laurenzi, A., Oriolo, G., Tsagarakis, N. G., & Mingo Hoffman, E. (2023). Multi-contact planning and control for humanoid robots: Design and validation of a complete framework. Robotics and Autonomous Systems, 166 , Article 104448. https://doi.org/10.1016/j.robot.2023.10444...
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.