REVIEW 3 major objections 3 minor 29 references
Hydra: Marker-Free RGB-D Hand-Eye Calibration
T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Hydra computes marker-free hand-eye calibration by aligning robot meshes to fused RGB-D point clouds with a robust point-to-plane ICP on SE(3), reaching roughly 90% success from only three arm poses and sub-second convergence.
desk verdict A practical marker-free hand-eye calibration pipeline built on standard robust ICP, with an honest experimental design but headline accuracy and success claims that outrun what the metric and data actually show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a robust point-to-plane iterative closest point objective written on the Lie algebra se(3). At each iteration, correspondences between mesh vertices and observed point-cloud points are found per robot configuration, and the pose update is the least-squares solution of ||W(A delta-$\theta$ - B)||^2, where delta-$\theta$ = (delta-omega, delta-tau) is an SE(3) increment, A is built from plane normals and the linearized exponential map, and W is a Huber-weight matrix with MAD-estimated scale. The Lie-group parameterization keeps every iterate a valid rigid transform, while the robust weighting and boundary-only erosion of the segmentation let the algorithm tolerate outliers and slide surfaces into alignment.
What would settle it
Take any calibrated setup from the paper's protocol and, in addition to the AprilTag-centric 2D reprojection, move the tag along the camera axis by a known amount and compare the predicted and observed depth displacements; if the depth error is much larger than the 5 mm figure while the 2D metric stays near 5 mm, the central accuracy claim fails.
Extended reading notes
Core claim
The paper's central claim is that marker-free hand-eye calibration can be solved as a direct 3D registration problem. Instead of detecting fiducials or training per-robot keypoints, Hydra segments the robot from RGB using SAM 2, erodes the segmentation so that only boundary depth points remain, fuses those point clouds across multiple robot configurations, and registers the robot's CAD mesh to the fused cloud with an ICP variant whose point-to-plane residual is minimized on the Lie group SE(3). The optimization is made robust by Huber reweighting of residuals, and a Kabsch-Umeyama centroid alignment supplies the initial guess. The paper reports that this formulation yields approximately 90% successful calibrations from only three randomly chosen configurations, a 2-3x higher convergence rate to the global optimum than both marker-based and marker-free baselines, convergence in 0.8 +/- 0.4 seconds for nine configurations, and task-space accuracy of about 5 mm, compared with around 7 mm for classical marker-based approaches.
Load-bearing premise
The headline task-space accuracy is computed only in a 2D view centred on an AprilTag, and the paper itself states this does not capture errors along the camera axis, so a calibration that is wrong in depth could still score 5 mm on the reported metric.
Editorial extensions
If this is right
- With only three randomly chosen robot poses, about nine in ten calibrations converge to the global optimum, so marker-free calibration no longer requires carefully hand-picked motions.
- On nine poses, calibration converges in about 0.8 seconds, two orders of magnitude faster than the differentiable-rendering marker-free baseline, making online or repeated calibration practical.
- Task-space reprojection accuracy is around 5 mm across three different arms and two RGB-D cameras, beating the classical marker-based baselines while using no marker at any stage.
- Success rates saturate with roughly six observed configurations for most robots, whereas marker-based baselines improve more slowly with sample count, so the method needs fewer data points.
- Because the objective treats the robot mesh as the calibration target, the same pipeline transfers to any serial manipulator that has a mesh, without per-robot training or marker placement.
Reading between the lines
- A natural extension the paper does not develop: the same point-to-plane registration cost is agnostic to which side of the kinematic chain is unknown, so the approach should also solve eye-in-hand calibration by treating the camera-to-gripper pose as the unknown, which the paper lists as future work.
- The depth-axis blind spot in the evaluation metric suggests a practical user should validate with a depth-varying test target or a grasp task before relying on the 5 mm figure; the speed and marker-free properties would survive even if the accuracy claim needs qualification.
- Because 90% success at three poses does not require specially chosen configurations, calibration could be re-run continuously in the background during normal robot operation, turning recalibration after any displacement into an automatic step.
- The method's practical ceiling is tied to segmentation quality: if the foundation segmentation model mislabels thin links or obstructing cables, the robust IRLS weighting may not fully compensate, which is a testable failure mode for dense or cluttered scenes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hydra, a marker-free eye-to-hand calibration method that segments a robot in RGB images with SAM 2, extracts corresponding point clouds from depth maps, and registers an articulated robot mesh to the fused point cloud using a robust point-to-plane ICP objective formulated on the SE(3) Lie algebra. The method is evaluated on three serial manipulators and two RGB-D cameras with a Monte Carlo cross-validation protocol, comparing against Tsai, Shah, PnP-based, and EasyHeC* baselines. The central claims are that with three robot configurations Hydra achieves approximately 90% successful calibrations, that it converges two to three times more often to the global optimum than baselines, that it is about two orders of magnitude faster than marker-free baselines, and that it reaches 5 mm task-space accuracy versus about 7 mm for classical approaches.
Significance. If the accuracy and success-rate claims held as stated, this would be a strong practical contribution: marker-free hand-eye calibration that is sample-efficient, sub-second, and applicable across different robots and cameras. The mathematical derivation in Section II-B is standard and appears correct, and the Monte Carlo cross-validation on real hardware is a sensible protocol. The open-source release of the benchmarking dataset, code, and ROS 2 integration is valuable for reproducibility and deployment. The main caveats are that the headline accuracy metric excludes the depth axis, the aggregate success-rate figure in the abstract is not supported by the data in Fig. 5, and the speed comparison is made against a modified, not the published, EasyHeC pipeline.
major comments (3)
- [III-A2] The quasi task-space metric used for all quantitative accuracy comparisons explicitly "does not capture errors along the camera axis." Because the metric reprojects AprilTag centers into the image and expresses the error in an AprilTag-centric 2D coordinate system scaled by tag size, a translation error of the hand-eye transform along the optical axis projects only weakly for the camera-to-robot distances shown in Fig. 3. Consequently, the headline "5 mm in task space" versus "7 mm in task space" comparison in the abstract and in Table I is not established for the depth axis. The authors should report a true 3D task-space error, an optical-axis error component, or clearly restrict the accuracy claims to the image-plane-projected component.
- [Abstract and Fig. 5] The abstract's claim of "approximately 90% successful calibrations" with three randomly chosen robot configurations is not supported by the reported data. At N=3 in Fig. 5, the Meca500 attains 0.53 with the RealSense and 0.67 with the ZED, and the average across the six robot-camera pairs is about 0.80, not 0.90. Please correct the aggregate success-rate statement or report the robot- and camera-specific values as the headline result.
- [IV-B and Table I] The claimed speed advantage is measured against EasyHeC*, which is not the published EasyHeC pipeline: the authors replaced PointRend with SAM 2 and PVNet initialization with a marker-based Shah initialization. The statement "2 orders of magnitude faster convergence time ... over other marker-free methods" should therefore be qualified as being against this adapted baseline on the xArm 7 DoF only. In addition, the sentence in Section IV-B1 that Hydra scores lower AprilTag center distances with smaller variations "across all baselines, except for the less practical PnP approach" is contradicted by Table I, where Shah is better than Hydra on the Meca500 with the ZED (4.3±1.5 mm versus 6.7±2.7 mm). The accuracy superiority claim needs to be restricted to the setups where the data support it.
minor comments (3)
- [II-A] The notation paragraph lists sigma and s as examples of both scalars and vectors without distinguishing which symbol is which; please use disjoint symbols or explicit type annotations.
- [Fig. 5] The numerical labels in Fig. 5 are very small and hard to read in the bottom two rows; enlarging them would make the Monte Carlo results much easier to verify.
- [IV-B2] The claim that "within standard deviation, there is little improvement in accuracy ... with number of robot configurations" is not accompanied by a statistical test; given the overlapping standard deviations, this is a qualitative observation and should be labeled as such.
Circularity Check
No significant circularity: Hydra's ICP registration is self-contained and evaluated against independent AprilTag reprojection.
full rationale
Hydra's derivation is a self-contained least-squares registration. Equation (9), solved via (10)-(11), minimizes a robust point-to-plane residual between robot mesh vertices and observed point clouds, with correspondences recomputed by nearest-neighbor search and weights derived from Huber/MAD residuals. The estimated quantity is the hand-eye transform Theta, fit directly against externally observed depth data and known forward kinematics; nothing in the objective is defined in terms of the AprilTag evaluation target. The quasi task-space metric in Section III-A2 re-projects AprilTag centers using the calibrated pose and compares them with detected AprilTag centers, so the evaluation signal is independent of the fitted residuals and of the point-to-plane objective. The only author self-citations, LBR-Stack [25] and OpTaS [26], support hardware and software setup, not any load-bearing derivation step. The runtime and success-rate comparisons are empirical benchmark results, not analytic consequences of the objective. The MAD-based sigma in Eq. (13) is an internal IRLS weighting parameter, not a fitted target quantity, and no fitted parameter is renamed as a prediction. Therefore no circular step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption Forward kinematics and robot mesh models accurately represent the physical robot.
- domain assumption SAM 2 segmentation and depth maps correctly isolate the robot and provide accurate depth.
- domain assumption AprilTag pose estimation provides ground truth for the evaluation metric.
- standard math The local linearization of the SE(3) exponential map and the Gauss-Newton/IRLS solution converge to a meaningful SE(3) optimum.
Cite this review
Pith. "Pith review of Hydra: Marker-Free RGB-D Hand-Eye Calibration." pith.science (2026). https://pith.science/paper/DL4DWXTP
@misc{pith2026250420584,
author = {Pith},
title = {Pith review of: Hydra: Marker-Free RGB-D Hand-Eye Calibration},
year = {2026},
howpublished = {\url{https://pith.science/paper/DL4DWXTP}},
note = {Machine review of arXiv:2504.20584}
}
read the original abstract
This work presents an RGB-D imaging-based approach to marker-free hand-eye calibration using a novel implementation of the iterative closest point (ICP) algorithm with a robust point-to-plane (PTP) objective formulated on a Lie algebra. Its applicability is demonstrated through comprehensive experiments using three well known serial manipulators and two RGB-D cameras. With only three randomly chosen robot configurations, our approach achieves approximately 90% successful calibrations, demonstrating 2-3x higher convergence rates to the global optimum compared to both marker-based and marker-free baselines. We also report 2 orders of magnitude faster convergence time (0.8 +/- 0.4 s) for 9 robot configurations over other marker-free methods. Our method exhibits significantly improved accuracy (5 mm in task space) over classical approaches (7 mm in task space) whilst being marker-free. The benchmarking dataset and code are open sourced under Apache 2.0 License, and a ROS 2 integration with robot abstraction is provided to facilitate deployment.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Redundancy-Aware Action Spaces for Robot Learning,
P. Mazzaglia, N. Backshall, X. Ma, and S. James, “Redundancy-Aware Action Spaces for Robot Learning,” IEEE Robotics and Automation Letters, vol. 9, no. 8, pp. 6912–6919, 2024
work page 2024
-
[2]
Universal Manipulation Interface: In-The- Wild Robot Teaching Without In-The-Wild Robots,
C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song, “Universal Manipulation Interface: In-The- Wild Robot Teaching Without In-The-Wild Robots,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10329
arXiv 2024
-
[3]
Camera-to-Robot Pose Estimation from a Single Image,
T. E. Lee, J. Tremblay, T. To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield, “Camera-to-Robot Pose Estimation from a Single Image,” 2020. [Online]. Available: https://arxiv.org/abs/1911.09231
arXiv 2020
-
[4]
Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer,
J. Lu, F. Richter, and M. C. Yip, “Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), June 2023, pp. 21 296–21 306
work page 2023
-
[5]
CtRNet-X: Camera-to-Robot Pose Estimation in Real-World Conditions Using a Single Camera,
J. Lu, Z. Liang, T. Xie, F. Ritcher, S. Lin, S. Liu, and M. C. Yip, “CtRNet-X: Camera-to-Robot Pose Estimation in Real-World Conditions Using a Single Camera,” 2024. [Online]. Available: https://arxiv.org/abs/2409.10441
arXiv 2024
-
[6]
Kalib: Markerless Hand-Eye Calibration with Keypoint Tracking,
T. Tang, M. Liu, W. Xu, and C. Lu, “Kalib: Markerless Hand-Eye Calibration with Keypoint Tracking,” 2024. [Online]. Available: https://arxiv.org/abs/2408.10562
arXiv 2024
-
[7]
Automatic Robot Hand- Eye Calibration Enabled by Learning-Based 3D Vision,
L. Li, X. Yang, R. Wang, and X. Zhang, “Automatic Robot Hand- Eye Calibration Enabled by Learning-Based 3D Vision,” Journal of Intelligent & Robotic Systems , vol. 110, no. 3, p. 130, 2024
work page 2024
-
[8]
S. Xing, F. Jing, and M. Tan, “RegHEC: Hand-Eye Calibration via Simultaneous Multi-view Point Clouds Registration of Arbitrary Object,” 2023. [Online]. Available: https://arxiv.org/abs/2304.14092
arXiv 2023
Show all 29 references
-
[9]
SAM 2: Segment Anything in Images and Videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer, “SAM 2: Segment Anything in Images and Videos,” 2024. [Online]. Avail...
2024 arXiv
-
[10]
EasyHeC: Accurate and Automatic Hand-Eye Calibration Via Differentiable Rendering and Space Exploration,
L. Chen, Y . Qin, X. Zhou, and H. Su, “EasyHeC: Accurate and Automatic Hand-Eye Calibration Via Differentiable Rendering and Space Exploration,” IEEE Robotics and Automation Letters , vol. 8, no. 11, pp. 7234–7241, 2023
2023
-
[11]
EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models,
Z. Hong, K. Zheng, and L. Chen, “EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 816–823
2024
-
[12]
Efficient Variants of the ICP Algo- rithm,
S. Rusinkiewicz and M. Levoy, “Efficient Variants of the ICP Algo- rithm,” in Proceedings Third International Conference on 3-D Digital Imaging and Modeling , 2001, pp. 145–152
2001
-
[13]
The Geometry of the Newton Method on Non-Compact Lie Groups,
R. Mahony and J. H. Manton, “The Geometry of the Newton Method on Non-Compact Lie Groups,” Journal of Global Optimization , vol. 23, no. 3-4, pp. 309–327, 2002
2002
-
[14]
Insight into Efficient Image Registration Techniques and the Demons Algorithm,
T. Vercauteren, X. Pennec, E. Malis, A. Perchant, and N. Ayache, “Insight into Efficient Image Registration Techniques and the Demons Algorithm,” in Information Processing in Medical Imaging . Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 495–506
2007
-
[15]
A Tutorial on SE(3) Transformation Parameterizations and On-Manifold Optimization,
J. L. Blanco-Claraco, “A Tutorial on SE(3) Transformation Parameterizations and On-Manifold Optimization,” 2022. [Online]. Available: https://arxiv.org/abs/2103.15980
2022 arXiv
-
[16]
Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives,
P. J. Green, “Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives,” Journal of the Royal Statistical Society: Series B (Methodological) , vol. 46, no. 2, pp. 149–170, 1984. [Online]. Available: https://rss. onlinel...
1984
-
[17]
PyTorch Kinematics,
S. Zhong, T. Power, A. Gupta, and P. Mitrano, “PyTorch Kinematics,” July 2024
2024
-
[18]
A Purely Algebraic Justification of the Kabsch-Umeyama Algorithm,
J. Lawrence, J. Bernal, and C. Witzgall, “A Purely Algebraic Justification of the Kabsch-Umeyama Algorithm,” Journal of Research of the National Institute of Standards and Technology , vol. 124, Oct
-
[19]
A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration,
R. Tsai and R. Lenz, “A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration,” IEEE Transactions on Robotics and Automation , vol. 5, no. 3, pp. 345–358, 1989
1989
-
[20]
Robust Hand- Eye Calibration via Iteratively Re-weighted Rank-Constrained Semi- Definite Programming,
C. Samant, A. Habed, M. d. Mathelin, and L. Goffin, “Robust Hand- Eye Calibration via Iteratively Re-weighted Rank-Constrained Semi- Definite Programming,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2019, pp. 4482–4489
2019
-
[21]
Solving the Robot-World/Hand-Eye Calibration Problem Using the Kronecker Product,
M. Shah, “Solving the Robot-World/Hand-Eye Calibration Problem Using the Kronecker Product,” Journal of Mechanisms and Robotics, vol. 5, no. 3, p. 031007, 06 2013. [Online]. Available: https://doi.org/10.1115/1.4024473
2013 doi
-
[22]
Robot Operating System 2: Design, Architecture, and Uses in the Wild,
S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, “Robot Operating System 2: Design, Architecture, and Uses in the Wild,” Science Robotics , vol. 7, no. 66, p. eabm6074,
-
[23]
AprilTag: A Robust and Flexible Visual Fiducial System,
E. Olson, “AprilTag: A Robust and Flexible Visual Fiducial System,” in 2011 IEEE International Conference on Robotics and Automation , 2011, pp. 3400–3407
2011
-
[24]
Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study,
D. Coleman, I. Sucan, S. Chitta, and N. Correll, “Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study,” 2014. [Online]. Available: https://arxiv.org/abs/1404.3785
2014 arXiv
-
[25]
LBR-Stack: ROS 2 and Python Integration of KUKA FRI for Med and IIW A Robots,
M. Huber, C. E. Mower, S. Ourselin, T. Vercauteren, and C. Bergeles, “LBR-Stack: ROS 2 and Python Integration of KUKA FRI for Med and IIW A Robots,”Journal of Open Source Software , vol. 9, no. 103, p. 6138, 2024. [Online]. Available: https://doi.org/10.21105/joss.06138
2024 doi
-
[26]
OpTaS: An Optimization-based Task Spec- ification Library for Trajectory Optimization and Model Predictive Control,
C. E. Mower, J. Moura, N. Z. Behabadi, S. Vijayakumar, T. Ver- cauteren, and C. Bergeles, “OpTaS: An Optimization-based Task Spec- ification Library for Trajectory Optimization and Model Predictive Control,” in 2023 IEEE International Conference on Robotics and Automation (ICR...
2023
-
[27]
Modular Primitives for High-Performance Differentiable Rendering,
S. Laine, J. Hellsten, T. Karras, Y . Seol, J. Lehtinen, and T. Aila, “Modular Primitives for High-Performance Differentiable Rendering,” ACM Trans. Graph. , vol. 39, no. 6, Nov. 2020. [Online]. Available: https://doi.org/10.1145/3414685.3417861
2020
-
[2019]
Available: http://dx.doi.org/10.6028/jres.124.028
[Online]. Available: http://dx.doi.org/10.6028/jres.124.028
-
[2022]
Available: https://www.science.org/doi/abs/10.1126/ scirobotics.abm6074
[Online]. Available: https://www.science.org/doi/abs/10.1126/ scirobotics.abm6074
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.