Bi-Manual Joint Camera Calibration and Scene Representation

Haozhan Tang; Matthew Johnson-Roberson; Tianyi Zhang; Weiming Zhi

REVIEW 3 major objections 6 minor 44 references

Bi-Manual Joint Camera Calibration and Scene Representation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A marker-free calibration pipeline for dual-arm robots recovers both hand-eye transforms, the inter-base pose, and a metric 3D scene from RGB images and forward kinematics alone.

desk verdict Useful bi-manual extension with an encouraging demo, but the printed base-to-base derivation doesn't work as written and the evaluation needs tightening. read the letter →

arxiv 2505.24819 v1 pith:7KPECSIN submitted 2025-05-30 cs.RO cs.CVcs.LG

Haozhan Tang , Tianyi Zhang , Matthew Johnson-Roberson , Weiming Zhi This is my paper

classification cs.ROcs.CVcs.LG

keywords hand-eyecalibrationbimanualmanipulation3Dfoundationmodelsmarker-freemetricscalerecoveryscenerepresentationDUSt3Rdual-armrobots

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents Bi-JCR, a pipeline that calibrates two robot arms with wrist-mounted cameras without any calibration markers. From the same set of RGB images, it recovers the camera-to-hand transform for each arm, the relative pose between the two robot bases, and a metric-scale dense 3D reconstruction of the shared tabletop. The method replaces checkerboards with dense correspondences from a 3D foundation model, then solves a joint scale-recovery and hand-eye calibration problem using only forward kinematics. If it works as claimed, bimanual robots can be set up for coordinated grasping and handovers from plain photos alone.

What carries the argument

The load-bearing mechanism is the hand-eye equation $T^{E,i+1}_{E,i} T^E_C = T^E_C T^{P_w,i+1}_{P_w,i}(\lambda)$, an $AX = XB$ system whose right side is scaled by the unknown $\lambda$ recovered from the 3D foundation model. A 3D foundation model (DUSt3R in the main experiments) supplies dense multi-view correspondences and unscaled camera poses from the RGB images, after which rotation components are solved through the Lie-algebra log map and translations plus scale by least squares, followed by a manifold-constrained gradient descent refinement.

What would settle it

Run Bi-JCR on a dual-arm rig with AprilTag ground truth for the base-to-base pose, implement Eq. (17) exactly as printed, and compare $T^{b_1}_{b_2}$ to the tag-measured value; if the residual is large while hand-eye residuals are small, the printed formula is the culprit. A cheaper check is symbolic: substitute Eqs. (6)-(8) into the right-hand side of Eq. (17) and observe that it equals $T^E_C$ rather than $T^b_w$.

Watch

Extended reading notes

Core claim

Bi-JCR claims that the classical hand-eye calibration problem can be extended to two independently moving arms and solved jointly with scene reconstruction by treating the foundation model's output scale as an unknown. The paper formulates the hand-eye constraint for each arm, $AX = XB$ with the right-hand side depending on a shared scale factor $\lambda$, solves for the rotation components via the $\mathrm{SO}(3)$ log map, then refines translations and $\lambda$ by gradient descent on the manifold of transformations. In the same optimization it recovers the world-to-base transform for each arm, yielding the secondary base pose in the primary frame through $P^{b_1}_{b_2} = T^{b_1}_w (T^{b_2}_w)^{-1}$. The authors report end-to-end demonstration of metric-scale reconstruction with object-size errors at most 2.98%, and successful bimanual grasps and handovers in real tabletop scenes.

Load-bearing premise

The load-bearing premise is that Eq. (17), which converts the optimized hand-eye solutions into each arm's world-to-base transform, is mathematically correct; as written the substitution of Eqs. (6) and (7) into its averaged term yields the hand-eye transform itself rather than $T^b_w$, so the recovered inter-base pose would be wrong if the implemented code follows the printed formula.

Editorial extensions

If this is right

If the recovered base-to-base transform is correct, the secondary arm's end-effector poses can be planned directly in the primary arm's frame, enabling coordinated grasps without external motion capture.
The metric scale factor means the reconstructed point cloud can be used for collision checking and grasp planning in simulation with real-world units.
Removing markers makes recalibration practical whenever cameras are bumped or remounted, since only new RGB images and forward kinematics are needed.
The framework is agnostic to the underlying 3D foundation model, so improvements in these models should transfer directly to calibration accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The world-to-base formula in Eq. (17) appears to collapse to the hand-eye transform under substitution; if that is a typographical error in the paper rather than the implemented code, the published derivation needs a correction before the base-pose claim is reproducible.
The framework could be extended to more than two arms with a factor-graph formulation, where each arm adds hand-eye constraints and a shared scale factor.
Confidence masks from the foundation model could drive active view selection, as the paper notes for future work, suggesting a testable extension where images are chosen to minimize predicted calibration residual.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bi-JCR is a reasonable next step from the authors' single-arm calibration paper, and the bi-manual joint estimation of hand-eye transforms, shared scale, and base-to-base pose from marker-free RGB images is genuinely new. The real-robot handover and grasping demos are encouraging evidence that the pipeline works end-to-end. The literature review is fair, and citing their own [8] for the scale-recovery subproblem is appropriate.

That said, the reader and the stress-test are right about Eq. (17). As printed, the averaged term A_i X B_i^{-1} collapses to the hand-eye transform X via Eq. (8), so it cannot recover T^b_w. This is load-bearing: the reconstruction is placed in the robot frames using that transform. The fact that the demos succeed suggests the code uses absolute camera poses rather than the relative form in Eq. (17), i.e., the printed equation is likely a typo. But without code or a corrected derivation, a reader cannot verify the central claim. This needs to be fixed before publication.

The evaluation is thinner than the claims. Table I's residual is essentially the optimized cost, so it doesn't independently validate accuracy. The object-size errors are an independent scale check, but the recovered base-to-base pose never gets a quantitative error against Apriltag ground truth; the PyBullet visualization and successful handovers are qualitative. Three tabletop scenes and no error bars make the robustness claims modest. These are fixable with more experiments and clearer metrics.

Who is this for? Robotics labs that use two wrist-mounted cameras and want to skip marker-based calibration. They will find the method appealing and the demos persuasive. As a referee, I would engage with it, but I'd ask for a corrected derivation, a non-circular evaluation metric, and ideally code before acceptance. The paper deserves peer review rather than desk rejection, because the contribution is real and the flaws are repairable.

Referee Report

3 major / 6 minor

Summary. The paper proposes Bi-JCR, a method for jointly calibrating the hand-eye transforms of two wrist-mounted cameras, recovering the relative pose between the two robot bases, and building a metric-scale dense 3D reconstruction of the shared workspace, using only RGB images and forward kinematics. The pipeline runs a 3D foundation model (DUSt3R) to obtain unscaled camera poses and point clouds, solves an initial closed-form hand-eye calibration with scale recovery, refines the result by gradient descent on SE(3), and then derives the world-to-base and base-to-base transforms. The authors evaluate on real tabletop scenes with three different environments, report residuals for calibration, object-size errors for scale recovery, ablations over foundation models and gradient descent, and demonstrate bimanual grasping and handover tasks.

Significance. If the method works as described, it is a practically valuable step toward marker-free calibration in bimanual manipulation, replacing checkerboard/AprilTag procedures with foundation-model correspondences. The paper includes real-robot experiments, downstream manipulation demonstrations, and ablations over foundation models and refinement, which are strengths. The scale-recovery validation via object sizes is convincing evidence that the representation is metric when the calibration is correct. However, the central derivation of the world-to-base transform is algebraically inconsistent as written, which undermines the paper's main claim until corrected.

major comments (3)

[III.D, Eq. (17)] Equation (17) as written does not recover the world-to-base transform. Substituting the hand-eye equation (8), A_i X = X B_i, into each summand gives A_i X B_i^{-1} = X, i.e., the hand-eye transform T^{Em}_{Cm}, independent of i. The SE(3) average therefore equals the hand-eye transform, not T^{bm}_w. Consequently, Eqs. (18)-(20) would transform points by the wrong transformation and Eq. (20) would give X_1 X_2^{-1}, not the base-to-base pose. If this is a typographical error and the intended expression is an average of E_{m,i} T^{Em}_{Cm} (P^w_{m,i}(λ))^{-1} (or the equivalent), the correct equation and a derivation must be provided. Since this step is the only route from the foundation-model frame to the robot base frames, the base-to-base recovery and all downstream bimanual claims rest on fixing this point.
[IV.A, Table I (and Tables III/IV)] The residual metric used in the quantitative tables is not specified unambiguously. The text states that residuals are computed 'via Equation (15) with ground truth, obtained via Apriltags, on the right-hand side,' but Equation (15) is the same expression minimized in the cost function (16). If the reported residual uses the estimated transforms, then Tables I, III, and IV report the training objective on the training data, which is a consistency measure rather than an independent calibration error. If ground-truth transforms are substituted into Equation (15), the reader needs a precise description of which quantities are replaced and how the residual is computed. Please clarify the metric and, ideally, report errors of the estimated T^{Em}_{Cm} and P^{b1}_{b2} directly against the AprilTag ground truth.
[IV.B and IV.C] The scale factor is validated quantitatively through object-size errors, but the base-to-base transformation P^{b1}_{b2} and the world-to-base transformations T^{b1}_w and T^{b2}_w are not evaluated against ground truth. The only support for the base-to-base recovery is a qualitative PyBullet visualization and the success of downstream tasks. Given that Equation (17) is problematic and the base-to-base transform is a central output, a quantitative evaluation (e.g., translation/rotation error of the recovered base-to-base pose with respect to a measured ground truth) should be reported before the method can be accepted as accurate.

minor comments (6)

[IV.A title] The heading 'Eyes-to-Hands Calibration' mixes singular and plural; 'Eye-to-Hand Calibration' or 'Eye-to-Hand Calibration of Both Arms' would be more standard.
[III.C, Eq. (12)] The scale recovery problem is attributed to the authors' prior work [8] without a self-contained description. Since the formulation in Eq. (12) is central to the initial solution, a brief derivation or restatement of the assumptions (e.g., single global scale, known rotations fixed from Eq. (11)) would help.
[III.D, Eq. (16)] The weighting parameter α in the cost function is never assigned a value or ablated in the experiments. Please state the default value and, ideally, its sensitivity.
[IV.B, Table II] The text reports a 'median of 1.48%' over the six objects, but it is unclear whether this median is over all objects and scenes or just one scene; also, stating the full distribution (e.g., min/max) would be more informative than only the maximum.
[IV.A] The 'Converged' criterion for COLMAP in Table I is not defined (e.g., whether it requires a minimum number of registered poses or a residual threshold). Please define it so the readers can interpret the NA entries.
[Reproducibility] No code or data are provided; given the ambiguity in Equation (17), releasing the implementation or the derivation script would substantially help verify the correctness of the base-to-base recovery.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (17) collapses to the already-solved hand-eye transform by the paper's own Eq. (8), so the claimed world-to-base and inter-base recovery reduces by construction to a fitted quantity.

self definitional [Section III.D, Eq. (17), together with Eq. (8)]
"With the solutions that minimize the cost function, we can then obtain the world-to-base transformations T b1 w , and T b2 w via T bm w = A VGSE3 i∈{1,··· ,N −1} ( T Em,i+1 Em,i T Em Cm T P w m,i+1 P w m,i −1 ) (17). The hand-eye equation (8) states: T Em,i+1 Em,i T Em Cm = T Em Cm T P w m,i+1 P w m,i (λ)."

Substituting Eq. (8) into the summand of Eq. (17) gives T^{Em,i+1}_{Em,i} T^{Em}_{Cm} (T^{Pw,m,i+1}_{Pw,m,i})^{-1} = (T^{Em}_{Cm} T^{Pw,m,i+1}_{Pw,m,i}) (T^{Pw,m,i+1}_{Pw,m,i})^{-1} = T^{Em}_{Cm}. Every term in the SE(3) average is therefore the same constant, the hand-eye transform X_m = T^{Em}_{Cm} that was already solved in the same optimization. Thus T^{bm}_w as written is not an independent estimate of the world-to-base transform; it is exactly the fitted hand-eye transform. Eqs. (18) and (20) then use this X_m as T^{b1}_w and compute P^{b1}_{b2} = X_1 X_2^{-1}, so the claimed recovery of the base-frame alignment and the inter-base pose reduces by construction to the already-known hand-eye solutions. The paper's assertion that Eq.

full rationale

Bi-JCR's scale recovery is not circular: the scale factor lambda is estimated jointly with hand-eye translations from forward kinematics and DUSt3R's unscaled poses through Eqs. (12)-(14), and the object-size measurements in Table II are used only for evaluation, not as fitting targets. The citation of the authors' prior work [8] for the 'similar scale recovery problem' is also not load-bearing by itself, because the objective is restated in full in the paper. The one structural reduction is Eq. (17), where, by the paper's own hand-eye equation (8), each averaged term equals the hand-eye transform already recovered in the same optimization. As written, this makes the world-to-base transform and the downstream inter-base pose P^{b1}_{b2} equal to the fitted hand-eye transforms by construction, which is a genuine circular step in the claimed derivation chain. If the implemented code averages a different expression, Eq. (17) would be a typo, but no code is released, so the written derivation is the only evidence for the claimed recovery of the world-to-base and base-to-base transformations. No other step in the paper fits a parameter to the validation targets or imports a uniqueness theorem from the authors, so the rest of the pipeline is self-contained.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the pre-trained 3D foundation model producing scale-consistent geometry, on accurate forward kinematics, and on the correctness of the SE(3) algebra in Eq. (17). No new physical entities are introduced; the scale factor is a fitted parameter, not an entity.

free parameters (2)

Scale factor lambda = recovered by least squares, reported scale error 0.26-2.98%
Shared scale factor aligning the foundation model's unscaled reconstruction with metric units; estimated from forward kinematics and DUSt3R correspondences.
Confidence threshold for point filtering = not specified in the paper
Used to filter low-confidence points from the foundation model output; the threshold is chosen by hand and not reported in the text.

assumptions (4)

domain assumption The chosen 3D foundation model (DUSt3R) produces globally scale-consistent camera poses and point clouds for the tabletop scenes.
The method relies on the foundation model's multi-view geometry being correct up to a single global scale; the paper tests DUSt3R, MASt3R, and VGGT but does not prove this property.
domain assumption Robot forward kinematics are known and accurate.
End-effector poses E_i are used as ground-truth motion in the hand-eye equations; errors in kinematics would bias the calibration.
domain assumption The hand-eye transform T^E_C is constant during data collection.
Cameras are rigidly mounted to the grippers; any flex or slip would violate the model.
domain assumption The executed motions provide sufficient rotation and translation diversity to make the AX=XB and scale recovery problems well-posed.
Degenerate motions would make calibration underdetermined; the paper does not analyze this condition.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bi-Manual Joint Camera Calibration and Scene Representation." pith.science (2026). https://pith.science/paper/7KPECSIN

@misc{pith2026250524819,
  author       = {Pith},
  title        = {Pith review of: Bi-Manual Joint Camera Calibration and Scene Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KPECSIN}},
  note         = {Machine review of arXiv:2505.24819}
}

read the original abstract

Robot manipulation, especially bimanual manipulation, often requires setting up multiple cameras on multiple robot manipulators. Before robot manipulators can generate motion or even build representations of their environments, the cameras rigidly mounted to the robot need to be calibrated. Camera calibration is a cumbersome process involving collecting a set of images, with each capturing a pre-determined marker. In this work, we introduce the Bi-Manual Joint Calibration and Representation Framework (Bi-JCR). Bi-JCR enables multiple robot manipulators, each with cameras mounted, to circumvent taking images of calibration markers. By leveraging 3D foundation models for dense, marker-free multi-view correspondence, Bi-JCR jointly estimates: (i) the extrinsic transformation from each camera to its end-effector, (ii) the inter-arm relative poses between manipulators, and (iii) a unified, scale-consistent 3D representation of the shared workspace, all from the same captured RGB image sets. The representation, jointly constructed from images captured by cameras on both manipulators, lives in a common coordinate frame and supports collision checking and semantic segmentation to facilitate downstream bimanual coordination tasks. We empirically evaluate the robustness of Bi-JCR on a variety of tabletop environments, and demonstrate its applicability on a variety of downstream tasks.

Figures

Figures reproduced from arXiv: 2505.24819 by the authors.

**Figure 3.** Left: We have cameras (outlined in red) rigidly attached to [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

**Figure 4.** 3D foundation model taking a set of RGB images, and output [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (6 more)

**Figure 6.** Figure 6: Qualitative evaluation of the camera calibration in the three [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: We observe that Bi-JCR reconstructs the relative [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 7.** Figure 7: Qualitative evaluations of the recovered 3D reconstructions, including scene A, scene B, scene C from left to right. The reconstruction [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Visualization of the base to base transformation recovery [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 12.** Figure 12: We demonstrate passing of the objects: wrench, spoon, [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]

**Figure 11.** Figure 11: Top: Selected objects (wrench, spoon, balance meter and [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 37 canonical work pages

[8]

Unifying representation and calibration with 3d foundation models,

W. Zhi, H. Tang, T. Zhang, and M. Johnson-Roberson, “Unifying representation and calibration with 3d foundation models,” IEEE Robotics and Automation Letters , 2024

work page 2024
[1]

Apriltag: A robust and flexible visual fiducial system,

E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in IEEE International Conference on Robotics and Automation , 2011

work page 2011
[2]

A new technique for fully autonomous and efficient 3d robotics hand/eye calibration,

R. Y . Tsai and R. K. Lenz, “A new technique for fully autonomous and efficient 3d robotics hand/eye calibration,” IEEE Trans. Robotics Autom., 1988

work page 1988
[3]

Hand-eye calibration,

R. Horaud and F. Dornaika, “Hand-eye calibration,” I. J. Robotic Res., 1995

work page 1995
[4]

Robot Sensor Calibration: Solving AX = XB on the Euclidean Group,

F. Park and B. Martin, “Robot Sensor Calibration: Solving AX = XB on the Euclidean Group,” IEEE Transactions on Robotics and Automation, 1994

work page 1994
[5]

Learning eye-in-hand camera calibration from a single image,

E. Valassakis, K. Dreczkowski, and E. Johns, “Learning eye-in-hand camera calibration from a single image,” in Proceedings of the 5th Conference on Robot Learning , 2022

work page 2022
[6]

Decision transformer: Re- inforcement learning via sequence modeling,

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Re- inforcement learning via sequence modeling,” in Advances in Neural Information Processing Systems , 2021

work page 2021
[7]

Implicit behavioral cloning,

P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” in Conference on Robot Learning (CoRL) , 2021

work page 2021

Show all 44 references

[9]

3d foundation models enable simultaneous geometry and pose estimation of grasped objects,

W. Zhi, H. Tang, T. Zhang, and M. Johnson-Roberson, “3d foundation models enable simultaneous geometry and pose estimation of grasped objects,” arXiv preprint arXiv:2407.10331 , 2024

2024 arXiv
[10]

Simultaneous hand–eye, tool–flange, and robot–robot calibration for comanipulation by solving the axb=ycz problem,

L. Wu, J. Wang, L. Qi, K. Wu, H. Ren, and M. Q.-H. Meng, “Simultaneous hand–eye, tool–flange, and robot–robot calibration for comanipulation by solving the axb=ycz problem,” IEEE TRansactions on robotics, vol. 32, no. 2, pp. 413–428, 2016

2016
[11]

A dual quaternion-based approach for coordinate calibration of dual robots in collaborative motion,

Z. Fu, J. Pan, E. Spyrakos-Papastavridis, X. Chen, and M. Li, “A dual quaternion-based approach for coordinate calibration of dual robots in collaborative motion,” IEEE Robotics and Automation Letters , vol. 5, no. 3, pp. 4086–4093, 2020

2020
[12]

Simultaneously calibration of multi hand–eye robot system based on graph,

Z. Zhou, L. Ma, X. Liu, Z. Cao, and J. Yu, “Simultaneously calibration of multi hand–eye robot system based on graph,” IEEE Transactions on Industrial Electronics , vol. 71, no. 5, pp. 5010–5020, 2023

2023
[13]

Sonar-based real-world mapping and navigation,

A. Elfes, “Sonar-based real-world mapping and navigation,” IEEE Journal on Robotics and Automation , 1987

1987
[14]

Shape modeling with front propagation: a level set approach,

R. Malladi, J. A. Sethian, and B. C. Vemuri, “Shape modeling with front propagation: a level set approach,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 1995

1995
[15]

Kinectfusion: Real-time dense surface mapping and tracking,

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, 2011

2011
[16]

Contextual oc- cupancy maps using gaussian processes,

S. O’Callaghan, F. T. Ramos, and H. Durrant-Whyte, “Contextual oc- cupancy maps using gaussian processes,” in 2009 IEEE International Conference on Robotics and Automation , 2009

2009
[17]

Continuous occupancy map fusion with fast bayesian hilbert maps,

W. Zhi, L. Ott, R. Senanayake, and F. Ramos, “Continuous occupancy map fusion with fast bayesian hilbert maps,” in International Confer- ence on Robotics and Automation (ICRA) , 2019

2019
[18]

Spatiotemporal learning of directional uncertainty in urban environments with kernel recurrent mixture density networks,

W. Zhi, R. Senanayake, L. Ott, and F. Ramos, “Spatiotemporal learning of directional uncertainty in urban environments with kernel recurrent mixture density networks,” IEEE Robotics and Automation Letters , 2019

2019
[19]

Deepsdf: Learning continuous signed distance functions for shape representation,

J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

2019
[20]

Pointnet: Deep learning on point sets for 3d classification and segmentation,

R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017

2017
[21]

Pointnet++: deep hierarchical feature learning on point sets in a metric space,

C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: deep hierarchical feature learning on point sets in a metric space,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017

2017
[22]

Global and reactive motion generation with geometric fabric command se- quences,

W. Zhi, I. Akinola, K. van Wyk, N. Ratliff, and F. Ramos, “Global and reactive motion generation with geometric fabric command se- quences,” in IEEE International Conference on Robotics and Automa- tion, ICRA, 2023

2023
[23]

Nerf: Representing scenes as neural radiance fields for view synthesis,

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoor- thi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in ECCV, 2020

2020
[24]

Instant neural graphics primitives with a multiresolution hash encoding,

T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., 2022

2022
[25]

Darkgs: Learning neural illumination and 3d gaussians relighting for robotic exploration in the dark,

T. Zhang, K. Huang, W. Zhi, and M. Johnson-Roberson, “Darkgs: Learning neural illumination and 3d gaussians relighting for robotic exploration in the dark,” 2024

2024
[26]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” CoRR, 2023

2023
[27]

Learning transferable visual models from natural language supervi- sion,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” CoRR, 2021

2021
[28]

Dust3r: Geometric 3d vision made easy,

S. Wang, V . Leroy, Y . Cabon, B. Chidlovskii, and J. Revaud, “Dust3r: Geometric 3d vision made easy,” in CVPR, 2024

2024
[29]

Depth anything: Unleashing the power of large-scale unlabeled data,

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

2024
[30]

Grounding image matching in 3d with mast3r,

V . Leroy, Y . Cabon, and J. Revaud, “Grounding image matching in 3d with mast3r,” 2024

2024
[31]

Vggt: Visual geometry grounded transformer,

J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

2025
[32]

On the opportunities and risks of foundation models,

R. Bommasani and et al., “On the opportunities and risks of foundation models,” CoRR, 2021

2021
[33]

Foundation models in robotics: Applications, challenges, and the future,

R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y . Zhu, S. Song, A. Kapoor, K. Hausman, B. Ichter, D. Driess, J. Wu, C. Lu, and M. Schwager, “Foundation models in robotics: Applications, challenges, and the future,” CoRR, 2023

2023
[34]

Pybullet, a python module for physics simulation for games, robotics and machine learning

E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning.” http:// pybullet.org, 2016–2019

2016
[35]

C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine learning, vol. 4. Springer, 2006

2006
[36]

Deep regression on manifolds: a 3D rotation case study,

R. Br ´egier, “Deep regression on manifolds: a 3D rotation case study,” 2021

2021
[37]

Structure-from-motion revisited,

J. L. Sch ¨onberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2016

2016
[38]

Cameras as rays: Pose estimation via ray diffusion,

J. Y . Zhang, A. Lin, M. Kumar, T.-H. Yang, D. Ramanan, and S. Tulsiani, “Cameras as rays: Pose estimation via ray diffusion,” in International Conference on Learning Representations (ICLR) , 2024

2024
[39]

Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,

J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny, “Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction,” in International Conference on Computer Vision , 2021

2021
[40]

C. M. Bishop, Pattern recognition and machine learning, 5th Edition . Information science and statistics, Springer, 2007

2007
[41]

Graphseg: Segmented 3d representations via graph edge addition and contraction,

H. Tang, T. Zhang, O. Kroemer, M. Johnson-Roberson, and W. Zhi, “Graphseg: Segmented 3d representations via graph edge addition and contraction,” arXiv preprint arXiv:2504.03129 , 2025

2025 arXiv
[42]

Grasp pose detection in point clouds,

A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,” The International Journal of Robotics Research, vol. 36, no. 13-14, pp. 1455–1473, 2017

2017
[43]

Diffeomorphic transforms for generalised imitation learning,

W. Zhi, T. Lai, L. Ott, and F. Ramos, “Diffeomorphic transforms for generalised imitation learning,” in Learning for Dynamics and Control Conference, L4DC, 2022

2022
[44]

Instructing robots by sketching: Learning from demonstration via probabilistic diagram- matic teaching,

W. Zhi, T. Zhang, and M. Johnson-Roberson, “Instructing robots by sketching: Learning from demonstration via probabilistic diagram- matic teaching,” in IEEE International Conference on Robotics and Automation, 2024

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [8]

Unifying representation and calibration with 3d foundation models,

W. Zhi, H. Tang, T. Zhang, and M. Johnson-Roberson, “Unifying representation and calibration with 3d foundation models,” IEEE Robotics and Automation Letters , 2024

work page 2024

[2] [1]

Apriltag: A robust and flexible visual fiducial system,

E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in IEEE International Conference on Robotics and Automation , 2011

work page 2011

[3] [2]

A new technique for fully autonomous and efficient 3d robotics hand/eye calibration,

R. Y . Tsai and R. K. Lenz, “A new technique for fully autonomous and efficient 3d robotics hand/eye calibration,” IEEE Trans. Robotics Autom., 1988

work page 1988

[4] [3]

Hand-eye calibration,

R. Horaud and F. Dornaika, “Hand-eye calibration,” I. J. Robotic Res., 1995

work page 1995

[5] [4]

Robot Sensor Calibration: Solving AX = XB on the Euclidean Group,

F. Park and B. Martin, “Robot Sensor Calibration: Solving AX = XB on the Euclidean Group,” IEEE Transactions on Robotics and Automation, 1994

work page 1994

[6] [5]

Learning eye-in-hand camera calibration from a single image,

E. Valassakis, K. Dreczkowski, and E. Johns, “Learning eye-in-hand camera calibration from a single image,” in Proceedings of the 5th Conference on Robot Learning , 2022

work page 2022

[7] [6]

Decision transformer: Re- inforcement learning via sequence modeling,

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Re- inforcement learning via sequence modeling,” in Advances in Neural Information Processing Systems , 2021

work page 2021

[8] [7]

Implicit behavioral cloning,

P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” in Conference on Robot Learning (CoRL) , 2021

work page 2021