Pith. sign in

REVIEW 5 major objections 6 minor 45 references

Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One hybrid representation lets a single differentiable pipeline recover object meshes, appearance, and robot and camera poses directly from raw, imperfect robot trajectories.

desk verdict A genuinely useful new hybrid representation (mesh + surface-bound Gaussians) and an honest end-to-end pipeline, but the real-world novel-view numbers are weakened by test-time pose alignment and the abstract oversells 'physical parameters.' read the letter →

arxiv 2506.04120 v2 pith:NN54VLB6 submitted 2025-06-04 cs.RO cs.GR

Ben Moran , Mauro Comi , Arunkumar Byravan , Steven Bohez , Tom Erez , Zhibin Li , Leonard Hasenclever This is my paper
classification cs.ROcs.GR
keywords 3DGaussianSplattingdifferentiablerenderingphysicsMuJoCoreal-to-simrobotcalibrationmeshreconstructionALOHA2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that real-to-sim — creating a physically usable simulation from real robot observations — can be solved as a single differentiable optimization instead of a chain of separate reconstruction, calibration, and meshing steps. Its central object is SplatMesh, a hybrid representation that couples the photorealistic appearance of 3D Gaussian Splatting with explicit triangle meshes suited to physics simulation, with Gaussians constrained to lie on the mesh surface. Because both the renderer and the MuJoCo physics backend are differentiable, pixel-level photometric errors propagate back through the whole system to mesh vertices, camera extrinsics, and robot joint angles. If this works, a robot that simply films an object while moving its arms can yield, from its own imperfect onboard video, a metric-scale object mesh, photorealistic novel views, and calibrated robot and camera poses, with no extra data collection, no structure-from-motion preprocessing, and no manual annotation. That would make high-fidelity simulation assets cheap enough to generate from ordinary robot operation, which is what makes large-scale robot learning from simulation practical.

What carries the argument

SplatMesh, a hybrid scene representation in which a deformable triangle mesh fixes the geometry and 3D Gaussians carry the appearance. Gaussian means are sampled on mesh faces via barycentric weights, so appearance follows the mesh as it deforms; covariance is kept axis-aligned to the face normal with the normal-direction scale clamped near zero, turning each Gaussian into a surface element rather than a volumetric blob. The same differentiable pipeline then couples the 3D Gaussian Splatting rasterizer with MuJoCo's JAX-based MJX physics, so losses on rendered RGB, silhouettes, and estimated surface normals, together with Laplacian mesh smoothing, back-propagate in one pass to mesh vertices, Gaussian parameters, camera extrinsics, and robot joint angles.

What would settle it

Deliberately corrupt the robot model with a known systematic error — for example, lengthen one arm link by 2 cm or add a fixed 0.05 rad offset to one joint encoder — then run the pipeline on a real trajectory and measure the recovered object mesh against ground truth. If the joint-angle and camera-extrinsic parameterization cannot absorb systematic (as opposed to random) error, the recovered geometry will be measurably biased, which directly tests the paper's premise that the simulator needs only 'reasonably accurate' geometry and kinematics.

Watch

Extended reading notes

Core claim

The central claim is that visual and physical reconstruction of a dynamic robot scene should be one problem, not two: the same scene is simultaneously a set of 3D Gaussians for rendering and a triangle mesh for simulation, and the two are coupled because Gaussian means live on mesh faces through barycentric coordinates while Gaussian covariance stays axis-aligned to the face normal with near-zero normal extent. Minimizing a weighted sum of photometric, silhouette, normal-consistency, and Laplacian-smoothing losses — through differentiable rasterization and differentiable MuJoCo kinematics — jointly refines object geometry, appearance, robot joint angles, and camera poses directly from raw RGB and proprioception. On a low-cost ALOHA 2 bi-manual platform with two fixed and two wrist-mounted RGB cameras, the authors report reconstructed YCB object meshes with square-root Chamfer distances between 3.1 and 7.4 mm, novel-view PSNR between 21.6 and 25.5 dB versus 14.0 to 20.2 dB without pose calibration, and, in simulation, tool-center-point error roughly halved under joint-angle noise. The same framework doubles as an asset generator, turning single images or text prompts into textured meshes importable into MuJoCo.

Load-bearing premise

The pipeline assumes the supplied MuJoCo model of the ALOHA 2 robot is accurate enough in link lengths, joint offsets, and kinematics that every real-robot error can be absorbed by optimizing joint angles and camera extrinsics alone; systematic kinematic errors in that model would silently bias the recovered scene.

Editorial extensions

If this is right

  • A single optimization pass recovers metric object meshes, photorealistic novel views, and robot and camera calibration at once, from raw RGB and proprioception alone, with no dedicated scanning session, COLMAP, or manual pose annotation.
  • Because recovered objects come out at metric scale with 6-DoF pose in the robot workspace, the meshes are simulation-ready as produced, and where a simulator cannot render Gaussians directly, the appearance can be baked into a standard texture map.
  • On the simulated YCB benchmark the full method outperforms both NeRFacto and vanilla 3DGS at the same 15,000-iteration budget (PSNR 30.91 vs 30.29 and 26.97), and the ablations show that both mesh regularization and the surface-aligned (surfel) constraint are needed for that margin.
  • Jointly optimizing poses makes the reconstruction itself feasible on the low-cost platform: freezing the cameras at nominal values leaves real-object geometry essentially unconverged (proprio-only Chamfer errors of 11.7 to 18.9 mm) and drops held-out PSNR by roughly 5 to 8 dB.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I read the calibration results as implying an untested corollary: the same optimization could serve as a passive health monitor for a robot fleet, with slowly growing joint-offset residuals flagging kinematic wear or mounting drift over time — the paper does not test this.
  • The fixed-topology limitation (a sphere-initialized mesh stays topologically sphere-like) suggests a concrete extension the authors mention only as future work: initialize from a coarse multi-view or single-view estimate instead of a sphere, which would extend the method to objects with handles or holes.
  • Because the method deliberately avoids segmenting the robot arm and instead lets the Menagerie model explain the scene, I expect calibration to degrade gracefully rather than catastrophically as background clutter grows, since the model already explains the arm's appearance and the photometric loss will favor poses that keep it aligned.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a hybrid scene representation called SplatMesh, which combines 3D Gaussian Splatting for appearance with an explicit triangle mesh for geometry, and an end-to-end optimization pipeline that uses differentiable rendering and differentiable MuJoCo (MJX) physics to jointly refine object geometry, appearance, robot joint angles, and camera extrinsics directly from raw RGB images and proprioceptive robot states. The authors validate the approach on a simulated dataset of 64 YCB objects and on real ALOHA 2 bimanual robot trajectories involving six YCB objects, reporting novel-view synthesis metrics, geometry reconstruction errors, and a 3D asset generation pipeline via CAT3D. The central claim is that a single pipeline can recover metric object meshes, photorealistic novel views, and annotation-free robot/camera pose calibration from imperfect, low-cost robot data without additional data collection.

Significance. If the claims hold, the framework is a valuable contribution to real-to-sim: it unifies appearance and physics-ready geometry in a single differentiable representation, requires no separate data collection or manual calibration, and demonstrates results on a low-cost ALOHA 2 platform with onboard RGB sensors. The simulation results on 64 YCB objects are clean and the method outperforms NeRFacto and 3DGS on several novel-view metrics while also producing explicit meshes. The real-data reconstructions are qualitatively plausible and the comparison against a TRELLIS baseline is informative. The paper also ships a reproducible experimental setup and ablation studies. However, the headline claims about pose calibration and physical parameter refinement are not fully supported by the presented evidence, and several evaluation choices make the reported numbers optimistic.

major comments (5)
  1. [Appendix D.4 / Table 3] The real-world novel-view PSNR values in Table 3 are computed after an additional optimization step that aligns the camera poses for the held-out views, as explicitly stated in Appendix D.4. This means the reported PSNR measures the quality of the SplatMesh representation given test-time fitted poses, not the generalization of the poses calibrated at training time. Since 'annotation-free robot pose calibration' is one of the three headline contributions, the evaluation conflates representation quality with calibration quality. Please report PSNR without the test-time alignment and, if possible, quantify the pose error on held-out views (e.g., by evaluating the optimized camera extrinsics against a reference) so that the calibration claim is directly tested.
  2. [Abstract / Section 4.1 / Appendix B.2] The abstract and Section 4.1 claim joint refinement of 'physical parameters' and use of 'differentiable physics,' but Appendix B.2 states the work focuses on 'object reconstruction and kinematics,' and no experiment recovers or validates mass, friction, contact, or other physical parameters. The real and simulated experiments involve static posed objects; no trajectory with contacts or object dynamics is shown. Either add a dynamics experiment (for example, releasing or pushing a reconstructed object and comparing simulated motion against real observations) or revise the claims to say 'kinematic parameters' and 'differentiable kinematics' where appropriate. As written, the scope mismatch is load-bearing because 'physical parameters' is part of the advertised contribution.
  3. [Section 5.1.2 / Table 2] The reported novel-view metrics in Table 2 use object-specific regularization weights (lambda_LL in [0.1, 1.0] and lambda_E in [0.01, 0.1]) that appear tuned per object. If these weights were selected using the test set or with access to the ground-truth meshes, the reported PSNR is optimistic and the comparison against NeRFacto and 3DGS is unfair. Please clarify the weight selection protocol (for example, a split-off validation set or a fixed schedule) or report results with a single fixed weight, as done in the 'Ours w/o mesh reg.' ablation.
  4. [Appendix D.1 / Section 3.2] The calibration experiments in Appendix D.1 add zero-mean Gaussian noise to joint angles, but the central motivation is robustness to 'imperfect' and 'inaccurate' robot models, which includes systematic errors such as wrong link lengths, joint offsets, or camera intrinsics. The paper states the setup requires a simulator with 'reasonably accurate geometry and kinematics, but not perfect,' yet no experiment perturbs the kinematic structure itself. A concrete stress test would be to modify link lengths or joint offsets in the simulated ALOHA model and measure whether the end-to-end fit still recovers accurate poses and object geometry. Without such a test, the method's robustness to systematic model mismatch—a key part of the real-to-sim claim—is not established.
  5. [Section 5.2 / Table 3] The real-world evaluation is limited to six objects with no repeated runs, no error bars, and no report of variance across random initializations or different ALOHA trajectories. Given the stochastic nature of gradient-based optimization and the mask pipeline, a single run per object is insufficient to support the quantitative geometry and PSNR comparisons in Table 3, especially for the claimed improvement over the Proprio-only baseline. Please provide multiple runs (or a variance estimate) and state the number of seeds used; this is important because the real-data evidence is the main support for the paper's central claim.
minor comments (6)
  1. [Section 6] The conclusion contains a typo: 'the the feasibility' should be 'the feasibility.'
  2. [Appendix A.1] There is a typo in the first sentence: 'optimizaiton' should be 'optimization.'
  3. [Appendix D.3] The sentence 'shows results for novel-viewy synthesis' contains a typo; 'novel-viewy' should be 'novel-view.'
  4. [References] SuGaR is listed twice: reference [26] and reference [36] refer to the same paper by Guédon and Lepetit; please consolidate.
  5. [Section 5.1.2] The ablation description mentions 'a uniform weight (lambda_L1 = 0.1)' but no L1 regularization term is defined in the loss equations in Appendix B.2. Please clarify whether this refers to the photometric loss weight (L_photo).
  6. [Section 5.3] The text says the framework is 'open source,' but no code repository or link is provided; please include the project URL or state where the code will be released.

Circularity Check

1 steps flagged · score 4.0 of 10

Real-data novel-view PSNR is computed after test-time pose alignment (Appendix D.4), so Table 3 does not validate the claimed annotation-free pose calibration; the rest of the pipeline is a direct fit to external RGB, mask, and ground-truth geometry data.

  1. fitted input called prediction [Appendix D.4, Real-World Object Reconstruction; reported in Table 3, Section 5.2]
    "Because of these non-negligible errors for the uncalibrated camera poses we additionally note that the PSNR values reported in the novel-view synthesis results in the main text are calculated after an additional optimization step, to align the camera poses for the held-out views."

    The abstract claims the pipeline performs 'annotation-free robot pose calibration' and jointly refines 'robot poses and physical parameters' from raw trajectories. In the real-world evaluation, the held-out PSNR is computed after fitting the camera poses to the held-out views. The test-time camera poses are exactly the quantity the method claims to calibrate, so the reported PSNR measures representation quality given test-fitted poses, not the generalization of the training-time calibration. The novel-view numbers are therefore partly forced by the additional pose fit, and the calibration error on unseen views is never measured. This is a partial reduction of the headline evaluation to a fitted input, disclosed but not isolated.

full rationale

The core reconstruction pipeline is a direct optimization against external real-world observations: L1 RGB loss, SAM2 masks, and estimated normals, with geometry evaluated against ground-truth YCB meshes via Chamfer distance. Robot-pose calibration is independently tested in simulation (Appendix D.1) against ground-truth synthetic poses. No load-bearing self-citation chain is present: the MuJoCo Menagerie ALOHA 2 model is an external asset, and the NeRF2Real citation is related work. The one substantive circularity concern is the real-data novel-view evaluation: Appendix D.4 states that the reported PSNR uses an extra optimization to align camera poses on held-out views. Because pose calibration is a stated contribution, this makes the headline real-data PSNR a partially fitted quantity rather than a pure prediction of the calibrated model. The paper is transparent about the step, and the sim-to-sim calibration experiment and ground-truth geometry metrics provide independent content, so the circularity is partial rather than total. Score 4 reflects one evaluation protocol that reduces a headline claim to a test-time fit, while the central method still has independent evidence.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by domain assumptions about the robot model, mask quality, and rigidity. The central method does not introduce new physical entities, but it does depend on several hand-chosen weights and initializations that are not derived from theory.

free parameters (4)
  • Per-object mesh regularization weights (lambda_Laplacian in [0.1,1.0], lambda_edge in [0.01,0.1]) = object-specific ranges
    Chosen per object to accommodate varying levels of guidance in Section 5.1.2; tuning to metrics weakens the ablation claim.
  • Real scene loss weights (L_mask=10, L_smask=1e-2, L_normal=3, L_laplacian=3) = fixed weights
    Hand-chosen weights for the real-world objective in Appendix B.2.2, Table 4; the central reconstruction relies on these balances.
  • Gaussians per mesh face (6 to 20) = sampled with stochastic rounding
    Chosen heuristically for appearance-geometry coupling in Appendix B.1.
  • Initial mesh sphere (642 vertices, 5 cm radius) = 5 cm radius
    Initialization chosen for all objects; topology is fixed to a sphere, a stated limitation in Appendix B.1.
assumptions (5)
  • domain assumption The pre-existing MuJoCo Menagerie ALOHA 2 model has reasonably accurate geometry and kinematics, with all error absorbable by joint-angle and camera-extrinsic perturbations.
    Stated in Section 3.2: 'access to a simulator with reasonably accurate geometry and kinematics, but not perfect system identification or camera calibration.' Link lengths and camera intrinsics are not optimized.
  • domain assumption Objects are rigid and topologically equivalent to the initialization mesh, so mesh connectivity and topology never change.
    Stated in Limitations: meshes learned this way are limited to those homeomorphic to the topology fixed at initialization. MJX also restricts the current implementation to rigid objects.
  • domain assumption Text-prompted OWL-ViT bounding boxes plus SAM2 masks provide reliable ground-truth silhouettes for the target object in each frame.
    Real reconstruction uses these masks as supervision in Appendix C.2.1; Appendix C.2.2 shows the same pipeline fails on robot parts, so object-mask reliability is load-bearing.
  • domain assumption The differentiable rasterizer and MJX physics gradients accurately model image formation and the kinematic chain.
    The pipeline relies on 3DGS rasterization and MuJoCo MJX gradients as black-box differentiable engines; software correctness is not independently verified.
  • domain assumption Estimated surface normals from a pre-trained diffusion model are accurate enough to supervise geometry.
    L_normal is weighted 3 in the real objective (Table 4); normal estimator errors propagate directly into mesh vertex updates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data." pith.science (2026). https://pith.science/paper/NN54VLB6

@misc{pith2026250604120,
  author       = {Pith},
  title        = {Pith review of: Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NN54VLB6}},
  note         = {Machine review of arXiv:2506.04120}
}
read the original abstract

Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challenging. Real robot data suffers from occlusions, noisy camera poses, dynamic scene elements, which hinder the creation of geometrically accurate and photorealistic digital twins of unseen objects. We introduce a novel real-to-sim framework tackling all these challenges at once. Our key insight is a hybrid scene representation merging the photorealistic rendering of 3D Gaussian Splatting with explicit object meshes suitable for physics simulation within a single representation. We propose an end-to-end optimization pipeline that leverages differentiable rendering and differentiable physics within MuJoCo to jointly refine all scene components - from object geometry and appearance to robot poses and physical parameters - directly from raw and imprecise robot trajectories. This unified optimization allows us to simultaneously achieve high-fidelity object mesh reconstruction, generate photorealistic novel views, and perform annotation-free robot pose calibration. We demonstrate the effectiveness of our approach both in simulation and on challenging real-world sequences using an ALOHA 2 bi-manual manipulator, enabling more practical and robust real-to-simulation pipelines.

Figures

Figures reproduced from arXiv: 2506.04120 by the authors.

Figure 1
Figure 1. Overview of our framework. Starting from an uncalibrated robot and real-world sensor [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Fitting with differentiable rendering. We optimize the scene parameters [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Recovering real world assets from robot data only. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: 3D asset generation with SplatMesh. Assets, which are generated from text or a single [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Example images from dataset. 2. Bounding Box Proposal: We used OWL-ViT [38] to generate bounding box proposals conditioned on the text prompt for each camera frame. The model outputs bounding boxes and associated confidence scores, and we discarded all but the highest …
Figure 6
Figure 6. Figure 6: Examples of segmentation difficulties on the ALOHA 2 robot arm using text prompts [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Effect of calibration on wrist camera view for a sample pose with 0.01 radians noise. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Distribution of TCP errors (mm) before (Initial) and after (Calibrated) optimization for [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Additional details on 3D asset generation. For each object: (Left) Input text prompt or [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Reconstruction of YCB objects in simulation. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Object reconstruction from real robot data fails in the ablation when camera pose is not [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Grid of real and learned YCB object images. Our method recovers metrically accurate [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Real object image prompts, ground truth meshes and TRELLIS meshes. While TRELLIS [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 26 canonical work pages

  1. [1]

    Mildenhall, P

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020. URLhttps://arxiv. org/abs/2003.08934

    arXiv 2020

  2. [2]

    Kerbl, G

    B. Kerbl, G. Kopanas, T. Leimkuehler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. , 42(4), July 2023. ISSN 0730-0301. doi:10. 1145/3592433. URL https://doi.org/10.1145/3592433

    doi:10.1145/3592433 2023

  3. [3]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026–

    2012

  4. [4]

    Dogaru, M

    A. Dogaru, M. ¨Ozer, and B. Egger. Generalizable 3D scene reconstruction via divide and conquer from a single view

    work page

  5. [5]

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http: //github.com/google/brax

    work page 2021

  6. [6]

    J. L. Sch ¨onberger and J.-M. Frahm. Structure-from-motion revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 9

    work page 2016

  7. [7]

    J. L. Sch ¨onberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016

    work page 2016

  8. [8]

    R. Liu, A. Canberk, S. Song, and C. V ondrick. Differentiable robot rendering

    work page

Show all 45 references
  1. [9]

    Y . Wu, L. Pan, W. Wu, G. Wang, Y . Miao, and H. Wang. Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning, 2024. URL https://arxiv. org/abs/2409.20291

    2024 arXiv

  2. [10]

    X. Li, J. Li, Z. Zhang, R. Zhang, F. Jia, T. Wang, H. Fan, K.-K. Tseng, and R. Wang. RoboGSim: A Real2Sim2Real robotic gaussian splatting simulator

  3. [11]

    H. Lou, Y . Liu, Y . Pan, Y . Geng, J. Chen, W. Ma, C. Li, L. Wang, H. Feng, L. Shi, L. Luo, and Y . Shi. Robo-GS: A physics consistent spatial-temporal model for robotic arm with hybrid representation

  4. [12]

    Y . Zhu, T. Xiang, A. Dollar, and Z. Pan. One-shot real-to-sim via end-to-end differentiable simulation and rendering, 2024. URL https://arxiv.org/abs/2412.00259

    2024 arXiv

  5. [13]

    M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting, 2024. URL https: //arxiv.org/abs/2409.10161

    2024 arXiv

  6. [14]

    Byravan, J

    A. Byravan, J. Humplik, L. Hasenclever, A. Brussee, F. Nori, T. Haarnoja, B. Moran, S. Bo- hez, F. Sadeghi, B. Vujatovic, and N. Heess. NeRF2Real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields

  7. [15]

    Sato and Y

    Y . Sato and Y . Yaguchi. Rapidsim: Enhancing robotic simulation with photorealistic 3d envi- ronments via smartphone-captured nerf and ue5 integration. In 2024 International Conference on Image Processing and Robotics (ICIPRoB) , pages 1–6, 2024. doi:10.1109/ICIPRoB62548. 202...

    2024

  8. [16]

    Rosinol, J

    A. Rosinol, J. J. Leonard, and L. Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields, 2022. URL https://arxiv.org/abs/2210.13641

    2022 arXiv

  9. [17]

    X. Kong, S. Liu, M. Taher, and A. J. Davison. vmap: Vectorised object mapping for neural field slam, 2023. URL https://arxiv.org/abs/2302.01838

    2023 arXiv

  10. [18]

    X. Zhou, R. Girdhar, A. Joulin, P. Kr¨ahenb¨uhl, and I. Misra. Detecting twenty-thousand classes using image-level supervision, 2022. URL https://arxiv.org/abs/2201.02605

    2022 arXiv

  11. [19]

    M. Ye, M. Danelljan, F. Yu, and L. Ke. Gaussian grouping: Segment and edit anything in 3D scenes. 2023

    2023

  12. [20]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. Girshick. Segment anything, 2023. URL https: //arxiv.org/abs/2304.02643

    2023 arXiv

  13. [21]

    L. Qi, J. Kuen, W. Guo, T. Shen, J. Gu, J. Jia, Z. Lin, and M.-H. Yang. High-quality entity segmentation, 2023. URL https://arxiv.org/abs/2211.05776

    2023 arXiv

  14. [22]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. URL https://arxiv.org/abs/2304.13705

    2023 arXiv

  15. [24]

    Zakka, Y

    K. Zakka, Y . Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A col- lection of high-quality simulation models for MuJoCo, 2022. URL http://github.com/ google-deepmind/mujoco_menagerie

    2022

  16. [25]

    P. Dai, J. Xu, W. Xie, X. Liu, H. Wang, and W. Xu. High-quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  17. [26]

    Gu ´edon and V

    A. Gu ´edon and V . Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5354–5363, 2024

    2024

  18. [27]

    G. M. Garcia, K. A. Zeid, C. Schmidt, D. de Geus, A. Hermans, and B. Leibe. Fine-tuning image-conditional diffusion models is easier than you think, 2025. URL https://arxiv. org/abs/2409.11355

    2025

  19. [28]

    SAM 2: Segment anything in images and videos

  20. [29]

    Calli, A

    B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. arXiv preprint arXiv:1502.03143, 2015

    2015 arXiv

  21. [30]

    Aldaco, T

    J. Aldaco, T. Armstrong, R. Baruch, J. Bingham, S. Chan, K. Draper, D. Dwibedi, C. Finn, P. Florence, S. Goodrich, et al. Aloha 2: An enhanced low-cost hardware for bimanual teleop- eration. arXiv preprint arXiv:2405.02292, 2024

    2024 arXiv

  22. [31]

    Tancik, E

    M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings , SIGGRAPH ’23, 2023

    2023

  23. [32]

    Xiang, Z

    J. Xiang, Z. Lv, S. Xu, Y . Deng, R. Wang, B. Zhang, D. Chen, X. Tong, and J. Yang. Structured 3d latents for scalable and versatile 3d generation, 2025. URL https://arxiv.org/abs/ 2412.01506

    2025 arXiv

  24. [33]

    R. Gao, A. Holynski, P. Henzler, A. Brussee, R. Martin-Brualla, P. Srinivasan, J. T. Barron, and B. Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024

    2024 arXiv

  25. [34]

    Saito, G

    S. Saito, G. Schwartz, T. Simon, J. Li, and G. Nam. Relightable gaussian codec avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130–141, 2024

    2024

  26. [35]

    Moenne-Loccoz, A

    N. Moenne-Loccoz, A. Mirzaei, O. Perel, R. de Lutio, J. Martinez Esturo, G. State, S. Fi- dler, N. Sharp, and Z. Gojcic. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Transactions on Graphics (TOG), 43(6):1–19, 2024

    2024

  27. [36]

    Gu ´edon and V

    A. Gu ´edon and V . Lepetit. SuGaR: Surface-aligned gaussian splatting for efficient 3D mesh reconstruction and high-quality mesh rendering

  28. [37]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  29. [38]

    Minderer, A

    M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. Ad- vances in Neural Information Processing Systems , 36:72983–73007, 2023

    2023

  30. [39]

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  31. [40]

    Dawson-Haggerty et al. trimesh. URL https://trimesh.org/. 11 A Implementation Details A.1 Hardware and Software All experiments detailed in the main paper were conducted on a single machine equipped with an NVIDIA H100 GPU possessing 80GB of VRAM, using JAX version 0.6.0 and M...

  32. [42]

    14 (a) Overhead camera (b) Table camera (c) Left wrist camera (d) Right wrist camera Figure 5: Example images from dataset

    Text Prompting: For each target object (e.g., ”banana”, ”blue tuna can”), a textual de- scription was used as a prompt. 14 (a) Overhead camera (b) Table camera (c) Left wrist camera (d) Right wrist camera Figure 5: Example images from dataset

  33. [43]

    The model outputs bounding boxes and associated confidence scores, and we discarded all but the highest confidence box

    Bounding Box Proposal: We used OWL-ViT [38] to generate bounding box proposals conditioned on the text prompt for each camera frame. The model outputs bounding boxes and associated confidence scores, and we discarded all but the highest confidence box

  34. [44]

    Frame Filtering: Frames where the maximum detection confidence score for the target object fell below 0.01 were discarded

  35. [45]

    Instance Segmentation: The selected bounding box was passed as a prompt to SAM2 [28] in multimask mode, generating three candidate segmentation masks within that box

  36. [46]

    This pipeline provided the ground truth masks used for the mask loss ( Lmask) during real-world object reconstruction

    Mask Postprocessing: The SAM2 masks were averaged together pixel-wise, and thresh- olded at 0.5 to obtain binary masks. This pipeline provided the ground truth masks used for the mask loss ( Lmask) during real-world object reconstruction. C.2.2 Segmentation Challenges on Robot...

  37. [5033]

    doi:10.1109/IROS.2012.6386109

    IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.