REVIEW 4 major objections 5 minor 34 references
Three real trajectories are enough to train a mesh GNN that matches calibrated MuJoCo and beats differentiable Brax on rigid contact while remaining fully differentiable.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-15 13:55 UTC pith:HWRSYR74
load-bearing objection Solid few-shot real-to-sim pipeline for a mesh GNN contact simulator that matches identified MuJoCo on real cubes and beats Brax; the differentiability claim rests on an untested fixed-contact-set assumption. the 4 major comments →
Few-Shot Neural Differentiable Simulator: Real-to-Sim Rigid-Contact Modeling
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
A mesh GNN trained solely on trajectories generated by a MuJoCo instance whose contact parameters were identified from three real cube-push sequences reproduces real-world rigid-contact trajectories at least as accurately as the identified MuJoCo itself and far more accurately than identified Brax, while the derived surrogate gradients through discrete collision detection make the whole pipeline differentiable for gradient-based contact optimization.
What carries the argument
Surrogate gradients of nearest contact points: treat the discrete-collision-detection contact set as fixed inside a time step (justified by a slightly enlarged distance threshold), then back-propagate through the contact Jacobian relating object generalized positions to the nearest-point locations, rendering non-differentiable collision detection usable inside an end-to-end differentiable mesh GNN.
Load-bearing premise
The method assumes that a modestly larger collision-distance threshold always supplies enough redundant contact pairs so that small state changes never alter which pairs are active; if that fails, the surrogate gradients become wrong.
What would settle it
Collect a new real multi-cube collision sequence whose motions are large enough that the active contact set changes within a single time step under the paper's distance threshold; if the GNN's predicted trajectories and back-propagated gradients then diverge sharply from both real data and the calibrated analytical engine, the surrogate-gradient claim fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a few-shot real-to-sim pipeline for rigid-contact simulation: contact parameters of MuJoCo (solimp, solref, friction) are identified from only three real cube trajectories via CMA-ES; the calibrated MuJoCo then generates a large synthetic dataset used to train a mesh-based GNN (FIGNet-style) that predicts node accelerations. Full differentiability is claimed by deriving surrogate gradients of nearest contact points through a contact Jacobian (Eq. 7), under the assumption that the discrete-collision-detection contact set remains fixed within a time step when the distance threshold is enlarged. On a 14-trajectory real held-out set the GNN matches or slightly beats identified MuJoCo and substantially outperforms all three Brax pipelines; a simple SGD example optimizes an initial push velocity through the simulator (Fig. 1), and a multi-cube bowling rollout is shown against MuJoCo-generated ground truth.
Significance. If the claims hold, the work offers a practical route to data-efficient, differentiable contact simulators that can be grounded on minimal real pose data and then used for gradient-based contact optimization—valuable for robotic manipulation where analytical contact models are brittle and pure learning methods are data-hungry. Strengths that deserve credit include: (i) a transparent few-shot identification protocol with a clear before/after ablation (Figs. 4–5), (ii) evaluation against held-out real trajectories rather than only simulation, and (iii) an explicit, implementable surrogate-gradient construction that makes a mesh GNN end-to-end differentiable in principle. The combination of real-to-sim scaling with a fully differentiable mesh GNN is a useful contribution to the real-to-sim and differentiable-simulation literature, provided the differentiability assumption and the scope of the optimization experiments are tightened.
major comments (4)
- [Sec. III-C.2, Eq. (7)] The differentiability claim rests on treating the GJK/EPA contact-pair set as fixed inside a time step (Sec. III-C.2, Eq. 7), justified only by the informal claim that a slightly larger d_ε yields enough redundant pairs that small state perturbations leave the active set unchanged. No ablation, sensitivity study, or finite-difference check is reported for the cube trajectories, the bowling scene, or—most critically—the finite velocity updates performed by SGD in Fig. 1. If the active set changes under those updates, the surrogate Jacobian is incorrect and the reported optimization success is not evidence of true end-to-end differentiability. A concrete validation (e.g., comparison of surrogate vs. finite-difference gradients under the same motions, or an active-set change rate under the Fig. 1 updates) is needed before the differentiability half of the central claim can be accepted.
- [Abstract; Sec. V; Fig. 1] The abstract and conclusion state that the differentiable design is validated through “simulation-based policy learning in multi-object interaction scenarios” and that the framework “increases the efficiency of policy learning.” The only gradient-based experiment shown is open-loop SGD on a single initial push velocity (Fig. 1). There is no policy network, no closed-loop control, no multi-object policy task, and no comparison of sample efficiency against a non-differentiable baseline. Either the abstract/conclusion language must be revised to match the actual experiment, or a genuine policy-learning result (even a simple residual policy or trajectory optimization with a multi-step horizon) must be added.
- [Sec. IV-D, Fig. 6] The multi-object generalization claim (bowling-like array of ten cubes, Fig. 6) uses trajectories generated by the same identified MuJoCo that produced the training data as ground truth. Consequently the experiment only demonstrates sim-to-sim consistency, not transfer of real-world contact dynamics to multi-body scenes. Given that the paper’s strongest real-world evidence is restricted to two-cube quasi-planar contacts, the multi-object claim should be clearly labeled as simulation-only, or a real multi-object test set should be collected.
- [Sec. IV-D, Fig. 7] The GNN is trained exclusively on trajectories from the identified MuJoCo; the real test-set comparison (Fig. 7) is therefore the decisive non-circular evidence. That comparison is informative, but the paper should more carefully separate (a) fidelity to the calibrated analytical model from (b) residual improvement over that model on real data. Currently the narrative sometimes attributes “learning real-world contact dynamics” to the GNN when the dominant effect may simply be faithful imitation of the already-identified MuJoCo. A short residual analysis (GNN vs. MuJoCo error correlation on the real test set) would clarify how much genuine real-world residual is being captured.
minor comments (5)
- [Table I] Table I lists “Ours” contact accuracy as “High” while MuJoCo is also “High”; the table would be more informative if it reported the actual mean trajectory errors from Fig. 7 rather than qualitative labels.
- [Eq. (1)] The positional scaling factors w_i in Eq. (1) are never specified numerically; a short note on how they are chosen would aid reproducibility.
- [Fig. 1] Fig. 1 caption and the surrounding text refer to “policy learning,” which is inconsistent with the open-loop velocity optimization actually performed; align terminology.
- [Sec. I; Sec. III-A] Minor typos: “IssacLab” → “Isaac Lab”; “solrefspecifies” missing space; “bDk” / “bxi” notation is hard to parse in text—consider clearer hats or tildes.
- [Sec. II-D] The related-work discussion of differentiable collision detection (Montaut et al., Tracy et al., Zimmermann et al.) is brief; a one-sentence contrast of how the surrogate Jacobian differs from randomized smoothing or convex-primitive methods would help place Eq. (7).
Circularity Check
Mild partial circularity: GNN trained exclusively on identified-MuJoCo rollouts is then reported as 'comparable' to that same MuJoCo on real test trajectories; the match is expected once the student approximates the teacher, while real-vs-Brax and differentiability claims remain independent.
specific steps
-
fitted input called prediction
[Sec. IV-D / Fig. 7 and accompanying text]
"Even though our simulator is trained only on the scaled data generated by the MuJoCo baseline, it still achieves similar error levels and even slightly lower positional errors compared to the identified MuJoCo. Thus, data scaling enables our simulator to learn the real-world rigid-contact dynamics well, thanks to the diverse configurations and relatively accurate contact modeling of the identified MuJoCo."
The GNN is a supervised student of the identified MuJoCo (3000 synthetic trajectories). Once the student approximates the teacher, its trajectory error relative to any fixed real test set is forced to be nearly identical to the teacher's error. Reporting that near-match as independent confirmation that 'data scaling enables learning real-world dynamics' therefore reduces, by construction of the training objective, to the statement that the GNN fit the MuJoCo data well. The slight numerical improvement does not break the reduction; it is still measured against the same teacher that supplied the labels.
full rationale
The derivation chain is largely non-circular. Contact-parameter identification (CMA-ES on three real trajectories) is ordinary system ID against an external objective; the subsequent synthetic scaling and GNN training are standard distillation; evaluation on a held-out 14-trajectory real test set is an independent external benchmark; the surrogate-gradient formula (Eq. 7) is an explicit approximation under a fixed-contact-set assumption, not a tautology; and no load-bearing uniqueness theorem or self-citation is invoked. The single mild circularity is that the paper repeatedly presents 'comparable (or slightly better) positional/angular error versus the identified MuJoCo' as supporting evidence for the real-to-sim pipeline, even though the GNN was supervised exclusively on trajectories generated by that MuJoCo. Perfect imitation of the teacher forces identical real-world error; the observed near-match is therefore largely by construction of a successful fit rather than an independent validation of the analytical model. Because the decisive claims (outperformance of Brax on real data, few-shot data efficiency, and successful gradient-based optimization) rest on external real trajectories and do not reduce to this comparison, the circularity is only partial and does not collapse the central results.
Axiom & Free-Parameter Ledger
free parameters (3)
- MuJoCo contact parameter vector θ = {solimp, solref, μ}
- collision distance threshold d_ε
- positional scaling factors w_i
axioms (4)
- domain assumption Rigid-body dynamics with Coulomb friction and MuJoCo’s convex contact solver adequately approximate the real quasi-planar cube contacts once solimp/solref/μ are identified.
- ad hoc to paper Inside one simulation time step the set of contact pairs returned by GJK/EPA remains constant under small state perturbations when d_ε is enlarged, so differentiation through discrete collision detection can be replaced by the contact Jacobian alone.
- domain assumption Verlet integration plus shape matching recovers rigid motion from predicted mesh-node accelerations.
- domain assumption AprilTag/TagSLAM 6-D poses at 60 Hz are sufficiently accurate ground truth for both identification and evaluation.
invented entities (1)
-
surrogate gradient of nearest points via contact Jacobian (Eq. 7)
no independent evidence
read the original abstract
Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to capture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differentiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi-object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.
Figures
Reference graph
Works this paper leans on
-
[1]
Brian Acosta, William Yang, and Michael Posa. Vali- dating robotics simulators on real-world impacts.IEEE Robotics and Automation Letters, 7(3):6471–6478, 2022. doi: 10.1109/LRA.2022.3174367
-
[2]
Optuna: A next-generation hyperparameter optimization framework
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InThe 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631, 2019
2019
-
[3]
Graph network simulators can learn discontinuous, rigid contact dynamics
Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 ofProceedings of Machine Learnin...
2023
-
[4]
Learning rigid dynamics with face interaction graph networks
Kelsey R Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William F Whitney, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. InThe Eleventh Inter- national Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=J7Uh781A05p
2023
-
[5]
Interaction networks for learning about objects, relations and physics
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. InProceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 4509–4517, Red Hook, NY , USA, 2016. Curran Associates Inc. ISBN 9781510838819
2016
-
[6]
Proximity queries and penetration depth computation on 3d game objects
Gino Bergen. Proximity queries and penetration depth computation on 3d game objects. 01 2001
2001
-
[7]
Si- multaneous learning of contact and continuous dynamics
Bibit Bianchini, Mathew Halm, and Michael Posa. Si- multaneous learning of contact and continuous dynamics. In7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=-3G6 D66Aua
2023
-
[8]
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation. InThirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https://openreview.net/forum?id= VdvDlnnjzIN
2021
-
[9]
E.G. Gilbert, D.W. Johnson, and S.S. Keerthi. A fast procedure for computing the distance between complex objects in three-dimensional space.IEEE Journal on Robotics and Automation, 4(2):193–203, 1988. doi: 10.1109/56.2083
doi:10.1109/56.2083 1988
-
[10]
Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S. Sukhatme. Neuralsim: Aug- menting differentiable simulators with neural networks. In2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9474–9481, 2021. doi: 10.1109/ICRA48506.2021.9560935
-
[11]
Howell, Simon Le Cleac’h, Jan Br ¨udigam, Qianzhong Chen, Jiankai Sun, J
Taylor A. Howell, Simon Le Cleac’h, Jan Br ¨udigam, Qianzhong Chen, Jiankai Sun, J. Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differen- tiable physics engine for robotics, 2025. URL https: //arxiv.org/abs/2203.00806
Pith/arXiv arXiv 2025
-
[12]
Flexible layouts for fiducial tags
Maximilian Krogius, Acshi Haggenmiller, and Edwin Olson. Flexible layouts for fiducial tags. In2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1898–1903, 2019. doi: 10.1109/IROS40897.2019.8967787
-
[13]
Simon Le Cleac’h, Mac Schwager, Zachary Manchester, Vikas Sindhwani, Pete Florence, and Sumeet Singh. Single-level differentiable contact simulation.IEEE Robotics and Automation Letters, 8(7):4012–4019, 2023. doi: 10.1109/LRA.2023.3268824
-
[14]
Simon Le Cleac’h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics-augmented neural objects.IEEE Robotics and Automation Letters, 8(5):2780–2787, 2023. doi: 10.1109/ LRA.2023.3257707
arXiv 2023
-
[15]
Quentin Le Lidec, Igor Kalevatykh, Ivan Laptev, Cordelia Schmid, and Justin Carpentier. Differentiable simulation for physical system identification.IEEE Robotics and Automation Letters, 6(2):3413–3420, 2021. doi: 10.1109/ LRA.2021.3062323
arXiv 2021
-
[16]
Tenen- baum, and Antonio Torralba
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenen- baum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. InInternational Conference on Learning Repre- sentations, 2019. URL https://openreview.net/forum?id= rJgbSn09Ym
2019
-
[17]
Residual policy learning for perceptive quadruped control using differen- tiable simulation, 2024
Jing Yuan Luo, Yunlong Song, Victor Klemm, Fan Shi, Davide Scaramuzza, and Marco Hutter. Residual policy learning for perceptive quadruped control using differen- tiable simulation, 2024. URL https://arxiv.org/abs/2410. 03076
2024
-
[18]
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments.IEEE Robotics and Au- tomation Letters, 8(6):3740–3747, 2023. do...
arXiv 2023
-
[19]
Dif- ferentiable collision detection: a randomized smoothing approach
Louis Montaut, Quentin Le Lidec, Antoine Bambade, Vladimir Petrik, Josef Sivic, and Justin Carpentier. Dif- ferentiable collision detection: a randomized smoothing approach. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3240–3246,
-
[20]
doi: 10.1109/ICRA48891.2023.10160251
-
[21]
Meshless deformations based on shape matching.ACM Trans
Matthias M ¨uller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. Meshless deformations based on shape matching.ACM Trans. Graph., 24(3):471–478, July 2005. ISSN 0730-0301. doi: 10.1145/1073204. 1073216. URL https://doi-org.libproxy1.nus.edu.sg/10. 1145/1073204.1073216
-
[22]
Position based dynamics.J
Matthias M ¨uller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. Position based dynamics.J. Vis. Comun. Image Represent., 18(2):109–118, April
-
[23]
doi: 10.1016/j.jvcir.2007.01.005
ISSN 1047-3203. doi: 10.1016/j.jvcir.2007.01.005. URL https://doi-org.libproxy1.nus.edu.sg/10.1016/j.jvcir. 2007.01.005
-
[24]
J. Krishna Murthy, Miles Macklin, Florian Golemo, Vikram V oleti, Linda Petrini, Martin Weiss, Brean- dan Considine, J ´erˆome Parent-L ´evesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differen- tiable simulation for system identification and visuo- motor control. InInternational Conference ...
2021
-
[25]
Coal: an extension of the flexible collision library
Jia Pan, Sachin Chitta, Dinesh Manocha, Florent Lami- raux, Joseph Mirabel, Justin Carpentier, Louis Montaut, et al. Coal: an extension of the flexible collision library. https://github.com/coal-library/coal, 2015–2024
2015
-
[26]
Learning mesh-based simula- tion with graph networks
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simula- tion with graph networks. InInternational Confer- ence on Learning Representations, 2021. URL https: //openreview.net/forum?id=roNqYL0 XP
2021
-
[27]
Tagslam: Robust slam with fiducial markers, 2019
Bernd Pfrommer and Kostas Daniilidis. Tagslam: Robust slam with fiducial markers, 2019. URL https://arxiv.org/ abs/1910.00679
Pith/arXiv arXiv 2019
-
[28]
Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations, 2020
Samuel Pfrommer, Mathew Halm, and Michael Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations, 2020. URL https: //arxiv.org/abs/2009.11193
Pith/arXiv arXiv 2020
-
[29]
Graph networks as learnable physics engines for inference and control
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Had- sell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In Jennifer Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 ofProceedings of Machine Le...
2018
-
[30]
Learning quadruped locomotion using differen- tiable simulation
Yunlong Song, Sang bae Kim, and Davide Scara- muzza. Learning quadruped locomotion using differen- tiable simulation. In8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id= XopATjibyz
2024
-
[31]
Mu- joco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mu- joco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109
-
[32]
Howell, and Zachary Manchester
Kevin Tracy, Taylor A. Howell, and Zachary Manchester. Differentiable collision detection for a set of convex primitives. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3663–3670,
-
[33]
doi: 10.1109/ICRA48891.2023.10160716
-
[34]
Differentiable collision avoidance using collision primitives
Simon Zimmermann, Matthias Busenhart, Simon Hu- ber, Roi Poranne, and Stelian Coros. Differentiable collision avoidance using collision primitives. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8086–8093, 2022. doi: 10.1109/IROS47612.2022.9981093
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.