Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PIN-WM is a physics-informed world model that identifies 3D rigid-body dynamics from RGB video and transfers push and flip policies to a real robot without fine-tuning.

desk verdict A genuinely useful integration of differentiable LCP physics with Gaussian splatting; the simulation evidence is strong, but the action-execution assumption is untested and the real-world eval is thin. read the letter →

arxiv 2504.16693 v2 pith:BBI7DRTN submitted 2025-04-23 cs.LG cs.RO

Wenxuan Li , Hang Zhao , Zhiyuan Yu , Yu Du , Qin Zou , Ruizhen Hu , Kai Xu This is my paper · ORCID
classification cs.LGcs.RO
keywords physics-informedworldmodelnon-prehensilemanipulationdifferentiablephysicssystemidentification2DGaussianSplattingsim-to-realtransferdomainrandomizationmodel-basedreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PIN-WM is a physics-informed world model that claims to identify the 3D rigid-body dynamics of a manipulated object directly from RGB video, using end-to-end differentiability through a velocity-based linear complementarity problem (LCP) solver and a 2D Gaussian Splatting renderer. The paper's core claim is that a few task-agnostic pushes, in simulation as few as a single straight-line shove, are enough to estimate mass, inertia, friction, and restitution, and that no separate state-estimation module is needed. Around the identified parameters the method creates 'physics-aware digital cousins' by narrow random perturbation, and trains vision-based push and flip policies in those simulated variations. On real hardware the resulting policies transfer without fine-tuning, with 75% success on pushing a T-shaped object and 65% on flipping a mug, outperforming the compared Real2Sim2Real baselines. If true, this means a robot could acquire accurate interaction dynamics from almost no task-specific data, making non-prehensile skills practical to deploy.

What carries the argument

The engine of PIN-WM is a differentiable velocity-based linear complementarity problem (LCP) rigid-body solver, which computes contact, friction, and joint constraint impulses from object state and end-effector translation, and propagates gradients from the rendering loss back to physics parameters. Around it sits a 2D Gaussian Splatting renderer that generates images from object pose, so the identification objective is simply the pixel difference between rendered and observed frames. The transfer mechanism is the 'physics-aware digital cousin': the identified physics and rendering parameters become the mean of a narrow uniform perturbation (±10%), and policies are trained across those sampled variants with PPO. Together these pieces turn a short task-agnostic video into an interactive simulator that supports zero-shot real-world policy deployment.

What would settle it

Run the same single-trajectory identification on a robot arm with noticeable compliance, such as a lightweight or series-elastic arm, and compare the learned model's one-step object-pose prediction against motion capture after identical pushes; if the prediction error is substantially larger than the paper reports for its rigid-arm setup, the assumed 'robot mass much greater than object mass, end-effector follows commanded translation' premise is the failing link.

Watch

Extended reading notes

Core claim

The paper claims that end-to-end gradient flow from rendered RGB images through a differentiable velocity-based LCP rigid-body simulator can recover the physics parameters that govern non-prehensile manipulation—mass and inertia, friction coefficients, and restitution—from a few short, task-agnostic pushes, in simulation as few as one. Because the identification is driven by a rendering loss against 2D Gaussian Splatting images rather than by state estimates, no pose-tracking or state-estimation module is needed. The learned world model is then perturbed around the identified parameters to form 'physics-aware digital cousins,' and PPO-trained vision policies in those variations transfer directly to the real world, with reported success rates of 75% for pushing a T-shaped object and 65% for flipping a mug, surpassing the compared ASID, 2D-physics, RoboGSim, and Dreamer V2 baselines.

Load-bearing premise

The pipeline assumes that the robot arm goes exactly where commanded, treating its push as a constant-velocity input and ignoring how the arm itself flexes or yields on contact; if the arm bends or absorbs impact rather than staying rigid, the identified object physics will not match reality.

Editorial extensions

If this is right

  • A robot can obtain a usable interactive model of a manipulated object from one short, task-agnostic interaction, so acquiring new manipulation skills need not require task-specific demonstrations.
  • The identified parameters plug into standard rigid-body simulators, so downstream policy learning can exploit fast parallel simulation rather than the differentiable solver.
  • Perturbing identified parameters in a narrow band yields more transferable policies than broad domain randomization, reducing the sim-to-real gap without inflating the training distribution.
  • Because the identification handles full 3D rotations, the same pipeline covers tasks that 2D physics models cannot, such as flipping an object over.
  • Without fine-tuning, the trained vision policy reaches 75% and 65% success on real push and flip tasks, suggesting the learned world model already captures the dominant real-world dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper leaves implicit: the same pixel-to-parameters gradient flow could identify a deformable object's constitutive parameters if the LCP solver were replaced by a differentiable material-point method, an option the paper names as future work.
  • The digital-cousin idea may apply beyond physics: any learned world model with a small set of identified parameters could benefit from narrow perturbation around the identified values, instead of uniform sampling over a broad prior.
  • A testable prediction follows from the paper's assumption that robot dynamics can be ignored: on a compliant arm the one-step prediction error should grow specifically during the impact phase of a poke, which would expose the missing arm-compliance term.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes PIN-WM, a two-stage Real2Sim2Real framework for non-prehensile manipulation. In the first stage, a physics-informed world model is identified end-to-end from RGB observations: a differentiable rigid-body LCP simulator provides the dynamics, a 2D Gaussian Splatting renderer provides the visual observation model, and the physics parameters (mass, friction, restitution) are optimized through a rendering loss using a small number of task-agnostic interaction trajectories and no explicit state estimation. In the second stage, the identified model is turned into a family of 'digital cousins' by perturbing the physics and rendering parameters around the identified values, and PPO policies are trained in the resulting randomized environments and transferred zero-shot to the target domain. The paper reports simulation results on push and flip tasks, as well as real-robot experiments, and claims higher success rates and fewer steps than Dreamer V2, Diffusion Policy, RoboGSim, Domain Randomization, 2D Physics, and ASID.

Significance. If the results hold, the contribution is valuable: it is a clean demonstration of differentiable 3D rigid-body system identification driven directly by a rendering loss, combined with a concrete mechanism for zero-shot policy transfer. The simulation evaluation is comparatively careful: one-step prediction errors are measured on actions not used for identification, parameter-identification accuracy is compared under a one-at-a-time protocol, robustness to geometry noise is tested, and the PADC ablation isolates the effect of the proposed randomization scheme. The real-robot demonstrations support the feasibility of the approach, but they are not strong enough on their own to establish the broad claim of surpassing state-of-the-art methods with quantified confidence. The paper does not release code or data, so the engineering choices behind the strong results cannot be independently checked.

major comments (3)
  1. [Section III-B, Eqs. (9)-(10)] The pipeline treats the commanded end-effector translation d as if it were executed exactly, converting it to velocity xi_e = d/H on the grounds that the robot's mass is much larger than the object's mass. This assumption is load-bearing for both identification and transfer, but the paper provides no measurement of actual end-effector motion, no characterization of the controller's compliance, and no experiment varying arm stiffness. A back-drivable or impedance-controlled arm would produce contact-induced tracking errors that the LCP solver would incorrectly attribute to object physics during identification, and during deployment the policy would rely on phantom actions in the simulator. Since PADC (Eq. (12)) perturbs only physics and rendering parameters, it does not cover this systematic gap. Please report commanded-versus-measured end-effector trajectories for identification and deployment, and quantify sensitivity by injecting action-execution noise (for example, bounded deviations or low-pass filtered tracking errors) into the identified world model.
  2. [Section IV-B, Table III] The real-world deployment conclusions are based on 20 trials per task under identical initial conditions, with no confidence intervals, no per-trial success/failure details, and no statistical tests; the additional real-world results in Table VII do not state the number of trials. With this sample size and setup, the reported differences from baselines cannot support a broad, quantified claim of surpassing state-of-the-art methods with measured uncertainty. Please report per-rollout variability, vary initial poses and lighting conditions, and provide error bars or an explicit statement that the real-world comparison is exploratory rather than statistically decisive.
  3. [Appendix B, Table V] The parameter-identification accuracy is assessed by estimating one parameter at a time while holding the others fixed at their ground-truth values, but the deployed system optimizes all parameters jointly from a single straight-line push trajectory. This protocol therefore does not validate the joint estimate that defines the mean of the PADC perturbations. Please report joint identification results (for example, all fitted parameter values, a validation-set predictive error, or a sensitivity analysis over plausible parameter sets), and show that policy transfer is robust to replacing theta* with other parameter sets that fit the identification video equally well.
minor comments (4)
  1. [Eq. (10)] The contact constraint is written as a single chain 'Jc xi_{t+1} >= -theta_k Jc xi_t >= -c', which appears to conflate two inequalities; please separate them or explain the compact notation.
  2. [Section IV-B] The comparison to 'RoboGSim [60]' should cite reference [45], which is the RoboGSim paper; reference [60] is the Domain Randomization paper.
  3. [Section V] Minor wording issues: 'rendering alignment places a key role' should be 'plays a key role', and 'our current framework focus on rigid-body dynamics' should be 'focuses'.
  4. [Section IV] The paper does not specify the policy network architecture, image resolution, reward scaling, or training hyperparameters, and no code release is mentioned; adding at least key implementation details to an appendix would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: system identification is fitted to observations, evaluated on held-out actions, and digital-cousin perturbations are standard domain randomization around identified values.

full rationale

The paper's derivation is self-contained as an identification-plus-transfer pipeline rather than a circular prediction. Rendering parameters alpha are optimized on static multi-view images (Eq. 6), then physics parameters theta are estimated by minimizing the rendering loss against a single task-agnostic push video (Eq. 8), with gradients propagated through a differentiable LCP (Eqs. 10-11). This is a system-identification fit, not a 'prediction' of the fitted quantity. The one-step error reported in Table II is evaluated on randomly sampled actions after identification, which are held out from the straight-line identification trajectory, so the accuracy metric tests generalization to new actions under the identified dynamics. The physics-aware digital cousins (Eq. 12) perturb the identified physics and rendering parameters within a uniform band; this is domain randomization centered at identified values, and the target-domain evaluation is conducted in the real or ground-truth simulator, not inside the perturbed training ensemble, so the target answer is not hidden in the training set. The simplification that the robot end-effector velocity equals the commanded translation divided by the action horizon (Section III-B) is a modeling assumption whose validity can be questioned, but it is not a circular reduction of any output to an input. The only self-citations ([82], [83]) appear in a future-work sentence about industrial automation and are not load-bearing. No equation or claim reduces a predicted quantity to its own fitting target by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions that are not validated independently: the rigid-body and known-geometry assumptions, the correctness of the LCP contact model, and the sufficiency of the rendering loss for identification. The physics parameters (mass, friction, restitution) are fitted to a single trajectory, so they are not derivable from first principles. No new physical entities are introduced.

free parameters (7)
  • Friction coefficient (mu) = 3.05e-2 (push), 2.11e-1 (flip) in simulation
    Identified by optimizing the rendering loss (Eq. 8); listed in Table V.
  • Mass (kg) = 0.76 (push), 1.19 (flip) vs GT 1.00
    Identified mass deviates from ground truth, indicating compensation effects; Table V.
  • Restitution coefficient = 4.18e-4 (push), 1.69e-5 (flip) vs GT 0.00
    Identified via Eq. 8; Table V.
  • Perturbation magnitude delta = 0.1
    Chosen by hand for digital cousin generation (Eq. 12).
  • 2DGS rendering parameters alpha = thousands of Gaussian parameters
    Fitted to static multi-view images (Eq. 6); not enumerated.
  • Loss weights omega_d, omega_n = not reported
    Hyperparameters in Eq. 6 for depth distortion and normal consistency.
  • Time horizon H and simulation step h = not reported in main text
    Set for LCP integration (Eq. 9); required to convert action to velocity.
assumptions (5)
  • domain assumption Rigid body assumption: object, floor, and end-effector are rigid bodies obeying Newton-Euler equations.
    Stated in Section III-B; excludes deformable objects, acknowledged in limitations.
  • domain assumption Object geometry (mesh) is known.
    Section III-A: 'the geometry is assumed to be known'; used for contact computation and Gaussian splat initialization.
  • domain assumption The LCP formulation with joint, contact, and friction constraints accurately models non-prehensile interactions.
    Section III-B Eq. 10; relies on the constraints capturing push and poke dynamics.
  • ad hoc to paper Rendering loss from 2DGS provides sufficient gradient signal to identify physics parameters.
    Eq. 8; the authors note in limitations that shadows can distort this loss.
  • domain assumption End-effector commanded translation maps directly to a constant velocity in simulation, ignoring robot dynamics.
    Section III-B: 'the robot's mass is typically much greater... allowing its own dynamics to be ignored.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation." pith.science (2026). https://pith.science/paper/BBI7DRTN

@misc{pith2026250416693,
  author       = {Pith},
  title        = {Pith review of: PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBI7DRTN}},
  note         = {Machine review of arXiv:2504.16693}
}
read the original abstract

While non-prehensile manipulation (e.g., controlled pushing/poking) constitutes a foundational robotic skill, its learning remains challenging due to the high sensitivity to complex physical interactions involving friction and restitution. To achieve robust policy learning and generalization, we opt to learn a world model of the 3D rigid body dynamics involved in non-prehensile manipulations and use it for model-based reinforcement learning. We propose PIN-WM, a Physics-INformed World Model that enables efficient end-to-end identification of a 3D rigid body dynamical system from visual observations. Adopting differentiable physics simulation, PIN-WM can be learned with only few-shot and task-agnostic physical interaction trajectories. Further, PIN-WM is learned with observational loss induced by Gaussian Splatting without needing state estimation. To bridge Sim2Real gaps, we turn the learned PIN-WM into a group of Digital Cousins via physics-aware randomizations which perturb physics and rendering parameters to generate diverse and meaningful variations of the PIN-WM. Extensive evaluations on both simulation and real-world tests demonstrate that PIN-WM, enhanced with physics-aware digital cousins, facilitates learning robust non-prehensile manipulation skills with Sim2Real transfer, surpassing the Real2Sim2Real state-of-the-arts.

Figures

Figures reproduced from arXiv: 2504.16693 by the authors.

Figure 1
Figure 1. PIN-WM is learned from few-shot and task-agnostic physical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our Real2Sim2Real framework for learning non-prehensile manipulation policies. (a) The robot in the target domain moves around [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Manipulation trajectories in simulation obtained by our [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Transition and orientation errors of push task during training. find a good solution in a finite time due to the large search space. Although 2D Physics [67] adopts a differentiable frame￾work, the 2D model finds difficulties in handling 3D rigid body dynamics, resulti…
Figure 5
Figure 5. Figure 5: Our real-world experiment setup. Experiment setup: Our hardware setup consists of a robot, an eye-in-hand camera, and an eye-to-hand camera, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Real-world trajectories of different methods on the [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Real-world trajectories of pushing T-shaped objects of different sizes obtained by our method. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Real-world trajectories of different methods on the [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Push T-shaped object on a slippery plane. Time lapse ASID PIN-WM [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Push cube object on a slippery plane. Time lapse ASID PIN-WM [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Flip a multicolored cube to change its top-surface color [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Energy-Structured Latent World Models with Neural Time Fields for Physically Constistent Open-World Motion Planning

    cs.RO 2026-08 conditional novelty 5.0 of 10

    Energy-structuring a latent world model with port-Hamiltonian dynamics and conditioning Eikonal neural time fields on its predictions improves open-world navigation success and physical collision rate in simulation, t...

  2. Where to Touch, How to Contact: A Hierarchical RL-MPC Framework for Geometry-Aware Sim-to-Real Manipulation

    cs.RO 2026-01 conditional novelty 5.0 of 10

    A hierarchical RL-MPC framework with a 'contact intention' interface achieves data-efficient, robust non-prehensile manipulation that transfers zero-shot to a real robot.

  3. From World Models to World Action Models: A Concise Tutorial for Robotics

    cs.RO 2026-07 unverdicted novelty 4.0 of 10

    World models are action-conditioned predictors of task-relevant futures; world action models couple those futures to robot actions via four paradigms: imagine-then-execute, feature-conditioned, joint, and auxiliary pr...

Reference graph

Works this paper leans on

89 extracted references · 63 canonical work pages · cited by 3 Pith papers

  1. [60]

    Sim-to-real transfer of robotic control with dynamics randomization

    Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation , 2018

    work page 2018

  2. [1]

    Physically embodied gaussian splat- ting: A visually learnt and physically grounded 3d repre- sentation for robotics

    Jad Abou-Chakra, Krishan Rana, Feras Dayoub, and Niko Suenderhauf. Physically embodied gaussian splat- ting: A visually learnt and physically grounded 3d repre- sentation for robotics. In Conference on Robot Learning, 2024

    2024

  3. [2]

    Cosmos world foundation model platform for physical ai

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Bal- aji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. IEEE Conference on Computer Vision and Pattern Recognition, 2025

    2025

  4. [3]

    Posing polygonal objects in the plane by pushing

    Srinivas Akella and Matthew T Mason. Posing polygonal objects in the plane by pushing. The International Journal of Robotics Research , 1998. Domain Rand.RoboGSimASID2D PhysicsPIN-WM (ours) Time lapse Result Fig. 6: Real-world trajectories of different methods on the push task. Large T Small T Time lapse Result Fig. 7: Real-world trajectories of pushing T...

    1998

  5. [4]

    Optnet: Differentiable optimization as a layer in neural networks

    Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In Interna- tional Conference on Machine Learning , 2017

    2017

  6. [5]

    Incremental Few-Shot Adaptation for Non-Prehensile Object Manipulation using Parallelizable Physics Simulators

    Fabian Baumeister, Lukas Mack, and Joerg Stueckler. Incremental few-shot adaptation for non-prehensile ob- ject manipulation using parallelizable physics simulators. arXiv preprint arXiv:2409.13228 , 2024

    work page Pith review arXiv 2024

  7. [6]

    Data quality in imitation learning

    Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in Neural Infor- mation Processing Systems , 2024

    2024

  8. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 2020

    2020

Show all 89 references
  1. [8]

    Neuma: Neural material adaptor for visual grounding of intrinsic dynamics

    Junyi Cao, Shanyan Guan, Yanhao Ge, Wei Li, Xiaokang Yang, and Chao Ma. Neuma: Neural material adaptor for visual grounding of intrinsic dynamics. In Advances in Neural Information Processing Systems , 2024

    2024

  2. [9]

    Neural radiance fields for dynamic view synthesis using local temporal priors

    Rongsen Chen, Junhong Zhao, Fang-Lue Zhang, Andrew Chalmers, and Taehyun Rhee. Neural radiance fields for dynamic view synthesis using local temporal priors. In Computational Visual Media , 2024

    2024

  3. [10]

    Towards Domain Rand.RoboGSimASID2D PhysicsPIN-WM Time lapse Fig

    Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards Domain Rand.RoboGSimASID2D PhysicsPIN-WM Time lapse Fig. 8: Real-world trajectories of different methods on the flip task. hum...

    2022

  4. [11]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2023

    2023

  5. [12]

    Policy transfer via modularity and reward guiding

    Ignasi Clavera, David Held, and Pieter Abbeel. Policy transfer via modularity and reward guiding. In IEEE International Conference on Intelligent Robots and Sys- tems, 2017

    2017

  6. [13]

    Rigid body simulation with contact and constraints

    Michael Bradley Cline. Rigid body simulation with contact and constraints. PhD thesis, University of British Columbia, 2002

    2002

  7. [14]

    Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    2016

  8. [15]

    Automated creation of digital cousins for robust policy learning

    Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. In Conference on Robot Learning , 2024

    2024

  9. [16]

    End-to-end differentiable physics for learning and control

    Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in Neural Information Processing Systems , 2018

    2018

  10. [17]

    A framework for push-grasping in clutter

    Mehmet Remzi Dogar and Siddhartha S Srinivasa. A framework for push-grasping in clutter. In Robotics: Science and Systems , 2011

    2011

  11. [18]

    Finding formations for the non- prehensile object transportation with differentially-driven mobile robots

    Henrik Ebel, Daniel Niklas Fahse, Mario Rosenfelder, and Peter Eberhard. Finding formations for the non- prehensile object transportation with differentially-driven mobile robots. In Symposium on Robot Design, Dynam- ics and Control , 2022

    2022

  12. [19]

    Con- text is everything: Implicit identification for dynamics adaptation

    Ben Evans, Abitha Thankaraj, and Lerrel Pinto. Con- text is everything: Implicit identification for dynamics adaptation. In International Conference on Robotics and Automation, 2022

    2022

  13. [20]

    Rigid body dynamics algorithms

    Roy Featherstone. Rigid body dynamics algorithms . Springer, 2014

    2014

  14. [21]

    Learning visuotactile estimation and control for non-prehensile manipulation under occlusions

    Juan Del Aguila Ferrandis, Joao Moura, and Sethu Vi- jayakumar. Learning visuotactile estimation and control for non-prehensile manipulation under occlusions. In Annual Conference on Robot Learning , 2024

    2024

  15. [22]

    Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing

    Jian Gao, Chun Gu, Youtian Lin, Zhihao Li, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision , 2024

    2024

  16. [23]

    Learning nonprehen- sile dynamic manipulation: Sim2real vision-based policy with a surgical robot

    Radian Gondokaryono, Mustafa Haiderbhai, Sai Aneesh Suryadevara, and Lueder A Kahrs. Learning nonprehen- sile dynamic manipulation: Sim2real vision-based policy with a surgical robot. IEEE Robotics and Automation Letters, 2023

    2023

  17. [24]

    Digital twin: Miti- gating unpredictable, undesirable emergent behavior in complex systems

    Michael Grieves and John Vickers. Digital twin: Miti- gating unpredictable, undesirable emergent behavior in complex systems. Transdisciplinary Perspectives on Complex Systems: New Findings and Approaches , 2017

    2017

  18. [25]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  19. [26]

    Lillicrap, Jimmy Ba, and Mo- hammad Norouzi

    Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mo- hammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020

    2020

  20. [27]

    Lillicrap, Mohammad Norouzi, and Jimmy Ba

    Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021

    2021

  21. [28]

    Temporal difference learning for model predictive control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning , 2022

    2022

  22. [29]

    TD- MPC2: scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD- MPC2: scalable, robust world models for continuous control. In International Conference on Learning Repre- sentations, 2024

    2024

  23. [30]

    Multiple view geometry in computer vision

    Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003

    2003

  24. [31]

    Neuralsim: Augmenting differentiable simulators with neural networks

    Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In IEEE International Conference on Robotics and Automation , 2021

    2021

  25. [32]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020

    2020

  26. [33]

    Data scaling laws in imitation learning for robotic manipulation

    Yingdong Hu, Fanqi Lin, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. In Workshop on X-Embodiment Robot Learning , 2024

    2024

  27. [34]

    2d gaussian splatting for geometri- cally accurate radiance fields

    Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometri- cally accurate radiance fields. In SIGGRAPH, 2024

    2024

  28. [35]

    Imitation learning: A survey of learning methods

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys , 2017

    2017

  29. [36]

    Rlbench: The robot learning bench- mark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning bench- mark & learning environment. IEEE Robotics and Automation Letters, 2020

    2020

  30. [37]

    State: Learning structure and texture representations for novel view synthesis

    Xinyi Jing, Qiao Feng, Yu-Kun Lai, Jinsong Zhang, Yuanqiang Yu, and Kun Li. State: Learning structure and texture representations for novel view synthesis. Computational Visual Media , 2023

    2023

  31. [38]

    Frnerf: Fusion and regularization fields for dynamic view synthesis

    Xinyi Jing, Tao Yu, Renyuan He, Yukun Lai, and Kun Li. Frnerf: Fusion and regularization fields for dynamic view synthesis. Computational Visual Media , 2024

    2024

  32. [39]

    Scalable deep reinforcement learning for vision- based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning, 2018

    2018

  33. [40]

    Physics-based rigid body object tracking and friction filtering from rgb-d videos

    Rama Krishna Kandukuri, Michael Strecke, and Joerg Stueckler. Physics-based rigid body object tracking and friction filtering from rgb-d videos. In International Conference on 3D Vision , 2024

    2024

  34. [41]

    Scal- ing laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal- ing laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  35. [42]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 2023

    2023

  36. [43]

    Robotic control with partial visual information

    Keisuke Kinoshita and Michael Lindenbaum. Robotic control with partial visual information. International Journal of Computer Vision , 2000

    2000

  37. [44]

    Investigating compounding prediction errors in learned dynamics models

    Nathan Lambert, Kristofer Pister, and Roberto Calandra. Investigating compounding prediction errors in learned dynamics models. arXiv preprint arXiv:2203.09637 , 2022

    2022 arXiv

  38. [45]

    Robogsim: A real2sim2real robotic gaussian splatting simulator

    Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, and Ruip- ing Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411.11839 , 2024

    2024 arXiv

  39. [46]

    Lin, Chenfanfu Jiang, and Chuang Gan

    Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Kr- ishna Murthy Jatavallabhula, Ming C. Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In International Conference on Learning Representa...

    2023

  40. [47]

    Difffr: Differentiable sph-based fluid-rigid coupling for rigid body control

    Zhehao Li, Qingyu Xu, Xiaohan Ye, Bo Ren, and Ligang Liu. Difffr: Differentiable sph-based fluid-rigid coupling for rigid body control. ACM Transactions on Graphics , 2023

    2023

  41. [48]

    Deep lagrangian networks: Using physics as model prior for deep learning

    Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2019

    2019

  42. [49]

    Isaac gym: High performance GPU based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Neural Information Process...

    2021

  43. [50]

    Mechanics and planning of manip- ulator pushing operations

    Matthew T Mason. Mechanics and planning of manip- ulator pushing operations. The International Journal of Robotics Research, 1986

    1986

  44. [51]

    Cvxgen: A code generator for embedded convex optimization

    Jacob Mattingley and Stephen Boyd. Cvxgen: A code generator for embedded convex optimization. Optimiza- tion and Engineering , 2012

    2012

  45. [52]

    Active domain randomiza- tion

    Bhairav Mehta, Manfred Diaz, Florian Golemo, Christo- pher J Pal, and Liam Paull. Active domain randomiza- tion. In Conference on Robot Learning , 2020

    2020

  46. [53]

    ASID: Active explo- ration for system identification in robotic manipulation

    Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Dieter Fox, and Abhishek Gupta. ASID: Active explo- ration for system identification in robotic manipulation. In International Conference on Learning Representa- tions, 2024

    2024

  47. [54]

    Structured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. In Robotics: Science and Systems , 2023

    2023

  48. [55]

    Learning robust perceptive locomotion for quadrupedal robots in the wild

    Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022

    2022

  49. [56]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 2021

    2021

  50. [57]

    Neural 3d reconstruction from sparse views using geometric priors

    Tai-Jiang Mu, Hao-Xiang Chen, Jun-Xiong Cai, and Ning Guo. Neural 3d reconstruction from sparse views using geometric priors. Computational Visual Media , 2023

    2023

  51. [58]

    Instant neural graphics primitives with a multiresolution hash encoding

    Thomas M ¨uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics, 2022

    2022

  52. [59]

    J. Krishna Murthy, Miles Macklin, Florian Golemo, Vikram V oleti, Linda Petrini, Martin Weiss, Brean- dan Considine, J ´erˆome Parent-L ´evesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differen- tiable simulation...

    2021

  53. [61]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International Conference on Ma- chine Learning, 2021

    2021

  54. [62]

    Offline reinforcement learning from im- ages with latent space models

    Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from im- ages with latent space models. In Learning for Dynamics and Control, 2021

    2021

  55. [63]

    Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations

    Maziar Raissi, Paris Perdikaris, and George E Karni- adakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 2019

    2019

  56. [64]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Ro- man R ¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  57. [65]

    The cross- entropy method: a unified approach to combinatorial op- timization, Monte-Carlo simulation, and machine learn- ing

    Reuven Y Rubinstein and Dirk P Kroese. The cross- entropy method: a unified approach to combinatorial op- timization, Monte-Carlo simulation, and machine learn- ing. Springer, 2004

    2004

  58. [66]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  59. [67]

    Learning to slide unknown objects with differentiable physics simu- lations

    Changkyu Song and Abdeslam Boularias. Learning to slide unknown objects with differentiable physics simu- lations. In Robotics: Science and Systems , 2020

    2020

  60. [68]

    Diffsdfsim: Dif- ferentiable rigid-body dynamics with implicit shapes

    Michael Strecke and Joerg Stueckler. Diffsdfsim: Dif- ferentiable rigid-body dynamics with implicit shapes. In International Conference on 3D Vision , 2021

    2021

  61. [69]

    Reinforcement learning: An introduc- tion

    Richard S Sutton. Reinforcement learning: An introduc- tion. A Bradford Book , 2018

    2018

  62. [70]

    Sukhatme, Fabio Ramos, and Yashraj S

    Bingjie Tang, Iretiayo Akinola, Jie Xu, Bowen Wen, Ankur Handa, Karl Van Wyk, Dieter Fox, Gaurav S. Sukhatme, Fabio Ramos, and Yashraj S. Narang. Au- tomate: Specialist and generalist assembly policies over diverse geometries. In Robotics: Science and Systems , 2024

    2024

  63. [71]

    Domain ran- domization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain ran- domization for transferring deep neural networks from simulation to the real world. In IEEE International Conference on Intelligent Robots and Systems , 2017

    2017

  64. [72]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In IEEE Conference on Computer Vision and Pattern Recognition , 2024

    2024

  65. [73]

    Model predictive path integral control: From theory to parallel computation

    Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics , 2017

    2017

  66. [74]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, 2022

    2022

  67. [75]

    Recent advances in 3d gaussian splatting

    Tong Wu, Yu-Jie Yuan, Ling-Xiao Zhang, Jie Yang, Yan- Pei Cao, Ling-Qi Yan, and Lin Gao. Recent advances in 3d gaussian splatting. Computational Visual Media , 2024

    2024

  68. [76]

    Jnerf: An efficient heterogeneous nerf model zoo based on jittor

    Guo-Wei Yang, Zheng-Ning Liu, Dong-Yang Li, and Hao-Yang Peng. Jnerf: An efficient heterogeneous nerf model zoo based on jittor. Computational Visual Media, 2023

    2023

  69. [77]

    Visual imitation made easy

    Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. In Conference on Robot Learning , 2021

    2021

  70. [78]

    More than a million ways to be pushed

    Kuan-Ting Yu, Maria Bauza, Nima Fazeli, and Alberto Rodriguez. More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing. In IEEE International Conference on Intelligent Robots and Systems, 2016

    2016

  71. [79]

    Mopo: Model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 2020

    2020

  72. [80]

    Rearrangement with nonprehensile manipulation using deep reinforcement learning

    Weihao Yuan, Johannes A Stork, Danica Kragic, Michael Y Wang, and Kaiyu Hang. Rearrangement with nonprehensile manipulation using deep reinforcement learning. In IEEE International Conference on Robotics and Automation, 2018

    2018

  73. [81]

    Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing tar- get domain data

    Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing tar- get domain data. In IEEE International Conference on Computer Vision, 2019

    2019

  74. [82]

    Learn- ing physically realizable skills for online packing of general 3d shapes

    Hang Zhao, Zherong Pan, Yang Yu, and Kai Xu. Learn- ing physically realizable skills for online packing of general 3d shapes. ACM Transactions on Graphics, 2023

    2023

  75. [83]

    Deliberate planning of 3d bin packing on packing configuration trees

    Hang Zhao, Juzhan Xu, Kexiong Yu, Ruizhen Hu, Chenyang Zhu, and Kai Xu. Deliberate planning of 3d bin packing on packing configuration trees. arXiv preprint arXiv:2504.04421, 2025

    2025 arXiv

  76. [84]

    Dino-wm: World models on pre-trained vi- sual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained vi- sual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024

    2024 arXiv

  77. [85]

    Pushing revisited: Differential flatness, trajectory planning, and stabilization

    Jiaji Zhou, Yifan Hou, and Matthew T Mason. Pushing revisited: Differential flatness, trajectory planning, and stabilization. The International Journal of Robotics Research, 2019

    2019

  78. [86]

    Domain Rand + I

    Wenxuan Zhou, Bowen Jiang, Fan Yang, Chris Paxton, and David Held. Hacman: Learning hybrid actor-critic maps for 6d non-prehensile manipulation. In Conference on Robot Learning , 2023. APPENDIX A. Implementation Details for Baselines All the baselines are implemented carefully...

    2023

  79. [87]

    Superiority over uniform randomization: The only dif- ference between our method and the DR baseline is the different ranges of physics parameters that are used for domain randomization, where DR uses a large range R to ensure that it covers the target parameters θ† while our ...

  80. [88]

    To fairly compare the accuracy across different methods, we estimate one parameter at a time while keeping the others fixed at their GT values

    Identified physical parameters: System identification is inherently ill-posed, as multiple parameter sets can explain the same observations. To fairly compare the accuracy across different methods, we estimate one parameter at a time while keeping the others fixed at their GT ...

  81. [89]

    Robustness to geometry noise: Geometry noise will affect collision detection accuracy and, consequently, system identification. To evaluate this, we conduct experiments on noisy inputs, in which the noise conforms to a Gaussian distribution with a mean of 0 and a variance of σ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.