Pith. sign in

REVIEW 4 major objections 5 minor 84 references

A single forward pass identifies deformable-object material and predicts its motion, replacing slow per-object optimization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:42 UTC pith:IBDHCL3F

load-bearing objection Clever combination of feed-forward material inference and residual correction inside a differentiable MPM simulator, with a real speedup claim—but the missing train/test split undermines the generalization claim as written. the 4 major comments →

arxiv 2607.20653 v1 pith:IBDHCL3F submitted 2026-07-22 cs.RO cs.CVcs.LG

PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics

Haocheng Yin , Shuohan Tao , Yongsheng Chen , Lu Gan This is my paper
classification cs.RO cs.CVcs.LG
keywords deformable objectsworld modelsmaterial identificationdifferentiable MPMresidual physicsrobotic manipulationactive explorationsim-to-real gap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PhysCoRe couples a differentiable Material Point Method simulator with two feed-forward neural networks: one infers per-particle elasticity from a short window of observed motion, and the other corrects the simulator's internal dynamics with a bounded velocity residual. The paper claims this replaces slow per-object material optimization while improving prediction accuracy on real manipulation sequences. Against a strong per-object optimization baseline, Chamfer distance drops by about 44% on elastic objects and 30% on elastoplastic objects, and material identification takes about 11 seconds instead of hundreds or thousands of seconds. A per-particle confidence output rises only where an object has actually deformed, providing a signal for active exploration. If correct, the model offers a fast, physically grounded path to adapting a robot's world model to each new deformable object.

Core claim

PhysCoRe is a world model that keeps a differentiable MLS-MPM simulator at its core and delegates material identification to a feed-forward module, Material from Motion (MfM), which predicts per-particle Young's modulus, Poisson's ratio, and a per-particle confidence from visual observations. A second module, Residual from Dynamics (RfD), predicts a bounded velocity residual on the simulation grid to absorb systematic sim-to-real error such as friction and contact mismatch. MfM is pre-trained on synthetically augmented episodes generated by re-rolling the simulator under Perlin-noise material fields, then RfD is trained on real episodes with MfM frozen. The paper reports that PhysCoRe outper

What carries the argument

The central mechanism is the differentiable MLS-MPM simulator with APIC transfers, which carries particle state through explicit substeps. Inside this simulator, MfM encodes canonical positions and per-frame displacements via Fourier features, processes them with a graph U-Net and a temporal GRU module, and decodes per-particle material and confidence. RfD reads a compact 19-dimensional feature per active grid cell, runs a sparse 3D U-Net with FiLM conditioning, and outputs a scaled-tanh grid-velocity residual that is zero-initialized so it begins as an identity correction. Training combines a confidence-weighted material regression loss with a plasticity-classification loss for MfM, and a C

Load-bearing premise

The load-bearing premise is that synthetic material fields generated from 14 real episodes, using Perlin noise and episode-level statistics, span the real stiffness and plasticity of rope, towel, plush bear, and Play-Doh closely enough that a single feed-forward pass correctly identifies material on novel objects.

What would settle it

Run PhysCoRe on a held-out object category not present in the training set, such as a cloth or a foam cube, and compare its prediction error and identification time against a per-object optimization baseline; if the feed-forward material estimate does not improve over a random material initialization, or if MfM confidence stays uniformly high despite poor predictions, the generalization claim is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the paper's claims hold, a robot can estimate a new deformable object's material in roughly 11 seconds from a short motion window, instead of minutes of per-object optimization.
  • Predicted rollouts stay physically consistent over long horizons because the material-grounded MPM backbone carries the dynamics, while the learned residual absorbs only the remaining systematic error.
  • The per-particle confidence map directly indicates which regions of an object have not been reliably identified, enabling an active exploration policy to deform exactly those regions.
  • The residual module's bounded, zero-initialized correction preserves the simulator's stability and momentum structure, making it a safe drop-in component for differentiable planning.
  • The two-stage training recipe—synthetic material augmentation followed by real-episode residual correction—could transfer to other physics simulators and constitutive models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is closed-loop control: continuously re-run MfM as new observations arrive, letting the robot re-plan with refreshed material estimates while RfD keeps the simulation on track.
  • The confidence-guided exploration signal could be formalized as an information-gain objective, where the robot chooses actions that maximize expected reduction in material uncertainty across the object's surface.
  • Because the augmentation scheme varies material fields while keeping geometry and controller motion fixed, the method's generalization depends on how well real objects' material distributions are represented by Perlin-noise fields with episode-level statistics; objects with sharply heterogeneous or history-dependent materials may require richer augmentation.
  • The residual correction's separation of labor—MfM handles object-specific material, RfD handles generic sim-to-real bias—suggests RfD could be reused across object categories once trained on a sufficiently diverse set of residual dynamics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PhysCoRe, a dynamics model that combines a differentiable MLS-MPM simulator with two feed-forward neural modules: MfM, which infers per-particle material parameters (logE, ν) and a per-particle confidence from a short window of RGB-D observations, and RfD, which predicts a bounded grid-velocity residual injected into the MPM cycle. MfM is pre-trained on synthetic augmented episodes generated from 14 real PhysTwin captures; RfD is then trained on real episodes with MfM frozen. The system is evaluated on 12 real manipulation episodes spanning elastic objects (rope, towel, plush bear) and elastoplastic objects (Play-Doh), comparing against PhysTwin and PGND. The main reported results are lower Chamfer distance and tracking loss, faster identification (11.4 s vs 930 s and 8280 s), an ablation showing RfD improves over MfM alone, and qualitative confidence maps for online exploration.

Significance. If the central claims hold, the contribution is substantial: a feed-forward material estimator plus a residual correction that matches or beats per-object optimization on real deformable-object prediction, while preserving physical structure and enabling online adaptation. The design has several concrete strengths: the residual is zero-initialized and bounded, preserving the analytical simulator at the start; the correction is applied on the grid through the same APIC transfer; the ablation isolates the contribution of RfD; and the per-particle confidence signal is a useful and relatively novel output. The inclusion of real manipulation episodes with multiple objects is also a strength. However, the paper's load-bearing evidence depends on evaluation protocol details that are currently under-specified: the train/test split between RfD's real-episode training data and the 12 evaluation episodes is not stated, and the reported comparisons lack variance or statistical testing. These omissions prevent the reader from verifying that the reported gains reflect generalization rather than in-sample fitting.

major comments (4)
  1. [§4.2.4, §5.1, §5.2] The paper does not state whether the 12 real episodes used for the Table 2 evaluation are disjoint from the episodes used to train RfD in §4.2.4. Section 5.1 says the 12 episodes 'drive our offline evaluations,' and Section 5.2 evaluates each method on the second 50% of each episode in time, but no episode-level train/test split is reported. If RfD was trained on the same 12 episodes (or if the MfM augmentation in §4.2.3 was generated from the same PhysTwin captures), the reported CD reductions of 43.7%/30.5% and tracking improvements could be in-sample fitting rather than generalization to unseen objects or interactions. This is load-bearing for the abstract's claim of generalization. Please state explicitly the episode-level split, report per-episode results, and ideally provide a leave-one-out or held-out-object evaluation.
  2. [§5.1, Table 2] Table 2 reports single scalar values for each metric with no standard deviations, confidence intervals, or statistical significance tests. With only 12 episodes total, split across elastic and elastoplastic categories, the reader cannot assess whether the reported differences are meaningful. For example, on elastic objects PhysCoRe's IoU (68.2) and SSIM (0.302) are actually lower than PhysTwin's (70.5 and 0.309), while the text in §5.2 states PhysCoRe 'attains the best overall accuracy.' Please clarify that 'accuracy' refers specifically to the geometric CD/Track metrics, and add per-episode standard deviations and per-object-category breakdowns for all metrics.
  3. [§4.2.3, §1, §7] MfM's generalization claim rests on synthetic material augmentation derived from 14 real PhysTwin episodes, with Perlin-noise material fields and episode-level statistics (Eq. 6). The text does not specify whether the 12 real evaluation episodes share object identities, geometries, or controller trajectories with these 14 source episodes. If they do overlap, the experiment does not test zero-shot object-level generalization, only material variation on seen geometries. The paper's own Limitations section concedes that zero-shot category-level transfer is not tested. This is not an error by itself, but the abstract and contributions should be calibrated to the actual claim, and the authors should provide evidence on held-out object categories or clearly state that only in-category material generalization is demonstrated.
  4. [§5.5, Figure 4] The confidence-guided exploration section is purely qualitative: Figure 4 shows that confidence rises in deformed regions, but there is no quantitative evaluation that this confidence signal improves exploration efficiency or material identification compared to a random or uniform probing baseline. Since 'confidence-guided active exploration' is listed as a contribution, an absence of any metric or comparison leaves this claim unsupported. Please add a quantitative measure (e.g., identification error versus number of interactions, or comparison against random exploration) or explicitly re-scope the claim to a qualitative demonstration.
minor comments (5)
  1. [General] The phrase 'best overall accuracy' in §5.2 is imprecise because Table 2 shows mixed visual metrics (IoU and SSIM lower than PhysTwin on elastic objects). Recommend wording such as 'best geometric accuracy' or 'best on the primary geometry and tracking metrics.'
  2. [Table 2] The table formatting makes CD values hard to parse (no leading zeros, variable decimal places). Add units and consistent formatting, and consider reporting mean ± std.
  3. [§4.2.1] The phrase 'Poisson's Ratio' has a typo (should be 'Poisson's ratio'). Also, in Eq. (8) there is an extraneous double space in the L2 term.
  4. [§4.2.4] The loss in Eq. (7) uses a per-particle confidence c_p multiplied by SL1 and subtracts λ log c_p. The text says the log term 'keeps c_p from inflating,' but the sign of the log term encourages larger c_p; the bound comes from the trade-off with the weighted SL1. This explanation could be clarified.
  5. [§5.1] The evaluation is on only 12 real episodes, but no mention is made of the number of frames per episode or the length of the prediction horizon. This context is needed to interpret the magnitudes of CD and tracking loss.

Circularity Check

1 steps flagged

RfD is trained on the same 12 evaluation episodes with the same Chamfer/L2 reconstruction loss used in Table 2; no held-out episode split is stated, so the headline accuracy gains may be in-sample fitting rather than prediction.

specific steps
  1. fitted input called prediction [Section 4.2.4 (Eq. 8), Section 5.1, Section 5.2]
    "With MfM frozen, RfD is trained end-to-end across MPM substeps. Each episode is rolled out with the MPM simulator from the first observed frame, and we process this rollout in windows of K camera frames. At the start of each window MfM refreshes its predictions(ϕ,c, π), and RfD applies its correction every H_r substeps. At each frame the loss compares the predicted configuration X_t to the observations, combining a one-sided Chamfer distance from the point cloud Ŷ_t with a mean L2 error on the visible tracked particles T_t,"

    Eq. (8) is the same Chamfer/L2 reconstruction error that Table 2 reports as the headline CD/tracking metrics. Section 5.1 says the same 12 real episodes 'drive our offline evaluations of future prediction,' and Section 5.2 defines the only held-out part as the second 50% of each episode in time, not as episodes withheld from RfD training. Because RfD training rolls out 'each episode' and applies the loss 'at each frame,' the evaluation frames could have contributed gradients to the residual network. The paper never states a train/eval episode split. Consequently, the reported 43.7%/30.5% CD reductions over PhysTwin may be in-sample fitting of RfD to the evaluation episodes rather than a test of feed-forward generalization; the 'prediction' is not independent of the fitted residual.

full rationale

The central accuracy claim in Section 5.2 ('PhysCoRe attains the best overall accuracy') depends on evaluating the learned residual module RfD on episodes it has not been trained on. The paper does not provide that split. Section 4.2.4 says RfD is trained by rolling out each real episode and supervising 'at each frame' with the Chamfer/L2 loss of Eq. (8); Section 5.1 says the same 12 real episodes drive the offline evaluations; and Section 5.2's 'held-out remainder' is defined only as the later 50% of each episode in time. Thus the training objective and the evaluation metric coincide, and the evaluation episodes may have directly contributed to the RfD weights. This is a fitted-input-called-prediction pattern: the residual is optimized to minimize the exact error later reported as a prediction result. The issue is partial rather than total: MfM is pre-trained on synthetic augmented rollouts, so the material-inference component has some independent content, and the MfM-only ablation row provides some evidence not contaminated by RfD's in-sample fit. However, the full-model headline numbers and the RfD ablation gains (13.1%/17.8%) are precisely the quantities affected. Section 7 acknowledges the residual is trained on 'a limited set of deformable objects' but does not state that evaluation objects are disjoint, so the limitation does not resolve the circularity concern. No self-citation uniqueness or ansatz-import circularity was found.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central accuracy claim rests on many hand-set simulator constants and loss weights, a self-generated synthetic material distribution, and an asserted-but-uncalibrated confidence signal. No new physical entity is introduced. The main dependency is that MfM's synthetic pre-training transfers to the real objects and that RfD's corrections remain stable.

free parameters (6)
  • Material augmentation per-episode statistics (μ_logE, σ_logE, μ_ν, σ_ν) = drawn from fixed uniform ranges; exact ranges not reported
    Eq. (6) uses these to map Perlin-noise fields to per-particle Young's modulus and Poisson's ratio; MfM is pre-trained on this synthetic distribution, so the generalization claim depends on this choice.
  • Material bounds logE∈[5,11], ν∈[0.05,0.45] = logE in [5,11], ν in [0.05,0.45]
    Bounded sigmoid Eq. (11) clips MfM's outputs; if real materials fall outside, predictions are truncated.
  • Residual bound δ_max = not reported
    Eq. (5) caps the learned velocity residual; its value controls stability and expressiveness but is not given.
  • RfD correction period H_r = 10 substeps
    Appendix C; hand-chosen, affects how often the learned correction acts.
  • Loss weights w_C, w_L2, λ, β = not reported
    Eqs. (7)-(8); balance Chamfer, tracking, confidence regularization, and plasticity cross-entropy.
  • Simulator hand-tuned parameters = grid 32, Δt=6.66e-4 s, α=0.999, ρ=100, μ=0.5, e=0
    Appendix A Table 5; manually chosen, set the analytical baseline RfD corrects.
axioms (5)
  • domain assumption MLS-MPM with fixed corotated elasticity and von Mises plasticity is an adequate physics backbone for the tested objects.
    Section 3; if the constitutive family cannot represent real behaviors, RfD must absorb more than residuals.
  • ad hoc to paper Synthetic material augmentation (Perlin-noise fields mapped via Eq. (6)) spans real material variations of rope, towel, bear, and Play-Doh.
    Section 4.2.3; no real material ground truth is used; MfM is trained entirely on this generated distribution.
  • domain assumption CoTracker3 tracks and Grounded SAM2 masks provide sufficiently accurate particle supervision for training and evaluation.
    Section 5.1; errors in tracking propagate to losses and confidence.
  • domain assumption RfD's zero-initialized bounded residual preserves MPM physical structure and stability.
    Section 4.2.2; asserted, not proven.
  • ad hoc to paper Confidence reweighting in Eq. (7) yields a meaningful uncertainty that correlates with identification quality.
    No calibration benchmark; the claimed reliability is inferred from qualitative maps.

pith-pipeline@v1.3.0-alltime-deepseek · 15066 in / 12263 out tokens · 98358 ms · 2026-08-01T09:42:49.088542+00:00 · methodology

0 comments
read the original abstract

Predicting how deformable objects evolve under robotic manipulation is a longstanding challenge. Existing approaches typically rely on per-object optimization to fit material parameters, which can be slow and cannot generalize, while end-to-end learned alternatives extrapolate poorly and often violate basic physical structure. We present PhysCoRe, a physics-corrected residual world model that couples a differentiable Material Point Method (MPM) simulator with two feed-forward neural networks. A material refinement module, Material from Motion (MfM), infers per-particle elasticity from visual observations, grounding the simulator in object-specific physics. A residual correction module, Residual from Dynamics (RfD), learns the discrepancy and predicts corrections to the simulator's internal dynamics, absorbing systematic biases that the analytical model cannot capture. This design also supports online material identification on novel objects. MfM adapts from limited interactions, and its predictive uncertainty steers further exploration toward the regions where its estimate is least confident. Experiments on real deformable-object manipulation sequences show that PhysCoRe outperforms state-of-the-art baselines in prediction accuracy, and that its predicted confidence forms a reliable distribution across the object's geometry, providing a natural signal for future confidence-guided exploration.

Figures

Figures reproduced from arXiv: 2607.20653 by Haocheng Yin, Lu Gan, Shuohan Tao, Yongsheng Chen.

Figure 1
Figure 1. Figure 1: Overview of PhysCoRe. From segmented RGB-D observations, MfM infers the object’s per-particle material. A differentiable MPM simulator then rolls out the action-conditioned dynamics, with RfD injecting a velocity residual ∆vi to correct the grid velocity. The two modules are supervised by a material regression loss and a reconstruction loss respectively against the observations. in a single forward pass th… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative rollout comparison. Predicted object positions are rendered with 3DGS. Compared with PhysTwin, PhysCoRe predicts future dynamics that more closely match the real observations. Method Elastic Objects Elastoplastic Objects CD↓ Track↓ IoU%↑ PSNR↑ SSIM↑ LPIPS↓ CD↓ Track↓ IoU%↑ PSNR↑ SSIM↑ LPIPS↓ PGND 0.03420 0.13175 37.8 10.74 0.103 0.416 0.04311 0.13263 23.9 9.87 0.081 0.440 PhysTwin 0.01617 0.030… view at source ↗
Figure 3
Figure 3. Figure 3: MfM confidence during material identification. The per-particle confidence is rendered onto the object with 3DGS. As more motion is observed, MfM becomes more confident in its material estimates. Brighter colors indicate higher confidence. low high Rope Towel Bear [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Confidence-guided exploration on a real robot. As a KUKA arm probes each object, MfM confi￾dence starts low everywhere and rises only in regions the arm has actively deformed. It stays low elsewhere, so the map indicates where further interaction is most informative. Brighter colors indicate higher confidence. in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 7 linked inside Pith

  1. [1]

    Y . Hu, Y . Fang, Z. Ge, Z. Qu, Y . Zhu, A. Pradhana, and C. Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling.ACM Transactions on Graphics (TOG), 37(4):1–14, 2018

    2018

  2. [2]

    Jiang, H.-Y

    H. Jiang, H.-Y . Hsu, K. Zhang, H.-N. Yu, S. Wang, and Y . Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7219–7230, 2025

    2025

  3. [3]

    Sanchez-Gonzalez, J

    A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. InInternational Conference on Machine Learning (ICML), pages 8459–8468. PMLR, 2020

    2020

  4. [4]

    Pfaff, M

    T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia. Learning mesh-based simu- lation with graph networks. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  5. [5]

    Zhang, B

    K. Zhang, B. Li, K. Hauser, and Y . Li. Particle-grid neural dynamics for learning deformable object models from rgb-d videos. InProceedings of Robotics: Science and Systems (RSS), 2025

    2025

  6. [6]

    Y . Chen, Y . Hu, L. Sun, T. Kusnur, L. Herlant, and C. Jiang. Empm: Embodied mpm for modeling and simulation of deformable objects.IEEE Robotics and Automation Letters (RA- L), 11(4):4179–4186, 2026

    2026

  7. [7]

    X. Provot. Deformation constraints in a mass-spring model to describe rigid cloth behaviour. InGraphics Interface, pages 147–154, 1995

    1995

  8. [8]

    T. Liu, A. W. Bargteil, J. F. O’Brien, and L. Kavan. Fast simulation of mass-spring systems. ACM Transactions on Graphics (TOG), 32(6):1–7, 2013

    2013

  9. [9]

    García, C

    M. García, C. Mendoza, L. Pastor, and A. Rodríguez. Optimized linear FEM for modeling deformable objects.Computer Animation and Virtual Worlds, 17(3-4):393–402, 2006

    2006

  10. [10]

    Sifakis and J

    E. Sifakis and J. Barbi ˇc. FEM simulation of 3d deformable solids: A practitioner’s guide to theory, discretization and model reduction. InACM SIGGRAPH 2012 Courses, pages 1–50, 2012

    2012

  11. [11]

    Müller, B

    M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff. Position based dynamics.Journal of Visual Communication and Image Representation, 18(2):109–118, 2007

    2007

  12. [12]

    Sulsky, S.-J

    D. Sulsky, S.-J. Zhou, and H. L. Schreyer. Application of a particle-in-cell method to solid mechanics.Computer Physics Communications, 87(1-2):236–252, 1995

    1995

  13. [13]

    Jiang, C

    C. Jiang, C. Schroeder, J. Teran, A. Stomakhin, and A. Selle. The material point method for simulating continuum materials. InACM SIGGRAPH 2016 Courses, pages 1–52, 2016

    2016

  14. [14]

    Zhong, H.-X

    L. Zhong, H.-X. Yu, J. Wu, and Y . Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. InEuropean Conference on Computer Vision (ECCV), pages 407–

  15. [15]

    Y . Lin, C. Lin, J. Xu, and Y . Mu. OmniphysGS: 3d constitutive gaussians for general physics- based dynamics generation. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  16. [16]

    P. Ma, P. Y . Chen, B. Deng, J. B. Tenenbaum, T. Du, C. Gan, and W. Matusik. Learning neural constitutive laws from motion observations for generalizable PDE dynamics. InInternational Conference on Machine Learning (ICML), 2023. 9

    2023

  17. [17]

    K. M. Jatavallabhula, M. Macklin, F. Golemo, V . V oleti, L. Petrini, M. Weiss, B. Considine, J. Parent-Lévesque, K. Xie, K. Erleben, L. Paull, F. Shkurti, D. Nowrouzezahrai, and S. Fi- dler. gradSim: Differentiable simulation for system identification and visuomotor control. In International Conference on Learning Representations (ICLR), 2021

    2021

  18. [18]

    Z. Gao, J. Mao, H.-X. Yu, H. Lou, E. Y .-t. Jia, J. Barbiˇc, J. Wu, and Y . Wang. Seeing the wind from a falling leaf. InAdvances in Neural Information Processing Systems (NeurIPS), pages 48278–48298, 2025

    2025

  19. [19]

    Li, Y .-L

    X. Li, Y .-L. Qiao, P. Y . Chen, K. M. Jatavallabhula, M. Lin, C. Jiang, and C. Gan. PAC-NeRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identifica- tion. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  20. [20]

    H.-y. Chen, E. Tretschk, T. Stuyck, P. Kadlecek, L. Kavan, E. V ouga, and C. Lassner. Virtual elastic objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15827–15837, 2022

    2022

  21. [21]

    X. Yang, Z. Ji, and Y .-K. Lai. Differentiable physics-based system identification for robotic manipulation of elastoplastic materials.International Journal of Robotics Research (IJRR), 44 (13):2126–2155, 2025

    2025

  22. [22]

    Zhang, H.-X

    T. Zhang, H.-X. Yu, R. Wu, B. Y . Feng, C. Zheng, N. Snavely, J. Wu, and W. T. Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. InEuropean Conference on Computer Vision (ECCV), pages 388–406. Springer, 2024

    2024

  23. [23]

    Huang, H

    T. Huang, H. Zhang, Y . Zeng, Z. Zhang, H. Li, W. Zuo, and R. W. H. Lau. DreamPhysics: Learning physics-based 3d dynamics with video diffusion priors. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025

    2025

  24. [24]

    F. Liu, H. Wang, S. Yao, S. Zhang, J. Zhou, and Y . Duan. Physics3D: Learning physical properties of 3d gaussians via video diffusion.arXiv preprint arXiv:2406.04338, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    C. Lv, Z. Chen, D. Di, W. Zhang, H. Li, W. Chen, and C. Li. Physgm: Large physical gaus- sian model for feed-forward 4d synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  26. [26]

    B. Ai, S. Tian, H. Shi, Y . Wang, T. Pfaff, C. Tan, H. I. Christensen, H. Su, J. Wu, and Y . Li. A review of learning-based dynamics models for robotic manipulation.Science Robotics, 10 (106):eadt1497, 2025

    2025

  27. [27]

    P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu. Interaction networks for learning about objects, relations and physics. InAdvances in Neural Information Processing Systems (NeurIPS), 2016

    2016

  28. [28]

    Y . Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  29. [29]

    X. Lin, Y . Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. InConference on Robot Learning (CoRL), pages 256–266. PMLR, 2021

    2021

  30. [30]

    X. Ma, D. Hsu, and W. S. Lee. Learning latent graph dynamics for visual manipulation of deformable objects. InInternational Conference on Robotics and Automation (ICRA), pages 8266–8273. IEEE, 2022

    2022

  31. [31]

    H. Shi, H. Xu, S. Clarke, Y . Li, and J. Wu. RoboCook: Long-horizon elasto-plastic object manipulation with diverse tools. InConference on Robot Learning (CoRL), 2023. 10

    2023

  32. [32]

    Y . Wang, Y . Li, K. Driggs-Campbell, L. Fei-Fei, and J. Wu. Dynamic-resolution model learn- ing for object pile manipulation. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  33. [33]

    H. Shi, H. Xu, Z. Huang, Y . Li, and J. Wu. RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3d with graph networks.International Journal of Robotics Research (IJRR), 43(4):533–549, 2024

    2024

  34. [34]

    Zhang, K

    M. Zhang, K. Zhang, and Y . Li. Dynamic 3d gaussian tracking for graph-based neural dynam- ics modeling. InConference on Robot Learning (CoRL), pages 1851–1862, 2024

    2024

  35. [35]

    S. Xue, S. Cheng, P. Kachana, and D. Xu. Neural field dynamics model for granular object piles manipulation. InConference on Robot Learning (CoRL), pages 2821–2837. PMLR, 2023

    2023

  36. [36]

    T. Tian, H. Li, B. Ai, X. Yuan, Z. Huang, and H. Su. Diffusion dynamics models with genera- tive state estimation for cloth manipulation. InConference on Robot Learning (CoRL), pages 1703–1725, 2025

    2025

  37. [37]

    Bauer, Z

    D. Bauer, Z. Xu, and S. Song. DoughNet: A visual predictive model for topological manipu- lation of deformable objects. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  38. [38]

    M. Yan, Y . Zhu, N. Jin, and J. Bohg. Self-supervised learning of state estimation for manipulat- ing deformable linear objects.IEEE Robotics and Automation Letters (RA-L), 5(2):2372–2379, 2020

    2020

  39. [39]

    Orozco, T

    S. Orozco, T. Kusnur, B. May, G. Konidaris, and L. Herlant. Learning equivariant neural- augmented object dynamics from few interactions.arXiv preprint arXiv:2605.02699, 2026

    Pith/arXiv arXiv 2026

  40. [40]

    W. F. Whitney, T. Lopez-Guevara, T. Pfaff, Y . Rubanova, T. Kipf, K. Stachenfeld, and K. R. Allen. Learning 3d particle-based simulators from RGB-D videos. InInternational Conference on Learning Representations (ICLR), pages 50009–50034, 2024

    2024

  41. [41]

    W. F. Whitney, J. Varley, D. Jain, K. Choromanski, S. Singh, and V . Sindhwani. Modeling the real world with high-density visual particle dynamics. InConference on Robot Learning (CoRL), 2024

    2024

  42. [42]

    Zhang, B

    K. Zhang, B. Li, K. Hauser, and Y . Li. AdaptiGraph: Material-adaptive graph-based neural dynamics for robotic manipulation. InProceedings of Robotics: Science and Systems (RSS), 2024

    2024

  43. [43]

    Longhini, M

    A. Longhini, M. Moletta, A. Reichlin, M. C. Welle, D. Held, Z. Erickson, and D. Kragic. EDO- Net: Learning elastic properties of deformable objects from graph dynamics. InInternational Conference on Robotics and Automation (ICRA), pages 3875–3881. IEEE, 2023

    2023

  44. [44]

    C. Wang, Y . Zhang, X. Zhang, Z. Wu, X. Zhu, S. Jin, T. Tang, and M. Tomizuka. Offline- online learning of deformation model for cable manipulation with graph neural networks.IEEE Robotics and Automation Letters (RA-L), 7(2):5544–5551, 2022

    2022

  45. [45]

    Huang, Y .-W

    W. Huang, Y .-W. Chao, A. Mousavian, M.-Y . Liu, D. Fox, K. Mo, and F.-F. Li. Point- world: Scaling 3d world models for in-the-wild robotic manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  46. [46]

    Huang, Q

    S. Huang, Q. Chen, X. Zhang, J. Sun, and M. Schwager. Particleformer: A 3d point cloud world model for multi-object, multi-material robotic manipulation. InConference on Robot Learning (CoRL), pages 4941–4957, 2025

    2025

  47. [47]

    Daniel, C

    T. Daniel, C. Qi, D. Haramati, A. Zadeh, C. Li, A. Tamar, D. Pathak, and D. Held. Latent particle world models: Self-supervised object-centric stochastic dynamics modeling. InInter- national Conference on Learning Representations (ICLR), 2026. 11

    2026

  48. [48]

    T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models. In International Conference on Learning Representations (ICLR), 2020

    2020

  49. [49]

    G. Zhou, H. Pan, Y . LeCun, and L. Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. InInternational Conference on Machine Learning (ICML), 2025

    2025

  50. [50]

    G. Lu, B. Jia, P. Li, Y . Chen, Z. Wang, Y . Tang, and S. Huang. GWM: Towards scalable gaus- sian world models for robotic manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9263–9274, 2025

    2025

  51. [51]

    T. Yin, Z. Mei, Z. Zheng, M. Yamane, D. Wang, J. Sceats, S. M. Bateman, L. Zha, A. Badithela, O. Shorinwa, and A. Majumdar. Playworld: Learning robot world models from autonomous play.arXiv preprint arXiv:2603.09030, 2026

    Pith/arXiv arXiv 2026

  52. [52]

    Z. He, B. Ai, T. Mu, Y . Liu, W. Wan, J. Fu, Y . Du, H. I. Christensen, and H. Su. Scaling cross- embodiment world models for dexterous manipulation.arXiv preprint arXiv:2511.01177, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Driess, Z

    D. Driess, Z. Huang, Y . Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. InConference on Robot Learning (CoRL), pages 1755–1768, 2022

    2022

  54. [54]

    B. Ai, S. Tian, H. Shi, Y . Wang, C. Tan, Y . Li, and J. Wu. RoboPack: Learning tactile-informed dynamics models for dense packing. InProceedings of Robotics: Science and Systems (RSS), 2024

    2024

  55. [55]

    H. Qi, H. Yin, A. Zhu, Y . Du, and H. Yang. Inference-time enhancement of generative robot policies via predictive world modeling.IEEE Robotics and Automation Letters (RA-L), 11 (5):5534–5541, 2026. Previously titled: Strengthening Generative Robot Policies through Predictive World Modeling

    2026

  56. [56]

    Z. Liu, G. Zhou, J. He, T. Marcucci, F.-F. Li, J. Wu, and Y . Li. Model-based control with sparse neural dynamics. InAdvances in Neural Information Processing Systems (NeurIPS), pages 6280–6296, 2023

    2023

  57. [57]

    Abou-Chakra, K

    J. Abou-Chakra, K. Rana, F. Dayoub, and N. Suenderhauf. Physically embodied gaussian splatting: A visually learnt and physically grounded 3d representation for robotics. InConfer- ence on Robot Learning (CoRL), pages 513–530, 2024

    2024

  58. [58]

    Abou-Chakra, L

    J. Abou-Chakra, L. Sun, K. Rana, B. May, K. Schmeckpeper, N. Suenderhauf, M. V . Minniti, and L. Herlant. Real-is-Sim: Bridging the sim-to-real gap with a dynamic digital twin.arXiv preprint arXiv:2504.03597, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Patel, K

    S. Patel, K. Zhang, S. Pokkali, S. Lazebnik, and Y . Li. Learning physics-guided residual dynamics for deformable object simulation.arXiv preprint arXiv:2607.13451, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    Jiang, C

    C. Jiang, C. Schroeder, A. Selle, J. Teran, and A. Stomakhin. The affine particle-in-cell method. ACM Transactions on Graphics (TOG), 34(4):1–10, 2015

    2015

  61. [61]

    Y . Hu, J. Liu, A. Spielberg, J. B. Tenenbaum, W. T. Freeman, J. Wu, D. Rus, and W. Matusik. ChainQueen: A real-time differentiable physical simulator for soft robotics. InInternational Conference on Robotics and Automation (ICRA), pages 6265–6271. IEEE, 2019

    2019

  62. [62]

    Schenck and D

    C. Schenck and D. Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning (CoRL), pages 317–335, 2018

    2018

  63. [63]

    H. Su, X. Li, T. Xue, C. Jiang, and M. Aanjaneya. A generalized constitutive model for versatile mpm simulation and inverse learning with differentiable physics.Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1–20, 2023. 12

    2023

  64. [64]

    Mittal, P

    H. Mittal, P. Zhuang, H.-Y . Lee, and S. Tulsiani. UniPhy: Learning a unified constitutive model for inverse physics simulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16208–16218, 2025

    2025

  65. [65]

    Huang, Y

    Z. Huang, Y . Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan. PlasticineLab: A soft-body manipulation benchmark with differentiable physics. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  66. [66]

    Luiten, G

    J. Luiten, G. Kopanas, B. Leibe, and D. Ramanan. Dynamic 3d gaussians: Tracking by persis- tent dynamic view synthesis. InInternational Conference on 3D Vision (3DV), pages 800–809. IEEE, 2024

    2024

  67. [67]

    Pumarola, E

    A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10318–10327, 2021

    2021

  68. [68]

    G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20310–20320, 2024

    2024

  69. [69]

    Z. Yang, X. Gao, W. Zhou, S. Jiao, Y . Zhang, and X. Jin. Deformable 3d gaussians for high- fidelity monocular dynamic scene reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20331–20341, 2024

    2024

  70. [70]

    Longhini, M

    A. Longhini, M. Büsching, B. P. Duisterhof, J. Lundell, J. Ichnowski, M. Björkman, and D. Kragic. Cloth-Splatting: 3d cloth state estimation from rgb supervision. InConference on Robot Learning (CoRL), pages 2845–2865, 2024

    2024

  71. [71]

    T. Xie, Z. Zong, Y . Qiu, X. Li, Y . Feng, Y . Yang, and C. Jiang. PhysGaussian: Physics- integrated 3d gaussians for generative dynamics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4389–4398, 2024

    2024

  72. [72]

    Huang, Y

    B. Huang, Y . Chen, R. Lu, G. Zeng, H. Zha, Y . Pei, and S. Huang. Gaussianfluent: Gaus- sian simulation for dynamic scenes with mixed materials. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  73. [73]

    Golemo, A

    F. Golemo, A. A. Taiga, A. Courville, and P.-Y . Oudeyer. Sim-to-real transfer with neural- augmented robot simulation. InConference on Robot Learning (CoRL), pages 817–828, 2018

    2018

  74. [74]

    A. Ajay, J. Wu, N. Fazeli, M. Bauza, L. P. Kaelbling, J. B. Tenenbaum, and A. Rodriguez. Augmenting physical simulators with stochastic neural networks: Case study of planar push- ing and bouncing. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3066–3073. IEEE, 2018

    2018

  75. [75]

    J. Gao, M. Y . Michelis, A. Spielberg, and R. K. Katzschmann. Sim-to-real of soft robots with learned residual physics.IEEE Robotics and Automation Letters (RA-L), 2024

    2024

  76. [76]

    Catmull and R

    E. Catmull and R. Rom. A class of local interpolating splines. InComputer Aided Geometric Design, pages 317–326. Academic Press, 1974

    1974

  77. [77]

    Gao and S

    H. Gao and S. Ji. Graph U-Nets. InProceedings of the International Conference on Machine Learning (ICML), pages 2083–2092, 2019

    2083

  78. [78]

    Perez, F

    E. Perez, F. Strub, H. de Vries, V . Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018

    2018

  79. [79]

    Çiçek, A

    Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. 3D U-Net: Learning dense volumetric segmentation from sparse annotation. InInternational Conference on Medi- cal Image Computing and Computer-Assisted Intervention (MICCAI), pages 424–432, 2016. 13

    2016

  80. [80]

    Graham, M

    B. Graham, M. Engelcke, and L. van der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9224–9232, 2018

    2018

Showing first 80 references.