Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Gaussian-splat simulator reproduces robot soft-body motion without physics

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 05:20 UTC pith:H4DOCCZA

load-bearing objection A credible action-conditioned GS simulator, but the paper never directly tests whether the action input drives the simulation. the 5 major comments →

arxiv 2602.02402 v2 pith:H4DOCCZA submitted 2026-02-02 cs.RO cs.AIcs.CVphysics.app-ph

SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation

classification cs.RO cs.AIcs.CVphysics.app-ph
keywords Gaussian splattingreal-to-sim simulationdeformable objectsrobot manipulationneural dynamicsaction-conditioned simulationsoft-body dynamicslong-horizon stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SoMA claims that soft-body manipulation can be simulated from real video by learning a force-driven dynamics model directly on 3D Gaussian splats, conditioned on robot joint actions. The paper argues this unifies object, environment, and robot in one latent space, avoiding predefined physical parameters. If right, it gives a controllable real-to-sim simulator that stays stable over long horizons and generalizes to unseen actions, supporting policy learning without fresh real-world data. The authors report roughly 20% improvement over physics-based and state-based neural baselines on real robot manipulation benchmarks.

Core claim

In the paper's own terms, SoMA establishes that robot-conditioned, force-driven dynamics over hierarchical Gaussian splats can serve as an end-to-end real-to-sim simulator for deformable objects. It reconstructs the scene, maps it to a unified simulation frame via a scale factor, rigid transform, and gravity direction recovered from the mounted camera and fitted table plane, then predicts per-splat forces (gravity, support, and robot contact) through a graph network. Training with multi-resolution temporal sampling and a momentum-conservation regularizer yields long-horizon stability and generalization to unseen manipulation sequences, demonstrated on rope, cloth, doll, and a full T-shirt fo

What carries the argument

The central mechanism is the action-conditioned force-driven Gaussian-splat dynamics: a hierarchical graph network that, at each step, predicts linear and angular velocities for Gaussian splats from their history and a total interaction force. Environment forces (gravity, table support) are computed analytically; robot forces are predicted from an interaction graph between robot control points and splats, after aligning robot, camera, and reconstruction frames with a similarity transformation and a resolved gravity direction.

Load-bearing premise

The robot-to-simulation mapping (scale factor, rigid alignment, and gravity direction) is assumed to be recovered accurately from the mounted-camera pose and fitted table plane; if any of these is off, every action-conditioned force is applied in the wrong frame, and the learned dynamics cannot be causally correct regardless of network capacity.

What would settle it

In a scene with a deliberately introduced calibration error (for instance, rotating the robot-to-sim transform by a few degrees or fitting the table plane from a misaligned point cloud), measure rollout PSNR and depth RMSE; if the simulator's accuracy degrades sharply or the object drifts off the observed trajectory, that would confirm the alignment step is load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • SoMA provides a real-to-sim backend that can be controlled directly by joint-space actions, enabling open-loop simulation of manipulation strategies without predefined physics.
  • The method generalizes to unseen actions and contact configurations on held-out test sequences, suggesting data-driven simulation can transfer beyond memorized trajectories.
  • Long-horizon stability on T-shirt folding indicates the simulator can handle self-contact and large deformations, supporting task-level analysis and policy development.
  • The same simulator, driven by external motion trajectories rather than robot joint states, still outperforms baselines, showing the action-conditioning mechanism is not limited to robot kinematics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because forces are modeled as explicit environment terms plus a learned robot-contact channel, the approach could plausibly be extended to other external influences (air drag, multiple robots, liquid interaction) by adding analogous force terms; this is a testable extension the paper does not pursue.
  • The alignment step (scale, rigid transform, gravity) is hand-set rather than learned and its error is not propagated into the reported metrics; a stress-test that deliberately perturbs the calibrated transform would directly measure how much the action-conditioned dynamics depend on this mapping.
  • The momentum-consistency regularizer is a self-supervised physical constraint; ablating it and evaluating view-consistency across the multiple cameras would clarify whether it stabilizes occluded dynamics or merely improves visible-region fitting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. SoMA proposes a real-to-sim neural simulator for deformable objects manipulated by a robot. It reconstructs the object as hierarchical 3D Gaussian splats, maps robot joint states and gripper state into a unified simulation coordinate system via forward kinematics and a similarity transform, and trains a graph-neural-network dynamics model that predicts future Gaussian splat states conditioned on past states and on robot-induced forces. Training uses multi-view RGB videos with an occlusion-aware masked image loss and a momentum-consistency regularizer, in a coarse-to-fine temporal schedule. The method is evaluated on a new real-world dataset (rope, doll, cloth, T-shirt) for resimulation and generalization against PhysTwin and GausSim, with additional results on the PhysTwin dataset. The paper reports best average PSNR/SSIM/LPIPS/AbsRel/RMSE and claims a 20% improvement.

Significance. If the action-conditioning result is substantiated, SoMA is a meaningful step toward controllable neural simulators for soft-body manipulation: it explicitly couples joint-space actions to learned Gaussian-splat dynamics and demonstrates stable long-horizon rollouts on cloth folding. Strengths of the paper include end-to-end supervision from real multi-view images (with held-out generalization, so the result is not fitted into existence), a new real-world dataset, multi-view qualitative results, and a clear architectural separation of environmental and robot forces. The main risk is that the central action-conditioning claim is not directly tested; the missing action-ablation study, lack of uncertainty quantification, and an unsupported 20% headline prevent the current version from establishing the claim at the level required by a journal paper.

major comments (5)
  1. [Abstract; §5.2, Table 1] The headline "20% improvement" is not directly supported by the reported numbers. In Table 1, SoMA's PSNR improvement over GausSim is 5.7% on resimulation and 5.1% on generalization; over PhysTwin it is 16.5% and 23.9%. Depth Abs Rel improvements range from 12% to 22%. The paper does not define which metric, baseline, or aggregation yields the 20% figure. Please provide the exact definition and report relative improvements with error bars.
  2. [§4.2.2, Eq. (11); §5.2, Table 3] The central claim is that SoMA is an action-conditioned simulator, but no experiment isolates the effect of the robot action R_t. Eq. (4) includes R_t in the state recurrence and Eq. (11) computes robot forces, yet the ablations in Table 3 remove supervision or multi-resolution training, never actions. A direct test is needed: e.g., shuffle or zero out R_t while keeping the same visual history, or compare against a no-action variant. Without it, the reported generalization could be largely driven by visual-state history rather than by joint-state control.
  3. [§4.2.1, Eq. (6)–(8), Eq. (14)] The robot-to-simulation mapping relies on a scale factor s and rigid transform (R,t) recovered from the mounted-camera pose and a fitted table plane. No sensitivity analysis is provided for this calibration. If s, R, or t is mis-estimated, the robot force f_rob in Eq. (11) is applied in the wrong reference frame, so the learned dynamics cannot be causally correct regardless of network capacity. Please report calibration error or an ablation that perturbs s/R/t and measures the effect on rollout metrics.
  4. [§5.1; Tables 1, 2, 7; Appendix B.3] Quantitative claims lack uncertainty information. Tables 1, 2, and 7 report averages with no standard deviations, number of sequences, or significance tests. Some differences are small (e.g., SSIM 0.971 vs 0.947; LPIPS 0.055 vs 0.086), so it is unclear whether improvements are consistent across sequences. Please provide per-sequence distributions or error bars and paired significance tests where possible.
  5. [§5.1; Appendix B.3] Baselines are adapted "for fair comparison (details in the supplementary material)", but no supplementary material is included with this version. In particular, GausSim cannot consume action inputs and is evaluated without actions in the generalization setting; this is a different task setting rather than a controlled comparison. Please include the baseline implementation details in the main text or appendix and, ideally, a variant of GausSim or another neural simulator augmented with the same action conditioning.
minor comments (6)
  1. [§4.1] "Problem Defination" should be "Problem Definition".
  2. [§4.2.4] The heading "IMMIGRATED SUPERVISION" appears to be a typo; it should likely read "Blended Supervision".
  3. [Table 3] Typos: "Genral" should be "General"; "w/o MRF" should be "w/o MRT"; the references to "Sec. 4.4" and "Sec. 4.3" are inconsistent with the actual numbering (§4.2.4 and §4.2.3).
  4. [Table 5] The caption says "Quantitative results on the cloth folding task" but the table reports clustering parameters for all datasets. The caption does not match the content.
  5. [Fig. 4] "More Reults" should be "More Results"; "phystwin datatests" should be "PhysTwin datasets".
  6. [Table 6] The RMSE values are not monotonically consistent with PSNR (e.g., k=5 gives lower RMSE but lower PSNR than k=10). Please clarify whether these are averages over different rollouts or computed on the same set of frames.

Circularity Check

0 steps flagged

No significant circularity: the dynamics are learned end-to-end from real images and evaluated on held-out sequences; the only self-citation (GausSim backbone/baseline) is not load-bearing.

full rationale

The central claim is empirical rather than definitional. SoMA is trained end-to-end by minimizing an image reconstruction loss (Eq. 5) over real multi-view observations, with actions provided as inputs in Eq. (4) and robot forces computed in Eq. (11); generalization is then measured on held-out manipulation sequences under a 7:3 train/test split. No fitted parameter is renamed as a prediction: the robot-to-sim scale factor, rigid transform, and gravity direction (Eqs. 6–8 and Eq. 14) are geometric calibrations obtained from camera poses, URDF kinematics, and a fitted table plane, not quantities fitted to the reported metrics. The momentum-consistency regularizer (Eq. 13) is a self-supervised constraint, not a supervised target, and it does not by construction equal zero because cluster and child positions are predicted independently by the network. The main self-citation is the hierarchical GS propagation of Sec. 3 (Eqs. 2–3), attributed to the authors' own GausSim work, which is also used as a baseline. This is a build-on-prior-work relation rather than a load-bearing circular justification: the paper's claimed contribution is the force-driven robot-conditioned dynamics, and it is tested against GausSim and PhysTwin on unseen actions. The lack of an action-ablation or action-shuffle experiment is a validation gap for the 'action-conditioned' claim, but it is not a circularity—nothing in the equations makes the output equal to the input by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical entities; its support is from learned latent states and an interaction graph. The free parameters listed are hand-chosen or tuned per dataset, and the core assumptions are that GS representation, simplified environment forces, and the robot interaction graph are sufficient to capture real manipulation dynamics.

free parameters (4)
  • Support distance threshold τ
    In Eq. 10, the support force is activated when distance d_i < τ; τ is not fitted but hand-chosen; its value is not reported.
  • Temporal stride k = 10 (5 for T-shirt)
    Hyperparameter for coarse training stage; ablation shows insensitivity but it is still a hand-chosen knob.
  • Number of robot control points = 30
    Fixed number of control points for the interaction graph; not justified by scaling or ablation.
  • Downsampling rate / cluster counts per object = [0.02,0.2] and [8,800]-[60,3000]
    Hand-tuned per dataset; part of the scene graph construction.
axioms (4)
  • domain assumption The 3D Gaussian splat representation and its hierarchical clustering capture the deformable object's geometry and dynamics sufficiently for simulation.
    Introduced in Sec. 3 and 4.2.2; the entire method operates on this representation; if the splats are not a faithful state (e.g., under heavy occlusion), the simulator cannot succeed.
  • domain assumption The environment force model (Eq. 10) with a single threshold τ and the table-plane gravity direction (Eq. 8) correctly represents environmental contact.
    Sec. 4.2.2; support forces and gravity are modeled explicitly and assumed to be the dominant environmental effects.
  • domain assumption Robot–object interaction is fully captured by a graph between 30 control points and splats, with gripper state as an additional input.
    Sec. 4.2.2, Eq. 11; complex contacts (grasping, sliding, self-contact) are reduced to this interaction graph.
  • ad hoc to paper The momentum consistency regularizer (Eq. 13) imposes physically plausible dynamics in occluded regions.
    Sec. 4.2.4; this is a heuristic that is not derived from physics and is not validated against ground-truth forces.

pith-pipeline@v1.3.0-alltime-deepseek · 13851 in / 10535 out tokens · 90190 ms · 2026-08-03T05:20:49.491106+00:00 · methodology

0 comments
read the original abstract

Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.

Figures

Figures reproduced from arXiv: 2602.02402 by Bo Dai, Hui Wang, Jiangmiao Pang, Kerui Ren, Linning Xu, Mu Huang, Mulin Yu, Yunsong Zhou.

Figure 1
Figure 1. Figure 1: SoMA is a GS neural simulator that reconstructs and simulates deformable object dynamics from real-world robot manipulation. Learning from multi-view RGB observations, it performs action-conditioned simulation directly on Gaussian splats, enabling interaction￾consistent, stable long-horizon resimulation with higher-fidelity rendering under both seen and unseen manipulations. Abstract Simulating deformable … view at source ↗
Figure 2
Figure 2. Figure 2: Framework of SoMA. SoMA takes RGB observations and robot joint-space actions collected from real-world manipulation as input (Left). It reconstructs deformable objects as hierarchical Gaussian splats, and propagates them through a neural simulator with supervision from rendering and dynamics (Middle). Object motion is driven by force-based interactions, where environmental and robot-induced forces act on s… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative resimulation and generalization under robot manipulation. Left: resimulation on training trajectories. Right: generalization to unseen robot actions and contact configurations. Across diverse soft-body objects, including near-linear (rope), near￾planar (cloth), and volumetric (doll) objects, SoMA produces stable, long-horizon simulations that closely match observed dynamics. PhysTwin shows devi… view at source ↗
Figure 4
Figure 4. Figure 4: More Reults (a) multi-view results; (b) T-shirt folding comparison results (c) results on phystwin datatests. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds

    cs.RO 2026-04 unverdicted novelty 6.0

    SIM1 converts sparse real demonstrations into high-fidelity synthetic data through physics-aligned simulation, yielding policies that match real-data performance at a 1:15 ratio with 90% zero-shot success on deformabl...

Reference graph

Works this paper leans on

22 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    J., Tulyakov, S., Wetzstein, G., et al

    Bahmani, S., Liu, X., Yifan, W., Skorokhodov, I., Rong, V ., Liu, Z., Liu, X., Park, J. J., Tulyakov, S., Wetzstein, G., et al. Tc4d: Trajectory-conditioned text-to-4d generation. InEuropean Conference on Computer Vision, pp. 53–72. Springer, 2024a. Bahmani, S., Skorokhodov, I., Siarohin, A., Menapace, W., Qian, G., Vasilkovsky, M., Lee, H.-Y ., Wang, C.,...

  2. [5]

    Table 5.Quantitative results on the cloth folding task under robot manipulation. rope cloth doll T-shirt downsample rate [0.02, 0.2] [0.02, 0.2] [0.02, 0.2] [0.02, 0.2] num of splats 13000 13000 13000 13000 num of clusters [8, 800] [30, 2400] [22, 2200] [60, 3000] 12 Submission and Formatting Instructions for ICML 2026 Network Architecture.Our simulator b...

  3. [10]

    Ravi, N., Gabeur, V ., Hu, Y .-T., Hu, R., Ryali, C., Ma, T., Khedr, H., R ¨adle, R., Rolland, C., Gustafson, L., et al

    Available at https://developer.nvidia.com/isaac-sim. Ravi, N., Gabeur, V ., Hu, Y .-T., Hu, R., Ryali, C., Ma, T., Khedr, H., R ¨adle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714,

  4. [12]

    C., and Dai, B

    Shao, Y ., Huang, M., Loy, C. C., and Dai, B. Gaussim: Fore- seeing reality by gaussian simulator for elastic objects. arXiv preprint arXiv:2412.17804,

  5. [13]

    Text-to-4d dynamic scene generation.arXiv preprint arXiv:2301.11280,

    Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh, D., Johnson, J., et al. Text-to-4d dynamic scene generation.arXiv preprint arXiv:2301.11280,

  6. [17]

    Discussion A.1

    10 Submission and Formatting Instructions for ICML 2026 A. Discussion A.1. Distinguishing Different Technical Paradigms Recent advances in dynamic scene modeling and simulation have led to increasingly similar visual results across different methods, which may obscure their fundamental differences. Despite comparable visual quality, these approaches are m...

  7. [18]

    Consequently, they are not suitable for controllable manipulation or physical reasoning

    focus on recovering temporally consistent geometry and appearance from visual observations, without explicitly modeling physical states or interactions. Consequently, they are not suitable for controllable manipulation or physical reasoning. World Models vs. Physics-Oriented Simulation.World models (Huang et al., 2026; Li et al., 2025; Yu et al.,

  8. [19]

    are typically trained on large-scale video or interaction data and excel at predicting perceptually plausible future observations. However, they are optimized for appearance realism rather than physical or geometric consistency, which can lead to view inconsistency and physically implausible behaviors under external interventions. These characteristics li...

  9. [20]

    with text prompts. We generate two types of masks: object masks, which provide the primary visual observations for end-to-end supervision, and robot masks, which are used to identify occluded regions caused by the manipulator during interaction. Robot State Processing.Given the robot joint states, we compute the end-effector pose and gripper opening param...

  10. [22]

    The model estimates the previous state GSt−1 from the initial splats and regresses the full sequence accordingly

    does not support explicit control or action inputs and instead predicts future Gaussian splat states in a purely state-based autoregressive manner. The model estimates the previous state GSt−1 from the initial splats and regresses the full sequence accordingly. For the resimulation setting, we train GausSim on full sequences following its original protoco...

  11. [1993]

    Dreamgaussian4d: Generative 4d gaussian splatting.arXiv preprint arXiv:2312.17142,

    Ren, J., Pan, L., Tang, J., Zhang, C., Cao, A., Zeng, G., and Liu, Z. Dreamgaussian4d: Generative 4d gaussian splatting.arXiv preprint arXiv:2312.17142,

  12. [1994]

    Vggt: Visual geometry grounded transformer

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., and Novotny, D. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294–5306, 2025a. Wang, Y ., Zhou, J., Zhu, H., Chang, W., Zhou, Y ., Li, Z., Chen, J., Pang, J., Shen, C., and He, T. Pi3: Permutation- equivariant visual geom...

  13. [2004]

    4dgen: Grounded 4d content generation with spatial-temporal consistency.arXiv preprint arXiv:2312.17225,

    Yin, Y ., Xu, D., Wang, Z., Zhao, Y ., and Wei, Y . 4dgen: Grounded 4d content generation with spatial-temporal consistency.arXiv preprint arXiv:2312.17225,

  14. [2007]

    Nvidia isaac sim: A robot simulation frame- work for robotics research,

    9 Submission and Formatting Instructions for ICML 2026 NVIDIA. Nvidia isaac sim: A robot simulation frame- work for robotics research,

  15. [2011]

    Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935,

    Hu, Y ., Anderson, L., Li, T.-M., Sun, Q., Carr, N., Ragan- Kelley, J., and Durand, F. Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935,

  16. [2014]

    Gasp: Gaussian splatting for physic- based simulations.arXiv preprint arXiv:2409.05819,

    Borycki, P., Smolak, W., Waczy´nska, J., Mazur, M., Tadeja, S., and Spurek, P. Gasp: Gaussian splatting for physic- based simulations.arXiv preprint arXiv:2409.05819,

  17. [2016]

    Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos.arXiv preprint arXiv:2503.17973, 2025a

    Jiang, H., Hsu, H.-Y ., Zhang, K., Yu, H.-N., Wang, S., and Li, Y . Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos.arXiv preprint arXiv:2503.17973, 2025a. Jiang, L., Mao, Y ., Xu, L., Lu, T., Ren, K., Jin, Y ., Xu, X., Yu, M., Pang, J., Zhao, F., et al. Anysplat: Feed-forward 3d gaussian splatting from unconstra...

  18. [2019]

    Pointworld: Scaling 3d world models for in-the-wild robotic manipulation.arXiv preprint arXiv:2601.03782,

    Huang, W., Chao, Y .-W., Mousavian, A., Liu, M.-Y ., Fox, D., Mo, K., and Fei-Fei, L. Pointworld: Scaling 3d world models for in-the-wild robotic manipulation.arXiv preprint arXiv:2601.03782,

  19. [2023]

    Wonderplay: Dynamic 3d scene generation from a single image and actions.arXiv preprint arXiv:2505.18151,

    Li, Z., Yu, H.-X., Liu, W., Yang, Y ., Herrmann, C., Wet- zstein, G., and Wu, J. Wonderplay: Dynamic 3d scene generation from a single image and actions.arXiv preprint arXiv:2505.18151,

  20. [2024]

    Worldmirror: Universal 3d world reconstruction with any-prior prompting.arXiv preprint arXiv:2510.10726,

    Liu, Y ., Min, Z., Wang, Z., Wu, J., Wang, T., Yuan, Y ., Luo, Y ., and Guo, C. Worldmirror: Universal 3d world reconstruction with any-prior prompting.arXiv preprint arXiv:2510.10726,

  21. [2025]

    Dynamic 3d gaussian track- ing for graph-based neural dynamics modeling.arXiv preprint arXiv:2410.18912, 2024a

    Zhang, M., Zhang, K., and Li, Y . Dynamic 3d gaussian track- ing for graph-based neural dynamics modeling.arXiv preprint arXiv:2410.18912, 2024a. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni...

  22. [2026]

    The material point method for simulating continuum materials

    Jiang, C., Schroeder, C., Teran, J., Stomakhin, A., and Selle, A. The material point method for simulating continuum materials. InAcm siggraph 2016 courses, pp. 1–52