REVIEW 5 major objections 6 minor 1 cited by
Gaussian-splat simulator reproduces robot soft-body motion without physics
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 05:20 UTC pith:H4DOCCZA
load-bearing objection A credible action-conditioned GS simulator, but the paper never directly tests whether the action input drives the simulation. the 5 major comments →
SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In the paper's own terms, SoMA establishes that robot-conditioned, force-driven dynamics over hierarchical Gaussian splats can serve as an end-to-end real-to-sim simulator for deformable objects. It reconstructs the scene, maps it to a unified simulation frame via a scale factor, rigid transform, and gravity direction recovered from the mounted camera and fitted table plane, then predicts per-splat forces (gravity, support, and robot contact) through a graph network. Training with multi-resolution temporal sampling and a momentum-conservation regularizer yields long-horizon stability and generalization to unseen manipulation sequences, demonstrated on rope, cloth, doll, and a full T-shirt fo
What carries the argument
The central mechanism is the action-conditioned force-driven Gaussian-splat dynamics: a hierarchical graph network that, at each step, predicts linear and angular velocities for Gaussian splats from their history and a total interaction force. Environment forces (gravity, table support) are computed analytically; robot forces are predicted from an interaction graph between robot control points and splats, after aligning robot, camera, and reconstruction frames with a similarity transformation and a resolved gravity direction.
Load-bearing premise
The robot-to-simulation mapping (scale factor, rigid alignment, and gravity direction) is assumed to be recovered accurately from the mounted-camera pose and fitted table plane; if any of these is off, every action-conditioned force is applied in the wrong frame, and the learned dynamics cannot be causally correct regardless of network capacity.
What would settle it
In a scene with a deliberately introduced calibration error (for instance, rotating the robot-to-sim transform by a few degrees or fitting the table plane from a misaligned point cloud), measure rollout PSNR and depth RMSE; if the simulator's accuracy degrades sharply or the object drifts off the observed trajectory, that would confirm the alignment step is load-bearing.
If this is right
- SoMA provides a real-to-sim backend that can be controlled directly by joint-space actions, enabling open-loop simulation of manipulation strategies without predefined physics.
- The method generalizes to unseen actions and contact configurations on held-out test sequences, suggesting data-driven simulation can transfer beyond memorized trajectories.
- Long-horizon stability on T-shirt folding indicates the simulator can handle self-contact and large deformations, supporting task-level analysis and policy development.
- The same simulator, driven by external motion trajectories rather than robot joint states, still outperforms baselines, showing the action-conditioning mechanism is not limited to robot kinematics.
Where Pith is reading between the lines
- Because forces are modeled as explicit environment terms plus a learned robot-contact channel, the approach could plausibly be extended to other external influences (air drag, multiple robots, liquid interaction) by adding analogous force terms; this is a testable extension the paper does not pursue.
- The alignment step (scale, rigid transform, gravity) is hand-set rather than learned and its error is not propagated into the reported metrics; a stress-test that deliberately perturbs the calibrated transform would directly measure how much the action-conditioned dynamics depend on this mapping.
- The momentum-consistency regularizer is a self-supervised physical constraint; ablating it and evaluating view-consistency across the multiple cameras would clarify whether it stabilizes occluded dynamics or merely improves visible-region fitting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SoMA proposes a real-to-sim neural simulator for deformable objects manipulated by a robot. It reconstructs the object as hierarchical 3D Gaussian splats, maps robot joint states and gripper state into a unified simulation coordinate system via forward kinematics and a similarity transform, and trains a graph-neural-network dynamics model that predicts future Gaussian splat states conditioned on past states and on robot-induced forces. Training uses multi-view RGB videos with an occlusion-aware masked image loss and a momentum-consistency regularizer, in a coarse-to-fine temporal schedule. The method is evaluated on a new real-world dataset (rope, doll, cloth, T-shirt) for resimulation and generalization against PhysTwin and GausSim, with additional results on the PhysTwin dataset. The paper reports best average PSNR/SSIM/LPIPS/AbsRel/RMSE and claims a 20% improvement.
Significance. If the action-conditioning result is substantiated, SoMA is a meaningful step toward controllable neural simulators for soft-body manipulation: it explicitly couples joint-space actions to learned Gaussian-splat dynamics and demonstrates stable long-horizon rollouts on cloth folding. Strengths of the paper include end-to-end supervision from real multi-view images (with held-out generalization, so the result is not fitted into existence), a new real-world dataset, multi-view qualitative results, and a clear architectural separation of environmental and robot forces. The main risk is that the central action-conditioning claim is not directly tested; the missing action-ablation study, lack of uncertainty quantification, and an unsupported 20% headline prevent the current version from establishing the claim at the level required by a journal paper.
major comments (5)
- [Abstract; §5.2, Table 1] The headline "20% improvement" is not directly supported by the reported numbers. In Table 1, SoMA's PSNR improvement over GausSim is 5.7% on resimulation and 5.1% on generalization; over PhysTwin it is 16.5% and 23.9%. Depth Abs Rel improvements range from 12% to 22%. The paper does not define which metric, baseline, or aggregation yields the 20% figure. Please provide the exact definition and report relative improvements with error bars.
- [§4.2.2, Eq. (11); §5.2, Table 3] The central claim is that SoMA is an action-conditioned simulator, but no experiment isolates the effect of the robot action R_t. Eq. (4) includes R_t in the state recurrence and Eq. (11) computes robot forces, yet the ablations in Table 3 remove supervision or multi-resolution training, never actions. A direct test is needed: e.g., shuffle or zero out R_t while keeping the same visual history, or compare against a no-action variant. Without it, the reported generalization could be largely driven by visual-state history rather than by joint-state control.
- [§4.2.1, Eq. (6)–(8), Eq. (14)] The robot-to-simulation mapping relies on a scale factor s and rigid transform (R,t) recovered from the mounted-camera pose and a fitted table plane. No sensitivity analysis is provided for this calibration. If s, R, or t is mis-estimated, the robot force f_rob in Eq. (11) is applied in the wrong reference frame, so the learned dynamics cannot be causally correct regardless of network capacity. Please report calibration error or an ablation that perturbs s/R/t and measures the effect on rollout metrics.
- [§5.1; Tables 1, 2, 7; Appendix B.3] Quantitative claims lack uncertainty information. Tables 1, 2, and 7 report averages with no standard deviations, number of sequences, or significance tests. Some differences are small (e.g., SSIM 0.971 vs 0.947; LPIPS 0.055 vs 0.086), so it is unclear whether improvements are consistent across sequences. Please provide per-sequence distributions or error bars and paired significance tests where possible.
- [§5.1; Appendix B.3] Baselines are adapted "for fair comparison (details in the supplementary material)", but no supplementary material is included with this version. In particular, GausSim cannot consume action inputs and is evaluated without actions in the generalization setting; this is a different task setting rather than a controlled comparison. Please include the baseline implementation details in the main text or appendix and, ideally, a variant of GausSim or another neural simulator augmented with the same action conditioning.
minor comments (6)
- [§4.1] "Problem Defination" should be "Problem Definition".
- [§4.2.4] The heading "IMMIGRATED SUPERVISION" appears to be a typo; it should likely read "Blended Supervision".
- [Table 3] Typos: "Genral" should be "General"; "w/o MRF" should be "w/o MRT"; the references to "Sec. 4.4" and "Sec. 4.3" are inconsistent with the actual numbering (§4.2.4 and §4.2.3).
- [Table 5] The caption says "Quantitative results on the cloth folding task" but the table reports clustering parameters for all datasets. The caption does not match the content.
- [Fig. 4] "More Reults" should be "More Results"; "phystwin datatests" should be "PhysTwin datasets".
- [Table 6] The RMSE values are not monotonically consistent with PSNR (e.g., k=5 gives lower RMSE but lower PSNR than k=10). Please clarify whether these are averages over different rollouts or computed on the same set of frames.
Circularity Check
No significant circularity: the dynamics are learned end-to-end from real images and evaluated on held-out sequences; the only self-citation (GausSim backbone/baseline) is not load-bearing.
full rationale
The central claim is empirical rather than definitional. SoMA is trained end-to-end by minimizing an image reconstruction loss (Eq. 5) over real multi-view observations, with actions provided as inputs in Eq. (4) and robot forces computed in Eq. (11); generalization is then measured on held-out manipulation sequences under a 7:3 train/test split. No fitted parameter is renamed as a prediction: the robot-to-sim scale factor, rigid transform, and gravity direction (Eqs. 6–8 and Eq. 14) are geometric calibrations obtained from camera poses, URDF kinematics, and a fitted table plane, not quantities fitted to the reported metrics. The momentum-consistency regularizer (Eq. 13) is a self-supervised constraint, not a supervised target, and it does not by construction equal zero because cluster and child positions are predicted independently by the network. The main self-citation is the hierarchical GS propagation of Sec. 3 (Eqs. 2–3), attributed to the authors' own GausSim work, which is also used as a baseline. This is a build-on-prior-work relation rather than a load-bearing circular justification: the paper's claimed contribution is the force-driven robot-conditioned dynamics, and it is tested against GausSim and PhysTwin on unseen actions. The lack of an action-ablation or action-shuffle experiment is a validation gap for the 'action-conditioned' claim, but it is not a circularity—nothing in the equations makes the output equal to the input by construction.
Axiom & Free-Parameter Ledger
free parameters (4)
- Support distance threshold τ
- Temporal stride k =
10 (5 for T-shirt)
- Number of robot control points =
30
- Downsampling rate / cluster counts per object =
[0.02,0.2] and [8,800]-[60,3000]
axioms (4)
- domain assumption The 3D Gaussian splat representation and its hierarchical clustering capture the deformable object's geometry and dynamics sufficiently for simulation.
- domain assumption The environment force model (Eq. 10) with a single threshold τ and the table-plane gravity direction (Eq. 8) correctly represents environmental contact.
- domain assumption Robot–object interaction is fully captured by a graph between 30 control points and splats, with gripper state as an additional input.
- ad hoc to paper The momentum consistency regularizer (Eq. 13) imposes physically plausible dynamics in occluded regions.
read the original abstract
Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.
Figures
Forward citations
Cited by 1 Pith paper
-
SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds
SIM1 converts sparse real demonstrations into high-fidelity synthetic data through physics-aligned simulation, yielding policies that match real-data performance at a 1:15 ratio with 90% zero-shot success on deformabl...
Reference graph
Works this paper leans on
-
[1]
J., Tulyakov, S., Wetzstein, G., et al
Bahmani, S., Liu, X., Yifan, W., Skorokhodov, I., Rong, V ., Liu, Z., Liu, X., Park, J. J., Tulyakov, S., Wetzstein, G., et al. Tc4d: Trajectory-conditioned text-to-4d generation. InEuropean Conference on Computer Vision, pp. 53–72. Springer, 2024a. Bahmani, S., Skorokhodov, I., Siarohin, A., Menapace, W., Qian, G., Vasilkovsky, M., Lee, H.-Y ., Wang, C.,...
-
[5]
Table 5.Quantitative results on the cloth folding task under robot manipulation. rope cloth doll T-shirt downsample rate [0.02, 0.2] [0.02, 0.2] [0.02, 0.2] [0.02, 0.2] num of splats 13000 13000 13000 13000 num of clusters [8, 800] [30, 2400] [22, 2200] [60, 3000] 12 Submission and Formatting Instructions for ICML 2026 Network Architecture.Our simulator b...
2026
-
[10]
Available at https://developer.nvidia.com/isaac-sim. Ravi, N., Gabeur, V ., Hu, Y .-T., Hu, R., Ryali, C., Ma, T., Khedr, H., R ¨adle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714,
-
[12]
Shao, Y ., Huang, M., Loy, C. C., and Dai, B. Gaussim: Fore- seeing reality by gaussian simulator for elastic objects. arXiv preprint arXiv:2412.17804,
-
[13]
Text-to-4d dynamic scene generation.arXiv preprint arXiv:2301.11280,
Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh, D., Johnson, J., et al. Text-to-4d dynamic scene generation.arXiv preprint arXiv:2301.11280,
-
[17]
Discussion A.1
10 Submission and Formatting Instructions for ICML 2026 A. Discussion A.1. Distinguishing Different Technical Paradigms Recent advances in dynamic scene modeling and simulation have led to increasingly similar visual results across different methods, which may obscure their fundamental differences. Despite comparable visual quality, these approaches are m...
2026
-
[18]
Consequently, they are not suitable for controllable manipulation or physical reasoning
focus on recovering temporally consistent geometry and appearance from visual observations, without explicitly modeling physical states or interactions. Consequently, they are not suitable for controllable manipulation or physical reasoning. World Models vs. Physics-Oriented Simulation.World models (Huang et al., 2026; Li et al., 2025; Yu et al.,
2026
-
[19]
are typically trained on large-scale video or interaction data and excel at predicting perceptually plausible future observations. However, they are optimized for appearance realism rather than physical or geometric consistency, which can lead to view inconsistency and physically implausible behaviors under external interventions. These characteristics li...
2026
-
[20]
with text prompts. We generate two types of masks: object masks, which provide the primary visual observations for end-to-end supervision, and robot masks, which are used to identify occluded regions caused by the manipulator during interaction. Robot State Processing.Given the robot joint states, we compute the end-effector pose and gripper opening param...
2022
-
[22]
The model estimates the previous state GSt−1 from the initial splats and regresses the full sequence accordingly
does not support explicit control or action inputs and instead predicts future Gaussian splat states in a purely state-based autoregressive manner. The model estimates the previous state GSt−1 from the initial splats and regresses the full sequence accordingly. For the resimulation setting, we train GausSim on full sequences following its original protoco...
2004
-
[1993]
Dreamgaussian4d: Generative 4d gaussian splatting.arXiv preprint arXiv:2312.17142,
Ren, J., Pan, L., Tang, J., Zhang, C., Cao, A., Zeng, G., and Liu, Z. Dreamgaussian4d: Generative 4d gaussian splatting.arXiv preprint arXiv:2312.17142,
-
[1994]
Vggt: Visual geometry grounded transformer
Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., and Novotny, D. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294–5306, 2025a. Wang, Y ., Zhou, J., Zhu, H., Chang, W., Zhou, Y ., Li, Z., Chen, J., Pang, J., Shen, C., and He, T. Pi3: Permutation- equivariant visual geom...
-
[2004]
Yin, Y ., Xu, D., Wang, Z., Zhao, Y ., and Wei, Y . 4dgen: Grounded 4d content generation with spatial-temporal consistency.arXiv preprint arXiv:2312.17225,
-
[2007]
Nvidia isaac sim: A robot simulation frame- work for robotics research,
9 Submission and Formatting Instructions for ICML 2026 NVIDIA. Nvidia isaac sim: A robot simulation frame- work for robotics research,
2026
-
[2011]
Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935,
Hu, Y ., Anderson, L., Li, T.-M., Sun, Q., Carr, N., Ragan- Kelley, J., and Durand, F. Difftaichi: Differentiable programming for physical simulation.arXiv preprint arXiv:1910.00935,
Pith/arXiv arXiv 1910
-
[2014]
Gasp: Gaussian splatting for physic- based simulations.arXiv preprint arXiv:2409.05819,
Borycki, P., Smolak, W., Waczy´nska, J., Mazur, M., Tadeja, S., and Spurek, P. Gasp: Gaussian splatting for physic- based simulations.arXiv preprint arXiv:2409.05819,
-
[2016]
Jiang, H., Hsu, H.-Y ., Zhang, K., Yu, H.-N., Wang, S., and Li, Y . Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos.arXiv preprint arXiv:2503.17973, 2025a. Jiang, L., Mao, Y ., Xu, L., Lu, T., Ren, K., Jin, Y ., Xu, X., Yu, M., Pang, J., Zhao, F., et al. Anysplat: Feed-forward 3d gaussian splatting from unconstra...
-
[2019]
Huang, W., Chao, Y .-W., Mousavian, A., Liu, M.-Y ., Fox, D., Mo, K., and Fei-Fei, L. Pointworld: Scaling 3d world models for in-the-wild robotic manipulation.arXiv preprint arXiv:2601.03782,
-
[2023]
Li, Z., Yu, H.-X., Liu, W., Yang, Y ., Herrmann, C., Wet- zstein, G., and Wu, J. Wonderplay: Dynamic 3d scene generation from a single image and actions.arXiv preprint arXiv:2505.18151,
-
[2024]
Liu, Y ., Min, Z., Wang, Z., Wu, J., Wang, T., Yuan, Y ., Luo, Y ., and Guo, C. Worldmirror: Universal 3d world reconstruction with any-prior prompting.arXiv preprint arXiv:2510.10726,
-
[2025]
Zhang, M., Zhang, K., and Li, Y . Dynamic 3d gaussian track- ing for graph-based neural dynamics modeling.arXiv preprint arXiv:2410.18912, 2024a. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni...
-
[2026]
The material point method for simulating continuum materials
Jiang, C., Schroeder, C., Teran, J., Stomakhin, A., and Selle, A. The material point method for simulating continuum materials. InAcm siggraph 2016 courses, pp. 1–52
2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.