Pith. sign in

REVIEW 3 major objections 3 minor 34 references

GPU-batched robot simulators are not deterministic data sources: identical setups diverge across parallel environments and repeated runs, and scaling widens the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-02 08:27 UTC pith:YRWLLH5R

load-bearing objection A solid, useful benchmark for GPU-simulator reproducibility that overclaims in the abstract: Type 1 'variability' is deterministic batching bias, not non-determinism, but the typology and metrics are worth keeping. the 3 major comments →

arxiv 2607.13059 v1 pith:YRWLLH5R submitted 2026-07-06 cs.RO

GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI

classification cs.RO
keywords GPU-accelerated simulationrobot learningsim-to-real transferdeterminismreproducibilityparallel simulation benchmarkEarth Mover's Distancestochasticity regimes
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the massively parallel GPU simulators now used as data infrastructure for embodied AI carry hidden trade-offs between speed, physical fidelity, and determinism, and it builds a benchmark, GPUSimBench, to expose them. Using controlled inclined-plane collisions matched to a real apparatus, it measures how closely each of seven simulators reproduces the real final distribution, and it quantifies two kinds of drift that survive fixed seeds: parallel variability across environments in one run, and run-to-run variability across repeated runs. It finds four empirical regimes of stochasticity, from fully deterministic to doubly variable, and shows that simply scaling environment counts can compromise reproducibility. A sympathetic reader comes away with a practical ranking of simulators for sim-to-real work and a warning that batched GPU execution is not a neutral data source.

Core claim

The central discovery is that batched execution on a GPU injects non-determinism into physics simulation even when random seeds are fixed and task-level randomization is disabled. Across seven simulators under unified conditions, GPUSimBench records each simulator's throughput, memory footprint, distribution-level agreement with real-world data (measured as Earth Mover's Distance on the planar cube positions), and two variability metrics. Four regimes emerge: some simulators show drift between parallel environments but stable aggregate distributions across runs; others show both forms of drift; one shows neither; one shows only run-to-run drift. The drift is traced to operation-order-depende

What carries the argument

The load-bearing machinery is the benchmark's two metrics together with the real-world EMD. Parallel Variability is the average pairwise Earth Mover's Distance (a measure, in centimeters, of how much planar mass must be moved to make one environment's final cube positions match another's) between all pairs of environments in a single run; Run-to-Run Variability is the same distance between the aggregate distributions of independent runs. Crossing these two presence/absence axes yields the four stochasticity regimes, and comparing each simulator's aggregate distribution to the measured real-world distribution yields the physical-alignment ranking. These three numbers turn 'the simulator is no

Load-bearing premise

The physical-alignment ranking assumes that the single set of lab-measured material and contact parameters transfers faithfully to all seven simulators' different solvers without per-simulator calibration.

What would settle it

Re-run a simulator classified as Type 2 (both variabilities absent) on two different GPU models with identical seeds; if the run-to-run Earth Mover's Distance becomes nonzero, the claim that this regime is stable across runs is a configuration artifact rather than an inherent property.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, reproducibility in GPU-batched simulation must be measured per simulator and per scale; assuming determinism from fixed seeds is unsafe.
  • The four regimes imply that 'deterministic' is not a single property: a simulator can be internally consistent within a run yet unstable across runs, or vice versa, so reproducibility claims need to specify which axis.
  • Sim-to-real contact benchmarks should expect large engine-to-engine spread in physical alignment; choices made for throughput can dominate the physical error.
  • For repeated-run comparisons or parameter tuning, prefer simulators with neither variability; for single-run evaluation with plausible contact behavior, other simulators may suffice.
  • Scaling to larger environment counts can push a simulator into a noisier regime, so scaling studies should report variability alongside throughput.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The variability metrics likely understate the effect on end-to-end robot learning, since policy gradients average over environments and runs; if environments drift, the effective sample size and gradient noise change in ways current RL pipelines do not track.
  • The physical-alignment ranking mixes intrinsic solver fidelity with parameter transfer; a follow-up that calibrates friction and restitution per simulator would separate engine realism from parameter mismatch, and could reorder the ranking.
  • A natural extension is to test whether variability grows with contact richness or simulation duration; the inclined-collision test is one contact-rich scene, while a simple cube-free-fall scene would be a lower-bound check.
  • The four regimes suggest a design criterion for simulator builders: report which regime a release occupies at a given batch size, so users can decide whether to average across environments or across runs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper introduces GPUSimBench, a benchmark for GPU-accelerated robotic simulators, evaluating seven simulators (Isaac Lab, ManiSkill, Genesis, Madrona, MuJoCo Warp, MJX, MuJoCo Playground) along three axes: parallel scalability (throughput Eq. (1) and memory Eq. (2)), physical fidelity to a real-world inclined-plane collision experiment (EMD, Eq. (10)), and execution variability (parallel and run-to-run EMD, Eqs. (8) and (11)). The authors find large throughput and memory differences, a wide range of sim-to-real EMD values (2.52 cm to 211.60 cm), and four empirical determinism regimes based on the presence or absence of the two variability metrics. They conclude that GPU-batched execution introduces inherent non-determinism and provide simulator-selection guidelines.

Significance. GPUSimBench is a useful and well-scoped empirical contribution. The methodology is controlled in important ways: fixed seeds, disabled rendering, a unified timestep, matched scenes, warmup phases, and an external real-world apparatus. The variability metrics are clearly defined, and the four regimes give practitioners a practical framework for reasoning about reproducibility. The strongest and most defensible finding is that reproducibility is not guaranteed by setting a seed alone and must be measured per simulator and per scale. Explicit credit is due for the authors' care in controlling task-level randomization and for openly acknowledging the benchmark's scope limits in Section VII. If the results hold, the paper provides valuable guidance for choosing simulators in large-scale robot learning. However, the headline claim that GPU batching is 'inherently non-deterministic' goes beyond the data: only Type 3 simulators show run-to-run variability, and Type 1's parallel-only variability is more naturally interpreted as deterministic batching bias. The sim-to-real ranking also rests on a single parameter set transferred to heterogeneous contact models. Both issues are fixable by ref

major comments (3)
  1. [Abstract; Section VI.A; Table IV] The abstract and Section I claim that GPU-batched execution has 'inherent non-determinism' characterized by 'significant run-to-run and inter-environment variability.' Table IV does not support this universal claim. Isaac Lab and ManiSkill (Type 1) have Parallel Variability 4.21 and 4.76 cm, respectively, but Run-to-Run Variability 0.00±0.00; Genesis and MJX (Type 2) have both 0.00; Playground (Type 4) has run-to-run variability only. Type 1's pattern is expected from a deterministic mapping between environment index and numerical outcome (e.g., fixed thread ordering), not from stochastic execution. The robust conclusion is the weaker one: reproducibility must be measured per simulator and scale. Please revise the abstract and the 'inherent non-determinism' phrasing, and present the four regimes as observed patterns rather than evidence of universal stochasticity.
  2. [Section IV.B; Table II; Table IV] The physical-alignment ranking uses one lab-identified set of material/contact parameters (Table II) for all seven simulators. Since PhysX, XPBD, Taichi, and MuJoCo implement contact and friction quite differently, the same nominal friction/restitution values do not transfer with equal meaning. The EMD values in Table IV—e.g., ManiSkill 2.52 cm vs. Madrona 211.60 cm—therefore conflate simulator fidelity with parameter-transfer mismatch. The manuscript acknowledges 'best-effort parameter matching' (Section IV.B) but the guidelines in Section VI.C still rely on this ranking. Please add a per-simulator calibration experiment or, at minimum, explicitly reframe the metric as 'sim-to-real alignment under a common, uncalibrated parameter set' and temper the corresponding recommendations.
  3. [Section VI.B] The proposed mechanism for Parallel Variability is 'different thread schedules' and 'floating-point accumulation depends on operation order.' This is inconsistent with the measured Type 1 data: if thread schedules differed across runs, Run-to-Run Variability would not be 0.00±0.00. For Isaac Lab and ManiSkill, the effective execution order is stable across runs, so the nonzero Parallel Variability is a deterministic per-environment batching bias. The section should distinguish (i) stable intra-run heterogeneity (deterministic batching), (ii) unstable inter-run numerics (true non-determinism), and (iii) absence of both, and adjust the causal language accordingly.
minor comments (3)
  1. [Table III] The maximum-environment counts appear to be rendered as '2 11', '2 12', etc., which is likely exponent notation but is confusing as typeset. Please clarify with superscripts or a note such as '2^11'.
  2. [Fig. 2 caption] The caption mentions 'dice-drop joint pose distribution and Gaussian ball-drop distribution setups' that are 'omitted here due to space constraints.' Either include these experiments in the release or remove the dangling reference to avoid confusion about the benchmark's completeness.
  3. [Table IV caption] The checkmarks and ✗ markers are defined only in the text. Please state in the caption that a checkmark means the corresponding variability is nonzero after rounding at the reported precision, since values like 0.00±0.00 are precision-dependent.

Circularity Check

0 steps flagged

No significant circularity: GPUSimBench is an empirical benchmark whose claims rest on external measurements, not on self-referential fits.

full rationale

The paper is an empirical benchmark, not a derivation. Its central claims—scalability, physical alignment as measured by EMD against a real-world apparatus, and the four empirical regimes of variability—are computed directly from simulator outputs using the definitions in Eqs. (7)–(11). The variability metrics (Parallel Variability and Run-to-Run Variability) are descriptive aggregates of measured cube positions, and the four regimes are labels on whether those metrics are nonzero after rounding. Nothing in the classification is fitted to the claim it supports. The lab-identified material and contact parameters in Table II are experimental inputs, not outputs of the analysis, and the paper explicitly cautions that the EMD values are distribution-level agreement with a measured reference under best-effort parameter matching. The skeptical concern that Type 1 'Parallel Variability' combined with zero Run-to-Run Variability may reflect deterministic batching bias rather than true stochasticity is a question of physical interpretation and benchmark validity, not a circularity: the empirical observation does not reduce by construction to the paper's inputs. Self-citations in the introduction ([4], [5]) are background references to the authors' prior work and are not load-bearing for any benchmark conclusion. No uniqueness theorem, ansatz, or fitted parameter is imported to force a result. The limitation section is explicit about the benchmark's scope rather than masking any circular step. Therefore the appropriate score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The benchmark rests on standard EMD math plus domain assumptions about parameter transferability, seed-controlled stochasticity, and equal-sized distribution comparison. The one fitted input is the lab-measured material/contact parameter set.

free parameters (1)
  • Real-world material/contact parameters (friction, restitution, mass) = Table II: s=0.65/0.35/0.35, k=0.45/0.25/0.25, e=0.40/0.40/0.55
    Obtained by lab identification ('obtained through identification in professional laboratory facilities') and applied as simulator inputs. If these values are wrong or need per-engine recalibration, every sim2real EMD ranking in Table IV shifts.
axioms (4)
  • domain assumption A single lab-identified parameter set transfers to all seven simulators' contact models without per-simulator calibration.
    Table II and Section IV.B: one set of friction/restitution/mass values is used across PhysX, XPBD, Taichi, and MuJoCo solvers; no per-engine tuning is performed.
  • domain assumption Fixing random seeds and disabling task-level randomization isolates numerical/execution nondeterminism from intentional stochasticity.
    Section VI.A: 'we fix random seeds and disable all task-level randomization, so any observed variability reflects numerical and execution-level effects.'
  • domain assumption Aggregating Nenv=16 parallel environments and K=16 real trials yields comparable planar point sets for one-to-one EMD.
    Section III.D: equal point counts (Nenv*n = K*n) make assignment-based EMD well-defined; distributional equivalence is assumed.
  • domain assumption Disabling rendering does not alter physics stepping behavior.
    Section IV.A: 'we disable rendering and measure physics stepping time only'; this is standard practice but assumes rendering path does not affect solver behavior.

pith-pipeline@v1.3.0-alltime-deepseek · 3041 in / 3843 out tokens · 135040 ms · 2026-08-02T08:27:12.976342+00:00 · methodology

0 comments
read the original abstract

Data-driven embodied AI is rapidly transitioning into a paradigm that scales training through massively parallel simulation, where GPU-accelerated simulators serve as the foundational data infrastructure. However, as computational throughput scales, the underlying trade-offs between parallel efficiency, physical fidelity, and execution determinism remain largely unexamined, hindering the development of reliable robot learning. In this paper, we expose the hidden limits of mainstream GPU-based robotic simulators (e.g., Isaac Lab, Genesis) by introducing GPUSimBench, which focuses on scalability, physical consistency, and computational determinism. First, GPUSimBench establishes a physical grounding evaluation with a controlled inclined-plane task, quantifying the distributional alignment between simulated dynamics and their real-world counterparts. Second, we benchmark parallel scalability by measuring throughput and memory footprints across scaling environment counts. Crucially, beyond standard performance metrics, we unveil and quantify the inherent non-determinism introduced by GPU-batched execution, characterized by significant run-to-run and inter-environment variability even under identical initial conditions. Finally, we identify four empirical regimes of stochasticity within current simulator stacks, highlighting that unbounded scaling can compromise reproducibility without explicit constraints.

Figures

Figures reproduced from arXiv: 2607.13059 by Dmitry Yudin, Hengjie Li, Huzhenyu Zhang, Jingcheng Pang, Li Ma, Shenghai Yuan, Wenrui Yan.

Figure 1
Figure 1. Figure 1: An overview of key features of GPUSimBench: physical consistency between simulator and the real world, scalability for sampling efficiency, and determinism quantified by parallel and run-to-run variability. highly concurrent nature of GPU-batched execution intro￾duces subtle numerical variations, parallel synchronization artifacts, and floating-point non-determinism. Without a rig￾orous understanding of th… view at source ↗
Figure 2
Figure 2. Figure 2: Benchmark scenes used to stress parallel physics stepping: (up left) a 3×3×3 cube stack released for free fall and ground impacts, and (up right) a Franka Panda manipulator driven by random actions. Additional real￾to-sim distribution experiments, including dice-drop joint pose distribution and Gaussian ball-drop distribution setups, are omitted here due to space constraints and will be released on the pro… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution-level outcomes of the inclined-collision benchmark. For each simulator and the real-world reference, we aggregate final cube positions across Nenv parallel environments, project them to the xy plane, and visualize the resulting log-density estimate alongside a representative simulator screenshot. Border colors indicate the determinism regimes Type 1–4 defined in Table IV. x y [PITH_FULL_IMAGE… view at source ↗
Figure 4
Figure 4. Figure 4: Composite summary of scalability and distribution results. Left: free-fall benchmark throughput and GPU memory increment versus the number of parallel environments Nenv. Middle: Franka random-action benchmark throughput and GPU memory increment versus Nenv. Each simulator is evaluated up to its maximum supported parallelism Nmax env . Right: one-dimensional marginal distributions along x and y for the incl… view at source ↗
Figure 5
Figure 5. Figure 5: An overall statistics of the key features for different simulators. Larger values indicate better performance. We provide the guidelines below based on the variability regimes and the physical-alignment ranking in Table IV. Use [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 linked inside Pith

  1. [1]

    Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration,

    A. O’Neill, A. Rehmanet al., “Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration,” in IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, 2024

  2. [2]

    RT-2: vision-language-action models transfer web knowledge to robotic control,

    B. Zitkovich, T. Yuet al., “RT-2: vision-language-action models transfer web knowledge to robotic control,” inCoRL, 2023

  3. [3]

    RT-1: robotics transformer for real-world control at scale,

    A. Brohan, N. Brownet al., “RT-1: robotics transformer for real-world control at scale,” inRSS, 2023

  4. [4]

    Learning view-invariant world models for visual robotic manipulation,

    J. Pang, N. Tanget al., “Learning view-invariant world models for visual robotic manipulation,” inICLR, 2025

  5. [5]

    Object-oriented option framework for robotics manipulation in clutter,

    J. Pang, S. Yanget al., “Object-oriented option framework for robotics manipulation in clutter,” inIROS, 2023

  6. [6]

    Scaling robot learning with semantically imagined experience,

    T. Yu, T. Xiaoet al., “Scaling robot learning with semantically imagined experience,” inRSS, 2023

  7. [7]

    Data scaling laws in imitation learning for robotic manipulation,

    F. Lin, Y . Huet al., “Data scaling laws in imitation learning for robotic manipulation,” inICLR, 2025

  8. [8]

    Rebot: Scaling robot learning with real-to- sim-to-real robotic video synthesis,

    Y . Fang, Y . Yanget al., “Rebot: Scaling robot learning with real-to- sim-to-real robotic video synthesis,” inIROS, 2025

  9. [9]

    Isaac gym: High performance gpu-based physics simulation for robot learning,

    V . Makoviychuk, L. Wawrzyniaket al., “Isaac gym: High performance gpu-based physics simulation for robot learning,” inNeurIPS, 2021

  10. [10]

    Genesis: A generative and universal physics engine for robotics and beyond,

    G. Authors, “Genesis: A generative and universal physics engine for robotics and beyond,” December 2024. [Online]. Available: https://github.com/Genesis-Embodied-AI/Genesis

  11. [11]

    Residual MPC: blending reinforce- ment learning with gpu-parallelized model predictive control,

    S. H. Jeon, H. J. Leeet al., “Residual MPC: blending reinforce- ment learning with gpu-parallelized model predictive control,”arXiv preprint arXiv:2510.12717

  12. [12]

    Maniskill3: GPU parallelized robotics sim- ulation and rendering for generalizable embodied AI,

    S. Tao, F. Xianget al., “Maniskill3: GPU parallelized robotics sim- ulation and rendering for generalizable embodied AI,”arXiv preprint arXiv:2410.00425

  13. [13]

    Learning to walk in minutes using massively parallel deep reinforcement learning,

    N. Rudin, D. Hoelleret al., “Learning to walk in minutes using massively parallel deep reinforcement learning,” inCoRL, 2021

  14. [14]

    GEAR: A gpu-centric experience replay system for large reinforcement learning models,

    H. Wang, M. Sitet al., “GEAR: A gpu-centric experience replay system for large reinforcement learning models,” inICML, 2023

  15. [15]

    Faster training for robotic manipulation in GPU paral- lelized robotics simulation,

    A. Pitkevich, “Faster training for robotic manipulation in GPU paral- lelized robotics simulation,”Int. J. Softw. Sci. Comput. Intell., vol. 17, no. 1, pp. 1–24, 2025

  16. [16]

    Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning,

    H. Geng, F. Wanget al., “Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning,” inRSS, 2025

  17. [17]

    Realm: A real-to-sim validated benchmark for generalization in robotic manipulation,

    M. Sedlacek, P. Yefanovet al., “Realm: A real-to-sim validated benchmark for generalization in robotic manipulation,”arXiv preprint arXiv:2512.19562, 2025

  18. [18]

    Performance comparison of typical physics engines using robot models with multiple joints,

    Y . Liu, Q. Wanget al., “Performance comparison of typical physics engines using robot models with multiple joints,”IEEE Robot. Autom. Lett., 2023

  19. [19]

    A comparative study on physics engines for robot simulation with mechanical interaction,

    T. Yoon, J. Lee, and J. Park, “A comparative study on physics engines for robot simulation with mechanical interaction,”Sensors, 2023

  20. [20]

    Benchmarking population-based reinforcement learning across robotic tasks with gpu-accelerated sim- ulation,

    A. A. Shahid, Y . Naranget al., “Benchmarking population-based reinforcement learning across robotic tasks with gpu-accelerated sim- ulation,” inCASE, 2025

  21. [21]

    Isaac lab: A GPU-accelerated simulation framework for multi-modal robot learning,

    M. Mittal, P. Rothet al., “Isaac lab: A GPU-accelerated simulation framework for multi-modal robot learning,” 2025

  22. [22]

    Maniskill3: GPU parallelized robotics simu- lation and rendering for generalizable embodied AI,

    S. Tao, F. Xianget al., “Maniskill3: GPU parallelized robotics simu- lation and rendering for generalizable embodied AI,” inRSS, 2025

  23. [23]

    An extensible, data-oriented ar- chitecture for high-performance, many-world simulation,

    B. Shacklett, L. G. Rosenzweiget al., “An extensible, data-oriented ar- chitecture for high-performance, many-world simulation,”ACM Trans. Graph., vol. 42, no. 4, 2023

  24. [24]

    Warp: A high-performance python framework for gpu simulation and graphics,

    M. Macklin, “Warp: A high-performance python framework for gpu simulation and graphics,” https://github.com/nvidia/warp, March 2022, nVIDIA GPU Technology Conference (GTC)

  25. [25]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” inIROS. IEEE, 2012

  26. [26]

    Mujoco playground: An open-source framework for gpu-accelerated robot learning and sim-to-real transfer

    K. Zakka, B. Tabanpouret al., “Mujoco playground: An open-source framework for gpu-accelerated robot learning and sim-to-real transfer.” 2025

  27. [27]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” inIROS, vol. 3. Ieee, 2004, pp. 2149–2154

  28. [28]

    Pybullet, a python module for physics sim- ulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “Pybullet, a python module for physics sim- ulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021

  29. [29]

    JAX: composable transformations of Python+NumPy programs,

    J. Bradbury, R. Frostiget al., “JAX: composable transformations of Python+NumPy programs,” 2018. [Online]. Available: http: //github.com/jax-ml/jax

  30. [30]

    Brax – a differentiable physics engine for large scale rigid body simulation,

    C. D. Freeman, E. Freyet al., “Brax – a differentiable physics engine for large scale rigid body simulation,” inNeurIPS, 2021

  31. [31]

    Taichi: a language for high-performance computation on spatially sparse data structures,

    Y . Hu, T.-M. Liet al., “Taichi: a language for high-performance computation on spatially sparse data structures,”ACM Trans. Graph., 2019

  32. [32]

    Taccel: Scaling up vision-based tactile robotics via high-performance gpu simulation,

    Y . Li, W. Duet al., “Taccel: Scaling up vision-based tactile robotics via high-performance gpu simulation,” inNeurIPS, 2025

  33. [33]

    SAPIEN: A simulated part-based interactive environment,

    F. Xiang, Y . Qinet al., “SAPIEN: A simulated part-based interactive environment,” inCVPR, 2020

  34. [34]

    Xpbd: Position-based simulation of compliant constrained dynamics,

    M. Macklin, M. M”uller, and N. Chentanez, “Xpbd: Position-based simulation of compliant constrained dynamics,” inMIG, 2016