{"id":"53e50996-c9d7-4cca-b6a6-c9479ce4817f","arxiv_id":"2607.13059","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Seven GPU robot simulators were benchmarked on speed, memory, and distribution-level physics; batched GPU execution produces measurable parallel and run-to-run variability that standard throughput benchmarks miss.","lead":"GPUSimBench tests seven GPU-accelerated robot simulators on throughput, memory, and whether repeated runs of the same scene produce the same results. It finds that batched GPU execution introduces measurable parallel and run-to-run variability, and sorts simulators into four reproducibility regimes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Type 1 'Parallel Variability' is reproducible across runs, so the abstract's 'inherent non-determinism' conflates deterministic batching bias with stochasticity; only Type 3 supports true non-determinism.","rationale":"The reader's weakest assumption (sim-to-real EMD parameter transfer) is legitimate, but it challenges the physical-fidelity ranking, not the central 'non-determinism' claim. The load-bearing issue for the strongest claim is the interpretation of the variability metrics. The paper is otherwise honest and includes a limitations section; no evidence of misconduct. The empirical measurements are useful, but the abstract overgeneralizes from deterministic batching bias to 'inherent non-determinism.' Since the verdict is already CONDITIONAL and this concern is addressable by revision and release of code, no verdict change is needed. Partial agreement with the reader because parameter transfer is secondary to the reproducibility claim.","tokens_in":10438,"tokens_out":8799,"duration_ms":97583,"concrete_test":"Run Isaac Lab's inclined-collision scene at Nenv=16 with the paper's fixed seed for 10 runs, dumping per-cube xy positions at ts=5.0s; then permute environment indices (swap env IDs 3 and 7) and rerun. Check bitwise equality of final positions across runs and whether outcomes follow env ID under permutation. If cross-run positions are bitwise identical and permuting env IDs permutes outcomes, Parallel Variability is deterministic indexing bias, not non-determinism. Repeat for ManiSkill and, for comparison, Madrona (expected true non-determinism).","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table IV, Isaac Lab and ManiSkill have Parallel Variability 4.21 and 4.76 cm while Run-to-Run Variability is reported as 0.00±0.00. At the paper's precision, the aggregate outcome distribution is identical across runs while individual environments within a run diverge by centimeters. This combination is expected from a deterministic mapping between environment index and numerical outcome (e.g., fixed block/thread ordering), not from stochastic execution. The abstract's 'inherent non-determinism introduced by GPU-batched execution' is therefore not established for Type 1; Type 2 (Genesis, MJX) shows no variability, and Type 4 (Playground) shows only run-to-run variability. Only Type 3 (Madrona, MuJoCo Warp) demonstrates genuine run-to-run non-determinism. Section VI.B attributes Parallel Variability to 'different thread schedules' and 'floating-point accumulation depends on operation order,' but with run-to-run variability at the rounding floor, the thread schedule must be effectively stable across runs; calling this 'non-determinism' conflates deterministic batching bias with stochasticity. The robust conclusion is the weaker one: reproducibility must be measured per simulator and scale, not 'GPU-batched execution is inherently non-deterministic.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces GPUSimBench, a benchmark for GPU-accelerated robotic simulators, evaluating seven simulators (Isaac Lab, ManiSkill, Genesis, Madrona, MuJoCo Warp, MJX, MuJoCo Playground) along three axes: parallel scalability (throughput Eq. (1) and memory Eq. (2)), physical fidelity to a real-world inclined-plane collision experiment (EMD, Eq. (10)), and execution variability (parallel and run-to-run EMD, Eqs. (8) and (11)). The authors find large throughput and memory differences, a wide range of sim-to-real EMD values (2.52 cm to 211.60 cm), and four empirical determinism regimes based on the presence or absence of the two variability metrics. They conclude that GPU-batched execution introduces inherent non-determinism and provide simulator-selection guidelines.","tokens_in":10767,"tokens_out":6302,"duration_ms":68661,"significance":"GPUSimBench is a useful and well-scoped empirical contribution. The methodology is controlled in important ways: fixed seeds, disabled rendering, a unified timestep, matched scenes, warmup phases, and an external real-world apparatus. The variability metrics are clearly defined, and the four regimes give practitioners a practical framework for reasoning about reproducibility. The strongest and most defensible finding is that reproducibility is not guaranteed by setting a seed alone and must be measured per simulator and per scale. Explicit credit is due for the authors' care in controlling task-level randomization and for openly acknowledging the benchmark's scope limits in Section VII. If the results hold, the paper provides valuable guidance for choosing simulators in large-scale robot learning. However, the headline claim that GPU batching is 'inherently non-deterministic' goes beyond the data: only Type 3 simulators show run-to-run variability, and Type 1's parallel-only variability is more naturally interpreted as deterministic batching bias. The sim-to-real ranking also rests on a single parameter set transferred to heterogeneous contact models. Both issues are fixable by ref","major_comments":[{"comment":"The abstract and Section I claim that GPU-batched execution has 'inherent non-determinism' characterized by 'significant run-to-run and inter-environment variability.' Table IV does not support this universal claim. Isaac Lab and ManiSkill (Type 1) have Parallel Variability 4.21 and 4.76 cm, respectively, but Run-to-Run Variability 0.00±0.00; Genesis and MJX (Type 2) have both 0.00; Playground (Type 4) has run-to-run variability only. Type 1's pattern is expected from a deterministic mapping between environment index and numerical outcome (e.g., fixed thread ordering), not from stochastic execution. The robust conclusion is the weaker one: reproducibility must be measured per simulator and scale. Please revise the abstract and the 'inherent non-determinism' phrasing, and present the four regimes as observed patterns rather than evidence of universal stochasticity.","section":"Abstract; Section VI.A; Table IV"},{"comment":"The physical-alignment ranking uses one lab-identified set of material/contact parameters (Table II) for all seven simulators. Since PhysX, XPBD, Taichi, and MuJoCo implement contact and friction quite differently, the same nominal friction/restitution values do not transfer with equal meaning. The EMD values in Table IV—e.g., ManiSkill 2.52 cm vs. Madrona 211.60 cm—therefore conflate simulator fidelity with parameter-transfer mismatch. The manuscript acknowledges 'best-effort parameter matching' (Section IV.B) but the guidelines in Section VI.C still rely on this ranking. Please add a per-simulator calibration experiment or, at minimum, explicitly reframe the metric as 'sim-to-real alignment under a common, uncalibrated parameter set' and temper the corresponding recommendations.","section":"Section IV.B; Table II; Table IV"},{"comment":"The proposed mechanism for Parallel Variability is 'different thread schedules' and 'floating-point accumulation depends on operation order.' This is inconsistent with the measured Type 1 data: if thread schedules differed across runs, Run-to-Run Variability would not be 0.00±0.00. For Isaac Lab and ManiSkill, the effective execution order is stable across runs, so the nonzero Parallel Variability is a deterministic per-environment batching bias. The section should distinguish (i) stable intra-run heterogeneity (deterministic batching), (ii) unstable inter-run numerics (true non-determinism), and (iii) absence of both, and adjust the causal language accordingly.","section":"Section VI.B"}],"minor_comments":[{"comment":"The maximum-environment counts appear to be rendered as '2 11', '2 12', etc., which is likely exponent notation but is confusing as typeset. Please clarify with superscripts or a note such as '2^11'.","section":"Table III"},{"comment":"The caption mentions 'dice-drop joint pose distribution and Gaussian ball-drop distribution setups' that are 'omitted here due to space constraints.' Either include these experiments in the release or remove the dangling reference to avoid confusion about the benchmark's completeness.","section":"Fig. 2 caption"},{"comment":"The checkmarks and ✗ markers are defined only in the text. Please state in the caption that a checkmark means the corresponding variability is nonzero after rounding at the reported precision, since values like 0.00±0.00 are precision-dependent.","section":"Table IV caption"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical benchmark with a clearly specified protocol, and the issues are fixable within the manuscript's scope. The main revision is to soften the abstract's 'inherent non-determinism' claim so it matches the per-simulator evidence, and to add a caveat or calibration study for the shared-parameter sim-to-real comparison. I would not reject; the contribution is useful for the community. Please ensure the benchmark code and parameter files are released with the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you care about GPU-accelerated simulation for robot learning. The genuinely new piece is the benchmark itself: EMD-based metrics for parallel variability and run-to-run variability, applied across seven mainstream GPU simulators with a controlled protocol (fixed seeds, disabled rendering, unified timestep, same scenes). Prior comparisons focused on speed or task success; nobody had quantified distribution-level consistency across environments and runs in a unified way. The four-regime typology is a useful organizing device, and the scalability measurements are clean.\n\nThat said, the paper's own data undercut its headline claim. Type 1 simulators (Isaac Lab, ManiSkill) show non-zero Parallel Variability but Run-to-Run Variability at 0.00±0.00. If the aggregate outcome distribution is identical across runs while environments within a run diverge by centimeters, that is a deterministic mapping from environment index to numerical outcome — a stable batching bias, not stochastic execution. The abstract's phrase 'inherent non-determinism introduced by GPU-batched execution' conflates the two. Only Type 3 (Madrona, MuJoCo Warp) demonstrates genuine run-to-run non-determinism. The robust conclusion is the weaker one: reproducibility must be measured per simulator and per scale. That conclusion survives the correction, and the typology remains useful.\n\nThe sim-to-real ranking is softer. A single lab-identified parameter set (Table II) is applied across heterogeneous contact solvers; if those parameters don't transfer equally, the EMD gap between ManiSkill (2.52 cm) and Madrona (211.6 cm) reflects parameter-transfer mismatch as much as intrinsic fidelity. The authors do acknowledge 'best-effort parameter matching,' so this is a limitation rather than a hidden fatal flaw.\n\nOther gaps: no linked code/data, and all results come from one GPU (RTX 5070). The 0.00±0.00 values are treated as determinism without a statistical test; at cm precision, that's probably acceptable but should be defended. Madrona's huge EMD is attributed to its XPBD solver's friction behavior in this setup, so it shouldn't be read as a general verdict on that engine.\n\nThis is a solid empirical contribution that deserves a serious referee. The framing needs revision — the author should be pushed to distinguish deterministic batching bias from true stochasticity — and the release of artifacts should be mandatory. But the benchmark and typology are real contributions, and the paper's practical guidance is mostly sensible. I'd send it to review with a request for major revisions rather than desk reject.","headline":"A solid, useful benchmark for GPU-simulator reproducibility that overclaims in the abstract: Type 1 'variability' is deterministic batching bias, not non-determinism, but the typology and metrics are worth keeping.","tokens_in":11225,"tokens_out":2392,"would_cite":true,"duration_ms":25530,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPU-batched robot simulators are not deterministic data sources: identical setups diverge across parallel environments and repeated runs, and scaling widens the gap.","keywords":["GPU-accelerated simulation","robot learning","sim-to-real transfer","determinism","reproducibility","parallel simulation benchmark","Earth Mover's Distance","stochasticity regimes"],"falsifier":"Re-run a simulator classified as Type 2 (both variabilities absent) on two different GPU models with identical seeds; if the run-to-run Earth Mover's Distance becomes nonzero, the claim that this regime is stable across runs is a configuration artifact rather than an inherent property.","tokens_in":10388,"feed_emoji":"🎲","tokens_out":6281,"duration_ms":64157,"temperature":0.7,"pith_summary":"This paper argues that the massively parallel GPU simulators now used as data infrastructure for embodied AI carry hidden trade-offs between speed, physical fidelity, and determinism, and it builds a benchmark, GPUSimBench, to expose them. Using controlled inclined-plane collisions matched to a real apparatus, it measures how closely each of seven simulators reproduces the real final distribution, and it quantifies two kinds of drift that survive fixed seeds: parallel variability across environments in one run, and run-to-run variability across repeated runs. It finds four empirical regimes of stochasticity, from fully deterministic to doubly variable, and shows that simply scaling environment counts can compromise reproducibility. A sympathetic reader comes away with a practical ranking of simulators for sim-to-real work and a warning that batched GPU execution is not a neutral data source.","feed_headline":"GPU robot simulators are not deterministic at scale","feed_subtitle":"A new benchmark measures run-to-run and inter-environment drift in seven simulators, with a 2.52–211.6 cm sim-to-real gap.","key_machinery":"The load-bearing machinery is the benchmark's two metrics together with the real-world EMD. Parallel Variability is the average pairwise Earth Mover's Distance (a measure, in centimeters, of how much planar mass must be moved to make one environment's final cube positions match another's) between all pairs of environments in a single run; Run-to-Run Variability is the same distance between the aggregate distributions of independent runs. Crossing these two presence/absence axes yields the four stochasticity regimes, and comparing each simulator's aggregate distribution to the measured real-world distribution yields the physical-alignment ranking. These three numbers turn 'the simulator is no","core_discovery":"The central discovery is that batched execution on a GPU injects non-determinism into physics simulation even when random seeds are fixed and task-level randomization is disabled. Across seven simulators under unified conditions, GPUSimBench records each simulator's throughput, memory footprint, distribution-level agreement with real-world data (measured as Earth Mover's Distance on the planar cube positions), and two variability metrics. Four regimes emerge: some simulators show drift between parallel environments but stable aggregate distributions across runs; others show both forms of drift; one shows neither; one shows only run-to-run drift. The drift is traced to operation-order-depende","pith_inferences":["The variability metrics likely understate the effect on end-to-end robot learning, since policy gradients average over environments and runs; if environments drift, the effective sample size and gradient noise change in ways current RL pipelines do not track.","The physical-alignment ranking mixes intrinsic solver fidelity with parameter transfer; a follow-up that calibrates friction and restitution per simulator would separate engine realism from parameter mismatch, and could reorder the ranking.","A natural extension is to test whether variability grows with contact richness or simulation duration; the inclined-collision test is one contact-rich scene, while a simple cube-free-fall scene would be a lower-bound check.","The four regimes suggest a design criterion for simulator builders: report which regime a release occupies at a given batch size, so users can decide whether to average across environments or across runs."],"forward_implications":["If the central claim is right, reproducibility in GPU-batched simulation must be measured per simulator and per scale; assuming determinism from fixed seeds is unsafe.","The four regimes imply that 'deterministic' is not a single property: a simulator can be internally consistent within a run yet unstable across runs, or vice versa, so reproducibility claims need to specify which axis.","Sim-to-real contact benchmarks should expect large engine-to-engine spread in physical alignment; choices made for throughput can dominate the physical error.","For repeated-run comparisons or parameter tuning, prefer simulators with neither variability; for single-run evaluation with plausible contact behavior, other simulators may suffice.","Scaling to larger environment counts can push a simulator into a noisier regime, so scaling studies should report variability alongside throughput."],"fun_headline_variants":["GPU simulators show unpredictable drift at scale","Benchmark exposes GPU simulator non-determinism","Seven GPU sims fail reproducibility at scale","GPU sim drift quantified: 2.5–211 cm gap","Parallel GPU sims: hidden variability uncovered"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The physical-alignment ranking assumes that the single set of lab-measured material and contact parameters transfers faithfully to all seven simulators' different solvers without per-simulator calibration.","fun_headline_variants_meta":{"raw":{"variants":["GPU simulators show unpredictable drift at scale","Benchmark exposes GPU simulator non-determinism","Seven GPU sims fail reproducibility at scale","GPU sim drift quantified: 2.5–211 cm gap","Parallel GPU sims: hidden variability uncovered"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1421,"prompt_tokens":734,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":625}},"tokens_in":478,"tokens_out":687,"duration_ms":6281,"temperature":1.0,"reasoning_tokens":625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:27:12.976342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run a simulator classified as Type 2 (both variabilities absent) on two different GPU models with identical seeds; if the run-to-run Earth Mover's Distance becomes nonzero, the claim that this regime is stable across runs is a configuration artifact rather than an inherent property.","supporting_citations":[],"review_version":1}