{"id":"08791da2-8242-4f74-9ef6-17a08aed6b1c","arxiv_id":"2506.07781","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SMaRCSim is a Unity-based simulation suite for maritime robotics that integrates multi-domain vehicles, mission planning, and RL training, with code released on GitHub.","lead":"SMaRCSim is a new open-source maritime robotics simulator built on Unity that combines underwater, surface, and aerial vehicles in one scene, with a command-and-control interface and support for faster-than-real-time reinforcement learning. The paper presents the tool and its design rationale, but includes no quantitative evaluation of its claimed performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline advantage, ~50xRT RL speed, is asserted without a measurement protocol, hardware specification, or workload description; this unsupported quantitative claim is load-bearing because the contribution's value for learning-based methods depends on it.","rationale":"The reader's weakest-assumption analysis identifies exactly the same load-bearing concern: the ~50xRT RL speed claim in Table I and §III-C is asserted without a measurement protocol, hardware specification, or workload description. I agree with this assessment. The paper is a software-presentation and the existence of the open-source GitHub repository is verifiable evidence for the tool's availability, which the reader correctly credits. The multi-domain, mixed-fidelity, and C2 integration claims are supported by figures and architectural descriptions, and are less consequential than the quantitative speed claim. The RL speedup is the specific differentiator over HoloOcean (2xRT) and MARUS (Partial), so an unverified number weakens the central comparative argument. The paper's own footnote that speed is 'subject to CPU resources and complexity' undercuts the categorical table entry, because no representative measurement is reported. My concrete test would settle whether the claim is reproducible on standard hardware. Since the reader's verdict is already CONDITIONAL and this concern only reinforces that condition, I do not recommend changing the verdict.","tokens_in":6163,"tokens_out":2941,"duration_ms":37681,"concrete_test":"Clone the SMaRCSim repository and run the RL experience-collection scenario used for Fig. 1c: 64 AUVs in the provided multi-agent scene. On a documented commodity GPU/CPU system, measure wall-clock time with `Time.realtimeSinceStartup` against simulation time advanced via `Time.timeScale`, and record the achieved speedup. Repeat with the same scene and hardware in HoloOcean (or Stonefish) under an equivalent RL workload. If the measured speedup for SMaRCSim falls outside a factor of two of 50xRT on the standard scene, then the `~50xRT` entry in Table I is not representative and should be replaced with a measured range plus workload/hardware specification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution of SMaRCSim, as presented in Table I and §III-C, is that it enables faster-than-real-time reinforcement learning, with the specific quantitative claim `RL feasible (Yes, ~50xRT)`. This figure is not supported by any measurement in the paper. No hardware configuration, scene complexity, number of vehicles, episode length, or Unity time-scale setting is reported, and the footnote in §III-C ('Subject to CPU resources and complexity of the particular simulation') explicitly acknowledges that speed depends on unstated factors. The claim that Unity natively supports up to 100xRT is a statement about the engine, not a measurement of SMaRCSim on a representative maritime scenario. The adjacent comparison to HoloOcean (~2xRT) and MARUS (Partial) is therefore not grounded in a common benchmark. Since the paper's stated motivation is to enable learning-based methods through computational efficiency, and the abstract and Fig. 1c highlight 64 AUVs running faster than real-time, the entire value proposition for RL depends on this unverified number. Without a reproducible benchmark, an independent user cannot determine whether SMaRCSim actually delivers the advertised speedup or whether the table entry is an optimistic estimate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SMaRCSim, a set of open-source Unity-based simulation modules developed at SMaRC for maritime robotics. The authors motivate the tool by three use cases: multi-domain mixed-fidelity simulation, integration with command-and-control (C2) software for field operations, and faster-than-real-time simulation for reinforcement learning. The main claims are that SMaRCSim supports simultaneous simulation of aerial, surface, and underwater vehicles; provides an integrated C2 interface through WARA-PS; and enables RL at roughly 50 times real-time speed. The paper includes a comparison table of existing simulators, a system description with three application scenarios, and a link to the public GitHub repository.","tokens_in":6509,"tokens_out":4001,"duration_ms":46166,"significance":"If the claims hold, SMaRCSim would fill a genuine gap: an open-source, extensible, multi-domain maritime simulator with ROS 2 support, integrated mission planning, and faster-than-real-time RL capability. The public GitHub repository and the authors' concrete field-experience background are strengths; the tool is likely to be reused by the community if the code is well documented. However, the paper's value proposition rests heavily on the quantitative 'RL feasible (Yes, ~50xRT)' claim in Table I and Section III-C, which is not supported by any measurement or benchmark data in the manuscript. The paper also lacks an evaluation section entirely, providing no verification that the described modules function as advertised. These gaps are load-bearing because the stated motivation is computational efficiency for learning-based methods. The paper is a useful software description, but it needs a reproducible performance evaluation before the central claims can be accepted.","major_comments":[{"comment":"The headline capability 'RL feasible (Yes, ~50xRT)' is asserted without any measurement. No hardware configuration, scene complexity, number of vehicles, episode length, or Unity time-scale setting is provided, and the footnote in Section III-C ('Subject to CPU resources and complexity of the particular simulation') only reinforces that the figure is context-dependent. Because the abstract and Fig. 1c advertise faster-than-real-time RL, and the Table I comparison is the main quantitative basis for the tool's value proposition, the authors must report a reproducible benchmark: for example, a fixed scene with N agents, the exact Unity and physics settings, wall-clock versus simulation time over a defined episode, and the median and spread over multiple runs. This benchmark should also be used to substantiate the '~50xRT' entry or to correct it if that is only an optimistic upper bound.","section":"Section III-C, Table I"},{"comment":"The paper contains no experiments or evaluation section. The claims of multi-domain mixed-fidelity support, C2 integration, and Real-2-Sim are supported only by static images in Fig. 1 and by anecdotal descriptions. A software paper should at least provide a minimal reproducible example, such as a launch script and a one-vehicle or multi-vehicle scenario, together with quantitative evidence that the modules run as described (e.g., vehicle trajectories, sensor messages, C2 commands). Consider adding a small demonstration scenario with measurable outputs to validate the architecture and to show that the tool is usable by others.","section":"Whole manuscript"},{"comment":"The comparison methodology for Table I is not transparent. The 'RL feasible' entries for HoloOcean ('Partial (~2xRT)') and MARUS ('Partial') are not cited or measured, and the SMaRCSim entry 'Yes (~50xRT)' appears to be inferred from Unity's native 100x real-time capability mentioned in Section III-C, not from a SMaRCSim-specific measurement. State explicitly which entries are measured by the authors, which are taken from the respective public repositories, and which are from the literature. Without this distinction, the table cannot be independently checked and the comparison is not meaningful.","section":"Table I"}],"minor_comments":[{"comment":"The GitHub link points to 'smarc-project/smarc2', but the paper's name is SMaRCSim; clarify the relationship between the repository name and the paper title, and consider adding a release tag or a DOI for versioning and reproducibility.","section":"Software Release section"},{"comment":"There is a typo: 'Inaccesible' should be 'Inaccessible'. Also, the phrase 'RL feasible (Yes, ~50xRT)' uses 'xRT' without defining the term; state that RT denotes the real-time factor (simulation time divided by wall-clock time) near the table or in the text.","section":"Table I"},{"comment":"The acronym 'LLAPI' is used without expansion; the authors should write 'Low-Level API (LLAPI)' on first use, and clarify the relationship between the MLAgents platform and the Unity engine.","section":"Section III-C"},{"comment":"The caption claims '64 AUVs running faster than real-time' but provides no quantitative speedup; give the measured factor in the caption or refer to the benchmark requested in Major Comment 1.","section":"Fig. 1c"},{"comment":"The conclusion states that SMaRCSim 'enables the investigation of learning-based methods for control through its computational efficiency'; this is stronger than the evidence supports, as no RL experiment is reported anywhere in the paper.","section":"Section IV"},{"comment":"The names 'DA VE' and 'W A VE' appear with inserted spaces in the body and in the reference titles; these should be written as 'DAVE' and 'WAVE'. The author name 'Karta ˇsev' in the byline also contains a corrupted character.","section":"References and text"},{"comment":"For the 'Latest update' row, the SMaRCSim entry is given as 2025, but the paper does not state the commit date of the repository used; specify the date of the last commit or a release version so the entry is checkable.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short system description with no evaluation section. The central quantitative claim, the ~50xRT RL speedup, is unsupported by measurement and is used as the headline feature in the abstract, Fig. 1c, and Table I. This is fixable within the scope of the paper by adding a benchmark section, but without that the paper would not meet the standards of the journal for a systems contribution. The tool itself appears to be open-source and potentially useful, so a major revision with added evaluation is the appropriate outcome. I would also suggest the editor ask the authors to clarify the exact version of the repository reviewed, since the paper lacks a version identifier."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe practical thing to know: SMaRCSim is a real open-source Unity-based maritime simulator that combines multi-domain vehicles, a command-and-control interface, and a reinforcement-learning path in one package. That combination is genuinely uncommon; the paper's Table I makes a plausible case that no single cited simulator offers all three. If you work with AUVs and ROS2, this is worth a look.\n\nWhat the paper does well: it is a clean, honest software description. The GitHub link is there, the architecture choices are explained (Unity for onboarding, prefabs for rapid modification, ROS2/MQTT for integration, WARAPS API for C2), and the authors are explicit that some claims are aspiration rather than finished results—the footnote about CPU resources and the admission that dynamic fidelity results \"remain heuristic at best\" are more candor than most software papers manage. The comparison table, while self-reported, is a useful map of the current simulator landscape.\n\nThe soft spot is the performance claim. The headline \"~50xRT\" for RL and the figure of 64 AUVs faster than real-time appear without a hardware specification, scene complexity, or measurement protocol. The footnote says speed depends on CPU and complexity, which is true but does not replace a benchmark. Since the paper's motivation for learning-based methods rests on computational efficiency, a referee should require a reproducibility section with a typical scenario, hardware, and measured timings. The comparison to HoloOcean's ~2xRT is not grounded in a common test. This is a load-bearing gap, not a cosmetic one.\n\nA minor issue: the feature table is based on GitHub repos with some entries inaccessible, so the comparison is not fully auditable. That is minor because the central artifact is the open repo itself.\n\nWho this is for: researchers in maritime robotics, especially those doing sim-to-real or multi-vehicle mission planning. It deserves a serious referee—the combination of features is genuinely useful—but the review should insist on benchmarks for the speed claims before publication.\n\nRecommendation: send it to peer review with a request for major revision, focusing on measurement of the RT claims.","headline":"A useful open-source maritime simulator contribution whose headline RL speedup needs a measurement protocol before the performance claims can be trusted.","tokens_in":6951,"tokens_out":2145,"would_cite":true,"duration_ms":26440,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents SMaRCSim, an open-source Unity-based maritime simulation suite that claims to combine multi-domain mixed-fidelity simulation, integrated command-and-control mission planning, and faster-than-real-time reinforcement…","keywords":["maritime robotics","simulation","reinforcement learning","autonomous underwater vehicles","mission planning","ROS 2","multi-domain simulation","sim-to-real"],"falsifier":"Measure the wall-clock-to-simulated-time ratio for the paper's own 64-AUV reinforcement-learning scene on the same hardware and check whether the ROS 2 connection between the command-and-control interface and a simulated vehicle stays alive; a ratio far below 50x or a broken C2 link would refute the core integration claim.","tokens_in":5970,"feed_emoji":"🌊","tokens_out":7281,"duration_ms":79604,"temperature":0.7,"pith_summary":"The paper argues that existing underwater simulators each serve a narrow slice of the maritime robotics workflow, so no single tool supports learning-based control, multi-vehicle mixed-domain missions, and field mission planning together. SMaRCSim is presented as the missing combination: a set of open-source Unity modules that simulate underwater, surface, and aerial vehicles at mixed fidelities, connect to ROS 2 and a 3D command-and-control interface, and run reinforcement learning roughly 50x faster than real time. A sympathetic reader would care because the package offers a potential single platform to carry a maritime robotics project from controller development and policy training through simulation-based field planning.","feed_headline":"Maritime robot sim claims 50x real-time RL and full-stack missions","feed_subtitle":"Open-source Unity modules put underwater, surface and aerial robots in one scene with mission planning and RL training.","key_machinery":"The carrying mechanism is the Unity game engine used as the base: its prefab system lets researchers assemble new vehicles and environments from existing objects without deep source changes, its PhysX physics backend accepts hand-computed forces on rigid bodies, and its machine-learning interface exposes the simulation loop directly to Python for reinforcement learning. On top of this, a ROS 2/MQTT bridge and an extensible multi-domain command-and-control API connect simulation to real field operations, and Unity's native support for simulation speeds up to 100x real time provides the headroom behind the ~50x reinforcement-learning speed figure.","core_discovery":"SMaRCSim's central claim is that one simulation stack can cover the full lifecycle of maritime robotics research: multi-domain scenes with aerial, surface, and underwater vehicles simulated together; mixed-fidelity setups where, say, sensors are rendered more accurately than vehicle dynamics; an integrated command-and-control mission planner that works with 3D waypoints and imported bathymetry; and a faster-than-real-time mode, reported at about 50x, for reinforcement learning through Unity's low-level Python API. The authors would state this as a solved engineering integration problem rather than a new algorithm: the same code and interfaces used in simulation connect to real vehicles through ROS 2 and MQTT, with custom physics forces for control surfaces, currents, and winds added directly to Unity's PhysX rigid bodies.","pith_inferences":["If the reported ~50x real-time speed survives a reproducible benchmark, SMaRCSim would close a gap between high-fidelity simulators and lightweight RL simulators; until then it is best read as an integration claim rather than a measured result.","The sim-to-real-to-sim loop suggests a practical workflow where learned residual dynamics from real trials are inserted back into the Unity simulation, which could make simulation-backed field planning more reliable.","A natural test is to run a standard RL control task such as station-keeping or path following in SMaRCSim and in one existing simulator on identical hardware to verify speed and learning efficiency.","Because the suite is open-source, the field could adopt it as a shared benchmark for multi-domain maritime scenarios, which existing single-domain simulators do not support."],"forward_implications":["A project can develop and test controllers, train RL policies, and plan field missions inside one toolchain, eliminating simulator-switching costs.","RL-trained underwater, surface, and aerial policies can be deployed through the same ROS 2 interfaces used in simulation, easing the transfer path.","Multi-domain operations such as an aerial vehicle recovering an AUV or coordinating with a surface vessel become testable in a single scene.","Field-collected bathymetry can be dragged into the command-and-control scene immediately, making mission planning possible at remote sites without a server connection.","Custom hydrodynamics and environmental disturbances can be added without rewriting the engine, supporting a sim-to-real-to-sim residual correction loop."],"supporting_citations":[{"why":"Stonefish is the main high-fidelity underwater baseline the paper contrasts with; it provides a ROS 1 interface and Bullet-based hydrodynamics but is said to lack ROS 2 and multi-agent speed.","marker":"[5]"},{"why":"Unity's machine-learning platform supplies the Low-Level API that gives SMaRCSim its direct Python interface for RL and supports faster-than-real-time training.","marker":"[11]"},{"why":"MARUS is the closest Unity-based maritime simulator comparison; the paper argues SMaRCSim differs by being open-source and by adding integrated command-and-control and faster RL.","marker":"[14]"},{"why":"HoloOcean is the Unreal-based simulator with RL interfaces and high-fidelity visuals used for the RL-capability comparison row.","marker":"[18]"},{"why":"ROS 2 is the communication standard the modules build on; the paper cites this design-and-architecture reference to justify the bridge between simulation and real vehicles.","marker":"[15]"},{"why":"Neptus demonstrates an existing mission-planning framework for underwater field tests, providing the planner capability that SMaRCSim's command-and-control interface builds on.","marker":"[17]"},{"why":"The champion-level drone racing work is cited as the inspiration for residual dynamic modeling with learned corrections, feeding SMaRCSim's sim-to-real-to-sim direction.","marker":"[13]"},{"why":"A prior field demonstration of a hydrobatic AUV cyber-physical system is used to argue that existing C2 software lacks flexibility for new vehicle maneuvers, a gap SMaRCSim addresses.","marker":"[3]"}],"fun_headline_variants":["Maritime sim runs RL 50x real-time, unifies robot fleets","One Unity sim: aerial, surface, underwater, 50x RL","Full-stack maritime sim: multi-domain scenes and 50x RL","Sim for underwater teams: 50x RL, Unity, ROS 2, MQTT","Open-source sim covers maritime robots from RL to field ops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument's load-bearing premise is that the reported ~50x faster-than-real-time speed and the integrated command-and-control workflows hold up in ordinary academic use, since the paper states these capabilities without giving a hardware specification, a workload description, or a measurement protocol.","fun_headline_variants_meta":{"raw":{"variants":["Maritime sim runs RL 50x real-time, unifies robot fleets","One Unity sim: aerial, surface, underwater, 50x RL","Full-stack maritime sim: multi-domain scenes and 50x RL","Sim for underwater teams: 50x RL, Unity, ROS 2, MQTT","Open-source sim covers maritime robots from RL to field ops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3595,"prompt_tokens":797,"completion_tokens":2798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":2698}},"tokens_in":413,"tokens_out":2798,"duration_ms":21175,"temperature":1.0,"reasoning_tokens":2698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:24:19.790078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the wall-clock-to-simulated-time ratio for the paper's own 64-AUV reinforcement-learning scene on the same hardware and check whether the ROS 2 connection between the command-and-control interface and a simulated vehicle stays alive; a ratio far below 50x or a broken C2 link would refute the core integration claim.","supporting_citations":[{"cited_title":"Stonefish: An Advanced Open-Source Simulation Tool Designed for Marine Robotics, With a ROS Interface — IEEE Conference Publication — IEEE Xplore","cited_arxiv_id":null,"evidence_quote":"Stonefish is the main high-fidelity underwater baseline the paper contrasts with; it provides a ROS 1 interface and Bullet-based hydrodynamics but is said to lack ROS 2 and multi-agent speed."},{"cited_title":"Benchmarking Hamiltonian Noise in the D-Wave Quantum Annealer","cited_arxiv_id":"2006.16421","evidence_quote":"Neptus demonstrates an existing mission-planning framework for underwater field tests, providing the planner capability that SMaRCSim's command-and-control interface builds on."},{"cited_title":"Champion-level drone racing using deep re- inforcement learning.Nature, pages 982–987, August","cited_arxiv_id":null,"evidence_quote":"The champion-level drone racing work is cited as the inspiration for residual dynamic modeling with learned corrections, feeding SMaRCSim's sim-to-real-to-sim direction."},{"cited_title":"A Cyber-Physical System for Hy- drobatic AUVs: System Integration and Field Demon- stration","cited_arxiv_id":null,"evidence_quote":"A prior field demonstration of a hydrobatic AUV cyber-physical system is used to argue that existing C2 software lacks flexibility for new vehicle maneuvers, a gap SMaRCSim addresses."}],"review_version":1}