{"id":"c33efbb2-dccd-43a0-a288-fdd8df9516d2","arxiv_id":"2505.06787","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NTNU's Marine Cybernetics Laboratory presents an open digital-physical testbed with simulators, hydrodynamic datasets, and a small-scale vessel fleet for developing and validating ship autonomy algorithms.","lead":"The authors built an open, modular testbed at NTNU that links small-scale model ships in a wave basin to simulation layers and a Unity digital twin. A four-corner test shows physical and simulated vessels tracking the same path, with the simulator reporting smaller errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run RMSE comparison cannot establish simulation fidelity; replicate trials and add a bollard-pull calibration before claiming validated simulation-to-physical transfer.","rationale":"The reader's weakest assumption and my independent assessment converge on the same point: the simulators' hydrodynamic and thrust coefficients are uncalibrated and the experimental validation is a single run without uncertainty quantification. This is the load-bearing weakness because the paper's value proposition is that researchers can develop and validate algorithms in simulation before transferring them to the physical vessel; if the simulator does not faithfully represent the physical vessel, that transfer promise is not demonstrated. The paper is honest about its limitations (missing bollard-pull test, approximate meshes, hand-set added-mass factors), and the software and datasets are openly released, which supports reproducibility. The single four-corner run is a sensible proof of concept, and the paper makes no quantitative claim that simulation matches physics beyond the RMSE numbers. My concern is not that the results are fabricated or that the testbed is useless; it is that the quantitative fidelity evidence is too thin to support the stronger wording in the abstract and conclusions about verification, validation, and seamless transition. The natural verdict is therefore CONDITIONAL, matching the reader: the testbed is a credible open infrastructure, but it should be presented as an initial demonstration pending calibration and repeated trials. My concrete test addresses exactly the two missing ingredients: repeated runs for uncertainty quantification and a bollard-pull test for thruster calibration. Agreement with the reader is 'agree' because the reader identified the same weakness in essentially the same terms.","tokens_in":9574,"tokens_out":1612,"duration_ms":14509,"concrete_test":"Repeat the four-corner stationkeeping experiment for N = 10 runs in both the reduced-order simulator and the physical basin under identical controller gains and reference trajectories, and report mean ± std of position, yaw, and velocity RMSE. Additionally, run a bollard-pull test on C/S Voyager that measures force versus commanded thrust, and compare the resulting thrust curve against the thruster model used in the simulator. If the physical and simulated RMSE distributions overlap within one standard deviation or if the bollard-pull curve deviates by more than roughly 20% from the model, the claimed quantitative fidelity is not yet established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the testbed enables simulation-to-physical transfer of MASS algorithms is supported only by one four-corner stationkeeping run per platform (Section 3.3.2, Fig. 4). The paper reports a position RMSE of 0.062 m in simulation versus 0.180 m in the basin, but these are single trajectories without repeated trials, error bars, or uncertainty quantification, so the reported difference between sim and physical cannot be separated from run-to-run variability, initial-condition sensitivity, and unmodeled basin effects. The paper's own limitations (Section 5) explicitly state that no bollard-pull test has been conducted, leaving thruster force modeling uncalibrated, and Section 3.4 states the WAMIT hull meshes are approximate half-hull meshes with geometric simplifications while the reduced-order model in Section 3.1.2 uses hand-set added-mass and damping scale factors. These are acknowledged, so the paper is not misleading, but the quantitative fidelity claim—the testbed 'facilitates verification and validation' and 'allows transitioning seamlessly'—is stronger than the evidence. A single uncalibrated demonstration can show the pipeline is operational, but it cannot establish that the simulators reproduce the physical vessel's dynamics at the claimed accuracy. The load-bearing assumption is that one good-looking run plus openly released approximate models suffices to validate the digital-physical pipeline. That assumption is under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes an open, modular digital-physical testbed at NTNU's Marine Cybernetics Laboratory for developing and validating MASS autonomy algorithms. It comprises a fleet of small-scale C/S vessels, a high-fidelity simulation environment (mcsimpy) with hydrodynamic data from WAMIT/VERES, a reduced-order simulator (shoeboxpy), a Stonefish sensor simulator, and a Unity digital twin with remote-control capabilities. The software stack is ROS 2 based and includes thrust allocation, velocity- and pose-control modules, and an EKF observer. The central demonstration is a four-corner stationkeeping test of C/S Voyager in the reduced-order simulator and in the physical basin, reporting position/yaw/velocity RMSEs of 0.062 m/0.439 deg/0.031 m/s in simulation and 0.180 m/4.295 deg/0.036 m/s in the basin. The authors claim that the testbed facilitates a seamless simulation-to-physical transition and supports verification and validation of autonomy algorithms.","tokens_in":9825,"tokens_out":5374,"duration_ms":53026,"significance":"The main strength is the infrastructure itself: the software and hydrodynamic datasets are openly released with DOIs, making this a potentially reusable community benchmark and lowering the entry barrier for maritime autonomy testing. The stability analysis of the velocity controller (Eqs. 5-8) is standard and correct. The four-corner experiment demonstrates that the integrated pipeline - observer, controller, thrust allocation, and actuator drivers - operates end to end in both simulation and the physical basin. However, the quantitative fidelity claims rest on a single uncalibrated four-corner run, and the hydrodynamic and thruster models are acknowledged to be approximate. As a functional demonstration the paper is convincing; as a validation of simulation-to-physical fidelity it is not yet supported.","major_comments":[{"comment":"Only one four-corner run is reported for each platform, with no repeated trials, error bars, or uncertainty quantification. The reported difference between simulated (0.062 m) and physical (0.180 m) position RMSE cannot be separated from run-to-run variability, initial-condition sensitivity, or unmodeled basin effects. Because the paper uses this result to claim that the testbed facilitates verification and validation and allows seamless transition, the demonstration needs either replicate runs with summary statistics or a correspondingly explicit caveat that the result is a functional check rather than a fidelity validation.","section":"3.3.2, Fig. 4"},{"comment":"The fidelity chain is not quantitatively established. Section 3.4 states that WAMIT meshes are approximate half-hull meshes with geometric simplifications; Section 3.1.2 sets added-mass and damping scale factors by hand (20% surge, 100% sway and roll); and the final paragraph of Section 5 states that no bollard-pull test has been conducted, leaving the thruster model uncalibrated. These are clearly disclosed, but they imply that the four-corner comparison cannot quantify how well the simulators reproduce the physical vessel's dynamics. The paper should add a sensitivity analysis or a calibration experiment (e.g., bollard-pull) before claiming quantitative simulation-to-physical correspondence; alternatively, the fidelity claims should be explicitly limited to qualitative or functional correspondence.","section":"3.1.2, 3.4, 5"},{"comment":"The demonstration uses only the reduced-order simulator; the high-fidelity WAMIT-based mcsimpy environment is not exercised or validated in the presented experiment. The paper's full design-and-verification-pipeline claim therefore remains unsubstantiated for the high-fidelity layer. A simulation-to-simulation comparison or a high-fidelity four-corner run would close this gap, or the claim should be scoped to the reduced-order layer.","section":"3.1.1, Fig. 4"}],"minor_comments":[{"comment":"In the text below Eq. (6), 'feedforward compansation' is a typo for 'feedforward compensation'.","section":"3.3.2"},{"comment":"The pipeline direction is described inconsistently: the abstract states that the pipeline 'starts from low-fidelity and moves up to high-fidelity' while the article text says it 'starts with high-fidelity simulation models'; please harmonize these statements.","section":"Abstract"},{"comment":"The software name appears as 'mcsimpy2' in Section 3.1.1 and as 'mcsimpy' elsewhere; please use one consistent name.","section":"3.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is primarily an infrastructure and development contribution rather than a new control-theoretic algorithm. If the journal's scope requires a novel algorithmic contribution, the editor may wish to weigh fit, but the open-release aspect and the reproducible demonstration give it archival value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an infrastructure paper, and judged as infrastructure it mostly works. The genuinely new thing is the integrated package — open simulators (mcsimpy, shoeboxpy), publicly released hydrodynamic datasets, a shared ROS 2 control stack, and a digital twin — all tied to a physical fleet in a wave basin. That combination gives the community something reusable rather than another one-off demo. Credit where due: the control laws are standard but the stability argument is correct, and the four-corner run on C/S Voyager shows the whole pipeline, from reduced-order model to physical vessel, actually executes. The paper is also honest about its limits; the missing bollard-pull test and the approximate WAMIT meshes are stated outright, not buried.\n\nSoft spots, in proportion. The main one is that the sim-to-physical fidelity claim rests on essentially one demonstration: a single four-corner run per platform, with no error bars, no repeated trials, and no uncertainty quantification. The reported 0.062 m vs 0.180 m RMSE difference could easily shift with run-to-run variability, initial conditions, or unmodeled basin effects. The paper's own language — \"transition seamlessly,\" \"facilitates verification and validation\" — overshoots what a single uncalibrated run can support. Also, the hydrodynamic coefficients come from simplified half-hull meshes and hand-set added-mass/damping scale factors; that is fine for a progress report, but it means the sim fidelity is asserted rather than demonstrated. These are not fatal flaws; they are gaps between evidence and language.\n\nWho this is for: anyone building similar lab infrastructure, or anyone wanting a ready-made benchmark for MASS algorithms, will get real value from the released artifacts. The paper itself is more of a capability statement than a scientific advance. I would want to see repeated trials, a bollard-pull calibration, and at least one external comparison before treating the testbed as a validated benchmark. But the package is open, modular, and the authors are explicit about what is missing.\n\nRecommendation: send it to peer review. It is not a breakthrough, but it is a solid, honest infrastructure contribution that the community can build on, and the weaknesses are addressable through standard experimental practice rather than conceptual error.","headline":"Open, modular testbed with genuinely useful artifacts; the quantitative fidelity claims are thinner than the language, but the package deserves a serious look.","tokens_in":10389,"tokens_out":1051,"would_cite":true,"duration_ms":12598,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a digital-physical testbed—model-scale vessels, layered simulators, sensor simulation, and a digital twin—can carry autonomous-ship algorithms from cost-effective simulation to physical basin trials, and…","keywords":["Maritime Autonomous Surface Ships (MASS)","Simulation","Testing","Validation","Model Testing","Digital Twin","Maritime Digitalization","Hydrodynamic modeling"],"falsifier":"Run a bollard-pull test on C/S Voyager and compare the measured thrust-versus-command curve with the reduced-order simulator's assumed thruster scaling; if the measured curve differs substantially, the 0.180 m physical and 0.062 m simulated RMSEs would be a coincidental match rather than a validated model.","tokens_in":9266,"feed_emoji":"⛴️","tokens_out":10334,"duration_ms":91809,"temperature":0.7,"pith_summary":"The paper describes an in-house digital-physical testbed intended to support development, testing, verification, and validation of Maritime Autonomous Surface Ship (MASS) algorithms. The testbed combines a fleet of small-scale model vessels, a laboratory wave basin, a reduced-order simulator, a high-fidelity hydrodynamic simulator, a sensor-rich robotics simulator, and a 3D digital twin, all sharing a common software interface. The central demonstration is a four-corner stationkeeping maneuver on the model vessel C/S Voyager: the physical vessel followed the reference with 0.180 m position RMSE, while the reduced-order simulator achieved 0.062 m, which the authors present as evidence that the simulation-to-physical pipeline is operational. A sympathetic reader would care because the testbed is meant to be an open, reusable benchmark platform, with hydrodynamic datasets and software released publicly, lowering the barrier for other laboratories to conduct model-scale autonomy experiments.","feed_headline":"Basin testbed puts ship autonomy through sim-to-physical trials","feed_subtitle":"A four-corner stationkeeping run in the basin and simulator tracks the same reference within 0.180 m and 0.062 m.","key_machinery":"The argument is carried by a layered simulation hierarchy connected to a shared software and hardware stack. Hydrodynamic coefficients are produced by boundary-element potential-flow analyses of approximate half-hull meshes and feed a high-fidelity simulator; a reduced-order simulator treats each vessel as a uniform rectangular prism with diagonal added-mass and damping terms (20 percent of rigid-body mass in surge, 100 percent in sway and roll) and integrates the 6DOF rigid-body equations of motion with a fourth-order Runge–Kutta step. The reduced-order simulator and the physical vessels expose the same interface, and the physical fleet runs a common onboard stack with an extended Kalman filter observer, a proportional-integral velocity controller, a proportional-derivative pose controller, a control-Lyapunov/control-barrier-function thrust allocator, and motion-capture-based pose feedback. The four-corner stationkeeping test on C/S Voyager is the concrete demonstration that a controller can be developed in the reduced-order model and executed in the basin.","core_discovery":"The paper claims that a single integrated pipeline, running from low-fidelity simulation up to physical model-scale testing, can support the full design and validation loop for autonomous ship guidance, navigation, and control. The evidence is a four-corner stationkeeping test executed with the same pose controller in both the reduced-order simulator and the physical basin vessel: the simulator produced a position RMSE of 0.062 m, a yaw RMSE of 0.439 degrees, and a velocity RMSE of 0.031 m/s, while the physical vessel produced 0.180 m, 4.295 degrees, and 0.036 m/s. On the authors' terms, this shows that an algorithm developed in the simulator can be transferred to the physical testbed with the same control stack, though the gap between simulated and physical tracking error also indicates where model fidelity remains incomplete.","pith_inferences":["The 0.062 m versus 0.180 m RMSE gap suggests the reduced-order model's thruster and wave-disturbance terms, not the control algorithm, dominate the remaining error; a bollard-pull calibration plus added wave excitation would be a direct test of that reading.","The shared interface between the reduced-order simulator and the physical vessel makes the testbed a natural benchmark for sim-to-real transfer in maritime reinforcement learning, where the same policy could be trained in fast simulation and evaluated in the basin.","Because the testbed publishes its hydrodynamic data, cross-laboratory comparisons of autonomy algorithms become possible, provided the basin conditions and motion-capture ground truth are also standardized.","Forward-speed hydrodynamic data, which the paper says it plans to add, would extend the pipeline's validity from stationkeeping to transit maneuvers, where damping and wave encounter effects are stronger."],"forward_implications":["An algorithm developed in the reduced-order simulator can be run on a physical model vessel with the same software stack, making iteration between simulation and experiment inexpensive.","Publicly released hydrodynamic datasets and open-source simulators let other laboratories reconstruct the same vessels digitally, making the testbed reproducible rather than a one-off setup.","The same pipeline can in principle be extended from model scale to semi-full-scale ferries and full-scale research vessels, reusing the architecture rather than starting from scratch.","The standardized hardware and software architecture means new model vessels can be added to the fleet with minimal re-engineering, so the testbed's capability grows incrementally."],"supporting_citations":[{"why":"Supplies the 6DOF equations of motion, added-mass and damping modeling, and the reference filter used in the simulators and pose controller.","marker":"Fossen (2021)"},{"why":"Defines the four-corner stationkeeping test that serves as the paper's demonstration of simulation-to-basin transfer.","marker":"Skjetne et al. (2017)"},{"why":"The potential-flow boundary-element analysis used to generate the hydrodynamic coefficients for the high-fidelity and reduced-order simulators.","marker":"WAMIT User's Manual, Version 7.5 (2023)"},{"why":"The simulator framework on which the high-fidelity Python simulation environment is built.","marker":"Perez et al. (2006)"},{"why":"Supplies the extended Kalman filter implementation used as the default observer in the shared control stack.","marker":"Moore et al. (2014)"},{"why":"The middleware that integrates the modular control, sensing, and estimation components across vessels.","marker":"Macenski et al. (2022)"},{"why":"The sensor-rich robotics simulator used to test perception and control algorithms before basin trials.","marker":"Cieślak (2019)"},{"why":"Establishes the milliAmpere1 ferry as the semi-full-scale platform the pipeline is intended to extend toward.","marker":"Brekke et al. (2022)"},{"why":"Documents milliAmpere2 as another target platform in the validation pipeline.","marker":"Eide et al. (2025)"}],"fun_headline_variants":["Ship autonomy testbed bridges simulator and basin trials","Digital-physical pipeline validates ship GNC algorithms","Sim-to-basin testbed for autonomous ship control","Lab basin and digital twin team up for ship autonomy","Testbed links low-fidelity sims to physical ship trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulator coefficients reflect the real vessels: they come from potential-flow analyses of approximate half-hull meshes with hand-set added-mass and damping scale factors (20 percent surge, 100 percent sway and roll), and the paper states that no bollard-pull test has been run, so the thruster force model is uncalibrated; if those coefficients are wrong, the claimed simulation-to-physical correspondence is not established.","fun_headline_variants_meta":{"raw":{"variants":["Ship autonomy testbed bridges simulator and basin trials","Digital-physical pipeline validates ship GNC algorithms","Sim-to-basin testbed for autonomous ship control","Lab basin and digital twin team up for ship autonomy","Testbed links low-fidelity sims to physical ship trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1326,"prompt_tokens":991,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":607,"tokens_out":335,"duration_ms":4137,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:32:43.476680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a bollard-pull test on C/S Voyager and compare the measured thrust-versus-command curve with the reduced-order simulator's assumed thruster scaling; if the measured curve differs substantially, the 0.180 m physical and 0.062 m simulated RMSEs would be a coincidental match rather than a validated model.","supporting_citations":[],"review_version":1}