{"id":"6ae97c11-c265-4c8d-9d3f-0f30034abc3f","arxiv_id":"2505.15465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new synthetic-plus-real sonar dataset for underwater 3D reconstruction, plus a regression variant of ElevateNET that outperforms prior methods in simulation.","lead":"The paper introduces SEE, a large synthetic sonar dataset with a small real-world subset, built to support 3D reconstruction research in dark underwater environments. It also reports a modified neural network that reconstructs objects more accurately in simulated tests, though real-data transfer is not evaluated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central sim-to-real claim is never tested: all training and evaluation uses synthetic data, so the paper does not yet demonstrate that SEE mitigates the simulation-to-reality gap.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the paper assumes HoloOcean's simulated sonar faithfully represents the BlueView P900 in a tank, yet all experiments are run on synthetic data and the real subset is unused. This is the single most important threat to the abstract's central claim, because the dataset's stated purpose is to bridge simulation-to-reality. No amount of synthetic benchmark success can establish that bridge without real-data validation. The paper's own text in Section IV acknowledges the real subset is 'not the focus of the current evaluation,' confirming the gap. I agree with the reader's CONDITIONAL verdict: the dataset and code release are potentially useful, the synthetic experiments are coherent, and the identified weakness is closable by adding real-data evaluation. I do not think the issue warrants REJECT because the resource may still be valuable as a synthetic benchmark, and the proposed fix is straightforward. I am also not raising the lack of error bars or the limited object subset as the primary concern, since those are secondary to the untested sim-to-real premise. My concrete test—evaluating the trained model on the real SEE subset—would directly settle whether the central claim holds.","tokens_in":9591,"tokens_out":1825,"duration_ms":20674,"concrete_test":"Train ElevateNET R on the SEE synthetic partition exactly as described in Section III.B, then evaluate the trained model on the SEE real-world partition. For each of the same props, compare reconstruction accuracy against the CAD ground truth (using the same Hausdorff and RMS metrics as Table I, with ICP alignment if needed). If the real-data errors are comparable to the synthetic errors (e.g., within a factor of 2), the sim-to-real claim is supported; if real-data reconstruction degrades substantially or fails qualitatively, the central claim fails as currently stated. A secondary check would be to compute a quantitative domain gap (e.g., FID or intensity statistics) between synthetic and real polar images to document the fidelity of the simulated sensor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated purpose is to 'mitigate the gap between simulated and real-world sonar data' and to enable methods to become 'feasible for real-world underwater applications.' The load-bearing condition is that HoloOcean's simulated BlueView P900 imagery faithfully reproduces the physical tank and sonar behavior described in Section III.A. That condition is asserted but not verified anywhere in the paper. Section IV states explicitly that 'all data utilized in this study's training and evaluation processes are exclusively synthetic,' and the real-world subset, though collected in a mirrored environment, is set aside for future work. Therefore every reported advantage of ElevateNET R and ElevateNET R* is measured on synthetic images only. If the simulator's echo generation, multipath, noise, or sonar beam pattern differ materially from the real BlueView P900 in the enclosed tank, then the dataset's headline contribution—reducing the sim-to-real gap—is unsupported. This is not a disagreement with consensus; it is an internal gap between the claim and the evaluation. The paper even concedes this by deferring real-world generalization to future work. A dataset can still be useful as a synthetic benchmark, but the specific central claim of the abstract requires quantitative evidence from the real subset that the paper does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Synthetic Enclosed Echoes (SEE), a dataset consisting of 15,536 synthetic sonar images generated with the HoloOcean simulator in a virtual replica of an indoor tank, plus a smaller set of real BlueView P900 sonar images collected in the corresponding physical tank. The synthetic portion includes 40 objects, four placement scenarios, CAD-based ground-truth point clouds, and per-image metadata. The authors benchmark three existing reconstruction methods (a classical intensity-threshold method, Neusis, and ElevateNET) and propose ElevateNET R, a regression variant of ElevateNET, together with a held-out-object training scheme called ElevateNET R*. Quantitative evaluation on synthetic test data reports mean and RMS Hausdorff distances, and the paper claims that the dataset bridges the simulation-to-reality gap for underwater sonar perception.","tokens_in":9848,"tokens_out":4881,"duration_ms":42232,"significance":"If the sim-to-real claim were established, SEE would be a valuable community resource: it is large (15,536 images), publicly released with code, uses an independently developed simulator (HoloOcean), and provides ground truth in a form compatible with both ElevateNET-style and Neusis-style methods. The R* generalization test is a genuine effort beyond simple train/test splits. However, the significance as claimed in the title and abstract is conditional on evidence that the synthetic images behave like real BlueView P900 data in the same tank; the paper currently offers no such evidence, so the contribution is better described as a synthetic benchmark with a collection of unused real data.","major_comments":[{"comment":"The central claim is unsupported: Section IV states that \"all data utilized in this study's training and evaluation processes are exclusively synthetic,\" and the real-world subset is never used. The abstract and title assert that SEE bridges the simulation-to-reality gap and improves feasibility for real-world applications. Because the physical tank and BlueView P900 are described in Section III.A as the reference for the simulation, the absence of any comparison between real and simulated images (e.g., image statistics, detection of the same object, fine-tuning on real data, or domain-shift metrics) leaves the headline contribution untested. Please either add such an evaluation or revise the claims to describe SEE as a purely synthetic benchmark with a separately collected real dataset.","section":"Section IV and Abstract"},{"comment":"All reported errors are point estimates from a single run per method, with no standard deviations, confidence intervals, or significance tests. The claimed consistent superiority of ElevateNET R over ElevateNET is therefore not statistically supported; some individual rows differ by orders of magnitude, but others (e.g., Cone-2 and Cone-4) are close enough that run-to-run variability could change the ranking. Please report results over multiple training seeds and, for the classical method, a sweep over the 95% intensity threshold, so that the comparisons are not artifacts of a single configuration.","section":"Table I"},{"comment":"The text says Neusis \"failed to produce any discernible 3D reconstructions,\" yet Table I reports numeric mean and RMS Hausdorff distances for Neusis on all 12 test objects. Please clarify what these numbers represent (e.g., the best output across epochs, a partial reconstruction, or some other criterion) and, if the method truly fails, whether including these numbers in the comparison is appropriate. The stated hypothesis that Neusis fails because it does not handle multiple-object environments is plausible but is presented without supporting evidence; a diagnostic experiment or a reference to prior results would strengthen the claim.","section":"Section IV, Neusis paragraph"}],"minor_comments":[{"comment":"The sentence describing the R* split, \"The remaining data is split into 90%,\" is incomplete; please specify the full split and the number of props held out.","section":"Section III.B"},{"comment":"The text says \"Table IV summarizes the numerical results,\" but the table is labeled Table I; please correct the cross-reference.","section":"Section IV"},{"comment":"The citation markers \"[4], [5]\" appear at the end of the introduction in an odd position; please move them to the appropriate supporting sentence.","section":"Section I"},{"comment":"Please provide basic statistics for the real subset (number of images, objects, and trajectories) even if it is not used in the current evaluation; this would help readers assess its usability for future domain-adaptation work.","section":"Section III.A"},{"comment":"Capitalization of \"Vehicle\" is inconsistent (e.g., \"the Vehicle was programmed\" vs. \"the vehicle\"), and \"blueview\" is capitalized inconsistently; please unify the formatting.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the paper's title and abstract promise more than the evaluation delivers. The dataset itself is plausibly useful, and the code release is a plus. I would encourage the editor to require either a minimal real-data validation (even a qualitative image comparison or a fine-tuning experiment on the real subset) or a revised framing that explicitly limits the contribution to synthetic evaluation. A secondary concern is the single-run reporting in Table I; the authors should address this in revision regardless of which framing they choose."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a dataset contribution, and the dataset part is solid. SEE gives you 15.5k synthetic sonar images of 40 objects in four tank scenarios, with ground truth, metadata, and an expandable HoloOcean-based generator, plus a smaller real-world subset collected in a matching tank. All of that is real, publicly available, and likely useful to anyone working on sonar 3D reconstruction. The ElevateNET R regression modification is modest but clearly described, and the synthetic benchmarks show it beating ElevateNET, Neusis, and a classical method; the R* split, which excludes test objects from training, is a good check and the improvement holds. The soft spot is exactly where the reader's report lands, and it is load-bearing. The abstract promises to 'mitigate the gap between simulated and real-world sonar data' and improve 'feasibility for real-world underwater applications.' But Section IV states plainly that all training and evaluation data are exclusively synthetic, and the real-world subset is set aside for future work. So every quantitative result is a synthetic-to-synthetic benchmark. The simulator may faithfully model the BlueView P900, and the tank geometry may be right, but that is asserted, not demonstrated. If the echo generation or noise differs from the physical sonar, the headline claim fails. The paper even concedes this by deferring real generalization. That is not a minor omission; it is the central promise. A few smaller things: Table I reports point estimates without variance, so the differences between methods could be noise. The classical method's 95% intensity threshold is a free parameter. The real-world data collection is described but never compared to the synthetic output, which would have been a cheap and informative sanity check. And there are minor copyediting issues, like referencing 'Table IV' when the results are in Table I. Who gets value from this? Researchers who need a large, labeled, synthetic sonar dataset and a flexible generator for training and benchmarking. It is less useful right now to people who want evidence that models trained on SEE transfer to real sonar, because that evidence does not exist yet. My recommendation: this deserves serious peer review, not a desk reject. The resource is valuable and the gap is fixable. Send it to reviewers, but make clear that the sim-to-real claim needs either real-data evaluation or a noticeable softening of the abstract and title. As it stands, it is a strong synthetic benchmark in search of a validation story.","headline":"A genuinely useful synthetic sonar dataset and generator, but the paper's central sim-to-real claim is untested because every reported result uses only synthetic data.","tokens_in":754,"tokens_out":3025,"would_cite":true,"duration_ms":37139,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Synthetic Enclosed Echoes (SEE), a dataset of simulated and real sonar images that aims to close the gap between laboratory training and real underwater 3D reconstruction, and shows a modified ElevateNET regression…","keywords":["Synthetic Enclosed Echoes dataset","underwater sonar imaging","3D reconstruction","simulation-to-reality gap","forward-looking sonar","ElevateNET","HoloOcean","deep learning for sonar"],"falsifier":"Take the real-world SEE subset and run ElevateNET R, trained only on the synthetic images, to reconstruct the same objects; measure the mean Hausdorff distance against the real tank's ground truth. If that error is close to the synthetic error, the sim-to-real claim holds; if it jumps by an order of magnitude, the claim falls.","tokens_in":9416,"feed_emoji":"🌊","tokens_out":8216,"duration_ms":63620,"temperature":0.7,"pith_summary":"SEE (Synthetic Enclosed Echoes) is a dataset of 15,536 simulated sonar images covering 40 underwater objects in four indoor-tank scenarios, paired with a smaller set of real sonar images from the same type of tank. The paper argues this is the first comprehensive sonar dataset for 3D reconstruction that couples high-fidelity simulation with real-world data, giving learning-based methods the labeled ground truth they need. On the synthetic portion, the paper shows that a regression-based variant of the ElevateNET network, called ElevateNET R, reconstructs objects with lower mean and root-mean-square Hausdorff error than the original ElevateNET, the neural implicit method Neusis, and a classical intensity-thresholding baseline. A version trained without seeing the test objects still beats ElevateNET, which the paper takes as evidence that the learned mapping generalizes. The real-world subset is collected but deliberately not used in the evaluation, leaving the sim-to-real claim as the paper's motivation rather than a demonstrated result.","feed_headline":"Synthetic sonar dataset aims to close the sim-to-real gap","feed_subtitle":"SEE offers 15,536 labeled sonar images and a regression variant of ElevateNET that beats existing methods.","key_machinery":"The load-bearing objects are the SEE dataset and the ElevateNET R architecture. SEE is generated in HoloOcean, an Unreal Engine-based underwater robotics simulator configured to mirror a 7 m by 7 m by 5 m indoor tank and the BlueView P900 imaging sonar; ground truth is produced by simulated rangefinder arrays that emit point clouds aligned with the sonar's field of view. ElevateNET R is the ElevateNET convolutional neural network converted from an elevation-angle classifier into a regressor, so it outputs a continuous elevation map for each polar sonar image. The design is intended to let the network absorb the sonar's ambiguity, reverberation, and noise implicitly instead of modeling them analytically.","core_discovery":"The central discovery claimed is that a dataset as simple as simulated enclosed-tank sonar, with accurate CAD-based ground truth, is sufficient to improve and evaluate 3D reconstruction methods in realistic underwater scenarios. On SEE's synthetic data, ElevateNET R attains a mean Hausdorff distance as low as 0.0076 on one cone scenario, compared to 0.0343 for the original ElevateNET and 0.3230 for the classical method; in every scenario it outperforms both the classical baseline and ElevateNET. ElevateNET R*, trained after removing all images of the test props, still surpasses the original ElevateNET, indicating that the improvement is not merely memorization. The paper further claims that Neusis, a neural implicit surface method, fails on this dataset because its design assumes a single object rather than the multiple objects and tank walls present in SEE.","pith_inferences":["If ElevateNET R*'s advantage survives on the real-world subset, synthetic enclosed-tank sonar could replace much of the per-site data collection now needed for underwater inspection robots.","The paper's own numbers show the real data are collected but unused; a quick test of synthetic-trained models on the real images would settle whether SEE actually mitigates the simulation-to-reality gap.","Because the dataset includes 40 objects and four scenarios, it could double as a benchmark for domain adaptation and generalization, not just reconstruction.","The Neusis failure suggests that neural implicit surface methods need architectural changes to handle cluttered, enclosed sonar scenes, which is an opportunity for follow-up work."],"forward_implications":["Researchers can use the simulator to generate endless labeled sonar data by adding new objects and sonar configurations, reducing the cost of collecting real underwater data.","The regression-based ElevateNET R offers a stronger baseline than the original classification-based ElevateNET for sonar elevation estimation in enclosed scenes.","The four scenario families let developers test whether reconstruction methods degrade when objects are near walls or the bottom, isolating specific acoustic challenges.","Because SEE provides polar and Cartesian images plus point-cloud ground truth, it can support evaluation of other sonar perception tasks, though the paper only demonstrates reconstruction."],"supporting_citations":[{"why":"Supplies the original ElevateNET architecture and the elevation-map ground-truth representation that SEE adopts, and serves as the main comparison baseline.","marker":"[20]"},{"why":"Provides Neusis, the neural implicit surface reconstruction method used as a benchmark, and motivates SEE's dataset structure.","marker":"[25]"},{"why":"Describes HoloOcean, the underwater robotics simulator used to build the tank worlds and generate synthetic sonar.","marker":"[26]"},{"why":"Documents HoloOcean's realistic sonar simulation capabilities that the paper relies on for fidelity to the BlueView P900.","marker":"[29]"},{"why":"Defines a classical BlueView-sonar 3D reconstruction method that the paper uses as a non-learning baseline.","marker":"[15]"},{"why":"Surveys sonar deep-learning datasets and highlights the scarcity of 3D-reconstruction-oriented sonar datasets, justifying SEE.","marker":"[28]"}],"fun_headline_variants":["Synthetic sonar dataset narrows sim-to-real gap for underwater robots","SEE: Hybrid sonar dataset boosts 3D reconstruction accuracy","ElevateNET R outperforms on synthetic enclosed-tank sonar data","Simulated tank sonar with real-world subset enables better reconstruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that HoloOcean's simulated sonar images faithfully reproduce the BlueView P900 sonar's behavior in the real tank, so a model that works on the synthetic images will also work on the real ones.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic sonar dataset narrows sim-to-real gap for underwater robots","SEE: Hybrid sonar dataset boosts 3D reconstruction accuracy","ElevateNET R outperforms on synthetic enclosed-tank sonar data","Simulated tank sonar with real-world subset enables better reconstruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1716,"prompt_tokens":922,"completion_tokens":794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":719}},"tokens_in":538,"tokens_out":794,"duration_ms":6645,"temperature":1.0,"reasoning_tokens":719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:16:45.263548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the real-world SEE subset and run ElevateNET R, trained only on the synthetic images, to reconstruct the same objects; measure the mean Hausdorff distance against the real tank's ground truth. If that error is close to the synthetic error, the sim-to-real claim holds; if it jumps by an order of magnitude, the claim falls.","supporting_citations":[{"cited_title":"Elevatenet: A convolutional neural network for estimating the missing dimension in 2d underwater sonar images,","cited_arxiv_id":null,"evidence_quote":"Supplies the original ElevateNET architecture and the elevation-map ground-truth representation that SEE adopts, and serves as the main comparison baseline."},{"cited_title":"Neural implicit surface reconstruction using imaging sonar,","cited_arxiv_id":null,"evidence_quote":"Provides Neusis, the neural implicit surface reconstruction method used as a benchmark, and motivates SEE's dataset structure."},{"cited_title":"Holoocean: An underwater robotics simulator,","cited_arxiv_id":null,"evidence_quote":"Describes HoloOcean, the underwater robotics simulator used to build the tank worlds and generate synthetic sonar."},{"cited_title":"Holoocean: Realistic sonar simulation,","cited_arxiv_id":null,"evidence_quote":"Documents HoloOcean's realistic sonar simulation capabilities that the paper relies on for fidelity to the BlueView P900."},{"cited_title":"Underwater 3d reconstruction using blueview imaging sonar,","cited_arxiv_id":null,"evidence_quote":"Defines a classical BlueView-sonar 3D reconstruction method that the paper uses as a non-learning baseline."},{"cited_title":"Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges","cited_arxiv_id":"2412.11840","evidence_quote":"Surveys sonar deep-learning datasets and highlights the scarcity of 3D-reconstruction-oriented sonar datasets, justifying SEE."}],"review_version":1}