Pith. sign in

REVIEW 3 major objections 5 minor 29 references

Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper introduces Synthetic Enclosed Echoes (SEE), a dataset of simulated and real sonar images that aims to close the gap between laboratory training and real underwater 3D reconstruction, and shows a modified ElevateNET regression…

desk verdict A genuinely useful synthetic sonar dataset and generator, but the paper's central sim-to-real claim is untested because every reported result uses only synthetic data. read the letter →

arxiv 2505.15465 v1 pith:BVJXCOUW submitted 2025-05-21 cs.RO

classification cs.RO
keywords SyntheticEnclosedEchoesdatasetunderwatersonarimaging3Dreconstructionsimulation-to-realitygapforward-lookingElevateNETHoloOceandeeplearningfor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEE (Synthetic Enclosed Echoes) is a dataset of 15,536 simulated sonar images covering 40 underwater objects in four indoor-tank scenarios, paired with a smaller set of real sonar images from the same type of tank. The paper argues this is the first comprehensive sonar dataset for 3D reconstruction that couples high-fidelity simulation with real-world data, giving learning-based methods the labeled ground truth they need. On the synthetic portion, the paper shows that a regression-based variant of the ElevateNET network, called ElevateNET R, reconstructs objects with lower mean and root-mean-square Hausdorff error than the original ElevateNET, the neural implicit method Neusis, and a classical intensity-thresholding baseline. A version trained without seeing the test objects still beats ElevateNET, which the paper takes as evidence that the learned mapping generalizes. The real-world subset is collected but deliberately not used in the evaluation, leaving the sim-to-real claim as the paper's motivation rather than a demonstrated result.

What carries the argument

The load-bearing objects are the SEE dataset and the ElevateNET R architecture. SEE is generated in HoloOcean, an Unreal Engine-based underwater robotics simulator configured to mirror a 7 m by 7 m by 5 m indoor tank and the BlueView P900 imaging sonar; ground truth is produced by simulated rangefinder arrays that emit point clouds aligned with the sonar's field of view. ElevateNET R is the ElevateNET convolutional neural network converted from an elevation-angle classifier into a regressor, so it outputs a continuous elevation map for each polar sonar image. The design is intended to let the network absorb the sonar's ambiguity, reverberation, and noise implicitly instead of modeling them analytically.

What would settle it

Take the real-world SEE subset and run ElevateNET R, trained only on the synthetic images, to reconstruct the same objects; measure the mean Hausdorff distance against the real tank's ground truth. If that error is close to the synthetic error, the sim-to-real claim holds; if it jumps by an order of magnitude, the claim falls.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a dataset as simple as simulated enclosed-tank sonar, with accurate CAD-based ground truth, is sufficient to improve and evaluate 3D reconstruction methods in realistic underwater scenarios. On SEE's synthetic data, ElevateNET R attains a mean Hausdorff distance as low as 0.0076 on one cone scenario, compared to 0.0343 for the original ElevateNET and 0.3230 for the classical method; in every scenario it outperforms both the classical baseline and ElevateNET. ElevateNET R*, trained after removing all images of the test props, still surpasses the original ElevateNET, indicating that the improvement is not merely memorization. The paper further claims that Neusis, a neural implicit surface method, fails on this dataset because its design assumes a single object rather than the multiple objects and tank walls present in SEE.

Load-bearing premise

The load-bearing premise is that HoloOcean's simulated sonar images faithfully reproduce the BlueView P900 sonar's behavior in the real tank, so a model that works on the synthetic images will also work on the real ones.

Editorial extensions

If this is right

  • Researchers can use the simulator to generate endless labeled sonar data by adding new objects and sonar configurations, reducing the cost of collecting real underwater data.
  • The regression-based ElevateNET R offers a stronger baseline than the original classification-based ElevateNET for sonar elevation estimation in enclosed scenes.
  • The four scenario families let developers test whether reconstruction methods degrade when objects are near walls or the bottom, isolating specific acoustic challenges.
  • Because SEE provides polar and Cartesian images plus point-cloud ground truth, it can support evaluation of other sonar perception tasks, though the paper only demonstrates reconstruction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ElevateNET R*'s advantage survives on the real-world subset, synthetic enclosed-tank sonar could replace much of the per-site data collection now needed for underwater inspection robots.
  • The paper's own numbers show the real data are collected but unused; a quick test of synthetic-trained models on the real images would settle whether SEE actually mitigates the simulation-to-reality gap.
  • Because the dataset includes 40 objects and four scenarios, it could double as a benchmark for domain adaptation and generalization, not just reconstruction.
  • The Neusis failure suggests that neural implicit surface methods need architectural changes to handle cluttered, enclosed sonar scenes, which is an opportunity for follow-up work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Synthetic Enclosed Echoes (SEE), a dataset consisting of 15,536 synthetic sonar images generated with the HoloOcean simulator in a virtual replica of an indoor tank, plus a smaller set of real BlueView P900 sonar images collected in the corresponding physical tank. The synthetic portion includes 40 objects, four placement scenarios, CAD-based ground-truth point clouds, and per-image metadata. The authors benchmark three existing reconstruction methods (a classical intensity-threshold method, Neusis, and ElevateNET) and propose ElevateNET R, a regression variant of ElevateNET, together with a held-out-object training scheme called ElevateNET R*. Quantitative evaluation on synthetic test data reports mean and RMS Hausdorff distances, and the paper claims that the dataset bridges the simulation-to-reality gap for underwater sonar perception.

Significance. If the sim-to-real claim were established, SEE would be a valuable community resource: it is large (15,536 images), publicly released with code, uses an independently developed simulator (HoloOcean), and provides ground truth in a form compatible with both ElevateNET-style and Neusis-style methods. The R* generalization test is a genuine effort beyond simple train/test splits. However, the significance as claimed in the title and abstract is conditional on evidence that the synthetic images behave like real BlueView P900 data in the same tank; the paper currently offers no such evidence, so the contribution is better described as a synthetic benchmark with a collection of unused real data.

major comments (3)
  1. [Section IV and Abstract] The central claim is unsupported: Section IV states that "all data utilized in this study's training and evaluation processes are exclusively synthetic," and the real-world subset is never used. The abstract and title assert that SEE bridges the simulation-to-reality gap and improves feasibility for real-world applications. Because the physical tank and BlueView P900 are described in Section III.A as the reference for the simulation, the absence of any comparison between real and simulated images (e.g., image statistics, detection of the same object, fine-tuning on real data, or domain-shift metrics) leaves the headline contribution untested. Please either add such an evaluation or revise the claims to describe SEE as a purely synthetic benchmark with a separately collected real dataset.
  2. [Table I] All reported errors are point estimates from a single run per method, with no standard deviations, confidence intervals, or significance tests. The claimed consistent superiority of ElevateNET R over ElevateNET is therefore not statistically supported; some individual rows differ by orders of magnitude, but others (e.g., Cone-2 and Cone-4) are close enough that run-to-run variability could change the ranking. Please report results over multiple training seeds and, for the classical method, a sweep over the 95% intensity threshold, so that the comparisons are not artifacts of a single configuration.
  3. [Section IV, Neusis paragraph] The text says Neusis "failed to produce any discernible 3D reconstructions," yet Table I reports numeric mean and RMS Hausdorff distances for Neusis on all 12 test objects. Please clarify what these numbers represent (e.g., the best output across epochs, a partial reconstruction, or some other criterion) and, if the method truly fails, whether including these numbers in the comparison is appropriate. The stated hypothesis that Neusis fails because it does not handle multiple-object environments is plausible but is presented without supporting evidence; a diagnostic experiment or a reference to prior results would strengthen the claim.
minor comments (5)
  1. [Section III.B] The sentence describing the R* split, "The remaining data is split into 90%," is incomplete; please specify the full split and the number of props held out.
  2. [Section IV] The text says "Table IV summarizes the numerical results," but the table is labeled Table I; please correct the cross-reference.
  3. [Section I] The citation markers "[4], [5]" appear at the end of the introduction in an odd position; please move them to the appropriate supporting sentence.
  4. [Section III.A] Please provide basic statistics for the real subset (number of images, objects, and trajectories) even if it is not used in the current evaluation; this would help readers assess its usability for future domain-adaptation work.
  5. [Throughout] Capitalization of "Vehicle" is inconsistent (e.g., "the Vehicle was programmed" vs. "the vehicle"), and "blueview" is capitalized inconsistently; please unify the formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dataset and method evaluations are externally benchmarked, and the sim-to-real gap is unverified but not circular.

full rationale

The paper's central derivation chain is a dataset construction plus an empirical method comparison, not a mathematical derivation. No fitted parameter is renamed as a prediction: ElevateNET R is a regression modification of the external ElevateNET baseline, and the comparison baselines (Neusis, classical method, ElevateNET) are all external to this paper. The R* evaluation explicitly excludes the reconstructed props from training, so the generalization claim is not forced by construction. The simulator, HoloOcean, is an independent external tool, and the paper does not invoke a self-authored uniqueness theorem or an ansatz smuggled in through self-citation. The self-citations in the introduction and references are background context and are not load-bearing for the dataset's validity or for the reported performance improvements. The paper's genuine weakness is that it never tests the central sim-to-real bridging claim, since Section IV states 'all data utilized in this study's training and evaluation processes are exclusively synthetic' and the real subset is deferred to future work; that is a correctness or external-validity gap, not circularity. No equation or construction in the paper reduces a claimed result to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central dataset claims rest primarily on simulator fidelity and on treating synthetic evaluation as evidence for real-world use. Training and reconstruction hyperparameters are hand-chosen, and no invented entities are introduced.

free parameters (3)
  • ElevateNET R training hyperparameters = learning rate 1e-6, batch size 32, 150 epochs
    Hand-selected training settings with no ablation; reported ElevateNET R errors depend on them.
  • Classical reconstruction intensity threshold = 95% of maximum intensity
    Hand-set filter that directly controls how many sonar pixels become 3D points in the classical baseline.
  • Data collection trajectory parameters = 2 m radius, 0.3 m ring spacing, 10 degree waypoint interval
    Chosen acquisition geometry determines the viewpoint coverage and total image count of the dataset.
assumptions (3)
  • domain assumption HoloOcean sonar simulation accurately represents the BlueView P900 imaging sonar in an enclosed tank.
    Invoked in Section III.A when simulations are designed to represent this sensor; if false, synthetic data cannot stand in for real sonar.
  • ad hoc to paper Synthetic reconstruction performance is a valid proxy for real-world suitability.
    The conclusion claims feasibility for real-world applications even though Section IV tests only synthetic data; no real-data validation is provided.
  • domain assumption The simulated rangefinder array provides accurate 3D ground truth.
    Ground truth is generated by rangefinders at sonar angles in Section III.A; no procedure to verify their accuracy is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data." pith.science (2026). https://pith.science/paper/BVJXCOUW

@misc{pith2026250515465,
  author       = {Pith},
  title        = {Pith review of: Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BVJXCOUW}},
  note         = {Machine review of arXiv:2505.15465}
}
read the original abstract

This paper introduces Synthetic Enclosed Echoes (SEE), a novel dataset designed to enhance robot perception and 3D reconstruction capabilities in underwater environments. SEE comprises high-fidelity synthetic sonar data, complemented by a smaller subset of real-world sonar data. To facilitate flexible data acquisition, a simulated environment has been developed, enabling the generation of additional data through modifications such as the inclusion of new structures or imaging sonar configurations. This hybrid approach leverages the advantages of synthetic data, including readily available ground truth and the ability to generate diverse datasets, while bridging the simulation-to-reality gap with real-world data acquired in a similar environment. The SEE dataset comprehensively evaluates acoustic data-based methods, including mathematics-based sonar approaches and deep learning algorithms. These techniques were employed to validate the dataset, confirming its suitability for underwater 3D reconstruction. Furthermore, this paper proposes a novel modification to a state-of-the-art algorithm, demonstrating improved performance compared to existing methods. The SEE dataset enables the evaluation of acoustic data-based methods in realistic scenarios, thereby improving their feasibility for real-world underwater applications.

Figures

Figures reproduced from arXiv: 2505.15465 by the authors.

Figure 1
Figure 1. Synthetic Enclosed Echoes: a new dataset of synthetic and real [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Indoor Tank Facilities. Simulation was employed to facilitate the development of simulated-to-real transfer strategies. This approach leverages the ease of generating highly reliable synthetic data, mini￾mizing the need for extensive real-world missions and exper￾iments, leading to significant cost reductions and accelerated development cycles. To this end, a simulated environment was created to replicate an indoor … view at source ↗
Figure 3
Figure 3. An overview of the SEE structure, exemplifying the scenarios developed and their respective collected data. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An overview of all props present in the SEE dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Proposed scenarios for the dataset and their respective way-points [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Vehicle used while collecting real data for the dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: A visual comparison between the results obtained with all tested methodologies and Ground Truth. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 28 canonical work pages

  1. [1]

    V olume change estimation of underwater structures using 2-d sonar data,

    I. P. Maurell, M. M. dos Santos, P. J. D. de Oliveira Evald, B. H. Justo, J. Arigony-Neto, A. W. Vieira, S. S. Botelho, and P. L. Drews- Jr, “V olume change estimation of underwater structures using 2-d sonar data,” IEEE Sensors Journal, vol. 22, no. 23, pp. 23 380–23 392, 2022

  2. [2]

    Optical sensors and meth- ods for underwater 3d reconstruction,

    M. Massot-Campos and G. Oliver-Codina, “Optical sensors and meth- ods for underwater 3d reconstruction,” Sensors, vol. 15, no. 12, pp. 31 525–31 557, 2015

  3. [3]

    Overview of underwater 3d reconstruction technology based on optical images,

    K. Hu, T. Wang, C. Shen, C. Weng, F. Zhou, M. Xia, and L. Weng, “Overview of underwater 3d reconstruction technology based on optical images,” Journal of Marine Science and Engineering , vol. 11, no. 5, p. 949, 2023

  4. [4]

    Transmission estimation in underwater single images,

    P. Drews-Jr, E. do Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” in IEEE ICCVw, June 2013

  5. [5]

    Underwater depth estimation and image restoration based on single images,

    P. L. J. Drews-Jr, E. R. Nascimento, S. S. C. Botelho, and M. F. M. Campos, “Underwater depth estimation and image restoration based on single images,” IEEE CG&A , vol. 36, no. 2, pp. 24–35, 2016

  6. [6]

    Un- derwater slam: Challenges, state of the art, algorithms and a new biologically-inspired approach,

    F. Guth, L. Silveira, S. Botelho, P. Drews-Jr, and P. Ballester, “Un- derwater slam: Challenges, state of the art, algorithms and a new biologically-inspired approach,” in IEEE BioRob , 2014, pp. 981–986

  7. [7]

    A probabilistic underwater localisation based on cross-view and cross-domain acoustic and aerial images,

    M. M. dos Santos, P. J. D. de Oliveira Evald, G. G. De Giacomo, P. L. J. Drews-Jr, and S. S. da Costa Botelho, “A probabilistic underwater localisation based on cross-view and cross-domain acoustic and aerial images,” Journal of Intelligent & Robotic Systems , vol. 108, no. 3, p. 34, 2023

  8. [8]

    Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images,

    M. M. Dos Santos, G. G. De Giacomo, P. L. Drews-Jr, and S. S. Botelho, “Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images,” IEEE RA-L, vol. 7, no. 2, pp. 4969–4974, 2022

Show all 29 references
  1. [9]

    Matching color aerial images and underwater sonar images using deep learning for underwater localization,

    M. Machado Dos Santos, G. G. De Giacomo, P. L. J. Drews-Jr, and S. S. C. Botelho, “Matching color aerial images and underwater sonar images using deep learning for underwater localization,” IEEE RA-L , vol. 5, no. 4, pp. 6365–6370, 2020

  2. [10]

    Underwater place recognition using forward-looking sonar images: A topological approach,

    M. M. Santos, G. B. Zaffari, P. O. Ribeiro, P. L. Drews-Jr, and S. S. Botelho, “Underwater place recognition using forward-looking sonar images: A topological approach,” Journal of Field Robotics , vol. 36, no. 2, pp. 355–369, 2019

  3. [11]

    Object classification in semi structured enviroment using forward- looking sonar,

    M. Dos Santos, P. O. Ribeiro, P. N ´u˜nez, P. Drews-Jr, and S. Botelho, “Object classification in semi structured enviroment using forward- looking sonar,” Sensors, vol. 17, no. 10, p. 2235, 2017

  4. [12]

    A theory of active sonar detection,

    J. L. Stewart and E. C. Westerfield, “A theory of active sonar detection,” Proceedings of the IRE , vol. 47, no. 5, pp. 872–881, 1959

  5. [13]

    Forward looking sonar scene matching using deep learning,

    P. O. Ribeiro, M. M. dos Santos, P. L. Drews-Jr, and S. S. Botelho, “Forward looking sonar scene matching using deep learning,” in IEEE ICMLA, 2017, pp. 574–579

  6. [14]

    Depth anything v2,

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” arXiv:2406.09414, 2024

  7. [15]

    Underwater 3d reconstruction using blueview imaging sonar,

    T. Guerneve and Y . Petillot, “Underwater 3d reconstruction using blueview imaging sonar,” in IEEE OCEANS , 2015, pp. 1–7

  8. [16]

    Contour based reconstruction of underwater structures using sonar, visual, inertial, and depth sensor,

    S. Rahman, A. Q. Li, and I. Rekleitis, “Contour based reconstruction of underwater structures using sonar, visual, inertial, and depth sensor,” in IEEE/RSJ IROS, 2019, pp. 8054–8059

  9. [17]

    Fusing concurrent orthogonal wide-aperture sonar images for dense underwater 3d re- construction,

    J. McConnell, J. D. Martin, and B. Englot, “Fusing concurrent orthogonal wide-aperture sonar images for dense underwater 3d re- construction,” in IEEE/RSJ IROS, 2020, pp. 1653–1660

  10. [18]

    High-precision underwater 3d map- ping using imaging sonar for navigation of autonomous underwater vehicle,

    B. Kim, H. Joe, and S.-C. Yu, “High-precision underwater 3d map- ping using imaging sonar for navigation of autonomous underwater vehicle,” International Journal of Control, Automation and Systems , vol. 19, no. 9, pp. 3199–3208, 2021

  11. [19]

    Underwater-sonar- image-based 3d point cloud reconstruction for high data utilization and object classification using a neural network,

    M. Sung, J. Kim, H. Cho, M. Lee, and S.-C. Yu, “Underwater-sonar- image-based 3d point cloud reconstruction for high data utilization and object classification using a neural network,” Electronics, vol. 9, no. 11, p. 1763, 2020

  12. [20]

    Elevatenet: A convolutional neural network for estimating the missing dimension in 2d underwater sonar images,

    R. DeBortoli, F. Li, and G. A. Hollinger, “Elevatenet: A convolutional neural network for estimating the missing dimension in 2d underwater sonar images,” in IEEE/RSJ IROS, 2019, pp. 8040–8047

  13. [21]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI, 2015, pp. 234–241

  14. [22]

    A novel gpu-based sonar simulator for real-time applica- tions,

    R. Cerqueira, T. Trocoli, G. Neves, S. Joyeux, J. Albiez, and L. Oliveira, “A novel gpu-based sonar simulator for real-time applica- tions,” Computers & Graphics , vol. 68, pp. 66–76, 2017

  15. [23]

    Learning pseudo front depth for 2d forward-looking sonar-based multi-view stereo,

    Y . Wang, Y . Ji, H. Tsuchiya, H. Asama, and A. Yamashita, “Learning pseudo front depth for 2d forward-looking sonar-based multi-view stereo,” in IEEE/RSJ IROS, 2022, pp. 8730–8737

  16. [24]

    Elevation angle estimation in 2d acoustic images using pseudo front view,

    Y . Wang, Y . Ji, D. Liu, H. Tsuchiya, A. Yamashita, and H. Asama, “Elevation angle estimation in 2d acoustic images using pseudo front view,” IEEE RA-L , vol. 6, no. 2, pp. 1535–1542, 2021

  17. [25]

    Neural implicit surface reconstruction using imaging sonar,

    M. Qadri, M. Kaess, and I. Gkioulekas, “Neural implicit surface reconstruction using imaging sonar,” in IEEE ICRA , 2023, pp. 1040– 1047

  18. [26]

    Holoocean: An underwater robotics simulator,

    E. Potokar, S. Ashford, M. Kaess, and J. G. Mangelson, “Holoocean: An underwater robotics simulator,” in IEEE ICRA . IEEE, 2022, pp. 3040–3046

  19. [27]

    Motion degeneracy in self-supervised learning of elevation angle estimation for 2d forward-looking sonar,

    Y . Wang, Y . Ji, C. Wu, H. Tsuchiya, H. Asama, and A. Yamashita, “Motion degeneracy in self-supervised learning of elevation angle estimation for 2d forward-looking sonar,” in IEEE/RSJ IROS , 2023, pp. 6133–6140

  20. [28]

    Sonar-based deep learning in underwater robotics: Overview, robustness and challenges,

    M. Aubard, A. Madureira, L. Teixeira, and J. Pinto, “Sonar-based deep learning in underwater robotics: Overview, robustness and challenges,” arXiv preprint arXiv:2412.11840 , 2024

  21. [29]

    Holoocean: Realistic sonar simulation,

    E. Potokar, K. Lay, K. Norman, D. Benham, T. B. Neilsen, M. Kaess, and J. G. Mangelson, “Holoocean: Realistic sonar simulation,” in IEEE/RSJ IROS, 2022, pp. 8450–8456

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.