REVIEW 3 major objections 5 minor 29 references
Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper introduces Synthetic Enclosed Echoes (SEE), a dataset of simulated and real sonar images that aims to close the gap between laboratory training and real underwater 3D reconstruction, and shows a modified ElevateNET regression…
desk verdict A genuinely useful synthetic sonar dataset and generator, but the paper's central sim-to-real claim is untested because every reported result uses only synthetic data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the SEE dataset and the ElevateNET R architecture. SEE is generated in HoloOcean, an Unreal Engine-based underwater robotics simulator configured to mirror a 7 m by 7 m by 5 m indoor tank and the BlueView P900 imaging sonar; ground truth is produced by simulated rangefinder arrays that emit point clouds aligned with the sonar's field of view. ElevateNET R is the ElevateNET convolutional neural network converted from an elevation-angle classifier into a regressor, so it outputs a continuous elevation map for each polar sonar image. The design is intended to let the network absorb the sonar's ambiguity, reverberation, and noise implicitly instead of modeling them analytically.
What would settle it
Take the real-world SEE subset and run ElevateNET R, trained only on the synthetic images, to reconstruct the same objects; measure the mean Hausdorff distance against the real tank's ground truth. If that error is close to the synthetic error, the sim-to-real claim holds; if it jumps by an order of magnitude, the claim falls.
Extended reading notes
Core claim
The central discovery claimed is that a dataset as simple as simulated enclosed-tank sonar, with accurate CAD-based ground truth, is sufficient to improve and evaluate 3D reconstruction methods in realistic underwater scenarios. On SEE's synthetic data, ElevateNET R attains a mean Hausdorff distance as low as 0.0076 on one cone scenario, compared to 0.0343 for the original ElevateNET and 0.3230 for the classical method; in every scenario it outperforms both the classical baseline and ElevateNET. ElevateNET R*, trained after removing all images of the test props, still surpasses the original ElevateNET, indicating that the improvement is not merely memorization. The paper further claims that Neusis, a neural implicit surface method, fails on this dataset because its design assumes a single object rather than the multiple objects and tank walls present in SEE.
Load-bearing premise
The load-bearing premise is that HoloOcean's simulated sonar images faithfully reproduce the BlueView P900 sonar's behavior in the real tank, so a model that works on the synthetic images will also work on the real ones.
Editorial extensions
If this is right
- Researchers can use the simulator to generate endless labeled sonar data by adding new objects and sonar configurations, reducing the cost of collecting real underwater data.
- The regression-based ElevateNET R offers a stronger baseline than the original classification-based ElevateNET for sonar elevation estimation in enclosed scenes.
- The four scenario families let developers test whether reconstruction methods degrade when objects are near walls or the bottom, isolating specific acoustic challenges.
- Because SEE provides polar and Cartesian images plus point-cloud ground truth, it can support evaluation of other sonar perception tasks, though the paper only demonstrates reconstruction.
Reading between the lines
- If ElevateNET R*'s advantage survives on the real-world subset, synthetic enclosed-tank sonar could replace much of the per-site data collection now needed for underwater inspection robots.
- The paper's own numbers show the real data are collected but unused; a quick test of synthetic-trained models on the real images would settle whether SEE actually mitigates the simulation-to-reality gap.
- Because the dataset includes 40 objects and four scenarios, it could double as a benchmark for domain adaptation and generalization, not just reconstruction.
- The Neusis failure suggests that neural implicit surface methods need architectural changes to handle cluttered, enclosed sonar scenes, which is an opportunity for follow-up work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Synthetic Enclosed Echoes (SEE), a dataset consisting of 15,536 synthetic sonar images generated with the HoloOcean simulator in a virtual replica of an indoor tank, plus a smaller set of real BlueView P900 sonar images collected in the corresponding physical tank. The synthetic portion includes 40 objects, four placement scenarios, CAD-based ground-truth point clouds, and per-image metadata. The authors benchmark three existing reconstruction methods (a classical intensity-threshold method, Neusis, and ElevateNET) and propose ElevateNET R, a regression variant of ElevateNET, together with a held-out-object training scheme called ElevateNET R*. Quantitative evaluation on synthetic test data reports mean and RMS Hausdorff distances, and the paper claims that the dataset bridges the simulation-to-reality gap for underwater sonar perception.
Significance. If the sim-to-real claim were established, SEE would be a valuable community resource: it is large (15,536 images), publicly released with code, uses an independently developed simulator (HoloOcean), and provides ground truth in a form compatible with both ElevateNET-style and Neusis-style methods. The R* generalization test is a genuine effort beyond simple train/test splits. However, the significance as claimed in the title and abstract is conditional on evidence that the synthetic images behave like real BlueView P900 data in the same tank; the paper currently offers no such evidence, so the contribution is better described as a synthetic benchmark with a collection of unused real data.
major comments (3)
- [Section IV and Abstract] The central claim is unsupported: Section IV states that "all data utilized in this study's training and evaluation processes are exclusively synthetic," and the real-world subset is never used. The abstract and title assert that SEE bridges the simulation-to-reality gap and improves feasibility for real-world applications. Because the physical tank and BlueView P900 are described in Section III.A as the reference for the simulation, the absence of any comparison between real and simulated images (e.g., image statistics, detection of the same object, fine-tuning on real data, or domain-shift metrics) leaves the headline contribution untested. Please either add such an evaluation or revise the claims to describe SEE as a purely synthetic benchmark with a separately collected real dataset.
- [Table I] All reported errors are point estimates from a single run per method, with no standard deviations, confidence intervals, or significance tests. The claimed consistent superiority of ElevateNET R over ElevateNET is therefore not statistically supported; some individual rows differ by orders of magnitude, but others (e.g., Cone-2 and Cone-4) are close enough that run-to-run variability could change the ranking. Please report results over multiple training seeds and, for the classical method, a sweep over the 95% intensity threshold, so that the comparisons are not artifacts of a single configuration.
- [Section IV, Neusis paragraph] The text says Neusis "failed to produce any discernible 3D reconstructions," yet Table I reports numeric mean and RMS Hausdorff distances for Neusis on all 12 test objects. Please clarify what these numbers represent (e.g., the best output across epochs, a partial reconstruction, or some other criterion) and, if the method truly fails, whether including these numbers in the comparison is appropriate. The stated hypothesis that Neusis fails because it does not handle multiple-object environments is plausible but is presented without supporting evidence; a diagnostic experiment or a reference to prior results would strengthen the claim.
minor comments (5)
- [Section III.B] The sentence describing the R* split, "The remaining data is split into 90%," is incomplete; please specify the full split and the number of props held out.
- [Section IV] The text says "Table IV summarizes the numerical results," but the table is labeled Table I; please correct the cross-reference.
- [Section I] The citation markers "[4], [5]" appear at the end of the introduction in an odd position; please move them to the appropriate supporting sentence.
- [Section III.A] Please provide basic statistics for the real subset (number of images, objects, and trajectories) even if it is not used in the current evaluation; this would help readers assess its usability for future domain-adaptation work.
- [Throughout] Capitalization of "Vehicle" is inconsistent (e.g., "the Vehicle was programmed" vs. "the vehicle"), and "blueview" is capitalized inconsistently; please unify the formatting.
Circularity Check
No circularity: the dataset and method evaluations are externally benchmarked, and the sim-to-real gap is unverified but not circular.
full rationale
The paper's central derivation chain is a dataset construction plus an empirical method comparison, not a mathematical derivation. No fitted parameter is renamed as a prediction: ElevateNET R is a regression modification of the external ElevateNET baseline, and the comparison baselines (Neusis, classical method, ElevateNET) are all external to this paper. The R* evaluation explicitly excludes the reconstructed props from training, so the generalization claim is not forced by construction. The simulator, HoloOcean, is an independent external tool, and the paper does not invoke a self-authored uniqueness theorem or an ansatz smuggled in through self-citation. The self-citations in the introduction and references are background context and are not load-bearing for the dataset's validity or for the reported performance improvements. The paper's genuine weakness is that it never tests the central sim-to-real bridging claim, since Section IV states 'all data utilized in this study's training and evaluation processes are exclusively synthetic' and the real subset is deferred to future work; that is a correctness or external-validity gap, not circularity. No equation or construction in the paper reduces a claimed result to its own inputs, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- ElevateNET R training hyperparameters =
learning rate 1e-6, batch size 32, 150 epochs
- Classical reconstruction intensity threshold =
95% of maximum intensity
- Data collection trajectory parameters =
2 m radius, 0.3 m ring spacing, 10 degree waypoint interval
assumptions (3)
- domain assumption HoloOcean sonar simulation accurately represents the BlueView P900 imaging sonar in an enclosed tank.
- ad hoc to paper Synthetic reconstruction performance is a valid proxy for real-world suitability.
- domain assumption The simulated rangefinder array provides accurate 3D ground truth.
Cite this review
Pith. "Pith review of Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data." pith.science (2026). https://pith.science/paper/BVJXCOUW
@misc{pith2026250515465,
author = {Pith},
title = {Pith review of: Synthetic Enclosed Echoes: A New Dataset to Mitigate the Gap Between Simulated and Real-World Sonar Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/BVJXCOUW}},
note = {Machine review of arXiv:2505.15465}
}
read the original abstract
This paper introduces Synthetic Enclosed Echoes (SEE), a novel dataset designed to enhance robot perception and 3D reconstruction capabilities in underwater environments. SEE comprises high-fidelity synthetic sonar data, complemented by a smaller subset of real-world sonar data. To facilitate flexible data acquisition, a simulated environment has been developed, enabling the generation of additional data through modifications such as the inclusion of new structures or imaging sonar configurations. This hybrid approach leverages the advantages of synthetic data, including readily available ground truth and the ability to generate diverse datasets, while bridging the simulation-to-reality gap with real-world data acquired in a similar environment. The SEE dataset comprehensively evaluates acoustic data-based methods, including mathematics-based sonar approaches and deep learning algorithms. These techniques were employed to validate the dataset, confirming its suitability for underwater 3D reconstruction. Furthermore, this paper proposes a novel modification to a state-of-the-art algorithm, demonstrating improved performance compared to existing methods. The SEE dataset enables the evaluation of acoustic data-based methods in realistic scenarios, thereby improving their feasibility for real-world underwater applications.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
V olume change estimation of underwater structures using 2-d sonar data,
I. P. Maurell, M. M. dos Santos, P. J. D. de Oliveira Evald, B. H. Justo, J. Arigony-Neto, A. W. Vieira, S. S. Botelho, and P. L. Drews- Jr, “V olume change estimation of underwater structures using 2-d sonar data,” IEEE Sensors Journal, vol. 22, no. 23, pp. 23 380–23 392, 2022
work page 2022
-
[2]
Optical sensors and meth- ods for underwater 3d reconstruction,
M. Massot-Campos and G. Oliver-Codina, “Optical sensors and meth- ods for underwater 3d reconstruction,” Sensors, vol. 15, no. 12, pp. 31 525–31 557, 2015
work page 2015
-
[3]
Overview of underwater 3d reconstruction technology based on optical images,
K. Hu, T. Wang, C. Shen, C. Weng, F. Zhou, M. Xia, and L. Weng, “Overview of underwater 3d reconstruction technology based on optical images,” Journal of Marine Science and Engineering , vol. 11, no. 5, p. 949, 2023
work page 2023
-
[4]
Transmission estimation in underwater single images,
P. Drews-Jr, E. do Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” in IEEE ICCVw, June 2013
work page 2013
-
[5]
Underwater depth estimation and image restoration based on single images,
P. L. J. Drews-Jr, E. R. Nascimento, S. S. C. Botelho, and M. F. M. Campos, “Underwater depth estimation and image restoration based on single images,” IEEE CG&A , vol. 36, no. 2, pp. 24–35, 2016
work page 2016
-
[6]
F. Guth, L. Silveira, S. Botelho, P. Drews-Jr, and P. Ballester, “Un- derwater slam: Challenges, state of the art, algorithms and a new biologically-inspired approach,” in IEEE BioRob , 2014, pp. 981–986
work page 2014
-
[7]
M. M. dos Santos, P. J. D. de Oliveira Evald, G. G. De Giacomo, P. L. J. Drews-Jr, and S. S. da Costa Botelho, “A probabilistic underwater localisation based on cross-view and cross-domain acoustic and aerial images,” Journal of Intelligent & Robotic Systems , vol. 108, no. 3, p. 34, 2023
work page 2023
-
[8]
M. M. Dos Santos, G. G. De Giacomo, P. L. Drews-Jr, and S. S. Botelho, “Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images,” IEEE RA-L, vol. 7, no. 2, pp. 4969–4974, 2022
work page 2022
Show all 29 references
-
[9]
Matching color aerial images and underwater sonar images using deep learning for underwater localization,
M. Machado Dos Santos, G. G. De Giacomo, P. L. J. Drews-Jr, and S. S. C. Botelho, “Matching color aerial images and underwater sonar images using deep learning for underwater localization,” IEEE RA-L , vol. 5, no. 4, pp. 6365–6370, 2020
2020
-
[10]
Underwater place recognition using forward-looking sonar images: A topological approach,
M. M. Santos, G. B. Zaffari, P. O. Ribeiro, P. L. Drews-Jr, and S. S. Botelho, “Underwater place recognition using forward-looking sonar images: A topological approach,” Journal of Field Robotics , vol. 36, no. 2, pp. 355–369, 2019
2019
-
[11]
Object classification in semi structured enviroment using forward- looking sonar,
M. Dos Santos, P. O. Ribeiro, P. N ´u˜nez, P. Drews-Jr, and S. Botelho, “Object classification in semi structured enviroment using forward- looking sonar,” Sensors, vol. 17, no. 10, p. 2235, 2017
2017
-
[12]
A theory of active sonar detection,
J. L. Stewart and E. C. Westerfield, “A theory of active sonar detection,” Proceedings of the IRE , vol. 47, no. 5, pp. 872–881, 1959
1959
-
[13]
Forward looking sonar scene matching using deep learning,
P. O. Ribeiro, M. M. dos Santos, P. L. Drews-Jr, and S. S. Botelho, “Forward looking sonar scene matching using deep learning,” in IEEE ICMLA, 2017, pp. 574–579
2017
-
[14]
Depth anything v2,
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” arXiv:2406.09414, 2024
2024 arXiv
-
[15]
Underwater 3d reconstruction using blueview imaging sonar,
T. Guerneve and Y . Petillot, “Underwater 3d reconstruction using blueview imaging sonar,” in IEEE OCEANS , 2015, pp. 1–7
2015
-
[16]
Contour based reconstruction of underwater structures using sonar, visual, inertial, and depth sensor,
S. Rahman, A. Q. Li, and I. Rekleitis, “Contour based reconstruction of underwater structures using sonar, visual, inertial, and depth sensor,” in IEEE/RSJ IROS, 2019, pp. 8054–8059
2019
-
[17]
Fusing concurrent orthogonal wide-aperture sonar images for dense underwater 3d re- construction,
J. McConnell, J. D. Martin, and B. Englot, “Fusing concurrent orthogonal wide-aperture sonar images for dense underwater 3d re- construction,” in IEEE/RSJ IROS, 2020, pp. 1653–1660
2020
-
[18]
High-precision underwater 3d map- ping using imaging sonar for navigation of autonomous underwater vehicle,
B. Kim, H. Joe, and S.-C. Yu, “High-precision underwater 3d map- ping using imaging sonar for navigation of autonomous underwater vehicle,” International Journal of Control, Automation and Systems , vol. 19, no. 9, pp. 3199–3208, 2021
2021
-
[19]
Underwater-sonar- image-based 3d point cloud reconstruction for high data utilization and object classification using a neural network,
M. Sung, J. Kim, H. Cho, M. Lee, and S.-C. Yu, “Underwater-sonar- image-based 3d point cloud reconstruction for high data utilization and object classification using a neural network,” Electronics, vol. 9, no. 11, p. 1763, 2020
2020
-
[20]
Elevatenet: A convolutional neural network for estimating the missing dimension in 2d underwater sonar images,
R. DeBortoli, F. Li, and G. A. Hollinger, “Elevatenet: A convolutional neural network for estimating the missing dimension in 2d underwater sonar images,” in IEEE/RSJ IROS, 2019, pp. 8040–8047
2019
-
[21]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI, 2015, pp. 234–241
2015
-
[22]
A novel gpu-based sonar simulator for real-time applica- tions,
R. Cerqueira, T. Trocoli, G. Neves, S. Joyeux, J. Albiez, and L. Oliveira, “A novel gpu-based sonar simulator for real-time applica- tions,” Computers & Graphics , vol. 68, pp. 66–76, 2017
2017
-
[23]
Learning pseudo front depth for 2d forward-looking sonar-based multi-view stereo,
Y . Wang, Y . Ji, H. Tsuchiya, H. Asama, and A. Yamashita, “Learning pseudo front depth for 2d forward-looking sonar-based multi-view stereo,” in IEEE/RSJ IROS, 2022, pp. 8730–8737
2022
-
[24]
Elevation angle estimation in 2d acoustic images using pseudo front view,
Y . Wang, Y . Ji, D. Liu, H. Tsuchiya, A. Yamashita, and H. Asama, “Elevation angle estimation in 2d acoustic images using pseudo front view,” IEEE RA-L , vol. 6, no. 2, pp. 1535–1542, 2021
2021
-
[25]
Neural implicit surface reconstruction using imaging sonar,
M. Qadri, M. Kaess, and I. Gkioulekas, “Neural implicit surface reconstruction using imaging sonar,” in IEEE ICRA , 2023, pp. 1040– 1047
2023
-
[26]
Holoocean: An underwater robotics simulator,
E. Potokar, S. Ashford, M. Kaess, and J. G. Mangelson, “Holoocean: An underwater robotics simulator,” in IEEE ICRA . IEEE, 2022, pp. 3040–3046
2022
-
[27]
Motion degeneracy in self-supervised learning of elevation angle estimation for 2d forward-looking sonar,
Y . Wang, Y . Ji, C. Wu, H. Tsuchiya, H. Asama, and A. Yamashita, “Motion degeneracy in self-supervised learning of elevation angle estimation for 2d forward-looking sonar,” in IEEE/RSJ IROS , 2023, pp. 6133–6140
2023
-
[28]
Sonar-based deep learning in underwater robotics: Overview, robustness and challenges,
M. Aubard, A. Madureira, L. Teixeira, and J. Pinto, “Sonar-based deep learning in underwater robotics: Overview, robustness and challenges,” arXiv preprint arXiv:2412.11840 , 2024
2024 arXiv
-
[29]
Holoocean: Realistic sonar simulation,
E. Potokar, K. Lay, K. Norman, D. Benham, T. B. Neilsen, M. Kaess, and J. G. Mangelson, “Holoocean: Realistic sonar simulation,” in IEEE/RSJ IROS, 2022, pp. 8450–8456
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.