REVIEW 4 major objections 5 minor 38 references
MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Adding 70,117 synthetic flood images to real training data improves flood-level detection for YOLOv10-B.
desk verdict MultiFloodSynth is a genuinely useful dataset contribution, but the evaluation protocol is too thin to confirm the claimed mAP gain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is parameter-controllable scene composition in a 3D urban simulator: lighting, camera view, flood level, wave texture, roughness, opacity, and other settings are varied to generate diverse flood scenes. Domain randomization over these parameters, combined with image-to-3D generation for vehicles and city generation for base layouts, supplies the variety that makes the synthetic data useful for training. The simulator also auto-renders nine aligned annotation types, including segmentation, depth, normal maps, and 2D/3D bounding boxes, which avoids the inconsistent manual bounding boxes the paper attributes to real flood datasets.
What would settle it
Fix the real train/test split and random seeds, then train YOLOv10-B on Dreal alone and on Dreal plus Dsynth multiple times; if the reported mAP50 gain of 58.61 versus 56.66 does not reproduce, the central improvement claim is refuted. Separately, publishing the exact Realistic Score computation would allow checking whether 93.17% (with the real set normalized to 100) is a valid, reproducible value.
Extended reading notes
Core claim
The central discovery claimed here is that a synthetic flood dataset generated with editable scene parameters can augment, rather than replace, real flood imagery for object-localized flood-level detection. MultiFloodSynth contains 70,117 images spanning five flood levels and nine paired annotation types, and the paper's main result is that training on Dreal plus Dsynth outperforms training on Dreal alone: mAP50 rises from 56.66 to 58.61 for YOLOv10-B, and mAP50-95 rises from 40.64 to 42.71. The paper also reports that synthetic images reach 93.17% of the real dataset's Realistic Score, supporting the claim of on-par realism.
Load-bearing premise
The central claim collapses if the real dataset's five flood levels, defined by the percentage of a vehicle submerged, do not mean the same thing when applied to synthetic scenes; the paper does not provide a protocol for transferring or verifying that label definition.
Editorial extensions
If this is right
- Training on Dreal plus Dsynth gives YOLOv10-B a mAP50 of 58.61 versus 56.66 for real data alone, so synthetic data acts as a useful supplement for flood-level detection.
- Synthetic-only training performs far worse than real-only training (mAP50 6.94 versus 56.66 for YOLOv10-B), so MultiFloodSynth is not a replacement for real flood imagery.
- The 70,117-image dataset with nine aligned annotation types provides consistently auto-labeled supervision for downstream tasks such as segmentation, depth estimation, and 3D box detection.
- The reported Realistic Score of 93.17% implies that, under the cited metric, the synthetic frames are visually close to real flood scenes.
Reading between the lines
- If the mAP gain is reproducible across random seeds and held-out test sets, the parameter-controlled simulator could be used to synthesize rare extreme flood levels that are scarce in real footage; the paper does not report per-level gains, so this remains an open test.
- The 93.17% realism figure depends on a metric whose formal definition and normalization are not fully specified in the paper; a reproducible protocol would be needed to treat the realism claim as settled.
- A similar controllable pipeline could plausibly extend to neighboring water-hazard domains such as storm surge or tsunami scenes, since the same wave, lighting, and camera controls apply.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents MultiFloodSynth, a synthetic urban flood dataset generated in NVIDIA Omniverse with 70,117 images across five flood-level classes and nine annotation types, including 2D/3D bounding boxes, segmentation maps, normal maps, and depth. The framework uses image-to-3D (Unique3D) for object generation, CityDreamer for city layouts, and a set of controllable parameters for urban and flood settings, with domain randomization to increase diversity. Experiments train YOLOv10-N and YOLOv10-B on real data only, synthetic data only, and their combination, reporting that mixing MultiFloodSynth with real data improves flood-level detection performance (e.g., YOLOv10-B mAP50 rises from 56.66 to 58.61 in Table 4b). Realism is assessed with a 'Realistic Score' borrowed from UrbanWorld (Shang et al. 2024), reporting 93.17% for the synthetic dataset relative to real data (Table 5).
Significance. If the dataset and its evaluation are fully validated, MultiFloodSynth would be a valuable contribution to flood hazard detection, offering a large, consistently annotated synthetic dataset that addresses a domain where real data collection is difficult and labels are often inconsistent. The multi-annotation design could support multiple downstream tasks beyond detection. However, the central empirical claims are not yet established because the evaluation protocol is under-specified and the realism metric is not reproducible from the manuscript alone. The potential is real, but the current evidence falls short of what is needed to support the headline conclusions.
major comments (4)
- [Comparison on Detection Performance (Table 4b)] The central claim that adding MultiFloodSynth to real data improves flood-level detection (mAP50 56.66 to 58.61 for YOLOv10-B) is not supported by the reported protocol: the paper does not state the real train/test split size, the number of random seeds or runs, or whether the same test split was used for all three training configurations. Given that the real dataset has only 2,000 frames (Table 2), the observed gain could easily be due to stochasticity or an accidental favorable split, so the reader cannot verify the main result.
- [Evaluation of MultiFloodSynth (Table 5)] The 'Realistic Score' is cited to Shang et al. 2024 but is not defined anywhere in this manuscript, and the normalization described only as 'We normalized the scores of Dsynth based on the scores of the real dataset' lacks the formula, the sampling protocol, and the code. Consequently, the reported 93.17% realism value cannot be independently recomputed or checked, making the 'on-par realism' claim unverifiable as presented.
- [Challenges of Real-World Flood Hazard Scenarios / Simulating Flood Wave] The real dataset's flood-level labels are defined by the percentage of a vehicle submerged (Wan et al. 2024), but the manuscript does not provide any bridging protocol to confirm that the synthetic flood-level labels correspond to the same physical definition. The large gap between synthetic-only training (mAP50 6.94 in Table 4b) and real-only training (56.66) also indicates a substantial domain shift that is never analyzed, so the interpretation of the mixed-training improvement as evidence of 'flood-level recognition' transfer is not yet justified.
- [Comparison on Detection Performance (Table 4)] The mixed-training improvement could stem from generic regularization or from simply adding more training data rather than from the specific content of MultiFloodSynth, since no comparison is made against standard data augmentation, an equal-size additional real-data baseline, or a synthetic dataset with different content. Such baselines are needed to attribute the observed mAP gains to the dataset's properties rather than to the extra training volume.
minor comments (5)
- [Abstract / Conclusions] The text contains several typos and ungrammatical phrases, e.g., 'our dataset demonstrate' (Abstract), 'alternating data requirements of real-world dataset' (Conclusions), and 'charactersize' (Conclusions), which should be corrected.
- [Table 2] The availability footnote '†Dataset will be available under the acceptance' is unclear about when and how the dataset will be released, and the Flood-Level row shows '4' for (Gao et al. 2024) with no explanation of what that number means relative to the other entries.
- [Related Works] The reference to Shang et al. 2024 appears twice in the first paragraph of Related Works, and the repeated citation should be consolidated.
- [Evaluation of MultiFloodSynth] When stating 'By randomly selecting 1K samples in each dataset, we average the score', the paper should specify whether the 1K selection was stratified by flood level or class and should report the variance of the averaged score rather than only the mean.
- [Figure 5] Figure 5 shows a point cloud in panel (h), but Table 2 does not list point clouds as an annotation type, so the annotation inventory is inconsistent between the table and the figure.
Circularity Check
No material circularity; the one minor self-citation is not load-bearing and the realism score is an external relative comparison.
full rationale
The paper's central claim is empirical: adding MultiFloodSynth to real training data improves flood-level detection mAP (Table 4). This is not derived from any fitted parameter or equation; the synthetic labels are set by simulator control parameters, and the evaluation is performed on the real test set. The 'Realistic Score' in Table 5 is borrowed from UrbanWorld (Shang et al. 2024), and normalizing the synthetic score by the real score (6.69/7.18 = 93.17%) is a relative comparison protocol rather than a definition that forces the claimed realism value. The only author self-citation is Jung et al. (2024) in the introduction, used as an example of a data-collection-intensive domain; the present framework and experiments do not rely on it. Missing experimental details, such as the real train/test split, number of runs, and the formal definition of the Realistic Score, are verifiability concerns rather than circularity. Accordingly, I find no step in which a predicted quantity reduces by construction to an input or to a load-bearing self-citation; the low score reflects only the incidental non-load-bearing self-citation.
Assumptions & free parameters
free parameters (3)
- flood_level_definition =
5 levels described as percentage of vehicle submerged
- realistic_score_threshold =
93.17 percent
- Simulator control parameters =
Not disclosed
assumptions (3)
- domain assumption The real flood-level labels in Wan et al. 2024 are reliable enough to serve as the ground-truth definition for synthetic labels.
- domain assumption The YOLOv10 metrics (mAP50, mAP50-95) computed on a single run are stable enough to compare training configurations.
- domain assumption The Realistic Score from Shang et al. 2024 is a valid measure of realism for flood imagery.
Cite this review
Pith. "Pith review of MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation." pith.science (2026). https://pith.science/paper/7YRZHDL3
@misc{pith2026250203966,
author = {Pith},
title = {Pith review of: MultiFloodSynth: Multi-Annotated Flood Synthetic Dataset Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/7YRZHDL3}},
note = {Machine review of arXiv:2502.03966}
}
read the original abstract
In this paper, we present synthetic data generation framework for flood hazard detection system. For high fidelity and quality, we characterize several real-world properties into virtual world and simulate the flood situation by controlling them. For the sake of efficiency, recent generative models in image-to-3D and urban city synthesis are leveraged to easily composite flood environments so that we avoid data bias due to the hand-crafted manner. Based on our framework, we build the flood synthetic dataset with 5 levels, dubbed MultiFloodSynth which contains rich annotation types like normal map, segmentation, 3D bounding box for a variety of downstream task. In experiments, our dataset demonstrate the enhanced performance of flood hazard detection with on-par realism compared with real dataset.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
A.; Klein, J.; Liu, D.; Pa ubicki, W.; Pirk, S.; and Michels, D
Amador Herrera, J. A.; Klein, J.; Liu, D.; Pa ubicki, W.; Pirk, S.; and Michels, D. L. 2024. Cyclogenesis: Simulating Hurricanes and Tornadoes. ACM Transactions on Graphics (TOG), 43(4): 1--16
work page 2024
-
[4]
P.; Schindler, K.; and Wegner, J
Chaudhary, P.; D’Aronco, S.; Leit \ a o, J. P.; Schindler, K.; and Wegner, J. D. 2020. Water level prediction from social media images with a multi-task ranking approach. ISPRS Journal of Photogrammetry and Remote Sensing, 167: 252--262
work page 2020
-
[5]
Delussu, R.; Putzu, L.; and Fumera, G. 2024. Synthetic Data for Video Surveillance Applications of Computer Vision: A Review. International Journal of Computer Vision, 1--37
work page 2024
-
[6]
Ebadi, S. E.; Dhakad, S.; Vishwakarma, S.; Wang, C.; Jhang, Y.-C.; Chociej, M.; Crespi, A.; Thaman, A.; and Ganguly, S. 2022. PSP-HDRI + : A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models. arXiv:2207.05025
arXiv 2022
-
[7]
Gao, K.; Yang, Z.; Gao, X.; Shao, W.; Wei, H.; and Xu, T. 2024. Measuring urban waterlogging depths from video images based on reference objects. Journal of Flood Risk Management, 17(1): e12948
work page 2024
-
[8]
J.; Gnanapragasam, D.; Golemo, F.; Herrmann, C.; Kipf, T.; Kundu, A.; Lagun, D.; Laradji, I
Greff, K.; Belletti, F.; Beyer, L.; Doersch, C.; Du, Y.; Duckworth, D.; Fleet, D. J.; Gnanapragasam, D.; Golemo, F.; Herrmann, C.; Kipf, T.; Kundu, A.; Lagun, D.; Laradji, I. H.; Liu, H.-T.; Meyer, H.; Miao, Y.; Nowrouzezahrai, D.; Oztireli, C.; Pot, E.; Radwan, N.; Rebain, D.; Sabour, S.; Sajjadi, M. S. M.; Sela, M.; Sitzmann, V.; Stone, A.; Sun, D.; Vor...
work page 2022
Show all 38 references
-
[9]
Hamza, A.; Lojo, A.; N \'u \ n ez-Marcos, A.; and Atutxa, A. 2024. Ali-AUG: Innovative Approaches to Labeled Data Augmentation using One-Step Diffusion Model. arXiv:2410.18678
2024 arXiv
-
[10]
Hao, S.; Han, W.; Jiang, T.; Li, Y.; Wu, H.; Zhong, C.; Zhou, Z.; and Tang, H. 2024. Synthetic data in AI: Challenges, applications, and ethical implications. arXiv:2401.01629
2024 arXiv
-
[11]
Hong, Z.; Hamdan, E.; Zhao, Y.; Ye, T.; Pan, H.; and Cetin, A. E. 2024. Wildfire detection via transfer learning: a survey. Signal, Image and Video Processing, 18(1): 207--214
2024
-
[12]
Hummel, M.; and van Kooten, K. 2019. Leveraging nvidia omniverse for in situ visualization. In High Performance Computing: ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019, Revised Selected Papers 34, 634--642. Springer
2019
-
[13]
Z.; Mahmood, A.; and Nandakumar, K
Islam, K.; Zaheer, M. Z.; Mahmood, A.; and Nandakumar, K. 2024. DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27621--27630
2024
-
[14]
U.; and Seo, S
Jung, Y.; Byun, S.; Kim, B.; Amin, S. U.; and Seo, S. 2024. Harnessing synthetic data for enhanced detection of Pine Wilt Disease: An image classification approach. Computers and Electronics in Agriculture, 218: 108690
2024
-
[15]
Karanjit, R.; Pally, R.; and Samadi, S. 2023. FloodIMG: flood image DataBase system. Data in brief, 48: 109164
2023
-
[16]
Khullar, D.; Sokhandan, N.; Kulkarni, N.; and Shah, Y. 2023. Synthetic data generation for scarce road scene detection scenarios
2023
-
[17]
Kim, H.-C.; Lam, H.-K.; Lee, S.-H.; and Ok, S.-Y. 2024. Early Fire Detection System by Using Automatic Synthetic Dataset Generation Model Based on Digital Twins. Applied Sciences, 14(5): 1801
2024
-
[18]
L.; Pirk, S.; and Palubicki, W
Kokosza, A.; Wrede, H.; Gonzalez Esparza, D.; Makowski, M.; Liu, D.; Michels, D. L.; Pirk, S.; and Palubicki, W. 2024. Scintilla: Simulating Combustible Vegetation for Wildfires. ACM Transactions on Graphics (TOG), 43(4): 1--21
2024
-
[19]
Lee, K.; Shin, U.; and Lee, B.-U. 2024. Learning to Control Camera Exposure via Reinforcement Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2975--2983
2024
-
[20]
Lin, C.-H.; Gao, J.; Tang, L.; Takikawa, T.; Zeng, X.; Huang, X.; Kreis, K.; Fidler, S.; Liu, M.-Y.; and Lin, T.-Y. 2023. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 300--309
2023
-
[21]
Lo, S.-W.; Wu, J.-H.; Chang, J.-Y.; Tseng, C.-H.; Lin, M.-W.; and Lin, F.-P. 2021. Deep sensing of urban waterlogging. IEEE Access, 9: 127185--127203
2021
-
[22]
L.; Singh, R.; Guo, Y.; Mazhar, H.; et al
Mittal, M.; Yu, C.; Yu, Q.; Liu, J.; Rudin, N.; Hoeller, D.; Yuan, J. L.; Singh, R.; Guo, Y.; Mazhar, H.; et al. 2023. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6): 3740--3747
2023
-
[23]
B.; and Yeasin, M
Muhammad, M. B.; and Yeasin, M. 2020. Eigen-cam: Class activation map using principal components. In 2020 international joint conference on neural networks (IJCNN), 1--7. IEEE
2020
-
[24]
Pally, R.; and Samadi, S. 2022. Application of image processing and convolutional neural networks for flood image classification and semantic segmentation. Environmental modelling & software, 148: 105285
2022
-
[25]
Rawal, P.; Sompura, M.; and Hintze, W. 2023. Synthetic data generation for bridging Sim2Real gap in a production environment. arXiv:2311.11039
2023 arXiv
-
[26]
R.; AlHaija, H
Richter, S. R.; AlHaija, H. A.; and Koltun, V. 2022. Enhancing photorealism enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 1700--1715
2022
-
[27]
C.; Kleinbeck, C.; Yang, S
Schieber, H.; Demir, K. C.; Kleinbeck, C.; Yang, S. H.; and Roth, D. 2024. Indoor synthetic data generation: A systematic review. Computer Vision and Image Understanding, 103907
2024
-
[28]
Shang, Y.; Lin, Y.; Zheng, Y.; Fan, H.; Ding, J.; Feng, J.; Chen, J.; Tian, L.; and Li, Y. 2024. UrbanWorld: An Urban World Model for 3D City Generation. arXiv:2407.11965
2024 arXiv
-
[29]
Valvano, G.; Agostino, A.; De Magistris, G.; Graziano, A.; and Veneri, G. 2024. Controllable Image Synthesis of Industrial Data using Stable Diffusion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 5354--5363
2024
-
[30]
Wan, J.; Qin, Y.; Shen, Y.; Yang, T.; Yan, X.; Zhang, S.; Yang, G.; Xue, F.; and Wang, Q. J. 2024. Automatic detection of urban flood level with YOLOv8 using flooded vehicle dataset. Journal of Hydrology, 639: 131625
2024
-
[31]
Wang, A.; Chen, H.; Liu, L.; Chen, K.; Lin, Z.; Han, J.; and Ding, G. 2024 a . Yolov10: Real-time end-to-end object detection. arXiv:2405.14458
2024 arXiv
-
[32]
Wang, Z.; Draghi, B.; Rotalinti, Y.; Lunn, D.; and Myles, P. 2024 b . High-fidelity synthetic data applications for data augmentation
2024
-
[33]
Wu, K.; Liu, F.; Cai, Z.; Yan, R.; Wang, H.; Hu, Y.; Duan, Y.; and Ma, K. 2024 a . Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image. arXiv:2405.20343
2024 arXiv
-
[34]
Wu, L.; Liu, Y.; Zhang, J.; Zhang, B.; Wang, Z.; Tong, J.; Li, M.; and Zhang, A. 2024 b . Identification of flood depth levels in urban waterlogging disaster caused by rainstorm using a CBAM-improved ResNet50. Expert Systems with Applications, 124382
2024
-
[35]
Xie, H.; Chen, Z.; Hong, F.; and Liu, Z. 2024. Citydreamer: Compositional generative model of unbounded 3d cities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9666--9675
2024
-
[36]
Zhang, S.; Zhou, M.; Wang, Y.; Luo, C.; Wang, R.; Li, Y.; Yin, X.; Zhang, Z.; and Peng, J. 2024. CityX: Controllable Procedural Content Generation for Unbounded 3D Cities. arXiv:2407.17572
2024 arXiv
-
[37]
Zhong, P.; Liu, Y.; Zheng, H.; and Zhao, J. 2024. Detection of urban flood inundation from traffic images using deep learning methods. Water Resources Management, 38(1): 287--301
2024
-
[38]
Zhu, J.; Li, S.; Liu, Y.; Huang, P.; Shan, J.; Ma, H.; and Yuan, J. 2024. ODGEN: Domain-specific Object Detection Data Generation with Diffusion Models. arXiv:2405.15199
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.