REVIEW 4 major objections 6 minor 26 references
Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that synthetic, programmatically generated aerial images can train deep-learning models to recognize forest fires and count houses in real drone photos, providing a workaround when labeled UAV imagery is scarce or costly.
desk verdict A small, honest proof-of-concept that applies synthetic-data training to two UAV tasks, but the reported numbers are validation-set tuning results, not independent generalization evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a synthetic-to-real feedback loop. The authors generate aerial-like scenes with drawing routines, train an adapted Inception-v3 convolutional network on those scenes, test on real images, use network-visualization results to see which image characteristics drive classification, and then add those missing characteristics back into the synthetic scenes before retraining. For counting, the architecture is further adapted—no pretraining on a large generic image collection, dropout, max pooling instead of average pooling, larger initial filters and stride, and a single ReLU output—because the authors found standard filters unsuitable for counting houses.
What would settle it
Repeat the described training procedure with a fixed synthetic dataset, then evaluate the trained models on a fresh set of real aerial photos that were not used during development—for example, forest-fire photos from a different region and house images from a different country. If the fire classifier's accuracy falls well below 96% or the counting model's mean squared error rises well above 20, the reported results measure tuning to the validation set rather than successful transfer from synthetic to real data.
Extended reading notes
Core claim
The central discovery, stated in the paper's own terms, is that synthetic data can prove useful for training deep-learning models, particularly for UAV-based aerial imagery. The paper demonstrates this with two tasks: distinguishing forest from forest fire in aerial photos, and counting houses in urban aerial photos. On the fire task, synthetic training images composed of generated fire and smoke over real forest backgrounds raised validation accuracy from about 86% to about 96%. On the counting task, an adapted Inception-v3 network trained on synthetic scenes reached a mean squared error of 20 after an iterative process of adding details such as trees, grass, swimming pools, and fences. The authors explicitly caution that their models were optimized to perform well on the specific validation datasets, so how well the approach generalizes to new real-world datasets remains untested.
Load-bearing premise
The reported accuracy and error assume the real-world validation images are an independent test, but the authors state the models were optimized against those exact images through an iterative feedback loop, so the numbers may reflect tuning rather than transfer.
Editorial extensions
If this is right
- Synthetic-only training can get a drone-vision model started: 2,000 generated fire/forest images produced 96% validation accuracy on a real 100-image set, meaning scarce or dangerous fire footage need not be collected first.
- Counting tasks can be bootstrapped with synthetic scenes built from simple shapes; the paper's model counted houses with an error of roughly ±4.5 houses in images containing 0–38 houses.
- The feedback loop gives a practical recipe: train on synthetic data, validate on a small set of real images, identify missing visual details, add them to the generator, and retrain.
- Because the models were tuned to their validation sets, the reported figures are best read as upper bounds; evaluation on independent real-world datasets is required before the method's generality is known.
Reading between the lines
- The same recipe probably transfers to other aerial counting problems—vehicles, solar panels, trees, or crowds—because the house counter learned from scenes assembled from basic geometric shapes and textures; this extension is not tested in the paper.
- The fire-detection accuracy may depend mostly on color and texture cues such as flames and smoke, which are inexpensive to synthesize; one way to test this is to replace the real forest backgrounds with fully synthetic terrain and see whether accuracy collapses.
- Combining the paper's feedback loop with structured domain randomization or GAN-based image style transfer, both cited in the paper as neighbouring techniques, is the natural next step toward closing the synthetic-to-real gap.
- An error breakdown per image would likely show whether the counting model's ±4.5-house error is concentrated in dense scenes or spreads evenly; the paper reports only aggregate mean squared error.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes using procedurally generated synthetic images to train deep learning models for UAV-based aerial imagery, targeting two tasks: forest-fire classification and house counting. For fire classification, the authors train an adapted Inception-v3 on 2,000 synthetic images and report 96% classification accuracy on 100 real-world aerial photos. For house counting, they train on 10,000 synthetic images and report an MSE of 20 on 60 real-world cropped images. The paper also surveys related work on synthetic data for deep learning and discusses the iterative refinement of the synthetic datasets based on validation performance.
Significance. If the reported results were obtained on a properly independent test set, the paper would make a useful contribution to an active area: generating synthetic training data to mitigate the scarcity of labeled aerial imagery. The related-work survey in Section 2 is a helpful entry point, and the explicit discussion of limitations in Section 5 is more candid than usual. However, the paper's central claim that 'synthetic data can prove useful for training DL models, particularly related to UAV-based aerial imagery' is currently supported only by validation-tuned metrics on two small datasets, with no baselines and no independent evaluation. The contribution is therefore preliminary rather than established.
major comments (4)
- [Section 4 and Section 5, Table 2 vs. Figures 3-4] The reported performance numbers are computed on the validation datasets that were used for iterative model and synthetic-data selection, not on an independent test set. Section 5 explicitly states that 'the DL models were optimized to perform well in the specific validation datasets' and that the counting MSE was reached 'after many weeks of observations and the iterative process of adding more details to the generated synthetic dataset.' Since the synthetic dataset composition, dropout rate, filter size, stride, and the number of training images were all adjusted by looking at these validation images, the reported CA=96% and MSE=20 are optimistically biased estimates and cannot support the central claim of generalization.
- [Section 3 and Section 5, fire identification] For the fire identification scenario, the real-world test set consists of 100 Google Images photos (50 forest, 50 fire), and Section 5 reports that validation accuracy rose from about 86% to 96% after 'adding background of real forest images to the generated smoke and fire.' The paper does not describe the provenance of these real forest backgrounds. If they were drawn from the same Google Images search pool as the test images, the training data overlaps the evaluation distribution, which would make the accuracy an artifact of data leakage rather than a measure of synthetic-data generalization.
- [Section 4, counting houses] The counting result is quantitatively weak. An MSE of 20 on labels ranging from 0 to 38 houses corresponds to an RMSE of about 4.47 houses, which is a large error relative to the label range. This number is reported on a single validation set of only 60 images, with no confidence intervals, no cross-validation, and no breakdown by count. The absence of any baseline (e.g., training on real data, fine-tuning from ImageNet, or a standard counting method) makes it impossible to attribute the observed error level to the synthetic-data approach rather than to the architecture or the difficulty of the task.
- [Sections 1 and 5, overall evidence] The central claim is broad: synthetic data can prove useful for UAV-based aerial imagery. The paper supports it with only two tasks, each with a small validation set, and neither task is compared against a model trained on real data or against a published baseline. Without such comparisons, the results cannot distinguish the effect of synthetic training data from the effect of having ample training data plus a powerful CNN. The paper should either add such baselines or substantially temper the central claim to reflect that these are preliminary feasibility studies.
minor comments (6)
- [Section 3] The phrase 'Scalable Vector Machines' should be 'Support Vector Machines'.
- [Section 4 and Figures 3-4] Table 2 labels these images as 'Testing,' but Section 4 and the figure legends refer to 'validation dataset'; the terminology should be made consistent.
- [Section 5] There is a typo: 'Finallly' should be 'Finally'.
- [Section 5] The phrase 'state of art' should be written as 'state-of-the-art' for consistency.
- [Section 3] The adaptations to Inception-v3 are described qualitatively; exact hyperparameters (e.g., learning rate, batch size, optimizer, data augmentation settings) should be reported for reproducibility.
- [General] The paper does not state whether the synthetic-data generation scripts or the real-world datasets will be made publicly available; providing these would greatly increase the reproducibility of the work.
Circularity Check
The supporting 96% accuracy and MSE=20 are measured on the same validation sets used to iteratively tune both the model and the synthetic data, so they are fitted scores presented as testing results rather than independent evidence of generalization.
-
fitted input called prediction
[Section 5 (Discussion), with Table 2 and Figures 3-4; Section 3 methodology loop (Figure 1)]
"Nevertheless, we need to be cautious with these indications, because the DL models were optimized to perform well in the specific validation datasets. It is questionable (and it has not been tested) whether the DL models can produce similar results in different real-world datasets that focus on similar problems and applications [10]."
The paper's central claim that synthetic data can train DL models for UAV-based aerial imagery rests on CA=96% for fire identification and MSE=20 for house counting. Both numbers are reported on the 'validation dataset,' yet the methodology in Section 3 explicitly feeds observations from testing real-world data back into synthetic-dataset creation, and Section 5 states the counting MSE was reached 'after many weeks of observations and the iterative process of adding more details to the generated synthetic dataset,' with fire accuracy rising from about 86% to 96% after adding real forest backgrounds.
full rationale
This is not a case of self-citation circularity: the only author self-citations ([7], [10]) are background/related-work references and are not load-bearing for the reported results. The core idea of training on synthetic images and evaluating on real images is independently meaningful, and the experimental procedure is transparent enough that the paper itself acknowledges the limitation. However, the quantitative evidence for the central claim is compromised by evaluation leakage. The real-world images are called 'testing' in Table 2 but 'validation' in Section 4 and Figures 3-4, and Section 5 admits the models 'were optimized to perform well in the specific validation datasets.' Furthermore, the synthetic data were iteratively enriched (trees, grass, swimming pools, fences, real forest backgrounds) specifically to reduce error on those same validation images over many weeks. There is no separate held-out test set, no repeated cross-validation, no confidence intervals, and no baseline comparison. The reported numbers therefore measure tuning to a small validation set rather than generalization to unseen real-world imagery. That makes the support for the paper's conclusion partially circular: the prediction is, by the paper's own account, fitted to the set on which it is evaluated. Score 6 reflects partial circularity: the methodology demonstration remains real, but the headline results reduce, by construction, to validation-set optimization.
Assumptions & free parameters
free parameters (5)
- synthetic dataset composition =
trees, grass, swimming pools, fences (added iteratively)
- dropout rate =
35%
- initial convolution filter size =
7x7
- stride =
5
- number of synthetic training images =
2,000 (fire), 10,000 (counting)
assumptions (3)
- domain assumption Synthetic images rendered with PIL and OpenCV can capture enough real-world variation for transfer to real aerial photos.
- domain assumption The real-world datasets are correctly labeled, based on Google Images search labels and manual house counts.
- domain assumption The validation dataset used for tuning is representative of the deployment domain.
Cite this review
Pith. "Pith review of Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles." pith.science (2026). https://pith.science/paper/LMJ3UAUB
@misc{pith2026190806472,
author = {Pith},
title = {Pith review of: Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles},
year = {2026},
howpublished = {\url{https://pith.science/paper/LMJ3UAUB}},
note = {Machine review of arXiv:1908.06472}
}
read the original abstract
This paper describes preliminary work in the recent promising approach of generating synthetic training data for facilitating the learning procedure of deep learning (DL) models, with a focus on aerial photos produced by unmanned aerial vehicles (UAV). The general concept and methodology are described, and preliminary results are presented, based on a classification problem of fire identification in forests as well as a counting problem of estimating number of houses in urban areas. The proposed technique constitutes a new possibility for the DL community, especially related to UAV-based imagery analysis, with much potential, promising results, and unexplored ground for further research.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Amara, J., Bouaziz, B., Algergawy, A., et al.: A deep learning-based approach for banana leaf diseases classification. In: BTW (Workshops). pp. 79–88 (2017)
work page 2017
-
[2]
arXiv preprint arXiv:1605.07678 (2016)
Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 (2016)
arXiv 2016
- [3]
-
[4]
In: Proceedings of the International Conference on Agricultural Engineering, Aarhus, Denmark
Dyrmann, M., Mortensen, A.K., Midtiby, H.S., Jorgensen, R.N., et al.: Pixel-wise classification of weeds and crops in images by using a fully convolutional neural net- work. In: Proceedings of the International Conference on Agricultural Engineering, Aarhus, Denmark. pp. 26–29 (2016)
work page 2016
-
[5]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4340–4349 (2016)
work page 2016
-
[6]
In: Advances in neural information processing systems
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
2014
-
[7]
In: Machine Learning in the Environmental Sciences Workshop, in Proc
Kamilaris, A.: Simulating training data for deep learning models. In: Machine Learning in the Environmental Sciences Workshop, in Proc. of EnviroInfo 2018. Munich, Germany (September 2018)
work page 2018
- [8]
Show all 26 references
-
[9]
In: Disaster Management for Resilience and Public Safety Workshop, in Proc
Kamilaris, A., Prenafeta-Boldu, F.X.: Disaster monitoring using unmanned aerial vehicles and deep learning. In: Disaster Management for Resilience and Public Safety Workshop, in Proc. of EnviroInfo2017. Luxembourg (September 2017)
2017
-
[10]
Com- puters and Electronics in Agriculture 147, 70–90 (2018)
Kamilaris, A., Prenafeta-Boldu, F.X.: Deep learning in agriculture: A survey. Com- puters and Electronics in Agriculture 147, 70–90 (2018)
2018
-
[11]
arXiv preprint arXiv:1904.11621 (2019)
Kar, A., Prakash, A., Liu, M.Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., Fidler, S.: Meta-sim: Learning to generate synthetic datasets. arXiv preprint arXiv:1904.11621 (2019)
2019 arXiv
-
[12]
IEEE transactions on medical imaging 26(7), 1010–1016 (2007) 7 Unity
Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., Yli-Harja, O.: Com- putational framework for simulating fluorescence microscope images with cell pop- ulations. IEEE transactions on medical imaging 26(7), 1010–1016 (2007) 7 Unity. https://unity.com 10 A. Kamilaris
2007
-
[13]
In: Advances in neural information processing systems
Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in neural information processing systems. pp. 1324–1332 (2010)
2010
-
[14]
arXiv preprint arXiv:1801.01726 (2018)
Li, P., Liang, X., Jia, D., Xing, E.P.: Semantic-aware grad-gan for virtual-to-real urban scene adaption. arXiv preprint arXiv:1801.01726 (2018)
2018 arXiv
-
[15]
Distill 3(3), e10 (2018)
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., Mordv- intsev, A.: The building blocks of interpretability. Distill 3(3), e10 (2018)
2018
-
[16]
arXiv preprint arXiv:1810.10093 (2018)
Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., Birchfield, S.: Structured domain randomization: Bridging the reality gap by context-aware synthetic data. arXiv preprint arXiv:1810.10093 (2018)
2018 arXiv
-
[17]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., Torralba, A.: Virtual- home: Simulating household activities via programs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8494–8502 (2018)
2018
-
[18]
Sensors 17(4), 905 (2017)
Rahnemoonfar, M., Sheppard, C.: Deep count: fruit counting based on deep sim- ulated learning. Sensors 17(4), 905 (2017)
2017
-
[19]
In: European Conference on Computer Vision
Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: European Conference on Computer Vision. pp. 102–
-
[20]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3234–3243 (2016)
2016
-
[21]
Neural networks 61, 85–117 (2015)
Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85–117 (2015)
2015
-
[22]
Washigton: Microsoft Research (2017)
Shah, S., Dey, D., Lovett, C., Kapoor, A.: Aerial informatics and robotics platform. Washigton: Microsoft Research (2017)
2017
-
[23]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
2016
-
[24]
In: European conference on computer vision
Walach, E., Wolf, L.: Learning to count with cnn boosting. In: European conference on computer vision. pp. 660–676. Springer (2016)
2016
-
[25]
arXiv preprint arXiv:1801.02209 (2018)
Wu, Y., Wu, Y., Gkioxari, G., Tian, Y.: Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209 (2018)
2018 arXiv
-
[26]
In: Proceedings of the IEEE interna- tional conference on computer vision
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 2223–2232 (2017)
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.