Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that synthetic, programmatically generated aerial images can train deep-learning models to recognize forest fires and count houses in real drone photos, providing a workaround when labeled UAV imagery is scarce or costly.

desk verdict A small, honest proof-of-concept that applies synthetic-data training to two UAV tasks, but the reported numbers are validation-set tuning results, not independent generalization evidence. read the letter →

arxiv 1908.06472 v1 pith:LMJ3UAUB submitted 2019-08-18 cs.CV cs.LGeess.IV

classification cs.CVcs.LGeess.IV
keywords deeplearningsynthetictrainingdataunmannedaerialvehiclesimageryforestfiredetectionobjectcountingdomaintransferconvolutionalneuralnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports initial evidence that fully synthetic aerial images, generated by drawing simple shapes rather than captured by drones, can train deep-learning models that later analyze real UAV imagery. A fire-classification model trained on 2,000 synthetic images reached about 96% validation accuracy on 100 real aerial photos, after hybrid images mixed real forest backgrounds with generated fire and smoke. A house-counting model trained on 10,000 synthetic images reached a mean squared error of 20 on 60 real images, corresponding to roughly ±4.5 houses. The authors argue that synthetic training data is a viable route for UAV-based aerial analysis, where labeled ground-truth imagery is often expensive to collect.

What carries the argument

The load-bearing mechanism is a synthetic-to-real feedback loop. The authors generate aerial-like scenes with drawing routines, train an adapted Inception-v3 convolutional network on those scenes, test on real images, use network-visualization results to see which image characteristics drive classification, and then add those missing characteristics back into the synthetic scenes before retraining. For counting, the architecture is further adapted—no pretraining on a large generic image collection, dropout, max pooling instead of average pooling, larger initial filters and stride, and a single ReLU output—because the authors found standard filters unsuitable for counting houses.

What would settle it

Repeat the described training procedure with a fixed synthetic dataset, then evaluate the trained models on a fresh set of real aerial photos that were not used during development—for example, forest-fire photos from a different region and house images from a different country. If the fire classifier's accuracy falls well below 96% or the counting model's mean squared error rises well above 20, the reported results measure tuning to the validation set rather than successful transfer from synthetic to real data.

Watch

Extended reading notes

Core claim

The central discovery, stated in the paper's own terms, is that synthetic data can prove useful for training deep-learning models, particularly for UAV-based aerial imagery. The paper demonstrates this with two tasks: distinguishing forest from forest fire in aerial photos, and counting houses in urban aerial photos. On the fire task, synthetic training images composed of generated fire and smoke over real forest backgrounds raised validation accuracy from about 86% to about 96%. On the counting task, an adapted Inception-v3 network trained on synthetic scenes reached a mean squared error of 20 after an iterative process of adding details such as trees, grass, swimming pools, and fences. The authors explicitly caution that their models were optimized to perform well on the specific validation datasets, so how well the approach generalizes to new real-world datasets remains untested.

Load-bearing premise

The reported accuracy and error assume the real-world validation images are an independent test, but the authors state the models were optimized against those exact images through an iterative feedback loop, so the numbers may reflect tuning rather than transfer.

Editorial extensions

If this is right

  • Synthetic-only training can get a drone-vision model started: 2,000 generated fire/forest images produced 96% validation accuracy on a real 100-image set, meaning scarce or dangerous fire footage need not be collected first.
  • Counting tasks can be bootstrapped with synthetic scenes built from simple shapes; the paper's model counted houses with an error of roughly ±4.5 houses in images containing 0–38 houses.
  • The feedback loop gives a practical recipe: train on synthetic data, validate on a small set of real images, identify missing visual details, add them to the generator, and retrain.
  • Because the models were tuned to their validation sets, the reported figures are best read as upper bounds; evaluation on independent real-world datasets is required before the method's generality is known.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe probably transfers to other aerial counting problems—vehicles, solar panels, trees, or crowds—because the house counter learned from scenes assembled from basic geometric shapes and textures; this extension is not tested in the paper.
  • The fire-detection accuracy may depend mostly on color and texture cues such as flames and smoke, which are inexpensive to synthesize; one way to test this is to replace the real forest backgrounds with fully synthetic terrain and see whether accuracy collapses.
  • Combining the paper's feedback loop with structured domain randomization or GAN-based image style transfer, both cited in the paper as neighbouring techniques, is the natural next step toward closing the synthetic-to-real gap.
  • An error breakdown per image would likely show whether the counting model's ±4.5-house error is concentrated in dense scenes or spreads evenly; the paper reports only aggregate mean squared error.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using procedurally generated synthetic images to train deep learning models for UAV-based aerial imagery, targeting two tasks: forest-fire classification and house counting. For fire classification, the authors train an adapted Inception-v3 on 2,000 synthetic images and report 96% classification accuracy on 100 real-world aerial photos. For house counting, they train on 10,000 synthetic images and report an MSE of 20 on 60 real-world cropped images. The paper also surveys related work on synthetic data for deep learning and discusses the iterative refinement of the synthetic datasets based on validation performance.

Significance. If the reported results were obtained on a properly independent test set, the paper would make a useful contribution to an active area: generating synthetic training data to mitigate the scarcity of labeled aerial imagery. The related-work survey in Section 2 is a helpful entry point, and the explicit discussion of limitations in Section 5 is more candid than usual. However, the paper's central claim that 'synthetic data can prove useful for training DL models, particularly related to UAV-based aerial imagery' is currently supported only by validation-tuned metrics on two small datasets, with no baselines and no independent evaluation. The contribution is therefore preliminary rather than established.

major comments (4)
  1. [Section 4 and Section 5, Table 2 vs. Figures 3-4] The reported performance numbers are computed on the validation datasets that were used for iterative model and synthetic-data selection, not on an independent test set. Section 5 explicitly states that 'the DL models were optimized to perform well in the specific validation datasets' and that the counting MSE was reached 'after many weeks of observations and the iterative process of adding more details to the generated synthetic dataset.' Since the synthetic dataset composition, dropout rate, filter size, stride, and the number of training images were all adjusted by looking at these validation images, the reported CA=96% and MSE=20 are optimistically biased estimates and cannot support the central claim of generalization.
  2. [Section 3 and Section 5, fire identification] For the fire identification scenario, the real-world test set consists of 100 Google Images photos (50 forest, 50 fire), and Section 5 reports that validation accuracy rose from about 86% to 96% after 'adding background of real forest images to the generated smoke and fire.' The paper does not describe the provenance of these real forest backgrounds. If they were drawn from the same Google Images search pool as the test images, the training data overlaps the evaluation distribution, which would make the accuracy an artifact of data leakage rather than a measure of synthetic-data generalization.
  3. [Section 4, counting houses] The counting result is quantitatively weak. An MSE of 20 on labels ranging from 0 to 38 houses corresponds to an RMSE of about 4.47 houses, which is a large error relative to the label range. This number is reported on a single validation set of only 60 images, with no confidence intervals, no cross-validation, and no breakdown by count. The absence of any baseline (e.g., training on real data, fine-tuning from ImageNet, or a standard counting method) makes it impossible to attribute the observed error level to the synthetic-data approach rather than to the architecture or the difficulty of the task.
  4. [Sections 1 and 5, overall evidence] The central claim is broad: synthetic data can prove useful for UAV-based aerial imagery. The paper supports it with only two tasks, each with a small validation set, and neither task is compared against a model trained on real data or against a published baseline. Without such comparisons, the results cannot distinguish the effect of synthetic training data from the effect of having ample training data plus a powerful CNN. The paper should either add such baselines or substantially temper the central claim to reflect that these are preliminary feasibility studies.
minor comments (6)
  1. [Section 3] The phrase 'Scalable Vector Machines' should be 'Support Vector Machines'.
  2. [Section 4 and Figures 3-4] Table 2 labels these images as 'Testing,' but Section 4 and the figure legends refer to 'validation dataset'; the terminology should be made consistent.
  3. [Section 5] There is a typo: 'Finallly' should be 'Finally'.
  4. [Section 5] The phrase 'state of art' should be written as 'state-of-the-art' for consistency.
  5. [Section 3] The adaptations to Inception-v3 are described qualitatively; exact hyperparameters (e.g., learning rate, batch size, optimizer, data augmentation settings) should be reported for reproducibility.
  6. [General] The paper does not state whether the synthetic-data generation scripts or the real-world datasets will be made publicly available; providing these would greatly increase the reproducibility of the work.

Circularity Check

1 steps flagged · score 6.0 of 10

The supporting 96% accuracy and MSE=20 are measured on the same validation sets used to iteratively tune both the model and the synthetic data, so they are fitted scores presented as testing results rather than independent evidence of generalization.

  1. fitted input called prediction [Section 5 (Discussion), with Table 2 and Figures 3-4; Section 3 methodology loop (Figure 1)]
    "Nevertheless, we need to be cautious with these indications, because the DL models were optimized to perform well in the specific validation datasets. It is questionable (and it has not been tested) whether the DL models can produce similar results in different real-world datasets that focus on similar problems and applications [10]."

    The paper's central claim that synthetic data can train DL models for UAV-based aerial imagery rests on CA=96% for fire identification and MSE=20 for house counting. Both numbers are reported on the 'validation dataset,' yet the methodology in Section 3 explicitly feeds observations from testing real-world data back into synthetic-dataset creation, and Section 5 states the counting MSE was reached 'after many weeks of observations and the iterative process of adding more details to the generated synthetic dataset,' with fire accuracy rising from about 86% to 96% after adding real forest backgrounds.

full rationale

This is not a case of self-citation circularity: the only author self-citations ([7], [10]) are background/related-work references and are not load-bearing for the reported results. The core idea of training on synthetic images and evaluating on real images is independently meaningful, and the experimental procedure is transparent enough that the paper itself acknowledges the limitation. However, the quantitative evidence for the central claim is compromised by evaluation leakage. The real-world images are called 'testing' in Table 2 but 'validation' in Section 4 and Figures 3-4, and Section 5 admits the models 'were optimized to perform well in the specific validation datasets.' Furthermore, the synthetic data were iteratively enriched (trees, grass, swimming pools, fences, real forest backgrounds) specifically to reduce error on those same validation images over many weeks. There is no separate held-out test set, no repeated cross-validation, no confidence intervals, and no baseline comparison. The reported numbers therefore measure tuning to a small validation set rather than generalization to unseen real-world imagery. That makes the support for the paper's conclusion partially circular: the prediction is, by the paper's own account, fitted to the set on which it is evaluated. Score 6 reflects partial circularity: the methodology demonstration remains real, but the headline results reduce, by construction, to validation-set optimization.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on a synthetic training set whose content was iteratively tuned to the validation data, and on hand-chosen network hyperparameters. These are fitted choices rather than input constants. No independent benchmark is used to separate learning from overfitting.

free parameters (5)
  • synthetic dataset composition = trees, grass, swimming pools, fences (added iteratively)
    The paper states that MSE was reduced over weeks by iteratively adding details to the generated synthetic dataset, so these choices are tuned to the validation data.
  • dropout rate = 35%
    Hand-chosen adaptation for counting; no ablation is reported.
  • initial convolution filter size = 7x7
    Hand-selected for counting; no ablation is reported.
  • stride = 5
    Hand-selected for counting; no ablation is reported.
  • number of synthetic training images = 2,000 (fire), 10,000 (counting)
    Chosen without a stated basis; no data scaling study is reported.
assumptions (3)
  • domain assumption Synthetic images rendered with PIL and OpenCV can capture enough real-world variation for transfer to real aerial photos.
    This is the core premise of the paper, invoked in Section 3, Methodology.
  • domain assumption The real-world datasets are correctly labeled, based on Google Images search labels and manual house counts.
    Section 3 describes labels as manual or search-based; no quality control is described.
  • domain assumption The validation dataset used for tuning is representative of the deployment domain.
    Section 5, Discussion explicitly questions this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles." pith.science (2026). https://pith.science/paper/LMJ3UAUB

@misc{pith2026190806472,
  author       = {Pith},
  title        = {Pith review of: Training Deep Learning Models via Synthetic Data: Application in Unmanned Aerial Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMJ3UAUB}},
  note         = {Machine review of arXiv:1908.06472}
}
read the original abstract

This paper describes preliminary work in the recent promising approach of generating synthetic training data for facilitating the learning procedure of deep learning (DL) models, with a focus on aerial photos produced by unmanned aerial vehicles (UAV). The general concept and methodology are described, and preliminary results are presented, based on a classification problem of fire identification in forests as well as a counting problem of estimating number of houses in urban areas. The proposed technique constitutes a new possibility for the DL community, especially related to UAV-based imagery analysis, with much potential, promising results, and unexplored ground for further research.

Figures

Figures reproduced from arXiv: 1908.06472 by the authors.

Figure 1
Figure 1. Basic methodology in generating data for training DL models. 2. A counting problem of estimating number of houses from aerial photos. The former is useful for UAV which monitor forest areas for fires and smoke, while the latter would be useful for policy-makers who want to understand dis￾tribution of houses in urban areas, possibilities for photovoltaic systems, urban gardening in roofs etc. For the problems under s… view at source ↗
Figure 2
Figure 2. Example images from the synthetic datasets (top). Example images from the real-world datasets (bottom). Images on the left are for the fire identification scenario, while images on the right for the case of the estimation of number of houses. As a DL model, we used the Inception-v3 convolutional neural network (CNN) architecture [23] (with some adaptations, see below), as it is one of the fastest CNN architectures a… view at source ↗
Figure 3
Figure 3. Training and validation classification accuracy at the fire identification chal￾lenge. iterative process of adding more details to the generated synthetic dataset (see [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training and validation MSE at the counting houses challenge. related specifically to counting, the work in [13] evades the hard task of learning to detect and localize individual object instances. Instead, it casts the problem as that of estimating an image density wh…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

  1. [1]

    In: BTW (Workshops)

    Amara, J., Bouaziz, B., Algergawy, A., et al.: A deep learning-based approach for banana leaf diseases classification. In: BTW (Workshops). pp. 79–88 (2017)

  2. [2]

    arXiv preprint arXiv:1605.07678 (2016)

    Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 (2016)

  3. [3]

    bioRxiv p

    Douarre, C., Schielein, R., Frindel, C., Gerth, S., Rousseau, D.: Deep learning based root-soil segmentation from x-ray tomography. bioRxiv p. 071662 (2016)

  4. [4]

    In: Proceedings of the International Conference on Agricultural Engineering, Aarhus, Denmark

    Dyrmann, M., Mortensen, A.K., Midtiby, H.S., Jorgensen, R.N., et al.: Pixel-wise classification of weeds and crops in images by using a fully convolutional neural net- work. In: Proceedings of the International Conference on Agricultural Engineering, Aarhus, Denmark. pp. 26–29 (2016)

  5. [5]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4340–4349 (2016)

  6. [6]

    In: Advances in neural information processing systems

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)

  7. [7]

    In: Machine Learning in the Environmental Sciences Workshop, in Proc

    Kamilaris, A.: Simulating training data for deep learning models. In: Machine Learning in the Environmental Sciences Workshop, in Proc. of EnviroInfo 2018. Munich, Germany (September 2018)

  8. [8]

    In: Proc

    Kamilaris, A., Assumpcio, A., Blasi, A.B., Torrellas, M., Prenafeta-Boldu, F.X.: Estimating the environmental impact of agriculture by means of geospatial and big data analysis: The case of catalonia. In: Proc. of EnviroInfo. Luxembourg (September 2017)

Show all 26 references
  1. [9]

    In: Disaster Management for Resilience and Public Safety Workshop, in Proc

    Kamilaris, A., Prenafeta-Boldu, F.X.: Disaster monitoring using unmanned aerial vehicles and deep learning. In: Disaster Management for Resilience and Public Safety Workshop, in Proc. of EnviroInfo2017. Luxembourg (September 2017)

  2. [10]

    Com- puters and Electronics in Agriculture 147, 70–90 (2018)

    Kamilaris, A., Prenafeta-Boldu, F.X.: Deep learning in agriculture: A survey. Com- puters and Electronics in Agriculture 147, 70–90 (2018)

  3. [11]

    arXiv preprint arXiv:1904.11621 (2019)

    Kar, A., Prakash, A., Liu, M.Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., Fidler, S.: Meta-sim: Learning to generate synthetic datasets. arXiv preprint arXiv:1904.11621 (2019)

  4. [12]

    IEEE transactions on medical imaging 26(7), 1010–1016 (2007) 7 Unity

    Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., Yli-Harja, O.: Com- putational framework for simulating fluorescence microscope images with cell pop- ulations. IEEE transactions on medical imaging 26(7), 1010–1016 (2007) 7 Unity. https://unity.com 10 A. Kamilaris

  5. [13]

    In: Advances in neural information processing systems

    Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in neural information processing systems. pp. 1324–1332 (2010)

  6. [14]

    arXiv preprint arXiv:1801.01726 (2018)

    Li, P., Liang, X., Jia, D., Xing, E.P.: Semantic-aware grad-gan for virtual-to-real urban scene adaption. arXiv preprint arXiv:1801.01726 (2018)

  7. [15]

    Distill 3(3), e10 (2018)

    Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., Mordv- intsev, A.: The building blocks of interpretability. Distill 3(3), e10 (2018)

  8. [16]

    arXiv preprint arXiv:1810.10093 (2018)

    Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., Birchfield, S.: Structured domain randomization: Bridging the reality gap by context-aware synthetic data. arXiv preprint arXiv:1810.10093 (2018)

  9. [17]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., Torralba, A.: Virtual- home: Simulating household activities via programs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8494–8502 (2018)

  10. [18]

    Sensors 17(4), 905 (2017)

    Rahnemoonfar, M., Sheppard, C.: Deep count: fruit counting based on deep sim- ulated learning. Sensors 17(4), 905 (2017)

  11. [19]

    In: European Conference on Computer Vision

    Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: European Conference on Computer Vision. pp. 102–

  12. [20]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3234–3243 (2016)

  13. [21]

    Neural networks 61, 85–117 (2015)

    Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85–117 (2015)

  14. [22]

    Washigton: Microsoft Research (2017)

    Shah, S., Dey, D., Lovett, C., Kapoor, A.: Aerial informatics and robotics platform. Washigton: Microsoft Research (2017)

  15. [23]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)

  16. [24]

    In: European conference on computer vision

    Walach, E., Wolf, L.: Learning to count with cnn boosting. In: European conference on computer vision. pp. 660–676. Springer (2016)

  17. [25]

    arXiv preprint arXiv:1801.02209 (2018)

    Wu, Y., Wu, Y., Gkioxari, G., Tian, Y.: Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209 (2018)

  18. [26]

    In: Proceedings of the IEEE interna- tional conference on computer vision

    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 2223–2232 (2017)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.