Pith. sign in

REVIEW 4 major objections 6 minor 21 references

The effects of using created synthetic images in computer vision training

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Rendered synthetic images can replace most real training photos in deep computer vision models without hurting test accuracy and with a smaller train-test accuracy gap.

desk verdict A practical, honest case study on mixing rendered and real images, but the quantitative headline claims are not established by the reported point estimates. read the letter →

arxiv 2506.03449 v1 pith:CYZTXIJ7 submitted 2025-06-03 cs.CV

John W. Smutny This is my paper
classification cs.CV
keywords syntheticimagegenerationUnrealEnginecomputervisiontrainingVGG16MobileNetV3-smallwelddefectdetectiondomaingapdatasetaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether synthetic images generated in a rendering engine can substitute for real photographs when training deep computer vision classifiers. Across two binary tasks (cat vs dog and weld defect detection) and two models (large VGG16, small MobileNetV3-small), the author finds that supplementing a real training set with more than 60% synthetic images leaves test accuracy essentially unchanged while shrinking the test-training accuracy gap to about 1–2%. Conversely, adding fewer than 10% real images to a synthetic-only training set halves the classification error, and the results suggest that as little as 10% real data is needed instead of the usual 50–70%. The paper also shows that frozen pre-trained models can be used as auditors, scoring real and synthetic images and comparing confidence-score histograms to approximate the real-synthetic domain gap.

What carries the argument

The argument rests on a procedural synthetic-image pipeline plus a controlled mixing protocol. In Unreal Engine, a Python script varies camera angle, lighting, background, and the weld seam, and weld defects are simulated by editing texture maps—Normal, Metallic, Roughness, and Ambient Occlusion—to create porosity, cracks, and incomplete welds; cat and dog images come from low-polygon public models across 100 camera angles, 10 backgrounds, and 8 lighting settings. The mixing protocol holds one image type (real or synthetic) constant while increasing the other, and trains VGG16 and MobileNetV3-small with early stopping and augmentation. A secondary mechanism is the auditor: frozen pre-trained models whose top-3 confidence histograms on real vs synthetic cats/dogs are used to approximate feature-space differences, though the two auditors disagree on whether the real and synthetic distributions are aligned.

What would settle it

Train the same models on synthetic-only weld images after making the texture-map defects deliberately unrealistic—for example, crack widths and porosity sizes far outside any range seen in real weld photos. If test accuracy on real welds stays near the reported levels, the claimed transfer does not depend on defect realism; if it collapses, fidelity is doing the work. A complementary check is to compare auditor confidence histograms of synthetic defects against real defect photos: essentially disjoint histograms would contradict the feature-space overlap the claim assumes.

Watch

Extended reading notes

Core claim

The central claim is that user-created rendered images are a viable partial substitute for real training images in deep CV classification. In the tested cases, models trained on mixed real/synthetic sets reached approximately the same peak test accuracy as models trained on real images alone, but with 13–40% fewer real samples; the smaller MobileNetV3-small model gained the most, needing about 60 real images per 100 synthetic images for a 1.8–2.8% test-accuracy drop. Adding synthetic images to a fixed real set did not consistently change test accuracy, but it consistently reduced the test-training accuracy gap, which the author interprets as improved generalization and reduced overfitting. Synthetic-only training was weak, but a small injection of real images improved it sharply, and the author concludes that synthetic rendering lets researchers train on roughly 10% real images instead of the traditional 50–70%.

Load-bearing premise

The load-bearing premise is that the procedurally generated weld defects and low-polygon cat/dog renders occupy the same feature space as real photographs, so models trained on them transfer to real-world images; the paper does not quantitatively validate this overlap for weld defects.

Editorial extensions

If this is right

  • Data-scarce projects can train on roughly 10% real images supplemented by synthetic renders and still approach the accuracy of fully real training sets.
  • Adding a small number of real images to a synthetic-only training set—under 10%—cuts classification error by about half, so real samples act as an anchor.
  • Adding synthetic images to a fixed real set does not reliably change test accuracy but narrows the train-test gap, indicating better generalization and less overfitting.
  • Because synthetic images are reproducible, controllable, and locally generated, they avoid the poisoning and licensing risks of scraping images from the internet.
  • Synthetic padding performs comparably to oversampling for rebalancing imbalanced weld-defect data, giving another use beyond pure dataset expansion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A default mixing ratio of roughly 60–80 synthetic images per 100 real images may generalize beyond these two tasks; the paper reports peak generalization in that band but tests only cat/dog and weld classification.
  • The auditor confidence histograms could be converted into a selection filter—discard synthetic images the frozen model scores low—since the paper's own results suggest only a small number of varied synthetic images are needed for the benefit.
  • The author's quality-over-quantity reading implies that investing in varied environments, poses, and lighting rather than raw render volume is the scalable path; that is an editorial extrapolation, not a tested claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper investigates whether synthetic images rendered with Unreal Engine 4 can supplement or partially replace real training images for deep computer vision classifiers. The authors train VGG16 and MobileNetV3-small on binary classification tasks (Cat vs Dog and Weld Defect Detection) with varying proportions of real and synthetic training images, and report test accuracy and the test-training accuracy gap. They claim that adding more than 60% synthetic images can narrow the train-test gap to roughly 1-2% without conclusive harm to test accuracy, and that adding less than 10% real images to synthetic-only training halves the error rate. The paper also proposes using pre-trained ImageNet models as 'auditors' to quantify the domain gap between real and synthetic images via confidence-score histograms.

Significance. If the quantitative claims hold, the paper provides practical, low-cost guidance for building training sets in data-scarce vision applications and introduces a lightweight method for assessing synthetic-data quality. The strengths are the reproducible synthetic-generation pipeline, the comparison of two very different model capacities, and the explicit cost-benefit framing. However, the core quantitative claims currently rest on unreplicated point estimates, the weld-defect evaluation is compromised by the imbalanced data and accuracy-only reporting, and the post-hoc removal of the 'minor imperfections' class after observing test performance biases the reported numbers. The qualitative trend that mixing synthetic and real images can improve the train-test gap is plausible, but the evidence as presented does not establish the specific quantitative headline claims.

major comments (4)
  1. [Section IV.A (Fig. 4.1-4.4)] All reported accuracies are single point estimates with no indication of repeated runs, seeds, confidence intervals, or significance tests, despite the stochastic training procedure described in Section III.C (augmentation, early stopping, learning-rate decay). The key differences are often small (e.g., 96.4% vs 96.5% in Fig. 4.1; 1.8% and 2.8% drops in Section IV.A) and could easily be run-to-run noise. The abstract's quantitative claims that adding >60% synthetic images narrows the gap to ~1-2% and that <10% real data halves the error rate are therefore not statistically supported. I recommend reporting mean and standard deviation over at least 3-5 training seeds for each condition, and ideally showing effect sizes or confidence intervals.
  2. [Section III.A] The 'minor imperfections' class was removed after observing test performance: the paper states the class 'and its created images, were removed from the synthetic training set because during testing the class did not appear to improve model accuracy or generalization.' This is a post-hoc selection on the test set, which biases the reported test accuracies upward. The final numbers are not an honest estimate of the pipeline as described. The class should either be included in the final reported results, or the decision should be justified using only validation data and the final test numbers recomputed without the removed class.
  3. [Section III.A and Fig. 4.1] The Weld Defect Detection dataset is imbalanced at 70% Defect / 30% Good, yet only test accuracy is reported. A trivial always-Defect classifier would achieve 70% accuracy, and the synthetic-only training accuracies (61.4% for VGG16, 40.2% for MobileNet) are below that baseline. Consequently, the claims that adding real images 'halves the error rate' and that only 10% real data suffices are not substantiated without per-class precision/recall, ROC-AUC, or balanced accuracy. Please report class-wise metrics, or at least balanced accuracy, for both use cases.
  4. [Section IV.B and Fig. 4.6/4.7] The auditor method is presented inconsistently. The body text states that 'both model auditors show there is a significant difference in top-3 confidence and distribution shape between the real and synthetic cat & dog images,' but the caption of Fig. 4.7 says that 'similar curves imply that the model's extracted feature spaces for real & synthetic images are closely aligned.' This contradiction undermines the proposed domain-gap metric. Additionally, the method is described as a hypothesis rather than validated against the training outcomes. Please reconcile the captions with the text, and if the auditor is intended to predict transfer success, demonstrate that relationship quantitatively.
minor comments (6)
  1. [References [3]] Reference [3] is listed as an anonymous ECCV submission ID; this is not a citable published reference and should be replaced with the actual paper (e.g., the Nguyen et al. work mentioned) or removed.
  2. [Section III.B] The phrase 'Every 30 frames a sequence of commands is sent ... (resulting in 80 uniq ue ~1980x1240 screenshots a minute)' appears to contain a typo; the intended meaning is likely '80 unique screenshots per minute.' Please clarify.
  3. [Section III.C] The training procedure does not state the batch size or the maximum number of epochs, which are needed to reproduce the experiments. Please add these details.
  4. [Section IV.A] The sentence 'it's possible to reach the approximate peak test accuracy of exclusively real image trained models with 13-40% fewer real samples' is vague; please specify the baseline for each model and use-case, and give the exact ratios from the figures.
  5. [Figure 3.3] The phrase 'in the Right image, that the 'weld' not 3-dimensional' contains a grammatical error that obscures the intended meaning; please revise.
  6. [Section III.B (and final 'Unreal Engine meshes & models' list)] The statement that 'all other 3D meshes & models ... were free to download and license' is too vague; please provide specific license information or URLs for the cat and dog models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the paper is an empirical measurement study whose claims are not forced by construction or by self-citation.

full rationale

The paper reports a set of controlled training experiments, not a derivation. The central claims about test accuracy, the test-training accuracy gap, and the percentage of real images needed are all measured by training models on explicit mixtures of real and synthetic images and evaluating on held-out real test sets. No equation in the paper defines a predicted quantity from a fitted quantity, and no fitted parameter is relabeled as a prediction. The pre-trained auditor method in Section III.D is explicitly presented as a hypothesis-generating tool, and its histograms are observational rather than inputs to the training pipeline; it is not load-bearing for the headline accuracy results. The removal of the 'minor imperfections' class from the synthetic weld set is a post-hoc experimental selection that may bias the reported point estimates, but that is a validity and robustness concern, not circularity. The references to prior work are external related research and are not used as a self-citation chain to justify the paper's own results. Since no circular step can be exhibited with quoted evidence, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper makes no mathematical derivation and introduces no free parameters or invented entities. The main assumptions are domain assumptions about the validity of the train-test gap, the auditor method, and the synthetic weld defect simulations.

assumptions (3)
  • domain assumption The test-training accuracy gap is a valid measure of generalization and overfitting.
    The paper uses reductions in the train-test gap as evidence of improved generalization throughout Section IV.A.
  • domain assumption Pre-trained ImageNet classifiers can meaningfully score the real-synthetic domain gap through top-3 confidence histograms.
    Section IV.B introduces the auditor method and itself labels the interpretation as a hypothesis ('This paper hypothesizes...').
  • domain assumption The synthetic weld defect images, created by texture-map edits, are representative of real weld defects.
    Section III.B describes creating porosity, crack, and incomplete-weld defects procedurally; the paper provides no quantitative validation that these match real defect distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The effects of using created synthetic images in computer vision training." pith.science (2026). https://pith.science/paper/CYZTXIJ7

@misc{pith2026250603449,
  author       = {Pith},
  title        = {Pith review of: The effects of using created synthetic images in computer vision training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYZTXIJ7}},
  note         = {Machine review of arXiv:2506.03449}
}
read the original abstract

This paper investigates how rendering engines, like Unreal Engine 4 (UE), can be used to create synthetic images to supplement datasets for deep computer vision (CV) models in image abundant and image limited use cases. Using rendered synthetic images from UE can provide developers and businesses with a method of accessing nearly unlimited, reproducible, agile, and cheap training sets for their customers and applications without the threat of poisoned images from the internet or the cost of collecting them. The validity of these generated images are examined by testing the change in model test accuracy in two different sized CV models across two binary classification cases (Cat vs Dog and Weld Defect Detection). In addition, this paper provides an implementation of how to measure the quality of synthetic images by using pre-trained CV models as auditors. Results imply that for large (VGG16) and small (MobileNetV3-small) parameter deep CV models, adding >60% additional synthetic images to a real image dataset during model training can narrow the test-training accuracy gap to ~1-2% without a conclusive effect on test accuracy compared to using real world images alone. Likewise, adding <10% additional real training images to synthetic only training sets decreased the classification error rate in half, then decreasing further when adding more real training images. For these cases tested, using synthetic images from rendering engines allow researchers to only use 10% of their real images during training, compared to the traditional 50-70%. This research serves as an example of how to create synthetic images, guidelines on how to use the images, potential restrictions and possible performance improvements for data-scarce projects.

Figures

Figures reproduced from arXiv: 2506.03449 by the authors.

Figure 3
Figure 3. outlines the differences between the two [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. ). This [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 4
Figure 4. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 7.2
Figure 7.2. Figure 7.2: Examples of high & low confidence cat dog images. B. Advice on making synthetic images This paper makes the following recommendations to improve generated synthetic data. First, utilize randomness in every facet of ‘setting up’ a screenshot (like the Welding collecti…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

  1. [1]

    Does Polygon Count matter in 3D Modeling for Game Assets?

    3D Ace. “Does Polygon Count matter in 3D Modeling for Game Assets?”. Nicosia, Cyprus. Jan 3rd, 2024. https://3d-ace.com/blog/polygon-count-in- 3d-modeling-for-game-assets/

    work page 2024

  2. [2]

    Python in Unreal Tips

    DowlingSoka, Ryan. “Python in Unreal Tips”. Published on Mar 18th,

    work page

  3. [3]

    Learning image classifiers from (limited) real and (abundant) synthetic data

    ECCV-18 Unknown author submission ID 2903. “Learning image classifiers from (limited) real and (abundant) synthetic data.” Semantic Scholar. 2018. Corpus ID: 53472409. https://www.semanticscholar.org/paper/and-(abundant)-synthetic- data/1f759dbd74ad3907457dcbf50d4be54b470e33f4. Citation possibly for Nguyen, Tan, Chen, H, et al. “Learning image classifiers...

    work page 2018

  4. [4]

    How Low Can You Go? Using Synthetic 3D Imagery to Drastically Reduce Real-World Training Data for Object Detection

    Gastelum, Zoe Nellie, and Shead, Timothy. How Low Can You Go? Using Synthetic 3D Imagery to Drastically Reduce Real-World Training Data for Object Detection. United States: N. p., 2020. Web. doi:10.2172/1670874

    work page doi:10.2172/1670874 2020

  5. [5]

    Deep residual learning for image recognition

    He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

    2016

  6. [6]

    Searching for mobilenetv3

    Howard, Andrew, et al. "Searching for mobilenetv3." Proceedings of the IEEE/CVF international conference on computer vision. 2019

    work page 2019

  7. [7]

    A Review of Synthetic Image Data and Its Use in Computer Vision

    Man, Keith, and Javaan Chahl. “A Review of Synthetic Image Data and Its Use in Computer Vision.” Journal of imaging vol. 8,11 310. 21 Nov. 2022, doi:10.3390/jimaging8110310

    work page doi:10.3390/jimaging8110310 2022

  8. [8]

    Metal 3: MIG Welding

    OpenWorks, Baltimore City. Mar 16th, 2024. Metal 3 with Brandon Padayao. Articles: “Metal 3: MIG Welding” and “Metal 3: TIG Welding”. 1400 Greenmount Ave, Baltimore, MD 21202

    work page 2024

Show all 21 references
  1. [9]

    Professional Welder & Welding Instructor

    Padayao, Brandon. Professional Welder & Welding Instructor. Personal interview at Baltimore Openworks. Mar 16th, 2024

    2024

  2. [10]

    UnrealCV: Virtual Worlds for Computer Vision

    Qiu, Weichao, Zhong, Fangwei, Zhang, Yi, et al. “UnrealCV: Virtual Worlds for Computer Vision”. ACM Multimedia Open Source Software Competition. Mountain View, CA, USA. 2017. unrealcv.org

    2017

  3. [11]

    Very deep convolutional networks for large -scale image recognition

    Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large -scale image recognition." arXiv preprint arXiv:1409.1556 (2014)

    2014 arXiv

  4. [12]

    Domain Adaptation of Unreal Images for Image Classification (Dissertation)

    Thornström, J. “Domain Adaptation of Unreal Images for Image Classification (Dissertation).” Linköping University Department of Electrical Engineering, Computer Vision (Sweden). 2019. URN: 165758

    2019

  5. [13]

    Cost -Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?

    Weiss, Gary M, McCarthy, Kate, and Zabar, Bibi. “Cost -Sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?”. Department of Computer and Information Science, Fordham University. 2007. https://storm.cis.fordham.edu/~gweiss/pap...

    2007

  6. [14]

    Welding Images Dataset

    15DEM20F. “Welding Images Dataset”. Dataset 01. Publisher Roboflow. Roboflow Universe. Jul 2022. https://universe.roboflow.com/15dem20f- ab0x1/welding-images

    2022

  7. [15]

    Proyek Akhir - OFA5 Dataset

    afiys. “Proyek Akhir - OFA5 Dataset”. Dataset 01. Publisher Roboflow. Roboflow Universe. Oct 2023. https://universe.roboflow.com/afiys - oxbaf/proyek-akhir-ofa5

    2023

  8. [16]

    Welding_Detection_v2 Dataset

    Celebal Workspace. “Welding_Detection_v2 Dataset”. Publisher Roboflow. Roboflow Universe. Dec 2022. https://universe.roboflow.com/celebal-workspace/welding_detection_v2

    2022

  9. [17]

    Cats vs Dogs

    Elson, Jeremy and Douceur, John (JD) and Howell, Jon and Saul, Jared. “Cats vs Dogs”. Proceedings of 14th ACM Conference on Computer and Communications Security (CCS). October 2007. Published by Association for Computing Machinery, Inc. Distributed by Micr osoft. https://www.m...

    2007

  10. [18]

    Weld_detections Dataset

    Welddefect. “Weld_detections Dataset”. Publisher Roboflow. Roboflow Universe. Jul 2023. https://universe.roboflow.com/welddefect - 4pvic/weld_detections/dataset/1

    2023

  11. [19]

    Final Dataset Dataset

    Weld 1. “Final Dataset Dataset”. Dataset 01. Publisher Roboflow. Roboflow Universe. April 2023. https://universe.roboflow.com/weld-1- mba6s/final-dataset-rc2vm Unreal Engine meshes & models

    2023

  12. [20]

    Samur Art

    4K Metal Material Pack. Samur Art. Publisher Epic Games (now FAB), September 26th, 2024. Unreal Engine ver4.27. https://www.fab.com/listings/0aedc3d3-4971-4ae0-827f-0e2386385b97 A list of all other 3D meshes & models (includes all cat & dog models) were free to download and license

    2024

  13. [2024]

    https://ryandowlingsoka.com/unreal/python-in-unreal/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.