REVIEW 3 major objections 6 minor 32 references
Development of Hybrid Artificial Intelligence Training on Real and Synthetic Data: Benchmark on Two Mixed Training Strategies
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fine-tuning usually beats simple mixed training for hybrid real-and-synthetic data, but a large domain gap can reverse the result.
desk verdict A solid, controlled benchmark confirming FT usually beats SM, but the paper's own domain-gap metric contradicts its third conclusion, so that part needs major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a controlled benchmark design that isolates the mixing strategy from other variables. For each of three hybrid datasets, the authors create eleven equal-sized subsets spanning synthetic proportions from 0% to 100%, match class distributions across subsets, keep total dataset size constant, and train vanilla multilayer perceptron (MLP), convolutional neural network (CNN), and vision transformer (ViT) models with plain stochastic gradient descent and no augmentation or regularization. The two strategies are simple mixed (SM), which samples both domains jointly in every batch, and fine-tuned (FT), which pretrains only on synthetic data with early stopping on the real validation set and then retrains on the real part. The domain gap is operationalized as the difference between purely real and purely synthetic baseline accuracy. That operationalization, together with the controlled ratio sweep, is what lets the paper attribute performance differences to the strategy rather than to dataset size, class imbalance, or pretraining on external data.
What would settle it
Take a fourth hybrid dataset whose synthetic images are visually close to the real ones but whose synthetic-only baseline accuracy is nearly as low as the DomainNet sketches, a large measured gap with a small visual gap. If FT does not beat SM on that dataset, the proposed link between domain-gap size and strategy choice fails; alternatively, compute the correlation between the baseline gap and the FT-minus-SM advantage across many datasets and check whether a single threshold reliably separates the two regimes.
Extended reading notes
Core claim
The central discovery is that fine-tuning is generally, but not universally, superior to simple mixed training for hybrid real/synthetic image datasets, with the exception shaped by both architecture and the size of the domain gap. Using equally sized subsampled hybrid datasets with synthetic-to-real ratios from 10% to 90%, the study finds FT ahead in 69 of 81 averaged configurations, and in 635 of 810 individual comparisons. Gains from adding real data follow diminishing returns: the first 10% of real data yields the largest accuracy jump, and each further increase gives less. On the two datasets whose synthetic images were designed to resemble the real ones (CiFake from Stable Diffusion and CAD-rendered LEGO bricks), FT consistently wins; on DomainNet, where synthetic data are black-and-white sketches, a CNN trained with SM beats FT. The paper interprets this as evidence that the optimal mixing strategy interacts with the domain gap, and suggests the strategy could in principle be chosen ahead of time from a quantifiable gap measure.
Load-bearing premise
The paper's actionable conclusion rests on assuming that three datasets, one per synthetic generation mechanism, represent synthetic data generally, and that the baseline accuracy gap between purely synthetic and purely real training measures the domain gap that decides which mixing strategy wins.
Editorial extensions
If this is right
- Practitioners mixing realistic synthetic renderings with real images should default to fine-tuning over joint training, especially when the synthetic fraction is high.
- A small real-data addition, around 10% of the training set, captures most of the performance benefit, so budgets can start there before collecting more real samples.
- When the synthetic domain is visually far from the real domain, as with sketches, simple mixed training can beat fine-tuning, particularly for CNNs, so the default should be re-evaluated per dataset.
- The optimal strategy can in principle be selected ahead of training from a quantifiable domain-gap measure, turning trial-and-error into a more systematic workflow.
Reading between the lines
- If the proposed domain-gap rule generalizes, any existing real/synthetic dataset pair plus a few baseline runs would be enough to check whether the FT-minus-SM gap correlates with the synthetic-versus-real baseline gap, giving a cheap way to validate the rule on new domains.
- The kernel-level explanation for the CNN/DomainNet reversal, that early layers lock onto binary edge detectors during synthetic pretraining, predicts that lighter pretraining, domain-randomized synthetic images, or kernel reinitialization before fine-tuning should reduce or remove the SM advantage.
- The diminishing-returns curve implies an economic rule for data collection: the marginal value of a real image is highest when the current real fraction is near zero, which could guide active learning or data-budgeting decisions.
- Because the benchmark uses unregularized, augmentation-free networks, production models with normalization, augmentation, and modern optimizers may show a different ordering of the two strategies than the one reported here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a benchmark study comparing two mixed training strategies for hybrid real/synthetic datasets: simple mixed (SM), in which real and synthetic data are trained together, and sequential fine-tuning (FT), in which a model is pretrained on synthetic data and then fine-tuned on real data. Using three network architectures (MLP, CNN, ViT) and three hybrid datasets generated by different mechanisms (GenAI, CAD, hand drawings), the authors sample eleven synthetic-to-real proportions and report test accuracies over ten repetitions. The central empirical claim is that FT usually outperforms SM (635 of 810 individual comparisons, 69 of 81 averaged configurations), with a notable counter-example for CNNs on DomainNet, where SM performs better. The paper further claims that the optimal strategy can be selected based on a quantifiable domain gap, defined as the difference between purely real and purely synthetic baseline accuracies.
Significance. The paper contributes a controlled and fairly extensive benchmark: dataset size is held constant across ratios, class balance is enforced, baselines are external, and the full code and numerical results are made available. The repeated runs and the cross-dataset, cross-architecture design are genuine strengths, and the reported FT-over-SM advantage with a documented counter-example is a useful practical data point. However, the paper's third conclusion—that the optimal strategy can be chosen from a quantifiable domain gap—is not supported by the paper's own operationalization of that gap, which is a load-bearing issue for the advertised actionable guidance. The mechanistic explanation for the CNN/DomainNet exception is also presented without supporting evidence, and the headline comparisons lack uncertainty quantification.
major comments (3)
- [Sections 4–6] Section 4 states that 'the domain gap can be quantified as the difference between the purely real and purely synthetic results' (Table 2). Under that definition, DomainNet has the smallest gap, not the largest: for the CNN, Cifar-10/CiFake gap = 0.6413 − 0.1430 = 0.4983, LegoBricks gap = 0.5582 − 0.0130 = 0.5452, and DomainNet gap = 0.2870 − 0.0353 = 0.2517; the same ordering holds for the MLP and ViT. Yet Section 5 asserts 'DomainNet exhibits a larger domain gap compared to Cifar-10/CiFake and LegoBricks', and Section 6 concludes that the optimal strategy could be chosen based on a quantifiable domain gap. The stated metric contradicts the qualitative claim, so the third conclusion is unsupported by the reported data. The authors must either revise the gap measure to one that actually orders the datasets as claimed, or present the DomainNet result purely as an architecture–data interaction without linking it to gap magnitude.
- [Section 4] The headline comparison counts (635/810 and 69/81) are reported without confidence intervals, standard deviations, or significance tests. Several boxplots, notably Fig. 5 for CNN/DomainNet and many appendix figures, show largely overlapping distributions between SM and FT at individual ratios, so it is unclear whether the aggregate counts reflect reliable differences or are dominated by small, noisy effects. Given that each configuration was repeated ten times, paired tests or confidence intervals on the per-configuration accuracy differences should be provided to support the first conclusion that 'FT is a preferable strategy'.
- [Section 5] The proposed explanation for the CNN/DomainNet counter-example—that pretraining on simple black-and-white sketches biases early convolutional layers toward edge detectors, leading to ill-conditioned gradients during fine-tuning—is presented as the reason for the results, but no evidence is offered: there are no kernel visualizations, probing experiments, or ablations that would distinguish this mechanism from alternative explanations. As written, this is a plausible post-hoc hypothesis, not a substantiated finding. The authors should either clearly label the discussion as speculative or add targeted analyses to support it, especially since the paper's main contribution is its controlled experimental comparison.
minor comments (6)
- [Section 6] The text says 'Across 89 individual configurations FT surpassed SM in 69 (78%)', but the experimental design yields 81 averaged configurations (3 datasets × 3 architectures × 9 hybrid ratios), matching the '69 of 81' reported in Section 4. The number 89 appears to be a typo and should be corrected.
- [Section 5] The reference '(Fig. 3.3)' near the description of DomainNet's synthetic images appears to be an incorrect cross-reference; it should likely point to Fig. 1 or a relevant panel.
- [Figures 4, 5, 12–14] The figure captions label the dataset as 'domain' instead of 'DomainNet'; please use the full dataset name for clarity.
- [Throughout] The dataset name is written inconsistently as 'Cifake', 'CiFake', and 'Cifar-10/Cifake' in different places; standardize the capitalization and the hybrid dataset label.
- [Various] There are several typographical and grammatical issues, including 'Therefor' (Section 3.2), 'It's real part' (Section 3.3), 'relaying on' (Section 1), and 'sgriculture' in reference [29]. A careful copyedit is recommended.
- [Section 3.3] The sentence 'LEGO Bricks [2] for training classification networks was selected as the second dataset' is awkwardly phrased; consider rewording to make the subject and intent clear.
Circularity Check
No circularity: the benchmark results are measured, not derived from their own inputs.
full rationale
This is an empirical benchmark, not a derivation chain. The central claims (FT outperforms SM in 635/810 comparisons, ratio effects, baselines) are measured on held-out real test sets over ten repetitions, with purely synthetic and purely real training as external baselines. No quantity in the paper is defined in terms of the result it is meant to explain, and no fitted parameter is relabeled as a prediction. The only self-citation, [29] Wachter et al., appears in Related Work as an example of synthetic-data generation and is not load-bearing for any conclusion. For completeness, the paper does contain a consistency problem that is not circularity: Section 4 states that 'the domain gap can be quantified as the difference between the purely real and purely synthetic results' (Table 2), but Section 5 asserts 'DomainNet exhibits a larger domain gap compared to Cifar-10/CiFake and LegoBricks,' which contradicts that quantification (CNN real-minus-synthetic accuracy: Cifar-10/CiFake 0.6413-0.1430=0.4983, LegoBricks 0.5582-0.0130=0.5452, DomainNet 0.2870-0.0353=0.2517; similar ordering for MLP and ViT). Consequently, the Section 6 conclusion that the optimal strategy 'could be chosen based on a quantifiable domain gap measure' is unsupported by the paper's own metric. This is a correctness/consistency issue, not circular reasoning, because the conclusion is not assumed in constructing the measurements or in defining the strategies; the empirical comparisons themselves remain self-contained.
Assumptions & free parameters
assumptions (3)
- domain assumption The three selected hybrid datasets (Cifar-10/CiFake, LegoBricks, DomainNet) are representative of synthetic data generation mechanisms (GenAI, CAD, hand drawings) and their associated domain gaps.
- domain assumption The difference between purely synthetic and purely real baseline test accuracy is a valid operationalization of the domain gap.
- domain assumption The training protocol (vanilla SGD, no regularization/data augmentation, fixed hyperparameters) isolates the effect of the mixing strategy.
Cite this review
Pith. "Pith review of Development of Hybrid Artificial Intelligence Training on Real and Synthetic Data: Benchmark on Two Mixed Training Strategies." pith.science (2026). https://pith.science/paper/CHJHX4NR
@misc{pith2026250624093,
author = {Pith},
title = {Pith review of: Development of Hybrid Artificial Intelligence Training on Real and Synthetic Data: Benchmark on Two Mixed Training Strategies},
year = {2026},
howpublished = {\url{https://pith.science/paper/CHJHX4NR}},
note = {Machine review of arXiv:2506.24093}
}
read the original abstract
Synthetic data has emerged as a cost-effective alternative to real data for training artificial neural networks (ANN). However, the disparity between synthetic and real data results in a domain gap. That gap leads to poor performance and generalization of the trained ANN when applied to real-world scenarios. Several strategies have been developed to bridge this gap, which combine synthetic and real data, known as mixed training using hybrid datasets. While these strategies have been shown to mitigate the domain gap, a systematic evaluation of their generalizability and robustness across various tasks and architectures remains underexplored. To address this challenge, our study comprehensively analyzes two widely used mixing strategies on three prevalent architectures and three distinct hybrid datasets. From these datasets, we sample subsets with varying proportions of synthetic to real data to investigate the impact of synthetic and real components. The findings of this paper provide valuable insights into optimizing the use of synthetic data in the training process of any ANN, contributing to enhancing robustness and efficacy.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
IEEE Access12, 15642–15650 (2024)
Bird, J.J., Lotfi, A.: Cifake: Image classification and explainable identification of ai- generated synthetic images. IEEE Access12, 15642–15650 (2024). https://doi. org/10.1109/access.2024.3356122
-
[2]
https://doi.org/10.34808/RCZA-JY08
Boiński, T., Zaraziński, S., Śledź, B.: Lego bricks for training classification network (2021). https://doi.org/10.34808/RCZA-JY08
-
[3]
Burdorf, S., Plum, K., Hasenklever, D.: Reducing the amount of real world data for object detector training with synthetic data (2022).https://doi.org/10.48550/ ARXIV.2202.00632
work page Pith review arXiv doi:10.48550/arxiv.2202.00632 2022
-
[4]
https://doi.org/10.48550/ARXIV.2208.04052
De Roovere, P., Moonen, S., Michiels, N., Wyffels, F.: Dataset of industrial metal objects (2022). https://doi.org/10.48550/ARXIV.2208.04052
-
[5]
In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=YicbFdNTTy
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021),https: //openreview.net/forum?id=YicbFdNTTy
2021
-
[6]
Ettedgui, S., Abu-Hussein, S., Giryes, R.: ProCST: Boosting semantic segmen- tation using progressive cyclic style-transfer (2022).https://doi.org/10.48550/ ARXIV.2204.11891
work page Pith review arXiv doi:10.48550/arxiv.2204.11891 2022
-
[7]
Sensors 21(23), 7901 (Nov 2021)
Eversberg, L., Lambrecht, J.: Generating images with physics-based rendering for an industrial object detection task: Realism versus domain randomization. Sensors 21(23), 7901 (Nov 2021). https://doi.org/10.3390/s21237901
-
[8]
Biological Cybernetics 36(4), 193–202 (Apr 1980).https://doi.org/10.1007/bf00344251
Fukushima, K.: Neocognitron: A self-organizing neural network model for a mech- anism of pattern recognition unaffected by shift in position. Biological Cybernetics 36(4), 193–202 (Apr 1980).https://doi.org/10.1007/bf00344251
Show all 32 references
-
[9]
Automation in Construc- tion 149, 104771 (May 2023).https://doi.org/10.1016/j.autcon.2023.104771
Kim,J.,Kim,D.,Lee,S.,Chi,S.:Hybriddnntrainingusingbothsyntheticandreal construction images to overcome training data shortage. Automation in Construc- tion 149, 104771 (May 2023).https://doi.org/10.1016/j.autcon.2023.104771
2023
-
[10]
Frontiers in Plant Science15 (Sep 2024)
Klein, J., Waller, R., Pirk, S., Pałubicki, W., Tester, M., Michels, D.L.: Synthetic data at scale: a development model to efficiently leverage machine learning in agriculture. Frontiers in Plant Science15 (Sep 2024). https://doi.org/10.3389/ fpls.2024.1360113
2024
-
[11]
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny im- ages. Tech. rep., University of Toronto, Toronto, Ontario (2009),https://www.cs. toronto.edu/~kriz/learning-features-2009-TR.pdf
2009
-
[12]
In: 2019 19th International Conference on Advanced Robotics (ICAR)
Lambrecht, J., Kästner, L.: Towards the usage of synthetic data for marker-less pose estimation of articulated robots in rgb images. In: 2019 19th International Conference on Advanced Robotics (ICAR). pp. 240–247. IEEE (Dec 2019).https: //doi.org/10.1109/icar46387.2019.8981600
2019
-
[13]
LDraw.org: Content and licensing information (2020),https://www.ldraw.org/, accessed: 2025-04-24
2020
-
[14]
Springer International Publish- ing (2021)
Nikolenko, S.I.: Synthetic Data for Deep Learning. Springer International Publish- ing (2021). https://doi.org/10.1007/978-3-030-75178-4
2021 doi
-
[15]
https://doi.org/10.48550/ARXIV.1907
Nowruzi, F.E., Kapoor, P., Kolhatkar, D., Hassanat, F.A., Laganiere, R., Rebut, J.: How much real data do we actually need: Analyzing object detection performance using synthetic and real data (2019). https://doi.org/10.48550/ARXIV.1907. 07061
2019 doi
-
[16]
In: 2019 IEEE/CVF International Conference 20 P
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., Wang, B.: Moment matching for multi-source domain adaptation. In: 2019 IEEE/CVF International Conference 20 P. Wachter et al. on Computer Vision (ICCV). pp. 1406–1415. IEEE (Oct 2019).https://doi.org/ 10.1109/iccv.2019.00149
2019
-
[17]
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Peng, X., Usman, B., Kaushik, N., Wang, D., Hoffman, J., Saenko, K.: Visda: A synthetic-to-real benchmark for visual domain adaptation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 2102–21025. IEEE (Jun 2018).https://doi.org/10...
2018
-
[18]
In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Poucin, F., Kraus, A., Simon, M.: Boosting instance segmentation with synthetic data: A study to overcome the limits of real world data sets. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). pp. 945–953. IEEE (Oct 2021).https://doi.org/10.1109/i...
2021
- [19]
-
[20]
In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10674–10685. IEEE (Jun 2022). https://doi.org/10.1109/cvpr52688.2022.01042
2022
-
[21]
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (Jun 2016).https://doi.org/1...
2016 doi
-
[22]
Schmidhuber, J.: Annotated history of modern ai and deep learning (2022).https: //doi.org/10.48550/ARXIV.2212.11279
2022 doi
-
[23]
Applied Sciences15(1), 354 (Jan 2025)
Staniszewski, M., Kempski, A., Marczyk, M., Socha, M., Foszner, P., Cebula, M., Labus, A., Cogiel, M., Golba, D.: Searching for the ideal recipe for preparing syn- thetic data in the multi-object detection problem. Applied Sciences15(1), 354 (Jan 2025). https://doi.org/10.3390...
2025 doi
-
[24]
Springer International Publishing, 2nd edn
Szeliski, R.: Computer Vision: Algorithms and Applications. Springer International Publishing, 2nd edn. (2022).https://doi.org/10.1007/978-3-030-34372-9
2022 doi
-
[25]
In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., Birchfield, S.: Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In: 2018 IEEE/CVF Conference on Computer Vision and Patte...
2018
-
[26]
In: 33rd British Machine Vision Confer- ence 2022, BMVC 2022, London, UK, November 21-24, 2022
Vanherle, B., Moonen, S., Reeth, F.V., Michiels, N.: Analysis of training object detection models with synthetic data. In: 33rd British Machine Vision Confer- ence 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press (2022), https://bmvc2022.mpi-inf.mpg.de/0833.pdf
2022
-
[27]
In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processi...
2017
-
[28]
In: 2022 International Conference on Smart Systems and Technologies (SST)
Vuletic, J., Polic, M., Orsag, M.: Procedural generation of synthetic dataset for robotic applications in sweet pepper cultivation. In: 2022 International Conference on Smart Systems and Technologies (SST). pp. 309–314. IEEE (Oct 2022).https: //doi.org/10.1109/sst55530.2022.9954643
2022
-
[29]
In: Informatik in der Land-, Forst-und Ernährungswirtschaft-Fokus: Biodiversität fördern durch digitale Landwirtschaft
Wachter, P., Kruse, N., Schöning, J.: Synthetic fields, real gains: Enhancing smart sgriculture through hybrid datasets. In: Informatik in der Land-, Forst-und Ernährungswirtschaft-Fokus: Biodiversität fördern durch digitale Landwirtschaft. Development of Hybrid AI Training on...
2024
-
[30]
Neurocomputing 312, 135–153 (Oct 2018).https://doi.org/10.1016/j.neucom.2018.05.083
Wang, M., Deng, W.: Deep visual domain adaptation: A survey. Neurocomputing 312, 135–153 (Oct 2018).https://doi.org/10.1016/j.neucom.2018.05.083
2018 doi
-
[31]
In: Proceedings of the 37th International Conference on Machine Learning
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., Liu, T.Y.: On layer normalization in the transformer architecture. In: Proceedings of the 37th International Conference on Machine Learning. ICML’20, JMLR.org (2020), https://dl.acm.org/...
2020
-
[32]
In: 2017 IEEE International Con- ference on Computer Vision (ICCV)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Con- ference on Computer Vision (ICCV). pp. 2242–2251. IEEE (Oct 2017). https: //doi.org/10.1109/iccv.2017.244
2017 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.