{"id":"67c63e12-e8bc-40e2-bc54-1551936f648f","arxiv_id":"2506.24093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning on real data after synthetic pretraining beats mixing both data types in most settings, but simple mixing wins for CNNs on large-gap sketch data.","lead":"This study benchmarks two ways of mixing synthetic and real images when training vision models: using both at once versus training on synthetic first and then fine-tuning on real. Across 1,800 trained models, fine-tuning usually wins, but a convolutional network on hand-drawn sketches is a clear exception, so the best strategy depends on the size of the domain gap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Domain-gap ordering contradicts the paper's own metric: Table 2 shows DomainNet has the smallest real-minus-synthetic gap, not the largest, so the Sec. 6 strategy-selection claim is unsupported.","rationale":"The paper's primary empirical finding (FT beats SM in the large majority of configurations) is credible given the controlled design and repeated trials. The stress-test pass focused on the third conclusion, which converts the benchmark into actionable advice: pick FT when the domain gap is small and SM when it is large. This conclusion depends on a quantifiable gap measure, and Section 4 offers one: the difference between purely real and purely synthetic baseline accuracy. Computing that difference from Table 2 shows DomainNet has the smallest, not the largest, gap for every architecture. For example, CNN: Cifar-10/CiFake 0.4983, LegoBricks 0.5452, DomainNet 0.2517. This contradicts Section 5's assertion that DomainNet exhibits a larger domain gap. Alternative normalized metrics (relative to real accuracy or relative to chance) also fail to place DomainNet first, so the inconsistency is not an artifact of one arbitrary normalization. The practical upshot is that the data distinguish at most two coarse categories (mimic-like synthetic sources vs hand-drawn sketches) rather than supporting a continuous gap-based rule. This does not invalidate the first two findings, but it means the third conclusion is currently unsupported. The reader's conditional verdict already anticipated uncertainty around the domain-gap operationalization; this check confirms and sharpens that uncertainty, so no verdict change is recommended.","tokens_in":11501,"tokens_out":5211,"duration_ms":51837,"concrete_test":"Recompute all three candidate gap metrics from Table 2: (1) absolute difference, (2) relative difference (real-synth)/real, (3) chance-normalized (real-synth)/(real-chance). If any metric ranks DomainNet as the largest gap, state it explicitly and use it; if none does, the paper must either revise the third conclusion or collect a fourth dataset with a pre-registered gap measure to test whether gap magnitude predicts FT-vs-SM preference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that 'the domain gap can be quantified as the difference between the purely real and purely synthetic results' (Table 2). Using that definition, DomainNet does not exhibit the largest gap; it exhibits the smallest. For the CNN, Cifar-10/CiFake gap = 0.6413 - 0.1430 = 0.4983, LegoBricks gap = 0.5582 - 0.0130 = 0.5452, DomainNet gap = 0.2870 - 0.0353 = 0.2517. Similar ordering holds for MLP and ViT. Yet Section 5 asserts 'DomainNet exhibits a larger domain gap compared to Cifar-10/CiFake and LegoBricks', and Section 6 concludes the optimal strategy could be chosen based on a quantifiable domain gap. Because the stated metric contradicts the qualitative claim, the third conclusion is not supported by the reported data. The issue is load-bearing: the paper's actionable advice (choose FT for small gaps, SM for large gaps) rests on this domain-gap/strategy interaction, and the only quantitative support offered is the baseline accuracy difference, which points the other way. Alternative gap definitions (relative or chance-normalized) also do not place DomainNet first, as its synthetic accuracy is near chance while its real accuracy is low in absolute terms, so the metric needs a task-difficulty correction before any comparison across datasets is valid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a benchmark study comparing two mixed training strategies for hybrid real/synthetic datasets: simple mixed (SM), in which real and synthetic data are trained together, and sequential fine-tuning (FT), in which a model is pretrained on synthetic data and then fine-tuned on real data. Using three network architectures (MLP, CNN, ViT) and three hybrid datasets generated by different mechanisms (GenAI, CAD, hand drawings), the authors sample eleven synthetic-to-real proportions and report test accuracies over ten repetitions. The central empirical claim is that FT usually outperforms SM (635 of 810 individual comparisons, 69 of 81 averaged configurations), with a notable counter-example for CNNs on DomainNet, where SM performs better. The paper further claims that the optimal strategy can be selected based on a quantifiable domain gap, defined as the difference between purely real and purely synthetic baseline accuracies.","tokens_in":11760,"tokens_out":4603,"duration_ms":52007,"significance":"The paper contributes a controlled and fairly extensive benchmark: dataset size is held constant across ratios, class balance is enforced, baselines are external, and the full code and numerical results are made available. The repeated runs and the cross-dataset, cross-architecture design are genuine strengths, and the reported FT-over-SM advantage with a documented counter-example is a useful practical data point. However, the paper's third conclusion—that the optimal strategy can be chosen from a quantifiable domain gap—is not supported by the paper's own operationalization of that gap, which is a load-bearing issue for the advertised actionable guidance. The mechanistic explanation for the CNN/DomainNet exception is also presented without supporting evidence, and the headline comparisons lack uncertainty quantification.","major_comments":[{"comment":"Section 4 states that 'the domain gap can be quantified as the difference between the purely real and purely synthetic results' (Table 2). Under that definition, DomainNet has the smallest gap, not the largest: for the CNN, Cifar-10/CiFake gap = 0.6413 − 0.1430 = 0.4983, LegoBricks gap = 0.5582 − 0.0130 = 0.5452, and DomainNet gap = 0.2870 − 0.0353 = 0.2517; the same ordering holds for the MLP and ViT. Yet Section 5 asserts 'DomainNet exhibits a larger domain gap compared to Cifar-10/CiFake and LegoBricks', and Section 6 concludes that the optimal strategy could be chosen based on a quantifiable domain gap. The stated metric contradicts the qualitative claim, so the third conclusion is unsupported by the reported data. The authors must either revise the gap measure to one that actually orders the datasets as claimed, or present the DomainNet result purely as an architecture–data interaction without linking it to gap magnitude.","section":"Sections 4–6"},{"comment":"The headline comparison counts (635/810 and 69/81) are reported without confidence intervals, standard deviations, or significance tests. Several boxplots, notably Fig. 5 for CNN/DomainNet and many appendix figures, show largely overlapping distributions between SM and FT at individual ratios, so it is unclear whether the aggregate counts reflect reliable differences or are dominated by small, noisy effects. Given that each configuration was repeated ten times, paired tests or confidence intervals on the per-configuration accuracy differences should be provided to support the first conclusion that 'FT is a preferable strategy'.","section":"Section 4"},{"comment":"The proposed explanation for the CNN/DomainNet counter-example—that pretraining on simple black-and-white sketches biases early convolutional layers toward edge detectors, leading to ill-conditioned gradients during fine-tuning—is presented as the reason for the results, but no evidence is offered: there are no kernel visualizations, probing experiments, or ablations that would distinguish this mechanism from alternative explanations. As written, this is a plausible post-hoc hypothesis, not a substantiated finding. The authors should either clearly label the discussion as speculative or add targeted analyses to support it, especially since the paper's main contribution is its controlled experimental comparison.","section":"Section 5"}],"minor_comments":[{"comment":"The text says 'Across 89 individual configurations FT surpassed SM in 69 (78%)', but the experimental design yields 81 averaged configurations (3 datasets × 3 architectures × 9 hybrid ratios), matching the '69 of 81' reported in Section 4. The number 89 appears to be a typo and should be corrected.","section":"Section 6"},{"comment":"The reference '(Fig. 3.3)' near the description of DomainNet's synthetic images appears to be an incorrect cross-reference; it should likely point to Fig. 1 or a relevant panel.","section":"Section 5"},{"comment":"The figure captions label the dataset as 'domain' instead of 'DomainNet'; please use the full dataset name for clarity.","section":"Figures 4, 5, 12–14"},{"comment":"The dataset name is written inconsistently as 'Cifake', 'CiFake', and 'Cifar-10/Cifake' in different places; standardize the capitalization and the hybrid dataset label.","section":"Throughout"},{"comment":"There are several typographical and grammatical issues, including 'Therefor' (Section 3.2), 'It's real part' (Section 3.3), 'relaying on' (Section 1), and 'sgriculture' in reference [29]. A careful copyedit is recommended.","section":"Various"},{"comment":"The sentence 'LEGO Bricks [2] for training classification networks was selected as the second dataset' is awkwardly phrased; consider rewording to make the subject and intent clear.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The experimental data and controlled protocol are valuable, and the FT-over-SM result with a well-documented counter-example is a useful contribution. However, the paper's most general actionable claim—strategy selection from a quantifiable domain gap—is contradicted by the paper's own metric in Table 2. This is a substantive issue that affects the conclusion, but it is correctable by reframing the DomainNet result or by developing a defensible gap measure. I recommend major revision rather than rejection, as the underlying benchmark remains sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThis paper is a genuinely controlled benchmark of two mixed training strategies, FT and SM, across three architectures and three synthetic-to-real datasets, with 1,800 trained networks. That is real work and mostly well done. The central empirical finding—FT beats SM in 635/810 cases—is clear and consistent, and the CNN/DomainNet counter-example is honest and interesting. The diminishing-returns result, where the first 10% of real data gives the biggest jump, is practically valuable and looks robust across the plots. So the paper earns its place as a reference point for anyone navigating synthetic/real training.\n\nThe soft spots are in the interpretation, not the experiment. The paper explicitly defines the domain gap in Section 4 as the difference between purely real and purely synthetic test accuracy (Table 2). Under that definition, DomainNet has the smallest gap, not the largest: for CNNs the gap is 0.252, versus 0.498 for Cifar-10/CiFake and 0.545 for LegoBricks. The ordering is the same for MLP and ViT. Yet Section 5 claims “DomainNet exhibits a larger domain gap” and Section 6 concludes that strategy choice could be based on a quantifiable domain gap. That is an internal contradiction. The stress-test note checks out. Alternative normalizations (relative or chance-corrected) don’t rescue the ordering either. So the paper’s third conclusion is not supported by its own data. This is load-bearing because that conclusion is the paper’s most actionable advice.\n\nOther concerns are more minor. No statistical significance tests or confidence intervals are reported; with ten repetitions and overlapping accuracies, some FT advantages could be noise. The “first 10% real data” claim is descriptive, not formally tested. And the abstract overclaims generality to “any ANN”—the study covers three architectures on image classification, which is a start, not universal.\n\nWho is this for? Practitioners working with synthetic data who want guidance on FT vs SM, and benchmarks looking for a solid experimental template. It deserves a serious referee. The main finding is useful, and the flaw in the third conclusion is fixable: either drop that conclusion or replace it with a more modest statement about visual similarity, and do the necessary metric work.\n\nRecommendation: send to peer review, ask for a revision that addresses the domain-gap contradiction and adds at least basic statistical reporting.","headline":"A solid, controlled benchmark confirming FT usually beats SM, but the paper's own domain-gap metric contradicts its third conclusion, so that part needs major revision.","tokens_in":12287,"tokens_out":4576,"would_cite":false,"duration_ms":44236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning usually beats simple mixed training for hybrid real-and-synthetic data, but a large domain gap can reverse the result.","keywords":["hybrid datasets","mixed training","fine-tuning","simple mixed","domain gap","synthetic data","image classification","benchmark"],"falsifier":"Take a fourth hybrid dataset whose synthetic images are visually close to the real ones but whose synthetic-only baseline accuracy is nearly as low as the DomainNet sketches, a large measured gap with a small visual gap. If FT does not beat SM on that dataset, the proposed link between domain-gap size and strategy choice fails; alternatively, compute the correlation between the baseline gap and the FT-minus-SM advantage across many datasets and check whether a single threshold reliably separates the two regimes.","tokens_in":11286,"feed_emoji":"📊","tokens_out":6232,"duration_ms":63889,"temperature":0.7,"pith_summary":"The paper asks which of two ways of mixing synthetic and real images—training on both at once (simple mixed, SM) or pretraining on synthetic data and then fine-tuning on real data (FT)—better closes the reality gap in image classification. Across three datasets whose synthetic parts come from generative AI, CAD renderings, and hand-drawn sketches, three architectures, and eleven synthetic-to-real ratios, FT outperformed SM in 635 of 810 individual comparisons and in 69 of 81 averaged configurations. The one clear reversal is a CNN on the hand-drawn DomainNet sketches, where SM wins. The authors conclude that FT is usually the better strategy when the synthetic data closely mimics the real domain, and that the choice between strategies may be driven by a quantifiable domain gap. If right, this gives practitioners a rule of thumb: prefer fine-tuning for realistic synthetic data, and consider simple mixing when the gap is large.","feed_headline":"Fine-tuning beats mixed training on 69 of 81 configurations","feed_subtitle":"A three-dataset benchmark shows the winning strategy depends on how large the synthetic-real gap is.","key_machinery":"The load-bearing mechanism is a controlled benchmark design that isolates the mixing strategy from other variables. For each of three hybrid datasets, the authors create eleven equal-sized subsets spanning synthetic proportions from 0% to 100%, match class distributions across subsets, keep total dataset size constant, and train vanilla multilayer perceptron (MLP), convolutional neural network (CNN), and vision transformer (ViT) models with plain stochastic gradient descent and no augmentation or regularization. The two strategies are simple mixed (SM), which samples both domains jointly in every batch, and fine-tuned (FT), which pretrains only on synthetic data with early stopping on the real validation set and then retrains on the real part. The domain gap is operationalized as the difference between purely real and purely synthetic baseline accuracy. That operationalization, together with the controlled ratio sweep, is what lets the paper attribute performance differences to the strategy rather than to dataset size, class imbalance, or pretraining on external data.","core_discovery":"The central discovery is that fine-tuning is generally, but not universally, superior to simple mixed training for hybrid real/synthetic image datasets, with the exception shaped by both architecture and the size of the domain gap. Using equally sized subsampled hybrid datasets with synthetic-to-real ratios from 10% to 90%, the study finds FT ahead in 69 of 81 averaged configurations, and in 635 of 810 individual comparisons. Gains from adding real data follow diminishing returns: the first 10% of real data yields the largest accuracy jump, and each further increase gives less. On the two datasets whose synthetic images were designed to resemble the real ones (CiFake from Stable Diffusion and CAD-rendered LEGO bricks), FT consistently wins; on DomainNet, where synthetic data are black-and-white sketches, a CNN trained with SM beats FT. The paper interprets this as evidence that the optimal mixing strategy interacts with the domain gap, and suggests the strategy could in principle be chosen ahead of time from a quantifiable gap measure.","pith_inferences":["If the proposed domain-gap rule generalizes, any existing real/synthetic dataset pair plus a few baseline runs would be enough to check whether the FT-minus-SM gap correlates with the synthetic-versus-real baseline gap, giving a cheap way to validate the rule on new domains.","The kernel-level explanation for the CNN/DomainNet reversal, that early layers lock onto binary edge detectors during synthetic pretraining, predicts that lighter pretraining, domain-randomized synthetic images, or kernel reinitialization before fine-tuning should reduce or remove the SM advantage.","The diminishing-returns curve implies an economic rule for data collection: the marginal value of a real image is highest when the current real fraction is near zero, which could guide active learning or data-budgeting decisions.","Because the benchmark uses unregularized, augmentation-free networks, production models with normalization, augmentation, and modern optimizers may show a different ordering of the two strategies than the one reported here."],"forward_implications":["Practitioners mixing realistic synthetic renderings with real images should default to fine-tuning over joint training, especially when the synthetic fraction is high.","A small real-data addition, around 10% of the training set, captures most of the performance benefit, so budgets can start there before collecting more real samples.","When the synthetic domain is visually far from the real domain, as with sketches, simple mixed training can beat fine-tuning, particularly for CNNs, so the default should be re-evaluated per dataset.","The optimal strategy can in principle be selected ahead of training from a quantifiable domain-gap measure, turning trial-and-error into a more systematic workflow."],"supporting_citations":[{"why":"Supplies the GenAI-synthetic half of the Cifar-10/CiFake hybrid dataset.","marker":"[1]"},{"why":"Provides the real and CAD-synthetic LEGO brick images used as the second hybrid dataset.","marker":"[2]"},{"why":"Defines the vision transformer architecture trained in the benchmark.","marker":"[5]"},{"why":"Defines the convolutional architecture whose DomainNet behavior produces the SM counter-example.","marker":"[8]"},{"why":"Supplies the real half of the Cifar-10/CiFake hybrid dataset and its class structure.","marker":"[11]"},{"why":"Supplies the CAD program used to render the synthetic LEGO brick images, fixing the CAD generation mechanism.","marker":"[13]"},{"why":"Earlier comparison of SM and FT on synthetic/real object detection that concluded FT is better, the result this study extends and qualifies.","marker":"[15]"},{"why":"Provides the Real and Quickdraw domains used as the third hybrid dataset with a large domain gap.","marker":"[16]"},{"why":"Describes the latent diffusion model used to generate CiFake, fixing the GenAI generation mechanism.","marker":"[20]"},{"why":"Previous SM-vs-FT comparison on a synthetic object-detection dataset whose confounds motivated the controlled design here.","marker":"[26]"}],"fun_headline_variants":["Fine-tuning beats mixed training in 69 of 81 configs","First 10% real data yields the largest accuracy jump","Domain gap size decides: fine-tuning vs mixed training","Fine-tuning wins unless domain gap is large","Mixed training edges fine-tuning on high-gap datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's actionable conclusion rests on assuming that three datasets, one per synthetic generation mechanism, represent synthetic data generally, and that the baseline accuracy gap between purely synthetic and purely real training measures the domain gap that decides which mixing strategy wins.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning beats mixed training in 69 of 81 configs","First 10% real data yields the largest accuracy jump","Domain gap size decides: fine-tuning vs mixed training","Fine-tuning wins unless domain gap is large","Mixed training edges fine-tuning on high-gap datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2719,"prompt_tokens":913,"completion_tokens":1806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":1728}},"tokens_in":529,"tokens_out":1806,"duration_ms":15125,"temperature":1.0,"reasoning_tokens":1728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:24:07.924241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fourth hybrid dataset whose synthetic images are visually close to the real ones but whose synthetic-only baseline accuracy is nearly as low as the DomainNet sketches, a large measured gap with a small visual gap. If FT does not beat SM on that dataset, the proposed link between domain-gap size and strategy choice fails; alternatively, compute the correlation between the baseline gap and the FT-minus-SM advantage across many datasets and check whether a single threshold reliably separates the two regimes.","supporting_citations":[{"cited_title":"IEEE Access12, 15642–15650 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the GenAI-synthetic half of the Cifar-10/CiFake hybrid dataset."},{"cited_title":"https://doi.org/10.34808/RCZA-JY08","cited_arxiv_id":null,"evidence_quote":"Provides the real and CAD-synthetic LEGO brick images used as the second hybrid dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CAD program used to render the synthetic LEGO brick images, fixing the CAD generation mechanism."},{"cited_title":"In: 33rd British Machine Vision Confer- ence 2022, BMVC 2022, London, UK, November 21-24, 2022","cited_arxiv_id":null,"evidence_quote":"Previous SM-vs-FT comparison on a synthetic object-detection dataset whose confounds motivated the controlled design here."}],"review_version":1}