{"id":"728949e5-e30a-4d7b-8781-6654e103e65d","arxiv_id":"2506.17165","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"As the share of GAN-generated brain MRI images in training data rises, CNN tumor classification accuracy falls, with a small 10% synthetic addition giving the best 95.2% accuracy.","lead":"This study mixes real and GAN-generated brain MRI images in different proportions to train a CNN for tumor detection, and finds that more synthetic data gradually lowers accuracy. The best result occurs with a small 10% synthetic share, suggesting GANs help only when used sparingly.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal split counts are contradictory (1,000 + 3,000 + 500 = 4,500 from 3,000 images; test later said 1,000); possible test-set overlap with GAN/CNN training makes Table 2 unverifiable.","rationale":"The reader's weakest assumption is exactly the data-split inconsistency, and I agree that it is the most load-bearing issue: every number in Table 2 depends on the test set being unseen. A leak between GAN training and test would not necessarily destroy the qualitative trend, but it would invalidate the specific 95.2% optimum and could change the shape of the ratio-response curve. The check I propose settles this without needing new architecture work. If the splits are disjoint, the paper still lacks statistical support (single runs, no error bars, no GAN fidelity metric), so the existing CONDITIONAL verdict remains the right call; if overlap exists, the experimental section must be redone before the claim can be evaluated. I therefore keep the reader's verdict unchanged rather than escalating, because the concern is concrete and repairable.","tokens_in":7108,"tokens_out":6703,"duration_ms":74924,"concrete_test":"Obtain from the authors the exact image identifiers or file paths assigned to the GAN training set, CNN training set, CNN validation set, and test set used to produce Table 2. Compute pairwise intersections. If any intersection is non-empty, rerun the full 11-ratio sweep with a held-out test set of 1,000 real images excluded from all GAN training and all CNN training/validation, then compare the recomputed Table 2 to the published one. If all intersections are empty, the concern is resolved; the remaining limitation is the lack of multiple seeds and error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 2 presumes a fixed, independent real test set. That presumption is not internally supported. Section 2.1 says the 3,000-image yes/no partition was split into 1,000 images for CNN training, 3,000 for GAN training, and 500 for testing; these numbers sum to 4,500 and cannot all come from 3,000 images. Section 2.4 then says testing used 1,000 real images, and Section 3 says each training configuration used 800 training / 200 validation images, so the CNN-side data alone is 2,000 images if test is 1,000. If the GAN really was trained on all 3,000 images as Section 2.1 states, then the test set overlaps the GAN training set. Such overlap is not a cosmetic issue. A GAN trained on test images can memorize or reproduce test-like features, so a CNN trained on its synthetic outputs may be evaluated on data whose distribution has already been seen in training. That would directly bias the comparison across GAN ratios: higher synthetic fractions could appear better (or worse) than they would with a clean held-out test set. If instead the CNN training or validation split overlaps the test split, every row of Table 2 is invalid. Either way, the claimed 10%-GAN optimum and the gradual-decline trend cannot be assessed until the actual partition is specified and proven disjoint.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a DCGAN on the BR35H brain MRI dataset, generates 1,000 synthetic images, and constructs 11 training sets with real-to-GAN ratios from 100:0 to 0:100. A custom CNN with three convolutional layers is trained on each mixture and evaluated on a real test set. The reported best performance is at 10% GAN (95.20% accuracy), with performance generally declining as the synthetic fraction increases, down to 65.40% at 100% GAN. The authors conclude that small amounts of GAN data can regularize training, while heavy reliance on synthetic data hurts generalization.","tokens_in":7375,"tokens_out":3347,"duration_ms":40583,"significance":"If the central trend holds, the paper addresses a practically important question for medical imaging: how much synthetic data should be mixed with real data when training classifiers. The systematic sweep over 11 ratios, the presentation of full metric tables, and the reproducible architecture description are strengths. However, the core quantitative claims currently rest on an internally inconsistent data-split description and on single-run results without uncertainty estimates. The finding that 10% GAN is better than 0% GAN is a 0.5 percentage-point difference that may be within run-to-run noise, so the paper's main recommendation is not yet established at the level of rigor expected for a journal publication.","major_comments":[{"comment":"The quality of the synthetic images is not quantitatively assessed. The paper relies only on visual inspection and loss curves, but no Frechet Inception Distance, Inception Score, or similar measure is reported. Given that the central argument is about how synthetic data quality affects downstream CNN performance, a quantitative quality measure would substantially strengthen the interpretation of the ratio-dependent results. If such measures are unavailable, the authors should at least acknowledge this limitation explicitly in the Discussion.","section":"§2.2, §3.1"}],"minor_comments":[{"comment":"The sentence 'The dataset was partitioned into 1,000 images for further training, 3,000 for GAN training, and 500 reserved as an independent test set' is also internally unclear because 'further training' is not defined; please rewrite to give exact counts for each role and clarify that all subsets are disjoint.","section":"§2.1"},{"comment":"The notation alternates between 'DCGAN' and 'DC-GAN'; please use one consistent abbreviation throughout.","section":"Abstract and §2.2"},{"comment":"The column header 'Data Distribution' would be clearer as 'Real:GAN Ratio' or 'Percentage of GAN Data'.","section":"Table 2"},{"comment":"Line 11 says 'Optionally evaluate on validation set and store metrics,' but the paper does not describe how validation is used (e.g., for early stopping or model selection). Please specify whether validation influenced the final model or was only used for reporting.","section":"Algorithm 1"},{"comment":"Several references are missing spaces or have typographical issues (e.g., 'Synthetic Data· Brain Tumor Classification' in the keywords line and the reference formatting in Ref. [8]). Please run a final copyediting pass.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical study whose central question is of interest to the community, but the data-split inconsistency is a serious correctness risk that must be resolved before the results can be trusted. If the authors can provide a coherent, disjoint split and ideally release the code and data partition, the paper could become a useful contribution. I did not find evidence of deliberate misreporting, but the current text does not allow a reader to verify the independence of the test set. The lack of multi-run statistics further weakens the headline 10%-GAN conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. My take: the 11-point DCGAN ratio sweep on BR35H is a practical, useful addition to the GAN-augmentation literature, but the data-split inconsistencies are serious enough that the quantitative claims should not be trusted until fixed.\n\nWhat is actually new: most prior work compares one augmentation fraction against none, or reports a single synthetic/real mix. This paper traces the full curve from 0% to 100% GAN on a fixed CNN architecture, with accuracy, precision, recall, F1, and AUC at each point. The fully synthetic baseline (65.4% accuracy) and the monotone decline beyond 10% give a concrete mapping of the trade-off for this dataset. That is worth having, even if the qualitative direction is not surprising.\n\nWhere it falls down: the split arithmetic does not hold. Section 2.1 says the 3,000 images were partitioned into 1,000 for CNN training, 3,000 for GAN training, and 500 for testing. Those numbers sum to 4,500. Section 2.4 then says the test set contains 1,000 real images, and Section 3 says each configuration used 800 training / 200 validation. You cannot fit all of that into 3,000 images with disjoint sets. If the GAN really trained on all 3,000, the test set overlaps GAN training. If not, the actual partition is unspecified. Either way, Table 2 is unverifiable, and the claimed 10% optimum could be an artifact of leakage. This is not a cosmetic complaint; it directly affects every row of the central comparison.\n\nOther soft spots: single-run experiments with no error bars, no quantitative evaluation of GAN output quality (only visual inspection), and no released code or data. The regularization explanation in Section 3.3 is attached post hoc to the observed argmax, so it is interpretation rather than a tested hypothesis. The citation pattern looks reasonable; the self-citations are to related recent work and not load-bearing.\n\nFor all that, the paper shows sensible experimental intent and the broad conclusion (small synthetic fractions help, large ones hurt) is consistent with the literature. It deserves a serious referee who will require a clean re-run with explicitly disjoint splits, multiple seeds, and quantitative GAN fidelity metrics. As it stands, I would not cite it, and I would not rely on the numbers. But it is a good reading-group case study on why split documentation matters, and with the arithmetic fixed it could be a solid empirical contribution.\n\nRecommendation: send to peer review only after the authors resolve the split contradictions; desk rejection would be premature given the usefulness of the sweep.","headline":"A useful GAN-ratio sweep undermined by inconsistent data-split arithmetic; the trend is plausible, but the headline numbers are not currently reproducible.","tokens_in":7981,"tokens_out":2259,"would_cite":false,"duration_ms":28017,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding 10% synthetic MRI images yields the best tumor-classification accuracy in this study's experiments.","keywords":["brain tumor classification","GAN augmentation","DCGAN","synthetic medical images","convolutional neural network","data augmentation ratio","MRI"],"falsifier":"Compare the actual file names in the BR35H 'yes' and 'no' folders with the images assigned to GAN training, CNN training, and testing. If any real test image was also used to train or validate the GAN or the CNN, the reported accuracy numbers cannot be interpreted as generalization to unseen real scans.","tokens_in":1466,"feed_emoji":"🧠","tokens_out":3416,"duration_ms":77826,"temperature":0.7,"pith_summary":"The paper seeks to determine how much GAN-generated data a CNN can tolerate when learning to classify brain tumor MRI scans. It reports that the best result comes from a small 10% synthetic share, yielding 95.2% accuracy on real test images, slightly above the 94.7% accuracy of a fully real training set. Beyond that point, accuracy falls steadily to 65.4% when all training images are synthetic. The point matters because medical datasets are often too small, and knowing the safe ceiling for synthetic augmentation guides how far GANs can stretch limited data.","feed_headline":"10% synthetic MRI beats all-real training in tumor CNN","feed_subtitle":"Beyond 10% synthetic share, accuracy slides from 95.2% to 65.4% on real brain scans.","key_machinery":"The machinery is a DCGAN (a deep convolutional generative adversarial network) trained over 1,000 epochs to synthesize 1,000 brain MRI images, which are then blended with real images into 11 training sets covering real-to-GAN ratios from 100:0 to 0:100. Each training set contains exactly 1,000 images, and a fixed custom CNN with three convolutional layers, ReLU activations, max-pooling, dropout, and a sigmoid output is trained on each and evaluated on a real test set. The ratio grid is the load-bearing device: it makes the performance decline visible as a function of synthetic share, and the evaluation metrics (accuracy, precision, recall, F1-score, AUC) tie that decline to the model's sensitivity to domain shift.","core_discovery":"The paper's central claim is that CNN performance on real brain tumor MRI is sensitive to the proportion of GAN-generated images mixed into the training set. With 900 real and 100 synthetic images (10% GAN), the custom CNN reaches 95.20% accuracy, 94.31% precision, 96.20% recall, 95.25% F1-score, and an AUC of 0.98, slightly better than the 94.70% accuracy of fully real training. Every further increase in synthetic share lowers accuracy, down to 65.40% at 100% GAN, with AUC falling to 0.71. The authors interpret this as small-scale GAN augmentation acting as a regularizer, while heavy reliance on synthetic data introduces a domain shift that hurts generalization to real scans.","pith_inferences":["Editorial inference: because the reported partition sizes (1,000 for CNN training, 3,000 for GAN training, and 500 or 1,000 for testing) do not sum to the 3,000 available images, the performance differences could partly reflect data leakage rather than the synthetic ratio itself.","Editorial inference: the 0.5 percentage-point gap between the 10%-GAN and 0%-GAN conditions is small, so the claimed regularizing benefit may not survive repeated runs with different random seeds.","Editorial inference: a direct testable extension would repeat the ratio grid on a fully disjoint split, with held-out real images never shown to the GAN or the CNN, to confirm whether the monotone decline in accuracy is robust.","Editorial inference: the optimal synthetic share likely depends on GAN fidelity and dataset size; with higher-quality generators, the performance peak could shift toward larger synthetic proportions."],"forward_implications":["A small dose of GAN-generated images, around 10% of the training set, can slightly improve accuracy over all-real training, suggesting a mild regularizing effect.","Increasing the synthetic share beyond roughly 50% leads to a pronounced drop in accuracy, recall, and F1-score, so synthetic data should not dominate training in this setting.","Even at 100% synthetic training, the model retains some discriminative ability (65.4% accuracy, AUC 0.71), meaning GAN images carry real signal but not enough to replace real scans.","The monotone decline across all metrics gives practitioners a concrete warning that domain shift from synthetic to real data grows with synthetic proportion.","Future generators, such as diffusion models or domain-adapted GANs, would need to close the remaining realism gap before higher synthetic ratios become safe."],"supporting_citations":[{"why":"Supplies the BR35H brain MRI dataset whose 'yes' and 'no' folders provide the 3,000 real images used throughout the study.","marker":"[8]"},{"why":"Provides the DCGAN architecture used to generate the 1,000 synthetic tumor and non-tumor images.","marker":"[16]"},{"why":"Introduces the GAN generator-discriminator formalism and minimax objective on which the paper bases its synthetic image generation.","marker":"[7]"},{"why":"Offers prior evidence on how much of real-data performance GAN-generated medical images can support, serving as the baseline the paper's ratio comparison extends.","marker":"[3]"},{"why":"Documents a custom CNN benchmark on the same BR35H dataset, providing the prior performance level the paper's CNN and evaluation protocol build on.","marker":"[14]"}],"fun_headline_variants":["10% synthetic MRI maximizes tumor CNN accuracy","Too much GAN data degrades brain tumor classifier","Small GAN dose beats all-real for tumor MRI CNN","Proportional GAN synthesis: 10% ideal for brain scans","Tumor CNN peaks at 10% GAN images, then falls"],"cache_read_input_tokens":9984,"weakest_assumption_plain":"The load-bearing premise is that the real images used for GAN training, CNN training, and testing are disjoint; the paper's stated counts (1,000 for CNN training, 3,000 for GAN training, and 500 or 1,000 test images) exceed the 3,000 images in the dataset, so overlap is possible and would make the test set not independent.","fun_headline_variants_meta":{"raw":{"variants":["10% synthetic MRI maximizes tumor CNN accuracy","Too much GAN data degrades brain tumor classifier","Small GAN dose beats all-real for tumor MRI CNN","Proportional GAN synthesis: 10% ideal for brain scans","Tumor CNN peaks at 10% GAN images, then falls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1620,"prompt_tokens":914,"completion_tokens":706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":624}},"tokens_in":530,"tokens_out":706,"duration_ms":8016,"temperature":1.0,"reasoning_tokens":624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:30:12.116112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the actual file names in the BR35H 'yes' and 'no' folders with the images assigned to GAN training, CNN training, and testing. If any real test image was also used to train or validate the GAN or the CNN, the reported accuracy numbers cannot be interpreted as generalization to unseen real scans.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BR35H brain MRI dataset whose 'yes' and 'no' folders provide the 3,000 real images used throughout the study."},{"cited_title":"Scientific Data 11, 259 (2024)","cited_arxiv_id":null,"evidence_quote":"Offers prior evidence on how much of real-data performance GAN-generated medical images can support, serving as the baseline the paper's ratio comparison extends."},{"cited_title":"Heliyon10(20), e38997 (2024)","cited_arxiv_id":null,"evidence_quote":"Documents a custom CNN benchmark on the same BR35H dataset, providing the prior performance level the paper's CNN and evaluation protocol build on."}],"review_version":1}