{"id":"e06d6d36-6194-423f-8476-1b459f595592","arxiv_id":"2502.03825","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding GAN-generated synthetic MRI to U-Net training data degrades brain tumor segmentation performance, but the paper's evidence for a monotonic, significant decline is weakened by contradictory table values and confounded experimental design.","lead":"This study trains U-Net brain tumor segmenters on datasets that mix real MRI scans with GAN-generated synthetic MRI, and reports that segmentation quality falls as the share of synthetic images rises. It is a cautionary empirical result for medical teams considering synthetic data augmentation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported poisoning effect is confounded: D′(p)=D∪X′ varies training-set size and never specifies how synthetic MRIs are labeled, so 'Dice decreases with p' cannot be attributed to synthetic proportion.","rationale":"The reader's weakest assumption flags the D′(p) construction, and I agree this is the load-bearing issue. The paper's own Algorithm 1 and §4.1 show D′(p)=D∪X′; no step describes label assignment for synthetic images or controls total training size. Both omissions are directly tied to the central claim because p is the only explanatory variable the paper names. If synthetic labels are inherited from CT/MRI pairs, the GAN's anatomical edits make the labels partially wrong, and the amount of wrong-label data grows with p; if labels are generated automatically, their quality is unknown. Either way the dose-response trend is not evidence for 'synthetic proportion' as an intrinsic data property. The non-monotonic rows in Table 1 (accuracy flat, sensitivity peaking at p=50%, Dice rising from 16.67% to 33.33%) further weaken the causal reading. A fixed-size, explicitly labeled re-run is the minimal check that would separate the effect of p from label/size confounds. The paper deserves credit for public data, standard metrics, and reporting standard deviations; those do not rescue the uncontrolled protocol. Verdict remains REJECT; no adjustment beyond the reader's.","tokens_in":10835,"tokens_out":5249,"duration_ms":56715,"concrete_test":"Re-run the §4.1 experiment with total training-set size fixed at N=150 for every condition: for each p, sample (1−p)N real MRI/mask pairs from D and pN synthetic images from X′, and assign each synthetic image a mask by registering/transferring the ET label from the real MRI paired with its CT input (or by manual annotation), validating a random subset for alignment. If Dice(p) does not decline monotonically under this controlled protocol, the reported 'poisoning rate' effect is an artifact of total dataset size or unvalidated label assignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that the proportion p of synthetic MRI data degrades U-Net segmentation—depends on isolating p in the §4.1 protocol. The protocol does not do so. Algorithm 1 constructs D′(p) ← D ∪ X′ with no subsampling or weighting, so the poisoned training sets have different total sizes (for fixed D, p=83.33% contains roughly five times as many training samples as p=16.67%) and, more importantly, every synthetic sample enters with an unstated label. The paper never says how the ET masks are assigned to the 150 GAN-generated T1-Ce images. If masks are copied from paired source MRIs or CTs, any geometric or contrast inconsistency introduced by the GAN becomes label noise whose magnitude scales with p; the observed Dice drop would then measure label misalignment, not 'synthetic proportion' as a property of the images. The data themselves already strain the monotonic claim: Dice is 0.8939 at p=0, 0.8650 at p=16.67, then 0.8937 at p=33.33 before falling to 0.7474 at p=83.33; accuracy is essentially flat (0.9983→0.9967) and sensitivity rises until p=50%. These patterns are consistent with random variation or label-noise effects, not with a clean dose–response. Because the protocol changes two independent variables at once and omits the label-generation step, the abstract's headline drop cannot be attributed specifically to synthetic data proportion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript investigates whether adding GAN-generated synthetic T1-contrast-enhanced (T1-Ce) MRI scans to the training set degrades U-Net brain tumor segmentation. The authors use a cross-domain image translation model from Xie et al. (2023) to generate 150 synthetic T1-Ce images, train a baseline U-Net on the real BraTS2021 dataset, and train 'poisoned' U-Nets on mixtures with synthetic proportions p = 16.67% to 83.33%. Performance is measured with Dice, Jaccard, accuracy, and sensitivity on a real MRI test set. The paper concludes that increasing synthetic data proportion causes significant segmentation degradation, citing a Dice drop from 0.8937 at p=33.33% to 0.7474 at p=83.33%, and states that accuracy and sensitivity show similar downward trends.","tokens_in":11164,"tokens_out":7214,"duration_ms":57544,"significance":"The paper addresses a timely and practically relevant question: whether synthetic data augmentation in medical imaging can act as a form of data poisoning. Its use of the public BraTS2021 benchmark and standard segmentation metrics is appropriate, and the metric equations are clearly stated. The appendix provides qualitative examples of synthetic MRIs. However, the study's internal validity is severely compromised: the poisoned-dataset construction confounds synthetic proportion with total training-set size, and the labeling procedure for synthetic images is never specified. The reported numbers in Table 1 also do not support the claimed monotonic degradation, making the central causal conclusion unsupported as presented.","major_comments":[{"comment":"The construction D′(p) ← D ∪ X′ confounds the synthetic proportion p with the total training-set size, and the paper never specifies how p is realized. For a fixed D of 150 real scans and X′ of 150 synthetic scans, p=50% corresponds to D ∪ X′ (300 images), whereas p=16.67% and p=83.33% require either subsampling D or X′ or a different mixing procedure that is not described. Any performance difference across p could therefore be caused by the number of training examples rather than by the proportion of synthetic content. In addition, the paper does not state how the enhanced tumor (ET) masks are assigned to the 150 synthetic T1-Ce images. If the masks are inherited from the source CT or a paired real MRI, GAN-induced geometric or contrast inconsistencies become label noise that scales with p, providing an alternative explanation for the observed Dice drop. The central causal claim requires controlling both variables, and the current protocol does not do so.","section":"Section 4.1, Algorithm 1"},{"comment":"The data in Table 1 do not support the abstract's claim of 'significant performance degradation as synthetic data increases.' Dice is 0.8939 at p=0, falls to 0.8650 at p=16.67%, then rises to 0.8937 at p=33.33% before declining to 0.7474 at p=83.33%; the 16.67% condition is lower than the 33.33% condition, contrary to a monotonic dose-response relationship. Sensitivity increases from 0.9136 (p=0) to 0.9292 (p=50%), and accuracy is almost constant (0.9983, 0.9981, 0.9979, 0.9978, 0.9967) apart from an anomalous 0.9638 at p=16.67%. Consequently, the statement that 'accuracy and sensitivity exhibit similar downward trends' is contradicted by the table.","section":"Table 1"},{"comment":"The abstract's headline comparison selects the starting point after the initial non-monotonic variation: the drop is reported from 0.8937 (33.33% synthetic) to 0.7474 (83.33% synthetic), which excludes the p=16.67% row. If the trend were presented from the baseline p=0, the pattern would be non-monotonic (0.8939, 0.8650, 0.8937, 0.8572, 0.8146, 0.7474). The claim that models with p ≤ 33.33% 'maintain relatively stable performance' is also inconsistent with the 0.8650 Dice at p=16.67%, which is already below the baseline. The paper should present the full data without selective endpoints or revise the conclusion accordingly.","section":"Abstract and Section 4.2"},{"comment":"No statistical support is provided for the word 'significant.' The table reports means and standard deviations but no number of independent training runs, p-values, confidence intervals, or effect sizes. The standard deviations are large and overlapping across conditions (e.g., Dice at p=0 is 0.8939±0.1243 vs. 0.8572±0.1580 at p=50%), so the observed differences may not be statistically reliable. Without repeated runs or a significance test, the central conclusion of significant degradation is not established.","section":"Table 1 and Section 4.2"}],"minor_comments":[{"comment":"Figure 3 includes a 'Poisoning Rate: 100%' panel, but Table 1 reports results only up to p=83.33%; the figure should be reconciled with the table or the missing condition should be reported.","section":"Figure 3"},{"comment":"The Introduction describes the GAN as 'a novel GAN-based model (Xie et al., 2023)'; since the GAN is taken verbatim from Xie et al. (2023), it should not be called novel in this work. Please rephrase to avoid implying novelty.","section":"Introduction"},{"comment":"The paper does not describe the U-Net training details (number of epochs, optimizer, learning rate, loss function, data preprocessing, train/validation/test split), which are needed for reproducibility and for assessing whether the observed variability might be due to training randomness.","section":"Section 4.1"},{"comment":"In the Data Poisoning Attack paragraph, the sentence 'poisoning can occur through mislabeled (Tolpegin et al., 2020; Lin et al., 2021), perturbed (Martinelli et al., 2023; Bortsova et al., 2021)' is grammatically incomplete and should be rewritten.","section":"Section 2"},{"comment":"The data lineage is unclear: the GAN is trained on 660 paired CT-MRI datasets, but the relationship of these pairs to the 150 BraTS2021 patients whose T1-Ce scans are used for training is not explained. Please clarify whether the synthetic MRIs are generated from the same patients or from a separate cohort.","section":"Section 4.1"},{"comment":"The appendix shows qualitative examples of fake MRI but provides no quantitative assessment of synthetic image quality (e.g., FID, SSIM, or a reader study), which would help interpret how label misalignment might arise.","section":"Appendix A"}],"recommendation":"reject","confidential_remarks":"This manuscript is largely an empirical report built on an existing GAN (Xie et al., 2023) and a standard U-Net. The primary concern is internal validity: the protocol confounds synthetic proportion with training-set size and does not specify label assignment for synthetic images. Even if these issues were repaired, the current data pattern is non-monotonic and would require re-analysis. The topic could be suitable for a venue on trustworthy medical AI after a substantially revised study, but the present version does not support its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honestly, this paper asks a sensible question—does mixing synthetic MRI into a U-Net training set hurt segmentation?—but the experiments don't answer it. The central claim in the abstract is not supported by the table.\n\nWhat's new: not much. The qualitative finding that out-of-distribution or low-quality synthetic data can degrade segmentation is already in the literature the authors cite. The proportion sweep (16.67% to 83.33%) is a small, incremental parameter scan. They do report standard deviations on four metrics, which is better than nothing, and they use a public dataset.\n\nThe problems are load-bearing. First, the protocol never says how the synthetic MRIs are labeled. The generative model is trained on paired CT-MRI data and produces fake MRI from real CT, but the segmentation masks for those fake images are not described. If masks are inherited from the source CTs or from paired real MRIs, any misalignment becomes label noise that scales with p, and the observed Dice drop could just be measuring that. Second, D'(p) = D ∪ X' means training set size grows with p. At p=83.33% you have roughly five times as many training samples as at p=16.67%, so you're changing two variables at once. Third, the data themselves are non-monotonic: Dice at 16.67% is 0.8650, then rises to 0.8937 at 33.33%, before falling. Accuracy is flat except for one anomalous 0.9638 at 16.67%. Sensitivity at 50% is above baseline. So the 'significant performance degradation as synthetic data increases' claim doesn't hold up.\n\nThe abstract also says accuracy and sensitivity exhibit similar downward trends; they don't. That's an internal inconsistency.\n\nWho is this for? A reader looking for a cautionary tale about sloppy experimental design might get something out of it, but not the medical imaging practitioner who wants guidance on when synthetic augmentation is safe.\n\nRecommendation: desk reject, or send back with a request for major revision that fixes the confounding and the label problem. As is, I wouldn't cite it.","headline":"A useful question undermined by a confounded protocol and unsupported claims.","tokens_in":11693,"tokens_out":3550,"would_cite":false,"duration_ms":35519,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding GAN-generated MRI to a U-Net training set lowers brain-tumor segmentation quality, with Dice falling from 0.8937 at 33% synthetic data to 0.7474 at 83%.","keywords":["synthetic data poisoning","GAN-based MRI synthesis","U-Net brain tumor segmentation","data augmentation risk","BraTS2021","T1-contrast-enhanced MRI","Dice coefficient","label fidelity"],"falsifier":"Train two U-Nets at 83.33% synthetic content with identical total size, one on real MRIs and one on synthetic MRIs paired with the same ground-truth masks; if Dice on the real validation set does not fall to roughly 0.7474 in the synthetic-conditioned model, the effect is not specifically synthetic image content. A cheaper check is to inspect the synthetic masks: if the fake MRIs inherit masks from the source CTs, retraining with manually corrected labels would test whether misalignment, not realism, causes the drop.","tokens_in":10668,"feed_emoji":"🧠","tokens_out":8728,"duration_ms":77216,"temperature":0.7,"pith_summary":"This paper claims that MRI scans produced by a generative adversarial network (GAN) act as a form of data poisoning when mixed into the training set of a U-Net for brain tumor segmentation. The authors train models on progressively contaminated datasets in which synthetic T1-contrast-enhanced MRI make up 16.67% to 83.33% of the data, and they report mean Dice falling from 0.8937 at 33.33% synthetic content to 0.7474 at 83.33%, with Jaccard and sensitivity declining as well. They interpret the decline as evidence that unregulated synthetic augmentation introduces bias and reduces segmentation reliability, and they argue for quality control and hybrid real-synthetic training strategies. The finding matters because synthetic augmentation is widely proposed as a fix for medical data scarcity, and this paper quantifies the downside when it is applied without oversight.","feed_headline":"Fake MRIs in training cut tumor-segmentation Dice by 0.15","feed_subtitle":"Raising GAN-made T1-Ce scans from 33% to 83% of the training set sinks overlap and recall on real brain MRIs.","key_machinery":"The central object is the poisoned training set $D'(p) = D \\cup X'$, where $X'$ are synthetic T1-contrast-enhanced MRI produced by a GAN-based cross-domain CT-to-MRI translation model with a shared encoding-decoding structure and shortest-path regularization. Varying the synthetic proportion $p \\in \\{16.67\\%, 33.33\\%, 50\\%, 66.67\\%, 83.33\\%\\}$ while keeping the real validation set fixed isolates, in the authors' design, the amount of contamination the U-Net is exposed to. The mechanism is downstream: fake images enter training, and the segmentation metrics on real test images—Dice, Jaccard, accuracy, sensitivity—reveal how much the model's tumor boundary predictions drift.","core_discovery":"The paper's central discovery is a quantitative degradation curve: as the proportion of synthetic MRI in the training mixture rises, a U-Net's real-image segmentation quality falls. Formally, the authors hypothesize $\\mathrm{Dice}(M') < \\mathrm{Dice}(M)$, where $M$ is trained on real data only and $M'$ on $D \\cup X'$. Their experiments support this most clearly at high contamination levels: the worst condition, $p = 83.33\\%$, gives Dice 0.7474 and sensitivity 0.7577, against 0.8939 and 0.9136 for real-only training. Accuracy remains near 0.998 throughout, so the damage is concentrated in overlap and recall, i.e., in the model's ability to delineate tumor boundaries rather than to classify background pixels. The authors conclude that synthetic content should be regulated and that controlled low-proportion augmentation may be acceptable.","pith_inferences":["The paper's poisoning account would be sharper if it ruled out label-source effects; a natural follow-up is to compare synthetic MRIs labeled by the generator's source CT against the same synthetic images labeled by a human expert, holding image content fixed.","Because the 16.67% condition shows a mean Dice (0.8650) below the 33.33% condition (0.8937) with notably larger standard deviation, the monotonic-trend reading rests mainly on the later proportions; a safe-threshold claim would need more replicates.","The result suggests that quality control should target semantic fidelity—whether the anatomy in the generated image matches its label—rather than visual realism alone; this is testable by having radiologists rate realism and correlating those ratings with Dice drops."],"forward_implications":["Above 50% synthetic content, segmentation quality drops steeply: at 83.33% the mean Dice falls to 0.7474 and sensitivity to 0.7577, meaning the model misses substantially more tumor tissue.","Up to about one-third synthetic content, mean Dice (0.8937 at 33.33%) stays close to the real-only baseline (0.8939), suggesting a tolerable range for augmentation.","Pixel accuracy remains near 0.998 across conditions, so auditing synthetic augmentation with accuracy alone would miss the damage; overlap and recall metrics are needed.","Medical pipelines should screen synthetic samples and cap their fraction, because unregulated generative augmentation can behave like a silent poisoning attack."],"supporting_citations":[{"why":"Supplies the generative model S: a GAN with shared encoder-decoder and shortest-path regularization that translates real CT into the synthetic T1-Ce MRI used as poisoning samples.","marker":"Xie et al., 2023"},{"why":"Defines the U-Net architecture whose segmentation performance is the measured outcome.","marker":"Ronneberger et al., 2015"},{"why":"Provides the BraTS2021 T1-Ce MRI scans and enhanced-tumor masks forming the real dataset D and the validation set.","marker":"Menze et al., 2014"},{"why":"Establishes the GAN framework underlying the synthesis pipeline.","marker":"Goodfellow et al., 2014"}],"fun_headline_variants":["Synthetic MRI in training drops U-Net tumor Dice by 0.15","Poisoned MRI data: U-Net tumor segmentation degrades sharply","GAN-made MRI in training hurts brain tumor overlap and recall","More synthetic scans, lower Dice: U-Net tumor maps suffer","Rising synthetic MRI ratio cuts tumor Dice to 0.7474"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed performance drop is caused by the synthetic proportion of the training set; the poisoned sets are larger than the baseline $D$, and the paper never states how segmentation masks are assigned to the synthetic MRIs, so label misalignment or dataset size, rather than synthetic content itself, could drive the degradation.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic MRI in training drops U-Net tumor Dice by 0.15","Poisoned MRI data: U-Net tumor segmentation degrades sharply","GAN-made MRI in training hurts brain tumor overlap and recall","More synthetic scans, lower Dice: U-Net tumor maps suffer","Rising synthetic MRI ratio cuts tumor Dice to 0.7474"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3175,"prompt_tokens":996,"completion_tokens":2179,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2085}},"tokens_in":612,"tokens_out":2179,"duration_ms":15109,"temperature":1.0,"reasoning_tokens":2085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:34:44.463393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two U-Nets at 83.33% synthetic content with identical total size, one on real MRIs and one on synthetic MRIs paired with the same ground-truth masks; if Dice on the real validation set does not fall to roughly 0.7474 in the synthetic-conditioned model, the effect is not specifically synthetic image content. A cheaper check is to inspect the synthetic masks: if the fake MRIs inherit masks from the source CTs, retraining with manually corrected labels would test whether misalignment, not realism, causes the drop.","supporting_citations":[{"cited_title":"U-net: Convolutional networks for biomed- ical image segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the U-Net architecture whose segmentation performance is the measured outcome."}],"review_version":1}