{"id":"923a7080-3912-4d88-8da5-0b815d3f2a14","arxiv_id":"2505.22926","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A simplified diffusion model generates synthetic protein images, but classifiers trained on mixes of real and synthetic data fail to generalize, performing far below ResNet baselines on Kaggle test sets.","lead":"This preprint tests whether diffusion-generated synthetic images can improve multi-label classification of protein subcellular localization. In experiments, classifiers trained with the synthetic data performed far worse on unseen test data than simple ResNet baselines, indicating the approach does not help in this setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's near-zero F1 is confounded by an inconsistent Mix Loss objective (Eq. 7 does not train z_mix from Eq. 6) and identical degenerate test scores across all four MixModel variants, so the central negative claim about diffusion augmentation is not supported.","rationale":"The reader identified the weak 32x32 simplified DDPM as the main limitation, but the more load-bearing problem is internal to the evaluation of MixModel. The Mix Loss objective in Eq. (7) does not optimize the prediction head on the mixed logits z_mix defined in Eq. (6), so the model is never trained for the inference procedure it claims to use. This is a concrete, demonstrable inconsistency rather than a question of generator quality. In addition, all four MixModel rows in Table 4 report bit-identical public and private F1 scores despite varying backbone and mixing strategy; this strongly indicates that the test-time predictions are degenerate, making the F1 a property of a trivial predictor rather than of the learned representations. The paper even contains an internal contradiction: Section 7.2.1 claims the simplified DDPM produces visibly different outputs for different classes, while the caption of Figure 6 says the generation results show minimal distinction across labels. If the synthetic images are not label-consistent, then the MixModel trains on mismatched targets, and the negative result is expected regardless of whether diffusion augmentation could work with a better generator. The reader's weak-generator concern is valid but secondary; a weak generator would not explain why four distinct training setups converge to exactly the same test F1. The central claim is therefore not established by the reported experiments. I recommend REJECT for the current manuscript, with the clear path that a corrected Mix Loss objective plus an analysis of test-prediction diversity could produce a more reliable negative result.","tokens_in":6923,"tokens_out":9748,"duration_ms":112443,"concrete_test":"Download the released code and checkpoints, run inference on the Kaggle test set with all four MixModel configurations, and compare the predicted multi-label vectors. Compute the fraction of all-zero predictions and the pairwise agreement of the prediction vectors across configurations; if all four configurations produce identical or all-zero predictions, the reported F1 of 0.019 is a degenerate artifact and Table 4 cannot support any claim about diffusion augmentation.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central negative claim rests entirely on Table 4, but that table does not provide a valid test of diffusion augmentation. In the Mix Loss variant, Eq. (6) defines the prediction logits as z_mix = λ z_r + (1−λ) z_g, while Eq. (7) defines the training loss as L_mix = λ L_r + (1−λ) L_g, where L_r and L_g are BCE losses on the unmixed real and synthetic logits. L_mix does not depend on z_mix, so z_mix receives no gradient and the model is never trained to make accurate mixed-logit predictions. The paper also does not specify how z_mix is formed at test time, where no synthetic counterpart for a test image exists; if a random generated image is mixed in, test predictions are degraded by construction rather than by any property of the synthetic data. Moreover, all four MixModel configurations report exactly the same public (0.01924) and private (0.01955) F1-scores despite using different backbones and different mixing strategies. That is the signature of a degenerate predictor (e.g., a constant output), not of four distinct learned models. The conclusion that diffusion-based augmentation fails is therefore unsupported: the reported failure is confounded with an inconsistent training objective and an unspecified, likely trivial test-time prediction rule.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether diffusion-generated synthetic images can improve multi-label classification of protein subcellular localization, using only the official HPA Kaggle dataset. The authors implement a simplified class-conditional DDPM that is trained at 32x32 resolution and upsample its outputs to 512x512, then train ResNet-based classifiers with two proposed mixing strategies, Mix Loss and Mix Representation. They report validation scores on a 90/10 split and public/private Kaggle leaderboard scores. The central claim is that the proposed MixModel variants fail to generalize to unseen test data, with Kaggle F1-scores around 0.019, far below the ResNet18 BCE baseline (public F1 about 0.457). I find that the central empirical claim is not supported as presented, because the Mix Loss objective is internally inconsistent, the test-time prediction rule is unspecified, and the identical near-zero test scores across all MixModel configurations indicate a degenerate prediction process rather than four independently learned classifiers.","tokens_in":7248,"tokens_out":5736,"duration_ms":59546,"significance":"If the negative result were rigorously established, it would provide a useful caution about naive diffusion-based augmentation in high-resolution biomedical multi-label classification. The paper reports external Kaggle leaderboard scores, which is a strength, and it makes code and generated data available, which supports reproducibility. However, the current significance is limited by the invalid Mix Loss formulation, the unexplained identical test scores, and the lack of confidence intervals or repeated runs. The study also uses a very lightweight 32x32 generator, so the scope of any supported conclusion is restricted to this specific generation setup, not to diffusion augmentation in general.","major_comments":[{"comment":"The Mix Loss objective is internally inconsistent. Eq. (6) defines the prediction logits as z_mix = lambda*z_r + (1-lambda)*z_g, but Eq. (7) defines the training loss as L_mix = lambda*L_r + (1-lambda)*L_g, where L_r and L_g are losses on the unmixed logits. Consequently, z_mix receives no gradient during training and the model is never optimized to make accurate mixed-logit predictions. The paper also does not specify how z_mix is formed at test time, where no paired synthetic image exists. This makes Table 4 an invalid test of diffusion augmentation. Please correct the objective to a loss that depends on z_mix, and specify the exact test-time prediction rule.","section":"Section 5.2.2, Eqs. (6)-(7)"},{"comment":"All four MixModel configurations report exactly the same public F1-score (0.01924) and private F1-score (0.01955), despite different backbones, different mixing strategies, and materially different validation scores. This identical pattern is the signature of a degenerate constant prediction rule, not of four independently trained classifiers. The authors should report the per-configuration test-time outputs and explain why the scores are identical; without this, the failure cannot be attributed to properties of the synthetic data or the mixing strategy.","section":"Table 4"},{"comment":"The simplified DDPM is trained on images downsampled to 32x32 and then upsampled to 512x512, and Figure 6's caption states that the class-conditional generation results show 'minimal distinction across labels.' This directly contradicts the claim in Section 7.2.1 that the simplified DDPM successfully leverages class conditioning and produces visibly different outputs across classes. The paper should reconcile this contradiction and provide quantitative evidence of class-conditional fidelity, such as label-consistency rates measured by a pretrained classifier or FID-style distributional metrics.","section":"Section 5.1 and Figures 4-6"},{"comment":"The dataset is multi-label, with 28 categories that can co-occur, but the generation protocol is described as producing '384 images per class,' which suggests single-label conditioning. The paper must clarify how multi-label annotations are encoded in the class-conditional input to the diffusion model and whether synthetic labels are single- or multi-label. If synthetic labels are single-label, then Eq. (3) mixes a single-label synthetic target with a multi-label real target, producing interpolated targets that may lie outside the feasible multi-label distribution and can corrupt supervision.","section":"Sections 3 and 5.1"},{"comment":"All reported metrics are single-run point estimates with no confidence intervals or repeated-seed experiments. The decisive Kaggle numbers for MixModels are near zero, and the baseline numbers are also variable across configurations, so the paper should include at least a small number of repeated runs (or otherwise justify why single runs are sufficient). Without this, the quantitative comparisons between baselines and MixModels are not statistically grounded, and the central negative claim is not established.","section":"Section 6 and Tables 1-4"}],"minor_comments":[{"comment":"The text attributes MixUp to Berthelot et al. (2019), but the cited work is MixMatch, not MixUp; a proper MixUp reference (e.g., Zhang et al., 2018) should be added and cited correctly.","section":"Section 2 and References"},{"comment":"ArcFace is cited as Deng et al. (2019), but no matching entry appears in the reference list; the full citation should be added.","section":"Section 4.2 and References"},{"comment":"ViT-B/16 and Swin-B are trained for only 10 epochs with no learning-rate scheduler, while ResNet models are trained for 30-80 epochs, so the poor transformer test scores should not be interpreted as evidence against transformer architectures; this comparison is strongly confounded.","section":"Section 7.1.3 and Table 3"},{"comment":"The description of early stopping and ReduceLROnPlateau is not accompanied by the monitored metric or patience details for the diffusion model; please specify the stopping criteria and the validation metric used.","section":"Section 6.4"},{"comment":"The data/code link is provided as a short URL; a stable repository DOI or permanent archive link would be more appropriate for reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting question, but the current empirical evidence for the central negative claim is not convincing. The inconsistent Mix Loss objective and the identical degenerate test scores in Table 4 suggest that the MixModel evaluation may be broken rather than informative. I would like to see corrected experiments with a properly defined loss on z_mix, a clearly specified test-time rule, and repeated runs before this can be considered publishable. The scope of the conclusion should also be narrowed to the specific simplified 32x32 generator, since the paper does not compare with stronger diffusion augmentation methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the central negative result in this paper is not a valid experiment. In the Mix Loss variant, Eq. (6) defines the mixed logits z_mix = λ z_r + (1−λ) z_g, but Eq. (7) defines the loss as λ L_r + (1−λ) L_g, where L_r and L_g are BCE losses on the unmixed logits. So z_mix receives no gradient, and the model is never trained to produce accurate mixed predictions. The paper then says \"Final predictions are derived from z_mix\" but never specifies how z_mix is computed at test time, where no synthetic image exists. Worse, all four MixModel configurations report identical public (0.01924) and private (0.01955) F1 scores despite different backbones and mixing strategies. Identical scores to five decimal places are the signature of a constant/trivial predictor, not of four distinct learned models. The claim that diffusion augmentation fails is therefore not supported.\n\nTo be fair, something is here. The paper documents a real attempt to use a simplified class-conditional DDPM on HPA, with code and generated data linked. The baseline comparisons are mostly useful: ResNet18 beating ResNet34 under short training, ArcFace and FocalLoss underperforming BCE, and transformer backbones collapsing on the Kaggle test set. The authors honestly report that validation performance does not transfer.\n\nThe soft spots beyond the fatal confound: no repeated runs or confidence intervals; the diffusion generator is trained at 32x32 and upsampled, so it is weak; no comparison to DiffuseMix or other diffusion-augmentation baselines; and the validation scores for MixModel (max 0.325) are not \"promising\" relative to the BCE baseline (0.638). The paper also attributes the failure to overfitting and distributional mismatch when the actual cause is likely the broken training objective and undefined test-time prediction.\n\nWho is this for? Practitioners wanting a cautionary tale about naive diffusion augmentation might find hints in the baseline results, but the main experiment cannot be relied on. This paper needs a rewritten method with a correct mixing loss and a clearly specified test-time rule before it is worth a serious referee.\n\nMy recommendation: desk reject in current form, or if you want to be generous, require major revision that fixes the objective and re-runs all MixModel configurations with proper error bars. As published, it would mislead rather than inform.","headline":"The paper's central negative claim about diffusion augmentation is invalid because the Mix Loss objective never trains the mixed logits, and the identical degenerate test scores across all four variants confirm the result is an artifact.","tokens_in":7680,"tokens_out":3189,"would_cite":false,"duration_ms":32800,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion-generated images fail to improve protein localization classification","keywords":["diffusion models","data augmentation","protein subcellular localization","multi-label classification","DDPM","MixUp","synthetic data","generalization gap"],"falsifier":"Train a state-of-the-art class-conditional diffusion model (e.g., a full UNet with timestep conditioning and attention) to generate 512x512 or at least 128x128 images, integrate them with the same Mix Representation and Mix Loss strategies, and evaluate on the Kaggle public test split; if the F1-score rises above the ResNet-BCE baseline of roughly 0.457, the paper's negative conclusion is specific to its lightweight generator rather than to diffusion augmentation in general.","tokens_in":6766,"feed_emoji":"🧬","tokens_out":2772,"duration_ms":26359,"temperature":0.7,"pith_summary":"The paper investigates whether synthetic images produced by a simplified class-conditional diffusion model can improve multi-label classification of protein subcellular localization, using only the official Kaggle training data without external labels. The authors propose MixModel, which mixes real and generated images either in feature space (Mix Representation) or in loss space (Mix Loss). Although MixModel achieves moderate validation F1-scores, all variants collapse on the Kaggle public and private test sets with F1-scores near 0.019, far below ResNet baselines with standard BCE loss (public F1 around 0.457). The paper establishes that naive diffusion-based augmentation, at least with a lightweight generator and simple mixing strategies, does not transfer to this high-resolution, multi-label biomedical task.","feed_headline":"Diffusion data augmentation fails protein localization test","feed_subtitle":"MixModel scores near 0.019 F1 on Kaggle while ResNet baselines reach 0.457, exposing the validation-to-test gap.","key_machinery":"The central object is the MixModel, a two-stage pipeline combining a simplified class-conditional DDPM generator with two MixUp-inspired integration strategies. The generator is a shallow three-layer convolutional network that concatenates a class embedding at the input, omits timestep conditioning, and is trained on 32x32 images before bicubic upsampling. The integration strategies are Mix Representation, which linearly interpolates feature embeddings from real and synthetic images with a Beta(0.3, 0.3) mixing coefficient, and Mix Loss, which interpolates logits and combines binary cross-entropy losses from the real and synthetic branches. These mechanisms carry the argument because the paper attributes the generalization failure to distributional mismatch introduced by interpolation and to artifacts in the low-fidelity synthetic data.","core_discovery":"On its own terms, the paper claims that a simplified class-conditional DDPM, trained on 32x32 downsampled images and upsampled to 512x512, produces blurry but label-consistent synthetic samples, and that integrating these samples into classifier training via Mix Loss or Mix Representation yields models that overfit to synthetic artifacts and fail on unseen test data. The central discovery is a negative result: generative augmentation with limited computational resources does not outperform conventional supervised training, and validation performance is a poor predictor of external test performance in this setting.","pith_inferences":["The 32x32-to-512x512 upsampling step likely destroys class-discriminative texture, so the negative result may be specific to this resolution mismatch; a moderate generation resolution such as 128x128 or 256x256 might behave differently and is a direct testable extension.","The near-identical test scores (0.01924 public, 0.01955 private) across all four MixModel variants suggest that the classifier is essentially ignoring the synthetic branch and predicting near the majority or label-average baseline, which could be verified by examining per-class predictions.","Mixing in feature space may require feature normalization or alignment before interpolation, since raw feature distributions of real and synthetic images are likely separated; adding a domain-adversarial or contrastive alignment step could rescue the Mix Representation approach.","The paper's negative result is conditional on the specific Kaggle competition data and evaluation metric; applying the same pipeline to a dataset with cleaner labels or fewer classes might yield different conclusions."],"forward_implications":["If the paper is correct, diffusion-generated synthetic data from lightweight generators cannot substitute for external labeled data in this task, and simple interpolation mixing amplifies rather than repairs the domain gap.","Validation F1-scores on a held-out split are not reliable indicators of leaderboard performance for this biomedical classification benchmark, since MixModel validation scores reach 0.325 while test scores stagnate near 0.019.","ResNet backbones trained with BCEWithLogitsLoss provide a more stable and generalizable baseline than more complex loss functions or semi-supervised mixing, suggesting that robust supervision matters more than added synthetic diversity.","Future attempts to use generative augmentation should focus on higher-fidelity generation, stronger class conditioning, and explicit distribution alignment between synthetic and real feature spaces, rather than on mixing strategies alone.","The observed collapse across all MixModel configurations implies that the failure is not specific to one mixing method or backbone depth, but stems from the synthetic data itself or its integration procedure."],"supporting_citations":[{"why":"Supplies the original class-conditional DDPM formulation that the paper simplifies into its lightweight generator.","marker":"Ho et al., 2020"},{"why":"Provides the MixUp and MixMatch framework that motivates the Mix Loss and Mix Representation interpolation strategies.","marker":"Berthelot et al., 2019"},{"why":"Demonstrates that pretrained diffusion models can improve few-shot classification, serving as the positive result the paper aims to test in a constrained setting.","marker":"Trabucco et al., 2024"},{"why":"Proposes DIFFUSEMIX, a label-preserving diffusion augmentation method, offering a direct comparison point for the paper's mixing design.","marker":"Islam et al., 2024"},{"why":"Introduces ArcFace, the metric-learning loss the paper evaluates as a baseline and contrasts with BCEWithLogitsLoss.","marker":"Deng et al., 2019"}],"fun_headline_variants":["Diffusion synthetic data fails to boost protein localization","Synthetic images from DDPM hurt protein localization models","MixModel flops: diffusion augmentation doesn't generalize","Blurry diffusion data sinks protein localization accuracy","Validation gains vanish: diffusion augmentation fails test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire negative conclusion rests on the assumption that the simplified, low-resolution diffusion model produces synthetic images good enough to train a downstream classifier; if a stronger generator were used, the result could flip.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion synthetic data fails to boost protein localization","Synthetic images from DDPM hurt protein localization models","MixModel flops: diffusion augmentation doesn't generalize","Blurry diffusion data sinks protein localization accuracy","Validation gains vanish: diffusion augmentation fails test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000534,"raw_usage":{"total_tokens":2481,"prompt_tokens":770,"completion_tokens":1711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":386,"completion_tokens_details":{"reasoning_tokens":1640}},"tokens_in":386,"tokens_out":1711,"duration_ms":11686,"temperature":1.0,"reasoning_tokens":1640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:56:09.638052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a state-of-the-art class-conditional diffusion model (e.g., a full UNet with timestep conditioning and attention) to generate 512x512 or at least 128x128 images, integrate them with the same Mix Representation and Mix Loss strategies, and evaluate on the Kaggle public test split; if the F1-score rises above the ResNet-BCE baseline of roughly 0.457, the paper's negative conclusion is specific to its lightweight generator rather than to diffusion augmentation in general.","supporting_citations":[{"cited_title":"Mixmatch: A holistic approach to semi-supervised learning","cited_arxiv_id":null,"evidence_quote":"Provides the MixUp and MixMatch framework that motivates the Mix Loss and Mix Representation interpolation strategies."},{"cited_title":"Effective data augmentation with diffusion models","cited_arxiv_id":null,"evidence_quote":"Demonstrates that pretrained diffusion models can improve few-shot classification, serving as the positive result the paper aims to test in a constrained setting."},{"cited_title":"Diffusemix: Label-preserving data augmentation with diffusion models","cited_arxiv_id":null,"evidence_quote":"Proposes DIFFUSEMIX, a label-preserving diffusion augmentation method, offering a direct comparison point for the paper's mixing design."}],"review_version":1}