Leveraging Diffusion Models for Synthetic Data Augmentation in Protein Subcellular Localization Classification

Sylvey Lin; Zhi-Yi Cao

REVIEW 5 major objections 5 minor 6 references

Leveraging Diffusion Models for Synthetic Data Augmentation in Protein Subcellular Localization Classification

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Diffusion-generated images fail to improve protein localization classification

desk verdict The paper's central negative claim about diffusion augmentation is invalid because the Mix Loss objective never trains the mixed logits, and the identical degenerate test scores across all four variants confirm the result is an artifact. read the letter →

arxiv 2505.22926 v1 pith:KUAZM2UZ submitted 2025-05-28 cs.CV

Sylvey Lin , Zhi-Yi Cao This is my paper

classification cs.CV

keywords diffusionmodelsdataaugmentationproteinsubcellularlocalizationmulti-labelclassificationDDPMMixUpsyntheticgeneralizationgap

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether synthetic images produced by a simplified class-conditional diffusion model can improve multi-label classification of protein subcellular localization, using only the official Kaggle training data without external labels. The authors propose MixModel, which mixes real and generated images either in feature space (Mix Representation) or in loss space (Mix Loss). Although MixModel achieves moderate validation F1-scores, all variants collapse on the Kaggle public and private test sets with F1-scores near 0.019, far below ResNet baselines with standard BCE loss (public F1 around 0.457). The paper establishes that naive diffusion-based augmentation, at least with a lightweight generator and simple mixing strategies, does not transfer to this high-resolution, multi-label biomedical task.

What carries the argument

The central object is the MixModel, a two-stage pipeline combining a simplified class-conditional DDPM generator with two MixUp-inspired integration strategies. The generator is a shallow three-layer convolutional network that concatenates a class embedding at the input, omits timestep conditioning, and is trained on 32x32 images before bicubic upsampling. The integration strategies are Mix Representation, which linearly interpolates feature embeddings from real and synthetic images with a Beta(0.3, 0.3) mixing coefficient, and Mix Loss, which interpolates logits and combines binary cross-entropy losses from the real and synthetic branches. These mechanisms carry the argument because the paper attributes the generalization failure to distributional mismatch introduced by interpolation and to artifacts in the low-fidelity synthetic data.

What would settle it

Train a state-of-the-art class-conditional diffusion model (e.g., a full UNet with timestep conditioning and attention) to generate 512x512 or at least 128x128 images, integrate them with the same Mix Representation and Mix Loss strategies, and evaluate on the Kaggle public test split; if the F1-score rises above the ResNet-BCE baseline of roughly 0.457, the paper's negative conclusion is specific to its lightweight generator rather than to diffusion augmentation in general.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a simplified class-conditional DDPM, trained on 32x32 downsampled images and upsampled to 512x512, produces blurry but label-consistent synthetic samples, and that integrating these samples into classifier training via Mix Loss or Mix Representation yields models that overfit to synthetic artifacts and fail on unseen test data. The central discovery is a negative result: generative augmentation with limited computational resources does not outperform conventional supervised training, and validation performance is a poor predictor of external test performance in this setting.

Load-bearing premise

The entire negative conclusion rests on the assumption that the simplified, low-resolution diffusion model produces synthetic images good enough to train a downstream classifier; if a stronger generator were used, the result could flip.

Editorial extensions

If this is right

If the paper is correct, diffusion-generated synthetic data from lightweight generators cannot substitute for external labeled data in this task, and simple interpolation mixing amplifies rather than repairs the domain gap.
Validation F1-scores on a held-out split are not reliable indicators of leaderboard performance for this biomedical classification benchmark, since MixModel validation scores reach 0.325 while test scores stagnate near 0.019.
ResNet backbones trained with BCEWithLogitsLoss provide a more stable and generalizable baseline than more complex loss functions or semi-supervised mixing, suggesting that robust supervision matters more than added synthetic diversity.
Future attempts to use generative augmentation should focus on higher-fidelity generation, stronger class conditioning, and explicit distribution alignment between synthetic and real feature spaces, rather than on mixing strategies alone.
The observed collapse across all MixModel configurations implies that the failure is not specific to one mixing method or backbone depth, but stems from the synthetic data itself or its integration procedure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 32x32-to-512x512 upsampling step likely destroys class-discriminative texture, so the negative result may be specific to this resolution mismatch; a moderate generation resolution such as 128x128 or 256x256 might behave differently and is a direct testable extension.
The near-identical test scores (0.01924 public, 0.01955 private) across all four MixModel variants suggest that the classifier is essentially ignoring the synthetic branch and predicting near the majority or label-average baseline, which could be verified by examining per-class predictions.
Mixing in feature space may require feature normalization or alignment before interpolation, since raw feature distributions of real and synthetic images are likely separated; adding a domain-adversarial or contrastive alignment step could rescue the Mix Representation approach.
The paper's negative result is conditional on the specific Kaggle competition data and evaluation metric; applying the same pipeline to a dataset with cleaner labels or fewer classes might yield different conclusions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: the central negative result in this paper is not a valid experiment. In the Mix Loss variant, Eq. (6) defines the mixed logits z_mix = λ z_r + (1−λ) z_g, but Eq. (7) defines the loss as λ L_r + (1−λ) L_g, where L_r and L_g are BCE losses on the unmixed logits. So z_mix receives no gradient, and the model is never trained to produce accurate mixed predictions. The paper then says "Final predictions are derived from z_mix" but never specifies how z_mix is computed at test time, where no synthetic image exists. Worse, all four MixModel configurations report identical public (0.01924) and private (0.01955) F1 scores despite different backbones and mixing strategies. Identical scores to five decimal places are the signature of a constant/trivial predictor, not of four distinct learned models. The claim that diffusion augmentation fails is therefore not supported.

To be fair, something is here. The paper documents a real attempt to use a simplified class-conditional DDPM on HPA, with code and generated data linked. The baseline comparisons are mostly useful: ResNet18 beating ResNet34 under short training, ArcFace and FocalLoss underperforming BCE, and transformer backbones collapsing on the Kaggle test set. The authors honestly report that validation performance does not transfer.

The soft spots beyond the fatal confound: no repeated runs or confidence intervals; the diffusion generator is trained at 32x32 and upsampled, so it is weak; no comparison to DiffuseMix or other diffusion-augmentation baselines; and the validation scores for MixModel (max 0.325) are not "promising" relative to the BCE baseline (0.638). The paper also attributes the failure to overfitting and distributional mismatch when the actual cause is likely the broken training objective and undefined test-time prediction.

Who is this for? Practitioners wanting a cautionary tale about naive diffusion augmentation might find hints in the baseline results, but the main experiment cannot be relied on. This paper needs a rewritten method with a correct mixing loss and a clearly specified test-time rule before it is worth a serious referee.

My recommendation: desk reject in current form, or if you want to be generous, require major revision that fixes the objective and re-runs all MixModel configurations with proper error bars. As published, it would mislead rather than inform.

Referee Report

5 major / 5 minor

Summary. The paper asks whether diffusion-generated synthetic images can improve multi-label classification of protein subcellular localization, using only the official HPA Kaggle dataset. The authors implement a simplified class-conditional DDPM that is trained at 32x32 resolution and upsample its outputs to 512x512, then train ResNet-based classifiers with two proposed mixing strategies, Mix Loss and Mix Representation. They report validation scores on a 90/10 split and public/private Kaggle leaderboard scores. The central claim is that the proposed MixModel variants fail to generalize to unseen test data, with Kaggle F1-scores around 0.019, far below the ResNet18 BCE baseline (public F1 about 0.457). I find that the central empirical claim is not supported as presented, because the Mix Loss objective is internally inconsistent, the test-time prediction rule is unspecified, and the identical near-zero test scores across all MixModel configurations indicate a degenerate prediction process rather than four independently learned classifiers.

Significance. If the negative result were rigorously established, it would provide a useful caution about naive diffusion-based augmentation in high-resolution biomedical multi-label classification. The paper reports external Kaggle leaderboard scores, which is a strength, and it makes code and generated data available, which supports reproducibility. However, the current significance is limited by the invalid Mix Loss formulation, the unexplained identical test scores, and the lack of confidence intervals or repeated runs. The study also uses a very lightweight 32x32 generator, so the scope of any supported conclusion is restricted to this specific generation setup, not to diffusion augmentation in general.

major comments (5)

[Section 5.2.2, Eqs. (6)-(7)] The Mix Loss objective is internally inconsistent. Eq. (6) defines the prediction logits as z_mix = lambda*z_r + (1-lambda)*z_g, but Eq. (7) defines the training loss as L_mix = lambda*L_r + (1-lambda)*L_g, where L_r and L_g are losses on the unmixed logits. Consequently, z_mix receives no gradient during training and the model is never optimized to make accurate mixed-logit predictions. The paper also does not specify how z_mix is formed at test time, where no paired synthetic image exists. This makes Table 4 an invalid test of diffusion augmentation. Please correct the objective to a loss that depends on z_mix, and specify the exact test-time prediction rule.
[Table 4] All four MixModel configurations report exactly the same public F1-score (0.01924) and private F1-score (0.01955), despite different backbones, different mixing strategies, and materially different validation scores. This identical pattern is the signature of a degenerate constant prediction rule, not of four independently trained classifiers. The authors should report the per-configuration test-time outputs and explain why the scores are identical; without this, the failure cannot be attributed to properties of the synthetic data or the mixing strategy.
[Section 5.1 and Figures 4-6] The simplified DDPM is trained on images downsampled to 32x32 and then upsampled to 512x512, and Figure 6's caption states that the class-conditional generation results show 'minimal distinction across labels.' This directly contradicts the claim in Section 7.2.1 that the simplified DDPM successfully leverages class conditioning and produces visibly different outputs across classes. The paper should reconcile this contradiction and provide quantitative evidence of class-conditional fidelity, such as label-consistency rates measured by a pretrained classifier or FID-style distributional metrics.
[Sections 3 and 5.1] The dataset is multi-label, with 28 categories that can co-occur, but the generation protocol is described as producing '384 images per class,' which suggests single-label conditioning. The paper must clarify how multi-label annotations are encoded in the class-conditional input to the diffusion model and whether synthetic labels are single- or multi-label. If synthetic labels are single-label, then Eq. (3) mixes a single-label synthetic target with a multi-label real target, producing interpolated targets that may lie outside the feasible multi-label distribution and can corrupt supervision.
[Section 6 and Tables 1-4] All reported metrics are single-run point estimates with no confidence intervals or repeated-seed experiments. The decisive Kaggle numbers for MixModels are near zero, and the baseline numbers are also variable across configurations, so the paper should include at least a small number of repeated runs (or otherwise justify why single runs are sufficient). Without this, the quantitative comparisons between baselines and MixModels are not statistically grounded, and the central negative claim is not established.

minor comments (5)

[Section 2 and References] The text attributes MixUp to Berthelot et al. (2019), but the cited work is MixMatch, not MixUp; a proper MixUp reference (e.g., Zhang et al., 2018) should be added and cited correctly.
[Section 4.2 and References] ArcFace is cited as Deng et al. (2019), but no matching entry appears in the reference list; the full citation should be added.
[Section 7.1.3 and Table 3] ViT-B/16 and Swin-B are trained for only 10 epochs with no learning-rate scheduler, while ResNet models are trained for 30-80 epochs, so the poor transformer test scores should not be interpreted as evidence against transformer architectures; this comparison is strongly confounded.
[Section 6.4] The description of early stopping and ReduceLROnPlateau is not accompanied by the monitored metric or patience details for the diffusion model; please specify the stopping criteria and the validation metric used.
[General] The data/code link is provided as a short URL; a stable repository DOI or permanent archive link would be more appropriate for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical negative result measured against an external Kaggle benchmark, not a prediction derived from fitted inputs.

full rationale

This paper is an empirical study with no derivation chain to walk. The central claim, that MixModel generalizes poorly, is supported by Kaggle public and private F1 scores in Table 4, which are external benchmark measurements rather than quantities constructed from the model's inputs by definition. Hyperparameters such as Beta(0.3), learning rate, and epoch counts are chosen by hand and are not fitted to test labels, so no fitted input is renamed as a prediction. The paper cites MixMatch (Berthelot et al., 2019) and DDPM (Ho et al., 2020) as external prior work, and there are no load-bearing self-citations. The apparent inconsistency in Eq. (7), where the Mix Loss objective does not train the mixed logits defined in Eq. (6), and the identical degenerately low test scores across all MixModel variants are genuine correctness and validity concerns, but they are not circularity: the reported F1 values are not equivalent by construction to the training loss, the mixing coefficients, or the synthetic data. The conclusion that diffusion-based augmentation failed in this setup is empirically grounded, however confounded. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. The main free parameters are hyperparameters chosen by hand, and the key assumption is that the weak low-resolution generator produces useful synthetic samples.

free parameters (3)

Beta (mixing coefficient) = 0.3
Chosen by hand for Beta distribution in MixUp; controls interpolation strength.
Number of synthetic images per class = 384
Arbitrary choice; total 10,752 generated samples. No sensitivity analysis.
Diffusion training resolution = 32x32
Downsampling chosen to reduce computational cost; likely a major factor in generation quality.

assumptions (3)

domain assumption HPA dataset labels are accurate and complete
The paper uses official Kaggle competition labels without external validation.
domain assumption The 90/10 split of the official training set is representative of the test distribution
Validation scores are used to select models, but the large gap between validation and Kaggle scores suggests this assumption is shaky.
ad hoc to paper Upsampled 32x32 synthetic images preserve enough class semantics for classifier training
The simplified DDPM generates at low resolution and then upsampled; this design choice is central to the negative result but not justified with evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Diffusion Models for Synthetic Data Augmentation in Protein Subcellular Localization Classification." pith.science (2026). https://pith.science/paper/KUAZM2UZ

@misc{pith2026250522926,
  author       = {Pith},
  title        = {Pith review of: Leveraging Diffusion Models for Synthetic Data Augmentation in Protein Subcellular Localization Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KUAZM2UZ}},
  note         = {Machine review of arXiv:2505.22926}
}

read the original abstract

We investigate whether synthetic images generated by diffusion models can enhance multi-label classification of protein subcellular localization. Specifically, we implement a simplified class-conditional denoising diffusion probabilistic model (DDPM) to produce label-consistent samples and explore their integration with real data via two hybrid training strategies: Mix Loss and Mix Representation. While these approaches yield promising validation performance, our proposed MixModel exhibits poor generalization to unseen test data, underscoring the challenges of leveraging synthetic data effectively. In contrast, baseline classifiers built on ResNet backbones with conventional loss functions demonstrate greater stability and test-time performance. Our findings highlight the importance of realistic data generation and robust supervision when incorporating generative augmentation into biomedical image classification.

Figures

Figures reproduced from arXiv: 2505.22926 by the authors.

**Figure 1.** Model Structure Overview 5.1 Diffusion Model The original class-conditional DDPM (Ho et al., 2020) was computationally expensive and failed to generate class-distinct outputs under our resource constraints. We therefore design a simplified variant that preserves core denoising capabilities while reducing architectural complexity. Compared to the original UNet—which employs a full encoder–decoder structure, skip … view at source ↗

**Figure 2.** Simplified ConvDout structure [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Simplified class-conditional UNet dings. A mixing coefficient λ ∼ Beta(β, β) is sampled for each pair: fmix = λfr + (1 − λ)fg (4) ymix = λyr + (1 − λ)yg (5) Where fr, fg are feature vectors from the real and synthetic images, and yr, yg their respective labels. The mixed feature vector fmix is passed to a linear classification head, and training is performed using binary cross-entropy loss on the interpolated targe… view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Visualization of the red channel from im [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Class-conditional generation results (red [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Training and validation loss curve for Mix [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 4 canonical work pages

[1]

Mixmatch: A holistic approach to semi-supervised learning

David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver, Nicolas Papernot, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). Curran Associates Inc., 2019

work page 2019
[2]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada, 2020. Curran Associates Inc

work page 2020
[3]

Diffusemix: Label-preserving data augmentation with diffusion models

Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label-preserving data augmentation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

work page 2024
[4]

Effective data augmentation with diffusion models

Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=ZWzUA9zeAg

work page 2024
[5]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[6]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Mixmatch: A holistic approach to semi-supervised learning

David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver, Nicolas Papernot, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). Curran Associates Inc., 2019

work page 2019

[2] [2]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada, 2020. Curran Associates Inc

work page 2020

[3] [3]

Diffusemix: Label-preserving data augmentation with diffusion models

Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label-preserving data augmentation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

work page 2024

[4] [4]

Effective data augmentation with diffusion models

Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=ZWzUA9zeAg

work page 2024

[5] [5]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[6] [6]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...