REVIEW 3 major objections 5 minor 24 references
Contrastive Learning with Diffusion Features for Weakly Supervised Medical Image Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that CLDF, by contrastively training a small pixel decoder on pixels selected from the intersection of a CAM and a mean diffusion gradient map, maps frozen diffusion features to embeddings whose K-means clusters…
desk verdict Plausible contrastive diffusion-feature method with real FLARE21 gains, but the BraTS21 SOTA claim rests on an unfair 4-modality vs 1-modality comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a high-confidence pixel selector feeding a supervised contrastive loss. Foreground pixels are the intersection of a CAM (AME-CAM on BraTS21, LayerCAM on FLARE21) with the mean gradient map obtained by averaging external-classifier gradients over timesteps 10 to 200; background pixels are sampled from regions activated by neither source. These labels train a four-layer MLP pixel decoder that takes aggregated diffusion features formed by noising the image at timesteps 1, 10, 50, and 100, extracting decoder-layer features from the frozen CDM, upsampling and concatenating them, and averaging across timesteps. The supervised contrastive loss pulls together embeddings of pixels with the same pseudo-label and pushes apart those with different pseudo-labels. During inference the decoder's 16-dimensional embeddings are clustered with K-means to produce the binary mask.
What would settle it
A reader could test the intersection assumption by computing, on a dataset with pixel-level ground truth, the per-lesion recall of the CAM-gradient intersection used to select foreground pixels; if K-means clusters fail to recover lesions that appear in only one of the two maps, while union-based selection recovers them, the confidence-pixel strategy is the bottleneck. Alternatively, on a synthetic dataset with small, low-contrast targets, a Dice drop concentrated on targets missed by either source would confirm the mechanism's limit.
Extended reading notes
Core claim
The central claim is that a pixel decoder trained with contrastive learning on high-confidence foreground and background pixels can convert aggregated features of a frozen conditional diffusion model into an embedding space where simple K-means clustering yields accurate segmentation masks, using only image-level labels. The paper reports that this recipe consistently beats seven WSSS baselines across four binary tasks: Dice of 0.880 on BraTS21, 0.740 on FLARE21 kidney, 0.623 on spleen, and 0.778 on liver, against 0.920, 0.949, 0.936, and 0.952 for fully supervised UNet. The ablation in Section 3.3 supports the two signal-combination claims: diffusion features outperform ResNet and UNet-encoder features, and fusing the CAM with the mean gradient map improves over contrastive learning on either signal alone.
Load-bearing premise
The method assumes that the intersection of the CAM and the mean diffusion gradient map marks true foreground with almost no false positives, and that everything outside both maps is reliably background; if a real lesion appears in only one of the two maps, the contrastive loss trains on biased pixels and the final clusters inherit that bias.
Editorial extensions
If this is right
- On the four tested binary tasks, CLDF outperforms all seven compared WSSS baselines, including the diffusion-only CDM and DiG methods whose masks suffer from false positives.
- The frozen diffusion model is the key feature source: under the same CLDF pipeline, CDM features give 0.880 BraTS21 Dice, while ResNet18, ResNet50, and UNet encoder features reach at most 0.775.
- Fusing CAM and the mean gradient map is load-bearing: on BraTS21, AME-CAM plus mean-gradient contrastive learning improves Dice from 0.847 to 0.880, and on FLARE21 kidney, LayerCAM plus mean-gradient improves from 0.692 to 0.740.
- Because CLDF is a post-processing refinement over any CAM, improvements in the CAM source are expected to translate into better final segmentation masks.
- A four-layer pixel decoder with output embedding dimension 16 is sufficient, making the added component light at inference time.
Reading between the lines
- The same fuse-and-contrast recipe may transfer to non-diffusion feature extractors or other dense prediction tasks, but the paper only demonstrates the full pipeline on frozen conditional diffusion model features.
- Because foreground labels are defined by an intersection, true objects that appear in only one of the two maps are excluded from the contrastive signal, so small or atypical lesions could set a hidden ceiling on the reported gains.
- A direct stress test would be multi-class or small-lesion segmentation, where the approach might need per-class pipelines or a union-based selector to avoid under-segmentation of subtle structures.
- One could also test the sensitivity of the method to the thresholding of the CAM and gradient maps, since the intersection labels change abruptly with those thresholds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Contrastive Learning with Diffusion Features (CLDF), a weakly supervised semantic segmentation method that trains a pixel decoder on frozen features from a conditional diffusion model, using a contrastive loss whose foreground/background pixel sets are derived from the intersection of a class activation map (CAM) and a mean gradient map obtained from the diffusion model's external classifier. The trained pixel decoder maps pixel features into an embedding space, and K-means clustering produces the final binary segmentation masks. Experiments on BraTS21 (brain tumor) and FLARE21 (abdominal organs) report large improvements over seven WSSS baselines, with CLDF approaching fully supervised performance on BraTS21 (Dice 0.880 vs. 0.920). Ablations analyze the contribution of diffusion features, the fusion of CAM and gradient maps, and the pixel decoder structure.
Significance. If the reported results are reliable, CLDF would be a meaningful advance in weakly supervised medical image segmentation: it leverages rich diffusion features with a light trainable decoder, avoids the need for pixel-level labels, and reports substantial gains on multiple binary segmentation tasks. The paper includes valuable ablation studies (Table 3) that decompose the contributions of contrastive learning and CAM/gradient fusion, and the design is practical in that it uses a frozen CDM and a small MLP. However, the central empirical claim of state-of-the-art performance is weakened by two evaluation issues: the BraTS21 comparison is confounded by input channel differences, and the CAM method is selected per dataset based on empirical performance. These issues are fixable but must be addressed before the claimed SOTA can be accepted.
major comments (3)
- [Section 3.1, Table 1] The BraTS21 comparison is not like-for-like. The paper states that unlike [4,9], which used single-modality slices, CLDF concatenates all four MRI modalities into a 4-channel input. All seven baselines in Table 1 are evaluated on single-modality input, so the reported gain over AME-CAM (0.880 vs. 0.827 Dice) could be due to the additional input information rather than the proposed fusion of CAM with the mean gradient map or the contrastive learning on diffusion features. To support the claim that CLDF "consistently achieves optimal results across all tasks," the authors should either re-run the baselines with 4-channel input, or report CLDF results with single-modality input, or explicitly frame the comparison as an upper-bound-style evaluation. This is the main load-bearing issue for the BraTS21 SOTA claim.
- [Section 3.1, Table 3] The CAM method is chosen per dataset after observing performance: "We use AME-CAMs [4] for Brats21 and LayerCAM [10] for FLARE21 to select pixels for CLDF as they empirically perform better." No validation protocol is described, and the final numbers in Table 1 use the better-performing CAM on each dataset. Since the method is claimed to work with any CAM, this dataset-dependent selection can introduce selection bias and makes the comparison difficult to interpret. The authors should either fix one CAM for all datasets, or use a held-out validation split to choose the CAM, and report the selection criterion explicitly.
- [Section 2, Table 3] The foreground pixel selection relies on the intersection of the CAM and the mean gradient map, which assumes that true foreground is covered by both maps. Pixels that are true foreground but appear in only one of the two maps (e.g., small, low-contrast, or atypical lesions) will be excluded from positive supervision, potentially biasing the learned embeddings. The ablation in Table 3 shows that fusion improves average Dice, but the paper does not quantify the false-positive/false-negative rates of the selected pseudo-masks, nor does it analyze failure cases. A per-image analysis of pseudo-mask quality or a sensitivity analysis of the intersection criterion would strengthen the central mechanism claim.
minor comments (5)
- [Section 2] In Section 2, "The extracted features are unsampled to H×W" should read "upsampled to H×W."
- [Table 1] The year for SEAM is inconsistent: the first sub-table gives "SEAM (2021)" while the second sub-table gives "SEAM (2020)"; reference [20] is a CVPR 2020 paper, so 2020 should be used consistently.
- [Section 3.1] The paper does not report the computational cost of training the CDM and external classifier (four days) or the added inference overhead of extracting diffusion features and computing gradient maps, which is relevant for practical deployment of this weakly supervised method.
- [Equation (1)] The notation in the supervised contrastive loss is mostly clear, but the denominator sum over k=1 to N with the indicator 1_{i≠k} should state that k runs over all selected pixels in the current batch; this is standard but a one-sentence clarification would help readers.
- [Section 3.2] The claim that CLDF "significantly outperforms" baselines is not supported by statistical significance tests; several Dice scores in Table 1 have overlapping standard deviations (e.g., FLARE21 Spleen CLDF 0.623±0.15 vs. AME-CAM 0.560±0.13). Reporting paired tests or confidence intervals would strengthen the claim.
Circularity Check
No circularity found: CLDF is evaluated on held-out ground-truth masks, and the contrastive pseudo-labels are training inputs, not evaluation targets.
full rationale
The paper's derivation chain is not circular. The pixel decoder is trained with a contrastive loss on pixels selected from the intersection of a CAM and a mean gradient map, but the final segmentation is obtained by K-means clustering on the learned embeddings and is evaluated against held-out ground-truth masks from BraTS21 and FLARE21. The CAM and gradient maps are used only to choose training pixels; they do not set evaluation constants or directly produce the reported Dice/IoU numbers. The method relies on self-cited components, notably AME-CAM [4] and CDM [9], as building blocks and as baselines, but those are published methods with external benchmark existence, and the central claim is an external comparison against them and other baselines. The ablation in Table 3 shows that combining CAM, mean gradient, and contrastive learning changes performance in ways that are not forced by construction. The BraTS21 comparison uses 4-channel input for CLDF while baselines use a single modality, which is a legitimate fairness confound for the SOTA claim, but it is an external-validity issue, not circularity: the reported masks are still judged by ground truth rather than by the method's own inputs. Therefore, no pattern of self-definition, fitted-input-as-prediction, or load-bearing self-citation is present.
Assumptions & free parameters
free parameters (6)
- Feature extraction timesteps =
{1, 10, 50, 100}
- Gradient map timesteps =
{10, 20, ..., 200}
- Contrastive temperature tau =
0.1
- Pixel decoder architecture and training schedule =
MLP (D,16,16,16,16), 5 epochs, SGD LR 1.0, batch 4
- Background pixel sampling count =
5000
- CAM method per dataset =
AME-CAM for BraTS21, LayerCAM for FLARE21
assumptions (4)
- domain assumption Diffusion features from a class-conditioned diffusion model carry pixel-level semantic information sufficient for segmentation.
- domain assumption Gradient maps from the CDM external classifier highlight class-relevant objects in noisy images.
- domain assumption Pixels in the intersection of CAM and mean gradient map are high-confidence foreground, and pixels outside both are high-confidence background.
- domain assumption K-means on the learned embeddings yields a valid binary segmentation.
Cite this review
Pith. "Pith review of Contrastive Learning with Diffusion Features for Weakly Supervised Medical Image Segmentation." pith.science (2026). https://pith.science/paper/QW6DGOJI
@misc{pith2026250623460,
author = {Pith},
title = {Pith review of: Contrastive Learning with Diffusion Features for Weakly Supervised Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/QW6DGOJI}},
note = {Machine review of arXiv:2506.23460}
}
read the original abstract
Weakly supervised semantic segmentation (WSSS) methods using class labels often rely on class activation maps (CAMs) to localize objects. However, traditional CAM-based methods struggle with partial activations and imprecise object boundaries due to optimization discrepancies between classification and segmentation. Recently, the conditional diffusion model (CDM) has been used as an alternative for generating segmentation masks in WSSS, leveraging its strong image generation capabilities tailored to specific class distributions. By modifying or perturbing the condition during diffusion sampling, the related objects can be highlighted in the generated images. Yet, the saliency maps generated by CDMs are prone to noise from background alterations during reverse diffusion. To alleviate the problem, we introduce Contrastive Learning with Diffusion Features (CLDF), a novel method that uses contrastive learning to train a pixel decoder to map the diffusion features from a frozen CDM to a low-dimensional embedding space for segmentation. Specifically, we integrate gradient maps generated from CDM external classifier with CAMs to identify foreground and background pixels with fewer false positives/negatives for contrastive learning, enabling robust pixel embedding learning. Experimental results on four segmentation tasks from two public medical datasets demonstrate that our method significantly outperforms existing baselines.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Chen, Y.J., Hu, X., Shi, Y., Ho, T.Y.: Ame-cam: Attentive multiple-exit cam for weakly supervised segmentation on mri brain tumor. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 173–182. Springer (2023)
work page 2023
-
[10]
IEEE Transactions on Image Processing 30, 5875–5888 (2021)
Jiang, P.T., Zhang, C.B., Hou, Q., Cheng, M.M., Wei, Y.: Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing 30, 5875–5888 (2021)
work page 2021
-
[1]
arXiv preprint arXiv:2107.02314 (2021)
Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colak, E., Farahani, K., Kalpathy-Cramer, J., Kitamura, F.C., Pati, S., et al.: The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classifica- tion. arXiv preprint arXiv:2107.02314 (2021)
arXiv 2021
-
[2]
arXiv preprint arXiv:2112.03126 (2021)
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., Babenko, A.: Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126 (2021)
arXiv 2021
-
[3]
In: 2018 IEEE winter conference on applications of computer vision (WACV)
Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV). pp. 839–847. IEEE (2018)
2018
-
[5]
Advances in neural information processing systems34, 8780–8794 (2021)
Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021)
2021
-
[6]
Advances in Neural Information Processing Systems35, 14715–14728 (2022)
Graikos, A., Malkin, N., Jojic, N., Samaras, D.: Diffusion models as plug-and- play priors. Advances in Neural Information Processing Systems35, 14715–14728 (2022)
work page 2022
-
[7]
Advances in neural information processing systems33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)
2020
Show all 24 references
-
[8]
arXiv preprint arXiv:2207.12598 (2022)
Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
2022 arXiv
-
[9]
In: International Conference on Medi- cal Image Computing and Computer-Assisted Intervention
Hu, X., Chen, Y.J., Ho, T.Y., Shi, Y.: Conditional diffusion models for weakly supervised medical image segmentation. In: International Conference on Medi- cal Image Computing and Computer-Assisted Intervention. pp. 756–765. Springer (2023)
2023
-
[11]
Advances in neural information processing systems33, 18661–18673 (2020)
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems33, 18661–18673 (2020)
2020
-
[12]
In: Proceedings of the IEEE/CVF international conference on computer vision
Li, Y., Kuang, Z., Liu, L., Chen, Y., Zhang, W.: Pseudo-mask matters in weakly- supervised semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6964–6973 (2021) 10 D Zeng, et al
2021
-
[13]
Medical Image Analysis82, 102616 (2022)
Ma, J., Zhang, Y., Gu, S., An, X., Wang, Z., Ge, C., Wang, C., Zhang, F., Wang, Y., Xu, Y., et al.: Fast and low-gpu-memory abdomen ct organ segmentation: the flare challenge. Medical Image Analysis82, 102616 (2022)
2022
-
[14]
In: International conference on machine learning
Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International conference on machine learning. pp. 8162–8171. PMLR (2021)
2021
-
[15]
arXiv preprint arXiv:1908.01224 (2019)
Omeiza, D., Speakman, S., Cintas, C., Weldermariam, K.: Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224 (2019)
2019 arXiv
-
[16]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Pinaya, W.H., Graham, M.S., Gray, R., Da Costa, P.F., Tudosiu, P.D., Wright, P., Mah, Y.H., MacKinnon, A.D., Teo, J.T., Jager, R., et al.: Fast unsupervised brain anomaly detection and segmentation with diffusion models. In: International Conference on Medical Image Computing ...
2022
-
[17]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[18]
In: Proceedings of the IEEE international conference on computer vision
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
2017
-
[19]
In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition workshops
Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., Hu, X.: Score-cam: Score-weighted visual explanations for convolutional neural net- works. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition workshops. pp. 24–25 (2020)
2020
-
[20]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wang, Y., Zhang, J., Kan, M., Shan, S., Chen, X.: Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12275–12284 (2020)
2020
-
[21]
In: International Conference on Medical Imaging with Deep Learning
Wolleb, J., Sandkühler, R., Bieder, F., Valmaggia, P., Cattin, P.C.: Diffusion mod- els for implicit image segmentation ensembles. In: International Conference on Medical Imaging with Deep Learning. pp. 1336–1348. PMLR (2022)
2022
-
[22]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xie,J.,Xiang,J.,Chen,J.,Hou,X.,Zhao,X.,Shen,L.:C2am:Contrastivelearning of class-agnostic activation map for weakly supervised object localization and se- mantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 989–998 (2022)
2022
-
[23]
In: European Conference on Computer Vision
Yoon, S.H., Kwon, H., Jeong, J., Park, D., Yoon, K.J.: Diffusion-guided weakly supervised semantic segmentation. In: European Conference on Computer Vision. pp. 393–411. Springer (2024)
2024
-
[24]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)
2016
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.