REVIEW 4 major objections 5 minor 30 references
No Masks Needed: Explainable AI for Deriving Segmentation from Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read ExplainSeg derives segmentation masks from classification labels alone: fine-tune a DINO ViT, turn Integrated Gradients heatmaps into masks via normalized cuts, and beat three baselines on two of three datasets without pixel annotations.
desk verdict Plausible pipeline, but the 'No Masks Needed' claim collapses on Kvasir-SEG because its patch labels come from the ground-truth masks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the attribution map: Integrated Gradients with a noise tunnel, applied to the fine-tuned classifier's prediction, assigns each pixel a relevance score for the predicted class. This is the step that turns 'why did the network predict this class' into 'where is the object'. The backbone is a DINO-pretrained Vision Transformer fine-tuned with a linear classification head, whose self-supervised features give the heatmaps spatial structure. Two mechanisms carry the rest: multiplying the relevance map by the ViT's intermediate feature map to denoise outlier pixels, and converting relevance values into a clean binary mask either by normalized-cut spectral clustering or by
What would settle it
The direct test is a label-shuffle control: retrain the pipeline with the fine-tuning labels randomly permuted, keeping images and all other settings identical. If the derived masks still match the ground-truth masks about as well as in the reported runs, then the fine-tuned classifier is not what localizes the object and the central mechanism fails. A complementary observational check: split results by whether the classifier's prediction was correct; if images classified wrongly yield equally good masks against the ground truth, classification and localization have decoupled.
Extended reading notes
Core claim
The paper claims that a classification network fine-tuned on image-label pairs keeps enough spatial information to yield a segmentation mask. ExplainSeg implements this: a DINO-pretrained Vision Transformer is fine-tuned on class labels only; Integrated Gradients with a noise tunnel turns the prediction into a relevance map; and normalized cuts converts that map into a binary mask, refined by DenseCRF. The best variant (XAI + NCut) reports 31.2% mIoU / 43.7% Dice on CBIS-DDSM, roughly double the best baseline's Dice, and beats all three baselines on NuInsSeg as well, staying competitive on Kvasir-SEG where MaskCut leads. The fine-tuning stage uses no ground-truth masks — that is the sense of
Load-bearing premise
The load-bearing premise is that, once the classifier is fine-tuned on image-level labels, its attribution heatmaps concentrate on the pixels of the target structure — and not on global cues such as breast density, background tissue, or staining variation that can also predict the class.
Editorial extensions
If this is right
- If the claim holds, segmentation training no longer needs pixel-wise masks: any dataset with diagnostic labels becomes a candidate for building a segmenter.
- On CBIS-DDSM the best variant (31.2% mIoU, 43.7% Dice) more than doubles the best baseline, indicating the largest gains appear on low-contrast, non-RGB modalities where general-purpose unsupervised methods collapse.
- The two-stage design (explanation, then post-processing) is modular: the NCut variant wins on mammograms, the morphology variant posts the top mIoU on histopathology, and the fusion-with-features variant leads on endoscopy — so the configuration can be chosen per modality.
- Because the masks are literally explanations of the classification decision, the clinician sees not only where the model segments but why the model made its diagnosis, coupling segmentation with interpretability.
Reading between the lines
- Editorial note: the 'no masks' framing is strictest for CBIS-DDSM and NuInsSeg. For Kvasir-SEG, Section 4.1 shows the patch-level classification labels are generated from the ground-truth polyp masks, so mask-derived supervision enters indirectly; the literal claim that holds is 'no masks during fine-tuning,' not 'no mask-derived labels anywhere.'
- Editorial note: Section 4.1 states that both training and validation images are passed through the model to generate the evaluated masks, so the reported scores include in-sample images; a validation-only rerun would give the cleaner estimate of generalization.
- Editorial extension: the mechanism should transfer to other medical tasks whose diagnostic label is solved from the lesion's own pixels, and should degrade where global context predicts the label; the large CBIS-DDSM gain suggests low-contrast, non-RGB modalities are the sweet spot.
- Editorial note: in the text as provided, the method section jumps from the problem formulation (Section 3.1) to the feature-relevance fusion (Section 3.5); the fine-tuning objective and the integrated-gradients formula are not shown, so the exact training signal is asserted rather than demonstrated.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ExplainSeg, a pipeline that fine-tunes a DINO-pretrained Vision Transformer with a classification head on image-level labels, applies Integrated Gradients with a noise tunnel to obtain attribution maps, and converts these maps into binary segmentation masks via morphology or normalized cuts followed by DenseCRF. The method is evaluated on CBIS-DDSM, NuInsSeg, and Kvasir-SEG, with comparisons to TokenCut, MICRA-Net, and MaskCut. The authors claim state-of-the-art performance and assert that no ground-truth segmentation mask information is used during classification fine-tuning.
Significance. If fully substantiated, the approach would be practically valuable: it offers a way to derive pixel-level segmentations from image-level labels in medical imaging, where dense annotation is expensive. The paper uses standard, reproducible components (DINO, Integrated Gradients, NCut, DenseCRF) and reports an ablation over four variants. However, the central claims are not established as stated: one dataset's classification labels are derived directly from the ground-truth masks, the reported Kvasir-SEG comparison is won by MaskCut, and no statistical significance or variance information is provided. The baseline set is also too narrow to support a general 'state-of-the-art' claim.
major comments (4)
- [Section 4.1, Kvasir-SEG paragraph] This paragraph states that Kvasir-SEG patches were labeled 'based on the presence of polyp pixels in the corresponding mask,' and then states in the same section that 'no information from the ground truth segmentation masks is used during the classification network finetuning stage.' These statements directly contradict each other. Since Kvasir-SEG is one of the three evaluation datasets, the title/abstract claim 'No Masks Needed' is not supported for the full evaluation. To fix this, the authors must either construct Kvasir-SEG classification labels without using mask-derived localization (e.g., using image-level labels or external negative samples) and rerun all affected experiments, or explicitly restrict the no-mask claim to CBIS-DDSM and NuInsSeg. As written, this is a load-bearing inconsistency.
- [Section 4.2, Table 2] On Kvasir-SEG, MaskCut achieves mIoU 41.3 and Dice 48.4, while ExplainSeg (XNCut) achieves mIoU 28.6 and Dice 41.4, a 12.7-point mIoU deficit. This contradicts the Section 1 claim that ExplainSeg 'achieves state-of-the-art performance and significantly outperforms other segmentation methods using pre-trained medical image segmentation models.' The text later acknowledges that ExplainSeg 'does not attain the top score on Kvasir-SEG,' but the abstract and introduction still make an unqualified state-of-the-art claim. The claims must be revised to match the reported results, or the scope of the claims must be narrowed.
- [Section 4.2, Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported; all numbers appear to come from a single run. For example, the NuInsSeg mIoU gap between ExplainSeg (13.1) and MaskCut (9.3) is modest and could easily be within run-to-run variation, especially with morphological post-processing and NCut. The phrase 'significantly outperforms' cannot be assessed without multiple seeds and dispersion measures. At minimum, report mean plus/minus standard deviation over several training seeds and, where feasible, a paired test over the evaluation images.
- [Section 4.2, Baselines Selection] The comparison set is too narrow to support a general state-of-the-art conclusion. The authors compare against two unsupervised object-discovery methods (TokenCut, MaskCut) and MICRA-Net, a supervised method from a different microscopy domain. Prior XAI-to-segmentation works [15, 16] are excluded because no public implementation is available, but the contribution is framed generically. The authors should either broaden the baseline set to include weakly supervised segmentation methods and other XAI-based segmentation approaches, or soften the state-of-the-art claim to a claim of competitiveness on the considered baselines and datasets.
minor comments (5)
- [Section 3] The method section is incomplete: equations (1) and (2) are referenced but missing, and the text jumps from the problem formulation to equation (3). Please restore the full derivation and label equations consistently.
- [Algorithm 1] The pseudo-code contains undefined or garbled symbols (e.g., the placeholder characters shown as '�'), and the inputs 'optional training' versus 'input image' are confusing. Use standard mathematical notation and clearly distinguish training-time and inference-time inputs.
- [Section 4.1, Implementation details] Several post-processing hyperparameters are not specified: morphology kernel sizes, threshold selection, NCut segment count, and DenseCRF parameters. These are essential for reproducibility. Please provide a table of all hyperparameters or release the configuration.
- [Table 2] The 'MIOU' header is capitalized inconsistently; use 'mIoU' throughout.
- [References] Some references appear only tangentially used (e.g., [17] for 'Growing a brain') and the reference list has formatting inconsistencies. Please check that every reference is cited and that citation indices match the bibliography.
Circularity Check
Kvasir-SEG patch labels are built from ground-truth masks, making the Kvasir-SEG 'segmentation from classification' output a function of the masks by construction.
-
self definitional
[Section 4.1, Datasets (Kvasir-SEG paragraph)]
"As the dataset contains only positive samples, we split each image into smaller patches and labeled them as positive or negative based on the presence of polyp pixels in the corresponding mask."
The paper claims (Section 4.1) that 'no information from the ground truth segmentation masks is used during the classification network finetuning stage,' yet the Kvasir-SEG fine-tuning labels are constructed directly from the ground-truth masks: each patch's binary label is defined by whether polyp pixels exist in the corresponding mask. The fine-tuned classifier therefore learns mask-derived spatial localization, and the Integrated-Gradients-with-Noise-Tunnel heatmaps (which become the segmentation after NCut/CRF post-processing) reflect that mask-informed signal. Hence for Kvasir-SEG the claimed derivation (mask from classification) is partially circular: the classification inputs are defined in terms of the mask output, so the predicted mask is a function of the evaluation ground truth
full rationale
The paper's central claim is that pixel-level medical segmentation is derived from classification labels and XAI heatmaps without using ground-truth masks. This holds as stated for CBIS-DDSM (pathology benign/malignant labels, not mask-derived) and plausibly for NuInsSeg (image-level presence labels). It fails for Kvasir-SEG by the paper's own description: patch labels are 'based on the presence of polyp pixels in the corresponding mask,' so mask information enters the fine-tuning stage through the label construction and is then re-extracted by the XAI heatmaps. The same section's assertion that 'no information from the ground truth segmentation masks is used during the classification network finetuning stage' is directly contradicted for one of the three datasets, weakening the title claim. No load-bearing self-citations exist: DINO, TokenCut, MaskCut, Captum, Integrated Gradients, DenseCRF, and the datasets are all external works, and the baseline comparisons are independent external benchmarks. The circularity is therefore partial but real: one of the three reported 'predictions' reduces by construction to mask-derived inputs, so the score is 6 rather than 0-2.
Assumptions & free parameters
free parameters (5)
- learning_rate =
0.005
- lr_decay_factor =
0.1 every 50 epochs
- noise_tunnel_samples =
5
- morphology_parameters
- ncut_segment_count
assumptions (4)
- domain assumption DINO ViT features transfer to medical imaging modalities
- domain assumption Integrated Gradients attribution identifies pixels relevant to the segmentation target
- domain assumption Normalized cut on the relevance graph separates foreground from background
- domain assumption DenseCRF refinement improves boundary accuracy
Cite this review
Pith. "Pith review of No Masks Needed: Explainable AI for Deriving Segmentation from Classification." pith.science (2026). https://pith.science/paper/UXPFFJKP
@misc{pith2026250804534,
author = {Pith},
title = {Pith review of: No Masks Needed: Explainable AI for Deriving Segmentation from Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/UXPFFJKP}},
note = {Machine review of arXiv:2508.04534}
}
read the original abstract
Medical image segmentation is vital for modern healthcare and is a key element of computer-aided diagnosis. While recent advancements in computer vision have explored unsupervised segmentation using pre-trained models, these methods have not been translated well to the medical imaging domain. In this work, we introduce a novel approach that fine-tunes pre-trained models specifically for medical images, achieving accurate segmentation with extensive processing. Our method integrates Explainable AI to generate relevance scores, enhancing the segmentation process. Unlike traditional methods that excel in standard benchmarks but falter in medical applications, our approach achieves improved results on datasets like CBIS-DDSM, NuInsSeg and Kvasir-SEG.
Reference graph
Works this paper leans on
-
[1]
Deep learning for medical image segmentation: State-of-the-art advancements and challenges,
Md Eshmam Rayed, SM Sajibul Islam, Sadia Islam Niha, Jamin Rahman Jim, Md Mohsin Kabir, and MF Mridha, “Deep learning for medical image segmentation: State-of-the-art advancements and challenges,” Informatics in Medicine Unlocked, p. 101504, 2024
work page 2024
-
[2]
A brief introduction to weakly supervised learning,
Zhi-Hua Zhou, “A brief introduction to weakly supervised learning,” National science review, vol. 5, no. 1, pp. 44–53, 2018. 12 International Conference on Digital Image Processing and Vision (ICDIPV)
work page 2018
-
[3]
A comprehensive survey on transfer learning,
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He, “A comprehensive survey on transfer learning,” Proceedings of the IEEE, vol. 109, no. 1, pp. 43–76, 2021
work page 2021
-
[4]
Self- supervised transformers for unsupervised object discovery using normalized cut,
Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz, “Self- supervised transformers for unsupervised object discovery using normalized cut,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14543–14553
work page 2022
-
[5]
Cut and learn for unsupervised object detection and instance segmentation,
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra, “Cut and learn for unsupervised object detection and instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 3124–3134
work page 2023
-
[6]
Unsuper- vised object localization: Observing the background to discover objects,
Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín V obeck`y, Éloi Zablocki, and Patrick Pérez, “Unsuper- vised object localization: Observing the background to discover objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023
work page 2023
-
[7]
Explainable artificial intelligence approaches: A survey,
Sheikh Rabiul Islam, William Eberle, Sheikh Khaled Ghafoor, and Mohiuddin Ahmed, “Explainable artificial intelligence approaches: A survey,” 2021
work page 2021
-
[8]
Medical image segmentation: a review,
Dinesh D Patil and Sonal G Deore, “Medical image segmentation: a review,” International Journal of Computer Science and Mobile Computing, vol. 2, no. 1, pp. 22–27, 2013
work page 2013
Show all 30 references
-
[9]
A review of medical image segmentation algorithms,
KKD Ramesh, G Kiran Kumar, K Swapna, Debabrata Datta, and S Suman Rajest, “A review of medical image segmentation algorithms,” EAI Endorsed Transactions on Pervasive Health and Technology, vol. 7, no. 27, pp. e6–e6, 2021
2021
-
[10]
Handbook of medical imaging: V olume 2, medical image processing and analysis,
Milan Sonka, J Michael Fitzpatrick, et al., “Handbook of medical imaging: V olume 2, medical image processing and analysis,” SPIE, 2000
2000
-
[11]
An image is worth 16x16 words: Transformers for image recognition at scale,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, “An image is worth 16x16 words: Transformers for image recognition a...
2021
-
[12]
Emerging properties in self-supervised vision transformers,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660
2021
-
[13]
Axiomatic attribution for deep networks,
Mukund Sundararajan, Ankur Taly, and Qiqi Yan, “Axiomatic attribution for deep networks,” 2017
2017
-
[14]
Grad-cam: Why did you say that?,
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra, “Grad-cam: Why did you say that?,” 2017
2017
-
[15]
From explanations to segmentation: Using explainable ai for image segmentation,
Clemens Seibold, Johannes Künzel, Anna Hilsmann, and Peter Eisert, “From explanations to segmentation: Using explainable ai for image segmentation,” in Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applicati...
2022
-
[16]
From classification to segmentation with explainable ai: A study on crack detection and growth monitoring,
Florent Forest, Hugo Porta, Devis Tuia, and Olga Fink, “From classification to segmentation with explainable ai: A study on crack detection and growth monitoring,” 2024
2024
-
[17]
Growing a brain: Fine-tuning by increasing model capacity,
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert, “Growing a brain: Fine-tuning by increasing model capacity,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2471–2480
2017
-
[18]
Captum: A unified and generic model interpretability library for pytorch,
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson, “Captum: A unified and generic model interpretability library for pytorch,” 2020
2020
-
[19]
Deep vit features as dense visual descriptors,
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel, “Deep vit features as dense visual descriptors,” arXiv preprint arXiv:2112.05814, vol. 2, no. 3, pp. 4, 2021
2021 arXiv
-
[20]
Survey over image thresholding techniques and quantitative perfor- mance evaluation,
Mehmet Sezgin and Bülent Sankur, “Survey over image thresholding techniques and quantitative perfor- mance evaluation,” J. Electronic Imaging, vol. 13, pp. 146–168, 2004
2004
-
[21]
Adaptive histogram equalization and its variations,
Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld, “Adaptive histogram equalization and its variations,” Computer vision, graphics, and image processing, vol. 39, no. 3, p...
1987
-
[22]
Normalized cuts and image segmentation,
Jianbo Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888–905, 2000. 13 International Conference on Digital Image Processing and Vision (ICDIPV)
2000
-
[23]
Efficient inference in fully connected crfs with gaussian edge potentials,
Philipp Krähenbühl and Vladlen Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” Advances in neural information processing systems, vol. 24, 2011
2011
-
[24]
A curated mammography data set for use in computer-aided detection and diagnosis research,
Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin, “A curated mammography data set for use in computer-aided detection and diagnosis research,” Scientific Data, vol. 4, pp. 170177, 2017
2017
-
[25]
Nuinsseg: A fully annotated dataset for nuclei instance segmentation in h&e-stained histological images,
Amirreza Mahbod, Christine Polak, Katharina Feldmann, Rumsha Khan, Katharina Gelles, Georg Dorffner, Ramona Woitek, Sepideh Hatamikia, and Isabella Ellinger, “Nuinsseg: A fully annotated dataset for nuclei instance segmentation in h&e-stained histological images,” 2023
2023
-
[26]
Kvasir-seg: A segmented polyp dataset,
Debesh Jha, Pia H. Smedsrud, Michael A. Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D. Johansen, “Kvasir-seg: A segmented polyp dataset,” 2019
2019
-
[27]
Automatic differentiation in pytorch,
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer, “Automatic differentiation in pytorch,” 2017
2017
-
[28]
A stochastic approximation method,
Herbert Robbins and Sutton Monro, “A stochastic approximation method,” The annals of mathematical statistics, 1951
1951
-
[29]
Microscopy analysis neural network to solve detection, enumeration and segmentation from image-level annotations,
Anthony Bilodeau, Constantin VL Delmas, Martin Parent, Paul De Koninck, Audrey Durand, and Flavie Lavoie-Cardinal, “Microscopy analysis neural network to solve detection, enumeration and segmentation from image-level annotations,” Nature Machine Intelligence, vol. 4, no. 5, pp...
2022
-
[30]
Neuronal activity remodels the f-actin based submembrane lattice in dendrites but not axons of hippocampal neurons,
Flavie Lavoie-Cardinal, Anthony Bilodeau, Mado Lemieux, Marc-André Gardner, Theresa Wiesner, Gabrielle Laramée, Christian Gagné, and Paul De Koninck, “Neuronal activity remodels the f-actin based submembrane lattice in dendrites but not axons of hippocampal neurons,” Scientifi...
2020
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.