REVIEW 3 major objections 7 minor 42 references
A hinge regularizer that borrows coarse anatomy masks from pretrained segmentation models steers medical classifiers toward the right tissue and raises accuracy across eight datasets.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 08:46 UTC pith:V5ITZBIT
load-bearing objection Clean plug-and-play hinge regularizer that turns frozen medical SFMs into anatomy priors; consistent gains, real but bounded novelty. the 3 major comments →
Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Locus establishes that an adaptive hinge comparing aggregate attention inside versus outside a foundation-model anatomical mask is sufficient spatial supervision for medical image classification: it yields consistent gains in accuracy and F1-score together with more anatomically grounded attention maps across eight diverse datasets, without dedicated segmentation training or expert pixel annotations.
What carries the argument
The anatomy-centric hinge regularizer R(M,A)=max(0,A_bg−A_fg), which activates only when background attention dominates foreground attention; masks M are obtained by union of patch-wise text prompts to a frozen segmentation foundation model.
Load-bearing premise
The method assumes that the coarse, un-fine-tuned masks from patch-wise prompting of the foundation model correctly cover the diagnostically relevant anatomy on every dataset; if those masks systematically miss or misplace the true structures, the hinge supplies wrong spatial guidance.
What would settle it
On a held-out set where the foundation model systematically omits the true lesion or organ, measure whether Locus accuracy drops below plain cross-entropy while Grad-CAM still concentrates on the incorrect masked region; that outcome would refute the claim that the free masks supply useful supervision.
If this is right
- Medical classifiers can obtain anatomical focus and higher scores with zero extra pixel annotation cost.
- The same plug-in regularizer works across CNN and transformer backbones and multiple saliency extractors.
- Coarse foundation-model masks suffice when the loss only demands anatomy-dominant rather than pixel-perfect attention.
- Robustness to brightness, contrast, blur and noise improves because background-correlated shortcuts are discouraged.
- Temporal sequences such as cardiac cine MRI can be regularized with the same spatial hinge.
Where Pith is reading between the lines
- As segmentation foundation models improve, free anatomical priors of this kind could become default training practice for any anatomy-dependent medical task.
- The hinge idea may transfer outside medicine whenever a cheap off-the-shelf detector can supply a soft region prior.
- Learning the patch grid end-to-end, as the authors note for future work, would likely close remaining cases of missed small or sparse structures.
- Uncertainty-aware down-weighting of unreliable foundation masks would let the regularizer back off automatically on out-of-distribution anatomy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Locus, an anatomy-guided attention regularization framework for medical image classification. Pretrained segmentation foundation models (BiomedParse, Medical SAM3) supply anatomical masks via a patch-wise text-prompting strategy; a hinge regularizer R = max(0, A_bg − A_fg) (Eq. 3) then penalizes the classifier only when Grad-CAM background attention exceeds foreground attention. The regularizer is added to standard cross-entropy with a warm-up schedule (Eq. 4). The method is claimed to be plug-and-play across backbones and saliency extractors, requiring neither manual pixel annotations nor fine-tuning of the foundation model. Empirical validation spans multiple public datasets (dermoscopy, X-ray, histopathology, ultrasound) plus an in-house cardiac cine MRI set, with reported accuracy/F1 gains over CE-only, masked-input, and MGA baselines, plus qualitative improvements in attention localization and robustness under test-time perturbations.
Significance. If the reported gains hold under stronger statistical scrutiny and more complete reporting, Locus is a practically useful contribution: it converts frozen, off-the-shelf medical SFMs into spatial supervision for classifiers without the annotation cost of mask-guided training or the optimization friction of joint segmentation–classification. The hinge design is deliberately tolerant of coarse masks, the method is architecture- and saliency-agnostic, and the multi-modality evaluation (including temporal cine MRI) is broader than typical single-domain studies. Code is promised (anonymous repository). These are genuine engineering strengths for MICCAI-style venues. The work does not introduce a new theoretical principle; its value is empirical and methodological.
major comments (3)
- Section 2.2 and Fig. 3: The central claim rests on the assumption that un-fine-tuned SFM masks obtained by patch-wise text prompting are sufficiently complete and correctly localized for the diagnostically relevant anatomy on every dataset. Only qualitative examples are shown; there is no quantitative mask-quality evaluation (e.g., coverage of lesion/organ, false-positive background rate, or failure-case rate) on the eight evaluation sets. Because R (Eq. 3) uses these masks as the sole spatial prior, systematic mislocalization would supply incorrect supervision. A short quantitative audit of mask quality (or explicit failure analysis) is needed to bound this risk.
- Table 1: Accuracy and F1 gains are reported as single point estimates with no standard deviations across seeds/runs and no statistical significance tests. Several improvements are modest (e.g., PneumoniaMNIST ResNet Acc 89.90 → 92.15; Cardiac DenseNet Acc 78.32 → 82.02). Without multi-seed statistics it is difficult to judge whether the “consistent gains” claim is reliable. At minimum, report mean ± std over ≥3 seeds and a paired test or confidence interval for the main comparisons.
- Abstract and §3 claim validation on eight datasets, yet Table 1 reports only six (HAM10000, PneumoniaMNIST, BTXRD, PanNuke, ChestX-ray8, Cardiac). BUSI and CheXpert appear in figures and text but lack the same quantitative baseline comparison. Either complete the main table for all eight datasets or revise the claim to match the reported numbers.
minor comments (7)
- Table 2 (λ ablation): performance is non-monotonic (dips at λ = 0.3 and 0.7). A brief discussion of why intermediate strengths can over-constrain the classifier would help readers choose λ.
- §3.1 Implementation: number of patches P is “selected empirically per dataset” with no sensitivity study or default rule. A short ablation or heuristic would improve reproducibility.
- Fig. 1 caption and several figure panels use “SOTA classifiers” without naming which models are shown; label the baselines explicitly.
- Leftover running head “Title Suppressed Due to Excessive Length” appears on multiple pages; remove before camera-ready.
- Medical SAM3 is cited as arXiv:2601.10880 (2026); confirm the citation is stable and publicly available at submission time.
- §3 Datasets: exclusion of “No Finding” / normal classes is reasonable but should be stated once with a clear criterion so readers can judge selection bias.
- Clarify whether Grad-CAM is computed only on the predicted class or on the ground-truth class during training of R; this affects the gradient path.
Circularity Check
No circularity: empirical regularization method whose hinge and SFM masks are independent of the classification objective and whose gains are measured on external benchmarks.
full rationale
Locus is a purely empirical method paper. The anatomy-centric regularizer is defined as R(M,A)=max(0,A_bg-A_fg) (Eq. 3) from Grad-CAM attention aggregates over an externally supplied, frozen SFM mask M; this expression does not contain the class labels y, is not fitted to any classification metric, and is not claimed to be a first-principles derivation of anything. The network loss (Eq. 4) simply adds a weighted hinge to ordinary cross-entropy; the reported accuracy/F1 gains (Table 1) and attention maps (Figs. 4–7) are ordinary empirical comparisons against CE, Masked and MGA baselines on eight held-out datasets. Patch-wise prompting is a fixed, non-learned heuristic. Author self-citations ([7],[8]) appear only as background and are not load-bearing for uniqueness, ansatz or any claimed prediction. Nothing reduces by construction to its own inputs.
Axiom & Free-Parameter Ledger
free parameters (3)
- λ_max (regularization strength) =
0.1
- number of patches P =
dataset-specific
- T_warmup / T_ramp =
5 / 5
axioms (3)
- domain assumption Grad-CAM (or other listed saliency maps) is a faithful enough proxy for the spatial regions the classifier uses for its decision.
- domain assumption Pretrained medical segmentation foundation models localize the diagnostically relevant anatomy sufficiently well across the eight modalities without any fine-tuning.
- ad hoc to paper Penalizing only the case A_bg > A_fg is sufficient; no pixel-wise alignment or fixed foreground/background ratio is required.
invented entities (2)
-
Locus hinge regularizer R = max(0, A_bg − A_fg)
no independent evidence
-
Patch-wise anatomical prompting (PAP)
no independent evidence
read the original abstract
Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework, Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier's attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention.
Figures
Reference graph
Works this paper leans on
-
[1]
Data in brief28, 104863 (2020)
Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A.: Dataset of breast ultrasound images. Data in brief28, 104863 (2020)
2020
-
[2]
In: 2018 IEEE winter conference on applications of computer vision (WACV)
Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV). pp. 839–847. IEEE (2018)
2018
-
[3]
Nature Machine Intelligence3(7), 610–619 (2021)
DeGrave, A.J., Janizek, J.D., Lee, S.I.: Ai for radiographic covid-19 detection se- lects shortcuts over signal. Nature Machine Intelligence3(7), 610–619 (2021)
2021
-
[4]
In: European congress on digital pathology
Gamper, J., Alemi Koohbanani, N., Benet, K., Khuram, A., Rajpoot, N.: Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classi- fication. In: European congress on digital pathology. pp. 11–19. Springer (2019)
2019
-
[5]
IEEE Transactions on Image Processing (2025)
Gao, H., Sheng, R., Su, Y., Chen, Z., Xu, S., Gao, L.: Multiscale segmentation- guided fusion network for hyperspectral image classification. IEEE Transactions on Image Processing (2025)
2025
-
[6]
Nature Machine In- telligence2(11), 665–673 (2020)
Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine In- telligence2(11), 665–673 (2020)
2020
-
[7]
arXiv preprint arXiv:2411.12201 (2024)
Hossain, T., Ma, J., Li, J., Zhang, M.: Invariant shape representation learning for image classification. arXiv preprint arXiv:2411.12201 (2024)
Pith/arXiv arXiv 2024
-
[8]
In: 2019 1st international conference on advances in science, engineering and robotics technology (ICASERT)
Hossain, T., Shishir, F.S., Ashraf, M., Al Nasim, M.A., Shah, F.M.: Brain tumor detection using convolutional neural network. In: 2019 1st international conference on advances in science, engineering and robotics technology (ICASERT). pp. 1–6. IEEE (2019)
2019
-
[9]
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al.: Segment anything model for medical images? Medical Image Analysis92, 103061 (2024)
2024
-
[10]
Biomedical Signal Processing and Control75, 103553 (2022)
Inan, M.S.K., Alam, F.I., Hasan, R.: Deep integrated pipeline of segmentation guided classification of breast cancer from ultrasound images. Biomedical Signal Processing and Control75, 103553 (2022)
2022
-
[11]
In: Proceedings of the AAAI conference on artificial intelligence
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)
2019
-
[12]
arXiv preprint arXiv:2601.10880 (2026)
Jiang, C., Ding, T., Song, C., Tu, J., Yan, Z., Shao, Y., Wang, Z., Shang, Y., Han, T., Tian, Y.: Medical sam3: A foundation model for universal prompt-driven medical image segmentation. arXiv preprint arXiv:2601.10880 (2026)
arXiv 2026
-
[13]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Jiang, H., Miao, C.: Anatomy-aware gating network for explainable alzheimer’s disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 90–100. Springer (2024)
2024
-
[14]
In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
Jiang, H., Xu, J., Shi, R., Yang, K., Zhang, D., Gao, M., Ma, H., Qian, W.: A multi-label deep learning model with interpretable grad-cam for diabetic retinopa- thy classification. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). pp. 1560–1563. IEEE (2020) Title Suppressed Due to Excessive Length 15
2020
-
[15]
cell172(5), 1122–1131 (2018)
Kermany, D.S., Goldbaum, M., Cai, W., Valentim, C.C., Liang, H., Baxter, S.L., McKeown, A., Yang, G., Wu, X., Yan, F., et al.: Identifying medical diagnoses and treatable diseases by image-based deep learning. cell172(5), 1122–1131 (2018)
2018
-
[16]
arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Pith/arXiv arXiv 2014
-
[17]
In: Proceedings of the IEEE/CVF international conference on computer vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)
2023
-
[18]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Li, W., Ju, L., Tang, F., Xia, P., Xiong, X., Hu, M., Zhu, L., Ge, Z.: Towards realistic semi-supervised medical image classification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 4968–4976 (2025)
2025
-
[19]
Medical Image Analysis p
Lin, M., Holste, G., Wang, S., Zhou, Y., Wei, Y., Banerjee, I., Chen, P., Dai, T., Du, Y., Dvornek, N.C., et al.: Cxr-lt 2024: A miccai challenge on long-tailed, multi- label, and zero-shot disease classification from chest x-ray. Medical Image Analysis p. 103739 (2025)
2024
-
[20]
IEEE Transactions on Medical Imaging43(2), 674–685 (2023)
Ling, Y., Wang, Y., Dai, W., Yu, J., Liang, P., Kong, D.: Mtanet: Multi-task attention network for automatic medical image segmentation and classification. IEEE Transactions on Medical Imaging43(2), 674–685 (2023)
2023
-
[21]
Nature Communications15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
2024
-
[22]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Madan, C., Satia, A., Basu, S., Gupta, P., Dutta, U., Arora, C.: Focus on texture: rethinking pre-training in masked autoencoders for medical image classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 207–217. Springer (2025)
2025
-
[23]
Com- puter Methods and Programs in Biomedicine197, 105725 (2020)
Mahbod, A., Tschandl, P., Langs, G., Ecker, R., Ellinger, I.: The effects of skin le- sion segmentation on the performance of dermatoscopic image classification. Com- puter Methods and Programs in Biomedicine197, 105725 (2020)
2020
-
[24]
In: European conference on computer vision
Müller, P., Kaissis, G., Zou, C., Rueckert, D.: Joint learning of localized represen- tations from medical images and reports. In: European conference on computer vision. pp. 685–701. Springer (2022)
2022
-
[25]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Müller, P., Meissen, F., Brandt, J., Kaissis, G., Rueckert, D.: Anatomy-driven pathology detection on chest x-rays. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 57–66. Springer (2023)
2023
-
[26]
IEEE trans- actions on medical imaging39(12), 4011–4022 (2020)
Noothout, J.M., De Vos, B.D., Wolterink, J.M., Postma, E.M., Smeets, P.A., Takx, R.A., Leiner, T., Viergever, M.A., Išgum, I.: Deep learning-based regression and classification for automatic landmark localization in medical images. IEEE trans- actions on medical imaging39(12), 4011–4022 (2020)
2020
-
[27]
ACM Computing Surveys56(4), 1–41 (2023)
Patrício,C.,Neves,J.C.,Teixeira,L.F.:Explainabledeeplearningmethodsinmed- ical image classification: A survey. ACM Computing Surveys56(4), 1–41 (2023)
2023
-
[28]
PLoS medicine15(11), e1002686 (2018)
Rajpurkar, P., Irvin, J., Ball, R.L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C.P., et al.: Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLoS medicine15(11), e1002686 (2018)
2018
-
[29]
In: Proceedings of the IEEE international conference on computer vision
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
2017
-
[30]
In- ternational journal of computer vision128(2), 336–359 (2020) 16 Hossain et al
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: visual explanations from deep networks via gradient-based localization. In- ternational journal of computer vision128(2), 336–359 (2020) 16 Hossain et al
2020
-
[31]
Scientific data5(1), 180161 (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data5(1), 180161 (2018)
2018
-
[32]
In: MICCAI Workshop on Data Engineering in Medical Imaging
Uddin, I.I., Wang, L., Santosh, K.: Expert-guided explainable few-shot learning for medical image diagnosis. In: MICCAI Workshop on Data Engineering in Medical Imaging. pp. 95–104. Springer (2025)
2025
-
[33]
Cardiovascular Imaging15(3), 413–427 (2022)
Wang, S., Patel, H., Miller, T., Ameyaw, K., Narang, A., Chauhan, D., Anand, S., Anyanwu, E., Besser, S.A., Kawaji, K., et al.: Ai based cmr assessment of biven- tricular function: clinical significance of intervendor variability and measurement errors. Cardiovascular Imaging15(3), 413–427 (2022)
2022
-
[34]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classi- fication and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2097–2106 (2017)
2097
-
[35]
Medical Image Analysis98, 103304 (2024)
Xie, Y., Gu, L., Harada, T., Zhang, J., Xia, Y., Wu, Q.: Rethinking masked image modelling for medical image representation. Medical Image Analysis98, 103304 (2024)
2024
-
[36]
Scientific Data10(1), 41 (2023)
Yang,J.,Shi,R.,Wei,D.,Liu,Z.,Zhao,L.,Ke,B.,Pfister,H.,Ni,B.:Medmnistv2- a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data10(1), 41 (2023)
2023
-
[37]
IEEE Transactions on Medical Imaging44(5), 2244–2255 (2025)
Yang, Y., Fu, H., Aviles-Rivero, A.I., Xing, Z., Zhu, L.: Diffmic-v2: Medical im- age classification via improved diffusion network. IEEE Transactions on Medical Imaging44(5), 2244–2255 (2025)
2025
-
[38]
Scientific Data12(1), 88 (2025)
Yao, S., Huang, Y., Wang, X., Zhang, Y., Paixao, I.C., Wang, Z., Chai, C.L., Wang, H., Lu, D., Webb, G.I., et al.: A radiograph dataset for the classification, localization, and segmentation of primary bone tumors. Scientific Data12(1), 88 (2025)
2025
-
[39]
PLoS medicine15(11), e1002683 (2018)
Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J.J., Oermann, E.K.: Variable generalization performance of a deep learning model to detect pneumo- nia in chest radiographs: a cross-sectional study. PLoS medicine15(11), e1002683 (2018)
2018
-
[40]
IEEE transactions on medical imaging40(6), 1618–1631 (2021)
Zhang, Y., Li, H., Du, J., Qin, J., Wang, T., Chen, Y., Liu, B., Gao, W., Ma, G., Lei, B.: 3d multi-attention guided multi-task learning network for automatic gastric tumor segmentation and lymph node classification. IEEE transactions on medical imaging40(6), 1618–1631 (2021)
2021
-
[41]
Nature methods22(1), 166–176 (2025)
Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H.H., Kiblawi, S., Naumann, T., Gao, J., Crabtree, A., Abel, J., et al.: A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods22(1), 166–176 (2025)
2025
-
[42]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)
2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.