REVIEW 3 major objections 5 minor 13 references
Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims hyper-reflective foci in OCT can be segmented from image-level labels alone by prompting SAM2 with the most relevant pixel from a relevance map, and that a compact convolutional transformer more than doubles Dice over…
desk verdict The crop-and-upsample SAM2 prompting is a genuine technical adaptation, but the paper never measures whether the max-LRP pixel actually lands on an HRF, and that unmeasured assumption likely explains most of the reported gains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the relevance map produced by layer-wise relevance propagation (LRP), a backpropagation-style rule that assigns each input pixel a score measuring how much it drove the image-level prediction. LRP turns the classifier's global decision into per-pixel evidence, and the pixel with the highest relevance becomes the SAM2 prompt; without this map there is no spatially precise cue for the foundation model. Around the prompt, a fixed-size box (4 pixels) and a cropped, upsampled region (50–100 pixels) make the HRF large enough for SAM2, which is otherwise poor at small objects. The supporting machinery is the Compact Convolutional Transformer (CCT), which forms tokens through convolutional layers and includes positional encoding, allowing attention to exchange information between patches so that small structures on patch borders are less likely to be missed.
What would settle it
For each test B-scan, compare the maximally relevant pixel against the annotated HRF masks and compute the fraction of prompts that fall inside an HRF; if that fraction is low yet SAM2 still outputs useful masks, the reported Dice gain is not coming from the claimed mechanism. A complementary ceiling check is to prompt SAM2 with the true centers of all annotated HRFs and compare the resulting masks with the weakly supervised ones.
Extended reading notes
Core claim
The central discovery, in the authors' framing, is that a coarse image-level decision can be converted into a precise pixel-level segmentation by treating the classifier's explanation as a prompt for a foundation model. They implement this with layer-wise relevance propagation on either an attention-based MIL classifier or a Compact Convolutional Transformer, take the pixel of maximum relevance, crop and upsample a local region so that the tiny HRF becomes large enough for SAM2, and use a small box around that pixel as the prompt. Iterative inference occludes already-detected HRFs and re-runs the pipeline to find additional foci, which mainly helps images containing more than one HRF (Dice 0.22 after the first iteration, 0.28 after three). Replacing MIL with CCT, which adds positional encoding and lets different image regions exchange information through attention, raises Dice from 0.13 to 0.33 on the 2-class task; the authors interpret this as the CCT catching HRFs that fall on patch borders. They also report that binary presence/absence labels are sufficient, with multi-class or count-based labels offering no additional segmentation benefit.
Load-bearing premise
The load-bearing premise is that the most influential pixel identified by the classifier's pixel-level explanation actually lies inside a hyper-reflective focus whenever the classifier says one is present, and the paper never measures that point-level hit rate.
Editorial extensions
If this is right
- Binary image-level labels are enough: adding extra classes or an HRF-count regression target does not improve segmentation Dice.
- Prompting SAM2 with the most relevant LRP pixel substantially outperforms thresholding the relevance map directly (0.33 vs 0.16 Dice), and reaches near the 0.35 ceiling of an oracle threshold tuned per sample with ground truth.
- Iterative inference mostly helps difficult multi-HRF images, raising their Dice from 0.22 after one iteration to 0.28 after three, while on the average B-scan recall gains are offset by precision losses.
- Full-resolution processing matters: downsampling to 256×256 or 224×224 would discard 16% or 22% of annotated HRFs, so a full-resolution classifier like CCT is a prerequisite for this approach.
- The authors expect the same pipeline to transfer to other small-structure segmentation problems in OCT, not just HRF.
Reading between the lines
- The paper does not test whether the maximally relevant pixel usually lands inside an HRF; computing that hit rate on the test set would isolate whether the Dice gain comes from accurate localization or from SAM2 segmenting plausible bright spots even when prompted off-target.
- The closeness of the SAM-based Dice (0.33) to the oracle-threshold ceiling (0.35) suggests that with a perfect prompt the remaining gap is set by SAM2's mask quality, pointing to fine-tuning SAM2 on small objects or using multiple prompt points as unexplored next steps.
- The CCT was trained from scratch while the MIL encoder used ImageNet-pretrained weights, so the reported architecture advantage may partly reflect training protocol; an ablation with a from-scratch full-resolution MIL would separate the two.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a weakly supervised segmentation framework for hyper-reflective foci (HRF) in OCT images. The framework first trains a multiple instance learning (MIL) model or a compact convolutional transformer (CCT) on image-level labels (binary, multi-class, or count regression). Layer-wise relevance propagation (LRP) then produces pixel-level relevance maps, from which the most relevant pixel is used to define a bounding-box prompt for SAM 2; images are iteratively re-run after occluding detected HRFs. On a dataset of 962 HRF-positive B-scans, the authors report that the CCT with binary labels achieves a test Dice of 0.33 after six iterations, compared with 0.13 for MIL and 0.16 for simple relevance-map thresholding. They conclude that replacing MIL with CCT yields a substantial improvement in segmentation accuracy.
Significance. The proposed pipeline is methodologically transparent: prompt geometry is tuned on training data only, the threshold baseline is calibrated on validation data, and an oracle threshold is reported as an upper bound. The comparison of SAM 2 prompting against relevance-map thresholding is a useful sanity baseline, and the study covers three weak-label types. However, the paper's central mechanism -- prompting SAM 2 at the maximum-relevance pixel -- is not validated by any point-localization metric. The reported Dice gain of CCT over MIL and of SAM 2 over thresholding is therefore conditional on an untested assumption about the relevance localization quality. If the assumption holds, the framework is a practical step toward full-resolution weakly supervised segmentation of small structures; the paper provides evidence for the relative benefit of CCT over MIL, but not yet for the reliability of the prompting strategy.
major comments (3)
- [Sec. 2.3 and Table 1] The prompting strategy relies on the most relevant pixel from the LRP relevance map lying on an HRF, yet the paper never measures this point-level localization accuracy. The authors report a mean Dice of about 0.67 when prompting with ground-truth HRF centers (Sec. 2.3), while the full pipeline achieves 0.33 (Table 1), a 0.34 gap that is likely dominated by prompt-localization error. If the max-relevance pixel frequently falls on background or on non-HRF bright structures, the iterative occlusion loop cannot recover from the first miss, and the headline comparison between CCT (0.33) and MIL (0.13) may reflect differences in relevance-map noise rather than a genuine segmentation improvement. Please add a quantitative analysis of prompt localization (e.g., hit rate within a few pixels of an HRF, or distance from the selected pixel to the nearest HRF) for both CCT and MIL, and condition the segmentation results on whether the prompt hit.
- [Section 3 and Table 1] All classification and segmentation metrics are point estimates from a single 80/20 split. The paper's central claim that CCT 'consistently outperforms' MIL and that SAM 2 postprocessing yields a 'substantial increase' in Dice is made without confidence intervals or multiple-seed training. Given the small test set and the high variance typical of Dice on small structures, the reported differences (e.g., CCT 0.33 vs. MIL 0.13 in Table 1) need at least bootstrap confidence intervals over B-scans or a multi-seed analysis to assess stability.
- [Introduction and Section 3] The paper cites TSSK-Net and MS-CAM as earlier weakly supervised segmentation methods, but it does not include any of these as a baseline. Without a comparison to at least one existing method, the contribution is positioned only against relevance-map thresholding and the MIL variant; the reader cannot judge whether the proposed framework advances the state of the art in weakly supervised HRF segmentation. If a direct comparison is infeasible, the authors should state this explicitly and justify the omission.
minor comments (5)
- [Section 3] In the Segmentation paragraph, 'even more pronounced than for the segmentation task' should presumably read 'than for the classification task'; the sentence is comparing segmentation differences to classification differences.
- [Section 2.1] The phrase 'manually annoted' is a typo for 'manually annotated'.
- [Section 2.3] The stopping threshold of 0.05 and the maximum of six iterations are introduced without explaining how these values were chosen; please state whether they were fixed a priori or tuned on validation data.
- [Section 4] The sentence 'Comparing Dice scores to those from an oracle-based thresholding ... suggests that we cannot expect to outperform SAM 2' confuses the two postprocessing approaches; it should read '...cannot expect to outperform the oracle thresholding'.
- [Table 1] The columns 'Segm. 1 Iteration' and 'Segm. 6 Iterations' should clarify that the latter is subject to the early-stopping threshold; reporting the average number of iterations would help interpret the recall/precision trade-off.
Circularity Check
No significant circularity: reported test Dice values are not fitted to test data, the oracle threshold is explicitly an upper bound, and the grid search is confined to training HRFs.
full rationale
The derivation chain is empirically self-contained. The classifiers are trained only on image-level labels; LRP relevance maps are then used to prompt SAM2 on the held-out test set, and the reported Dice values in Tab. 1 and Tab. 2 are computed on that test set. The crop and box sizes were selected via a grid search over the training HRFs (Sec. 2.3), not the test set, so no test metric is fitted. The oracle threshold in Tab. 2 is explicitly an upper-bound comparison ('the threshold is determined per sample using the ground truth segmentation as a guide'), so it cannot serve as a fitted input for the test numbers. The stopping threshold of 0.05 is a hand-set hyperparameter and the iterative inference is judged by the same held-out metrics. The unmeasured point-level localization accuracy of the max-relevance pixel is a genuine limitation and a risk to the mechanism, but it is an unsupported assumption rather than a definitional or self-citational reduction: no equation equates the predicted quantity to a fitted parameter, and no fitted value is renamed as a prediction. The only self-citation, reference [12] for the HRF annotations, is data provenance and is not load-bearing for the algorithmic comparison between CCT, MIL, thresholding, and SAM prompting. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (3)
- SAM2 prompt box size =
4 pixels
- SAM2 crop size =
50-100 pixels (range)
- Iterative inference stopping threshold =
0.05
assumptions (3)
- domain assumption Manual HRF annotations in the LEAD dataset (excluding those under 5 pixels) are a reliable ground truth.
- domain assumption The LRP extension for transformers [13] yields conservative, meaningfully localizing relevance maps on the CCT.
- domain assumption SAM2, pretrained on natural images, can segment OCT HRFs when prompted with a box and upsampled crop.
Cite this review
Pith. "Pith review of Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2." pith.science (2026). https://pith.science/paper/Q6TWZTTZ
@misc{pith2026250105933,
author = {Pith},
title = {Pith review of: Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q6TWZTTZ}},
note = {Machine review of arXiv:2501.05933}
}
read the original abstract
Weakly supervised segmentation has the potential to greatly reduce the annotation effort for training segmentation models for small structures such as hyper-reflective foci (HRF) in optical coherence tomography (OCT). However, most weakly supervised methods either involve a strong downsampling of input images, or only achieve localization at a coarse resolution, both of which are unsatisfactory for small structures. We propose a novel framework that increases the spatial resolution of a traditional attention-based Multiple Instance Learning (MIL) approach by using Layer-wise Relevance Propagation (LRP) to prompt the Segment Anything Model (SAM~2), and increases recall with iterative inference. Moreover, we demonstrate that replacing MIL with a Compact Convolutional Transformer (CCT), which adds a positional encoding, and permits an exchange of information between different regions of the OCT image, leads to a further and substantial increase in segmentation accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
Prevalence and IncidenceofAge-RelatedMacularDegenerationinEurope:ASystematicReviewandMeta- Analysis
Li JQ, Welchowski T, Schmid M, Mauschitz MM, Holz FG, Finger RP. Prevalence and IncidenceofAge-RelatedMacularDegenerationinEurope:ASystematicReviewandMeta- Analysis. Br J Ophthalmol. 2020;104(8):1077–84
work page 2020
-
[2]
Verma A, Corradetti G, He Y, Nittala MG, Nassisi M, Velaga SB et al. Relationship between the Distribution of Intra-Retinal Hyper-Reflective Foci and the Progression of Intermediate Age-Related Macular Degeneration. Graefes Arch Clin Exp Ophthalmol. 2023;261(12):3437–47
work page 2023
-
[3]
MaX,JiZ,NiuS,LengT,RubinDL,ChenQ.MS-CAM:Multi-ScaleClassActivationMaps for Weakly-Supervised Segmentation of Geographic Atrophy Lesions in SD-OCT Images. IEEE J Biomed Health Inform. 2020;24(12):3443–55
work page 2020
-
[4]
Wang J, Li W, Chen Y, Fang W, Kong W, He Y et al. Weakly Supervised Anomaly Seg- mentation in Retinal OCT Images Using an Adversarial Learning Approach. Biomed Opt Express. 2021;12(8):4713
work page 2021
-
[5]
Liu X, Liu Q, Zhang Y, Wang M, Tang J, Liu X et al. TSSK-Net: Weakly Supervised Biomarker Localization and Segmentation with Image-Level Annotation in Retinal OCT Images. Comput Biol Med. 2023:106467
work page 2023
-
[6]
A Foundation Model for Generalizable Disease Detection from Retinal Images
Zhou Y, Chia MA, Wagner SK, Ayhan MS, Williamson DJ, Struyven RR et al. A Foundation Model for Generalizable Disease Detection from Retinal Images. Nature. 2023;622(7981):156–63
work page 2023
-
[7]
Attention-Based Deep Multiple Instance Learning
Ilse M, Tomczak J, Welling M. Attention-Based Deep Multiple Instance Learning. Proceed- ings of the 35th International Conference on Machine Learning. PMLR, 2018:2127–36. Weakly Supervised Hyper-Reflective Foci Segmentation 7
work page 2018
-
[8]
On Pixel-Wise Ex- planations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On Pixel-Wise Ex- planations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS One. 2015;10(7):e0130140
work page 2015
Show all 13 references
-
[9]
SAM 2: Segment Anything in Images and Videos
Ravi N, Gabeur V, Hu Y, Hu R, Ryali C, Ma T et al. SAM 2: Segment Anything in Images and Videos. CoRR. 2024;abs/2408.00714
2024 arXiv
-
[10]
Escaping the Big Data Paradigm with Compact Transformers
Hassani A, Walton S, Shah N, Abuduweili A, Li J, Shi H. Escaping the Big Data Paradigm with Compact Transformers. CoRR. 2021;abs/2104.05704
2021 arXiv
-
[11]
Ophthalmology
GuymerRH,WuZ,HodgsonLAB,CarusoE,BrassingtonKH,TindillNetal.Subthreshold NanosecondLaserInterventioninAge-RelatedMacularDegeneration:TheLEADRandom- ized Controlled Clinical Trial. Ophthalmology. 2019;126(6):829–38
2019
-
[12]
Hyper- reflective Foci not Seen as Hyperpigmentary Abnormalities on Color Fundus Photographs in Age-Related Macular Degeneration
Goh KL, Wintergerst MWM, Abbott CJ, Hadoux X, Jannaud M, Kumar H et al. Hyper- reflective Foci not Seen as Hyperpigmentary Abnormalities on Color Fundus Photographs in Age-Related Macular Degeneration. Retina. 2024;44(2):214–21
2024
-
[13]
XAI for Transformers: Better Explanations through Conservative Propagation
Ali A, Schnake T, Eberle O, Montavon G, Müller KR, Wolf L. XAI for Transformers: Better Explanations through Conservative Propagation. Proc. 39th International Conference on Machine Learning. Vol. 162. PMLR, 2022:435–51
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.