REVIEW 3 major objections 7 minor 44 references
SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection
T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read SAUGE claims a frozen SAM can serve as a multi-granularity edge detector, producing edges at any desired granularity with state-of-the-art results on standard benchmarks.
desk verdict Solid empirical adapter on SAM with real SOTA results, but the 'linear blending' label construction is actually cumulative OR — fix the story and the paper holds up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is the Side Transfer Network (STN), a stack of Feature Fusion Blocks inserted into the frozen SAM. Each Feature Fusion Block applies cross-attention and a gated-convolution feed-forward layer to fuse SAM's shallow features, image embedding, and mask embedding into edge-aware features, which a shared head turns into coarse, medium, and fine side outputs. The arbitrary-granularity output is a linear interpolation of the three side outputs indexed by α, and the supervision includes the OR-blended pseudo-labels from Eq. 5, a pairwise diversity loss among side outputs, and a mask-guided loss on the final fused output.
What would settle it
Train the same STN architecture on BSDS500 but shuffle the granularity assignment, e.g., supervise the coarse side output with the finest OR-blended label and the fine side output with the coarsest label; if the model still yields the same side-output ordering and comparable performance, then the multi-granularity behavior comes from the loss structure, not from SAM's features. Alternatively, run a linear probe on each frozen SAM layer against the three pseudo-label levels and check whether coarse-to-fine structure appears in the feature ordering.
Extended reading notes
Core claim
The paper's central claim is that SAM's intermediate features—the shallow encoder features, the image embedding, and the mask embedding—carry object-edge information at distinct granularities, and that these can be regressed into three side outputs ordered from coarse to fine. Those three side outputs can then be linearly combined to produce an edge map for any granularity level α in [0,1], where 0 is coarsest and 1 is finest. Supervision comes from sorted human annotations that are progressively OR-combined into coarse, medium, and fine pseudo-labels, together with a diversity loss that forces the side outputs to genuinely differ and a guide loss that uses SAM's own object masks to down-weight confusing pixels. The paper reports that this setup outperforms prior edge detectors on BSDS500, produces competitive results on Multicue, and transfers to NYUDv2 without any fine-tuning.
Load-bearing premise
The approach assumes that SAM's internal features genuinely represent edges at different levels of detail and that the progressively merged human labels correspond to those levels; if either is false, the multi-granularity outputs are artifacts of the label construction rather than uncertainty-aligned edge options.
Editorial extensions
If this is right
- A frozen foundation model plus a tiny adapter can replace heavy task-specific training for edge detection.
- Users can select edge density at inference time by changing α, with best-matched multi-candidate scores improving as more candidates are sampled.
- Because the SAM backbone stays frozen, cross-dataset generalization follows from SAM's priors, and the model transfers to a different domain without fine-tuning.
- The pseudo-label construction is independent of the backbone, so the same label-normalization recipe could be reused for other dense prediction tasks with multi-annotator labels.
Reading between the lines
- Editorial extension: the method treats granularity as a single scalar axis, which implicitly conflates 'amount of edge detail' with 'degree of annotator disagreement'; it would be worth testing whether α correlates with per-pixel annotator variance.
- Editorial extension: because the coarse-to-fine side outputs are supervised by OR-blended labels rather than by any direct measure of uncertainty, the 'uncertainty alignment' could be a descriptive slogan; a clean test is whether the same side-output ordering appears when annotations are shuffled.
- Editorial extension: the same STN and pseudo-label recipe could be attached to any frozen vision backbone, and comparing such a variant against SAUGE would isolate how much of the gain comes from SAM's features versus from the multi-granularity supervision itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAUGE, a lightweight adapter (about 1.5% of SAM parameters) injected into a frozen SAM for multi-granularity edge detection. The Side Transfer Network (STN) progressively fuses SAM's intermediate features to produce coarse, medium, and fine side outputs, which are supervised by pseudo-labels derived from the multiple human annotations. A post-hoc linear combination of the three side outputs yields edge maps at an arbitrary granularity α. The final output is trained with a mask-guided loss using SAM's object masks. Experiments on BSDS500, Multicue, and NYUDv2 report state-of-the-art or competitive results, including a zero-shot evaluation on NYUDv2 using the model trained on BSDS500 and PASCAL VOC.
Significance. If the results hold, the paper makes a useful empirical contribution: it shows that a very small trainable module on frozen SAM can outperform prior dedicated edge detectors while providing granularity control, and its zero-shot generalization to NYUDv2 is notable. The method is clearly specified and the evaluation protocol follows prior work. However, the central conceptual claim—that SAM's intermediate features inherently encode edge granularity and that the pseudo-label construction 'linearly blends' annotations to align uncertainty with granularity—is not supported by the actual equations, and no code is released. The empirical gains are plausible, but the interpretation needs substantial clarification before the scientific claim can be accepted.
major comments (3)
- [Method, Eq. (5)] The text repeatedly describes the pseudo-label construction as 'linear blending' (Abstract, Introduction, Method), but Eq. (5) defines Y^c = Y^S_1, Y^m = Y^c ∨ Y^S_ceil(N/2), and Y^f = Y^m ∨ Y^S_N, where ∨ is the element-wise OR. This is a cumulative union, not a linear blend: no weights are involved, and the medium and fine labels are obtained by monotonically adding edge pixels from sparser annotations. Consequently, the claimed 'normalization' of granularity and the connection to annotator uncertainty is a hand-crafted heuristic rather than an interpolation of labels. This mismatch is load-bearing because the multi-granularity supervision is central to the method's conceptual contribution. Please either correct the terminology and provide a principled justification for the cumulative-union construction, or compare against a true linear blending of labels (e.g., weighted averages of the binary maps) to show that the proposed construction is not an artifact of the label design.
- [Introduction / Method, 'key insight'] The paper asserts that 'intermediate SAM features inherently correspond to object edges at various granularities' and that this is the basis for aligning uncertainty with granularity. However, no direct evidence is provided for this correspondence; the STN is trained on the hand-designed pseudo-labels of Eq. (5), so the coarse-to-fine ordering of the outputs may simply reflect the label construction rather than an inherent property of SAM features. To substantiate the key insight, please include a layer-wise analysis (e.g., linear probing of SAM features at different depths against edge maps of different granularity, or an ablation that removes one SAM feature source and shows the granularity ordering degrades). Without such evidence, the claim that SAM provides a strong prior for granularity remains an interpretation rather than a demonstrated mechanism.
- [Method, Eq. (8)] The diversity loss L_diff is designed to make side outputs differ on pixels where the pseudo-labels differ. Because the pseudo-labels are nested (Y^c ⊆ Y^m ⊆ Y^f under Eq. 5), the XOR masks in Eq. (8) simply mark the newly added edge pixels. Thus the 'diversity' is fully predetermined by the cumulative-union construction, and the loss does not independently discover uncertainty structure. This reinforces the concern in the first comment: the diversity of the side outputs is a consequence of the label design. Please clarify what additional role L_diff plays beyond enforcing the already-imposed label hierarchy, and consider an ablation that removes L_diff while keeping the Eq. (5) labels to measure its marginal contribution.
minor comments (7)
- [Abstract] The dataset name 'Muticue' is a typo; it should be 'Multicue'.
- [Figure 4 and Table 3] 'DifussionEdge' is misspelled; the correct name is 'DiffusionEdge'.
- [Method, Eq. (2)] The symbol E_m is used for both the SAM mask embedding (Eq. 1, E^e_m) and the intermediate feature produced by FFB1 (Eq. 2, 'E_m = FFB1(...)'). Please use distinct notations to avoid confusion.
- [Experiments, Implementation Details] The BSDS500 and NYUDv2 results are reported from a single run without error bars or multiple seeds. Given the small performance gaps over prior methods, please report the variance or at least state that the differences are stable across runs.
- [Experiments, Multicue] The paper notes the lower AP on Multicue 'may be attributed to overfitting' but provides no supporting evidence. Please either provide a diagnostic (e.g., training/validation AP curves) or temper the claim.
- [Method, Eq. (10)] The definitions of ψ_j and ω_j are hard to parse: ψ_j uses both Y_mask and Y^mask with similar notation, and the role of the frequency map is unclear. Please clarify the notation and explain the intuition of the guide loss weight in more detail.
- [General] No code is released. Given the several non-obvious implementation details (e.g., exact SAM version, prompt grid, label construction), providing code or at least a detailed pseudocode in the supplementary material would substantially improve reproducibility.
Circularity Check
No circularity found: SAUGE trains a supervised adapter on pseudo-labels derived from human annotations and evaluates on standard splits; the multi-granularity outputs are explicit convex combinations, not fitted quantities.
full rationale
The paper's core derivation is a supervised learning pipeline: intermediate SAM features are fed through a lightweight Side Transfer Network and regressed to pseudo-labels constructed from human edge annotations (Eq. 5), with the final output supervised by a Gaussian-derived label and a mask-guide loss. This is standard supervised training, not a prediction that reduces to its inputs by construction, and the reported results are measured on held-out test splits (BSDS500 test, Multicue test, NYUDv2 zero-shot). The 'arbitrary granularity' output in Eq. 4 is an explicit piecewise linear interpolation of the three learned side outputs, so generating an edge map at any alpha is a declared interpolation operation rather than a fitted constant disguised as a prediction. The paper's text says pseudo-labels are created by 'linear blending,' while Eq. 5 actually defines cumulative OR operations; that is a real descriptive inconsistency and a scientific-correctness concern, but it is not circularity, because the pseudo-labels are training targets rather than claims derived from the model's own outputs. No load-bearing self-citation or imported uniqueness theorem is used: citations to UAED and MuGE concern standard loss designs and evaluation protocols, and the central architectural contribution is benchmarked against external methods without relying on the present authors' prior results. The acknowledged limitation (dependency on the SAM backbone) further indicates the authors are not presenting the framework as assumption-free. Overall, no circular step is exhibited by the paper's equations or claims.
Assumptions & free parameters
free parameters (3)
- threshold zeta for binarizing sampled final label =
0.2 (BSDS500), 0.3 (Multicue)
- loss weights lambda and beta =
lambda=0.1, beta=0.5
- points grid prompt for SAM =
8x8 grid
assumptions (4)
- domain assumption Intermediate SAM features encode edges at multiple granularities
- domain assumption Linear blending of binary human labels yields valid intermediate-granularity pseudo labels
- domain assumption Edge granularity is ordered by the number of edge pixels
- domain assumption Gaussian sampling of multiple annotations models label uncertainty
Cite this review
Pith. "Pith review of SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection." pith.science (2026). https://pith.science/paper/EZOWIZYD
@misc{pith2026241212892,
author = {Pith},
title = {Pith review of: SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/EZOWIZYD}},
note = {Machine review of arXiv:2412.12892}
}
read the original abstract
Edge labels are typically at various granularity levels owing to the varying preferences of annotators, thus handling the subjectivity of per-pixel labels has been a focal point for edge detection. Previous methods often employ a simple voting strategy to diminish such label uncertainty or impose a strong assumption of labels with a pre-defined distribution, e.g., Gaussian. In this work, we unveil that the segment anything model (SAM) provides strong prior knowledge to model the uncertainty in edge labels. Our key insight is that the intermediate SAM features inherently correspond to object edges at various granularities, which reflects different edge options due to uncertainty. Therefore, we attempt to align uncertainty with granularity by regressing intermediate SAM features from different layers to object edges at multi-granularity levels. In doing so, the model can fully and explicitly explore diverse ``uncertainties'' in a data-driven fashion. Specifically, we inject a lightweight module (~ 1.5% additional parameters) into the frozen SAM to progressively fuse and adapt its intermediate features to estimate edges from coarse to fine. It is crucial to normalize the granularity level of human edge labels to match their innate uncertainty. For this, we simply perform linear blending to the real edge labels at hand to create pseudo labels with varying granularities. Consequently, our uncertainty-aligned edge detector can flexibly produce edges at any desired granularity (including an optimal one). Thanks to SAM, our model uniquely demonstrates strong generalizability for cross-dataset edge detection. Extensive experimental results on BSDS500, Muticue and NYUDv2 validate our model's superiority.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Arbelaez, P.; Maire, M.; Fowlkes, C.; and Malik, J. 2010. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5): 898--916
2010
-
[4]
Canny, J. 1986. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6): 679--698
work page 1986
-
[5]
Cetinkaya, B.; Kalkan, S.; and Akbas, E. 2024. RankED: Addressing Imbalance and Uncertainty in Edge Detection Using Ranking-based Losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3239--3249
work page 2024
-
[6]
Chen, W.-T.; Vong, Y.-J.; Kuo, S.-Y.; Ma, S.; and Wang, J. 2024. RobustSAM: Segment Anything Robustly on Degraded Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4081--4091
work page 2024
-
[7]
Deng, R.; and Liu, S. 2020. Deep structural contour detection. In Proceedings of the 28th ACM international conference on multimedia, 304--312
work page 2020
-
[8]
Deng, R.; Liu, S.; Wang, J.; Wang, H.; Zhao, H.; and Zhang, X. 2021. Learning to decode contextual information for efficient contour detection. In Proceedings of the 29th ACM International Conference on Multimedia, 4435--4443
work page 2021
Show all 44 references
-
[9]
K.; Winn, J.; and Zisserman, A
Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303--338
2010
-
[10]
Gu, P.; Zhao, Z.; Wang, H.; Peng, Y.; Zhang, Y.; Sapkota, N.; Wang, C.; and Chen, D. Z. 2024. Boosting Medical Image Classification with Segmentation Foundation Model. arXiv preprint arXiv:2406.11026
2024 arXiv
-
[11]
Hallman, S.; and Fowlkes, C. C. 2015. Oriented edge forests for boundary detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1732--1740
2015
-
[12]
He, J.; Zhang, S.; Yang, M.; Shan, Y.; and Huang, T. 2019. Bi-directional cascade network for perceptual edge detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3828--3837
2019
-
[13]
Jiang, C.; Gao, F.; Ma, B.; Lin, Y.; Wang, N.; and Xu, G. 2023. Masked and adaptive transformer for exemplar based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22418--22427
2023
-
[14]
Ke, L.; Ye, M.; Danelljan, M.; Tai, Y.-W.; Tang, C.-K.; Yu, F.; et al. 2024. Segment anything in high quality. Advances in Neural Information Processing Systems, 36
2024
-
[15]
B.; et al
Kinga, D.; Adam, J. B.; et al. 2015. A method for stochastic optimization. In International conference on learning representations (ICLR), volume 5, 6. San Diego, California
2015
-
[16]
C.; Lo, W.-Y.; et al
Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015--4026
2023
-
[17]
Kittler, J. 1983. On the accuracy of the Sobel edge detector. Image and Vision Computing, 1(1): 37--42
1983
-
[18]
Kokkinos, I. 2015. Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386
2015 arXiv
-
[19]
Liu, J.-J.; Hou, Q.; and Cheng, M.-M. 2020. Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Transactions on Image Processing, 29: 8652--8667
2020
-
[20]
Liu, Y.; Cheng, M.-M.; Hu, X.; Wang, K.; and Bai, X. 2017. Richer convolutional features for edge detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3000--3009
2017
-
[21]
Liu, Y.; and Lew, M. S. 2016. Learning relaxed deep supervision for better edge detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 231--240
2016
-
[22]
Ma, J.; He, Y.; Li, F.; Han, L.; You, C.; and Wang, B. 2024. Segment anything in medical images. Nature Communications, 15(1): 654
2024
-
[23]
A.; Kim, J.; McGill, M.; Guo, Y.; and Serre, T
M \'e ly, D. A.; Kim, J.; McGill, M.; Guo, Y.; and Serre, T. 2016. A systematic comparison between visual cues for boundary detection. Vision research, 120: 93--107
2016
-
[24]
Nazeri, K.; Ng, E.; Joseph, T.; Qureshi, F.; and Ebrahimi, M. 2019. Edgeconnect: Structure guided image inpainting using edge prediction. In Proceedings of the IEEE/CVF international conference on computer vision workshops, 0--0
2019
-
[25]
Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32
2019
-
[26]
Pu, M.; Huang, Y.; Liu, Y.; Guan, Q.; and Ling, H. 2022. Edter: Edge detection with transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1402--1412
2022
-
[27]
Qin, X.; Zhang, Z.; Huang, C.; Gao, C.; Dehghan, M.; and Jagersand, M. 2019. Basnet: Boundary-aware salient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7479--7489
2019
-
[28]
Shen, W.; Wang, X.; Wang, Y.; Bai, X.; and Zhang, Z. 2015. Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3982--3991
2015
-
[29]
Silberman, N.; Hoiem, D.; Kohli, P.; and Fergus, R. 2012. Indoor segmentation and support inference from rgbd images. In Computer Vision--ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, 746--760. Springer
2012
-
[30]
Simonyan, K.; and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
2014 arXiv
-
[31]
Su, Z.; Liu, W.; Yu, Z.; Hu, D.; Liao, Q.; Tian, Q.; Pietik \"a inen, M.; and Liu, L. 2021. Pixel difference networks for efficient edge detection. In Proceedings of the IEEE/CVF international conference on computer vision, 5117--5127
2021
-
[32]
Wang, Y.; Zhao, X.; and Huang, K. 2017. Deep crisp boundaries. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3892--3900
2017
-
[33]
Wu, Y.; Pan, C.; Wang, G.; Yang, Y.; Wei, J.; Li, C.; and Shen, H. T. 2023. Learning semantic-aware knowledge guidance for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1662--1671
2023
-
[34]
Xie, S.; and Tu, Z. 2015. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, 1395--1403
2015
-
[35]
Xu, X.; Wang, R.; and Lu, J. 2023. Low-light image enhancement via structure modeling and guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9893--9903
2023
-
[36]
Yang, W.; Chen, X.-D.; Wu, W.; Qin, H.; Yan, K.; Mao, X.; and Song, H. 2024. Boosting Deep Unsupervised Edge Detection via Segment Anything Model. IEEE Transactions on Industrial Informatics
2024
-
[37]
Yao, Z.; and Wang, L. 2023. Object localization and edge refinement network for salient object detection. Expert Systems with Applications, 213: 118973
2023
-
[38]
Ye, Y.; Xu, K.; Huang, Y.; Yi, R.; and Cai, Z. 2024. DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 6675--6683
2024
-
[39]
Yu, T.; Feng, R.; Feng, R.; Liu, J.; Jin, X.; Zeng, W.; and Chen, Z. 2023. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790
2023 arXiv
-
[40]
Yu, Z.; Huang, R.; Byeon, W.; Liu, S.; Liu, G.; Breuel, T.; Anandkumar, A.; and Kautz, J. 2021. Coupled segmentation and edge learning via dynamic graph propagation. Advances in Neural Information Processing Systems, 34: 4919--4932
2021
-
[41]
W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F
Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; and Yang, M.-H. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5728--5739
2022
-
[42]
Zhang, M.; Wang, Y.; Guo, J.; Li, Y.; Gao, X.; and Zhang, J. 2024. IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection. arXiv preprint arXiv:2407.07520
2024 arXiv
-
[43]
Zhou, C.; Huang, Y.; Pu, M.; Guan, Q.; Deng, R.; and Ling, H. 2024. MuGE: Multiple Granularity Edge Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 25952--25962
2024
-
[44]
Zhou, C.; Huang, Y.; Pu, M.; Guan, Q.; Huang, L.; and Ling, H. 2023. The treasure beneath multiple annotations: An uncertainty-aware edge detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15507--15517
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.