REVIEW 3 major objections 4 minor 1 cited by
Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Tuning SAM to draft and then refine its own coarse anomaly mask, with a visual-relation-aware adapter, is reported to improve promptable anomaly segmentation by more than 15 percentage points on average across six industrial datasets.
desk verdict Solid, honest engineering contribution; the private training set and missing variance/per-dataset numbers are the real soft spots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the draft-then-refine loop of Self-Draft Tuning (SDT), organized into display, draft, and refine stages: the image and prompt encoders extract features, a draft decoder initialized from the original mask decoder generates a coarse anomaly mask, and the original mask decoder refines it using the draft's embedding as additional dense knowledge. The second component is the Visual-Relation-Aware Adapter (VRA-Adapter), which builds a relation matrix from cosine similarities between image-region embeddings, suppresses weak relations by a threshold $\alpha/d$, and adds relation-aggregated features back into the decoder with a learned scale $\beta$. These two mechanisms are what the paper claims let a lightly tuned SAM perceive industrial anomalies rather than merely interpolating prompt points.
What would settle it
Train SPT on a publicly available substitute training set, such as the training images from one of the six benchmark datasets, and test on the other five; if the average mIoU gain over zero-shot SAM falls far below the reported 15-plus points, the private training distribution rather than the method itself is what carries the result.
Extended reading notes
Core claim
The central claim is that SAM's poor anomaly segmentation on industrial images is a perception problem, not merely a prompt problem, and that it can be fixed by letting the model perceive anomalies through its own coarse draft and through explicit visual-relation cues. SPT augments SAM with a draft decoder that produces an initial anomaly mask; that mask is encoded and fed back to the original mask decoder as a dense embedding, and a Visual-Relation-Aware Adapter computes a cosine-similarity relation matrix between image regions, thresholds weak relations, and aggregates related features during decoding. Fine-tuned with a PEFT method while nearly all SAM weights stay frozen, the model is reported to outperform both zero-shot SAM and the PEFT baselines on MVTec, VisA, MTD, KSDD2, BTAD, and MPDD, with the largest relative gains on sparse point prompts. The paper frames SPT as a flexible tuning framework: any PEFT method can be wrapped by SPT and inherit the draft-then-refine gains.
Load-bearing premise
The load-bearing premise is that the private, unreleased training set of about 15,000 real-factory defect images is representative enough of the six public test benchmarks that the reported cross-dataset gains transfer; if that match fails, the measured improvements would not reproduce for other industrial imaging conditions.
Editorial extensions
If this is right
- A single SPT-tuned SAM can be prompted by boxes or points and segment defects across different product lines, including electronics, photovoltaics, magnetic tiles, and metal parts, without per-dataset retraining.
- The reported point-prompt gains imply that the model does not need the prompt to trace the full defect: its internal draft and relation cues complete regions the prompt only partially covers.
- Because SPT wraps existing PEFT methods, new PEFT designs can be plugged into the same self-draft loop and likely inherit the same relative lift.
- The added cost is small: on the smallest backbone the full SPT model trains roughly 0.4% of the parameters and keeps throughput close to the untuned model, so the gain is not bought with a large fine-tuning budget.
- Replacing the untuned SAM inside an existing automatic-prompt pipeline with SPT-tuned SAM also raises detection metrics, so the improvement survives when prompts come from a detector instead of a human.
Reading between the lines
- Editorial inference: if the private 15,000-image factory set is what carries the gains, then releasing a comparable public industrial anomaly corpus could turn SPT into a general pre-training recipe; the paper itself does not release the training set.
- Editorial inference: the draft-then-refine idea is not SAM-specific, since any promptable segmenter with a mask decoder could feed its own draft back as a conditioning input, so the mechanism may transfer to other out-of-domain segmentation tasks.
- Editorial inference: the VRA-Adapter's threshold $\alpha/d$ assumes anomalies have coherent regional structure, so extremely fragmented or heavily textured defects may need re-tuning; the paper does not test that regime.
- Editorial inference: the point-prompt results suggest a stress test the paper does not run, namely perturbing prompt points and measuring how much mIoU drops, which would quantify how much internal perception compensates for noisy prompts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Self-Perception Tuning (SPT), a PEFT-based adaptation of SAM for promptable anomaly segmentation. SPT adds a draft decoder that first produces a coarse anomaly mask; this draft is encoded and fed as dense embedding into a refine decoder. A Visual-Relation-Aware Adapter (VRA-Adapter) enhances the decoder image features using a relation matrix computed from cosine similarities. The method is evaluated on six industrial benchmarks with box and point prompts, claiming over 15% average improvement in mIoU/mBIoU over zero-shot SAM and 2.0/2.2 point gains over DoRA, with small trainable-parameter overhead. Code is released, but the training set is private.
Significance. If the claims hold, SPT provides a simple, parameter-efficient recipe for adapting a segmentation foundation model to industrial defect segmentation, with the draft-then-refine self-prompting idea being reusable beyond this specific task. The paper's strengths include clean ablations isolating the SDT and VRA components, cost/efficiency analysis, experiments across three SAM backbones, and public code. The significance is moderate: the reported gains are consistent and the method is practical, but the lack of a public training set and the aggregate-only reporting substantially limit independent verification.
major comments (3)
- [§4.1, Appx. A.1] The load-bearing premise is that the private, unreleased ~15,000-image training set is representative of and disjoint from the six public test benchmarks. The text only asserts that the training categories are 'distinct' without demonstrating appearance-level disjointness; the listed private categories (3C, photovoltaics, display screens, lithium batteries) plausibly share visual structure with test categories such as PCBs, transistors, capsules, and metal parts. Because no images or annotations are released, independent readers cannot verify whether the reported 15% and 2% gains reflect the SPT mechanism or train/test overlap. Please release the data or a representative subset, or provide a per-category/per-dataset transfer analysis quantifying the train-test distribution gap.
- [§4.2, Table 1] The headline claim that SPT improves anomaly segmentation 'across six benchmark datasets' is supported only by dataset-averaged scores. With six datasets and four prompt modes, a reader cannot tell whether SPT wins on each dataset or whether the average is driven by a few favorable cases. Report per-dataset mIoU/mBIoU for all methods (at least for the main Table 1 configurations), and include run-to-run variance or significance tests; the 2.0% mIoU gain over DoRA is small enough that it could be within noise.
- [Appx. C.4, Table 8] There is an internal data inconsistency: the 'Ours' row in Table 8 reports Point=5 70.1/64.5 and Point=10 64.6/59.8, which are the reverse of the Point=5/Point=10 values for SPTLoRA in Table 1 and for 'Ours' in Table 9. This must be corrected. Additionally, it is not stated which SPT variant (SPTLoRA, SPTDoRA, SPTAdapter) is used in Tables 8–11 where the label 'Ours' appears; specify this for each table.
minor comments (4)
- [Abstract] There is a typo in the abstract: 'Self-Perceptinon' should be 'Self-Perception', and the spacing in '( SPT)' should be fixed.
- [Fig. 7, Appx. C.3] The sensitivity-analysis figure appears garbled by rendering tokens; ensure the figure is legible in the camera-ready version.
- [§3.3, Eq. (9)] The threshold α/d is not motivated; clarify whether d is the feature dimension and report the selected α values per backbone/PEFT method in a small table, including how α was chosen (validation set vs. test benchmarks).
- [§4.1] The training description says 'a batch of 8 images' with 8 GPUs; please clarify whether this means 8 total images or 8 per GPU, and report the number of training iterations or effective batch size.
Circularity Check
No significant circularity: SPT's benchmark gains are empirical held-out results, and the self-draft loop is an architectural mechanism rather than a definitional shortcut.
full rationale
SPT is an empirical fine-tuning proposal evaluated on held-out public benchmarks; none of its headline numbers (e.g., "SPTDoRA surpasses DoRA by an average of 2.0% in mIoU and 2.2% in mBIoU," Section 4.2) is a fitted parameter renamed as a prediction. The self-draft mechanism (Eqs. 2-6) feeds mdraft through the prompt encoder and re-injects it as a dense embedding edraft; this is an architectural feedback loop within the model, not a circular argument, because the test-set metrics measure generalization to datasets not used for training. VRA-Adapter (Eqs. 7-11) is a relation-aware feature aggregation module validated by ablations; its hyperparameter α is scanned over a range and reported robust, so it is not a constant fitted to the test set. The only self-citations (e.g., Xiong et al. 2025; Hao et al. 2023; Wang et al. 2023/2024; Yang et al. 2024) appear as related-work or method-background citations and carry none of the load of the central claim. The substantive limitation is external validity: the ~15k-image private training set is unreleased, and the paper only asserts that its categories are "distinct from those in the test datasets" (Section 4.1; Appendix A.1), so independent verification of distribution match and disjointness is impossible. That is a reproducibility and evidence-quality concern, not circularity. Similarly, Table 1 reports only averaged numbers, which weakens per-dataset scrutiny but does not make the results tautological. Therefore, no circular step is present.
Assumptions & free parameters
free parameters (2)
- VRA threshold alpha =
0 to 0.5 (tuned per PEFT method and model size)
- Adapter rank =
8
assumptions (3)
- domain assumption The private training set of about 15,000 industrial images is representative of the six public test datasets.
- domain assumption Encoding a coarse self-drafted mask as a dense prompt embedding improves the refinement decoder.
- domain assumption The cosine-similarity based relation matrix captures structure that helps separate normal from anomalous regions.
Cite this review
Pith. "Pith review of Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning." pith.science (2026). https://pith.science/paper/N55QPUGM
@misc{pith2026241117217,
author = {Pith},
title = {Pith review of: Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning},
year = {2026},
howpublished = {\url{https://pith.science/paper/N55QPUGM}},
note = {Machine review of arXiv:2411.17217}
}
read the original abstract
Segment Anything Model (SAM) has made great progress in anomaly segmentation tasks due to its impressive generalization ability. However, existing methods that directly apply SAM through prompting often overlook the domain shift issue, where SAM performs well on natural images but struggles in industrial scenarios. Parameter-Efficient Fine-Tuning (PEFT) offers a promising solution, but it may yield suboptimal performance by not adequately addressing the perception challenges during adaptation to anomaly images. In this paper, we propose a novel \textbf{S}elf-\textbf{P}erceptinon \textbf{T}uning (\textbf{SPT}) method, aiming to enhance SAM's perception capability for anomaly segmentation. The SPT method incorporates a self-drafting tuning strategy, which generates an initial coarse draft of the anomaly mask, followed by a refinement process. Additionally, a visual-relation-aware adapter is introduced to improve the perception of discriminative relational information for mask generation. Extensive experimental results on several benchmark datasets demonstrate that our SPT method can significantly outperform baseline methods, validating its effectiveness.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 1 Pith paper
-
YOLO-UniOW: Efficient Universal Open-World Object Detection
YOLO-UniOW unifies open-vocabulary and open-world detection by removing cross-modal fusion and adding a self-supervised wildcard embedding, reaching 34.6 AP on LVIS at 69.6 FPS.
Reference graph
Works this paper leans on
-
[1]
Bergmann, P.; Fauser, M.; Sattlegger, D.; and Steger, C. 2019. MVTec AD--A comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9592--9600
2019
-
[2]
Bo z i c , J.; Tabernik, D.; and Sko c aj, D. 2021. Mixed supervision for surface-defect detection: From weakly to fully supervised learning. Computers in Industry, 129: 103459
work page 2021
-
[3]
Cao, Y.; Xu, X.; Sun, C.; Cheng, Y.; Du, Z.; Gao, L.; and Shen, W. 2023. Segment any anomaly without training via hybrid prompt regularization. arXiv preprint arXiv:2305.10724
arXiv 2023
-
[4]
Chen, S.; Ge, C.; Tong, Z.; Wang, J.; Song, Y.; Wang, J.; and Luo, P. 2022. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 16664--16678
2022
-
[5]
Chen, T.; Zhu, L.; Deng, C.; Cao, R.; Wang, Y.; Zhang, S.; Li, Z.; Sun, L.; Zang, Y.; and Mao, P. 2023 a . Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3367--3375
work page 2023
-
[6]
Chen, T.; Zhu, L.; Ding, C.; Cao, R.; Wang, Y.; Li, Z.; Sun, L.; Mao, P.; and Zang, Y. 2023 b . SAM Fails to Segment Anything?--SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More. arXiv preprint arXiv:2304.09148
arXiv 2023
-
[7]
Chen, X.; Zhang, J.; Tian, G.; He, H.; Zhang, W.; Wang, Y.; Wang, C.; Wu, Y.; and Liu, Y. 2023 c . Clip-ad: A language-guided staged dual-path model for zero-shot anomaly detection. arXiv preprint arXiv:2311.00453
arXiv 2023
-
[8]
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
arXiv 2020
Show all 41 references
-
[9]
Farahani, A.; Voghoei, S.; Rasheed, K.; and Arabnia, H. R. 2021. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, 877--894
2021
-
[10]
Hao, T.; Chen, H.; Guo, Y.; and Ding, G. 2023. Consolidator: Mergable Adapter with Group Connections for Visual Adaptation. In The Eleventh International Conference on Learning Representations
2023
-
[11]
Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning, 2790--2799. PMLR
2019
-
[12]
J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685
2021 arXiv
-
[13]
W.; Liu, Y.; and Guo, Q
Huang, Y.; Cao, Y.; Li, T.; Juefei-Xu, F.; Lin, D.; Tsang, I. W.; Liu, Y.; and Guo, Q. 2023. On the robustness of segment anything. arXiv preprint arXiv:2305.16220
2023 arXiv
-
[14]
Huang, Y.; Qiu, C.; and Yuan, K. 2020. Surface defect saliency of magnetic tile. The Visual Computer, 36(1): 85--96
2020
-
[15]
Jeong, J.; Zou, Y.; Kim, T.; Zhang, D.; Ravichandran, A.; and Dabeer, O. 2023. Winclip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19606--19616
2023
-
[16]
Jezek, S.; Jonak, M.; Burget, R.; Dvorak, P.; and Skotak, M. 2021. Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions. In 2021 13th International congress on ultra modern telecommunications and control systems and workshops (I...
2021
-
[17]
Ke, L.; Ye, M.; Danelljan, M.; Tai, Y.-W.; Tang, C.-K.; Yu, F.; et al. 2024. Segment anything in high quality. Advances in Neural Information Processing Systems, 36
2024
-
[18]
C.; Lo, W.-Y.; et al
Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015--4026
2023
-
[19]
A.; Navaneet, K.; Nooralinejad, P.; Kolouri, S.; and Pirsiavash, H
Koohpayegani, S. A.; Navaneet, K.; Nooralinejad, P.; Kolouri, S.; and Pirsiavash, H. 2024. NOLA: Compressing LoRA using Linear Combination of Random Basis. arXiv:2310.02556
2024 arXiv
-
[20]
Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9579--9589
2024
-
[21]
Li, S.; Cao, J.; Ye, P.; Ding, Y.; Tu, C.; and Chen, T. 2024. ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation. arXiv preprint arXiv:2401.12665
2024 arXiv
-
[22]
Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Li, C.; Yang, J.; Su, H.; Zhu, J.; et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499
2023 arXiv
-
[23]
F.; Cheng, K.-T.; and Chen, M.-H
Liu, S.-Y.; Wang, C.-Y.; Yin, H.; Molchanov, P.; Wang, Y.-C. F.; Cheng, K.-T.; and Chen, M.-H. 2024. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353
2024 arXiv
-
[24]
Mishra, P.; Verk, R.; Fornasier, D.; Piciarelli, C.; and Foresti, G. L. 2021. VT-ADL: A vision transformer network for image anomaly detection and localization. In 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), 01--06. IEEE
2021
-
[25]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR
2021
-
[26]
Sun, Y.; Chen, J.; Zhang, S.; Zhang, X.; Chen, Q.; Zhang, G.; Ding, E.; Wang, J.; and Li, Z. 2024. VRP-SAM: SAM with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23565--23574
2024
-
[27]
Wang, A.; Chen, H.; Lin, Z.; Han, J.; and Ding, G. 2023. Repvit-sam: Towards real-time segmenting anything. arXiv preprint arXiv:2312.05760
2023 arXiv
-
[28]
Wang, A.; Chen, H.; Lin, Z.; Han, J.; and Ding, G. 2024. Repvit: Revisiting mobile cnn from vit perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15909--15920
2024
-
[29]
Xiong, Y.; Chen, H.; Hao, T.; Lin, Z.; Han, J.; Zhang, Y.; Wang, G.; Bao, Y.; and Ding, G. 2025. Pyra: Parallel yielding re-activation for training-inference efficient task adaptation. In European Conference on Computer Vision, 455--473. Springer
2025
-
[30]
Xiong, Y.; Chen, H.; Lin, Z.; Zhao, S.; and Ding, G. 2023. Confidence-based Visual Dispersal for Few-shot Unsupervised Domain Adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11621--11631
2023
-
[31]
Yang, H.-Y.; Chen, H.; Liu, L.; Lin, Z.; Chen, K.; Wang, L.; Han, J.; and Ding, G. 2024. Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection. In ECAI 2024, 2098--2105. IOS Press
2024
-
[32]
Yao, X.; Li, R.; Qian, Z.; Luo, Y.; and Zhang, C. 2023. Focus the discrepancy: Intra-and inter-correlation learning for image anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6803--6813
2023
-
[33]
You, Z.; Cui, L.; Shen, Y.; Yang, K.; Lu, X.; Zheng, Y.; and Le, X. 2022. A unified model for multi-class anomaly detection. Advances in Neural Information Processing Systems, 35: 4571--4584
2022
-
[34]
B.; Ravfogel, S.; and Goldberg, Y
Zaken, E. B.; Ravfogel, S.; and Goldberg, Y. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199
2021
-
[35]
Zhang, H.; Su, Y.; Xu, X.; and Jia, K. 2024. Improving the generalization of segmentation foundation model under distribution shift via weakly supervised adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23385--23395
2024
-
[36]
Zhang, X.; Li, S.; Li, X.; Huang, P.; Shan, J.; and Chen, T. 2023. Destseg: Segmentation guided denoising student-teacher for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3914--3923
2023
-
[37]
Zhou, Q.; Pang, G.; Tian, Y.; He, S.; and Chen, J. 2023. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310.18961
2023
-
[38]
Zou, X.; Yang, J.; Zhang, H.; Li, F.; Li, L.; Wang, J.; Wang, L.; Gao, J.; and Lee, Y. J. 2024. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36
2024
-
[39]
Zou, Y.; Jeong, J.; Pemula, L.; Zhang, D.; and Dabeer, O. 2022. Spot-the-difference self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision, 392--408. Springer
2022
-
[40]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[41]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.