REVIEW 5 minor 42 references
ET-SAM replaces thousands of random SAM point prompts with a few peaks from a word heatmap, cutting inference time by about 3× while matching HierText and lifting average F-score 11% on three single-level benchmarks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-13 18:22 UTC pith:FDW5V4DK
load-bearing objection Clean engineering fix to Hi-SAM: sparse heatmap points + task-prompt joint training deliver the claimed 3 imes speed-up and ~11-point F-score gains with solid ablations.
ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Sparse, high-confidence points taken from the peaks of a predicted word heatmap are sufficient visual prompts for a SAM-style hierarchical mask decoder to recover accurate word, word-group, text-line, and paragraph masks, eliminating the need for thousands of randomly sampled points and for pixel-level text segmentation during both training and inference.
What carries the argument
A streamlined point decoder (derived from SAM’s mask decoder) that emits a single word-centric heatmap whose local maxima become the sparse point prompts, together with three sets of learnable task-prompt tokens that condition both the point decoder and the hierarchical mask decoder on multi-level, word-only, or line-only annotation regimes.
Load-bearing premise
The method assumes that the few local peaks of a predicted word heatmap already give the hierarchical decoder enough and correct visual prompts to reconstruct accurate line and paragraph masks on both dense and curved text.
What would settle it
On a held-out set of heavily curved or densely overlapping text images, if the sparse peaks selected by the point decoder systematically miss many word centres or produce large numbers of false peaks, and hierarchical F-scores plus layout PQ therefore fall well below the dense-sampling baseline, the central efficiency claim would be falsified.
If this is right
- Inference latency for hierarchical text segmentation drops by roughly a factor of three relative to the prior SAM-based architecture.
- Public datasets that supply only word-level or only line-level boxes can be mixed with multi-level data without requiring new dense annotations.
- Competitive panoptic-quality and F-scores are retained on dense HierText while layout analysis slightly improves.
- Average F-score on Total-Text, CTW1500, and ICDAR15 rises by about 11 points after joint training and fine-tuning.
- The same sparse-prompt idea can be dropped into other SAM-based pipelines that currently rely on dense foreground sampling.
Where Pith is reading between the lines
- Heatmap-based prompt selection may generalise to other dense object classes where random sampling is currently the latency bottleneck.
- The residual gap to specialised single-level detectors is largely a data-volume issue; further scaling of mixed-granularity corpora could close it without architectural change.
- Task-prompt tokens offer a lightweight route for multi-task SAM adapters when annotation taxonomies differ across sources.
- For highly curved or overlapping text an adaptive multi-scale peak extractor could further reduce missed centres that the fixed centre-line heatmap may still produce.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ET-SAM is a SAM-based dual-decoder architecture for unified scene text detection and layout analysis. A lightweight point decoder produces word-centric heatmaps (Gaussian kernels along center-lines) from which sparse local-maxima points are extracted as visual prompts; a hierarchical mask decoder then segments word, word-group, text-line and paragraph masks under the guidance of those points and three sets of learnable task-prompt tokens. Joint training mixes multi-level (HierText), word-only and line-only datasets in parallel, with task prompts used to mitigate annotation and density discrepancies. On HierText the method is competitive with (and after fine-tuning slightly superior to) Hi-SAM while reducing average points from ~1500 to ~280 and latency from 4.4 s to 1.4 s (~3 imes). On Total-Text, CTW1500 and ICDAR15 it reports an average +11.0 % F-score over Hi-SAM zero-shot / limited-data baselines.
Significance. The work cleanly removes two practical bottlenecks of the prior SAM-based hierarchical text system: dense random point sampling and the requirement for fully multi-level annotations. The point-decoder replacement (Table 3), task-prompt ablations (Table 4), data-mixture study (Table 5), threshold sensitivity (Table 6) and center-line versus center-point comparison (Appendix A) form a coherent empirical package that substantiates both the speed claim and the improved data utilization. The fair re-training of DPText-DETR on the same data mixture (Appendix B) further strengthens the data-efficiency argument. These contributions are of clear engineering and methodological value for SAM-based document and scene-text pipelines.
minor comments (5)
- Abstract and §1 claim “about 3× inference acceleration”; Table 3 reports 4.4 s → 1.4 s (≈3.1×). A single consistent figure (or explicit statement that the factor is measured on HierText with the reported point counts) would avoid any ambiguity.
- §3.5 describes the construction of pseudo word heatmaps for line-only data, yet no quantitative check of pseudo-label quality (e.g., precision/recall of extracted peaks versus held-out word annotations) is supplied. A short diagnostic would increase confidence.
- Tables 1–2 and 8 report point estimates only. Even a single-run standard deviation over a few random seeds, or a note that variance is negligible, would strengthen the 11-point average claim.
- Fig. 6–8 captions and the main text occasionally use “word group” and “intra-line words” interchangeably; a one-sentence definition in §3.4 would improve readability.
- Several OCR-style artifacts remain in the compiled PDF (e.g., “��������� �������”, “��inference”). Cleaning these before camera-ready would be desirable.
Circularity Check
No significant circularity: purely empirical CV method paper whose claims are measured on held-out public test sets.
full rationale
ET-SAM proposes an architectural change (lightweight point decoder producing word heatmaps whose local maxima become sparse SAM prompts) plus a joint-training recipe with task-prompt tokens, then reports wall-clock latency and standard detection/layout metrics (PQ, F, P, R) on the public HierText, Total-Text, CTW1500 and ICDAR15 test sets. The 3 imes acceleration claim is a direct measurement of fewer prompts (∼1500 → ∼280) and the resulting mask-decoder/post-processing cost; the 11 % average F-score gain is likewise an empirical comparison against the prior SAM-based baseline under identical evaluation protocols. No equation, loss term or “prediction” is defined in terms of the quantity it is later said to predict; no parameter is fitted on a subset and then re-reported as an independent forecast; no uniqueness theorem or ansatz is imported from the authors’ own prior work to force the result. Self-citations to Hi-SAM serve only as the natural baseline being improved upon and do not close any logical loop. The derivation chain is therefore empty of circular steps; the paper is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (4)
- heatmap peak threshold =
0.6 / 0.7
- max points per word / batch size of 100 =
≤2 points/word, batches of 100
- IoU / NMS thresholds =
0.5
- learning-rate schedule and epoch counts =
1e-4, decay at 100/35/60 epochs
axioms (3)
- domain assumption Pre-trained SAM ViT-L image encoder and mask-decoder weights provide a transferable feature space for text after adapter fine-tuning.
- ad hoc to paper Word-center heatmaps generated by Gaussian kernels along center-lines are adequate supervision for locating prompt points.
- ad hoc to paper Three discrete task-prompt embeddings suffice to disambiguate multi-level, word-only and line-only annotation regimes.
invented entities (2)
-
lightweight point decoder (heatmap head)
no independent evidence
-
learnable task-prompt tokens (three sets)
no independent evidence
read the original abstract
Previous works based on Segment Anything Model (SAM) have achieved promising performance in unified scene text detection and layout analysis. However, the typical reliance on pixel-level text segmentation for sampling thousands of foreground points as prompts leads to unsatisfied inference latency and limited data utilization. To address above issues, we propose ET-SAM, an Efficient framework with two decoders for unified scene Text detection and layout analysis based on SAM. Technically, we customize a lightweight point decoder that produces word heatmaps for achieving a few foreground points, thereby eliminating excessive point prompts and accelerating inference. Without the dependence on pixel-level segmentation, we further design a joint training strategy to leverage existing data with heterogeneous text-level annotations. Specifically, the datasets with multi-level, word-level only, and line-level only annotations are combined in parallel as a unified training set. For these datasets, we introduce three corresponding sets of learnable task prompts in both the point decoder and hierarchical mask decoder to mitigate discrepancies across datasets.Extensive experiments demonstrate that, compared to the previous SAM-based architecture, ET-SAM achieves about 3$\times$ inference acceleration while obtaining competitive performance on HierText, and improves an average of 11.0% F-score on Total-Text, CTW1500, and ICDAR15.
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the AAAI conference on artificial intelligence
Bu, Q., Park, S., Khang, M., Cheng, Y.: Srformer: text detection transformer with incorporated segmentation and regression. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 855–863 (2024) 3, 11
2024
-
[2]
arXiv preprint arXiv:2511.16719 (2025) 4
Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala,K.V.,Khedr,H.,Huang,A.,etal.:Sam3:Segmentanythingwithconcepts. arXiv preprint arXiv:2511.16719 (2025) 4
Pith/arXiv arXiv 2025
-
[3]
In: ECCV
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: ECCV. pp. 213–229 (2020) 2, 3
2020
-
[4]
In: NeurIPS
Cen, J., Zhou, Z., Fang, J., Yang, C., Shen, W., Xie, L., Zhang, X., Tian, Q.: Segment anything in 3d with nerfs. In: NeurIPS. vol. 36, pp. 25971–25990 (2023) 4
2023
-
[5]
Cheng, Y., Li, L., Xu, Y., Li, X., Yang, Z., Wang, W., Yang, Y.: Segment and track anything. arXiv:2305.06558 (2023) 4
Pith/arXiv arXiv 2023
-
[6]
In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR)
Ch’ng, C.K., Chan, C.S.: Total-text: A comprehensive dataset for scene text de- tection and recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 935–942. IEEE (2017) 9, 10
2017
-
[7]
In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition
Dai, P., Zhang, S., Zhang, H., Cao, X.: Progressive contour regression for arbitrary- shape scene text detection. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 7393–7402 (2021) 11
2021
-
[8]
IEEE Transactions on Geoscience and Remote Sensing (2025) 4
Gao, J., Zhang, D., Wang, F., Ning, L., Zhao, Z., Li, X.: Combining sam with lim- ited data for change detection in remote sensing. IEEE Transactions on Geoscience and Remote Sensing (2025) 4
2025
-
[9]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.Y., Cubuk, E.D., Le, Q.V., Zoph, B.: Simple copy-paste is a strong data augmentation method for instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2918–2928 (2021) 10
2021
-
[10]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Huang, M., Zhang, J., Peng, D., Lu, H., Huang, C., Liu, Y., Bai, X., Jin, L.: Estextspotter: Towards better scene text spotting with explicit synergy in trans- former. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19495–19505 (2023) 11
2023
-
[11]
In: 2015 13th international conference on document analysis and recognition (ICDAR)
Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwa- mura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., Lu, S., et al.: Icdar 2015 competition on robust reading. In: 2015 13th international conference on document analysis and recognition (ICDAR). pp. 1156–1160. IEEE (2015) 9, 10
2015
-
[12]
In: 2013 12th international conference on document analysis and recognition
Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust read- ing competition. In: 2013 12th international conference on document analysis and recognition. pp. 1484–1493. IEEE (2013) 9
2013
-
[13]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 9404–9413 (2019) 10
2019
-
[14]
In: Proceedings of the IEEE/CVF international conference on computer vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023) 2, 3, 4, 9
2023
-
[15]
In: CVPR
Li, J., Jain, J., Shi, H.: Matting anything. In: CVPR. pp. 1775–1785 (2024) 4
2024
-
[16]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Li, X.H., Yin, F., Liu, C.L.: Docsam: Unified document image segmentation via query decomposition and heterogeneous mixed learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 15021–15032 (2025) 11 16 X. Zhang et al
2025
-
[17]
IEEE transactions on image processing27(8), 3676–3690 (2018) 1, 3
Liao, M., Shi, B., Bai, X.: Textboxes++: A single-shot oriented scene text detector. IEEE transactions on image processing27(8), 3676–3690 (2018) 1, 3
2018
-
[18]
In: Proceedings of the AAAI conference on artificial intelligence
Liao, M., Shi, B., Bai, X., Wang, X., Liu, W.: Textboxes: A fast text detector with a single deep neural network. In: Proceedings of the AAAI conference on artificial intelligence. vol. 31 (2017) 3
2017
-
[19]
In: Proceedings of the AAAI conference on artificial intelligence
Liao, M., Wan, Z., Yao, C., Chen, K., Bai, X.: Real-time scene text detection with differentiable binarization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11474–11481 (2020) 3
2020
-
[20]
IEEE transactions on pattern analysis and machine intelligence45(1), 919–931 (2022) 1, 3, 11
Liao, M., Zou, Z., Wan, Z., Yao, C., Bai, X.: Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE transactions on pattern analysis and machine intelligence45(1), 919–931 (2022) 1, 3, 11
2022
-
[21]
In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liu, Y., Chen, H., Shen, C., He, T., Jin, L., Wang, L.: Abcnet: Real-time scene text spotting with adaptive bezier-curve network. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9809–9818 (2020) 3
2020
-
[22]
Pattern Recognition90, 337–345 (2019) 5, 9, 10
Liu, Y., Jin, L., Zhang, S., Luo, C., Zhang, S.: Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition90, 337–345 (2019) 5, 9, 10
2019
-
[23]
IEEE Transactions on Pattern Analysis and Machine Intelligence44(11), 8048–8064 (2021) 1, 3, 11
Liu, Y., Shen, C., Jin, L., He, T., Chen, P., Liu, C., Chen, H.: Abcnet v2: Adaptive bezier-curve network for real-time end-to-end text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence44(11), 8048–8064 (2021) 1, 3, 11
2021
-
[24]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Long, S., Qin, S., Panteleev, D., Bissacco, A., Fujii, Y., Raptis, M.: Towards end- to-end unified scene text detection and layout analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1049– 1059 (2022) 1, 2, 3, 5, 7, 9, 10, 11
2022
-
[25]
arXiv preprint arXiv:1711.05101 (2017) 9
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 9
Pith/arXiv arXiv 2017
-
[26]
In: 2016 fourth international confer- ence on 3D vision (3DV)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international confer- ence on 3D vision (3DV). pp. 565–571. Ieee (2016) 8
2016
-
[27]
In: The Thirteenth International Conference on Learning Representations 4
Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. In: The Thirteenth International Conference on Learning Representations 4
-
[28]
In: Advances in Neural Information Processing Systems
Wang, D., Zhang, J., Du, B., Xu, M., Liu, L., Tao, D., Zhang, L.: Samrs: Scaling-up remote sensing segmentation dataset with segment anything model. In: Advances in Neural Information Processing Systems. vol. 36, pp. 8815–8827 (2023) 4
2023
-
[29]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wang, W., Xie, E., Li, X., Hou, W., Lu, T., Yu, G., Shao, S.: Shape robust text detection with progressive scale expansion network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9336–9345 (2019) 3
2019
-
[30]
In: Proceedings of the IEEE/CVF international conference on computer vision
Wang,W.,Xie,E.,Song,X.,Zang,Y.,Wang,W.,Lu,T.,Yu,G.,Shen,C.:Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8440–8449 (2019) 1, 3, 11
2019
-
[31]
Advances in Neural information processing systems33, 17721–17732 (2020) 9
Wang, X., Zhang, R., Kong, T., Li, L., Shen, C.: Solov2: Dynamic and fast instance segmentation. Advances in Neural information processing systems33, 17721–17732 (2020) 9
2020
-
[32]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Xu, X., Zhang, Z., Wang, Z., Price, B., Wang, Z., Shi, H.: Rethinking text segmen- tation: A novel dataset and a text-specific refinement approach. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12045– 12055 (2021) 9 Efficient Point Prompt Prediction in SAM for Text Detection 17
2021
-
[33]
Yang, J., Gao, M., Li, Z., Gao, S., Wang, F., Zheng, F.: Track anything: Segment anything meets videos. arXiv:2304.11968 (2023) 4
Pith/arXiv arXiv 2023
-
[34]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) 1, 2, 3, 4, 7, 9, 10, 11
Ye, M., Zhang, J., Liu, J., Liu, C., Yin, B., Liu, C., Du, B., Tao, D.: Hi-sam: Marry- ing segment anything model for hierarchical text segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) 1, 2, 3, 4, 7, 9, 10, 11
2024
-
[35]
In: Proceedings of the AAAI conference on artificial intelligence
Ye, M., Zhang, J., Zhao, S., Liu, J., Du, B., Tao, D.: Dptext-detr: Towards better scene text detection with dynamic points in transformer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 37, pp. 3241–3249 (2023) 1, 3, 11
2023
-
[36]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Ye, M., Zhang, J., Zhao, S., Liu, J., Liu, T., Du, B., Tao, D.: Deepsolo: Let trans- former decoder with explicit points solo for text spotting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19348– 19357 (2023) 1, 3, 11
2023
-
[37]
Yue, W., Zhang, J., Hu, K., Wu, Q., Ge, Z., Xia, Y., Luo, J., Wang, Z.: Part to whole: Collaborative prompting for surgical instrument segmentation. arXiv:2312.14481 (2023) 4
Pith/arXiv arXiv 2023
-
[38]
In: AAAI (2023) 4
Yue, W., Zhang, J., Hu, K., Xia, Y., Luo, J., Wang, Z.: Surgicalsam: Efficient class promptable surgical instrument segmentation. In: AAAI (2023) 4
2023
-
[39]
IEEE Transactions on Multimedia26, 1747–1760 (2023) 1, 3, 11
Zhang, S.X., Yang, C., Zhu, X., Yin, X.C.: Arbitrary shape text detection via boundary transformer. IEEE Transactions on Multimedia26, 1747–1760 (2023) 1, 3, 11
2023
-
[40]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Zhang, X., Su, Y., Tripathi, S., Tu, Z.: Text spotting transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9519–9528 (2022) 11
2022
-
[41]
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition
Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., Liang, J.: East: an efficient and accurate scene text detector. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 5551–5560 (2017) 3
2017
-
[42]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Zhu, Y., Chen, J., Liang, L., Kuang, Z., Jin, L., Zhang, W.: Fourier contour em- bedding for arbitrary-shaped text detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3123–3131 (2021) 3, 11 18 X. Zhang et al. A Center Point Heatmap We have already introduced a method for generating the target heatmap based...
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.