Pith. sign in

REVIEW 5 major objections 6 minor 16 references

Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that injecting text-derived semantic similarity maps as prompts into SAM-HQ consistently improves segmentation of complex-shaped objects, by up to +5.9% IoU and +8.3% boundary IoU.

desk verdict A plausible idea with a confounded evaluation: the SAM-HQ baseline is never fine-tuned, so the reported text-driven gains are not isolated. read the letter →

arxiv 2506.05396 v1 pith:M2ZBTZIU submitted 2025-06-03 cs.CV eess.IV

Luka Vetoshkin , Dmitry Yudin This is my paper
classification cs.CVeess.IV
keywords text-guidedsegmentationSAM-HQcomplex-shapedobjectsthinobjectCLIPDINOv2semanticpromptsvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a text prompt naming the target object can be converted into a dense semantic map and fed to the SAM-HQ mask decoder as an extra prompt, producing better masks on thin and complex shapes where point or box prompts alone fail. The method, Talk2SAM, projects a CLIP text embedding into DINOv2's feature space through a learned nonlinear mapping, computes a per-patch similarity map, and retrains the mask-prompt encoder to consume that map. On ThinObject5K, BIG, and DIS5K, it reports consistent gains over SAM-HQ, up to +5.9% mIoU and +8.3% mBIoU. If these results hold, natural language becomes a practical lever for refining mask quality and disambiguating objects inside one bounding box.

What carries the argument

The load-bearing object is the semantic similarity map produced by the CLIP-to-DINO projection. The text embedding $t$ is warped by $\psi(t) = W_b^\top \tanh(W_a^\top t + b_a) + b_b$, then compared by cosine similarity to attention-weighted visual embeddings $v_{A_i}$ computed from DINOv2's [CLS]-to-patch attention heads, taking the maximum score over heads. The resulting $37\times 37$ map is upsampled to $256\times 256$ and encoded by a retrained mask-prompt encoder before entering the SAM-HQ decoder; retraining that encoder to accept continuous dense semantic values instead of binary masks is what lets the decoder use language at all.

What would settle it

Run Talk2SAM and SAM-HQ with a fixed set of mismatched text prompts, such as object names that are absent or only partly visible in the image, on the same three benchmarks; if mIoU does not drop below SAM-HQ's in that setting, the claim that text guidance is the cause of the improvement is falsified, while a clear drop would confirm that the method's advantage depends on correct text-object correspondence.

Watch

Extended reading notes

Core claim

The central claim is that language-derived semantic guidance, injected through the mask-prompt channel of SAM-HQ, fixes a class of failures that spatial prompts cannot fix: thin structures, fine boundaries, and visually similar overlapping objects. Concretely, the paper reports Talk2SAM reaching 0.929 mIoU and 0.842 mBIoU on ThinObject5K with a ViT-H backbone, versus 0.870 and 0.759 for SAM-HQ, and gains of +1.8 mIoU and +4.2 mBIoU on BIG. The mechanism works by taking the maximum cosine similarity between the projected CLIP text embedding and DINOv2 attention-weighted patch embeddings, upsampling that map, and letting a trainable prompt encoder feed it to the frozen SAM-HQ decoder.

Load-bearing premise

The entire gain rests on the text prompt actually naming the object the user wants; if the filename or query does not match the visible content, the similarity map will point the decoder at the wrong region, and Talk2SAM can fall below SAM-HQ.

Editorial extensions

If this is right

  • Text prompts become a first-class prompt modality for promptable segmentation, alongside points, boxes, and masks.
  • Objects that are structurally similar or overlap inside one bounding box can be separated by the user's words rather than by geometry alone.
  • The method is modular: any SAM-based variant could accept the same semantic map without architectural changes, so the gains may transfer to other backbones.
  • Fine-boundary metrics such as mBIoU improve alongside mIoU, which matters for thin wires, ropes, and fences.
  • Training stays light, with less than 1% of parameters updated, so the semantic adapter can be learned on a single GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benefit comes from semantic disambiguation rather than boundary detail, then the method should gain most when the text prompt is an exact object name and degrade when it is not; a systematic experiment with shuffled or wrong prompts could test this directly.
  • The similarity-map channel is a general interface: the same CLIP-to-DINO projection could feed other dense predictors, such as interactive editing or referring segmentation, without retraining the visual backbone; the paper lists this as future work but does not demonstrate it.
  • The paper's own DIS5K filtering implies that unfiltered evaluations with scene-level or action-level filenames would likely show smaller gains or losses, which is consistent with the load-bearing role of text-object correspondence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. Talk2SAM is an empirical method that augments SAM-HQ with text-derived semantic guidance for segmenting thin or complex-shaped objects. The pipeline uses a CLIP text encoder to embed a user-provided category name, learns a nonlinear projection (Eq. 1) into DINOv2 attention-based visual features, derives a dense similarity map from the projected text embedding, and feeds this map as an additional prompt to a fine-tuned SAM-HQ decoder. The paper reports consistent improvements over SAM-HQ on ThinObject5K, BIG, and DIS5K, with up to +5.9% mIoU and +8.3% mBIoU (Tables 2-4), plus qualitative examples and a discussion of limitations. The authors state that the method is modular and can be applied to any SAM-based variant.

Significance. If the reported gains are reproducible on genuinely held-out data, Talk2SAM would be a practical, lightweight way to add language control to SAM-family models while preserving their spatial precision. The paper's strengths are its clear modular design, explicit training details (freezing most backbones, reporting parameter counts, memory usage), and the release of source code on GitHub. The method also addresses a real limitation of SAM-HQ: the inability to disambiguate visually similar or overlapping objects from a single box prompt. However, the central empirical claim is currently under-supported because the evaluation protocol is underspecified: train/test splits for two of the three datasets are not stated, the DIS5K evaluation uses a manually filtered subset, and no ablation isolates the contribution of text guidance from the effect of fine-tuning the decoder and prompt encoder. These gaps make the claimed consistency and the magnitude of the improvements difficult to assess.

major comments (5)
  1. [§3.1 and Tables 2–4] The manuscript never states the train/test split for ThinObject5K and BIG, and for DIS5K it only describes a filtered training subset (2,777 images) and a validation subset (457 images). Since the learned projection ψ(·), the semantic prompt encoder, and the SAM-HQ decoder are all fine-tuned on these datasets, the headline gains in Table 2 could in principle be measured on images or categories seen during training. Please specify the exact official splits used for each dataset, report results separately on train/validation/test portions, and add a category-held-out experiment to demonstrate that the text projection transfers to object classes not seen during training of ψ(·).
  2. [§4.1 and Table 4] DIS5K is manually filtered from 5,470 images to 2,777 training and 457 validation images, which removes about 41% of the dataset, yet Table 4 does not indicate whether SAM-HQ and Talk2SAM are evaluated on the same filtered subset or on the full DIS5K test set. This changes the comparison task and could bias the result in favor of whichever method benefits more from the filtering. Please report baselines on exactly the same subset used for Talk2SAM, and ideally also report results on the full DIS5K test set with automatically or manually verified prompts, so that the effect of the filtering is explicit.
  3. [§4.2 and Tables 2–4] No error bars, number of runs, or significance tests are reported. On DIS5K the mBIoU difference in Table 4 is 0.663 vs 0.660, and on BIG the mIoU difference is 0.947 vs 0.929; these differences may be within run-to-run variation, so the claim of consistent improvement is not yet statistically grounded. Please report per-image standard errors, multiple seeds, or paired significance tests, and state how many random seeds were used for each configuration.
  4. [§4.1 and Tables 2–4] The box prompt generation procedure is never described. Since both SAM-HQ and Talk2SAM use box prompts, and the claimed improvement is relative to SAM-HQ 'using the same image encoders', the reader cannot tell whether the boxes are ground-truth boxes, detector outputs, or entire image bounds, nor whether identical boxes are supplied to both methods. Please specify the box source and confirm that exactly the same box prompts are used for every method in all reported tables.
  5. [§3.1] A critical missing control is a SAM-HQ baseline fine-tuned under the same protocol (same epochs, loss, optimizer, and fine-tuned decoder/prompt encoder) but without the text-derived similarity map, or with a random or constant semantic prompt. Without this ablation, the gains in Tables 2–4 could be due to the additional fine-tuning of the decoder and prompt encoder rather than to the text guidance itself, which is the paper's central claim. Please add this control to isolate the contribution of the semantic map.
minor comments (6)
  1. [Eq. (1)] Please define the dimensions of W_a, W_b, b_a, b_b and state whether tanh is applied element-wise to the vector W_a^T t + b_a or to the matrix; the current notation is ambiguous.
  2. [Eq. (2)] Specify whether the softmax is taken over all spatial positions (h, w) or independently per attention head, and clarify how the attention maps A_i are extracted from the final DINOv2 layer.
  3. [Abstract and Tables 2–4] The reported improvements such as '+5.9% IoU' are percentage-point differences (0.870 to 0.929); please describe them as '+5.9 percentage points' to avoid confusion with relative improvement.
  4. [§4.1] The statement that text prompts are 'automatically extracted from the filenames' is followed by a description of manual filtering; clarify the exact extraction rule, how many distinct categories remain after filtering for each dataset, and whether the same manual filtering was applied when evaluating the SAM-HQ baseline.
  5. [Figure 7 and §5] The caption calls these 'challenging cases where similarity maps align well with the text prompts but segmentation performance suffers', but the following paragraph says Talk2SAM 'mitigates' these failures; please revise the caption and text to state clearly which method fails in each example and what the limitation is.
  6. [§3.1] The claim that 'less than 1% of parameters are updated' is close to the boundary for the partial-CLIP configuration (10.64M/1.096B = 0.97%); consider phrasing it as 'about 1%' or reporting the exact percentage for each row.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Talk2SAM's reported gains are empirical post-training evaluations, not quantities encoded in its training objective.

full rationale

The paper's central claim is empirical: a trained model is evaluated under a supervised protocol, and the reported IoU and boundary IoU figures are measured segmentation outputs rather than fitted quantities. The text prompt is an input to the pipeline, not a parameter derived from the ground-truth mask, and the learned mapping psi(t) is trained to map CLIP embeddings into DINO space, but the reported improvements are not defined by that mapping's training loss. No equation in the paper equates the predicted mask with the training objective, with the text-derived similarity map, or with any fitted value; the similarity map is an intermediate prompt fed into a separately trained decoder. The paper relies on Talk2DINO, but that is external prior work by Barsellotti et al., not a self-citation, and adopting a published projection method is a normal use of prior art rather than a circular import. The stated limitations in Section 6 and Figure 7 acknowledge dependence on CLIP embeddings and on the SAM-HQ decoder, which are architectural assumptions rather than self-referential reductions. The evaluation does have validity concerns that are outside circularity: the SAM-HQ decoder and prompt encoder are retrained while the SAM-HQ baseline is frozen, and train/test splits for ThinObject5K and BIG are not explicitly stated, so the comparison may partly reflect fine-tuning rather than text guidance. These concerns do not make the derivation circular, because the reported numbers are still post-training evaluations and no prediction reduces to a fitted quantity by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the quality of pretrained CLIP and DINOv2 representations, on a learned projection ψ trained on the target benchmarks, and on a retrained prompt encoder and decoder. No new theoretical entities are introduced. The main domain assumptions are that filename-derived text prompts match the target objects (which the authors enforce by filtering DIS5K) and that DINOv2 attention maps capture meaningful semantic regions.

free parameters (4)
  • Learned projection ψ(t) (W_a, W_b, b_a, b_b) = Learned during training on the three benchmarks
    Defined in Eq. (1), maps CLIP text embeddings into DINOv2 feature space; jointly trained with the decoder. Its values are not reported.
  • Semantic prompt encoder (convolutional module) = Learned during training
    Replaces SAM-HQ's mask prompt encoder to accept continuous 256x256 similarity maps; fully trainable.
  • Fine-tuned SAM-HQ decoder weights = Learned during training
    Decoder updated with BCE and dice loss; accounts for the trainable parameter count in Table 1.
  • CLIP text encoder final projection layer (large-scale setting) = Fine-tuned only in large-scale regimes
    Section 3.1: in low-data regimes CLIP is frozen; when data is large, only the final projection layer of the CLIP text encoder is updated.
assumptions (4)
  • domain assumption CLIP text embeddings capture the semantics of the target object category
    Stage 1 in Section 3 encodes a user-provided category such as 'wire' or 'bicycle' using CLIP; if this embedding is semantically misaligned with the image content, the similarity map is uninformative.
  • domain assumption DINOv2 attention maps highlight semantic regions that support localization
    Stage 2 in Section 3 computes weighted visual embeddings from attention maps; the method assumes these maps decompose the image into useful semantic regions.
  • domain assumption Filenames provide text prompts that match the target object
    Section 4.1 automatically extracts prompts from filenames; the authors filter DIS5K because some filenames describe scenes rather than objects, acknowledging this assumption often fails.
  • domain assumption SAM-HQ decoder can fuse a continuous semantic map with box prompts after retraining
    Stage 3 in Section 3 retrains the mask prompt encoder and decoder; this assumes the decoder can exploit the new modality without architectural changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation." pith.science (2026). https://pith.science/paper/M2ZBTZIU

@misc{pith2026250605396,
  author       = {Pith},
  title        = {Pith review of: Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2ZBTZIU}},
  note         = {Machine review of arXiv:2506.05396}
}
read the original abstract

Segmenting objects with complex shapes, such as wires, bicycles, or structural grids, remains a significant challenge for current segmentation models, including the Segment Anything Model (SAM) and its high-quality variant SAM-HQ. These models often struggle with thin structures and fine boundaries, leading to poor segmentation quality. We propose Talk2SAM, a novel approach that integrates textual guidance to improve segmentation of such challenging objects. The method uses CLIP-based embeddings derived from user-provided text prompts to identify relevant semantic regions, which are then projected into the DINO feature space. These features serve as additional prompts for SAM-HQ, enhancing its ability to focus on the target object. Beyond improving segmentation accuracy, Talk2SAM allows user-controllable segmentation, enabling disambiguation of objects within a single bounding box based on textual input. We evaluate our approach on three benchmarks: BIG, ThinObject5K, and DIS5K. Talk2SAM consistently outperforms SAM-HQ, achieving up to +5.9\% IoU and +8.3\% boundary IoU improvements. Our results demonstrate that incorporating natural language guidance provides a flexible and effective means for precise object segmentation, particularly in cases where traditional prompt-based methods fail. The source code is available on GitHub: https://github.com/richlukich/Talk2SAM

Figures

Figures reproduced from arXiv: 2506.05396 by the authors.

Figure 1
Figure 1. Graphical Abstract. The core idea of Talk2SAM is to guide open-world segmentation models using textual queries, enabling precise selection of target objects. This approach overcomes the limitations of prompt-based methods such as SAM or HQ￾SAM, where spatial prompts (e.g., boxes) may be insufficient to disambiguate complex object boundaries or distinguish between visually similar objects in close proximity. To addre… view at source ↗
Figure 2
Figure 2. Overview of the Talk2SAM framework. Given an input image and a user-defined text prompt, CLIP extracts a global text embedding and computes a relevance map with the image. This relevance is projected into DINO’s visual space to produce a dense semantic prompt, which is injected into SAM-HQ’s decoder for refined segmentation. 3 Method In this section, we present Talk2SAM, a vision-language segmentation method that en… view at source ↗
Figure 3
Figure 3. Representative image and mask pairs from the datasets used in our study. Although not shown visually, textual categories used as prompts in our method were automatically extracted from the filenames of these samples. The figure highlights the diversity and structural complexity of the segmented objects, motivating the use of text-guided segmentation. 4.2 Evaluation Metrics We evaluate segmentation quality using two … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative results on the BIG dataset. Each row shows: the input image, the text query, the similarity map from CLIP-DINOv2 features, and the predicted masks from SAM, SAM-HQ, and Talk2SAM. While SAM and SAM-HQ often produce incomplete or imprecise masks, Talk2SAM lev…
Figure 5
Figure 5. Figure 5: Qualitative results on the ThinObject5K dataset. Each row includes: the input image, the corresponding text query, the similarity map obtained from projected CLIP￾DINOv2 features, and segmentation masks from SAM, SAM-HQ, and Talk2SAM. Talk2SAM outperforms baselines in …
Figure 6
Figure 6. Figure 6: Qualitative results on the DIS5K dataset. Each row displays: the input image, the corresponding text query, the similarity map, and the segmentations by SAM, SAM-HQ, and Talk2SAM. Talk2SAM maintains high segmentation fidelity in scenes with complex structure and fine d…
Figure 7
Figure 7. Figure 7: Challenging cases where similarity maps align well with the text prompts but segmentation performance suffers. Each row shows: the input image, text query, simi￾larity map from CLIP-DINOv2, and predicted masks from SAM-HQ and Talk2SAM. These examples highlight that acc…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages

  1. [1]

    arXiv preprint arXiv:2411.19331 (2024)

    Barsellotti, L., Bianchi, L., Messina, N., Carrara, F., Cornia, M., Baraldi, L., Falchi,F.,Cucchiara,R.:Talkingtodino:Bridgingself-supervisedvisionbackbones with language for open-vocabulary segmentation. arXiv preprint arXiv:2411.19331 (2024)

    arXiv 2024

  2. [2]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)

    2021

  3. [3]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Girshick, R., Dollár, P., Berg, A.C., Kirillov, A.: Boundary iou: Im- proving object-centric image segmentation evaluation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15334– 15342 (2021)

    work page 2021

  4. [4]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, H.K., Chung, J., Tai, Y.W., Tang, C.K.: Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8890–8899 (2020)

    work page 2020

  5. [5]

    Advances in Neural Information Processing Systems36, 29914–29934 (2023)

    Ke, L., Ye, M., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F., et al.: Segment anything in high quality. Advances in Neural Information Processing Systems36, 29914–29934 (2023)

    work page 2023

  6. [6]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    Kirillov, A., Mintun, E., Ravi, N., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023)

    work page 2023

  7. [7]

    arXiv preprint arXiv:2201.03546 (2022)

    Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 (2022)

    arXiv 2022

  8. [8]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Liew, J.H., Cohen, S., Price, B., Mai, L., Feng, J.: Deep interactive thin object selection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 305–314 (2021)

    work page 2021

Show all 16 references
  1. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 7086–7096 (2022)

    2022

  2. [10]

    In: European Conference on Computer Vision

    Qin, X., Dai, H., Hu, X., Fan, D.P., Shao, L., Van Gool, L.: Highly accurate di- chotomous image segmentation. In: European Conference on Computer Vision. pp. 38–56. Springer (2022)

    2022

  3. [11]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 14...

    2021

  4. [12]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., Lu, J.: Denseclip: Language-guided dense prediction with context-aware prompting. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 18082–18091 (2022)

    2022

  5. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sun,Y.,Chen,J.,Zhang,S.,Zhang,X.,Chen,Q.,Zhang,G.,Ding,E.,Wang,J.,Li, Z.: Vrp-sam: Sam with visual reference prompt. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23565–23574 (2024)

    2024

  6. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, H., Vasu, P.K.A., Faghri, F., Vemulapalli, R., Farajtabar, M., Mehta, S., Rastegari, M., Tuzel, O., Pouransari, H.: Sam-clip: Merging vision foundation mod- els towards semantic and spatial understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  7. [15]

    In: European Conference on Computer Vision

    Xiao, A., Xuan, W., Qi, H., Xing, Y., Ren, R., Zhang, X., Shao, L., Lu, S.: Cat- sam: Conditional tuning for few-shot adaptation of segment anything model. In: European Conference on Computer Vision. pp. 189–206. Springer (2024)

    2024

  8. [16]

    In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition

    Zou, X., Dou, Z.Y., Yang, J., Gan, Z., Li, L., Li, C., Dai, X., Behl, H., Wang, J., Yuan, L., et al.: Generalized decoding for pixel, image, and language. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15116–15127 (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.