{"id":"08ede148-ca2c-49a4-bd47-14c1a6112bf8","arxiv_id":"2505.06280","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The Show or Tell benchmark compares text and visual prompts for semantic segmentation on 14 datasets, finding visual prompts yield higher average mIoU but with high variance and much higher computational cost.","lead":"Show or Tell is a new benchmark that directly compares two ways of prompting a segmentation model: typing a class name (text) versus showing a reference image with a mask (visual). Across 14 datasets, visual prompts generally score higher on average, especially for specialized objects like tools and waste, but text prompts are much faster and win on everyday scenes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The confidence-based mask merging in Sec. 3.2 is the load-bearing step: if mean-backbone confidences are not comparable across classes, the visual-vs-textual comparison in Table 3 is biased and needs a calibration check.","rationale":"The reader's weakest assumption is exactly the step on which the paper's headline comparison depends: the multi-class adaptation of Sec. 3.2. Every visual method in Table 3 is evaluated through the confidence-based merging, so an artifact there propagates to the average (38.7 mIoU) and to the per-domain comparisons. I do not see a more load-bearing issue: the backbone mismatch between DINOv2/SAM and CLIP is real but is a property of the chosen methods, and the unweighted average is a reporting choice that the reader already flagged. The merging rule, in contrast, is a methodological construction that the paper introduces as a contribution, and it has a specific failure mode: mean-of-mask confidence is not a calibrated cross-class score. The proposed concrete test would settle whether this failure mode changes the Table 3 conclusions. The reader's conditional verdict already asks for validation of this merging strategy, so my pass does not move the verdict.","tokens_in":17230,"tokens_out":6299,"duration_ms":66171,"concrete_test":"Run GFSAM and Matcher on PASCAL VOC, Cityscapes, and Toolkits under two alternative merges: (i) per-pixel max confidence (softmax over the n class-confidence maps at each pixel) and (ii) mean-confidence normalized per class by the class's average confidence over the validation set before argmax. Compare the resulting mIoU with Table 3 and with an oracle merge that assigns each pixel the class whose binary mask is locally most confident as measured per-pixel. If the visual-vs-textual gap for these datasets shifts by more than ~3 mIoU points under (i) or (ii), or if the oracle merge exceeds the mean-based merge by more than ~5 points, the Sec. 3.2 merging rule is not neutral and the headline comparison requires re-evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sec. 4.2: 'visual prompting methods generally outperform textual approaches') rests on Table 3, and every visual-method number in Table 3 is produced by the multi-class adaptation of Sec. 3.2. That adaptation merges per-class binary masks by argmax over the mean visual-backbone confidence within each mask (DINOv2/SAM scores for training-free methods, decoder probability for SINE). This is only a fair multi-class extension if those mean confidences are comparable across classes and methods. They are not calibrated: DINOv2 patch-similarity scores and SAM confidence outputs have no common scale with CLIP text-image alignment, and within one method the mean-over-mask statistic favors classes whose instances are large and homogeneous (e.g., 'road', 'building') over small or partially visible classes (e.g., 'traffic light', 'Allen key'). Overlapping predictions are assigned wholesale to the class with the highest mean confidence, so a confident large-class mask can erase a correct small-class prediction. The largest visual margins in the paper are in exactly these structured scenes (Cityscapes, Toolkits), so the headline comparison may be an artifact of the merging rule rather than a property of the prompt modality. Without a demonstration that the rule tracks per-pixel oracle assignment, the benchmark's central comparison is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Show or Tell (SoT), a benchmark for directly comparing textual prompts (via open-vocabulary semantic segmentation methods) and visual reference prompts (via few-shot visual prompting methods) for semantic segmentation across 14 datasets in 7 domains. It evaluates 5 open-vocabulary methods (MaskCLIP, TCL, CLIP-DINoiser, NACLIP, ProxyCLIP) and 4 visual reference methods (SINE, PerSAM, Matcher, GFSAM), and adapts the latter to multi-class segmentation by generating per-class binary masks and merging them via a confidence-based argmax over mean backbone scores. The main reported finding is that visual prompting methods generally outperform textual approaches on average, with GFSAM achieving the highest average mIoU of 38.7, while textual methods remain competitive on common scenes. The paper also analyzes inference cost and prompt sensitivity, and releases code.","tokens_in":17499,"tokens_out":8408,"duration_ms":77280,"significance":"If the results are reliable, SoT would be a valuable community resource: it is the first benchmark to compare the two prompt modalities under a common evaluation protocol, spans a broad range of domains, samples support sets five times and reports standard deviations, and releases code. The paper also provides a useful decomposition of where each modality fails (e.g., specialized tools and waste favor visual prompts; common categories yield to text). However, the central quantitative comparison rests on the multi-class adaptation in Sec. 3.2, whose fairness is not established; the aggregation over datasets of very different sizes is not justified; and some reported numbers are internally inconsistent. The contribution is real, but the load-bearing claims need additional verification before the benchmark's headline conclusion can be accepted.","major_comments":[{"comment":"The confidence-based mask merging rule is load-bearing: every visual-method result in Table 3 is produced by taking the argmax over per-class masks using the mean backbone confidence within each mask. The paper provides no evidence that these mean confidences are comparable across classes or across methods. Mean-over-mask statistics favor large, homogeneous classes (e.g., 'road' in Cityscapes, 'building' in LoveDA-Urban) over small or partially visible classes (e.g., 'traffic light', 'Allen key'), so overlapping predictions can be assigned to the wrong class even when the per-pixel mask is correct. The confidence scales also differ across DINOv2 similarity, SAM scores, and SINE's decoder probabilities. The paper should report a sensitivity analysis of the main claim to the merging rule, e.g., compare with per-pixel max without mean aggregation, with class-wise calibration, and with an oracle merging rule that uses ground-truth overlap to resolve conflicts. Without such a check, the headline 'visual generally outperforms textual' (Sec. 4.2) may be an artifact of this adaptation rather than a property of the prompt modality.","section":"Sec. 3.2, Table 3"},{"comment":"The headline average mIoU is an unweighted mean over 14 datasets that differ by more than two orders of magnitude in validation-set size: Toolkits has 6 validation images, ADE20K has 2000, and Cityscapes has 500. Equal weighting lets the 88.9 mIoU on Toolkits count the same as the 23.4 mIoU on ADE20K for GFSAM. The paper should report per-domain averages, a size-weighted average, or otherwise justify the unweighted aggregation. It should also flag the statistical fragility on the smallest datasets; for example, GFSAM's ±14.1 standard deviation on Pizza indicates that its 62.2 mIoU there is not a stable estimate.","section":"Sec. 4.2, Table 2, Table 3"},{"comment":"For Matcher, two entries (ADE20K and UECFood) are marked '†' (excessive time), but the AVG is reported as 27.2 for 1 prompt and 33.5 for 5 prompts without stating the aggregation rule. The arithmetic mean of the twelve numeric entries shown for Matcher 1-prompt is 27.0 (not 27.2), and for 5-prompt it is 33.4 (not 33.5). The paper must state how averages are computed when dataset entries are missing (e.g., average over available datasets only), and should report the number of datasets included in each average. This is necessary for readers to interpret the central comparison.","section":"Table 3, 'AVG' column"},{"comment":"The abstract and introduction state that the methods are evaluated under 'identical experimental conditions,' but the methods use different backbones: NACLIP and ProxyCLIP use ViT-L/14, the other open-vocabulary methods use ViT-B/16, and the visual methods use DINOv2 ViT-L/14 with SAM ViT-H. Since larger backbones generally improve segmentation quality, this confounds the prompt-modality comparison with model capacity. The authors should either match backbones across modalities for a cleaner head-to-head, or explicitly acknowledge this as a limitation and analyze how it might affect the conclusions.","section":"Sec. 1 and Supplementary Sec. 7"},{"comment":"The visual reference methods (SINE, PerSAM, Matcher, GFSAM) were 'reimplemented within our codebase,' but the paper does not report any verification that these reimplementations reproduce the original methods' published behavior on a standard few-shot segmentation setting. For a benchmark whose main output is a set of scores, such a reproducibility check is important to rule out that the visual-method results (and hence the headline conclusion) are artifacts of the reimplementation rather than the original algorithms. The authors should compare their implementations against the official code on at least one representative dataset (e.g., PASCAL-5i or COCO-20i) and report the deviation.","section":"Sec. 4.1"}],"minor_comments":[{"comment":"The class list for Trash is internally inconsistent: the text states 'The Trash [42] dataset is made up of 12 classes,' but then enumerates 22 class names. Similarly, the House-Parts list claims 22 classes but enumerates 12 names. Please correct these counts and ensure the lists exactly match the dataset versions used.","section":"Supplementary Sec. 8"},{"comment":"The caption says PerSAM reports only 1-prompt results 'due to its limitation in handling multiple prompts,' but the table shows full rows of hyphens for the 5-prompt configuration. Please state explicitly what the hyphens mean (e.g., 'not applicable / not evaluated') and clarify the reason that multiple prompts are not supported.","section":"Table 3 caption"},{"comment":"The sentence 'The performance gap between modalities dramatically varies across domains' is vague. Consider reporting a concrete measure, such as the per-domain difference between the best visual and best textual method, or a table of these gaps.","section":"Sec. 4.2"},{"comment":"The benchmark name is used inconsistently: 'Toolkit' (Table 2) vs. 'Toolkits' (Table 3, Fig. 3), and 'Toolkit [39]' in Sec. 3.3 vs. 'Toolkits [39]' elsewhere. Please standardize the spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically relevant question, and the benchmark itself is a useful resource if the reported numbers are trustworthy. The main risk is the unvalidated multi-class merging rule in Sec. 3.2, which affects every visual-method score in Table 3; the authors must add a sensitivity/calibration analysis before the central claim can be accepted. I would also press for a clear statement on how missing entries are aggregated into the AVG column and a correction of the class-count inconsistencies in the supplementary. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read of Show or Tell.\n\nWhat's new: this is the first head-to-head, identical-condition comparison of textual and visual prompting for semantic segmentation, across 14 datasets in 7 domains. The authors adapt binary visual-reference methods to multi-class by sampling per-class support sets and merging masks via argmax over mean confidence. They also do five support-set samples and report standard deviations, give implementation details, release code, and are refreshingly honest about open challenges. The qualitative section is thoughtful and the compute table is a useful practical addition. That is real value.\n\nThe soft spots are in the central claim. The merging rule in Sec. 3.2 is the load-bearing step for every visual method in Table 3, and it is not validated. Mean backbone confidence (DINOv2/SAM similarity or decoder probability) is not scale-comparable across classes or methods, and mean-over-mask naturally favors large homogeneous classes like road or building over small ones. The argmax can therefore erase a correct small-class prediction because a confident large mask wins the whole region. The biggest visual margins are in exactly these structured scenes (Cityscapes, Toolkits), so the headline 'visual generally outperforms' may be an artifact of the merging rule. That is not a manufactured flaw; it is the core methodological assumption and it deserves a calibration check, e.g., comparing the rule against per-pixel oracle assignment or per-class normalization.\n\nSecond, the aggregate '38.7 mIoU on average' is a simple mean over datasets that range from 6 validation images (Toolkits) to 2,000 (ADE20K). Unweighted average over such heterogeneous domains is a weak basis for 'generally outperform.' A weighted or per-domain breakdown would be more honest.\n\nThird, the conclusion says ProxyCLIP variants achieved the highest scores on ADE20K and Cityscapes, but Table 3 shows GFSAM (5 prompts) higher on both. That's a factual slip, easy to fix. Minor: PerSAM is reported only with one prompt and Matcher is missing on ADE20K and UECFood, so 'all methods under identical conditions' is slightly overstated.\n\nBottom line: the benchmark itself is worth having, and the paper deserves a serious referee. The central comparison needs the merging validation and a fairer aggregate before the claim should be published as-is. If I were handling it, I'd send it out and ask for those revisions. I'd bring it to a reading group to discuss evaluation pitfalls, and I'd cite it as a benchmark resource, though not for the visual-beats-textual conclusion.","headline":"Useful benchmark and a mostly transparent protocol, but the central visual-vs-textual claim rests on an unvalidated merging rule and an unweighted average; referee it, don't desk-reject it.","tokens_in":17968,"tokens_out":2681,"would_cite":true,"duration_ms":27184,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual prompts beat text prompts for semantic segmentation on average, with the edge concentrated in specialized domains.","keywords":["semantic segmentation","visual prompting","textual prompting","open-vocabulary segmentation","benchmark","vision foundation models","prompt sensitivity","multi-class adaptation"],"falsifier":"Re-run the SoT comparison with confidence scores normalized per class, for example rank-based or z-scored within each class, before the argmax merge, and check whether GFSAM's average 38.7 mIoU and the visual-method lead over the best textual method shrink or disappear; if they do, the reported advantage is an artifact of the merging rule.","tokens_in":17027,"feed_emoji":"🖼️","tokens_out":4807,"duration_ms":48258,"temperature":0.7,"pith_summary":"The paper sets out to settle a simple question: when a segmentation model is told what to look for, is a picture better than a sentence? It introduces SoT, a benchmark that runs nine prompt-driven segmentation methods head-to-head on 14 datasets in seven domains. The central finding is that visual reference prompts outperform open-vocabulary text prompts on average, with the best visual method reaching 38.7 mIoU with five prompts. The advantage is not uniform: text wins for everyday objects, while visuals win by large margins for tools, waste, and X-ray scans. The paper's adaptation of single-class visual methods to multi-class segmentation is what makes the head-to-head possible, and that adaptation carries the main load-bearing assumption of the comparison.","feed_headline":"Visual prompts beat text prompts in segmentation benchmark","feed_subtitle":"Across 14 datasets, reference images outperform class names on average, except for common scenes.","key_machinery":"The central object is the Show or Tell (SoT) benchmark itself, together with a confidence-based mask merging strategy that adapts binary visual-reference methods to multi-class segmentation. Each class is handled by a separate run that produces a binary mask, and the binary masks are merged by taking an argmax over confidence scores; for training-free methods the confidence is the mean visual-backbone (DINOv2 or SAM) score inside the mask, and for trained methods it is the decoder probability. This merging rule is what allows single-class visual prompting methods to be compared fairly with open-vocabulary text methods on multi-class scenes.","core_discovery":"The paper claims that, under identical benchmark conditions, visual reference prompt methods generally outperform open-vocabulary textual prompting for semantic segmentation. The best visual method, GFSAM, reaches 38.7 mIoU on average across the 14 datasets with five prompts, while the best open-vocabulary method trails on most specialized domains. Textual prompting remains competitive or superior in common scenes such as PASCAL VOC and Cityscapes, but visual prompting dominates waste, tools, and X-ray imagery, where verbal descriptions poorly capture the visual cues. The paper also shows that visual methods are highly sensitive to prompt selection, with standard deviations exceeding 14 mIoU in some datasets, and that they require many forward passes because each class is processed separately.","pith_inferences":["A natural test of the paper's comparison is to replace raw confidence values with calibrated or per-class normalized scores before the argmax merge; if the visual methods' average lead shrinks, part of the reported advantage belongs to the merging rule rather than to visual prompting itself.","The domain split suggests a hybrid system: use text prompts as a cheap default, and switch to visual prompts only when text confidence is low or the domain is specialized; the benchmark provides the per-domain breakpoints such a controller would need.","Because the benchmark randomizes support-set selection, its standard deviations quantify prompt sensitivity; a follow-up could search for support-set selection heuristics, such as diversity of pose or background, that reduce the observed swings.","The same confidence-merging recipe could be applied to combined text-plus-visual prompting, letting a model fall back to textual descriptions for small objects in cluttered scenes, where the paper's qualitative results show visual prompts failing."],"forward_implications":["If the central result holds, practitioners working in specialized domains such as security X-ray, waste sorting, and tool identification should prefer visual reference prompts over text descriptions when accuracy is the priority.","Textual prompts remain the practical choice for common scenes and urban settings, where they match or beat visual methods at a fraction of the inference cost.","Visual prompt selection becomes a first-class concern: five randomly chosen support images shift results by more than 14 mIoU in some datasets, so deployment should include prompt ensembling or automatic support-set selection.","The multi-class adaptation means single-class visual reference methods can be applied to broad semantic segmentation without retraining, at a cost that grows linearly with the number of classes.","Training-free methods generalize better across domains than trained ones in this comparison, suggesting that foundation-model priors matter more than task-specific fine-tuning for prompt-based segmentation."],"supporting_citations":[{"why":"Supplies GFSAM, the best-performing visual reference method; its 38.7 mIoU average with five prompts is the paper's headline quantitative claim.","marker":"[70]"},{"why":"Supplies Matcher, the other strong visual reference method; its 88.9 mIoU on Toolkits anchors the tools-domain advantage.","marker":"[34]"},{"why":"DINOv2 is the visual backbone whose mean confidence scores are used to merge binary masks for training-free visual methods.","marker":"[43]"},{"why":"SAM is the mask generator used by PerSAM, Matcher, and GFSAM; its confidence output feeds the merging step.","marker":"[25]"},{"why":"CLIP provides the text and image encoders behind all five open-vocabulary methods compared in the benchmark.","marker":"[47]"},{"why":"CLIP-DINoiser gives the best text-prompt result on PASCAL VOC (62.1 mIoU), the paper's case for textual strength in common scenes.","marker":"[65]"},{"why":"ProxyCLIP is the best average open-vocabulary method and the main textual baseline that visual methods must beat.","marker":"[27]"},{"why":"SINE is the one trained visual method; its cross-domain drop supports the paper's claim that training-free methods generalize better.","marker":"[35]"},{"why":"The prior multi-domain zero-shot segmentation benchmark motivates SoT but does not compare prompt modalities, which is the gap the paper targets.","marker":"[4]"}],"fun_headline_variants":["Show beats Tell in segmentation benchmark","Visual prompts outperform text prompts in segmentation","Image references win over class names in segmentation","Benchmark: visual prompting dominates text on most tasks","Segmentation: visual cues beat verbal descriptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The head-to-head comparison assumes that the confidence values used to merge per-class masks, mean DINOv2 or SAM scores for training-free methods and decoder probabilities for SINE, are comparable across classes and methods, so that taking the argmax does not systematically favor visual reference methods over textual ones.","fun_headline_variants_meta":{"raw":{"variants":["Show beats Tell in segmentation benchmark","Visual prompts outperform text prompts in segmentation","Image references win over class names in segmentation","Benchmark: visual prompting dominates text on most tasks","Segmentation: visual cues beat verbal descriptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1414,"prompt_tokens":895,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":511,"tokens_out":519,"duration_ms":5707,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:41:32.575221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SoT comparison with confidence scores normalized per class, for example rank-based or z-scored within each class, before the argmax merge, and check whether GFSAM's average 38.7 mIoU and the visual-method lead over the best textual method shrink or disappear; if they do, the reported advantage is an artifact of the merging rule.","supporting_citations":[{"cited_title":"Bridge the points: Graph-based few-shot segment anything semantically.NeurIPS, 37:33232–33261, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies GFSAM, the best-performing visual reference method; its 38.7 mIoU average with five prompts is the paper's headline quantitative claim."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM is the mask generator used by PerSAM, Matcher, and GFSAM; its confidence output feeds the merging step."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP provides the text and image encoders behind all five open-vocabulary methods compared in the benchmark."},{"cited_title":"Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"CLIP-DINoiser gives the best text-prompt result on PASCAL VOC (62.1 mIoU), the paper's case for textual strength in common scenes."},{"cited_title":"Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation","cited_arxiv_id":null,"evidence_quote":"ProxyCLIP is the best average open-vocabulary method and the main textual baseline that visual methods must beat."},{"cited_title":"A simple im- age segmentation framework via in-context examples","cited_arxiv_id":null,"evidence_quote":"SINE is the one trained visual method; its cross-domain drop supports the paper's claim that training-free methods generalize better."},{"cited_title":"What a mess: Multi-domain evaluation of zero-shot semantic segmentation.Advances in Neural Infor- mation Processing Systems, 36:73299–73311, 2023","cited_arxiv_id":null,"evidence_quote":"The prior multi-domain zero-shot segmentation benchmark motivates SoT but does not compare prompt modalities, which is the gap the paper targets."}],"review_version":1}