{"id":"9cd256d7-9584-4bcd-913a-0ac45b4c340a","arxiv_id":"2505.04905","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A weakly supervised localizer that builds a coarse class-aware map with a transformer, then picks the best SAM mask from a grid-prompted gallery by pixel-level overlap.","lead":"This paper uses the Segment Anything Model (SAM) as a test-time postprocessor to turn coarse, class-trained attention maps into fine-grained object masks for weakly supervised object localization. It reports large gains in high-IoU localization accuracy on CUB-200-2011 and ILSVRC, though the abstract's state-of-the-art claim is contradicted by its own table on CUB.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mask-gallery generation step, Eq. (10), is inconsistent with SAM's forward interface: feeding all grid points as a single prompt set returns at most three masks, not a gallery; the reported results depend on an unspecified auto-mask pipeline.","rationale":"Pro2SAM is a test-time SAM post-processing pipeline for WSOL: GTFormer produces a coarse foreground map, SAM with grid points produces a mask gallery, and pixel-level similarity selects the best mask. The reported gains are substantial and the ablations (GTFormer components, loss weights, token counts, matching mechanism) are informative, so the paper deserves follow-up. The reader's weakest assumption was that the mask gallery must contain a mask that accurately covers the foreground object; that is a valid upper-bound concern. My concern is more fundamental: the paper's formal specification of how the gallery is generated, Eq. (10), does not match SAM's documented behavior, and the actual hyperparameters of the likely automatic-mask-generator pipeline are omitted. This makes the central construction ambiguous: a faithful reader cannot reproduce the mask gallery from the text, and the claimed fine-grained localization results cannot be verified. This is not an accusation of misconduct; the authors may have used SAM's automatic mask generator and reported results honestly, but the method as written is internally inconsistent on a load-bearing step. The reader's verdict of CONDITIONAL already requires specification or code release, and my concern reinforces that requirement rather than changing it. I therefore keep the verdict unchanged, with the concrete expectation that the authors clarify the prompt mechanism, release the grid configuration, and add an oracle upper-bound analysis of the gallery so that the reader can see how much headroom the mask matching step actually exploits.","tokens_in":13590,"tokens_out":6882,"duration_ms":77745,"concrete_test":"Run the released SAM ViT-H on 100 ILSVRC validation images in the two ways: (1) feed all grid points as one prompt set exactly as written in Eq. (10) and count the number of masks returned per image; (2) run SAM's automatic mask generator with the same grid and the paper's implied post-processing, recording all hyperparameters needed. If variant (1) returns at most 3 masks per image while variant (2) requires parameters that the paper does not state, Eq. (10) is not a valid description of the method. Additionally, recompute Table 2's IOU90 while varying the binarization threshold for the GTFormer coarse map M_b over 0.3, 0.5, and 0.7; if the reported 45-48% IOU90 shifts materially with the threshold, the mask matching step of Eq. (11) is under-specified and the headline numbers are not uniquely defined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the mask gallery as M_L_SAM = SAM(I, [point_1, ..., point_G]), implying that all grid points are supplied as one sparse prompt set. In SAM's released interface, a single forward pass through the mask decoder with a set of points returns at most 3 masks (whole, part, subpart) plus confidence scores, not a gallery of all object masks. The paper then states that \"SAM uses a complex post-processing operation to exclude some noisy masks,\" which describes SAM's automatic mask generator (amg.py), where grid points are processed as individual prompts over many forward passes, followed by duplicate removal and NMS. The text therefore conflates two distinct SAM modes. If Eq. (10) is taken literally, the gallery collapses to at most 3 masks per image and the reported IOU90 numbers (48.43% on CUB, 45.94% on ILSVRC) cannot be produced. If the authors instead used the automatic mask generator, the procedure is underspecified: grid density (points_per_side), crop overlap, prediction IoU threshold, stability score threshold, and NMS settings are not reported, and the promised supplementary comparison of single point/box/grid prompts is not present in arXiv v1. This is load-bearing because the mask gallery is the sole source of fine-grained masks in the pipeline; without a faithful specification of this step, the central claim that selected SAM masks achieve state-of-the-art localization cannot be reproduced or independently verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Pro2SAM, a weakly supervised object localization pipeline that combines a newly designed Global Token Transformer (GTFormer) with the frozen Segment Anything Model (SAM). GTFormer is trained with image-level labels to produce a coarse foreground map, which is then used as a mask prompt to select, from a set of SAM-generated masks, the mask that best matches the coarse map via a pixel-level intersection/union score. The selected mask is the final localization output. The authors report strong results on CUB-200-2011 and ILSVRC, including 84.03% Top-1 Loc on CUB and 66.85% on ILSVRC, and notably high MaxBoxAccV2 IOU90 scores. The core idea is coherent and the three-stage pipeline is clearly presented, but there are several load-bearing ambiguities and claim inconsistencies that need to be resolved before the paper is publishable.","tokens_in":13851,"tokens_out":6006,"duration_ms":59773,"significance":"If the reported results are reproducible, the paper makes a valuable contribution by showing that a frozen, zero-shot foundation model can be used to refine coarse WSOL activation maps without any additional training. The GTFormer design is a reasonable extension of transformer-based WSOL, and the proposed mask-matching mechanism is simple and effective in principle. The reported IOU90 MaxBoxAccV2 numbers are substantially higher than prior WSOL methods, which is a meaningful advance for fine-grained localization evaluation. However, the significance is currently tempered by (i) an inaccurate state-of-the-art claim in the abstract relative to the paper's own Table 1, and (ii) a central methodological ambiguity in how the SAM mask gallery is generated, without which the experimental results cannot be reproduced or even unambiguously interpreted.","major_comments":[{"comment":"Equation (10) defines the mask gallery as M_L_SAM = SAM(I, [point_1, ..., point_G]), implying that all grid points are passed as a single sparse prompt set. In the released SAM interface, a single forward pass through the mask decoder with a set of points returns at most three masks (whole, part, and subpart), not a mask gallery whose size L varies per image. The text's statement that 'SAM uses a complex post-processing operation to exclude some noisy masks' describes the automatic mask generator (amg.py), where grid points are processed as individual prompts over many forward passes, followed by duplicate removal and NMS. If Eq. (10) is taken literally, the gallery collapses to at most three masks per image and the reported IOU90 results cannot be produced; if the automatic mask generator was used, this is the load-bearing implementation detail and it must be specified exactly. Please clarify how the grid prompts are issued, how many forward passes are made per image, and what post-processing parameters (points_per_side, prediction IoU threshold, stability score threshold, NMS settings, crop overlap) are used.","section":"§3.3, Eq. (10) and §3.1"},{"comment":"The abstract claims state-of-the-art performance on both CUB-200-2011 and ILSVRC, but Table 1 shows that LocLoc reports 84.40% Top-1 Loc and 98.10% GT-Known on CUB, while Pro2SAM reports 84.03% and 95.67%. Pro2SAM is therefore not state-of-the-art on CUB; the claim should be restricted to ILSVRC or stated as second-best on CUB, and the comparison with LocLoc should be discussed in the text rather than only appearing in the table.","section":"Abstract and Table 1"},{"comment":"Several parameters essential for reproducibility are missing. The grid-point prompt density G defined in Eq. (10) is never assigned a value or described in terms of spatial density; the SAM checkpoint is only described as 'ViT-H model' without the specific released version; and no details are given for the automatic mask generator settings if that is what was used. In addition, no error bars or number of runs are reported, which is particularly important because Table 1 contains comparisons with differences under one point (e.g., Ours-GTFormer 80.48 vs SAT 80.43 Top-1 Loc on CUB). Please provide the missing settings and report variance or at least multiple-seed results for the main claims.","section":"§4.1, Implementation Details"}],"minor_comments":[{"comment":"The table labelled 'T able 4: Ablation studies for numbers of Global Token on ILSVRC dataset' actually contains the (mu, lambda) hyperparameter sweep, while the subsequent Table 5 reports the number of global tokens; the Table 4 caption appears to be a copy-paste error and should be corrected.","section":"§4.3, captions of Tables 4 and 5"},{"comment":"The sentence claiming IOU70 and IOU90 improvements of '14.92% and 18.80% compared with the second method TokenCut on CUB' does not match the table: for IOU90, Pro2SAM's 48.43 vs TokenCut's 25.96 is a relative improvement of about 86.5%, not 18.80%. The text should state whether improvements are absolute percentage points or relative percentages and correct the numbers accordingly.","section":"§4.2, paragraph on Fine-grained Localization"},{"comment":"The paper promises 'detailed quantitative and qualitative comparisons of single point, single box, and grid points in Supplementary Material,' but no supplementary material is available with this arXiv version. Please either include the supplementary material referenced by the paper or remove the reference.","section":"§4.4, Discussions"},{"comment":"The summation indices H and W in Eq. (11) are typed in the same style as image height and width, which is confusing; the limits h/p and w/p suggest these are patch indices, so a notation such as i and j or a clarification of the resize/resampling step would improve readability.","section":"Eq. (11)"},{"comment":"The phrase 'we initiatively leverage the capability' is awkward and should be rephrased to 'we are the first to leverage' or 'we propose to leverage' for clarity.","section":"§1, Introduction"}],"recommendation":"major_revision","confidential_remarks":"The central methodological ambiguity in Eq. (10) is serious and must be resolved before the results can be trusted. In addition, the abstract's SOTA claim is contradicted by the paper's own Table 1 on CUB. These are fixable with a careful revision, so I am not recommending rejection, but the revision must include an exact specification of the SAM prompting protocol, ideally with code or a detailed algorithmic description, and a corrected statement of the claimed state of the art."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2505.04905. The hook is real: using a frozen SAM with grid points to generate a mask gallery, then selecting the mask that best matches a coarse WSOL map, is a sensible and apparently powerful post-processing step. The reported IOU90 numbers (48.43 on CUB, 45.94 on ILSVRC) are an order of magnitude better than previous WSOL methods, and the ablations confirm that mask matching, not just SAM, is driving the gain. GTFormer with its global tokens is a modest but real architectural addition, and the paper is clearly written.\n\nThe soft spots are substantial. The abstract's SOTA claim is contradicted by Table 1: on CUB Top-1 Loc, LocLoc gets 84.40, Pro2SAM gets 84.03. So 'state-of-the-art on both' is false. More importantly, the mask-gallery step is underspecified. Eq. (10) writes M = SAM(I, [point_1..point_G]), which reads as a single SAM call with all points as one prompt set. But SAM's standard mask decoder returns at most three masks for a prompt set, not a gallery. The paper then mentions SAM's 'complex post-processing operation', which is the automatic mask generator—a different code path that runs many prompts and NMS. If the authors used the automatic generator, they need to report grid density, points_per_side, prediction IoU threshold, stability score, NMS settings, and so on. None are given. Without that, nobody can reproduce the 45.94 IOU90, and the method's core selection mechanism cannot be evaluated. The comparison also mixes protocols: baselines are pure WSOL models, while Pro2SAM gets a frozen SAM at test time, a model trained on 1.1B masks. A simple SAM-only baseline (e.g., largest predicted mask, or mask chosen by GT boxes) would show how much the GTFormer coarse map actually contributes.\n\nStill, the core idea is worth pursuing. The gains are big enough that the paper deserves a serious referee, but it needs major revision: correct the abstract, fully specify or release the SAM configuration, add a SAM baseline, and ideally release code. I'd send it to review, not desk-reject—an expert referee can check whether the numbers hold up once the setup is clear.","headline":"Pro2SAM's mask-matching idea is promising and the IOU90 gains are striking, but the paper overclaims SOTA and leaves the SAM gallery generation too underspecified to reproduce.","tokens_in":14438,"tokens_out":3601,"would_cite":true,"duration_ms":34740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a frozen SAM, prompted by a grid of points and steered by a transformer's coarse foreground map, produces fine-grained object localization from image-level labels alone.","keywords":["weakly supervised object localization","Segment Anything Model","mask prompt","grid point prompt","Global Token Transformer","fine-grained localization","MaxBoxAccV2","class activation map"],"falsifier":"Take an image set where SAM is known to produce incomplete or noisy masks (for example, translucent objects, heavy occlusion, or out-of-distribution classes), then compare Pro2SAM's IOU90 against a variant that skips SAM and uses the GTFormer map directly. If the two accuracies converge or the GTFormer-only variant wins, the SAM mask gallery lacked a covering mask, falsifying the claim that mask matching is responsible for the gain.","tokens_in":13340,"feed_emoji":"🎯","tokens_out":7091,"duration_ms":62993,"temperature":0.7,"pith_summary":"This paper tries to establish that a frozen Segment Anything Model (SAM) can supply the fine-grained object boundaries that weakly supervised object localization (WSOL) methods lack. The proposed pipeline, Pro2SAM, first trains a Global Token Transformer (GTFormer) to emit a coarse foreground map from image-level labels alone. It then prompts SAM with a grid of points to create a gallery of candidate masks and selects the mask whose pixel-level overlap with the coarse map is highest. The paper reports that this selection lifts Top-1 localization accuracy to 84.03% on CUB-200-2011 and 66.85% on ILSVRC, with the largest gains at strict IoU thresholds such as IOU90. A reader should care because the result suggests that precise localization can be obtained without any box annotation by pairing a weak class-conditional map with a strong zero-shot segmenter.","feed_headline":"Frozen SAM lifts weak-supervision localization to 84% Top-1 on CUB","feed_subtitle":"The coarse map only selects among SAM's zero-shot masks, sharply raising strict-IoU localization accuracy.","key_machinery":"The load-bearing machinery is a three-stage selection pipeline. Stage one is GTFormer, a DeiT-S transformer augmented with four learned global tokens; their attention to patch tokens produces a coarse foreground probability map that is trained with classification, edge, and region losses. Stage two feeds a preset grid of point prompts into SAM's frozen prompt encoder and mask decoder to obtain a mask gallery, where the grid avoids the omissions seen with a single point or box. Stage three computes a pixel-level similarity score between the coarse map and each SAM mask, using logical AND over OR (Eq. 11), and returns the highest-scoring mask as the final localization map. The argument rests on this gallery containing at least one mask that closely covers the true foreground object, so the matching step only has to rank it first.","core_discovery":"The central claim is that SAM's semantic ambiguity—its habit of returning background and distractor masks alongside foreground—can be resolved by a mask-prompt matching scheme. GTFormer's output map serves as a mask prompt; a dense grid of point prompts makes SAM produce many candidate masks; and a logical AND/OR overlap score (Eq. 11) picks the candidate most similar to the prompt. The chosen mask is declared the localization result. On CUB-200-2011 this procedure reaches 48.43% MaxBoxAccV2 at IOU90, and on ILSVRC 45.94%, higher than the compared methods, while also improving Top-1 Loc to 84.03% and 66.85% respectively. The paper claims this shows a frozen SAM, never fine-tuned, can overcome the partial-activation problem of CAM and the boundary-dilation problem of transformer attention maps.","pith_inferences":["One could test the coarse map's role by replacing GTFormer with a plain CAM or self-attention map; if the matching step still selects good SAM masks, the transformer's global tokens are not essential to the localization gain.","Grid-point density is a promising free parameter: too sparse a grid may miss small objects, and too dense a grid may oversegment, so measuring performance across object scales could expose a scale limit that the current benchmarks do not.","The same mask-matching scheme could be applied to unsupervised object discovery, where no image-level labels exist at all, by ranking SAM masks with a self-supervised similarity signal instead of a trained coarse map.","The large IOU90 margins suggest a new evaluation regime for WSOL, but they are reported on two datasets; replicating on fine-grained domains beyond birds (e.g., cars, aircraft) would show whether SAM's mask gallery is consistently rich enough."],"forward_implications":["WSOL can be decomposed into a coarse map generator and an off-the-shelf frozen segmenter, so fine-grained localization no longer requires training a model to draw sharp boundaries.","Strict-IoU metrics (IOU70, IOU90) become the informative benchmark for WSOL, since the method's advantage over prior work is concentrated at those thresholds.","Because SAM is not fine-tuned, the same mask gallery and matching step can be reused with any coarse localization source, including CAMs from CNN classifiers.","The method's localization error is upper-bounded by SAM's mask gallery: on any image where SAM fails to segment the object, Pro2SAM cannot recover a better box.","The reported IOU90 gains, if they persist, call for re-ranking WSOL methods by MaxBoxAccV2 at high thresholds rather than Top-1 Loc alone."],"supporting_citations":[{"why":"Supplies the frozen SAM model whose prompt encoder and mask decoder generate the candidate mask gallery.","marker":"[15]"},{"why":"Provides the edge loss and region loss used to train GTFormer and serves as a transformer-based baseline that Pro2SAM outperforms.","marker":"[34]"},{"why":"Introduces the first transformer-based WSOL method and the classification-loss convention GTFormer follows.","marker":"[43]"},{"why":"Defines the CAM baseline that establishes the partial-activation problem Pro2SAM addresses.","marker":"[48]"},{"why":"Provides a strong CAM-based baseline used in quantitative and IOU-threshold comparisons.","marker":"[49]"},{"why":"Supplies the unsupervised TokenCut method whose IOU90 score is the direct comparison point for Pro2SAM's strict-IoU gains.","marker":"[30]"},{"why":"Defines the MaxBoxAccV2 evaluation metric the paper uses to measure fine-grained localization at IOU50/70/90.","marker":"[7]"},{"why":"Gives the LocLoc class-agnostic baseline that Pro2SAM is compared against on CUB-200-2011.","marker":"[3]"}],"fun_headline_variants":["Mask-prompted SAM nails weak-supervision localization","Grid prompts make SAM a top weakly-supervised localizer","SAM's zero-shot masks boost WSOL to 84% on CUB","Pro2SAM: SAM with grid points beats CAM and attention","Mask prompt + SAM: state-of-the-art weak localization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the mask gallery that SAM generates: if no candidate mask accurately covers the full foreground object, no similarity score can select a good localization map, and the reported gains would vanish.","fun_headline_variants_meta":{"raw":{"variants":["Mask-prompted SAM nails weak-supervision localization","Grid prompts make SAM a top weakly-supervised localizer","SAM's zero-shot masks boost WSOL to 84% on CUB","Pro2SAM: SAM with grid points beats CAM and attention","Mask prompt + SAM: state-of-the-art weak localization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2455,"prompt_tokens":1041,"completion_tokens":1414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":1328}},"tokens_in":657,"tokens_out":1414,"duration_ms":10801,"temperature":1.0,"reasoning_tokens":1328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:18:53.774316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image set where SAM is known to produce incomplete or noisy masks (for example, translucent objects, heavy occlusion, or out-of-distribution classes), then compare Pro2SAM's IOU90 against a variant that skips SAM and uses the GTFormer map directly. If the two accuracies converge or the GTFormer-only variant wins, the SAM mask gallery lacked a covering mask, falsifying the claim that mask matching is responsible for the gain.","supporting_citations":[{"cited_title":"In: Processing of the CVPR (2016)","cited_arxiv_id":null,"evidence_quote":"Defines the CAM baseline that establishes the partial-activation problem Pro2SAM addresses."},{"cited_title":"In: Proceedings of the ICCV","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen SAM model whose prompt encoder and mask decoder generate the candidate mask gallery."},{"cited_title":"In: Proceedings of the ICCV","cited_arxiv_id":null,"evidence_quote":"Provides the edge loss and region loss used to train GTFormer and serves as a transformer-based baseline that Pro2SAM outperforms."},{"cited_title":"IEEE Transactions on Neural Networks and Learning Systems p","cited_arxiv_id":null,"evidence_quote":"Introduces the first transformer-based WSOL method and the classification-loss convention GTFormer follows."},{"cited_title":"In: Proceedings of the ECCV","cited_arxiv_id":null,"evidence_quote":"Provides a strong CAM-based baseline used in quantitative and IOU-threshold comparisons."},{"cited_title":"In: Proceedings of the CVPR","cited_arxiv_id":null,"evidence_quote":"Supplies the unsupervised TokenCut method whose IOU90 score is the direct comparison point for Pro2SAM's strict-IoU gains."},{"cited_title":"In: Proceedings of the CVPR","cited_arxiv_id":null,"evidence_quote":"Defines the MaxBoxAccV2 evaluation metric the paper uses to measure fine-grained localization at IOU50/70/90."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the LocLoc class-agnostic baseline that Pro2SAM is compared against on CUB-200-2011."}],"review_version":1}