{"id":"9b4ac61a-9fde-4ccc-a0ad-a09fe08677ac","arxiv_id":"2607.03595","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"TokAG selects the LVLM output token whose aggregated cross-attention is most concentrated on a CLIPSeg object mask, converting that map into a zero-shot affordance heatmap that outperforms weakly supervised baselines.","lead":"TokAG turns the internal attention of large vision-language models into zero-shot heatmaps that mark which parts of an object support a given action. The method beats prior weakly supervised affordance grounders and could let robots and AR systems reason about interactions without labeled training data.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the already-flagged CLIPSeg dependence.","rationale":"The strongest claim is an empirical performance statement that is directly supported by Tables 2–3 and the ablations in Tables 5–6. The reader correctly identifies the only material vulnerability—dependence on an off-the-shelf object mask—and the paper already quantifies its impact. No additional load-bearing concern (mathematical, circular, or evaluation-protocol) surfaces under careful reading. Therefore the ACCEPT verdict stands; the concrete test above simply makes the already-acknowledged sensitivity fully transparent.","tokens_in":23344,"tokens_out":436,"duration_ms":4004,"concrete_test":"On the 540-image AGD20K Unseen split, recompute the full pipeline once with ground-truth object masks (or a high-quality oracle segmenter) in place of CLIPSeg and once with deliberately corrupted masks (IoU < 0.3). If oracle masks raise NSS only modestly above 1.514 while corrupted masks drop it below the prior SOTA (1.368), the headline zero-shot claim is robust; a large swing would confirm that mask quality, not token selection, is doing the heavy lifting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader already isolates the paper’s real soft spot: the binary object mask M_obj from CLIPSeg is the spatial filter that decides which LVLM token is kept (Eqs. 1–3). When that mask fails, token selection becomes unreliable and NSS collapses (1.168 vs 1.665). The paper itself reports that 37.7 % of its 77 failure cases are segmentation errors and that swapping CLIPSeg for SAM3 still beats the prior SOTA (NSS 1.416). No deeper internal inconsistency, circular evaluation, or hidden supervision appears; the quantitative gains on AGD20K-Set 2 and HICO-IIF remain intact under the stated zero-shot protocol. The single-token assumption and slow FPS are acknowledged limitations, not load-bearing flaws in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes TokAG, a training-free zero-shot framework for affordance grounding that extracts action-conditioned spatial heatmaps from the cross-modal attention of a frozen LVLM (primarily Qwen3-VL). Given an egocentric image, object name and action, the model is prompted to generate a short descriptive sentence; attention maps of every output token are aggregated across all layers and heads, masked by a CLIPSeg object mask (Eqs. 1–3), and the single token with the highest object-region score is retained as the affordance map. On AGD20K (Seen/Unseen) and HICO-IIF the method reports clear gains over recent weakly-supervised baselines (e.g., NSS 1.514 vs 1.368 on AGD20K Unseen; 1.655 vs 1.234 on HICO-IIF). Ablations isolate the contributions of full layer/head aggregation and Top-1 spatial token selection; failure analysis, multi-instance examples, and a user study are also supplied.","tokens_in":23575,"tokens_out":915,"duration_ms":7591,"significance":"If the reported zero-shot gains hold under independent re-implementation, TokAG supplies a practical, annotation-free route from off-the-shelf LVLMs to pixel-level affordance maps—an advance that is immediately useful for embodied AI and robotic manipulation. The work is transparent about its auxiliary mask dependence, provides extensive ablations (Tables 5–6, S1–S2), a failure-case breakdown, and a public-code commitment; these strengths make the contribution reproducible and falsifiable. The single-token and FPS limitations are acknowledged and do not undermine the central empirical claim.","major_comments":[{"comment":"The central claim rests on the reliability of the CLIPSeg binary mask M_obj that filters every token score (Sec. 3.1, Eqs. 1–3). The authors themselves report that 37.7 % of 77 failure cases are segmentation errors and that NSS collapses from 1.665 to 1.168 when the mask fails. While the SAM3 substitution still beats prior SOTA (NSS 1.416), a more systematic quantification of mask quality versus final NSS—ideally with an oracle-mask upper bound—would strengthen confidence that the gains are not largely an artifact of the auxiliary segmenter.","section":null},{"comment":"No uncertainty estimates (error bars, multiple random seeds, or bootstrap intervals) accompany the headline numbers in Tables 2–3. Because the method is deterministic once the LVLM and CLIPSeg are fixed, the absence of variability measures is less critical than for trained models, yet a short sensitivity study over prompt paraphrases or temperature would still help readers judge the stability of the 10.7 % / 29.7 % NSS improvements.","section":null}],"minor_comments":[{"comment":"The free post-processing parameters (power 0.75, Gaussian σ = 0.05 × min(H,W)) are stated only in the supplement; a brief mention in Sec. 3.1 or an ablation would improve reproducibility.","section":null},{"comment":"Fig. 1 caption claims “Directly prompting an LVLM \to CLIPSeg” produces inaccurate localization; the exact prompt and CLIPSeg threshold used for that baseline should be listed for fairness.","section":null},{"comment":"Table 1 semantic categories were assigned by GPT then verified by one human; inter-annotator agreement or a short protocol would increase trust in the 54–65 % “meaningful token” claim.","section":null},{"comment":"Runtime (0.59 / 0.20 FPS) is reported honestly; a short note on whether attention can be cached or extracted only from a subset of layers without re-running generation would be useful for practitioners.","section":null},{"comment":"Minor typos: “semanticallysimilaractions” (p. 2), “theLVLM’simplicitsemanticsignals” (abstract), and inconsistent capitalization of “Set 1 / Set 2” versus “Seen / Unseen”.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is a solid systems contribution that sits comfortably within the journal’s scope. The CLIPSeg dependence is the only real soft spot, but the authors already quantify it and show that an alternative segmenter still outperforms prior work; I therefore view the issue as addressable by a short revision rather than a fundamental flaw. No novelty or citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: they take a frozen LVLM (Qwen3-VL, also InternVL3), aggregate every layer and head of the cross-attention for each generated token, score those maps against a CLIPSeg object mask, keep the single highest-scoring token, and get heatmaps that beat recent weakly-supervised methods by a clear margin (NSS 1.514 vs 1.368 on AGD20K-Unseen, 1.655 vs 1.234 on HICO-IIF). No training on the target datasets.\n\nWhat is actually new is not “use LVLM attention” (Kang, Zhang already did variants of that). It is the systematic Top-1 object-region token selection plus full aggregation, plus the empirical demonstration that this works for action-specific parts rather than whole objects. The ablations (Tables 5–6 and the long S2 layer/head sweep) isolate the pieces cleanly: single heads or fixed tokens lose, averaging all tokens loses, Top-1 with the mask wins. They also show the selected tokens often decode to functional words (“blade”, “seat”, “bristle”), which is a nice interpretability check, and they test multi-instance cases and an EPIC-Aff sample. Failure analysis is honest: 37.7 % of their 77 errors are CLIPSeg, 62.3 % token choice; swapping to SAM3 still beats prior SOTA.\n\nSoft spots are real but proportional. The method inherits CLIPSeg’s mistakes; when the mask fails, NSS drops from 1.665 to 1.168. Single-token assumption is acknowledged as a limit for compositional or bimanual affordances. Inference is slow (0.59 / 0.20 FPS). Two post-processing constants (0.75 power, 0.05 relative sigma) exist but are not load-bearing. Code is promised, not yet out. None of this is circular fitting or a math error; the zero-shot claim holds under the stated protocol.\n\nThis is for people working on affordance, HOI, or embodied perception who want a training-free baseline that actually moves the numbers. It deserves a serious referee. I would engage with it and expect to cite the selection idea once the repo appears.","headline":"Clean zero-shot affordance pipeline that turns LVLM token attention into real SOTA numbers on AGD20K-Unseen and HICO-IIF; the CLIPSeg mask is the main soft spot, not a hidden flaw.","tokens_in":24179,"tokens_out":577,"would_cite":true,"duration_ms":5782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A zero-shot method turns LVLM output-token attention into action-specific affordance heatmaps that beat weakly supervised baselines.","keywords":["affordance grounding","vision-language models","zero-shot learning","token-level attention","egocentric perception","weakly supervised baselines","cross-modal localization"],"falsifier":"On the AGD20K unseen split, replace the CLIPSeg mask with random or empty masks (or deliberately broken segmentations) and check whether NSS collapses toward the paper’s own failure-case figure of ~1.17; if performance stays high, the object-mask filter is not load-bearing.","tokens_in":24255,"feed_emoji":"🎯","tokens_out":888,"duration_ms":10599,"temperature":0.7,"pith_summary":"Affordance grounding asks where in an image a given action can happen—sit on the seat, not the handle; cut with the blade, not the grip. Prior work trains on weak action labels from third-person photos and often fails when several actions co-occur in one image or when short phrases cannot separate similar actions on the same object. This paper claims that large vision–language models already encode the needed action semantics inside their cross-attention maps, but different output tokens look at different places, many of them irrelevant. TokAG therefore aggregates attention across layers and heads, scores each generated token by how strongly it activates inside an off-the-shelf object mask, and keeps only the single most object-focused map as the affordance heatmap—no affordance labels or task training required. On standard benchmarks the resulting zero-shot maps improve NSS by roughly 11% on unseen AGD20K objects and nearly 30% on HICO-IIF relative to previous weakly supervised systems.","feed_headline":"Zero-shot LVLM tokens beat trained affordance models","feed_subtitle":"Pick the output token most focused on the object and its attention becomes the action map—no labels needed.","key_machinery":"Spatial-aware token selection: for each generated token, aggregate its token-to-image attention over all layers and heads, mask by a CLIPSeg object region, score total mass inside the object, and retain the Top-1 token’s masked map as the affordance heatmap.","core_discovery":"The authors show that the cross-modal attention maps of LVLM output tokens already contain usable action-specific spatial cues, and that a simple spatial filter—selecting the single token whose aggregated attention is most concentrated on the target object—converts those cues into competitive zero-shot affordance heatmaps that outperform methods trained with weak action supervision.","pith_inferences":["Compositional or bimanual actions that need several simultaneous parts will force an extension beyond single-token selection—the paper’s own limitation section already points there.","If token selection can be made mask-free (e.g., by internal objectness or multi-token consensus), the method would become a pure LVLM readout usable in robotics without a separate segmenter.","The observation that articles like “a”/“the” often carry the strongest spatial signal suggests autoregressive “look-ahead” could be exploited for other dense grounding tasks."],"forward_implications":["Affordance grounding can be obtained from frozen LVLMs without any exocentric training images or pixel labels.","Larger LVLM backbones select more semantically meaningful tokens and yield better localization, so future model scale should improve the maps further.","The same token-selection recipe can be swapped onto other LVLM families (e.g., InternVL) and still beat prior weakly supervised scores.","Direct coordinate or bounding-box prompting of LVLMs, and simple LVLM+SAM pipelines, are insufficient; attention extraction is required for fine-grained functional parts."],"fun_headline_variants":["Select object-focused LVLM token for zero-shot affordance maps","LVLM output token filter beats weakly supervised affordance models","Token attention selection turns LVLMs into affordance grounders","Spatial-aware token pick yields zero-shot action heatmaps","One object-centric LVLM token outperforms trained affordance methods"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"An off-the-shelf object mask is accurate enough that the token with the highest mass inside that mask is the one carrying the true functional region.","fun_headline_variants_meta":{"raw":{"variants":["Select object-focused LVLM token for zero-shot affordance maps","LVLM output token filter beats weakly supervised affordance models","Token attention selection turns LVLMs into affordance grounders","Spatial-aware token pick yields zero-shot action heatmaps","One object-centric LVLM token outperforms trained affordance methods"]},"model":"grok-4.5","effort":"low","cost_usd":0.0048,"raw_usage":{"total_tokens":1388,"prompt_tokens":834,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":48000000,"prompt_tokens_details":{"text_tokens":834,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":484,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":834,"tokens_out":70,"duration_ms":3922,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:18:12.491582+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the AGD20K unseen split, replace the CLIPSeg mask with random or empty masks (or deliberately broken segmentations) and check whether NSS collapses toward the paper’s own failure-case figure of ~1.17; if performance stays high, the object-mask filter is not load-bearing.","supporting_citations":[],"review_version":1}