Pith. sign in

REVIEW 2 major objections 5 minor 50 references

A zero-shot method turns LVLM output-token attention into action-specific affordance heatmaps that beat weakly supervised baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:18 UTC pith:KZHCXSEG

load-bearing objection Clean zero-shot affordance pipeline that turns LVLM token attention into real SOTA numbers on AGD20K-Unseen and HICO-IIF; the CLIPSeg mask is the main soft spot, not a hidden flaw. the 2 major comments →

arxiv 2607.03595 v1 pith:KZHCXSEG submitted 2026-07-03 cs.CV cs.AIcs.RO

Token-Based Affordance Grounding with Large Vision-Language Models

Seung Il Lee , Qinqian Lei , Daguang Xu , Dong Yang , Robby T. Tan , Yixin Chen , Bo Wang This is my paper
classification cs.CV cs.AIcs.RO
keywords affordance groundingvision-language modelszero-shot learningtoken-level attentionegocentric perceptionweakly supervised baselinescross-modal localization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Affordance grounding asks where in an image a given action can happen—sit on the seat, not the handle; cut with the blade, not the grip. Prior work trains on weak action labels from third-person photos and often fails when several actions co-occur in one image or when short phrases cannot separate similar actions on the same object. This paper claims that large vision–language models already encode the needed action semantics inside their cross-attention maps, but different output tokens look at different places, many of them irrelevant. TokAG therefore aggregates attention across layers and heads, scores each generated token by how strongly it activates inside an off-the-shelf object mask, and keeps only the single most object-focused map as the affordance heatmap—no affordance labels or task training required. On standard benchmarks the resulting zero-shot maps improve NSS by roughly 11% on unseen AGD20K objects and nearly 30% on HICO-IIF relative to previous weakly supervised systems.

Core claim

The authors show that the cross-modal attention maps of LVLM output tokens already contain usable action-specific spatial cues, and that a simple spatial filter—selecting the single token whose aggregated attention is most concentrated on the target object—converts those cues into competitive zero-shot affordance heatmaps that outperform methods trained with weak action supervision.

What carries the argument

Spatial-aware token selection: for each generated token, aggregate its token-to-image attention over all layers and heads, mask by a CLIPSeg object region, score total mass inside the object, and retain the Top-1 token’s masked map as the affordance heatmap.

Load-bearing premise

An off-the-shelf object mask is accurate enough that the token with the highest mass inside that mask is the one carrying the true functional region.

What would settle it

On the AGD20K unseen split, replace the CLIPSeg mask with random or empty masks (or deliberately broken segmentations) and check whether NSS collapses toward the paper’s own failure-case figure of ~1.17; if performance stays high, the object-mask filter is not load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Affordance grounding can be obtained from frozen LVLMs without any exocentric training images or pixel labels.
  • Larger LVLM backbones select more semantically meaningful tokens and yield better localization, so future model scale should improve the maps further.
  • The same token-selection recipe can be swapped onto other LVLM families (e.g., InternVL) and still beat prior weakly supervised scores.
  • Direct coordinate or bounding-box prompting of LVLMs, and simple LVLM+SAM pipelines, are insufficient; attention extraction is required for fine-grained functional parts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Compositional or bimanual actions that need several simultaneous parts will force an extension beyond single-token selection—the paper’s own limitation section already points there.
  • If token selection can be made mask-free (e.g., by internal objectness or multi-token consensus), the method would become a pure LVLM readout usable in robotics without a separate segmenter.
  • The observation that articles like “a”/“the” often carry the strongest spatial signal suggests autoregressive “look-ahead” could be exploited for other dense grounding tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes TokAG, a training-free zero-shot framework for affordance grounding that extracts action-conditioned spatial heatmaps from the cross-modal attention of a frozen LVLM (primarily Qwen3-VL). Given an egocentric image, object name and action, the model is prompted to generate a short descriptive sentence; attention maps of every output token are aggregated across all layers and heads, masked by a CLIPSeg object mask (Eqs. 1–3), and the single token with the highest object-region score is retained as the affordance map. On AGD20K (Seen/Unseen) and HICO-IIF the method reports clear gains over recent weakly-supervised baselines (e.g., NSS 1.514 vs 1.368 on AGD20K Unseen; 1.655 vs 1.234 on HICO-IIF). Ablations isolate the contributions of full layer/head aggregation and Top-1 spatial token selection; failure analysis, multi-instance examples, and a user study are also supplied.

Significance. If the reported zero-shot gains hold under independent re-implementation, TokAG supplies a practical, annotation-free route from off-the-shelf LVLMs to pixel-level affordance maps—an advance that is immediately useful for embodied AI and robotic manipulation. The work is transparent about its auxiliary mask dependence, provides extensive ablations (Tables 5–6, S1–S2), a failure-case breakdown, and a public-code commitment; these strengths make the contribution reproducible and falsifiable. The single-token and FPS limitations are acknowledged and do not undermine the central empirical claim.

major comments (2)
  1. The central claim rests on the reliability of the CLIPSeg binary mask M_obj that filters every token score (Sec. 3.1, Eqs. 1–3). The authors themselves report that 37.7 % of 77 failure cases are segmentation errors and that NSS collapses from 1.665 to 1.168 when the mask fails. While the SAM3 substitution still beats prior SOTA (NSS 1.416), a more systematic quantification of mask quality versus final NSS—ideally with an oracle-mask upper bound—would strengthen confidence that the gains are not largely an artifact of the auxiliary segmenter.
  2. No uncertainty estimates (error bars, multiple random seeds, or bootstrap intervals) accompany the headline numbers in Tables 2–3. Because the method is deterministic once the LVLM and CLIPSeg are fixed, the absence of variability measures is less critical than for trained models, yet a short sensitivity study over prompt paraphrases or temperature would still help readers judge the stability of the 10.7 % / 29.7 % NSS improvements.
minor comments (5)
  1. The free post-processing parameters (power 0.75, Gaussian σ = 0.05 × min(H,W)) are stated only in the supplement; a brief mention in Sec. 3.1 or an ablation would improve reproducibility.
  2. Fig. 1 caption claims “Directly prompting an LVLM o CLIPSeg” produces inaccurate localization; the exact prompt and CLIPSeg threshold used for that baseline should be listed for fairness.
  3. Table 1 semantic categories were assigned by GPT then verified by one human; inter-annotator agreement or a short protocol would increase trust in the 54–65 % “meaningful token” claim.
  4. Runtime (0.59 / 0.20 FPS) is reported honestly; a short note on whether attention can be cached or extracted only from a subset of layers without re-running generation would be useful for practitioners.
  5. Minor typos: “semanticallysimilaractions” (p. 2), “theLVLM’simplicitsemanticsignals” (abstract), and inconsistent capitalization of “Set 1 / Set 2” versus “Seen / Unseen”.

Circularity Check

0 steps flagged

No circularity: zero-shot extraction of LVLM attention maps, filtered by an off-the-shelf external masker, evaluated on held-out external benchmarks with no parameters fitted to the target heatmaps.

full rationale

TokAG's derivation chain is: (1) prompt a frozen pretrained LVLM (Qwen3-VL / InternVL3) with an action-object query to obtain output tokens and their cross-attention maps; (2) aggregate those maps across all layers/heads; (3) score each token by the sum of its attention inside a binary object mask produced by an off-the-shelf CLIPSeg (or SAM3) model that was never trained on AGD20K/HICO-IIF affordance labels; (4) keep the single highest-scoring token's masked map as the heatmap. No free parameters are fitted to the evaluation ground-truth heatmaps, no uniqueness theorem is imported from the authors' prior work, and no self-citation is load-bearing for the central claim. The quantitative gains (NSS 1.514 / 1.655) are measured against independent external benchmarks under a stated zero-shot protocol. Dependence on CLIPSeg mask quality is an acknowledged empirical limitation (37.7 % of failure cases), not a definitional or fitted circularity. The method is therefore self-contained; score 0.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The central claim rests on three external components (pretrained LVLM, CLIPSeg, and the assumption that action-conditioned tokens encode functional parts) plus two hand-chosen post-processing constants. No free parameters are fitted to the evaluation heatmaps themselves.

free parameters (2)
  • fractional power scaling exponent = 0.75
    Fixed at 0.75 in supplementary A.2 to soften attention peaks; chosen by hand, not derived.
  • Gaussian blur relative sigma = 0.05
    Set to 0.05 × min(image dimensions) for spatial continuity; hand-chosen scale factor.
axioms (3)
  • domain assumption Cross-attention maps of LVLM output tokens contain usable action-specific spatial cues that can be isolated by object masking.
    Stated as observation in Sec. 3.1 and Fig. 3; never proved, only empirically verified on the benchmarks.
  • domain assumption An off-the-shelf CLIPSeg binary mask is a sufficiently faithful approximation of the target object region for token scoring.
    Invoked in Eq. 1 and Sec. 3.1; the paper itself reports that 37.7% of failures are CLIPSeg errors.
  • ad hoc to paper Aggregating attention across all layers and heads is preferable to selecting any single layer or head in the zero-shot regime.
    Justified by ablation Table 5; alternative selection would require supervision, which the method forbids.
invented entities (1)
  • spatial-aware token-selection mechanism (Top-1 object-region attention score) no independent evidence
    purpose: Chooses which LVLM output token’s attention map becomes the final affordance heatmap.
    Defined by Eqs. 1–3; the paper’s core algorithmic contribution. Independent evidence is limited to the reported correlation with meaningful part names (Table 1).

pith-pipeline@v1.1.0-grok45 · 27472 in / 2657 out tokens · 24627 ms · 2026-07-12T01:18:12.491582+00:00 · methodology

0 comments
read the original abstract

Affordance grounding aims to localize image regions that support a specific action, serving as a core capability for physical intelligence and embodied perception. Previous studies have primarily relied on weakly supervised learning with action labels from exocentric images. However, these methods often struggle with visually ambiguous exocentric images containing co-occurring actions; moreover, they fail to distinguish semantically similar actions because existing methods typically rely on brief action phrases that lack rich semantic details for action-specific localization. Although large vision-language models (LVLMs) encode rich action semantics and their action-conditioned textual outputs implicitly contain spatial cues, they do not directly provide action-specific spatial localization. To address these problems, we propose TokAG, a zero-shot affordance grounding framework that exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision. We observe that attention maps associated with different LVLM output tokens vary significantly, with many attending to irrelevant regions such as the background. Thus, we introduce a spatial-aware token-selection mechanism to systematically evaluate each output token and select the one whose attention maps exhibit dominant activation over the target object, instead of relying on arbitrary attention maps. By extracting these object-focused attention maps, we transform the LVLM's implicit semantic signals into zero-shot affordance heatmaps. Our zero-shot framework consistently outperforms prior weakly supervised approaches across multiple benchmarks, improving NSS by 10.7% on the unseen split of AGD20K and by 29.7% on HICO-IIF. The code and models will be made publicly available.

Figures

Figures reproduced from arXiv: 2607.03595 by Bo Wang, Daguang Xu, Dong Yang, Qinqian Lei, Robby T. Tan, Seung Il Lee, Yixin Chen.

Figure 1
Figure 1. Figure 1: Comparison of affordance grounding results. LOCATE [24] struggles with action co-occurrence in exocentric images, while WSMA [43] fails to distinguish se￾mantically similar actions due to its reliance on short action phrases. Directly prompt￾ing a large vision–language model (LVLM, Qwen3-VL [3]) also produces inaccurate localization. Since LVLMs do not output heatmaps, the results are obtained using CLIPSe… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the TokAG framework. Given an egocentric image and a text prompt, we extract output tokens (e.g., t1, t2, t3) from the language layers of a frozen LVLM. We aggregate token-to-image attention maps across all layers and heads. With object masks generated by CLIPSeg [28], we compute object-region attention weights for each token. The token exhibiting the highest score is selected (e.g., t2, corres… view at source ↗
Figure 3
Figure 3. Figure 3: Attention maps across various layers and heads. (a) Attention maps extracted from early, middle, and deep layers show different spatial activation patterns. (b) Dif￾ferent attention heads also focus on distinct image regions. These observations indicate that affordance-relevant cues are distributed across layers and heads, while some acti￾vations attend to task-irrelevant background regions. where ⊙ denote… view at source ↗
Figure 4
Figure 4. Figure 4: Examples of selected tokens. For each object–action pair, our spatial-aware token selection mechanism identifies the output token whose attention is most con￾centrated on the target object region. The selected token is then decoded into text for visualization. Interestingly, many decoded tokens correspond to meaningful object parts or functional regions (e.g., “blade”, “mattress”, “seat”, “handle”, and “co… view at source ↗
Figure 5
Figure 5. Figure 5: Examples of Multi-objects and Multi-part scenarios Token Selection Strategy. Existing visual grounding approaches typically rely on a predefined token, such as the [CLS] token in vision transformers [8, 41]. However, generative LVLMs do not contain a dedicated classification token, and therefore require alternative strategies. Previous work has explored using the first output token [47] or the last prompt … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 9 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2004.07400 (2020)

    Ardón, P., Pairet, È., Lohan, K.S., Ramamoorthy, S., Petrick, R.: Affordances in robotic tasks–a survey. arXiv preprint arXiv:2004.07400 (2020)

    Pith/arXiv arXiv 2004

  2. [2]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Bahl, S., Mendonca, R., Chen, L., Jain, U., Pathak, D.: Affordances from human videos as a versatile representation for robotics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13778–13790 (2023)

    2023

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al.: Do as i can, not as i say: Grounding language inroboticaffordances.In:Conferenceonrobotlearning.pp.287–318.PMLR(2023)

    2023

  5. [5]

    arXiv preprint arXiv:2605.23891 (2026)

    Cao, X., Qu, Y., Xiao, W., Hu, J., Li, H., Liu, J., Huang, Z., Li, X., et al.: Smart- insertion-v: Photorealistic video insertion via a closed-loop feedback dual-stream framework. arXiv preprint arXiv:2605.23891 (2026)

    Pith/arXiv arXiv 2026

  6. [6]

    Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., Lei, J., Ma, T., Guo, B., Kalla, A., Marks, M., Greer, J., Wang, M., Sun, P., Rädle, R., Afouras, T., Mavroudi, E., Xu, K., Wu, T.H., Zhou, Y., Momeni, L., Hazra, R., Ding, S., Vaze, S., Porcher, F., Li, F., Li, S., Kamath, A., Cheng, H.K., ...

    2025

  7. [7]

    In: 2018 ieee winter conference on applications of computer vision (wacv)

    Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: 2018 ieee winter conference on applications of computer vision (wacv). pp. 381–389. IEEE (2018)

    2018

  8. [8]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visu- alization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 782–791 (2021)

    2021

  9. [9]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Chuang, C.Y., Li, J., Torralba, A., Fidler, S.: Learning to act properly: Predicting and explaining affordances from images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 975–983 (2018)

    2018

  10. [10]

    IEEE Transactions on Pattern Analysis and Machine Intelligence43(11), 4125–4141 (2020)

    Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence43(11), 4125–4141 (2020)

    2020

  11. [11]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., Ye, Q.: Ts-cam: Token semantic coupled attention map for weakly supervised object localization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2886–2895 (2021)

    2021

  12. [12]

    arXiv preprint arXiv:2212.06727 (2022)

    Ghiasi, A., Kazemi, H., Borgnia, E., Reich, S., Shu, M., Goldblum, M., Wilson, A.G., Goldstein, T.: What do vision transformers learn? a visual exploration. arXiv preprint arXiv:2212.06727 (2022)

    Pith/arXiv arXiv 2022

  13. [13]

    Psychol- ogy press (2014)

    Gibson, J.J.: The ecological approach to visual perception: classic edition. Psychol- ogy press (2014)

    2014

  14. [14]

    ACM Computing Surveys (CSUR)54(3), 1–35 (2021)

    Hassanin, M., Khan, S., Tahtali, M.: Visual affordance and function understanding: A survey. ACM Computing Surveys (CSUR)54(3), 1–35 (2021)

    2021

  15. [15]

    In: European Conference on Computer Vision

    Jang, J.H., Seo, H., Chun, S.Y.: Intra: Interaction relationship-aware weakly su- pervised affordance grounding. In: European Conference on Computer Vision. pp. 18–34. Springer (2024) TokAG 17

    2024

  16. [16]

    Jiang, M., Ruan, Y., Huang, S., Liao, S., Pitis, S., Grosse, R.B., Ba, J.: Calibrating language models via augmented prompt ensembles (2023)

    2023

  17. [17]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Kang, S., Kim, J., Kim, J., Hwang, S.J.: See what you are told: Visual attention sink in large multimodal models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  18. [18]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Kang, S., Kim, J., Kim, J., Hwang, S.J.: Your large vision-language model only needs a few attention heads for visual grounding. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 9339–9350 (2025)

    2025

  19. [19]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)

    2023

  20. [20]

    The International journal of robotics research32(8), 951–970 (2013)

    Koppula, H.S., Gupta, R., Saxena, A.: Learning human activities and object af- fordances from rgb-d videos. The International journal of robotics research32(8), 951–970 (2013)

    2013

  21. [21]

    In: In Proceedings of the IEEE/CVF inter- national conference on computer vision (2025)

    Lei, Q., Wang, B., Robby T., T.: Hola: Zero-shot hoi detection with low-rank decomposed vlm feature adaptation. In: In Proceedings of the IEEE/CVF inter- national conference on computer vision (2025)

    2025

  22. [22]

    Advances in Neural Information Processing Systems37, 55831–55857 (2024)

    Lei, Q., Wang, B., Tan, R.: Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection. Advances in Neural Information Processing Systems37, 55831–55857 (2024)

    2024

  23. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)

    Lei, Q., Wang, B., Tan, R.T.: Crosshoi-bench: A unified benchmark for hoi eval- uation across vision-language models and hoi-specific methods. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)

    2026

  24. [24]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, G., Jampani, V., Sun, D., Sevilla-Lara, L.: Locate: Localize and transfer ob- ject parts for weakly supervised affordance grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10922– 10931 (2023)

    2023

  25. [25]

    arXiv preprint arXiv:2602.09609 (2026)

    Liu, J., Li, T., Cao, X., Ma, Y., Shang, G., Huang, H., Zhang, C., Chang, X., Huang, Z., Hu, J., et al.: Tele-omni: a unified multimodal framework for video generation and editing. arXiv preprint arXiv:2602.09609 (2026)

    arXiv 2026

  26. [26]

    In: European conference on computer vision

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024)

    2024

  27. [27]

    arXiv preprint arXiv:2310.02255 (2023)

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.W., Galley, M., Gao, J.: Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 (2023)

    Pith/arXiv arXiv 2023

  28. [28]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 7086–7096 (2022)

    2022

  29. [29]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Luo, H., Zhai, W., Zhang, J., Cao, Y., Tao, D.: Learning affordance grounding from exocentric images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2252–2261 (2022)

    2022

  30. [30]

    In: European Conference on Computer Vision

    Ma, K., Dong, H., Mu, Y.: Local occupancy-enhanced object grasping with multi- ple triplanar projection. In: European Conference on Computer Vision. pp. 1–18. Springer (2024)

    2024

  31. [31]

    In: CVPR

    Mai, J., Yang, M., Luo, W.: Erasing integrated learning: A simple yet effective ap- proach for weakly supervised object localization. In: CVPR. pp. 8766–8775 (2020)

    2020

  32. [32]

    Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? Ad- vances in neural information processing systems32(2019) 18 Lee et al

    2019

  33. [33]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Moon, W., Seong, H.S., Heo, J.P.: Selective contrastive learning for weakly su- pervised affordance grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5210–5220 (2025)

    2025

  34. [34]

    In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision

    Mur-Labadia, L., Guerrero, J.J., Martinez-Cantin, R.: Multi-label affordance map- ping from egocentric vision. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 5238–5249 (2023)

    2023

  35. [35]

    In: International Conference on Robotics and Automa- tion2

    Myers, A., Kanazawa, A., Fermuller, C., Aloimonos, Y.: Affordance of object parts from geometric features. In: International Conference on Robotics and Automa- tion2. pp. 5–6 (2015)

    2015

  36. [36]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Pan, X., Gao, Y., Lin, Z., Tang, F., Dong, W., Yuan, H., Huang, F., Xu, C.: Unveiling the potential of structure preserving for weakly supervised object lo- calization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11642–11651 (2021)

    2021

  37. [37]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  38. [38]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Sawatzky, J., Srikantha, A., Gall, J.: Weakly supervised affordance detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2795–2804 (2017)

    2017

  39. [39]

    In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision

    Sun, P., Chen, S., Zhu, C., Xiao, F., Luo, P., Xie, S., Yan, Z.: Going denser with open-vocabulary part segmentation. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. pp. 15453–15465 (2023)

    2023

  40. [40]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Tang, J., Wei, Z., Zheng, G., Yang, S.: Closed-loop transfer for weakly-supervised affordance grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9530–9539 (2025)

    2025

  41. [41]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, J., Kang, W., Tang, H., Hong, Y., Yan, Y.: On the faithfulness of vision trans- former explanations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10936–10945 (2024)

    2024

  42. [42]

    arXiv preprint arXiv:2412.10302 (2024)

    Wu,Z.,Chen,X.,Pan,Z.,Liu,X.,Liu,W.,Dai,D.,Gao,H.,Ma,Y.,Wu,C.,Wang, B., et al.: Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024)

    Pith/arXiv arXiv 2024

  43. [43]

    In: Proceedings of the AAAI Conference on Ar- tificial Intelligence

    Xu, L., Gao, Y., Song, W., Hao, A.: Weakly supervised multimodal affordance grounding for egocentric images. In: Proceedings of the AAAI Conference on Ar- tificial Intelligence. vol. 38, pp. 6324–6332 (2024)

    2024

  44. [44]

    arXiv preprint arXiv:2505.24103 (2025)

    Xu, P., Mu, Y.: Weakly-supervised affordance grounding guided by part-level se- mantic priors. arXiv preprint arXiv:2505.24103 (2025)

    Pith/arXiv arXiv 2025

  45. [45]

    In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision

    Yang, Y., Zhai, W., Luo, H., Cao, Y., Luo, J., Zha, Z.J.: Grounding 3d object affordance from 2d interactions in images. In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision. pp. 10905–10915 (2023)

    2023

  46. [46]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9556–9567 (2024)

    2024

  47. [47]

    arXiv preprint arXiv:2502.17422 (2025)

    Zhang, J., Khayatkhoei, M., Chhikara, P., Ilievski, F.: Mllms know where to look: Training-free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502.17422 (2025)

    Pith/arXiv arXiv 2025

  48. [48]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016) TokAG 19

    2016

  49. [49]

    When people perform {action} with {object_name}, which part of the {object_name} is used for ‘{action}’? Answer in one sentence

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) 20 Lee et al. Supplementary Material for Token-Based Affordance Grounding with Large Vision-Language Models A Implementati...

    Pith/arXiv arXiv 2025

  50. [50]

    a task-oriented visual AI

    or “a task-oriented visual AI” (Case 2), leads to slight performance improve- ments (e.g., KLD of 1.039 for Case 1). However, the “Original” prompt, which specifies the action-object relationship without strict formatting constraints, also achieves competitive results. We therefore use it as the default configuration in the main experiments to demonstrate...

    arXiv