Pith. sign in

REVIEW 4 major objections 5 minor 75 references

RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAGNet adds 273k multi-domain images and 26k reasoning instructions to affordance learning, and AffordanceNet, trained on it, claims zero-shot grasp-region segmentation and 70 percent real-robot grasping success.

desk verdict The dataset is a real resource, but the label pipeline needs an audit before the generalization claims hold. read the letter →

arxiv 2507.23734 v1 pith:LCHEUUNB submitted 2025-07-31 cs.CV cs.RO

Dongming Wu , Yanping Fu , Saike Huang , Yingfei Liu , Fan Jia , Nian Liu , Feng Dai , Tiancai Wang
show 3 more authors
Rao Muhammad Anwer Fahad Shahbaz Khan Jianbing Shen
This is my paper
classification cs.CVcs.RO
keywords affordancesegmentationroboticgraspingvision-languagemodelreasoninginstructionsopen-worldgeneralizationbenchmarkdatasetzero-shotembodiedAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a sufficiently large and diverse collection of grasping-oriented affordance images, paired with instructions that describe an object's function rather than its name, lets a vision-language model segment graspable regions in categories and scenes it has never seen. To support that claim, it builds RAGNet, 273,000 images spanning 180 categories and four embodied data domains (wild photos, robot logs, egocentric video, and simulation), with 26,000 reasoning instructions generated by a large language model. It then trains AffordanceNet, a two-part system: a VLM that turns an image and instruction into an affordance mask, and a pose generator that converts that mask plus depth into a 3D grasp pose. The reported results show the model outperforming prior segmentation baselines and reaching 70 percent average success on a real-robot grasping benchmark with no in-domain training, which is the concrete payoff the paper is trying to establish.

What carries the argument

The load-bearing object is the RAGNet benchmark itself, built by pooling images from four embodied domains and annotating graspable affordance regions with a priority-ordered tool suite: original masks where available, an interactive segmentation model on handle-free objects, a dense-prediction model plus segmenter for language-annotated robot data, a part-level detector plus segmenter for handles, and human annotation only when automatic tools fail. On top of that, the instruction set is engineered to force reasoning: 'easy' instructions name the object, 'hard' instructions describe only function and intention, and 26,000 such instructions are generated synthetically. The model side is carried by the <AFF> token, a special output token distinct from the generic segmentation token, which funnels the language-conditioned mask embedding toward affordance-specific regions, with a pose generator projecting the resulting mask through camera intrinsics and extrinsics to produce 3D grasp poses.

What would settle it

A concrete check is to sample images from each automatic annotation path, have independent human annotators draw affordance masks on them, and compute per-tool agreement; if the automatic outputs match human labels on only a minority of instances, the benchmark's ground truth fails and the reported zero-shot gains need to be re-measured on a human-only training subset.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that dataset scale and instruction difficulty together unlock open-world affordance grounding: RAGNet's combination of many object categories, multiple visual domains, and 'hard' instructions that omit category names teaches a VLM to locate the functional part of an object—handle, body, or door—from a description of what the user wants done. The paper demonstrates this with AffordanceNet, whose AffordanceVLM component is built on a reasoning-segmentation architecture, augmented with a dedicated <AFF> token and an embodied-robot system prompt, and whose pose generator projects the affordance mask into 3D to close the loop with a physical gripper. The claimed payoff is that the system generalizes zero-shot to unseen object categories and unseen image domains, and transfers to real-robot manipulation on objects and scenes not present in training, outperforming standard grasping and affordance-segmentation baselines.

Load-bearing premise

The load-bearing premise is that RAGNet's automatically generated affordance masks are accurate enough to serve as training ground truth, since the paper reports no human quality audit or inter-annotator agreement for the large fraction of masks produced by automatic tools, and if those masks are systematically wrong the model learns their errors and the comparison against the same tools becomes partly circular.

Editorial extensions

If this is right

  • If the benchmark's masks are reliable, the released 273k-image corpus becomes a reusable training resource for any affordance-segmentation method, not just the proposed model.
  • Zero-shot category generalization means a robot can be asked for a category it has never seen and still localize a graspable region, so instruction-driven grasping scales to novel household objects.
  • The hard-instruction format shows that functional descriptions can substitute for category names in grounding, which matters for more natural human-robot dialogue.
  • The same affordance map can be plugged into generic grasp-pose generators, decoupling semantic understanding from low-level manipulation planning.
  • Real-robot success at 70 percent average across ten tasks, with zero demonstration from that scene, is the concrete evidence that dataset scale transfers to physical action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the instruction set deliberately removes category names, a natural extension is to vary the described function for the same object (a mug to drink coffee versus a mug to scoop flour) and test whether the model shifts the grasp region accordingly; that would separate true functional reasoning from scene-context guessing.
  • Editorial inference: The annotation pipeline's reliance on automatic tools suggests a cheap scaling path: re-run the same tool suite on additional video corpora to expand RAGNet beyond 273k images, with human cleanup reserved for the failure cases.
  • Editorial inference: The reported robot-grasping numbers come from ten trials per task on a single scene; repeating the protocol across multiple scenes, object instances, and viewpoints would tell whether the 70 percent average is robust or an artifact of the particular arrangement.
  • Editorial inference: Training on a human-annotated-only subset and comparing zero-shot performance with the full pseudo-labeled training would identify how much of the generalization gain comes from the automatic labels themselves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RAGNet, a large-scale reasoning-based affordance segmentation benchmark containing 273k images across 180 categories and 26k reasoning instructions, drawn from wild, robot, ego-centric, and simulation domains. The authors also present AffordanceNet, a two-stage framework comprising AffordanceVLM (a LISA-based VLM fine-tuned on RAGNet with a dedicated <AFF> token) and a pose generator that converts predicted 2D affordance masks plus depth into 3D grasp poses. Experiments cover affordance segmentation on HANDAL, GraspNet, and 3DOI, real-robot grasping with a UR5, and RLBench simulation, reporting improvements over generalist MLLMs and foundation-model baselines.

Significance. If the annotations are reliable, RAGNet would be a substantial training resource for grasp-oriented affordance perception, with a scale and diversity (multiple domains, 180 categories, reasoning instructions without category names) that goes beyond existing benchmarks. The proposed AffordanceNet shows strong quantitative results on the evaluated benchmarks, and the authors have committed to releasing data and code, which supports reproducibility. The paper also contains useful zero-shot and out-of-domain evaluations. However, the absence of a ground-truth quality audit for the automatically generated masks and the limited statistical power of the robot experiments leave the strongest generalization claims not fully established.

major comments (4)
  1. [§3.2, Table 11] The ground-truth affordance masks in RAGNet are largely produced by automatic tools (SAM2, Florence2+SAM2, VLPart+SAM2), with human annotation used only when automatic tools fail, yet no quality audit is reported. This is load-bearing because the dataset is claimed to be 'carefully annotated' and because AffordanceNet is trained on these masks; systematic errors in the pseudo-labels would be inherited by the model and would undermine the benchmark's validity. The authors should provide per-tool precision/recall or IoU against a human-annotated subset, report inter-annotator agreement, and state the percentage of images annotated by each tool. It should also be noted that the HANDAL validation set uses the original HANDAL masks, so the comparison with VLPart+SAM2 on HANDAL is not directly circular; however, the lack of audit remains a serious gap.
  2. [§3.4, Table 3] The zero-shot evaluation on 'GraspNet novel' is ambiguous because GraspNet is one of the training sources in Table 2. The paper must specify which GraspNet categories are held out from training and confirm that the validation images do not overlap with the training set; otherwise the zero-shot claim is unsupported. Similarly, the relationship between HANDAL and HANDAL† should be clarified: the text says HANDAL† is a random subset of HANDAL, but the former is used for both in-domain and reasoning-based evaluation in different rows of Table 3, so the reader cannot tell which images are seen during training.
  3. [§6.1, Table 7] The real-robot results are based on 10 trials per task, and the paper reports only average success rates without confidence intervals or significance testing. Given the small sample size, the difference between AffordanceNet (70%) and GraspNet (32%) could be driven by a few trials; for example, the wok success rates are 70% vs 60%, within sampling error. The authors should report confidence intervals (e.g., Wilson interval) and ideally increase the number of trials, or temper the claim of superior open-world generalization. Table 8 ablations are also limited to five tasks and should be accompanied by uncertainty estimates.
  4. [§5.1, Table 5] The ablation in Table 5 cumulatively adds data sources, but the final row 'Ours' includes additional task-specific modifications (system prompt and <AFF> token), making it impossible to attribute the improvement to the dataset alone. The authors should present a proper ablation that isolates the effect of each component: the system prompt, the <AFF> token, and each data source independently, with the same training budget and evaluation protocol.
minor comments (5)
  1. [§5.1, Table 4] The caption of Table 4 contains a typo: 'categoty' should be 'category'. The related work section also refers to 'HANDANL' instead of 'HANDAL', and Table 7's caption has 'A verage' with an extra space.
  2. [§4.2, Eq. (1)] Equation (1) is missing a word: 'd is the along the axis' should be 'd is the depth along the axis'. Also, using K and T as 4×4 matrices is nonstandard; clarifying the coordinate convention would improve reproducibility.
  3. [§3.3, Tables 12-13] The GPT-4 prompts in Tables 12 and 13 contain inconsistencies: the system prompt for easy instructions requires both <1> and <2> to include the category name, but the example for 'screwdriver' generates <1> without the name; the authors should report the exact prompts used and ensure they match the described protocol.
  4. [§3.1, Table 2] The paper claims 180 categories, but the category lists in Tables 2 and 11 sum to fewer than 180 (HANDAL 17, Open-X ~124, GraspNet 32, EgoObjects 74, RLBench 10). Clarify whether the count refers to unique categories after merging or to something else.
  5. [Supplementary, Fig. 9] The supplementary states that 'the highlighted words are category names at most times,' but no highlighting appears in Figure 9; consider adding visual cues or clarifying the instruction generation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark and model claims rest on external zero-shot and real-robot evaluations; auto-labeling overlap is a quality risk, not a circular step.

full rationale

The paper's central contribution is a dataset and a trained model rather than a mathematical derivation, so the usual circularity patterns (self-definitional identities, fitted parameters renamed as predictions, load-bearing self-citations) do not apply in an obvious way. I checked the claimed derivation chain: the annotation pipeline in Sec. 3.2 and Table 11 uses existing masks, SAM2, Florence2+SAM2, VLPart+SAM2, and human annotation to produce training labels, but no evaluation metric is defined as the same function of those labels by construction. The zero-shot generalization claims are supported by GraspNet-novel categories, the 3DOI unseen-domain set, real-robot grasping in an unseen scene, and RLBench simulation tasks, all of which lie outside the training sources. HANDAL and GraspNet-seen are explicitly marked as seen validation sets in Table 3, so evaluating on them is a standard supervised benchmark rather than a disguised prediction. The overlap between the VLPart+SAM2 baseline and the use of VLPart+SAM2 to label some EgoObjects handle categories is a legitimate correctness and comparison concern, but it does not make the model's predictions statistically forced on the HANDAL evaluation, whose masks come from the original HANDAL annotations. The absence of inter-annotator agreement or per-tool error analysis is a data-quality limitation, not an instance of the paper's conclusions reducing to its inputs. No load-bearing premise is justified only by a self-citation, and no equation in the paper defines an output as a fitted input. Therefore, no specific circular step is established.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central contribution is a labeled benchmark, so the ledger is dominated by annotation-quality assumptions rather than fitted constants. The only hand-chosen numeric input identified is the data sampling ratio during training. The load-bearing premises are that pseudo-labels are accurate, that the full-object-versus-handle annotation convention matches how robots should grasp, and that GPT-4 instructions are representative of human commands. No new physical entities are introduced.

free parameters (1)
  • Data sampling ratios = 3:1:1:1:9:3 (Table 10)
    The ratios of semantic segmentation, referring segmentation, reasoning segmentation, VQA, affordance segmentation, and reasoning-based affordance segmentation data during training are hand-chosen and affect model performance. No sensitivity analysis is provided.
assumptions (3)
  • domain assumption The graspable affordance of an object is fully represented by either the full object mask for handle-free objects or the handle region for handled objects.
    Section 3.2 defines this labeling rule, which determines all masks in RAGNet. It is not validated against human grasp-preference data or grasp success rates.
  • domain assumption Automatic annotations from SAM2, Florence2 plus SAM2, and VLPart plus SAM2 are accurate enough to serve as training ground truth.
    Table 11 and section 3.2 rely on these tools for a large share of masks, with no inter-annotator agreement or quality audit reported.
  • domain assumption GPT-4 generated reasoning instructions are valid proxies for natural human instructions for grasping.
    Section 3.3 and Tables 12-13 use GPT-4 to generate all easy and hard instructions, but no human study evaluates their naturalness, ambiguity, or coverage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping." pith.science (2026). https://pith.science/paper/LCHEUUNB

@misc{pith2026250723734,
  author       = {Pith},
  title        = {Pith review of: RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCHEUUNB}},
  note         = {Machine review of arXiv:2507.23734}
}
read the original abstract

General robotic grasping systems require accurate object affordance perception in diverse open-world scenarios following human instructions. However, current studies suffer from the problem of lacking reasoning-based large-scale affordance prediction data, leading to considerable concern about open-world effectiveness. To address this limitation, we build a large-scale grasping-oriented affordance segmentation benchmark with human-like instructions, named RAGNet. It contains 273k images, 180 categories, and 26k reasoning instructions. The images cover diverse embodied data domains, such as wild, robot, ego-centric, and even simulation data. They are carefully annotated with an affordance map, while the difficulty of language instructions is largely increased by removing their category name and only providing functional descriptions. Furthermore, we propose a comprehensive affordance-based grasping framework, named AffordanceNet, which consists of a VLM pre-trained on our massive affordance data and a grasping network that conditions an affordance map to grasp the target. Extensive experiments on affordance segmentation benchmarks and real-robot manipulation tasks show that our model has a powerful open-world generalization ability. Our data and code is available at https://github.com/wudongming97/AffordanceNet.

Figures

Figures reproduced from arXiv: 2507.23734 by the authors.

Figure 1
Figure 1. Large-scale benchmark for reasoning-based affordance segmentation, which sources from various embodied domains: wild, robot, ego-centric indoor, and simulation. By leveraging the extensive high-quality dataset for training, our model AffordanceNet exhibits remarkable open-world generalization capabilities, steering further towards robust general-purpose object grasping. Abstract General robotic grasping systems requ… view at source ↗
Figure 2
Figure 2. Overview of our data annotation pipeline and manipulation model. We collect data from several public datasets, including HANDAL, GraspNet, Open-X, etc. A variety of models and manual annotation are utilized to annotate affordance masks. Subsequently, we refine textual instructions using templates and GPT4 to emulate human-like commands. For grasp operations, the VLM model is employed to identify affordance regions, … view at source ↗
Figure 3
Figure 3. Easy (left) v.s. Hard (right) Reasoning Instruction. The hard version has no category name itself. detailed affordance masks for grasping. In these cases, we utilize the original masks as our affordance annotations without further refinement. ❷ SAM2: For objects that lack handles, SAM2 [50] can still be utilized to generate a mask, when only the ground-truth bounding box is available (like EgoObjects [68]). ❸ Floren… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Affordance segmentation from our AffordanceNet. Even though they source from various data sources, such as wild, robot, ego-centric and simulation, our model can accurately capture their affordance region. More visualizations are included in supplementary. I need a for…
Figure 5
Figure 5. Figure 5: Reasoning-based affordance segmentation from our AffordanceNet. The left examples represent easy reasoning-based instructions with referent name, while the right are hard instructions that include object function or intention rather than the name itself. performs all o…
Figure 6
Figure 6. Figure 6: Object grasping results from our AffordanceNet on robot arm UR5. The instructions are “I need a screwdriver for repairing”, “Can you hand me the wok, please?”, “Give me the circle”, “Please hand me a computer mouse”, respectively. Method Can Pen Screwdriver Hammer Wok …
Figure 7
Figure 7. Figure 7: Object grasping results from our AffordanceNet on RLBench. The instruction of the top video is “Open the top drawer”, and the bottom one refers to “Close the green jar”. Data Semantic Seg Referring Seg Reasoning-based Seg VQA Affordance Seg Reasoning-based Affordance S…
Figure 8
Figure 8. Figure 8: More affordance segmentation examples from our RAGNet. It covers various data sources, like robot, wild, and ego-centric domains [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: More reasoning-based affordance segmentation examples from our RAGNet. It includes two types of reasoning instructions: easy instructions and hard instructions [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: More experiment results from our model. We use template-based, easy reasoning-based, and hard reasoning-based instruc￾tions, respectively [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 58 canonical work pages

  1. [1]

    CVGIP: Image Understanding, 1994

    Function-based generic recognition for multiple object cate- gories. CVGIP: Image Understanding, 1994. 3

    work page 1994

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Affordances from human videos as a versa- tile representation for robotics

    Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versa- tile representation for robotics. In CVPR, 2023. 1

    work page 2023

  4. [4]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2

    arXiv 2022

  5. [5]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 1

    work page 2018

  6. [6]

    Affordance grounding from demonstration video to target image

    Joya Chen, Difei Gao, Kevin Qinghong Lin, and Mike Zheng Shou. Affordance grounding from demonstration video to target image. In CVPR, 2023. 3

    work page 2023

  7. [7]

    Learning to act properly: Predicting and explaining affordances from images

    Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. Learning to act properly: Predicting and explaining affordances from images. In CVPR, 2018. 3

    work page 2018

  8. [8]

    Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and chal- lenges for epic-kitchens-100. IJCV, 2022. 2, 5

    work page 2022

Show all 75 references
  1. [9]

    Affordancenet: An end-to-end deep learning approach for object affordance detection

    Thanh-Toan Do, Anh Nguyen, and Ian Reid. Affordancenet: An end-to-end deep learning approach for object affordance detection. In ICRA, 2018. 3

    2018

  2. [10]

    Graspnet-1billion: A large-scale benchmark for general ob- ject grasping

    Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general ob- ject grasping. In CVPR, 2020. 4, 5, 7, 8, 1

    2020

  3. [11]

    Demo2vec: Reasoning object affordances from online videos

    Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph J Lim. Demo2vec: Reasoning object affordances from online videos. In CVPR, 2018. 3

    2018

  4. [12]

    Learning visual at- tributes

    Vittorio Ferrari and Andrew Zisserman. Learning visual at- tributes. NeurIPS, 2007. 3

    2007

  5. [13]

    The ecological approach to visual percep- tion: classic edition

    James J Gibson. The ecological approach to visual percep- tion: classic edition. Psychology press, 2014. 3

    2014

  6. [14]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 2

    2022

  7. [15]

    HAN- DAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions

    Andrew Guo, Bowen Wen, Jianhe Yuan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. HAN- DAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. In IROS, 2023. 2, 3, 4, 5, 1

    2023

  8. [16]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024. 1

    2024 arXiv

  9. [17]

    Manipvqa: Injecting robotic affordance and physi- cally grounded information into multi-modal large language models

    Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xi- aoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physi- cally grounded information into multi-modal large language models. In IROS, 2024. 2, 3, 5

    2024

  10. [18]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 1

    2023 arXiv

  11. [19]

    Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024. 1

    2024 arXiv

  12. [20]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Let- ters, 2020. 4, 8

    2020

  13. [21]

    Affordpose: A large-scale dataset of hand-object inter- actions with affordance-driven hand pose

    Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordpose: A large-scale dataset of hand-object inter- actions with affordance-driven hand pose. In ICCV, 2023. 1

    2023

  14. [22]

    Robo-abc: Affordance gener- alization beyond categories via semantic correspondence for robot manipulation

    Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Min- grun Jiang, and Huazhe Xu. Robo-abc: Affordance gener- alization beyond categories via semantic correspondence for robot manipulation. In ECCV, 2024. 3

    2024

  15. [23]

    Hotr: End-to-end human-object in- teraction detection with transformers

    Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object in- teraction detection with transformers. In CVPR, 2021. 1

    2021

  16. [24]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 5

    2023

  17. [25]

    Vi- sual object-action recognition: Inferring object affordances from human demonstration

    Hedvig Kjellstr ¨om, Javier Romero, and Danica Kragi ´c. Vi- sual object-action recognition: Inferring object affordances from human demonstration. Computer Vision and Image Un- derstanding, 2011. 3

    2011

  18. [26]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In CVPR, 2024. 2, 5, 6, 8, 1

    2024

  19. [27]

    Locate: Localize and transfer object parts for weakly super- vised affordance grounding

    Gen Li, Varun Jampani, Deqing Sun, and Laura Sevilla-Lara. Locate: Localize and transfer object parts for weakly super- vised affordance grounding. In CVPR, 2023. 3

    2023

  20. [28]

    One-shot open affordance learning with foundation models

    Gen Li, Deqing Sun, Laura Sevilla-Lara, and Varun Jampani. One-shot open affordance learning with foundation models. In CVPR, 2024. 2, 3

    2024

  21. [29]

    Learning precise affordances from egocentric videos for robotic manipulation

    Gen Li, Nikolaos Tsagkas, Jifei Song, Ruaridh Mon- Williams, Sethu Vijayakumar, Kun Shao, and Laura Sevilla- Lara. Learning precise affordances from egocentric videos for robotic manipulation. arXiv preprint arXiv:2408.10123,

  22. [30]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  23. [31]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yux- ing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In CVPR,

  24. [32]

    Laso: Language-guided affordance seg- mentation on 3d object

    Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language-guided affordance seg- mentation on 3d object. In CVPR, 2024. 3

    2024

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024. 2, 1

    2024

  26. [34]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6

    2023 arXiv

  27. [35]

    Learning to seg- ment affordances

    Timo Luddecke and Florentin Worgotter. Learning to seg- ment affordances. In ICCV Workshops, 2017. 3

    2017

  28. [36]

    Learning affordance grounding from exocen- tric images

    Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocen- tric images. In CVPR, 2022. 2, 3

    2022

  29. [37]

    Leverage interactive affinity for affordance learning

    Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Leverage interactive affinity for affordance learning. In CVPR, 2023. 3

    2023

  30. [38]

    Affordance detection of tool parts from geomet- ric features

    Austin Myers, Ching L Teo, Cornelia Ferm ¨uller, and Yiannis Aloimonos. Affordance detection of tool parts from geomet- ric features. In ICRA, 2015. 2, 3

    2015

  31. [39]

    Pivot: Iterative visual prompt- ing elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompt- ing elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024. 1

    2024 arXiv

  32. [40]

    Object-based affordances detection with convolutional neural networks and dense conditional random fields

    Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In IROS, 2017. 3

    2017

  33. [41]

    Llarva: Vision-action instruction tuning enhances robot learning

    Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. arXiv preprint arXiv:2406.11815, 2024. 8

    2024 arXiv

  34. [42]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023. 2, 4, 1

    arXiv

  35. [43]

    Understanding 3d object interaction from a single image

    Shengyi Qian and David F Fouhey. Understanding 3d object interaction from a single image. In ICCV, 2023. 2, 3, 5

    2023

  36. [44]

    Understanding 3d object articulation in in- ternet videos

    Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F Fouhey. Understanding 3d object articulation in in- ternet videos. In CVPR, 2022. 5

    2022

  37. [45]

    Affordancellm: Grounding affordance from vision language models

    Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. Affordancellm: Grounding affordance from vision language models. In CVPR, 2024. 2, 3, 5

    2024

  38. [46]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 5

    2021

  39. [47]

    Paco: Parts and attributes of common objects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Mar- quez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In CVPR, 2023. 2, 1

    2023

  40. [48]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In CVPR,

  41. [49]

    Language embedded radiance fields for zero-shot task- oriented grasping

    Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Gold- berg. Language embedded radiance fields for zero-shot task- oriented grasping. In CoRL, 2023. 3

    2023

  42. [50]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 4, 6, 1

    2024 arXiv

  43. [51]

    Weakly supervised affordance detection

    Johann Sawatzky, Abhilash Srikantha, and Juergen Gall. Weakly supervised affordance detection. In CVPR, 2017. 3

    2017

  44. [52]

    Understanding human hands in contact at internet scale

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In CVPR, 2020. 1

    2020

  45. [53]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In CoRL, 2022. 1

    2022

  46. [54]

    Going denser with open-vocabulary part segmentation

    Peize Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xie, and Zhicheng Yan. Going denser with open-vocabulary part segmentation. In ICCV, 2023. 2, 4, 6, 8, 1

    2023

  47. [55]

    Grasp-anything: Large-scale grasp dataset from foundation models

    An Dinh Vuong, Minh Nhat Vu, Hieu Le, Baoru Huang, Binh Huynh, Thieu V o, Andreas Kugi, and Anh Nguyen. Grasp-anything: Large-scale grasp dataset from foundation models. arXiv preprint arXiv:2309.09818, 2023. 3

    2023 arXiv

  48. [56]

    Language- driven grasp detection

    An Dinh Vuong, Minh Nhat Vu, Baoru Huang, Nghia Nguyen, Hieu Le, Thieu V o, and Anh Nguyen. Language- driven grasp detection. In CVPR, 2024. 3

    2024

  49. [57]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, An- dre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In CoRL, 2023. 2

    2023

  50. [58]

    An interactive navigation method with effect-oriented affordance

    Xiaohan Wang, Yuehu Liu, Xinhang Song, Yuyi Liu, Six- ian Zhang, and Shuqiang Jiang. An interactive navigation method with effect-oriented affordance. In CVPR, 2024. 1

    2024

  51. [59]

    Move as you say interact as you can: Language-guided human motion generation with scene af- fordance

    Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: Language-guided human motion generation with scene af- fordance. In CVPR, 2024. 1

    2024

  52. [60]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In CVPR, 2024. 4, 6, 1

    2024

  53. [61]

    Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation

    Ran Xu, Yan Shen, Xiaoqi Li, Ruihai Wu, and Hao Dong. Naturalvlm: Leveraging fine-grained natural language for affordance-guided visual manipulation. arXiv preprint arXiv:2403.08355, 2024. 2, 3

    2024 arXiv

  54. [62]

    Grounding 3d object affordance from 2d interactions in images

    Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d object affordance from 2d interactions in images. In ICCV, 2023. 3

    2023

  55. [63]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, 2016. 1

    2016

  56. [64]

    Uniaff: A unified representation of affor- dances for tool usage and articulation with vision-language models

    Qiaojun Yu, Siyuan Huang, Xibin Yuan, Zhengkai Jiang, Ce Hao, Xin Li, Haonan Chang, Junbo Wang, Liu Liu, Hong- sheng Li, et al. Uniaff: A unified representation of affor- dances for tool usage and articulation with vision-language models. arXiv preprint arXiv:2409.20551, 2024. 2, 3

    2024 arXiv

  57. [65]

    Taskonomy: Disentangling task transfer learning

    Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 5

    2018

  58. [66]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023. 5

    2023

  59. [67]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 1

    2017

  60. [68]

    Egoobjects: A large-scale egocentric dataset for fine-grained object understanding

    Chenchen Zhu, Fanyi Xiao, Andr ´es Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri, Sean Culatana, Roshan Sumbaly, and Zhicheng Yan. Egoobjects: A large-scale egocentric dataset for fine-grained object understanding. In ICCV, 2023. 4, 1 RAGNet: Large-scale Reasoning-based ...

    2023

  61. [69]

    From these data, we emphasize grasping-oriented objects, encompassing both those with handles and those without

    Details of Data Annotation As the original data sources, such as HANDAL [15], Open- X [42], EgoObjects [68], GraspNet [10], provide original annotation information (e.g., ground-truth boxes or masks), we make full use of them for minimal human intervention. From these data, we...

  62. [70]

    Affordance Annotation Examples Since our benchmark RAGNet includes a significant num- ber of grasping-oriented objects from various domains (like robot, wild, and ego-centric domains), we highlight this as- pect by showcasing additional examples of affordance seg- mentation an...

  63. [71]

    microwave, open the door

    Reasoning-based Affordance Examples More reasoning-based affordance segmentation examples are shown in Fig. 10. It contains two types of instructions, easy instructions and hard instructions. As seen, the easy instructions include the target object name, while the hard ones on...

  64. [72]

    Implementation Details of AffordanceNet Beyond our reasoning-based affordance segmentation data, we also incorporate a variety of generic segmentation datasets into our training. This diverse generic set includes data for semantic segmentation (e.g., ADE20k [67], COCO- Stuff [...

  65. [73]

    More Results on Visual Affordance We provide more visualization results of affordance seg- mentation from our AffordanceVLM model in Fig. 10. The testing images are selected from multiple validation sets, such as GraspNet Novel, 3DOI, and HANDAL. We employ template-based, easy...

  66. [74]

    Open the top drawer

    More Results on Real Robot Beyond the evaluation tasks in our main manuscript, such as grasping can, pen, screwdriver, hammer, and wok, we also evaluate the open-world generalization capabilities of our model by utilizing a broader range of instructions en- compassing various ...

  67. [75]

    open the top drawer,

    More Results on RLBench We present several visualization results from the simulation task RLBench in Figure 7. The top video demonstrates the task “open the top drawer,” while the bottom video illus- trates “close the green jar”. As shown, our model success- fully completes bo...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.