Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A task-conditioned affordance model trained with zero manual labels—using vision-language pseudo-labels on rendered 3D objects—predicts pixel-level affordances that transfer to real scenes and, as an observation space, lets a…

desk verdict Solid, genuinely useful paper with a scalable VLM annotation pipeline; the zero-shot claims are softer than stated due to AGD20K rephrasing and indirect label validation. read the letter →

arxiv 2506.09284 v2 pith:WP2RZHSE submitted 2025-06-10 cs.RO cs.AIcs.CV

Yihe Tang , Wenlong Huang , Yingke Wang , Chengshu Li , Roy Yuan , Ruohan Zhang , Jiajun Wu , Li Fei-Fei This is my paper
classification cs.ROcs.AIcs.CV MSC 68T4068T45
keywords affordanceroboticmanipulationfoundationmodelsunsupervisedannotationtask-conditionedpredictionDINOv2FiLMconditioningimitationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UAD claims that fine-grained pixel-level affordances—knowing exactly where on an unseen object to grasp, pour, insert, or open—can be learned without any human annotation by distilling knowledge from off-the-shelf foundation models. The authors render 3D objects, cluster DINOv2 features into functional regions, and ask a vision-language model to propose task instructions and match each instruction to a region; the resulting pseudo-labels train a lightweight language-conditioned decoder on top of frozen DINOv2 features. Trained only on single objects rendered in simulation, the model predicts continuous per-pixel affordance maps zero-shot on real robot scenes and on human-activity benchmarks, and these maps agree with human labels better than CLIP-based or segmentation baselines. Fed into a multi-view transformer policy trained on ten demonstrations per task, the affordance maps support generalization to unseen poses, instances, categories, and even new instructions, both in simulation and on a real robot.

What carries the argument

The load-bearing mechanism is the extract-and-distill loop. For extraction, multi-view DINOv2 features are fused into a 3D feature field, PCA-reduced and clustered into functional regions; a vision-language model then generates instruction-region pairs, and each pair is converted into a continuous affordance map by computing cosine similarity between the cluster's mean feature and every point's feature, projected back to pixels. For distillation, a lightweight decoder of three FiLM layers—feature-wise linear modulation that scales and shifts channels according to the language embedding—is trained with binary cross-entropy on frozen DINOv2 features. The continuous, per-pixel nature of the resulting map is what the downstream policy consumes as its observation space, and the language conditioning is what lets a single frozen backbone answer many open-ended instructions.

What would settle it

Take the rendered objects and instructions used to train UAD, collect human affordance masks for them (as the paper already does for its evaluation sets), and compute per-pair agreement; if a substantial fraction of object-instruction pairs show low or chance-level agreement, the pseudo-label premise fails. A second check would retrain the downstream 10-demonstration policies with affordance maps from randomly shuffled instruction-region assignments: if success rates stay unchanged, affordance quality is not the mechanism driving the reported generalization.

Watch

Extended reading notes

Core claim

The central claim is that a task-conditioned affordance model can be distilled from the complementary strengths of a self-supervised vision model and a vision-language model, replacing manual affordance datasets. Concretely, DINOv2 patch features from multiple rendered views are fused into a 3D feature field and clustered into semantic regions; GPT-4o, prompted with the overlaid clusters, proposes relevant instructions and picks the region each instruction refers to; the discrete choice is turned into a continuous map by cosine similarity to the mean feature of the chosen region. The map trains a three-layer FiLM decoder on frozen DINOv2 features. The paper reports AUC at least 0.92 on rendered evaluation sets spanning training data, novel instances, novel categories, and novel instructions; an AUC of 0.840 on DROID real-robot scenes versus 0.836 for OpenSeeD and 0.500 for CLIP; competitive scores on AGD20K's unseen split; and real-world task success averaging 73 percent for policies trained with ten demonstrations.

Load-bearing premise

Everything rests on the assumption that GPT-4o's region-to-instruction mappings on DINOv2 clusters of synthetic single-object renders are accurate affordance labels and that these labels transfer to multi-object real-world scenes; the pseudo-labels are never validated directly against human labels on the training distribution.

Editorial extensions

If this is right

  • Affordance labels can be produced at scale without human effort: any 3D object render can be turned into many instruction-affordance map pairs, so the bottleneck shifts from annotation cost to asset diversity.
  • A single affordance model trained only on rendered single objects can serve as a drop-in observation space for real-world manipulation policies, replacing task-agnostic visual features with task-conditioned pixel attention.
  • Generalization to novel instructions is achievable with very few demonstrations, because the language-conditioned decoder reuses frozen visual features instead of learning new visual semantics per task.
  • The same affordance representation supports precise 6-DoF actions such as pen insertion, drawer opening, and pouring, indicating that per-pixel continuous maps carry the spatial detail needed for contact-level control.
  • Because the decoder is small and the backbone is frozen, updating UAD for new tasks requires re-labeling and re-distilling rather than retraining the vision encoder, making continual expansion cheap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pipeline's accuracy holds beyond the tested assets, the same extract-and-distill recipe could scale to large object corpora (the paper's Objaverse-XL case study is a step in that direction), effectively synthesizing an open-vocabulary affordance dataset from unlabeled 3D models alone.
  • A testable vulnerability follows from the pipeline's dependence on the vision-language model: measuring how much UAD's predictions and downstream success vary when the same clustering is labeled by different prompts, model versions, or temperature settings would quantify how much of the result is knowledge in the VLM versus knowledge in the frozen vision features.
  • Affordance-as-observation may generalize beyond keyframe imitation: because it is a dense map rather than a task-specific feature, it could plausibly condition reinforcement learning, diffusion policies, or trajectory optimizers, though the paper only demonstrates the imitation setting.
  • If DINOv2 clusters are driven more by texture than by function for some object classes, UAD's real-world transfer could degrade precisely on those classes; a texture-ablation study (rendering the same geometry with different materials) would separate functional from appearance-driven affordances.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes UAD (Unsupervised Affordance Distillation), a two-stage pipeline that first extracts pixel-level affordance annotations from frozen DINOv2 features and GPT-4o on single-object synthetic renders, then trains a lightweight task-conditioned FiLM decoder on top of frozen DINOv2 features. The affordance model is evaluated zero-shot on rendered-object sanity checks, the DROID real-world robot dataset, and the AGD20K human-activity affordance benchmark. The paper also integrates UAD as the observation space for an RVT-style imitation learning policy and reports generalization to unseen object instances, categories, and instruction variations in simulation and on three real-world tasks.

Significance. If the claims hold, UAD addresses a real bottleneck: scaling fine-grained, open-vocabulary affordance supervision without manual annotation. The paper's central idea of combining the complementary strengths of a self-supervised vision model (DINOv2) and a VLM (GPT-4o) is timely and well-motivated. The manuscript's strengths include a multi-stage evaluation (affordance prediction with MTurk validation, DROID, AGD20K, simulation policy learning with several baselines, and real-world policy demonstrations), the use of a frozen backbone with a lightweight trainable module, and a case study scaling the annotation pipeline to Objaverse-XL. The policy experiments with 10 demonstrations show a plausible path to few-shot manipulation generalization. However, several evaluation choices weaken the strength of the generalization claims, particularly the AGD20K protocol and the confounded instruction-change experiments.

major comments (4)
  1. [Table I and Appendix IV-A] The AGD20K comparison is not a clean zero-shot evaluation. The authors reformulate instructions to match the benchmark's labeling conventions (e.g., 'hit - axe -> handle of axe to hold during hitting') and introduce a new metric NSS-0.5 with a threshold of 0.5. Under the standard NSS threshold, UAD (1.092) is below LOCATE (1.157), so the statement that UAD 'outperforms LOCATE' relies entirely on the new metric. The instruction rewriting also means the model does not see the original AGD20K prompts. Please report results with the original action labels as-is, justify the new metric independently of the observed results, or soften the claim to 'competitive on SIM' rather than 'outperforming LOCATE.'
  2. [Section III-A and Section IV-A] The 'unsupervised' annotation claim depends on the quality of GPT-4o's region-to-instruction mapping, but this mapping is never directly scored against human labels on the training distribution. The MTurk evaluation in Section IV-A validates the trained model's predictions, not the generated labels used for training. Because the instructions in the evaluation pairs are also generated by GPT-4o, a systematic bias in instruction phrasing (e.g., always preferring a handle over a body when both are valid) would be shared between training and evaluation and would not surface as low AUC. A direct comparison between the automatically generated affordance maps and the MTurk masks on the same (image, instruction) pairs would isolate label-source quality. Please add this experiment or argue explicitly why the current AUC evidence is sufficient to rule out such shared VLM biases.
  3. [Section IV-B, Pouring and Insertion novel-instruction settings] The instruction-generalization evaluation is confounded with object and environment changes. For Pouring, the novel-instruction condition changes 'pour to bowl' to 'water plant', which replaces the target object (bowl vs. pot plant); for Insertion, 'insert pen' becomes 'insert carrot', changing the manipulated object from a pen to a carrot. As a result, the reported success rates under 'novel instruction' do not isolate the model's ability to condition on instruction wording from its ability to generalize to new objects and categories. Please add a condition in which the instruction is rephrased while the objects remain the same (e.g., 'pour to bowl' vs. 'fill the bowl'), or explicitly acknowledge this confound and temper the claim of generalizing to 'variations in task instructions.'
  4. [Section I contributions and Section IV-A] The contribution statement 'outperforms prior methods on existing benchmarks' is overstated relative to the reported numbers. On AGD20K, UAD has higher KLD (1.878 vs. 1.405) and lower standard NSS (1.092 vs. 1.157) than LOCATE; the only clearly superior standard metric is SIM. On DROID, the comparison is against CLIP and OpenSeeD, where OpenSeeD is a segmentation model rather than an affordance model, and the AUC margin over OpenSeeD is small (0.840 vs. 0.836). Please either provide a benchmark where UAD is clearly superior under standard metrics or revise the contribution statement to describe the results as competitive or as strong on specific metrics.
minor comments (5)
  1. [Section III-A] The sentence 'Post paper acceptance, we additionally conduct a case study of scaling to more diverse object database...' is inappropriate for a manuscript under review; it should be integrated into the main text or appendix as a normal part of the study, without the post-acceptance framing.
  2. [Figure 3 and Section IV-C] The real-world success rates are shown visually but not reported numerically in the text; please include the per-task success rates and the number of trials in the main text for reproducibility.
  3. [Table I] The NSS-0.5 metric is used to support a headline comparison but is only defined in the Appendix; please define it in the main text where it first appears.
  4. [Section IV-A] The evaluation set labeled 'training data' is ambiguous: please clarify whether these are image-instruction pairs seen during training or held-out pairs from the same distribution, since memorization would inflate AUC in the former case.
  5. [References] The reference list contains a formatting artifact in 'V oltron' and 'Affordance grounding'; please ensure all reference names are typeset correctly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; external human and benchmark evaluation grounds the central generalization claims.

full rationale

The paper's annotation pipeline defines the supervision target A as a cosine similarity between per-point DINOv2 features and the mean feature of a GPT-4o-selected region, and UAD trains a FiLM decoder on frozen DINOv2 features to predict A. This makes the target a function of the same feature space used as input, but the semantic region-to-instruction association is supplied by GPT-4o, not by the decoder, and the learned model is evaluated against independent human labels (MTurk on the four rendered evaluation sets and on DROID) and against the external AGD20K benchmark. These are external ground truths, not the paper's own fitted values, so the generalization claims are not forced by construction. The only self-citations in the method, D3fields [115] for multi-view feature fusion and GenDP [80] as related work, are implementation details and contextual observations rather than load-bearing assumptions or uniqueness arguments. The AGD20K instruction disambiguation is a dataset-convention fairness adjustment and does not relabel or retrain on the test set. The absence of direct human validation of GPT-4o labels on the training distribution is a validity risk, not a circularity, because held-out external evaluations partially mitigate it.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the reliability of auto-generated labels from GPT-4o and DINOv2, plus the transferability of those synthetic labels to real scenes. The free parameters listed are hand-chosen hyperparameters in the annotation pipeline that shape the training labels.

free parameters (4)
  • Affordance binarization threshold = 0.5
    Ground-truth affordance maps are thresholded at 0.5 to focus training on the most relevant regions (Appendix A), affecting the learned labels.
  • Minimum cluster count = 5
    Mean shift clustering is re-run with k-means if fewer than 5 clusters are found, a hand-chosen value that influences region granularity (Appendix A).
  • Number of render views = 14
    Each object is rendered from 14 views for feature fusion, a hyperparameter that affects the pointcloud feature field (Section III-A).
  • Gaussian blur kernel = 3
    Affordance maps are smoothed with a 3x3 Gaussian kernel after thresholding, affecting label boundaries (Appendix A).
assumptions (5)
  • domain assumption GPT-4o provides accurate task proposals and region associations from visual prompts of clustered object images.
    The entire annotation pipeline relies on the VLM correctly matching instructions to colored regions on rendered objects (Section III-A).
  • domain assumption DINOv2 features on rendered single objects form clusters that correspond to functional object parts useful for manipulation.
    The region proposal step uses DINOv2 embeddings as the basis for segmentation and cosine-similarity affordance scores (Section III-A).
  • domain assumption Affordances learned from single-object synthetic renders transfer to multi-object real-world scenes.
    The paper trains only on BEHAVIOR-1K single-object renders and evaluates on DROID and real-world multi-object scenes (Sections IV-A, IV-C).
  • ad hoc to paper The continuous affordance map computed as cosine similarity to a region's reference DINOv2 feature is a valid training target.
    The conversion from discrete VLM region selection to per-pixel scores uses cosine similarity in feature space, a choice specific to this method (Section III-A).
  • domain assumption Rendered 3D assets from BEHAVIOR-1K cover the distribution of target objects and instructions.
    The dataset includes 206 objects, 76 categories, 667 instructions; generalizing beyond this set is a central assumption (Section III-A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation." pith.science (2026). https://pith.science/paper/WP2RZHSE

@misc{pith2026250609284,
  author       = {Pith},
  title        = {Pith review of: UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WP2RZHSE}},
  note         = {Machine review of arXiv:2506.09284}
}
abstract

Understanding fine-grained object affordances is imperative for robots to manipulate objects in unstructured environments given open-ended task instructions. However, existing methods of visual affordance predictions often rely on manually annotated data or conditions only on a predefined set of tasks. We introduce UAD (Unsupervised Affordance Distillation), a method for distilling affordance knowledge from foundation models into a task-conditioned affordance model without any manual annotations. By leveraging the complementary strengths of large vision models and vision-language models, UAD automatically annotates a large-scale dataset with detailed $<$instruction, visual affordance$>$ pairs. Training only a lightweight task-conditioned decoder atop frozen features, UAD exhibits notable generalization to in-the-wild robotic scenes and to various human activities, despite only being trained on rendered objects in simulation. Using affordance provided by UAD as the observation space, we show an imitation learning policy that demonstrates promising generalization to unseen object instances, object categories, and even variations in task instructions after training on as few as 10 demonstrations. Project website: https://unsup-affordance.github.io/

Figures

Figures reproduced from arXiv: 2506.09284 by the authors.

Figure 1
Figure 1. Unsupervised Affordance Distillation (UAD) extracts affordance annotations from large pre-trained models and distills them into a task-conditioned affordance model, which is capable of predicting fine-grained affordance in open-world scenes with open-ended instructions, enabling diverse generalization properties in downstream policy learning. Abstract— Understanding fine-grained object affordances is imperative for … view at source ↗
Figure 2
Figure 2. Overview of Unsupervised Affordance Distillation (UAD). Using renderings of 3D objects, we first perform multi-view fusion of DINOv2 features and clustering to obtain fine-grained semantic regions of objects, which are then fed to VLM for proposing relevant tasks and corresponding regions (a). The extracted affordance is then distilled by training a language-conditioning FiLM atop frozen DINOv2 features (b). The lea… view at source ↗
Figure 3
Figure 3. Tasks for evaluating UAD. Left: tasks in simulation along with different generalization requirements. Right: tasks in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Task-conditioned affordance prediction results on the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Zero-shot generalization to affordance predictions in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Generalization performance of UAD in three sim [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Amazon MTurk Annotation Interface. affordance instead of per-image affordance as in other works, we post-process our model’s prediction by clipping per-pixel prediction to [ϵ, 1 − ϵ] before normalization. We format instructions as ”region to action the object” to match…
Figure 8
Figure 8. Figure 8: Comparison of ground truth fixation regions at [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning 3D Affordances for Blade Insertion in Cluttered Stowing

    cs.CV 2026-06 conditional novelty 6.5 of 10

    VulcanVoxel reconstructs blade occupancy with a 3D masked autoencoder, recovering multi-modal free-space affordances from unimodal warehouse stow data and raising top-5 coverage from 0.71 to 0.89.

  2. O$^3$Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation

    cs.RO 2025-09 conditional novelty 6.0 of 10

    A one-shot training regime with DINOv2-enriched point clouds and joint cross-attention predicts 3D object-to-object affordance maps that guide optimization-based robotic manipulation.

Reference graph

Works this paper leans on

135 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    To afford or not to afford: A new formalization of affordances toward affordance-based robot control,

    E. S ¸ahin, M. Cakmak, M. R. Do ˘gar, E. U ˘gur, and G. ¨Uc ¸oluk, “To afford or not to afford: A new formalization of affordances toward affordance-based robot control,” Adaptive Behavior , vol. 15, no. 4, pp. 447–472, 2007

    2007

  2. [2]

    Affordances from human videos as a versatile representation for robotics,

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak, “Affordances from human videos as a versatile representation for robotics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 778–13 790

    2023

  3. [3]

    Ditto in the house: Building articulation models of indoor scenes through interactive perception,

    C.-C. Hsu, Z. Jiang, and Y . Zhu, “Ditto in the house: Building articulation models of indoor scenes through interactive perception,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 3933–3939

    2023

  4. [4]

    Affordances in psychology, neuroscience, and robotics: A survey,

    L. Jamone, E. Ugur, A. Cangelosi, L. Fadiga, A. Bernardino, J. Piater, and J. Santos-Victor, “Affordances in psychology, neuroscience, and robotics: A survey,” IEEE Transactions on Cognitive and Develop- mental Systems, vol. 10, no. 1, pp. 4–25, 2016

    2016

  5. [5]

    A brief review of affordance in robotic manipulation research,

    N. Yamanobe, W. Wan, I. G. Ramirez-Alpizar, D. Petit, T. Tsuji, S. Akizuki, M. Hashimoto, K. Nagata, and K. Harada, “A brief review of affordance in robotic manipulation research,” Advanced Robotics, vol. 31, no. 19-20, pp. 1086–1101, 2017

    2017

  6. [6]

    Visual affordance and function understanding: A survey,

    M. Hassanin, S. Khan, and M. Tahtali, “Visual affordance and function understanding: A survey,”ACM Computing Surveys (CSUR), vol. 54, no. 3, pp. 1–35, 2021

    2021

  7. [7]

    A survey of visual affordance recognition based on deep learning,

    D. Chen, D. Kong, J. Li, S. Wang, and B. Yin, “A survey of visual affordance recognition based on deep learning,” IEEE Transactions on Big Data , 2023

    2023

  8. [8]

    A survey of semantic reasoning frameworks for robotic systems,

    W. Liu, A. Daruna, M. Patel, K. Ramachandruni, and S. Chernova, “A survey of semantic reasoning frameworks for robotic systems,” Robotics and Autonomous Systems , vol. 159, p. 104294, 2023

    2023

Show all 135 references
  1. [9]

    Recent advances of deep robotic affordance learning: a reinforcement learning perspective,

    X. Yang, Z. Ji, J. Wu, and Y .-K. Lai, “Recent advances of deep robotic affordance learning: a reinforcement learning perspective,” IEEE Transactions on Cognitive and Developmental Systems, vol. 15, no. 3, pp. 1139–1149, 2023

    2023

  2. [10]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” arXiv, 2023

    2023

  3. [11]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  4. [12]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  5. [13]

    Emerging properties in self-supervised vision trans- formers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision trans- formers,” in Proceedings of the International Conference on Com- puter Vision (ICCV) , 2021

    2021

  6. [14]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  7. [15]

    Grounded affordance from exocentric view,

    H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Grounded affordance from exocentric view,” arXiv preprint arXiv:2208.13196 , 2022

    2022 arXiv

  8. [16]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” in Robotics: Science and Systems , 2024

    2024

  9. [17]

    Learning state representations with robotic priors,

    R. Jonschkowski and O. Brock, “Learning state representations with robotic priors,” Autonomous Robots, vol. 39, pp. 407–428, 2015

    2015

  10. [18]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763

    2021

  11. [19]

    The theory of affordances,

    J. J. Gibson, “The theory of affordances,” The Ecological Approach to Visual Perception, 1977

    1977

  12. [20]

    Norman, The design of everyday things: Revised and expanded edition

    D. Norman, The design of everyday things: Revised and expanded edition. Basic books, 2013

    2013

  13. [21]

    Contrastively learning visual attention as affordance cues from demonstrations for robotic grasping,

    Y . Zha, S. Bhambri, and L. Guan, “Contrastively learning visual attention as affordance cues from demonstrations for robotic grasping,” CoRR, vol. abs/2104.00878, 2021. [Online]. Available: https://arxiv.org/abs/2104.00878

    2021 arXiv

  14. [22]

    Self-supervised visual descriptor learning for dense correspondence,

    T. Schmidt, R. Newcombe, and D. Fox, “Self-supervised visual descriptor learning for dense correspondence,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 420–427, 2016

    2016

  15. [23]

    Dense object nets: Learning dense visual object descriptors by and for robotic manipu- lation,

    P. R. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learning dense visual object descriptors by and for robotic manipu- lation,” arXiv preprint arXiv:1806.08756 , 2018

    2018 arXiv

  16. [24]

    kpam: Keypoint affordances for category-level robotic manipulation,

    L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “kpam: Keypoint affordances for category-level robotic manipulation,” in The Interna- tional Symposium of Robotics Research . Springer, 2019, pp. 132– 157

    2019

  17. [25]

    Unsupervised learning of object key- points for perception and control,

    T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V . Mnih, “Unsupervised learning of object key- points for perception and control,” Advances in neural information processing systems, vol. 32, 2019

    2019

  18. [26]

    Keto: Learning keypoint representations for tool manipulation,

    Z. Qin, K. Fang, Y . Zhu, L. Fei-Fei, and S. Savarese, “Keto: Learning keypoint representations for tool manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 7278–7285

    2020

  19. [27]

    Learning rope manipulation policies using dense object descriptors trained on synthetic depth data,

    P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg, “Learning rope manipulation policies using dense object descriptors trained on synthetic depth data,” in 2020 IEEE International Conference on Robotics and Automati...

    2020

  20. [28]

    Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning,

    L. Manuelli, Y . Li, P. Florence, and R. Tedrake, “Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning,” arXiv preprint arXiv:2009.05085 , 2020

    2009 arXiv

  21. [29]

    Unsupervised learning of visual 3d keypoints for control,

    B. Chen, P. Abbeel, and D. Pathak, “Unsupervised learning of visual 3d keypoints for control,” in International Conference on Machine Learning. PMLR, 2021, pp. 1539–1549

    2021

  22. [30]

    Neural descriptor fields: Se (3)- equivariant object representations for manipulation,

    A. Simeonov, Y . Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V . Sitzmann, “Neural descriptor fields: Se (3)- equivariant object representations for manipulation,” in 2022 Inter- national Conference on Robotics and Automation (ICRA) . IEEE, 2022, pp. 6394–6400

    2022

  23. [31]

    Se (3)-equivariant relational rearrangement with neural descriptor fields,

    A. Simeonov, Y . Du, Y .-C. Lin, A. R. Garcia, L. P. Kaelbling, T. Lozano-P ´erez, and P. Agrawal, “Se (3)-equivariant relational rearrangement with neural descriptor fields,” in Conference on Robot Learning. PMLR, 2023, pp. 835–846

    2023

  24. [32]

    Robotap: Tracking arbitrary points for few-shot visual imitation,

    M. Vecerik, C. Doersch, Y . Yang, T. Davchev, Y . Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz, “Robotap: Tracking arbitrary points for few-shot visual imitation,” arXiv preprint arXiv:2308.15975, 2023

    2023 arXiv

  25. [33]

    Local neural descriptor fields: Locally conditioned object represen- tations for manipulation,

    E. Chun, Y . Du, A. Simeonov, T. Lozano-Perez, and L. Kaelbling, “Local neural descriptor fields: Locally conditioned object represen- tations for manipulation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 1830–1836

    2023

  26. [34]

    Any-point trajectory modeling for policy learning,

    C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y . Gao, and P. Abbeel, “Any-point trajectory modeling for policy learning,” arXiv preprint arXiv:2401.00025, 2023

    2023 arXiv

  27. [35]

    Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation,

    H. Bharadhwaj, R. Mottaghi, A. Gupta, and S. Tulsiani, “Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation,” 2024

    2024

  28. [36]

    An affordance keypoint detection network for robot manipulation,

    R. Xu, F.-J. Chu, C. Tang, W. Liu, and P. A. Vela, “An affordance keypoint detection network for robot manipulation,” IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 2870–2877, 2021

    2021

  29. [37]

    Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation,

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation,” arXiv preprint arXiv:2409.01652 , 2024

    2024 arXiv

  30. [38]

    Affordance detection of tool parts from geometric features,

    A. Myers, C. L. Teo, C. Ferm ¨uller, and Y . Aloimonos, “Affordance detection of tool parts from geometric features,” in 2015 IEEE In- ternational Conference on Robotics and Automation (ICRA) . IEEE, 2015, pp. 1374–1381

    2015

  31. [39]

    Affordance detection for task-specific grasping using deep learning,

    M. Kokic, J. A. Stork, J. A. Haustein, and D. Kragic, “Affordance detection for task-specific grasping using deep learning,” in 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). IEEE, 2017, pp. 91–98

    2017

  32. [40]

    Object-based affordances detection with convolutional neural net- works and dense conditional random fields,

    A. Nguyen, D. Kanoulas, D. G. Caldwell, and N. G. Tsagarakis, “Object-based affordances detection with convolutional neural net- works and dense conditional random fields,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 5908–5915

    2017

  33. [41]

    Affordancenet: An end-to-end deep learning approach for object affordance detection,

    T.-T. Do, A. Nguyen, and I. Reid, “Affordancenet: An end-to-end deep learning approach for object affordance detection,” in 2018 IEEE international conference on robotics and automation (ICRA) . IEEE, 2018, pp. 5882–5889

    2018

  34. [42]

    Toward affordance detection and ranking on novel objects for real-world robotic manip- ulation,

    F.-J. Chu, R. Xu, L. Seguin, and P. A. Vela, “Toward affordance detection and ranking on novel objects for real-world robotic manip- ulation,” IEEE Robotics and Automation Letters , vol. 4, no. 4, pp. 4070–4077, 2019

    2019

  35. [43]

    Learning dexterous grasping with object-centric visual affordances,

    P. Mandikal and K. Grauman, “Learning dexterous grasping with object-centric visual affordances,” in 2021 IEEE international con- ference on robotics and automation (ICRA) . IEEE, 2021, pp. 6169– 6176

    2021

  36. [44]

    Affordance learning for end-to-end visuomotor robot control,

    A. H ¨am¨al¨ainen, K. Arndt, A. Ghadirzadeh, and V . Kyrki, “Affordance learning for end-to-end visuomotor robot control,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019, pp. 1781–1788

    2019

  37. [45]

    Affordance learning from play for sample-efficient policy learning,

    J. Borja-Diaz, O. Mees, G. Kalweit, L. Hermann, J. Boedecker, and W. Burgard, “Affordance learning from play for sample-efficient policy learning,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 6372–6378

    2022

  38. [46]

    Visual affordance pre- diction for guiding robot exploration,

    H. Bharadhwaj, A. Gupta, and S. Tulsiani, “Visual affordance pre- diction for guiding robot exploration,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 3029–3036

    2023

  39. [47]

    Hrp: Human af- fordances for robotic pre-training,

    M. K. Srirama, S. Dasari, S. Bahl, and A. Gupta, “Hrp: Human af- fordances for robotic pre-training,” arXiv preprint arXiv:2407.18911, 2024

    2024 arXiv

  40. [48]

    Learning relational affordance models for robots in multi-object manipulation tasks,

    B. Moldovan, P. Moreno, M. Van Otterlo, J. Santos-Victor, and L. De Raedt, “Learning relational affordance models for robots in multi-object manipulation tasks,” in 2012 ieee international confer- ence on robotics and automation . IEEE, 2012, pp. 4373–4378

    2012

  41. [49]

    Affordance-based grasping and manipulation in real world applications,

    C. Pohl, K. Hitzler, R. Grimm, A. Zea, U. D. Hanebeck, and T. Asfour, “Affordance-based grasping and manipulation in real world applications,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 9569–9576

    2020

  42. [50]

    Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross- domain image matching,

    A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo et al., “Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross- domain image matching,” The International Journal of Robotics Research, vol. 41...

    2022

  43. [51]

    Metagrasp: Data efficient grasping by affordance interpreter network,

    J. Cai, H. Cheng, Z. Zhang, and J. Su, “Metagrasp: Data efficient grasping by affordance interpreter network,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 4960–4966

    2019

  44. [52]

    Learning affordance space in physical world for vision-based robotic object manipulation,

    H. Wu, Z. Zhang, H. Cheng, K. Yang, J. Liu, and Z. Guo, “Learning affordance space in physical world for vision-based robotic object manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2020, pp. 4652–4658

    2020

  45. [53]

    Learning multi- object dense descriptor for autonomous goal-conditioned grasping,

    S. Yang, W. Zhang, R. Song, J. Cheng, and Y . Li, “Learning multi- object dense descriptor for autonomous goal-conditioned grasping,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 4109–4116, 2021

    2021

  46. [54]

    Deep affordance foresight: Planning through what can be done in the future,

    D. Xu, A. Mandlekar, R. Mart ´ın-Mart´ın, Y . Zhu, S. Savarese, and L. Fei-Fei, “Deep affordance foresight: Planning through what can be done in the future,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 6206–6213

    2021

  47. [55]

    Where2act: From pixels to actions for articulated 3d objects,

    K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani, “Where2act: From pixels to actions for articulated 3d objects,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6813–6823

    2021

  48. [56]

    Do as i can and not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine,...

    2022 arXiv

  49. [57]

    Grounding language with visual affordances over unstructured data,

    O. Mees, J. Borja-Diaz, and W. Burgard, “Grounding language with visual affordances over unstructured data,” arXiv preprint arXiv:2210.01911, 2022

    2022 arXiv

  50. [58]

    Learning generalizable dexterous manipulation from human grasp affordance,

    Y .-H. Wu, J. Wang, and X. Wang, “Learning generalizable dexterous manipulation from human grasp affordance,” in Conference on Robot Learning. PMLR, 2023, pp. 618–629

    2023

  51. [59]

    Learning affordance grounding from exocentric images,

    H. Luo, W. Zhai, J. Zhang, Y . Cao, and D. Tao, “Learning affordance grounding from exocentric images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022

    2022

  52. [60]

    Locate: Localize and transfer object parts for weakly supervised affordance grounding,

    G. Li, V . Jampani, D. Sun, and L. Sevilla-Lara, “Locate: Localize and transfer object parts for weakly supervised affordance grounding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  53. [61]

    Affordpose: A large- scale dataset of hand-object interactions with affordance-driven hand pose,

    J. Jian, X. Liu, M. Li, R. Hu, and J. Liu, “Affordpose: A large- scale dataset of hand-object interactions with affordance-driven hand pose,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 14 713–14 724

    2023

  54. [62]

    3d affordancenet: A benchmark for visual object affordance understanding,

    S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia, “3d affordancenet: A benchmark for visual object affordance understanding,” in proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 1778–1787

    2021

  55. [63]

    Gift: Generalizable interaction-aware functional tool affordances without labels,

    D. Turpin, L. Wang, S. Tsogkas, S. Dickinson, and A. Garg, “Gift: Generalizable interaction-aware functional tool affordances without labels,” arXiv preprint arXiv:2106.14973 , 2021

    2021 arXiv

  56. [64]

    Rrl: Resnet as representation for reinforce- ment learning,

    R. Shah and V . Kumar, “Rrl: Resnet as representation for reinforce- ment learning,” arXiv preprint arXiv:2107.03380 , 2021

    2021 arXiv

  57. [65]

    The sur- prising effectiveness of representation learning for visual imitation,

    J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto, “The sur- prising effectiveness of representation learning for visual imitation,” arXiv preprint arXiv:2112.01511 , 2021

    2021 arXiv

  58. [66]

    The unsurprising effectiveness of pre-trained vision models for control,

    S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta, “The unsurprising effectiveness of pre-trained vision models for control,” in international conference on machine learning . PMLR, 2022, pp. 17 359–17 371

    2022

  59. [67]

    Masked visual pre- training for motor control,

    T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, “Masked visual pre- training for motor control,” arXiv preprint arXiv:2203.06173 , 2022

    2022 arXiv

  60. [68]

    Real-world robot learning with masked visual pre-training,

    I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell, “Real-world robot learning with masked visual pre-training,” in Conference on Robot Learning . PMLR, 2023, pp. 416–426

    2023

  61. [69]

    Vip: Towards universal visual reward and representation via value-implicit pre-training,

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang, “Vip: Towards universal visual reward and representation via value-implicit pre-training,” arXiv preprint arXiv:2210.00030 , 2022

    2022 arXiv

  62. [70]

    Where are we in the search for an artificial visual cortex for embodied intelligence?

    A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V .-P. Berges, T. Wu, J. Vakil et al. , “Where are we in the search for an artificial visual cortex for embodied intelligence?” Advances in Neural Information Processing Systems , vol. 36, pp. 655–677, 2023

    2023

  63. [71]

    What makes pre-trained visual representations successful for robust manipulation?

    K. Burns, Z. Witzel, J. I. Hamid, T. Yu, C. Finn, and K. Hausman, “What makes pre-trained visual representations successful for robust manipulation?” arXiv preprint arXiv:2312.12444 , 2023

    2023 arXiv

  64. [72]

    Offline visual represen- tation learning for embodied navigation,

    K. Yadav, R. Ramrakhya, A. Majumdar, V .-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymets, “Offline visual represen- tation learning for embodied navigation,” in Workshop on Reincar- nating Reinforcement Learning at ICLR 2023 , 2023

    2023

  65. [73]

    R3m: A universal visual representation for robot manipulation,

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” arXiv preprint arXiv:2203.12601, 2022

    2022 arXiv

  66. [74]

    Language-driven representation learning for robotics,

    S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang, “Language-driven representation learning for robotics,” arXiv preprint arXiv:2302.12766 , 2023

    2023 arXiv

  67. [75]

    Liv: Language-image representations and rewards for robotic control,

    Y . J. Ma, V . Kumar, A. Zhang, O. Bastani, and D. Jayaraman, “Liv: Language-image representations and rewards for robotic control,” in International Conference on Machine Learning . PMLR, 2023, pp. 23 301–23 320

    2023

  68. [76]

    Simple but effective: Clip embeddings for embodied ai,

    A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, “Simple but effective: Clip embeddings for embodied ai,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2022, pp. 14 829–14 838

    2022

  69. [77]

    Can foundation models perform zero-shot task specification for robot manipulation?

    Y . Cui, S. Niekum, A. Gupta, V . Kumar, and A. Rajeswaran, “Can foundation models perform zero-shot task specification for robot manipulation?” in Learning for dynamics and control conference . PMLR, 2022, pp. 893–905

    2022

  70. [78]

    Cliport: What and where pathways for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on robot learning. PMLR, 2022, pp. 894–906

    2022

  71. [79]

    Spawnnet: Learning generalizable visuomotor skills from pre-trained network,

    X. Lin, J. So, S. Mahalingam, F. Liu, and P. Abbeel, “Spawnnet: Learning generalizable visuomotor skills from pre-trained network,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 4781–4787

    2024

  72. [80]

    GenDP: 3d semantic fields for category-level generalizable diffusion policy,

    Y . Wang, G. Yin, B. Huang, T. Kelestemur, J. Wang, and Y . Li, “GenDP: 3d semantic fields for category-level generalizable diffusion policy,” in8th Annual Conference on Robot Learning, 2024. [Online]. Available: https://openreview.net/forum?id=7wMlwhCvjS

    2024

  73. [81]

    Robots pre-train robots: Manipulation-centric robotic representation from large-scale robot datasets,

    G. Jiang, Y . Sun, T. Huang, H. Li, Y . Liang, and H. Xu, “Robots pre-train robots: Manipulation-centric robotic representation from large-scale robot datasets,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.22325

    2024 arXiv

  74. [82]

    Otter: A vision-language-action model with text-aware visual feature extraction,

    H. Huang, F. Liu, L. Fu, T. Wu, M. Mukadam, J. Malik, K. Goldberg, and P. Abbeel, “Otter: A vision-language-action model with text-aware visual feature extraction,” 2025. [Online]. Available: https://arxiv.org/abs/2503.03734

    2025

  75. [83]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms,

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9568–9578

    2024

  76. [84]

    Winoground: Probing vision and language models for visio-linguistic compositionality,

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross, “Winoground: Probing vision and language models for visio-linguistic compositionality,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5238–5248

    2022

  77. [85]

    When and why vision-language models behave like bags-of-words, and what to do about it?

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” in The Eleventh International Conference on Learning Representations , 2023

    2023

  78. [86]

    Sugarcrepe: Fixing hackable benchmarks for vision-language com- positionality,

    C.-Y . Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna, “Sugarcrepe: Fixing hackable benchmarks for vision-language com- positionality,” Advances in neural information processing systems , vol. 36, 2024

    2024

  79. [87]

    Toward general-purpose robots via foundation models: A survey and meta-analysis,

    Y . Hu, Q. Xie, V . Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y . Xie, T. Zhang, Z. Zhao et al. , “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782, 2023

    2023 arXiv

  80. [88]

    Foundation models in robotics: Applications, challenges, and the future,

    R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y . Zhu, S. Song, A. Kapoor, K. Hausman et al. , “Foundation models in robotics: Applications, challenges, and the future,” arXiv preprint arXiv:2312.07843, 2023

    2023 arXiv

  81. [89]

    Real-world robot applications of foundation models: A review,

    K. Kawaharazuka, T. Matsushima, A. Gambardella, J. Guo, C. Pax- ton, and A. Zeng, “Real-world robot applications of foundation models: A review,” arXiv preprint arXiv:2402.05741 , 2024

    2024 arXiv

  82. [90]

    Foundation models for decision making: Problems, methods, and opportunities,

    S. Yang, O. Nachum, Y . Du, J. Wei, P. Abbeel, and D. Schuurmans, “Foundation models for decision making: Problems, methods, and opportunities,” arXiv preprint arXiv:2303.04129 , 2023

    2023 arXiv

  83. [91]

    Copa: General robotic manipulation through spatial constraints of parts with foun- dation models,

    H. Huang, F. Lin, Y . Hu, S. Wang, and Y . Gao, “Copa: General robotic manipulation through spatial constraints of parts with foun- dation models,” arXiv preprint arXiv:2403.08248 , 2024

    2024 arXiv

  84. [92]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting,

    F. Liu, K. Fang, P. Abbeel, and S. Levine, “Moka: Open-vocabulary robotic manipulation through mark-based visual prompting,” arXiv preprint arXiv:2403.03174, 2024

    2024 arXiv

  85. [93]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms,

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu et al. , “Pivot: Iterative visual prompting elicits actionable knowledge for vlms,” arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  86. [94]

    Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning,

    Y . Hu, F. Lin, T. Zhang, L. Yi, and Y . Gao, “Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning,” arXiv preprint arXiv:2311.17842 , 2023

    2023 arXiv

  87. [95]

    Video language planning,

    Y . Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum et al., “Video language planning,” arXiv preprint arXiv:2310.10625 , 2023

    2023 arXiv

  88. [96]

    3d-llm: Injecting the 3d world into large language models,

    Y . Hong, H. Zhen, P. Chen, S. Zheng, Y . Du, Z. Chen, and C. Gan, “3d-llm: Injecting the 3d world into large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 20 482–20 494, 2023

    2023

  89. [97]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 14 455– 14 465

    2024

  90. [98]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,” arXiv preprint arXiv:2307.05973 , 2023

    2023 arXiv

  91. [99]

    Rt-2: Vision- language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choro- manski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “Rt-2: Vision- language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023

    2023 arXiv

  92. [100]

    Physically grounded vision-language models for robotic manipulation,

    J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, “Physically grounded vision-language models for robotic manipulation,” arXiv preprint arXiv:2309.02561 , 2023

    2023 arXiv

  93. [101]

    Grounding language plans in demonstrations through counterfactual perturba- tions,

    Y . Wang, T.-H. Wang, J. Mao, M. Hagenow, and J. Shah, “Grounding language plans in demonstrations through counterfactual perturba- tions,” arXiv preprint arXiv:2403.17124 , 2024

    2024 arXiv

  94. [102]

    Ns3d: Neuro-symbolic grounding of 3d objects and relations,

    J. Hsu, J. Mao, and J. Wu, “Ns3d: Neuro-symbolic grounding of 3d objects and relations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 2614–2623

    2023

  95. [103]

    Physically grounded vision-language models for robotic manipulation,

    J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, “Physically grounded vision-language models for robotic manipulation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 12 462–12 469

    2024

  96. [104]

    Robopoint: A vision-language model for spatial affordance prediction for robotics,

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Mu- rali, A. Mousavian, and D. Fox, “Robopoint: A vision-language model for spatial affordance prediction for robotics,” arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  97. [105]

    Manipulate-anything: Automating real-world robots using vision-language models,

    J. Duan, W. Yuan, W. Pumacay, Y . R. Wang, K. Ehsani, D. Fox, and R. Krishna, “Manipulate-anything: Automating real-world robots using vision-language models,” arXiv preprint arXiv:2406.18915 , 2024

    2024 arXiv

  98. [106]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International conference on machine learning . Pmlr, 2021, pp. 8821–8831

    2021

  99. [107]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation,” in International conference on machine learning . PMLR, 2022, pp. 12 888–12 900

    2022

  100. [108]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  101. [109]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 000–16 009

    2022

  102. [110]

    Unsupervised learning of visual features by contrasting cluster assignments,

    M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in neural information processing systems , vol. 33, pp. 9912–9924, 2020

    2020

  103. [111]

    Data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Ma- chine Learning. PMLR, 2022, pp. 1298–1312

    2022

  104. [112]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning . PMLR, 2020, pp. 1597–1607

    2020

  105. [113]

    Vision trans- formers need registers,

    T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, “Vision trans- formers need registers,” arXiv preprint arXiv:2309.16588 , 2023

    2023 arXiv

  106. [114]

    Segment anything without supervision,

    X. Wang, J. Yang, and T. Darrell, “Segment anything without supervision,” 2024. [Online]. Available: https://arxiv.org/abs/2406. 20081

    2024

  107. [115]

    D3fields: Dynamic 3d descriptor fields for zero-shot gen- eralizable robotic manipulation,

    Y . Wang, Z. Li, M. Zhang, K. Driggs-Campbell, J. Wu, L. Fei-Fei, and Y . Li, “D3fields: Dynamic 3d descriptor fields for zero-shot gen- eralizable robotic manipulation,” arXiv preprint arXiv:2309.16118 , 2023

    2023 arXiv

  108. [116]

    Find any part in 3d,

    Z. Ma, Y . Yue, and G. Gkioxari, “Find any part in 3d,” 2024. [Online]. Available: https://arxiv.org/abs/2411.13550

    2024 arXiv

  109. [117]

    Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,

    C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Mart ´ın- Mart´ın, C. Wang, G. Levine, M. Lingelbach, J. Sun et al., “Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation,” in Conference on Robot Learning . PMLR, 2023, pp. 80–93

    2023

  110. [118]

    Objaverse-xl: A universe of 10m+ 3d objects,

    M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusu- pati, A. Fan, C. Laforte, V . V oleti, S. Y . Gadre, E. VanderBilt, A. Kembhavi, C. V ondrick, G. Gkioxari, K. Ehsani, L. Schmidt, and A. Farhadi, “Objaverse-xl: A universe of 10m+ 3d objects,” arXiv preprint arXiv...

    2023 arXiv

  111. [119]

    What do different evaluation metrics tell us about saliency models?

    Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 41, no. 3, pp. 740–757, 2019

    2019

  112. [120]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Pro- ceedings of the AAAI conference on artificial intelligence , 2018

    2018

  113. [121]

    Rvt: Robotic view transformer for 3d object manipulation,

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox, “Rvt: Robotic view transformer for 3d object manipulation,” in Conference on Robot Learning . PMLR, 2023, pp. 694–710

    2023

  114. [122]

    Rvt- 2: Learning precise manipulation from few demonstrations,

    A. Goyal, V . Blukis, J. Xu, Y . Guo, Y .-W. Chao, and D. Fox, “Rvt- 2: Learning precise manipulation from few demonstrations,” arXiv preprint arXiv:2406.08545, 2024

    2024 arXiv

  115. [123]

    A simple framework for open-vocabulary segmentation and detection,

    H. Zhang, F. Li, X. Zou, S. Liu, C. Li, J. Yang, and L. Zhang, “A simple framework for open-vocabulary segmentation and detection,” in Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2023, pp. 1020–1031

    2023

  116. [124]

    Locate: Localize and transfer object parts for weakly supervised affordance grounding,

    G. Li, V . Jampani, D. Sun, and L. Sevilla-Lara, “Locate: Localize and transfer object parts for weakly supervised affordance grounding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 922–10 931

    2023

  117. [125]

    Understanding 3d object interaction from a single image,

    S. Qian and D. F. Fouhey, “Understanding 3d object interaction from a single image,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023

    2023

  118. [126]

    Affor- dancellm: Grounding affordance from vision language models,

    S. Qian, W. Chen, M. Bai, X. Zhou, Z. Tu, and L. E. Li, “Affor- dancellm: Grounding affordance from vision language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7587–7597

    2024

  119. [127]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  120. [128]

    Perceiver-actor: A multi- task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi- task transformer for robotic manipulation,” in Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022

    2022

  121. [129]

    Q-attention: Enabling efficient learning for vision-based robotic manipulation,

    S. James and A. J. Davison, “Q-attention: Enabling efficient learning for vision-based robotic manipulation,” IEEE Robotics and Automa- tion Letters, vol. 7, no. 2, pp. 1612–1619, 2022

    2022

  122. [130]

    Coarse-to- fine q-attention: Efficient learning for visual robotic manipulation via discretisation,

    S. James, K. Wada, T. Laidlow, and A. J. Davison, “Coarse-to- fine q-attention: Efficient learning for visual robotic manipulation via discretisation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 739–13 748

    2022

  123. [131]

    Pybullet, a python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016

    2016

  124. [132]

    Viola: Imitation learning for vision-based manipulation with object proposal priors,

    Y . Zhu, A. Joshi, P. Stone, and Y . Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” 6th Annual Conference on Robot Learning , 2022. APPENDIX A. Details on Unsupervised Affordance Annotation Extrac- tion Pipeline (Sec III-A) Implemen...

    2022

  125. [133]

    Details for Evaluation on AGD20K Dataset – Evalu- ation setting and Processing

    if more than three workers label it accordingly. Details for Evaluation on AGD20K Dataset – Evalu- ation setting and Processing. We evaluate our model on the Unseen test split of AGD20K and compare with the baseline performance reported in [126]. To avoid numerical instability...

  126. [134]

    handle of plastic bag −− region for agent to hold and lift the bag

    Identify specific regions of the object that serve different purposes in various manipulation tasks. − Focus on crucial parts and offer detailed and fine−grained descriptions of the regions of interest. − For each identified region, provide both a Region Description and a Regi...

  127. [135]

    ANSWER:

    Match the colored region in (c) with the proposed task in step 1, considering the functionality and the granularity of the task. The requirements are as follows: − Compare the original image to the proposals to find the colored region that matches the description, considering ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.