Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a vision-language-action policy improves when it must first produce a chain of four affordances—what to manipulate and where, how to grasp, where to place, and how to move—before choosing an action.

desk verdict A plausible affordance-reasoning synthesis for VLA policies, but the real-robot headline gain is five successes out of 77 trials and the evaluation protocol needs clarifying. read the letter →

arxiv 2412.20451 v2 pith:7IN57SP6 submitted 2024-12-29 cs.RO

Jinming Li , Yichen Zhu , Zhibin Tang , Junjie Wen , Minjie Zhu , Xiaoyu Liu , Chengmeng Li , Ran Cheng
show 3 more authors
Yaxin Peng Yan Peng Feifei Feng
This is my paper · ORCID
classification cs.RO
keywords vision-language-actionmodelchain-of-affordanceaffordancereasoningrobotmanipulationdiffusionpolicyvisualpromptinggeneralizationmulti-tasklearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that a robot manipulation policy becomes more accurate, not by adding data or parameters, but by making the model produce a short reasoning chain before it acts. The chain consists of four robot affordances—which object to touch and where, which part of it to grasp, where to place it, and what collision-free path to follow—expressed both as text and as visual overlays on the camera image. This reasoning is injected into a diffusion-based action predictor, and the paper reports that the resulting model beats existing generalist robot policies on seven real-robot tasks, improves on its own backbone by 14.29 percent, and reaches a 79.8 percent success rate on the LIBERO simulation benchmark. A sympathetic reader would care because the recipe is cheap: it adds an intermediate reasoning stage to a vision-language-action model rather than scaling the model, and it appears to help precisely where robot policies usually fail—unseen object poses, cluttered scenes, and obstacle avoidance.

What carries the argument

The carrying mechanism is the chain-of-affordance: a fixed sequence of four affordance predictions interleaved as intermediate reasoning between the observation and the action. The visual-textual co-injection module is the other load-bearing piece: it takes text embeddings from the vision-language model and visual-affordance patch tokens from a small vision transformer, passes them through two transformer blocks, and injects the result into the diffusion action head with FiLM (feature-wise linear modulation) conditioning. Dynamic affordance selection, driven by proprioceptive state such as gripper and joint data, decides which affordances are needed at each timestep, keeping the added inference cheap. Together these pieces convert the model's own affordance guesses into explicit conditioning for action generation.

What would settle it

Take the trained model and, at test time, feed it deliberately wrong affordance overlays and texts; if success rates do not drop, the chain is not doing the causal work the paper attributes to it. A second decisive check is to compare the automatically generated affordance labels against human annotations on a small sample—if agreement is low, the training signal itself is unverified.

Watch

Extended reading notes

Core claim

The paper's central claim is that sequential affordance reasoning is a sufficient intermediate representation to improve vision-language-action policy learning. Before predicting an action, the model must first identify the target object and its location (object affordance), decide where to grasp it (grasp affordance), identify a collision-free placement region (spatial affordance), and trace a feasible motion path (movement affordance). The four affordances are generated by the model itself, rendered in both natural-language and pixel-aligned visual form, and fused into the action head. The discovery, as the paper reports it, is that this conditioning raises average success rates over the baselines it is compared with in both real-world and simulated tasks, and that the largest gaps appear during visual generalization, free-space placement, and obstacle avoidance.

Load-bearing premise

The entire result rests on the automatic annotation pipeline being accurate: if the object boxes, grasp points, placement points, and movement tracks it produces are noisy, the model is trained on corrupted reasoning, and the reported gains may come from the visual overlays acting as hints rather than from genuine affordance understanding.

Editorial extensions

If this is right

  • Any vision-language-action model that can emit tokens and condition a diffusion head could adopt the chain-of-affordance pattern, since the additional module is small compared with the backbone.
  • Dynamic affordance selection keeps test-time reasoning cheap, making the method practical on a single GPU; the paper reports roughly 6 Hz inference versus 1 Hz when all affordances are forced.
  • The four-part taxonomy gives robot policies an inspectable intermediate output: failures can be attributed to the object, grasp, placement, or movement stage instead of the action decoder alone.
  • Explicit spatial and object affordances appear to reduce sensitivity to visual appearance, which is exactly the regime where the paper reports the largest gains over baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: part of the measured gain may come from the visual overlays acting as spatial priors rather than from the semantic chain; ablating overlays with shuffled coordinates would separate the two.
  • Inference: because the training labels are generated by an unmeasured automatic pipeline, the method's ceiling is tied to that pipeline; a small human-verified label set would quantify how much label noise remains.
  • Inference: the same chain could be generated once per subtask rather than per timestep and used by a slower high-level planner, which would reduce compute cost below even the paper's 6 Hz figure.
  • Inference: applying the chain to 3D or contact-rich affordances would be a natural extension, since 2D pixel points may under-specify grasps and placements in clutter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CoA-VLA, a vision-language-action model that generates a chain of four affordances—object, grasp, spatial, and movement—in both textual and visual formats before predicting actions. The affordances are produced through a dynamic selection mechanism and injected into a diffusion-based policy via a visual-textual co-injection module. The authors evaluate the method on seven real-robot tasks and on the LIBERO benchmark, reporting improvements over OpenVLA, Octo, Diffusion Policy, and the DiffusionVLA backbone, as well as qualitative generalization results for unseen poses, free-space placement, and obstacle avoidance.

Significance. The paper addresses a timely and important question: whether explicit, structured affordance reasoning can improve VLA policies. The proposed four-part affordance taxonomy and the visual-textual co-injection module are well motivated, and the dynamic affordance selection is a sensible efficiency mechanism. The LIBERO evaluation uses 1500 trials with standard errors and three-seed averaging, and the ablations in Tables 4 and 5 provide useful evidence about the contribution of each component. The real-robot study covers seven tasks and includes both in-distribution and visual-generalization settings. If the results were rigorously established, CoA-VLA would be a meaningful advance in reasoning-augmented robot manipulation.

major comments (4)
  1. [Section 5.1, Table 1] The headline real-robot claim is not statistically supported. In the in-distribution setting, CoA-VLA succeeds in 64/77 trials versus 59/77 for DiffusionVLA, a difference of only 5 successes concentrated in three tasks (PourTea +2, PlaceBread +3, WipeWater +3) with one task showing a loss (NailHammer -1). For n=77, the standard error of the difference in proportions is approximately 0.065, giving p>0.15; the visual-generalization comparison (36/63 vs 28/63) is also marginal (roughly p=0.08). No confidence intervals or significance tests are reported. Without additional trials or a more careful statistical analysis, the abstract's statement that CoA-VLA 'exceeds its own diffusion backbone DiffusionVLA by 14.29% on real-robot tasks' is not established. The qualitative claims in Section 5.3 about obstacle avoidance and free-space placement are based on very small numbers of scenarios and should be reported with explicit success counts.
  2. [Supplementary Section 6.4] The sentence 'We filter out the failure data and increase the image resolution to 224 x 224' is ambiguous and potentially damaging. If this filtering applies to evaluation rollouts, the reported success rates in Table 2 would be invalid. If it applies only to training-data cleaning or to the affordance-generation pipeline, that should be stated explicitly. The authors must clarify exactly where this filtering occurs and confirm that the same protocol is applied to all compared methods.
  3. [Section 4.3] The automated affordance-annotation pipeline is a load-bearing component of the method, but its output quality is never quantified. The paper does not report any accuracy, agreement, or noise-level measurement for the GPT-4o, Grounding DINOv2/SAM, RoboPoint, and CoTracker labels, nor does it measure how often the model's self-generated affordances at test time are correct. Since the policy is trained on these labels and conditioned on them at inference, noisy or biased annotations could be the source of the observed gains rather than the proposed reasoning structure. I recommend adding an evaluation of affordance-prediction quality, an ablation with corrupted labels, or at minimum a manual inspection statistics for a held-out subset.
  4. [Section 6.4, Table 2] The LIBERO results for Diffusion Policy, Octo, and OpenVLA are cited from the OpenVLA paper rather than re-run under the authors' exact protocol. Given that the authors also report using a different image resolution and a filtered dataset, the cross-paper comparison may not be apples-to-apples. The authors should either re-run the baselines with their own evaluation pipeline or provide evidence that the protocol differences do not affect the comparison.
minor comments (4)
  1. [Section 5.1 vs Section 6.3] There is an apparent inconsistency: Section 5.1 states that DiffusionVLA is 'the same model our approach is built upon,' while Section 6.3 says the method is trained 'without relying on pre-trained weights from DiffusionVLA.' The authors should clarify whether the comparison is architectural or weight-initialization-based.
  2. [General] There are several typos, including 'Acknowledegments' in the acknowledgments heading, 'Wi U Water' in Section 6.2, and 'It can also represented' in Section 2. These should be corrected.
  3. [References] Reference [29] has a malformed author list that appears to concatenate many names without proper formatting; this should be fixed.
  4. [Table 5] The inference-speed measurement is based on only 5 trials per task, which is a very small sample for a timing claim; adding more trials and reporting a variance estimate would strengthen the efficiency claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are closed-loop empirical comparisons, not outputs forced by fitted parameters or by the authors' prior results.

full rationale

The paper's central claim is that adding a visual-textual chain-of-affordance intermediate output to a DiffusionVLA backbone improves task success. The derivation chain is architectural and empirical rather than algebraic: Section 4.1 defines the affordance decomposition z = {zobj, zgrasp, zspat, zmove} and the action model a ~ p(a|τ,g,z), but no equation in the paper defines the reported success-rate gain as an identity in terms of z or of any fitted parameter. Section 4.3 describes an offline annotation pipeline (GPT-4o, Grounding DINOv2 + SAM, RoboPoint + GPT-4o point clustering, CoTracker) that generates training labels from expert demonstrations; this is a data-generation procedure and does not by construction determine closed-loop evaluation outcomes. The only same-author citation that is load-bearing for the method's structure is DiffusionVLA [46], used as the backbone; however, the paper evaluates CoA-VLA against DiffusionVLA as a directly trained baseline on the same data ('all models are fine-tuned on the same dataset...the last checkpoint is used for evaluation'), so the 14.29% improvement is an empirical comparison, not an imported theorem. No uniqueness claim, ansatz, or fitted input is justified by self-citation. One passage in the supplementary material, Section 6.4, states 'We filter out the failure data'; if this refers to evaluation rollouts it would undermine the LIBERO success-rate numbers, but this is an evaluation-integrity and statistical-reporting concern, not a circularity of derivation, and the surrounding text indicates the filtering concerns data/resolution processing rather than a stated equivalence between inputs and outputs. I therefore find no circular step warranting a nonzero score.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on empirical training and evaluation rather than a derivation. No numerical constants are fitted to the target benchmarks; the hyperparameters reported (learning rates 2e-5/2e-6, 5 epochs) are standard choices. The key assumptions are domain assumptions about the affordance taxonomy, the annotation pipeline, and the sufficiency of proprioception for dynamic selection.

assumptions (5)
  • domain assumption The four affordance types (object, grasp, spatial, movement) are sufficient to guide the target manipulation tasks.
    The entire framework, data generation pipeline, and prompting formats are built around these four categories (Section 4.1). If a task requires a different kind of reasoning (e.g., force or contact), the method may not cover it.
  • domain assumption The automated annotation pipeline produces accurate affordance labels.
    Section 4.3 relies on GPT-4o, Grounding DINOv2, SAM, RoboPoint, and CoTracker to create training labels. The paper does not quantify annotation accuracy or agreement with human labels.
  • domain assumption The model can learn dynamic affordance selection from proprioception and vision without explicit supervision.
    Section 4.1 states that concatenating a proprioceptive token with visual tokens lets the model 'intelligently select relevant affordances'. No supervision or metric for selection quality is provided.
  • domain assumption The Qwen2-VL and DiffusionVLA backbones can effectively ground and use the visual-textual affordance prompts.
    Section 3 builds on the pre-trained DiffusionVLA model; the paper provides no analysis of which components of the VLM are responsible for the gain.
  • domain assumption The baseline results cited from prior papers are measured under the same conditions as the in-house runs.
    Table 2 cites Diffusion Policy, Octo, and OpenVLA results from OpenVLA's paper; the paper states the same test data is used but does not rerun these baselines, so comparability is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance." pith.science (2026). https://pith.science/paper/7IN57SP6

@misc{pith2026241220451,
  author       = {Pith},
  title        = {Pith review of: CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7IN57SP6}},
  note         = {Machine review of arXiv:2412.20451}
}
read the original abstract

Robot foundation models, particularly Vision-Language-Action (VLA) models, have garnered significant attention for their ability to enhance robot policy learning, greatly improving robot's generalization and robustness. OpenAI's recent model, O1, showcased impressive capabilities in solving complex problems by utilizing extensive reasoning chains. This prompts an important question: can robot models achieve better performance in multi-task , complex environments by reviewing prior observations and then providing task-specific reasoning to guide action prediction? In this paper, we introduce Chain-of-Affordance (CoA-VLA) , a novel approach to scaling robot models by incorporating reasoning in the format of sequential robot affordances to facilitate task completion. Specifically, we prompt the model to consider the following four types of affordances before taking action: (1) object affordance - what object to manipulate and where it is ; (2) grasp affordance - the specific object part to grasp ; (3) spatial affordance - the optimal space to place the object ; and (4) movement affordance-the collision - free path for movement. We further transform each affordance into two prompting formats: visual affordance and textual affordance. We introduce a novel vision-language co-injection module that integrates this knowledge into the policy network. This allows the robot to leverage essential contextual information during action inference, resulting in improved precision and robustness. Our experiments demonstrate that CoA-VLA outperforms state-of-the-art robot foundation models, including OpenVLA and Octo, on a variety of tasks. Furthermore, CoA-VLA exhibits strong generalization capabilities, including recognizing unseen object poses, identifying free space, and avoiding obstacles in novel environments.

Figures

Figures reproduced from arXiv: 2412.20451 by the authors.

Figure 1
Figure 1. This figure illustrates the overall framework of our CoA-VLA model, which empowers vision-language-action models with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of the chain-of-affordance for the PourTea task. The first row presents the text affordance and the second row shows the visual affordance. By employing a dynamic affordance selection mechanism, our method avoids generating redundant affordances at every timestep. object to interact with and where it is located, particularly in scenarios where user instructions lack explicit spatial or semantic details. T… view at source ↗
Figure 3
Figure 3. Robot setup and examples for real-world manipulation tasks. We evaluate seven real-world tasks on Franka robot arm equipped with two external Zed cameras and a Realsense 435i wrist camera [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Spatial affordance for CoA-VLA. CoA-VLA can identify free space for object placement.. Method \ Obstacle Avoidance OpenVLA DiffusionVLA CoA-VLA [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Movement generalization for CoA-VLA. CoA￾VLA can avoid obstacles and operate safely. robot is presented with a plate on which three distinct ob￾jects are already placed, and it is instructed to add a piece of bread onto the plate. Our method successfully identifies ope…
Figure 6
Figure 6. Figure 6: Visual Generalization. We evaluate each method on multi-task learning and visual generalization, which includes handling additional distractors and interference from colored light. We also test the ability to grasp objects of the same type but with varying shapes, such…
Figure 7
Figure 7. Figure 7: More detailed examples of successful Chain-of-Affordance. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Generalization on object pose. CoA can pick up ob￾jects with unseen poses, benefiting from grasp affordance. all other implementations kept the same. 7. More Experiments 7.1. Ablation Study on Visual-Textual Affordance Our primary contribution lies in the introduction …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think Proprioceptively: State-Grounded Visual Token Selection for VLA Policies

    cs.RO 2026-02 conditional novelty 6.0 of 10

    Using tokenized proprioception plus instruction to select ~15% of visual patches matches or beats full-token VLA baselines and cuts latency by ~58%.

  2. FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies

    cs.RO 2025-09 conditional novelty 6.0 of 10

    A compact 950-million-parameter robot policy trained in about 200 GPU-hours matches or beats multi-billion-parameter baselines on most manipulation benchmarks, including a new best score on CALVIN ABC.

  3. ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge

    cs.RO 2025-05 conditional novelty 6.0 of 10

    ChatVLA-2 uses dynamic mixture-of-experts and a two-stage training recipe to let a vision-language-action model retain pretrained reasoning while following robot instructions.

  4. Data Pyramid for Embodied Manipulation: A Survey

    cs.RO 2026-07 conditional novelty 3.0 of 10

    Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.

  5. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

Reference graph

Works this paper leans on

56 extracted references · 24 canonical work pages · cited by 5 Pith papers

  1. [1]

    TraceVLA: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies

    Anonymous. TraceVLA: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. under review. 3

    work page 2024

  2. [2]

    Affordances from human videos as a versatile representation for robotics

    Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778–13790, 2023. 3

    work page 2023

  3. [3]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gersten- berger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, pages 17682–17690, 2024. 3

    work page 2024

  4. [4]

    Roboa- gent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking

    Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboa- gent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788–4795. IEEE, 2024. 5

    work page 2024

  5. [5]

    Training diffusion models with reinforce- ment learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning. arXiv preprint arXiv:2305.13301, 2023. 3

    arXiv 2023

  6. [6]

    Zero-shot robotic manipulation with pretrained image- editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image- editing diffusion models. arXiv preprint arXiv:2310.10639,

    arXiv

  7. [7]

    π0: A vision-language-action flow model for general robot control, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0: A visi...

    work page 2024

  8. [8]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 3

    arXiv 2023

Show all 56 references
  1. [9]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 144...

  2. [10]

    Diffu- sion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffu- sion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023. 2, 6, 7, 8

    2023 arXiv

  3. [11]

    Manipulate- anything: Automating real-world robots using vision- language models

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate- anything: Automating real-world robots using vision- language models. arXiv preprint arXiv:2406.18915 , 2024. 3

    2024 arXiv

  4. [12]

    Graspnet-1billion: A large-scale benchmark for general ob- ject grasping

    Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general ob- ject grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11444– 11453, 2020. 3

    2020

  5. [13]

    Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains

    Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spa- tial and temporal domains. IEEE Transactions on Robotics,

  6. [14]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024. 3

    2024 arXiv

  7. [15]

    A3vlm: Actionable articulation-aware vision language model

    Siyuan Huang, Haonan Chang, Yuhan Liu, Yimeng Zhu, Hao Dong, Peng Gao, Abdeslam Boularias, and Hongsheng Li. A3vlm: Actionable articulation-aware vision language model. arXiv preprint arXiv:2406.07549, 2024

    2024 arXiv

  8. [16]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  9. [17]

    Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024. 3

    2024 arXiv

  10. [18]

    Synergies between affordance and geome- try: 6-dof grasp detection via implicit representations

    Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geome- try: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542, 2021. 3, 4

    2021 arXiv

  11. [19]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023. 6

    2023 arXiv

  12. [20]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. arXiv preprint arXiv:2410.11831 ,

    arXiv

  13. [21]

    Droid: A large-scale in-the-wild robot manipulation dataset

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun- liang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:24...

    2024 arXiv

  14. [22]

    OpenVLA: An open- source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Fos- ter, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open- source vision-language-action model. 2, 3, 6, 7, 8

  15. [23]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 6

    2023

  16. [24]

    Chain of code: Reasoning with a language model-augmented code emulator

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei 9 Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474, 2023. 3

    2023 arXiv

  17. [25]

    Cogact: A foundational vision-language- action model for synergizing cognition and action in robotic manipulation, 2024

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jian- min Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. Cogact: A foundational vision-language- action model f...

    2024

  18. [26]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378,

    arXiv

  19. [27]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yux- ing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  20. [28]

    Data scaling laws in imitation learning for robotic manipulation, 2024

    Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Ji- acheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation, 2024. 3

    2024

  21. [29]

    Cot-vla visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao Yao Lu Moo Jin Kim Zipeng Fu Zhuoyang Zhang Yecheng Wu Max Li Qianli Ma Song Han Chelsea Finn Ankur Handa Ming-Yu Liu Donglai Xiang* Gordon Wetzstein* Tsung-Yi Lin*. Cot-vla visual chain-of-thought reasoning for vision-language-action models. 2024. 3

    2024

  22. [30]

    LIBERO: Benchmarking knowl- edge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowl- edge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024. 2, 7

    2024

  23. [31]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024. 3

    2024

  24. [32]

    Marrying dino with grounded pre-training for open-set object detection

    S Liu, Z Zeng, T Ren, F Li, H Zhang, J Yang, C Li, J Yang, H Su, J Zhu, et al. Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499. 6

    arXiv

  25. [33]

    Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects

    Weiyu Liu, Chris Paxton, Tucker Hermans, and Dieter Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. In 2022 Inter- national Conference on Robotics and Automation (ICRA) , pages 6322–6329. IEEE, 2022. 3

    2022

  26. [34]

    Same object, differ- ent grasps: Data and semantic knowledge for task-oriented grasping

    Adithyavairavan Murali, Weiyu Liu, Kenneth Marino, So- nia Chernova, and Abhinav Gupta. Same object, differ- ent grasps: Data and semantic knowledge for task-oriented grasping. In Conference on robot learning , pages 1540–

  27. [35]

    Rt- affordance: Affordances are versatile intermediate represen- tations for robot manipulation, 2024

    Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. Rt- affordance: Affordances are versatile intermediate represen- tations for robot manipulation, 2024. 3

    2024

  28. [36]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo...

    2024

  29. [37]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023. 3

    arXiv

  30. [39]

    Fast: Efficient action tokenization for vision- language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- language-action models. arXiv e-prints, pages arXiv–2501,

  31. [40]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 6

    2024 arXiv

  32. [41]

    Yell at your robot: Improving on-the-fly from language corrections

    Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403.12910 ,

    arXiv

  33. [42]

    Contact-graspnet: Efficient 6-dof grasp gen- eration in cluttered scenes

    Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-graspnet: Efficient 6-dof grasp gen- eration in cluttered scenes. In 2021 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 13438– 13444. IEEE, 2021. 3

    2021

  34. [43]

    Contact-graspnet: Efficient 6-dof grasp gen- eration in cluttered scenes

    Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-graspnet: Efficient 6-dof grasp gen- eration in cluttered scenes. In 2021 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 13438– 13444. IEEE, 2021. 4

    2021

  35. [44]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 3

    2024 arXiv

  36. [45]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 3

    2022

  37. [46]

    DiffusionVLA: Scal- ing robot foundation models via unified diffusion and autore- gression

    Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Chengmeng Li, Zhongyi Zhou, Xiaoyu Liu, Chaomin Shen, Yaxin Peng, and Feifei Feng. DiffusionVLA: Scal- ing robot foundation models via unified diffusion and autore- gression. 2024. 2, 3, 6, 7

    2024

  38. [47]

    Tinyvla: Towards fast, data-efficient vision- language-action models for robotic manipulation

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, et al. Tinyvla: Towards fast, data-efficient vision- language-action models for robotic manipulation. arXiv preprint arXiv:2409.12514, 2024. 3, 7 10

    2024 arXiv

  39. [48]

    Gradient boosted feature selection

    Zhixiang Xu, Gao Huang, Kilian Q Weinberger, and Alice X Zheng. Gradient boosted feature selection. InProceedings of the 20th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 522–531, 2014. 4

    2014

  40. [49]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Sys- tems, 36, 2024. 3

    2024

  41. [50]

    M2t2: Multi-task masked trans- former for object-centric pick and place

    Wentao Yuan, Adithyavairavan Murali, Arsalan Mousa- vian, and Dieter Fox. M2t2: Multi-task masked trans- former for object-centric pick and place. arXiv preprint arXiv:2311.00926, 2023. 3

    2023 arXiv

  42. [51]

    Robopoint: A vision-language model for spatial affordance prediction for robotics

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousa- vian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024. 3, 4, 6

    2024 arXiv

  43. [52]

    Robotic control via embodied chain-of-thought reasoning

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08693, 2024. 3

    2024 arXiv

  44. [53]

    Transporter networks: Rearranging the visual world for robotic manipu- lation

    Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipu- lation. In Conference on Robot Learning , pages 7...

    2021

  45. [54]

    Cot-vla: Visual chain-of- thought reasoning for vision-language-action models, 2024

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Max Li Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming-Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung-Yi Lin. Cot-vla: Visual chain-of- thought reasoning for vision-language-action models, 2024. 3

    2024

  46. [55]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Sch ¨arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022. 3

    2022 arXiv

  47. [56]

    Scaling diffusion policy in transformer to 1 billion parameters for robotic manipulation

    Minjie Zhu, Yichen Zhu, Jinming Li, Junjie Wen, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, et al. Scaling diffusion policy in transformer to 1 billion parameters for robotic manipulation. arXiv preprint arXiv:2409.14411, 2024. 7, 8 11 CoA-VLA: Impr...

    2024 arXiv

  48. [57]

    More Experiments 7.1. Ablation Study on Visual-Textual Affordance Our primary contribution lies in the introduction of tex- tual affordances and visual affordances, paired with a novel visual-textual co-injection module designed to synergisti- cally integrate these modalities ...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.