Pith. sign in

REVIEW 4 major objections 3 minor 67 references

PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that automatically extracted geometric primitives, anchored to task semantics by a vision-language model, perform as well as manual annotations for robotic manipulation.

desk verdict A sensible-sounding framework that I cannot verify: the full text is garbled and the abstract reports no numbers. read the letter →

arxiv 2508.05976 v1 pith:FNQDTVEA submitted 2025-08-08 cs.CV cs.RO

Zhihao Zhu , Yifan Zheng , Siyu Pan , Yaohui Jin , Yao Mu This is my paper
classification cs.CVcs.RO
keywords roboticmanipulationvision-languagemodelsemanticaffordancegeometricprimitiveextractionkeypointdetectiongroundingclosed-loopframeworkspatial-semanticreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a robot can acquire the semantic-affordance knowledge needed for manipulation—where and how to grasp or use an object—without per-object manual annotation. PASG automatically extracts geometric primitives (keypoints and axes) from objects via geometric feature aggregation, then a vision-language model anchors those primitives to functional affordances and task-relevant descriptions. On practical manipulation tasks across diverse scenarios, the authors report performance comparable to manual annotations, plus finer-grained semantic-affordance understanding. If the claim holds, it removes a major labeling bottleneck and lets robots reason about novel objects in a canonical geometric space rather than through memorized object templates.

What carries the argument

The central object is the PASG framework itself, a closed-loop pipeline with two coupled stages. The first stage, automatic primitive extraction, aggregates geometric features to detect keypoints and axes across object categories without category-specific supervision; this provides a canonical geometric ground. The second stage, VLM-driven semantic anchoring, couples those primitives to functional affordances and task-relevant descriptions using a fine-tuned vision-language model (Qwen2.5VL-PA). The load-bearing mechanism is the dynamic coupling: task semantics attach to detected geometry rather than to image pixels or memorized object names, which is what allows the system to generalize acr

What would settle it

Measure keypoint and axis localization error on held-out object categories and correlate it with manipulation success: if large localization errors coexist with success rates equal to manual annotations, the semantic anchoring is not actually carrying the load. A direct check is to replace the extracted primitives with noise or random positions and see whether task performance collapses.

Watch

Extended reading notes

Core claim

The central claim is that the gap between high-level task semantics and low-level geometry can be closed without manual affordance labels. PASG first detects manipulation-relevant geometric primitives—keypoints and axes—across object categories using geometric feature aggregation; this is the automatic step intended to replace human annotation. A VLM then performs semantic anchoring, dynamically coupling those primitives with functional affordances and task descriptions, and the coupled result guides manipulation. The paper also contributes a spatial-semantic reasoning benchmark and a fine-tuned VLM, Qwen2.5VL-PA, for the anchoring stage. Evaluated in practical robotic manipulation scenarios

Load-bearing premise

The automatic primitive extractor must reliably locate manipulation-relevant keypoints and axes on objects it was not specifically trained on; if those primitives are noisy or category-biased, the VLM anchoring inherits the errors and the claimed parity with manual annotations fails.

Editorial extensions

If this is right

  • Manual semantic-affordance annotation can be replaced or at least augmented by automatic primitive extraction plus VLM anchoring, cutting the labeling cost for manipulation datasets.
  • Because primitives are detected across categories, the same geometric vocabulary can support reasoning about novel objects instead of category-specific templates.
  • The spatial-semantic reasoning benchmark gives a shared measure for comparing how well a system understands where an object affords an action.
  • If parity with manual annotations holds in practice, robots can be set up for new manipulation tasks faster: an object can be understood from its geometry and a task description, with no curated semantic map.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test would feed the VLM deliberately misplaced primitives while keeping the image fixed; if task reasoning does not change, the extracted geometry is not actually driving the semantics.
  • The closed-loop framing suggests a self-improving variant: failed manipulations could update primitive-semantic bindings, turning the framework into a learning loop that further reduces the need for curated affordance data.
  • The same keypoint-and-axis vocabulary might extend beyond grasping to tool use and assembly, where shared geometric roles such as handle, opening, and rotation axis are equally task-relevant.
  • The benchmark could be pushed to deformable or articulated objects, where canonical axes are less stable; that would test whether the geometric grounding argument is a general principle or a rigid-object result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes PASG, a closed-loop framework for automatic geometric primitive extraction and semantic anchoring in robotic manipulation. The stated contributions are (1) cross-category primitive extraction via geometric feature aggregation, (2) VLM-driven semantic anchoring that couples primitives with affordances and task descriptions, and (3) a spatial-semantic reasoning benchmark together with a fine-tuned VLM, Qwen2.5VL-PA. The abstract's central claim is that PASG achieves performance comparable to manual annotations in practical robotic manipulation tasks. However, the abstract provides no quantitative results, and the supplied full text is corrupted and unreadable: it appears as mojibake and carries a header for arXiv:2508.05987v1 [cs.CL], not arXiv:2508.05976 cs.CV. Consequently, no method details, derivations, experiments, or benchmark descriptions can be checked.

Significance. If the central claim were supported, the work would be practically significant: replacing manual semantic-affordance annotations with an automatic pipeline of geometric primitive extraction and VLM semantic anchoring would reduce annotation cost and potentially improve generalization across object categories. The motivation is sensible and the proposed architecture is plausible. However, the submitted manuscript does not demonstrate these benefits. There are no reported success rates, no baseline comparisons, no error bars, no dataset description, no primitive-localization accuracy, and no external validation. The paper introduces both the evaluation benchmark and the fine-tuned model, so the evaluation is a self-consistency check without independent grounding. No code, proofs, or reproducible artifacts are provided. The significance is therefore potential, not demonstrated, in the current submission.

major comments (4)
  1. [Abstract] The central claim, 'achieving performance comparable to manual annotations,' is asserted without any quantitative metric, dataset definition, baseline, success rate, or error bar. This is the load-bearing result of the paper and cannot be evaluated from the abstract or the supplied text. A table comparing PASG against manual annotations (and against at least one prior automatic method) on explicit manipulation success metrics is required, along with variance and statistical significance.
  2. [Full text (all sections)] The provided full text is unreadably corrupted (mojibake), so no equation, table, figure, loss function, or experimental protocol can be verified. The embedded header 'arXiv:2508.05987v1 [cs.CL] 8 Aug 2025' mismatches the claimed paper identifier arXiv:2508.05976 cs.CV. This prevents any check of the derivations behind the primitive extraction or semantic anchoring, and makes the experimental section effectively absent. A clean, correctly identified manuscript is a prerequisite for further review.
  3. [Benchmark and model introduced together (per abstract)] The evaluation appears to use a spatial-semantic reasoning benchmark introduced in the same paper and the fine-tuned model Qwen2.5VL-PA introduced in the same paper. This is not inherently circular, but it means the headline parity claim is a self-consistency check unless the benchmark is externally grounded or compared to existing datasets. To support the claim, the authors should evaluate on an established manipulation benchmark, include human-annotation agreement, and provide cross-split generalization results that rule out overfitting to the in-house benchmark.
  4. [Primitive extraction (abstract)] The abstract claims automatic cross-category detection of keypoints and axes via geometric feature aggregation, but reports no localization accuracy for these primitives. Since the VLM semantic anchoring is downstream of the primitive extractor, noisy or category-biased primitives would propagate into the final affordance predictions. Task-level success rates alone cannot validate this component. The paper needs per-category keypoint/axis localization error, ablations removing the primitive extractor, and a direct comparison of PASG outputs against manual primitive annotations.
minor comments (3)
  1. [Abstract and general terminology] Terms such as 'spatial-semantic reasoning benchmark' and 'Qwen2.5VL-PA' are introduced without definitions or release details. If these are new entities, they should be described with dataset size, annotation procedure, and model architecture in the main text.
  2. [Title and framing] The phrase 'closed-loop framework' is prominent, but the abstract does not identify the feedback signal or loop closure mechanism. State explicitly where the loop is closed and what information flows back.
  3. [arXiv header] The full text contains a different arXiv identifier and subject class than the one in the paper's title. This needs correction; it also raises concerns about whether the correct file was submitted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; headline evaluation is an empirical claim, not a derivation that reduces to its inputs.

full rationale

PASG is an empirical systems paper, not a derivation chain. Its stated components—automatic geometric feature aggregation, VLM-driven semantic anchoring, a benchmark, and a fine-tuned Qwen2.5VL-PA—are inputs to an experimental evaluation, not quantities that are defined in terms of each other or fitted and then renamed as predictions. I find no quoted equation or table in which a fitted parameter is later called a predicted result, and no load-bearing argument that reduces to a self-citation chain. The abstract's headline claim of 'performance comparable to manual annotations' is an empirical claim; whether it is well supported is a correctness/verification issue, not a circularity issue. The supplied full text is heavily mojibake and contains an arXiv header mismatch ('arXiv:2508.05987v1 [cs.CL] 8 Aug 2025'), so I could not verify the experimental details; this is a provenance concern and does not by itself demonstrate circularity. Introducing an in-house benchmark and a fine-tuned VLM is self-referential in an evaluation sense, but nothing in the available text shows that the benchmark's ground truth is the same output used to supervise the model in a way that would force the reported result by construction. Absent a specific reduction, no circular step can be claimed under the hard rules.

Assumptions & free parameters 1 free parameters · 3 assumptions · 2 invented entities

Central claim rests on an in-house fine-tuned VLM, an in-house benchmark, and the domain assumption that geometric primitives are sufficient semantic anchors. No external dataset or third-party evaluation visible in the abstract. Moderate, not extreme, reliance on self-produced components.

free parameters (1)
  • Qwen2.5VL-PA fine-tuned weights = not disclosed
    The VLM is fine-tuned by the authors for semantic anchoring; the trained weights encode the core semantic mapping and are fitted to the authors' data, but no training details or weights are given in the abstract.
assumptions (3)
  • domain assumption Geometric primitives (keypoints, axes) extracted by geometric feature aggregation are sufficient and stable anchors for semantic-affordance reasoning.
    The entire PASG pipeline assumes that low-level geometric primitives carry the information needed for VLM semantic anchoring; stated in the abstract's contribution list.
  • domain assumption Manual annotations are a reliable ground-truth baseline and the bottleneck the framework removes.
    The motivation, 'reliance on manual annotations severely limits,' treats manual annotation as the reference quality target; no independent evidence is given.
  • domain assumption Vision-language models can map geometric primitives to functional affordances given task descriptions.
    The semantic anchoring component presumes Qwen2.5VL-PA can perform the mapping from primitives to affordances; the abstract offers no failure analysis.
invented entities (2)
  • Spatial-semantic reasoning benchmark
    purpose: Evaluation of spatial-semantic grounding quality
    The benchmark is introduced by this paper; no external standardization, public availability, or adoption is indicated in the abstract.
  • Qwen2.5VL-PA fine-tuned VLM
    purpose: VLM-driven semantic anchoring of geometric primitives
    A fine-tuned variant of Qwen2.5VL; its weights and training data are internal to this work, so it has no falsifiable handle outside the paper as described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation." pith.science (2026). https://pith.science/paper/FNQDTVEA

@misc{pith2026250805976,
  author       = {Pith},
  title        = {Pith review of: PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNQDTVEA}},
  note         = {Machine review of arXiv:2508.05976}
}
read the original abstract

The fragmentation between high-level task semantics and low-level geometric features remains a persistent challenge in robotic manipulation. While vision-language models (VLMs) have shown promise in generating affordance-aware visual representations, the lack of semantic grounding in canonical spaces and reliance on manual annotations severely limit their ability to capture dynamic semantic-affordance relationships. To address these, we propose Primitive-Aware Semantic Grounding (PASG), a closed-loop framework that introduces: (1) Automatic primitive extraction through geometric feature aggregation, enabling cross-category detection of keypoints and axes; (2) VLM-driven semantic anchoring that dynamically couples geometric primitives with functional affordances and task-relevant description; (3) A spatial-semantic reasoning benchmark and a fine-tuned VLM (Qwen2.5VL-PA). We demonstrate PASG's effectiveness in practical robotic manipulation tasks across diverse scenarios, achieving performance comparable to manual annotations. PASG achieves a finer-grained semantic-affordance understanding of objects, establishing a unified paradigm for bridging geometric primitives with task semantics in robotic manipulation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Satr: Zero-shot semantic segmentation of 3d shapes, 2023

    Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes, 2023

    work page 2023

  3. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    work page 2022

  5. [5]

    Claude 3.5: Technical overview, 2024

    Anthropic. Claude 3.5: Technical overview, 2024

    work page 2024

  6. [6]

    Zs6d: Zero-shot 6d object pose estimation using vision transformers

    Philipp Ausserlechner, David Haberger, Stefan Thalhammer, Jean-Baptiste Weibel, and Markus Vincze. Zs6d: Zero-shot 6d object pose estimation using vision transformers. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 463--469, 2024

    work page 2024

  7. [7]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025

    2025

  8. [8]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

Show all 67 references
  1. [9]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  2. [10]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 144...

    2024

  3. [11]

    Autotamp: Autoregressive task and motion planning with llms as translators and checkers

    Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. Autotamp: Autoregressive task and motion planning with llms as translators and checkers. In 2024 IEEE International conference on robotics and automation (ICRA), pages 6695--6702. IEEE, 2024 b

    2024

  4. [12]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objavers...

    2023

  5. [13]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  6. [14]

    Task and motion planning with large language models for object rearrangement

    Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2086--2092. IEEE, 2023

    2023

  7. [15]

    Manipulate-anything: Automating real-world robots using vision-language models, 2024

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models, 2024

    2024

  8. [16]

    A vision-language-guided robotic action planning approach for ambiguity mitigation in human--robot collaborative manufacturing

    Junming Fan and Pai Zheng. A vision-language-guided robotic action planning approach for ambiguity mitigation in human--robot collaborative manufacturing. Journal of Manufacturing Systems, 74: 0 1009--1018, 2024

    2024

  9. [17]

    Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers, 2021

    Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers, 2021

    2021

  10. [18]

    Physically grounded vision-language models for robotic manipulation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462--12469. IEEE, 2024

    2024

  11. [19]

    Integrated task and motion planning

    Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom \'a s Lozano-P \'e rez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4 0 (1): 0 265--293, 2021

    2021

  12. [20]

    Recent trends in task and motion planning for robotics: A survey

    Huihui Guo, Fan Wu, Yunchuan Qin, Ruihui Li, Keqin Li, and Kenli Li. Recent trends in task and motion planning for robotics: A survey. ACM Computing Surveys, 55 0 (13s): 0 1--36, 2023

    2023

  13. [21]

    Multi-modal latent space learning for chain-of-thought reasoning in language models

    Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain-of-thought reasoning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18180--18187, 2024

    2024

  14. [22]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models

    Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9488--9495. IEEE, 2024 a

    2024

  15. [23]

    Language is not all you need: Aligning perception with language models

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36: 0 72096--72...

    2023

  16. [24]

    Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023 b

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023 b

    2023

  17. [25]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024 b

    2024 arXiv

  18. [26]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  19. [27]

    Smart-llm: Smart multi-agent robot task planning using large language models

    Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min. Smart-llm: Smart multi-agent robot task planning using large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12140--12147. IEEE, 2024

    2024

  20. [28]

    Partstad: 2d-to-3d part segmentation task adaptation, 2024

    Hyunjin Kim and Minhyuk Sung. Partstad: 2d-to-3d part segmentation task adaptation, 2024

    2024

  21. [29]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision ...

    2023

  22. [30]

    Openpifpaf: Composite fields for semantic keypoint detection and spatio-temporal association

    Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Openpifpaf: Composite fields for semantic keypoint detection and spatio-temporal association. IEEE Transactions on Intelligent Transportation Systems, 23 0 (8): 0 13498--13511, 2022

    2022

  23. [31]

    Semantic-sam: Segment and recognize anything at any granularity

    Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767, 2023 a

    2023 arXiv

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR, 2023 b

    2023

  25. [33]

    Uni3d-llm: Unifying point cloud perception, generation and editing with large language models, 2024 a

    Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models, 2024 a

    2024

  26. [34]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024 b

    2024

  27. [35]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  28. [36]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019

    2019

  29. [37]

    Latent space translation via semantic alignment

    Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodol \`a . Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 36: 0 55394--55414, 2023

    2023

  30. [38]

    Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders

    Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and St \'e phane Marchand-Maillet. Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. ACM Transactions on Multimedia Computing, Communications, and Applicat...

    2021

  31. [39]

    Robotwin: Dual-arm robot benchmark with generative digital twins (early version), 2024

    Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (early version), 2024

    2024

  32. [40]

    Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024

    Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024

    2024

  33. [41]

    Lip-llm: Integrating linear programming and dependency graph with large language models for multi-robot task planning

    Kazuma Obata, Tatsuya Aoki, Takato Horii, Tadahiro Taniguchi, and Takayuki Nagai. Lip-llm: Integrating linear programming and dependency graph with large language models for multi-robot task planning. IEEE Robotics and Automation Letters, 2024

    2024

  34. [42]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023

    2023

  35. [43]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024

    2024

  36. [44]

    Dinov2: Learning robust visual features without supervision, 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabba...

    2024

  37. [45]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE Internat...

    2024

  38. [46]

    Omnimanip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints, 2025

    Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. Omnimanip: Towards general robotic manipulation via object-centric interaction primitives as spatial constraints, 2025

    2025

  39. [47]

    Cross-modal transformers for infrared and visible image fusion

    Seonghyun Park, An Gia Vien, and Chul Lee. Cross-modal transformers for infrared and visible image fusion. IEEE Transactions on Circuits and Systems for Video Technology, 34 0 (2): 0 770--785, 2024

    2024

  40. [48]

    Derpanis, and Kostas Daniilidis

    Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G. Derpanis, and Kostas Daniilidis. 6-dof object pose from semantic keypoints. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2011--2018, 2017

    2017

  41. [49]

    Sofar: Language-grounded orientation bridges spatial reasoning and object manipulation, 2025

    Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. Sofar: Language-grounded orientation bridges spatial reasoning...

    2025

  42. [50]

    Learning transferable visual models from natural language supervision, 2021 a

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021 a

    2021

  43. [51]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (IC...

    2021

  44. [52]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automatio...

    2023

  45. [53]

    Functo: Function-centric one-shot imitation learning for tool manipulation, 2025

    Chao Tang, Anxing Xiao, Yuhong Deng, Tianrun Hu, Wenlong Dong, Hanbo Zhang, David Hsu, and Hong Zhang. Functo: Function-centric one-shot imitation learning for tool manipulation, 2025

    2025

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  47. [55]

    Llm-based robot task planning with exceptional handling for general purpose service robots

    Ruoyu Wang, Zhipeng Yang, Zinan Zhao, Xinyan Tong, Zhi Hong, and Kun Qian. Llm-based robot task planning with exceptional handling for general purpose service robots. In 2024 43rd Chinese Control Conference (CCC), pages 4439--4444. IEEE, 2024 a

    2024

  48. [56]

    Llmˆ 3: Large language model-based task and motion planning with motion failure reasoning

    Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, and Hangxin Liu. Llmˆ 3: Large language model-based task and motion planning with motion failure reasoning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1208...

    2024

  49. [57]

    Optimizing alignment of speech and language latent spaces for end-to-end speech recognition and understanding

    Wei Wang, Shuo Ren, Yao Qian, Shujie Liu, Yu Shi, Yanmin Qian, and Michael Zeng. Optimizing alignment of speech and language latent spaces for end-to-end speech recognition and understanding. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Pro...

    2022

  50. [58]

    Foundationpose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17868--17879, 2024

    2024

  51. [59]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

    2023

  52. [60]

    Manipose: A comprehensive benchmark for pose-aware object manipulation in robotics, 2024

    Qiaojun Yu, Ce Hao, Junbo Wang, Wenhai Liu, Liu Liu, Yao Mu, Yang You, Hengxu Yan, and Cewu Lu. Manipose: A comprehensive benchmark for pose-aware object manipulation in robotics, 2024

    2024

  53. [61]

    Unsupervised learning of 3d semantic keypoints with mutual reconstruction

    Haocheng Yuan, Chen Zhao, Shichao Fan, Jiaxi Jiang, and Jiaqi Yang. Unsupervised learning of 3d semantic keypoints with mutual reconstruction. In Computer Vision -- ECCV 2022, pages 534--549, Cham, 2022. Springer Nature Switzerland

    2022

  54. [62]

    Robopoint: A vision-language model for spatial affordance prediction for robotics

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  55. [63]

    Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers

    Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE Transactions on intelligent transportation systems, 24 0 (12): 0 14679--14694, 2023

    2023

  56. [64]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  57. [65]

    3d-vla: A 3d vision-language-action generative world model, 2024

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model, 2024

    2024

  58. [66]

    Open3d: A modern library for 3d data processing, 2018

    Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing, 2018

    2018

  59. [67]

    Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning

    Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2081--2088. IEEE, 2024

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.