Pith. sign in

REVIEW 4 major objections 6 minor 115 references

A multi-agent coding harness can assemble and self-refine modular robot computation graphs that outperform model-free policies on variational automation tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 07:10 UTC pith:6TFVCMZ3

load-bearing objection Solid systems paper: multi-agent graph harness + sim self-learning + MORSL beats VLAs/CaP/TAMP on new VA benchmarks, but the big gaps partly reflect privileged geometry and skills rather than the harness alone. the 4 major comments →

arxiv 2607.05369 v1 pith:6TFVCMZ3 submitted 2026-07-06 cs.RO cs.AIcs.CLcs.LG

GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Kaiyuan Chen , Shuangyu Xie , Letian Fu , Justin Yu , William Pacini , Sandeep Bajamahal , Hudson Kim , Jaimyn Drake
show 16 more authors
Daehwa Kim Haoru Xue Jonathan Francis Christian Juette Peter Schaldenbrand Muhammet Yunus Seker Ruwan Wickramarachchi Uksang Yoo Guanzhi Wang Adithyavairavan Murali Balakumar Sundaralingam S. Shankar Sastry Spencer Huang Yuke Zhu Linxi "Jim" Fan Ken Goldberg
This is my paper
classification cs.RO cs.AIcs.CLcs.LG
keywords Variational AutomationGraph-as-Policyagentic codingself-learningrobot computation graphsskill librariesTask and Motion Planningmulti-agent systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Commercial robots need policies that run for long periods on tasks with real variation in object geometry and pose—not fixed factory motions and not open-ended household generalism. The paper calls this class Variational Automation and argues that end-to-end model-free policies often fail under that variation while classical programming is too expensive to retune for every instance distribution. Graph-as-Policy answers by having multiple coding agents build a directed graph of perception, planning, and control nodes from an open skill library, then rehearse many sampled instances in simulation and rewrite graph structure and parameters until success rate and throughput stop improving. On eight new open benchmarks (four simulated, four physical)—grocery order fulfillment and packing, making popcorn, USB-C cable insertion, and dual-arm crate washing—the refined graphs reach high success rates that substantially exceed single-agent code generation, modular planners, and finetuned vision-language-action models. A sympathetic reader cares because the approach keeps policies interpretable and composable while still using agents and learned skills where they help, aiming to cut the human setup cost of reliable automation.

Core claim

Representing a robot policy as a directed computation graph of modular skills, authored by a multi-agent coding harness and iteratively refined by parallel simulation rehearsal with failure analysis, yields persistent policies for variational automation whose success rates significantly outperform free-form coding agents, modular task-and-motion planners, and vision-language-action baselines under large pose and object variation.

What carries the argument

Graph-as-Policy (GaP): a policy is a directed computation graph whose nodes are atomic skills from the Modular Open Robot Skill Library (MORSL). An orchestration multi-agent system decomposes a language task into subgraphs, wires them, then runs self-learning—sample belief-space instances, parallel sim rollouts, analyze contacts and state differences, update graph topology and parameters—until a weighted success-and-throughput objective plateaus; the resulting graph is interpreted on the edge without further agents.

Load-bearing premise

The workcell, robot, sensors, object models, and the distribution of instance poses are already known and fixed, so the system only has to handle variation inside that closed operating envelope.

What would settle it

On grocery fulfillment and packing with mixed large pose variation, if GaP success collapsed to the level of the vision-language-action baselines (near 0.2 over the same hundred-instance cells) while those baselines remained high, or if make-popcorn self-learning gains in simulation failed to transfer so real success stayed near the initial one-third rather than near 90 percent, the central reliability claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Structured multi-agent graph authoring with static validation can avoid the total collapse to zero success that free-form single-agent script generation shows on the same tasks.
  • Graph-based approach and camera-centering steps can more than double vision-language-action success under pose variation by bringing wrist-camera observations into distribution before handing off control.
  • Self-learning from simulation contact and state feedback can raise a long-horizon kitchen graph from roughly one-third to over 90 percent success under pan pose variation, with transfer to physical trials.
  • The same harness can emit ROS-compatible force-aware insertion workflows and dual-arm coordination graphs whose reliability approaches a hand-engineered industrial baseline on crate washing.
  • Open variational benchmarks with controlled larger pose variation expose reliability gaps that near-fixed-pose evaluations hide.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • As skill libraries grow, hierarchical multi-agent graph harnesses may become the practical bridge between classical automation engineering and learned skills, rather than pure end-to-end generalists for factory-like settings.
  • Sim-to-real fidelity of contact and state feedback is the hinge of self-learning; sites without usable digital twins may need physical rehearsal or the loop will optimize the wrong failures.
  • Cycle time remains far below industrial unit-per-hour targets; cutting vision-model and motion-planning latency at execution may matter as much as raising success rate for adoption.
  • Adding force-rich, deformable, and moving-target nodes would test whether the same authoring-and-rehearsal loop covers a larger share of industrial work beyond quasi-static pick-and-place.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Graph-as-Policy (GaP), a multi-agent LLM coding harness that synthesizes directed computation graphs of perception, planning, and control nodes drawn from a Modular Open Robot Skill Library (MORSL), then refines graph structure and parameters via parallel Isaac-Lab rehearsal (Algorithm 1). It defines a Variational Automation (VA) task class with known workcell, robot/sensors, object models, and pose belief (tuple T in §3), and evaluates on eight new open benchmarks (grocery fulfill/pack, make popcorn, USB-C insertion, dual-arm crate wash; 4 sim / 4 real). Reported results claim large gains over CaP-X, π0.5, MolmoAct2, and TipTop (e.g., Table 1: 0.93–0.99 vs ~0.20 under large pose variation; real make-popcorn 18/20; cable insertion 121/130; crate wash 0.95 matching a hand-engineered graph), with ablations that graphless or single-agent generation collapses to zero.

Significance. If the results hold under fairer controls, GaP is a practical bridge between classical modular stacks (ROS/TAMP-style graphs) and agentic coding for commercial-style tasks with bounded geometric variation. Concrete strengths include: large sim trial counts (100 instances per cell, >5000 trials for grocery tables), real-robot tables, explicit ablations of graph structure and multi-agent authoring, a documented self-learning case study on make-popcorn (33%→94% sim), hybrid GaP+VLA rows showing scaffolding value, an open MORSL catalog (51 skills), and promised code/benchmarks. The VA framing and open benchmarks are useful even if the absolute reliability/throughput remain below industrial targets (as the Limitations section acknowledges).

major comments (4)
  1. The headline claim that GaP “significantly outperform[s]” CaP-X, π0.5, MolmoAct2, and TipTop (Abstract; §5.2–5.3; Table 1) is not cleanly isolated to the multi-agent graph harness. §5.2 itself states CaP-X is “not a fair comparison” (single agent, image+language only, no self-learning). VLAs are LIBERO-finetuned without GaP’s known E/R/O/B, MORSL skills, or interactive-perception centering; TipTop failures are attributed largely to M2T2/cuRobo/cuTAMP planning limits on cubic/tall objects that GaP’s skill stack handles. Ablations in §5.4 show graphless/single-agent collapse to zero inside GaP’s stack, but that does not show the harness—not privileged models, skills, and sim edits—drives the 0.93–0.99 vs ~0.20 gap. A load-bearing revision is either (i) baselines given the same O/B/MORSL/sim loop, or (ii) a carefully scoped claim that GaP is an effective integration/self-refinement system u
  2. Self-learning (Algorithm 1; §4.2; Fig. 2) is central to the paper’s contribution narrative, yet evidence is concentrated on Make Popcorn (initial ~33% → 94% sim after ~10 iterations; 18/20 real). §5.2 states that for Benchmarks I–II the first generated graph already achieves high performance, so self-learning is not exercised. Without multi-task statistics on when GraphUpdate changes topology vs parameters, how often sim-improved G* transfers, and failure modes of AnalyzeFailure under contact/state feedback, the claim that parallel rehearsal “iteratively refine[s] the graph… to improve success rates and throughput” remains under-supported outside one long-horizon example.
  3. §3 Assumptions and the VA tuple T=⟨L,E,R,O,X,B,J⟩ grant known fixed workcell, robot/sensor config, object models (including articulated joints), and a known belief B over instance poses. The weakest assumption for deployment is that Isaac-Lab state/contact feedback is faithful enough that graph edits raising sim success transfer to real hardware. Real N is modest relative to sim (Table 2: 25/25, 28/30, 18/20; Table 3: 130 insertions; Table 4: 150 crate trials). The manuscript should quantify sim-to-real discrepancy for self-refined graphs (e.g., make-popcorn IK/kinematic failures) and state more clearly which results use self-learning vs first-shot generation, so readers can judge generalization under misspecified B or imperfect sim.
  4. Reward J = ws·I(success) + wt·Φ (§3) and free parameters (ws, wt; M, N; cable force/depth thresholds in §5.6; LLM choice/temperature 0.1) are not systematically ablated. Throughput claims (Table 4: ~18 successes/hr; Limitations: far below ~500 units/hr industrial targets) depend on cycle time dominated by VLM/IK. Without sensitivity of G* to J weights and rehearsal budget, it is hard to assess robustness of the optimization objective that defines G* = arg max_G E[J].
minor comments (6)
  1. Figure 1 and several captions use corrupted glyphs (e.g., “V A”, “Pck”, “tributinterpreted”); clean for camera-ready.
  2. Table 1 header/column labels mix LIBERO, LIBERO-Pro, and new VA variants; a short legend defining “mixed all” and packing success (items/6 after 6 attempts) in the table caption would help.
  3. Related Work repeats CaP/self-improving agent paragraphs almost verbatim between §2 and Appendix A; consolidate.
  4. MORSL is said to have “51 initial skills” (§1, §4.1) while Appendix D lists many primitives; a single inventory table mapping composite skills to gRPC methods would improve reproducibility.
  5. Notation: belief B and instance sampling xi∼p(x|X) (§3) vs later {ŝi}∼B in Algorithm 1; keep symbols consistent.
  6. Project URL and “code and data will be posted” should be checked for availability at review time; if not yet public, provide an anonymized artifact or clear release plan.

Circularity Check

0 steps flagged

No circular derivation: GaP is an empirical systems paper whose success rates are measured against external task criteria and independent baselines, not forced by construction from fitted inputs or self-citation.

full rationale

The load-bearing claim is empirical outperformance of GaP-generated graphs on eight VA benchmarks (Tables 1–4), not a first-principles derivation. Success is defined by external task outcomes (item in basket, pan covering burner, USB-C insertion depth/force, dual-arm crate placement) via J = w_s·I(success) + w_t·Φ and by comparison to CaP-X, π0.5, MolmoAct2, TipTop, and a hand-engineered crate graph. Self-learning (Alg. 1) samples instances from belief B, rolls out in Isaac-Lab, and edits graph topology/parameters from contact/state feedback; that is iterative optimization against physical sim outcomes, not redefinition of success as the graph itself. MORSL skills (GraspGen, cuRobo, DINO/SAM, ROS nodes) are library components, including some from overlapping NVIDIA coauthors; using prior tools as nodes is normal engineering and does not make reported success rates tautological. Ablations (graphless/single-agent → zero) show structure helps inside the stack but do not close a circular loop. No equation equates a claimed prediction to a fitted input by construction; no uniqueness theorem is imported to forbid alternatives. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 4 invented entities

The central claim rests on domain assumptions that define VA (known cell and object set), on the adequacy of modular skill libraries and LLM coding agents, and on simulation as a valid rehearsal oracle. Free parameters are mostly engineering thresholds and reward weights, not a single fitted constant that forces the headline success rates. Invented entities are methodological (graph-as-policy, harness, MORSL, VA class), not new physical particles.

free parameters (4)
  • Reward weights ws, wt in J = ws·I(success) + wt·Φ
    Multi-objective tradeoff between success and throughput used to drive graph optimization; values not uniquely derived.
  • Self-learning iterations M and parallel rollouts N
    Algorithm 1 hyperparameters that control how long/how widely graphs are refined before plateau.
  • Cable-insertion force/depth thresholds (e.g. >3 mm at <10 N; success depth >6 mm at 30 N; extract lateral <20 N)
    Hand-set contact criteria that define success for Benchmark IV policy nodes.
  • LLM temperature 0.1 and model choice (Gemini-3.1-Flash-Lite)
    Generation stochasticity and capability are fixed by experiment setup; results depend on this agent stack.
axioms (5)
  • domain assumption VA tasks have known fixed workcell E, robot/sensors R, object set O with models, and known belief B over instance poses.
    Section 3 Assumptions; without this, graph generation and sim rehearsal as described do not apply to open generalist settings.
  • domain assumption A robot policy can be represented as a directed computation graph of typed skill nodes with data/control edges (ROS/TAMP-inspired).
    Problem Formulation and Section 4; load-bearing design choice that enables multi-agent decomposition and validation.
  • domain assumption Isaac-Lab (or equivalent) physics, contacts, and rendered observations provide failure signals sufficient to improve real-world graphs.
    Self-learning Algorithm 1 and make-popcorn transfer; sim-to-real is assumed workable for the reported tasks.
  • domain assumption MORSL skills expose correct typed I/O contracts and implement the named perception/planning/control behaviors.
    Graph correctness depends on skill library fidelity; ablations show unstructured code fails structural checks.
  • standard math Standard probability and SE(3) geometry for sampling xi ~ B and evaluating success indicators.
    Used in formal task tuple and expectation G* = arg max_G E[J].
invented entities (4)
  • Variational Automation (VA) task class no independent evidence
    purpose: Position the problem between fixed automation and generalist robotics with bounded variation.
    New framing with formal tuple T; independent evidence is the eight benchmarks, not external prior definition.
  • Graph-as-Policy (GaP) multi-agent harness no independent evidence
    purpose: Generate, validate, and self-refine robot computation graphs from language + MORSL.
    Core system contribution; evidence is empirical performance, not a prior named system.
  • Modular Open Robot Skill Library (MORSL) no independent evidence
    purpose: Provide 51 typed atomic/composite skills for agents to wire into graphs.
    Library is introduced and catalogued in the paper/appendix; skills wrap prior models (SAM, GraspGen, cuRobo) but the library contract is new.
  • Eight open VA benchmarks (grocery fulfill/pack, make popcorn, cable insert, wash crates) no independent evidence
    purpose: Evaluate reliability under pose/geometry variation in sim and real.
    New evaluation suite; some assets from LIBERO but variations and tasks are paper-defined.

pith-pipeline@v1.1.0-grok45 · 43985 in / 3831 out tokens · 36565 ms · 2026-07-11T07:10:46.902892+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks." pith.science (2026). https://pith.science/paper/6TFVCMZ3

@misc{pith2026260705369,
  author       = {Pith},
  title        = {Pith review of: GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TFVCMZ3}},
  note         = {Machine review of arXiv:2607.05369}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

For robots to work reliably in commercial and industrial applications, can recent advances in agentic coding systems combine interpretable robot programming with the open-world adaptability of model-free policies? We focus on "Variational Automation" (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation. Model-free policies often struggle to close the reliability gap for VA tasks, which must be executed persistently and reliably in commercial and industrial applications. Motivated by prior work on Task and Motion Planning (TAMP) and the Robot Operating System (ROS), we introduce Graph-as-Policy (GaP), a multi-agent coding harness that generates directed computation graphs with perception, planning, and control nodes from a Modular Open Robot Skill Library (MORSL). GaP then generates an internal simulation environment to rehearse task instances with different graphs in parallel to iteratively refine the graph structure and parameters to improve success rates and throughput. Evaluation with 8 new open VA task benchmarks, 4 in-simulation and 4 in real-world, suggests that GaP can achieve success rates that significantly outperform baselines. Details, code, and data can be found online: https://graph-robots.github.io/gap

Figures

Figures reproduced from arXiv: 2607.05369 by Adithyavairavan Murali, Balakumar Sundaralingam, Christian Juette, Daehwa Kim, Guanzhi Wang, Haoru Xue, Hudson Kim, Jaimyn Drake, Jonathan Francis, Justin Yu, Kaiyuan Chen, Ken Goldberg, Letian Fu, Linxi "Jim" Fan, Muhammet Yunus Seker, Peter Schaldenbrand, Ruwan Wickramarachchi, Sandeep Bajamahal, Shuangyu Xie, Spencer Huang, S. Shankar Sastry, Uksang Yoo, William Pacini, Yuke Zhu.

Figure 1
Figure 1. Figure 1: GaP system architecture. Given a “Variational Automation (VA)” task specification, GaP uses a multi-agent harness for coding tools such as Claude and Gemini to automatically generate computation graphs that include “skill” nodes from a Modular Open Robot Skill Library (MORSL), which includes model-based procedures (e.g. ROS [1]) and model-free policies (e.g. GraspGen [2]). GaP then orchestrates self-learni… view at source ↗
Figure 2
Figure 2. Figure 2: Self-Learning for Making Popcorn Benchmark. Pan pose variations (left) drives an 10-iteration sequence graph update ; (blue, left axis) iteration phases shaded by class of edit. The Make Popcorn task requires the robot to grasps the stove knob and rotate it to turn on the burner, then the robot must find and pick up the handle of the JiffyPop popcorn pan, place it on the stove burner, wait, and then turn o… view at source ↗
Figure 3
Figure 3. Figure 3: Seven-ports cable insertion setup [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: ROS computation graph that is hand-engineered using traditional ROS nodes and topics. Middle: [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

115 extracted references · 1 canonical work pages

  1. [1]

    Macenski, T

    S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall. Robot operating system 2: Design, architecture, and uses in the wild.Science robotics, 7(66):eabm6074, 2022

    2022

  2. [2]

    Murali, B

    A. Murali, B. Sundaralingam, Y .-W. Chao, J. Yamada, W. Yuan, M. Carlson, F. Ramos, S. Birch- field, D. Fox, and C. Eppner. Graspgen: A diffusion-based framework for 6-dof grasping with on-generator training. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026

    2026

  3. [3]

    Isaac Sim, 2025

    NVIDIA. Isaac Sim, 2025. URL https://github.com/isaac-sim/IsaacSim. Version 5.1.0

    2025

  4. [4]

    M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    P. I. Team.π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    O. X.-E. Co-Authors. Open X-Embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024

    2024

  7. [7]

    H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W.-C. Tsai, S. Chen, Y . R. Wang, et al. Molmoact2: Action reasoning models for real-world deployment.arXiv preprint arXiv:2605.02881, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Black, N

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fu- sai, M. Y . Galliker, et al.π0.5: a vision-language-action model with open-world generalization. In9th Annual Conference on Robot Learning, 2025

    2025

  9. [9]

    Zawalski, W

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine. Robotic control via embodied chain-of-thought reasoning. InConference on Robot Learning, pages 3157–3181. PMLR, 2025

    2025

  10. [10]

    J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh. A taxonomy for evaluating generalist robot manipulation policies.IEEE Robotics and Automation Letters (RA-L), 2026

    2026

  11. [11]

    Goldberg

    K. Goldberg. Should robot generalists get off their high horse?, 2026. Available online. 11

    2026

  12. [12]

    Solowjow, I

    E. Solowjow, I. Ugalde, Y . Shahapurkar, J. Aparicio, J. Mahler, V . Satish, K. Goldberg, and H. Claussen. Industrial robot grasping with deep learning using a programmable logic controller (plc). In2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 97–103, 2020. doi:10.1109/CASE48305.2020.9216902

    doi:10.1109/case48305.2020.9216902 2020

  13. [13]

    Adebola, T

    S. Adebola, T. Sadjadpour, K. El-Refai, W. Panitch, Z. Ma, R. Lin, T. Qiu, S. Ganti, C. Le, J. Drake, and K. Goldberg. Automating deformable gasket assembly. In2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), pages 4146–4153. IEEE, 2024

    2024

  14. [14]

    S. Xie, K. Goldberg, and D. Song. Energy efficient planning for repetitive heterogeneous tasks in precision agriculture. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7139–7145, 2025. doi:10.1109/ICRA55743.2025.11128083

    doi:10.1109/icra55743.2025.11128083 2025

  15. [15]

    Adebola, R

    S. Adebola, R. Parikh, M. Presten, S. Sharma, S. Aeron, A. Rao, S. Mukherjee, T. Qu, C. Wistrom, E. Solowjow, and K. Goldberg. Can machines garden? systematically comparing the alphagarden vs. professional horticulturalists. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11779–11785, 2023. doi:10.1109/ICRA48891.2023. 10161497

    doi:10.1109/icra48891.2023 2023

  16. [16]

    Q. Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors,Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 ofProceedings of Machine Lear...

    2021

  20. [20]

    Bordes, R

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma ˜nas, Z. Lin, A. Mah- moud, B. Jayaraman, et al. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    G. R. Team, S. Abeyruwan, et al. Gemini robotics: Bringing ai into the physical world, 2025. URLhttps://arxiv.org/abs/2503.20020

    Pith/arXiv arXiv 2025

  22. [22]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. InConference on Robot Learning, pages 540–562. PMLR, 2023

    2023

  23. [23]

    C. Ning, K. Fang, and W.-C. Ma. Prompting with the future: Open-world model predictive control with interactive digital twins. InProceedings of Robotics: Science and Systems (RSS), 2025

    2025

  24. [24]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/product/claude-code, June 2024

    2024

  25. [25]

    K. Chen, S. Xie, Z. Ma, P. R. Sanketi, and K. Goldberg. Robo2vlm: Improving visual question answering using large-scale robot manipulation data. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

  26. [26]

    Y . Ji, H. Tan, J. Shi, X. Hao, Y . Zhang, H. Zhang, P. Wang, M. Zhao, Y . Mu, P. An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete.CVPR, 2025. 12

    2025

  27. [27]

    Harness design for long-running application development

    Anthropic. Harness design for long-running application development. https://www. anthropic.com/engineering/harness-design-long-running-apps, Mar. 2026

    2026

  28. [28]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  29. [31]

    M. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, F.-F. Li, G. Shi, et al. Cap-x: A framework for benchmarking and improving coding agents for robot manipulation. arXiv preprint arXiv:2603.22435, 2026

    Pith/arXiv arXiv 2026

  30. [32]

    L. P. Kaelbling and T. Lozano-P´erez. Hierarchical task and motion planning in the now. In2011 IEEE international conference on robotics and automation, pages 1470–1477. IEEE, 2011

    2011

  31. [33]

    C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez. Integrated task and motion planning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    2021

  32. [34]

    W. Shen, N. Kumar, S. Chintalapudi, J. Wang, C. Watson, E. S. Hu, J. Cao, D. Jayaraman, L. P. Kaelbling, and T. Lozano-P ´erez. TiPToP: A modular open-vocabulary planning system for robotic manipulation.arXiv preprint arXiv:2603.09971, 2026

    arXiv 2026

  33. [35]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36: 44776–44791, 2023

    2023

  34. [36]

    Levine, C

    S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016

    2016

  35. [37]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  36. [38]

    N. J. Nilsson.Artificial intelligence: a new synthesis. Elsevier, 1998

    1998

  37. [39]

    N. J. Nilsson.Principles of artificial intelligence. Morgan Kaufmann, 2014

    2014

  38. [40]

    Bucker, P

    A. Bucker, P. Ortega-Kral, J. Francis, and J. Oh. Grappa: Generalizing and adapting robot policies via online agentic guidance.IEEE Robotics and Automation Letters, 2026

    2026

  39. [41]

    J. Shi, R. Yang, K. Chao, B. S. Wan, Y . S. Shao, J. Lei, J. Qian, L. Le, P. Chaudhari, K. Daniilidis, et al. Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots. InNeurIPS 2025 Workshop on Space in Vision, Language, and Embodied AI, 2025

    2025

  40. [42]

    S. Wang, M. Han, Z. Jiao, Z. Zhang, Y . N. Wu, S.-C. Zhu, and H. Liu. LLM3: Large Language Model-based Task and Motion Planning with Motion Failure Reasoning. In2024 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 12086–12092. IEEE, 2024

    2024

  41. [43]

    Curtis, N

    A. Curtis, N. Kumar, J. Cao, T. Lozano-P´erez, and L. P. Kaelbling. Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. InConference on Robot Learning, pages 1362–1383. PMLR, 2025. 13

    2025

  42. [44]

    Z. Yang, C. Garrett, D. Fox, T. Lozano-P´erez, and L. P. Kaelbling. Guiding long-horizon task and motion planning with vision language models. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16847–16853. IEEE, 2025

    2025

  43. [45]

    Kumar, W

    N. Kumar, W. Shen, F. Ramos, D. Fox, T. Lozano-P´erez, L. P. Kaelbling, and C. R. Garrett. Open-world task and motion planning via vision-language model generated constraints.IEEE Robotics and Automation Letters, 2026

    2026

  44. [46]

    N. Cote, J. Drake, and S. Chitta. Agentic language-grounded adaptive robotic assembly. In The first CVPR workshop on 3D Vision Language Models (VLMs) for Robotics Manipulation: Opportunities and Challenges, 2024

    2024

  45. [47]

    G. Yin, Y . Li, Y . Wang, D. Mcconachie, P. Shah, K. Hashimoto, H. Zhang, K. Liu, and Y . Li. CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity. InProceedings of Robotics: Science and Systems, LosAngeles, CA, USA, June

  46. [48]

    doi:10.15607/RSS.2025.XXI.072

    work page doi:10.15607/rss.2025.xxi.072 2025

  47. [49]

    B. Wu, A. Jones, A. Renault, H. Tay, J. Noble, N. Picard, S. Jiang, et al. Introducing advanced tool use on the claude developer platform, 2024

    2024

  48. [50]

    X. Hou, Y . Zhao, S. Wang, and H. Wang. Model context protocol (mcp): Landscape, security threats, and future research directions.ACM Transactions on Software Engineering and Methodology, 2025

    2025

  49. [51]

    GPT-4o System Card

    OpenAI. GPT-4o System Card. https://openai.com/index/gpt-4o-system-card/ , Aug. 2024

    2024

  50. [52]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , June 2024

    2024

  51. [53]

    Kavukcuoglu

    K. Kavukcuoglu. Gemini 2.5: Our most intelligent AI model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , Mar. 2025

    2025

  52. [54]

    Zhang, J

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, et al. Aflow: Automating agentic workflow generation. InInternational Conference on Learning Representations, volume 2025, pages 34040–34077, 2025

    2025

  53. [55]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    2024

  54. [56]

    Zehle, T

    T. Zehle, T. Heiß, M. Schlager, M. Aßenmacher, and M. Feurer. promptolution: A unified, mod- ular framework for prompt optimization. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 282–296, 2026

    2026

  55. [57]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. InFirst Workshop on Foundations of Reasoning in Language Models, 2025

    2025

  56. [58]

    Y . Lee, J. Boen, and C. Finn. Feedback descent: Open-ended text optimization via pairwise comparison.arXiv preprint arXiv:2511.07919, 2025

    arXiv 2025

  57. [59]

    Goldberg, K

    A. Goldberg, K. Kondap, T. Qiu, Z. Ma, L. Fu, J. Kerr, H. Huang, K. Chen, K. Fang, and K. Goldberg. Blox-net: Generative design-for-robot-assembly using vlm supervision, physics, simulation, and a robot with reset. In2025 International Conference on Robotics and Automation (ICRA). IEEE, 2025. 14

    2025

  58. [60]

    Equipping agents for the real world with Agent Skills

    Anthropic. Equipping agents for the real world with Agent Skills. https://www.anthropic. com/engineering/equipping-agents-for-the-real-world-with-agent-skills , Oct. 2025

    2025

  59. [61]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. InInternational Conference on Learning Representations, volume 2025, pages 28085–28128, 2025

    2025

  60. [62]

    Carion, L

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. R¨adle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Doll ´ar, N. Ravi, K. ...

    Pith/arXiv arXiv 2025

  61. [63]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  62. [64]

    Minderer, A

    M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European conference on computer vision, pages 728–755. Springer, 2022

    2022

  63. [65]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 91–104. IEEE, 2025

    2025

  64. [66]

    Kamath, J

    A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram´e, M. Rivi`ere, L. Rouillard, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 4, 2025

    Pith/arXiv arXiv 2025

  65. [67]

    Sundermeyer, A

    M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In2021 IEEE international conference on robotics and automation (ICRA), pages 13438–13444. IEEE, 2021

    2021

  66. [68]

    W. Yuan, A. Murali, A. Mousavian, and D. Fox. M2t2: Multi-task masked transformer for object-centric pick and place. In7th Annual Conference on Robot Learning, 2023

    2023

  67. [69]

    Sundaralingam, S

    B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V . Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8112–8119. IEEE, 2023

    2023

  68. [70]

    Sundaralingam, A

    B. Sundaralingam, A. Murali, and S. Birchfield. curobov2: Dynamics-aware motion generation with depth-fused distance fields for high-dof robots.arXiv preprint arXiv:2603.05493, 2026

    Pith/arXiv arXiv 2026

  69. [71]

    X. Zhou, Y . Xu, G. Tie, Y . Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization.[arXiv preprint arXiv:2510.03827], 2025

    Pith/arXiv arXiv 2025

  70. [72]

    Minkowski.Geometrie der Zahlen

    H. Minkowski.Geometrie der Zahlen. B.G. Teubner, Leipzig, 1896

  71. [73]

    W. Shen, C. Garrett, N. Kumar, A. Goyal, T. Hermans, L. P. Kaelbling, T. Lozano-P´erez, and F. Ramos. Differentiable gpu-parallelized task and motion planning. InRobotics science and systems. Robotics; Science and Systems, 2025

    2025

  72. [74]

    A. Pun, K. Deng, R. Liu, D. Ramanan, C. Liu, and J.-Y . Zhu. Generating physically stable and buildable brick structures from text. InICCV, 2025. 15 Appendix Contents A More Related Work 17 B Integrating ROS with GaP in Cable Insertion Benchmark 17 C Sample Generated Graphs 18 C.1 Fulfill Grocery Orders . . . . . . . . . . . . . . . . . . . . . . . . . . ...

    2025

  73. [75]

    orchestrator

    and multi-agent prompt optimization frameworks [55, 56]. To improve code generation quality and performance, CaP-X [ 30] used a VLM to provide feedback before and after execution (i.e. Visual Differencing), but the VLM can suffer from hallucinations and cannot handle geometric and numerical information such as motion feasibility. Building on the structure...

  74. [76]

    The block MUST end with a module-level variable named ‘‘spec‘‘ bound to a ‘‘WorkflowSpec‘‘ instance

  75. [77]

    Imports: ‘‘from vos.builder import WorkflowSpec, START‘‘ and ‘‘from vos.runtime.workflow import ServiceCall‘‘ are pre-provided in the sandbox

  76. [78]

    No I/O, no other imports. 33

  77. [79]

    OrientedBoundingBox

    Subgraph ‘inputs‘ / ‘outputs‘ MUST be dicts of ‘{name: proto_type_str}‘. Each value is a **bare proto type string** (e.g. ‘‘"OrientedBoundingBox"‘‘, ‘‘"Mask"‘‘, ‘‘"PointCloud"‘‘) -- never a ‘‘Ref‘‘ and never a nested dict. Cross-subgraph data flow is established implicitly by ‘add_edge‘ / ‘add_conditional_edges‘ plus matching input/output **names** betwee...

  78. [80]

    done"]‘ and ‘on_error=

    Every ‘declare_subgraph(...)‘ call MUST pass BOTH ‘exit_success_values=‘ AND ‘on_error=‘ as keyword arguments -- they are required and have no defaults. Omitting either raises ‘declare_subgraph() missing 2 required keyword-only arguments‘ and the whole spec rejects. Conventional values: ‘exit_success_values=["done"]‘ and ‘on_error="abort"‘, paired with ‘s...

  79. [81]

    Every end node must be reachable from ‘START‘

    **Edges and conditional_edges.** For each subgraph node, every one of its ‘exit_success_values‘ PLUS its ‘on_error‘ symbol must appear as a key in the corresponding ‘add_conditional_edges‘ mapping, and every mapping target must be a node declared at the top level. Every end node must be reachable from ‘START‘

  80. [82]

    **The entry node must be a subgraph node** (not an end node)

Showing first 80 references.