Pith. sign in

REVIEW 4 major objections 8 minor 49 references

Natural-language corrections plus a growing skill library let an LLM write steerable robot programs that beat black-box VLAs on hard manipulation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 12:36 UTC pith:DAZ22S2P

load-bearing objection Solid systems paper on steerable LLM robot code with a real skill-library story; the headline “beats VLAs” claim rests on an asymmetric adaptation budget the paper under-reports. the 4 major comments →

arxiv 2607.23784 v1 pith:DAZ22S2P submitted 2026-07-26 cs.RO cs.AI

A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

Daphne Chen , Archit Ritesh Jain , Eric Goossen , Emma Romig , Michael Murray , Nick Walker , Maya Cakmak This is my paper
classification cs.RO cs.AI
keywords human in the looplanguage correctionsprogram synthesisskill libraryrobot manipulationvision-language-actionmodular policiesinteractive code generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Black-box vision-language-action models can look capable zero-shot, but when they fail they are hard to interpret, correct, or adapt without expensive new data and retraining. This paper argues that robot policy acquisition can instead be treated as interactive program synthesis: an LLM coding agent writes modular robot programs over a fixed suite of perception and control tools, a human gives short natural-language corrections when execution fails, and those corrections are grounded in execution traces and stored in a persistent skill library. The library acts as long-term in-context memory so later tasks reuse earlier fixes with far less new human input. On a Franka Panda across eight real tasks that include drawer closing, cloth folding, and long-horizon occlusion, the resulting policies outperform Code-as-Policies and leading VLAs without any robot-specific training data, and the skill library measurably amortizes human effort across related novel scenes.

Core claim

Interactive LLM program synthesis guided by sparse natural-language corrections and a persistent skill library produces modular, interpretable robot policies that outperform both one-shot code-synthesis baselines and state-of-the-art vision-language-action models on complex real-world manipulation, while transferring to related tasks with sharply reduced human intervention and no robot demonstration data.

What carries the argument

ARCHITECT's skill library: natural-language corrections are grounded against program execution traces, distilled into reusable skill entries, and reloaded into the LLM's context on every subsequent synthesis so prior fixes become high-level primitives rather than one-off patches.

Load-bearing premise

A fixed generic toolkit of detectors, grasp samplers, placers, and motion planners, plus at most a handful of natural-language corrections, is enough to close the gap between an underspecified instruction and reliable execution for the claimed task classes.

What would settle it

On the same Franka setup and eight-task suite, run the final skill-library policies after at most five expert corrections and check whether success rate, goal-condition recall, and normalized task progression still exceed Code-as-Policies and the π0 / π0.5 baselines, especially on articulated, deformable, and long-horizon tasks; if the library no longer reduces corrections on the held-out human-eval transfer pair, the amortization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Users can steer and repair robot behavior with ordinary language instead of collecting new demonstration datasets or fine-tuning large models.
  • Once a skill library is populated on one task family, related novel objects and scenes require far fewer (sometimes zero) new corrections.
  • Failures become localizable to particular tool calls or code lines rather than cascading as opaque distribution shift inside an end-to-end network.
  • Robot competence can accumulate as an interpretable, inspectable repertoire of skills without robot-specific training data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach trades end-to-end generality for an explicit dependence on the quality of the underlying grasp, depth, and motion modules; better off-the-shelf primitives should raise the ceiling without changing the correction loop.
  • Human corrections appear to encode physical intuitions (contact, slip, clearance) that current vision-language models miss from pixels alone, suggesting hybrid human-plus-VLM feedback schedules as a natural next experiment.
  • If skill libraries remain readable, they could serve as audit trails or safety contracts for deployed manipulators in settings where black-box policies are unacceptable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper presents ARCHITECT, a framework in which an LLM coding agent (Claude Opus 4.6) synthesizes modular robot programs over a fixed tool suite (Grounded SAM2 + AnyGrasp detection/grasping, AnyPlace placement, cuRobo motion, GPT-5.4 VQA, proprioception), and a human supervisor steers the policy through natural-language corrections grounded in execution traces. Corrections are distilled into a persistent skill library that is loaded into context for subsequent tasks. The authors evaluate on a Franka Panda across 8 real manipulation tasks (N=10 rollouts/task, SR/GCR/NTP metrics) against Code as Policies, π0, π0.5 (Table 1) and Inner Monologue, ProgPrompt, MolmoAct2, GR00T N1.7 (Table 2), plus a VLM-correction ablation, a language-perturbation ablation (Table 3), and a within-subjects human study (N=6) measuring skill-library transfer and amortization of human effort. The headline claim is that ARCHITECT outperforms SOTA VLAs and program-synthesis baselines on complex tasks without robot-specific training data, and that the skill library transfers to related tasks with reduced human intervention.

Significance. If the results hold, this is a useful contribution to the growing literature on steerable, modular robot policies: the evaluation is on a real robot (not simulation), spans genuinely hard categories (articulated drawer closing, cloth folding, occlusion-resolving long-horizon tasks), reports partial-credit metrics (GCR/NTP) alongside SR, includes a broad baseline set, and ships concrete artifacts — full system prompts (App. F), tool-suite tables (App. H), rubrics (App. C), and a worked skill example (App. G) — that make the mechanism inspectable. The skill-library amortization result (queries per trial 4.67 → 0.83) and the zero-shot transfer result (0/6 → 4/6) are falsifiable, directly measured claims about human effort, which is the right quantity to measure for this class of system. The VLM-vs-human correction ablation is also a genuinely informative comparison. The main caveat on significance is that the headline benchmark comparison is conducted under an asymmetric adaptation protocol (detailed in major comments), which currently limits how much of the measured gap can be attributed to the architecture rather than to expert supervision; this is quantifiable from logs the authors

major comments (4)
  1. [§4.3 / Table 1] The headline comparison is run under an asymmetric adaptation budget that the paper does not quantify. Per §4.3, ARCHITECT-HiTL's evaluated policy is the final policy after 'a maximum of 5 rollouts with an expert correction in each rollout,' while CaP, π0, and π0.5 are evaluated zero-shot on their first attempt. Table 1 therefore compares an interactively expert-debugged, task-specific program against policies with no task-specific adaptation. The size of the gap (e.g., close drawer 0.90 vs 0.10/0.00; fold cloth 0.80 vs 0.00) is exactly what the abstract's 'outperforms state-of-the-art VLA models' rests on, and the reader cannot separate the contribution of the synthesis architecture from the contribution of up to five rounds of free expert supervision. Two reportable quantities would largely resolve this, and both should exist in the logs: (a) the number of corrections actually consumed
  2. [§4.3] The evaluation-hygiene protocol for the correction rollouts is ambiguous. §4.3 states that object position/orientation and instruction phrasing 'are perturbed for each new rollout.' It is not stated whether ARCHITECT's up-to-5 correction rollouts were drawn from the same perturbed configuration distribution (or even the same configurations) later used for the 10 evaluation episodes. If so, the corrected policy and skill library have seen eval-adjacent experience, which would inflate Table 1 relative to baselines that never see the eval distribution. Please state explicitly whether correction rollouts and evaluation rollouts were drawn from disjoint configuration sets, and if not, rerun or bound the effect.
  3. [§5.1 / Abstract] The 'outperforms' claim is not supported per-task and should be reframed to match the table. In Table 1, π0.5 exceeds ARCHITECT-HiTL on banana→plate (SR 0.90 vs 0.80) and block→basket (1.00 vs 0.70), and ties or beats it on GCR/NTP for those rows; ARCHITECT-VLM beats ARCHITECT-HiTL on fold-cloth SR (0.70 vs 0.80 is fine, but close drawer 0.40 vs 0.90 involves overlapping budgets). The defensible claim is that ARCHITECT wins on average and dominates on the articulated/deformable/long-horizon categories where all baselines score ~0. The abstract and §5.1 should say this, ideally with an aggregate row (mean SR/GCR/NTP across tasks) and confidence intervals — with N=10 per cell, per-task differences of 0.1–0.2 SR are within binomial noise, and no uncertainty quantification is given anywhere in Tables 1–3.
  4. [§4.3 / §5.2–5.3] Two evaluation-protocol details are load-bearing for the numbers and currently unspecified. (1) Success adjudication: App. C gives rubrics, but the manuscript never states who or what scores SR/GCR/NTP on the benchmark — human annotator, VQA, or the authors — and whether scoring was blinded to method. Given that §4.4 uses VQA for subgoal verification in the human study, the benchmark scoring procedure needs its own statement, including any inter-rater or VQA-agreement check. (2) The human study (N=6) reports p=0.036 for the query-count reduction without naming the statistical test, and the within-subjects design (Task 1 always precedes Task 2, skill library always populated from the participant's own Task 1) confounds transfer with ordering/familiarity; with 6 participants and 3/6 at zero corrections, the result is suggestive but fragile. At minimum, name the test, report per-participant
minor comments (8)
  1. [§4.1 / Ref [27]] The orchestrator is described as 'Claude Opus 4.6' but reference [27] is the Claude 3 model family card (Opus/Sonnet/Haiku, 2024). Please cite the correct model card; the orchestrator identity is a free parameter of the system and should be pinned precisely for reproducibility.
  2. [Fig. 2] The radar chart scoring ARCHITECT, CaP, and VLAs on six axes (generalizability, interpretability, etc.) has no stated quantitative basis. Either ground the scores in measured quantities from the paper or remove/label the figure as a qualitative schematic.
  3. [§5.2 / Fig. 5 vs Fig. 7] The text refers to human-evaluation results as 'Fig. 7a/7b/7c,' but the results panels are Figure 5; Figures 6 and 7 in the appendix are task photos. Please fix the cross-references.
  4. [Table 3 / App. B] The language-perturbation ablation complicates the robustness claim in ways worth acknowledging: ARCHITECT itself drops to SR 0.00 on the long instruction and on 'Move the apple.' Also, the number of rollouts per cell in Table 3 is not stated (canonical instructions suggest small N). Please report N per condition and temper the 'robust to variations' wording in §5.1 accordingly.
  5. [§3.3 / §6] The skill library loads 'all skills' into context before each generation. Scalability of this design (context length, skill retrieval/selection as the library grows, interference between contradictory skills) is a natural concern for the 'long-term in-context learning' framing and deserves at least a paragraph in §6.
  6. [Table 5] VQA is listed as 'GPT-5.4 [40]' but [40] is the GPT-5 system card; again, pin the exact model version. More broadly, the system's dependence on two proprietary frontier models (orchestrator + VQA) means the results may not be reproducible as these APIs drift; a short reproducibility statement (model snapshots, dates, total API cost per task) would strengthen the paper.
  7. [App. E] The failure analysis is candid and useful (grasp sampling, depth-induced placement error, cuRobo planning failures). It would be strengthened by a quantitative breakdown: what fraction of ARCHITECT-HiTL's residual failures in Table 1 fall into each category, and which were correctable vs. uncorrectable by language feedback. This bears directly on the paper's core assumption that residual failures are localizable and correctable in language space.
  8. [Typos] Minor text issues: 'perturbance' (§1) should be 'perturbation'; spaced-letter artifacts ('V oyager', 'p r i m i t i v e s' in App. F) from PDF extraction or source formatting; 'There is a apple' in Table 3 (keep if verbatim, but mark as such).

Circularity Check

0 steps flagged

No circularity: empirical robot-trial results, not identities or self-justifying fits.

full rationale

ARCHITECT is an empirical systems paper. Its load-bearing claims (Table 1 SR/GCR/NTP vs CaP and π0/π0.5; human-eval skill-library transfer and query amortization in §5.2–5.3) are measured outcomes of real Franka rollouts under an explicit protocol, not quantities derived from equations that redefine the target as an input. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from overlapping authors to forbid alternatives, and no ansatz smuggled in via self-citation. Self-citations (e.g., Murray et al. for the GCR definition) supply a metric or related-work context and are not load-bearing for the performance claims. Asymmetric adaptation budgets (up to five expert-correction rollouts for ARCHITECT vs zero-shot baselines) are a fairness/protocol concern, not circularity by construction. Residual dependence on working tools and LLM coding quality is ordinary domain dependence, not a circular reduction.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

Central claims rest on engineering assumptions about tools, LLM coding agents, human correction quality, and evaluation protocol—not on fitted physical constants. The ‘theory’ is that modular programs plus trace-grounded language corrections and a skill library yield steerable, transferable policies; free choices are mostly protocol knobs (correction budget, models, success rubrics) and the postulated skill-library memory mechanism.

free parameters (4)
  • Max HiTL correction rollouts before final policy eval = 5
    Main benchmark allows up to 5 expert-corrected rollouts then evaluates the resulting policy; this budget is a design choice that strongly affects reported SR vs zero-shot baselines.
  • Orchestrator and VQA model identities = Claude Opus 4.6; GPT-5.4
    Results depend on chosen LLMs/VLMs (Claude Opus 4.6; GPT-5.4 for VQA) rather than a model-agnostic proof; swapping models would change synthesis and verification quality.
  • Guarded-move force threshold / placement heuristics in skills = ~5.0 N; distance -0.25 m (example skill)
    Example skills encode numeric thresholds (e.g., ~5N force, -0.25 m descent) chosen for the hardware/objects; these are hand-set control parameters entering successful policies.
  • Human-eval task pair and success verification via VQA = cup-on-table → pasta-box-on-shelf; N=6
    Transfer conclusions depend on choosing two structurally similar upright-placement tasks and automated subgoal checks; different task distance would change amortization claims.
axioms (6)
  • domain assumption A pretrained LLM coding agent can map language instructions, tool docs, traces, and skill text into executable robot programs that invoke the exposed API correctly enough to be iteratively repaired.
    Core of §3; no formal completeness claim—assumed from LLM coding ability and demonstrated empirically.
  • domain assumption The fixed control/perception/proprioception tool suite is expressive enough for the evaluated manipulation classes when composed in code.
    §3.4 and App. H; limitations §6 admit grasp/depth/motion-planning failures when tools fail.
  • domain assumption Natural-language corrections plus execution traces are sufficient to localize faults and update programs/skills without demonstration data or weight updates.
    §3.2–3.3; contrasted with VLM-only corrections in §5.4.
  • ad hoc to paper Skills distilled from one task remain valid and helpful on related tasks/scenes (long-term in-context learning via skill library).
    Central transfer claim §3.3, §5.2–5.3; supported only within narrow within-domain human-eval tasks.
  • domain assumption Binary/stage success rubrics (SR, GCR, NTP) and VQA subgoal checks faithfully measure task completion.
    §4.3 and App. C; standard in the area but still an evaluation modeling choice.
  • standard math Standard robotics/math facts: rigid-body poses, IK/motion planning feasibility, gripper aperture semantics, camera–base transforms.
    Implicit in control and perception tool implementations (cuRobo, TF, etc.).
invented entities (2)
  • ARCHITECT agentic loop (synthesize → execute with traces → language correct → skill-library update) no independent evidence
    purpose: Name the end-to-end interactive policy acquisition process claimed to outperform black-box VLAs and one-shot program synthesis.
    Framework label for an orchestration pattern; not a new physical entity. Evidence is empirical system performance, not an independent natural phenomenon.
  • Persistent skill library as long-term in-context memory of corrections no independent evidence
    purpose: Store distilled natural-language/code patterns so future tasks reuse prior human fixes and amortize effort.
    Operational memory structure inspired by Voyager-like libraries; falsifiable only via transfer/sample-efficiency experiments in this and follow-on work.

pith-pipeline@v1.2.0-grok45-kimik3 · 26699 in / 4090 out tokens · 77675 ms · 2026-07-30T12:36:50.244754+00:00 · methodology

0 comments
read the original abstract

While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail. In this work, we propose ARCHITECT, a framework that treats robot policy acquisition as an interactive program synthesis task. ARCHITECT leverages the reasoning capabilities of LLM coding agents to synthesize modular robot programs that utilize a suite of perception and control tools. Unlike end-to-end models where distribution shift leads to unpredictable, cascading failures, our modular architecture allows users to isolate failures and localize feedback at the level of abstraction required. We introduce an iterative process where a human supervisor provides natural language corrections to steer the policy. These corrections are grounded in the policy code by program execution traces and distilled into a persistent skill library, a form of long-term in-context learning which enables the agent to accumulate a repertoire of reusable, interpretable behaviors. In a benchmark evaluation on a Franka Panda robot, ARCHITECT outperforms state-of-the-art VLA models and program synthesis baselines on complex, long-horizon tasks, including articulated object manipulation and cloth folding. Our results demonstrate that the synthesized skill library enables the system to transfer to novel tasks with decreasing human intervention, providing a steerable and data-efficient alternative to black-box robot learning. Website: https://robo-architect.github.io/

Figures

Figures reproduced from arXiv: 2607.23784 by Archit Ritesh Jain, Daphne Chen, Emma Romig, Eric Goossen, Maya Cakmak, Michael Murray, Nick Walker.

Figure 1
Figure 1. Figure 1: System overview of ARCHITECT. The pipeline takes natural language input and syn [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of ARCHITECT, Code as Policies, and Vision-Language Ac￾tion Models across 6 axes of desirable traits for robot policies. The tools primarily consist of robot control, pro￾prioception, and perception modules. They are de￾fined to be simple and generic so as to not require task-specific assumptions, and can thus be used to compose flexible and diverse task logic. Functions are parameterized with a… view at source ↗
Figure 3
Figure 3. Figure 3: The 8 evaluation tasks and their challenge categories. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of ARCHITECT performance against CaP, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ARCHITECT results from human evaluation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Task 1 from human evaluation. The instruction is ”place the cup upright on the table”. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Task 2 from human evaluation. The instruction is ”place the pasta box upright on the [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 24 linked inside Pith

  1. [1]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246

    Pith/arXiv arXiv 2024

  2. [2]

    Maslej, L

    N. Maslej, L. Fattorini, R. Perrault, Y . Gil, V . Parli, N. Kariuki, E. Capstick, A. Reuel, E. Bryn- jolfsson, J. Etchemendy, K. Ligett, T. Lyons, J. Manyika, J. C. Niebles, Y . Shoham, R. Wald, T. Walsh, A. Hamrah, L. Santarlasci, J. B. Lotufo, A. Rome, A. Shi, and S. Oak. Artificial intelligence index report 2025, 2025. URLhttps://arxiv.org/abs/2504.07139

    arXiv 2025

  3. [3]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control, 2023. URLhttps://arxiv. org/abs/2209.07753

    Pith/arXiv arXiv 2023

  4. [4]

    H. Wang, K. Kedia, J. Ren, R. Abdullah, A. Bhardwaj, A. Chao, K. Y . Chen, N. Chin, P. Dan, X. Fan, G. Gonzalez-Pumariega, A. Kompella, M. A. Pace, Y . Sharma, X. Sun, N. Sunkara, and S. Choudhury. Mosaic: Modular foundation models for assistive and interactive cooking,

  5. [5]

    Yasnitsky.Vygotsky, Lev, pages 843–845

    A. Yasnitsky.Vygotsky, Lev, pages 843–845. 09 2014. ISBN 9781452230894. doi:10.4135/ 9781483346229

    2014

  6. [6]

    Novack and S

    M. Novack and S. Waxman. Becoming human: human infants link language and cognition, but what about the other great apes?Philosophical Transactions of the Royal Society B, 375: 20180408, 11 2019. doi:10.1098/rstb.2018.0408

    arXiv 2019

  7. [7]

    Lupyan, H

    G. Lupyan, H. Gentry, and M. Zettersten. How important is language for human-like intelli- gence?, 2025. URLhttps://arxiv.org/abs/2509.15560

    arXiv 2025

  8. [8]

    Singh, R

    S. Singh, R. Lewis, and A. Barto. Where do rewards come from? 01 2009

    2009

  9. [9]

    R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URLhttp://incompleteideas.net/book/the-book-2nd.html

    2018

  10. [10]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models, 2023. URLhttps: //arxiv.org/abs/2305.16291

    Pith/arXiv arXiv 2023

  11. [11]

    Grannen, S

    J. Grannen, S. Karamcheti, S. Mirchandani, P. Liang, and D. Sadigh. V ocal sandbox: Continual learning and adaptation for situated human-robot collaboration, 2024. URLhttps://arxiv. org/abs/2411.02599

    Pith/arXiv arXiv 2024

  12. [12]

    Huang, F

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022. URL https://arxiv.org/abs/2207.05608

    Pith/arXiv arXiv 2022

  13. [13]

    Singh, V

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022. URLhttps://arxiv.org/abs/2209.11302. 9

    Pith/arXiv arXiv 2022

  14. [14]

    H. Wang, G. Gonzalez-Pumariega, Y . Sharma, and S. Choudhury. Demo2code: From sum- marizing demonstrations to synthesizing code via extended chain-of-thought, 2023. URL https://arxiv.org/abs/2305.16744

    Pith/arXiv arXiv 2023

  15. [15]

    P. Liu, Y . Orru, J. Vakil, C. Paxton, N. Shafiullah, and L. Pinto. Demonstrating ok-robot: What really matters in integrating open-knowledge models for robotics. InRobotics: Science and Systems XX, RSS2024. Robotics: Science and Systems Foundation, July 2024. doi:10.15607/ rss.2024.xx.091. URLhttp://dx.doi.org/10.15607/RSS.2024.XX.091

    doi:10.15607/rss.2024.xx.091 2024

  16. [16]

    Murray, A

    M. Murray, A. Gupta, and M. Cakmak. Teaching robots with show and tell: Using foundation models to synthesize robot policies from language and visual demonstration. In P. Agrawal, O. Kroemer, and W. Burgard, editors,Proceedings of The 8th Conference on Robot Learning, volume 270 ofProceedings of Machine Learning Research, pages 4033–4050. PMLR, 06–09 Nov ...

    2025

  17. [17]

    A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners, 2023. URLhttps://arxiv.org/abs/2307. 01928

    2023

  18. [19]

    J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: personalized robot assistance with large language mod- els.Autonomous Robots, 47(8):1087–1102, Nov. 2023. ISSN 1573-7527. doi:10.1007/ s10514-023-10139-z. URLhttp://dx.doi.org/10.1007/s10514-023-10139-z

    doi:10.1007/s10514-023-10139-z 2023

  19. [20]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine, Y . Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Retti...

  20. [21]

    M. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, F.-F. Li, G. Shi, J. Wu, S. Sastry, Y . Zhu, K. Goldberg, and L. J. Fan. Cap-x: A framework for benchmarking and improving coding agents for robot manipulation, 2026. URLhttps://arxiv.org/abs/ 2603.22435

    Pith/arXiv arXiv 2026

  21. [22]

    L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly from language corrections, 2024. URLhttps://arxiv. org/abs/2403.12910

    Pith/arXiv arXiv 2024

  22. [23]

    Y . Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. InProceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’23, page 93–101. ACM, Mar. 2023. doi:10.1145/3568162.3578623. URLhttp://dx.doi.org/10. 1145/3568...

    arXiv 2023

  23. [24]

    W. Chen, J. S. Bhatia, C. Glossop, N. Mathihalli, R. Doshi, A. Tang, D. Driess, K. Pertsch, and S. Levine. Steerable vision-language-action policies for embodied reasoning and hierarchical control, 2026. URLhttps://arxiv.org/abs/2602.13193. 10

    Pith/arXiv arXiv 2026

  24. [25]

    Winfield, S

    A. Winfield, S. Booth, L. Dennis, T. Egawa, H. Hastie, N. Jacobs, R. Muttram, J. Olszewska, F. Rajabiyazdi, A. Theodorou, M. Underwood, R. Wortham, and E. Watson. Ieee p7001: A proposed standard on transparency.Frontiers in Robotics and AI, 8:665729, 07 2021. doi: 10.3389/frobt.2021.665729

    arXiv 2021

  25. [26]

    Khazatsky, K

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y . J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y . Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. ...

    Pith/arXiv arXiv 2025

  26. [27]

    The Claude 3 model family: Opus, Sonnet, Haiku

    Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Model card, 2024. URL https://api.semanticscholar.org/CorpusID:268232499

    2024

  27. [28]

    Sundaralingam, S

    B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. V . Wyk, V . Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, N. Ratliff, and D. Fox. curobo: Parallelized collision- free minimum-jerk robot motion generation, 2023. URLhttps://arxiv.org/abs/2310. 17274

    2023

  28. [29]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π 0: A vision-language-action flow model for general robot control, 2026. URLhttps://arxiv. o...

    Pith/arXiv arXiv 2026

  29. [30]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. V...

    Pith/arXiv arXiv 2025

  30. [31]

    Y . R. Wang, C. Ung, C. Tan, G. Tannert, J. Duan, J. Li, A. Le, R. Oswal, M. Grotz, W. Pumacay, Y . Deng, R. Krishna, D. Fox, and S. Srinivasa. Roboeval: Where robotic manipulation meets structured and scalable evaluation, 2026. URLhttps://arxiv.org/abs/2507.00435

    Pith/arXiv arXiv 2026

  31. [32]

    X. Yang, R. Dagli, A. Zook, H. Hadfield, A. Goyal, S. Birchfield, F. Ramos, and J. Tremblay. Robolab: A high-fidelity simulation benchmark for analysis of task generalist policies, 2026. URLhttps://arxiv.org/abs/2604.09860

    Pith/arXiv arXiv 2026

  32. [33]

    H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y . Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains, 2023. URLhttps: //arxiv.org/abs/2212.08333

    Pith/arXiv arXiv 2023

  33. [34]

    Murali, B

    A. Murali, B. Sundaralingam, Y .-W. Chao, W. Yuan, J. Yamada, M. Carlson, F. Ramos, S. Birchfield, D. Fox, and C. Eppner. Graspgen: A diffusion-based framework for 6-dof grasp- ing with on-generator training, 2025. URLhttps://arxiv.org/abs/2507.13097. 11

    Pith/arXiv arXiv 2025

  34. [35]

    W. Yuan, A. Murali, A. Mousavian, and D. Fox. M2t2: Multi-task masked transformer for object-centric pick and place. In7th Annual Conference on Robot Learning, 2023

    2023

  35. [36]

    H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W.-C. Tsai, S. Chen, Y . R. Wang, S. Xing, J. Cho, J. S. Park, A. Eftekhar, P. Sushko, K. Farley, A. Wadhwa, C. Harrison, W. Han, Y .-C. Lee, E. VanderBilt, R. Hendrix, S. Ellawela, L. Ngoo, J. Chai, Z. Ren, A. Farhadi, D. Fox, and R. Krishna. Molmoact2: Action reasoning models for real-world d...

    Pith/arXiv arXiv 2026

  36. [37]

    Bjorck, F

    NVIDIA, :, J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y . L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y . Xie, Y . Xu, Z. Xu, S. Ye, Z...

    Pith/arXiv arXiv 2025

  37. [38]

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024. URLhttps://arxiv.org/abs/2401.14159

    Pith/arXiv arXiv 2024

  38. [39]

    Y . Zhao, M. Bogdanovic, C. Luo, S. Tohme, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Anyplace: Learning generalized object placement for robot manipulation, 2025. URLhttps://arxiv.org/abs/2502.04531

    arXiv 2025

  39. [40]

    Singh, A

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, A. Nathan, A. Luo, A. Helyar, A. Madry, A. Efremov, A. Spyra, A. Baker-Whitcomb, A. Beutel, A. Karpenko, A. Makelov, A. Neitz, A. Wei, A. Barr, A. Kirch- meyer, A. Ivanov, A. Christakis, A. Gillespie, A. Tam, A. Bennett, A. Wan, A. Huang, A. M....

    Pith/arXiv arXiv 2026

  40. [41]

    L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan. Emergent correspondence from image diffusion, 2023. URLhttps://arxiv.org/abs/2306.03881

    Pith/arXiv arXiv 2023

  41. [42]

    Huang, C

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation, 2024. URLhttps://arxiv.org/ abs/2409.01652. 13 Table 2: Success Rate (SR), Goal Condition Recall (GCR), and Normalized Task Progression (NTP) across tasks. We compare ARCHITECT against Inner Monologue (IM) [12]...

    Pith/arXiv arXiv 2024

  42. [45]

    Call l i s t _ p r i m i t i v e s () to see the full robot API and any p r i m i t i v e s already \ r e g i s t e r e d this session

  43. [46]

    pick up the banana and put it in the frying pan

    Call q u e r y _ r o b o t _ s t a t e () and g e t _ s c e n e _ d e s c r i p t i o n () to ground your plan in the current robot state and scene . Then call the a p p r o p r i a t e p e r c e p t i o n \ fu nc ti on s up - front before writing the program : - d e t e c t _ o b j e c t s () - ONLY for objects that must be GRASPED ( picked up ) . \ 18 D...

  44. [47]

    grasp object

    Identify sub - o p e r a t i o n s . For EACH sub - o pe ra ti on that involves more than one \ API call ( e . g . " grasp object " , " place on shelf " , " navigate to position " ) , you MUST \ call w r i t e _ p r i m i t i v e ( name , code , d oc st ri ng ) to register it before using it in the \ program . Do not inline multi - step logic into the top...

  45. [48]

    position

    Only after all p r i m i t i v e s are registered , call s u b m i t _ p r o g r a m ( code ) with a \ short top - level program that reads as a sequence of named p ri mi tiv e calls . Tool r ef er en ce : - l i s t _ p r i m i t i v e s - see all a va il ab le robot API + session - r e g i s t e r e d p r i m i t i v e s - r e a d _ p r i m i t i v e ( n...

  46. [49]

    diffusion features; captures RGB-D, looks up the camera-to-base transform, and returns keypoint coordinates in the robot base frame get keypoints trajectory() text prompt,task (natural language), camera(wrist/scene) Computes staged waypoint trajectories via a ReKep-based planner [42] over DIFT keypoints; returns gripper-action- annotated waypoints grouped...

  47. [50]

    from a pre-computed segmentation mask; transforms the highest-scored grasp pose to the robot base frame 22 Table 6: ARCHITECT tool suite primitives: Proprioception Name Input Description get current ee pose()None Returns the current end-effector pose (po- sition + quaternion) by extracting and con- verting theO T EEhomogeneous trans- form fromFrankaState ...

  48. [2022]

    URLhttps://arxiv.org/abs/2204.01691

    Pith/arXiv arXiv

  49. [2025]

    URLhttps://arxiv.org/abs/2402.18796

    arXiv