Pith. sign in

REVIEW 4 major objections 6 minor 103 references

A frozen vision-language-action policy becomes reliable under language and layout shifts when used as a retryable contact skill inside a fixed analytic planner with memory.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 07:17 UTC pith:S53W3UDI

load-bearing objection Solid systems paper: fixed primitives + frozen VLA as a retryable contact specialist, with real multi-benchmark gains; seed-0 bootstrap is the main caveat, not a collapse of the claim. the 4 major comments →

arxiv 2607.08448 v1 pith:S53W3UDI submitted 2026-07-09 cs.RO

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Yixian Zhang , Huanming Zhang , Feng Gao , Xiao Li , Zhihao Liu , Chunyang Zhu , Jiaxing Qiu , Yuchen Yan
show 8 more authors
Jiyuan Liu Wenhao Tang Zhengru Fang Yi Nie Changxu Wei Yu Wang Wenbo Ding Chao Yu
This is my paper
classification cs.RO
keywords vision-language-action modelsrobotic manipulationagentic planningfrozen policiesmemory-augmented agentslanguage-conditioned controlanalytic primitivescontact-rich skills
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

End-to-end robot policies are strong at local contact but brittle when instructions, goals, or object layouts change, because they must do language grounding, long-horizon composition, and low-level control in one shot. Pure coding agents reason well about tasks but struggle when every delicate grasp or fixture move must be hand-scripted. This paper claims the right division of labor is to freeze the pretrained policy, expose it as one retryable contact primitive, and let a memory-guided planner compose it with a small fixed set of analytic moves for staging, transport, navigation, and release. The harness does not invent new skills; it learns when each fixed primitive works from reference traces, success rules, and failure models. On perturbed tabletop, kitchen, and bimanual benchmarks, that design lifts success far above the strongest comparable baselines without finetuning the visuomotor model.

Core claim

Harness VLA extends a pretrained vision-language-action model beyond its original trajectory distribution without finetuning by casting the frozen policy as a single contact-rich primitive (VLA ACT) and composing it with a small fixed analytic library. Semantic re-grounding, free-space motion, and re-staging after failed contacts stay with the planner; the VLA is reserved for local contact-rich phases and may be retried after the planner changes approach pose or viewpoint. Task-specific traces and global success and failure rules teach the planner the operating range of those fixed primitives rather than expanding the skill vocabulary at deployment.

What carries the argument

VLA ACT: a frozen vision-language-action call treated as a retryable local contact primitive, orchestrated by an agentic planner over a fixed analytic library and backed by Task Specific Memory (parameterized successful traces) plus Global Memory (reusable success rules and failure models).

Load-bearing premise

One successful reference-seed exploration, stored as a re-groundable primitive trace plus a few global heuristics, is enough for the planner to handle held-out seeds and scene or language perturbations without new skills or policy finetuning.

What would settle it

Under the same frozen VLA backends and fixed primitive library, if memory-guided re-grounding and re-staging fail to raise success well above the direct frozen-VLA baseline on LIBERO-Pro instruction-redirection and position-swap cells, the claim that planner-staged composition extends the policy would be refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Many language and layout shifts can be absorbed by re-binding targets and re-staging contact rather than by finetuning the visuomotor backbone.
  • A small fixed primitive vocabulary is enough if the planner learns each primitive's operating range from traces and failure models.
  • Sparse, planner-chosen VLA invocations with retries can beat continuous monolithic rollouts on long-horizon and contact-heavy tasks.
  • Analytic control can own transport, navigation, and release while the frozen policy owns grasping and fixture actuation across tabletop, kitchen, and bimanual settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same fixed-vocabulary harness may transfer across robot embodiments if VLA ACT and arm or base binding remain stable interfaces.
  • The paper's own open planner-to-VLA feedback loop points to a next step: sample-efficient joint training that rewards good staging and early returns without abandoning the fixed primitive contract.
  • If one clean reference trace generalizes under re-grounding, many structured manipulation suites may need less large-scale policy adaptation than end-to-end training assumes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. Harness VLA proposes a memory-augmented agentic framework that treats a frozen Vision-Language-Action model as a single retryable contact-rich primitive (VLA ACT) and composes it with a small fixed library of analytic primitives (transport, posture, gripper, navigation, release). An LLM planner orchestrates these primitives via a JSON/file-mediated REPL, using Task Specific Memory (parameterized successful traces from one reference seed) and Global Memory (success rules and failure models). The central claim is that this factorization extends pretrained VLAs beyond their original trajectory distributions under semantic retargeting, layout shifts, and long-horizon composition without VLA finetuning or deployment-time skill expansion. Empirically, the system reports competitive standard LIBERO performance, large gains on LIBERO-Pro (up to 82.4% vs. RATS 43.8% and RLinf 50.0%) and RoboCasa365 (55.4% vs. RLDX-1 30.0%), and 58.4% on RoboTwin clean-to-randomized transfer, with mechanism analyses of re-grounding, staged VLA retries, and analytic/VLA division of labor.

Significance. If the results hold under the stated protocols, the paper offers a practically important alternative to both monolithic VLA finetuning and ever-expanding skill libraries: keep the VLA frozen as a local contact specialist and lift grounding, staging, and recovery to a memory-guided planner. Strengths include multi-benchmark coverage (tabletop, kitchen, bimanual C2R), same-backend frozen-VLA baselines, transparent few-shot vs. zero-shot regimes, and quantitative evidence that sparse re-staged VLA invocations improve reliability (Figure 4). The fixed-vocabulary design and explicit Task Specific / Global Memory split are useful contributions relative to coding-agent lines that grow skills at deployment time. The work is significant for robotics systems research even if some gains partly reflect privileged reference-seed search, provided that dependence is measured and framed carefully.

major comments (4)
  1. [§2.2; Appendix C; Tables 3–4] Section 2.2 and Appendix C make seed-0 exploratory bootstrapping (RESET enabled, generous budget) load-bearing for the few-shot headline results in Tables 3–4. The abstract and introduction state that Harness VLA “extends pretrained VLAs… without finetuning,” but the 38.6 and 25.4 pp gains measure the full harness after privileged task-level search, not closed-loop composition from a frozen interface alone. Please quantify sensitivity to the reference seed (e.g., multiple seed-0 traces, weak/failed exploration, or withheld Task Specific Memory on the full LIBERO-Pro and RoboCasa365 suites), and state explicitly in the abstract/results which numbers are few-shot-after-bootstrap versus zero-shot.
  2. [§2.2; §3.2 Table 5; Key Findings 1–3] The paper lacks a systematic ablation of the memory modules that are claimed to teach the operating range of the fixed primitives. Table 5 withholds Task Specific Memory only on LIBERO-Pro GOAL, and Global Memory is not ablated. For the central claim that “rather than expanding the skill library, the harness learns the operating range… from task-specific execution traces, global success rules, and failure models,” report success with (i) no Task Specific Memory, (ii) no Global Memory, and (iii) trace-only without failure models on at least one full perturbed suite. Without this, it remains unclear whether gains come from memory-guided re-grounding or from online frontier-LLM search over a good API.
  3. [§3.1–3.2; Tables 3–4, 6] Baseline fairness under the same protocol is incompletely specified. RATS and Cap-X omit LIBERO-10 cells (Table 3 caption), yet the headline “+38.6 pp over RATS” mixes overall aggregates. Clarify whether Cap-X/RATS (and other coding-agent baselines) receive an analogous reference-seed exploration budget, RESET, and memory write, or only zero-shot/API access. Also report RLinf/RLDX-1/LingBot under identical step budgets and observation interfaces as Harness VLA so that the gain is attributable to primitive composition rather than evaluation asymmetry.
  4. [§2.3; §3.3 Key Finding 2; Figure 4] Key Finding 2 argues that planner-staged retries make the frozen VLA reliable, and Figure 4 supports sparse multi-invocation gains. However, max_chunks, stop predicate τ, and the episode step budget are free parameters of VLA ACT (Appendix B/D) and are not held fixed or ablated across benchmarks. Please report how success depends on chunk budget and allowed VLA calls under a matched total action/step budget against the continuous frozen-VLA baseline, so that “retryable primitive” is not confounded with simply giving the agent more attempts or wall-clock.
minor comments (6)
  1. [Figure 1] Figure 1 and the bottom rollout strip are informative but dense; a single annotated timeline of analytic vs. VLA calls for one LIBERO-Pro and one RoboCasa task would help readers parse the intended sparse invocation pattern.
  2. [§3.2 Table 2] Table 2 shows Harness VLA slightly below AtomVLA overall on standard LIBERO; a short discussion of when the harness preserves vs. slightly trades off in-distribution performance would strengthen the “no free lunch” framing.
  3. [§4] Several related-work citations are concurrent arXiv preprints (Cap-X, RATS, ASPIRE, RoboCasa365). Ensure versioned citations and a clear novelty paragraph distinguishing fixed-vocabulary + frozen VLA ACT from skill-library expansion (ASPIRE) and pure coding agents.
  4. [Appendix A; §2.2] Appendix A’s file-mediated REPL is valuable for reproducibility; consider moving a short protocol box into the main text so the isolation from privileged simulator state is not appendix-only.
  5. [Throughout; Appendix C] Minor consistency: “V oyager” spacing, “π 0.5” vs “π0.5”, and “Harness VLA (CC)” vs “Claude Code” should be normalized; also fix “clean-to-randomized” protocol wording where “seeds 0 (seed 0)” is repeated awkwardly in Appendix C.
  6. [§5] Limitations mention open feedback between planner and VLA and lack of joint RL; a concrete failure-case taxonomy (empty grasp, false success, irrecoverable displacement) with frequencies would make the limitation section more actionable.

Circularity Check

0 steps flagged

No circular derivation: headline gains are external-benchmark success rates of a systems composition, not quantities forced by definition or self-cited uniqueness.

full rationale

Harness VLA is an empirical agentic systems paper, not a first-principles derivation. The central claim—that a frozen VLA exposed as VLA ACT plus fixed analytic primitives and memory-guided re-staging extends beyond the VLA’s training trajectory distribution without finetuning—is evaluated by binary completion predicates supplied by LIBERO, LIBERO-Pro, RoboCasa365, and RoboTwin (Appendix C), against independent and frozen-VLA baselines (Tables 2–6). Task Specific Memory is built on a reference seed and re-grounded on held-out seeds; that is standard few-shot transfer, not a fit that defines the reported success rate. Using authors’ own frozen backends (RLinf, LingBot-VLA) is disclosed and scored as direct baselines that the harness still beats; those citations do not force the 38.6 / 25.4 pp deltas by construction. Concerns about privileged seed-0 exploration affect evaluation fairness, not circularity of a derivation chain. No self-definitional identity, fitted-input-as-prediction, uniqueness import, or ansatz-via-self-citation reduces the result to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

The central claim rests on engineering assumptions about division of labor (VLA for contact, analytic for non-contact), a privileged reference-seed memory construction protocol, and commercial LLM planners operating over a hand-specified fixed API—not on free physical constants or invented particles. Free parameters are mostly operational budgets and planner/backend choices. Invented entities are software abstractions (memories, VLA ACT interface, harness) with independent operational handles in the REPL logs.

free parameters (5)
  • VLA ACT max_chunks / stop predicate τ
    Planner-chosen chunk budgets and early-return predicates control how long the frozen VLA runs; success curves in Figure 4 depend on allowing multiple invocations.
  • Episode step budget and strict evaluation reset policy
    Deployment phase shortens budget and disables RESET; reported rates are defined under these hand-set limits (Section 2.2).
  • Reference seed (seed 0) for Task Specific Memory
    One exploratory seed per task supplies the JSONL skeleton; evaluation seeds re-ground that skeleton. Choice of reference seed is a free experimental degree of freedom (Appendix C).
  • Planner backbone (Codex vs Claude Code)
    Headline numbers differ by planner (e.g., LIBERO-Pro 72.1% vs 82.4%); the harness is not planner-agnostic in practice.
  • Benchmark-specific frozen VLA checkpoint choice
    RLinf π0.5-SFT, RLDX-1, and post-trained LingBot-VLA are selected per suite; gains are relative to those backends.
axioms (5)
  • domain assumption Frozen VLAs are strongest on local contact-rich phases and weak on semantic rebinding, long-horizon composition, and layout shift.
    Stated in Introduction and Related Work; motivates reserving VLA ACT for contact only.
  • domain assumption A small fixed analytic primitive library (MOVE TO, ROTATE, SET GRIPPER, RELEASE, optional base motion) can cover non-contact structure across tabletop, kitchen, and bimanual settings.
    Table 1 / Section 2.3; vocabulary is fixed before evaluation and not expanded at deployment.
  • domain assumption Frontier multimodal LLM coding agents can parse language, ground RGB-D, and emit valid JSON primitive calls in closed loop.
    Harness architecture Section 2.2 and Related Work on coding agents.
  • domain assumption Task success is defined solely by each benchmark's binary completion predicate, not by visual proximity or primitive post-conditions.
    Appendix C evaluation protocol; Global Memory failure models warn against false visual success.
  • ad hoc to paper JSON-serialized primitive interface and file-mediated REPL correctly isolate the planner from privileged simulator state.
    Appendix A protocol; required for claiming realistic partial observability.
invented entities (4)
  • VLA ACT primitive independent evidence
    purpose: Uniform interface that runs a frozen VLA in short bursts with prompt and stop predicate for contact-rich phases.
    Core abstraction of the paper; independent_evidence true because it is an executable API with logs and success attribution.
  • Task Specific Memory (JSONL trace + audit summary) independent evidence
    purpose: Store parameterized successful primitive order from reference seed for few-shot re-grounding.
    Section 2.2; operational memory object, not a physical entity.
  • Global Memory (success rules and failure models) independent evidence
    purpose: Cross-task heuristics such as empty-grasp detection and full-instruction prompting.
    Section 2.2 and Appendix A examples; software knowledge base.
  • Harness VLA agentic framework independent evidence
    purpose: REPL-style planner loop that composes fixed primitives without expanding the skill library.
    Overall system name; evaluated via external benchmarks.

pith-pipeline@v1.1.0-grok45 · 33573 in / 3821 out tokens · 39340 ms · 2026-07-10T07:17:45.846420+00:00 · methodology

0 comments
read the original abstract

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R.

Figures

Figures reproduced from arXiv: 2607.08448 by Changxu Wei, Chao Yu, Chunyang Zhu, Feng Gao, Huanming Zhang, Jiaxing Qiu, Jiyuan Liu, Wenbo Ding, Wenhao Tang, Xiao Li, Yi Nie, Yixian Zhang, Yuchen Yan, Yu Wang, Zhengru Fang, Zhihao Liu.

Figure 1
Figure 1. Figure 1: Harness VLA system overview. Given a task description, RGB-D observations, and robot state, the agentic planner selects structured calls from a fixed primitive library rather than emitting low-level ac￾tions directly. The library exposes the frozen VLA as VLA ACT for contact-rich behaviors and uses analytic primitives such as MOVE TO, ROTATE, and SET GRIPPER for perception-conditioned staging, transport, p… view at source ↗
Figure 2
Figure 2. Figure 2: Primitive composition extends a frozen VLA beyond its trajectory distribution. Deployment perturbations expand the possible task configurations beyond the in-distribution trajectories covered by the frozen VLA. A direct VLA rollout may attempt to bridge the perturbed space and fail before reaching the target. Harness VLA instead decomposes the task into local contact-rich VLA invocations and analytic primi… view at source ↗
Figure 3
Figure 3. Figure 3: Terminal-state frames for two LIBERO-Pro cells. The first triplet compares RLinf on the standard [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Adaptive VLA invocation improves success across benchmarks. Each panel plots cumulative task success as a function of the maximum number of VLA primitive invocations allowed per episode. The blue dashed line marks the corresponding frozen-policy baseline, while the gray dashed line marks full Harness VLA performance with all planner-selected invocations. Across LIBERO-Pro, RoboCasa365, and RoboTwin C2R, su… view at source ↗
Figure 5
Figure 5. Figure 5: Representative rollout frames for adaptive VLA invocation. Top row: a Harness VLA rollout on LIBERO-PRO OBJECT task 4. The planner repeatedly invokes VLA ACT around the milk carton after intermediate grasping or placement attempts leave the object outside or only partially inside the basket; after re-staging the end-effector and retrying the local contact-rich operation, the milk carton is finally placed s… view at source ↗
Figure 6
Figure 6. Figure 6: Task completion attribution across benchmarks. Bars show the fraction of successful rollouts whose final benchmark completion predicate fires after an analytic primitive (blue) or after a VLA primitive (orange). LIBERO Pro-family tasks are mostly finished by analytic primitives after the VLA has established stable contact, whereas RoboCasa365 and RoboTwin C2R contain more terminal contact-rich operations s… view at source ↗
Figure 7
Figure 7. Figure 7: Representative rollout frames for analytic decomposition around contact-rich phases. Top row: on a LIBERO-10-PRO swap task, the agent first invokes VLA ACT and starts moving toward the basket, then detects during MOVE TO that the VLA has not actually grasped the cream-cheese box. The planner moves back, retries VLA ACT, and, after a successful grasp, completes the subtask with MOVE TO and RELEASE. Bottom r… view at source ↗
Figure 8
Figure 8. Figure 8: Overview of representative environments across the four benchmark families used in our evalua [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

103 extracted references · 103 canonical work pages · 33 internal anchors

  1. [1]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning (CoRL), 2023

    work page 2023

  2. [2]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control. InRobotics: Science and Systems (RSS), 2025

    work page 2025

  4. [4]

    $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

    Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    C.-Y . Hung, Q. Sun, P. Hong, A. Zadeh, C. Li, U. Tan, N. Majumder, S. Poria, et al. NORA: A small open-sourced generalist vision language action model for embodied tasks.arXiv preprint arXiv:2504.19854, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    J. Lee, J. Duan, H. Fang, Y . Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y . R. Wang, S. Lee, W. Han, W. Pumacay, A. Wu, R. Hendrix, K. Farley, E. VanderBilt, A. Farhadi, D. Fox, and R. Krishna. Mol- moAct: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    J. Wang, M. Leonard, K. Daniilidis, D. Jayaraman, and E. S. Hu. Evaluatingπ 0 in the wild: Strengths, problems, and the future of generalist robot policies.Online, 2025

    work page 2025

  8. [8]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. InIEEE International Conference on Robotics and Automation (ICRA), 2023

    work page 2023

  9. [9]

    Singh, V

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. InIEEE International Conference on Robotics and Automation (ICRA), 2023

    work page 2023

  10. [10]

    Gupta and A

    T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning without training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    work page 2023

  11. [11]

    J. Shi, R. Yang, K. Chao, B. S. Wan, Y . S. Shao, J. Lei, J. Qian, L. Le, P. Chaudhari, K. Daniilidis, et al. Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots. InNeurIPS 2025 Workshop on Space in Vision, Language, and Embodied AI, 2025

    work page 2025

  12. [12]

    M. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, F.-F. Li, G. Shi, et al. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation.arXiv preprint arXiv:2603.22435, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    Playful Agentic Robot Learning

    J. Zhang, J. Ge, H. Yoo, L. Fu, Z. Yang, Y . Liu, R. Saravanan, S. Yin, J. Yu, D. Niu, et al. Playful agentic robot learning.arXiv preprint arXiv:2606.19419, 2026. 16

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [14]

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji. Executable code actions elicit better LLM agents. InInternational Conference on Machine Learning (ICML), 2024

    work page 2024

  15. [15]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  16. [16]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An open platform for AI software developers as generalist agents. InInternational Conference on Learning Representa...

    work page 2025

  18. [18]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems (NeurIPS), 36, 2023

    work page 2023

  19. [19]

    LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

    Y . Zhou et al. LIBERO-Pro: Towards realistic robotic manipulation benchmarks via systematic per- turbations.arXiv preprint arXiv:2510.03827, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Nasiriany, S

    S. Nasiriany, S. Nasiriany, A. Maddukuri, and Y . Zhu. RoboCasa365: A large-scale simulation frame- work for training and benchmarking generalist robots.arXiv preprint arXiv:2603.04356, 2026

    work page arXiv 2026

  21. [21]

    Y . Mu, T. Chen, Z. Chen, S. Peng, Z. Lan, Z. Gao, Z. Liang, Q. Yu, Y . Zou, M. Xu, et al. RoboTwin: Dual-arm robot benchmark with generative digital twins. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    work page 2025

  22. [22]

    C. Yu, Y . Wang, Z. Guo, H. Lin, S. Xu, H. Zang, Q. Zhang, Y . Wu, C. Zhu, J. Hu, et al. Rlinf: Flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation.arXiv preprint arXiv:2509.15965, 2025

    work page arXiv 2025

  23. [23]

    D. Kim, H. Jang, M. Koo, S. Jang, T. Kim, B. Kim, B. Yoon, C. Jang, D. Choi, D. Han, et al. Rldx-1 technical report.arXiv preprint arXiv:2605.03269, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  24. [24]

    W. Wu, F. Lu, Y . Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y . Wang, S. Ma, et al. A pragmatic VLA foundation model.arXiv preprint arXiv:2601.18692, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    X. Sun, Z. Xu, C. Cao, Z. Liu, Y . Sun, J. Pang, R. Zhang, Z. Yang, K. Pang, D. He, et al. AtomVLA: Scalable post-training for robotic manipulation via predictive latent world models.arXiv preprint arXiv:2603.08519, 2026

    work page arXiv 2026

  26. [26]

    X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model

    J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y . Feng, et al. X-VLA: Soft-prompted transformer as scalable cross-embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    X. Wang, Z. Zhu, G. Huang, B. Wang, X. Chen, and J. Lu. Worlddreamer: Towards general world models for video generation via predicting masked tokens.arXiv preprint arXiv:2401.09985, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

    J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025. 17

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing

    S. Community. Starvla: A lego-like codebase for vision-language-action model developing.arXiv preprint arXiv:2604.05014, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  30. [30]

    RT-1: Robotics Transformer for Real-World Control at Scale

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  31. [31]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy.Online, 2023

    work page 2023

  32. [32]

    Open X-Embodiment: Robotic learning datasets and RT-X mod- els.Online, 2023

    Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X mod- els.Online, 2023

    work page 2023

  33. [33]

    Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

    S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vlms: Investi- gating the design space of visually-conditioned language models.arXiv preprint arXiv:2402.07865, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    Gemini Robotics: Bringing AI into the Physical World

    Gemini Robotics Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Bal- akrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan. 3d-vla: 3d vision-language- action generative world model.arXiv preprint arXiv:2403.09631, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-E: An embodied multimodal language model. InInternational Conference on Machine Learning ...

    work page 2023

  37. [37]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, H. Dong, H. Hu, W. Zhan, H. Wu, Y . Han, and T. Kong. Vision-language foundation models as effective robot imitators. InInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  38. [38]

    Huang, S

    J. Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y . Wang, Q. Li, S.-C. Zhu, B. Jia, and S. Huang. An embodied generalist agent in 3d world. InInternational Conference on Machine Learning (ICML), 2024

    work page 2024

  39. [39]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning (CoRL), 2022

    work page 2022

  40. [40]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. InRobotics: Science and Systems (RSS), 2023

    work page 2023

  41. [41]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRobotics: Science and Systems (RSS), 2023

    work page 2023

  42. [42]

    Introducing GPT-5.2.Online, 2025

    OpenAI. Introducing GPT-5.2.Online, 2025

    work page 2025

  43. [43]

    Pichai, D

    S. Pichai, D. Hassabis, and K. Kavukcuoglu. A new era of intelligence with Gemini 3.Google Blog, 2025. 18

    work page 2025

  44. [44]

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  45. [45]

    Introducing Claude Sonnet 4.5.Online, 2025

    Anthropic. Introducing Claude Sonnet 4.5.Online, 2025

    work page 2025

  46. [46]

    Llama 4 Herd.Meta Blog, 2025

    Meta. Llama 4 Herd.Meta Blog, 2025

    work page 2025

  47. [47]

    GPT-4o System Card

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Gemini Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  49. [49]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    work page 2025

  50. [50]

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  51. [51]

    Introducing deep research.OpenAI Blog, 2025

    OpenAI. Introducing deep research.OpenAI Blog, 2025

    work page 2025

  52. [52]

    Gemini Deep Research.Google Blog, 2024

    Google. Gemini Deep Research.Google Blog, 2024

    work page 2024

  53. [53]

    J. Wu, Z. Deng, W. Li, Y . Liu, B. You, B. Li, Z. Ma, and Z. Liu. Mmsearch-r1: Incentivizing lmms to search.arXiv preprint arXiv:2506.20670, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    X. Geng, P. Xia, Z. Zhang, X. Wang, Q. Wang, R. Ding, C. Wang, J. Wu, Y . Zhao, K. Li, et al. Webwatcher: Breaking new frontier of vision-language deep research agent.arXiv preprint arXiv:2508.05748, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    Huang, Y

    W. Huang, Y . Zeng, Q. Wang, Z. Fang, S. Cao, Z. Chu, Q. Yin, S. Chen, Z. Yin, L. Chen, et al. Vision- deepresearch: Incentivizing deepresearch capability in multimodal large language models.arXiv preprint arXiv:2601.22060, 2026

    work page arXiv 2026

  56. [56]

    B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  57. [57]

    K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. Websailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  58. [58]

    G. R. Team, A. Abdolmaleki, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, A. Balakr- ishna, N. Batchelor, A. Bewley, J. Bingham, et al. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer.arXiv preprint arXiv:2510.03342, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [59]

    Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

    S. Huang, Z. Jiang, H. Dong, Y . Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model.arXiv preprint arXiv:2305.11176, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  60. [60]

    ChatGPT for Robotics: Design Principles and Model Abilities

    S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. ChatGPT for robotics: Design principles and model abilities.arXiv preprint arXiv:2306.17582, 2023. 19

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  61. [61]

    Y . Mu, J. Chen, Q. Zhang, S. Chen, Q. Yu, C. Ge, R. Chen, Z. Liang, M. Hu, C. Tao, P. Sun, H. Yu, C. Yang, W. Shao, W. Wang, J. Dai, Y . Qiao, M. Ding, and P. Luo. RoboCodeX: Multimodal code generation for robotic behavior synthesis.arXiv preprint arXiv:2402.16117, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  62. [62]

    Sur ´ıs, S

    D. Sur ´ıs, S. Menon, and C. V ondrick. ViperGPT: Visual inference via Python execution for reasoning. InIEEE/CVF International Conference on Computer Vision (ICCV), 2023

    work page 2023

  63. [63]

    VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  64. [64]

    Goldberg, K

    A. Goldberg, K. Kondap, T. Qiu, Z. Ma, L. Fu, J. Kerr, H. Huang, K. Chen, K. Fang, and K. Goldberg. Blox-net: Generative design-for-robot-assembly using VLM supervision, physics simulation, and a robot with reset. InIEEE International Conference on Robotics and Automation (ICRA), 2025

    work page 2025

  65. [65]

    R. Li, Y . Zhou, Y . Zhu, K. Chen, J. Wang, S. Wang, K. Hu, M. Yu, B. Jiang, Z. Su, J. Ma, X. He, Y . Shen, Y . Yang, G. Ren, M. Yao, W. Wang, and Y . Mu. RoboClaw: An agentic framework for scalable long-horizon robotic tasks.arXiv preprint arXiv:2603.11558, 2026

    work page arXiv 2026

  66. [66]

    R. Lu, Y . Wu, E. Kou, L. Fu, W. Xiao, A. Mandlekar, Y . Xu, G. Shi, K. Goldberg, A. Chen, et al. ASPIRE: Agentic/skills discovery for robotics.arXiv preprint arXiv:2607.00272, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  67. [67]

    Shinn, F

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    work page 2023

  68. [68]

    Madaan, N

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, et al. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    work page 2023

  69. [69]

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou. Teaching large language models to self-debug. InInterna- tional Conference on Learning Representations (ICLR), 2024

    work page 2024

  70. [70]

    Yoneda, J

    T. Yoneda, J. Fang, P. Li, H. Zhang, T. Jiang, S. Lin, B. Picker, D. Yunis, H. Mei, and M. R. Walter. Statler: State-maintaining language models for embodied reasoning and planning. InIEEE Interna- tional Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  71. [71]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. Jauregui Ruano, K. Jef- frey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine, Y . Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Re...

    work page 2022

  72. [72]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Cheb- otar, et al. Inner monologue: Embodied reasoning through planning with language models.arXiv preprint arXiv:2207.05608, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  73. [73]

    Y . Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y . Qiao, and P. Luo. Embod- iedGPT: Vision-language pre-training via embodied chain of thought. InAdvances in Neural Informa- tion Processing Systems (NeurIPS), volume 36, 2023. 20

    work page 2023

  74. [74]

    A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani, J. Lee, V . Vanhoucke, and P. Florence. Socratic models: Composing zero- shot multimodal reasoning with language. InInternational Conference on Learning Representations (ICLR), 2023

    work page 2023

  75. [75]

    B. Liu, Y . Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. LLM+P: Empowering large language models with optimal planning proficiency.arXiv preprint arXiv:2304.11477, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  76. [76]

    Y . Chen, J. Arkin, C. Dawson, Y . Zhang, N. Roy, and C. Fan. AutoTAMP: Autoregressive task and motion planning with LLMs as translators and checkers. InIEEE International Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  77. [77]

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf. SayPlan: Grounding large language models using 3D scene graphs for scalable task planning. InConference on Robot Learning (CoRL), 2023

    work page 2023

  78. [78]

    task":"put the black bowl on the wooden tray

    Z. Mandi, S. Jain, and S. Song. RoCo: Dialectic multi-robot collaboration with large language models. InIEEE International Conference on Robotics and Automation (ICRA), 2024. 21 A File-Mediated REPL Protocol The harness of Section 2.2 implements the execution loop of Section 2.1 as a synchronous file-mediated Read-Eval-Print Loop (REPL). A long-running en...

    work page 2024

  79. [79]

    seen” and “unseen

    LIBERO-SPATIALcontains tasks that vary spatial relations, LIBERO-OBJECTvaries the target object identity, LIBERO-GOALvaries the goal predicate under related scenes, and LIBERO-10 contains longer- horizon compositional manipulation tasks. 25 Each suite contains 10 language-conditioned tasks. For each task, seeds0 (seed 0) is used only to explore the task a...

    work page 2048

  80. [80]

    Identify the relevant object, fixture, target surface, or relation landmark from RGB

    work page

Showing first 80 references.