Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A language scratchpad—an evolving text record of positions, a plan, and completed sub-actions—lets stateless and recurrent robot policies succeed on memory-dependent manipulation tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:08 UTC pith:6YKIA2WY

load-bearing objection A simple, plausible scratchpad memory for VLAs with large headline gains, but test-time note fidelity is unmeasured — worth a serious referee, conditional on that measurement. the 4 major comments →

arxiv 2602.21013 v2 pith:6YKIA2WY submitted 2026-02-24 cs.RO

Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks

Sanjay Haresh , Daniel Dijkman , Apratim Bhattacharyya , Roland Memisevic This is my paper
classification cs.RO
keywords language scratchpadvision-language-action modelsmemory-dependent manipulationnon-Markovian tasksspatial memorytemporal memoryrecurrent policiesrobot manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that stateless vision-language-action models fail on non-Markovian manipulation tasks because they lack memory, and that this can be fixed by giving the model a running text scratchpad. The scratchpad stores object positions, the plan of sub-tasks, and which sub-tasks are already done; the model itself writes to it by emitting a special 'done' token that appends new notes. On a simulated suite of memory-dependent tasks, this improves average success by roughly 49% for a transformer-based policy and 11% for a recurrent policy, and on a real-world pick-place-restore task it raises success from 0% to 65%. A sympathetic reader would care because it proposes a cheap, architecture-agnostic way to add memory, and because it shows explicit symbolic traces help even models that already have internal recurrence.

Core claim

The central claim is that a policy modeled as p(a_t, d_t | o_t, S_t, l), where S_t is an evolving scratchpad of grounded positions, a plan, and completed sub-actions, can solve tasks that require remembering initial object positions and progress through sub-goals. The model emits a special 'done' token to trigger appending its current description to the scratchpad, so the scratchpad becomes an explicit, readable memory of the episode. Experiments show that this converts a stateless transformer VLA from near-zero to substantial success on memory-dependent tasks, and also improves a recurrent state-space VLA, though the method does not help tasks requiring fine-grained low-level temporal memor

What carries the argument

The scratchpad itself: a textual record divided into three sections—'grounding' (initial object and end-effector positions), 'plan' (ordered sub-tasks), and 'act' (completed sub-tasks). The update mechanism is a special <done> token that, when predicted, appends the current description to the scratchpad (Algorithm 1). This turns a stateless policy into a stateful one by allowing the model to condition every future action on its own past linguistic notes, giving both spatial memory (positions) and temporal memory (progress tracking).

Load-bearing premise

At test time the model must write an accurate scratchpad from the first frame—object positions, the plan, and the right moments to emit 'done' tokens—because the paper's simulation results use oracle-generated scratchpads during training but let the model predict the plan autonomously during evaluation, and the paper never measures how correct those self-generated notes are.

What would settle it

If an experiment corrupts or scrambles the scratchpad content at test time (for instance, deleting all 'done' updates or writing wrong positions) and the policy's success rate does not drop, then the reported gains are not actually caused by the scratchpad memory.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Stateless vision-language-action models can be endowed with memory for non-Markovian tasks without architectural change, by leveraging their existing language output abilities.
  • Recurrent policies, despite having an internal hidden state, still benefit from an explicit scratchpad on long-horizon tasks, suggesting that explicit memory complements rather than replaces recurrence.
  • The scratchpad is flexible about what it stores—3D coordinates or 2D image pointers, semantic or arbitrary action labels—so the recipe may transfer across embodiments and task families.
  • The gap between the scratchpad policy and perfect performance on a precise-manipulation benchmark is mostly motor precision, not memory recall: with a small assist on the hard motor action, the scratchpad policy reaches 100% success.
  • The approach yields an inspectable text trace of the robot's reasoning, which could be used for debugging or verification of the policy's behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the scratchpad can be generated autonomously from the first observation without oracle initialization, the same mechanism could provide open-ended robots with a lightweight episodic memory that is independent of hidden-state capacity.
  • The 'done' token acts as a learned sub-goal segmentation signal; one testable extension is whether it can be trained from reward or self-supervision rather than oracle subtask labels, which would broaden applicability.
  • Because the scratchpad is textual, a human operator could in principle read and correct it mid-episode; this suggests an intervention-based safety mechanism the paper does not explore.
  • A natural stress test would be to scale up task horizon and measure whether scratchpad performance stays flat where recurrent hidden-state memory degrades, and whether the choice of storing coordinates versus object names changes robustness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes augmenting vision-language-action (VLA) models with a language scratchpad that stores object grounding, a sub-task plan, and an evolving action history. The policy conditions on this scratchpad and emits structured text tokens (plan, act, <done>) that update the memory at test time. The authors evaluate the approach on a new ClevrSkills-Mem split, on MemoryBench's Put-Block-Back task, and on a real-world pick-place-restore task, comparing T-VLA (PaliGemma-2 3B) and R-VLA (Mamba 130M) with and without the scratchpad. They report large gains for the non-recurrent model and moderate gains for the recurrent model, and they claim parity with specialized methods on MemoryBench when using a relaxed 'sim-eval' protocol.

Significance. If validated, the work would provide a simple and broadly applicable way to endow stateless VLAs with spatial and temporal memory, leveraging existing language heads rather than architectural change. The ClevrSkills-Mem benchmark and the real-world task are useful contributions, and the method is reproduced from clearly described hyperparameters. However, the evidence as presented does not fully support the headline claims: success rates are single-run point estimates without confidence intervals, the MemoryBench 100% result relies on an oracle button-press intervention, and no ablation separates the scratchpad context from the auxiliary language prediction task. The idea is promising, but the current evaluation needs strengthening.

major comments (4)
  1. [Fig. 3; Sec. V.B] All ClevrSkills-Mem results are point estimates from 50 rollouts with no standard errors, confidence intervals, or repeated seeds. The text states 'significant improvements' and reports an average gain of 48.8%, but the R-VLA gain is 11% and several task-level differences are between a near-zero baseline and a moderate value. Without variance or a significance test, these claims are not statistically supported. Please report per-seed results or confidence intervals, and avoid 'significant' unless tested.
  2. [Table I; Sec. V.C] The 100% success for 'T-VLA w/ SP (sim-eval)' is obtained by replacing the model's predicted button-press with the ground-truth action when the model gets close. This is an oracle intervention that is not applied to the RVT-2 or SAM2Act baselines. In the non-intervention row, T-VLA w/ SP achieves 40%, below RVT-2's 50%. The claim that a generalist VLA with scratchpad 'achieve[s] performance similar to highly specialized, task-specific methods' is therefore supported only by the oracle-augmented number. The standard evaluation should be the headline, and sim-eval should be explicitly labeled as a diagnostic, not a competitive result.
  3. [Sec. III.A; Sec. V.A; Alg. 1] The proposed method combines two changes: the model is trained to output structured language descriptions (plan/act/<done>), and the resulting scratchpad is fed back as input at test time. The comparison between T-VLA and T-VLA w/ SP does not control for the auxiliary prediction task itself. A control that predicts the same descriptions but does not condition on the scratchpad (e.g., fixed or empty scratchpad context) would isolate whether the benefit comes from the memory channel or from additional supervision/regularization. An oracle-scratchpad condition would also bound the benefit. Without such ablations, the central claim that the scratchpad imparts memory is not fully established.
  4. [Sec. V.A; Alg. 1] The paper does not measure the fidelity of the self-generated scratchpad at test time: accuracy of predicted object coordinates, correctness of plan tokens, or precision/recall of <done> at sub-task boundaries. While ClevrSkills evaluation uses self-generated notes (not oracle-initialized ones), this missing measurement makes failures hard to attribute and raises robustness concerns for transfer. Please report scratchpad token/coordinate accuracy and <done> timing metrics, and, if possible, success conditioned on correct versus incorrect scratchpad updates.
minor comments (5)
  1. [Sec. III (Preliminaries)] There is a typo: 'disctretized' should be 'discretized'.
  2. [Alg. 1] The notation 'd_t ∈ d_update' is unclear. Define what set d_update contains (e.g., a special <done> token) before using it.
  3. [Table II] The 'Sub-task CR' column should specify units: average number of completed sub-tasks out of 3, or a fraction, and how it is computed.
  4. [Sec. V.C] The 'sim-eval' criterion 'if the VLA gets close to the button' is vague. Specify the distance threshold used to trigger the ground-truth button press.
  5. [Sec. IV] The benchmark name appears as 'ClevrSkills-Mem' in the abstract and as 'ClevrSkills-Mem' later; ensure the hyphenation is consistent throughout.

Circularity Check

0 steps flagged

No significant circularity: scratchpad gains are measured by environment success, not by reproducing the scratchpad labels.

full rationale

The paper's central claim is an empirical comparison: policies with and without the scratchpad are evaluated by success rate on ClevrSkills-Mem, MemoryBench, and a real-world task. Success is defined by task completion (e.g., object placement within a distance threshold), not by matching the oracle-generated scratchpad, so there is no self-definitional reduction. The scratchpad is trained from oracle solver traces, and at evaluation the model must generate its own plan and <done> tokens; if these are inaccurate the policy can fail, which is an unmeasured robustness gap, not a construction that forces the reported result. The ClevrSkills-Mem split is built from the authors' own ClevrSkills benchmark, but MemoryBench ([7], external work) and the real-world Pick-Place-Restore task provide independent support for the central claim. The 'sim-eval' MemoryBench row replaces only the button-press action with a ground-truth action and is explicitly disclosed; it does not define success as matching the scratchpad. No circular step is exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claim rests on hand-designed choices and domain assumptions rather than a derivation: the scratchpad's three sections, the oracle subtask segmentation, and test-time generation of the plan are all assumed to work. External anchoring comes from the MemoryBench task and the real-world rollouts, both with small N.

free parameters (4)
  • Scratchpad structure (grounding/plan/act sections + <done> trigger) = hand-designed
    The three-section scratchpad and the update token are chosen by the authors to match the oracle's sub-task segmentation; no ablation shows which component drives the gain.
  • MemoryBench sim-eval oracle button threshold = unspecified ("gets close to the button")
    The relaxed evaluation replaces the model's button press with a ground-truth action when the model is close; the closeness threshold is not defined, so the reported 100% success depends on a hand-set intervention.
  • Base model selection (PaliGemma-2 3B for T-VLA, Mamba 130M for R-VLA) = 3B / 130M
    Authors report trying 1B/3B Mamba and selecting 130M because of 'little to no performance difference'; model size is chosen empirically on the same task distribution.
  • LoRA rank 32 for real-world OpenVLA tuning = 32
    Standard LoRA rank used for the real-world experiment; no sweep or sensitivity report is given.
axioms (4)
  • domain assumption ClevrSkills oracle solver trajectories and subtask segmentation are correct and unbiased for generating scratchpad labels
    Section V.B relies on 'an oracle solver to generate data including actions, object positions, and solver traces which are used to get sub-task segmentation to create scratchpad.' If the oracle segmentation leaks the desired plan in a way that differs from what a model could infer from images, the scratchpad gains are inflated.
  • domain assumption The 50 ClevrSkills-Mem rollouts and 20 real-world rollouts with unseen or randomized positions are representative of generalization
    Sections V.B and V.D report single-run success rates without confidence intervals; results could shift noticeably with different seeds.
  • domain assumption RGB-only OpenVLA is comparable to point-cloud baselines on MemoryBench
    Table I compares against RVT-2 and SAM2Act, which take 3D point clouds, while the paper's T-VLA uses RGB images. The paper attributes failures to button-press precision, but the comparison is not apples-to-apples.
  • domain assumption The pretrained VLM can faithfully generate and parse English scratchpad text containing coordinates and plan labels
    The method relies on the VLM's language capabilities (Section III.A); if the coordinate or plan format is outside the VLM's pretraining distribution, the scratchpad could mislead rather than help.
invented entities (1)
  • Language scratchpad memory (grounding/plan/act context) no independent evidence
    purpose: External, evolving textual memory of object positions, subgoal plan, and completed steps provided to the policy at every step
    This is an internal mechanism rather than an independently falsifiable scientific entity; its utility is only evidenced by the model performance reported in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 11817 in / 11345 out tokens · 107410 ms · 2026-08-02T21:08:12.255723+00:00 · methodology

0 comments
read the original abstract

Many dexterous manipulation tasks are non-markovian in nature, yet little attention has been paid to this fact in the recent upsurge of the vision-language-action (VLA) paradigm. Although they are successful in bringing internet-scale semantic understanding to robotics, existing VLAs are primarily "stateless" and struggle with memory-dependent long horizon tasks. In this work, we explore a way to impart both spatial and temporal memory to a VLA by incorporating a language scratchpad. The scratchpad makes it possible to memorize task-specific information, such as object positions, and it allows the model to keep track of a plan and progress towards subgoals within that plan. We evaluate this approach on a split of memory-dependent tasks from the ClevrSkills environment, on MemoryBench, as well as on a challenging real-world pick-and-place task. We show that incorporating a language scratchpad significantly improves generalization on these tasks for both non-recurrent and recurrent models.

Figures

Figures reproduced from arXiv: 2602.21013 by Apratim Bhattacharyya, Daniel Dijkman, Roland Memisevic, Sanjay Haresh.

Figure 1
Figure 1. Figure 1: Scratchpad-augmented VLAs. The VLA generates and updates scratchpad which is stored and provided as part of the input context for all [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example memory dependent tasks from ClevrSkills-Mem we evaluate on. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results on the ClevrSkills-Mem benchmark. We show results for all the models considered with and without scratchpad on 5 tasks of ClevrSkills-Mem benchmark. We report success rate of each model on 50 rollouts on unseen starting positions objects. On the right we show the average performance across all tasks. the challenging nature of the memory-dependent ClevrSkills tasks, T-VLA struggles on all the tasks,… view at source ↗
Figure 4
Figure 4. Figure 4: Average trajectory length of tasks in ClevrSkills-Mem. Here, TRP [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Key frames of the real world task: Pick the tomato, place it in the bowl and then restore it to the initial position. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation

    cs.RO 2026-07 conditional novelty 7.0

    A vision-language-action robot policy that stores compressed wrist-force histories as memory tokens can count contact events that are visually ambiguous, achieving 83.3% average success across three tasks.

  2. $\mu$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models

    cs.LG 2026-06 unverdicted novelty 6.0

    Adding recurrent memory tokens to VLA models raises success rates on partially observable manipulation tasks from 0.42 to 0.84 on training and 0.07 to 0.23 on held-out tasks while preserving performance under full obs...

Reference graph

Works this paper leans on

44 extracted references · 22 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis,et al., “Rt-1: Robotics transformer for real-world control at scale,”arXiv preprint arXiv:2212.06817, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Rt- 2: Vision-language-action models transfer web knowledge to robotic control. arxiv,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen,et al., “Rt- 2: Vision-language-action models transfer web knowledge to robotic control. arxiv,”arXiv preprint arXiv:2307.15818, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, et al., “Openvla: An open-source vision-language-action model,”arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    π 0: A vision-language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi,et al., “π 0: A vision-language-action flow model for general robot control,”arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain,et al., “Open x-embodiment: Robotic learning datasets and rt-x models: Open x- embodiment collaboration 0,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903

    2024

  6. [6]

    Clevrskills: Compositional language and visual reasoning in robotics,

    S. Haresh, D. Dijkman, A. Bhattacharyya, and R. Memise- vic, “Clevrskills: Compositional language and visual reasoning in robotics,”Advances in Neural Information Processing Systems, vol. 37, pp. 38 235–38 266, 2024

    2024

  7. [7]

    Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation,

    H. Fang, M. Grotz, W. Pumacay, Y . R. Wang, D. Fox, R. Krishna, and J. Duan, “Sam2act: Integrating visual foundation model with a memory architecture for robotic manipulation,”arXiv preprint arXiv:2501.18564, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    A generalist agent,

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg, et al., “A generalist agent,”arXiv preprint arXiv:2205.06175, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Octo: An open-source generalist robot policy,

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine, “Octo: An open-source generalist robot policy,” inProceedings of Robotics: Science and Systems, Delft, Netherlands, 2024

    2024

  10. [10]

    Jack of all trades, master of some, a multi-purpose transformer agent,

    Q. Gallou ´edec, E. Beeching, C. Romac, and E. Dellandr ´ea, “Jack of all trades, master of some, a multi-purpose transformer agent,”arXiv preprint arXiv:2402.09844, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    π0.5: a vision- language-action model with open-world generalization,

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker,et al., “π0.5: a vision- language-action model with open-world generalization,”arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Spatialvla: Exploring spatial representations for visual- language-action model,

    D. Qu, H. Song, Q. Chen, Y . Yao, X. Ye, Y . Ding, Z. Wang, J. Gu,et al., “Spatialvla: Exploring spatial representations for visual- language-action model,”arXiv preprint arXiv:2501.15830, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    3d-vla: A 3d vision-language-action generative world model,

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan, “3d-vla: A 3d vision-language-action generative world model,”arXiv preprint arXiv:2403.09631, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Vision-language foundation models as effective robot imitators,

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu,et al., “Vision-language foundation models as effective robot imitators,”arXiv preprint arXiv:2311.01378, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,

    J. Liu, M. Liu, Z. Wang, P. An, X. Li, K. Zhou, S. Yang, R. Zhang, Y . Guo, and S. Zhang, “Robomamba: Efficient vision-language-action model for robotic reasoning and manipulation,”Advances in Neural Information Processing Systems, vol. 37, pp. 40 085–40 110, 2024

    2024

  16. [16]

    A large recurrent action model: xlstm enables fast inference for robotics tasks,

    T. Schmied, T. Adler, V . Patil, M. Beck, K. P ¨oppel, J. Brandstetter, G. Klambauer, R. Pascanu, and S. Hochreiter, “A large recurrent action model: xlstm enables fast inference for robotics tasks,”arXiv preprint arXiv:2410.22391, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Robotic control via embodied chain-of-thought reasoning,

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,”arXiv preprint arXiv:2407.08693, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Llara: Su- percharging robot learning data for vision-language policy,

    X. Li, C. Mata, J. Park, K. Kahatapitiya, Y . S. Jang, J. Shang, K. Ranasinghe, R. Burgert, M. Cai, Y . J. Lee,et al., “Llara: Su- percharging robot learning data for vision-language policy,”arXiv preprint arXiv:2406.20095, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Towards an episodic memory for cognitive robots,

    S. Jockel, M. Weser, D. Westhoff, and J. Zhang, “Towards an episodic memory for cognitive robots,” inProc. of 6th Cognitive Robotics workshop at 18th European Conf. on Artificial Intelligence (ECAI), 2008, pp. 68–74

    2008

  20. [20]

    Simultaneous localization and mapping (slam): Part ii,

    T. Bailey and H. Durrant-Whyte, “Simultaneous localization and mapping (slam): Part ii,”IEEE robotics & automation magazine, vol. 13, no. 3, pp. 108–117, 2006

    2006

  21. [21]

    Object goal navigation using goal-oriented semantic exploration,

    D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” Advances in Neural Information Processing Systems, vol. 33, pp. 4247–4258, 2020

    2020

  22. [22]

    Visual simultaneous localization and mapping: a survey,

    J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rend ´on-Mancha, “Visual simultaneous localization and mapping: a survey,”Artificial intelligence review, vol. 43, no. 1, pp. 55–81, 2015

    2015

  23. [23]

    Memory-augmented reinforcement learning for image-goal navigation,

    L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari, “Memory-augmented reinforcement learning for image-goal navigation,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 3316–3323

    2022

  24. [24]

    Scene memory trans- former for embodied agents in long-horizon tasks,

    K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese, “Scene memory trans- former for embodied agents in long-horizon tasks,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 538–547

    2019

  25. [25]

    Emergence of maps in the memories of blind navigation agents,

    E. Wijmans, M. Savva, I. Essa, S. Lee, A. S. Morcos, and D. Batra, “Emergence of maps in the memories of blind navigation agents,”AI Matters, vol. 9, no. 2, pp. 8–14, 2023

    2023

  26. [26]

    Copa: General robotic manipulation through spatial constraints of parts with foundation models,

    H. Huang, F. Lin, Y . Hu, S. Wang, and Y . Gao, “Copa: General robotic manipulation through spatial constraints of parts with foundation models,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 9488–9495

    2024

  27. [27]

    V oxposer: Composable 3d value maps for robotic manipulation with language models,

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei, “V oxposer: Composable 3d value maps for robotic manipulation with language models,”arXiv preprint arXiv:2307.05973, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Out of sight, still in mind: Reasoning and planning about unobserved objects with video tracking enabled memory models,

    Y . Huang, J. Yuan, C. Kim, P. Pradhan, B. Chen, L. Fuxin, and T. Hermans, “Out of sight, still in mind: Reasoning and planning about unobserved objects with video tracking enabled memory models,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 3108–3115

    2024

  29. [29]

    Robot-r1: Reinforcement learning for enhanced embodied reasoning in robotics,

    D. Kim, S. Park, H. Jang, J. Shin, J. Kim, and Y . Seo, “Robot-r1: Reinforcement learning for enhanced embodied reasoning in robotics,” arXiv preprint arXiv:2506.00070, 2025

    arXiv 2025

  30. [30]

    Show your work: Scratchpads for intermediate computation with language models,

    M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, et al., “Show your work: Scratchpads for intermediate computation with language models,”arXiv preprint arXiv:2112.00114, 2021

    Pith/arXiv arXiv 2021

  31. [31]

    Towards generalist robot policies: What matters in building vision-language-action models,

    H. Liu, X. Li, P. Li, M. Liu, D. Wang, J. Liu, B. Kang, X. Ma, T. Kong, and H. Zhang, “Towards generalist robot policies: What matters in building vision-language-action models,”arXiv preprint arXiv:2412.14058, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions,

    C. Fan, X. Jia, Y . Sun, Y . Wang, J. Wei, Z. Gong, X. Zhao, M. Tomizuka, X. Yang, J. Yan,et al., “Interleave-vla: Enhancing robot manipulation with interleaved image-text instructions,”arXiv preprint arXiv:2505.02152, 2025

    arXiv 2025

  33. [33]

    Neural networks and the chomsky hierarchy,

    G. Deletang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and P. A. Ortega, “Neural networks and the chomsky hierarchy,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=WbxHAzkeQcn

    2023

  34. [34]

    Maniskill2: A unified benchmark for generalizable manipulation skills,

    J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y . Tang, S. Tao, X. Wei, Y . Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su, “Maniskill2: A unified benchmark for generalizable manipulation skills,” inInter- national Conference on Learning Representations, 2023

    2023

  35. [35]

    Paligemma 2: A family of versatile vlms for transfer,

    A. Steiner, A. S. Pinto, M. Tschannen, D. Keysers, X. Wang,et al., “Paligemma 2: A family of versatile vlms for transfer,”arXiv preprint arXiv:2412.03555, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Gemma 2: Improving open language models at a practical size,

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin,et al., “Gemma 2: Improving open language models at a practical size,”arXiv preprint arXiv:2408.00118, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  38. [38]

    Adam: A method for stochastic optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,”arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  39. [39]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  41. [41]

    Py- torch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury,et al., “Py- torch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019

    2019

  42. [42]

    Rlbench: The robot learning benchmark & learning environment,

    S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,”IEEE Robotics and Automation Letters, 2020

    2020

  43. [43]

    Rvt2: Learning precise manipulation from few demonstrations,

    A. Goyal, V . Blukis, J. Xu, Y . Guo, Y .-W. Chao, and D. Fox, “Rvt2: Learning precise manipulation from few demonstrations,”RSS, 2024

    2024

  44. [44]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022