Pith. sign in

REVIEW 4 major objections 4 minor 11 cited by

SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fusing the SAM2 vision encoder's multi-resolution embeddings into a coarse-to-fine manipulation transformer produces a policy with 86.8% average success on 18 tasks, and that adding a SAM2-style memory bank lets it…

desk verdict Strong RLBench gains from SAM2 multi-res upsampling, but the MemoryBench headline rests on per-task tuning and the Colosseum 4.3% claim hides a 58-point combined drop. read the letter →

arxiv 2501.18564 v4 pith:ZSKGITFO submitted 2025-01-30 cs.RO

Haoquan Fang , Markus Grotz , Wilbert Pumacay , Yi Ru Wang , Dieter Fox , Ranjay Krishna , Jiafei Duan This is my paper · ORCID
classification cs.RO
keywords roboticmanipulationimitationlearningbehaviorcloningmulti-viewtransformerspatialmemorymemory-basedarchitecturegeneralizationvisualfoundationmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a robotic manipulation policy can get both higher precision and better generalization by reusing the multi-resolution visual embeddings of a large video-segmentation model rather than training a task-specific visual encoder. It introduces SAM2Act, a language-conditioned multi-view behavior-cloning policy whose coarse-to-fine transformer fuses those embeddings at three resolutions during heatmap upsampling; on an 18-task benchmark it reports an average 86.8% success, and under a suite of environmental perturbations it drops only 4.3%. It then extends the same architecture into SAM2Act+, adding a memory bank, encoder, and attention module so the policy can recall past predicted positions. On three deliberately non-Markovian tasks built to require spatial memory, SAM2Act+ reports 94.3% average success, far above memory-less baselines. The paper also contributes that benchmark, MemoryBench, for measuring spatial memory in behavior-cloning agents.

What carries the argument

The mechanism is the SAM2Act module: three cascaded convex upsamplers that double spatial resolution at each stage, each combining the upsampled feature map with the matching multi-resolution embedding from the SAM2 image encoder via $X^{l+1} = \mathrm{LayerNorm}(U(X^l) \oplus E^l)$. The memory extension adds a per-view FIFO memory bank, a memory encoder that compresses the predicted translation heatmap with the observation embedding, and memory attention that cross-attends current features to stored past memories. The heatmap plays the role that predicted masks play in SAM2's video tracking, so the policy conditions its next translation prediction on where it previously acted.

What would settle it

Run SAM2Act+ and the best memory-less baseline on MemoryBench under a single fixed memory window size for all tasks and the same epoch-selection rule; if the gap over the memory-less model falls well below the reported 39.3 points or near chance, the memory claim as stated does not hold.

Watch

Extended reading notes

Core claim

The central claim is that multi-resolution embeddings from the SAM2 image encoder, injected through cascaded convex upsamplers, give a manipulation policy materially better spatial precision and robustness, and that the same video model's memory components can be transplanted to give policies episodic spatial memory. The authors state this as a state-of-the-art 86.8% average over 18 RLBench tasks, with the largest gains on precision-heavy peg insertion and shape sorting; a 4.3% average degradation under thirteen perturbation types on the Colosseum suite; and a 94.3% average on the three MemoryBench tasks, compared with 55.0% for the memory-less SAM2Act and 54.0% for the prior baseline. They stress that the memory tasks are non-Markovian: the same current observation can require different next actions, so success is impossible without recalling which action history led to it.

Load-bearing premise

The headline memory advantage assumes fair evaluation under a uniform protocol, yet the reported SAM2Act+ numbers use per-task memory-window sizes and best-epoch checkpoint selection in a way that is not applied equally to the baselines.

Editorial extensions

If this is right

  • A single behavior-cloning policy can handle 18 varied manipulation tasks with an average success rate near 87%, with the largest gains on high-precision tasks such as inserting a ring onto a peg (84% vs 40% for the previous best baseline).
  • The same policy loses on average only 4.3% when evaluated under 13 perturbation categories such as lighting, table texture, distractors, and camera pose, indicating that visual-feature robustness transfers to manipulation.
  • Tasks that violate the Markov property, where the same frame can require different actions, are solvable by explicit memory; without it, the best tested policy stays near chance on those tasks.
  • Memory can be added without retraining the full model: the visual encoder and fine branch are frozen, and only the memory components plus the coarse branch are fine-tuned, reducing the cost of acquiring episodic recall.
  • A benchmark with three memory tasks, reopening a drawer, putting a block back, and rearranging blocks, can separate memory-capable from memory-less policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its MemoryBench numbers are tuned: the memory window size is chosen per task and the best-performing epoch is selected for every model, so the 94.3% headline is best read as an upper bound under a tuned protocol.
  • Because the memory bank stores only translation heatmaps, not rotation, gripper state, or object identity, the architecture is naturally suited to spatial recall; extending the memory encoder to consume those other signals might let the same design solve tasks requiring semantic memory.
  • The fixed-length FIFO memory makes the approach most applicable to tasks with short, bounded horizons; tasks with unbounded histories or continuous control would likely need learned memory compression or a recurrent state.
  • Task-by-task inspection of the perturbation table shows several tasks with low baseline success even without perturbations, so the 4.3% average gap is not uniform across tasks; future work could separate robustness gains from tasks that are simply easier.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SAM2Act, a behavior-cloning policy built on RVT-2 that replaces the visual backbone with a LoRA-fine-tuned SAM2 image encoder and adds a cascaded multi-resolution convex upsampling module. It reports 86.8% average success across 18 RLBench tasks, a 4.3% average performance drop under individually applied Colosseum perturbations, and a 94.3% average success on MemoryBench, a new three-task suite of non-Markovian manipulation tasks, using the memory-augmented variant SAM2Act+. Real-world experiments on a Franka Panda compare SAM2Act with RVT-2 and include a memory-style button-pressing task. The paper includes ablations of the SAM2 encoder and multi-resolution upsampling, full comparison tables in appendices, and a disclosure of the MemoryBench evaluation protocol.

Significance. If the RLBench results hold, SAM2Act would be a meaningful advance over RVT-2 and other keyframe-based BC policies, with particularly large gains on precision tasks such as Insert Peg and Sort Shape. The multi-resolution upsampling ablation (Table 9) and the Colosseum robustness results support the architectural claims. The MemoryBench benchmark is a useful idea, and the authors are transparent about its evaluation protocol. However, the memory claim is currently under-supported: the protocol uses best-epoch selection and per-task window sizes, the baselines are all memory-less, and one of the three tasks appears partly solvable without explicit memory. These issues need to be addressed before the 94.3% result can be taken at face value.

major comments (4)
  1. [§5.4, Appendix B.2, Table 6] The 94.3% MemoryBench headline rests on a protocol in which the best-performing epoch is selected for every model and the memory window size differs per task (window 8 for reopen_drawer, 10 for the other two tasks; learning rates and batch sizes also differ). This can inflate the reported average, particularly with only 25 held-out demonstrations and 4 evaluation runs, and the large standard deviation reported for SAM2Act (24.3 on the average) shows that epoch-level noise is substantial. Appendix H says the window was later standardized to 10 for SAM2Act+ and results updated, but Appendix B.2 and Table 6 still describe per-task windows, so the paper is internally inconsistent. Please report results with a pre-specified epoch-selection rule applied identically to all models, a single memory window, and per-epoch performance curves.
  2. [§5.4, Table 3] The memory comparison includes only memory-less baselines (RVT-2 and SAM2Act without memory). Since MemoryBench is deliberately non-Markovian, a memory-less policy is handicapped by construction, and the 39.3% margin does not isolate the contribution of the proposed memory bank, memory encoder, and memory attention. Please add a history-aware baseline, such as a frame-stacking variant of SAM2Act or RVT-2, a recurrent policy, or an existing history-based method (e.g., HiveFormer or the memory model cited as [41]), to show that the gains come from the memory architecture rather than from merely receiving past observations.
  3. [§3.2, Table 3] SAM2Act without memory reaches 82% on rearrange_block, where random chance is 50%, despite the task being designed to be non-Markovian. This suggests that the current observation may contain residual cues or that the two blocks are visually distinguishable, so the task may not be a pure memory test. Please verify that the decision-point observation is truly ambiguous (for example, by checking that the blocks are visually identical and that a memory-less policy operates near chance), or revise the task and environment accordingly.
  4. [§4.1, update equation] The update equation X^{l+1} = LayerNorm(U(X^l) ⊕ E^l) is dimensionally inconsistent as written: U(X^l) has C_l/2 channels while E^l is defined as having C_l channels, so element-wise addition is undefined. Please specify how E^l is projected or otherwise aligned before addition, or correct the notation. This is necessary to reproduce the multi-resolution upsampling mechanism that underlies the reported RLBench gains.
minor comments (4)
  1. [§5.2, Table 1] The main text says SAM2Act surpasses RVT-2 by 5.4%, while the Table 1 caption states a margin of 5.8%; please make these numbers consistent.
  2. [§5.4, Table 3 caption] The caption states an average improvement of 37.6% across all three tasks, but the per-task differences in the table give 40.3 percentage points over RVT-2 and 39.3 percentage points over SAM2Act; please clarify how the 37.6% figure is computed.
  3. [§5.3, Table 2] The abstract emphasizes a 4.3% average drop, but Table 2 shows a 58.3% drop for SAM2Act in the All Perturbations column when perturbations are combined; please state this distinction explicitly in the main text. Additionally, the Camera Pose value for SAM2Act is -4.3±3.6, identical to the Average value, which appears to be a transcription error.
  4. [§5.5, Table 4] The real-world memory task (d) is evaluated with only 10 in-distribution and 10 out-of-distribution trials; please acknowledge the limited statistical power or provide confidence intervals.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity: SAM2Act's and SAM2Act+'s claims are empirical benchmark results, with the main caveats being evaluation-protocol choices rather than input-output equivalence.

full rationale

The paper contains no derivation chain whose outputs are equivalent to its inputs by construction. The central claims are empirical success rates measured on RLBench, The Colosseum, MemoryBench, and real-robot trials. The architectural equations, such as X^{l+1} = LayerNorm(U(X^l) ⊕ E^l) for multi-resolution upsampling and Algorithm 1 for the memory forward pass, are constructive definitions rather than theorems, and their support comes from comparative experiments against baselines trained under the same benchmark protocols. MemoryBench tasks are designed to violate the Markov property (Section 3.1), but all compared policies are evaluated on the same tasks, so the memory mechanism's advantage is an empirical outcome, not an identity. The paper's citations to RVT-2 as the backbone and to The Colosseum as a robustness benchmark involve overlapping authors, but these are used as reproducible prior tools and are not invoked as unverified premises or uniqueness arguments to force the authors' choices. The most significant qualification in the manuscript is the MemoryBench evaluation protocol: Appendix B.2 states that models are 'evaluate[d] across all epochs and select the best-performing one for evaluation' and that the memory window size 'is also decided to be different for each task in MemoryBench,' which can inflate the reported 94.3% average and should be weighed as an evaluation-fairness and generalization risk rather than as circularity. Appendix H reports that the reopen_drawer task was retrained with a uniform window size of 10 and that 'all results are updated to Table 3,' partially addressing the nonuniformity, while the best-epoch selection remains. Section 6 also candidly notes the fixed, task-dependent memory window as a limitation. These are correctness concerns about statistical selection, not cases where a predicted quantity reduces to a fitted parameter by definition. No circular step can be quoted and exhibited, so the correct score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No physical entities are invented. The method introduces software components (MemoryBank, Memory Encoder, Memory Attention) but these are not postulates requiring independent evidence. The only free parameter that materially affects the central memory claim is the per-task memory window size.

free parameters (1)
  • memory window size = 8 for reopen_drawer, 10 for the other two tasks
    Chosen per task in Appendix B.2 and Table 6; the method's performance depends on this and no single fixed window is used across tasks.
assumptions (3)
  • domain assumption SAM2 multi-resolution image embeddings improve action heatmap prediction in manipulation beyond other encoders.
    This is the central design premise of SAM2Act and is tested only through ablations on the same benchmark, not on an independent benchmark.
  • domain assumption Translation heatmaps can be treated analogously to object masks for the SAM2 memory mechanism.
    Section 4.2 states the conceptual alignment between predicted heatmaps and SAM2 masks, with no formal justification for why the transfer works.
  • domain assumption Virtual camera rendering from point clouds is a sufficient representation for 6-DoF manipulation.
    Inherited from RVT-2 and used without re-justification in Sections 4.1 and 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation." pith.science (2026). https://pith.science/paper/ZSKGITFO

@misc{pith2026250118564,
  author       = {Pith},
  title        = {Pith review of: SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZSKGITFO}},
  note         = {Machine review of arXiv:2501.18564}
}
read the original abstract

Robotic manipulation systems operating in diverse, dynamic environments must exhibit three critical abilities: multitask interaction, generalization to unseen scenarios, and spatial memory. While significant progress has been made in robotic manipulation, existing approaches often fall short in generalization to complex environmental variations and addressing memory-dependent tasks. To bridge this gap, we introduce SAM2Act, a multi-view robotic transformer-based policy that leverages multi-resolution upsampling with visual representations from large-scale foundation model. SAM2Act achieves a state-of-the-art average success rate of 86.8% across 18 tasks in the RLBench benchmark, and demonstrates robust generalization on The Colosseum benchmark, with only a 4.3% performance gap under diverse environmental perturbations. Building on this foundation, we propose SAM2Act+, a memory-based architecture inspired by SAM2, which incorporates a memory bank, an encoder, and an attention mechanism to enhance spatial memory. To address the need for evaluating memory-dependent tasks, we introduce MemoryBench, a novel benchmark designed to assess spatial memory and action recall in robotic manipulation. SAM2Act+ achieves an average success rate of 94.3% on memory-based tasks in MemoryBench, significantly outperforming existing approaches and pushing the boundaries of memory-based robotic systems. Project page: sam2act.github.io.

Figures

Figures reproduced from arXiv: 2501.18564 by the authors.

Figure 1
Figure 1. SAM2Act is a multi-view, language-conditioned behavior cloning policy trained with fewer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Simulation and Real Tasks. We demonstrate the effectiveness of SAM2Act+ in solving memory-based tasks by evaluating it against baselines on the three benchmark memory tasks (shown at the top). Additionally, we validate our approach using a Franka Panda robot on four real-world tasks (shown at the bottom), including tests under out-of-distribution perturbations. require different subsequent actions. This forces the a… view at source ↗
Figure 3
Figure 3. Overview of the SAM2Act (top) and SAM2Act+ (bottom) architectures. The SAM2Act architecture leverages the SAM2 image encoder to generate prompt-conditioned, multi-resolution embeddings, fine-tuned with LoRA for efficient adaptation to manipulation tasks. A multi-view transformer aligns spatial coordinates with language instructions, while a cascaded multi-resolution upsampling mechanism refines feature maps and gene… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: SAM2Act Module and multi-resolution upsampling mechanism. A cascade of three convex upsamplers processes feature maps at increasing resolutions, integrating multi-resolution embeddings from the SAM2 image encoder through elementwise addition and layer normalization. Th…
Figure 5
Figure 5. Figure 5: Real-world Robot Setup. A Franka Panda robot with a Robotiq Gripper. A RealSense D455 depth sensor captures the scene. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Repurposing a VLA’s vision encoder to emit one action-supervised memory token per historical frame-view yields long-horizon manipulation with large success gains and low latency.

  2. DIM-WAM: World-Action Modeling with Diverse Historical Event Memory

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    Multi-bank similarity-merged event memory plus progress supervision raises long-horizon WAM success from 28.4% to 69.8% on RMBench and full-task success from 52.5% to 80% on real Franka tasks.

  3. Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks

    cs.RO 2026-02 conditional novelty 6.0 of 10

    A language scratchpad that records object positions, the plan, and completed subgoals lets vision-language-action policies succeed on memory-dependent manipulation tasks that stateless baselines fail.

  4. LMPVC and Policy Bank: Adaptive voice control for industrial robots with code generating LLMs and reusable Pythonic policies

    cs.RO 2025-06 conditional novelty 6.0 of 10

    LMPVC and the Policy Bank let users control an industrial robot by voice, teach it reusable Python policies, and have a local code-generating LLM call those policies automatically.

  5. EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulation

    cs.RO 2025-05 conditional novelty 6.0 of 10

    EquAct embeds SE(3) equivariance into a multi-task keyframe manipulation transformer with language conditioning, improving spatial generalization over non-equivariant baselines.

  6. Differential Amplifier-Inspired AmpAttention for Multi-View Robotic Manipulation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    AmpAttention and RVAF raise multi-view robotic manipulation success and cut training time by suppressing attention noise with a differential-amplifier-style mechanism plus a CMRR loss.

  7. MemoAct: Atkinson-Shiffrin-Inspired Hierarchical Memory-Augmented Policy for Robotic Manipulation

    cs.RO 2026-03 conditional novelty 5.0 of 10

    MemoAct combines a lossless short-term memory bank with a compressed long-term memory bank, enabling robotic manipulation policies to track task state precisely and retain long-horizon context, outperforming prior mem...

  8. RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design

    cs.RO 2026-03 conditional novelty 5.0 of 10

    A benchmark and modular policy show that explicit memory components substantially improve robotic manipulation on tasks requiring recall of past observations.

  9. RoboPearls: Editable Video Simulation for Robot Manipulation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    RoboPearls is a 3D Gaussian Splatting based framework that edits demonstration videos into varied photorealistic simulations, and training on them improves robot manipulation success rates on RLBench and COLOSSEUM.

  10. Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges

    cs.RO 2025-08 conditional novelty 4.0 of 10

    A survey that taxonomizes robotic manipulation policies trained by imitation learning, traces their evolution, and compiles benchmark comparisons.

  11. Data Pyramid for Embodied Manipulation: A Survey

    cs.RO 2026-07 conditional novelty 3.0 of 10

    Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.

Reference graph

Works this paper leans on

59 extracted references · 22 canonical work pages · cited by 11 Pith papers

  1. [41]

    Huang, J

    Y . Huang, J. Yuan, C. Kim, P. Pradhan, B. Chen, L. Fuxin, and T. Hermans. Out of sight, still in mind: Reasoning and planning about unobserved objects with video tracking enabled memory models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 3108–3115. IEEE, 2024

    work page 2024

  2. [1]

    Smith and M

    L. Smith and M. Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13–29, 2005

    work page 2005

  3. [2]

    A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V . Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726–747. PMLR, 2021

    2021

  4. [3]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on robot learning, pages 894–906. PMLR, 2022

    work page 2022

  5. [4]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  6. [5]

    Goyal, J

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  7. [6]

    Pumacay, I

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. The colosseum: A bench- mark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191, 2024

    arXiv 2024

  8. [7]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    arXiv 2023

Show all 59 references
  1. [8]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  2. [9]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  3. [10]

    S. Chen, R. Garcia, C. Schmid, and I. Laptev. Polarnet: 3d point clouds for language-guided robotic manipulation. arXiv preprint arXiv:2309.15596, 2023

    2023 arXiv

  4. [11]

    W. Yuan, A. Murali, A. Mousavian, and D. Fox. M2t2: Multi-task masked transformer for object-centric pick and place. arXiv preprint arXiv:2311.00926, 2023. 11

    2023 arXiv

  5. [12]

    J. Duan, W. Yuan, W. Pumacay, Y . R. Wang, K. Ehsani, D. Fox, and R. Krishna. Manipulate- anything: Automating real-world robots using vision-language models. arXiv preprint arXiv:2406.18915, 2024

    2024 arXiv

  6. [13]

    James and P

    S. James and P. Abbeel. Coarse-to-fine q-attention with learned path ranking. arXiv preprint arXiv:2204.01571, 2022

    2022 arXiv

  7. [14]

    Gervet, Z

    T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: Infinite resolution action detection transformer for robotic manipulation. arXiv preprint arXiv:2306.17817, 2023

    2023 arXiv

  8. [15]

    Z. Xian, N. Gkanatsios, T. Gervet, T.-W. Ke, and K. Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023

    2023

  9. [16]

    Goyal, V

    A. Goyal, V . Blukis, J. Xu, Y . Guo, Y .-W. Chao, and D. Fox. Rvt-2: Learning precise manipula- tion from few demonstrations. arXiv preprint arXiv:2406.08545, 2024

    2024 arXiv

  10. [17]

    Majumdar, K

    A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V .-P. Berges, T. Wu, J. Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36:655–677, 2023

    2023

  11. [18]

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022

    2022 arXiv

  12. [19]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022

    2022 arXiv

  13. [20]

    Laskin, K

    M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020

    2020

  14. [21]

    Yarats, I

    D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2021

    2021

  15. [22]

    Laskin, A

    M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pages 5639–5650. PMLR, 2020

    2020

  16. [23]

    Shang, K

    J. Shang, K. Schmeckpeper, B. B. May, M. V . Minniti, T. Kelestemur, D. Watkins, and L. Her- lant. Theia: Distilling diverse vision foundation models for robot learning. arXiv preprint arXiv:2407.20179, 2024

    2024 arXiv

  17. [24]

    Shah and V

    R. Shah and V . Kumar. Rrl: Resnet as representation for reinforcement learning. arXiv preprint arXiv:2107.03380, 2021

    2021 arXiv

  18. [25]

    C. Wang, X. Luo, K. Ross, and D. Li. Vrl3: A data-driven framework for visual deep rein- forcement learning. Advances in Neural Information Processing Systems, 35:32974–32988, 2022

    2022

  19. [26]

    Zhang, C

    J. Zhang, C. Bai, H. He, W. Xia, Z. Wang, B. Zhao, X. Li, and X. Li. Sam-e: Leveraging visual foundation model with sequence imitation for embodied manipulation. arXiv preprint arXiv:2405.19586, 2024

    2024 arXiv

  20. [27]

    D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant q learning in spatial action spaces. In Conference on Robot Learning, pages 1713–1723. PMLR, 2022

    2022

  21. [28]

    Dasari, M

    S. Dasari, M. K. Srirama, U. Jain, and A. Gupta. An unbiased look at datasets for visuo-motor pre-training. In Conference on Robot Learning, pages 1183–1198. PMLR, 2023. 12

    2023

  22. [29]

    what" and

    J. Shi, J. Qian, Y . J. Ma, and D. Jayaraman. Composing pre-trained object-centric repre- sentations for robotics from "what" and "where" foundation models. 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA) , pages 15424–15432, 2024. URL https://api.semant...

    2024

  23. [30]

    J. Qian, Y . Li, B. Bucher, and D. Jayaraman. Task-oriented hierarchical object decom- position for visuomotor control. In Conference on Robot Learning , 2024. URL https: //api.semanticscholar.org/CorpusID:273811817

    2024

  24. [31]

    Jockel, M

    S. Jockel, M. Weser, D. Westhoff, and J. Zhang. Towards an episodic memory for cognitive robots. In Proc. of 6th Cognitive Robotics workshop at 18th European Conf. on Artificial Intelligence (ECAI), pages 68–74. Citeseer, 2008

    2008

  25. [32]

    Henry, M

    P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. Rgb-d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. The international journal of Robotics Research, 31(5):647–663, 2012

    2012

  26. [33]

    S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas. Probabilistic data association for semantic slam. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 1722–1729. IEEE, 2017

    2017

  27. [34]

    D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247–4258, 2020

    2020

  28. [35]

    Peller-Konrad, R

    F. Peller-Konrad, R. Kartmann, C. R. Dreher, A. Meixner, F. Reister, M. Grotz, and T. Asfour. A memory system of a robot cognitive architecture and its implementation in armarx. Robotics and Autonomous Systems, 164:104415, 2023

    2023

  29. [36]

    Huang, F

    H. Huang, F. Lin, Y . Hu, S. Wang, and Y . Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.08248, 2024

    2024 arXiv

  30. [37]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  31. [38]

    P. Liu, Z. Guo, M. Warke, S. Chintala, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation. arXiv preprint arXiv:2411.04999, 2024

    2024 arXiv

  32. [39]

    Kerbl, G

    B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023

    2023

  33. [40]

    Shorinwa, J

    O. Shorinwa, J. Tucker, A. Smith, A. Swann, T. Chen, R. Firoozi, M. D. Kennedy, and M. Schwager. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. In 8th Annual Conference on Robot Learning, 2024

    2024

  34. [42]

    Lauri, D

    M. Lauri, D. Hsu, and J. Pajarinen. Partially observable markov decision processes in robotics: A survey. IEEE Transactions on Robotics, 39(1):21–40, 2022

    2022

  35. [43]

    James, Z

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020

    2020

  36. [44]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 13

    2024 arXiv

  37. [45]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  38. [46]

    Grotz, M

    M. Grotz, M. Shridhar, Y .-W. Chao, T. Asfour, and D. Fox. Peract2: Benchmarking and learning for robotic bimanual manipulation tasks. In CoRL 2024 Workshop on Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond , 2024. URL https://openreview.net...

    2024

  39. [47]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. URL https:/...

    2021

  40. [48]

    T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ArXiv, abs/2307.08691, 2023. URL https://api.semanticscholar.org/CorpusID: 259936734

    2023 arXiv

  41. [49]

    J. Su, Y . Lu, S. Pan, B. Wen, and Y . Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021. URL https://api.semanticscholar.org/ CorpusID:233307138

    2021 arXiv

  42. [50]

    B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024. URL https://api.semanticscholar. org/CorpusID:268536717

    2024

  43. [51]

    Zhang, Y

    X. Zhang, Y . Liu, H. Chang, L. Schramm, and A. Boularias. Autoregressive action sequence learning for robotic manipulation. IEEE Robotics and Automation Letters, 10:4898–4905, 2024. URL https://api.semanticscholar.org/CorpusID:273162644

    2024

  44. [52]

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. ArXiv, abs/2202.02005, 2022. URL https://api.semanticscholar.org/CorpusID:237257594

    2022 arXiv

  45. [53]

    James, K

    S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13729–13738, 2021. URL https: //api.semanticsc...

    2022

  46. [54]

    Guhur, S

    P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. ArXiv, abs/2209.04899, 2022. URL https: //api.semanticscholar.org/CorpusID:252200013

    2022 arXiv

  47. [55]

    T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. ArXiv, abs/2402.10885, 2024. URL https://api.semanticscholar.org/ CorpusID:267740176

    2024 arXiv

  48. [56]

    Garcia, S

    R. Garcia, S. Chen, and C. Schmid. Towards generalizable vision-language robotic manipulation: A benchmark and llm-guided 3d policy. ArXiv, abs/2410.01345, 2024. URL https://api. semanticscholar.org/CorpusID:273025962

    2024 arXiv

  49. [57]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollár, and R. B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992–4003, 2023. URL https: //api.semantics...

    2023

  50. [58]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. Q. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . B. Huang, S.-W. Li, I. Misra, M. G. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Lab...

    2023 arXiv

  51. [59]

    put the item in the drawer

    L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. ArXiv, abs/2406.09414, 2024. URL https://api.semanticscholar.org/CorpusID: 270440448. 15 A Model Architecture We will explain our model architecture in detail, including Multi-View Transformer...

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.