{"id":"0a03a410-4b19-4f0f-b3ec-b4fa7bdf04e8","arxiv_id":"2501.18564","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SAM2Act reports 86.8% average success across 18 RLBench tasks, and the memory variant SAM2Act+ reaches 94.3% on the new MemoryBench tasks.","lead":"The paper presents SAM2Act, a robot policy that combines the SAM2 vision model with multi-resolution upsampling to predict precise manipulation actions, reporting a top average success rate of 86.8% on 18 RLBench tasks. A memory-augmented version, SAM2Act+, solves tasks that require remembering past actions, and the authors introduce a new benchmark, MemoryBench, to measure this.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MemoryBench's 94.3% headline rests on per-task best-epoch and memory-window selection; a fixed-configuration evaluation could substantially shrink the claimed memory advantage.","rationale":"The reader's weakest assumption identified the MemoryBench evaluation protocol as the key vulnerability, and I agree. The two headline numbers are 86.8% on RLBench and 94.3% on MemoryBench. The RLBench result is supported by multiple baselines and ablations and is less sensitive to protocol choices. The MemoryBench result, however, depends on per-task hyperparameter selection (memory window 8 vs 10) and per-task best-epoch selection, with only three tasks and small evaluation sets. This is a genuine load-bearing concern because the memory architecture is a central contribution and the claimed 39.3% margin over SAM2Act is what motivates the memory components. A concrete test—fixing the configuration and epoch rule—would settle whether the margin persists. The Colosseum reporting issue (4.3% average vs 58.3% under all perturbations) is also real and mentioned by the reader, but it affects a secondary claim and is already visible in the table; it does not change the verdict beyond what the MemoryBench concern already motivates. Therefore, the conditional verdict remains appropriate: the paper should be accepted only after the MemoryBench protocol is clarified or re-run with a fixed configuration.","tokens_in":23444,"tokens_out":7199,"duration_ms":69687,"concrete_test":"Re-run the MemoryBench evaluation with a single fixed memory window size (e.g., 10) for all three tasks, one shared training configuration, and an epoch-selection rule defined before seeing test results (e.g., best epoch on a held-out validation split, or simply the final epoch). Report the average success rate and the margin over the SAM2Act baseline. If the margin shrinks substantially (e.g., from 39.3% to below 15%), the 94.3% headline is not robust to evaluation protocol choices.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central memory claim—SAM2Act+ reaching 94.3% average success on MemoryBench—is directly tied to an evaluation protocol that selects the best performing epoch for each task and tunes the memory window size separately per task (Appendix B.2, Table 6). The paper states: 'we use a universal set of hyperparameters for training but evaluate models across all epochs and select the best-performing one for evaluation,' and 'the window size of the memory mechanism is also decided to be different for each task.' Concretely, reopen_drawer uses a memory window of 8 while the other two tasks use 10, and learning rates also differ (3.2e-3 vs 4e-3). Because MemoryBench has only three tasks, each evaluated on 25 unseen demonstrations with 4 evaluation runs, per-task best-epoch selection can inflate the reported average, especially given the large variance seen in baselines (e.g., SAM2Act std of 24.3 on the average). If a single fixed window and a pre-specified epoch rule were applied uniformly, the 94.3% could drop substantially, and the claimed 39.3% margin over SAM2Act (without memory) might narrow. The paper discloses this protocol, but the disclosure does not remove the risk that the headline memory result is an artifact of per-task tuning rather than the architecture alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAM2Act, a behavior-cloning policy built on RVT-2 that replaces the visual backbone with a LoRA-fine-tuned SAM2 image encoder and adds a cascaded multi-resolution convex upsampling module. It reports 86.8% average success across 18 RLBench tasks, a 4.3% average performance drop under individually applied Colosseum perturbations, and a 94.3% average success on MemoryBench, a new three-task suite of non-Markovian manipulation tasks, using the memory-augmented variant SAM2Act+. Real-world experiments on a Franka Panda compare SAM2Act with RVT-2 and include a memory-style button-pressing task. The paper includes ablations of the SAM2 encoder and multi-resolution upsampling, full comparison tables in appendices, and a disclosure of the MemoryBench evaluation protocol.","tokens_in":23655,"tokens_out":11255,"duration_ms":109591,"significance":"If the RLBench results hold, SAM2Act would be a meaningful advance over RVT-2 and other keyframe-based BC policies, with particularly large gains on precision tasks such as Insert Peg and Sort Shape. The multi-resolution upsampling ablation (Table 9) and the Colosseum robustness results support the architectural claims. The MemoryBench benchmark is a useful idea, and the authors are transparent about its evaluation protocol. However, the memory claim is currently under-supported: the protocol uses best-epoch selection and per-task window sizes, the baselines are all memory-less, and one of the three tasks appears partly solvable without explicit memory. These issues need to be addressed before the 94.3% result can be taken at face value.","major_comments":[{"comment":"The 94.3% MemoryBench headline rests on a protocol in which the best-performing epoch is selected for every model and the memory window size differs per task (window 8 for reopen_drawer, 10 for the other two tasks; learning rates and batch sizes also differ). This can inflate the reported average, particularly with only 25 held-out demonstrations and 4 evaluation runs, and the large standard deviation reported for SAM2Act (24.3 on the average) shows that epoch-level noise is substantial. Appendix H says the window was later standardized to 10 for SAM2Act+ and results updated, but Appendix B.2 and Table 6 still describe per-task windows, so the paper is internally inconsistent. Please report results with a pre-specified epoch-selection rule applied identically to all models, a single memory window, and per-epoch performance curves.","section":"§5.4, Appendix B.2, Table 6"},{"comment":"The memory comparison includes only memory-less baselines (RVT-2 and SAM2Act without memory). Since MemoryBench is deliberately non-Markovian, a memory-less policy is handicapped by construction, and the 39.3% margin does not isolate the contribution of the proposed memory bank, memory encoder, and memory attention. Please add a history-aware baseline, such as a frame-stacking variant of SAM2Act or RVT-2, a recurrent policy, or an existing history-based method (e.g., HiveFormer or the memory model cited as [41]), to show that the gains come from the memory architecture rather than from merely receiving past observations.","section":"§5.4, Table 3"},{"comment":"SAM2Act without memory reaches 82% on rearrange_block, where random chance is 50%, despite the task being designed to be non-Markovian. This suggests that the current observation may contain residual cues or that the two blocks are visually distinguishable, so the task may not be a pure memory test. Please verify that the decision-point observation is truly ambiguous (for example, by checking that the blocks are visually identical and that a memory-less policy operates near chance), or revise the task and environment accordingly.","section":"§3.2, Table 3"},{"comment":"The update equation X^{l+1} = LayerNorm(U(X^l) ⊕ E^l) is dimensionally inconsistent as written: U(X^l) has C_l/2 channels while E^l is defined as having C_l channels, so element-wise addition is undefined. Please specify how E^l is projected or otherwise aligned before addition, or correct the notation. This is necessary to reproduce the multi-resolution upsampling mechanism that underlies the reported RLBench gains.","section":"§4.1, update equation"}],"minor_comments":[{"comment":"The main text says SAM2Act surpasses RVT-2 by 5.4%, while the Table 1 caption states a margin of 5.8%; please make these numbers consistent.","section":"§5.2, Table 1"},{"comment":"The caption states an average improvement of 37.6% across all three tasks, but the per-task differences in the table give 40.3 percentage points over RVT-2 and 39.3 percentage points over SAM2Act; please clarify how the 37.6% figure is computed.","section":"§5.4, Table 3 caption"},{"comment":"The abstract emphasizes a 4.3% average drop, but Table 2 shows a 58.3% drop for SAM2Act in the All Perturbations column when perturbations are combined; please state this distinction explicitly in the main text. Additionally, the Camera Pose value for SAM2Act is -4.3±3.6, identical to the Average value, which appears to be a transcription error.","section":"§5.3, Table 2"},{"comment":"The real-world memory task (d) is evaluated with only 10 in-distribution and 10 out-of-distribution trials; please acknowledge the limited statistical power or provide confidence intervals.","section":"§5.5, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central RLBench contribution appears solid, but the memory evaluation needs a protocol fix and stronger baselines before the headline MemoryBench numbers are publishable. The authors disclose the best-epoch and per-task window selection, so this is not a case of hidden evaluation, but the disclosure does not resolve the risk that the 39.3% margin is inflated. I would also ask the editor to weigh whether a three-task author-created benchmark is sufficient to support the breadth of the memory claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the RLBench results and the multi-resolution upsampling idea are genuinely strong; the memory benchmark and the Colosseum headline are less solid than the abstract suggests, and the paper contradicts itself about the memory window protocol.\n\nThe real contribution is architectural. Using SAM2's multi-resolution embeddings as auxiliary inputs to a cascaded convex upsampler in RVT-2, with LoRA fine-tuning, gives a clear jump on RLBench: 86.8 average across 18 tasks, beating RVT-2 by 5.4 points and also ARP+ and 3D-LOTUS in the full table. The ablations are honest: removing the multi-res input costs 1.1 points, replacing the upsampler 2.6, swapping SAM2 for SAM 6.0. That is a believable, well-isolated result. Adapting SAM2's memory bank/attention/encoder to action heatmaps is a legitimately new combination, and the three MemoryBench tasks are well designed to violate the Markov property.\n\nNow the soft spots. First, the MemoryBench evaluation protocol. Appendix B.2 says they select the best-performing epoch for every model and tune the memory window per task (Table 6: window 8 for reopen_drawer, 10 for the others). Appendix H says they later retrained reopen_drawer with window 10 \"to standardize the memory window size across all three tasks,\" which conflicts with Table 6 and with the conclusion's statement that the window \"differs from task to task.\" With 3 tasks, 25 test demos, and 4 evaluation runs, best-epoch selection can inflate the 94.3% headline and the 39.3% margin over the no-memory SAM2Act. Second, there are no memory-aware baselines. All comparisons on MemoryBench are against models without any memory mechanism, so the margin is not surprising. A simple recurrent policy or a history-stacking baseline would tell you whether the SAM2-style spatial memory is the right design. Third, the Colosseum claim: the 4.3% average gap is the mean of individual perturbation types. The paper's own \"All Perturbations\" column shows a 58.3 point drop. That is not \"only a 4.3% performance gap\" when perturbations are combined. It should be reported that way.\n\nWho is this for? Anyone working on visual foundation models for manipulation or on memory-augmented policies. The citation pattern is solid, covering the relevant SAM, RVT, and memory lines. It deserves a serious referee, but the MemoryBench protocol needs a fixed evaluation rule and memory-based baselines before the central memory claim can be taken at face value.","headline":"Strong RLBench gains from SAM2 multi-res upsampling, but the MemoryBench headline rests on per-task tuning and the Colosseum 4.3% claim hides a 58-point combined drop.","tokens_in":24235,"tokens_out":7322,"would_cite":true,"duration_ms":59243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fusing the SAM2 vision encoder's multi-resolution embeddings into a coarse-to-fine manipulation transformer produces a policy with 86.8% average success on 18 tasks, and that adding a SAM2-style memory bank lets it…","keywords":["robotic manipulation","imitation learning","behavior cloning","multi-view transformer","spatial memory","memory-based architecture","generalization","visual foundation model"],"falsifier":"Run SAM2Act+ and the best memory-less baseline on MemoryBench under a single fixed memory window size for all tasks and the same epoch-selection rule; if the gap over the memory-less model falls well below the reported 39.3 points or near chance, the memory claim as stated does not hold.","tokens_in":23228,"feed_emoji":"🤖","tokens_out":6541,"duration_ms":68337,"temperature":0.7,"pith_summary":"The paper argues that a robotic manipulation policy can get both higher precision and better generalization by reusing the multi-resolution visual embeddings of a large video-segmentation model rather than training a task-specific visual encoder. It introduces SAM2Act, a language-conditioned multi-view behavior-cloning policy whose coarse-to-fine transformer fuses those embeddings at three resolutions during heatmap upsampling; on an 18-task benchmark it reports an average 86.8% success, and under a suite of environmental perturbations it drops only 4.3%. It then extends the same architecture into SAM2Act+, adding a memory bank, encoder, and attention module so the policy can recall past predicted positions. On three deliberately non-Markovian tasks built to require spatial memory, SAM2Act+ reports 94.3% average success, far above memory-less baselines. The paper also contributes that benchmark, MemoryBench, for measuring spatial memory in behavior-cloning agents.","feed_headline":"Robot policy tops 18-task manipulation suite at 86.8%","feed_subtitle":"A memory-augmented version also reaches 94.3% on tasks that require recalling past actions.","key_machinery":"The mechanism is the SAM2Act module: three cascaded convex upsamplers that double spatial resolution at each stage, each combining the upsampled feature map with the matching multi-resolution embedding from the SAM2 image encoder via $X^{l+1} = \\mathrm{LayerNorm}(U(X^l) \\oplus E^l)$. The memory extension adds a per-view FIFO memory bank, a memory encoder that compresses the predicted translation heatmap with the observation embedding, and memory attention that cross-attends current features to stored past memories. The heatmap plays the role that predicted masks play in SAM2's video tracking, so the policy conditions its next translation prediction on where it previously acted.","core_discovery":"The central claim is that multi-resolution embeddings from the SAM2 image encoder, injected through cascaded convex upsamplers, give a manipulation policy materially better spatial precision and robustness, and that the same video model's memory components can be transplanted to give policies episodic spatial memory. The authors state this as a state-of-the-art 86.8% average over 18 RLBench tasks, with the largest gains on precision-heavy peg insertion and shape sorting; a 4.3% average degradation under thirteen perturbation types on the Colosseum suite; and a 94.3% average on the three MemoryBench tasks, compared with 55.0% for the memory-less SAM2Act and 54.0% for the prior baseline. They stress that the memory tasks are non-Markovian: the same current observation can require different next actions, so success is impossible without recalling which action history led to it.","pith_inferences":["The paper leaves implicit that its MemoryBench numbers are tuned: the memory window size is chosen per task and the best-performing epoch is selected for every model, so the 94.3% headline is best read as an upper bound under a tuned protocol.","Because the memory bank stores only translation heatmaps, not rotation, gripper state, or object identity, the architecture is naturally suited to spatial recall; extending the memory encoder to consume those other signals might let the same design solve tasks requiring semantic memory.","The fixed-length FIFO memory makes the approach most applicable to tasks with short, bounded horizons; tasks with unbounded histories or continuous control would likely need learned memory compression or a recurrent state.","Task-by-task inspection of the perturbation table shows several tasks with low baseline success even without perturbations, so the 4.3% average gap is not uniform across tasks; future work could separate robustness gains from tasks that are simply easier."],"forward_implications":["A single behavior-cloning policy can handle 18 varied manipulation tasks with an average success rate near 87%, with the largest gains on high-precision tasks such as inserting a ring onto a peg (84% vs 40% for the previous best baseline).","The same policy loses on average only 4.3% when evaluated under 13 perturbation categories such as lighting, table texture, distractors, and camera pose, indicating that visual-feature robustness transfers to manipulation.","Tasks that violate the Markov property, where the same frame can require different actions, are solvable by explicit memory; without it, the best tested policy stays near chance on those tasks.","Memory can be added without retraining the full model: the visual encoder and fine branch are frozen, and only the memory components plus the coarse branch are fine-tuned, reducing the cost of acquiring episodic recall.","A benchmark with three memory tasks, reopening a drawer, putting a block back, and rearranging blocks, can separate memory-capable from memory-less policies."],"supporting_citations":[{"why":"The SAM2 model supplies both the multi-resolution image embeddings used by the upsampling cascade and the memory attention, encoder, and bank design that SAM2Act+ adapts.","marker":"[44]"},{"why":"The RVT-2 multi-view transformer is the backbone SAM2Act builds on and the strongest baseline it must outperform on RLBench.","marker":"[16]"},{"why":"RLBench is the simulator and task source for the 18-task benchmark used to measure multi-task success.","marker":"[43]"},{"why":"The Colosseum provides the perturbation categories used to measure generalization under lighting, texture, distractor, and camera changes.","marker":"[6]"},{"why":"SAM-E is a baseline that already uses a SAM visual encoder for manipulation, establishing the comparison class for encoder choice.","marker":"[26]"},{"why":"LoRA is the low-rank adaptation method used to fine-tune the SAM2 encoder at rank 16 with modest compute.","marker":"[45]"}],"fun_headline_variants":["Robotic policy hits 86.8% across 18 tasks, then 94.3% with memory","From 86.8% to 94.3%: memory boosts robot manipulation","Memory-augmented robot policy scores 94.3% on recall tasks","SAM2Act: 86.8% on 18 tasks, memory version hits 94.3%","Robot policy with spatial memory: 94.3% on non-Markovian tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline memory advantage assumes fair evaluation under a uniform protocol, yet the reported SAM2Act+ numbers use per-task memory-window sizes and best-epoch checkpoint selection in a way that is not applied equally to the baselines.","fun_headline_variants_meta":{"raw":{"variants":["Robotic policy hits 86.8% across 18 tasks, then 94.3% with memory","From 86.8% to 94.3%: memory boosts robot manipulation","Memory-augmented robot policy scores 94.3% on recall tasks","SAM2Act: 86.8% on 18 tasks, memory version hits 94.3%","Robot policy with spatial memory: 94.3% on non-Markovian tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3565,"prompt_tokens":960,"completion_tokens":2605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2485}},"tokens_in":576,"tokens_out":2605,"duration_ms":18098,"temperature":1.0,"reasoning_tokens":2485,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:57:31.747360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAM2Act+ and the best memory-less baseline on MemoryBench under a single fixed memory window size for all tasks and the same epoch-selection rule; if the gap over the memory-less model falls well below the reported 39.3 points or near chance, the memory claim as stated does not hold.","supporting_citations":[],"review_version":1}