SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2

Alen Adamyan; Klara Janouskova; Martin Schmid; Matej Straka; Tom\'a\v{s} \v{C}\'i\v{z}ek

REVIEW 3 major objections 4 minor 12 references

SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing SAM 2's fixed memory-bank updates with a reinforcement learning policy overfitted per video raises tracking quality by +4.91% over SAM 2 and more than triples the gains of existing memory heuristics.

desk verdict A useful and honest upper-bound study of RL-driven memory control in SAM 2, but the '3x' claim rides on a SAMURAI baseline that needs verification before I would trust it. read the letter →

arxiv 2507.08548 v1 pith:RNFFU4L5 submitted 2025-07-11 cs.CV cs.LG

Alen Adamyan , Tomáš Čížek , Matej Straka , Klara Janouskova , Martin Schmid This is my paper

classification cs.CVcs.LG

keywords reinforcementlearningvisualobjecttrackingmemorybankcontrolSegmentAnythingModel2PPOvideosegmentationoverfittingupperboundquality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the memory bank of SAM 2, the fixed-size store of past frame memories that conditions each new prediction, is a much larger performance lever than its small size suggests. Rather than hand-crafting when to keep or evict memories, the authors treat each memory update as an action in a sequential decision problem and train a reinforcement learning controller with PPO on top of the frozen SAM 2 pipeline. In an intentional overfitting setup with one agent per video, SAM2RL raises tracking quality by +4.91%, accuracy by +2.52%, and robustness by +2.33% relative to SAM 2, while DAM4SAM gains +1.60% and SAMURAI loses -4.70% in their evaluation. A sympathetic reader should care because the improvement comes without changing any of SAM 2's trained weights, so the result isolates the untapped capacity of the memory update policy. The authors position these numbers as an upper bound and as motivation for scaling RL memory control to policies that generalize beyond individual videos.

What carries the argument

The load-bearing machinery is the RL memory controller wrapped around the frozen SAM 2 model. At each timestep the image encoder and mask decoder produce a predicted mask and a memory feature for the current frame; the controller observes a binary vector encoding which frame indices are currently in the memory bank plus the current timestep, and selects one of $N=7$ actions: discard the incoming memory or insert it by replacing one of the $N-1$ evictable slots, with the initial frame always kept. The policy is a two-layer MLP with 1024 hidden units trained with PPO, and the reward at each timestep is the per-frame VOTS tracking-quality term $q_t$, which gives IoU on frames with a visible object and applies a penalty convention when the object is absent. This decomposition isolates memory selection as the only learnable component, making the measured gains attributable to the update policy rather than to any change in SAM 2's parameters.

What would settle it

Run the published SAMURAI implementation with its intended configuration on the same 64 SA-V videos and recompute the mean tracking quality; if SAMURAI does not underperform SAM 2 by about 4.7 percentage points, the claim that SAM2RL's gain is more than three times the gains of existing heuristics collapses.

Watch

Extended reading notes

Core claim

The central discovery the authors claim is that memory selection in SAM 2 can be profitably learned: an RL agent that decides, frame by frame, whether to insert the incoming memory and which stored memory to evict achieves substantially higher tracking quality than SAM 2's fixed most-recent-frames rule or the published heuristic additions. On 64 videos from the SA-V training set, the overfitted agent reaches a mean tracking quality of 76.86%, against 71.95% for SAM 2, 73.55% for DAM4SAM, and 67.25% for SAMURAI, with accuracy and robustness also highest for SAM2RL. The paper's interpretation is that existing update rules underutilize the memory bank: the same frozen vision backbone, with only the memory controller changed, produces a multi-point gain on all three VOTS metrics. Because each policy is trained separately on one video, the result is an upper-bound estimate rather than a deployable tracker, and the authors present it as evidence that learned memory control is a promising replacement for hand-crafted rules.

Load-bearing premise

The load-bearing premise is that the SAMURAI baseline is configured and evaluated as its own authors intend, since if its reported -4.70% tracking quality is an artifact of the setup, the central comparison behind more than three times loses its footing.

Editorial extensions

If this is right

If the overfitting result is a true upper bound, memory-bank update policy is a substantial performance lever: choosing which memories to keep can move tracking quality by several points without retraining SAM 2.
RL-trained memory control can beat hand-crafted update rules such as motion-aware and distractor-aware selection in the authors' evaluation, suggesting learned policies are a viable alternative to hand-designed heuristics.
The gains in accuracy (+2.52%) and robustness (+2.33%) mean the learned policy improves both mask precision and the chance of finding the object, not just the aggregate quality score.
Scaling this approach to richer observations and network architectures is the authors' stated next step; if it works, a generalizing RL memory controller could improve SAM 2 without modifying its weights.
The per-video overfitting protocol provides an upper-bound estimate of the memory bank's exploitable capacity, which future generalizing policies can be measured against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the authors do not spell out is to replace hard memory eviction with learned soft weighting or learned retention scores, which could close part of the gap between the overfitted ceiling and the theoretical optimum.
Because the frozen SAM 2 backbone is unchanged, a memory controller trained with richer observations on diverse videos could act as a plug-in improvement for any SAM 2 variant; whether the per-video upper bound transfers to unseen videos is the natural next experiment.
The same RL framing could be applied to other fixed rule-based components of SAM 2, such as prompt conditioning or mask selection, suggesting that the untapped-capacity finding may extend beyond memory updates alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this is a fair, honest upper-bound study. The authors overfit an RL policy to control SAM 2's memory bank per video and openly call it an overfitting setup. That is the right frame. The genuinely new bit is the RL formulation itself—nobody has treated memory selection this way before. The result suggests the memory bank has headroom beyond hand-crafted rules like DAM4SAM, and the per-video quality gain (+4.91%) is larger than what the heuristics get. Credit where due: the method section is clear, the observation is simple, and they do not oversell generalization.

The soft spots are real but not fatal. The headline 'more than three times' rests entirely on the SAMURAI number being -4.70%. That contradicts the published SAMURAI results, and the paper gives no code, no evaluation config, and only a one-sentence attribution for the drop. If SAMURAI is actually positive, the factor falls below 3 and the central comparison weakens. This is load-bearing. Second, there are no error bars or repeated runs: 64 videos, one seed, one subset. The gaps between +1.60 and +4.91 could easily shift. Third, the reward is exactly the per-frame tracking quality, and with gamma close to 1 the policy is literally maximizing the reported metric. That is fine for an upper bound—they say this—but it means the number is a fitted optimum, not a prediction. No code or data makes checking harder.

In proportion, the core conceptual point holds: a learned memory update policy can beat fixed rules on training videos. That is a real, modest contribution. The exact magnitude and the 3x claim are not established by this evidence.

Who should read it: people working on SAM 2 memory mechanisms or RL for video tracking. It is a diagnostic, not a deployable method. I would send it to peer review—a serious editor would want baseline verification, error bars, and a code release—but it is not a desk reject. If I were reviewing, I would push hardest on the SAMURAI configuration.

Referee Report

3 major / 4 minor

Summary. The paper proposes SAM2RL, which frames memory-bank updates in SAM 2 as a sequential decision-making problem and trains a per-video PPO policy on 64 SA-V videos. The observation is a binary vector encoding the stored frame indices and the current timestep, and the action either discards the incoming memory or replaces one of the N-1 non-initial slots; the reward is the per-frame tracking quality q_t. In an explicitly overfitting setup, SAM2RL reports +4.91% tracking quality, +2.52% accuracy, and +2.33% robustness over SAM 2, compared with +1.60% for DAM4SAM and -4.70% for SAMURAI, and the abstract claims this exceeds the gains of existing heuristics by more than three times.

Significance. If the reported numbers are reliable, the result would demonstrate that the SAM 2 memory bank has substantial untapped capacity and that learned update rules are a promising direction for visual object tracking. Strengths of the paper include a clearly specified RL formulation, a simple and well-defined action space, and an honest acknowledgment that the policy is intentionally overfitted to individual videos. However, the headline comparison rests on a single unverified baseline number, the measurements lack error bars, and the oracle nature of the comparison (per-video overfitting with ground-truth rewards) limits the force of the claim that SAM2RL exceeds existing heuristics; the study is better characterized as an upper-bound feasibility analysis than as a practical tracking method.

major comments (3)

[Sec. 4, Table 1] The central claim of a more-than-three-fold improvement relies on the SAMURAI row reporting -4.70% tracking quality, which contradicts SAMURAI's published performance. The paper provides no evaluation configuration, no code, and no supporting analysis for the attribution to mask prediction during absent frames. The authors should evaluate the official SAMURAI implementation with the same protocol and report the result; if the published configuration improves over SAM 2, the factor-of-three claim must be revised.
[Sec. 4, Table 1] All quantitative results come from a single run on 64 videos, with no error bars, confidence intervals, or multiple seeds; the reported gaps between SAM2RL and DAM4SAM could be within run-to-run variation. The authors should report the mean and variance over at least three independent training runs, or provide a bootstrap over videos, to establish that the differences are statistically meaningful.
[Sec. 4, Training Setup and Quantitative Results] The comparison is between per-video overfitted policies that optimize the reported metric using oracle rewards and general-purpose heuristics that were not overfit to the test videos; this is not an equal-footing comparison, so the 'more than three times' statement measures the gap between an oracle and practical methods rather than the superiority of RL. Furthermore, since the reward equals the reported tracking quality, the improvement is a fitted training objective. A control under the same overfitting protocol, such as a random memory-update policy or a simple greedy selection rule, is needed to isolate the contribution of the RL optimization from the expressive action space.

minor comments (4)

[Table 1] The table mixes relative and absolute values in the same column; for example, SAMURAI appears as '-4.70 67.25' and SAM 2 appears as '71.95'. Use separate columns for absolute and relative values.
[Sec. 4, Quantitative Results] The text says 'SAMURAI, in our training setup', but SAMURAI is not trained in this work; this should read 'our evaluation setup'.
[Sec. 3, Memory Selection via Reinforcement Learning] The observation is described as 'the representation of the stored memories in the memory bank', but the implemented observation is a binary vector of frame indices and the current timestep; clarify whether any memory features are actually used.
[Sec. 4, Training Setup] The paper does not explicitly state that the reported metrics are computed on the same videos used for training the overfitted policy; since overfitting is the deliberate setup, this should be stated to avoid any ambiguity about evaluation on unseen videos.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported tracking-quality gain is the RL reward by construction; the overfitting is disclosed, but the headline quantitative claim is a fitted objective.

fitted input called prediction [Section 3 'Memory Selection via Reinforcement Learning' (Eq. 1 and reward definition); Section 4 'Quantitative Results' (Table 1)]
"To guide the learning process towards maximizing the tracking quality Q, we set the reward at timestep t to qt. The final episode return to maximize is thus equal to PT t=1 γt−1qt with γ ∈ (0,1]."

Equation (1) defines tracking quality as Q = (1/T) Σ q_t. The method then sets each per-timestep RL reward to q_t, so for γ=1 (a value the paper permits) the PPO episode return equals T·Q exactly. Table 1 reports SAM2RL's Q improvement (+4.91%) as the central result. This number is therefore the objective the policy was optimized to maximize on the same videos, not an independent prediction or held-out evaluation. The paper explicitly discloses the overfitting design, but the headline quantitative claim still reduces by construction to the fitted reward.

full rationale

The only structural circularity is the identity between the optimized reward and the reported metric: the reward is q_t and Q is the mean of q_t, so the reported improvement is the training objective, not an extrapolation. The paper is transparent that the policy is overfit per video and frames the result as an upper bound, which mitigates but does not remove the circularity of presenting the fitted objective as the main result. The SAMURAI baseline discrepancy is an external correctness risk, not a circularity, and is not counted in this score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the per-video policy weights being fit to the training reward, plus several domain assumptions about the frozen SAM 2 pipeline, the action space, and the reward signal. No new physical or algorithmic entities are introduced.

free parameters (2)

Per-video policy network weights = Trained by PPO to maximize Q on each video
The MLP policy is optimized separately on 64 videos; the reported tracking quality gain is the training objective, not an out-of-sample quantity.
Discount factor gamma = Not specified in paper
The return is defined as sum of gamma^(t-1) q_t with gamma in (0,1]; the chosen value affects whether the optimized return exactly equals the tracking quality Q. The paper does not report the value used.

assumptions (4)

domain assumption The frozen SAM 2 pipeline provides sufficient representation for the RL agent to make good memory decisions.
The agent only observes memory indices and current timestep, so any improvement must come through selecting which memories to retain; the frozen representations are assumed adequate.
ad hoc to paper The action space (discard current memory or replace one of the N-1 non-initial slots) covers all meaningful memory bank updates.
The upper-bound claim depends on this restriction; if other update operations (e.g., not retaining the initial frame or replacing it) were allowed, the bound could be different.
domain assumption The per-frame tracking quality q_t is an appropriate reward signal for sequential decision-making.
The paper equates reward with q_t and, with gamma=1, the return with Q, which is valid only if the metric decomposition is correct.
domain assumption PPO with the described setup reliably converges to a near-optimal policy for each video.
No convergence diagnostics or seed variance are provided; the claimed gain assumes training ran successfully and reached a good policy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2." pith.science (2026). https://pith.science/paper/RNFFU4L5

@misc{pith2026250708548,
  author       = {Pith},
  title        = {Pith review of: SAM2RL: Towards Reinforcement Learning Memory Control in Segment Anything Model 2},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNFFU4L5}},
  note         = {Machine review of arXiv:2507.08548}
}

read the original abstract

Segment Anything Model 2 (SAM 2) has demonstrated strong performance in object segmentation tasks and has become the state-of-the-art for visual object tracking. The model stores information from previous frames in a memory bank, enabling temporal consistency across video sequences. Recent methods augment SAM 2 with hand-crafted update rules to better handle distractors, occlusions, and object motion. We propose a fundamentally different approach using reinforcement learning for optimizing memory updates in SAM 2 by framing memory control as a sequential decision-making problem. In an overfitting setup with a separate agent per video, our method achieves a relative improvement over SAM 2 that exceeds by more than three times the gains of existing heuristics. These results reveal the untapped potential of the memory bank and highlight reinforcement learning as a powerful alternative to hand-crafted update rules for memory control in visual object tracking.

Figures

Figures reproduced from arXiv: 2507.08548 by the authors.

**Figure 2.** Accuracy vs. robustness plot comparing the performance of SAM 2-based models and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Visualization of predicted masks and the tracking quality of SAM2RL and DAM4SAM [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages

[1]

S am2 L ong: E nhancing S am 2 for L ong V ideo S egmentation with a T raining- F ree M emory T ree

Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. S am2 L ong: E nhancing S am 2 for L ong V ideo S egmentation with a T raining- F ree M emory T ree. arXiv preprint arXiv:2410.16268, 2024

arXiv
[2]

A N ew A pproach to L inear F iltering and P rediction P roblems, 1960

Rudolph Emil Kalman. A N ew A pproach to L inear F iltering and P rediction P roblems, 1960

work page 1960
[3]

Segment A nything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment A nything. In Proceedings of the IEEE/CVF I nternational C onference on C omputer V ision , pp.\ 4015--4026, 2023

work page 2023
[4]

The F irst V isual O bject T racking S egmentation VOTS2023 C hallenge R esults

Matej Kristan, Ji r \' Matas, Martin Danelljan, Michael Felsberg, Hyung Jin Chang, Luka C ehovin Zajc, Alan Luke z i c , Ondrej Drbohlav, Zhongqun Zhang, Khanh-Tung Tran, et al. The F irst V isual O bject T racking S egmentation VOTS2023 C hallenge R esults. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 1796--1818, 2023

work page 2023
[5]

Deep L earning for V isual T racking: A C omprehensive S urvey

Seyed Mojtaba Marvasti-Zadeh, Li Cheng, Hossein Ghanei-Yakhdan, and Shohreh Kasaei. Deep L earning for V isual T racking: A C omprehensive S urvey. IEEE T ransactions on I ntelligent T ransportation S ystems , 23 0 (5): 0 3943--3968, 2022. doi:10.1109/TITS.2020.3046478

arXiv 2022
[6]

S am 2: S egment A nything in I mages and V ideos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. S am 2: S egment A nything in I mages and V ideos. arXiv preprint arXiv:2408.00714, 2024

arXiv 2024
[7]

Hiera: A H ierarchical V ision T ransformer without the B ells-and- W histles

Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A H ierarchical V ision T ransformer without the B ells-and- W histles. arXiv preprint arXiv:2306.00989, 2023

arXiv 2023
[8]

Proximal P olicy O ptimization A lgorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal P olicy O ptimization A lgorithms. arXiv preprint arXiv:1707.06347, 2017

arXiv 2017

Show all 12 references

[9]

A D istractor- A ware M emory for V isual O bject T racking with SAM2

Jovana Videnovic, Alan Lukezic, and Matej Kristan. A D istractor- A ware M emory for V isual O bject T racking with SAM2 . arXiv preprint arXiv:2411.17576, 2024

2024 arXiv
[10]

S amurai: A dapting S egment A nything M odel for Z ero- S hot V isual T racking with M otion- A ware M emory

Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. S amurai: A dapting S egment A nything M odel for Z ero- S hot V isual T racking with M otion- A ware M emory. arXiv preprint arXiv:2411.11922, 2024

2024 arXiv
[11]

M o SAM : M otion- G uided S egment A nything M odel with S patial- T emporal M emory S election

Qiushi Yang, Yuan Yao, Miaomiao Cui, and Liefeng Bo. M o SAM : M otion- G uided S egment A nything M odel with S patial- T emporal M emory S election. arXiv preprint arXiv:2505.00739, 2025

2025 arXiv
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

S am2 L ong: E nhancing S am 2 for L ong V ideo S egmentation with a T raining- F ree M emory T ree

Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. S am2 L ong: E nhancing S am 2 for L ong V ideo S egmentation with a T raining- F ree M emory T ree. arXiv preprint arXiv:2410.16268, 2024

arXiv

[2] [2]

A N ew A pproach to L inear F iltering and P rediction P roblems, 1960

Rudolph Emil Kalman. A N ew A pproach to L inear F iltering and P rediction P roblems, 1960

work page 1960

[3] [3]

Segment A nything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment A nything. In Proceedings of the IEEE/CVF I nternational C onference on C omputer V ision , pp.\ 4015--4026, 2023

work page 2023

[4] [4]

The F irst V isual O bject T racking S egmentation VOTS2023 C hallenge R esults

Matej Kristan, Ji r \' Matas, Martin Danelljan, Michael Felsberg, Hyung Jin Chang, Luka C ehovin Zajc, Alan Luke z i c , Ondrej Drbohlav, Zhongqun Zhang, Khanh-Tung Tran, et al. The F irst V isual O bject T racking S egmentation VOTS2023 C hallenge R esults. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 1796--1818, 2023

work page 2023

[5] [5]

Deep L earning for V isual T racking: A C omprehensive S urvey

Seyed Mojtaba Marvasti-Zadeh, Li Cheng, Hossein Ghanei-Yakhdan, and Shohreh Kasaei. Deep L earning for V isual T racking: A C omprehensive S urvey. IEEE T ransactions on I ntelligent T ransportation S ystems , 23 0 (5): 0 3943--3968, 2022. doi:10.1109/TITS.2020.3046478

arXiv 2022

[6] [6]

S am 2: S egment A nything in I mages and V ideos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. S am 2: S egment A nything in I mages and V ideos. arXiv preprint arXiv:2408.00714, 2024

arXiv 2024

[7] [7]

Hiera: A H ierarchical V ision T ransformer without the B ells-and- W histles

Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A H ierarchical V ision T ransformer without the B ells-and- W histles. arXiv preprint arXiv:2306.00989, 2023

arXiv 2023

[8] [8]

Proximal P olicy O ptimization A lgorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal P olicy O ptimization A lgorithms. arXiv preprint arXiv:1707.06347, 2017

arXiv 2017