Pith. sign in

REVIEW 4 major objections 5 minor 62 references

SAM2's heavy memory and encoder work can be cut ~1.4x with target-anchored pruning and routing that protects against occlusion-driven drift.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:37 UTC pith:NV7GYZI4

load-bearing objection A legitimate engineering extension of Efficient-SAM2 with a plausible three-part mechanism, but the headline speedups and accuracy gains are tuned on the same validation set they are reported on, and the timing subset is too underspecified to trust the exact numbers. the 4 major comments →

arxiv 2607.19811 v1 pith:NV7GYZI4 submitted 2026-07-22 cs.CV

Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2

Xudong Ouyang , Wenlun Zhang , Yimin Xu , Huazhong Liu , Yunshan Zhong This is my paper
classification cs.CV
keywords video object segmentationSegment Anything Model 2efficient inferencememory pruningsparse window routingpost-training accelerationocclusion and distractor handlingtarget-anchored attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that SAM2's two computational bottlenecks—memory cross-attention and full-frame image encoding—can be reduced without retraining the base model, and that the accuracy losses from pruning and sparse routing can be mostly recovered by consulting a fixed set of target anchors derived from the initial prompt. Lean-SAM2 combines three mechanisms: target-anchored memory pruning (TAMP), visibility-gated temporal condensation with an insurance bank (TCIM), and anchor-aware window routing with a risk-aware full-frame fallback (TARR). If right, this yields overall inference speedups around 1.41–1.43x across SAM2.1 Large, Base+, and Small, while beating prior pruning-and-routing baselines by several J&F points on occlusion-heavy benchmarks. A sympathetic reader would care because it suggests post-training acceleration can stay accurate exactly in the hard cases—occlusions and lookalike distractors—where earlier pruning approaches failed.

Core claim

The central claim is that the failure mode of earlier pruning/routing accelerators is not pruning per se but pruning without an identity reference: when occluders or lookalikes corrupt attention scores or mask predictions, attention-only memory pruning discards target tokens and mask-driven window routing skips the target. Lean-SAM2 therefore derives a compact anchor set from the prompt foreground in the first frame and reuses it everywhere: TAMP re-scores memory tokens by cosine similarity to these anchors, TCIM fuses history with a visibility gate and keeps an insurance bank of high-confidence entries, and TARR routes windows through the heavy encoder only if a window contains tokens align

What carries the argument

The load-bearing object is the anchor set A—a sparse, diversity-selected collection of memory tokens sampled from the initial prompt foreground (Eq. 12–14) that serves as a frozen oracle for target identity. TAMP uses max cosine similarity to A to modulate attention-significance scores for memory-token pruning; TARR uses the same anchor similarity to decide which windows go to the heavy encoder and when to trigger a full-frame fallback; TCIM uses the predicted occlusion score as a visibility gate and a small FIFO insurance bank for high-confidence archived entries. The anchor set carries the whole argument: because it comes from the first-frame prompt rather than from drifting predictions, i

Load-bearing premise

The whole method leans on the assumption that a set of tokens sampled from the first-frame prompt foreground remains a faithful reference for what the target looks like for the rest of the video; if the object changes appearance or the initial prompt is imperfect, the anchor similarities driving both pruning and routing can point the model at the wrong regions.

What would settle it

Take a video where the target undergoes a gradual but large appearance change—for example a person changing pose and clothing over a long sequence—and track the per-frame average cosine similarity between true target-region memory tokens and the first-frame anchors. If that similarity falls toward or below the 0.5 threshold used in Eq. 22 while the model still tracks correctly, the same signal that triggers full-frame fallback will either degrade speed or, when ignored, degrade accuracy; either way, anchor staleness becomes measurable. A direct comparison with occasional anchor refreshment on

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Overall SAM2.1 inference can be accelerated by 1.41–1.43x on a single RTX 3090 with J&F drops of roughly 0.6–1.2 points relative to the unmodified baseline, with consistent gains over the previous unified pruning-and-routing accelerator.
  • Per-module, the memory-attention part reaches 1.61–1.68x speedups with TAMP+TCIM and the image-encoder part reaches 1.31–1.36x with TARR; the two combine to produce the reported overall speedups.
  • The risk-aware fallback policy runs at only about a 25% trigger rate at the default threshold, meaning the accuracy insurance costs relatively little speed.
  • Real pruning (rather than layer-wise mask retrieval) cuts the memory-attention GPU footprint to about 488 MB, a 55.9% reduction over baseline and roughly 77% versus the prior method.
  • Per-frame performance curves show the accelerated model hugs the original SAM2 curve over long videos instead of decaying, implying error accumulation is controlled.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: because the anchors are sampled once from the first-frame prompt and never updated, the scheme will likely degrade when the target's appearance changes substantially (pose, deformation, lighting) over the video; an adaptive re-anchoring rule—refreshing anchors when the visibility-gated fusion confidence stays high—would be a natural extension.
  • The same anchor-consistency score could double as a confidence signal for interactive applications: low mean anchor similarity in routed windows could prompt a user-in-the-loop re-click rather than a silent full-frame refresh.
  • Since TAMP and TCIM are training-free and only TARR's bypass network is trained, the framework could plausibly be stacked with quantization-based accelerators to push beyond the reported 1.4x; the paper does not test that composition.
  • The 0.5 alignment threshold in Eq. 22 and the occlusion/area fallback rule imply a measurable prediction: on videos with gradual appearance change but no occlusion, the fallback rate should spike and the realized speedup should collapse—a testable behavioral signature of anchor staleness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Lean-SAM2, a post-training acceleration framework for SAM2 that combines three mechanisms: Target-Anchored Memory Pruning (TAMP), which prunes memory tokens by modulating attention significance with cosine similarity to anchors sampled from the initial prompt foreground; Temporal Condensation with Insurance Memory (TCIM), which compresses the working memory into three entries plus a parallel insurance bank; and Target-Anchored Risk-Aware Routing (TARR), which routes image-encoder windows based on anchor alignment and triggers full-frame fallback under low occlusion scores or large mask-area changes. Experiments on LVOSv2, MOSEv1/v2, and SA-V report about 1.41x–1.43x overall speedups on SAM2.1-Large/Base+/Small with small J&F losses relative to the SAM2.1 baselines and larger gains over Efficient-SAM2.

Significance. If the reported results hold, this is a practically useful contribution: it addresses a real deployment bottleneck of SAM2, and the three mechanisms are clearly specified and largely training-free, making the approach easy to reproduce and adapt. The modular ablations give some support to the design choices, and the authors commit to releasing code. However, the headline numbers are weakened by two evaluation-protocol issues: hyperparameters are chosen by grid search on the same LVOSv2 valid split that is used for the main results, and all speedups are measured on an unspecified 20-video subset with no error bars. These issues need to be resolved before the claimed superiority over Efficient-SAM2 can be fully trusted.

major comments (4)
  1. [Section V-D, Tables III–IV vs. Table I] The default values of r_anchor, lambda_A, rho, theta_ins, T, insurance-bank size, and theta_full are all selected by grid search on LVOSv2 valid, and the highest-J&F configuration is chosen as the default (Tables III and IV). The same LVOSv2 valid split is then used for the headline results in Table I. Consequently, the reported 5.0% and 3.6% gains over Efficient-SAM2 and the absolute J&F numbers are in-sample optima, not unbiased estimates. This is load-bearing for the central claim. Please retune on a separate validation split (e.g., a held-out portion of LVOSv2 valid or a different dataset) and report the held-out numbers, or use a nested evaluation procedure.
  2. [Section V-A, 'Timing protocol'] All speedups are measured on 20 video sequences from LVOSv2 valid, but the list of videos is not given, and no error bars or per-video breakdown are reported. Since the fallback rate (25% at the default theta_full, Table IV) directly determines the speedup and varies with video content, the reported 1.412x/1.417x/1.433x speedups are not reproducible and may not be representative. Please specify the subset, report the mean and variance over multiple runs or over the per-video distribution, and justify that the subset is not selected to favor the method.
  3. [Table II] The full model (TAMP+TCIM+TARR) achieves 82.4 J&F, which is 2.1 points lower than TCIM alone (84.5 J&F) and 1.2 points below the baseline (83.6 J&F). The text states that the mechanisms 'harmonize their individual strengths,' but it does not explain why adding TAMP and TARR reduces accuracy relative to TCIM alone. This makes it difficult to attribute the accuracy-efficiency balance to the full framework; please provide a discussion or a Pareto-style analysis of the accuracy/speedup trade-off across the configurations in Table II.
  4. [Section IV-B, Eqs. (14) and (21)] The anchor set A is sampled once from the initial prompt foreground and never updated. If the target's appearance changes substantially over time (pose, lighting, deformation), the cosine-similarity anchors can mis-rank memory tokens and mis-route windows, potentially causing tracking drift. The paper does not analyze this failure mode or provide a re-anchoring mechanism. Please add a sensitivity analysis (e.g., split results by degree of appearance change over the video) or update anchors over time.
minor comments (5)
  1. [Eq. (18)] The formula for alpha is ambiguous: it is not clear whether sigma(1/T) is in the numerator or denominator. Please add parentheses and define sigma explicitly as the sigmoid function.
  2. [Section V-A] The 'Timing protocol' paragraph appears twice; please remove the duplicate.
  3. [Table II] Clarify that the TARR row speedup (1.228x) is the overall-model speedup while Table I's TARR row reports module-level speedup (1.351x); the two are not directly comparable and this should be stated to avoid confusion.
  4. [Figure 4] The CDF in Figure 4 is based on 'randomly selecting 20 videos' but no random seed is given. Please specify the seed or list the video IDs for reproducibility.
  5. [References] Several references are dated 2026; please verify that they are published or publicly available and update the citations if needed.

Circularity Check

0 steps flagged

No significant circularity: the paper's mechanisms and speedups are empirical heuristics, not results derived from their own inputs.

full rationale

Lean-SAM2 is an empirical post-training acceleration method; it does not claim a formal derivation of its accuracy or speedup from first principles. TAMP, TCIM, and TARR are explicitly defined heuristics: anchors are sampled from the initial prompt foreground, attention significance and cosine-similarity scores are combined, occlusion scores gate temporal fusion, and routing decisions are made from anchor alignment and risk-based fallback. None of these steps reduces to the reported J&F or speedup by construction. The paper's central comparisons are measured against official SAM2 baselines and Efficient-SAM2 on multiple external benchmarks, so the claims are independently evaluable rather than self-referential. The main methodological caveat — hyperparameters such as r_anchor, λ_A, ρ, θ_ins, T, and θ_full are tuned on LVOSv2 valid and then reported on the same split in Table I — is a potential overfitting / evaluation-protocol issue, not a circularity of the kind defined here: the reported numbers are empirical outcomes of a configured system, not quantities that are equal to their inputs by definition. No load-bearing self-citations or uniqueness arguments are present; references to the authors' prior quantization work are peripheral to the central efficiency claims.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 2 invented entities

The framework's value hinges on several domain assumptions: that cosine similarity to initial anchors tracks target identity, that predicted occlusion scores are reliable quality indicators, and that a 0.5 alignment threshold for routing is universally appropriate. No formal justification is given; the paper relies on empirical validation. Additionally, seven hyperparameters are tuned on the validation set, which inflates reported accuracy.

free parameters (9)
  • r_anchor = 0.05
    Anchor sampling ratio in Eq. 12; ablated in Table III(a) to maximize LVOSv2 valid J&F.
  • lambda_A = 2.0
    Anchor-similarity modulation strength in Eq. 15; ablated in Table III(b).
  • rho = 0.55
    Visibility-gating scaling in Eq. 18; ablated in Table III(c).
  • theta_ins = 0.7
    Insurance-bank archive threshold in Eq. 19; ablated in Table III(d).
  • T = 0.2
    Temperature in the visibility-gated fusion Eq. 18; ablated in Table III(f).
  • insurance_bank_size = 3
    Maximum entries B in insurance bank Eq. 19; ablated in Table III(e).
  • theta_full = 0.99
    Fallback threshold in Eq. 23; ablated in Table IV to trade off speed and accuracy.
  • alignment_threshold = 0.5
    Threshold for routing window to full encoder in Eq. 22; stated without ablation.
  • K_ret = not stated
    Number of memory tokens retained by TopK in Eq. 16; inherited from Efficient-SAM2 but not reported.
axioms (5)
  • domain assumption SAM2's memory attention and image encoder are the dominant inference costs (Fig. 2).
    Motivates the two-branch acceleration; if other components dominated, the speedups would be lower.
  • domain assumption Cosine similarity between memory tokens and initial-prompt anchors is a reliable measure of target identity across the video.
    Used in Eq. 14 and 21 for pruning and routing; no adaptation of anchors over time.
  • domain assumption The predicted occlusion score o_t is a trustworthy indicator of memory quality/visibility.
    Used to gate fusion in Eq. 18, archive in Eq. 19, and trigger fallback in Eq. 23.
  • ad hoc to paper A fixed maximum-alignment threshold 0.5 in Eq. 22 separates target-relevant from background windows across all scales.
    No scale/context dependence is modeled.
  • domain assumption The bypass branch can be trained on 30 SA-V videos and generalizes to other domains.
    The paper trains the shortcut network on 30 videos but does not specify selection or assess domain shift.
invented entities (2)
  • Insurance bank (M_ins) independent evidence
    purpose: Parallel FIFO buffer that archives high-confidence memory entries to prevent catastrophic loss during occlusions (Eq. 19).
    Ablation in Table III(e) shows removing it drops J&F from 82.4 to 76.4, confirming it has a causal effect.
  • Anchor set A independent evidence
    purpose: Diverse subset of initial foreground tokens used as an oracle for target identity in pruning and routing (Eq. 13-14).
    Ablations on lambda_A and the anchor sampling ratio show the anchors contribute to accuracy; however, they are internal to the paper and not externally falsifiable beyond benchmark numbers.

pith-pipeline@v1.3.0-alltime-deepseek · 20033 in / 13053 out tokens · 119978 ms · 2026-08-01T11:37:41.412971+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2." pith.science (2026). https://pith.science/paper/NV7GYZI4

@misc{pith2026260719811,
  author       = {Pith},
  title        = {Pith review of: Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NV7GYZI4}},
  note         = {Machine review of arXiv:2607.19811}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The Segment Anything Model 2 (SAM2) has advanced temporal promptable segmentation, yet its deployment remains hindered by heavy memory cross-attention overhead and redundant full-frame visual feature extraction. While recent methods explore efficiency via heuristic memory pruning and window-based sparse routing, they typically suffer from catastrophic performance degradation in complex segmentation scenarios replete with occlusions and distractors. To resolve these limitations, we propose \textbf{Lean-SAM2}, a holistic lightweight framework designed to address the above vulnerabilities while systematically eliminating computational redundancies. Specifically, Lean-SAM2 integrates three collaborative mechanisms: (1) Target-Anchored Memory Pruning (TAMP) safeguards target tokens against deceptive attention by modulating raw attention significance with semantic consistency against prompt-derived foreground anchors; (2) Temporal Condensation with Insurance Memory (TCIM) condenses historical context via a visibility-gated fusion while conditionally archiving high-confidence entries in a parallel insurance bank; and (3) Target-Anchored Risk-Aware Routing (TARR) selectively activates the heavy image encoder for target-related windows based on anchor similarity, utilizing a risk-aware fallback policy to trigger full-frame refreshes during volatile transitions. Extensive evaluations across multiple challenging benchmarks demonstrate that Lean-SAM2 establishes a superior balance between accuracy and efficiency. For example, on the LVOSv2 validation dataset, Lean-SAM2 achieves overall inference speedups of $1.412\times$ and $1.417\times$ on the SAM2.1-Large and SAM2.1-Base+, respectively, significantly outperforming Efficient-SAM2 while boosting the corresponding $\mathcal{J}\&\mathcal{F}$ scores by $5.0\%$ and $3.6\%$. Code is available at https://github.com/DeawhaleQwQ/Lean-SAM2.

Figures

Figures reproduced from arXiv: 2607.19811 by Huazhong Liu, Wenlun Zhang, Xudong Ouyang, Yimin Xu, Yunshan Zhong.

Figure 1
Figure 1. Figure 1: The framework of the proposed Lean-SAM2. Lean-SAM2 introduces three collaborative mechanisms, including Target-Anchored Memory Pruning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: In frames with distractors and small target regions, attention-only [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Runtime latency decomposition of SAM2 across various model [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cumulative distribution function (CDF) of token similarity within the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: In this frame, the predicted mask of Efficient-SAM2 drifts toward [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: GPU memory footprint comparison within the memory attention [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison between Efficient-SAM 2 and Lean-SAM2, with Ground Truth provided for reference. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The per-frame J &F scores of the predicted masks at different timestamps, where Lean-SAM2 demonstrates stronger resilience to video segmentation compared to Efficient-SAM2. Results are reported on the full LVOSv2 valid using the SAM2.1-Base+ model. robustness against error propagation. Its performance curve closely tracks the original SAM2 baseline throughout the video and achieves slightly superior accura… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 13 linked inside Pith

  1. [1]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,” inInternational Conference on Learning Repre- sentations, 2025, pp. 28 085–28 128

    2025

  2. [2]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    2023

  3. [3]

    A benchmark dataset and evaluation methodology for video object segmentation,

    F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, “A benchmark dataset and evaluation methodology for video object segmentation,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2016, pp. 724–732

    2016

  4. [4]

    Rethinking space-time networks with improved memory coverage for efficient video object segmentation,

    H. K. Cheng, Y .-W. Tai, and C.-K. Tang, “Rethinking space-time networks with improved memory coverage for efficient video object segmentation,”Advances in Neural Information Processing Systems, vol. 34, pp. 11 781–11 794, 2021

    2021

  5. [5]

    Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,

    H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” inEuropean conference on computer vision, 2022, pp. 640–658

    2022

  6. [6]

    Putting the object back into video object segmentation,

    H. K. Cheng, S. W. Oh, B. Price, J.-Y . Lee, and A. Schwing, “Putting the object back into video object segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3151–3161

    2024

  7. [7]

    Efficient-sam2: Accelerating sam2 with object-aware visual encoding and memory retrieval,

    J. Zhang, Z. Li, X. Liu, and Q. Gu, “Efficient-sam2: Accelerating sam2 with object-aware visual encoding and memory retrieval,” in International Conference on Learning Representations, 2026

    2026

  8. [8]

    Fast sam2 with text-driven token pruning,

    A. Mandal, C. Zhang, F. D. Puspitasari, X. Wang, J. Zhang, C. Qin, G. Wang, Y . Yang, and H. T. Shen, “Fast sam2 with text-driven token pruning,”arXiv preprint arXiv:2512.21333, 2025

    arXiv 2025

  9. [9]

    Tinysam 2: Extreme mem- ory compression for efficient track anything model,

    Z. Ding, Y . Yang, H. Shu, and X. Chen, “Tinysam 2: Extreme mem- ory compression for efficient track anything model,”arXiv preprint arXiv:2605.18013, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Surgical SAM 2: Real- time segment anything in surgical video by efficient frame pruning,

    H. Liu, E. Zhang, J. Wu, M. Hong, and Y . Jin, “Surgical SAM 2: Real- time segment anything in surgical video by efficient frame pruning,” in Advancements In Medical Foundation Models: Explainability, Robust- ness, Security, and Beyond, 2024

    2024

  11. [11]

    Tsms-sam2: multi-scale temporal sampling augmentation and memory-splitting pruning for promptable video object segmentation and tracking in surgical scenarios,

    G. Xu, H.-C. Shao, and Y . Zhang, “Tsms-sam2: multi-scale temporal sampling augmentation and memory-splitting pruning for promptable video object segmentation and tracking in surgical scenarios,”Machine Learning: Health, vol. 2, no. 1, p. 015003, 2026

    2026

  12. [12]

    Efficient track anything,

    Y . Xiong, C. Zhou, X. Xiang, L. Wu, C. Zhu, Z. Liu, S. Suri, B. Varadarajan, R. Akula, F. Iandolaet al., “Efficient track anything,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 11 513–11 524

    2025

  13. [13]

    Samu- rai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory,

    C.-Y . Yang, H.-W. Huang, W. Chai, Z. Jiang, and J.-N. Hwang, “Samu- rai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory,”arXiv preprint arXiv:2411.11922, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Ahcptq: Accurate and hardware-compatible post-training quantization for segment anything model,

    W. Zhang, Y . Zhong, S. Ando, and K. Yoshioka, “Ahcptq: Accurate and hardware-compatible post-training quantization for segment anything model,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 22 383–22 392

    2025

  15. [15]

    Q-sam2: Accurate quantization for segment anything model 2,

    N. Farronato, F. Scheidegger, M. Rigotti, C. Malossi, M. Magno, and H. Qin, “Q-sam2: Accurate quantization for segment anything model 2,” arXiv preprint arXiv:2506.09782, 2025

    arXiv 2025

  16. [16]

    Mix-qsam2: Mixed-precision quantization for high fidelity segmentation in resource constrained scenarios,

    Y . Duan, X. Ren, G. Dong, X. Yang, and Y . Yang, “Mix-qsam2: Mixed-precision quantization for high fidelity segmentation in resource constrained scenarios,” inProceedings of the AAAI Conference on Artificial Intelligence, 2026, pp. 3741–3749

    2026

  17. [17]

    Q-minisam2: A quantization-based benchmark for resource-efficient video segmenta- tion,

    X. Ren, X. Li, K. Wei, X. Yang, and Y . Yang, “Q-minisam2: A quantization-based benchmark for resource-efficient video segmenta- tion,” inProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025, pp. 1829–1837

    2025

  18. [18]

    Efficient video object segmentation and tracking with recurrent dynamic submodel,

    W. Tang, Z. Liang, X. Wan, C. Zhu, Z. Xu, P. Zhou, Y . Song, Y . You, and W. Zhao, “Efficient video object segmentation and tracking with recurrent dynamic submodel,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 20 912–20 921. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, 2026 12

    2026

  19. [19]

    Edgetam: On-device track anything model,

    C. Zhou, C. Zhu, Y . Xiong, S. Suri, F. Xiao, L. Wu, R. Krishnamoorthi, B. Dai, C. C. Loy, V . Chandraet al., “Edgetam: On-device track anything model,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 13 832–13 842

    2025

  20. [20]

    Ahcq-sam: Toward accurate and hardware-compatible post-training seg- ment anything model quantization,

    W. Zhang, Y . Zhong, W. Yan, S. Zhang, S. Ando, and K. Yoshioka, “Ahcq-sam: Toward accurate and hardware-compatible post-training seg- ment anything model quantization,”arXiv preprint arXiv:2503.03088, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Lvos: A benchmark for large-scale long-term video object segmentation,

    L. Hong, Z. Liu, W. Chen, C. Tan, Y . Feng, X. Zhou, P. Guo, J. Li, Z. Chen, S. Gaoet al., “Lvos: A benchmark for large-scale long-term video object segmentation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  22. [22]

    Sentry: Sam2-enhanced neighbor-aware and temporally reasoned memory for visual tracking,

    M. Alansari, Y . Michael, H. AlMarzouqi, M. Naseer, N. Werghi, and S. Javed, “Sentry: Sam2-enhanced neighbor-aware and temporally reasoned memory for visual tracking,”arXiv preprint arXiv:2606.24449, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Object tracking: A survey,

    A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,”Acm computing surveys, vol. 38, no. 4, pp. 13–es, 2006

    2006

  24. [24]

    A survey on deep learning technique for video segmentation,

    T. Zhou, F. Porikli, D. J. Crandall, L. Van Gool, and W. Wang, “A survey on deep learning technique for video segmentation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 6, pp. 7099– 7122, 2022

    2022

  25. [25]

    Medical sam 2: Segment medical images as video via segment anything model 2,

    J. Zhu, A. Hamdi, Y . Qi, Y . Jin, and J. Wu, “Medical sam 2: Segment medical images as video via segment anything model 2,”arXiv preprint arXiv:2408.00874, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Medsam2: Segment anything in 3d medical images and videos,

    J. Ma, Z. Yang, S. Kim, B. Chen, M. Baharoon, A. Fallahpour, R. Asakereh, H. Lyu, and B. Wang, “Medsam2: Segment anything in 3d medical images and videos,”arXiv preprint arXiv:2504.03600, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree,

    S. Ding, R. Qian, X. Dong, P. Zhang, Y . Zang, Y . Cao, Y . Guo, D. Lin, and J. Wang, “Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 13 614–13 624

    2025

  28. [28]

    A distractor-aware memory for visual object tracking with sam2,

    J. Videnovic, A. Lukezic, and M. Kristan, “A distractor-aware memory for visual object tracking with sam2,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 24 255–24 264

    2025

  29. [29]

    Fast segment anything,

    X. Zhao, W. Ding, Y . An, Y . Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,”arXiv preprint arXiv:2306.12156, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Faster segment anything: Towards lightweight sam for mobile applications,

    C. Zhang, D. Han, Y . Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong, “Faster segment anything: Towards lightweight sam for mobile applications,”arXiv preprint arXiv:2306.14289, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything,

    Y . Xiong, B. Varadarajan, L. Wu, X. Xiang, F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun, F. Iandolaet al., “Efficientsam: Leveraged masked image pretraining for efficient segment anything,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 111–16 121

    2024

  32. [32]

    Edgesam: Prompt-in-the- loop distillation for on-device deployment of sam,

    C. Zhou, X. Li, C. C. Loy, and B. Dai, “Edgesam: Prompt-in-the- loop distillation for on-device deployment of sam,”arXiv preprint arXiv:2312.06660, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Tinysam: Pushing the envelope for efficient segment anything model,

    H. Shu, W. Li, Y . Tang, Y . Zhang, Y . Chen, H. Li, Y . Wang, and X. Chen, “Tinysam: Pushing the envelope for efficient segment anything model,” inProceedings of the AAAI conference on artificial intelligence, 2025, pp. 20 470–20 478

    2025

  34. [34]

    Slimsam: 0.1% data makes segment anything slim,

    Z. Chen, G. Fang, X. Ma, and X. Wang, “Slimsam: 0.1% data makes segment anything slim,”Advances in Neural Information Processing Systems, vol. 37, pp. 39 434–39 461, 2024

    2024

  35. [35]

    Supersam: Crafting a sam supernetwork via structured pruning and unstructured parameter prioritization,

    W. Abebe, S. Jafari, S. Yu, A. Dutta, J. Strube, N. R. Tallent, L. Guo, P. Munoz, and A. Jannesari, “Supersam: Crafting a sam supernetwork via structured pruning and unstructured parameter prioritization,”arXiv preprint arXiv:2501.08504, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Sparsesam: Structured sparsification of activations in segment anything models,

    H.-C. Tran, C. H. Nguyen, D. M. Nguyen, M. Niepert, F. Lai, and K. D. Doan, “Sparsesam: Structured sparsification of activations in segment anything models,”arXiv preprint arXiv:2605.17633, 2026

    Pith/arXiv arXiv 2026

  37. [37]

    Structsam: structure-aware prompt adaptation for robust lung cancer lesion segmentation in ct,

    M. Liu, Y . Yao, J. Jia, J. Yao, Z. Huang, Z. Zeng, G. Pu, Y . Wu, Y . Bai, B. Wanget al., “Structsam: structure-aware prompt adaptation for robust lung cancer lesion segmentation in ct,”npj Digital Medicine, 2026

    2026

  38. [38]

    Car-sam: Cross-attention recon- struction for post-training quantization of the segment anything model,

    H. Wen, J. Yu, D. Yang, and J. Li, “Car-sam: Cross-attention recon- struction for post-training quantization of the segment anything model,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 33 632–33 641

    2026

  39. [39]

    Saq-sam: Semantically- aligned quantization for segment anything model,

    J. Zhang, Z. Li, C. Hu, X. Liu, and Q. Gu, “Saq-sam: Semantically- aligned quantization for segment anything model,” inProceedings of the AAAI Conference on Artificial Intelligence, 2026, pp. 12 547–12 555

    2026

  40. [40]

    Efficientvit-sam: Accelerated segment anything model without performance loss,

    Z. Zhang, H. Cai, and S. Han, “Efficientvit-sam: Accelerated segment anything model without performance loss,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7859–7863

    2024

  41. [41]

    Ptq4sam: Post-training quantization for segment anything,

    C. Lv, H. Chen, J. Guo, Y . Ding, and X. Liu, “Ptq4sam: Post-training quantization for segment anything,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 941–15 951

    2024

  42. [42]

    On efficient variants of segment anything model: A survey: X. sun et al

    X. Sun, J. Liu, H. Shen, X. Zhu, and P. Hu, “On efficient variants of segment anything model: A survey: X. sun et al.”International Journal of Computer Vision, vol. 133, no. 10, pp. 7406–7436, 2025

    2025

  43. [43]

    Repvit: Revisiting mobile cnn from vit perspective,

    A. Wang, H. Chen, Z. Lin, J. Han, and G. Ding, “Repvit: Revisiting mobile cnn from vit perspective,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 909–15 920

    2024

  44. [44]

    Sam2lora: Composite loss-guided, parameter-efficient finetuning of sam2 for retinal fundus segmentation,

    S. Mandal, D. Karthikeyan, and M. Paldhe, “Sam2lora: Composite loss-guided, parameter-efficient finetuning of sam2 for retinal fundus segmentation,”arXiv preprint arXiv:2510.10288, 2025

    arXiv 2025

  45. [45]

    Uniultra: Interactive parameter-efficient sam2 for universal ultrasound segmentation,

    Y . Li, Q. Xu, Y . Zhang, X. He, Q. Zhang, Y . Yao, F. B. Tesem, X. Chen, R. Wang, Z. Chenet al., “Uniultra: Interactive parameter-efficient sam2 for universal ultrasound segmentation,”IEEE Transactions on Multimedia, 2026

    2026

  46. [46]

    Prompt-free and efficient sam2 adaptation for biomedical semantic segmentation via dual adapters,

    H. Mitsuoka and K. Hotta, “Prompt-free and efficient sam2 adaptation for biomedical semantic segmentation via dual adapters,”arXiv preprint arXiv:2605.05979, 2026

    Pith/arXiv arXiv 2026

  47. [47]

    Sam2v-btr: Accelerat- ing sam 2 training for 3d medical image segmentation through bootstrap and memory annealing,

    E. Zhang, J. Iwasawa, K. Oda, and Y . Tokuoka, “Sam2v-btr: Accelerat- ing sam 2 training for 3d medical image segmentation through bootstrap and memory annealing,” inIEEE International Conference on Systems, Man, and Cybernetics, 2025, pp. 4456–4461

    2025

  48. [48]

    Mft: Memory- aware fine-tuning of sam2 for efficient long-sequence video object segmentation,

    G. Li, H. Yuan, S. Chen, Q. Hu, J. Wang, and K. Jiang, “Mft: Memory- aware fine-tuning of sam2 for efficient long-sequence video object segmentation,”IEEE Signal Processing Letters, 2026

    2026

  49. [49]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification,

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,”Ad- vances in Neural Information Processing Systems, vol. 34, pp. 13 937– 13 949, 2021

    2021

  50. [50]

    Not all patches are what you need: Expediting vision transformers via token reorganizations,

    Y . Liang, C. Ge, Z. Tong, Y . Song, J. Wang, and P. Xie, “Not all patches are what you need: Expediting vision transformers via token reorganizations,”arXiv preprint arXiv:2202.07800, 2022

    Pith/arXiv arXiv 2022

  51. [51]

    A-vit: Adaptive tokens for efficient vision transformer,

    H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022, pp. 10 809–10 818

    2022

  52. [52]

    Token merging: Your vit but faster,

    D. Bolya, C.-Y . Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman, “Token merging: Your vit but faster,” inInternational Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=JroZRaRw7Eu

    2023

  53. [53]

    Token merging for fast stable diffusion,

    D. Bolya and J. Hoffman, “Token merging for fast stable diffusion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 4599–4603

    2023

  54. [54]

    Algm: Adaptive local-then-global token merging for efficient semantic segmentation with plain vision transformers,

    N. Norouzi, S. Orlova, D. De Geus, and G. Dubbelman, “Algm: Adaptive local-then-global token merging for efficient semantic segmentation with plain vision transformers,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 773–15 782

    2024

  55. [55]

    Segformer++: Efficient token-merging strategies for high-resolution semantic segmen- tation,

    D. Kienzle, M. Kantonis, R. Sch ¨on, and R. Lienhart, “Segformer++: Efficient token-merging strategies for high-resolution semantic segmen- tation,” inIEEE 7th International Conference on Multimedia Information Processing and Retrieval, 2024, pp. 75–81

    2024

  56. [56]

    Efficient and robust video object segmentation through isogenous memory sampling and frame relation mining,

    J. Dang, H. Zheng, J. Lai, X. Yan, and Y . Guo, “Efficient and robust video object segmentation through isogenous memory sampling and frame relation mining,”IEEE Transactions on Image Processing, vol. 32, pp. 3924–3938, 2023

    2023

  57. [57]

    Beyond appearance: Multi-frame spatio-temporal context memory networks for efficient and robust video object segmentation,

    J. Dang, H. Zheng, X. Xu, L. Wang, and Y . Guo, “Beyond appearance: Multi-frame spatio-temporal context memory networks for efficient and robust video object segmentation,”IEEE Transactions on Image Processing, vol. 33, pp. 4853–4866, 2024

    2024

  58. [58]

    Region aware video object segmentation with deep motion modeling,

    B. Miao, M. Bennamoun, Y . Gao, and A. Mian, “Region aware video object segmentation with deep motion modeling,”IEEE Transactions on Image Processing, vol. 33, pp. 2639–2651, 2024

    2024

  59. [59]

    Prototypical matching networks for video object segmentation,

    F. Lin, Z. Qiu, C. Liu, T. Yao, H. Xie, and Y . Zhang, “Prototypical matching networks for video object segmentation,”IEEE Transactions on Image Processing, vol. 32, pp. 5623–5636, 2023

    2023

  60. [60]

    Delving deeper into mask utilization in video object segmentation,

    M. Wang, J. Mei, L. Liu, G. Tian, Y . Liu, and Z. Pan, “Delving deeper into mask utilization in video object segmentation,”IEEE Transactions on Image Processing, vol. 31, pp. 6255–6266, 2022

    2022

  61. [61]

    Mosev2: A more challenging dataset for video object segmentation in complex scenes,

    H. Ding, K. Ying, C. Liu, S. He, X. Jiang, Y .-G. Jiang, P. H. Torr, and S. Bai, “Mosev2: A more challenging dataset for video object segmentation in complex scenes,”arXiv preprint arXiv:2508.05630, 2025

    arXiv 2025

  62. [62]

    Mose: A new dataset for video object segmentation in complex scenes,

    H. Ding, C. Liu, S. He, X. Jiang, P. H. Torr, and S. Bai, “Mose: A new dataset for video object segmentation in complex scenes,” inProceedings IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, 2026 13 of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 20 224–20 234

    2026