REVIEW 4 major objections 6 minor 37 references
HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A training-free combination of hierarchical motion estimation and a split long/short memory bank makes SAM2 materially more robust in long-term video tracking, with relative AUC gains of 9.6% on LaSOT and 7.2% on LaSOText using the large…
desk verdict A plausible training-free SAM2 upgrade with consistent gains, but the evaluation overstates SOTA and the key thresholds are tuned on the same test set, so the exact numbers need scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a two-level scoring system for mask proposals. A coarse motion score $s_{\mathrm{coarse}}$ comes from IoU between the Kalman-filter predicted box and each SAM2 proposal, and is linearly combined with SAM2's IoU score via $\alpha$ (Eq. 1). When the combined score falls below threshold $\tau$, a fine motion score $s_{\mathrm{fine}}$ is computed by propagating farthest-point-sampled points backward through CoTracker3, reconstructing a soft mask with radial basis function kernels, and comparing to historical masks via the Dice coefficient (Eqs. 2 and 3). The same motion scores feed the memory filter: high-confidence frames go to short-term FIFO memory, while frames whose selected mask is far from alternative proposals under directed Hausdorff distance (Eq. 4) are stored as distinctive long-term memory to help later disambiguation.
What would settle it
On a held-out set of long LaSOT sequences with target disappearances, compare the full HiM2SAM-L against a variant whose long-term memory stores random high-confidence frames instead of Hausdorff-selected ones; if AUC does not drop, the distractor-selection heuristic is not the source of the gains.
Extended reading notes
Core claim
The paper's central claim is that treating motion and memory as two coupled problems, rather than patching SAM2's components in isolation, yields consistent gains across all SAM2 model sizes. The authors demonstrate that coarse Kalman-based box prediction combined with selective fine-grained point tracking outperforms either alone, and that a memory bank separated into high-confidence short-term frames and distractor-selected long-term frames recovers targets after occlusion and appearance change better than SAM2's uniform FIFO memory. The improvements hold without retraining and with only a few milliseconds of added latency, and they are largest on the smallest models, where the motion and memory modules compensate for limited representation capacity.
Load-bearing premise
The load-bearing premise is that the hand-designed heuristics choose good memory: a large directed Hausdorff distance between SAM2's proposal masks reliably marks distractor ambiguity worth storing for later, and high IoU-plus-motion scores reliably mark frames safe to keep in short-term memory.
Editorial extensions
If this is right
- If the reported gains replicate, SAM2-based trackers can be upgraded without retraining, preserving class-agnostic generalization.
- Small SAM2 variants (Tiny and Small) benefit proportionally more, so lightweight edge deployments gain the most from the motion and memory modules.
- The hierarchical trigger means the expensive point tracker runs on few frames, keeping real-time throughput; the reported +3.68 ms average latency on the large model is the cost of the full configuration.
- The ablation shows the full model exceeds the sum of individual gains, suggesting motion estimation and memory curation reinforce each other rather than acting independently.
Reading between the lines
- The paper leaves implicit that the same hierarchical gating should transfer to multi-object tracking with SAM2MOT-style pipelines, where expensive point tracking can be reserved for the few frames with low proposal confidence.
- A directly testable extension is to replace the directed Hausdorff criterion with a learned or simpler proxy such as mask IoU spread, and check whether the long-term memory gains persist, which would reveal whether exact distance computation matters.
- If the thresholds in the released code are not robust across datasets, the reported gains may depend on per-dataset tuning; sweeping $\tau$ and the memory intervals on VOT-LT would make that dependency visible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiM2SAM, a training-free enhancement of SAM2 for long-term video object tracking. It adds a hierarchical motion estimator that combines a lightweight Kalman-filter-based coarse prediction with a selectively triggered CoTracker3-based fine point-tracking refinement, and it splits SAM2's memory into short-term and long-term banks with motion-aware and distractor-aware (directed Hausdorff distance) selection rules. Experiments on LaSOT, LaSOText, VOT-LT2020, and VOT-LT2022 report AUC and F-score improvements over SAM2 baselines and prior zero-shot SAM2-based trackers across Tiny, Small, Base, and Large model scales, with the claimed gains attributed to better occlusion and reappearance handling.
Significance. If the reported results hold, the paper provides a practical, low-overhead recipe for improving SAM2-based long-term tracking without additional training, which would be valuable given SAM2's wide adoption. The strengths of the manuscript include multi-benchmark evaluation, an ablation study (Table 4) showing that each proposed module contributes, and consistent improvements across model scales. The authors also provide a code link, which supports reproducibility in principle. However, the paper overstates its state-of-the-art claim on LaSOT, does not report the numerical values of several key hyperparameters, and performs its sensitivity analysis on the same test set used for the headline results, which weakens confidence that the gains are not partly due to benchmark-specific threshold selection.
major comments (4)
- [§5.4, Figure 3] The sensitivity analysis is performed on the LaSOT test set, which is also the source of the headline AUC numbers in Table 1, and the final parameter values (α, β, τ, point-tracker window, LM/SM intervals, Hausdorff threshold) are not reported in the paper, only said to be in the released code. This makes it impossible to verify that the improvements are robust rather than the result of tuning on the evaluation set. Please report the exact threshold values in the manuscript and evaluate sensitivity on a held-out validation split (e.g., LaSOT train sequences or a different benchmark) or provide sequence-level error bars and a statistical comparison.
- [Abstract and §5.2, Table 1] The abstract claims "state-of-the-art performance on LaSOT and LaSOText with the large model," but Table 1 shows MCITrack-B achieves AUC 75.3 on LaSOT and DAM4SAM-L ties HiM2SAM-L at 75.1. The claim is therefore inaccurate for LaSOT; the method is state-of-the-art among zero-shot methods, not absolute SOTA. Please revise the wording to "best zero-shot performance" and explicitly acknowledge the tie with DAM4SAM-L and the higher supervised result.
- [§5.3, Table 4] The ablation reports latency averaged over a "representative LaSOT subset" and claims the full method adds only 3.68 ms, but the table also shows that applying the point tracker (PT) alone adds 165.95 ms. The average overhead therefore depends entirely on how often the fine-motion stage is triggered by the threshold τ. Without reporting the trigger rate or the distribution of per-frame latencies, the "low-overhead" claim is not fully supported. Please report the fraction of frames on which CoTracker3 is run and the variance of the latency.
- [§4.2, Eq. (4)] The long-term memory selection relies on the heuristic that a large directed Hausdorff distance between SAM2 proposal masks indicates distractor-induced ambiguity, but no evidence is provided that this specific criterion identifies useful memory frames. The paper also does not state the threshold or how it is normalized (e.g., relative to object size), which is important because the raw pixel distance will scale with object dimensions. Please provide an ablation that isolates the Hausdorff-distance selection rule and report the chosen threshold and normalization scheme.
minor comments (6)
- [General] Several table captions contain the typo "T able" instead of "Table" (Tables 1, 2, and 3).
- [Abstract] The term "trainless" is informal; consider using "training-free" for consistency with the rest of the paper.
- [§4.1, Eq. (3)] The parameters α and β are introduced but the text does not state their feasible range; please state that α, β ≥ 0 and α + β ≤ 1, and indicate where their values are set.
- [Figure 3] The right panel of Figure 3 is difficult to interpret because the color scale and circle-size legend for the AUC values are not explained in the caption; please make the axes and legend self-contained.
- [Table 2] SAM2Long is not included in the VOT-LT2020/VOT-LT2022 comparison; if its results are not available, please state this explicitly, or include them if they are reported in the literature.
- [§5.4] The sentence "The final selection of parameters is provided in our released code" is not sufficient for a self-contained paper; at least the numerical values of the main thresholds should appear in a table in the manuscript.
Circularity Check
No circular derivation: the claimed gains are empirical measurements against independent benchmarks, and the hyperparameter-tuning concern is an evidential limitation, not a definitional reduction.
full rationale
The paper's contribution is an engineering modification of SAM2, not a derivation from first principles whose output is defined by its input. The score-fusion equations (1) and (3) linearly combine motion confidence, SAM2 IoU, and point-tracker Dice scores; the selected mask is the one maximizing the combined score, and the reported AUC/F-score is measured against ground truth, so the result is not equivalent to the input by construction. The memory filter (Eq. 4) uses directed Hausdorff distance to select frames for long-term memory, but its effect is tested empirically in the ablation (Table 4), not assumed by definition. The main legitimate concern is evidential: Section 5.4 says 'The final selection of parameters is provided in our released code,' and the sensitivity analysis is run on the same LaSOT dataset used for the headline 9.6% gain, so the LaSOT number is not fully out-of-sample. However, this is a reproducibility and overfitting risk, not circularity: no fitted parameter is renamed as a prediction, and the method also improves on LaSOText, VOT-LT2020, and VOT-LT2022, which were not used for the reported parameter sensitivity. Self-citations ([28], [36]) appear only in related-work context and are not load-bearing; no uniqueness theorem from the authors is invoked to forbid alternatives. The derivation chain is therefore self-contained as an empirical claim, with no step reducing to its own inputs.
Assumptions & free parameters
free parameters (6)
- Alpha (alpha) in Eq. (1) and Eq. (3) =
Not reported in paper; final value in code
- Beta (beta) in Eq. (3)
- Threshold tau for fine motion trigger
- Point-tracker temporal window
- Long-term and short-term memory frame intervals
- Hausdorff distance threshold for long-term memory inclusion
assumptions (6)
- domain assumption SAM2.1 with its pre-trained weights provides the base tracker and memory attention as described in Section 3.
- domain assumption CoTracker3 provides sufficiently reliable pixel correspondences for backward point propagation.
- domain assumption The Kalman filter's constant-velocity box model is a reasonable coarse motion prior.
- domain assumption Benchmark AUC, P, Pnorm and F-score on LaSOT, LaSOText, VOT-LT2020, and VOT-LT2022 measure long-term tracking quality.
- standard math Farthest point sampling plus Gaussian RBF reconstruction yields a usable soft mask from visible tracked points.
- ad hoc to paper Large directed Hausdorff distance between SAM2 proposals indicates distractor-induced ambiguity.
Cite this review
Pith. "Pith review of HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking." pith.science (2026). https://pith.science/paper/TFSQG32W
@misc{pith2026250707603,
author = {Pith},
title = {Pith review of: HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/TFSQG32W}},
note = {Machine review of arXiv:2507.07603}
}
read the original abstract
This paper presents enhancements to the SAM2 framework for video object tracking task, addressing challenges such as occlusions, background clutter, and target reappearance. We introduce a hierarchical motion estimation strategy, combining lightweight linear prediction with selective non-linear refinement to improve tracking accuracy without requiring additional training. In addition, we optimize the memory bank by distinguishing long-term and short-term memory frames, enabling more reliable tracking under long-term occlusions and appearance changes. Experimental results show consistent improvements across different model scales. Our method achieves state-of-the-art performance on LaSOT and LaSOText with the large model, achieving 9.6% and 7.2% relative improvements in AUC over the original SAM2, and demonstrates even larger relative gains on smaller models, highlighting the effectiveness of our trainless, low-overhead improvements for boosting long-term tracking performance. The code is available at https://github.com/LouisFinner/HiM2SAM.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Cambridge University Press (2003)
Buhmann, M.: Radial Basis Functions: Theory and Implementations. Cambridge University Press (2003)
work page 2003
-
[2]
Cai, W., Liu, Q., Wang, Y.: Hiptrack: Visual tracking with historical prompts (2024), https://arxiv.org/abs/2311.02072
work page Pith review arXiv 2024
-
[3]
Cheng, H.K., Oh, S.W., Price, B., Lee, J.Y., Schwing, A.: Putting the object back into video object segmentation. In: arXiv (2023)
work page 2023
-
[4]
In: European Conference on Computer Vision (ECCV) (2022)
Cheng, H.K., Schwing, A.G.: XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In: European Conference on Computer Vision (ECCV) (2022)
work page 2022
-
[5]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Cuttano, C., Trivigno, G., Rosi, G., Masone, C., Averta, G.: Samwise: Infusing wis- dom in sam2 for text-driven video segmentation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 3395–3405 (2025) 14 R. Chen et al
work page 2025
-
[6]
arXiv preprint arXiv:2410.16268 (2024)
Ding, S., Qian, R., Dong, X., Zhang, P., Zang, Y., Cao, Y., Guo, Y., Lin, D., Wang, J.: Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree. arXiv preprint arXiv:2410.16268 (2024)
arXiv 2024
-
[7]
Fan, H., Bai, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Harshit, Huang, M., Liu, J., Xu, Y., Liao, C., Yuan, L., Ling, H.: Lasot: A high-quality large-scale single object tracking benchmark (2020),https://arxiv.org/abs/2009.03465
arXiv 2020
-
[8]
Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., Ling, H.: Lasot: A high-quality benchmark for large-scale single object tracking (2019), https://arxiv.org/abs/1809.07845
work page Pith review arXiv 2019
Show all 37 references
-
[9]
IEEE Transactions on Pattern Analysis and Machine Intelli- gence 15(9), 850–863 (1993).https://doi.org/10.1109/34.232073
Huttenlocher, D., Klanderman, G., Rucklidge, W.: Comparing images using the hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelli- gence 15(9), 850–863 (1993).https://doi.org/10.1109/34.232073
1993 doi
-
[10]
arXiv preprint arXiv:2504.04519 (2025)
Jiang, J., Wang, Z., Zhao, M., Li, Y., Jiang, D.: Sam2mot: A novel paradigm of multi-object tracking by segmentation. arXiv preprint arXiv:2504.04519 (2025)
2025
-
[11]
Trans- actions of the ASME–Journal of Basic Engineering82(Series D), 35–45 (1960)
Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans- actions of the ASME–Journal of Basic Engineering82(Series D), 35–45 (1960)
1960
-
[12]
In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)
Kang, B., Chen, X., Lai, S., Liu, Y., Liu, Y., Wang, D.: Exploring enhanced con- textual information for video-level object tracking. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)
2025
-
[13]
Karaev, N., Makarov, I., Wang, J., Neverova, N., Vedaldi, A., Rupprecht, C.: Co- Tracker3: Simpler and better point tracking by pseudo-labelling real videos (2024)
2024
-
[14]
In: European Conference on Computer Vision (ECCV) (2024)
Karaev, N., Rocco, I., Graham, B., Neverova, N., Vedaldi, A., Rupprecht, C.: Cotracker: It is better to track together. In: European Conference on Computer Vision (ECCV) (2024)
2024
-
[15]
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything (2023), https://arxiv.org/abs/2304.02643
2023 arXiv
-
[16]
In: European Conference on Com- puter Vision (ECCV)
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.K., Chang, H.J., Danelljan, M., Čehovin Zajc, L., Lukežič, A., et al.: The tenth visual object tracking vot2022 challenge results. In: European Conference on Com- puter Vision (ECCV). Springer (2022)
2022
-
[17]
In: European Conference on Com- puter Vision (ECCV Workshops)
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.K., Danelljan, M., Čehovin Zajc, L., Lukežič, A., Drbohlav, O., et al.: The eighth visual object tracking vot2020 challenge results. In: European Conference on Com- puter Vision (ECCV Workshops...
2020
-
[18]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Li, J., Cantu, F.J.P., Yu, E., Wong, A., Cui, Y., Chen, Y.: Samjam: Zero-shot video scene graph generation for egocentric kitchen videos. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 467–473 (2025)
2025
-
[19]
In: European Conference on Computer Vision (ECCV) (2024)
Lin, L., Fan, H., Zhang, Z., Wang, Y., Xu, Y., Ling, H.: Tracking meets lora: Faster training, larger model, stronger performance. In: European Conference on Computer Vision (ECCV) (2024)
2024
-
[20]
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks forvolumetricmedicalimagesegmentation(2016), https://arxiv.org/abs/1606. 04797
2016
-
[21]
arXiv preprint arXiv:1706.02413 (2017)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learn- ing on point sets in a metric space. arXiv preprint arXiv:1706.02413 (2017)
2017 arXiv
-
[22]
arXiv preprint arXiv:2408.00714 (2024),https://arxiv.org/ abs/2408.00714 Enhancing SAM2 with Motion and Memory Optimization 15
Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv...
2024 arXiv
-
[23]
In: Pro- ceedings of the 40th International Conference on Machine Learning (ICML) (2023)
Ryali, C., Hu, Y.T., Bolya, D., Wei, C., Fan, H., Huang, P.Y., Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoffman, J., Malik, J., Li, Y., Feichtenhofer, C.: Hiera: A hierarchical vision transformer without the bells-and-whistles. In: Pro- ceedings of the 40th International Co...
2023
-
[24]
arXiv preprint arXiv:2410.01806 (2024)
Segu, M., Piccinelli, L., Li, S., Yang, Y.H., Van Gool, L., Schiele, B.: Samba: Synchronized set-of-sequences modeling for end-to-end multiple object tracking. arXiv preprint arXiv:2410.01806 (2024)
2024 arXiv
-
[25]
In: Proceedings of the AAAI Conference on Artificial Intelligence (2024)
Shi, L., Zhong, B., Liang, Q., Li, N., Zhang, S., Li, X.: Explicit visual prompts for visual object tracking. In: Proceedings of the AAAI Conference on Artificial Intelligence (2024)
2024
-
[26]
2024 IEEE International Conference on Image Processing (ICIP) pp
Shim, K., Ko, K., Hwang, J., Kim, C.: Adaptrack: Adaptive thresholding-based matching for multi-object tracking. 2024 IEEE International Conference on Image Processing (ICIP) pp. 2222–2228 (2024)
2024
-
[27]
Machine Vision and Applications 35(3) (2024)
Stanojevic, V.D., Todorovic, B.T.: Boosttrack: boosting the similarity measure and detection confidence for improved multiple object tracking. Machine Vision and Applications 35(3) (2024)
2024
-
[28]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Sun, G., Liu, Y., Ding, H., Probst, T., Van Gool, L.: Coarse-to-fine feature mining for video semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3126–3137 (2022)
2022
-
[29]
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need (2023),https://arxiv.org/abs/1706. 03762
2023
-
[30]
In: Comp
Videnovic, J., Lukezic, A., Kristan, M.: A distractor-aware memory for visual ob- ject tracking with SAM2. In: Comp. Vis. Patt. Recognition (2025)
2025
-
[31]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Xie, J., Zhong, B., Mo, Z., Zhang, S., Shi, L., Song, S., Ji, R.: Autoregressive queries for adaptive tracking with spatio-temporal transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19300–19309 (2024)
2024
-
[32]
Yang, C.Y., Huang, H.W., Chai, W., Jiang, Z., Hwang, J.N.: Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory (2024), https://arxiv.org/abs/2411.11922
2024 arXiv
-
[33]
Proceedings of the AAAI Conference on Artificial Intelligence38(7), 6702–6710 (Mar 2024).https: //doi.org/10.1609/aaai.v38i7.28493
Yi, K., Luo, K., Luo, X., Huang, J., Wu, H., Hu, R., Hao, W.: Ucmctrack: Multi- object tracking with uniform camera motion compensation. Proceedings of the AAAI Conference on Artificial Intelligence38(7), 6702–6710 (Mar 2024).https: //doi.org/10.1609/aaai.v38i7.28493
2024 doi
-
[34]
In: European Conference on Computer Vision (ECCV) (2022)
Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., Wei, Y.: Motr: End-to-end multiple-object tracking with transformer. In: European Conference on Computer Vision (ECCV) (2022)
2022
-
[35]
In: Proceedings of the AAAI Confer- ence on Artificial Intelligence (2024)
Zheng, Y., Zhong, B., Liang, Q., Mo, Z., Zhang, S., Li, X.: Odtrack: Online dense temporal token learning for visual tracking. In: Proceedings of the AAAI Confer- ence on Artificial Intelligence (2024)
2024
-
[36]
Visual Intelligence3(1), 10 (2025)
Zhou, Y., Sun, G., Li, Y., Xie, G.S., Benini, L., Konukoglu, E.: When sam2 meets video camouflaged object segmentation: A comprehensive evaluation and adapta- tion. Visual Intelligence3(1), 10 (2025)
2025
-
[37]
IEEE Transactions on Circuits and Sys- tems for Video Technology (2024)
Zhu, W., Cao, J., Xie, J., Yang, S., Pang, Y.: Clip-vis: Adapting clip for open- vocabulary video instance segmentation. IEEE Transactions on Circuits and Sys- tems for Video Technology (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.