Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A SAM2 pipeline with backward tracking and tracklet interpolation achieves 89.4 AUC, first place on the 2024 ICPR Multi-Modal Tracking challenge.

desk verdict A thin but honest challenge report: SAM2-on-RGB is the real winner, and the two proposed 'tricks' add 0.8 AUC but with no ablations, no parameters, and an unvalidated core assumption. read the letter →

arxiv 2505.18111 v1 pith:SKHPKE7Y submitted 2025-05-23 cs.CV

Cheng-Yen Yang , Hsiang-Wei Huang , Pyong-Kun Kim , Chien-Kai Kuo , Jui-Wei Chang , Kwang-Ju Kim , Chung-I Huang , Jenq-Neng Hwang This is my paper · ORCID
classification cs.CV
keywords VisualobjecttrackingMulti-modalSegmentAnythingModel2BackwardTrackletinterpolationBoundingboxextractionICPRchallengeRGB-infrared-depth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes how to turn SAM2, a video segmentation model, into a bounding-box visual object tracker. The authors report a first-place AUC of 89.4 on Track 1 of the 2024 ICPR Multi-Modal Tracking challenge, well above the 74.1 of the ViPT baseline. Their additions are backward tracking, which re-runs the tracker in reverse from the last frame to recover missed detections, and tracklet interpolation, which replaces jittery bounding boxes with linear interpolation guided by stable anchor frames. The result is evidence that a segmentation model with light task-specific post-processing can outperform purpose-built multi-modal trackers on this benchmark.

What carries the argument

The machinery is SAM2's memory-attention segmentation together with the paper's tracklet interpolation rule. SAM2 is a promptable video segmentation model whose memory bank lets a prompt (here the first-frame box) propagate an object mask through the video; the adaptation extracts a bounding box from each mask by taking min/max coordinates. The interpolation rule computes $\Delta r_i$, the absolute percentage change in box width-to-height ratio from frame $i-1$ to $i$, forms a threshold $t$ as the mean of these changes, and flags frames with $\Delta r_i > \alpha t$. Those frames are replaced by $b'_i = b_a + \frac{i-a}{b-a}(b_b - b_a)$ anchored on the nearest unaffected frames, iterated until $\max_i \Delta r_i < \beta t$.

What would settle it

Run the pipeline on a tracking benchmark with sequences where targets legitimately deform (a person turning, a galloping animal, an opening umbrella) and compare AUC with tracklet interpolation on versus off; if interpolation-on scores lower, the aspect-ratio threshold is overwriting correct boxes. A simpler check is to compute $\Delta r_i$ on ground-truth boxes from such sequences and see how often a valid frame exceeds $\alpha t$.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that SAM2's video memory, already able to carry an object through occlusion and appearance change, can be repurposed for visual object tracking by converting its segmentation masks to bounding boxes and adding two steps. The pipeline first prompts SAM2 with the first-frame box and propagates masks forward; it then re-runs the sequence backward from the final box so that a different memory state can recover objects the forward pass lost. A post-processing pass computes the per-frame percentage change of the box aspect ratio, flags frames whose change exceeds a threshold based on the mean change, and linearly interpolates those frames between the nearest unaffected anchors. On the challenge test set, this full pipeline scores 89.4 AUC, while RGB-only SAM2-l scores 88.6 and the ViPT baseline scores 74.1; depth-only and infrared-only inputs score far lower at 19.8 and 56.5.

Load-bearing premise

The load-bearing premise is that a large jump in the bounding-box aspect ratio reliably signals a tracking failure rather than a real change in the target's shape, so replacing those frames with linear interpolation preserves valid track.

Editorial extensions

If this is right

  • Using RGB input alone outperforms infrared (56.5) and depth (19.8) on this benchmark, so the multi-modal fusion the challenge name suggests was not what drove the first-place result.
  • Backward tracking can recover objects after occlusion or motion blur because the reverse pass starts from the last frame's box and builds a different memory trajectory.
  • Tracklet interpolation removes jitter while preserving the overall trajectory, provided the target's box aspect ratio stays roughly constant.
  • The full pipeline beats the RGB-only SAM2 baseline by 0.8 AUC points and the ViPT baseline by 15.3 AUC points on the challenge test set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to ablate backward tracking and tracklet interpolation separately; the report gives only their combined effect, so the individual gain of each is unknown.
  • The aspect-ratio assumption may transfer to other trackers or datasets with rigid, slowly-deforming targets, but would need validation on sequences with legitimate shape changes, where interpolation could erase correct boxes.
  • Because the backward pass needs the whole video before producing output, the method is offline; a causal variant that only uses frames up to the current one would be a meaningful next step.
  • The large margin over ViPT suggests that on this benchmark, tight, stable ground-truth boxes favor segmenters; a dataset annotated with loose boxes or heavy deformation might not show the same ordering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports the first-place solution of the UWIPL_ETRI team on Track 1 (Multi-Modal Tracking) of the 2024 ICPR Multi-Modal Visual Pattern Recognition challenge. The method adapts SAM2 to single-object visual tracking by prompting it with the first-frame bounding box, extracting boxes from generated masks, running the tracker forward and then backward, and applying a tracklet interpolation post-processing that replaces frames where the bounding-box aspect ratio changes by more than α times the mean change with linearly interpolated boxes. The paper reports 89.4 AUC on the test set, ranked first among competitors, with an RGB SAM2-l baseline of 88.6 AUC and a ViPT baseline of 74.1 AUC. The core claim is that the proposed optimizations—backward tracking and tracklet interpolation—are responsible for the top performance.

Significance. The paper provides a useful data point that SAM2, a foundation segmentation model, can be adapted to VOT with modest modifications and achieve strong performance on a multi-modal tracking benchmark. The per-modality breakdown (RGB 88.6, infrared 56.5, depth 19.8) is a valuable empirical observation that challenges the value of adding infrared/depth modalities to a strong RGB tracker. However, the scientific contribution is limited by the absence of any ablation isolating the two proposed tricks, missing hyperparameter values and merge rules, and lack of a validation protocol. The 0.8 AUC gap between the RGB-only baseline and the full system is not attributed to any specific component. The reported leaderboard score is externally verified by the challenge organizer, which is a strength, but the internal claim of effectiveness is under-supported. If the ablations and parameter choices are provided and confirm the reported gain, the paper would be a solid challenge report; as written, it is more of an extended abstract than a complete technical description.

major comments (4)
  1. [Section 3.3, Eqs. (1)-(4), and Table 2] The central claim that the proposed optimizations cause the first-place result is not supported by any ablation. Table 2 reports 88.6 AUC for RGB SAM2-l and the text states that adding the tricks yields 89.4 AUC, but the individual contribution of backward tracking and tracklet interpolation is never measured. A reader cannot determine whether the 0.8-point gain comes from interpolation, backward tracking, both, or from evaluation details. The paper should include at least forward-only, forward+interpolation, forward+backward, and full-pipeline AUCs on a fixed validation split.
  2. [Section 3.3, Eq. (3)] The operational assumption that large changes in the aspect-ratio (Δr_i > αt) indicate tracking failure rather than genuine target deformation is asserted but never validated. The sentence 'Most objects in the dataset exhibit limited deformation and maintain relatively stable bounding box ratios and sizes' is not supported by any statistic. If the test set contains legitimate scale/aspect-ratio changes, the interpolation would overwrite correct boxes and potentially hurt performance. To make the claim load-bearing, the authors should report the distribution of Δr_i computed from ground-truth boxes, the number of frames flagged for interpolation on the test set, and the performance with interpolation disabled.
  3. [Section 3.2] The backward tracking procedure is underspecified. The three steps describe running SAM2 in reverse initialized from the last forward box, but no rule is provided for merging the forward and backward bounding-box sequences into a single output. The paper also does not report any experiment showing that backward tracking helps, or on which sequences. Without a concrete fusion rule (e.g., choosing per-frame the box with higher predicted IoU, or splitting sequences by a confidence threshold), the method is not reproducible and the reported 89.4 AUC cannot be attributed to this component. The authors should define the merge rule and give forward-only, backward-only, and merged results.
  4. [Section 4.2 and Table 1] The validation and evaluation protocol is incomplete. The paper does not state whether the test AUC is from a single challenge submission or an average over multiple runs, whether SAM2 inference involves stochastic components that require multiple trials, or whether α, β, and the number of interpolation iterations were tuned on the test set. It also reports no error bars or per-sequence variance. Given that the claimed improvement over the RGB-only baseline is only 0.8 AUC, the reader cannot assess whether this difference is reliable. The authors should describe the official evaluation procedure, report whether the final numbers were obtained by the challenge server, and provide standard deviations or per-sequence analyses.
minor comments (5)
  1. [Section 3.3, Eq. (2)] The threshold t is the mean of the same Δr_i values being thresholded, so the criterion is circular and sensitive to outliers; using the median or a robust estimator would be more defensible.
  2. [References] The paper contains duplicate references for the ViPT model: [6] and [19] (both 'Visual prompt multi-modal tracking') are the same paper; consolidate them.
  3. [References] Several references are incomplete or have typos, such as [4] lacking venue/page information and [6] having a malformed author name; additionally, 'T able 1' in the experiments section is a formatting error.
  4. [Related Work / Experiments] The paper does not provide a comparison with the authors' own SAMURAI [15], which is a closely related SAM2-based tracker; adding that baseline would substantially clarify the incremental contribution of the proposed tricks.
  5. [Section 3.3, Eq. (3)] The interpolation formula can produce bounding boxes outside the image boundaries; a clipping step or a validity check should be specified.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reported first-place AUC is an externally evaluated empirical result, and the tracklet-interpolation heuristic is a post-processing rule rather than a self-defined prediction.

full rationale

The central claim is an empirical result: an AUC of 89.4 on the external 2024 ICPR MMVPR Track 1 leaderboard. None of the paper's equations define the reported score in terms of the method's inputs. Eqs. (1)-(4) describe a post-processing heuristic: Delta r_i is the relative change in the predicted bounding-box aspect ratio, t is the mean of those changes, and frames exceeding alpha t are replaced by linear interpolation between anchor frames. This is a data-dependent smoothing rule, but the output bounding boxes are still evaluated against held-out ground truth through Eqs. (5)-(7); no fitted parameter is renamed as a prediction, and no quantity is defined circularly. The related-work section cites the authors' own SAMURAI paper [15], and the reference list contains several other self-citations, but none is load-bearing: the method builds on the externally developed SAM2 model and the challenge's official evaluation, and the self-citations are descriptive rather than used to justify the proposed tricks. The unvalidated assumption that large aspect-ratio jumps indicate tracking failure rather than genuine deformation is a correctness and robustness concern, not a circularity concern. Therefore no significant circularity is present; the minor non-load-bearing self-citation places this at the low end of the scale.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method introduces no new model components; it adds two post-processing heuristics controlled by alpha and beta. The assumptions are about dataset behavior, SAM2's transferability, the backward pass, and the evaluation protocol. No invented entities are introduced.

free parameters (2)
  • alpha (alpha) = not reported
    Threshold multiplier for flagging frames with large bounding-box ratio changes (Eq. 1 and 'alpha t' in Section 3.3). It is a tunable parameter; no value or validation procedure is given.
  • beta (beta) = not reported
    Convergence threshold multiplier in Eq. (4), 'typically set slightly higher than alpha'; no value or tuning procedure reported.
assumptions (5)
  • domain assumption Large changes in bounding-box aspect ratio indicate tracking failures rather than genuine target deformation.
    Explicitly asserted in Section 3.3: 'Most objects in the dataset exhibit limited deformation and maintain relatively stable bounding box ratios and sizes.' The interpolation replaces flagged frames with linear boxes, so if a target legitimately changes aspect ratio, the correction would erase the true motion.
  • domain assumption Re-running SAM2 backward from the last frame yields better or recoverable tracking results.
    Section 3.2 justifies backward tracking heuristically; there is no derivation, ablation, or explicit rule for how forward and backward outputs are combined.
  • domain assumption SAM2's pretrained video segmentation and memory module transfer successfully to single-object bounding-box tracking.
    The entire method depends on SAM2's masks being accurate enough that box conversion and memory propagation yield strong tracking; the paper does not fine-tune SAM2.
  • domain assumption The challenge's official evaluation protocol and leaderboard results are accurate.
    The AUC calculation and the leaderboard (Table 1) are taken as ground truth; no external verification or evaluation code is provided.
  • domain assumption RGB alone is the best modality for this dataset and multi-modal fusion is unnecessary.
    Table 2 shows depth and infrared scores far below RGB; the final method uses RGB only, but there is no fusion experiment to test whether combining modalities could improve further. The claim is thus an assumption about the dataset rather than a demonstrated design principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking." pith.science (2026). https://pith.science/paper/SKHPKE7Y

@misc{pith2026250518111,
  author       = {Pith},
  title        = {Pith review of: Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SKHPKE7Y}},
  note         = {Machine review of arXiv:2505.18111}
}
read the original abstract

We present an effective approach for adapting the Segment Anything Model 2 (SAM2) to the Visual Object Tracking (VOT) task. Our method leverages the powerful pre-trained capabilities of SAM2 and incorporates several key techniques to enhance its performance in VOT applications. By combining SAM2 with our proposed optimizations, we achieved a first place AUC score of 89.4 on the 2024 ICPR Multi-modal Object Tracking challenge, demonstrating the effectiveness of our approach. This paper details our methodology, the specific enhancements made to SAM2, and a comprehensive analysis of our results in the context of VOT solutions along with the multi-modality aspect of the dataset.

Figures

Figures reproduced from arXiv: 2505.18111 by the authors.

Figure 1
Figure 1. An example illustration of adapting SAM2 [11] on VOT task. This pipeline leverages SAM2’s powerful segmentation capabilities by using an initial bounding box prompt on the first frame, then utilizes its memory bank feature to propagate and refine object masks through subsequent video frames, enabling efficient and accurate object tracking. Compared to its predecessor, SAM2 introduces several key innovations that imp… view at source ↗
Figure 2
Figure 2. Sample of the multi-modal videos from the testing sequence. 4 Experiments 4.1 Dataset For our experiments, we utilized the track1 visual object tracking dataset pro￾vided by the 2024 ICPR Multi-Modal Visual Pattern Recognition Workshop. This dataset is specifically designed to tackle the challenges of tracking objects across different modalities, including RGB, infrared thermal, depth, and event data. It consists of… view at source ↗
Figure 3
Figure 3. Visualization of our tracking results on the ICPR multi-modal tracking dataset. We selected three different tracking cases with different level of difficulties, caused by the moving speed of object, occlusion, and distractor in the environment. When the target is visible, Rt measures the IoU value. If the target is out of view or occluded (Gt is empty), Rt is set to 0. We then compute success rates at different thre… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    In: Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14

    Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., Torr, P.H.: Fully- convolutional siamese networks for object tracking. In: Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14. pp. 850–865. Springer (2016)

    work page 2016

  2. [2]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition

    Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H.: Transformer tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 8126–8135 (2021)

    work page 2021

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(3), 3072–3089 (2023)

    Hu, W., Wang, Q., Zhang, L., Bertinetto, L., Torr, P.H.: Siammask: A framework for fast online object tracking and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence45(3), 3072–3089 (2023)

    work page 2023

  4. [4]

    arXiv preprint arXiv:2403.10826 (2024)

    Huang, H.W., Yang, C.Y., Chai, W., Jiang, Z., Hwang, J.N.: Exploring learning- based motion models in multi-object tracking. arXiv preprint arXiv:2403.10826 (2024)

    arXiv 2024

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, H.W., Yang, C.Y., Jiang, Z., Kim, P.K., Lee, K., Kim, K., Ramkumar, S., Mullapudi, C., Jang, I.S., Huang, C.I., et al.: Enhancing multi-camera peo- ple tracking with anchor-guided clustering and spatio-temporal consistency id re- assignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5239–5249 (2023)

    work page 2023

  6. [6]

    In: CVPR (2023)

    Jiawen, Z., Simiao, l., Xin, C., Wang, D., Lu, H.: Visual prompt multi-modal tracking. In: CVPR (2023)

    work page 2023

  7. [7]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)

    2023

  8. [8]

    In: 2024 IEEE Intelligent Vehicles Symposium (IV)

    Kuan, S.Y., Cheng, J.H., Huang, H.W., Chai, W., Yang, C.Y., Latapie, H., Liu, G., Wu, B.F., Hwang, J.N.: Boosting online 3d multi-object tracking through camera- radar cross check. In: 2024 IEEE Intelligent Vehicles Symposium (IV). pp. 2125–

    work page 2024

Show all 19 references
  1. [9]

    In: Proceedings of the IEEE conference on com- puter vision and pattern recognition

    Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition. pp. 8971–8980 (2018)

    2018

  2. [10]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine- Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–732 (2016)

    2016

  3. [11]

    arXiv preprint arXiv:2408.00714 (2024)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

    2024 arXiv

  4. [12]

    In: Proceedings of the Asian Conference on Computer Vision

    Sun, J., Huang, H.W., Yang, C.Y., Jiang, Z., Hwang, J.N.: Gta: Global track- let association for multi-object tracking in sports. In: Proceedings of the Asian Conference on Computer Vision. pp. 421–434 (2024)

    2024

  5. [13]

    arXiv preprint arXiv:1809.03327 (2018)

    Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., Huang, T.: Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327 (2018)

    2018 arXiv

  6. [14]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Yan, B., Peng, H., Fu, J., Wang, D., Lu, H.: Learning spatio-temporal transformer for visual tracking. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10448–10457 (2021) Title Suppressed Due to Excessive Length 11

    2021

  7. [15]

    arXiv preprint arXiv:2411.11922 (2024)

    Yang, C.Y., Huang, H.W., Chai, W., Jiang, Z., Hwang, J.N.: Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory. arXiv preprint arXiv:2411.11922 (2024)

    2024 arXiv

  8. [16]

    In: Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision

    Yang, C.Y., Huang, H.W., Jiang, Z., Kuo, H.C., Mei, J., Huang, C.I., Hwang, J.N.: Sea you later: Metadata-guided long-term re-identification for uav-based multi- object tracking. In: Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision. pp. 805–812 (2024)

    2024

  9. [17]

    In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Yang, C.Y., Huang, H.W., Kim, P.K., Jiang, Z., Kim, K.J., Huang, C.I., Du, H., Hwang, J.N.: An online approach and evaluation method for tracking people across cameras in extremely long video sequence. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern...

    2024

  10. [18]

    In: Proceedings of the IEEE/CVF International conference on computer vision workshops

    Zhang, L., Danelljan, M., Gonzalez-Garcia, A., Van De Weijer, J., Shahbaz Khan, F.: Multi-modal fusion for end-to-end rgb-t tracking. In: Proceedings of the IEEE/CVF International conference on computer vision workshops. pp. 0–0 (2019)

    2019

  11. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Zhu, J., Lai, S., Chen, X., Wang, D., Lu, H.: Visual prompt multi-modal tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9516–9526 (June 2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.