Pith. sign in

REVIEW 4 major objections 4 minor 6 cited by

R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3-billion-parameter vision-language model fine-tuned with reinforcement learning reaches a 0.680 average overlap on the GOT-10k tracking benchmark, close to expert trackers trained on over fifty times more data.

desk verdict A rough but honestly labeled technical report showing GRPO can turn a small MLLM into a decent GOT-10k tracker; the core result is plausible, but reproducibility gaps and the missing center-prior baseline mean the template-matching claim is not yet established. read the letter →

arxiv 2506.21980 v3 pith:QQTOU2CV submitted 2025-06-27 cs.CV

Biao Wang , Wenwen Li , Jiawei Ge This is my paper
classification cs.CV
keywords visualobjecttrackingmultimodallargelanguagemodelsGRPOreinforcementlearningrule-basedGIoUrewardGOT-10kbenchmarktemplatematchingvision-languagegrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual object tracking has usually meant training a dedicated network with classification and regression heads on large annotated video datasets. This paper asks whether a general-purpose multimodal language model can instead be pointed at tracking directly, and reports that the answer is yes. After a brief supervised warm-up on a few dozen examples and then fine-tuning Qwen2.5-VL-3B with GRPO reinforcement learning and a rule-based GIoU reward on 100,000 template-search image pairs, the authors obtain R1-Track, which reaches an average overlap of 0.680 on the GOT-10k benchmark. That score is above the reported TransT score and close to OSTrack, two expert trackers trained on more than fifty times as much tracking data. The same recipe already works at 5,000 pairs, and the model can be initialized either with a bounding box or with a text description, so tracking becomes one capability of a general model rather than a separate expert system.

What carries the argument

The mechanism is group relative policy optimization (GRPO), which samples several candidate answers for each template-search pair, normalizes each candidate's reward against the group mean and standard deviation, and updates the policy toward the better-than-average responses while penalizing drift from the original model. The only task-specific signal is a rule-based reward whose main component is GIoU between the predicted box and the ground-truth box, with extra bonuses for very high overlap and penalties for malformed output or overly long reasoning. The input format is two resized 336-by-336 images plus a prompt that states the first-frame bounding box, and the model is trained to output the box in the second image; the template is held fixed throughout inference with no re-localization. This combination of a geometric overlap reward and a relative-comparison update rule is what carries the argument from 'the base model cannot track' to 'a 3B multimodal language model scores 0.680 AO on GOT-10k.'

What would settle it

Take the released R1-Track-100k model and run it on sequences where the target's motion between frames exceeds the 20 percent center-shift seen in training, or where target scale falls outside the 2-to-8 search-region factor. If its average overlap collapses while a conventional tracker such as OSTrack degrades only mildly, the model was relying on training-crop priors rather than on target appearance.

Watch

Extended reading notes

Core claim

The paper's central claim is that tracking ability can be elicited from a pretrained vision-language model without any architecture change and without per-coordinate regression supervision. The authors show that native Qwen2.5-VL cannot relocate a boxed target across two images even with extensive prompting, but that GRPO fine-tuning with a piecewise GIoU reward converts the same 3-billion-parameter model into a tracker scoring 0.680 AO on GOT-10k in no-think mode. They further claim that GRPO outperforms standard supervised fine-tuning at the same data scale, that direct answer output beats an explicit reasoning chain at this model size, and that the fine-tuned model keeps most of its original general abilities while optionally accepting text descriptions as initialization. The intended upshot is that future tracking systems may not need dedicated expert architectures; incorporating tracking data into the post-training of a multimodal language model may be enough.

Load-bearing premise

The load-bearing assumption is that the model learns genuine template matching rather than the crop statistics of the GOT-10k training set, since every training example is an isolated pair with a fixed template, a search-region scale factor of 2 to 8, and a center shift of 0 to 20 percent, and no re-localization is used at inference.

Editorial extensions

If this is right

  • A 3-billion-parameter multimodal language model fine-tuned with GRPO reaches 0.680 AO on GOT-10k, which is above the reported TransT score and within reach of OSTrack despite using far less tracking-specific training data.
  • The GRPO recipe works with as few as 5,000 template-search pairs, so the data requirement for injecting tracking into a multimodal language model is modest by current standards.
  • At the same data scale, GRPO with a rule-based reward outperforms supervised fine-tuning, so coordinate-level regression labels are not necessary for this task.
  • Because the same model accepts either a bounding box or a text description as initialization, tracking can be combined with language-specified targets in a single model rather than requiring a separate tracker plus a detector.
  • If the finding generalizes, the tracking field's dedicated expert networks could be replaced or supplemented by post-training a generalist vision-language model on tracking data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of a purely geometric reward suggests the same GRPO recipe could transfer to other fine-grained localization tasks, such as referring-expression segmentation or open-vocabulary detection, where coordinate supervision is costly.
  • The fact that no-think beats think at the 3B scale is a size-dependent result; a larger base model with better reasoning may reverse that ordering, but this paper does not test that hypothesis.
  • Because the template is fixed and inference is frame-by-frame, the model leaves temporal continuity untouched; adding memory or re-detection would likely close most of the remaining gap to expert trackers.
  • The exclusive use of GOT-10k for both training and testing means the reported scores may overstate generalization; a direct evaluation on other benchmarks would separate template matching from dataset-prior learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces R1-Track, a method that applies the MLLM Qwen2.5-VL-3B-Instruct to visual object tracking by fine-tuning it with Group Relative Policy Optimization (GRPO) and a rule-based reward combining format, GIoU, and length terms. Training data are sampled from the GOT-10k training set as template-search image pairs, with a cold-start SFT stage followed by SFT or RL fine-tuning. The central empirical claim, reported in Table 1, is that the best configuration, R1-Track-GRPO on R1-Track-100k in no-think mode, achieves an AO of 0.680 on the GOT-10k benchmark, close to expert trackers trained on substantially larger data. The paper also claims flexible initialization via bounding boxes or text descriptions and retention of the base model's general capabilities, though these claims are not experimentally substantiated.

Significance. If the central result is robust, the paper is significant because it demonstrates that a relatively small MLLM can acquire competitive visual tracking behavior from a modest amount of data through RL with a simple rule-based reward, without task-specific architectural components. The public release of code, datasets, and model weights is a concrete strength that supports reproducibility. The study also provides a useful comparison of SFT versus GRPO on a grounding task. However, the key attribution of the reported AO improvement to genuine template matching is not yet established, and several advertised capabilities are asserted without evaluation. Because these gaps affect the interpretation of the headline numbers, the work needs additional controls and experiments before the claims can be accepted at face value.

major comments (4)
  1. [Section 2 and Table 1] The training pairs in R1-Track-100k are generated with a search-region scale factor of 2-8 and a center-shift factor of 0-0.2, which places the target near the center of the search crop and at a scale correlated with the template. The GIoU reward in Eq. (5) can therefore be partially satisfied by predicting a box near the crop center with a template-consistent scale, independent of target appearance. The paper reports no baseline that simply outputs the center of the search crop, and no control in which the template is replaced by a distractor or a different object. Without such controls, the AO 0.680 in Table 1 cannot be attributed to template matching, which is the central claim of the paper. Please add at least a center-prior baseline and a template-swap control, and report how the results change.
  2. [Section 3.3] The inference procedure is underspecified regarding how the search region is produced for each frame. The text says the template is cropped and cached and tracking is done frame by frame, but it does not state whether the search image is the full frame, a crop centered on the previous prediction, or a crop obtained by some other rule. This matters because a search region centered on the previous prediction recreates the near-centered training condition, whereas a full-frame input changes the task substantially. Please specify the exact inference-time cropping and state whether the same scale and shift statistics as in training are used.
  3. [Table 1 and Section 4] Every configuration is reported as a single run with no variance estimate. The difference between R1-Track-SFT and R1-Track-GRPO on R1-Track-100k is 0.667 versus 0.672/0.680, which is small relative to run-to-run variability typical of RL fine-tuning. The statement that GRPO yields better performance than SFT is therefore not statistically supported. Please provide multiple seeds with error bars or at least two runs per configuration, and adjust the claims accordingly if the differences are within noise.
  4. [Abstract, Section 3.3, and Section 5] The paper claims R1-Track supports text-description initialization and retains most of the base model's general capabilities, but neither claim is evaluated. The text-init scenario is described in Section 3.3 but no experiment or quantitative result is reported for it. Similarly, the conclusion states that GRPO causes less damage to the original model than SFT, yet no general-capability benchmark (e.g., visual question answering, grounding, or instruction following) is run. Please either add the missing evaluations or explicitly remove these claims from the abstract and conclusion.
minor comments (4)
  1. [Section 3.1] The sentence 'The MLLM serves as the reward function' appears to be a typo; the rewards in Section 3.2 are rule-based, not produced by the MLLM. Please rephrase to avoid confusion.
  2. [Equation (5)] The answer reward is constant zero on the interval 0 < GIoU <= 0.4, creating a plateau with no gradient. A brief comment on why this dead zone was chosen and how it affects training would improve the exposition.
  3. [Throughout] The paper inconsistently uses 'GOT10k' and 'GOT-10k'; please standardize. There are also minor grammatical issues such as 'a flagship MLLMs' in the abstract.
  4. [Table 1] The table reports parameter counts for R1-Track and the baselines, but the baseline rows are difficult to compare because training-data sizes and backbones differ widely; a column noting the reported training data scale for each baseline would help the reader interpret the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GOT-10k AO score is an external benchmark result produced by training on the permitted training split, and no load-bearing step reduces the reported result to the training inputs.

full rationale

R1-Track's central claim is an empirical benchmark score: fine-tuning Qwen2.5-VL with GRPO and a GIoU-based reward on R1-Track-100k yields AO 0.680 on GOT-10k. This is not circular. The training data are sampled from the GOT-10k training split, while the reported AO is computed on the GOT-10k test set under the official protocol. The paper does not fit any parameter to the test set; the reward thresholds in Eq. (5) are hand-set piecewise constants, and the GRPO advantage in Eq. (1) is computed within sampled response groups, not against test labels. Using GIoU as both training reward and evaluation metric is standard practice and does not make the evaluation result an artifact of the training objective. No equation in the paper defines a claimed output in terms of the input, and no fitted quantity is renamed as a prediction. The paper also contains no load-bearing self-citation chain: it cites external frameworks (Qwen2.5-VL, DeepSeek-R1, GRPO, DAPO, EasyR1) and does not invoke any prior uniqueness theorem by the same authors. The possible concern that the centered-crop and scale-factor sampling in Section 2 could let the model exploit dataset-specific priors is a generalization-validity question, not a circularity-of-derivation question; it would require additional control experiments, but its absence does not make the benchmark claim equivalent to the training data by construction. Overall, the derivation is self-contained with respect to circularity, and the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central result depends on a handful of hand-chosen training and reward design choices (piecewise GIoU thresholds, reward weights, length bounds, sampling factors, cold-start size) and on the assumption that pair-based training transfers to sequential tracking. No invented entities are introduced.

free parameters (5)
  • Reward thresholds in Eq. (5) = 0, 0.4, 0.75, 0.95 thresholds; +0.2 and +0.5 bonuses
    Hand-chosen piecewise boundaries in the GIoU answer reward; the paper gives no ablation or validation-based selection, so the central result depends on these values.
  • Reward weights a, b, c in Eq. (3) = not specified
    The overall reward is a weighted sum, but the weights are not reported; the method as stated is incomplete.
  • Length reward bounds (Lmin, Lcache, Lmax) in Eq. (6) = not specified
    Used to penalize overlong reasoning, but concrete values are not given.
  • Data sampling factors for R1-Track-100k = scale 2-8, center shift 0-0.2, resolutions 112/224/336/448
    These choices control the difficulty and distribution of training pairs and are not justified by ablations.
  • Cold-start dataset size = 110 points (100 no-think, 10 think)
    The model is bootstrapped on a very small cold-start set; if this is too small, the procedure may be fragile.
assumptions (3)
  • domain assumption The GOT-10k training set can be sampled into independent template-search pairs that provide sufficient coverage for learning tracking, and the official evaluation protocol is correctly applied.
    Section 2 constructs R1-Track-100k by random sampling; the transfer from pair training to sequential test inference is assumed, not proven.
  • domain assumption A fixed template from the first frame, without update or re-localization, is sufficient to track throughout the video.
    Section 3.3 states the template remains fixed and no target re-localization strategies are applied; this is a strong simplification.
  • domain assumption The vision encoder and language model of Qwen2.5-VL provide a good enough prior for grounding that RL fine-tuning converges with 100k pairs.
    The paper treats the base model's grounding ability as a given, which is standard but not proven here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning." pith.science (2026). https://pith.science/paper/QQTOU2CV

@misc{pith2026250621980,
  author       = {Pith},
  title        = {Pith review of: R1-Track: Direct Application of MLLMs to Visual Object Tracking via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQTOU2CV}},
  note         = {Machine review of arXiv:2506.21980}
}
read the original abstract

Visual single object tracking aims to continuously localize and estimate the scale of a target in subsequent video frames, given only its initial state in the first frame. This task has traditionally been framed as a template matching problem, evolving through major phases including correlation filters, two-stream networks, and one-stream networks with significant progress achieved. However, these methods typically require explicit classification and regression modeling, depend on supervised training with large-scale datasets, and are limited to the single task of tracking, lacking flexibility. In recent years, multi-modal large language models (MLLMs) have advanced rapidly. Open-source models like Qwen2.5-VL, a flagship MLLMs with strong foundational capabilities, demonstrate excellent performance in grounding tasks. This has spurred interest in applying such models directly to visual tracking. However, experiments reveal that Qwen2.5-VL struggles with template matching between image pairs (i.e., tracking tasks). Inspired by deepseek-R1, we fine-tuned Qwen2.5-VL using the group relative policy optimization (GRPO) reinforcement learning method on a small-scale dataset with a rule-based reward function. The resulting model, R1-Track, achieved notable performance on the GOT-10k benchmark. R1-Track supports flexible initialization via bounding boxes or text descriptions while retaining most of the original model's general capabilities. And we further discuss potential improvements for R1-Track. This rough technical report summarizes our findings as of May 2025.

Figures

Figures reproduced from arXiv: 2506.21980 by the authors.

Figure 1
Figure 1. Visualization of R1-Track-5k dataset. 2. Re-locate this target in <image_2>; 3. Return [x_min, y_min, x_max, y_max] coordinates of the target in <image_2>. The formatting prompt remains consistent with R1-Track-5k. Cold-Start Data. The SFT and RL fine-tuning processes use identical datasets; differences exist solely in their data assembly formats. Before SFT or RL, we performed a cold start for the base model, also … view at source ↗
Figure 2
Figure 2. The overall framework of R1-Track. using a relatively low learning rate. Subsequently, more refined SFT or RL fine-tuning is performed. Notably, jointly fine-tuning the vision encoder and the LLM significantly improves performance. More detailed parameter settings can be found in our provided code repository. During inference, we recommend deploying the service using vLLM [18] and invoking it on demand for tracking … view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Bridge-STG decouples spatio-temporal alignment via semantic bridging and query-guided localization modules to achieve state-of-the-art m_vIoU of 34.3 on VidSTG among MLLM methods.

  2. Debate-Enhanced Pseudo Labeling and Frequency-Aware Progressive Debiasing for Weakly-Supervised Camouflaged Object Detection with Scribble Annotations

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    D³ETOR combines debate-enhanced pseudo labeling from SAM with frequency-aware progressive debiasing in FADeNet to achieve state-of-the-art weakly-supervised camouflaged object detection using scribbles.

  3. Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    A language dependency parsing mechanism combined with Qwen-VL enables adaptive updates to textual descriptions for improved vision-language tracking performance on benchmarks like TNL2K and LaSOT.

  4. INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    INTENT mitigates cross-modal correspondence noise and modality-inherent noise in composed image retrieval via FFT-based visual invariant composition and bi-objective discriminative learning.

  5. ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    ReTrack calibrates directional bias in composed video features using semantic disentanglement and bidirectional evidence alignment to improve retrieval performance on CVR and CIR tasks.

  6. OneThinker: All-in-one Reasoning Model for Image and Video

    cs.CV 2025-12 unverdicted novelty 5.0 of 10

    OneThinker unifies image and video reasoning in one model across 10 tasks via a 600k corpus, CoT-annotated SFT, and EMA-GRPO reinforcement learning, reporting strong results on 31 benchmarks plus some cross-task transfer.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages · cited by 6 Pith papers

  1. [1]

    High-Speed Tracking with Kernelized Correlation Filters,

    J. F. Henriques, R. Caseiro, P. Martins and J. Batista, “High-Speed Tracking with Kernelized Correlation Filters,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 37, no. 3, pp. 583-596, 1 Mar. 2015. 6

    work page 2015

  2. [2]

    ECO: Efficient convolution operators for tracking,

    M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “ECO: Efficient convolution operators for tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , Jul. 2017, pp. 6931-6939

    work page 2017

  3. [3]

    SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks,

    B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan, “SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 4277-4286

    work page 2019

  4. [4]

    Transformer Tracking

    X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang and H. Lu, “Transformer Tracking”, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 8122-8131

    work page 2021

  5. [5]

    Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework

    B. Ye, H. Chang, B. Ma, S. Shan and X. Chen, “Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework.”, in Proc. Eur . Conf. Comput. Vis. (ECCV), 2022, pp. 341-357

    work page 2022

  6. [6]

    Language models are few-shot learners

    T. Brown, B. Mann, N. Ryder,et al. “Language models are few-shot learners.” Advances in neural information processing systems, 2020, pp: 1877-1901

    work page 2020

  7. [7]

    Visual instruction tuning

    H. Liu, C. Li, Q. Wu, et al. “Visual instruction tuning.” Advances in neural information processing systems, 2023, pp: 34892-34916

    work page 2023

  8. [8]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    J. Zhu, W. Wang, Z. Chen,et al. “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.” arXiv preprint arXiv:2504.10479, 2025

    arXiv 2025

Show all 19 references
  1. [9]

    Qwen2. 5-vl technical report

    S. Bai, K. Chen, X. Liu,et al. “Qwen2. 5-vl technical report.”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  2. [10]

    Openai o1 system card

    A. Jaech, A. Kalai, A. Lerer, et al. “Openai o1 system card.” arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    D. Guo, D. Yang, H. Zhang, et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.” arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Z. Shao, P. Wang, Q. Zhu,et al. “Deepseekmath: Pushing the limits of mathematical reasoning in open language models.” arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  5. [13]

    Got-10k: A large high-diversity benchmark for generic object tracking in the wild

    L. Huang, X. Zhao, K. Huang, “Got-10k: A large high-diversity benchmark for generic object tracking in the wild”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 5, 2019, pp. 1562-1577

    2019

  6. [14]

    Proximal policy optimization algorithms

    J. Schulman, F. Wolski, P. Dhariwal,et al. “Proximal policy optimization algorithms.” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  7. [15]

    EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework

    Y . Zheng, J. Lu, S. Wanget al. “EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework.” 2025, https://github.com/hiyouga/EasyR1

    2025

  8. [16]

    Dapo: An open-source llm reinforcement learning system at scale

    Q. Yu, Z. Zhang, R. Zhu, et al. “Dapo: An open-source llm reinforcement learning system at scale.” arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  9. [17]

    Generalized intersection over union: A metric and a loss for bounding box regression

    H. Rezatofighi, N. Tsoi, J. Y . Gwak,et al., “Generalized intersection over union: A metric and a loss for bounding box regression”, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 658-666

    2019

  10. [18]

    Efficient memory management for large language model serving with pagedattention

    W. Kwon, Z. Li, S. Zhuang, et al. “Efficient memory management for large language model serving with pagedattention” in Proc. 29th ACM Symp. Operating Syst. Principles . 2023: 611-626

    2023

  11. [19]

    Improved baselines with visual instruction tuning

    H. Liu, C. Li, Y . Li, et al. “Improved baselines with visual instruction tuning” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR). 2024: 26296-26306. 7

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.