{"id":"68a261bf-1cbe-4477-a2a0-0a2915e17f77","arxiv_id":"2507.20177","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":14,"one_line_summary":"A video-level tracker with propagated temporal tokens and gated cross-modal fusion reports state-of-the-art results on RGB, RGB-T, RGB-D, and RGB-E benchmarks after one joint training run.","lead":"This paper presents a single object tracker that handles ordinary video, thermal video, depth video, and event-camera video with one set of network weights. It reports top results on twelve benchmarks and introduces gated modules for cross-modal fusion alongside frame-to-frame token propagation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Eq. 11 temporal-token recurrence is validated only on training clip lengths up to 5, while inference propagates the token across full videos; the paper's own Table XVI shows performance degrading beyond length 3, leaving token drift over inference-time frame counts as the central untested risk.","rationale":"The reader identified the same load-bearing assumption: the temporal token propagation T_{t+1}=T_t+T_empty must preserve target information over long inference horizons, yet it is only validated through short clip-length ablations. I agree with that identification. The paper's own Tables VIII and XVI show that increasing the search clip length from 3 to 4 or 5 degrades performance, which is direct evidence that the token mechanism is sensitive to temporal accumulation. Since inference propagates the token over far longer spans than any training clip, the central claim that this mechanism provides robust online video-level association is not yet fully supported. This is not a fatal flaw; the benchmark results on long-term datasets such as LaSOT and DepthTrack are encouraging, and a reset ablation or temporal-quartile analysis would settle the question. The reader's CONDITIONAL verdict remains appropriate: the empirical claim is plausible but the token-drift risk needs a targeted check before full acceptance. I therefore recommend no change to the reader's verdict.","tokens_in":30925,"tokens_out":4471,"duration_ms":53061,"concrete_test":"Run a token-reset ablation at inference on LaSOT and DepthTrack: track each sequence with the current auto-regressive token propagation (Algorithm 1) and with the temporal token reset to zero every N frames for N=50, 100, and 200, keeping all other parameters identical. Additionally report success/F-score stratified by temporal quartile of each sequence. If resetting the token every N frames maintains or improves accuracy, or if late-quartile performance drops relative to early-quartile performance, then the propagated token is not delivering the claimed long-range benefit and Eq. 11's online update mechanism is not substantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The novelty of UM-ODTrack rests on the claim that online dense temporal token learning can replace explicit online model updates while propagating appearance and trajectory information across a video stream (Section III.F, Eqs. 11-12; Algorithm 1). The ablation evidence for this mechanism, however, is confined to training clip length: Tables VIII and XVI vary the number of search frames from 2 to 5 and show that performance peaks at length 3 and declines thereafter, which the paper attributes to 'accumulation of cross-modal temporal noise' (Section IV.C). But at inference the temporal token is propagated frame-by-frame over the entire test sequence (Algorithm 1), which can be hundreds of frames long. The paper never measures whether the token drifts, saturates, or corrupts over that much longer horizon. If the token accumulates harmful noise, the video-level gains over image-pair trackers would erode exactly in long-term sequences, which are central to the LaSOT and DepthTrack claims. The Limitation section also concedes that the video-level modeling is a 'global approximation', making the recurrence's long-horizon behavior the load-bearing assumption that is least supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"UM-ODTrack is a universal video-level tracker that extends the authors' prior ODTrack. It replaces image-pair sampling with video-clip sampling, propagates a learned temporal token across frames in an auto-regressive manner (Eq. 11), and introduces a conditional gate and a gated modal-scalable perceiver to fuse RGB with thermal, depth, or event data. A single model is trained jointly on LasHeR, DepthTrack, and VisEvent and then evaluated on five multi-modal benchmarks; a separate ODTrack model is evaluated on seven RGB benchmarks. The paper claims state-of-the-art results on both groups.","tokens_in":31215,"tokens_out":9871,"duration_ms":99839,"significance":"The practical significance is potentially high: if one checkpoint can serve RGB, RGB-T, RGB-D, and RGB-E tracking, it would remove the need for per-modality expert models and repeated training. The paper provides extensive comparisons and ablations (Tables I-XVI), and the controlled ablations for gated perceivers, gate layers, activation functions, and modality weights are informative. The central limitations are that the universal model is not actually evaluated on RGB benchmarks, and the long-horizon behavior of the temporal token is not validated.","major_comments":[{"comment":"Equation (11) defines T_{t+1}=T_t+T_empty but never specifies what T_empty is; if T_empty is a zero token, then no information is added to the propagated token, while if it is a nonzero learned vector, the token grows approximately linearly with frame index. Tables VIII and XVI show that the model's own performance already declines when the training clip length exceeds 3, which the text attributes to 'accumulation of cross-modal temporal noise' (Section IV.C). Since Algorithm 1 propagates the token across the entire test sequence (often hundreds of frames) rather than the training clip length, the paper's central claim that online token propagation replaces explicit online updates requires direct evidence about token drift over inference-time horizons. I request (i) a precise definition and initialization of T_empty, (ii) a plot or statistic of token norm/change versus frame index on long sequences, and (iii) a long-sequence performance comparison between propagating the token and resetting it periodically.","section":"Section III.F, Eq. (11); Algorithm 1; Tables VIII and XVI"},{"comment":"The abstract and conclusion state that UM-ODTrack achieves state-of-the-art results on seven visible and five multi-modal benchmarks, but all visible-benchmark tables (I, II, and III) report ODTrack, the RGB-only model, and no table reports RGB results for the jointly trained UM-ODTrack model. The paper therefore does not support the claim that the same trained parameters span RGB and multi-modal tracking. Either add RGB experiments with the universal model or revise the abstract and conclusion to state that UM-ODTrack is evaluated only on multi-modal benchmarks and that ODTrack serves as the RGB instantiation.","section":"Section IV.B; Tables I-III; Abstract and Conclusion"},{"comment":"Algorithm 1 is not executable as written: T and T' are initialized to None, and the first frame directly uses them in Eq. (11), which adds T_t to T_empty; 'if not memory is None' is always true for a list; and the update of T in the multi-modal branch is not clearly separated from the RGB branch. In addition, the paper does not specify the event representation (frames versus event stream), the kernel/stride of the shared modality tokenizer, or the architecture and initialization of T_empty and the gate networks. The 'code will be available' statement is not a substitute at review time; please provide the code/checkpoints or at least complete pseudocode and hyperparameter details so the central SOTA numbers can be independently reproduced.","section":"Algorithm 1 and Section III.D-G"},{"comment":"The comparison between single-task training (#4) and one-shot joint training (#5) in Table XIV confounds the training scheme with data quantity: the joint model is trained on three datasets, whereas each single-task expert is trained on one. The reported gains (e.g., 67.8 to 69.3 F-score on DepthTrack) may simply reflect three times more training data. Please add an equal-data control, e.g., a joint-trained model subsampled so each task sees the same number of samples as the single-task expert, or report per-task data volume and iterations.","section":"Table XIV, rows 4 and 5"}],"minor_comments":[{"comment":"Section III.B contains the typo 'mdoal', and the Abstract contains 'an near-global'; a copyedit pass is needed.","section":"Section III.B and Abstract"},{"comment":"Equation (5) uses the symbol bf for the gated output, which is easily confused with a bold vector notation; please choose a clearer symbol.","section":"Section III.E, Eq. (5)"},{"comment":"The GOT10K evaluation protocol for ODTrack is ambiguous because Section III.G describes training on LaSOT, GOT-10k, TrackingNet, and COCO, while Section IV.B says the GOT10K protocol is followed; please clarify which checkpoint was used for the GOT10K row.","section":"Table I"},{"comment":"The term 'one-shot training' should be defined explicitly, since the multi-modal stage is preceded by a 300-epoch RGB training stage.","section":"Section IV.A"},{"comment":"Fig. 5 lists 'Backgroudclutter'; this should be 'Background clutter'.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extension of the authors' own AAAI 2024 ODTrack paper, and the genuinely new elements are the gated perceivers and the joint multi-modal training. The main risk is overclaiming universality: the RGB-benchmark evidence comes from ODTrack, not from the jointly trained UM-ODTrack. I would support a major revision, not rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering extension of the authors' ODTrack to RGB-T/D/E tracking, and the most interesting result is that one-shot joint training beats separate per-modality training. But the \"first universal video-level modal tracking\" claim does not survive contact with Un-Track, which they cite, and the temporal token recurrence is the load-bearing component yet is validated only at training clip lengths up to 5 while inference propagates it over full sequences. That gap is the softest spot in the paper.\n\nWhat is actually new: combining their temporal token propagation with two gated perceivers (conditional gate and GMP) and joint training across thermal, depth, and event data in a single model. Table XIV's comparison of joint one-shot training against single-task training is the kind of ablation that matters: joint training improves each task, which is a real argument for the unified approach if the numbers hold up. The ablation work is otherwise thorough—token association, clip length, sampling range, gating layers, activation functions, modality weights—and the benchmark coverage is broad.\n\nWhere it is soft. First, novelty: the paper describes Un-Track as \"the first attempt to learn representations of multiple modalities... unify them into the same set of model parameters,\" then later claims \"truly universal modal tracking for the first time.\" That is an overclaim; the genuinely new piece is the video-level temporal token plus gated perceivers, not universality per se. Second, no code or checkpoints are released and the GitHub link points to the ODTrack repo, so the SOTA numbers are unverified. Third, the temporal token concern: Table XVI shows multi-modal performance peaks at search-clip length 3 and declines at 4-5, and the text attributes this to \"accumulation of cross-modal temporal noise.\" At inference, though, Algorithm 1 propagates the token frame-by-frame over the entire video, which can be hundreds of frames. The training evidence does not cover that horizon, and the paper's own explanation of the decline names the failure mode. This is worth a direct test—e.g., plotting performance against sequence length or measuring token drift—before the video-level gains are treated as robust. Minor: some RGB margins are thin (0.6 AUC on LaSOT over ARTrack) with no error bars, and the one-shot vs single-task comparison is partly a data-volume effect, since single-task training sees less data.\n\nWho this is for: tracking researchers, especially people working on multi-modal or unified trackers. It deserves a serious referee: the architecture is clearly described, the ablations are honest, and the one-shot training result is a useful datapoint even if the token-drift question remains open. My recommendation: send it to peer review, ask for code release, a rewording of the novelty claim, and some evaluation of long-sequence token behavior.","headline":"A competent and useful ODTrack-to-multimodal extension whose one-shot training result is the real prize, but it overclaims novelty against Un-Track and never tests its temporal token at inference-time horizons.","tokens_in":31799,"tokens_out":3485,"would_cite":true,"duration_ms":35226,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One video-level tracker, trained once, sets state-of-the-art results across RGB, thermal, depth, and event tracking.","keywords":["visual object tracking","multi-modal tracking","RGB-T tracking","RGB-D tracking","RGB-E tracking","temporal tokens","gated attention","one-shot training"],"falsifier":"Run the tracker on a long sequence with full occlusion and periodically reset the temporal token to a fresh empty vector, then compare success rates: if resetting the token produces no loss, the propagation rule is not what carries the performance; alternatively, if the token's nearest-neighbor feature drifts away from the target's appearance over time, the accumulation assumption fails.","tokens_in":30702,"feed_emoji":"🎯","tokens_out":5735,"duration_ms":55397,"temperature":0.7,"pith_summary":"The paper claims that a single video-level tracker, trained once on RGB, thermal, depth, and event data jointly, surpasses separate expert trackers that are trained per modality on their own benchmarks. It identifies image-pair sampling and per-task training as the main bottlenecks, and replaces them with video-clip sampling plus a temporal token that carries target appearance and trajectory cues from frame to frame. Two gated attention modules let one set of parameters fuse any auxiliary modality, so the same checkpoint runs RGB, RGB-T, RGB-D, and RGB-E tracking without fine-tuning. If the claim holds, the field can stop training one expert per sensor type and instead share one model across all of them.","feed_headline":"One checkpoint leads RGB, thermal, depth, and event tracking","feed_subtitle":"A propagating temporal token carries target memory frame-to-frame and replaces online model updates.","key_machinery":"The load-bearing object is the temporal token sequence $T_t$, a learned vector per frame that stores the target's appearance and trajectory information. The propagation rule $T_{t+1}=T_t+T_{empty}$ (Eq. 11) means the token is never re-initialized; it accumulates across frames and serves as a prompt in the next frame's attention, replacing online model update strategies. Two attention variants, concatenated and separated token attention, let the token interact with reference frames and the current search frame, and the gated modal-scalable perceiver with the conditional gate fuses RGB with thermal, depth, or event tokens. The token is what carries video-level information through the otherwise 2D transformer.","core_discovery":"UM-ODTrack reformulates visual tracking as an auto-regressive token sequence propagation task. Each frame contributes a temporal token that compresses the target's appearance, location, and motion trajectory; the token from frame $t$ is added to an empty token for frame $t+1$, so past information guides future inference without explicit online parameter updates. A universal modal encoder with a shared tokenizer processes RGB plus one auxiliary modality, and two gated perceivers, a conditional gate inserted between encoder layers and a gated modal-scalable perceiver after the encoder, adaptively weight the modalities and compress them into one shared feature space. One-shot joint training on LasHeR, DepthTrack, and VisEvent yields a single model that the paper reports as state-of-the-art on LasHeR, RGBT234, DepthTrack, VOT-RGBD2022, and VisEvent, while also improving the underlying RGB tracker on LaSOT, TrackingNet, GOT10K, TNL2K, OTB100, LaSOT_ext, and VOT2020.","pith_inferences":["A natural stress test is to measure token drift directly: if $T_t$ is a trajectory memory, its content should track the target's embedding over time; the paper's ablations only measure end-to-end accuracy, so a token-replacement experiment could separate the memory effect from the extra compute of video-level sampling.","The one-shot training result suggests that cross-modal datasets can be pooled even when their label distributions differ; a testable extension is to add a fourth modality dataset and see whether gains transfer to the RGB baseline without touching the architecture.","Because the paper reports equal weighting across depth, infrared, and event as robust, a practical route is to treat modality weights as learnable per input frame rather than fixed per dataset, letting the gate decide reliability frame-by-frame during occlusion or sensor failure."],"forward_implications":["A single UM-ODTrack checkpoint can replace separate RGB-T, RGB-D, and RGB-E expert trackers; deploying on a new sensor type requires no retraining.","Joint one-shot training improves each multi-modal task over the same model trained only on its own data, so data diversity across modalities, not task-specific architecture, is doing part of the work.","Longer video-clip inputs help only up to a clip length of three; beyond that, performance plateaus or drops, indicating that the token's benefit is bounded by accumulated temporal noise.","The token propagation mechanism transfers directly from RGB tracking to multi-modal tracking without changing the propagation rule, so a new auxiliary modality can likely be added by training a new tokenizer while reusing the gated perceivers."],"supporting_citations":[{"why":"Conference version that introduced ODTrack and the temporal token mechanism this paper extends to multiple modalities.","marker":"[87]"},{"why":"OSTrack one-stream transformer tracker, the architectural baseline and a principal RGB benchmark competitor.","marker":"[43]"},{"why":"Prior single-model any-modality tracker whose unified-parameter claim this paper directly compares against and improves.","marker":"[78]"},{"why":"ViPT prompt-based multi-modal tracker, the main baseline for prompt tuning across RGB-T/D/E tasks.","marker":"[62]"},{"why":"DepthTrack RGB-D dataset used in one-shot joint training and as a benchmark.","marker":"[74]"},{"why":"LasHeR RGB-T dataset used in joint training and as the main thermal benchmark.","marker":"[71]"},{"why":"VisEvent RGB-E dataset used in joint training and as the main event benchmark.","marker":"[73]"},{"why":"TBSI, a previous state-of-the-art RGB-T tracker, exceeded on LasHeR and RGBT234.","marker":"[64]"},{"why":"SeqTrack sequence-to-sequence RGB tracker, compared on RGB benchmarks to isolate the contribution of temporal token association.","marker":"[52]"}],"fun_headline_variants":["One token streams target memory across frames","Universal tracker: single model, four modalities","Temporal tokens replace online model updates","Auto-regressive tokens propagate tracking context","One-shot training unifies RGB, thermal, depth, event"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding the previous frame's temporal token to the next frame's empty token, $T_{t+1}=T_t+T_{empty}$, preserves target appearance and trajectory well enough across many frames that the tracker never needs an explicit online update, and the paper tests this only through end-to-end benchmark ablations.","fun_headline_variants_meta":{"raw":{"variants":["One token streams target memory across frames","Universal tracker: single model, four modalities","Temporal tokens replace online model updates","Auto-regressive tokens propagate tracking context","One-shot training unifies RGB, thermal, depth, event"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1600,"prompt_tokens":1030,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":646,"tokens_out":570,"duration_ms":5306,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:48:05.246219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the tracker on a long sequence with full occlusion and periodically reset the temporal token to a fresh empty vector, then compare success rates: if resetting the token produces no loss, the propagation rule is not what carries the performance; alternatively, if the token's nearest-neighbor feature drifts away from the target's appearance over time, the accumulation assumption fails.","supporting_citations":[{"cited_title":"Multi- adapter rgbt tracking,","cited_arxiv_id":null,"evidence_quote":"Conference version that introduced ODTrack and the temporal token mechanism this paper extends to multiple modalities."},{"cited_title":"Mixformer: End-to-end track- ing with iterative mixed attention,","cited_arxiv_id":null,"evidence_quote":"OSTrack one-stream transformer tracker, the architectural baseline and a principal RGB benchmark competitor."},{"cited_title":"Duality-gated mutual condition network for rgbt tracking,","cited_arxiv_id":null,"evidence_quote":"Prior single-model any-modality tracker whose unified-parameter claim this paper directly compares against and improves."},{"cited_title":"Hand posture recognition using finger geometric feature,","cited_arxiv_id":null,"evidence_quote":"ViPT prompt-based multi-modal tracker, the main baseline for prompt tuning across RGB-T/D/E tasks."},{"cited_title":"Object tracking by jointly exploiting frame and event domain,","cited_arxiv_id":null,"evidence_quote":"LasHeR RGB-T dataset used in joint training and as the main thermal benchmark."},{"cited_title":"Rgb-t object tracking: Benchmark and baseline,","cited_arxiv_id":null,"evidence_quote":"VisEvent RGB-E dataset used in joint training and as the main event benchmark."},{"cited_title":"Multi-modal fusion for end-to-end rgb-t tracking,","cited_arxiv_id":null,"evidence_quote":"TBSI, a previous state-of-the-art RGB-T tracker, exceeded on LasHeR and RGBT234."},{"cited_title":"Aiatrack: Attention in attention for transformer visual tracking,","cited_arxiv_id":null,"evidence_quote":"SeqTrack sequence-to-sequence RGB tracker, compared on RGB benchmarks to isolate the contribution of temporal token association."}],"review_version":1}