{"id":"88c103e6-3565-4af4-a9c1-13047924acb1","arxiv_id":"2501.18487","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A causal, frame-by-frame transformer point tracker with spatial and context memory achieves state-of-the-art online tracking and competitive offline-level accuracy on TAP-Vid and related benchmarks.","lead":"Track-On is an online point tracker that processes video frame by frame, using two memory modules to hold past appearance and location information for each point. It reports state-of-the-art results among online trackers and matches several offline trackers on standard benchmarks, which matters for robotics and augmented reality where future frames are unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The offline-competitive claim rests on per-dataset test-set selection of memory size and visibility threshold via untrained long-context extrapolation; the online-SOTA claim is better supported and survives without IME.","rationale":"The paper is a well-engineered contribution: patch classification plus offset refinement is a clean alternative to iterative regression, and the two memory modules are clearly motivated and ablated (Tables 2-3, Appendix E). The online-SOTA claim is strongly supported, especially on DAVIS, where the no-IME model already beats Online TAPIR by over 8 AJ and also edges CoTracker3 (64.9 vs 64.5). My stress-test confirms the reader's weakest assumption with a sharper observation: the paper's own Appendix D and Figure 12 sweep Ki on the test sets, and the final per-dataset values are read off those curves, with δv also changed to 0.5 for RGB-Stacking without any validation protocol. Since IME linearly interpolates positional embeddings from K=12 to unseen lengths, this is untrained behavior being tuned directly on the benchmark. On DAVIS this does not threaten the primary claim, but on RGB-Stacking IME contributes about 2.9 AJ, and without it the 'comparable or superior to offline' framing likely fails (AJ roughly 68.5 vs CoTracker3's 74.0). The absence of error bars and released checkpoints compounds the uncertainty. I therefore keep the reader's CONDITIONAL verdict: the online-SOTA result is credible, but the seven-dataset offline-competitive framing should be conditional on a described validation split and released checkpoints, or qualified to say 'competitive on most datasets but inferior on RGB-Stacking AJ.' The paper itself overstates in the abstract with 'superior or competitive results compared to offline approaches on seven datasets,' while Table 1 shows a clear AJ deficit on RGB-Stacking (71.4 vs 74.0), so the conditional verdict is the right calibration.","tokens_in":21110,"tokens_out":7649,"duration_ms":81932,"concrete_test":"Define a validation split inside each evaluation set (e.g., DAVIS 20/10, RGB-Stacking 35/15, Kinetics 800/200, RoboTAP 180/85, Dynamic Replica 14/6, BADJA 4/3). Sweep Ki in {12,16,32,48,64,80,96} and δv in {0.5,0.6,0.7,0.8} on the validation split only, then report test AJ/δavg/OA for (a) the training-time default (K=12, δv=0.8), (b) the validation-selected setting, and (c) the paper's reported setting, using the same query points and seeds. Also release the model weights so the check is reproducible. If RGB-Stacking AJ at K=12 falls to about 68.5 as Appendix D implies, the offline-competitive claim on that dataset fails; if the validation-selected margins over CoTracker3 shrink to zero, the seven-dataset claim needs a stated validation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Track-On's headline has two parts: (1) new SOTA among online trackers, and (2) superior or competitive with offline models on seven datasets. Part (1) is solid: on DAVIS, even the no-IME variant (Model-D, Table 3) gives AJ 64.9 vs 56.7 for Online TAPIR, a margin well beyond noise. Part (2) is where the argument is least secure. The model is trained with memory size K=12 and a visibility threshold implicitly 0.8, but at inference the paper sets Ki=48 for DAVIS/RoboTAP/Dynamic Replica, 80 for RGB-Stacking, 96 for Kinetics and PointOdyssey (Sections 3.1 and Appendix C), and δv=0.5 for RGB-Stacking (Appendix B.1), with no validation split described. Appendix D and Figure 12 sweep Ki on the test sets themselves: DAVIS AJ peaks at K=48, RGB-Stacking improves by +2.9 AJ up to K=80, and Section 3.3 says K=48 is where the model 'performs best on this dataset.' The positional embeddings γs and γc are linearly interpolated from K=12 to these unseen lengths, so the long-context behavior is entirely untrained. Because the datasets are small (30, 50, 20, and 7 videos for several of the seven), the 0.5 AJ margin on DAVIS and the RGB-Stacking δavg/OA gains are within plausible selection noise. The 'seven datasets' claim therefore depends on test-set-tuned, untrained hyperparameters; without this tuning, RGB-Stacking loses about 2.9 AJ and the Kinetics/RoboTAP/PointOdyssey numbers have no reported sensitivity. No code or weights are released, so the exact margins cannot be independently checked. This is a correctness-risk concern about the evidence, not a claim of misconduct; the architecture and ablations are otherwise careful and internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Track-On proposes a causal, frame-by-frame transformer for long-term point tracking. Queries are initialized from DINOv2/ViT-Adapter features, refined by a query decoder, and matched to patches via multi-scale cosine similarity with top-k re-ranking; a separate offset head refines the patch center, and visibility/uncertainty heads produce per-point estimates. Temporal information is carried by two FIFO memory modules: spatial memory stores features around recent predictions, and context memory stores decoded queries from the track history. The model is trained on 24-frame TAP-Vid Kubric clips with memory size K=12; at inference, the temporal positional embeddings are linearly interpolated to extend memory up to Ki=96. Experiments report state-of-the-art results among online trackers on TAP-Vid DAVIS/RGB-Stacking/Kinetics and additional comparisons on RoboTAP, Dynamic Replica, BADJA, and PointOdyssey, with the headline claim of matching or surpassing offline models on several benchmarks.","tokens_in":21435,"tokens_out":7635,"duration_ms":72239,"significance":"The paper's strongest and best-supported result is the online-vs-online comparison: on DAVIS, Track-On improves AJ by 8.3 points over Online TAPIR (64.9/65.0 vs 56.7), a margin far larger than plausible run-to-run noise. The design is clean, the ablations isolate the contributions of re-ranking, offset prediction, the visibility head, and the two memory modules, and the memory-efficiency figures (0.73 GB, 16.8 FPS at Ki=48 on an A100) are a useful practical contribution. If the offline-competitive claim were fully supported, the paper would establish an important fact: causal streaming tracking can match the accuracy of bidirectional window/video-level models on standard benchmarks. The current evidence for that claim is weakened by per-dataset test-set selection of memory size and visibility threshold, by reliance on untrained long-context extrapolation, and by the absence of uncertainty estimates on small datasets. These issues are addressable and do not undermine the online SOTA contribution, but they need to be resolved before the broader claim can be accepted.","major_comments":[{"comment":"The inference-time memory size Ki is selected per dataset on the evaluation sets. Section 3.1 sets Ki=48 for DAVIS, RoboTAP, and Dynamic Replica; Ki=80 for RGB-Stacking; Ki=96 for Kinetics and PointOdyssey; and Ki=12 for BADJA. Appendix D and Fig. 12 then sweep Ki directly on the TAP-Vid test sets and report that RGB-Stacking gains +2.9 AJ up to Ki=80 and DAVIS peaks at Ki=48. No validation split is described, so the long-video numbers in Tables 1, 4, and 5 are produced with hyperparameters selected on the evaluation data. This matters because the margins supporting the offline-competitive claim are small: 0.5 AJ over CoTracker3 on DAVIS and 1.2 AJ over LocoTrack on RoboTAP. Please select Ki on a validation split or by a principled length-based rule, report the resulting test numbers, and show how the conclusions change when Ki is fixed to 12 or chosen by a data-independent rule.","section":"Section 3.1, Appendix D, Fig. 12"},{"comment":"The model is trained with memory size K=12, and at inference the temporal positional embeddings gamma_s and gamma_c are linearly interpolated to sizes up to Ki=96. This is an untrained extrapolation, yet it is active in all long-video results that support the offline-competitive claim. The paper does not justify that linear interpolation preserves the memory read/write behavior learned at K=12; for example, the learned relative-position semantics of the embeddings may not transfer to unseen lengths. Please provide evidence that the interpolation is safe, ideally by training with K=16/32/48 or with variable-length memory on Kubric and comparing against the interpolated model on a held-out split, or by analyzing the stability of memory attention under interpolation.","section":"Section 2.3.3, Eqs. (8)-(9)"},{"comment":"The visibility threshold delta_v is set to 0.8 for all datasets except RGB-Stacking, where it is 0.5, justified only as 'due to its domain-specific characteristics.' No validation split is described for this choice. Since AJ and OA depend on the predicted visibility threshold, this is a hyperparameter that can change the reported numbers. Please report AJ/OA as a function of delta_v on a validation split, at least for DAVIS and RGB-Stacking, and state the chosen value before test evaluation.","section":"Appendix B.1"},{"comment":"All reported results are single-run point estimates without error bars, confidence intervals, or multiple seeds. The datasets are small (30 DAVIS videos, 50 RGB-Stacking videos, 20 Dynamic Replica sequences, 7 BADJA sequences, 12 PointOdyssey videos), and the headline offline comparisons are close: 65.0 vs 64.5 AJ on DAVIS, a 0.2 AJ gap on Kinetics, and a 1.2 AJ gap on RoboTAP. Without per-sequence variance, bootstrap intervals, or multiple seeds/checkpoints, it is not possible to assess whether these differences are meaningful. Please add uncertainty estimates for the main comparisons or state the variance across sequences.","section":"Tables 1, 4, 5"}],"minor_comments":[{"comment":"The sentence 'For DA VIS evaluation, we set the memory size Ki to 48, 80, and 96 for DA VIS, RGB-Stacking, and Kinetics' is grammatically unclear; it should say that for evaluation on DAVIS, RGB-Stacking, and Kinetics, Ki is set to 48, 80, and 96, respectively.","section":"Section 3.1"},{"comment":"The phrase 'the former setting grants access to any frame regardless of video length' is confusing because the preceding text does not clearly establish two numbered settings; use 'offline models with video-level access' instead.","section":"Table 1 caption"},{"comment":"The y-axis is labeled 'AJ' but the text describes 'the change in AJ compared to the default training memory size of 12'; the axis should be labeled 'ΔAJ' or the caption should explicitly state the plotted quantity.","section":"Figure 12"},{"comment":"The description of 'random key masking with a 0.1 ratio during attention calculations for memory read operations' is not defined in the method section; please specify which attention computations are masked.","section":"Appendix B.1"},{"comment":"The paragraph says 'we evaluate models using delta_vis, consistent with the TAP-Vid benchmark,' but delta_vis is not introduced in the main text and is not a TAP-Vid metric; please define it and provide a reference.","section":"Appendix C, Dynamic Replica"},{"comment":"No code or trained weights are released. Given that several headline margins are around 0.5 AJ, a reproducibility statement with code/weights would materially help independent verification of the reported numbers.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"I see no grounds for a deeper correctness concern: the online-vs-online comparison is credible, the method is well specified, and the ablations are informative. The main gap is between the broad offline-competitive claim and the evidence provided. If the authors can supply validation-based hyperparameter selection and uncertainty estimates, the paper would be a strong contribution. The lack of code/weights is a notable limitation for a paper with small reported margins, but it is not a blocker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The online part of the headline is the part that holds. Track-On beats Online TAPIR on DAVIS by 8.3 AJ, and the ablation table shows that even without inference-time memory extension it is at 64.9 AJ, so the online state-of-the-art claim does not lean on the fragile parts of the method. The architecture is a genuine new entry for point tracking: it uses patch classification plus offset refinement instead of the regression/iterative updates from TAPIR and CoTracker, and the two memory modules (spatial and context) have distinct, well-ablated roles. The related work is broad and the citation pattern looks fair. Efficiency numbers (16.8 FPS at K=48 on one A100) are useful, and the failure analysis is honest.\n\nThe soft spot is the second half of the abstract: 'superior or competitive results compared to offline approaches on seven datasets.' That is real but not as clean as it sounds. On RGB-Stacking, Track-On's AJ is 71.4 against CoTracker3's 74.0, so it trails the best offline model. On DAVIS the margin over CoTracker3 is 0.5 AJ on a 30-video dataset; without error bars that could easily be noise. The per-dataset memory sizes (Ki=48, 80, 96, etc.) and the visibility threshold (0.8, except 0.5 for RGB-Stacking) are chosen without a described validation split, and Appendix D sweeps Ki on the test sets themselves. The inference-time memory extension interpolates positional embeddings from K=12 to sizes never seen in training, which works in practice but is untrained behavior, so the long-video gains should be treated with some caution. None of this looks like misconduct; the paper is transparent. But the 'seven datasets' claim deserves a qualification and stronger evidence.\n\nWho gets value: anyone working on online/causal point tracking, robotics, or augmented reality. The online-vs-online comparison is the strongest part and stands alone. The offline-competitive framing should be pushed to include error bars, a real validation split, and released code. I'd send it to peer review and ask for those before publication.","headline":"The online-SOTA claim is solid; the offline-competitive claim is real but overstated, resting on test-set-tuned memory sizes and no error bars.","tokens_in":22075,"tokens_out":3505,"would_cite":true,"duration_ms":33813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A causal, frame-by-frame tracker matches offline point-tracking models that see the entire video, and beats them on TAP-Vid DAVIS.","keywords":["online point tracking","long-term point tracking","transformer","memory module","temporal correspondence","TAP-Vid benchmark","causal video processing","patch classification"],"falsifier":"Run the TAP-Vid DAVIS evaluation with one fixed memory size and visibility threshold for all datasets (for example, $K_i=48$ and threshold 0.8) and no per-dataset selection; if the reported AJ 65.0 margin over CoTracker3 shrinks or reverses, the state-of-the-art claim depends on test-set hyperparameter selection. Separately, evaluate on videos longer than 96 frames at $K_i=96$: if accuracy collapses beyond the trained memory length, the untrained interpolation only covers the tested range.","tokens_in":20800,"feed_emoji":"🎯","tokens_out":8722,"duration_ms":66907,"temperature":0.7,"pith_summary":"Track-On is a point-tracking model that processes video one frame at a time, with no access to future frames, and still reaches the accuracy of offline models that consume the whole video or a large window. The paper's central claim is that long-term tracking does not need bidirectional temporal modeling: two memory modules, one storing local patches around recent predictions and one storing the track's history of query embeddings, carry the temporal information. On the TAP-Vid benchmark, the model reports AJ 65.0 and $\\delta_{\\mathrm{avg}}$ 78.0 on DAVIS, ahead of the best offline model CoTracker3 (64.5 and 76.7) trained on comparable data, and it sets a new state of the art among online trackers. The significance, if true, is that streaming applications such as robotics and augmented reality can get offline-level tracking without waiting for future frames.","feed_headline":"Frame-by-frame tracker beats whole-video models on TAP-Vid","feed_subtitle":"Track-On runs causally with two memory modules, hitting AJ 65.0 on DAVIS without ever seeing future frames.","key_machinery":"The two memory modules are the load-bearing mechanism. Spatial memory $M_s$ holds up to $K$ feature vectors extracted from small neighborhoods around the model's predictions in past frames; before the query decoder runs, the initial query is updated by attending to this memory, which the paper shows increases similarity to the true correspondence by 24% on DAVIS, directly countering feature drift. Context memory $M_c$ stores the decoded query embeddings from previous frames and is attended by the query decoder, providing the track's history including occlusion status. Both are FIFO queues of size $K=12$ at training, and at inference the temporal positional embeddings are linearly interpolated to extend the memory to $K_i$ up to 96, which lets a model trained on 24-frame clips track videos of hundreds of frames. The correspondence pipeline of multi-scale patch classification, top-$k$ re-ranking, and offset refinement carries the localization.","core_discovery":"On its own terms, the paper establishes that point tracking can be formulated as a causal matching problem solved by a transformer: points are treated as queries that attend to the current frame, the correspondence is found by classifying which patch contains the point using cosine similarity at multiple scales, a re-ranking step re-scores the top-$k$ candidate patches, and an offset head refines the chosen patch center to sub-patch precision. Temporal continuity comes from two FIFO memories — spatial memory, which stores the features around the model's recent predictions and updates the initial query to counteract feature drift, and context memory, which stores decoded query embeddings from past frames so the decoder can see the track's history, including occlusions. The paper reports that this design sets a new state of the art among online trackers and delivers superior or competitive results against offline approaches on seven datasets including TAP-Vid, with the DAVIS numbers exceeding the offline CoTracker3.","pith_inferences":["The per-dataset memory sizes (48 for DAVIS, 80 for RGB-Stacking, 96 for Kinetics) suggest the authors treat $K_i$ as an inference hyperparameter matched to video length and motion speed; a testable extension would be a rule for choosing $K_i$ from clip statistics rather than per-dataset tuning.","The paper's own failure analysis singles out thin surfaces and uniform regions, both symptoms of stride-4 feature resolution; combining the memory design with learnable upsampling is the natural follow-up the authors flag as future work.","Because the largest gains over offline models appear on textureless robotics scenes (RGB-Stacking, RoboTAP), the approach looks particularly suited to robot perception; whether the same memory settings transfer to egocentric internet video is an open question."],"forward_implications":["Streaming deployment: robotics, augmented reality, and video editing can use a tracker that emits predictions frame by frame with bounded memory and no future frames.","Long-video scalability: GPU memory stays in the 0.61–1.03 GB range as the inference memory grows from 12 to 96, so very long videos can be tracked with roughly constant per-frame cost.","A new paradigm: patch classification plus offset refinement is a viable alternative to the iterative regression used by prior point trackers.","Training and inference length are decoupled: a model trained on 24-frame clips tracks 250-frame videos because the learned memory positional embeddings extrapolate by interpolation."],"supporting_citations":[{"why":"TAP-Vid benchmark; provides the Kubric training set and the evaluation protocol and metrics (AJ, δavg, OA) used in all main results.","marker":"Doersch et al. 2022"},{"why":"TAPIR; the offline baseline whose coarse-to-fine prediction paradigm Track-On adapts, and whose online variant is the closest online competitor on DAVIS.","marker":"Doersch et al. 2023"},{"why":"CoTracker3; the strongest offline competitor, trained on 64-frame videos, that Track-On must beat on DAVIS and match elsewhere.","marker":"Karaev et al. 2024a"},{"why":"DINOv2; the pretrained visual backbone whose features initialize the point queries.","marker":"Oquab et al. 2024"},{"why":"ViT-Adapter; produces the dense stride-4 feature map the query decoder and memories operate on.","marker":"Chen et al. 2022b"},{"why":"Deformable DETR; supplies the deformable attention blocks used in re-ranking, offset prediction, visibility head, and memory writing.","marker":"Zhu et al. 2021"},{"why":"CoTracker; supplies the training augmentations and the global support grid added at inference.","marker":"Karaev et al. 2024b"},{"why":"LocoTrack; region-to-region similarity offline baseline that motivates the correspondence formulation.","marker":"Cho et al. 2024"}],"fun_headline_variants":["Causal point tracker beats offline models on DAVIS","Track-On: Online point tracking with dual memory, SOTA","No future frames needed: Track-On sets online SOTA","Transformer-based track-on: causal, memory, beats CoTracker3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model is trained with a memory of 12 frames and then evaluated with memories of up to 96 frames created by linearly interpolating the temporal positional embeddings — a length extrapolation the model never trained on — and the per-dataset memory sizes and visibility thresholds are chosen without a described validation split.","fun_headline_variants_meta":{"raw":{"variants":["Causal point tracker beats offline models on DAVIS","Track-On: Online point tracking with dual memory, SOTA","No future frames needed: Track-On sets online SOTA","Transformer-based track-on: causal, memory, beats CoTracker3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1547,"prompt_tokens":927,"completion_tokens":620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":550}},"tokens_in":543,"tokens_out":620,"duration_ms":7283,"temperature":1.0,"reasoning_tokens":550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:18:01.422280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the TAP-Vid DAVIS evaluation with one fixed memory size and visibility threshold for all datasets (for example, $K_i=48$ and threshold 0.8) and no per-dataset selection; if the reported AJ 65.0 margin over CoTracker3 shrinks or reverses, the state-of-the-art claim depends on test-set hyperparameter selection. Separately, evaluate on videos longer than 96 frames at $K_i=96$: if accuracy collapses beyond the trained memory length, the untrained interpolation only covers the tested range.","supporting_citations":[{"cited_title":"TAP-Vid : A benchmark for tracking any point in a video","cited_arxiv_id":null,"evidence_quote":"TAP-Vid benchmark; provides the Kubric training set and the evaluation protocol and metrics (AJ, δavg, OA) used in all main results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DINOv2; the pretrained visual backbone whose features initialize the point queries."},{"cited_title":"Local all-pair correspondence for point tracking","cited_arxiv_id":null,"evidence_quote":"LocoTrack; region-to-region similarity offline baseline that motivates the correspondence formulation."}],"review_version":1}