Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

The paper claims that video diffusion transformers, trained only to generate videos, inherently contain strong point-tracking ability, and that a tracker built on them outperforms large-scale models while training on far less real data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:19 UTC pith:XRNC3X3Y

load-bearing objection DiTracker is a well-run empirical paper, but the zero-shot claim that video DiTs 'inherently' track points is undercut by selecting the layer/head on the evaluation benchmark itself. the 3 major comments →

arxiv 2512.20606 v2 pith:XRNC3X3Y submitted 2025-12-23 cs.CV

Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking

Soowon Son , Honggyu An , Jisu Nam , Hyunah Ko , Chaehyun Kim , Dahyun Chung , Siyoon Jin , Jung Yi
show 2 more authors
Junhwa Hur Seungryong Kim
This is my paper
classification cs.CV
keywords point trackingvideo diffusion transformerstemporal correspondencequery-key attentionLoRA adaptationcost fusionzero-shot evaluationITTO benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that video diffusion transformers, pretrained on large-scale real-world videos with full 3D spatio-temporal attention, already encode strong temporal correspondence and can serve as a superior feature backbone for point tracking. A zero-shot comparison shows their query-key attention produces more accurate initial matches than a ResNet backbone explicitly supervised on tracking data, especially under motion blur, large displacements, and frequent occlusions. Building on this, the paper introduces DiTracker, which extracts query-key features from a video DiT, computes softmax-based local 4D matching costs, adapts the backbone with LoRA, and fuses those costs with a ResNet branch at the cost level. DiTracker, trained only on synthetic Kubric data and with a fraction of the batch size and iterations, achieves state-of-the-art results on the challenging ITTO-MOSE benchmark and matches or exceeds existing trackers on TAP-Vid and corruption robustness tests. The implication is that generative video pretraining provides real-world motion priors that dramatically reduce the need for large-scale real-data supervision in point tracking.

Core claim

The central claim is that video diffusion transformers, through their full spatio-temporal attention and large-scale video pretraining, inherently exhibit a strong point tracking capability: the internal query-key attention, when used as a matching cost, outperforms a ResNet backbone explicitly trained for point tracking, even without any tracking-specific supervision. DiTracker operationalizes this by extracting query and key features from a single attention head of a frozen video DiT, constructing a hierarchical local 4D matching cost with the same softmax formulation, applying lightweight LoRA adaptation to shift the generative features toward tracking, and fusing DiT costs with ResNet co

What carries the argument

The central mechanism is the query-key attention matching cost from a video DiT's self-attention: C = Softmax(Q_i K_j^T / sqrt(d_head)) between frames i and j, which acts as a probabilistic correspondence map. The paper shows that a single attention head at an intermediate layer, selected for best zero-shot accuracy, captures temporally coherent matching. DiTracker preserves this internal matching mechanism by using the same softmax cost formulation, adding LoRA adaptation (low-rank weight updates) to shift the generative feature distribution toward tracking, and fusing DiT costs with ResNet costs at the cost level so the tracking head can learn complementary strengths: DiT for robust global

Load-bearing premise

The zero-shot analysis assumes that the DiT's correspondence ability is best measured by choosing the layer and attention head that perform best on the same benchmark used to report final numbers, so if that choice must be tuned per benchmark, the claimed superiority over supervised ResNet features may be overstated.

What would settle it

Fix the layer and head a priori without looking at TAP-Vid or ITTO results (e.g., always use layer 18, head 3) and evaluate DiTracker on a fresh benchmark not used in any selection; if the margin over a ResNet-based tracker shrinks or disappears, the inherent-superiority claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Video diffusion pretraining can serve as a strong backbone for point tracking, reducing dependence on large-scale real-world tracking supervision and enabling effective training on synthetic data alone.
  • The proposed tracker converges about 10 times faster than prior state-of-the-art methods, substantially lowering training cost and hardware requirements.
  • The zero-shot evaluation protocol provides a way to assess foundation-model features for correspondence tasks before any task-specific training.
  • Cost-level fusion of a generative model's matching cost with a classical backbone's cost is a general principle that could extend to other geometric tasks like optical flow or depth estimation.
  • Tracking performance scales with DiT backbone size, so larger generative video models are likely to yield further improvements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If video DiT features are this strong zero-shot, end-to-end finetuning of the full DiT (rather than only LoRA) might unlock even larger gains, though it would require more compute and risk forgetting the pretrained motion priors.
  • The layer/head selection procedure, which picks the best head on the evaluation benchmark itself, could introduce optimistic bias; a head fixed without peeking at test results would provide a cleaner measure of inherent capability.
  • The per-frame VAE encoding used here avoids temporal compression but strips the temporal context that 3D attention might exploit; a temporally-aware encoding could be essential for even longer-range or higher-frame-rate tracking.
  • The finding suggests video generative models could be repurposed for other dense correspondence tasks, such as optical flow or temporal depth, with similar efficiency gains over supervised backbones.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DiTracker, a point-tracking method that repurposes video diffusion transformer (DiT) features as the backbone. The authors first conduct a zero-shot analysis of several video DiTs (HunyuanVideo, CogVideoX-2B/5B, Wan-14B) and other visual foundation models on TAP-Vid-DA VIS and ITTO-MOSE, reporting that video DiT features outperform the supervised ResNet from CoTracker3 and other VFMs. They then design DiTracker by extracting query-key attention costs from a LoRA-adapted video DiT, fusing these costs with a ResNet branch, and feeding the fused cost into an existing iterative tracking head. DiTracker is trained solely on Kubric (36k steps, batch size 4) and is reported to achieve state-of-the-art results on ITTO-MOSE while being competitive on TAP-Vid benchmarks and corruption robustness tests. The paper argues that large-scale real-world video pretraining and full 3D spatio-temporal attention provide inherent temporal-correspondence priors that reduce the need for large-scale real-data supervision.

Significance. If the central claim holds—that video DiT features, without tracking-specific supervision, provide strong and robust matching priors—the paper would offer a concrete recipe for leveraging generative video models in dense correspondence tasks, potentially reducing the data and compute required to train point trackers. The paper is the first to systematically compare multiple video DiTs against supervised ResNet backbones and other VFMs for point tracking, and the proposed framework is simple and well motivated. The experiments include a useful ablation of LoRA and cost fusion, and the authors disclose the inference-time cost of using a large backbone. However, the significance is tempered by the fact that the zero-shot layer/head selection uses the evaluation benchmarks, and the causal attribution to 3D attention is not isolated. The paper also overstates its TAP-Vid results relative to its own Table 3.

major comments (3)
  1. [Sec. 4.1, Appendix B.1] The zero-shot comparison selects the DiT layer and attention head on the evaluation benchmarks themselves. Sec. 4.1 states: 'we select the optimal layer l and head m by analyzing zero-shot point tracking performance across all layers and heads,' and Appendix B.1 reports 'Figure 4 shows that the 3rd head achieves the highest accuracy.' This is a test-set hyperparameter selection. The margins in Tables 1 and 2 (e.g., WAN-14B 51.2% vs. ResNet 43.7%) could be inflated by this selection, and the same selected head/layer is later used for the DiTracker pipeline, so the final results also inherit this selection. The authors should either perform layer/head selection on a held-out validation set (e.g., a split of TAP-Vid-Kinetics) or report the full distribution over layers and heads, including the mean/median and the range, so that the reader can assess whether the 'inherent' advantage is limit
  2. [Sec. 4.3, Sec. 7] The paper's causal conclusion—that the performance comes from large-scale real-video pretraining and full 3D spatio-temporal attention—is not supported by the evidence presented. The comparison changes multiple factors simultaneously: architecture family (CNN/transformer), pretraining data, pretraining objective, and the per-frame VAE encoding procedure. Per-frame encoding deliberately changes the temporal stride of the latent sequence relative to the model's training distribution, so the observed performance could depend on this encoding rather than on the 3D attention itself. To support the attribution, the authors would need controlled ablations, such as (a) the same DiT architecture with and without video pretraining, (b) a 3D-attention DiT versus a frame-independent version, or (c) joint VAE encoding. As written, the 'inherently exhibit' claim is not isolated; the authors should eit
  3. [Abstract, Sec. 6.3, Table 3] The abstract claims DiTracker 'matches or outperforms state-of-the-art models on TAP-Vid benchmarks,' but the paper's own Table 3 contradicts this. On TAP-Vid-DA VIS, DiTracker's AJ is 62.7, below LocoTrack (64.8) and CoTracker3† (64.8). On TAP-Vid-Kinetics, DiTracker's AJ is 54.3, below BootsTAPIR (54.6) and CoTracker3† (54.7), and its δx_avg is also lower than both. DiTracker does outperform on ITTO-MOSE AJ and δx_avg (43.9/57.9 vs. 42.4/55.8 for CoTracker3†), but even there the OA is lower (79.3 vs. 80.4). The result summary should be revised to state that DiTracker is state-of-the-art on ITTO-MOSE and competitive with, but not generally superior to, prior methods on TAP-Vid.
minor comments (5)
  1. [General] No variance or multiple-seed results are reported. Since some claimed improvements are small (e.g., Table 5: DiTracker 65.0 vs. CoTracker3† 64.7 average corruption), a single training run may not be statistically distinguishable. Reporting at least 2–3 seeds or a variance estimate would strengthen the claims.
  2. [Table 5] The text says DiTracker 'consistently outperforms across most of severe corruption types,' but in several categories CoTracker3† is better (e.g., defocus, pixelate, JPEG); the claim should be softened to 'achieves the best average and outperforms on several key categories.'
  3. [Sec. 4.1] The zero-shot extraction uses argmax matching without refinement, which is a reasonable protocol, but the paper should make clearer that this measures a combination of feature quality and the hand-crafted matching cost, not a fully trained tracker. This is especially relevant when comparing against the supervised ResNet from CoTracker3, which was trained with a different cost module.
  4. [Appendix B.1, Figure 4] Figure 4 is referenced but not fully explained: the x-axis appears to enumerate attention heads, and the y-axis is omitted. Please label the axes and state the benchmark used for this head analysis.
  5. [Sec. 6.1] The caption of Figure 1 claims '10× Faster' convergence, but the figure's axis labels are small and the meaning of the 'Robust' annotation is unclear. Please provide a clearer definition of the convergence criterion and the reported 'Robust' metric.

Circularity Check

1 steps flagged

Zero-shot 'inherent DiT superiority' is partly constructed by selecting the layer/head on the evaluation benchmark; DiTracker's trained results remain externally tested.

specific steps
  1. fitted input called prediction [Sec. 4.1 and Appendix B.1; Tables 1-2]
    "Here, we select the optimal layer l and head m by analyzing zero-shot point tracking performance across all layers and heads (Appendix B.1). ... To select the optimal attention head, we analyze zero-shot point tracking performance across all attention heads in this layer. Figure 4 shows that the 3rd head achieves the highest accuracy."

    The zero-shot numbers used to conclude that video DiTs 'inherently exhibit strong point tracking capability' are produced by the layer/head (l, m) that the paper chooses by maximizing zero-shot point tracking performance on the same benchmark. Thus the reported δx_avg is a best-of-heads result selected using the evaluation labels, not an independent evaluation of a fixed DiT feature; the ResNet baseline is a single fixed backbone. The same selected head is reused in DiTracker (Sec. 5.1: 'apply LoRA ... up to and including layer l, and then extract ... as described in Sec. 4.1'), so evaluation-set selection leaks into the method's hyperparameters. The conclusion is therefore partially forced by the selection rule rather than by an inherent property.

full rationale

The paper's trained DiTracker is evaluated on external benchmarks (TAP-Vid, ITTO-MOSE, corruptions) and is not circular: the tracking head is trained on Kubric and tested on held-out benchmarks, and the reported gains are genuine empirical results. No self-citation chain or uniqueness theorem is load-bearing; [44] is used as a methodological prior and the analysis is repeated here. However, the central zero-shot analysis claim — that video DiT features inherently surpass supervised ResNet features — is weakened by an evaluation-set selection step: the optimal layer and attention head are chosen by measuring zero-shot accuracy on the same benchmark that is later used to demonstrate the 'inherent' superiority. Comparing the best of many heads against a single fixed ResNet baseline overstates the inherent property and makes part of the reported margin a selection artifact. This is a data-selection / partial circularity issue, not a full reduction by construction, so the score is moderate rather than high.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central results rest on several untested assumptions about why DiT features are good, plus a handful of hand-chosen hyperparameters. There are no invented physical or architectural entities: the paper repurposes existing video DiTs and ResNet backbones.

free parameters (6)
  • DiT feature layer/head for matching (CogVideoX-2B) = layer 18, head 3
    Chosen by zero-shot accuracy on the same benchmark family used for evaluation; the reported advantage depends on this selection (Sec. 4.1, Appendix B.1).
  • LoRA rank = 128
    Hand-chosen; controls how much of the DiT is adapted and is central to the method (Sec. 6.1).
  • Local correlation radius = delta = 3
    Determines neighborhood size for the local 4D cost; set by hand (Sec. 6.1).
  • Feature pyramid scales = S = 4
    Hand-chosen number of scales in the cost pyramid (Sec. 6.1).
  • Loss weighting = occluded weight 1/5; gamma=0.8; confidence threshold 12px; Huber threshold 6px
    Hand-chosen supervision weights that shape the trained tracker (Sec. 5.5).
  • Training input resolution = 480x720
    Chosen due to CogVideoX-2B zero-shot performance; may affect results (Sec. 6.1).
axioms (5)
  • domain assumption Video DiT query-key attention at a single layer/head, computed on per-frame VAE latents at the final denoising timestep, is a valid indicator of point-tracking capability.
    The entire zero-shot analysis in Sec. 4 rests on this protocol, inherited from DiffTrack [44].
  • domain assumption The superiority of DiT features is caused by large-scale video pretraining, full 3D spatio-temporal attention, and the diffusion objective.
    Stated as a hypothesis (Sec. 4.3) but never tested by controlled variation; V-JEPA2, also video-pretrained, underperforms, so the three factors are not independently validated.
  • domain assumption Synthetic Kubric MOVi-F training transfers sufficiently to real-world benchmarks to support the SOTA comparisons.
    DiTracker is trained only on Kubric; the claim of beating CoTracker3 assumes this sim-to-real transfer holds for DiT features (Sec. 6.1-6.3).
  • domain assumption TAP-Vid and ITTO benchmarks adequately measure real-world robustness.
    The paper uses these as ground truth for robustness; ITTO is a newer benchmark, and only the MOSE subset is used (Sec. 6.2).
  • domain assumption The CoTracker3 tracking head can be combined with arbitrary cost embeddings without architectural change.
    DiTracker reuses the tracking head from [26] and assumes it works with fused DiT+ResNet costs (Sec. 5.4).

pith-pipeline@v1.3.0-alltime-deepseek · 17622 in / 14453 out tokens · 135081 ms · 2026-08-03T14:19:39.438893+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking." pith.science (2026). https://pith.science/paper/XRNC3X3Y

@misc{pith2026251220606,
  author       = {Pith},
  title        = {Pith review of: Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRNC3X3Y}},
  note         = {Machine review of arXiv:2512.20606}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Despite achieving strong results on standard benchmarks, current point tracking methods rely on feature backbones that are rarely designed with the temporal coherence needed for robust real-world performance. While recent works incorporate powerful visual foundation model (VFM) features into tracking pipelines, no prior work has systematically analyzed which VFM provides the most robust representations for point tracking. We present the first such analysis, evaluating diverse VFMs in a zero-shot setting on both standard and robustness benchmarks for point tracking. Our study reveals that video diffusion transformers (DiTs) consistently yield the most temporally coherent and discriminative features, even surpassing ResNet backbones explicitly supervised on tracking data. We hypothesize this advantage stem from large-scale video pretraining, full 3D spatio-temporal attention, and a diffusion training objective. Motivated by this finding, we propose DiTracker, which integrates video DiT features into existing tracking frameworks through query-key matching cost computation, cost-level fusion with a lightweight ResNet branch, and LoRA adaptation. Under the same tracking head, DiTracker is trained solely on synthetic data with far fewer iterations, yet outperforms CoTracker3 trained with additional real-world videos, with the largest gains under challenging and corrupted scenarios. It further generalizes across tracking heads and scales with backbone size, confirming that generative video pretraining provides real-world priors that reduce the dependence on large-scale real-data supervision.

Figures

Figures reproduced from arXiv: 2512.20606 by Chaehyun Kim, Dahyun Chung, Honggyu An, Hyunah Ko, Jisu Nam, Jung Yi, Junhwa Hur, Seungryong Kim, Siyoon Jin, Soowon Son.

Figure 1
Figure 1. Figure 1: Enhanced robustness and training efficiency of our DiT-based point tracking framework. DiTracker leverages pre-trained video Diffusion Transformer (DiT) features to outperform state-of-the-art methods such as CoTracker3 [26] in challenging real-world scenarios involving complex motion and frequent occlusions, while achieving comparable final performance with 10× faster convergence, substantially reducing t… view at source ↗
Figure 2
Figure 2. Figure 2: Overall Architecture of DiTracker. For long video sequences, input frames are divided into N chunks with the global first frame prepended. Individual video frames are then encoded via VAE. These encoded latents are processed by a video DiT to extract query feature qi and key feature kj , which are then used to compute a hierarchical local 4D matching cost C DiT i,j . The video DiT local cost is subsequentl… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative results on ITTO-MOSE [9] benchmark. Our DiTracker predicts more accurate point trajectories under challenging real-world scenarios, including dynamic motions and occlusions, even surpassing CoTracker3 [26]. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attention head analysis in 17th layer of CogVideoX-2B [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative results on ITTO-MOSE [9] benchmark. Our DiTracker predicts more smooth and accurate point trajectories under challenging real-world scenarios, including large displacement, motion blur, and occlusions, even surpassing CoTracker3 [26]. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results on TAP-Vid-DAVIS [11]. Our DiTracker predicts more accurate point trajectories under large displacements between the query frame and final frame compared to CoTracker3 [26]. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative results on TAP-Vid-DAVIS [11] with corruptions from ImageNet-C [21]. Our DiTracker predicts more accurate point trajectories under gaussian noise and motion blur corruptions compared to CoTracker3 [26]. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking

    cs.CV 2026-05 unverdicted novelty 8.0

    TrackCraft3R is the first method to repurpose a video diffusion transformer as a feed-forward dense 3D tracker via dual-latent representations and temporal RoPE alignment, achieving SOTA performance with lower compute.

  2. DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents

    cs.CV 2026-08 conditional novelty 7.0

    A single RGB image plus a language instruction predicts a metric 6-DoF object trajectory, decoded from the intermediate latents of a frozen video diffusion model at an early denoising step.

  3. Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation

    cs.CV 2026-07 conditional novelty 5.0

    Massive activations in DiTs are timestep-driven detail channels; suppressing them guides finer sampling and AdaLN-modulating them yields more discriminative dense features.

Reference graph

Works this paper leans on

65 extracted references · 11 linked inside Pith · cited by 3 Pith papers

  1. [1]

    V-JEPA 2: Self-supervised video models enable understanding, predic- tion and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-JEPA 2: Self-supervised video models enable understanding, predic- tion and planning.arXiv preprint arXiv:2506.09985, 2025. 3, 4, 10

    Pith/arXiv arXiv 2025

  2. [2]

    Can visual foundation models achieve long-term point tracking? InEC- CVW, 2024

    G ¨orkay Aydemir, Weidi Xie, and Fatma G ¨uney. Can visual foundation models achieve long-term point tracking? InEC- CVW, 2024. 2

    2024

  3. [3]

    Track-On: Transformer-based online point tracking with memory

    G ¨orkay Aydemir, Xiongyi Cai, Weidi Xie, and Fatma G¨uney. Track-On: Transformer-based online point tracking with memory. InICLR, 2025. 2

    2025

  4. [4]

    DriveTrack: A benchmark for long-range point tracking in real-world videos

    Arjun Balasingam, Joseph Chandler, Chenning Li, Zhoutong Zhang, and Hari Balakrishnan. DriveTrack: A benchmark for long-range point tracking in real-world videos. InCVPR, pages 22488–22497, 2024. 2

    2024

  5. [5]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. InCVPR, pages 6299–6308, 2017. 6

    2017

  6. [6]

    FlowTrack: Revisiting optical flow for long- range dense tracking

    Seokju Cho, Jiahui Huang, Seungryong Kim, and Joon- Young Lee. FlowTrack: Revisiting optical flow for long- range dense tracking. InCVPR, pages 19268–19277, 2024. 2

    2024

  7. [7]

    Local all-pair correspon- dence for point tracking

    Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seun- gryong Kim, and Joon-Young Lee. Local all-pair correspon- dence for point tracking. InECCV, pages 306–325, 2024. 1, 2, 7, 8, 9

    2024

  8. [8]

    Seurat: From moving points to depth

    Seokju Cho, Jiahui Huang, Seungryong Kim, and Joon- Young Lee. Seurat: From moving points to depth. InCVPR, pages 7211–7221, 2025. 1, 2

    2025

  9. [9]

    Is this tracker on? a benchmark protocol for dynamic tracking

    Ilona Demler, Saumya Chauhan, and Georgia Gkioxari. Is this tracker on? a benchmark protocol for dynamic tracking. NeurIPS, 2025. 2, 4, 6, 7, 8, 12

    2025

  10. [10]

    MOSE: A new dataset for video object segmentation in complex scenes

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. InICCV, pages 20224–20234, 2023. 7

    2023

  11. [11]

    TAP-Vid: A benchmark for track- ing any point in a video.NeurIPS, 35:13610–13626, 2022

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: A benchmark for track- ing any point in a video.NeurIPS, 35:13610–13626, 2022. 1, 2, 3, 4, 6, 7, 9, 10, 13, 14

    2022

  12. [12]

    TAPIR: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. InICCV, pages 10061–10072, 2023. 1, 2, 7, 8, 9

    2023

  13. [13]

    BootsTAP: Bootstrapped training for tracking-any-point

    Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, Joao Carreira, et al. BootsTAP: Bootstrapped training for tracking-any-point. InACCV, pages 3257–3274,

  14. [14]

    Unleashing diffu- sion transformers for visual correspondence by modulating massive activations.NeurIPS, 2025

    Chaofan Gan, Yuanpeng Tu, Xi Chen, Tieyuan Chen, Yuxi Li, Mehrtash Harandi, and Weiyao Lin. Unleashing diffu- sion transformers for visual correspondence by modulating massive activations.NeurIPS, 2025. 2, 3

    2025

  15. [15]

    Motion prompting: Controlling video generation with motion trajec- tories

    Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, and Deqing Sun. Motion prompting: Controlling video generation with motion trajec- tories. InCVPR, 2025. 1

    2025

  16. [16]

    Ego4D: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012, 2022. 7

    2022

  17. [17]

    Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. InCVPR, pages 3749–3761,

  18. [18]

    Harley, Zhaoyuan Fang, and Katerina Fragkiadaki

    Adam W. Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. InECCV, pages 59–75. Springer, 2022. 2

    2022

  19. [19]

    Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, et al

    Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, et al. AllTracker: Effi- cient dense point tracking at high resolution. InICCV, 2025. 1

    2025

  20. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778, 2016. 1, 2, 3

    2016

  21. [21]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. InICLR, 2019. 2, 4, 7, 9, 11, 14

    2019

  22. [22]

    Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.NeurIPS, 33:6840–6851, 2020. 3

    2020

  23. [23]

    LVOS: A benchmark for long-term video object segmentation

    Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. LVOS: A benchmark for long-term video object segmentation. In ICCV, pages 13480–13492, 2023. 7

    2023

  24. [24]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language mod- els.ICLR, 1(2):3, 2022. 2, 5

    2022

  25. [25]

    Stereo4D: Learning how things move in 3D from internet stereo videos

    Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4D: Learning how things move in 3D from internet stereo videos. InCVPR, pages 10497–10509, 2025. 2

    2025

  26. [26]

    Co- Tracker3: Simpler and better point tracking by pseudo- labelling real videos.arXiv preprint arXiv:2410.11831,

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- Tracker3: Simpler and better point tracking by pseudo- labelling real videos.arXiv preprint arXiv:2410.11831,

    Pith/arXiv arXiv

  27. [27]

    Co- Tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- Tracker: It is better to track together. InECCV, pages 18–35,

  28. [28]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. InCVPR, pages 9492–9502, 2024. 2, 3 15

    2024

  29. [29]

    Exploring temporally-aware features for point tracking

    In `es Hyeonsu Kim, Seokju Cho, Jiahui Huang, Jung Yi, Joon-Young Lee, and Seungryong Kim. Exploring temporally-aware features for point tracking. InCVPR, pages 1962–1972, 2025. 1, 2

    1962

  30. [30]

    Learning to track any points from human motion.arXiv preprint arXiv:2507.06233, 2025

    In `es Hyeonsu Kim, Seokju Cho, Jahyeok Koo, Junghyun Park, Jiahui Huang, Joon-Young Lee, and Seungryong Kim. Learning to track any points from human motion.arXiv preprint arXiv:2507.06233, 2025. 2

    arXiv 2025

  31. [31]

    HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. HunyuanVideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    Pith/arXiv arXiv

  32. [32]

    Dense optical tracking: Connecting the dots

    Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. InCVPR, pages 19187–19197, 2024. 2

    2024

  33. [33]

    TAPTRv2: Attention-based position update improves tracking any point

    Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Bohan Li, Tianhe Ren, and Lei Zhang. TAPTRv2: Attention-based position update improves tracking any point. NeurIPS, 37:101074–101095, 2024. 2

    2024

  34. [34]

    TAPTR: Tracking any point with transformers as detection

    Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. TAPTR: Tracking any point with transformers as detection. InECCV, pages 57–75. Springer, 2024. 2, 7, 8, 9

    2024

  35. [35]

    SD4Match: Learning to prompt stable diffu- sion model for semantic matching

    Xinghui Li, Jingyi Lu, Kai Han, and Victor Adrian Prisacariu. SD4Match: Learning to prompt stable diffu- sion model for semantic matching. InCVPR, pages 27558– 27568, 2024. 3

    2024

  36. [36]

    Hunyuan-DIT: A power- ful multi-resolution diffusion transformer with fine-grained chinese understanding.arXiv preprint arXiv:2405.08748,

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-DIT: A power- ful multi-resolution diffusion transformer with fine-grained chinese understanding.arXiv preprint arXiv:2405.08748,

    Pith/arXiv arXiv

  37. [37]

    TSM: Temporal shift module for efficient video understanding

    Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. InICCV, pages 7083–7093, 2019. 2

    2019

  38. [38]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 3

    Pith/arXiv arXiv 2022

  39. [39]

    Mind the gap: Aligning vision foundation models to image feature match- ing.arXiv preprint arXiv:2507.10318, 2025

    Yuhan Liu, Jingwen Fu, Yang Wu, Kangyi Wu, Pengna Li, Jiayi Wu, Sanping Zhou, and Jingmin Xin. Mind the gap: Aligning vision foundation models to image feature match- ing.arXiv preprint arXiv:2507.10318, 2025. 3

    Pith/arXiv arXiv 2025

  40. [40]

    Diffusion hyperfeatures: Search- ing through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holyn- ski, and Trevor Darrell. Diffusion hyperfeatures: Search- ing through time and space for semantic correspondence. NeurIPS, 36:47500–47510, 2023. 3

    2023

  41. [41]

    Not all diffusion model activations have been evaluated as discriminative features.NeurIPS, 37: 55141–55177, 2024

    Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, and Qingming Huang. Not all diffusion model activations have been evaluated as discriminative features.NeurIPS, 37: 55141–55177, 2024. 2, 3

    2024

  42. [42]

    DreamMatcher: Ap- pearance matching self-attention for semantically-consistent text-to-image personalization

    Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seun- gryong Kim, and Seunggyu Chang. DreamMatcher: Ap- pearance matching self-attention for semantically-consistent text-to-image personalization. InCVPR, pages 8100–8110,

  43. [43]

    Diffusion model for dense matching

    Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hy- oungwon Cho, Seyeon Kim, and Seungryong Kim. Diffusion model for dense matching. InICLR, 2024. 2, 3, 7

    2024

  44. [44]

    Emergent tem- poral correspondences from video diffusion transformers

    Jisu Nam, Soowon Son, Dahyun Chung, Jiyoung Kim, Siy- oon Jin, Junhwa Hur, and Seungryong Kim. Emergent tem- poral correspondences from video diffusion transformers. NeurIPS, 2025. 2, 3, 10

    2025

  45. [45]

    DINOv2: Learning robust visual features without supervi- sion.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervi- sion.arXiv preprint arXiv:2304.07193, 2023. 2, 3, 4, 10

    Pith/arXiv arXiv 2023

  46. [46]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 6

    Pith/arXiv arXiv 2017

  47. [47]

    TAPTRv3: Spatial and temporal context foster robust tracking of any point in long video.arXiv preprint arXiv:2411.18671, 2024

    Jinyuan Qu, Hongyang Li, Shilong Liu, Tianhe Ren, Zhaoyang Zeng, and Lei Zhang. TAPTRv3: Spatial and temporal context foster robust tracking of any point in long video.arXiv preprint arXiv:2411.18671, 2024. 2

    arXiv 2024

  48. [48]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, pages 10684– 10695, 2022. 2, 3

    2022

  49. [49]

    Particle video: Long-range mo- tion estimation using point trajectories.IJCV, 80(1):72–91,

    Peter Sand and Seth Teller. Particle video: Long-range mo- tion estimation using point trajectories.IJCV, 80(1):72–91,

  50. [50]

    DINOv3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. DINOv3.arXiv preprint arXiv:2508.10104, 2025. 3, 4, 10

    Pith/arXiv arXiv 2025

  51. [51]

    CleanDIFT: Diffusion features without noise

    Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Bj ¨orn Ommer. CleanDIFT: Diffusion features without noise. InCVPR, pages 117–127, 2025. 3

    2025

  52. [52]

    Emergent correspondence from image diffusion.NeurIPS, 36:1363–1389, 2023

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion.NeurIPS, 36:1363–1389, 2023. 2, 3

    2023

  53. [53]

    DINO-Tracker: Taming DINO for self-supervised point tracking in a single video

    Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. DINO-Tracker: Taming DINO for self-supervised point tracking in a single video. InECCV, pages 367–385, 2024. 2

    2024

  54. [54]

    RoboTAP: Tracking arbitrary points for few-shot visual imitation

    Mel Vecerik, Carl Doersch, Yi Yang, Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. RoboTAP: Tracking arbitrary points for few-shot visual imitation. InICRA, pages 5397–5403, 2024. 1

    2024

  55. [55]

    Wan: Open and advanced large-scale video gen- erative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video gen- erative models.arXiv preprint arXiv:2503.20314, 2025. 1, 2, 3, 4

    Pith/arXiv arXiv 2025

  56. [56]

    Tracking everything everywhere all at once

    Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. InICCV, pages 19795–19806, 2023. 2 16

    2023

  57. [57]

    Shape of motion: 4D reconstruc- tion from a single video

    Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4D reconstruc- tion from a single video. InICCV, 2025. 1

    2025

  58. [58]

    SpatialTracker: Tracking any 2D pixels in 3D space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking any 2D pixels in 3D space. InCVPR, pages 20406– 20417, 2024. 1, 2

    2024

  59. [59]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 2, 3

    2023

  60. [60]

    MATCHA: Towards matching anything

    Fei Xue, Sven Elflein, Laura Leal-Taix ´e, and Qunjie Zhou. MATCHA: Towards matching anything. InCVPR, pages 27081–27091, 2025. 3

    2025

  61. [61]

    CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 1, 2, 3, 4, 10

    Pith/arXiv arXiv 2024

  62. [62]

    A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.NeurIPS, 36: 45533–45547, 2023

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Pola- nia Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.NeurIPS, 36: 45533–45547, 2023. 2, 3

    2023

  63. [63]

    Telling left from right: Identifying geometry-aware semantic corre- spondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. Telling left from right: Identifying geometry-aware semantic corre- spondence. InCVPR, pages 3076–3085, 2024. 3

    2024

  64. [64]

    TAPNext: Tracking any point (TAP) as next token prediction

    Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen He, Ignacio Rocco, Mehdi SM Sajjadi, Sarath Chandar, and Ross Goroshin. TAPNext: Tracking any point (TAP) as next token prediction. InICCV,

  65. [2024]

    1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14