Pith. sign in

REVIEW 5 major objections 5 minor 58 references

ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ProTracker claims that probabilistic integration of optical flow and long-term keypoint matches delivers state-of-the-art point-tracking accuracy, beating all compared methods on position precision across TAP-Vid and BADJA.

desk verdict Solid engineering contribution to point tracking, but the unreported correlation coefficient p in the fusion step makes the exact numbers non-reproducible; still worth refereeing. read the letter →

arxiv 2501.03220 v2 pith:N32DYVZD submitted 2025-01-06 cs.CV

Tingyang Zhang , Chen Wang , Zhiyang Dou , Qingzhe Gao , Jiahui Lei , Baoquan Chen , Lingjie Liu This is my paper · ORCID
classification cs.CV
keywords pointtrackingopticalflowlong-termcorrespondenceprobabilisticintegrationKalmanfilterocclusionhandlingTAP-Vidbenchmarktest-timeoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProTracker makes the case that long-term dense point tracking is best framed as a probabilistic fusion problem rather than a choice between local and global trackers. The paper's claim is that chaining optical-flow predictions from multiple past frames, treating each as a Gaussian, and merging them with high-confidence long-term keypoint matches yields the precision of iterative local trackers and the occlusion robustness of global heatmap methods simultaneously. On TAP-Vid DA VIS (first and strided queries), Kinetics-first, and BADJA, the reported position accuracy, measured by $\delta x_{\text{avg}}$, is the highest among all methods compared, and it is state-of-the-art among optimization-based trackers. The practical payoff is a test-time tracker that recovers points after occlusion and reappearance without drifting.

What carries the argument

The load-bearing mechanism is the bidirectional probabilistic integration rule, which treats every prediction as an isotropic Gaussian. For a chain from frame $j$ to $i$, the mean is $\mu_{ji} = \mu_j + f_{ji}$ and the variance is $\sigma_{ji}^2 = \sigma_j^2 + \sigma_{fji}^2$ (assuming an orthogonal flow Jacobian), and merging $N$ such predictions gives a combined variance of $\sqrt{(N-1)p + 1 / \sum_j 1/\sigma_{ji}^2}$, where $p$ is a constant correlation coefficient. The joint integration then treats a long-term keypoint as an independent Gaussian with $\sigma = 1$ and combines it with the flow-integrated estimate by inverse-variance weighting (Eq. 11). A hybrid filter removes predictions before integration, using object-level segmentation masks and geometry-aware features with thresholds 0.7 for keypoints and 0.3 for flow.

What would settle it

Run ProTracker on TAP-Vid DA VIS with $p$ fixed to 0, 0.5, and 1 and with the full flow Jacobian kept in Eq. (2); if $\delta x_{\text{avg}}$ does not peak near the paper's implicit $p$, or if the orthogonal-Jacobian version does not beat the full-Jacobian version, then the fusion rule as specified is not the mechanism behind the reported accuracy.

Watch

Extended reading notes

Core claim

The central claim is that the fusion framework itself, not any single component, produces the accuracy gain. Rough optical-flow predictions chained from frames at intervals $\{\infty, 1, 2, 4, 8, 16, 32\}$ are first filtered by an object-level mask and a geometry-aware feature similarity check, then merged into one Gaussian estimate by inverse-variance weighting with a constant correlation coefficient $p$, and the same update combines this flow estimate with long-term keypoint observations derived from a refined DINO heatmap. The author states that 'our approach achieves the highest $\delta x_{\text{avg}}$ across all datasets, demonstrating superior precision in tracking visible points.' The mechanism also runs backward from the last frame to recover points the forward pass marked occluded, and it re-localizes points that disappear and reappear.

Load-bearing premise

The accuracy gain rests on treating the chained flow predictions as correlated Gaussians with a single constant correlation coefficient $p$, whose value is never reported, while also simplifying the flow Jacobian to orthogonal; if the true error correlations deviate from that model, the integration weights that produce the reported accuracy are not actually specified.

Editorial extensions

If this is right

  • Points that disappear and later reappear can be re-localized by the long-term keypoint branch, after which optical flow resumes tracking, recovering trajectory segments that pure flow chaining would lose.
  • Because the fused variance shrinks as more valid predictions accumulate, accuracy should improve with longer integration spans without retraining, up to the limit where stale flow predictions violate the Gaussian error model.
  • The reported results imply that a test-time optimization tracker can beat supervised feed-forward trackers on position precision by combining cheap short-term flow with sparse long-term matches, shifting the design trade-off away from larger networks.
  • The same fusion update applies to any noisy source of point predictions, so swapping in a different optical-flow or correspondence model should preserve the robustness gains as long as the filter and the variance estimates are retained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The constant correlation coefficient $p$ is never given a value in the paper, so measuring it empirically from chained-flow error correlations would make the method fully reproducible and likely change the optimal balance between flow and keypoints.
  • The orthogonality assumption on the flow Jacobian is a known weak point for large rotations and non-rigid deformation; testing the full-Jacobian update would reveal how much of the gain depends on that simplification.
  • A natural extension is to apply the same probabilistic integration to other backbone pairs, for instance replacing RAFT with a learned long-range flow model, to test whether the fusion rule itself generalizes or whether the gain is tied to the specific flow and keypoint models used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. ProTracker is a test-time optimization point tracker that combines chained RAFT optical flow predictions with long-term keypoint correspondences from a DINO-based feature model. The pipeline first filters rough predictions with SAM2-style object masks and geometry-aware semantic features, then integrates filtered flow predictions from multiple temporal intervals in forward and backward directions using Gaussian inverse-variance weighting, and finally fuses the integrated flow estimate with high-confidence long-term keypoints through a second inverse-variance fusion. The authors evaluate on TAP-Vid DA VIS, TAP-Vid Kinetics, and BADJA, reporting state-of-the-art results among optimization-based trackers and the highest delta_x_avg among all compared methods, with component ablations and an efficiency comparison against DINO-Tracker.

Significance. If the reported results hold, ProTracker is a practically valuable contribution: it demonstrates that a probabilistic coupling of local optical flow and global semantic keypoints can achieve competitive or superior precision relative to supervised feed-forward trackers, without end-to-end supervised training of the whole pipeline. The paper's strengths include extensive comparisons against many recent baselines, component ablations that isolate the main algorithmic choices, and a clear efficiency advantage over DINO-Tracker. The probabilistic formulation is simple and appealing. However, the central claim that the fusion framework is the source of the SOTA precision is currently under-specified: the fusion depends on an unreported correlation coefficient and an arbitrary keypoint variance, and the independence assumption between the two fused signals is questionable given the self-supervised training signal.

major comments (5)
  1. [Sec. 3.2, Eq. (6)] The correlation coefficient p introduced in Eq. (6) is never assigned a value in the main text or the supplement, yet the variance inflation term (N-1)p+1 directly controls the relative weight of the integrated flow estimate in the final fusion Eq. (11). With the reported margins being small (e.g., 0.7 delta_x_avg on DA-VIS First and 0.2 on Kinetics First in Table 2), the conclusion that the fusion framework explains the SOTA precision requires either a specified p or a sensitivity analysis over a plausible range of p. As written, the experiments are not reproducible from the paper alone.
  2. [Sec. 3.2, Eqs. (2)-(3)] The derivation replaces the covariance propagation J sigma^2 J^T by the scalar sum sigma^2_j + sigma^2_fji via the assumption that J is orthogonal. This is only exact for translations and small rotations; under large rotations or non-rigid motion the Jacobian is not orthogonal, so the propagated variance is incorrect. Since sigma_i enters the inverse-variance weights in Eq. (4) and Eq. (11), the approximation is not merely cosmetic. Please justify the assumption or propagate the full covariance.
  3. [Sec. 3.3, Eqs. (8)-(12)] The long-term keypoint model is trained with optical flow as a self-supervised signal ('we train a feature extractor ... with the optical flow as a self-supervised signal'). The flow-integrated prediction and the keypoint observation are therefore not independent, because the keypoint model has been fit to the same RAFT flow that is being integrated. Treating the two as independent Gaussian likelihoods in Eq. (11) violates the independence assumption underlying the product-of-Gaussians fusion. Please quantify the correlation or discuss how the training signal is made independent, for example by using different flow frames or withholding the target flow.
  4. [Sec. 3.3, Eq. (10)] The keypoint variance is set to sigma_key = 1 without any calibration or justification. The ratio sigma_i / sigma_key determines how strongly the long-term keypoint pulls the final estimate in Eq. (11). A unit variance may be reasonable for a heatmap SoftArgMax error, but the paper should report how this value was chosen and how sensitive the final delta_x_avg is to it. Without this, the balance between flow and keypoint is under-specified.
  5. [Sec. 4.2, Tables 1-3] All reported numbers are point estimates without error bars or repeated-run statistics. Some headline margins are very small (e.g., Kinetics-First delta_x_avg 71.1 vs. 70.9 for CoTracker3 in Table 2), and the ablation gains in Table 3 are also single runs. To support the load-bearing claim that the full system 'achieves the highest delta_x_avg across all datasets,' please provide standard deviations over at least three runs or bootstrap confidence intervals per video.
minor comments (5)
  1. [Sec. 3.2, Eq. (6)] The symbol N is not defined in the main text; it appears to be the number of integrated predictions. Please define it.
  2. [Sec. 3.2, after Eq. (6)] The sentence 'where mu_i represents the final predicted position' should refer to mu^f_i, to avoid confusion with the preliminary refined mean in Eq. (4).
  3. [Sec. 3.1 vs. Supp. Sec. 2.1] The main text states that a prediction is invalid if feature similarity to the query falls below 0.5, while the supplement says flow predictions use a threshold of 0.3 and long-term keypoints use 0.7; please make the main-text description consistent with the supplement.
  4. [Supp. Sec. 2.2, Eq. (1)] The notation (Oji, p) > rho should be Oji(p) > rho, and the symbol rho is reused later for the keypoint similarity threshold; please use distinct symbols to avoid ambiguity.
  5. [Supp. Sec. 2.1] The sentence 'our method don't require any hyperparameter tuning' should read 'does not require'.

Circularity Check

1 steps flagged · score 6.0 of 10

Keypoint 'observation' is trained from the same RAFT flow it is later fused with, so the claimed independent probabilistic fusion partially reduces to re-averaging the flow signal.

  1. fitted input called prediction [Sec. 3.3, Eqs. (8)-(11); independence assumption in Sec. 3.2]
    "We train a feature extractor Φ∆ and heatmap refiner R of a long-term correspondence-based keypoint tracker based on DINO-Tracker [49] for the input video, with the optical flow as a self-supervised signal. ... whenever valid keypoints from the long-term correspondence are available, we treat them as another source of noisy observations besides optical flow."

    Eq. (10) obtains keypoints p_i from the flow-trained network, and Eq. (11) inverse-variance averages the flow-integrated estimate µ_i with these keypoints under the independence assumption introduced in Sec. 3.2 ('by assuming that they are independent'). Because the keypoint network is trained on the same RAFT flow that generates µ_i, the two 'sources' are not independent: the keypoint observation is a fitted function of the flow signal. The maximum-likelihood fusion therefore reduces, in large part, to averaging the flow with a flow-fitted copy of itself; any gain from the keypoint term cannot be cleanly attributed to an independent global re-localization. This is a fitted input renamed as an observation.

full rationale

The benchmark claims are measured against external TAP-Vid/BADJA ground truth, so the SOTA result is not a tautology. The circularity is confined to the method's derivation: the long-term keypoint model is trained per video with RAFT optical flow as its self-supervised signal, and the same flow-integrated estimate is then combined with these keypoints under an independence assumption. Thus the 'global observation' is fitted to the very signal it is supposed to complement, undermining the claim that the fusion combines independent evidence. The unreported correlation coefficient p and the fixed σ_key=1 are reproducibility and robustness concerns, not circularity. No load-bearing self-citation chain is present: DINO-Tracker and MFT are external prior work, and CaDeX++ is only a baseline. The partial circularity is real but does not make the whole derivation equivalent to its inputs, hence a score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on a handful of hand-chosen thresholds, an unspecified correlation coefficient, and several strong modeling assumptions: orthogonal flow Jacobian, correlated-Gaussian merge, and flow-supervised keypoints treated as independent evidence. No new physical entities are introduced.

free parameters (6)
  • Correlation coefficient p = not specified
    Introduced in Eq. (6) and Eq. (7) to account for correlations between chained predictions. It directly controls the integrated variance and therefore the weighting between flow and keypoints in Eq. (11). No value or tuning procedure is reported.
  • Flow feature similarity threshold = 0.3
    Used in the hybrid filter for flow predictions (Supplementary Sec. 2.1). Hand-chosen; affects which rough predictions are retained.
  • Long-term keypoint similarity threshold = 0.7
    Taken from DINO-Tracker (Eq. 10). Determines which long-term keypoints are accepted as observations.
  • Geometry-aware feature threshold = 0.5
    Used in the hybrid filter to reject ambiguous points (Supplementary Sec. 2.1).
  • Outlier distance threshold rho_dist = 10 pixels
    Supplementary Eq. (2). Discards rough predictions deviating more than 10 pixels from the most confident one.
  • Keypoint observation variance sigma_key = 1
    Fixed value in Eq. (10). Controls how strongly keypoints influence the final estimate relative to flow.
assumptions (4)
  • domain assumption The Jacobian of optical flow with respect to position is orthogonal (Eq. 2 to Eq. 3).
    Stated as 'for ease of computation' in Sec. 3.2. Invalid for large rotations or non-rigid deformations, yet used to simplify variance propagation.
  • ad hoc to paper Predictions from different frames can be modeled as Gaussians and combined via inverse-variance weighting with a constant correlation p.
    The merge rule in Eq. (6) assumes a specific correlated-Gaussian product form; this is asserted, not derived, and p is unspecified.
  • domain assumption The long-term keypoint model trained with RAFT flow provides re-localization evidence beyond flow itself.
    Sec. 3.3 trains the delta-DINO extractor and heatmap refiner with optical flow as self-supervision. The claim that this yields independent long-term observations is assumed and tested only indirectly via ablations.
  • domain assumption DINOv2 features plus a per-video delta feature generalize across appearance and occlusion.
    Relies on DINO-Tracker's finding; the paper follows that training recipe in Sec. 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking." pith.science (2026). https://pith.science/paper/N32DYVZD

@misc{pith2026250103220,
  author       = {Pith},
  title        = {Pith review of: ProTracker: Probabilistic Integration for Robust and Accurate Point Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N32DYVZD}},
  note         = {Machine review of arXiv:2501.03220}
}
read the original abstract

We propose ProTracker, a novel framework for accurate and robust long-term dense tracking of arbitrary points in videos. Previous methods relying on global cost volumes effectively handle large occlusions and scene changes but lack precision and temporal awareness. In contrast, local iteration-based methods accurately track smoothly transforming scenes but face challenges with occlusions and drift. To address these issues, we propose a probabilistic framework that marries the strengths of both paradigms by leveraging local optical flow for predictions and refined global heatmaps for observations. This design effectively combines global semantic information with temporally aware low-level features, enabling precise and robust long-term tracking of arbitrary points in videos. Extensive experiments demonstrate that ProTracker attains state-of-the-art performance among optimization-based approaches and surpasses supervised feed-forward methods on multiple benchmarks. The code and model will be released after publication.

Figures

Figures reproduced from arXiv: 2501.03220 by the authors.

Figure 1
Figure 1. Visualization of tracking trajectories in various videos.Our method achieves robust point tracking without suffering from drifting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline overview of our proposed method. (1) Sample & Chain: Key points are initially sampled and linked through optical [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Bidirectional Probabilistic Flow Integration. Top row: Optical flow effectively tracks a point in the short term but may fail under [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: We evaluate our method against state-of-the-art approaches, including feed-forward models (Co-Tracker3 [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation study on different components. 5. Conclusion and Future Work In this paper, we introduced a robust tracking framework that combines optical flow integration with long-term cor￾respondence through probabilistic integration to achieve ac￾curate and smooth point …
Figure 1
Figure 1. Figure 1: Results of tracking a single object. While DINO-Tracker may mispredict parts onto similar objects and TAPIR can be disrupted [PITH_FULL_IMAGE:figures/full_fig_p014_1.png]
Figure 2
Figure 2. Figure 2: Results of tracking a single object. While DINO-Tracker may lose some parts and TAPIR can be disrupted by multiple similar [PITH_FULL_IMAGE:figures/full_fig_p015_2.png]
Figure 3
Figure 3. Figure 3: Results of tracking at a higher frame rate. Sliding window based methods can easily lose track after occlusion and drift due to [PITH_FULL_IMAGE:figures/full_fig_p016_3.png]
Figure 4
Figure 4. Figure 4: Results of tracking at a higher frame rate. Sliding window based methods can mispredict points to other regions during occlusion [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 36 canonical work pages

  1. [1]

    Can visual foundation models achieve long-term point tracking?, 2024

    G ¨orkay Aydemir, Weidi Xie, and Fatma G ¨uney. Can visual foundation models achieve long-term point tracking?, 2024. 2

    work page 2024

  2. [2]

    Creatures great and SMAL: Recovering the shape and motion of animals from video

    Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and SMAL: Recovering the shape and motion of animals from video. In ACCV, 2018. 5, 6

    work page 2018

  3. [3]

    High accuracy optical flow estimation based on a theory for warping

    Thomas Brox, Andr ´es Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8 , pages 25–36. Springer, 2004. 2

    work page 2004

  4. [4]

    Large displacement optical flow

    Thomas Brox, Christoph Bregler, and Jitendra Malik. Large displacement optical flow. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 41–48. IEEE, 2009. 2

    work page 2009

  5. [5]

    End-to- end object detection with transformers, 2020

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers, 2020. 2

    2020

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2

    2021

  7. [7]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 5

    2017

  8. [8]

    Zero-shot image feature consensus with deep functional maps

    Xinle Cheng, Congyue Deng, Adam Harley, Yixin Zhu, and Leonidas Guibas. Zero-shot image feature consensus with deep functional maps. arXiv preprint arXiv:2403.12038 ,

    arXiv

Show all 58 references
  1. [9]

    Cats: Cost ag- gregation transformers for visual correspondence

    Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost ag- gregation transformers for visual correspondence. Advances in Neural Information Processing Systems , 34:9011–9023,

  2. [10]

    Flowtrack: Revisiting optical flow for long- range dense tracking

    Seokju Cho, Jiahui Huang, Seungryong Kim, and Joon- Young Lee. Flowtrack: Revisiting optical flow for long- range dense tracking. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 19268–19277, 2024. 1

    2024

  3. [11]

    Local all-pair correspon- dence for point tracking

    Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seun- gryong Kim, and Joon-Young Lee. Local all-pair correspon- dence for point tracking. arXiv preprint arXiv:2407.15420,

    arXiv

  4. [12]

    TAP-vid: A benchmark for track- ing any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for track- ing any point in a video. Advances in Neural Information Processing Systems, 35:13610–13626, 2022. 1, 2, 5, 6

    2022

  5. [13]

    Tapir: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pag...

    2023

  6. [14]

    Boot- sTAP: Bootstrapped training for tracking-any-point

    Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, Jo˜ao Carreira, and Andrew Zisserman. Boot- sTAP: Bootstrapped training for tracking-any-point. Asian Conference on Computer Vision, 2024. 6

    2024

  7. [15]

    Flownet: Learning optical flow with convolutional networks

    Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Pro- ceedings of the IEEE international conference on computer...

    2015

  8. [16]

    Videoswap: Customized video subject swapping with interactive semantic point cor- respondence

    Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point cor- respondence. In Proceedings of the IEEE/CVF Conference o...

    2024

  9. [17]

    Particle video revisited: Tracking through occlusions using point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vi- sion, pages 59–75. Springer, 2022. 1, 2, 6

    2022

  10. [18]

    Unsupervised semantic correspondence using stable diffu- sion

    Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffu- sion. Advances in Neural Information Processing Systems , 36, 2024. 2

    2024

  11. [19]

    Determining opti- cal flow

    Berthold KP Horn and Brian G Schunck. Determining opti- cal flow. Artificial intelligence, 17(1-3):185–203, 1981. 2

    1981

  12. [20]

    Flowformer: A transformer architecture for optical flow

    Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European conference on computer vision, pages 668–685. Springer, 2022. 2

    2022

  13. [21]

    Flownet 2.0: Evolu- tion of optical flow estimation with deep networks

    Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolu- tion of optical flow estimation with deep networks. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 2462–2470, 2017. 2

    2017

  14. [22]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960. 2, 4

    1960

  15. [23]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023. 1, 2, 6

    2023 arXiv

  16. [24]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. In Proc. arXiv:2410.11831, 2024. 6, 7

    2024 arXiv

  17. [25]

    Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds, 2024

    Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds, 2024. 1

    2024

  18. [26]

    Taptrv2: Attention-based position update improves tracking any point

    Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe Ren, Bohan Li, and Lei Zhang. Taptrv2: Attention-based position update improves tracking any point. arXiv preprint arXiv:2407.16291, 2024. 6

    2024 arXiv

  19. [27]

    Taptr: Tracking any point with transformers as detection

    Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers as detection. In Proceedings of the IEEE/CVF European Conference on Computer Vision, 2024. 1, 2, 7

    2024

  20. [28]

    Decomposition betters tracking every- thing everywhere

    Rui Li and Dong Liu. Decomposition betters tracking every- thing everywhere. arXiv preprint arXiv:2407.06531, 2024. 2, 3, 6

    2024 arXiv

  21. [29]

    David G. Lowe. Distinctive image features from scale- invariant keypoints. International Journal of Computer Vi- sion, 60:91–110, 2004. 1, 2

    2004

  22. [30]

    An iterative image reg- istration technique with an application to stereo vision

    Bruce D Lucas and Takeo Kanade. An iterative image reg- istration technique with an application to stereo vision. In IJCAI’81: 7th international joint conference on Artificial in- telligence, pages 674–679, 1981. 2

    1981

  23. [31]

    Diffusion hyperfeatures: Searching through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holyn- ski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Ad- vances in Neural Information Processing Systems, 36, 2024. 2

    2024

  24. [32]

    Dgc-net: Dense ge- ometric correspondence network

    Iaroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Marc Pollefeys, Esa Rahtu, and Juho Kannala. Dgc-net: Dense ge- ometric correspondence network. In 2019 IEEE Winter Con- ference on Applications of Computer Vision (WACV), pages 1034–1042. IEEE, 2019. 2

    2019

  25. [33]

    Mft: Long- term tracking of every pixel

    Michal Neoral, Jon ´aˇs ˇSer`ych, and Ji ˇr´ı Matas. Mft: Long- term tracking of every pixel. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6837–6847, 2024. 3, 4, 6, 1

    2024

  26. [34]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 3

    2023 arXiv

  27. [35]

    The 2017 davis challenge on video object segmentation

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017. 5

    2017 arXiv

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  30. [38]

    Efficient neighbourhood consensus networks via submanifold sparse convolutions

    Ignacio Rocco, Relja Arandjelovi ´c, and Josef Sivic. Efficient neighbourhood consensus networks via submanifold sparse convolutions. In Computer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part IX 16, pages 605–621. Springer, 2020. 2

    2020

  31. [39]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2

    2022

  32. [40]

    Towards longer long-range motion trajectories

    Michael Rubinstein and Ce Liu. Towards longer long-range motion trajectories. In Proceedings of the British Machine Vision Conference, pages 53.1–53.11. BMV A Press, 2012. 1

    2012

  33. [41]

    Orb: An efficient alternative to sift or surf

    Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference on computer vision , pages 2564–

    2011

  34. [42]

    Sand and S

    P. Sand and S. Teller. Particle video: Long-range motion es- timation using point trajectories. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR’06), pages 2195–2202, 2006. 1

    2006

  35. [43]

    Track everything everywhere fast and ro- bustly

    Yunzhou Song, Jiahui Lei, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Track everything everywhere fast and ro- bustly. arXiv preprint arXiv:2403.17931, 2024. 2, 3, 6, 7

    2024 arXiv

  36. [44]

    Dynamic gaussian marbles for novel view synthesis of casual monocular videos, 2024

    Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos, 2024. 1

    2024

  37. [45]

    Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume

    Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943,

  38. [46]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Pro- cessing Systems, 36:1363–1389, 2023. 2

    2023

  39. [47]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  40. [48]

    Glu- net: Global-local universal network for dense flow and corre- spondences

    Prune Truong, Martin Danelljan, and Radu Timofte. Glu- net: Global-local universal network for dense flow and corre- spondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6258–6268,

  41. [49]

    Dino-tracker: Taming dino for self-supervised point tracking in a single video, 2024

    Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in a single video, 2024. 2, 3, 5, 6, 7, 8, 1

    2024

  42. [50]

    Tracking everything everywhere all at once

    Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 19795–19806, 2023. 2, 3, 6

    2023

  43. [51]

    Shape of motion: 4d reconstruc- tion from a single video

    Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruc- tion from a single video. 2024. 1

    2024

  44. [52]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406–20417, 2024. 1, 2, 6, 7

    2024

  45. [53]

    Telling left from right: Identifying geometry-aware semantic corre- spondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. Telling left from right: Identifying geometry-aware semantic corre- spondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 307...

    2024

  46. [54]

    Video Results Please refer to our Supplementary Webpage for the corre- sponding videos of images illustrated in the paper and more results on different data

  47. [55]

    Hyperparameters During the dual filtering stage, we apply different thresholds to predictions from flow and long-term keypoints

    Implementation Details 2.1. Hyperparameters During the dual filtering stage, we apply different thresholds to predictions from flow and long-term keypoints. For long- term keypoints, we only need those with higher confidence to avoid mistakes. A prediction is first marked as i...

  48. [56]

    While optical flow and geometry-aware features can be computed densely, generat- ing masks for each pixel is both time-intensive and memory- intensive

    Dense Inference As discussed in Sec.3.1 in the main paper, we utilize a geometry-aware feature extractor and a video mask gen- erator for the dual-filter stage. While optical flow and geometry-aware features can be computed densely, generat- ing masks for each pixel is both ti...

  49. [57]

    The total time consumed for our method includes the time for keypoint extraction, mask generation, geometry-aware feature extraction and probabilistic integration

    Training and Inference Speed Our methods is more than 20x faster than DINO-Tracker during the inference stage, while maintaining the same training time. The total time consumed for our method includes the time for keypoint extraction, mask generation, geometry-aware feature ex...

  50. [58]

    We conduct experiments on more challenging cases and show the quali- tative results

    More Qualitative Results To further illustrate our methods’ robustness. We conduct experiments on more challenging cases and show the quali- tative results. Some of the previous methods rely on computing a heatmap between the query point and the target frame. However, the per-...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.