Online Long-term Point Tracking in the Foundation Model Era

G\"orkay Aydemir

REVIEW 3 major objections 4 minor 22 references

Online Long-term Point Tracking in the Foundation Model Era

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A causal frame-by-frame tracker can match offline models on long-term point tracking.

desk verdict Competent thesis packaging two peer-reviewed papers; Track-On's empirical core is solid, but the SOTA claim is overstated and the error-propagation mechanism is asserted, not demonstrated. read the letter →

arxiv 2507.09217 v1 pith:BITJJ7L3 submitted 2025-07-12 cs.CV

Görkay Aydemir This is my paper

classification cs.CV

keywords pointtrackinganyonlinecausalvideoprocessingtransformerspatialmemorycontextDINOv2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis tries to establish that long-term point tracking does not require future frames: a causal transformer that treats each tracked point as a query and carries compact memory from frame to frame can be as accurate as offline trackers that see the whole video. The first part probes visual foundation models and reports that DINOv2, with only lightweight LoRA adaptation, surpasses a fully supervised baseline, so foundation features are a usable geometric initialization. The second part builds Track-On on that backbone, framing correspondence as patch classification with top-k re-ranking and local offset refinement, and adding two memory modules: spatial memory refreshes the query from recent local appearance, and context memory records the track's history. The central claim is that this online design sets a new state of the art among online trackers and matches or exceeds offline models across seven benchmarks, making frame-by-frame tracking practical for robotics, augmented reality, and streaming video.

What carries the argument

The central object is the Track-On query decoder with its two memory modules. Treating each tracked point as a query lets a standard transformer decoder localize many points in one forward pass through the frame; the coarse-to-fine prediction pipeline (patch classification, top-k re-ranking, deformable offset head) replaces the iterative cost-volume regression used by prior trackers. Spatial memory $M_s$ is written by attending a local neighborhood around the current prediction and read by updating the initial query against all stored entries, which is how feature drift is counteracted. Context memory $M_c$ feeds past query embeddings into the decoder, giving the model a bounded history of appearance and occlusions. Because both memories are FIFO queues of fixed size with learnable positional embeddings, the model can be trained on short clips and run on arbitrarily long videos by interpolating the embeddings at inference time.

What would settle it

Deliberately shift the predicted location used to write spatial memory in the first few frames by 10-20 pixels, then compare downstream tracking error against the unperturbed run; if one early wrong prediction substantially increases error later, the claim that memory writes prevent error propagation is refuted.

Watch

Extended reading notes

Core claim

Track-On demonstrates that online, frame-by-frame point tracking can match offline performance. Each point of interest is a query token in a transformer decoder that attends to the current frame; the model predicts where the point is by classifying which patch of the feature map contains it, re-ranking the top-k candidate patches with a dedicated module, and then refining the chosen patch center with an offset head. Visibility and uncertainty are predicted separately. To keep tracks coherent over time without future access, Track-On maintains two FIFO memory modules: spatial memory $M_s$ stores local features around the model's previous predictions and updates the initial query to counter feature drift, while context memory $M_c$ stores previously decoded query embeddings so the decoder can see appearance changes and occlusion history. Memory is trained with size $K=12$ and extended at inference to up to $K=96$ by interpolating positional embeddings. Evaluated on TAP-Vid DAVIS, RGB-Stacking, and Kinetics plus RoboTAP, Dynamic Replica, BADJA, and PointOdyssey, Track-On achieves the best results among online trackers and is competitive with or better than offline baselines on the same training data.

Load-bearing premise

The spatial memory assumes the model's own recent predictions are close enough to the true point that features sampled around them refresh the query correctly, so a wrong early prediction could store misleading content and amplify drift instead of preventing it.

Editorial extensions

If this is right

Online point tracking can be deployed in streaming and embodied settings: Track-On processes one frame at a time with a single backbone forward pass and does not need a sliding window or full-video attention.
A tracker trained on 24-frame synthetic clips can generalize to videos with thousands of frames, since the memory size is extended at inference rather than retrained.
Patch classification with top-k re-ranking and offset refinement is a workable alternative to iterative regression, yielding fine-grained predictions at low error thresholds.
Foundation model features provide the geometric initialization, but temporal consistency still requires dedicated memory; frozen or lightly adapted backbones alone are not enough.
The reported efficiency (~16.8 FPS at $K=48$ with about 0.73 GB GPU memory while tracking roughly 400 points) puts online long-term tracking within reach of real-time applications.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the spatial memory writes from the model's own predicted location, an untested failure mode is error amplification: if an early prediction is wrong, the memory may store features from the wrong region and push later predictions further off; a testable extension is gating memory writes on the predicted uncertainty.
The same query-plus-memory architecture could transfer to other streaming dense-prediction tasks such as video object segmentation or action detection, where FIFO context memories are already used.
The remaining gap on Kinetics AJ suggests that combining this online architecture with pseudo-labeled real-world training data, following the direction of real-video baselines, is a natural next step.
The thesis names lifting tracks into 3D and reprojecting them as future work; if realized with dynamic scene reconstruction models, that would likely help textureless regions where appearance memory is weakest.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

It's a thesis, not a new research paper: Chapters 3 and 4 are the author's own published works (ECCVW 2024 and ICLR 2025). If you know Track-On, you already know the method and most of the numbers. What the thesis adds is a single, coherent write-up of the full pipeline and a consolidated seven-benchmark comparison, which is genuinely useful for the online point-tracking subfield.

The engineering is credible. Patch classification with re-ranking and a local offset head is a sensible alternative to iterative regression, and the two memory modules are clearly motivated and described. The ablations in Tables 4.4-4.7 do real work: they show which components matter, how much, and where memory extension helps. The online-vs-offline comparison is mostly fair, because the tables separate models trained with and without real-world data. But the abstract's "state of the art across seven benchmarks" is too strong. On RGB-Stacking and Kinetics, CoTracker3 trained on Kubric-L beats Track-On on Average Jaccard; "matches or surpasses offline baselines" is the accurate phrasing.

The real soft spot is the error-propagation claim. Section 4.1.3 asserts that updating the query from the initial query and the full memory "prevents error propagation", but no experiment traces what happens after a wrong prediction or whether the memory recovers from a bad frame. Table 4.7 and Figure 4.9 show average gains and a similarity ratio above 1 on visible points, which is supportive but not conclusive. This is a genuine weakness in the explanation, though not in the headline accuracy numbers: the ablations still show memory helps on average, and the main empirical claim (online tracking around 17 FPS with competitive accuracy) holds up. Reproducibility is another minor issue: no code is released and several hyperparameters (exact loss weight, top-k, training memory details) are not fully reported, which makes independent verification harder than it should be.

Who benefits? Someone new to the online tracking literature, or a practitioner wanting a self-contained reference for Track-On's architecture. Someone who has read the ICLR paper will not find new results here.

If this crossed my desk as a journal submission, I would send it to a referee rather than desk reject, with the explicit instruction that novelty must be scoped as a thesis compilation. As a conference submission it would be redundant with already-published work.

Referee Report

3 major / 4 minor

Summary. The thesis addresses online (causal) long-term point tracking. Chapter 3 probes visual foundation models under zero-shot, probing, and LoRA-adaptation regimes, concluding that DINOv2 and Stable Diffusion carry useful geometric priors. Chapter 4 introduces Track-On, a DINOv2 + ViT-Adapter transformer that tracks points frame-by-frame via patch classification, top-k re-ranking, offset refinement, and visibility/uncertainty heads, augmented by spatial and context memory modules and an inference-time memory extension. Experiments are reported on TAP-Vid DAVIS, RGB-Stacking, and Kinetics; RoboTAP, Dynamic Replica, BADJA, and PointOdyssey; plus ablations, memory-size analyses, and efficiency measurements. The central claim is that Track-On sets a new state of the art across seven public benchmarks and demonstrates the feasibility of long-term point tracking without future access.

Significance. If the empirical results stand, the contribution is significant: Track-On is a causal, frame-by-frame tracker that closes much of the accuracy gap to offline methods, with a simple coarse-to-fine formulation and a memory design that is effective in the reported ablations. The systematic benchmark coverage, component ablations, and efficiency analyses are strengths, as is the explicitly online evaluation protocol. However, the headline 'state of the art across seven benchmarks' overstates the tables, and the paper's causal claim that spatial memory 'prevents error propagation' is asserted without a corresponding failure-mode analysis. These issues are correctable, and the core approach remains plausible, so the needed revision is substantive but not a rejection.

major comments (3)

[Abstract; Section 1.3; Tables 4.1 and 4.2] The statement that Track-On 'sets a new state of the art across seven public benchmarks' is not supported by the reported numbers. On RGB-Stacking, Track-On's AJ of 71.4 is below CoTracker3 with video input (74.0 trained on Kub-L and 74.3 trained on Kub-L + real data). On Kinetics, Track-On's AJ of 53.9 is below CoTracker3 windowed (54.1) and BootsTAPIR (54.6). On RoboTAP, Track-On's AJ of 63.5 is below BootsTAPIR (64.9) and CoTracker3 windowed with real data (66.4). The defensible claim is that Track-On is state-of-the-art among online models and competitive with, but not uniformly superior to, offline models; please revise the abstract and conclusion to state the claim with this qualification.
[Section 4.1.3, Eqs. (4.8)-(4.9); Table 4.7; Figure 4.9] The claim that spatial memory 'prevents error propagation' is asserted rather than demonstrated. Because the memory is written using features around the model's own previous prediction, a localization error at an early frame will store features from the wrong region, and Eq. (4.9) subsequently conditions the updated query on that contaminated memory. The provided evidence consists of average AJ gains in Table 4.7 and the similarity-ratio score in Figure 4.9, but the latter is computed only at ground-truth visible locations and does not condition on prediction error or measure recovery after an erroneous frame. Please either temper the causal wording to 'reduces drift in the average case' or add an experiment that directly measures failure cascades, e.g., tracking accuracy conditioned on early prediction error or recovery rates after injected errors.
[Section 4.1.3, 'Inference-Time Memory Extension'; Figure 4.8] Linearly interpolating temporal positional embeddings from a training memory size of K=12 to inference sizes up to Ki=96 is an unvalidated extrapolation. Figure 4.8 shows that AJ is non-monotonic in Ki and degrades beyond Ki=48 on DAVIS and beyond Ki=80 on RGB-Stacking, which is at least consistent with the interpolation not fully preserving the training-time semantics. Since this mechanism is used to justify the claim that the model generalizes to arbitrarily long videos, please restrict the claim to the tested range or provide an analysis of the interpolation, for example by training with several K values and comparing interpolated embeddings against embeddings trained at the target size.

minor comments (4)

[Tables 4.1 and 4.2, footnote] The footnote sentence 'While online models process one frame at a time, enable frame-by-frame inference' is grammatically incomplete; it should read something like 'Online models process one frame at a time, enabling frame-by-frame inference.'
[Section 4.2.2, RGB-Stacking paragraph] The text says the model 'outperforms offline competitors' on RGB-Stacking, which is contradicted by Table 4.1, where CoTracker3 with video input achieves a higher AJ (74.0) than Track-On (71.4); please rephrase to list the specific offline models that are outperformed.
[Figure 4.8] The caption says memory sizes reported in Table 4.1 are marked with crosses, but the rendered figure does not show crosses; please either include the markers or clarify which points correspond to Table 4.1.
[Section 4.1.3, Eq. (4.8)] The notation [qinit, qt] for the concatenated query is not defined; please state the concatenation axis and the resulting dimension so that the memory-write operation is unambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Track-On's central claims rest on external benchmark evaluation and ablations, not on a self-referential derivation or fitted-input prediction.

full rationale

The paper's central claim—that Track-On achieves state-of-the-art online point tracking without future access—is supported by external benchmark numbers on TAP-Vid, RoboTAP, Dynamic Replica, BADJA, and PointOdyssey. The model is trained on synthetic TAP-Vid Kubric ground truth and evaluated on held-out public benchmarks; no test labels are used to define the method or fit its parameters. The memory modules in Section 4.1.3 are architectural and engineering choices: spatial memory stores features around the model's own past predictions and context memory stores past query embeddings, but these are not defined in terms of the target predictions, and the reported gains come from ablations on external metrics. The paper's self-citations to the author's prior work (Chapter 3's FoMo analysis and the Track-On conference paper) are not load-bearing in a circular sense: Chapter 3 is an independent empirical study re-used as motivation, and Chapter 4 reports its own experiments. The assertion in Section 4.1.3 that query updates conditioned on all previous predictions 'prevents error propagation' is not proven by a failure-cascade analysis, but that is an empirical robustness gap or unsupported assumption, not circularity—the memory does not encode the target labels or a fitted version of the evaluation metric. The similarity ratio score (Eq. 4.11) compares the memory-updated query against ground-truth features, which is a diagnostic, not a training target. No equation reduces to its own input, and no fitted parameter is renamed as a prediction. The paper is therefore self-contained against external benchmarks, and no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on standard supervised-learning assumptions and on two paper-specific design assumptions: the spatial memory write uses the model's own predictions as if correct, and positional embeddings interpolated at inference are assumed to remain valid. The manually chosen memory sizes, thresholds, and unreported loss weights are free parameters. No invented physical or latent entities are needed; the memory modules are implemented components with ablations.

free parameters (6)

Training memory size K = 12
Memory buffer size during training; inference is extended to 16-96. Figure 4.8 shows AJ varies with K, so the reported results depend on this choice.
Uncertainty threshold delta_u = 8 pixels
Used to label training predictions as uncertain in Section 4.1.2; manually set rather than learned.
Visibility threshold delta_v = not reported
Inference threshold for visible vs. occluded classification in Section 4.1.2; the thesis does not give the value, so reproducing the OA numbers requires tuning.
Total loss weight lambda = not reported
Weight balancing the six loss terms in Eq. 4.7; value is omitted from the experimental setup.
Top-k candidates in re-ranking = not reported
Number of highest-similarity patches fed to the re-ranking module in Section 4.1.2; not specified, though ablation shows re-ranking improves AJ.
Encoder stride S = 4
Stride of the ViT-Adapter feature map used for patch classification; ablation in Table 4.5 shows stride affects precision and AJ.

assumptions (5)

domain assumption TAP-Vid Kubric ground-truth trajectories and visibility labels are accurate and representative enough to train a general point tracker.
All training uses TAP-Vid Kubric (Section 4.2.1); if these synthetic annotations are biased, every downstream benchmark conclusion inherits the bias.
domain assumption Cosine similarity between DINOv2 patch features is a meaningful correspondence signal across appearance change and domain shift.
Patch classification in Section 4.1.2 and the Chapter 3 analysis both rely on feature cosine similarity; the thesis validates this empirically but does not prove it.
ad hoc to paper The model's own previous predictions can serve as correct locations for writing spatial memory.
Spatial memory write (Eq. 4.8) samples features around the predicted point; if a prediction is wrong, the memory stores off-target features. The thesis asserts this prevents error propagation but does not analyze failure cascades.
ad hoc to paper Linearly interpolating temporal positional embeddings from training size K=12 to inference sizes up to 96 preserves the semantics of the memory modules.
Inference-time memory extension relies on this interpolation (Section 4.1.3); only benchmark curves in Figure 4.8 support it, with no formal or worst-case analysis.
domain assumption Queried-first evaluation on TAP-Vid, RoboTAP, Dynamic Replica, BADJA, and PointOdyssey measures the intended real-world online tracking capability.
All SOTA claims are relative to these benchmarks at 256x256 resolution; if the protocol is not representative, the headline conclusion weakens.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Online Long-term Point Tracking in the Foundation Model Era." pith.science (2026). https://pith.science/paper/BITJJ7L3

@misc{pith2026250709217,
  author       = {Pith},
  title        = {Pith review of: Online Long-term Point Tracking in the Foundation Model Era},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BITJJ7L3}},
  note         = {Machine review of arXiv:2507.09217}
}

read the original abstract

Point tracking aims to identify the same physical point across video frames and serves as a geometry-aware representation of motion. This representation supports a wide range of applications, from robotics to augmented reality, by enabling accurate modeling of dynamic environments. Most existing long-term tracking approaches operate in an offline setting, where future frames are available to refine predictions and recover from occlusions. However, real-world scenarios often demand online predictions: the model must operate causally, using only current and past frames. This constraint is critical in streaming video and embodied AI, where decisions must be made immediately based on past observations. Under such constraints, viewpoint invariance becomes essential. Visual foundation models, trained on diverse large-scale datasets, offer the potential for robust geometric representations. While they lack temporal reasoning on their own, they can be integrated into tracking pipelines to enrich spatial features. In this thesis, we address the problem of long-term point tracking in an online setting, where frames are processed sequentially without access to future information or sliding windows. We begin by evaluating the suitability of visual foundation models for this task and find that they can serve as useful initializations and be integrated into tracking pipelines. However, to enable long-term tracking in an online setting, a dedicated design is still required. In particular, maintaining coherence over time in this causal regime requires memory to propagate appearance and context across frames. To address this, we introduce Track-On, a transformer-based model that treats each tracked point as a query and processes video frames one at a time. Track-On sets a new state of the art across seven public benchmarks, demonstrating the feasibility of long-term tracking without future access.

Figures

Figures reproduced from arXiv: 2507.09217 by the authors.

**Figure 1.1.** Tracking with Optical Flow. When short-term optical flow predictions are chained to achieve long-term tracking, they fail to recover from occlusions. In this example, all predictions (red) collapse onto the tree trunk, while the correct locations (green) are on the cycling child. video. This requires overcoming significant challenges such as appearance changes, occlusions, and complex motion. The task demands a hig… view at source ↗

**Figure 1.2.** Offline vs. Online Point Tracking. We propose an online model, tracking points frame-by-frame (right), unlike the dominant offline paradigm where models require access to all frames within a sliding window or the entire video (left). In contrast, our approach allows for frame-by-frame tracking in videos of any length. To capture temporal information, we introduce two memory modules: spatial memory, which tracks chan… view at source ↗

**Figure 2.1.** Iterative update operation introduced in RAFT [Teed and Deng, 2020]. illov et al., 2023]. The flexibility of these models allows them to perform different tasks, such as video object segmentation [Wang et al., 2023b, Wang et al., 2023c], estimating correspondence [Hedlin et al., 2023], object-centric learning [Aydemir et al., 2023b], discovering parts [Amir et al., 2021], autonomous driving [Barın et al., 2024], an… view at source ↗

Figures from the paper (14 more)

**Figure 2.2.** Figure 2.2: Overview of PIPs [Harley et al., 2022], iteratively updating the track feature and correspondence prediction in the temporal window. et al., 2024] took a different route, using DINO as a foundation for test-time optimization. TAPTRv2 [Li et al., 2024a], the successo…

**Figure 2.3.** Figure 2.3: LSTR [Xu et al., 2021] maintains causal representations by combining short-term and long-term memory modules to store and update information in a streaming video setting. introduce memory modules for more effective and efficient handling of complex tasks. For example…

**Figure 3.1.** Figure 3.1: Correlation Map. We compute a dense correlation map Ct by measuring cosine similarity between a query feature q and all spatial locations in the frame feature map Ft . The resulting map encodes the similarity distribution for a single point across the frame at time …

**Figure 3.2.** Figure 3.2: Probing the Correlation Map. We adopt lightweight convolutional heads, inspired by TAPNet [Doersch et al., 2022], to decode correlation maps into point predictions pˆt and occlusion logits oˆt . A shared encoder Φe extracts compact features, which are processed by tw…

**Figure 3.3.** Figure 3.3: Zero-Shot Tracking on TAP-Vid DAVIS. We show correlation maps Ct for query points from two videos using Stable Diffusion [Rombach et al., 2022], DINOv2 [Oquab et al., 2024], DINOv2-Reg [Darcet et al., 2023], and SAM [Kirillov et al., 2023]. We compute correlation map…

**Figure 4.1.** Figure 4.1: Overview. We introduce Track-On, a simple transformer-based method for online, frame-by-frame point tracking. The process involves three steps: (i) Visual Encoder, which extracts features from the given frame; (ii) Query Decoder, which decodes interest point queries…

**Figure 4.2.** Figure 4.2: Top-k Points. In certain cases, a patch with high similarity, though not the most similar, is closer to the ground-truth patch. The top-3 patch centers, ranked by similarity, are marked with dots, while the ground-truth is represented by a diamond. Re-ranking: We obs…

**Figure 4.3.** Figure 4.3: Re-ranking Module. The features around the top-k points (pˆ top t ) with the highest similarity are decoded using deformable attention to extract the corresponding top-k features (q top t ). These features are then fused with the decoded query q dec t using a transfo…

**Figure 4.4.** Figure 4.4: Offset Head. Starting with a rough estimation from patch classification (left), where lighter colors indicate higher correlation, we refine the prediction using the offset head (right). The selected patch center and the final prediction are marked by a blue dot and a…

**Figure 4.5.** Figure 4.5: Feature Drift. For the tracks shown below (start, middle, and final frames), the plot above illustrates the decreasing similarity between the features of the initial query and its correspondences over time, with the initial similarity indicated by horizontal dashed l…

**Figure 4.6.** Figure 4.6: Memory Modules. Spatial memory Ms t−1 (left) is used to update the initial query q init from the first frame to q init t on the current frame. The goal is to resolve feature drift by storing the content around the model’s predictions in previous frames. Context memor…

**Figure 4.7.** Figure 4.7: Efficiency. Inference speed (frames per second, FPS) vs. maximum GPU memory usage (in GB) where color represents the performance in AJ for different memory sizes (indicated near the nodes), while tracking approximately 400 points on the DAVIS dataset. leading to an o…

**Figure 4.8.** Figure 4.8: Memory Size. The effect of varying extended memory sizes during inference, on TAP-Vid DAVIS and TAP-Vid RGB-Stacking. consumer GPUs and real-time applications. Moreover, we observe that performance improves as the memory size increases up to K = 48, but declines beyo…

**Figure 4.9.** Figure 4.9: Similarity Ratio Score. The similarity ratio score ssr > 1 over frames for different tracks, demonstrates increased similarity with ground-truth location on the target frame when utilizing spatial memory [PITH_FULL_IMAGE:figures/full_fig_p062_4_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages

[1]

[Amir et al., 2021] Amir, S., Gandelsman, Y., Bagon, S., and Dekel, T. (2021). Deep ViT features as dense visual descriptors. arXiv preprint arXiv:2112.05814 . [Aydemir et al., 2023a] Aydemir, G., Akan, A. K., and G¨ uney, F. (2023a). ADAPT: Efficient multi-agent trajectory prediction with adaptation. In Proc. of the IEEE International Conf. on Computer V...

arXiv 2021
[2]

Qiao, Y. (2022b). Vision transformer adapter for dense predictions. In Proc. of the International Conf. on Learning Representations (ICLR) . [Cheng and Schwing, 2022] Cheng, H. K. and Schwing, A. G. (2022). XMem: Long- term video object segmentation with an atkinson-shiffrin memory model. In Proc. of the European Conf. on Computer Vision (ECCV) . [Cho et ...

arXiv 2022
[3]

Gupta, A., Heyward, J., Rocco, I., Goroshin, R., Carreira, J., and Zisserman, A. (2024). BootsTAP: Bootstrapped training for tracking-any-point. Proc. of the Asian Conf. on Computer Vision (ACCV) . Bibliography 55 [Doersch et al., 2023] Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A.,

work page 2024
[4]

Aytar, Y., Carreira, J., and Zisserman, A. (2023). TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proc. of the IEEE In- ternational Conf. on Computer Vision (ICCV) . [Dosovitskiy et al., 2015] Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. v. d., Cremers, D., and Brox, T. (2015)...

work page 2023
[5]

Rubinstein, M., Sun, D., Guibas, L., Johnson, J., and Jampani, V. (2024). Probing the 3D awareness of visual foundation models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Eun et al., 2020] Eun, H., Moon, J., Park, J., Jung, C., and Kim, C. (2020). Learn- ing to discriminate information for online action detection. In Proc. IE...

work page 2024
[6]

Tagliasacchi, A., and Yi, K. M. (2023). Unsupervised semantic correspondence using stable diffusion. In Advances in Neural Information Processing Systems (NeurIPS). [Horn and Schunck, 1981] Horn, B. K. and Schunck, B. G. (1981). Determining optical flow. In Artificial Intelligence (AI) . [Hu et al., 2022] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li...

work page 2023
[7]

Wang, L., and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In Proc. of the International Conf. on Learning Representations (ICLR) . [Jasinschi et al., 1998] Jasinschi, R. S., Veen, T. N., et al. (1998). Motion estimation methods for video compression—a review. Journal of the Franklin Institute . [Jin et al., 2025] Jin, L., Tucker, ...

arXiv 2022
[8]

C., Lo, W.-Y., et al

Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023). Bibliography 57 Segment anything. In Proc. of the IEEE International Conf. on Computer Vision (ICCV). [Kondratyuk et al., 2021] Kondratyuk, D., Yuan, L., Li, Y., Zhang, L., Tan, M.,

work page 2023

Show all 22 references

[9]

Brown, M., and Gong, B. (2021). MoViNets: Mobile video networks for effi- cient video recognition. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). [Lee et al., 2009] Lee, K.-Y., Chuang, Y.-Y., Chen, B.-Y., and Ouhyoung, M. (2009). Video stabilization usi...

2021
[10]

Zhang, L. (2024a). TAPTRv2: Attention-based position update improves tracking any point. In Advances in Neural Information Processing Systems (NeurIPS) . [Li et al., 2024b] Li, H., Zhang, H., Liu, S., Zeng, Z., Ren, T., Li, F., and Zhang, L. (2024b). TAPTR: Tracking any point ...

2024
[11]

Vedaldi, A. (2022). Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Neoral et al., 2024] Neoral, M., ˇSer` ych, J., and Matas, J. (2024). MFT...

2022
[12]

Huang, P.-Y., Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. (2024). DINOv2: Learning robust visual features without supervision. Transactions on Machine...

2024
[13]

Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In Proc. of the International Conf. on Machine learning (ICML) . [Ravi et al., 2025] Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryal...

2021
[14]

V., Carion, N., Wu, C.-Y., Girshick, R., Doll´ ar, P., and Feichtenhofer, C

Khedr, H., R¨ adle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Doll´ ar, P., and Feichtenhofer, C. (2025). SAM 2: Segment anything in images and videos. In Proc. of the Interna- tional Conf. on Learning Representati...

2025
[15]

Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Sand and Teller, 2008] Sand, P. and Teller, S. (2008). Particle video: Long-range motion estimation using point trajectories...

2022
[16]

Zhou, G., Hadsell, R., Agapito, L., and Scholz, J. (2023). RoboTAP: Tracking arbitrary points for few-shot visual imitation. In Proc. IEEE International Conf. on Robotics and Automation (ICRA) . [Wang et al., 2025] Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., an...

2023 arXiv
[17]

Sang, N. (2021). OadTR: Online action detection with transformers. In Proc. of the IEEE International Conf. on Computer Vision (ICCV) . [Wang et al., 2020] Wang, Z., Zheng, L., Liu, Y., Li, Y., and Wang, S. (2020). Towards real-time multi-object tracking. In Proc. of the Europ...

2021
[18]

Zhou, X. (2024). SpatialTracker: Tracking any 2D pixels in 3D space. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Xu et al., 2023] Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., and De Mello, S. (2023). Open-vocabulary panoptic segmentation with ...

2024 arXiv
[19]

Jin, X. (2024a). Flash-VStream: memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085 . [Zhang et al., 2024b] Zhang, J., Herrmann, C., Hur, J., Chen, E., Jampani, V., Sun, D., and Yang, M.-H. (2024b). Telling left from right: Identifying ...

2024 arXiv
[20]

Cole, F., Sun, D., and Yang, M.-H. (2025). Monst3r: A simple approach for Chapter 5: Conclusion 63 estimating geometry in the presence of motion. In Proc. of the International Conf. on Learning Representations (ICLR) . [Zhang et al., 2021b] Zhang, Z., Zhou, C., Ma, J., Lin, Z....

2025
[21]

Zhao, Z. (2021b). Learning to rehearse in long sequence memorization. In Proc. of the International Conf. on Machine learning (ICML) . [Zhao and Kr¨ ahenb¨ uhl, 2022] Zhao, Y. and Kr¨ ahenb¨ uhl, P. (2022). Real-time online video detection with temporal smoothing transformers....

2021
[22]

Nagrani, A., and Schmid, C. (2024). Streaming dense video captioning. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Zhu et al., 2021] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2021). Deformable DETR: Deformable transformers for end-to-e...

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

[Amir et al., 2021] Amir, S., Gandelsman, Y., Bagon, S., and Dekel, T. (2021). Deep ViT features as dense visual descriptors. arXiv preprint arXiv:2112.05814 . [Aydemir et al., 2023a] Aydemir, G., Akan, A. K., and G¨ uney, F. (2023a). ADAPT: Efficient multi-agent trajectory prediction with adaptation. In Proc. of the IEEE International Conf. on Computer V...

arXiv 2021

[2] [2]

Qiao, Y. (2022b). Vision transformer adapter for dense predictions. In Proc. of the International Conf. on Learning Representations (ICLR) . [Cheng and Schwing, 2022] Cheng, H. K. and Schwing, A. G. (2022). XMem: Long- term video object segmentation with an atkinson-shiffrin memory model. In Proc. of the European Conf. on Computer Vision (ECCV) . [Cho et ...

arXiv 2022

[3] [3]

Gupta, A., Heyward, J., Rocco, I., Goroshin, R., Carreira, J., and Zisserman, A. (2024). BootsTAP: Bootstrapped training for tracking-any-point. Proc. of the Asian Conf. on Computer Vision (ACCV) . Bibliography 55 [Doersch et al., 2023] Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A.,

work page 2024

[4] [4]

Aytar, Y., Carreira, J., and Zisserman, A. (2023). TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proc. of the IEEE In- ternational Conf. on Computer Vision (ICCV) . [Dosovitskiy et al., 2015] Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. v. d., Cremers, D., and Brox, T. (2015)...

work page 2023

[5] [5]

Rubinstein, M., Sun, D., Guibas, L., Johnson, J., and Jampani, V. (2024). Probing the 3D awareness of visual foundation models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) . [Eun et al., 2020] Eun, H., Moon, J., Park, J., Jung, C., and Kim, C. (2020). Learn- ing to discriminate information for online action detection. In Proc. IE...

work page 2024

[6] [6]

Tagliasacchi, A., and Yi, K. M. (2023). Unsupervised semantic correspondence using stable diffusion. In Advances in Neural Information Processing Systems (NeurIPS). [Horn and Schunck, 1981] Horn, B. K. and Schunck, B. G. (1981). Determining optical flow. In Artificial Intelligence (AI) . [Hu et al., 2022] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li...

work page 2023

[7] [7]

Wang, L., and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In Proc. of the International Conf. on Learning Representations (ICLR) . [Jasinschi et al., 1998] Jasinschi, R. S., Veen, T. N., et al. (1998). Motion estimation methods for video compression—a review. Journal of the Franklin Institute . [Jin et al., 2025] Jin, L., Tucker, ...

arXiv 2022

[8] [8]

C., Lo, W.-Y., et al

Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023). Bibliography 57 Segment anything. In Proc. of the IEEE International Conf. on Computer Vision (ICCV). [Kondratyuk et al., 2021] Kondratyuk, D., Yuan, L., Li, Y., Zhang, L., Tan, M.,

work page 2023