Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Trajectory geometry alone — positions, velocities, visibility — can say what moves, when, and where in a video, with no text query: TCAM retrieves and grounds multiple descriptions directly from dense point tracks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:04 UTC pith:NQKTODVE

load-bearing objection The paper's actual contribution—query-free multi-expression retrieval with trajectory grounding—is new and mostly solid; the arXiv title/abstract promise open-vocabulary generation that the method does not implement. the 4 major comments →

arxiv 2512.10607 v2 pith:NQKTODVE submitted 2025-12-11 cs.CV

Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation

Bishoy Galoaa , Sarah Ostadabbas This is my paper
classification cs.CV MSC 68T45
keywords motion-centric video understandingquery-free discoverydense point trajectoriestrajectory-conditioned captioningvideo-to-text retrievalspatial groundingcontrastive vision-language alignmentmulti-expression grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that dense point trajectories are not just geometry: once aligned with a contrastive vision–language embedding space, they become a semantic signal strong enough to drive automatic video understanding. The proposed system, TCAM, watches a video, discovers multiple simultaneous motion activities, produces a natural-language description for each, and spatially grounds each description to the specific trajectory that carries it — all without a user query or a region prompt. The load-bearing mechanism is motion-field attention, which encodes each track's position, velocity, and visibility into a fixed-length motion context compatible with text embeddings. On a motion-expression benchmark spanning over 50K clips, the system reports 58.4% video-to-text retrieval, 64.9 segmentation quality, and 4.8 relevant expressions per video at 84.7% precision, outperforming query-based dense captioning and grounding baselines. Because the architecture decodes motion descriptors into either segmentation masks or bounding boxes, the same learned representation transfers across task formats.

Core claim

TCAM's central discovery is that a set of dense point trajectories, after projection into the semantic space of a frozen vision–language model, can act as the bridge between motion and meaning. The tracker's raw outputs — positions, velocities, and visibility flags — are processed by motion-field attention into 512-dimensional motion descriptors that are directly comparable to text embeddings. At inference, the video's global embedding is matched against a precomputed bank of text descriptions, so multiple relevant expressions are retrieved without any query; a cross-attention step then scores each track against each expression, grounding language to motion. Training jointly optimizes global

What carries the argument

Motion Field Attention (MFA): a mechanism that converts each dense point trajectory — position, velocity, and visibility per frame — plus frozen vision features, into a 512-dimensional motion descriptor aligned to a contrastive vision–language space. Dual transformer encoders split spatial and temporal patterns, and a projection layer makes trajectories and text mutually comparable. MFA carries the argument: it is what turns movement geometry into semantics, enabling retrieval (via global video–text similarity) and grounding (via track–text cross-attention) from the same representation.

Load-bearing premise

The whole scheme stands or falls on whether a tracked point's position, velocity, and visibility, once projected into a vision-language embedding space, carry enough meaning to tell one action from another — and on whether retrieving from the dataset's own set of ground-truth phrases can count as open-vocabulary description.

What would settle it

Build a held-out test in which every expression in the evaluation videos is removed from the text bank; if discovery coverage drops toward the random baseline, the system is a lookup, not an open-vocabulary describer. Separately, measure grounding precision on a stress set of videos with frequent scene cuts: a sharp drop would confirm that track identity, not semantics, is the limiting factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Query-free discovery becomes practical: a single pass over a video yields several concurrent activity descriptions with reported 84.7% precision, so users can explore footage without knowing what to look for.
  • Motion descriptors act as a task-agnostic intermediate: the same learned representation decodes into segmentation masks or bounding boxes, so spatial grounding transfers across output formats.
  • Appearance becomes optional: trajectory dynamics alone carry enough semantics to describe actions under occlusion, camouflage, and fast motion where detectors fail.
  • Motion-centric retrieval is competitive with query-based pipelines: TCAM matches or exceeds dedicated grounding methods without being told what to describe.
  • Failure is bounded by track identity: scene cuts and viewpoint changes break trajectory continuity, so semantic grounding degrades exactly when the upstream tracker loses the points.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract's 'generative open-vocabulary' framing overstates the implemented system: inference retrieves from a precomputed bank of the dataset's own ground-truth expressions, so nothing is generated outside that vocabulary. A testable extension is to rebuild the bank from a large external caption corpus; the paper's bank-composition table suggests coverage and precision would keep climbing with
  • If MFA descriptors really are semantically discriminative, they could feed a language decoder directly instead of a retrieval step — turning TCAM into the generative captioner the abstract describes. That is a natural next experiment: replace the text-bank similarity with a small autoregressive head conditioned on the motion context.
  • Track identity is the weak point, not semantics. Fusing the trajectory stream with an object re-identification module, or adding temporal consistency across cuts, is a concrete fix that the paper names as future work; if it succeeds, the approach should extend from continuous wildlife or rafting scenes to cinematic content.
  • Because the representation is shared across tasks, the same motion descriptors could be probed for action-count estimation or anomaly detection without retraining, though the paper does not report such probes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TCAM is proposed as a query-free framework for video understanding that combines dense point trajectories (CoTracker3) with CLIP-based visual and text embeddings. The method consists of: (1) a Motion Field Attention module that encodes trajectory position/velocity/visibility and projects motion descriptors into a 512-D CLIP-aligned space; (2) a cross-attention mechanism that scores trajectory relevance for a text expression, used for spatial grounding; and (3) a query-free discovery mechanism that retrieves multiple expressions from a precomputed text bank by cosine similarity with a temporally pooled global video embedding. Training uses a global InfoNCE loss plus a spatial loss composed of diversity, sparsity, and ranking terms. The paper reports experiments on MeViS for retrieval, segmentation-based grounding, and multi-expression discovery, plus a cross-dataset grounding result on HC-STVG, and an extensive set of ablations. The title and metadata abstract advertise open-vocabulary generative spatiotemporal captioning, but the full text describes and evaluates a retrieval-and-grounding system with a fixed text bank.

Significance. If reframed accurately, TCAM addresses an interesting and relatively unexplored problem: discovering and spatially grounding multiple motion-related expressions in a video without requiring a user query. The use of dense trajectories as a spatial grounding medium, combined with a fixed text bank, is a plausible retrieval-based approach, and the reported gains over adapted dense-captioning baselines suggest the pipeline has merit. The paper also provides useful ablations (Tables 5–7) and releases code. However, the significance as stated is undermined by a mismatch between the advertised generative/open-vocabulary capability and the actual retrieval implementation, by the fact that the discovery pathway in Eq. (3) uses only global CLIP features rather than the motion descriptors, and by ambiguities in the text-bank construction and the HC-STVG protocol. These issues must be resolved before the contribution can be assessed at face value.

major comments (4)
  1. [Abstract / §3.3 / §5] The title and metadata abstract describe a "generative framework" that "describes each motion in open vocabulary" through a "language decoder." The actual method in §3.3 does not generate language: Eq. (3) retrieves expressions from a precomputed text bank B by cosine similarity, and the conclusion explicitly defers "open-vocabulary generalization" to future work. All evaluations (Tables 1, 2, 8) are retrieval-based. This is a load-bearing mismatch: the paper should either implement and evaluate a generative decoder or be reframed as a query-free retrieval-and-grounding system. The claims in the title/abstract must be revised to match the implemented system.
  2. [§3.3, Eq. (3), Figure 2] The query-free discovery similarity s_k in Eq. (3) is computed by temporally pooling the frame-level CLIP features {f_vis_t}. The MFA motion descriptors m_j do not enter this equation; they are used only in the spatial-grounding cross-attention of Eq. (2). The central claim that TCAM "decides what is moving" from trajectory dynamics is therefore not supported by the discovery pathway: retrieval is appearance-based via frozen CLIP features. This also makes the MFA ablation on V2T R@1 in Table 5 (51.7 vs 58.4) difficult to explain. Please clarify whether MFA influences e_video or correct the description/claims; otherwise restrict the motion-centric claim to spatial grounding.
  3. [§3.3, §4.3, Table 7] The construction of the text bank is underspecified. The text says "all unique descriptions in the dataset" (§3.3), which is ambiguous about whether validation/test expressions are included. If the bank includes the ground-truth expressions used to compute coverage and precision in Table 2 and Table 7, those numbers are inflated by exposing the answer key. The bank must be restricted to training-split expressions (or another clearly disjoint source), and the paper should state this explicitly. Additionally, because discovery is retrieval from a fixed bank, the system cannot caption an expression not present in the bank; this makes the "open-vocabulary" claim in the abstract even more problematic.
  4. [§4.1, Table 3] The HC-STVG transfer protocol is not described. It is unclear whether TCAM is fine-tuned on HC-STVG training splits or tested zero-shot after MeViS training, and how mask-based relevance scores are converted to bounding boxes for IoU metrics. Without this information, the cross-task generalization claim in Table 3 cannot be interpreted. Please provide the exact training/evaluation protocol for HC-STVG, including which splits are used and how the point/track relevance maps are turned into boxes.
minor comments (5)
  1. [Throughout] All tables report single point estimates without error bars or multiple-seed statistics. Given the relatively small MeViS set (2,006 videos) and the use of a pretrained tracker, the reported improvements should be accompanied by variance estimates or significance tests.
  2. [Abstract / §4.1] The metadata abstract states "On over 50K clips," but the body reports MeViS as containing 2,006 videos. This factual discrepancy should be reconciled.
  3. [Figure 1, §2] Typos: "chhallenging" in the Figure 1 caption, and "with with" in the Related Works section (in the description of spatio-temporal grounding methods). There are also several missing punctuation and spacing issues throughout.
  4. [Table 7] The "+ Generated variations" row does not specify how the variations are generated or whether they overlap with the ground-truth expressions. Provide details on the generation procedure and ensure no test-expression leakage.
  5. [References] References [4] and [5] both describe MeViS; one is an ICCV paper and the other is an arXiv preprint. This appears to be a duplicate or incorrect citation and should be cleaned up.

Circularity Check

0 steps flagged

No circularity: TCAM is a supervised retrieval-and-grounding system; the abstract's 'generative open-vocabulary' phrasing is unsupported by the body, but that is a claim-vs-implementation mismatch, not a derivation that reduces to its inputs.

full rationale

The paper's actual derivation chain is: dense trajectories (CoTracker3) plus frozen CLIP visual features are encoded by MFA into 512D motion descriptors; global video-text alignment (Eq. 5) trains pooled video features to match CLIP text embeddings; query-free output is produced by cosine retrieval from a precomputed text bank (Eq. 3); spatial grounding is supervised with ground-truth masks via a ranking loss (Eq. 7). Each component is trained or externally provided, and no output quantity is defined in terms of the target metric. The text bank is an explicit design input, not a fitted parameter masquerading as a prediction; evaluating retrieval coverage against the same expression set is a benchmark choice, and the paper openly frames the task as retrieval ('retrieves relevant expressions from a precomputed text bank'). The abstract's claim of a 'generative framework' with a 'language decoder' and 'open vocabulary' is not implemented in §3.3, and §5 concedes that open-vocabulary generalization is future work; this is a correctness/claim-support weakness, not circularity. §4.7 likewise states an inherited tracking limitation, which narrows but does not circularize the method. There are no self-citations used as load-bearing evidence and no uniqueness theorem invoked. The central mechanism is therefore independent of its conclusions; score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central method relies on external pretrained trajectory and CLIP models, on MeViS annotations, and on a text bank of ground-truth expressions. The system adds a small set of hand-set hyperparameters but no new physical or formal axioms.

free parameters (5)
  • spatial loss weight λ = 0.1
    Chosen to prioritize global alignment; ablation shows sensitivity (spatial-only λ=1 degrades J&F to 58.1).
  • InfoNCE temperature τ = 0.1
    Set by hand for the contrastive loss; no sensitivity analysis.
  • ranking loss margin γ = 0.2
    Margin in the spatial alignment loss; no sensitivity analysis.
  • discovery threshold / percentile = top 70th percentile; 'adaptive' variant
    Retrieval threshold is a post-hoc selection criterion; the adaptive variant yields the best discovery numbers in Table 2.
  • track grid density = 24×24 = 576 tracks
    Grid initialization for CoTracker3 is a design choice affecting motion coverage and is not swept.
axioms (4)
  • domain assumption CoTracker3 trajectory estimates preserve identity and capture action-relevant motion in continuous scenes.
    §3.1 processes p_{j,1:T} directly; if tracks drift or break, semantic grounding fails. The authors admit scene cuts break track identity (§4.7, Fig. 7).
  • domain assumption Frozen CLIP embeddings provide a semantic space in which trajectory-derived motion descriptors can be aligned by cosine similarity.
    §3.1–3.3 rely on this for Eq. 3 and Eq. 2; no evidence is given that trajectory descriptors are linearly simulable in CLIP space.
  • domain assumption MeViS ground-truth masks and expressions are sufficient supervision for generic motion semantics.
    §3.4 uses only existing segmentation annotations; generalization beyond dataset expressions is asserted but not demonstrated beyond two benchmarks.
  • ad hoc to paper The precomputed text bank B contains all expressions needed for inference.
    §3.3 Eq. 3 restricts output to B; this makes 'open-vocabulary' a retrieval among known captions, not generation.

pith-pipeline@v1.3.0-alltime-deepseek · 12076 in / 14635 out tokens · 150647 ms · 2026-08-03T17:04:57.035075+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation." pith.science (2026). https://pith.science/paper/NQKTODVE

@misc{pith2026251210607,
  author       = {Pith},
  title        = {Pith review of: Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQKTODVE}},
  note         = {Machine review of arXiv:2512.10607}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present TCAM (Track and Caption Any Motion), a generative framework that watches a video and with no text query and no region prompt decides what is moving, describes each motion in open vocabulary, locates it in time, and points to the exact trajectories that carry it. Two mature lines of work make this possible yet leave it unsolved: dense point trackers follow pixels with sub-object precision but emit no language, while video-language models produce fluent descriptions only when handed a query and only from clip-level features that cannot resolve which pixels move. Object-level captioners narrow the gap but still reason over detector boxes or masks, never reaching individual trajectories. TCAM couples tracking and language at point granularity through a Caption-Aware Resampler, where a small set of learnable queries cross-attends to dense point trajectory tokens and distills them into a fixed-length motion context that conditions a language decoder. The decoder generates an entire video's events in a single pass, each with a free-form caption, a start and end time, and a pointer to the trajectories it refers to, for sequential events and several subjects active at once. Training uses only existing segmentation annotations, with no extra event labeling, to supervise caption quality, pointer-mask alignment, and pointer diversity. On over 50K clips, TCAM outperforms dense video captioning baselines and matches dedicated, query-based grounding and point-tracking methods despite using no query, showing that trajectory-conditioned generation is a direct route to motion-driven video understanding.

Figures

Figures reproduced from arXiv: 2512.10607 by Bishoy Galoaa, Sarah Ostadabbas.

Figure 1
Figure 1. Figure 1: TCAM (Track and Caption Any Motion) automatically discovers and describes motion patterns without user queries. Our frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: TCAM pipeline overview. Training: Dense trajectories from the point tracker are processed by motion-field attention (MFA) and projected to CLIP space. CLIP Vision provides global features. Multi-head cross-attention aligns motion and text embeddings with unified loss (global InfoNCE + spatial BCE/diversity/sparsity). Inference (query-free): Motion descriptors and text embeddings share the joint 512D CLIP-a… view at source ↗
Figure 3
Figure 3. Figure 3: Text-Track Cross-Attention mechanism across consec [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results demonstrating TCAM’s query-free multi-expression discovery and spatial grounding. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Robust spatial grounding through occlusion. TCAM maintains accurate tracking and expression retrieval across five frames [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training progression of spatial grounding for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Failure case illustrating TCAM’s limitations during [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 5 linked inside Pith

  1. [1]

    V-jepa 2: Self- supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self- supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025. 2

    Pith/arXiv arXiv 2025

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. InICCV, 2021. 2, 8

    2021

  3. [3]

    Object segmentation by long term analysis of point trajectories

    Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. InECCV, 2010. 3

    2010

  4. [4]

    Mevis: A large-scale bench- mark for video segmentation with motion expressions.arXiv preprint, 2024

    Henghui Chen, Tsun-Hsuan Liu, Yutong Hu, Kuanchih Lin, Xiao Liu, and Peng Gao. Mevis: A large-scale bench- mark for video segmentation with motion expressions.arXiv preprint, 2024. 2

    2024

  5. [5]

    Mevis: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023. 5, 6, 7, 8

    2023

  6. [6]

    Tap-vid: A benchmark for tracking any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adri `a Re- casens, Lucas Smaira, Yusuf Aytar, Jo ˜ao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. InNeurIPS, 2022. 3

    2022

  7. [7]

    Tapir: Tracking any point with per-frame initialization and temporal refinement

    Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. InICCV, 2023. 2, 3

    2023

  8. [8]

    Step- former: Self-supervised step discovery and localization in instructional videos

    Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. Step- former: Self-supervised step discovery and localization in instructional videos. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 18952–18961, 2023. 2

    2023

  9. [9]

    Context-guided spatio-temporal video grounding

    Xin Gu, Heng Lan, and Jianzhuang Ma. Context-guided spatio-temporal video grounding. InCVPR, 2024. 3, 7

    2024

  10. [10]

    Harley, Zhaoyuan Fang, and Katerina Fragkiadaki

    Adam W. Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. InECCV, 2022. 2, 3

    2022

  11. [11]

    Pips++: Improved tracking through occlusions via extended point trajectories

    Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Pips++: Improved tracking through occlusions via extended point trajectories. InICCV, 2023. 3

    2023

  12. [12]

    A better use of audio-visual cues: Dense video captioning with bi-modal transformer

    Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer. In BMVC, 2020. 3

    2020

  13. [13]

    Videorag: Retrieval-augmented generation over video corpus.arXiv preprint arXiv:2501.05874, 2025

    Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval-augmented generation over video corpus.arXiv preprint arXiv:2501.05874, 2025. 2

    Pith/arXiv arXiv 2025

  14. [14]

    Embracing consistency: A one-stage approach for spatio- temporal video grounding

    Rui Jin, Wenjun Zhao, Zhedong Zheng, and Yang Wu. Embracing consistency: A one-stage approach for spatio- temporal video grounding. InNeurIPS, 2022. 2, 3, 7

    2022

  15. [15]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. InECCV, 2023. 2, 3

    2023

  16. [16]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. InICCV, 2024. 2, 3

    2024

  17. [17]

    Cospal: Co-optimizing spatio-temporal context prompting and adapt- ing for weakly supervised video grounding.arXiv preprint,

    Sachit Kumar, Haochen Wang, and Ming Tang. Cospal: Co-optimizing spatio-temporal context prompting and adapt- ing for weakly supervised video grounding.arXiv preprint,

  18. [18]

    Unsupervised object discovery and track- ing in video collections

    Suha Kwak, Minsu Cho, Ivan Laptev, Jean Ponce, and Cordelia Schmid. Unsupervised object discovery and track- ing in video collections. InICCV, 2015. 3

    2015

  19. [19]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 508:293–304, 2022

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 508:293–304, 2022. 1, 2, 3, 8

    2022

  20. [20]

    X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval. InACM Interna- tional Conference on Multimedia, pages 638–647, 2022. 1, 2, 3, 8

    2022

  21. [21]

    Delta: Dense efficient long-range 3d tracking for any video

    Tuan Duc Ngo, Peiye Hua, Shangzhe Xue, Yunzhi Zhang, and Xiaoke Zhang. Delta: Dense efficient long-range 3d tracking for any video. InICLR, 2025. 2, 3

    2025

  22. [22]

    Unsupervised discovery of actions in in- structional videos

    AJ Piergiovanni, Anelia Angelova, Alexander Toshev, and Michael S Ryoo. Unsupervised discovery of actions in in- structional videos. InBMVC, 2021. 3

    2021

  23. [23]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 3

    2021

  24. [24]

    Two-stream con- volutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream con- volutional networks for action recognition in videos. In NeurIPS, 2014. 3

    2014

  25. [25]

    Human-centric spatio-temporal video grounding with visual transformers

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. In TCSVT, 2021. 5, 6, 7

    2021

  26. [26]

    Human-centric spatio-temporal video grounding with visual transformers

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Tech- nology, 32(12):8238–8249, 2021. 2

    2021

  27. [27]

    Repre- sentation learning with contrastive predictive coding, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding, 2018. 4

    2018

  28. [28]

    Action recognition with trajectories

    Heng Wang and Cordelia Schmid. Action recognition with trajectories. InCVPR, 2013. 3

    2013

  29. [29]

    Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021. 1, 3

    Pith/arXiv arXiv 2021

  30. [30]

    Tracking everything everywhere all at once

    Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. InICCV, 2023. 2, 3

    2023

  31. [31]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. InICCV, 2021. 2, 3, 7 9

    2021

  32. [32]

    Language as queries for referring video object segmen- tation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmen- tation. InCVPR, 2022. 3, 5, 7

    2022

  33. [33]

    Spatialtracker: Tracking any 2d pixels in 3d space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406–20417, 2024. 2

    2024

  34. [34]

    Spatialtrackerv2: 3d point tracking made easy.arXiv preprint arXiv:2507.12462, 2025

    Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy.arXiv preprint arXiv:2507.12462, 2025. 2

    Pith/arXiv arXiv 2025

  35. [35]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. InEMNLP, 2021. 1, 3

    2021

  36. [36]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Ze- huan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. InCVPR, 2023. 3, 5, 7

    2023

  37. [37]

    Tubedetr: Spatio-temporal video ground- ing with transformers

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio-temporal video ground- ing with transformers. InCVPR, 2022. 2, 3, 7

    2022

  38. [38]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. InCVPR,

  39. [39]

    Tapip3d: Tracking any point in persistent 3d geome- try.arXiv preprint arXiv:2504.14717, 2025

    Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragki- adaki. Tapip3d: Tracking any point in persistent 3d geome- try.arXiv preprint arXiv:2504.14717, 2025. 2

    arXiv 2025

  40. [40]

    Where does it exist: Spatio-temporal video grounding for multi-form sentences

    Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10668–10677, 2020. 2

    2020

  41. [41]

    Video- text prompting for weakly supervised spatio-temporal video grounding

    Mengyao Zhao, Yiming Li, and Zhenlin Zhang. Video- text prompting for weakly supervised spatio-temporal video grounding. InEMNLP, 2024. 3

    2024

  42. [42]

    Unsupervised learning from video to detect foreground objects in single images

    Nannan Zhao, Xuesheng Wen, and Jie Yang. Unsupervised learning from video to detect foreground objects in single images. InSymmetry, 2021. 3

    2021

  43. [43]

    Tapnext: Tracking any point as next token predic- tion.arXiv preprint arXiv:2504.05579, 2025

    Ivan Zholus, Jia-Bin Yang, Tsung-Han Xue, and Yung-Yu Chuang. Tapnext: Tracking any point as next token predic- tion.arXiv preprint arXiv:2504.05579, 2025. 3

    Pith/arXiv arXiv 2025

  44. [44]

    Dense video object captioning from disjoint super- vision

    Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid. Dense video object captioning from disjoint super- vision. InInternational Conference on Learning Represen- tations (ICLR), 2025. 3 10

    2025