Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Trajectory tokens learned end-to-end outperform patch tokens in video models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:34 UTC pith:O5FPJH7I

load-bearing objection Solid empirical tokenizer paper wrapped in an overstated abstract: the 'duration-independent' headline is contradicted by the chunked inference, but the method and controls are worth a real look. the 4 major comments →

arxiv 2602.22779 v3 pith:O5FPJH7I submitted 2026-02-26 cs.CV

TrajTok: Learning Trajectory Tokens enables better Video Understanding

Chenhao Zheng , Jieyu Zhang , Jianing Zhang , Weikai Huang , Ashutosh Kumar , Quan Kong , Oncel Tuzel , Chun-Liang Li
show 1 more author
Ranjay Krishna
This is my paper
classification cs.CV
keywords video tokenizationtrajectory tokensobject-centric representationvideo transformerCLIPvideo-language modeltoken efficiencyvideo understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video models usually chop frames into fixed space–time patches, producing thousands of redundant tokens that grow with video length. This paper tries to establish that a better unit is the object trajectory: a compact token that follows an object through space and time, so token count tracks semantic complexity rather than duration. It introduces TrajTok, an end-to-end differentiable tokenizer whose segmenter clusters pixels in a single forward pass, and shows that a CLIP-style encoder built on it (TrajViT2) beats patch-based and token-merging baselines across classification and retrieval while matching their efficiency. The same tokenizer also works as a probing head for pretrained encoders and as a connector for video language models, with the largest gains in long-video QA. If right, this would make video transformers cheaper and more accurate, and give object-centric structure for free at pretraining time.

Core claim

Trajectory tokenization can be learned end-to-end rather than supplied by an external segmentation-and-tracking pipeline, and that learnability drives the gains. TrajTok's segmenter clusters pixels across space and time in one forward pass, producing a variable number of soft masks that a trajectory encoder turns into tokens; joint training with the CLIP objective reshapes grouping granularity for the task. As TrajViT2 (trained from scratch) it reports +4.8% on Kinetics-400 and +4.1% on Something-Something-v2 over a standard video ViT, higher retrieval recall, and better data scaling at roughly the inference cost of token-merging baselines. As TrajAdapter or TrajVLM it also improves probing

What carries the argument

The load-bearing mechanism is the segmenter–encoder pair. The segmenter builds a dense feature map with a lightweight ConvNeXt patch encoder, runs 128 learnable queries through Perceiver cross-attention with rotary temporal position embeddings, and softmaxes over queries to produce soft segmentation masks; empty queries drop out, so trajectory count varies with scene complexity. The encoder aggregates features by these masks into proposals, refines them with hard-masked cross-attention so each token reads only its own region, and can emit 1, 2, or 4 tokens per trajectory via Matryoshka-style queries with Fourier-initialized offsets. Training combines Dice+Focal segmentation loss on pseudo-gr

Load-bearing premise

Everything rests on the quality of the pseudo-ground-truth trajectory masks that supervise the segmenter, which come from an external segmentation-and-tracking pipeline filtered only by crude coverage and object-count thresholds; if those masks are systematically wrong or out of distribution, the trajectory proposals inherit the errors and the downstream objective cannot fully correct them because the segmenter is frozen or its patch features are detached.

What would settle it

Control the pseudo-label source: train identical TrajViT2 models with (a) the paper's external-pipeline masks, (b) the same masks randomly permuted across queries, and (c) no segmentation loss at all. If the classification and retrieval gains over ViT3D persist under (b) or (c), the trajectory grouping itself is not the cause; if they vanish only under (b), the specific label content matters. The paper's ablations remove loss components but never vary the label source, so this comparison is the missing experiment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Video token count stops growing linearly with length: 128 frames yield at most 128 trajectory tokens, and the same tokenizer handles variable durations in one pass.
  • Downstream objectives actively reshape segmentation granularity, so the same tokenizer can favor dancer body parts for fine action recognition or a whole-dancer token for group-formation tasks.
  • TrajViT2 keeps a consistent accuracy margin over patch-based ViT3D as pretraining data grows from 1M to 8M clips, where the fixed external-pipeline predecessor lost its edge.
  • The frozen pretrained segmenter transfers as a probing head, improving K400/SSv2 accuracy of frozen pretrained video backbones without full fine-tuning.
  • As a VLM connector, TrajTok gives the larger gains on long-video QA benchmarks, suggesting object-trajectory tokens help long-range reasoning more than short-clip recognition.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If trajectory tokens indeed carry object structure, a natural next step is to use them as the visual input for autoregressive video generation or world models, where compact object-centric latents could make temporal prediction easier; the paper does not test generation.
  • The pseudo-label dependency suggests a testable extension: replace the external masks with a purely self-supervised grouping objective (e.g., reconstruction or CLIP-based consistency) and measure how much of the gain survives; the paper only ablates loss components, not the label source.
  • Because the segmenter sees space and time uniformly, the same module could be applied to image-only inputs with no changes, which the paper already does for image retrieval and probing; a deeper implication is that trajectory tokens may serve as a unified image/video tokenizer for multimodal models.
  • The ImageNet dip the authors report hints at a boundary: in single-object, centered-image domains, trajectory grouping produces too few tokens, so gains may concentrate in multi-object, dynamic scenes; this could be probed by varying scene statistics systematically.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. TrajTok proposes a fully differentiable trajectory tokenizer that uses a lightweight segmenter to group video pixels into object-trajectory tokens via learnable Perceiver queries, followed by a trajectory encoder with adaptive per-trajectory token counts. The paper evaluates the tokenizer in three configurations: TrajViT2 (a CLIP-style video encoder trained from scratch), TrajAdapter (a probing head on frozen pretrained backbones), and TrajVLM (a LLaVA-style video-language connector). Reported results include consistent gains over patch-based and token-merging baselines, e.g., +4.8% on Kinetics-400 and +4.1% on SSv2 for TrajViT2, stronger retrieval numbers, improved data-scaling trends, and better long-video QA performance in TrajVLM. The central conceptual claim is that token count is decoupled from video duration and adapts only to semantic complexity.

Significance. If the empirical results are reproducible, the paper makes a meaningful contribution: it replaces slow external trajectory-extraction pipelines with a learned, end-to-end differentiable tokenizer and demonstrates gains across three disparate use cases. The paper also includes useful control baselines: a Perceiver module without trajectory priors, PatchVLM variants with matched token counts, and scaling curves against TrajViT and ViT3D. The segmenter is analyzed both qualitatively and quantitatively, including a scaled-up variant evaluated on ViPEntitySeg. However, the central duration-independence claim is contradicted by the implementation, and the pseudo-label dependence plus lack of uncertainty quantification prevent the current version from being accepted as-is.

major comments (4)
  1. [Abstract, §3.1, §4.3] The claim that TrajTok 'dynamically adapts its token granularity to semantic complexity, independent of video duration' is contradicted by the described chunking mechanism. Section 3.1 states that 'long videos are divided into temporal chunks,' and Section 4.3 confirms that TrajVLM processes 128 frames by truncating them into 16-frame clips, each proposing a maximum of 128 tokens. Thus the maximum token count is 8T for a T-frame video, scaling linearly with duration. Figure 6 also shows FLOPs increasing with frame count. The mechanism caps tokens per chunk, not per video. The paper should either revise the headline claim or redesign the tokenizer so that token count is truly independent of duration; as written, this is a load-bearing conceptual error.
  2. [§3.1, §7.1] The segmenter is supervised by pseudo-ground-truth trajectory masks from the TrajViT pipeline, filtered by coarse coverage and object-count thresholds. Since patch-feature gradients are detached before the Perceiver layers and the segmenter is frozen in TrajAdapter/TrajVLM, the downstream CLIP or QA objective cannot correct systematic grouping errors. The paper's own Section 8 acknowledges that the segmenter misses small objects, over-merges backgrounds, and produces imprecise boundaries. The dependency on this external pseudo-label source is therefore load-bearing. Please add robustness experiments, e.g., varying the filtering thresholds, corrupting pseudo-labels, or comparing a randomly initialized/frozen segmenter against a finetuned one, to show the method does not inherit TrajViT's failure modes.
  3. [Tables 1–5, Figs. 5 and 8] All experimental tables and scaling/figure curves are reported as single runs with no error bars, standard deviations, or seeds. Several differences are small enough that run-to-run noise could matter (e.g., Table 4: VEQ 42.3 vs. 42.1 with cross-entropy; Table 3: TrajAdapter 82.0 vs. 82.4 for 1 vs. 2 tokens/trajectory). The paper should report mean±std over multiple seeds or at least provide error bars on the main scaling and VLM comparisons. Releasing code and checkpoints would also materially strengthen the reproducibility of the empirical claims.
  4. [§4.2, Table 3] The Perceiver probing baseline is described as having 'identical size and number of learnable queries as our trajectory encoder but without trajectory priors,' yet in TrajAdapter the segmenter is pretrained on a large corpus (8M videos and 15M images) and frozen, while the trajectory encoder is trained jointly. The Perceiver baseline appears to be trained from scratch without this pretraining. This conflates architecture with pretraining data. A fairer control would train a Perceiver with access to the same pretrained segmenter features (e.g., a random or learned grouping initialized from the segmenter), or train the segmenter from scratch in the probing setup, to isolate the contribution of trajectory priors.
minor comments (4)
  1. [Throughout] Typos and inconsistencies: 'Segementer' in Section 7 heading, 'initializatoin' in Section 5, 'apple-to-apple' in Section 4.3, 'bur decreased' in Section 4.3, and inconsistent use of 'TrajViT-2' vs. 'TrajViT2' in Tables 1–2.
  2. [§4.1 vs. Fig. 5] Section 4.1 says the training corpus contains 4M video clips randomly sampled from Panda-70M, while Figure 5 describes partitioning Panda-10M. Please clarify which dataset was used for the main results and which for the scaling curves.
  3. [Figure 6] The caption says ViT3D has 'quadratic scaling,' but the text says 'quadratic scaling of patch-based ViT3D.' Please specify whether this refers to full attention over the spatiotemporal token sequence; if the figure plots actual FLOPs, a short explanation of the measurement would help.
  4. [§4.1] The phrase 'standard video ViT' is used for ViT3D with 16×16×2 patches. It would be useful to state the patch temporal stride and the resulting token count explicitly, to make the efficiency comparison in Figure 6 interpretable.

Circularity Check

0 steps flagged

No significant circularity: the central claims rest on external benchmark evaluations, not on equations or fits that reduce to their own inputs.

full rationale

The paper's derivation chain is: a lightweight segmenter computes soft masks via softmax query-feature similarity (Eq. 1); the trajectory encoder aggregates patch features with those masks (Eq. 2); and the full system is co-trained with a CLIP objective plus a segmentation loss. None of these steps defines its output in terms of the downstream accuracies it predicts, and the reported gains over ViT3D, TrajViT, TokenLearner, RLT, etc. are measured on held-out classification, retrieval, and QA benchmarks. The main author-overlap dependency is that pseudo-ground-truth masks are 'generated via the TrajViT [84] pipeline' (Section 7.1), but that is a training-supervision source rather than a derived result being renamed as a prediction; TrajViT is also an external baseline that TrajViT2 outperforms in the paper's own tables. The duration-independence claim appears inconsistent with the chunked inference described in Section 4.3, where token count grows linearly with the number of 16-frame chunks, but that is a correctness/architectural issue, not circularity. No load-bearing step reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests mainly on the quality of pseudo-labeled trajectories and the hypothesis that object-level grouping helps downstream understanding. There are a few hand-set hyperparameters but no fitted physical constants or invented entities; the main epistemic load is on external pseudo-label quality and transfer.

free parameters (4)
  • Number of learnable segmenter queries Nq = 128
    Fixed prototype count; controls the maximum/typical number of trajectory tokens; chosen by validation, not derived.
  • Adaptive tokens per trajectory n = 1, 2, or 4
    Compute-budget hyperparameter; randomly sampled during training in the Matryoshka-style scheme; selected by resource constraints.
  • Segmentation loss weighting (Dice + Focal) = not reported
    Paper states the Dice+Focal combination 'yields the best downstream understanding' but never reports the actual loss weights; empirically tuned.
  • Pseudo-label filtering thresholds = coverage >= 80%, object count >= 10
    Hand-set filters that determine the segmenter training set (2.5M images, 2.0M videos); no sensitivity analysis is given.
axioms (4)
  • domain assumption Pseudo-ground-truth trajectory masks produced by the TrajViT pipeline are accurate enough to supervise the segmenter.
    Section 7.1: annotations use DirectSAM + SAM2 + heuristic merging, with only crude coverage/object-count filters; no human verification.
  • domain assumption Grouping video tokens by object trajectories improves downstream understanding over patch tokens.
    Motivated by TrajViT [84] and human perceptual principles (Sections 1-2); not independently re-derived in this paper.
  • domain assumption Co-training with CLIP and segmentation objectives reshapes granularity beneficially.
    Figure 3 is qualitative; no metric directly shows that the CLIP objective changes masks in a way that causes the observed gains, though Table 4 shows segmentation loss components matter.
  • domain assumption A segmenter pretrained on pseudo-labels transfers to new domains without adaptation.
    TrajAdapter and TrajVLM keep the segmenter frozen (Sections 4.2, 4.3); this relies on the generalization of the universal segmenter.

pith-pipeline@v1.3.0-alltime-deepseek · 19172 in / 10946 out tokens · 96635 ms · 2026-08-02T20:34:22.744333+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of TrajTok: Learning Trajectory Tokens enables better Video Understanding." pith.science (2026). https://pith.science/paper/O5FPJH7I

@misc{pith2026260222779,
  author       = {Pith},
  title        = {Pith review of: TrajTok: Learning Trajectory Tokens enables better Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5FPJH7I}},
  note         = {Machine review of arXiv:2602.22779}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.

Figures

Figures reproduced from arXiv: 2602.22779 by Ashutosh Kumar, Chenhao Zheng, Chun-Liang Li, Jianing Zhang, Jieyu Zhang, Oncel Tuzel, Quan Kong, Ranjay Krishna, Weikai Huang.

Figure 1
Figure 1. Figure 1: (a) Traditional video tokenization splits a video into space-time patches, introducing large number of redundant tokens. (b) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the TrajTok architecture. TrajTok comprises a trajectory segmenter and a trajectory encoder. The segmenter proposes trajectory masks for all objects in an image or video within a single forward pass. The encoder then aggregates raw video pixels or encoded visual features (parameterized by f in the figure) according to these masks to produce trajectory tokens. The number of tokens per trajectory… view at source ↗
Figure 3
Figure 3. Figure 3: Training with downstream understanding tasks reshapes the segmentation granularity. We visualize the trajectory masks produced by our segmenter when trained with only segmentation supervision versus jointly with segmentation and CLIP objectives. The CLIP objective reshapes the segmentation granularity, producing finer foreground object masks while merging background regions. 3. TrajTok We aim to design an … view at source ↗
Figure 4
Figure 4. Figure 4: TrajTok is a versatile module applicable across pre￾training, feature adaptation, and finetuning stages. We demon￾strate its use in three scenarios: TrajViT2, which trains a visual encoder from scratch; TrajAdapter, which adapts pretrained fea￾tures for downstream tasks; and TrajVLM, which uses TrajTok as a connector in LLaVA-style large vision–language models. Mhard k,t,i,j = 1. This refinement recovers f… view at source ↗
Figure 6
Figure 6. Figure 6: Test time FLOPs compari￾son under different frame numbers. clips. We train TrajViT2, TrajViT, and ViT3D on all three scales and report their performance on video benchmarks. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trajectory masks produced by our segmenter vs. by TrajViT pipeline. While our segmenter produces coarser masks and may miss very small objects, it demonstrates strong semantic grouping ability that is sufficient for downstream understanding tasks. recognition accuracy on the Kinetics-400 and Something￾Something V2 (SSv2) benchmarks. Videos are uniformly sampled to 16 frames and sent to segmenter in one for… view at source ↗
Figure 8
Figure 8. Figure 8: VideoQA results for TrajTok applying to large vision-language model. VLM with TrajTok as connector (TrajVLM) notably outperforms patch pooling baseline (PatchVLM) in long-video benchmarks, while the performance is mixed for short-video benchmark. Module Variation VEQ (%) STQ (%) Retrieval (R@5) Default Architecture 42.3 70.1 22.1 Backbone no hierarchical features 39.3 (↓ 3.0) 66.2 (↓ 3.9) 19.2 (↓ 2.9) Outp… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative Examples of the trajectory masks produced by our segmenter. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Tracking and Understanding Object Transformations

    cs.CV 2026-07 conditional novelty 6.0

    FluxGraph detects object transformations reactively via SAM2's multi-mask disagreement, cutting TubeletGraph's inference cost by 3.3–10.7x with comparable tracking and state-graph quality.

Reference graph

Works this paper leans on

85 extracted references · 27 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Vivit: A video vi- sion transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vi- sion transformer. InICCV, 2021. 1, 2, 5 Table 8.Datasets used for training TrajVLM under the “fi- nal” mixture.This mixture combines a large set of academic VideoQA datasets, temporal reasoning datasets, and synthetic cap- tioning/QA corpora. C...

    2021

  2. [2]

    V-jepa 2: Self-supervised video models enable understanding, predic- tion and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, et al. V-jepa 2: Self-supervised video models enable understanding, predic- tion and planning.arXiv preprint arXiv:2506.09985, 2025. 1, 6

    Pith/arXiv arXiv 2025

  3. [3]

    V-jepa: Video joint-embedding predictive archi- tecture.arXiv preprint arXiv:2404.08471, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nico- las Ballas. V-jepa: Video joint-embedding predictive archi- tecture.arXiv preprint arXiv:2404.08471, 2024. 1

    Pith/arXiv arXiv 2024

  4. [4]

    Is 11 space-time attention all you need for video understanding? InICML, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is 11 space-time attention all you need for video understanding? InICML, 2021. 1, 2

    2021

  5. [5]

    Flexivit: One model for all patch sizes

    Lukas Beyer, Xiaohua Zhai, Alexander Kolesnikov, Joan Puigcerver, Alexander Steiner, Daniel Keysers, Barret Zoph, and Neil Houlsby. Flexivit: One model for all patch sizes. In CVPR, 2023. 1

    2023

  6. [6]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InICLR, 2023. 1, 2

    2023

  7. [7]

    Monet: Unsupervised scene decompo- sition and representation

    Chris Burgess, Hyunjik Kim, Loic Matthey, Nick Wat- ters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decompo- sition and representation. InarXiv:1901.11390, 2019. 2

    Pith/arXiv arXiv 1901

  8. [8]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 5, 6, 8

    2015

  9. [9]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021. 5, 9

    2021

  10. [10]

    Subobject-level image tokeniza- tion.arXiv preprint arXiv:2402.14327, 2024

    Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang, and Pascale Fung. Subobject-level image tokeniza- tion.arXiv preprint arXiv:2402.14327, 2024. 2, 9

    Pith/arXiv arXiv 2024

  11. [11]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13320–13331, 2024. 5, 9

    2024

  12. [12]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 2, 3

    2022

  13. [13]

    Joonmyung Choi, Sanghyeok Lee, Jaewon Chu, Minhyuk Choi, and Hyunwoo J. Kim. vid-tldr: Training free token merging for light-weight video transformer. InCVPR, 2024. 2

    2024

  14. [14]

    Kitani, and L´aszl´o A

    Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Ni- inuma, Kris M. Kitani, and L´aszl´o A. Jeni. Don’t look twice: Faster video transformers with run-length tokenization. In NeurIPS, 2024. 1, 2, 5

    2024

  15. [15]

    Jeni, and Kris M

    Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, L´aszl´o A. Jeni, and Kris M. Kitani. Accelerating vi- sion transformers with adaptive patch sizes.arXiv preprint arXiv:2510.18091, 2025. 1, 2

    Pith/arXiv arXiv 2025

  16. [16]

    Molmo 2: Open weights and open data for state-of- the-art video and image models, 2026

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Rohun Tripathi, Sangho Lee, Mohammadreza Salehi, Ja- son Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Ali Farhadi, and Ranjay Krishna. Molmo 2: Open weights and open data for state-of- the-art video an...

    Pith/arXiv arXiv 2026

  17. [17]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.arXiv preprint arXiv:2409.17146, 2024. 1, 7, 11

    Pith/arXiv arXiv 2024

  18. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InProceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2009. 5, 6

    2009

  19. [19]

    Adaptive token sampling for efficient vision transform- ers

    Mohsen Fayyaz, Soroush Abbasi Koohpayegani, and J ¨urgen Gall. Adaptive token sampling for efficient vision transform- ers. InECCV, 2022. 2

    2022

  20. [20]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recogni- tion Workshop (CVPRW), 2004. 5, 6

    2004

  21. [21]

    Video-mme: A comprehensive eval- uation benchmark of multi-modal llms in video analysis

    Chaoyou Fu et al. Video-mme: A comprehensive eval- uation benchmark of multi-modal llms in video analysis. arXiv:2405.21075, 2024. 7

    Pith/arXiv arXiv 2024

  22. [22]

    Dat- acomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Sys- tems, 36:27092–27112, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets.Advances in Neural Information Processing Sys- tems, 36:27092–27112, 2023. 9

    2023

  23. [23]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Sebastian Westphal, Heuna Kim, Valentin Haenel, Moritz Fr ¨uh, Peter Yianilos, Marcel Mueller-Freitag, et al. The “something something” video database for learning and evaluating visual common sense. InProceedings of the IEEE International Conference on Computer Vision (ICCV), 2...

    2017

  24. [24]

    Multi-object representation learning with iterative variational inference

    Klaus Greff, Raphael L Kaufman, and et al. Multi-object representation learning with iterative variational inference. InICML, 2019. 2

    2019

  25. [25]

    Prunevid: Visual to- ken pruning for efficient video large language models.arXiv preprint arXiv:2412.16117, 2024

    Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual to- ken pruning for efficient video large language models.arXiv preprint arXiv:2412.16117, 2024. 2, 7

    Pith/arXiv arXiv 2024

  26. [26]

    Perceiver: General perception with it- erative attention.ICML, 2021

    Andrew Jaegle et al. Perceiver: General perception with it- erative attention.ICML, 2021. 2, 3

    2021

  27. [27]

    Perceiver io: A general architecture for structured inputs & outputs.arXiv:2107.14795, 2021

    Andrew Jaegle et al. Perceiver io: A general architecture for structured inputs & outputs.arXiv:2107.14795, 2021. 2

    Pith/arXiv arXiv 2021

  28. [28]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, et al. The kinetics human action video dataset. InarXiv preprint arXiv:1705.06950, 2017. 5, 6

    Pith/arXiv arXiv 2017

  29. [29]

    Token fusion: Bridging the gap between token pruning and token merging

    Minchul Kim et al. Token fusion: Bridging the gap between token pruning and token merging. InWACV, 2024. 2

    2024

  30. [30]

    Elsayed, Aravindh Mahen- dran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jon- schkowski, Alexey Dosovitskiy, and Klaus Greff

    Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahen- dran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jon- schkowski, Alexey Dosovitskiy, and Klaus Greff. Condi- tional object-centric learning from video. InICLR, 2022. 2

    2022

  31. [31]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 2 12

    2023

  32. [32]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 5, 6

    2009

  33. [33]

    Trokens: Semantic- aware relational trajectory tokens for few-shot action recog- nition

    Pulkit Kumar, Shuaiyi Huang, Matthew Walmer, Sai Saketh Rambhatla, and Abhinav Shrivastava. Trokens: Semantic- aware relational trajectory tokens for few-shot action recog- nition. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 13544–13556, 2025. 2

    2025

  34. [34]

    Matryoshka representation learning.Advances in Neu- ral Information Processing Systems, 35:30233–30249, 2022

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning.Advances in Neu- ral Information Processing Systems, 35:30233–30249, 2022. 2, 4

    2022

  35. [35]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7, 8

    2024

  36. [36]

    Not all patches are what you need: Expediting vi- sion transformers via token reorganizations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, and Jue Wang. Not all patches are what you need: Expediting vi- sion transformers via token reorganizations. InICLR, 2022. 2

    2022

  37. [37]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. 5, 6

    2014

  38. [38]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 4

    2017

  39. [39]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 5, 7

    2023

  40. [40]

    Video-xl-pro: Reconstructive token compres- sion for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Reconstructive token compres- sion for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025. 2

    Pith/arXiv arXiv 2025

  41. [41]

    Tempcompass: Do video llms really under- stand videos? InFindings of the Association for Computa- tional Linguistics (ACL), 2024

    Yutao Liu et al. Tempcompass: Do video llms really under- stand videos? InFindings of the Association for Computa- tional Linguistics (ACL), 2024. 7, 8

    2024

  42. [42]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 11976–11986,

  43. [43]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. InCVPR,

  44. [44]

    Object- centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, and et al. Object- centric learning with slot attention. InNeurIPS, 2020. 2

    2020

  45. [45]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InInternational Conference on Learning Representations (ICLR), 2019. 11

    2019

  46. [46]

    Videoeval-pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025

    Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation.arXiv preprint arXiv:2505.14640, 2025. 7

    Pith/arXiv arXiv 2025

  47. [47]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. InAdvances in Neu- ral Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. 7

    2023

  48. [48]

    Spformer: Enhancing vision transformer with superpixel representation.arXiv preprint arXiv:2401.02931, 2024

    Jieru Mei, Liang-Chieh Chen, Alan Yuille, and Cihang Xie. Spformer: Enhancing vision transformer with superpixel representation.arXiv preprint arXiv:2401.02931, 2024. 1

    Pith/arXiv arXiv 2024

  49. [49]

    Large-scale video panoptic seg- mentation in the wild: A benchmark

    Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yun- chao Wei, and Yi Yang. Large-scale video panoptic seg- mentation in the wild: A benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21033–21043, 2022. 10

    2022

  50. [50]

    Atten- tion bottlenecks for multimodal fusion

    Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. Atten- tion bottlenecks for multimodal fusion. InNeurIPS, 2021. 2

    2021

  51. [51]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica P ˘atr˘aucean, Lucas Smaira, Ankush Gupta, et al. Per- ception test: A diagnostic benchmark for multimodal video models. InAdvances in Neural Information Processing Sys- tems (NeurIPS), Datasets and Benchmarks Track, 2023. 7, 8

    2023

  52. [52]

    Plummer, Liwei Wang, Christopher Cervantes, Juan C

    Bryan A. Plummer, Liwei Wang, Christopher Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. InPro- ceedings of the IEEE International Conference on Computer Vision (ICCV), 2015. 5, 6

    2015

  53. [53]

    Pylyshyn and Roger W

    Zenon W. Pylyshyn and Roger W. Storm. Tracking multiple independent targets: Evidence for a parallel tracking mecha- nism.Spatial Vision, 3(3):179–197, 1988. 2

    1988

  54. [54]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Qwen Team. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 7

    Pith/arXiv arXiv 2025

  55. [55]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, 2021. 2

    2021

  56. [56]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InNeurIPS,

  57. [57]

    Sam 2: Segment anything in images and videos.arXiv:2408.00714, 2024

    Nikhila Ravi et al. Sam 2: Segment anything in images and videos.arXiv:2408.00714, 2024. 2, 9, 10

    Pith/arXiv arXiv 2024

  58. [58]

    Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova

    Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos?NeurIPS, 2021. 1, 2, 5

    2021

  59. [59]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. InPro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018. 5, 9

    2018

  60. [60]

    Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta

    Gunnar A. Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in 13 homes: Crowdsourcing data collection for activity under- standing. InProceedings of the European Conference on Computer Vision (ECCV), 2016. 5, 6, 8

    2016

  61. [61]

    Scal- ing slot attention for unsupervised object discovery

    Satvik Singh, Chris Burgess, and Alexander Lerchner. Scal- ing slot attention for unsupervised object discovery. InICLR,

  62. [62]

    Elizabeth S. Spelke. Principles of object perception.Cogni- tive Science, 14(1):29–56, 1990. 2

    1990

  63. [63]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  64. [64]

    Generalised dice overlap as a deep learning loss function for highly unbalanced seg- mentations

    Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced seg- mentations. InInternational Workshop on Deep Learning in Medical Image Analysis, pages 240–248. Springer, 2017. 4

    2017

  65. [65]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025. 7

    Pith/arXiv arXiv 2025

  66. [66]

    Elder, Michael Kubovy, Stephen E

    Johan Wagemans, James H. Elder, Michael Kubovy, Stephen E. Palmer, Irving Biederman, et al. A century of gestalt psychology in visual perception: I. perceptual group- ing and figure–ground organization.Psychological Bulletin, 138(6):1172–1217, 2012. 2

    2012

  67. [67]

    Efficient video transformers with spatial- temporal token selection

    Junke Wang et al. Efficient video transformers with spatial- temporal token selection. InECCV, 2022. 2

    2022

  68. [68]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 14549–14560, 2023. 6

    2023

  69. [69]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 7

    Pith/arXiv arXiv 2025

  70. [70]

    Lvbench: An extreme long video un- derstanding benchmark.arXiv preprint arXiv:2406.08035,

    Weihan Wang et al. Lvbench: An extreme long video un- derstanding benchmark.arXiv preprint arXiv:2406.08035,

    Pith/arXiv arXiv

  71. [71]

    Vatex: A large-scale, high- quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high- quality multilingual dataset for video-and-language research. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 5, 6, 8

    2019

  72. [72]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023. 1

    Pith/arXiv arXiv 2023

  73. [73]

    Videollm-mod: Efficient video-language streaming with mixture-of-depths vision computation.arXiv preprint arXiv:2408.16730, 2024

    Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mod: Efficient video-language streaming with mixture-of-depths vision computation.arXiv preprint arXiv:2408.16730, 2024. 2

    Pith/arXiv arXiv 2024

  74. [74]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

  75. [75]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5, 6, 8

    2016

  76. [76]

    Slowfast-llava: A strong training-free base- line for video large language models.arXiv preprint arXiv:2407.15841, 2024

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models.arXiv preprint arXiv:2407.15841, 2024. 7

    Pith/arXiv arXiv 2024

  77. [77]

    Crosslmm: Decoupling long video sequences from lmms via dual cross- attention mechanisms.arXiv preprint arXiv:2505.17020,

    Shilin Yan, Jiaming Han, Joey Tsai, Hongwei Xue, Rongyao Fang, Lingyi Hong, Ziyu Guo, and Ray Zhang. Crosslmm: Decoupling long video sequences from lmms via dual cross- attention mechanisms.arXiv preprint arXiv:2505.17020,

    arXiv

  78. [78]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 7

    Pith/arXiv arXiv 2025

  79. [79]

    Xmem: Long- term video object segmentation with an atkinson–shiffrin memory model

    Zongxin Yang, Yunchao Wei, and Yi Yang. Xmem: Long- term video object segmentation with an atkinson–shiffrin memory model. InECCV, 2022. 2

    2022

  80. [80]

    Efficient video object segmentation via decomposing attention with optimized memory.arXiv preprint arXiv:2306.00961, 2023

    Zongxin Yang et al. Efficient video object segmentation via decomposing attention with optimized memory.arXiv preprint arXiv:2306.00961, 2023. 2

    Pith/arXiv arXiv 2023

Showing first 80 references.