Pith. sign in

REVIEW 3 major objections 6 minor 39 references

DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A video is better represented for classification as a temporal sequence of embeddings than as a single pooled vector, and a learned time-weighted dynamic-time-warping head on a frozen encoder can deliver that gain with little extra cost.

desk verdict A clever DTW-as-min-pooling reformulation with a real speedup, but the headline accuracy gains are overstated by a nonstandard baseline and inconsistent 'leading' claims. read the letter →

arxiv 2506.12585 v1 pith:R5JTKOMH submitted 2025-06-14 cs.CV

Darryl Ho , Samuel Madden This is my paper
classification cs.CV
keywords videoclassificationtemporalsequenceembeddingsdynamictimewarpingtime-weightedalignmentencoder-agnosticfrozenencodersactionrecognitioncentroidlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DejaVid's central claim is that a video is better represented for classification as a time-ordered sequence of embeddings, one per short clip, than as a single mean-pooled embedding, and that a lightweight head can exploit that representation without retraining the video encoder. The head learns, for each class, a centroid sequence and positive per-timestep, per-feature weights, and classifies a video by its time-weighted dynamic-time-warping distance to each centroid. On top of a frozen VideoMAE V2-g encoder, DejaVid reports top-1 accuracy of 77.2% on Something-Something V2, 89.1% on Kinetics-400, and 88.6% on HMDB51, with fewer than 1.8% additional learnable parameters and under three hours of training. The paper also shows that simply supersampling clips and mean-pooling the embeddings does not help, which supports the claim that learned temporal alignment, not extra samples, is what drives the gain.

What carries the argument

The load-bearing object is the time-weighted DTW distance $D_w(U, C, m)$ between a video's temporal sequence of embeddings $m$ and a class centroid sequence $C$, where $U$ is a positive per-timestep, per-feature weight tensor and the pointwise distance is $\sum_k u_{i,k} |c_{i,k} - m_{j,k}|$ along the optimal warping path. This distance is made trainable by rewriting the DTW recurrence as a stack of min-pooling layers (kernel size 2, stride 1) interlaced with additive skip-connections along diagonals, which reduces the critical path from $O(nm)$ to $O(n+m)$ and lets gradients flow back to both $U$ and $C$. Two stability choices, dropping the diagonal transition so every warping path has exactly $n+m-1$ steps and freezing the copies of centroids and weights used for path selection for an entire epoch, are presented as necessary for convergence. Centroid sequences are initialized with DTW barycenter averaging (DBA).

What would settle it

Re-run the Something-Something V2 comparison with the baseline fixed to the officially reported 77.0 top-1 instead of the paper's re-measured 76.7, repeat the full DejaVid training several times with different random seeds, and check whether the average gain over 77.0 exceeds the run-to-run standard deviation; if it does not, the headline +0.5% claim fails.

Watch

Extended reading notes

Core claim

DejaVid establishes that a frozen, state-of-the-art video transformer can be improved at the decision stage by replacing mean-pooling over clips with a learned, differentiable time-weighted dynamic time warping (DTW) classification head. The method treats each video as a temporal sequence of embeddings (TSE), initializes a per-class centroid TSE by DTW barycenter averaging (DBA), and then trains both the centroids and a positive weight tensor that modulates the per-timestep, per-feature contribution to the warping distance. Because gradients through DTW flow only along the optimal warping path, the paper reformulates the algorithm as a serial stack of min-pooling layers with additive skip-connections, reducing the critical path from quadratic to linear and making backpropagation practical. On top of VideoMAE V2-g, this head reaches 77.2, 89.1, and 88.6 top-1 on Something-Something V2, Kinetics-400, and HMDB51, with less than 1.8% additional parameters and under three hours of training; ablations show both centroid fine-tuning and time-weight learning contribute, and that diagonal transitions and per-batch warping-path updates hurt stability.

Load-bearing premise

The claimed accuracy gains are measured against the authors' own re-measured VideoMAE V2-g baseline (76.7 top-1 on Something-Something V2) rather than the officially reported 77.0, and the paper does not report run-to-run variance, so the headline +0.5% gain could be within noise if the official baseline is the fair comparator.

Editorial extensions

If this is right

  • If the reported gains hold, a frozen billion-parameter video encoder can be improved by 0.5 to 0.7 top-1 points with only a few million learned parameters and a few hours of training, making state-of-the-art upgrades cheap and accessible.
  • Because the head is encoder-agnostic, any pre-trained encoder with published weights can be upgraded the same way; the paper reports larger relative gains on smaller encoders such as TimeSformer and ViViT.
  • Temporal supersampling alone did not improve accuracy in the paper's control, so the benefit of DejaVid appears to come from learned alignment over the temporal sequence rather than from simply seeing more clips.
  • Ablations show that both learnable centroids and time-varying feature weights contribute independently, with weights helping most on the dataset with the shortest, most duration-variable videos.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave untested is that the learned weight tensor $U$ could double as a temporal-saliency map, revealing which time steps and embedding dimensions drive each class decision; this could be checked by visualizing the trained weights and comparing them to human annotations of salient moments.
  • The finding that time weights help most on Something-Something V2 suggests a testable prediction: datasets with high temporal variability and strong event ordering should show larger DejaVid gains over mean-pooling, for example in surgical or sports video.
  • Because DejaVid is a prototype-based classifier in DTW space, a natural extension is few-shot or streaming recognition, where DBA could initialize centroids from a handful of examples and the learned weights could be adapted online.
  • The min-pooling reformulation of DTW is not specific to video, so the same head could improve other sequence models that currently collapse time by averaging, a transfer that the paper does not claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DejaVid, an encoder-agnostic temporal head for video classification. A video is converted into a variable-length temporal sequence of embeddings (TSE) by sliding a window over a frozen backbone encoder; per-class centroid TSEs are initialized with DBA, and classification is performed by a learned time-weighted dynamic time warping distance to the centroids. The DTW recurrence is reformulated as a stack of min-pooling and additive skip-connection layers, enabling backpropagation and a custom CUDA kernel with a reported speedup of 161-750x. On VideoMAE V2-g, the paper reports top-1 accuracies of 77.2% on Something-Something V2, 89.1% on Kinetics-400, and 88.6% on HMDB51, with added learnable parameters below 1.8% of the encoder and training under 3 hours.

Significance. The core idea is valuable if the reported gains are robust: improving a frozen, billion-parameter video encoder with a cheap, trainable DTW-based head would be a practically useful contribution, and the DTW-as-neural-network formulation is a concrete and reusable algorithmic idea. The paper ships code, reports informative ablations in Tables 2 and 3, and demonstrates a large practical speedup from the CUDA kernel. The significance as stated, however, is not yet supported: the headline improvements are small (0.5-0.7%), the Something-Something V2 gain rests on a nonstandard re-measured baseline, no run-to-run variance is reported, and the 'leading' wording is contradicted by the paper's own Table 1. The method itself is coherent and the design choices are well motivated, but the strength of the central claim needs additional evidence.

major comments (3)
  1. [Section 4.3, Footnote 5, Tables 1-2] The central Something-Something V2 result is not robustly established. The paper compares DejaVid's 77.2% to the authors' re-measured VideoMAE V2-g baseline of 76.7% (footnote 5) while the official VideoMAE V2 release reports 77.0%; with the official comparator the gain drops from +0.5% to +0.2%. No seed variance or confidence interval is reported, and the two full-learning variants in Table 2 differ by 0.2% (77.2 vs 77.0), the same order of magnitude as the headline gain. Please report multiple runs with mean and standard deviation for all reported settings, and either use the official baseline or provide a direct head-to-head comparison with the official evaluation pipeline.
  2. [Abstract, Introduction, Conclusion, Table 1] The claim of 'leading Top-1 accuracy' is contradicted by the paper's own Table 1, which lists InternVideo2-6B at 77.5% on Something-Something V2 and 92.1% on Kinetics-400, and TubeViT at 90.9% on Kinetics-400. DejaVid's 77.2% and 89.1% are therefore not leading on those benchmarks. The statement in Section 4.3 that DejaVid achieves the highest HMDB51 accuracy is also not supported by any comparison in Table 1. Please replace these statements with accurate comparative claims, or supply a complete, dated leaderboard that substantiates the 'leading' and 'highest' wording.
  3. [Tables 1-2 and Abstract] The reported Kinetics-400 headline accuracy of 89.1% is achieved by the frozen-weights variant, not by the full-learning variant, which obtains 88.9% with random initialization and 89.0% with one initialization in Table 2. The abstract describes the method as learning per-timestep, per-feature weights, yet the paper's own text in Section 4.3 states that weight learning provides no additional benefit on Kinetics-400. Please present the main configuration consistently: either report the full-learning setting as the main system and explain the frozen-weights variant separately, or explicitly flag that the K400 result comes from the frozen-weights setting.
minor comments (6)
  1. [Table 4, Footnote 7] The naive DTW runtime entries 4.79e47 and 1.17e57 seconds for Something-Something V2 and Kinetics-400 are implausible and appear to be typographical errors (possibly missing an exponent or a decimal point). In addition, Footnote 7 says the naive runtime is estimated by running 3 batches, but the table labels the values as runtime per training epoch; please clarify which quantity is actually reported.
  2. [Section 3.2] The statement that the reformulation 'obtains the linear runtime O(n + m)' should be qualified as a reduction in critical-path length; the total number of cell computations per pairwise DTW distance remains O(nm) if the diagonals are processed sequentially.
  3. [Section 3.2] There is a typo, 'as descrbed' in the first paragraph; it should read 'as described'.
  4. [Supplementary Material] In the gradient formulas, the condition i + start + j = l is easy to misread; state explicitly that j = l - (i + start) and that these indices refer to the offset position in the l-th diagonal.
  5. [Table 1] The 'Clips×Crops' column is missing entries for the frozen-weights and full-learning (w-init-one) rows on Kinetics-400; please fill these in for completeness.
  6. [Section 4.3] The sentence 'we test our method on each dataset with three different settings' is incomplete because Table 2 also reports frozen-centroid variants; please clarify how the three settings relate to the ablation variants in Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DejaVid's temporal-matching method is trained and evaluated in a standard supervised manner, and its claims do not reduce to their inputs by construction.

full rationale

The paper's derivation chain is self-contained. The DTW and time-weighted DTW distances are fixed algorithmic definitions (Algorithm 1), the centroid TSEs are initialized by the external DBA algorithm, and both centroids and per-timestep per-feature weights are optimized on the training set via cross-entropy and then evaluated on validation sets; no target accuracy is fitted as a constant, and no fitted parameter is renamed as a prediction. The neural-network reformulation in Algorithm 2 is explicitly an equivalence to Algorithm 1, not a new assumption that secretly encodes the result. There are no load-bearing self-citations: the authors cite VideoMAE V2, DBA, and DTW as external prior work, and the method's own contributions are validated with ablations rather than by appeal to the authors' previous results. The reviewer's main concern, the Something-Something V2 baseline discrepancy described in footnote 5, is a measurement-comparability issue, not circularity: using a re-measured 76.7 baseline instead of the official 77.0 changes the size of the reported improvement but does not make the model's output equivalent to its inputs. Similarly, the 'leading Top-1 accuracy' wording conflicting with Table 1 is an accuracy or presentation issue, not a circular-derivation issue. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method's success depends on a handful of hand-chosen hyperparameters (Tc=8, windows, strides, learning rates) and on the domain assumption that frozen ViT embeddings and logits carry temporally discriminative structure. No new physical entities are introduced; the DTW distance is a fixed algorithmic definition with learned weights.

free parameters (5)
  • Tc (centroid TSE length) = 8
    Set to 8 for all experiments; not tuned per dataset. Controls the temporal resolution of each class prototype and is chosen by hand.
  • Sliding window stride (K400/HMDB) = |vid|/40
    Produces 33 timesteps per video; chosen to give a 336x1 clip-crop count.
  • Sliding window widths (SSv2) = {16, 32, 64, |vid|}
    Four parallel windows with stride 1; chosen to provide both constant-width and variable-width inputs.
  • Learning rates for log-weights and centroids = 1e-3 to 8e-3 (log-weights), one-third for centroids
    Chosen per dataset; no search or theory is given.
  • DBA iterations and sample count for centroid init = 100 iterations, 50 TSEs per class
    Used for centroid initialization; values chosen by hand.
assumptions (4)
  • domain assumption Frozen backbone per-clip embedding and logit vectors carry enough information for a DTW-to-centroid classifier to generalize.
    The entire method operates in the output space of VideoMAE V2-g; if the features are not temporally informative, the method fails. Invoked throughout Sec. 3 and 4.
  • domain assumption DTW distance with the diagonal transition removed is a suitable and stable similarity measure for multivariate TSEs.
    Sec. 3.4 removes diagonal transitions and justifies by stability; the ablation supports it empirically for these datasets, but there is no theoretical guarantee.
  • domain assumption DBA produces a good enough centroid initialization.
    Sec. 3.3 uses 100 DBA iterations on 50 resampled TSEs per class; the quality of this init is not independently evaluated beyond end accuracy.
  • standard math Standard backpropagation through the min-pooling formulation gives valid gradients.
    Sec. 3.2 and the Supplementary provide gradient formulas; min and add operations are subdifferentiable, which is standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification." pith.science (2026). https://pith.science/paper/R5JTKOMH

@misc{pith2026250612585,
  author       = {Pith},
  title        = {Pith review of: DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5JTKOMH}},
  note         = {Machine review of arXiv:2506.12585}
}
read the original abstract

In recent years, large transformer-based video encoder models have greatly advanced state-of-the-art performance on video classification tasks. However, these large models typically process videos by averaging embedding outputs from multiple clips over time to produce fixed-length representations. This approach fails to account for a variety of time-related features, such as variable video durations, chronological order of events, and temporal variance in feature significance. While methods for temporal modeling do exist, they often require significant architectural changes and expensive retraining, making them impractical for off-the-shelf, fine-tuned large encoders. To overcome these limitations, we propose DejaVid, an encoder-agnostic method that enhances model performance without the need for retraining or altering the architecture. Our framework converts a video into a variable-length temporal sequence of embeddings, which we call a multivariate time series (MTS). An MTS naturally preserves temporal order and accommodates variable video durations. We then learn per-timestep, per-feature weights over the encoded MTS frames, allowing us to account for variations in feature importance over time. We introduce a new neural network architecture inspired by traditional time series alignment algorithms for this learning task. Our evaluation demonstrates that DejaVid substantially improves the performance of a state-of-the-art large encoder, achieving leading Top-1 accuracy of 77.2% on Something-Something V2, 89.1% on Kinetics-400, and 88.6% on HMDB51, while adding fewer than 1.8% additional learnable parameters and requiring less than 3 hours of training time. Our code is available at https://github.com/darrylho/DejaVid.

Figures

Figures reproduced from arXiv: 2506.12585 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. 1. Each cell of the DTW 2D grid executes a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages

  1. [1]

    Youtube-8m: A large- scale video classification benchmark

    Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large- scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016. 8

    arXiv 2016

  2. [2]

    Timesformer-base-finetuned-ssv2

    Facebook AI. Timesformer-base-finetuned-ssv2. https: / / huggingface . co / facebook / timesformer - base-finetuned-ssv2 , 2021. Accessed: 2025-03-12. 1

    work page 2021

  3. [3]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 6836–6846,

  4. [4]

    The uea multivariate time series classification archive, 2018

    Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Ea- monn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075, 2018. 3

    arXiv 2018

  5. [5]

    Using dynamic time warping to find patterns in time series

    Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd international conference on knowledge discovery and data mining, pages 359–370, 1994. 2, 3

    work page 1994

  6. [6]

    Is space-time attention all you need for video understanding? In International Conference on Machine Learning , pages 813–824

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning , pages 813–824. PMLR, 2021. 1

    work page 2021

  7. [7]

    Dtwnet: a dynamic time warping network

    Xingyu Cai, Tingyang Xu, Jinfeng Yi, Junzhou Huang, and Sanguthevar Rajasekaran. Dtwnet: a dynamic time warping network. Advances in neural information processing sys- tems, 32, 2019. 3

    work page 2019

  8. [8]

    Few-shot video classification via tem- poral alignment

    Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-shot video classification via tem- poral alignment. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 10618–10627, 2020. 3

    work page 2020

Show all 39 references
  1. [9]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 1

    2017

  2. [10]

    D3tw: Discriminative differentiable dy- namic time warping for weakly supervised action alignment and segmentation

    Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dy- namic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogniti...

    2019

  3. [11]

    Soft-dtw: a differen- tiable loss function for time-series

    Marco Cuturi and Mathieu Blondel. Soft-dtw: a differen- tiable loss function for time-series. In International confer- ence on machine learning, pages 894–903. PMLR, 2017. 3

    2017

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1

    2010 arXiv

  5. [13]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 2, 8

    2019

  6. [14]

    Fine- grained temporal contrastive learning for weakly-supervised temporal action localization

    Junyu Gao, Mengyuan Chen, and Changsheng Xu. Fine- grained temporal contrastive learning for weakly-supervised temporal action localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19999–20009, 2022. 3

    2022

  7. [15]

    Video action transformer network

    Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zis- serman. Video action transformer network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 244–253, 2019. 8

    2019

  8. [16]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  9. [17]

    Large-scale video classification with convolutional neural networks

    Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014. 8

    2014

  10. [18]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  11. [19]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 2

    2012

  12. [20]

    Hmdb: a large video database for human motion recognition

    Hildegard Kuehne, Hueihan Jhuang, Est ´ıbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 Inter- national conference on computer vision , pages 2556–2563. IEEE, 2011. 1, 2, 5, 6

    2011

  13. [21]

    Tam: Temporal adaptive module for video recog- nition

    Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recog- nition. In Proceedings of the IEEE/CVF international con- ference on computer vision , pages 13708–13718, 2021. 2, 8

    2021

  14. [22]

    Action recognition on something-something v2 leaderboard

    Papers with Code. Action recognition on something-something v2 leaderboard. https : / / paperswithcode . com / sota / action - recognition - in - videos - on - something ,

  15. [23]

    A global averaging method for dynamic time warping, with ap- plications to clustering

    Franc ¸ois Petitjean, Alain Ketterlin, and Pierre Ganc ¸arski. A global averaging method for dynamic time warping, with ap- plications to clustering. Pattern recognition, 44(3):678–693,

  16. [24]

    Re- thinking video vits: Sparse video tubes for joint image and video learning

    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Re- thinking video vits: Sparse video tubes for joint image and video learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2214– 2224, 2023. 1, 2, 8

    2023

  17. [25]

    Vivit-b-16x2-kinetics400

    Google Research. Vivit-b-16x2-kinetics400. https : / / huggingface . co / google / vivit - b - 16x2 - kinetics400, 2021. Accessed: 2025-03-12. 1 9

    2021

  18. [26]

    Dynamic time warping algorithm review

    Pavel Senin. Dynamic time warping algorithm review. In- formation and Computer Science Department University of Hawaii at Manoa Honolulu, USA, 855(1-23):40, 2008. 3

    2008

  19. [27]

    The move-split-merge metric for time series

    Alexandra Stefan, Vassilis Athitsos, and Gautam Das. The move-split-merge metric for time series. IEEE transac- tions on Knowledge and Data Engineering , 25(6):1425– 1438, 2012. 3, 5

    2012

  20. [28]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022. 8

    2022

  21. [29]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE In- ternational Conference on Computer Vision (ICCV) , pages 4489–4497, 2015. 8

    2015

  22. [30]

    Implicit temporal modeling with learn- able alignment for video recognition

    Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learn- able alignment for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19936–19947, 2023. 2, 8

    2023

  23. [31]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14549–14560, 2023. 1, ...

    2023

  24. [32]

    Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning. In Proceedings of the IEEE/CVF conference on computer vi- ...

    2023

  25. [33]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 1, 2, 7, 8

    2024 arXiv

  26. [34]

    What can simple arithmetic oper- ations do for temporal modeling? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13712–13722, 2023

    Wenhao Wu, Yuxin Song, Zhun Sun, Jingdong Wang, Chang Xu, and Wanli Ouyang. What can simple arithmetic oper- ations do for temporal modeling? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13712–13722, 2023. 2, 8

    2023

  27. [35]

    Multiview transformers for video recognition

    Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3333–3343, 2022. 2, 8

    2022

  28. [36]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lu- cas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104–12113, 2022. 1 10 DejaVid: Encoder-Agnostic Learned Temporal Matching for Video C...

    2022

  29. [38]

    We now describe our choice of the temporal sliding win- dow widths and strides

    to produce the centroid. We now describe our choice of the temporal sliding win- dow widths and strides. For Kinetics-400 and HMDB51, given a video, VideoMAE V2 temporally segments the video into 5 clips of the same length and takes 3 crops at the left, center, and right to pr...

  30. [39]

    This means that the encoder is finetuned to an input win- dow width of |vid|, which complicates our sliding window application. The vast majority of Kinetics-400 videos are of length ∼ 300 frames, but videos in Something-Something V2 vary more in frame count, ranging from the ...

  31. [2024]

    Accessed: 2024-11-04. 7

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.