Pith. sign in

REVIEW 3 major objections 4 minor 76 references

Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A learned token sampler watches how video tokens move and keeps the motion-rich ones visible, so masked video autoencoders can pre-train with up to 95% of tokens hidden and still fine-tune to strong action-recognition accuracy.

desk verdict A genuinely new adaptive-masking recipe with solid empirical gains over two baselines, but the PPO objective as written is not a valid PPO update, and that flaw sits at the center of the paper's main methodological claim. read the letter →

arxiv 2505.08561 v2 pith:XM6Z7PS5 submitted 2025-05-13 cs.CV

Ayush K. Rai , Kyle Min , Tarun Krishna , Feiyan Hu , Alan F. Smeaton , Noel E. O'Connor This is my paper · ORCID
classification cs.CV
keywords maskedvideomodelingadaptivetokensamplingtrajectoryattentionproximalpolicyoptimizationactionrecognitionself-supervisedpre-trainingtransformermasking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that masked video pre-training does not need to guess where to mask: a small learned module can watch how tokens move across frames and choose which tokens to keep visible. That module, trained jointly with the video autoencoder by reinforcement learning, is meant to keep motion-carrying tokens available while hiding redundant ones, allowing pre-training at 85 to 95 percent mask ratios without losing downstream action-recognition accuracy. If this works, aggressive masking can be adaptive rather than random, and pre-training compute is spent on tokens that actually matter. The authors report consistent fine-tuning gains over a random-masking baseline and a REINFORCE-trained adaptive-masking baseline across four action-recognition benchmarks.

What carries the argument

The central object is TATS, a two-branch module whose policy branch runs Trajectory Attention (a linear-complexity attention that pools features along the probabilistic path of each token between frames) through a linear layer and softmax to produce a categorical distribution over space-time tokens, while a value branch predicts expected reward from the mean token representation. Sampling visible tokens from that distribution defines the mask. PPO with a clipped objective, a value loss, and an entropy term updates the sampler using reconstruction error as reward, and an alternating freeze-and-unfreeze schedule lets the autoencoder and sampler co-train from scratch. The machinery's job is to turn 'mask the right tokens' into a learnable policy rather than a hand-designed heuristic.

What would settle it

Compare the tokens TATS keeps visible against tokens marked by dense optical flow or frame differencing on a held-out set of videos: if the intersection-over-union between selected tokens and true motion regions is no better than random, and downstream accuracy is unchanged when the reward is shuffled, the claim that trajectory-guided selection drives the gains would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a trajectory-aware token sampler (TATS) can be inserted into a masked video autoencoder and optimized end-to-end with PPO so that it learns to select motion-centric space-time tokens. The reward for the sampler is the autoencoder's reconstruction error: tokens that are hard to reconstruct are treated as informative and worth keeping visible. With this reward and a two-phase alternating update scheme that keeps the two networks stable when trained from scratch, the paper reports that pre-training under mask ratios 0.85, 0.90, and 0.95 yields fine-tuned action-recognition accuracy above both a random tube-masking baseline and a REINFORCE-trained adaptive sampler, across UCF101, HMDB51, Kinetics-400, and Something-Something v2, with pre-training memory lower than the adaptive baseline.

Load-bearing premise

The load-bearing premise is that reconstruction error measures informativeness: the sampler is rewarded for keeping tokens that are hard to reconstruct, assuming those are exactly the motion-bearing tokens; if hard-to-reconstruct tokens are instead noisy or rare patches, the selection target is wrong.

Editorial extensions

If this is right

  • Adaptive trajectory-based masking makes very aggressive masking practical: at mask ratios up to 0.95, fine-tuned action-recognition accuracy stays above that of fixed or randomly masked baselines trained at the same ratio.
  • The learned masking policy transfers across datasets, so pre-training on one benchmark and fine-tuning on another gives consistent accuracy gains over the baselines in most settings.
  • No optical flow or external semantic model is required; the trajectory signal is learned from raw video, keeping the pre-training pipeline self-contained and its memory use comparable to a standard masked video autoencoder.
  • The two-phase PPO training recipe provides a stable way to jointly optimize a masking policy and the reconstruction network that supplies its reward, which is the main obstacle to adaptive masking in this setting.
  • The TATS module can be dropped into an existing masked autoencoder architecture without changing the reconstruction target or the encoder-decoder design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's comparisons, the reconstruction-error reward is doing double duty: it selects tokens and also acts as a curriculum signal, and it would be worth testing whether a cheap motion estimate such as frame differencing produces the same masks.
  • Because TATS only needs token trajectories, the same recipe could apply to masked image modeling with pseudo-motion from augmentations or multi-view images, or to point-cloud and 3D token streams where trajectories are explicit.
  • The alternating PPO schedule is a general template for any masking policy whose reward comes from a co-trained reconstruction network; a testable consequence is that replacing the categorical sampler with a differentiable Gumbel-softmax relaxation should match its performance if the reconstruction-error reward premise is correct.
  • If the reward is replaced by shuffled or noise-corrupted reconstruction errors, the selected masks and downstream accuracy should degrade; if they do not, reconstruction error is not the informative signal the paper claims.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TATS (Trajectory-Aware Adaptive Token Sampler), a token-selection module for masked video autoencoders, and a training recipe that alternates between optimizing the MAE and TATS with a PPO-style objective. TATS uses Trajectory Attention to define a per-token sampling distribution, masks space-time tokens accordingly, and is trained with a reward based on the reconstruction error of the masked tokens. The authors report fine-tuned action recognition results on UCF101, HMDB51, a K400 subset, a SSv2 subset, and full SSv2 at mask ratios 0.85/0.90/0.95, claiming consistent gains over VideoMAE and AdaMAE with pretraining memory below AdaMAE. They also provide ablations on attention type, decoder depth, reconstruction loss, TA block count, and memory usage, plus qualitative mask visualizations.

Significance. If the methodological issues are resolved, the work is a useful step toward learned, input-adaptive masking in video masked autoencoders. The paper provides internally consistent comparisons with baselines trained under the same schedule, releases source code, evaluates on four benchmarks with a large-scale SSv2 run, and includes qualitative evidence that the learned masks concentrate on moving regions. The memory comparison against AdaMAE is a concrete practical contribution. However, the central theoretical novelty, the PPO-based joint training, is not correctly specified in the current equations, and the UCF101/HMDB51 results are confounded by hyperparameter selection on the same datasets. The significance of the empirical claims therefore cannot be fully assessed in the present form.

major comments (3)
  1. [Section 3.3, Eq. (7)-(8); Algorithm 1] The PPO surrogate in Eq. (8) is not a valid policy-gradient objective as written. In Phase 1 the stored episode contains pi_theta_old(Im|X) and the reward LR(phi) for the masked set Im actually used. In Phase 2 the current policy samples a new set Im' and Eq. (8) sets r(theta)=pi_theta(Im'|X)/pi_theta_old(Im|X). A valid importance ratio for PPO must be pi_theta(a|s)/pi_old(a|s) for the same action a whose reward is used; here the numerator is evaluated at Im' while the denominator and the reward correspond to Im, so the ratio mixes probabilities of two different actions and the stored baseline psi_old is not the baseline of Im'. Consequently J_CLIP is not a surrogate for any well-defined policy-gradient objective, and the claim that PPO stabilizes joint training is unsupported by the equations. Please correct the update (e.g., use r(theta)=pi_theta(Im|X)/pi_old(Im|X) with the stored episodes, or sample a fresh set and recompute the reward), and define the probability of a full masked set under the without-replacement categorical sampling of Section 3.2, since it is not the product of per-token probabilities.
  2. [Supplementary Section 8.2, Tables 5 and 6; Table 1] The fine-tuning results on UCF101 and HMDB51 in Table 1 are not independent of hyperparameter selection. Tables 5 and 6 select (mo,k) and (c1,c2,c3) by minimizing reconstruction error on UCF101 and HMDB51, and the same datasets' downstream accuracies are then reported in Table 1. Although the selection uses an unsupervised reconstruction criterion rather than action labels, it is still a form of model selection on the datasets whose fine-tuning numbers are the headline results, and the baselines are run with default configurations. Please either tune hyperparameters on a held-out validation split (or on K400/SSv2 only) before reporting UCF101/HMDB51, or provide an explicit argument that reconstruction-error-based selection cannot bias the reported downstream comparison.
  3. [Section 3.3] The reward design assumes that reconstruction error is a valid proxy for token informativeness: tokens with low reconstruction error are called redundant, while motion-centric tokens are assumed to have high reconstruction error. This premise is asserted rather than demonstrated, and the whole policy objective rests on it. If high reconstruction error instead reflects noise, occlusion, or rare static texture, the sampler would not select motion. The paper should supply quantitative evidence for the proxy, for example by correlating TATS-selected tokens with optical flow or annotated motion regions, or by including an ablation with a reversed/inverted reward to show that the direction of the reward, not just the added capacity, drives the gains.
minor comments (4)
  1. [Section 3.2 and Algorithm 1, line 10] The text and pseudocode write Im = Iv, but Im should be the complement of Iv; as written, the masked and visible sets are identical.
  2. [Section 4.2, item 2 vs Table 3b] The text says the best decoder depth is '# Blocks = 1' with 81.75%, but Table 3b reports 81.75% for '# Blocks = 4' and 81.46% for '# Blocks = 1'. Please reconcile the text and table.
  3. [Eq. (7)] The entropy term H(X,pi_theta)(.) is not defined; write H(pi_theta(.|X)) and state whether it is averaged over the sampled set or over all tokens.
  4. [Abstract and Section 5; Table 3c] The claim that pretraining is memory efficient should be qualified as relative to AdaMAE only: Table 3c shows a saving of less than 1 GB over AdaMAE and an increase of about 4.4 GB over VideoMAE.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central results are empirical measurements against external baselines, not quantities derived from their own inputs by construction.

full rationale

The paper's central claims are empirical: TATS is trained with the PPO-style objective in Eqs. (7)-(8), and its value is measured by fine-tuning accuracy in Tables 1, 2, and 9. Those accuracies are not algebraic consequences of the reward definition or of the hyperparameters, so there is no equation-level circularity. The reward design in Sec. 3.3, where reconstruction error is used as a proxy for informativeness, is an explicit assumption rather than a derivation; the further claim that high reconstruction error corresponds to motion-centric tokens is testable and is supported only by qualitative mask visualizations, so it is not true by definition. The supplementary hyperparameter tuning of (mo, k) and (c1, c2, c3) on UCF101 and HMDB51 (Tables 5 and 6), followed by fine-tuning reports on those same datasets, is a mild selection loop, but it is not a fitted parameter renamed as a prediction: the reported accuracies are measured outcomes, and the transfer results to Kinetics-400 and Something-Something v2 provide independent evidence that the configuration is not solely selected on the reported datasets. The PPO importance ratio in Eq. (8) compares probabilities of different masked sets, pi_theta(Im'|X) / pi_theta_old(Im|X), which is not the standard importance-sampling ratio for a single recorded action; however, this is a correctness and soundness concern about the training recipe, not a circularity in the derivation chain. There are no load-bearing self-citations: Trajectory Attention, VideoMAE, AdaMAE, and PPO are all external prior works cited as component sources, and none is used to justify the paper's own conclusions. Overall, the derivation and evaluation chain is not circular.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method's additional ingredients are fixed hyperparameters (mo, k, c1, c2, c3) tuned on the benchmark datasets themselves, plus two domain assumptions about the reward signal and trajectory attention. No new physical or conceptual entities are postulated; TATS is a learned module rather than an invented entity.

free parameters (5)
  • mo (epochs to train MAE only before TATS) = 10
    Chosen by tuning on UCF101/HMDB51 reconstruction error (Table 5); affects when TATS starts training.
  • k (TATS update interval in steps) = 1
    Chosen by tuning on UCF101/HMDB51 (Table 5); controls how often the policy is updated.
  • c1 (policy loss coefficient) = 1e-4
    Chosen by tuning on UCF101 (Table 6); balances the PPO policy term.
  • c2 (value loss coefficient) = 1e-4
    Chosen by tuning on UCF101 (Table 6); balances the value network loss.
  • c3 (entropy coefficient) = 1e-4
    Chosen by tuning on UCF101 (Table 6); controls exploration in the sampler.
assumptions (3)
  • domain assumption Reconstruction error is a valid proxy for token informativeness: low-error tokens are redundant, high-error tokens are motion-centric.
    Invoked in Section 3.3 to define the reward; if false, TATS selects the wrong tokens.
  • domain assumption Trajectory Attention captures real motion trajectories without a motion-specific objective.
    Section 3.2 assumes TA's learned probabilistic paths correspond to physical motion; this is not independently verified.
  • domain assumption The alternating freeze/unfreeze PPO recipe converges stably and the hyperparameters tuned on UCF101 transfer to other datasets.
    Section 3.3 and Supplementary Section 8.2; the stability claim is empirical and not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection." pith.science (2026). https://pith.science/paper/XM6Z7PS5

@misc{pith2026250508561,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XM6Z7PS5}},
  note         = {Machine review of arXiv:2505.08561}
}
read the original abstract

Masked video modeling~(MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable Trajectory-Aware Adaptive Token Sampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2505.08561 by the authors.

Figure 1
Figure 1. A depicts our overall architecture with MAE (fϕ) and TATS (gθ). B illustrates the joint training (Algorithm 1) of fϕ and gθ using PPO. Until epoch mo, standard random space-time masking is applied. Afterward, every k steps, Phase 1 (gθ frozen,fϕ unfrozen) stores old state of gθ in memory buffer Mb as episodes, followed by Phase 2 (gθ unfrozen,fϕ frozen), where gθ is optimized via Ls(θ). The optimization process then… view at source ↗
Figure 2
Figure 2. Visualization of adaptive masks learned by [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the TA learnt by TATS. The figure com￾prises four blocks : K400, SSv2, UCF101, and HMDB51 in top to bottom order. In each block, the first row shows video frames, the second depicts the trajectory attention on space-time tokens averaged across different heads. (K400, SSv2, UCF101, HMDB51) for the downstream task of action recognition, showcasing the effectiveness, gen￾eralization, transferability, a… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Sample Visualization of a Kinetics 400 video with [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Sample Visualization of a Kinetics 400 video with [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Sample Visualization of a Kinetics 400 video with [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 13
Figure 13. Figure 13: Sample Visualization of a HMDB51 video with [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Sample Visualization of a HMDB51 video with [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Sample Visualization of a HMDB51 video with [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 62 canonical work pages

  1. [1]

    Learning to see by moving

    Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE international conference on computer vision, pages 37–45, 2015. 3

    work page 2015

  2. [2]

    Self-supervised object detection from egocentric videos

    Peri Akiva, Jing Huang, Kevin J Liang, Rama Kovvuri, Xingyu Chen, Matt Feiszli, Kristin Dana, and Tal Hassner. Self-supervised object detection from egocentric videos. In 2023 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 5202–5214, 2023. 1

    work page 2023

  3. [3]

    Self- supervised Object-centric Learning for Videos

    G¨orkay Aydemir, Weidi Xie, and Fatma G ¨uney. Self- supervised Object-centric Learning for Videos. In Advances in Neural Information Processing Systems, 2023. 1

    work page 2023

  4. [4]

    Adamae: Adaptive masking for efficient spatiotempo- ral learning with masked autoencoders

    Wele Gedara Chaminda Bandara, Naman Patel, Ali Gho- lami, Mehdi Nikkhah, Motilal Agrawal, and Vishal M Pa- tel. Adamae: Adaptive masking for efficient spatiotempo- ral learning with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14507–14517, 2023. 1, 2, 3, 4, 5, 6, 7, 8

    work page 2023

  5. [5]

    Clustering on the unit hypersphere using von mises-fisher distributions

    Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005. 3

    work page 2005

  6. [6]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022. 1, 3

    work page 2022

  7. [7]

    Improving masked autoencoders by learning where to mask

    Haijian Chen, Wendong Zhang, Yunbo Wang, and Xiaokang Yang. Improving masked autoencoders by learning where to mask. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 377–390. Springer, 2023. 1, 3

    work page 2023

  8. [8]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Confer- ence on Machine Learning, pages 1691–1703. PMLR, 2020. 3

    work page 2020

Show all 76 references
  1. [9]

    Dynamonet: Dynamic action and motion network

    Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelha- gen. Dynamonet: Dynamic action and motion network. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6192–6201, 2019. 3

    2019

  2. [10]

    Peco: Perceptual codebook for bert pre-training of vision transformers

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. Peco: Perceptual codebook for bert pre-training of vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 552–560,

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  4. [12]

    Motion- guided masking for spatiotemporal representation learning

    David Fan, Jue Wang, Shuai Liao, Yi Zhu, Vimal Bhat, Hec- tor Santos-Villalobos, Rohith MV , and Xinyu Li. Motion- guided masking for spatiotemporal representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5619–5629, 2023. 3, 4

    2023

  5. [13]

    A large-scale study on unsupervised spatiotemporal representation learning

    Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Gir- shick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3299–3309, 2021. 3

    2021

  6. [14]

    Masked autoencoders as spatiotemporal learners

    Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946–35958,

  7. [15]

    Evolved part masking for self-supervised learning

    Zhanzhou Feng and Shiliang Zhang. Evolved part masking for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10386–10395, 2023. 3

    2023

  8. [16]

    Self-supervised video representation learn- ing with odd-one-out networks

    Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learn- ing with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636–3645, 2017. 3

    2017

  9. [17]

    Omnimae: Single model masked pretraining on images and videos

    Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10406–1041...

    2023

  10. [18]

    Reinforcement learning for language mod- els

    Yoav Goldberg. Reinforcement learning for language mod- els. URL https : / / gist . github . com / yoavg / 6bff0fecd65950898eba1bb321cfbd81, 2023. 5

    2023

  11. [19]

    Unsupervised learning of spatiotemporally coherent metrics

    Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pages 4086–4093, 2015. 3

    2015

  12. [20]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In...

    2017

  13. [21]

    Video repre- sentation learning by dense predictive coding

    Tengda Han, Weidi Xie, and Andrew Zisserman. Video repre- sentation learning by dense predictive coding. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019. 3

    2019

  14. [22]

    Self- supervised co-training for video representation learning

    Tengda Han, Weidi Xie, and Andrew Zisserman. Self- supervised co-training for video representation learning. Ad- vances in neural information processing systems, 33:5679– 5690, 2020. 1

    2020

  15. [23]

    Masked autoencoders are scalable 9 vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable 9 vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 1, 3

    2022

  16. [24]

    Using self-supervised learning can improve model robustness and uncertainty

    Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural In- formation Processing Systems. Curran Associates, Inc., 2019. 1

    2019

  17. [25]

    Mgmae: Motion guided masking for video masked autoencoding

    Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 13493– 13504, 2023. 2, 3, 4

    2023

  18. [26]

    Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens

    Sunil Hwang, Jaehong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens. In International Conference on Machine Learning, 2024. 3

    2024

  19. [27]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. 3

    2016 arXiv

  20. [28]

    What to hide from your students: Attention- guided masked image modeling

    Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention- guided masked image modeling. In Computer Vision – ECCV 2022, pages 300–318, Cham, 2022. Springer Natur...

    2022

  21. [29]

    The kinetics hu- man action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics hu- man action video dataset. arXiv preprint arXiv:1705.06950,

    arXiv

  22. [30]

    Bert: Pre-training of deep bidirectional transform- ers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transform- ers for language understanding. In Proceedings of NAACL- HLT, pages 4171–4186, 2019. 1, 3

    2019

  23. [31]

    Hmdb: a large video database for human motion recognition

    Hildegard Kuehne, Hueihan Jhuang, Est ´ıbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In2011 International conference on computer vision , pages 2556–2563. IEEE,

  24. [32]

    Unsupervised representation learning by sorting sequences

    Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE international confer- ence on computer vision, pages 667–676, 2017. 3

    2017

  25. [33]

    Semmae: Semantic-guided mask- ing for learning masked autoencoders

    Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided mask- ing for learning masked autoencoders. Advances in Neural Information Processing Systems, 35:14290–14302, 2022. 1, 3

    2022

  26. [34]

    Uniform masking: Enabling mae pre-training for pyramid- based vision transformers with locality

    Xiang Li, Wenhai Wang, Lingfeng Yang, and Jian Yang. Uniform masking: Enabling mae pre-training for pyramid- based vision transformers with locality. arXiv preprint arXiv:2205.10063, 2022. 1, 3

    2022 arXiv

  27. [35]

    Deep predictive coding networks for video prediction and unsu- pervised learning

    William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsu- pervised learning. In International Conference on Learning Representations, 2017. 3

    2017

  28. [36]

    Cl-mae: Curriculum-learned masked autoencoders

    Neelu Madan, Nicolae-C ˘at˘alin Ristea, Kamal Nasrollahi, Thomas B Moeslund, and Radu Tudor Ionescu. Cl-mae: Curriculum-learned masked autoencoders. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 2492–2502, 2024. 1, 3

    2024

  29. [37]

    Deep multi-scale video prediction beyond mean square error

    Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In 4th International Conference on Learning Representations, ICLR 2016, 2016. 3

    2016

  30. [38]

    Shuffle and learn: unsupervised learning using temporal order veri- fication

    Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order veri- fication. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 527–544. Sp...

    2016

  31. [39]

    Oswald, Alexander Kirillov, Cees G

    Duy Kien Nguyen, Yanghao Li, Vaibhav Aggarwal, Martin R. Oswald, Alexander Kirillov, Cees G. M. Snoek, and Xinlei Chen. R-MAE: Regions meet masked autoencoders. In The Twelfth International Conference on Learning Representa- tions, 2024. 3

    2024

  32. [40]

    Maxime Oquab, Timoth´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicola...

  33. [41]

    Learning features by watching objects move

    Deepak Pathak, Ross Girshick, Piotr Doll´ar, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2701–2710, 2017. 3

    2017

  34. [42]

    Keeping your eye on the ball: Trajectory at- tention in video transformers

    Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory at- tention in video transformers. Advances in neural information processing systems, 34:12493–12506...

    2021

  35. [43]

    Spatiotempo- ral contrastive video representation learning

    Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotempo- ral contrastive video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6964–6974, 2021. 3

    2021

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning , pages 8821–8831. Pmlr, 2021. 3

    2021

  38. [46]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 2, 3, 5, 8

    2017 arXiv

  39. [47]

    Adversarial masking for self-supervised learning

    Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. Adversarial masking for self-supervised learning. In Interna- 10 tional Conference on Machine Learning, pages 20026–20040. PMLR, 2022. 3

    2022

  40. [48]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 6, 1

    2012 arXiv

  41. [49]

    Learning video representations using contrastive bidirectional transformer

    Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743,

    1906 arXiv

  42. [50]

    Masked motion encoding for self-supervised video representation learning

    Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H Li, Mingkui Tan, and Chuang Gan. Masked motion encoding for self-supervised video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2235–2245, 2023. 3, 4

    2023

  43. [51]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020. 2, 3, 4, 8

    2020

  44. [52]

    Video- MAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Video- MAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 4, 6, 7, 8

    2022

  45. [53]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3

    2017

  46. [54]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 4

    2017

  47. [55]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th Inter- national Conference on Machine Learning, page 1096–1103, New York, NY , USA, 2008. Association ...

    2008

  48. [56]

    An- ticipating visual representations from unlabeled video

    Carl V ondrick, Hamed Pirsiavash, and Antonio Torralba. An- ticipating visual representations from unlabeled video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 98–106, 2016. 3

    2016

  49. [57]

    Tracking emerges by col- orizing videos

    Carl V ondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by col- orizing videos. In Proceedings of the European conference on computer vision (ECCV), pages 391–408, 2018. 3

    2018

  50. [58]

    Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics

    Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recogni...

    2019

  51. [59]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14549–14560, 2023. 1, 2, 3

    2023

  52. [60]

    Bevt: Bert pretraining of video transformers

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14733–14743, 2022. 1, 3

    2022

  53. [61]

    Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning. In Proceedings of the IEEE/CVF conference on computer visi...

    2023

  54. [62]

    Unsupervised learning of visual representations using videos

    Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2015. 3

    2015

  55. [63]

    Learning correspondence from the cycle-consistency of time

    Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2566–2576, 2019. 3

    2019

  56. [64]

    Masked feature predic- tion for self-supervised visual pre-training

    Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature predic- tion for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668–14678, 2022. 1, 3

    2022

  57. [65]

    Learning and using the arrow of time

    Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8052–8060, 2018. 3

    2018

  58. [66]

    Efficient vision- language pre-training by cluster masking

    Zihao Wei, Zixuan Pan, and Andrew Owens. Efficient vision- language pre-training by cluster masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26815–26825, 2024. 3

    2024

  59. [67]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. 2, 4, 5

    1992

  60. [68]

    Slow feature analysis: Unsupervised learning of invariances

    Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural com- putation, 14(4):715–770, 2002. 3

    2002

  61. [69]

    Masked frequency modeling for self-supervised visual pre-training

    Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. In The Eleventh Inter- national Conference on Learning Representations, 2023. 1, 3

    2023

  62. [70]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9653–9663, 2022. 1, 3

    2022

  63. [71]

    How mask mat- ters: Towards theoretical understandings of masked autoen- coders

    Qi Zhang, Yifei Wang, and Yisen Wang. How mask mat- ters: Towards theoretical understandings of masked autoen- coders. Advances in Neural Information Processing Systems, 35:27127–27139, 2022. 3

    2022

  64. [72]

    Image BERT pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. 1, 3 11 Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Ada...

    2022

  65. [73]

    Each video depicts a single action with a duration ranging from 2 to 6 seconds

    Datasets Something-Something V2 (SSv2) [20] is a curated video dataset for human action classification, comprising 174 classes and a total of 220,847 videos. Each video depicts a single action with a duration ranging from 2 to 6 seconds. SSv2 is a motion-focused dataset, where...

  66. [74]

    Data Preprocessing Our data processing pipeline closely follows AdaMAE [4] for pre-training

    Additional Implementation Details 8.1. Data Preprocessing Our data processing pipeline closely follows AdaMAE [4] for pre-training. We extract 16 frames of dimension 224× 224 from the videos, using a temporal stride of 4 (K400) and 2 (HMDB51/UCF101/SSv2), with the starting fra...

  67. [75]

    Large Scale Pre-training Results We conduct pre-training (400 epochs) and finetuning (100 epochs) of our model on full SSv2 [20] dataset forρ = 0.95 on 8 Nvidia A100 GPUs. In order to ensure fairness in comparison, we also pre-train (400 epochs) and finetune (100 epochs) both ...

  68. [76]

    Mask Visualization Here we show visualizations adaptive sampling learned by our TATS module across benchmark dataset for different mask ratiosρ ={0.95, 0.9, 0.85} in Figure 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. In all of these Figures, first row represents input video fram...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.