Pith. sign in

REVIEW 5 major objections 5 minor 82 references

Extending Video Masked Autoencoders to 128 frames

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video MAEs can pre-train on 128 frames by reconstructing only the top 15% of tokens.

desk verdict A solid empirical paper on 128-frame video MAE with a real contribution, but the adaptive-masking mechanism is less well isolated than the long-context result. read the letter →

arxiv 2411.13683 v1 pith:LLOMJ5UB submitted 2024-11-20 cs.CV

Nitesh Bharadwaj Gundavarapu , Luke Friedman , Raghav Goyal , Chaitra Hegde , Eirikur Agustsson , Sagar M. Waghmare , Mikhail Sirotenko , Ming-Hsuan Yang
show 3 more authors
Tobias Weyand Boqing Gong Leonid Sigal
This is my paper · ORCID
classification cs.CV
keywords videomaskedautoencoderslong-contextpre-trainingadaptivedecodermaskingtokenimportancescoringFSQ-MagViTtokenizerself-supervisedrepresentationlearningactionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video masked autoencoders are usually pre-trained on 16 or 32 frames because decoding every masked token at longer lengths exhausts memory. This paper claims that pre-training can be extended to 128 frames by decoding only the 15% of tokens judged most important by a learned saliency scorer, and that doing so improves the resulting encoder. The ranking and the reconstruction targets come from a jointly trained adaptive FSQ-MagViT tokenizer, which is frozen and reused for MAE pre-training. The longer context beats 32-frame pre-training on the same downstream tasks and lifts state-of-the-art accuracy on Diving48 by 3.9 points and on EPIC-Kitchens-100 verb classification by 2.5 points, with a plain ViT encoder and video-only pre-training.

What carries the argument

The carrying mechanism is the adaptive FSQ-MagViT tokenizer, a video tokenizer that learns both quantized reconstruction targets and a rank order over spatio-temporal tokens. A MAGVIT 3D-CNN encoder produces the tokens, finite scalar quantization maps them to a discrete codebook, and a CNN token scorer computes, for each token, the Euclidean distance between its feature and the feature of the corresponding token in the previous frame, treating that distance as importance. A differentiable top-k layer selects the 768 largest-importance tokens on 16-frame training clips and zeroes the rest, and the module is trained end-to-end on Kinetics-600 with pixel, GAN, and perceptual losses. Once frozen, this scorer generates the adaptive decoder mask (top 15% plus 5% random tokens) and the quantized target latents for MAE pre-training, which is what makes the decoder sparse enough for 128-frame videos.

What would settle it

On a set of 128-frame videos with pure camera motion and no foreground change, compute the adaptive mask from the frozen scorer and compare it with random and flow-based masks; if the selected tokens follow the background and downstream accuracy matches random-mask training, the transferability claim is false. A more direct test is to measure per-token reconstruction error of a fully trained 128-frame MAE: if the scorer's top 15% tokens are not systematically harder to reconstruct than randomly chosen tokens, the priority signal is not doing the work the paper attributes to it.

Watch

Extended reading notes

Core claim

The central claim is that a content-dependent adaptive decoder mask, built from a learned token-importance ranking, makes 128-frame video MAE pre-training both feasible and better than short-context pre-training. With a 15% decoder token budget, the proposed adaptive masking outperforms random, uniform, optical-flow, and EVEREST-style masking at the same budget on 32-frame clips, and it matches or improves on a VideoMAE decoder that reconstructs all tokens. Pre-training at 128 frames then yields better fine-tuned accuracy than pre-training at 32 frames, including when the 32-frame model is fine-tuned with 128-frame context, and 128-frame multi-crop evaluation does not close the gap. On long-horizon action benchmarks the resulting LVMAE reaches state-of-the-art top-1 accuracy on Diving48 and EPIC-Kitchens-100 verb classification while using only unlabeled video pre-training and a single temporal crop at evaluation.

Load-bearing premise

The load-bearing premise is that the frozen token-importance scorer, trained on short Kinetics-600 clips, keeps ranking the most valuable tokens correctly when it is slid across longer, higher-resolution videos; if its ranking degrades, adaptive masking collapses to random masking and the reported gains vanish.

Editorial extensions

If this is right

  • 128-frame pre-training consistently outperforms 32-frame pre-training when both are fine-tuned on 128 frames, and it also beats 32-frame multi-crop evaluation.
  • At a 15% decoder token budget, adaptive masking is competitive with or better than decoding all masked tokens, so the memory savings transfer directly into longer temporal context rather than sacrificing accuracy.
  • The same recipe reaches state-of-the-art results on Diving48 and EPIC-Kitchens-100 verb classification with a standard ViT, one temporal crop, and no labeled or text-supervised pre-training.
  • Decoder masking ratios near the encoder masking ratio (about 90%) are needed to fit 128-frame pre-training in memory, and a 15% decoder budget outperforms more expensive budgets such as 50%.
  • Gains from extending context grow from 16 to 64 frames and then become small from 64 to 128, matching the short clip lengths of most current benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the importance scorer is trained once on 16-frame clips and applied by sliding a window over 128-frame videos, the same frozen scorer could plausibly transfer across MAE architectures, resolutions, and frame rates without retraining; the paper does not test this.
  • The paper's deliberate decoupling of scorer and MAE leaves open an untested variant in which the scorer is adapted during MAE pre-training, a direction that could improve masks but risks reintroducing the easy-token bias the decoupling avoids.
  • The finding that a 15% decoder budget beats a 50% budget suggests the optimal budget may track how redundant the video content is, so future work could vary the budget per clip rather than fixing it globally.
  • Current benchmarks contain few videos longer than 128 frames, so the method's benefit on truly long-form video is likely underestimated; datasets with longer episodes would be the natural test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes LVMAE, a long-video masked autoencoder that extends VideoMAE-style pre-training to 128 frames by decoding only a small, content-selected subset of tokens. A frozen FSQ-MagViT tokenizer is trained jointly with a CNN token scorer that ranks spatio-temporal tokens by inter-frame feature distance; the MAE decoder then reconstructs the top-ranked tokens plus a small random fraction, yielding a 15% decoder token budget that makes 128-frame pre-training memory-feasible. The authors report controlled comparisons at equal total frames, show that 128-frame pre-training beats 32-frame pre-training when both are fine-tuned on 128 frames, and claim state-of-the-art results on Diving48 (+3.9 points) and EPIC-Kitchens-100 verb classification (+2.5 points). Additional ablations cover masking strategies, reconstruction targets, decoder budgets, and model-size trade-offs.

Significance. If the central claims hold, the paper makes a useful empirical contribution: it demonstrates a practical recipe for long-context video MAE pre-training and shows that a learned, frozen token-importance scorer can outperform random and uniform decoder masks at a very low token budget. The design has a genuine methodological virtue: the tokenizer and scorer are trained before MAE pre-training and then frozen, which avoids the circularity of fitting the mask to the MAE reconstruction objective. The experimental setup is mostly well controlled, with matched evaluation protocols (16x8, 32x4, 64x2, 128x1) and matched fine-tuning protocols in Table 2, and the paper reports exhaustive ablations across datasets and masking strategies. The main risk is that the scorer's transfer from its training operating point (per-window top-768 at 49% retention, 128x128 resolution) to the inference operating point (global 15% budget across concatenated windows at 112x112) is not validated, so the claimed advantage of adaptive over random masking at 128 frames rests on an incompletely characterized mechanism.

major comments (5)
  1. [§3.3 / A.5.1 / Table 10] The token scorer is trained with top-k=768 out of 1,568 tokens per 16-frame window, i.e., roughly 49% retention, but at inference it is applied at a global 15% token budget across eight concatenated windows (plus 5% random tokens). No check is reported that scores learned under the permissive 49% threshold rank-order correctly at the much stricter 15% global operating point, and no per-window normalization is described. If scores are uncalibrated across windows, the global mask can be dominated by a few high-motion windows, and the adaptive-over-random margins in Table 4d (47.3 vs 46.4 on EK100; 87.9 vs 86.3 on D48) could reflect an uneven sparsity pattern rather than content prioritization. Please add an analysis of rank-order stability as a function of budget and window, or an ablation with per-window normalization.
  2. [§3.3 / A.5.1] During tokenizer training, all tokens in the first frame are kept, but the paper does not state how the first latent frame of each 16-frame sliding window is handled at inference. Since importance is defined as the Euclidean distance between corresponding tokens in adjacent frames, the first latent frame of every window has no score. If these boundary frames are force-kept, the effective decoder budget exceeds 15%; if they are unscored and excluded, the global top-k can systematically ignore one-eighth of the temporal positions. The paper should specify the inference-time treatment of boundary frames and, ideally, ablate whether keeping or excluding them changes the reported results.
  3. [§4.3 / Table 2 / A.5.3] The headline comparison of 128-frame versus 32-frame pre-training when both are fine-tuned on 128 frames is missing a key implementation detail: how a model pre-trained on 32-frame clips is adapted to 128-frame inputs. In particular, the handling of spatio-temporal positional embeddings and any temporal interpolation or padding is not described. Without this, the reported gaps (47.3 vs 45.0 on EK100; 87.9 vs 83.2 on D48) may be confounded by the adaptation procedure rather than by pre-training length alone. The paper should also state how videos shorter than 128 frames are sampled or padded, since many EK100 clips are shorter than 128 frames.
  4. [Table 4d] The 128-frame masking-strategy comparison reports no error bars or number of seeds, while Table 1 reports standard deviations around 0.1 points at 32 frames. The margins of Adaptive over Random in Table 4d are only 0.9 points on EK100 and 1.6 points on D48, so without a significance assessment the claim that adaptive masking is the best strategy at 128 frames is not fully supported. Please report mean and standard deviation over multiple seeds, or a statistical test, for the Table 4d comparisons.
  5. [A.5.1 / Table 10] The adaptive tokenizer is pre-trained at 128x128 spatial resolution but used at 112x112 at inference, with no description of how the tokenizer or scorer adapts to this resolution change. Because the scorer's CNN features and the token grid change with resolution, this mismatch could affect both the importance scores and the reconstruction targets. Please clarify whether the resolution difference is intentional and discuss its effect on score calibration and downstream accuracy.
minor comments (5)
  1. [§4.2] The sentence containing 'respectfully' should read 'respectively'.
  2. [§3.1 / Eq. (2)] The notation V Q(V) in Eq. (2) is introduced only informally; please define it clearly, for example as the quantized token embedding of the input video V.
  3. [§3.2] The text 'N d = 0 .85N' contains an odd spacing and should be formatted as N_d = 0.85N.
  4. [A.2.2] The FineGym288 experiments add a 1-second margin to temporal action boundaries; this modifies the benchmark and should be disclosed prominently in the main text or in the table caption, not only in the appendix.
  5. [A.3 / Table 8] Table 8 is described as using 'improved hyper-parameters' that yield better results than Table 4d; please state which hyperparameters changed and whether all models in Table 8 share the same settings, so readers can reconcile the two tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the mask-generating token-importance scorer is trained and frozen before MAE pre-training, the adaptive-vs-random comparisons use identical reconstruction targets and independent external benchmarks, and the MAGVIT/FSQ self-citations are backed by released code and by an in-paper LFQ-vs-FSQ ablation.

full rationale

The paper's derivation chain is not circular. The decoder mask M_d is produced by a token-importance module whose scores S are defined by an independent quantity - pairwise Euclidean distance between CNN features of corresponding tokens in adjacent frames (Section 3.3, 'This distance is considered as token importance as it signifies the extent of underlying change in the video as done in [17]') - not by the MAE's reconstruction error. Section 3.4 states the tokenizer and scorer are trained on Kinetics600 and 'Once trained, we keep this module frozen,' so the mask cannot be optimized against the MAE objective the paper later evaluates; the paper explicitly decouples the two to avoid 'selecting easily unmaskable tokens,' which is the opposite of a fitted-input circularity. The headline claims are empirical benchmark comparisons: 'adaptive > random' is measured in Table 1 (32 frames) and Table 4d (128 frames) with identical FSQ-MagViT targets across all masking schemes, and '128 frame pre-training outperforms 32 frames' (Table 2) compares models pre-trained with the same adaptive scheme and fine-tuned identically, so neither claim is true by construction. The SOTA numbers on EPIC-Kitchens-100 and Diving48 are external benchmarks evaluated with standard fine-tuning protocols, not predictions implied by the method's definitions. The self-citations present - MAGVIT [25], the tokenizer-for-generation study [56], and FSQ [36], all with overlapping authorship - justify the tokenizer and quantizer choice, but they are genuine independent evidence: MAGVIT and FSQ are published with released code, and the paper additionally re-tests FSQ against LFQ in Table 5 before adoption, so the citation chain is not load-bearing in the sense of resting on unverified same-author results. Concerns that could be raised - hyperparameters such as the 15% decoder budget and the 2:1 adaptive-to-random ratio are selected using the same benchmarks on which SOTA is reported (Tables 4a and 8), and the token scorer is deployed at a stricter 15% inference operating point than the top-768 used during tokenizer training (Table 10) - are validity or robustness risks, not circular reductions, since nothing in those choices makes the reported accuracies equal to the method's inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The method contributes a learned token scorer plus a composition of known components. The reader inherits MAE, dual masking, MAGVIT, FSQ, and the temporal-distance saliency criterion from prior work; the ledger counts the tuned ratios (rho_d, rho_r, top-k, codebook size, fine-tuning masks) as free parameters and the design assumptions as axioms. No new physical or architectural entities beyond the token scorer module.

free parameters (6)
  • Decoder token budget, 10% adaptive + 5% random (rho_d = 90%, rho_r = 5%) = 15% total budget
    Ablated over 5/15/25/50% budgets in Appendix Table 8; 15% is best on Diving48 and used throughout. Section 4.2 reports rho_d = 90% and rho_r = 5%.
  • Random token fraction rho_r = 5%
    Table 4a shows 10% adaptive + 5% random (43.2) beats 15% + 0% (42.5) and 0% + 15% (42.2) on EK100; the paper says the random mix helps with overfitting.
  • Top-k selection count in tokenizer training = 768 of 2048 tokens
    Fixed k = 768 in Section 3.3 for 16-frame training clips; no ablation is reported for this value.
  • FSQ codebook size = 18-bit, levels [8,8,4,4,4,4,4,4]
    Appendix Table 5 compares LFQ and FSQ at 14 and 18 bits; FSQ 18-bit is chosen for the highest PSNR and EK100 accuracy.
  • First-frame keep-all rule = all first-frame tokens kept
    Section 3.3: 'For the special case of the first frame we keep all the tokens during training.' Ad hoc rule justified only by tokenizer reconstruction quality.
  • Fine-tuning encoder mask ratio = 20% (25% for ViT-L)
    Appendix A.5.3: 20% for 128-frame fine-tuning, 0% for 32/64-frame fine-tuning, adjusted to avoid OOM for ViT-L. Different masking across compared protocols.
assumptions (5)
  • domain assumption Masked reconstruction of a small subset of high-saliency tokens yields representations that transfer to downstream action classification.
    The core MAE premise inherited from VideoMAE and VideoMAEv2 (Sections 3.1 and 4.2); the paper validates it empirically but does not derive it.
  • domain assumption Pairwise Euclidean distance between CNN features of adjacent spatial locations is a valid token-importance signal.
    Section 3.3 says 'as done in [17]' (EVEREST); the paper adds learned features and joint training but keeps the distance criterion.
  • domain assumption The frozen importance scorer trained on 16-frame 128x128 Kinetics-600 clips transfers to 128-frame 224x224 videos via sliding-window concatenation.
    Appendix A.5.1: 'we simply slide a window of 16 frames with a stride of 16 through the tokenizer and token scorer module.' No adaptation or fine-tuning on long videos.
  • ad hoc to paper Decoupling tokenizer and scorer training from MAE pre-training prevents the MAE from biasing selection toward easily reconstructable tokens.
    Section 3.4: 'we consciously choose to decouple... so that the gradients do not bias token selection towards selecting easily unmaskable tokens.' This is the stated reason Adaptive beats EVEREST, but it is a design justification, not proven.
  • domain assumption Discrete FSQ-MagViT latents are better MAE reconstruction targets than RGB pixels.
    Supported by Table 4b (39.9 RGB vs 43.2 MAGVIT tokens) and consistent with BEVT and MVD findings; accepted as an empirical premise.
invented entities (1)
  • CNN token scorer with differentiable top-k layer
    purpose: Ranks spatio-temporal tokens by learned temporal feature distance to define the decoder mask and, jointly with FSQ-MagViT, the reconstruction targets.
    New learned module validated only through this paper's downstream ablations; no external benchmark characterizes the importance ranking itself, and the top-k value (768) is fixed without ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extending Video Masked Autoencoders to 128 frames." pith.science (2026). https://pith.science/paper/LLOMJ5UB

@misc{pith2026241113683,
  author       = {Pith},
  title        = {Pith review of: Extending Video Masked Autoencoders to 128 frames},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LLOMJ5UB}},
  note         = {Machine review of arXiv:2411.13683}
}
read the original abstract

Video understanding has witnessed significant progress with recent video foundation models demonstrating strong performance owing to self-supervised pre-training objectives; Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused on relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding. One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the-art on Diving48 by 3.9 points and EPIC-Kitchens-100 verb classification by 2.5 points while relying on a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).

Figures

Figures reproduced from arXiv: 2411.13683 by the authors.

Figure 1
Figure 1. Left: Proposed Long Video MAE Decoder Masking. We leverage a jointly trained adaptive tokenizer and importance module to define a decoder mask and token targets for a video MAE pre-training strategy. The resulting sparsification in tokens (only 15%) allows pre-training with long videos (128-frames) and results in substantial performance gains. Right: Decoder masking and memory in long-video (128 frames) pre-training… view at source ↗
Figure 2
Figure 2. Illustration of Adaptive FSQ-MagViT Training. FSQ-MagViT adaptive tokenizer includes MAGVIT encoder and CNN-based token scorer with a differentiable top-k selection layer designating importance of tokens. During tokenizer training unselected tokens zeroed out and video is reconstructed using MAGVIT decoder. We then freeze this adaptive tokenizer and use it to generate target tokens for scalable pre-training of video… view at source ↗
Figure 3
Figure 3. Our Adaptive tokenizer visualized. We visualize the tokens masks by masking the [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Token selection strategies visualized. We can see that flow based token selection can be [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 74 canonical work pages

  1. [1]

    Towards long-form video understanding

    Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021. 1

    2021

  2. [2]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS, 2024. 1

    2024

  3. [3]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV, 2022. 1

    work page 2022

  4. [4]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, 2023. 1

    work page 2023

  5. [5]

    Memory consolidation enables long-context video understanding

    Ivana Balaževi´c, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Memory consolidation enables long-context video understanding. In ICML, 2024. 1, 4, 6, 8, 9

    work page 2024

  6. [6]

    Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In CVPR, 2022. 4, 9

    work page 2022

  7. [7]

    Token turing machines

    Michael S Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab. Token turing machines. In CVPR, 2023. 1

    work page 2023

  8. [8]

    Video recap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video recap: Recursive captioning of hour-long videos. In CVPR, 2024. 1

    work page 2024

Show all 82 references
  1. [9]

    A simple recipe for contrastively pre-training video-first encoders beyond 16 frames

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames. In CVPR, 2024. 1, 3, 4

    2024

  2. [10]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. In EMNLP, 2024

    2024

  3. [11]

    Long-form video- language pre-training with multimodal temporal contrastive learning

    Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video- language pre-training with multimodal temporal contrastive learning. In NeurIPS, 2022. 1

    2022

  4. [12]

    Koala: Key frame-conditioned long video-llm

    Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In CVPR, 2024. 1

    2024

  5. [13]

    Masked autoencoders as spatiotemporal learners

    Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. In NeurIPS, 2022. 2, 3, 5

    2022

  6. [14]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023. 2, 4, 5, 6, 7, 15, 16

    2023

  7. [15]

    VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. 2, 3, 4, 5, 15

    2022

  8. [16]

    How can objects help action recognition? In CVPR, 2023

    Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid. How can objects help action recognition? In CVPR, 2023. 2, 3, 5, 8

    2023

  9. [17]

    Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens

    Sunil Hwang, Jaehong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens. In ICML, 2024. 2, 3, 5, 6, 7

    2024

  10. [18]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. In IJCV, 20...

    2022

  11. [19]

    Resound: Towards action recognition without representation bias

    Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In ECCV, 2018. 2, 6

    2018

  12. [20]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 3

    2022

  13. [21]

    Bevt: Bert pretraining of video transformers

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, 2022. 3, 9

    2022

  14. [22]

    Girdhar, A

    R. Girdhar, A. El-Nouby, M. Singh, K.V . Alwala, A. Joulin, and I. Misra. Omnimae: Single model masked pretraining on images and videos. In CVPR, 2023. 3

    2023

  15. [23]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 3, 15

    2021

  16. [24]

    Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR, 2023. 3, 7

    2023

  17. [25]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR,

  18. [26]

    Tokenlearner: What can 8 learned tokens do for images and videos? In NeurIPS, 2021

    Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? In NeurIPS, 2021. 3

    2021

  19. [27]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In IC...

    2017

  20. [28]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 3, 4

    2021

  21. [29]

    Mgmae: Motion guided masking for video masked autoencoding

    Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In ICCV, 2023. 3, 5

    2023

  22. [30]

    Motion-guided masking for spatiotemporal representation learning

    David Fan, Jue Wang, Shuai Liao, Yi Zhu, Vimal Bhat, Hector Santos-Villalobos, Rohith MV , and Xinyu Li. Motion-guided masking for spatiotemporal representation learning. In ICCV, 2023. 3

    2023

  23. [31]

    Video codec design: developing image and video compression systems

    Iain E Richardson. Video codec design: developing image and video compression systems. John Wiley & Sons, 2002. 3, 5

    2002

  24. [32]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 3, 7

    2020

  25. [33]

    Videoprism: A foundational visual encoder for video understanding

    Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In ICML, 2024. 3, 7

    2024

  26. [34]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV, 2024. 3

    2024

  27. [35]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021. 3, 9

    2021

  28. [36]

    Finite scalar quantization: VQ-V AE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-V AE made simple. InICLR, 2024. 5, 15, 18

    2024

  29. [37]

    A short note about kinetics-600

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. 6, 16

    2018 arXiv

  30. [38]

    A short note on the kinetics-700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. 8

    1907 arXiv

  31. [39]

    Multiview transformers for video recognition

    Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In CVPR, 2022. 8, 9 12

    2022

  32. [40]

    Temporally-adaptive models for efficient video understanding

    Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Yingya Zhang, Ziwei Liu, and Marcelo H Ang Jr. Temporally-adaptive models for efficient video understanding. arXiv preprint arXiv:2308.05787, 2023. 8, 9

    2023 arXiv

  33. [41]

    Training a large video model on a single machine in a day

    Yue Zhao and Philipp Krähenbühl. Training a large video model on a single machine in a day. arXiv preprint arXiv:2309.16669, 2023. 8, 9, 10

    2023 arXiv

  34. [42]

    Imagenet-21k pretraining for the masses

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In NeurIPS Datasets and Benchmarks Track, 2021. 8

    2021

  35. [43]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 8

    2022

  36. [44]

    Verbs in action: Improving verb understanding in video-language models

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In ICCV, 2023. 8

    2023

  37. [45]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 9

    2019

  38. [46]

    Interactive prototype learning for egocentric action recognition

    Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocentric action recognition. In ICCV, 2021. 9

    2021

  39. [47]

    Movinets: Mobile video networks for efficient video recognition

    Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In CVPR, 2021. 9

    2021

  40. [48]

    Omnivore: A single model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, 2022. 9

    2022

  41. [49]

    Learning video representations from large language models

    Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023. 9

    2023

  42. [50]

    Is space-time attention all you need for video understanding? In ICML, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021. 9

    2021

  43. [51]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022. 9

    2022

  44. [52]

    Can an image classifier suffice for action recognition? In ICLR, 2022

    Quanfu Fan, Rameswar Panda, et al. Can an image classifier suffice for action recognition? In ICLR, 2022. 9

    2022

  45. [53]

    Object-region video transformers

    Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In CVPR, 2022. 9

    2022

  46. [54]

    Aim: Adapting image models for efficient video action recognition

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. In ICLR, 2023. 9

    2023

  47. [55]

    Video-focalnets: Spatio-temporal focal modulation for video action recognition

    Syed Talal Wasim, Muhammad Uzair Khattak, Muzammal Naseer, Salman Khan, Mubarak Shah, and Fahad Shahbaz Khan. Video-focalnets: Spatio-temporal focal modulation for video action recognition. In ICCV, 2023. 9

    2023

  48. [56]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. In ICLR, 2024. 15

    2024

  49. [57]

    Finegym: A hierarchical video dataset for fine-grained action understanding

    Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. In CVPR, 2020. 15

    2020

  50. [58]

    Learning temporal cues for fine-grained action recognition

    Zhihao Liu, Yi Zhang, Wenhui Huang, Yan Liu, Mengyang Pu, Chao Deng, and Junlan Feng. Learning temporal cues for fine-grained action recognition. In ICIP, 2024. 15, 17

    2024

  51. [59]

    Tsm: Temporal shift module for efficient video understanding

    Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019. 17

    2019

  52. [60]

    Temporal query networks for fine-grained video understanding

    Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Temporal query networks for fine-grained video understanding. In CVPR, 2021. 17

    2021

  53. [61]

    Combined cnn transformer encoder for enhanced fine-grained human action recognition

    Mei Chee Leong, Haosong Zhang, Hui Li Tan, Liyuan Li, and Joo Hwee Lim. Combined cnn transformer encoder for enhanced fine-grained human action recognition. In CVPRW, 2022. 17 13

    2022

  54. [62]

    Going deeper with image transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021. 17

    2021

  55. [63]

    Scenic: A jax library for computer vision research and beyond

    Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A jax library for computer vision research and beyond. In CVPR, 2022. 18 14 A Appendix A.1 Choice of tokenizer & quantizer We choose MAGVIT [25, 56] (which has a 3D CNN encoder) because it...

    2022

  56. [65]

    We found this slightly improved accuracy ( +0.5 points on EPIC-Kitchens-100 Verbs and +1.2 points on Diving48)

    We use a single layer of class attention [ 62] as the aggregation method when generating pre- logits as opposed to mean pooling. We found this slightly improved accuracy ( +0.5 points on EPIC-Kitchens-100 Verbs and +1.2 points on Diving48)

  57. [66]

    We use 25% encoder masking instead of 20% encoder masking to avoid going OOM with the larger model

  58. [67]

    17 Table 9: Model size vs frames

    For Diving48 we finetune for less steps (50 epochs instead of 200 epochs). 17 Table 9: Model size vs frames. We report top-1 performance of our proposed MAE pre-training with decoder masking & FSQ-MagViT with different model sizes and maximum frames for that model size given a...

  59. [68]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: See Section 4 Experiments. Guidelines: • The answer NA means that the abstract and introduction do not include the cla...

  60. [69]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 5, Limitations and Broader Impact Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the p...

  61. [70]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 21 Justification: No theoretical claims and results. Guidelines: • The answer NA means that the paper does n...

  62. [71]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  63. [72]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 22 Answer: [No] Justification: We will strive to make th...

  64. [73]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Refer to Appendix A.5 for imp...

  65. [74]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Reported for main ablation in Table 1. Guidelines:...

  66. [75]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: The compute resources a...

  67. [76]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Yes. Guidelines: • The answer NA means that the authors have not reviewed the N...

  68. [77]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Section 5 Guidelines: • The answer NA means that there is no societal impact of the work performed. •...

  69. [78]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We are not rele...

  70. [79]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The base codebas...

  71. [80]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: No new assets are released. Guidelines: • The answer NA means that the paper does not release new assets. • Researche...

  72. [81]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  73. [82]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  74. [2023]

    3, 5, 6, 7, 15, 16, 18

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.