{"id":"d3a37e3c-fe0c-4d31-9fb1-93507d5e06e4","arxiv_id":"2411.13683","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A frozen token-importance scorer trained with an FSQ-MagViT tokenizer enables 128-frame masked-autoencoder pre-training, which outperforms 32-frame pre-training and sets state-of-the-art results on Diving48 and EPIC-Kitchens-100 verbs.","lead":"Video models that learn by masking normally see only 16 or 32 frames at a time; this paper adds a learned token-importance scorer that lets them pre-train on 128-frame clips by reconstructing only 15 percent of tokens. On long-horizon action benchmarks Diving48 and EPIC-Kitchens-100 verb classification, the resulting models beat published state of the art using video-only pre-training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frozen scorer is used at a 15% global budget over 8 concatenated windows after training at a 49% per-window budget; if its ranking is uncalibrated there, the 128-frame adaptive-over-random gains may reflect sparsity pattern, not content prioritization.","rationale":"The paper is a well-executed systems paper: it includes a matched 32-frame-pre-trained/128-frame-fine-tuned baseline, variance over three seeds in Table 1, ablations over decoder budgets and targets, and a sensible decoupling of tokenizer/scorer training from MAE pre-training to avoid the easy-token bias it attributes to EVEREST. The headline comparisons are large and internally consistent (Table 2). My concern is not that the tables are wrong, but that the mechanism claimed for the adaptive mask is less well supported than the headline numbers. The strongest direct test the paper provides is Table 4d, where adaptive beats random by 0.9/1.6 points at 128 frames; without error bars or a matched sparsity control, that gap is too thin to establish that the frozen scorer's content ranking, rather than the shape of the mask distribution, drives the result. The scorer's training regime (top-49% per 16-frame window, first frame force-kept) is very different from its inference regime (top-15% across eight concatenated windows), and the paper does not analyze the mask's temporal distribution or boundary behavior. This is the load-bearing assumption because the entire long-video claim rests on reinvesting the 15% decoder budget intelligently; if the budget were spent by random or by an uncalibrated scorer, the paper would still demonstrate memory savings, but the 'adaptive masking enables long-video MAE' contribution would reduce to a sparsity trick. I therefore agree with the reader's conditional assessment, with a more specific mechanism. The proposed inverse-mask experiment would settle whether the ranking is load-bearing. No change to the verdict: conditional acceptance remains appropriate pending those checks and release of masks/code.","tokens_in":22120,"tokens_out":10659,"duration_ms":989903,"concrete_test":"Re-run the 128-frame LVMAE configuration of Table 4d (same hyperparameters, same FSQ-MagViT targets, same 15% total budget) but invert the adaptive component of the decoder mask: decode the 10% lowest-scoring tokens plus the same 5% random tokens instead of the top 10%. If fine-tuned EK100/D48 top-1 does not drop materially below the reported 47.3/87.9, the scorer's ranking is not selecting the most valuable tokens and the claimed adaptive gain over Random is not caused by content prioritization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the frozen FSQ-MagViT scorer: at 128 frames its masks decide which 15% of tokens the MAE tries to reconstruct, and the paper claims this content-dependent choice, not just the memory saving, is what makes long-video pre-training beat 32-frame pre-training. Two details make that mechanism less secure than the paper's presentation. First, the scorer was trained to keep top-k=768 of 1,568 tokens per 16-frame window, i.e. roughly 49%, while LVMAE selects a global top 10% (plus 5% random) across eight concatenated windows, and Table 10 confirms inference uses a 15% top-k. Nothing in the paper checks that scores learned under a permissive 49% threshold rank-order correctly at the much stricter 15% global operating point, and no per-window normalization is described, so the global mask can be dominated by a few high-motion windows. Second, importance is defined as Euclidean distance between corresponding tokens in adjacent frames within a window; the paper force-keeps the first frame during tokenizer training but does not state how the first latent frame of each sliding window is scored at inference. If those boundary frames receive no score, the global top-k can systematically ignore one eighth of the temporal positions. Either failure would make the adaptive mask little more than an uneven sparsity pattern; the observed margins over Random in Table 4d (EK100 47.3 vs 46.4, D48 87.9 vs 86.3) are small enough that such an artifact could explain them. The paper presents no experiment isolating whether the scorer's ranking, rather than the induced token distribution, causes the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LVMAE, a long-video masked autoencoder that extends VideoMAE-style pre-training to 128 frames by decoding only a small, content-selected subset of tokens. A frozen FSQ-MagViT tokenizer is trained jointly with a CNN token scorer that ranks spatio-temporal tokens by inter-frame feature distance; the MAE decoder then reconstructs the top-ranked tokens plus a small random fraction, yielding a 15% decoder token budget that makes 128-frame pre-training memory-feasible. The authors report controlled comparisons at equal total frames, show that 128-frame pre-training beats 32-frame pre-training when both are fine-tuned on 128 frames, and claim state-of-the-art results on Diving48 (+3.9 points) and EPIC-Kitchens-100 verb classification (+2.5 points). Additional ablations cover masking strategies, reconstruction targets, decoder budgets, and model-size trade-offs.","tokens_in":22411,"tokens_out":6140,"duration_ms":62479,"significance":"If the central claims hold, the paper makes a useful empirical contribution: it demonstrates a practical recipe for long-context video MAE pre-training and shows that a learned, frozen token-importance scorer can outperform random and uniform decoder masks at a very low token budget. The design has a genuine methodological virtue: the tokenizer and scorer are trained before MAE pre-training and then frozen, which avoids the circularity of fitting the mask to the MAE reconstruction objective. The experimental setup is mostly well controlled, with matched evaluation protocols (16x8, 32x4, 64x2, 128x1) and matched fine-tuning protocols in Table 2, and the paper reports exhaustive ablations across datasets and masking strategies. The main risk is that the scorer's transfer from its training operating point (per-window top-768 at 49% retention, 128x128 resolution) to the inference operating point (global 15% budget across concatenated windows at 112x112) is not validated, so the claimed advantage of adaptive over random masking at 128 frames rests on an incompletely characterized mechanism.","major_comments":[{"comment":"The token scorer is trained with top-k=768 out of 1,568 tokens per 16-frame window, i.e., roughly 49% retention, but at inference it is applied at a global 15% token budget across eight concatenated windows (plus 5% random tokens). No check is reported that scores learned under the permissive 49% threshold rank-order correctly at the much stricter 15% global operating point, and no per-window normalization is described. If scores are uncalibrated across windows, the global mask can be dominated by a few high-motion windows, and the adaptive-over-random margins in Table 4d (47.3 vs 46.4 on EK100; 87.9 vs 86.3 on D48) could reflect an uneven sparsity pattern rather than content prioritization. Please add an analysis of rank-order stability as a function of budget and window, or an ablation with per-window normalization.","section":"§3.3 / A.5.1 / Table 10"},{"comment":"During tokenizer training, all tokens in the first frame are kept, but the paper does not state how the first latent frame of each 16-frame sliding window is handled at inference. Since importance is defined as the Euclidean distance between corresponding tokens in adjacent frames, the first latent frame of every window has no score. If these boundary frames are force-kept, the effective decoder budget exceeds 15%; if they are unscored and excluded, the global top-k can systematically ignore one-eighth of the temporal positions. The paper should specify the inference-time treatment of boundary frames and, ideally, ablate whether keeping or excluding them changes the reported results.","section":"§3.3 / A.5.1"},{"comment":"The headline comparison of 128-frame versus 32-frame pre-training when both are fine-tuned on 128 frames is missing a key implementation detail: how a model pre-trained on 32-frame clips is adapted to 128-frame inputs. In particular, the handling of spatio-temporal positional embeddings and any temporal interpolation or padding is not described. Without this, the reported gaps (47.3 vs 45.0 on EK100; 87.9 vs 83.2 on D48) may be confounded by the adaptation procedure rather than by pre-training length alone. The paper should also state how videos shorter than 128 frames are sampled or padded, since many EK100 clips are shorter than 128 frames.","section":"§4.3 / Table 2 / A.5.3"},{"comment":"The 128-frame masking-strategy comparison reports no error bars or number of seeds, while Table 1 reports standard deviations around 0.1 points at 32 frames. The margins of Adaptive over Random in Table 4d are only 0.9 points on EK100 and 1.6 points on D48, so without a significance assessment the claim that adaptive masking is the best strategy at 128 frames is not fully supported. Please report mean and standard deviation over multiple seeds, or a statistical test, for the Table 4d comparisons.","section":"Table 4d"},{"comment":"The adaptive tokenizer is pre-trained at 128x128 spatial resolution but used at 112x112 at inference, with no description of how the tokenizer or scorer adapts to this resolution change. Because the scorer's CNN features and the token grid change with resolution, this mismatch could affect both the importance scores and the reconstruction targets. Please clarify whether the resolution difference is intentional and discuss its effect on score calibration and downstream accuracy.","section":"A.5.1 / Table 10"}],"minor_comments":[{"comment":"The sentence containing 'respectfully' should read 'respectively'.","section":"§4.2"},{"comment":"The notation V Q(V) in Eq. (2) is introduced only informally; please define it clearly, for example as the quantized token embedding of the input video V.","section":"§3.1 / Eq. (2)"},{"comment":"The text 'N d = 0 .85N' contains an odd spacing and should be formatted as N_d = 0.85N.","section":"§3.2"},{"comment":"The FineGym288 experiments add a 1-second margin to temporal action boundaries; this modifies the benchmark and should be disclosed prominently in the main text or in the table caption, not only in the appendix.","section":"A.2.2"},{"comment":"Table 8 is described as using 'improved hyper-parameters' that yield better results than Table 4d; please state which hyperparameters changed and whether all models in Table 8 share the same settings, so readers can reconcile the two tables.","section":"A.3 / Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the central mechanism—a frozen scorer trained at a permissive per-window budget and then applied at a stricter global budget—needs direct validation before the claims can be fully trusted. The missing details about boundary-frame handling, positional-embedding adaptation to 128 frames, and short-video padding are all load-bearing for the paper's main comparisons. If the authors can provide the requested analyses and clarifications, the revised manuscript would be suitable for publication; I do not see a fundamental flaw that would require rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result holds up: pre-training a video MAE on 128 frames, enabled by decoder masking, beats 32-frame pre-training under matched long-context fine-tuning. That is the new, useful finding, and the paper earns credit for the control in Table 2, where both rows share the same 128-frame fine-tuning protocol. The frozen scorer design also avoids the main circularity risk. I agree with the reader's conditional verdict, with one caveat about the stress-test concern.\n\nWhat the paper does well: it gives a clean answer to \"does longer MAE pre-training help?\" — yes, up to 128 frames, with diminishing returns after 64. The SOTA numbers on Diving48 (+3.9) and EK100 verb (+2.5) are clearly disclosed as using ViT-L and unlabeled Kinetics-710, so no overclaiming there. The ablation set is reasonably thorough: masking strategies, reconstruction targets, token budgets, frame counts.\n\nThe soft spots are real but not fatal. The stress-test point about the scorer calibration is the sharpest: the scorer trains at a 49% per-window top-k but is used at a 15% global budget across concatenated windows, and the paper never checks rank-order fidelity at that operating point. Per-window normalization is not described, and how boundary latent frames are scored is ambiguous. If the global mask is dominated by a few high-motion windows, the adaptive-over-random gains in Table 4d (0.9 and 1.6 points) could partly reflect sparsity artifacts, not content ranking. However, that concern is softened by Table 1 at 32 frames, where Adaptive beats Random by a full point with error bars around 0.1 — that is a stable, significant margin, and the same artifact story would have to explain it. So I would not call the mechanism claim load-bearing for the whole paper, but it is undersupported. The missing code/checkpoints and variance only on Table 1 are minor-to-moderate reproducibility issues.\n\nWho should read this: anyone in video self-supervised learning, especially those working on MAE pre-training or long-context encoders. It deserves a serious referee. A referee should push for an experiment that isolates content ranking from token distribution — e.g., comparing against a deliberately anti-correlated scorer, or controlling the raw sparsity pattern — and should ask for per-window statistics at the 15% budget. With those additions, the paper would be quite strong. Without them, it is still a worthwhile empirical contribution.","headline":"A solid empirical paper on 128-frame video MAE with a real contribution, but the adaptive-masking mechanism is less well isolated than the long-context result.","tokens_in":23161,"tokens_out":2053,"would_cite":true,"duration_ms":22860,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video MAEs can pre-train on 128 frames by reconstructing only the top 15% of tokens.","keywords":["video masked autoencoders","long-context video pre-training","adaptive decoder masking","token importance scoring","FSQ-MagViT tokenizer","self-supervised video representation learning","action recognition"],"falsifier":"On a set of 128-frame videos with pure camera motion and no foreground change, compute the adaptive mask from the frozen scorer and compare it with random and flow-based masks; if the selected tokens follow the background and downstream accuracy matches random-mask training, the transferability claim is false. A more direct test is to measure per-token reconstruction error of a fully trained 128-frame MAE: if the scorer's top 15% tokens are not systematically harder to reconstruct than randomly chosen tokens, the priority signal is not doing the work the paper attributes to it.","tokens_in":21804,"feed_emoji":"🎬","tokens_out":9537,"duration_ms":95128,"temperature":0.7,"pith_summary":"Video masked autoencoders are usually pre-trained on 16 or 32 frames because decoding every masked token at longer lengths exhausts memory. This paper claims that pre-training can be extended to 128 frames by decoding only the 15% of tokens judged most important by a learned saliency scorer, and that doing so improves the resulting encoder. The ranking and the reconstruction targets come from a jointly trained adaptive FSQ-MagViT tokenizer, which is frozen and reused for MAE pre-training. The longer context beats 32-frame pre-training on the same downstream tasks and lifts state-of-the-art accuracy on Diving48 by 3.9 points and on EPIC-Kitchens-100 verb classification by 2.5 points, with a plain ViT encoder and video-only pre-training.","feed_headline":"Decoding 15% of tokens lets video MAEs see 128 frames","feed_subtitle":"Adaptive token scoring ranks what to rebuild, and the longer context beats 32-frame baselines.","key_machinery":"The carrying mechanism is the adaptive FSQ-MagViT tokenizer, a video tokenizer that learns both quantized reconstruction targets and a rank order over spatio-temporal tokens. A MAGVIT 3D-CNN encoder produces the tokens, finite scalar quantization maps them to a discrete codebook, and a CNN token scorer computes, for each token, the Euclidean distance between its feature and the feature of the corresponding token in the previous frame, treating that distance as importance. A differentiable top-k layer selects the 768 largest-importance tokens on 16-frame training clips and zeroes the rest, and the module is trained end-to-end on Kinetics-600 with pixel, GAN, and perceptual losses. Once frozen, this scorer generates the adaptive decoder mask (top 15% plus 5% random tokens) and the quantized target latents for MAE pre-training, which is what makes the decoder sparse enough for 128-frame videos.","core_discovery":"The central claim is that a content-dependent adaptive decoder mask, built from a learned token-importance ranking, makes 128-frame video MAE pre-training both feasible and better than short-context pre-training. With a 15% decoder token budget, the proposed adaptive masking outperforms random, uniform, optical-flow, and EVEREST-style masking at the same budget on 32-frame clips, and it matches or improves on a VideoMAE decoder that reconstructs all tokens. Pre-training at 128 frames then yields better fine-tuned accuracy than pre-training at 32 frames, including when the 32-frame model is fine-tuned with 128-frame context, and 128-frame multi-crop evaluation does not close the gap. On long-horizon action benchmarks the resulting LVMAE reaches state-of-the-art top-1 accuracy on Diving48 and EPIC-Kitchens-100 verb classification while using only unlabeled video pre-training and a single temporal crop at evaluation.","pith_inferences":["Because the importance scorer is trained once on 16-frame clips and applied by sliding a window over 128-frame videos, the same frozen scorer could plausibly transfer across MAE architectures, resolutions, and frame rates without retraining; the paper does not test this.","The paper's deliberate decoupling of scorer and MAE leaves open an untested variant in which the scorer is adapted during MAE pre-training, a direction that could improve masks but risks reintroducing the easy-token bias the decoupling avoids.","The finding that a 15% decoder budget beats a 50% budget suggests the optimal budget may track how redundant the video content is, so future work could vary the budget per clip rather than fixing it globally.","Current benchmarks contain few videos longer than 128 frames, so the method's benefit on truly long-form video is likely underestimated; datasets with longer episodes would be the natural test."],"forward_implications":["128-frame pre-training consistently outperforms 32-frame pre-training when both are fine-tuned on 128 frames, and it also beats 32-frame multi-crop evaluation.","At a 15% decoder token budget, adaptive masking is competitive with or better than decoding all masked tokens, so the memory savings transfer directly into longer temporal context rather than sacrificing accuracy.","The same recipe reaches state-of-the-art results on Diving48 and EPIC-Kitchens-100 verb classification with a standard ViT, one temporal crop, and no labeled or text-supervised pre-training.","Decoder masking ratios near the encoder masking ratio (about 90%) are needed to fit 128-frame pre-training in memory, and a 15% decoder budget outperforms more expensive budgets such as 50%.","Gains from extending context grow from 16 to 64 frames and then become small from 64 to 128, matching the short clip lengths of most current benchmarks."],"supporting_citations":[{"why":"Supplies the dual masking formulation, the VideoMAE architecture and training recipe that this work extends to adaptive decoder masking and 128 frames; also the uniform masking baseline.","marker":"[14]"},{"why":"Supplies the MAGVIT 3D-CNN video tokenizer architecture and training recipe used as the base of the adaptive FSQ-MagViT tokenizer.","marker":"[25]"},{"why":"Supplies finite scalar quantization, the simple quantizer used with MAGVIT to form the discrete reconstruction targets.","marker":"[36]"},{"why":"Defines the nearest prior work on learned informative-token reduction via feature distance; used as a baseline and the contrast that motivates learning saliency jointly with the tokenizer.","marker":"[17]"},{"why":"Supplies the VideoMAE short-video baseline, random tube masking, and the 90% encoder masking convention the paper adopts.","marker":"[15]"},{"why":"Supplies the Kinetics-600 dataset used to train the adaptive tokenizer and scorer that are frozen and reused for MAE pre-training.","marker":"[37]"},{"why":"Provides the Diving48 benchmark used for the long-range action classification evaluation and state-of-the-art comparison.","marker":"[19]"},{"why":"Provides the EPIC-Kitchens-100 benchmark used for verb, noun, and action evaluation and the reported state-of-the-art comparison.","marker":"[18]"},{"why":"Supplies the unlabeled Kinetics-710 videos used as additional pre-training data for the state-of-the-art LVMAE models.","marker":"[38]"},{"why":"Provides the memory-consolidation long-context model that is a strong Diving48 baseline which LVMAE surpasses.","marker":"[5]"},{"why":"Provides the strong EPIC-Kitchens verb-classification baseline that LVMAE surpasses without its large-scale pre-training resources.","marker":"[41]"}],"fun_headline_variants":["Adaptive masking extends video MAEs to 128 frames","Prioritized token decoding trains video MAEs at 128 frames","Long-video MAE: adaptive decoder mask beats random at 128 frames","LVMAE: 128-frame video pre-training via token prioritization","Smart token selection makes 128-frame video MAEs work"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen token-importance scorer, trained on short Kinetics-600 clips, keeps ranking the most valuable tokens correctly when it is slid across longer, higher-resolution videos; if its ranking degrades, adaptive masking collapses to random masking and the reported gains vanish.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive masking extends video MAEs to 128 frames","Prioritized token decoding trains video MAEs at 128 frames","Long-video MAE: adaptive decoder mask beats random at 128 frames","LVMAE: 128-frame video pre-training via token prioritization","Smart token selection makes 128-frame video MAEs work"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2231,"prompt_tokens":1023,"completion_tokens":1208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":1118}},"tokens_in":639,"tokens_out":1208,"duration_ms":8892,"temperature":1.0,"reasoning_tokens":1118,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:01:01.200512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a set of 128-frame videos with pure camera motion and no foreground change, compute the adaptive mask from the frozen scorer and compare it with random and flow-based masks; if the selected tokens follow the background and downstream accuracy matches random-mask training, the transferability claim is false. A more direct test is to measure per-token reconstruction error of a fully trained 128-frame MAE: if the scorer's top 15% tokens are not systematically harder to reconstruct than randomly chosen tokens, the priority signal is not doing the work the paper attributes to it.","supporting_citations":[{"cited_title":"Videomae v2: Scaling video masked autoencoders with dual masking","cited_arxiv_id":null,"evidence_quote":"Supplies the dual masking formulation, the VideoMAE architecture and training recipe that this work extends to adaptive decoder masking and 128 frames; also the uniform masking baseline."},{"cited_title":"Magvit: Masked generative video transformer","cited_arxiv_id":null,"evidence_quote":"Supplies the MAGVIT 3D-CNN video tokenizer architecture and training recipe used as the base of the adaptive FSQ-MagViT tokenizer."},{"cited_title":"Finite scalar quantization: VQ-V AE made simple","cited_arxiv_id":null,"evidence_quote":"Supplies finite scalar quantization, the simple quantizer used with MAGVIT to form the discrete reconstruction targets."},{"cited_title":"Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens","cited_arxiv_id":null,"evidence_quote":"Defines the nearest prior work on learned informative-token reduction via feature distance; used as a baseline and the contrast that motivates learning saliency jointly with the tokenizer."},{"cited_title":"VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training","cited_arxiv_id":null,"evidence_quote":"Supplies the VideoMAE short-video baseline, random tube masking, and the 90% encoder masking convention the paper adopts."},{"cited_title":"Resound: Towards action recognition without representation bias","cited_arxiv_id":null,"evidence_quote":"Provides the Diving48 benchmark used for the long-range action classification evaluation and state-of-the-art comparison."},{"cited_title":"Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100","cited_arxiv_id":null,"evidence_quote":"Provides the EPIC-Kitchens-100 benchmark used for verb, noun, and action evaluation and the reported state-of-the-art comparison."},{"cited_title":"Memory consolidation enables long-context video understanding","cited_arxiv_id":null,"evidence_quote":"Provides the memory-consolidation long-context model that is a strong Diving48 baseline which LVMAE surpasses."}],"review_version":1}