{"id":"96c2b330-c430-49e4-b49c-a5457318bb87","arxiv_id":"2412.15212","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Scaling masked autoencoding video transformers from 20M to 22B parameters steadily improved camera pose, tracking, and depth estimation, while language-supervised and image-only models lagged on these tasks.","lead":"Researchers trained video masked autoencoders at sizes up to 22 billion parameters, the largest self-supervised video models reported, and found consistent gains on spatial and temporal tasks such as camera pose, tracking, and depth. The result challenges the belief that this type of pretraining stops improving near one billion parameters and reopens the case for scaling pixel-only video learning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 22B datapoint is not a controlled scale comparison: the 22B recipe changes resolution, decoding layers, update count, and data volume at once, so the headline monotonic scaling claim is underdetermined past 4B.","rationale":"Inside the 20M-to-4B family, the comparison is clean: the same resolution, decoding layers, updates, and data volume are used (Table 5), and the tables show consistent gains on RE10k, ScanNet, and Waymo in both frozen and finetuned evaluation. This is real evidence and should be credited. The paper is also honest about its own limits, including the absence of scaling laws and the reliance on personal communication for negative results on MAE scaling. The weak point is precisely the 22B point that the abstract foregrounds. The confounds in Table 5 are not cosmetic: Section C shows the decoding configuration alone is worth about 0.03 RE10k EPE and 0.2 ScanNet AbsRel for 4DS-e, and Table 4 shows data volume matters substantially (ScanNet AbsRel improves from 1.39 to 0.94 when training examples go from 250M to 1B). The 4B-to-22B increment could therefore plausibly be driven by doubled updates, doubled data, or higher resolution rather than by the 5.6x parameter increase. The reader's CONDITIONAL verdict already captures this concern, so I do not move the verdict, but I would make the controlled 4B/22B-recipe comparison an explicit condition of acceptance. I did not elevate other fragilities, such as the post-hoc readout-layer selection or the appearance-dominated ScanNet signal, because those would weaken but not break the central claim; the uncontrolled 22B recipe is the single most load-bearing issue.","tokens_in":26028,"tokens_out":5670,"duration_ms":50430,"concrete_test":"Train or obtain a 4B (4DS-e) checkpoint using the 22B training recipe except for model size: 256x256 input, 2 decoding layers, 1M updates, and 2B pretraining clips. Evaluate it frozen with the same attention readouts on RE10k, Perception Test, ScanNet, and Waymo, and compare against the reported 4DS-j numbers. If the 4B model matches or exceeds 4DS-j, the 22B scaling gain is not established; if it lands near the current 4DS-e numbers, the confounds are quantitatively minor and the scaling conclusion stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim, 'consistently improving performance on these 4D tasks, as model size increases from 20M all the way to the largest ... 22B parameters', leans most heavily on the 4B-to-22B interval, but Table 5 of the supplementary material changes four training-protocol variables simultaneously for the 22B model: input resolution 224 to 256, decoding layers 4 to 2, output patch size 2x16x16 to 4x32x32, training updates 488k to 1M, and pretraining clips 1B to 2B. The paper's own decoding ablation in Table 8 shows that the 22B-style decoding configuration is the worse one for 4DS-e on RE10k (0.32 vs 0.29), ScanNet (1.15 vs 0.95), and SSv2 (60.6 vs 65.8). With two effects pointing in opposite directions (worse decoder versus more updates, higher resolution, and more data), the observed 22B improvements cannot be cleanly attributed to parameter count. No controlled 4B run at 256 resolution, 1M updates, 2 decoding layers, and 2B clips is reported, and no seeds or error bars are provided, so the frozen Waymo gain (78.0 to 78.3) in particular is within plausible noise. The 20M-to-4B trend is much better controlled; the 'all the way to 22B' conclusion is the vulnerable part of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that pure self-supervised masked auto-encoding on video does scale, contrary to the common belief that MAE saturates, provided evaluation targets non-semantic 4D tasks (camera pose, point tracking, object tracking, depth) rather than classification. It introduces SimpleMAE, a stripped-down video MAE without target normalization, tube masking, or a separate decoder, and trains a family of 7 ViT models (4DS) from 20M to 22B parameters on 170M web videos. The authors evaluate frozen and finetuned attention-based readouts on RE10k pose, Perception Test point tracking, Waymo object tracking, ScanNet depth, and SSv2/Kinetics classification, comparing against DINOv2, SigLIP, VideoPrism, V-JEPA, and VideoMAE baselines. The empirical core is a large set of internally consistent tables showing that 4DS models improve with size on the 4D tasks, with the 22B model best, plus supporting studies: data scaling, readout-layer sweep, decoding hyperparameter ablation, and distillation from 4B to 90M.","tokens_in":26259,"tokens_out":4830,"duration_ms":37489,"significance":"If the central claim is accepted, the paper is a substantial empirical contribution: it provides the first demonstration that video-only self-supervised models can be scaled beyond 1B parameters with consistent downstream gains on geometric and temporal tasks, it offers a large family of released checkpoints, and it challenges the semantic-centric view of scaling by showing that classification benchmarks may have obscured MAE's scaling behavior. The evaluation protocol is unusually careful: all models share identical readout architectures, training example budgets, and optimization schedules, and the paper includes controlled ablation studies (data scaling, layer depth, decoding choices, distillation). The main fragility is the 22B data point, which changes several training variables at once, together with the absence of uncertainty estimates on the headline monotonic trends. The paper explicitly acknowledges that it does not produce proper scaling laws, which is an honest limitation rather than a defect.","major_comments":[{"comment":"The 4B-to-22B interval is not a controlled scale comparison. Supp. Tab. 5 changes five protocol variables simultaneously for the 22B model (input resolution 224 to 256, decoding layers 4 to 2, output patch size 2x16x16 to 4x32x32, training updates 488k to 1M, pretraining clips 1B to 2B, plus a different learning rate). Supp. Tab. 8, which the paper itself reports, shows that the 22B-style decoding configuration is worse than the default for a 4B model on RE10k (0.32 vs 0.29), ScanNet (1.15 vs 0.95), and SSv2 (60.6 vs 65.8). Because these changes pull in opposite directions, the observed 22B improvements in Tables 2, 3, and 9 cannot be cleanly attributed to parameter count; this is precisely the interval that the abstract's 'all the way to 22B' assertion relies on. Please add a controlled 4B run using the 22B recipe, or explicitly restate the headline as a well-controlled 20M-to-4B trend plus a separately-confounded 22B data point.","section":"Supp. Tab. 5, Supp. Tab. 8; Sec. 3.2, 4.1"},{"comment":"The paper claims 'consistent' improvement without reporting any uncertainty. In the frozen evaluation, 4DS-G (1.8B) and 4DS-e (3.8B) tie on RE10k (0.27) and 4DS-G is better than 4DS-e on Perception Test (82.7 vs 82.4); in short finetuning RE10k is 0.29 for 4DS-G versus 0.28 for 4DS-e. The top-end increments are often tiny, for example frozen Waymo 78.0 (4DS-e) to 78.3 (4DS-j). Without seeds or confidence intervals, the strict monotonic reading of the results is not supported. Please report multiple seeds with confidence intervals, or soften the monotonicity claim to a general trend with stated noise.","section":"Tables 2, 3, 9; Fig. 7"},{"comment":"The ScanNet depth task may not measure 4D geometric-temporal representation as cleanly as the other three tasks. The paper itself notes that image models are strong on ScanNet 'likely because monocular cues dominate in short clips', and indeed DinoV2-g, a pure image model, achieves 0.91 AbsRel frozen versus 4DS-e's 0.95, and remains competitive under long finetuning (0.66 vs 0.65). The scaling improvement on ScanNet may therefore partly reflect monocular appearance or shape cues rather than temporal 3D understanding. Please quantify the temporal contribution (for example, a frame-order-perturbation test) or explicitly relegate depth to a secondary role when claiming gains in 4D representation quality.","section":"Sec. 4.1, Table 2, Sec. A.2.6"},{"comment":"The readout layer at 95% depth (75% for semantic tasks) was selected after inspecting a sweep on the same downstream tasks. Because the same choice is applied to all models, the across-size comparison is probably not severely biased, but the reported absolute numbers may be optimistic. This is a moderate concern rather than a fatal one; reporting results for a second readout layer (e.g., the average over layers) would strengthen the robustness of the scaling conclusion.","section":"Sec. 4.2, Fig. 6"}],"minor_comments":[{"comment":"The third contribution is written as a fragment: 'which produced c) three new MAE-VIT models...' Please rewrite for grammatical completeness.","section":"Sec. 1, Contributions"},{"comment":"Fig. 4b and 4c include the 22B model, but the caption of Fig. 4a only states that the 22B model is omitted from the reconstruction panel; please clarify that the omission applies only to the MAE reconstruction subfigure.","section":"Fig. 4 (caption)"},{"comment":"There is an empty placeholder in the point tracking qualitative description: 'provided in .' The figure reference is missing.","section":"Supp. Sec. B"},{"comment":"The sentence 'Neither evaluates the method on ScanNet videos and they all evaluate zero-shot' is internally contradictory given the immediately following sentence that DUSt3R zero-shot 'gets 0.088' on ScanNet; please rephrase.","section":"Supp. Sec. A.2.6"},{"comment":"For the Perception Test readout, the table header says 'CrossAttention(qkv size=1024, num heads=8)' but the text in Sec. A.2.4 says 'cross-attention with 1024 parameters'; please use consistent terminology (channels vs parameters).","section":"Table 6"},{"comment":"The table does not list the learning rate chosen for the 22B model despite the text mentioning different learning rates; adding this row would make the protocol fully reproducible.","section":"Supp. Tab. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution, and the 20M-to-4B evidence is convincing. The main risk is the oversold 22B claim: the authors themselves provide the decoder ablation that shows the 22B configuration is suboptimal, and the lack of any controlled run at the 22B recipe makes the headline 'all the way to 22B' statement underdetermined. This is fixable within the manuscript's scope by either adding the controlled run or carefully rephrasing the claim, so I recommend major revision rather than rejection. The external-benchmark evaluation design makes circularity unlikely, and I see no citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first paper to push pure self-supervised video MAE past 1B parameters and evaluate on geometry/temporal tasks rather than just classification, and that 20M-to-4B curve is convincing. Second, the headline claim of consistent scaling \"all the way to 22B\" is not actually established: the 22B run changes resolution, decoding layers, output patch size, training updates, and data volume all at once, so the 4B-to-22B improvement cannot be cleanly attributed to parameter count. The stress-test note lands; the paper's own decoding ablation shows the 22B-style configuration is the worse one on RE10k, ScanNet, and SSv2, and the frozen Waymo gain (78.0 to 78.3) is within plausible noise with no seeds or error bars. The 20M-to-4B trend, by contrast, is much better controlled, with monotonic gains across all four 4D tasks and consistent behavior under short, medium, and long finetuning.\n\nWhat the paper does well: SimpleMAE is a genuinely clean simplification, removing target normalization, tube masking, and a custom decoder, and the paper includes a real decoding-hyperparameter ablation, a data-scaling study, a layer-depth readout study, and a distillation experiment that yields a 90M model beating same-size training. The evaluation protocol is apples-to-apples across baselines with shared readouts and budgets, which is more than most papers in this area do. The authors also disclose their limitations honestly: no scaling laws, and the community belief about MAE saturation is cited as personal communication rather than synthetic certainty. Released checkpoints are a concrete contribution.\n\nSoft spots, in proportion: the 22B confound is the main one; the readout layer at 95% depth was chosen after inspecting a sweep, which introduces hindsight bias into the reported numbers; ScanNet depth is partly driven by monocular appearance cues and the paper admits this; the pretraining corpus is proprietary, so replication on the training side is impossible. None of these kill the central 20M-to-4B result, but they do mean the paper's strongest sentence is also its weakest.\n\nThis is a paper for anyone working on video self-supervised learning, scaling studies, or representation evaluation beyond classification. It deserves a serious referee: the empirical footprint is large, the evaluation lens is new and useful, and the release of checkpoints makes independent downstream verification possible. The right outcome is probably acceptance after revision that either adds a controlled 4B run at 22B-style settings or softens the \"all the way to 22B\" claim accordingly. I would engage with it and would cite the 4B results.","headline":"The 20M-to-4B scaling story is solid and useful, but the 'all the way to 22B' claim is undercut by a confounded recipe change, so the headline conclusion should be read with more caution than the paper admits.","tokens_in":27109,"tokens_out":1752,"would_cite":true,"duration_ms":17926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that purely self-supervised masked auto-encoding from video scales: larger video transformers, up to 22B parameters, consistently improve four geometry-and-motion tasks—camera pose, point tracking, object tracking, and…","keywords":["self-supervised video representation learning","masked auto-encoding","model scaling","4D scene understanding","video transformers","camera pose estimation","point tracking","depth estimation"],"falsifier":"Train or re-evaluate the 22B model under the same recipe as the smaller models—224x224 input, four latent decoding layers, 2x16x16 reconstruction patches, and 488k training updates—and compare frozen-feature results on RE10k, Perception Test, ScanNet, and Waymo against 4DS-e. If the 22B model does not beat the 4B model, or if a static-image model matches it on ScanNet depth (where monocular cues dominate short clips), the scaling conclusion is not established.","tokens_in":25710,"feed_emoji":"🎥","tokens_out":7632,"duration_ms":52553,"temperature":0.7,"pith_summary":"Self-supervised video pretraining has been seen as saturating around 1B parameters, but that judgement comes mostly from semantic benchmarks such as action classification. This paper re-runs the comparison on non-semantic 4D tasks—camera pose estimation, point and object tracking, and monocular depth, which combine three spatial dimensions with time—and claims that a deliberately simplified masked autoencoder (SimpleMAE) improves consistently as the video transformer grows from 20M to 22B parameters, the largest self-supervised video model reported to date. Under a shared frozen-feature protocol with identical attention readouts, the 22B model outperforms every baseline, including language-supervised video models and the previous ~1B self-supervised video models, on all four 4D tasks. The same trend holds for short, medium, and long finetuning, and a 90M student distilled from the 4B teacher beats same-size models. If the scaling claim is right, it shifts attention from language-supervised, classification-centric pretraining toward pure video self-supervision and geometric-temporal evaluation.","feed_headline":"Bigger self-supervised video models keep winning on 4D tasks","feed_subtitle":"Frozen-feature tests on camera pose, point/object tracking, and depth improve from 20M to 22B parameters.","key_machinery":"The load-bearing object is SimpleMAE, a deliberately bare-bones video masked autoencoder: it masks 95% of 2x16x16 space-time patches at random, feeds the rest through a Vision Transformer, appends a learned grid of latent tokens in the last few self-attention blocks, and decodes each latent patch with a single linear layer—no target normalization, no tube masking, no separate decoder. For evaluation, features from the encoder (read at 95% depth for the 4D tasks) are fed to task-specific cross-attention readouts, which are trained on top of either frozen or finetuned backbones. This shared protocol is what lets the paper attribute differences in performance to the pretrained representation rather than to the readout design.","core_discovery":"On the paper's own terms, the discovery is that masked auto-encoding with video transformers scales when the targets are 4D: from a 20M-parameter model to a 22B-parameter model (4DS-j), frozen-feature performance improves monotonically on RealEstate10k camera pose (mean endpoint error 0.73 to 0.24), Perception Test point tracking (average Jaccard 75.9 to 83.4), ScanNet depth (absolute relative error 2.05 to 0.84), and Waymo Open object tracking (IoU 69.6 to 78.3). The same upward trend holds with finetuning, and the largest model is best on all four tasks. The paper argues this contradicts the common belief that MAE has mediocre scaling properties, and that prior work missed the trend because it evaluated on semantic tasks such as Something-Something v2 and Kinetics, where gains saturate earlier and language supervision helps more.","pith_inferences":["One testable extension is to fit the seven 4DS points with a power law; the paper does not claim a scaling law, but its data suggest a smooth, still-improving curve that would make such a law testable with one or two larger runs.","The 95%-depth readout finding suggests that precise geometric-temporal information is concentrated in late layers; probing intermediate layers on a fixed model could reveal where 4D structure emerges during pretraining.","Because the 22B checkpoint uses a different input resolution, decoding grid, and update count, a controlled rerun matching the smaller models' recipe would separate scale from recipe; the paper does not provide this control.","If the scaling transfers, tasks like optical flow, video object segmentation, and physical prediction—all 4D in nature—are natural next beneficiaries, even though the paper does not evaluate them."],"forward_implications":["The same frozen-feature protocol, applied to any new video backbone, now has a reference scaling curve on four 4D tasks instead of only classification benchmarks.","If the trend continues beyond 22B, larger self-supervised video models should keep improving camera pose, tracking, and depth without any language supervision.","Language-supervised pretraining, while strong on classification, is not necessary—and may be harmful—for geometric-temporal representation quality.","Distillation from the large 4DS models offers a practical route to small, strong 4D backbones, as shown by the 90M distilled model beating its same-size counterpart.","Semantic and 4D tasks rank models differently, so conclusions about scaling from action-recognition benchmarks should not be transferred to geometry-and-motion tasks."],"supporting_citations":[{"why":"Supplies the masked autoencoding formulation that SimpleMAE strips down to its bare bones.","marker":"[39]"},{"why":"Provides the video MAE baseline and the tube-masking scheme SimpleMAE deliberately omits.","marker":"[67]"},{"why":"Documents the prior scaling attempt to roughly 1B parameters and the saturation claim this paper extends past.","marker":"[71]"},{"why":"Provides a strong feature-prediction video baseline and the attention-probing readout protocol reused here.","marker":"[8]"},{"why":"Supplies a language-supervised video baseline that underperforms on the 4D tasks.","marker":"[82]"},{"why":"Supplies a strong image-only self-supervised baseline for the image-versus-video comparison.","marker":"[50]"},{"why":"Supplies the 2B language-supervised image baseline used to test whether scaling images alone helps 4D tasks.","marker":"[15]"},{"why":"Defines the Vision Transformer backbone on which all 4DS models are built.","marker":"[24]"},{"why":"Provides the architecture-scaling reference for very large vision transformers, up to 22B parameters.","marker":"[19]"}],"fun_headline_variants":["Video MAE scaling: 22B model excels at 4D tasks","Self-supervised video scaling works for 4D, not just semantics","From 20M to 22B: Video MAE improves pose, tracking, depth","22B video MAE: best for pose, tracking, depth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the premise that the four tasks, evaluated with one shared frozen-feature protocol, genuinely measure spatial-temporal (4D) representation quality, and that the improvements are caused by model scale rather than by protocol choices or appearance-based shortcuts such as monocular depth cues.","fun_headline_variants_meta":{"raw":{"variants":["Video MAE scaling: 22B model excels at 4D tasks","Self-supervised video scaling works for 4D, not just semantics","From 20M to 22B: Video MAE improves pose, tracking, depth","22B video MAE: best for pose, tracking, depth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2408,"prompt_tokens":922,"completion_tokens":1486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1403}},"tokens_in":538,"tokens_out":1486,"duration_ms":8366,"temperature":1.0,"reasoning_tokens":1403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:33:18.295745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or re-evaluate the 22B model under the same recipe as the smaller models—224x224 input, four latent decoding layers, 2x16x16 reconstruction patches, and 488k training updates—and compare frozen-feature results on RE10k, Perception Test, ScanNet, and Waymo against 4DS-e. If the 22B model does not beat the 4B model, or if a static-image model matches it on ScanNet depth (where monocular cues dominate short clips), the scaling conclusion is not established.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Supplies the masked autoencoding formulation that SimpleMAE strips down to its bare bones."},{"cited_title":"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training","cited_arxiv_id":null,"evidence_quote":"Provides the video MAE baseline and the tube-masking scheme SimpleMAE deliberately omits."},{"cited_title":"Videomae v2: Scaling video masked autoencoders with dual masking","cited_arxiv_id":null,"evidence_quote":"Documents the prior scaling attempt to roughly 1B parameters and the saturation claim this paper extends past."},{"cited_title":"Revisiting feature prediction for learning visual rep- resentations from video","cited_arxiv_id":null,"evidence_quote":"Provides a strong feature-prediction video baseline and the attention-probing readout protocol reused here."},{"cited_title":"Videoprism: A foundational visual encoder for video understanding","cited_arxiv_id":null,"evidence_quote":"Supplies a language-supervised video baseline that underperforms on the 4D tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a strong image-only self-supervised baseline for the image-versus-video comparison."},{"cited_title":"PaLI-3 vision language models: Smaller, faster, stronger","cited_arxiv_id":null,"evidence_quote":"Supplies the 2B language-supervised image baseline used to test whether scaling images alone helps 4D tasks."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Defines the Vision Transformer backbone on which all 4DS models are built."},{"cited_title":"Scaling vision transformers to 22 billion pa- rameters","cited_arxiv_id":null,"evidence_quote":"Provides the architecture-scaling reference for very large vision transformers, up to 22B parameters."}],"review_version":1}