Pith. sign in

REVIEW 9 cited by

MaskViT: Masked Visual Pre-Training for Video Prediction

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.11894 v2 pith:VR37FNSX submitted 2022-06-23 cs.CV cs.LGcs.RO

Agrim Gupta , Stephen Tian , Yunzhi Zhang , Jiajun Wu , Roberto Martín-Martín , Li Fei-Fei This is my paper · ORCID
classification cs.CVcs.LGcs.RO
keywords maskvitvisualmaskmaskedpredictionvideoagentsdemonstrate
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The ability to predict future visual observations conditioned on past observations and motor commands can enable embodied agents to plan solutions to a variety of tasks in complex environments. This work shows that we can create good video prediction models by pre-training transformers via masked visual modeling. Our approach, named MaskViT, is based on two simple design decisions. First, for memory and training efficiency, we use two types of window attention: spatial and spatiotemporal. Second, during training, we mask a variable percentage of tokens instead of a fixed mask ratio. For inference, MaskViT generates all tokens via iterative refinement where we incrementally decrease the masking ratio following a mask scheduling function. On several datasets we demonstrate that MaskViT outperforms prior works in video prediction, is parameter efficient, and can generate high-resolution videos (256x256). Further, we demonstrate the benefits of inference speedup (up to 512x) due to iterative decoding by using MaskViT for planning on a real robot. Our work suggests that we can endow embodied agents with powerful predictive models by leveraging the general framework of masked visual modeling with minimal domain knowledge.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 45 citations worldwide. Full citation record

  1. NAE: Normalizing AutoEncoder

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...

  2. Self-Guided Masked Autoencoder

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A Masked Autoencoder that masks the object cluster found by its own early patch-clustering signal learns better representations than random masking, with no external labels or models.

  3. Boosting Bot Detection via Heterophily-Aware Representation Learning and Prototype-Guided Cluster Discovery

    cs.AI 2025-06 conditional novelty 6.0 of 10

    BotHP combines a dual-encoder (graph and MLP) with prototype-guided clustering to pre-train graph bot detectors, improving F1 by 1.3-6.0 points on TwiBot-20 and MGTAB.

  4. TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    TIV-Diffusion adds object-centric slot alignment to a diffusion-based image-to-video generator and reports improved alignment and temporal-consistency metrics on MNIST, CATER, and Bridge datasets.

  5. RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulation

    cs.RO 2025-10 unverdicted novelty 5.0 of 10

    Abstract describes RoDyn but full text describes iMoWM; the record is internally inconsistent and the headline claims are absent from the body.

  6. OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A shared-backbone transformer with pairwise modality training reports top results across 25 datasets spanning 12 modalities.

  7. Pre-Trained Video Generative Models as World Simulators

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A lightweight action-conditioning module and a motion-reinforced loss convert pre-trained video generators into action-following world simulators that also speed up model-based reinforcement learning.

  8. CrossVideoMAE: Self-Supervised Image-Video Representation Learning with Masked Autoencoders

    cs.CV 2025-02 reject novelty 5.0 of 10

    CrossVideoMAE combines intra-modal and cross-modal contrastive learning with masked autoencoding between videos and sampled frames, reporting modest SOTA gains on UCF101, HMDB51, K400, and SSv2.

  9. On the Benefits of Instance Decomposition in Video Prediction Models

    cs.CV 2025-01 reject novelty 4.0 of 10

    Explicit instance decomposition with per-class shared weights improves latent-transformer video prediction in the paper's experiments, but the claimed advantage is weakened by mismatched parameter counts and test-set ...

Pith tools