Pith. sign in

REVIEW 5 cited by

Rethinking Patch Dependence for Masked Autoencoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.14391 v2 pith:UY5EMRF4 submitted 2024-01-25 cs.CV

Letian Fu , Long Lian , Renhao Wang , Baifeng Shi , Xudong Wang , Adam Yala , Trevor Darrell , Alexei A. Efros
show 1 more author
Ken Goldberg
This is my paper
classification cs.CV
keywords maskedautoencoderscross-attentioncrossmaedecoderpatchestokensencoder
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we examine the impact of inter-patch dependencies in the decoder of masked autoencoders (MAE) on representation learning. We decompose the decoding mechanism for masked reconstruction into self-attention between mask tokens and cross-attention between masked and visible tokens. Our findings reveal that MAE reconstructs coherent images from visible patches not through interactions between patches in the decoder but by learning a global representation within the encoder. This discovery leads us to propose a simple visual pretraining framework: cross-attention masked autoencoders (CrossMAE). This framework employs only cross-attention in the decoder to independently read out reconstructions for a small subset of masked patches from encoder outputs. This approach achieves comparable or superior performance to traditional MAE across models ranging from ViT-S to ViT-H and significantly reduces computational requirements. By its design, CrossMAE challenges the necessity of interaction between mask tokens for effective masked pretraining. Code and models are publicly available: https://crossmae.github.io

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

    cs.RO 2025-09 conditional novelty 7.0 of 10

    Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.

  2. Separating Representation from Reconstruction Enables Scalable Text Encoders

    cs.CL 2026-07 accept novelty 6.5 of 10

    Separating representation from token reconstruction via a bipartite CrossBERT architecture restores scalable frozen text embeddings and enables high-masking complementary training.

  3. Visual Pre-Training on Unlabeled Images using Reinforcement Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Casting image-crop consistency as temporal-difference value learning improves visual representations on unlabeled web, scene, and video data.

  4. Robust Representation Learning in Masked Autoencoders

    cs.LG 2026-02 conditional novelty 5.0 of 10

    Masked Autoencoders build class-separable representations across depth and keep their embeddings directionally stable under blur and occlusion, which tracks their robust classification.

  5. Improving Joint Embedding Predictive Architecture with Diffusion Noise

    cs.CV 2025-07 conditional novelty 4.0 of 10

    Injecting EDM-style noise into masked-token position embeddings and adding two auxiliary losses improves I-JEPA's linear-probing accuracy by about 1.5 points on ImageNet-1K.

Pith tools