Pith. sign in

REVIEW 3 major objections 4 minor 63 references

Self-Guided Masked Autoencoder

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masked autoencoders learn pattern-based patch clusters almost immediately, and the paper shows that using this internal signal to choose masks accelerates pretraining without external models or labels.

desk verdict Nice early-clustering analysis and consistent empirical gains, but the exploitation-rate metric that triggers the schedule is mathematically broken, so the paper's central mechanism is unsupported. read the letter →

arxiv 2507.19773 v1 pith:CUA66PR6 submitted 2025-07-26 cs.CV

Jeongwoo Shin , Inseo Lee , Junho Lee , Joonseok Lee This is my paper
classification cs.CV
keywords maskedautoencoderself-supervisedlearninginformedmaskingpatchclusteringnormalizedcutexploitationraterepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked autoencoders (MAEs) reconstruct randomly hidden image patches, but this paper argues that random masking is wasteful: from very early in pretraining the MAE encoder already organizes patches into clusters of similar texture and color. The authors propose a self-guided MAE that reads out its own clustering progress, then masks the cluster containing the main object while leaving a few hint patches, shifting training effort to harder, less separable patches. The switch from random to informed masking happens only when an internal attention-based 'exploitation rate' shows the encoder's clusters are reliable, at roughly epoch 50 on ImageNet-1K. With no external models or labels, this raises MAE's linear probing on ImageNet-1K from 61.4 to 62.9 and fine-tuning from 82.5 to 83.2, with gains also on detection and segmentation. If the mechanism is right, MAE-style pretraining is not mainly learning to reconstruct pixels but to sharpen pattern-based patch clusters, and masking can be treated as a curriculum learned from the model itself.

What carries the argument

The load-bearing mechanism is the 'exploitation rate', an attention-rollout-style quantity that recursively accumulates, across decoder layers, how much of the signal used to build one set of tokens comes from another set; the paper uses the moment when mask-token exploitation first exceeds visible-token exploitation as the internal trigger for switching from random to self-generated informed masks. The informed masks themselves are built by Normalized Cut on the cosine-similarity matrix of the second-last encoder layer's patch embeddings, masking the cluster that most likely contains the main object and keeping a few random 'hint' patches so reconstruction of the object remains possible.

What would settle it

Pre-train the same ViT-B MAE for 400 epochs on ImageNet-1K, fix the informed-mask generator, and sweep the switch epoch over {10, 30, 50, 100, 150}; if linear probing does not peak at or near the exploitation-rate crossing, the attention proxy is not what drives the benefit and the scheduling evidence for early clustering weakens.

Watch

Extended reading notes

Core claim

The paper's central claim is that MAE internally learns pattern-based patch-level clustering from an extremely early stage of pretraining, and that the encoder's own progress can be read out to build an informed mask that accelerates and strengthens representation learning. Concretely, the authors show that already around epoch 50 of a 400-epoch ImageNet run, the decoder's accumulated attention to mask tokens exceeds its attention to visible tokens, which they interpret as evidence that mask tokens now carry the shared, image-wide information the encoder has learned; at that point the model can bi-partition patches with Normalized Cut and mask the cluster containing the main object, leaving a few random hint patches. This self-guided scheme needs no external model, no labels, and no second stage: it is the same MAE with a different mask generator from epoch T onward. The paper reports consistent improvements over vanilla MAE and the external-free baseline AMT across linear probing, fine-tuning, object detection, and semantic segmentation, and shows the gain persists when pretraining is extended to 800 or 1600 epochs.

Load-bearing premise

The load-bearing premise is that the decoder's attention-based 'exploitation rate' faithfully measures how much useful shared information the mask tokens carry, so the epoch where mask-token exploitation exceeds visible-token exploitation is truly the moment when the encoder's patch clusters are reliable.

Editorial extensions

If this is right

  • If the discovery is correct, MAE's early training steps are spent repeatedly rediscovering the same coarse foreground/background split, so any masking scheme that focuses on the harder, less separable patches should speed up the same objective at equal epoch count.
  • The exploitation-rate threshold gives a training-time, model-generated criterion for curriculum scheduling; the same idea could schedule masking ratio, patch size, or reconstruction target as training progresses.
  • Because the scheme needs no external model or labels, it preserves MAE's fully self-supervised nature and adds only about 0.25% training time, so the gains come at negligible extra cost.
  • Extended pretraining shows the advantage widening at 800 epochs (65.9 vs 63.8 linear probing), suggesting the benefit compounds with longer training rather than saturating.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editors' note: a testable extension is to replace the attention-based exploitation-rate trigger with a direct measure of bi-partition quality, such as normalized mutual information between consecutive graph-cut results; if the same gains appear when the switch is made at materially different epochs, the attention proxy is not what carries the benefit.
  • Editors' note: if pattern-based clustering is the true objective, informed masking is only one instantiation; reweighting the reconstruction loss toward harder clusters or annealing the masking ratio could achieve the same curriculum without any mask generator at all.
  • Editors' note: the method's object-centric assumption suggests a sharper prediction the paper does not test: gains should be largest on object-centric datasets and smallest on scene-centric or highly fragmented images, which is consistent with the paper's own stated limitation.
  • Editors' note: the analysis implies that early-clustering behavior should be visible in any reconstruction objective, so the same exploitation-rate analysis could be applied to other MIM variants to sort out whether the phenomenon is specific to MAE's asymmetric design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper analyzes what MAE learns internally and claims that MAE forms pattern-based patch-level clusters very early in pretraining. Based on this analysis, it proposes "self-guided" masking: after roughly 50 epochs the model computes a similarity matrix from its own encoder embeddings, bipartitions the image with Normalized Cut, masks the object-like cluster, leaves a few hint tokens, and continues pretraining without any external model or label. The authors report consistent improvements over MAE and AMT on ImageNet-1K linear probing and fine-tuning, iNat2019, CIFAR-100, CUB, COCO detection, and ADE20K segmentation, with sustained gains under extended training to 800 and 1600 epochs.

Significance. If the reported gains hold under repeated runs, the paper would make a useful and reasonably practical contribution: an external-model-free way to accelerate MAE pretraining, together with a falsifiable analysis claim that MAE's patch embeddings become clusterable at an early stage. The breadth of the downstream evaluation and the extended-training results in Table V are genuine strengths, and the method's extra pretraining overhead is reported as negligible. The main weaknesses are that the quantitative trigger of the method rests on an information-flow metric whose definition is internally inconsistent, and that the empirical comparisons are single runs without variance estimates.

major comments (3)
  1. [Section 3.4, Eqs. (6)-(7)] The exploitation-rate metric is not a valid measure of information flow, and the switching rule derived from it is unsupported. With the stated base case R(0)_{A->B}=1, if A and B partition the token set O then row-stochasticity of the attention matrix gives r(l)_{A->B}+r(l)_{B->B}=1 for every l, so the recurrence yields R(l)_{A->B}=1 for all l by induction; Eq. (7) then gives R_V->O = R_M->O = 1 identically and Figure 4 cannot exhibit a crossover. The correct initial condition would be the indicator 1[A=B], but even with that correction Eq. (6) is not the attention-rollout product cited in [1]: it sums rather than multiplies and can exceed 1 (e.g., R(1)_{M->M}=2m under uniform attention). Moreover, with the corrected base and uniform row-stochastic attention, Eq. (7) already gives R_M->O > R_V->O for the masking ratio m=0.75, so the claim that the mask tokens "surpass" the visible tokens only after T≈50 is an artifact of the unnormalized set-size weighting. The method's switch epoch, and hence a large part of the self-guidance claim, therefore needs either a correctly defined metric or an explicit ablation over T.
  2. [Section 5.2, Table 2 and Appendix F, Table V] All reported numbers are single runs without multiple seeds or error bars. Several improvements are small relative to typical pretraining noise (CIFAR-100 FT 78.9 vs. 78.7; COCO APbox 43.3 vs. 43.0; ADE20K mIoU 45.2 vs. 45.0), so the "consistently outperforms" claim is not established as written. At minimum, repeated pretraining runs (e.g., three seeds) for the main comparison, or a clear statement of the expected variance in the protocol, are needed.
  3. [Section 4, Tables 3 and II] The method's only adaptive component is the switch epoch T, but T is never ablated. The paper states that T≈50 "may differ depending on the model or dataset," yet the ablations vary only the layer, target cluster, hint strategy, and masking ratio. The hint ratio is fixed at 0.05 (Figure II) without sensitivity analysis. Without an ablation over T (e.g., T in {0, 25, 50, 100} or a random schedule), the reported gains cannot be attributed to the self-guiding criterion rather than to a fixed early switch to a harder, object-centric masking distribution.
minor comments (4)
  1. [Section 3.4, Eq. (7)] The notation R_{V->O} is misleading: the equation is not an average over all tokens O but a weighted combination of R_{V->M} and R_{V->V} with weights m and 1-m; please clarify the definition.
  2. [Figure 4] Figure 4 reports curves for Layers 1, 2, 3, and 8, while the text says "every decoder layer"; please specify which layers are shown and whether multi-head attention scores are averaged over heads.
  3. [Section 3.3] The early-clustering analysis is contrasted only with MoCo and ViT; comparing with another MIM method (e.g., SimMIM or BEiT) would strengthen the claim that patch-level clustering is specific to MAE rather than a general transformer property.
  4. [Section 4] The paper states that the method is "completely free from any external models or extra information," but the choice of the second-last encoder layer is made from validation-set analyses of attention distance and NMI; please state this selection criterion explicitly when claiming parameter-freeness.

Circularity Check

1 steps flagged · score 4.0 of 10

Exploitation-rate readiness trigger is self-definitional: under uniform attention the metric's own recursion already satisfies the crossover used to set T≈50, while the central downstream benchmarking remains external.

  1. self definitional [Section 3.4 (Eqs. 5-7, Fig. 4) and Section 4, 'When to Start Informed Masking']
    "R(l) A→B = r(l) A→B · R(l−1) A→A + r(l) B→B · R(l−1) A→B, (6) where R(0) A→B = 1. ... R(l) V→O = m·R(l) V→M + (1−m)·R(l) V→V , R(l) M→O = m·R(l) M→M + (1−m)·R(l) M→V .(7) ... start informed masking when it becomes comparable to the information in visible tokens (R(L) M→O ≥ R(L) V→O ), i.e., around T epochs in Figure 4. ... Heavy exploitation of the mask tokens in decoder after T≈50 implies that the encoder is presently trained sufficiently to cluster the patches."

    With the unit base case R(0)_A→B = 1 in Eq. 6, the accumulated rates are not probabilities: under the near-uniform softmax attention expected at initialization with m = 0.75, R(1)_M→O = R(1)_V→O = 1.25 at layer 1 and R(2)_M→O ≈ 1.81 > R(2)_V→O ≈ 1.19 at layer 2. Thus the readiness condition R(L)_M→O ≥ R(L)_V→O that Section 4 declares to mean mask tokens 'have comparable amount of information to the visible tokens' — and that Section 3.4 says 'implies that the encoder is presently trained sufficiently to cluster the patches' — is already met by uniform, untrained attention, purely from the majority-mask weighting and the unit base case.

full rationale

The paper's central empirical claims are validated against external benchmarks with no parameters fitted to them: Table 2 (62.9 vs 61.4/61.7 linear probing; 83.2 vs 82.5/82.8 fine-tuning), Table V (65.9/68.7 at 800/1600 epochs), COCO and ADE20K gains, and the extended-training table. These results are genuine external evidence and do not reduce to the method's internal definitions. The 'MAE learns pattern-based patch-level clustering from early stages' analysis likewise has support independent of the contested metric: the bi-partitioning gap (Fig. 3a), KL-divergence convergence (Fig. 3b and Fig. IV), NMI, attention distance, and Fourier measurements (Figs. 6-7) directly measure the model's representations without presupposing the conclusion. The one load-bearing step that is constructed rather than measured is the exploitation-rate trigger: Eq. 6 sets R(0)_A→B = 1 for all token sets (not the identity base case of attention rollout, which the metric claims to specialize), and Eq. 7 mixes by the masking ratio m. Under the near-uniform attention at initialization, the recursion already yields R(L)_M→O ≥ R(L)_V→O at layer 2 onward, with rates exceeding 1, so the condition used to certify encoder readiness and set T≈50 is satisfied by the metric's own construction; the inference that mask tokens 'truly hold substantial shared information' does not follow. Because this same crossover is presented (Section 3.4 and Appendix A: 'This process is verified in Figure 4') as the verification that the encoder is 'presently trained sufficiently to cluster the patches,' the internal derivation is partially circular by construction. Self-citation is limited to [40] (an author-overlapping video-MIM entry in a Related Work enumeration) and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The paper's own caveat that T 'may differ depending on the model or dataset' concedes the trigger's fragility. Net: one partial, construction-based circularity in the schedule's internal trigger; the central external validation keeps the overall claim substantially independent.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The method rests on several hand-chosen hyperparameters (T, layer, hint ratio) and borrowed domain assumptions. The exploitation rate is a new metric rather than an entity; it is the main load-bearing proxy.

free parameters (4)
  • Switch epoch T = 50
    Chosen from the epoch where decoder attention on mask tokens overtakes visible tokens (Fig. 4); the paper notes it may vary with model or dataset.
  • Encoder layer for similarity matrix = Layer 11 (second-last of ViT-B)
    Selected because it has the highest attention distance and sufficient NMI among candidate layers; ablation shows only minor effects.
  • Hint ratio = 0.05
    Mentioned in the Figure II caption; number of hint patches kept visible inside the masked object cluster.
  • Masking ratio = 0.75
    Inherited from the MAE default; ablation shows 0.6 gives equal linear probing, so the choice is not tight.
assumptions (5)
  • standard math Ncut can be approximated by thresholding the second smallest eigenvector of the normalized graph Laplacian (Shi and Malik [44]).
    Used in Section 4 to bi-partition patches without labels.
  • domain assumption The hierarchical latent variable model of Kong et al. [29] correctly describes MAE, i.e., encoder estimates shared information c from visible patches.
    Adopted in Sections 2 and Appendix A to justify that mask tokens carry shared information.
  • domain assumption Higher variance of normalized patch embeddings and of pairwise similarities indicates stronger pattern-based clustering.
    Used in Section 3.2 to conclude that MAE clusters patches.
  • ad hoc to paper Attention mass on mask tokens at the decoder is a good indicator of the amount of shared information c used by the decoder.
    Stated in Section 3.4 as the basis for the exploitation rate metric and the choice of T.
  • domain assumption The token with the largest absolute entry in the Ncut eigenvector tends to belong to the main object.
    Borrowed from Wang et al. [55] in Section 4 to justify object-centric masking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Guided Masked Autoencoder." pith.science (2026). https://pith.science/paper/CUA66PR6

@misc{pith2026250719773,
  author       = {Pith},
  title        = {Pith review of: Self-Guided Masked Autoencoder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUA66PR6}},
  note         = {Machine review of arXiv:2507.19773}
}
read the original abstract

Masked Autoencoder (MAE) is a self-supervised approach for representation learning, widely applicable to a variety of downstream tasks in computer vision. In spite of its success, it is still not fully uncovered what and how MAE exactly learns. In this paper, with an in-depth analysis, we discover that MAE intrinsically learns pattern-based patch-level clustering from surprisingly early stages of pretraining. Upon this understanding, we propose self-guided masked autoencoder, which internally generates informed mask by utilizing its progress in patch clustering, substituting the naive random masking of the vanilla MAE. Our approach significantly boosts its learning process without relying on any external models or supplementary information, keeping the benefit of self-supervised nature of MAE intact. Comprehensive experiments on various downstream tasks verify the effectiveness of the proposed method.

Figures

Figures reproduced from arXiv: 2507.19773 by the authors.

Figure 1
Figure 1. Illustration of our self-guided MAE [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Relationships among the patch embeddings. (a) Pairwise similarity matrix for all 196 × 196 pairs of patches. (b) Similarity between the mean patch and all individual patches. (c) Attention score of the class token. the shared information cˆ from the visible patches Xv, and the decoder D([E(Xv); m]) performs the reconstruction task by inducing Xm from cˆ via the mask tokens. 3 Analysis of MAE We study what and how MA… view at source ↗
Figure 3
Figure 3. MAE learns patch clustering from very early stage of training process. (a) MAE widens the gap µintra − µinter. (b) Token relations drastically converge at early epochs and then gradually level off. Numbers in the legend denote the layer i. More details are provided in Section B. Figure 3b depicts δi(j) for even-numbered layers up to 400 epochs, measured with M and A. It clearly shows that δi monotonically decreases,… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Exploitation rate. Empirical Analysis. We measure the exploitation rate of visible tokens (R (l) V→O) and mask tokens (R (l) M→O) in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Examples of Self-guided Informed masking. More examples and detailed explanations on our method are displayed in Section A. separated clusters. We emphasize that MAE is still trained in a single stage; at epoch T, we begin generating informed masks and continue the tra…
Figure 6
Figure 6. Figure 6: MAE properties. Appropriate Layer for Patch Clustering. We consider attention distance and Normalized Mutual Information (NMI) in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Metrics explaining our performance gain. Layers left on the red dotted line belong to the encoder, and the rest to the decoder. 6 Related Work Masked Image Modeling (MIM). Inspired by Masked Language Modeling [13, 8], MIM has been widely applied in image [63, 11, 43, 7…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 42 canonical work pages

  1. [1]

    Abnar and W

    S. Abnar and W. Zuidema. Quantifying attention flow in transformers.arXiv:2005.00928, 2020

    arXiv 2005

  2. [2]

    Assran, M

    M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas. Masked siamese networks for label-efficient learning. InECCV, 2022

    work page 2022

  3. [3]

    MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning

    S. Atito, M. Awais, A. Farooq, Z. Feng, and J. Kittler. MC-SSL0.0: Towards multi-concept self-supervised learning.arXiv:2111.15340, 2021

    work page Pith review arXiv 2021

  4. [4]

    Atito, M

    S. Atito, M. Awais, and J. Kittler. SiT: Self-supervised vision transformer.arXiv:2104.03602, 2021

    arXiv 2021

  5. [5]

    Bachmann, D

    R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. InECCV, 2022

    work page 2022

  6. [6]

    W. G. C. Bandara, N. Patel, A. Gholami, M. Nikkhah, M. Agrawal, and V . M. Patel. AdaMAE: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. InCVPR, 2023

    work page 2023

  7. [7]

    H. Bao, L. Dong, S. Piao, and F. Wei. BEiT: Bert pre-training of image transformers. arXiv:2106.08254, 2021

    arXiv 2021

  8. [8]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. InNeurIPS, 2020

    work page 2020

Show all 63 references
  1. [9]

    S. Cao, P. Xu, and D. A. Clifton. How to understand masked autoencoders.arXiv:2202.03670, 2022

    2022 arXiv

  2. [10]

    H. Chen, W. Zhang, Y . Wang, and X. Yang. Improving masked autoencoders by learning where to mask.arXiv:2303.06583, 2023

    2023 arXiv

  3. [11]

    M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. InICML, 2020

    2020

  4. [12]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. InCVPR, 2009

    2009

  5. [13]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding.arXiv:1810.04805, 2018

    2018 arXiv

  6. [14]

    X. Dong, J. Bao, T. Zhang, D. Chen, W. Zhang, L. Yuan, D. Chen, F. Wen, N. Yu, and B. Guo. PeCo: Perceptual codebook for bert pre-training of vision transformers. InAAAI, 2023

    2023

  7. [15]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv:2010.11929, 2020

    2010 arXiv

  8. [16]

    Y . Fang, S. Yang, S. Wang, Y . Ge, Y . Shan, and X. Wang. Unleashing vanilla vision transformer with masked image modeling for object detection. InICCV, 2023

    2023

  9. [17]

    Feichtenhofer, Y

    C. Feichtenhofer, Y . Li, K. He, et al. Masked autoencoders as spatiotemporal learners. In NeurIPS, 2022

    2022

  10. [18]

    P. Gao, T. Ma, H. Li, Z. Lin, J. Dai, and Y . Qiao. MCMAE: Masked convolution meets masked autoencoders. InNeurIPS, 2022

    2022

  11. [19]

    Girdhar, A

    R. Girdhar, A. El-Nouby, M. Singh, K. V . Alwala, A. Joulin, and I. Misra. OmniMAE: Single model masked pretraining on images and videos. InCVPR, 2023

    2023

  12. [20]

    Gupta, S

    A. Gupta, S. Tian, Y . Zhang, J. Wu, R. Martín-Martín, and L. Fei-Fei. MaskViT: Masked visual pre-training for video prediction.arXiv:2206.11894, 2022

    2022 arXiv

  13. [21]

    Gupta, J

    A. Gupta, J. Wu, J. Deng, and L. Fei-Fei. Siamese masked autoencoders.arXiv:2305.14344, 2023. 11

    2023 arXiv

  14. [22]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. InCVPR, 2022

    2022

  15. [23]

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. InCVPR, 2020

    2020

  16. [24]

    K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. InICCV, 2017

    2017

  17. [25]

    Z. Hou, F. Sun, Y .-K. Chen, Y . Xie, and S.-Y . Kung. MILAN: Masked image pretraining on language assisted representation.arXiv:2208.06049, 2022

    2022 arXiv

  18. [26]

    Huang, Z

    B. Huang, Z. Zhao, G. Zhang, Y . Qiao, and L. Wang. MGMAE: Motion guided masking for video masked autoencoding. InICCV, 2023

    2023

  19. [27]

    Huang, X

    Z. Huang, X. Jin, C. Lu, Q. Hou, M.-M. Cheng, D. Fu, X. Shen, and J. Feng. Contrastive masked autoencoders are stronger vision learners.arXiv:2207.13532, 2022

    2022 arXiv

  20. [28]

    Kakogeorgiou, S

    I. Kakogeorgiou, S. Gidaris, B. Psomas, Y . Avrithis, A. Bursuc, K. Karantzalos, and N. Ko- modakis. What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022

    2022

  21. [29]

    L. Kong, M. Q. Ma, G. Chen, E. P. Xing, Y . Chi, L.-P. Morency, and K. Zhang. Understanding masked autoencoders via hierarchical latent variable models. InCVPR, 2023

    2023

  22. [30]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009

    2009

  23. [31]

    J. D. Lee, Q. Lei, N. Saunshi, and J. Zhuo. Predicting what you already know helps: Provable self-supervised learning. InNeurIPS, 2021

    2021

  24. [32]

    G. Li, H. Zheng, D. Liu, C. Wang, B. Su, and C. Zheng. SemMAE: Semantic-guided masking for learning masked autoencoders. InNeurIPS, 2022

    2022

  25. [33]

    X. Li, Y . Ge, K. Yi, Z. Hu, Y . Shan, and L.-Y . Duan. mc-BEiT: Multi-choice discretization for image bert pre-training. InECCV, 2022

    2022

  26. [34]

    Z. Li, Z. Chen, F. Yang, W. Li, Y . Zhu, C. Zhao, R. Deng, L. Wu, R. Zhao, M. Tang, et al. MST: Masked self-supervised transformer for visual representation. InNeurIPS, 2021

    2021

  27. [35]

    Liang, Y

    F. Liang, Y . Li, and D. Marculescu. SupMAE: Supervised masked autoencoders are efficient vision learners.arXiv:2205.14540, 2022

    2022 arXiv

  28. [36]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. InECCV, 2014

    2014

  29. [37]

    Y . Liu, S. Zhang, J. Chen, K. Chen, and D. Lin. PixMIM: Rethinking pixel reconstruction in masked image modeling.arXiv:2303.02416, 2023

    2023 arXiv

  30. [38]

    Z. Liu, J. Gui, and H. Luo. Good helper is around you: Attention-driven masked image modeling. InAAAI, 2023

    2023

  31. [39]

    Madan, N.-C

    N. Madan, N.-C. Ristea, K. Nasrollahi, T. B. Moeslund, and R. T. Ionescu. Cl-mae: Curriculum- learned masked autoencoders. InProceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 2492–2502, 2024

    2024

  32. [40]

    J. Mun, M. Shin, G. Han, S. Lee, S. Ha, J. Lee, and E.-S. Kim. BaSSL: Boundary-aware self-supervised learning for video scene segmentation. InACCV, 2022

    2022

  33. [41]

    J. Pan, P. Zhou, and S. Yan. Towards understanding why mask-reconstruction pretraining helps in downstream tasks.arXiv:2206.03826, 2022

    2022 arXiv

  34. [42]

    N. Park, W. Kim, B. Heo, T. Kim, and S. Yun. What do self-supervised vision transformers learn?arXiv:2305.00729, 2023

    2023 arXiv

  35. [43]

    Pathak, P

    D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. InCVPR, 2016. 12

    2016

  36. [44]

    Shi and J

    J. Shi and J. Malik. Normalized cuts and image segmentation.IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000

    2000

  37. [45]

    Y . Shi, N. Siddharth, P. Torr, and A. R. Kosiorek. Adversarial masking for self-supervised learning. InICML, 2022

    2022

  38. [46]

    Strehl and J

    A. Strehl and J. Ghosh. Cluster ensembles—a knowledge reuse framework for combining multiple partitions.Journal of Machine Learning Research, 3(12):583–617, 2002

    2002

  39. [47]

    Z. Tong, Y . Song, J. Wang, and L. Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. InNeurIPS, 2022

    2022

  40. [48]

    Van Horn, O

    G. Van Horn, O. Mac Aodha, Y . Song, Y . Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inaturalist species classification and detection dataset. InCVPR, 2018

    2018

  41. [49]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD birds-200-2011 dataset. Technical report, California Institute of Technology, 2011

    2011

  42. [50]

    H. Wang, K. Song, J. Fan, Y . Wang, J. Xie, and Z. Zhang. Hard patches mining for masked image modeling. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10375–10385, 2023

    2023

  43. [51]

    H. Wang, Y . Tang, Y . Wang, J. Guo, Z.-H. Deng, and K. Han. Masked image modeling with local multi-scale reconstruction. InCVPR, 2023

    2023

  44. [52]

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y . He, Y . Wang, Y . Wang, and Y . Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. InCVPR, 2023

    2023

  45. [53]

    R. Wang, D. Chen, Z. Wu, Y . Chen, X. Dai, M. Liu, Y .-G. Jiang, L. Zhou, and L. Yuan. BEVT: BERT pretraining of video transformers. InCVPR, 2022

    2022

  46. [54]

    Y . Wang, Z. Pan, X. Li, Z. Cao, K. Xian, and J. Zhang. Less is more: Consistent video depth estimation with masked frames modeling. InACM MM, 2022

    2022

  47. [55]

    Y . Wang, X. Shen, S. X. Hu, Y . Yuan, J. L. Crowley, and D. Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. InCVPR, 2022

    2022

  48. [56]

    C. Wei, H. Fan, S. Xie, C.-Y . Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. InCVPR, 2022

    2022

  49. [57]

    T. Xiao, Y . Liu, B. Zhou, Y . Jiang, and J. Sun. Unified perceptual parsing for scene understanding. InECCV, 2018

    2018

  50. [58]

    J. Xie, W. Li, X. Zhan, Z. Liu, Y . S. Ong, and C. C. Loy. Masked frequency modeling for self-supervised visual pre-training.arXiv:2206.07706, 2022

    2022 arXiv

  51. [59]

    Z. Xie, Z. Zhang, Y . Cao, Y . Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. SimMIM: A simple framework for masked image modeling. InCVPR, 2022

    2022

  52. [60]

    W. Yan, Y . Zhang, P. Abbeel, and A. Srinivas. VideoGPT: Video generation using vq-vae and transformers.arXiv:2104.10157, 2021

    2021 arXiv

  53. [61]

    Zhang, Y

    Q. Zhang, Y . Wang, and Y . Wang. How mask matters: Towards theoretical understandings of masked autoencoders. InNeurIPS, 2022

    2022

  54. [62]

    B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    2019

  55. [63]

    J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. iBOT: Image bert pre-training with online tokenizer.arXiv:2111.07832, 2021. 13 Appendix A Method Elaboration Detailed Reasoning for Our Method.As discussed in Section 2, the true shared information c exists for...

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.