Pith. sign in

REVIEW 3 major objections 5 minor 72 references

PR-MIM: Delving Deeper into Partial Reconstruction in Masked Image Modeling

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that masked-image pre-training can throw away half its masked tokens and still match full accuracy, if the thrown tokens are reconstructed from nearby retained tokens and the retained tokens are spread as far apart as…

desk verdict A simple, broadly validated efficiency trick for MAE-style pre-training, with the main caveats being single-run numbers, a missing ViT-L/16 control, and no code yet. read the letter →

arxiv 2411.15746 v1 pith:QB4EVB5T submitted 2024-11-24 cs.CV

Zhong-Yu Li , Yunheng Li , Deng-Ping Fan , Ming-Ming Cheng This is my paper · ORCID
classification cs.CV
keywords maskedimagemodelingpartialreconstructionprogressivefurthestsamplingself-supervisedrepresentationlearningvisiontransformerpre-trainingefficiencyNet-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Masked image modeling (MIM) pre-training is accurate but expensive, and a known shortcut called partial reconstruction throws away a subset of masked tokens so the decoder has less to process, saving compute but degrading the learned representations. This paper claims to eliminate that degradation with two additions: a progressive reconstruction step in which a tiny $7\times 7$ depthwise convolution reconstructs the thrown tokens from the retained ones, and a furthest-sampling rule that keeps the retained tokens as spread out as possible. With a 50% throwing ratio on ViT-B/16, the method reports ImageNet-1K accuracy identical to standard MAE while using about 72% of the FLOPs, 64% of the memory, and 75% of the pre-training time. If true, the result means MIM pre-training can be accelerated by roughly a quarter to a third without losing the quality of the learned representations.

What carries the argument

The load-bearing object is the progressive reconstruction scheme: after the standard MAE decoder reconstructs the retained masked tokens, a $7\times 7$ depthwise convolution aggregates the decoded unmasked and retained tokens in spatial layout to fill in the thrown token positions, costing about $7.3\times 10^{-3}$ GFLOPs. The companion mechanism is furthest sampling, a greedy farthest-point selection over the token grid that keeps retained masked tokens as dispersed as possible so that every thrown token has nearby support inside the $7\times 7$ window. Together they supply supervision from every masked token while keeping the decoder's input sequence short, which is what preserves representation quality at reduced pre-training cost.

What would settle it

Run Algorithm 1 on the $14\times 14$ token grid of a ViT-B/16 image with a 75% masking ratio and throwing ratios of 25%, 50%, and 65%, and count how many thrown tokens have no retained or unthrown token within a $7\times 7$ neighborhood; if a non-negligible fraction of thrown tokens is uncovered while the accuracy gain persists, then the mechanism the paper credits for the gain is not the one doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that the performance loss from partial reconstruction comes not from the shorter decoder sequence but from thrown tokens being abandoned, so the fix is to keep every masked token in the loss while preserving the cost saving. PR-MIM does this with a progressive reconstruction scheme: after the standard decoder reconstructs the retained masked tokens, a $7\times 7$ depthwise convolution rearranges the decoder outputs into spatial form, fills the thrown positions with zeros, and predicts those tokens from neighboring retained and unthrown tokens, at an added cost of only $7.3\times 10^{-3}$ GFLOPs. A furthest-sampling strategy then selects the retained masked tokens so they are dispersed as widely as possible, giving each thrown token local context to aggregate. At $\rho_d=50\%$, the paper reports ViT-B/16 ImageNet-1K fine-tuning accuracy of 83.3% at 800 epochs, matching full MAE, with FLOPs at 0.72x, memory at 0.64x, and pre-training time at 0.75x.

Load-bearing premise

The load-bearing premise is that the greedy furthest sampling leaves every thrown token with at least one retained or unthrown token inside the $7\times 7$ window of the depthwise convolution; Sec. 3.3 admits that a thrown token with no such neighbors cannot receive a meaningful loss, but no coverage statistics or worst-case guarantee are reported for the throwing ratios tested.

Editorial extensions

If this is right

  • At 50% throwing, PR-MIM reports ImageNet-1K fine-tuning accuracy equal to full MAE for ViT-B/16 (83.3% at 800 epochs) and ViT-L/16 (85.4% at 800 epochs), with decoder FLOPs cut to 0.72x and memory to 0.64x.
  • The same combination works across MAE, SimMIM, TEC, GreenMIM, LocalMIM, and MFF, so the cost saving is not tied to one decoder design and extends to Swin-based architectures.
  • At a 65% throwing ratio the reported 100-epoch ViT-B/16 accuracy (81.7%) matches the full-budget baseline, so the savings can be pushed beyond 50% without measured loss.
  • The method also closes most of the partial-reconstruction gap in robustness benchmarks, iNaturalist transfer, ADE20K segmentation, and COCO detection, with additional cost only of order $10^{-3}$ GFLOPs.
  • Because thrown tokens remain in the loss, gradient estimates are closer to those of standard MAE, which the paper argues makes stochastic pre-training more accurate at shorter effective epochs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the paper tests static images, but the same spatial aggregation should transfer to higher-resolution or video inputs, where the $7\times 7$ convolution cost is essentially independent of sequence length; this is a straightforward extension the paper does not run.
  • Editorial: furthest sampling's first retained token is chosen randomly, so coverage near image borders is not controlled; a deterministic seed, such as a corner or grid anchor, would remove that variance and is a one-line modification of Algorithm 1.
  • Editorial: because PR-MIM reconstructs thrown tokens spatially rather than discarding them, it could be combined with attention-based token selection to save even more compute; the paper shows compatibility with many frameworks but does not test that combination.
  • Editorial: the reported gradient analysis predicts that PR-MIM should also improve convergence under shorter schedules, which could be checked directly by comparing learning curves at 100 versus 400 pre-training epochs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PR-MIM, a modification of partial-reconstruction masked image modeling (MIM) that recovers the quality lost when a subset of masked tokens is dropped from the reconstruction loss. The method has two components: a progressive reconstruction scheme that reconstructs thrown tokens via a cheap 7×7 depth-wise convolution over the decoder's spatial output, and a furthest-sampling strategy that selects the retained masked tokens to be maximally dispersed. The authors apply PR-MIM to MAE, SimMIM, MFF, TEC, LocalMIM, and GreenMIM, reporting that at a 50% token-throwing ratio, ViT-B/16 pre-trained for 800 epochs matches standard MAE's 83.3% top-1 accuracy while saving 28% FLOPs and 36% memory. Downstream results on ImageNet robustness, iNaturalist, ADE20K segmentation, and COCO detection are also presented.

Significance. If the accuracy claims hold, PR-MIM is a practically valuable and conceptually clean contribution: it gives a near-cost-free way to retain the supervision signal of thrown tokens, is orthogonal to several existing MIM frameworks, and is supported by ablations of the depth-wise convolution kernel size, spatial aggregation design, and sampling strategy. The breadth of the evaluation (six frameworks, three model sizes, and 100/300/800-epoch schedules) is a strength, and the reported FLOPs/memory savings are concrete and reproducible from the tables. The main reservation is that the central 'lossless' claim is established only through single-run accuracy pairs, with no variance information, and one key scaling baseline (ViT-L/16 at 800 epochs under partial reconstruction) is missing. These are fixable by additional experiments or by tempering the abstract's wording.

major comments (3)
  1. [Abstract; Table 3] The 'lossless performance' claim rests entirely on single-run top-1 accuracies such as 83.3 (MAE) versus 83.3 (PR-MIM) for ViT-B/16 at 800 epochs, with the partial-reconstruction baseline at 83.0. Without seed-level variance or at least multiple runs, an exact match to one decimal place does not establish losslessness; typical seed variance for 800-epoch MAE pre-training is on the order of 0.1–0.3 top-1. Please report mean±std over at least three seeds for the headline comparisons (MAE, partial reconstruction, and PR-MIM at 800 epochs for ViT-B/16) and adjust the abstract's wording if the confidence intervals show a residual gap.
  2. [Table 3, ViT-L/16 rows] The scaling claim that larger models are more fragile to partial reconstruction and that PR-MIM recovers the degradation is supported at 100 epochs (82.4 vs 83.2 vs 83.3) but not at 800 epochs, where the table lists only the MAE baseline (85.4) and PR-MIM (85.4) and omits the MAE-with-partial-reconstruction row. Adding the ViT-L/16 800-epoch partial-reconstruction baseline is necessary to support the long-schedule scaling narrative.
  3. [Table 6] The comparison with AMT is not apples-to-apples: in the MAE rows PR-MIM uses a 50% throwing ratio versus AMT's 40%, and in the SimMIM rows the total pre-training epochs differ (200 for AMT vs 100 for PR-MIM). The text states that PR-MIM 'outperforms AMT using even a higher throwing ratio,' which is only meaningful if the comparison is made at matched training budgets. Please either run matched schedules and throwing ratios or provide a clear cost-equivalence argument (e.g., FLOPs-matched) that justifies the comparison.
minor comments (5)
  1. [Sec. 3.3, Eq. (1) and Alg. 1] Eq. (1) maximizes the sum of pairwise distances among retained tokens, while Alg. 1 greedily maximizes the minimum distance to already-selected tokens. These are different objectives; the text should explicitly state that Alg. 1 is a heuristic approximation to Eq. (1) and comment on the mismatch.
  2. [Sec. 3.3] The motivation for furthest sampling is that a thrown token may have no retained or unmasked token within the 7×7 window, making the loss meaningless. At the tested throwing ratios this event is exceedingly rare (for ρe=0.75 and ρd=0.5, the probability that all 48 surrounding positions are thrown is below 10^-20), so the paper should quantify the actual frequency of the insufficient-neighbor case to justify this motivation, or present it as a heuristic for improving aggregate supervision quality rather than as a necessity.
  3. [Sec. 4.2, Fig. 4] The y-axis 'L2 norm of gradient difference' is undefined. Please specify which gradients are compared (e.g., encoder backbone gradients at a particular layer, or total loss gradients) and how the norm is aggregated over the batch and over training steps before plotting.
  4. [Table 4] The SimMIM rows are formatted ambiguously: the 'Epochs' column value appears to apply to both the baseline and the '+PR-MIM' row, but the reader must infer this. Add an explicit entry or footnote for the +PR-MIM row (e.g., '800') to avoid confusion.
  5. [Throughout] Typographical and unit issues: 'we obverse' should be 'we observe' (end of Sec. 3.1 of the introduction); '7.3 · 10−3G floating-point operations per second' should be 'GFLOPs' (operations, not per second); and the GFLOPs value in Sec. 3.2 should be written as 7.3×10−3 GFLOPs for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: PR-MIM is an empirical architecture paper whose efficiency and accuracy claims are benchmarked against external MAE/SimMIM baselines, not derived from its own conclusions.

full rationale

PR-MIM's central claims are empirical and self-contained with respect to any derivation chain. The proposed progressive reconstruction (Sec. 3.2) and furthest sampling (Sec. 3.3, Alg. 1) are design choices whose components are justified by ablations (Tabs. 2, 11, 12) rather than by defining a target quantity in terms of itself. The 'lossless' claim is an observed match to an external baseline, MAE [24], under the same training schedule; it is not a fitted parameter renamed as a prediction. Although the paper cites prior work by its own authors ([18], [30], [35]), these citations are used as related frameworks or background and are not load-bearing for the main MAE efficiency comparison. The closest potential concern, the Sec. 3.3 assertion that a thrown token without retained neighbors yields no meaningful loss, is a design motivation tested by ablation, not an equation that equates the method's output to its input; whether the coverage worry is statistically real is a correctness/robustness matter, not a circularity matter. No load-bearing step reduces by construction or by self-citation to its own inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are architecture choices (kernel size) ablated on the target metric. The core empirical claims rest on assumptions about local reconstructability and sampling coverage that are plausible but unquantified.

free parameters (1)
  • Depth-wise convolution kernel size = 7x7
    Selected by ablation in Table 11; 3x3 and 5x5 give 81.7, 7x7 gives 81.8, larger kernels are no better.
assumptions (3)
  • domain assumption A thrown masked token can be reconstructed from neighboring tokens within a local 7x7 window.
    Sec. 3.2 and Fig. 5; the spatial aggregation module relies on local context, citing [6] that local information suffices for reconstruction.
  • ad hoc to paper The greedy furthest sampling approximation of Eq. (1) leaves enough retained neighbors around every thrown token at the tested throwing ratios.
    Sec. 3.3 acknowledges random sampling can leave a thrown token isolated; the paper asserts furthest sampling reduces this but gives no coverage guarantee or measurement.
  • domain assumption The decoder output for unthrown tokens lives in a space where a depth-wise convolution can produce valid targets for thrown tokens.
    Sec. 3.2 says channel-level modeling is not needed because the decoder has mapped tokens into reconstruction-target space; this is assumed, not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PR-MIM: Delving Deeper into Partial Reconstruction in Masked Image Modeling." pith.science (2026). https://pith.science/paper/QB4EVB5T

@misc{pith2026241115746,
  author       = {Pith},
  title        = {Pith review of: PR-MIM: Delving Deeper into Partial Reconstruction in Masked Image Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QB4EVB5T}},
  note         = {Machine review of arXiv:2411.15746}
}
read the original abstract

Masked image modeling has achieved great success in learning representations but is limited by the huge computational costs. One cost-saving strategy makes the decoder reconstruct only a subset of masked tokens and throw the others, and we refer to this method as partial reconstruction. However, it also degrades the representation quality. Previous methods mitigate this issue by throwing tokens with minimal information using temporal redundancy inaccessible for static images or attention maps that incur extra costs and complexity. To address these limitations, we propose a progressive reconstruction strategy and a furthest sampling strategy to reconstruct those thrown tokens in an extremely lightweight way instead of completely abandoning them. This approach involves all masked tokens in supervision to ensure adequate pre-training, while maintaining the cost-reduction benefits of partial reconstruction. We validate the effectiveness of the proposed method across various existing frameworks. For example, when throwing 50% patches, we can achieve lossless performance of the ViT-B/16 while saving 28% FLOPs and 36% memory usage compared to standard MAE. Our source code will be made publicly available

Figures

Figures reproduced from arXiv: 2411.15746 by the authors.

Figure 1
Figure 1. Performance on ImageNet-1K. The bubble area is pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Different strategies for masked image modeling and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. When partial reconstruction throws a subset of masked tokens, the proposed progressive reconstruction scheme reconstructs each [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: The furthest sampling strategy. The red box and arrows [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Partial fine-tuning results of ViT-B pre-trained for 800 [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Reconstruction visualization by different methods. The [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 54 canonical work pages

  1. [1]

    Asano, Christian Rupprecht, and Andrea Vedaldi

    Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 2

    2020

  2. [2]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023. 2

    work page 2023

  3. [3]

    BEit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022. 2

    work page 2022

  4. [4]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. In NeurIPS, 2020. 2

    2020

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 2

    2021

  6. [6]

    Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction

    Jun Chen, Ming Hu, Boyang Li, and Mohamed Elho- seiny. Efficient self-supervised vision pretraining with local masked reconstruction. arXiv preprint arXiv:2206.00790 ,

    work page Pith review arXiv

  7. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2

    2020

  8. [8]

    Exploring simple siamese rep- resentation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. In CVPR, 2021. 2

    work page 2021

Show all 72 references
  1. [9]

    Context autoencoder for self- supervised representation learning

    Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self- supervised representation learning. IJCV, 132(1):208–223,

  2. [10]

    Dalal and B

    N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005. 2

    2005

  3. [11]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In NAACL-HLT, 2019. 2

    2019

  4. [12]

    Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsu- pervised visual representation learning by context prediction. In ICCV, 2015. 2

    2015

  5. [13]

    Bootstrapped masked autoencoders for vision bert pretraining

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In ECCV, 2022. 2

    2022

  6. [14]

    Peco: Perceptual codebook for bert pre-training of vision transformers

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. Peco: Perceptual codebook for bert pre-training of vision transformers. In AAAI, 2023. 2

    2023

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  8. [16]

    With a little help from my friends: Nearest-neighbor contrastive learning of visual representations

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021. 2

    2021

  9. [17]

    Whitening for self-supervised representation learning

    Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML, 2021. 2

    2021

  10. [18]

    Towards sustainable self-supervised learn- ing

    Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Towards sustainable self-supervised learn- ing. arXiv preprint arXiv:2210.11016, 2022. 2, 5, 6

    2022 arXiv

  11. [19]

    Un- supervised representation learning by predicting image rota- tions

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. In ICLR, 2018. 2

    2018

  12. [20]

    Accurate, large mini- batch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll ´ar, Ross Girshick, Pieter Noord- huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large mini- batch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2018. 8

    2018 arXiv

  13. [21]

    Richemond, Elena Buchatskaya, Carl Doersch, Bernardo ´Avila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo ´Avila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko. Bootstrap your own latent -...

    2020

  14. [22]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Gir- shick. Mask r-cnn. In ICCV, 2017. 8

    2017

  15. [23]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In CVPR, 2020. 2

    2020

  16. [24]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1, 2, 3, 5, 6, 7

    2022

  17. [25]

    H ´enaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and Jo˜ao Carreira

    Olivier J. H ´enaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and Jo˜ao Carreira. Effi- cient visual pretraining with contrastive detection. In ICCV,

  18. [26]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. ICLR, 2019. 7

    2019

  19. [27]

    The many faces of robustness: A criti- cal analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In ICCV,

  20. [28]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 7

    2021

  21. [29]

    Green hierarchical vision transformer for masked image modeling

    Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. In NeurIPS, 2022. 2, 5, 6

    2022

  22. [30]

    Contrastive masked autoencoders are stronger vision learners

    Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. IEEE TPAMI, 46(4):2506–2517, 2024. 2

    2024

  23. [31]

    Mean shift for self-supervised learning

    Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for self-supervised learning. In ICCV, 2021. 2 9

    2021

  24. [32]

    Colorization as a proxy task for visual understanding

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017. 2

    2017

  25. [33]

    Semmae: Semantic-guided mask- ing for learning masked autoencoders

    Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided mask- ing for learning masked autoencoders. In NeurIPS, 2022. 2

    2022

  26. [34]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In ECCV, 2022. 7, 8

    2022

  27. [35]

    Sere: Exploring feature self-relation for self-supervised trans- former

    Zhong-Yu Li, Shanghua Gao, and Ming-Ming Cheng. Sere: Exploring feature self-relation for self-supervised trans- former. IEEE TPAMI, 45(12):15619–15631, 2023. 2

    2023

  28. [36]

    Improving pixel-based mim by reduc- ing wasted modeling capability

    Yuan Liu, Songyang Zhang, Jiacheng Chen, Zhaohui Yu, Kai Chen, and Dahua Lin. Improving pixel-based mim by reduc- ing wasted modeling capability. In ICCV, 2023. 2, 5, 6, 8

    2023

  29. [37]

    Swin trans- former: Hierarchical vision transformer using shifted win- dows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hierarchical vision transformer using shifted win- dows. ICCV, 2021. 6

    2021

  30. [38]

    Good helper is around you: attention-driven masked image modeling

    Zhengqi Liu, Jie Gui, and Hao Luo. Good helper is around you: attention-driven masked image modeling. In AAAI,

  31. [39]

    Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

    Jiawei Mao, Shujian Guo, Yuanqi Chang, Xuesong Yin, and Binling Nie. Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification. arXiv preprint arXiv:2305.05871, 2023. 1, 3

    2023 arXiv

  32. [40]

    Unsupervised learning of visual representions by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representions by solving jigsaw puzzles. In ECCV,

  33. [41]

    Rep- resentation learning by learning to count

    Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Rep- resentation learning by learning to count. In ICCV, 2017. 2

    2017

  34. [42]

    Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 2

    2016

  35. [43]

    Spatilly consistent representation learning

    Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatilly consistent representation learning. In CVPR,

  36. [44]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211–252,

  37. [45]

    Siamese image modeling for self-supervised vision represen- tation learning

    Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision represen- tation learning. In CVPR, 2023. 2

    2023

  38. [46]

    Designing bert for convolutional net- works: Sparse and hierarchical masked modeling

    Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional net- works: Sparse and hierarchical masked modeling. In ICLR,

  39. [47]

    Un- derstanding self-supervised learning dynamics without con- trastive pairs

    Yuandong Tian, Xinlei Chen, and Surya Ganguli. Un- derstanding self-supervised learning dynamics without con- trastive pairs. In ICML, 2020. 2

    2020

  40. [48]

    Integrally pre-trained transformer pyramid net- works

    Yunjie Tian, Lingxi Xie, Zhaozhi Wang, Longhui Wei, Xi- aopeng Zhang, Jianbin Jiao, Yaowei Wang, Qi Tian, and Qixiang Ye. Integrally pre-trained transformer pyramid net- works. In CVPR, 2023. 2

    2023

  41. [49]

    The inaturalist species classification and de- tection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and de- tection dataset. In CVPR, 2018. 7

    2018

  42. [50]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019. 7

    2019

  43. [51]

    Hard patches mining for masked image modeling

    Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In CVPR, 2023. 2

    2023

  44. [52]

    Masked image modeling with local multi-scale reconstruction

    Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. In CVPR, 2023. 2, 5, 6

    2023

  45. [53]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023. 1, 3

    2023

  46. [54]

    Image as a foreign language: BEiT pretraining for vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In CVPR, 2023. 2

    2023

  47. [55]

    Dense contrastive learning for self-supervised visual pre-training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021. 2

    2021

  48. [56]

    Masked feature predic- tion for self-supervised visual pre-training

    Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature predic- tion for self-supervised visual pre-training. In CVPR, 2022. 2

    2022

  49. [57]

    Mvp: Multimodality-guided visual pre-training

    Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-guided visual pre-training. In ECCV, 2022. 2

    2022

  50. [58]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, 2023. 2

    2023

  51. [59]

    Yu, and Dahua Lin

    Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 2

    2018

  52. [60]

    Masked frequency modeling for self-supervised visual pre-training

    Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. In ICLR, 2023. 2

    2023

  53. [61]

    Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning

    Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021. 2

    2021

  54. [62]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022. 1, 2, 5, 6

    2022

  55. [63]

    Decoupled contrastive learning

    Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In ECCV, 2022. 2

    2022

  56. [64]

    Masked image modeling with denoising contrast

    Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. ICLR, 2023. 2, 5, 7 10

    2023

  57. [65]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In PMLR, 2021. 2

    2021

  58. [66]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. 2

    2016

  59. [67]

    Self-supervised visual representations learning by contrastive mask prediction

    Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-supervised visual representations learning by contrastive mask prediction. In ICCV, 2021. 2

    2021

  60. [68]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 7

    2017

  61. [69]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022. 2

    2022

  62. [70]

    Mugs: A multi- granular self-supervised learning framework

    Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- granular self-supervised learning framework. In arXiv preprint arXiv:2203.14415, 2022. 2 11 PR-MIM: Delving Deeper into Partial Reconstruction in Masked Image Modeling Supplementary ...

    2022 arXiv

  63. [71]

    In implementation, we use a greedy strategy to find the approximate solution of Eq

    Implementation of furthest sampling Given Nm masked tokens, the furthest sampling throws Nt tokens and retains Nm − Nt tokens. In implementation, we use a greedy strategy to find the approximate solution of Eq. (1) of the manuscript, as shown in Alg. 1. After the first retaine...

  64. [72]

    13 and Tab

    Implementation details Tab. 13 and Tab. 14 summarize the hyperparameters used for pre-training and fine-tuning, respectively. These hy- perparameters are used for MAE-based experiments, and the other experiments follow the official settings of the corresponding papers. For fin...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.