Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Sample- and Parameter-Efficient Auto-Regressive Image Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read XTRA claims that switching auto-regressive image pre-training from next-patch to next-block prediction makes it much more sample- and parameter-efficient, beating AIM-0.6B on average across 15 benchmarks while trained on 152× fewer images.

desk verdict Block causal masking is a real, well-ablated improvement over next-patch AR pretraining, but the paper's 152x sample-efficiency headline rests on a cross-dataset comparison without a matched AIM baseline. read the letter →

arxiv 2411.15648 v2 pith:26PU5CCT submitted 2024-11-23 cs.CV

Elad Amrani , Leonid Karlinsky , Alex Bronstein This is my paper · ORCID
classification cs.CV
keywords XTRAblockcausalmaskingauto-regressiveimagemodelsself-supervisedrepresentationlearningsampleefficiencyparametervisiontransformernext-blockreconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the inefficiency of auto-regressive image models has a simple cause: they predict one image patch at a time, so most modeling capacity goes into local texture rather than structure. XTRA instead predicts blocks of k×k patches, using a block causal mask that lets tokens inside a block attend freely while enforcing raster-order causality between blocks. The authors claim this single change makes auto-regressive pre-training dramatically more sample-efficient (13.1M images beat 2B images on average across 15 benchmarks) and parameter-efficient (an 85M ViT-B/16 beats models with 7–16× more parameters on ImageNet-1K probing). If true, block causal masking is a cheap, scalable lever for vision pre-training without contrastive or masking tricks.

What carries the argument

The central mechanism is the Block Causal Mask. The image is split into a grid of k×k patch blocks; a transformer's attention mask allows any token to attend to all tokens inside its own block and all tokens in earlier blocks in a fixed raster order, and forbids attention to later blocks. Inside the decoder, each block's token embeddings are concatenated into one vector and passed through a shared MLP that predicts the pixel values of the next block, trained with per-block normalized MSE loss. The mechanism converts the auto-regressive task from "guess the next patch" to "reconstruct the next region from a coarser context," which is what the paper claims carries the efficiency gains.

What would settle it

Train AIM (or an otherwise identical next-patch autoregressive model) on the same 13.1M-filtered ImageNet-21K subset with the same 100 epochs and the same decoder; if its average over the 15 benchmarks reaches or exceeds XTRA's 76.2%, the block causal mask is not the cause of the claimed sample-efficiency gain. A second test would train XTRA on DFN-2B directly and compare against AIM-0.6B on identical data.

Watch

Extended reading notes

Core claim

The paper claims that next-block reconstruction is a better auto-regressive objective for visual representation learning than next-patch prediction. XTRA's ViT-H/14, trained only on ImageNet-21K filtered to 13.1M usable images, averages 76.2% top-1 accuracy under attentive probing across 15 recognition benchmarks, beating the AIM-0.6B baselines trained on 2B images (74.5% on DFN-2B and 75.6% on DFN-2B+), and matches or beats them on 9 of 15 datasets. On ImageNet-1K alone, XTRA ViT-B/16 gets 70.2% linear probing accuracy with 85M parameters versus iGPT-L's 65.2% with 1.36B, and 76.8% attentive probing versus AIM-0.6B's 73.5% with 600M. The paper interprets these results as showing that grouping tokens into blocks redirects the model toward low-frequency, semantically meaningful image structure, and that the block-size-to-resolution ratio, not patch size or absolute block size, is the controlling factor.

Load-bearing premise

The load-bearing premise is that comparing XTRA on 13.1M filtered ImageNet-21K images with AIM on 2B DFN-2B images isolates the effect of the block causal mask; if dataset curation and class balance, rather than the mask, explain part of the 1.7% average gain, the 152× sample-efficiency claim is weakened, since the paper does not train AIM on the same filtered 13.1M images with a matched budget.

Editorial extensions

If this is right

  • Block causal masking is a drop-in change to the attention pattern; any causal ViT vision model can adopt it without new losses or auxiliary heads.
  • If the block-size-to-resolution ratio is the controlling factor, then recipes tuned for one resolution and patch combination transfer to others by keeping that ratio fixed, not by copying absolute block sizes.
  • The 3.0% gain in Table 6a over the AIM re-implementation under identical training details locates the mask effect within ImageNet-1K, separate from the cross-dataset comparison.
  • Auto-regressive image models can compete with contrastive and masked-image-modeling methods on frozen-feature probing without multi-crop views, momentum encoders, or a tokenizer, while keeping the simple and scalable recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 152× headline compares different datasets; a matched-data AIM baseline could shrink or change the number, so the honest sample-efficiency statement may be "block masking helps within a fixed dataset" rather than "the mask alone buys 152× more sample efficiency."
  • The same block-causal idea should transfer to video or multi-scale token grids, where a 3D block over space-time could analogously let the model capture object structure over larger regions.
  • A natural test is whether block size should grow during training or with model scale; the paper fixes block size per model, so whether adapting it dynamically preserves the benefit is an open extension.
  • Because the decoder is discarded at probing time, the efficiency gain is in representation quality, not in downstream inference cost; the parameter-efficiency claim is about pre-training, not about the deployed trunk.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces XTRA, a vision transformer pre-trained with a block-wise auto-regressive objective in which the image is partitioned into k×k-token blocks and each block is reconstructed from preceding blocks under a block-causal mask. The authors report two headline findings: (i) sample efficiency—XTRA ViT-H/14, trained on ImageNet-21K filtered to 13.1M images, surpasses AIM-0.6B trained on DFN-2B/DFN-2B+ by 1.7%/0.6% average top-1 accuracy across 15 recognition benchmarks despite 152× fewer unique samples; and (ii) parameter efficiency—XTRA ViT-B/16 outperforms prior auto-regressive image models on ImageNet-1K with 7–16× fewer parameters. The paper includes ablations on block size, block-to-resolution ratio, loss function, auto-regressive pattern, and decoder depth/width, all on ImageNet-1K with ViT-B/16, plus a computational-cost comparison. The authors argue that block causal masking aligns attention with 2D image structure, yielding better low-frequency semantics than token-level next-patch prediction.

Significance. If the sample-efficiency claim holds, block causal masking would be a simple, impactful design choice for auto-regressive visual pre-training, potentially reducing the data and compute barriers to scaling such models on curated or uncurated data. The paper's internal ablation on ImageNet-1K (Table 6a) provides a same-data control in which moving from token-level to block-level prediction improves attentive-probing accuracy by 3.0%, and the raster-order ablation (Table 6d) is large and directionally clear. The method is simple, the architecture is standard, and the authors provide a public implementation link. However, the headline cross-dataset comparison in Table 3 is confounded by dataset composition and label structure, and the paper does not establish that the same mechanism transfers to the ViT-H/14 scale at which the headline sample-efficiency claim is made. The parameter-efficiency comparison against iGPT-L and AIM-0.6B on ImageNet-1K is more internally controlled, though it relies on single runs and an old, weak baseline.

major comments (4)
  1. [Section 5.1, Table 3] The central sample-efficiency claim is confounded by a cross-dataset comparison. XTRA-H is pre-trained on ImageNet-21K filtered to 13.1M images for 100 epochs, while AIM-0.6B is pre-trained on DFN-2B or DFN-2B+ for 2.5 epochs. ImageNet-21K is a curated, roughly class-balanced dataset, whereas DFN-2B is uncurated web data; the reported 1.7%/0.6% average-accuracy margins could be attributable to dataset curation, class balance, or epoch budget rather than to Block Causal Masking. The load-bearing missing control is an AIM-0.6B (or equivalent token-level causal model) trained on the same 13.1M ImageNet-21K subset with the same 100-epoch budget. Without that control, the phrase 'Despite being trained on 152× fewer samples ... surpasses' overstates what the experiment can establish.
  2. [Section 5.1, abstract, and Section 9] The '152× fewer samples' factor compares unique image counts (13.1M vs 2B) but ignores epochs. In terms of total image presentations, XTRA sees 13.1M × 100 = 1.31B samples and AIM sees 2B × 2.5 = 5B, a factor of about 3.8×, not 152×. The paper should define 'sample' explicitly and either temper the claim or justify why unique-image count is the appropriate efficiency measure.
  3. [Section 6, Table 6a and Section 5.1] The ablation that supports the block-masking mechanism is performed on ViT-B/16 with 100 epochs on ImageNet-1K only. The headline sample-efficiency claim is made for ViT-H/14 trained on ImageNet-21K for 100 epochs. The paper does not test whether the benefit of block size (the +3.0% in Table 6a) transfers to the larger architecture and dataset; it is possible that the effect is scale- or data-dependent. The authors should either provide a same-scale control (e.g., an AIM-style ViT-H/14 on the same 13.1M ImageNet-21K subset) or explicitly restrict the sample-efficiency claim to the scale at which the controlled ablation was performed.
  4. [Tables 3 and 6] All results are reported from single runs without error bars or multiple seeds. Several ablation comparisons are small relative to expected probe-training noise: e.g., Table 6a reports 67.3 vs 67.4 and 67.6 vs 67.4; Table 6b reports 67.6 vs 67.7; Table 6e reports 67.4, 67.9, 67.6, 67.6, 67.8. These differences are within a few tenths of a percent, and the paper uses them to support conclusions such as 'decoder depth has little effect' and 'block size to resolution ratio is the key factor.' The authors should report variance estimates (e.g., 2–3 seeds with standard deviation) for at least the ablation and main comparisons, or hedge conclusions accordingly.
minor comments (5)
  1. [Abstract and Section 3] The phrase 'novel auto-regressive objective' is imprecise: Eq. (1) is a standard MSE loss, and the novelty is in the Block Causal Mask, not the objective. Consider rephrasing to 'novel auto-regressive masking scheme.'
  2. [Eq. (1)] The text says losses are applied to pixel values 'normalized per block,' but Eq. (1) shows only a plain MSE without per-block normalization factors. Please make the normalization explicit or clarify whether the normalization is done inside the network before the loss.
  3. [Table 1] Table 1 lists the dataset as 'ImageNet-21K' but the text in Section 5.1 notes it is filtered to 13.1M samples due to broken URLs. The table should label this as 'ImageNet-21K (filtered to 13.1M)' to avoid ambiguity.
  4. [Section 6, Table 6a] Rows labeled '16×16' correspond to block sizes in pixels, but the explanation in the text says 'single token/patch, 2×2 tokens, and 4×4 tokens,' which is clear enough. However, the table header '# of blocks to predict' is somewhat cryptic; consider renaming to 'number of blocks predicted per step' for readability.
  5. [Figure 4 caption] The phrase 'post-generation per block de-normalization' is unclear; please explain the de-normalization step or rephrase to make the visualization procedure understandable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: XTRA's claims are supported by external comparisons and same-data ablations; the sample-efficiency comparison is a benchmarking choice, not a derivation from fitted inputs.

full rationale

The central derivation is an MSE next-block reconstruction loss with no parameters fitted to downstream targets; the reported gains are empirical comparisons against published AIM and iGPT numbers rather than consequences of the paper's own equations. The '152x fewer samples' figure compares unique pretraining images across datasets, which is a definitional choice about what to count, not a circular reduction. The internal ablation in Table 6a provides the needed same-data control: a block size of 16x16 is stated to replicate AIM, and increasing to 64x64 yields +3.0%, so the masking mechanism is independently tested rather than assumed. The one self-citation in the paper ([1], Self-Classifier) appears only in a related-work survey of contrastive methods and is not load-bearing. The absence of a same-data AIM-0.6B baseline at ViT-H scale is a possible confound for the headline efficiency claim, but that is an experimental-comparison weakness, not circularity; no prediction or claimed derivation reduces to its own input by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on one design parameter (block size) and one comparison assumption (cross-dataset evaluation). No new physical or architectural entities are introduced.

free parameters (2)
  • block size (tokens per block) = 4x4 tokens (64x64 px for ViT-B/16; 56x56 px for ViT-H/14)
    The method's central design choice; ablated in Table 6a but not derived. The paper claims block-to-resolution ratio (16/256) is the key factor based on two resolutions.
  • decoder width = 768 for ViT-B/16; 640 for ViT-H/14
    Ablation Table 6f shows width shifts attentive probing by 2.5 points (67.1 at 192 to 69.6 at 768); the final model uses 768 for B and 640 for H.
assumptions (3)
  • domain assumption Autoregressive block prediction is a useful proxy for semantic representation learning.
    Inherited from iGPT/AIM and central to the method; the paper does not prove that better reconstruction yields better representations beyond probing results.
  • domain assumption Frozen-trunk attentive probing is a valid and comparable evaluation across methods.
    XTRA results are compared to numbers taken from AIM [26] and CAE [19]; the paper states hyperparameters are taken from AIM, but does not re-run baselines.
  • ad hoc to paper ImageNet-21K filtered to 13.1M images is an appropriate control for DFN-2B in measuring sample efficiency.
    The 152x claim assumes dataset differences do not drive the accuracy gap; no AIM model is trained on the same ImageNet-21K filter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample- and Parameter-Efficient Auto-Regressive Image Models." pith.science (2026). https://pith.science/paper/26PU5CCT

@misc{pith2026241115648,
  author       = {Pith},
  title        = {Pith review of: Sample- and Parameter-Efficient Auto-Regressive Image Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26PU5CCT}},
  note         = {Machine review of arXiv:2411.15648}
}
abstract

We introduce XTRA, a vision model pre-trained with a novel auto-regressive objective that significantly enhances both sample and parameter efficiency compared to previous auto-regressive image models. Unlike contrastive or masked image modeling methods, which have not been demonstrated as having consistent scaling behavior on unbalanced internet data, auto-regressive vision models exhibit scalable and promising performance as model and dataset size increase. In contrast to standard auto-regressive models, XTRA employs a Block Causal Mask, where each Block represents k $\times$ k tokens rather than relying on a standard causal mask. By reconstructing pixel values block by block, XTRA captures higher-level structural patterns over larger image regions. Predicting on blocks allows the model to learn relationships across broader areas of pixels, enabling more abstract and semantically meaningful representations than traditional next-token prediction. This simple modification yields two key results. First, XTRA is sample-efficient. Despite being trained on 152$\times$ fewer samples (13.1M vs. 2B), XTRA ViT-H/14 surpasses the top-1 average accuracy of the previous state-of-the-art auto-regressive model across 15 diverse image recognition benchmarks. Second, XTRA is parameter-efficient. Compared to auto-regressive models trained on ImageNet-1k, XTRA ViT-B/16 outperforms in linear and attentive probing tasks, using 7-16$\times$ fewer parameters (85M vs. 1.36B/0.63B).

Figures

Figures reproduced from arXiv: 2411.15648 by the authors.

Figure 1
Figure 1. Sample and Parameter Efficiency of XTRA. (Left) XTRA-H/14 (0.6B parameters) outperforms prior state-of-the-art auto-regressive image model (AIM-0.6B [26]) in top-1 average accuracy across 15 diverse image recognition benchmarks, despite being trained on 152× fewer samples. (Right) XTRA-B/16 (85M parameters) outperforms prior auto-regressive image models trained on ImageNet-1k in linear and attentive probing tasks, w… view at source ↗
Figure 2
Figure 2. XTRA Architecture. Following ViT [24] an image is partitioned into a sequence of patches (numbered grid) and processed by a standard ViT encoder-decoder architecture with our proposed Block Causal Masking. I.e., causality is enforced at the block level with a rasterized pattern (see [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Block Causal Masking. In the image above the fine-grained grid represents a grid of patches (following ViT [24]) that are processed by the model. The coarse-grained (numbered) grid represents a grid of blocks, where each block represents 4×4 patches/tokens. Block Causal Masking enforces causality at the block level with a rasterized pattern (numbered sequence), ensuring that tokens can attend to others within the sa… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of XTRA’s Predictions on the ImageNet-1k Validation Set. XTRA generates predictions auto-regressively, producing one block of pixels at a time, with each new block conditioned on the preceding sequence of ground-truth blocks. Note that no loss is applied …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 42 canonical work pages

  1. [1]

    Self- supervised classification network

    Elad Amrani, Leonid Karlinsky, and Alex Bronstein. Self- supervised classification network. InEuropean Conference on Computer V ision, pages 116–132. Springer, 2022. 2

    work page 2022

  2. [2]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Y ann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 15619–15629, 2023. 2, 3, 7

    work page 2023

  3. [3]

    Data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning , pages 1298–1312. PMLR, 2022. 3, 7

    work page 2022

  4. [4]

    Efficient self-supervised learning with contextualized target representations for vision, speech and language

    Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. InInternational Conference on Machine Learning, pages 1416–1429. PMLR,

    work page

  5. [5]

    From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge

    Peter Bandi, Oscar Geessink, Quirine Manson, Marcory V an Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550–560, 2018. 1

    work page 2018

  6. [6]

    BEit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. 2, 3, 6

    work page 2022

  7. [7]

    Stochastic positional embeddings improve masked image modeling

    Amir Bar, Florian Bordes, Assaf Shocher, Mido Assran, Pascal Vincent, Nicolas Ballas, Trevor Darrell, Amir Globerson, and Y ann LeCun. Stochastic positional embeddings improve masked image modeling. In International conference on machine learning. PMLR, 2024. 7

    work page 2024

  8. [8]

    The iwildcam 2021 competition dataset

    Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.03494, 2021. 1

    arXiv 2021

Show all 52 references
  1. [9]

    Food- 101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc V an Gool. Food- 101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446–461. Springer, 2014. 1

    2014

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey ...

    1901

  3. [11]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. InAdvances in Neural Information Processing Systems, 2020. 2

    2020

  4. [12]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J ´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the International Conference on Computer V ision (ICCV),

  5. [13]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020. 2, 3, 6

    2020

  6. [14]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, ICML, 2020. 2

    2020

  7. [15]

    Big self-supervised models are strong semi-supervised learners

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems, 2020. 2

    2020

  8. [16]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer V ision and P attern Recognition, pages 15750–15758, 2021. 2

    2021

  9. [17]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. In arXiv preprint arXiv:2003.04297, 2020. 2

    2003 arXiv

  10. [18]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9640–9649, 2021. 7

    2021

  11. [19]

    Context autoencoder for self-supervised representation learning

    Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shen- tong Mo, Y unhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. International Journal of Computer V ision, 132(1):208–223, 2024. 2, 3, 7

    2024

  12. [20]

    Functional map of the world

    Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. InProceedings of the IEEE Conference on Computer V ision and P attern Recognition, pages 6172–6180, 2018. 1

    2018

  13. [21]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea V edaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 1

    2014

  14. [22]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 1

    2009

  15. [23]

    Discriminative unsupervised feature learning with convolutional neural networks

    Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pages 766–774, 2014. 2

    2014

  16. [24]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De- hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  17. [25]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan 9 Schelten, Amy Y ang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 2

    2024 arXiv

  18. [26]

    Scalable pre-training of large autoregressive image models

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel An- gel Bautista, Alexander Toshev, V aishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In International conference on machine learning. PMLR, 2024. 1, 2, ...

    2024

  19. [27]

    Boot- strap your own latent - a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal V alko. Boot- strap your own latent - a ne...

    2020

  20. [28]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Y uxin Wu, Saining Xie, and Ross Gir- shick. Momentum contrast for unsupervised visual representation learning. InProceedings of the IEEE/CVF Conference on Com- puter V ision and P attern Recognition, pages 9729–9738, 2020. 2

    2020

  21. [29]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Y anghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009,

  22. [30]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected T opics in Applied Earth Observations and Remote Sensing, 12 (7):2217–2226, 2019. 1

    2019

  23. [31]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 2

    2001 arXiv

  24. [32]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, page 2. Minneapolis, Minnesota, 2019. 2, 3

    2019

  25. [33]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013. 1

    2013

  26. [34]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 1

    2009

  27. [35]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, V asil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  28. [36]

    Cats and dogs

    Omkar M Parkhi, Andrea V edaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 1

    2012

  29. [37]

    Moment matching for multi-source do- main adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source do- main adaptation. InProceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019. 1

    2019

  30. [38]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 2, 3

    2019

  31. [39]

    Discrete variational autoencoders

    Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:1609.02200, 2016. 3

    2016 arXiv

  32. [40]

    Spreading vectors for similarity search

    Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Herv´e J´egou. Spreading vectors for similarity search. arXiv preprint arXiv:1806.03198, 2018. 2

    2018 arXiv

  33. [41]

    The effectiveness of mae pre-pretraining for billion-scale pretraining

    Mannat Singh, Quentin Duval, Kalyan V asudev Alwala, Haoqi Fan, V aibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Doll´ar, Christoph Feichtenhofer, Ross Girshick, et al. The effectiveness of mae pre-pretraining for billion-scale pretraining. In Proceedings of the IEEE/CVF ...

    2023

  34. [42]

    Rxrx1: An image set for cellular morphological variation across many experimental batches

    James Taylor, Berton Earnshaw, Ben Mabey, Mason Victors, and Jason Y osinski. Rxrx1: An image set for cellular morphological variation across many experimental batches. InInternational Con- ference on Learning Representations (ICLR), page 23, 2019. 1

    2019

  35. [43]

    Going deeper with image transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv ´e J ´egou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32–42, 2021. 2

    2021

  36. [44]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Y asmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  38. [46]

    iNaturalist 2018 competition dataset

    Grant V an Horn, Oisin Mac Aodha, Yin Cui, Y ang Song, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. iNaturalist 2018 competition dataset. https://github. com/visipedia/inat_comp/tree/master/2018,

    2018

  39. [47]

    Rotation equivariant cnns for digital pathology

    Bastiaan S V eeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. InMedical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proce...

    2018

  40. [48]

    Unsupervised feature learning via non-parametric instance dis- crimination

    Zhirong Wu, Y uanjun Xiong, Stella X Y u, and Dahua Lin. Unsupervised feature learning via non-parametric instance dis- crimination. InProceedings of the IEEE Conference on Computer V ision and P attern Recognition, pages 3733–3742, 2018. 2

    2018

  41. [49]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Y ue Cao, Y utong Lin, Jianmin Bao, Zhuliang Y ao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9653–9663, 2022. 2, 3, 7

    2022

  42. [50]

    ibot: Image bert pre-training with on- line tokenizer.arXiv preprint arXiv:2111.07832, 2021

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Y uille, and Tao Kong. ibot: Image bert pre-training with on- line tokenizer.arXiv preprint arXiv:2111.07832, 2021. 2, 3, 6, 7 10 Sample-and Parameter-Efficient Auto-Regressive Image Models Supplementary Material

    2021 arXiv

  43. [51]

    Benchmark Datasets Dataset train test classes Imagenet-1k [22] 1,281,167 50,000 1000 iNA T-18 [46] 437,513 24,426 8142 CIFAR-10 [34] 50,000 10,000 10 CIFAR-100 [34] 50,000 10,000 100 Food101 [9] 75,750 25,250 101 DTD [21] 3,760 1,880 47 Pets [36] 3,680 3,669 37 Cars [33] 8,144...

  44. [52]

    • Parameters (Linear): The number of parameters in the model determines the size of weight matrices involved in computation

    Computational Cost Estimation In Table 8, weestimatethe computational cost of each method using the following simplified formula: Cost=Parameters×Samples×Epochs×Views2×Tokens2 (2) This formula provides an approximate scaling relationship rather than an exact measurement, as it...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.