Pith. sign in

REVIEW 2 major objections 5 minor 68 references

Multi-Token Enhancing for Vision Representation Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single vision transformer can create its own ensemble with extra tokens, then distill them away, leaving a stronger model at zero inference cost.

desk verdict MTE is a well-tested plugin for SSL pre-training: multiple auxiliary tokens distilled into a single global token with a clean attention-masking trick, and the main weakness is missing reproducibility artifacts, not the mechanism. read the letter →

arxiv 2411.15787 v1 pith:LACPG73R submitted 2024-11-24 cs.CV

Zhong-Yu Li , Yu-Song Hu , Bo-Wen Yin , Ming-Ming Cheng This is my paper · ORCID
classification cs.CV
keywords multi-tokenensembleself-supervisedlearningvisiontransformerauxiliarytokensonlinedistillationadaptivepoolingrepresentationknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single vision transformer can supply its own ensemble: by adding several auxiliary tokens (extra CLS tokens with different initializations and adaptively pooled tokens from patch features), and fusing their losses during pre-training, the model learns richer representations. It then distills the fused auxiliary knowledge into the global CLS token online, so the auxiliary tokens can be discarded with no loss at inference and no added inference cost. Across DINO, iBOT, MoCo V3, BYOL, and MoBY, the paper reports consistent k-NN and linear-probing gains on ImageNet, and improved transfer to iNaturalist, COCO, and ADE20K. The contribution is a general plugin rather than a new loss family, minimally invasive to existing self-supervised pipelines.

What carries the argument

The load-bearing mechanism is the set of auxiliary tokens plus the distillation loss. Auxiliary CLS tokens are additional trainable class tokens with distinct random initializations, fed through the same transformer blocks; adaptively pooled tokens are computed from patch tokens using $K$ different learned pooling-weight maps generated by large-kernel convolutions. A Token Enhancing (TEN) module applies cross-attention to auxiliary CLS tokens to produce a stronger teacher. The total pre-training loss is $\mathcal{L}=\mathcal{L}(\hat{h}_t,\tilde{h}_t)+\mathcal{L}(\hat{h}_t,\tilde{h}_c)$ (Eq. 6), where the second term distills the fused auxiliary-token representation into the global token; an attention mask prevents the global token and patches from depending on the auxiliary tokens, so discarding them at inference is lossless.

What would settle it

Train the same MTE recipe but with all auxiliary CLS tokens sharing one initialization and all pooling convolutions sharing one weight initialization, keeping every other detail fixed; if the reported k-NN gain over DINO (61.2 to 66.6 in Table 5) largely disappears, the claim that auxiliary tokens are complementary by construction is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that ensemble-like diversity can be manufactured inside one encoder. With $M$ auxiliary CLS tokens and $K$ adaptively pooled tokens, each token develops its own expertise on different categories, shown by CKA differences and per-category accuracy analyses; fusing their outputs gives a stronger teacher than any single token. The online distillation term $\mathcal{L}(\hat{h}_t,\tilde{h}_c)$ makes the global CLS token imitate that teacher, and because the global token shares the encoder with the auxiliary tokens, the optimization is reciprocal: distillation improves the auxiliary tokens as well as the global token. After pre-training the auxiliary parts are removed, leaving an unchanged model whose representations are stronger than the baseline's.

Load-bearing premise

The entire no-inference-cost gain rests on the empirical premise that differently initialized auxiliary tokens stay complementary instead of collapsing into the same representation, and that the online distillation term transfers all of their task-relevant knowledge into the global token.

Editorial extensions

If this is right

  • Adopting MTE changes only the pre-training forward pass (roughly 6.1G to 6.4G FLOPs for a 224×224 image); the deployed model is identical to the baseline, so any downstream pipeline inherits the gains for free.
  • On DINO, 300 epochs with MTE already match the 800-epoch baseline's k-NN and linear accuracy, so the technique compresses long pre-training schedules.
  • The gains transfer beyond classification: iBOT+MTE pre-trained for 100 epochs improves COCO detection and segmentation mAP by about 0.9 and 1.0 points, respectively, and ADE20K mIoU by 1.2 points.
  • The method extends to supervised training and to non-CLS architectures such as Swin, where six adaptively pooled tokens are used with an average-pooled global token.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper only tests complementarity at the final token level, a natural next experiment is to apply the same auxiliary-token ensemble at intermediate transformer blocks; if per-token expertise is generic, mid-level tokens might give even larger dense-prediction gains.
  • The per-category expertise analysis suggests MTE could be a cheap fix for fine-grained and long-tail recognition; the largest relative gains might appear where different tokens specialize on rare classes, though the paper only measures aggregate accuracy.
  • The auxiliary tokens are removed after pre-training, but the same online-distillation idea could be reused during fine-tuning to transfer a teacher's diversity into a student without changing the deployed architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes Multi-Token Enhancing (MTE), a plug-in module for vision Transformers that adds M auxiliary CLS tokens and K adaptively pooled tokens during pre-training. These auxiliary tokens are used in the self-supervised or supervised loss through independent projection heads, and their fused representation is distilled online into the global CLS token (Eq. 6). At inference the auxiliary tokens are discarded; for plain ViTs an attention mask prevents the global/patch tokens from depending on the auxiliary CLS tokens, making removal lossless (Table 16). The method is evaluated on ImageNet with DINO, iBOT, MoCo V3, BYOL, MoBY, and Swin backbones, as well as on fine-tuning, detection/segmentation transfer, and a supervised classification setting, reporting consistent gains over baselines.

Significance. If the empirical claims hold, MTE is a low-cost, architecture-agnostic way to obtain ensemble-like improvements in self-supervised pre-training without inference overhead. The paper's main strengths are the breadth of the evaluation (five SSL losses, two architectures, and dense downstream tasks) and the clean construction: the adaptively pooled tokens are post-encoder, and the attention-mask design in Figure 4 is validated by Table 16, so the 'no additional inference costs' claim is convincingly supported. The ablation studies in Tables 13–15 are consistent with the proposed mechanism. The method is not a derivation with fitted constants; M=4 and K=6 are selected by ablation and reported as such. The main uncertainties are empirical reliability: there are no error bars, and the BYOL baseline is in-house. The paper promises code but does not provide it, which limits reproducibility.

major comments (2)
  1. [Table 5] The BYOL baseline is implemented using the DINO codebase, as stated in Table 5, rather than the official BYOL recipe. BYOL uses an asymmetric predictor head and stop-gradient on the target, whereas DINO uses a prototype/clustering loss with a momentum teacher. Because the reported +9.4 k-NN improvement over BYOL is the largest in Table 5, this gain may partly reflect a weak baseline. Please either replace the BYOL numbers with the official BYOL ViT-S/16 results under the same 100-epoch protocol, or provide a faithful re-implementation with the BYOL loss and report the exact hyperparameters and code configuration. Without this, the claim that MTE consistently improves over all five self-supervised baselines is not fully supported.
  2. [Tables 5–11] All reported results are single pre-training runs with no variance estimates. Several headline gains are small relative to typical run-to-run variability, e.g., +0.8 linear probing for MoCo V3, +0.6 for iBOT (300 epochs), +0.8 for MoBY in Table 5, and +0.3 for ViT-S/16 fine-tuning in Table 7. Please provide standard deviations over at least three independent runs for the primary comparisons (Table 5 and Table 7), or otherwise justify that the differences are beyond run-to-run variance. If computational budget is limited, reporting two seeds for the smallest gains and the corresponding spread would already strengthen the consistency claim.
minor comments (5)
  1. [Section 4.1 / Figures 7–10] Please state explicitly whether the CKA, per-category expertise, and fusion analyses are performed on the 300-class subset used for ablations or on full ImageNet; if the former, note that the mechanism analysis is illustrative and not carried out on the full benchmark.
  2. [Section 5.2 and Table 7 caption] The fine-tuning epoch counts are contradictory: Section 5.2 says ViT-S/16 and ViT-B/16 are fine-tuned for 100 and 200 epochs, respectively, while the Table 7 caption says 200 and 100. Please correct the inconsistency.
  3. [Table 12] The unit label 'FLOPS' should be 'FLOPs' (floating-point operations); the table reports quantities in GFLOPs, not operations per second. Please fix the unit.
  4. [Abstract / Section 1] The paper states that source code will be made publicly available, but no code or detailed pseudocode is included in the submission. Given the many implementation details (attention mask placement, adaptive pooling convolution, TEN, per-token heads), providing code or at least pseudocode in a supplement would aid reproducibility.
  5. [Equation (7)] There is a subscript typo: the classifier weights are written as 'W i 1 ∈ R D×C' but should presumably be 'W i c ∈ R D×C'. Please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MTE's gains are empirically benchmarked, and no fitted quantity or self-citation carries the derivation.

full rationale

The central claim is an empirical method claim: adding multiple auxiliary tokens and an online distillation loss (Eq. 6) improves representation quality on downstream tasks. The auxiliary tokens are defined directly from the encoder (Eq. 2 and Section 3.2), and the distillation objective in Eq. 5 is a standard teacher-student loss rather than a relabeled fit. The attention mask in Fig. 4 and the ablation in Tab. 16 make the inference-time removal of auxiliary tokens an architectural construction, not a circular prediction. The complementarity of the tokens is verified empirically through CKA, linear probing, and NMI, not asserted by definition. Hyperparameters M=4 and K=6 are chosen by ablation (Fig. 15), so they are model selection rather than fitted constants renamed as predictions. Self-citations appear for component choices such as large-kernel convolutions (refs. 19, 26) and for the HSSL baseline (ref. 32), but these are not load-bearing premises: no uniqueness theorem is invoked, and the headline comparisons are against external baselines and ImageNet benchmarks. Empirical caveats remain, including the absence of error bars, the in-house BYOL baseline implemented with the DINO codebase, and the 300-class subset used for the complementarity analysis, but these affect reliability and generality rather than circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and relies on standard self-supervised learning abstractions, a complementarity assumption that is only empirically validated, and an attention-masking assumption. The free parameters are the number of auxiliary CLS tokens and adaptively pooled tokens, selected by ablation on a 300-class subset. No new physical entities or external mediators are introduced; auxiliary tokens are internal model components.

free parameters (3)
  • Number of auxiliary CLS tokens M = 4
    Chosen from the ablation in Figure 15 on a 300-class ImageNet subset, where performance saturates with four auxiliary CLS tokens.
  • Number of adaptively pooled tokens K = 6
    Chosen from the ablation in Figure 15 on a 300-class ImageNet subset, where performance saturates with six adaptively pooled tokens alongside four CLS tokens.
  • Adaptive pooling kernel size = 11x11 depth-wise convolution plus 1x1 point-wise convolution
    Architecture choice inherited from prior large-kernel convolution works (references 19 and 26); it is not varied in the ablations.
assumptions (4)
  • domain assumption The general self-supervised loss L(h_hat, h_tilde) in Eq. 1 abstracts contrastive, clustering, and similarity losses.
    Section 3.1 states that most methods can be abstracted as L(h_hat, h_tilde), and MTE builds on this abstraction.
  • ad hoc to paper Randomly initialized auxiliary CLS tokens and different adaptive pooling weights produce complementary representations.
    This is the core mechanism of MTE; it is supported only by empirical CKA and category-expertise analysis in Section 4.1 on a 300-class subset.
  • domain assumption Attention masking between global and auxiliary tokens prevents reliance on auxiliary tokens, so removal at inference is lossless.
    Section 3.4 and Figure 4 describe the masking; it is validated on a 300-class subset in Table 16, not on full ImageNet.
  • ad hoc to paper Online distillation from auxiliary tokens to the global token does not collapse the auxiliary tokens and improves both.
    This is an empirical discovery reported in Section 4.2, Figures 12 and 13; it is not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Token Enhancing for Vision Representation Learning." pith.science (2026). https://pith.science/paper/LACPG73R

@misc{pith2026241115787,
  author       = {Pith},
  title        = {Pith review of: Multi-Token Enhancing for Vision Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LACPG73R}},
  note         = {Machine review of arXiv:2411.15787}
}
read the original abstract

Vision representation learning, especially self-supervised learning, is pivotal for various vision applications. Ensemble learning has also succeeded in enhancing the performance and robustness of the vision models. However, traditional ensemble strategies are impractical for representation learning, especially self-supervised representation learning that requires large-scale datasets and long schedules. This is because they require k times more training and inference computation costs for an ensemble of k models. Differently, we introduce Multi-Token Enhancing (MTE) that extracts multiple auxiliary tokens simultaneously from a single model to enhance representation learning, while incurring minimal additional training costs and no additional inference costs. These auxiliary tokens, including auxiliary CLS tokens and adaptively pooled tokens, capture complementary information due to their differences. Meanwhile, to address the increase in inference costs, we distill the knowledge acquired by the auxiliary tokens into a global token during pre-training. Consequently, we can discard the auxiliary tokens during inference without incurring additional costs. Our MTE is compatible with various self-supervised loss functions and architectures, consistently improving performances across different downstream tasks. Our source code will be made publicly available.

Figures

Figures reproduced from arXiv: 2411.15787 by the authors.

Figure 1
Figure 1. Increasing the number of the proposed auxiliary to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. k-NN Top-1 accuracies when cooperating MTE with dif￾ferent methods, including MoBY [58], DINO [5], and iBOT [66]. that leverage only a single token, i.e. a CLS token [9, 66, 67] or a pooled token [7, 23], to learn global representations. 2.2. Vision Architectures In recent years, various architectures have been proposed, with convolution neural networks [22, 35] and vision trans￾formers [12, 34, 49] as two main cate… view at source ↗
Figure 3
Figure 3. For effective training, our MTE employs additional auxiliary parts, which will be discarded during inference and fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The attention mask in the self-attention layers. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The visualization of the kernel weights in the large kernel [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 9
Figure 9. Figure 9: The complementarity between auxiliary tokens. We [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 7
Figure 7. Figure 7: The centered kernel alignment (CKA) [28] between dif￾ferent pairs of auxiliary tokens when using independent projection heads for different auxiliary tokens. We measure the CKA using the auxiliary tokens output by the projection heads. Ta and Tp are the auxiliary CLS t…
Figure 10
Figure 10. Figure 10: The complementarity between auxiliary tokens after [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 13
Figure 13. Figure 13: Training dynamic of the normalized mutual informa [PITH_FULL_IMAGE:figures/full_fig_p007_13.png]
Figure 14
Figure 14. Figure 14: The k-NN when using n patch tokens with the top￾n highest self-attention scores. The models are pre-trained on ImageNet-1K [40] for 100 epochs [PITH_FULL_IMAGE:figures/full_fig_p007_14.png]
Figure 15
Figure 15. Figure 15: The k-NN performance when using different numbers of auxiliary tokens. For all experiments of adaptively pooled to￾kens, four auxiliary CLS tokens are also used. dense predictions. For instance segmentation, we use Cas￾caded Mask RCNN [3] as the detector and fine-tune…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 54 canonical work pages

  1. [1]

    Asano, Christian Rupprecht, and Andrea Vedaldi

    Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020. 2

    work page 2020

  2. [2]

    BEit: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In ICLR, 2022. 9, 10

    work page 2022

  3. [3]

    Cascade r-cnn: Delving into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018. 10

    work page 2018

  4. [4]

    Unsupervised learn- ing of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learn- ing of visual features by contrasting cluster assignments. In NeurIPS, 2020. 4, 8, 9 11

    work page 2020

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 3, 4, 6, 7, 8, 9, 10

    work page 2021

  6. [6]

    Mixed autoencoder for self-supervised visual representation learning

    Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-Yan Yeung. Mixed autoencoder for self-supervised visual representation learning. In CVPR, pages 22742– 22751, 2023. 2

    work page 2023

  7. [7]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 3, 4

    work page 2020

  8. [8]

    Exploring simple siamese rep- resentation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. In CVPR, 2021. 2, 4, 8

    work page 2021

Show all 68 references
  1. [9]

    An empiri- cal study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empiri- cal study of training self-supervised vision transformers. In ICCV, 2021. 2, 4, 8, 9

    2021

  2. [10]

    Convit: Improving vision transformers with soft convolutional inductive biases

    St ´ephane D’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021. 4

    2021

  3. [11]

    Ensemble methods in machine learn- ing

    Thomas G Dietterich. Ensemble methods in machine learn- ing. In International workshop on multiple classifier systems, pages 1–15. Springer, 2000. 1, 2

    2000

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  5. [13]

    Whitening for self-supervised representation learning

    Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML, 2021. 2

    2021

  6. [14]

    Seed: Self-supervised dis- tillation for visual representation

    Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised dis- tillation for visual representation. In ICLR, 2021. 3

    2021

  7. [15]

    Evolved part masking for self-supervised learning

    Zhanzhou Feng and Shiliang Zhang. Evolved part masking for self-supervised learning. In CVPR, pages 10386–10395,

  8. [16]

    Ganaie, Minghui Hu, A.K

    M.A. Ganaie, Minghui Hu, A.K. Malik, M. Tanveer, and P.N. Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022. 1

    2022

  9. [17]

    Large-scale un- supervised semantic segmentation

    Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming- Ming Cheng, Junwei Han, and Philip Torr. Large-scale un- supervised semantic segmentation. IEEE TPAMI, 2022. 11

    2022

  10. [18]

    Richemond, Elena Buchatskaya, Carl Doersch, Bernardo ´Avila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo ´Avila Pires, Zhaohan Guo, Moham- mad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko. Bootstrap your own latent -...

    2020

  11. [19]

    Visual attention network

    Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.09741, 2022. 4

    2022 arXiv

  12. [20]

    Hansen and P

    L.K. Hansen and P. Salamon. Neural network ensembles. IEEE TPAMI, 12(10):993–1001, 1990. 1, 2

    1990

  13. [21]

    Training independent subnetworks for robust prediction

    Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In ICLR, 2021. 3

    2021

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  15. [23]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In CVPR, 2020. 2, 3, 4

    2020

  16. [24]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 10

    2022

  17. [25]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 3

    2015 arXiv

  18. [26]

    Conv2former: A simple transformer-style convnet for visual recognition

    Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. arXiv preprint arXiv:2211.11943, 2022. 4

    2022 arXiv

  19. [27]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018. 1, 2

    2018 arXiv

  20. [28]

    Similarity of neural network representa- tions revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representa- tions revisited. In ICML, 2019. 5, 6

    2019

  21. [29]

    Fractalnet: Ultra-deep neural networks without residuals

    Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. 3

    2017

  22. [30]

    Why m heads are bet- ter than one: Training a diverse ensemble of deep networks

    Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are bet- ter than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314, 2015. 3

    2015 arXiv

  23. [31]

    Sere: Exploring feature self-relation for self-supervised trans- former

    Zhong-Yu Li, Shanghua Gao, and Ming-Ming Cheng. Sere: Exploring feature self-relation for self-supervised trans- former. IEEE TPAMI, 2023. 2

    2023

  24. [32]

    Enhancing representa- tions through heterogeneous self-supervised learning

    Zhong-Yu Li, Bo-Wen Yin, Shanghua Gao, Yongxiang Liu, Li Liu, and Ming-Ming Cheng. Enhancing representa- tions through heterogeneous self-supervised learning. arXiv preprint arXiv:2310.05108, 2023. 2, 8

    2023

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 9, 10

    2014

  26. [34]

    Swin trans- former: Hierarchical vision transformer using shifted win- dows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hierarchical vision transformer using shifted win- dows. ICCV, 2021. 2, 3, 8, 9

    2021

  27. [35]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. CVPR, 2022. 2, 3, 4

    2022

  28. [36]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 9

    2019

  29. [37]

    Representation uncertainty in self-supervised learning as variational inference

    Hiroki Nakamura, Masashi Okada, and Tadahiro Taniguchi. Representation uncertainty in self-supervised learning as variational inference. In ICCV, pages 16484–16493, 2023. 2 12

    2023

  30. [38]

    Simreg: Regression as a sim- ple yet effective tool for self-supervised knowledge distilla- tion

    K L Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Te- jankar, and Hamed Pirsiavash. Simreg: Regression as a sim- ple yet effective tool for self-supervised knowledge distilla- tion. In BMVC, 2021. 3

    2021

  31. [39]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 2, 4

    2018 arXiv

  32. [40]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211–252,

  33. [41]

    Learning common rationale to improve self-supervised rep- resentation for fine-grained visual recognition problems

    Yangyang Shu, Anton van den Hengel, and Lingqiao Liu. Learning common rationale to improve self-supervised rep- resentation for fine-grained visual recognition problems. In CVPR, pages 11392–11401, 2023. 4

    2023

  34. [42]

    Ivor J. A. Simpson, Sara Vicente, and Neill D. F. Campbell. Learning structured gaussians to approximate deep ensem- bles. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 366– 374, 2022. 1

    2022

  35. [43]

    Multi- mode online knowledge distillation for self-supervised visual representation learning

    Kaiyou Song, Jin Xie, Shan Zhang, and Zimeng Luo. Multi- mode online knowledge distillation for self-supervised visual representation learning. In CVPR, 2023. 2

    2023

  36. [44]

    Semantics-consistent feature search for self-supervised visual representation learning

    Kaiyou Song, Shan Zhang, Zimeng Luo, Tong Wang, and Jin Xie. Semantics-consistent feature search for self-supervised visual representation learning. In ICCV, pages 16099– 16108, 2023. 2

    2023

  37. [45]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. 3

    1929

  38. [46]

    Siamese image modeling for self-supervised vision represen- tation learning

    Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision represen- tation learning. In CVPR, pages 2132–2141, 2023. 2

    2023

  39. [47]

    Un- derstanding self-supervised learning dynamics without con- trastive pairs

    Yuandong Tian, Xinlei Chen, and Surya Ganguli. Un- derstanding self-supervised learning dynamics without con- trastive pairs. In ICML, 2020. 2

    2020

  40. [48]

    The inaturalist species classification and de- tection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and de- tection dataset. In CVPR, 2018. 9, 10

    2018

  41. [49]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568–578,

  42. [50]

    Dense contrastive learning for self-supervised visual pre-training

    Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021. 2

    2021

  43. [51]

    Masked feature predic- tion for self-supervised visual pre-training

    Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature predic- tion for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133, 2021. 2

    2021 arXiv

  44. [52]

    Batchensemble: an alternative approach to efficient ensemble and lifelong learning

    Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2020. 3

    2020

  45. [53]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In CVPR, pages 16133–16142, 2023. 2

    2023

  46. [54]

    Cvt: Introducing con- volutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing con- volutions to vision transformers. In ICCV, 2021. 4

    2021

  47. [55]

    P2T: Pyramid pooling transformer for scene understanding

    Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2T: Pyramid pooling transformer for scene understanding. IEEE TPAMI, 2022. 4

    2022

  48. [56]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. In ECCV, 2018. 10

    2018

  49. [57]

    Detco: Unsu- pervised contrastive learning for object detection

    Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsu- pervised contrastive learning for object detection. In ICCV,

  50. [58]

    Self-supervised learning with swin transformers

    Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04553, 2021. 2, 8, 9

    2021 arXiv

  51. [59]

    Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning

    Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021. 2

    2021

  52. [60]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, pages 9653–9663, 2022. 2

    2022

  53. [61]

    Bag of instances aggregation boosts self-supervised distillation

    Haohang Xu, Jiemin Fang, XIAOPENG ZHANG, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts self-supervised distillation. In ICLR, 2022. 3

    2022

  54. [62]

    Joint unsuper- vised learning of deep representations and image clusters

    Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsuper- vised learning of deep representations and image clusters. In CVPR, 2016. 2

    2016

  55. [63]

    Decoupled contrastive learning

    Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In ECCV, 2022. 2

    2022

  56. [64]

    Online deep clustering for unsupervised representation learning

    Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR, 2020. 2

    2020

  57. [65]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 10

    2017

  58. [66]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022. 1, 2, 6, 8, 9, 10

    2022

  59. [67]

    Mugs: A multi- granular self-supervised learning framework

    Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi- granular self-supervised learning framework. In arXiv preprint arXiv:2203.14415, 2022. 2

    2022 arXiv

  60. [68]

    Multi-label self- supervised learning with scene images

    Ke Zhu, Minghao Fu, and Jianxin Wu. Multi-label self- supervised learning with scene images. In ICCV, pages 6694–6703, 2023. 2 13

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.