Pith. sign in

REVIEW 3 major objections 4 minor 75 references

On the Surprising Effectiveness of Attention Transfer for Vision Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-training works because of attention maps, not learned features.

desk verdict Attention transfer is a real finding with a clean causal design, but the headline match to fine-tuning rests on a 200-epoch distillation run versus a 100-epoch baseline; the phenomenon survives, the exact equality does not. read the letter →

arxiv 2411.09702 v1 pith:OMLDDTTW submitted 2024-11-14 cs.LG cs.AIcs.CVcs.NE

Alexander C. Li , Yuandong Tian , Beidi Chen , Deepak Pathak , Xinlei Chen This is my paper · ORCID
classification cs.LGcs.AIcs.CVcs.NE
keywords attentiontransfervisiontransformerspre-trainingknowledgedistillationself-supervisedlearningfine-tuningmapsrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the usual story that pre-training helps Vision Transformers by giving them useful features. Its claim is that the inter-token attention maps learned during pre-training are typically sufficient: a student that receives only those maps, and learns all features from scratch, can match full weight fine-tuning on ImageNet-1K. The authors show this with two simple methods, Attention Copy and Attention Distillation, and verify it across pretraining methods, model sizes, and a detection task. The finding matters because it separates what pre-training actually contributes—how information flows between patches—from the features themselves, and points to a way to reuse pretrained models without sharing weights. The paper also marks the boundary of the claim: attention transfer loses to fine-tuning under distribution shift and when the pretraining and downstream datasets are far apart.

What carries the argument

The key object is the attention map in each multi-head self-attention block, defined as $\mathrm{softmax}(QK^\top)$; it is the only computation in a Vision Transformer that exchanges information between tokens, since value projections, MLPs, and normalization act per token. Attention Copy replaces the student's attention maps with frozen teacher maps, fully decoupling inter-token routing from intra-token feature learning. Attention Distillation replaces that hard constraint with a cross-entropy loss that pushes the student's attention maps toward the teacher's, summed over layers and heads, and the teacher is discarded at inference. This decoupling is what lets the experiments attribute the pre-training gain to information flow rather than to the features that flow through it.

What would settle it

Train a ViT-L from scratch on ImageNet-1K while distilling only the attention maps of a MAE-pretrained teacher, reproducing the paper's recipe; if an independent run cannot reach the reported 85.7 in-distribution accuracy, the sufficiency claim fails. A sharper check: if a fixed, input-independent attention pattern also reaches 85.7, then the learned content of the attention maps is not what carries the benefit.

Watch

Extended reading notes

Core claim

The central discovery is that for a pre-trained Vision Transformer, the attention map matrix $\mathrm{softmax}(QK^\top)$—the only operation that routes information between tokens—carries enough of the pre-training benefit to recover the full fine-tuning accuracy. With a MAE-pretrained ViT-L on ImageNet-1K, training from scratch reaches 83.0, full weight fine-tuning reaches 85.7, Attention Copy reaches 85.1, and Attention Distillation reaches 85.7. Attention Distillation also recovers most of the pre-training gain in COCO object detection, and ensembling a distilled student with a fine-tuned teacher improves ImageNet accuracy to 86.3, showing the student is not merely relearning the teacher. The same pattern holds for MoCo-v3, FLIP, and DINO teachers and across ViT-B/L/H sizes. The paper states the limits explicitly: attention transfer is consistently worse than fine-tuning on ImageNet-A, ImageNet-R, ImageNet-Sketch, and ImageNet-V2, and the gap widens when pretraining and downstream datasets differ, so pretrained features do add robustness and distribution-mismatch tolerance beyond what attention maps provide.

Load-bearing premise

The load-bearing premise is that 'downstream performance' is in-distribution accuracy on benchmarks similar to the pre-training data; once robustness to distribution shift or transfer to a different dataset is included, the paper's own experiments show attention maps alone do not suffice.

Editorial extensions

If this is right

  • If attention maps are sufficient, the features learned in pre-training are not what fine-tuning needs, at least for in-distribution classification.
  • Attention Distillation can replace weight fine-tuning with the same inference-time model and no layerwise learning-rate decay, since the student learns its own features from scratch.
  • Ensembling an attention-distilled student with a fine-tuned teacher gives a concrete accuracy gain (86.3 on ImageNet-1K), because the two models make different errors.
  • The transfer benefit scales with model size and is not specific to MAE, holding for contrastive (MoCo-v3) and image-text (FLIP) pretraining as well.
  • Under distribution shift or large pretrain-to-downstream dataset differences, attention transfer underperforms fine-tuning, so pretrained features still carry robustness value.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: attention-map transfer could make cross-size transfer easier than weight fine-tuning, since attention maps are shaped by sequence length rather than embedding width; the paper raises this but does not test it.
  • Inference: a hybrid that distills attention maps while also transferring a small amount of feature information might recover the out-of-distribution robustness that pure attention transfer lacks.
  • Inference: the sufficiency result is naturally read as a statement about the optimization landscape—a randomly initialized student with good routing can fit in-distribution tasks as well as a pretrained network—so the mechanism is likely tied to the training dynamics of attention, not to the semantic content of the features.
  • Inference: if attention maps are the reusable unit, sharing a small set of maps instead of full weights could address settings where weight sharing is undesirable for security or memory reasons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes two methods, Attention Copy and Attention Distillation, to transfer only the attention maps of a pre-trained Vision Transformer teacher to a randomly initialized student, leaving all other weights to be learned from scratch. On ImageNet-1K with a MAE-pretrained ViT-L, Attention Copy reaches 85.1% top-1 accuracy and Attention Distillation reaches 85.7%, compared with 83.0% for training from scratch and 85.7% for full fine-tuning. The authors use these results to argue that pre-trained features are not essential and that the attention patterns alone largely account for the downstream benefit of pre-training. The paper also analyzes which layers, heads, and attention-related activations matter, shows that attention-transfer students are dissimilar to the fine-tuned teacher by CKA and ensembling, and documents settings where attention transfer underperforms fine-tuning, including cross-dataset transfer and out-of-distribution robustness.

Significance. If the central claim holds, the paper is significant: it challenges the standard feature-centric interpretation of pre-training for vision transformers and identifies inter-token attention routing as a separable, transferable component. The Attention Copy method is a genuinely clean decoupling of inter-token and intra-token computation, and the random-teacher control in Table 7 is a strong falsification check. The paper also honestly reports limitations in Section 5, and the code release will facilitate reproduction. However, the headline quantitative claim that Attention Distillation 'matches fine-tuning' currently rests on an unequal training budget, and one supporting baseline in the appendix is inconsistent with the main text. These issues affect the strength of the central claim and require correction before the paper can be accepted.

major comments (3)
  1. [Section 3; Appendix A.2; Appendix C, Table 16] The headline equality in Table 1 (Attention Distillation 85.7 = fine-tuning 85.7) is not an equal-budget comparison. The Attention Distillation recipe in Table 16 uses 200 epochs, warmup 20, base learning rate 1e-4, weight decay 0.3, drop path 0.2, distillation weight lambda=3, and distillation applied to the first 18 layers, while Appendix A.2 states that fine-tuning is 100 epochs of regular fine-tuning. No 200-epoch fine-tuning result is reported, so the equality may reflect the doubled training budget and selected hyperparameters rather than the sufficiency of attention maps. Please report fine-tuning under the same 200-epoch recipe, or provide an accuracy-versus-epoch curve showing that fine-tuning saturates by epoch 100, and also report Attention Distillation under the 100-epoch recipe used for the baselines. Without this control, the claim that attention transfer 'completely matches fine-tuning on ImageNet-1K' (Section 7) is not established. Note that the cleaner Attention Copy probe at 100 epochs reaches 85.1, recovering 77.8% of the gap, which remains a strong and interesting result even if the equality with fine-tuning is not confirmed.
  2. [Appendix B.2, Table 13] The scratch baseline in Table 13 is reported as 79.7, while the main text and Tables 1, 7, and 8 report scratch ViT-L accuracy as 83.0. Since Table 13 is used to argue that feature distillation (81.3) is not competitive with attention distillation (85.7), this inconsistency matters: if the 79.7 number comes from a different training recipe, the feature-distillation comparison is not controlled against the main scratch baseline. Please clarify which recipe produced the 79.7 result, or rerun the feature-distillation baseline with the same recipe as Table 15 and the main scratch experiments, and reconcile the discrepancy.
  3. [Abstract; Section 5.1 and 5.2, Tables 4-6] The abstract states that attention maps are 'sufficient for models to learn high quality features from scratch and achieve comparable downstream performance.' The paper's own Section 5 shows this fails under distribution shift: Attention Distillation is worse than fine-tuning on all four out-of-distribution benchmarks in Table 6, and Attention Copy degrades more than fine-tuning when the pre-training and downstream datasets differ in Tables 4 and 5. Since the authors explicitly acknowledge these limitations, this is not a fatal flaw, but the headline claim should be scoped to in-distribution ImageNet-1K classification or the term 'downstream performance' should be defined to exclude robustness. As written, the abstract overstates the generality of the sufficiency result.
minor comments (4)
  1. [Appendix C, Table 16] The table header 'layers copied' should read 'layers distilled' for the Attention Distillation recipe; the same wording appears in the surrounding text.
  2. [Section 4.1, Table 2 and Appendix B.3] The result that copying queries Q (85.6) outperforms copying the full attention map (85.1) is interesting, but the explanation in Appendix B.3 is speculative and the JSD analysis is acknowledged to have limitations. Consider stating more explicitly that Q transfer still transfers part of the attention computation, and that the result does not contradict the central inter-token-flow interpretation.
  3. [Figure 7 and Appendix B.3] The captions use 'PT' and 'FT' without defining them on first use; please spell out 'pre-trained teacher' and 'fine-tuned model'.
  4. [Section 5.3, Table 7] In the 'none' row, it would be helpful to state explicitly that the teacher is a randomly initialized ViT and that the student is also randomly initialized, to make the control condition unambiguous.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: central attention-transfer claims rest on direct ImageNet accuracy comparisons, not on equations that reduce to their inputs.

full rationale

The paper's central claim is an empirical sufficiency result: a ViT trained with only copied or distilled attention maps from a pre-trained teacher, while learning all other weights from scratch, reaches ImageNet-1K accuracy comparable to full fine-tuning (Table 1: scratch 83.0, attn. copy 85.1, attn. distill 85.7, fine-tune 85.7). The measured quantity, top-1 accuracy on an external benchmark, is independent of the method's definition; neither Eq. (2) nor the training recipes in Tables 15-16 define accuracy in terms of the claim. The only self-citations are to Tian et al. [52, 53], used as background on transformer training dynamics and as an interpretive gloss ('This reinforces the theory from Tian et al. [53]'), not as premises from which the accuracy numbers are derived. The paper also includes controls that break any definitional collapse: copying from a randomly initialized teacher hurts (72.7 vs 83.0 scratch, Table 7); transferring averaged attention maps drops to 79.7-82.7 (Table 12); and feature distillation reaches only 81.3 (Table 13). These falsifiable contrasts confirm that the result is the empirical transfer of specific attention maps, not an identity. The reviewer's observation that Attention Distillation uses 200 epochs and λ=3 (Table 16) against a 100-epoch fine-tune (A.2) is a potential budget-equality confound for the 85.7=85.7 headline, but it is a validity concern, not a circularity: it does not make the claim true by definition or by fitted-parameter renaming. Hence no circular step; the paper is self-contained against external benchmarks.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper is empirical, so the listed entries are domain and evaluation assumptions rather than mathematical axioms. The central claim does not rest on invented entities, and the only tuned quantities that affect the headline result are the distillation weight and the number of distilled layers, which are reported with sensitivity analysis.

free parameters (2)
  • distillation loss weight lambda = 3
    Used for the main Attention Distillation result (85.7); Appendix B.5 reports 84.7 at lambda=0.3 and 85.3 at lambda=1.0, so the match with fine-tuning depends on this chosen value.
  • number of distilled layers = 18 of 24
    The main result distills the first 18 layers; Appendix B.5 reports 85.3 for 18 layers, 85.5 for 21, and 85.1 for 24 under lambda=1, making the layer count a tuned design choice.
assumptions (3)
  • domain assumption Inter-token communication in a ViT is fully captured by the attention maps softmax(QK^T); all other operations are intra-token.
    Section 2.1 states this decoupling, which is the premise that isolating attention maps isolates the entire inter-token flow.
  • domain assumption ImageNet-1K top-1 accuracy is an adequate measure of downstream performance for the sufficiency claim.
    Used throughout Section 3; the paper later limits the claim when evaluating out-of-distribution shifts in Section 5.2.
  • domain assumption The selected teachers (MAE, MoCo-v3, FLIP, DINO) are representative of pre-training methods.
    Section 5.3 uses these methods to generalize the finding beyond MAE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Surprising Effectiveness of Attention Transfer for Vision Transformers." pith.science (2026). https://pith.science/paper/OMLDDTTW

@misc{pith2026241109702,
  author       = {Pith},
  title        = {Pith review of: On the Surprising Effectiveness of Attention Transfer for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMLDDTTW}},
  note         = {Machine review of arXiv:2411.09702}
}
read the original abstract

Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential. Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve comparable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from a pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systematically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope our exploration provides a better understanding of what pre-training accomplishes and leads to a useful alternative to the standard practice of fine-tuning

Figures

Figures reproduced from arXiv: 2411.09702 by the authors.

Figure 1
Figure 1. Using only attention is sufficient for full performance. By copying the attention maps (top) from a MAE [22] pre-trained ViT-L [12], a ViT-L can reach a top-1 accuracy of 85.1 on ImageNet-1K [10] – recovering 77.8% of the gap between no transfer (training from scratch, 83.0) and full transfer (fine￾tuning all the weights, 85.7). Distilling attention maps (bottom) can even fully match MAE weight tuning while only tra… view at source ↗
Figure 2
Figure 2. Two types of Attention transfer for Vision Transformers. Attention Copy (left): We simply “copy-and-paste” the attention maps from a pre-trained teacher model to a randomly initialized student one. Other weights of the student are then trained via supervised learning. This fully decouples inter-token learning (from the teacher) and intra-token learning (in the student); but is less practical. Attention Distillation … view at source ↗
Figure 3
Figure 3. Copy a subset of layers. By default, all 24 ViT-L layers are transferred. Here we only transfer a subset, and find: more layers always helps; and attention maps from top layers are more beneficial than those from bottom layers. 0 4 8 12 16 number of heads transferred 77.5 80.0 82.5 85.0 76.1 77.2 81.2 84.8 85.1 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: CKA representation similarity to the fine-tuned model. We use CKA [32] to measure the layer-wise similarity between representations learned in different models against the fine-tuned MAE model. Higher means more similar. We find that attention transfer methods are quit…
Figure 6
Figure 6. Figure 6: Ensemble accuracy with the fine￾tuned model. We plot the accuracy of ensem￾bling our attention transfer models and a fine￾tuned MAE. We use this to measure model pre￾diction similarity with the fine-tuned model. The ensemble yields notable accuracy gains over the base …
Figure 7
Figure 7. Figure 7: Attention map similarity across methods. Each column corresponds to a different way of matching up attention heads between two models. The top row shows the Jensen-Shannon divergence (JSD) with respect to the MAE pre-trained teacher, whereas the bottom row shows the JS…
Figure 8
Figure 8. Figure 8: Visualization of attention maps for different methods. We show what the [CLS] token attends to at various layers within the network. Darker patches indicate more attention weight. Notably, the pre-trained MAE model’s attention maps provide a significant prior over what…
Figure 9
Figure 9. Figure 9: Weight on distillation loss Layers distilled Accuracy 18 85.3 21 85.5 24 85.1 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 11
Figure 11. Figure 11: Attention map visualizations on more examples 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Attention map visualizations on more examples 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 49 canonical work pages

  1. [1]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450, 2016

    arXiv 2016

  2. [2]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...

    work page 2020

  3. [3]

    Burkholz

    R. Burkholz. Most activation functions can win the lottery without excessive depth. Advances in Neural Information Processing Systems, 35:18707–18720, 2022

    work page 2022

  4. [4]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021

    work page 2021

  5. [5]

    M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, 2020

    2020

  6. [6]

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020

    work page 2020

  7. [7]

    X. Chen, S. Xie, and K. He. An empirical study of training self-supervised Vision Transformers. In ICCV, 2021

    work page 2021

  8. [8]

    Clark, M.-T

    K. Clark, M.-T. Luong, Q. V . Le, and C. D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020

    work page 2020

Show all 75 references
  1. [9]

    E. D. Cubuk, B. Zoph, J. Shlens, and Q. V . Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020

    2020

  2. [10]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009

    2009

  3. [11]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019

    2019

  4. [12]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

    2021

  5. [13]

    Duval, I

    Q. Duval, I. Misra, and N. Ballas. A simple recipe for competitive low-compute self supervised vision models. arXiv preprint arXiv:2301.09451, 2023

    2023 arXiv

  6. [14]

    A. Fang, G. Ilharco, M. Wortsman, Y . Wan, V . Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, 2022

    2022

  7. [15]

    Frankle and M

    J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  8. [16]

    Girshick, J

    R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014

    2014

  9. [17]

    I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  10. [18]

    Goyal, P

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv:1706.02677, 2017

    2017 arXiv

  11. [19]

    Z. Hao, J. Guo, D. Jia, K. Han, Y . Tang, C. Zhang, H. Hu, and Y . Wang. Learning efficient vision transformers via fine-grained manifold distillation. In NeurIPS, 2022

    2022

  12. [20]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016

    2016

  13. [21]

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020

    2020

  14. [22]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022

    2022

  15. [23]

    Hendrycks, S

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021. 11

    2021

  16. [24]

    Hendrycks, K

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In CVPR, 2021

    2021

  17. [25]

    Hertz, R

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  18. [26]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  19. [27]

    Howard and S

    J. Howard and S. Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018

    2018 arXiv

  20. [28]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  21. [29]

    Huang, Y

    G. Huang, Y . Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016

    2016

  22. [30]

    Izmailov, P

    P. Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516–38532, 2022

    2022

  23. [31]

    Jelassi, M

    S. Jelassi, M. Sander, and Y . Li. Vision transformers provably learn spatial structure.Advances in Neural Information Processing Systems, 35:37822–37836, 2022

    2022

  24. [32]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In ICML, 2019

    2019

  25. [33]

    LeCun, B

    Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989

    1989

  26. [34]

    Y . Li, X. Lyu, N. Koren, L. Lyu, B. Li, and X. Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930, 2021

    2021 arXiv

  27. [35]

    Y . Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022

    2022

  28. [36]

    Y . Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. In CVPR, 2023

    2023

  29. [37]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014

    2014

  30. [38]

    Loshchilov and F

    I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017

    2017

  31. [39]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019

    2019

  32. [40]

    Mahajan, R

    D. Mahajan, R. Girshick, V . Ramanathan, K. He, M. Paluri, Y . Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018

    2018

  33. [41]

    Navaneet, S

    K. Navaneet, S. A. Koohpayegani, A. Tejankar, and H. Pirsiavash. Simreg: Regression as a simple yet effective tool for self-supervised knowledge distillation. arXiv preprint arXiv:2201.05131, 2022

    2022 arXiv

  34. [42]

    Orseau, M

    L. Orseau, M. Hutter, and O. Rivasplata. Logarithmic pruning is all you need. Advances in Neural Information Processing Systems, 33:2925–2934, 2020

    2020

  35. [43]

    N. Park, W. Kim, B. Heo, T. Kim, and S. Yun. What do self-supervised vision transformers learn? arXiv preprint arXiv:2305.00729, 2023

    2023 arXiv

  36. [44]

    Pensia, S

    A. Pensia, S. Rajput, A. Nagle, H. Vishwakarma, and D. Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. Advances in neural information processing systems, 33:2599–2610, 2020

    2020

  37. [45]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  38. [46]

    Ramanujan, M

    V . Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What’s hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11893–11902, 2020. 12

    2020

  39. [47]

    Recht, R

    B. Recht, R. Roelofs, L. Schmidt, and V . Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019

    2019

  40. [48]

    Schuhmann, R

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022

    2022

  41. [49]

    N. Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  42. [50]

    Srivastava, G

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, pages 1929–1958, 2014

    1929

  43. [51]

    Szegedy, V

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In CVPR, 2016

    2016

  44. [52]

    Y . Tian, Y . Wang, B. Chen, and S. Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. NeurIPS, 2023

    2023

  45. [53]

    Y . Tian, Y . Wang, Z. Zhang, B. Chen, and S. Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. ICLR, 2024

    2024

  46. [54]

    Torralba and A

    A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, 2011

    2011

  47. [55]

    Trockman and J

    A. Trockman and J. Z. Kolter. Mimetic initialization of self-attention layers. arXiv preprint arXiv:2305.09828, 2023

    2023 arXiv

  48. [56]

    Van Horn, O

    G. Van Horn, O. Mac Aodha, Y . Song, Y . Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018

    2018

  49. [57]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS, 2017

    2017

  50. [58]

    Venkataramanan, A

    S. Venkataramanan, A. Ghodrati, Y . M. Asano, F. Porikli, and A. Habibian. Skip-attention: Improving vision transformers by paying less attention. arXiv preprint arXiv:2301.02240, 2023

    2023 arXiv

  51. [59]

    Walmer, S

    M. Walmer, S. Suri, K. Gupta, and A. Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. In CVPR, 2023

    2023

  52. [60]

    H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019

    2019

  53. [61]

    K. Wang, F. Yang, and J. van de Weijer. Attention distillation: self-supervised vision transformer students need more guidance. arXiv preprint arXiv:2210.00944, 2022

    2022 arXiv

  54. [62]

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788, 2020

    2020

  55. [63]

    Z. Xie, Z. Geng, J. Hu, Z. Zhang, H. Hu, and Y . Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023

    2023

  56. [64]

    Z. Yang, J. Zhao, B. Dhingra, K. He, W. W. Cohen, R. Salakhutdinov, and Y . LeCun. Glomo: Unsupervisedly learned relational graphs as transferable representations. arXiv preprint arXiv:1806.05662v1, 2018

    2018 arXiv

  57. [65]

    S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y . Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019

    2019

  58. [66]

    Zagoruyko and N

    S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017

    2017

  59. [67]

    Zhang, M

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2018

    2018

  60. [68]

    feature distill

    Y . Zhang, A. Backurs, S. Bubeck, R. Eldan, S. Gunasekar, and T. Wagner. Unveiling transform- ers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301, 2022. 13 Appendix A Key Numbers A.1 Information in Attention Transfer How much information is transferred d...

    2022 arXiv

  61. [69]

    This can fail since similar heads may not be in the same order across models

    Direct pair: this is the naive approach of computing the JSD between the ith head of the first model and the ith head of the second model. This can fail since similar heads may not be in the same order across models

  62. [70]

    We then use bipartite matching to create a one-to-one pairing between the heads that minimizes the cumulative JSD

    Bipartite matching: for each layer, we compute the JSD between each of the 16 heads in the first model and the 16 heads in the second model. We then use bipartite matching to create a one-to-one pairing between the heads that minimizes the cumulative JSD. This solves the previ...

  63. [71]

    We call this Minimum because each head in the first model is paired with the head from the second model with the smallest JSD

    Minimum: instead of creating a one-to-one matching, we allow many-to-one matching between heads. We call this Minimum because each head in the first model is paired with the head from the second model with the smallest JSD. This allows our metric to potentially ignore extraneo...

  64. [72]

    This can still be thrown off by extraneous heads

    Averaged maps: we average the attention maps of all heads in a layer and compare the averaged maps across models. This can still be thrown off by extraneous heads. Figure 7 shows the results of comparing models against the pre-trained teacher (top row) or fine-tuned model (bot...

  65. [73]

    Head 1 attends uniformly over all locations, head 2 attends arbitrarily over locations, and the second head’s values are set to 0

  66. [74]

    Head 1 attends uniformly over the top half of the image, head 2 attends uniformly over the bottom half of the image, and both use values V /2

  67. [75]

    sweet spot

    Head 1 attends uniformly over the left half of the image, head 2 attends uniformly over the right half of the image, and both use values V /2. 16 Layer 1 input init scratch pre-train / copy fine-tune attn. distill. Layer 13 Layer 23 Figure 8: Visualization of attention maps fo...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.