{"id":"4a66edcb-62bd-4119-9593-e97896d5c8f2","arxiv_id":"2411.09702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Transferring only the attention maps of a pre-trained ViT recovers the accuracy gain of full fine-tuning on ImageNet-1K classification.","lead":"This paper tests whether pre-training Vision Transformers works by teaching features or by teaching where information should flow. It reports that transferring only the pre-trained attention maps lets a randomly initialized model match full fine-tuning on ImageNet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 85.7=85.7 headline match between Attention Distillation and fine-tuning is not an equal-budget comparison: Table 16 uses 200 epochs and λ=3 while the fine-tuning baseline is reported at 100 epochs (A.2), so the central equality may reflect longer training rather than attention-map sufficiency.","rationale":"The paper is a serious empirical study: it isolates attention maps from features with Attention Copy and Attention Distillation, includes a random-teacher control (Table 7, none row), a feature-distillation control (Table 13), CKA and ensembling analyses, and honestly reports distribution-shift and cross-dataset limitations. The central claim, however, is the in-distribution equality between Attention Distillation and fine-tuning on ImageNet-1K. For that equality to be evidence that pre-trained attention maps are sufficient, the two methods must be compared under comparable optimization budgets. They are not: the distillation recipe in Table 16 uses 200 epochs and a tuned λ=3, while the fine-tuning baseline is described as 100 epochs in Appendix A.2. The cleaner Attention Copy variant, which uses 100 epochs, closes 77.8% of the gap rather than matching it, so the full match is associated precisely with the longer, differently hyperparameterized distillation schedule. The reader's weakest assumption was out-of-distribution robustness; that is a real scope limitation and the authors acknowledge it, but it does not undermine the headline equality. The unequal-budget issue is more load-bearing because it threatens the exact in-distribution result that drives the paper's strongest claim. I therefore recommend keeping the verdict CONDITIONAL, with the added condition that the fine-tuning baseline be re-run at 200 epochs under the Table 16 recipe (without the distillation loss) and that multiple seeds be reported for the Table 1 comparisons.","tokens_in":17546,"tokens_out":9236,"duration_ms":98466,"concrete_test":"Re-run the fine-tuning baseline on ImageNet-1K with the exact Attention Distillation schedule from Table 16 (200 epochs, lr 1e-4, weight decay 0.3, drop path 0.2, EMA, augmentations) with the distillation loss set to zero, and report top-1 accuracy over at least 3 seeds. If the 200-epoch fine-tune exceeds 85.7 beyond seed noise, the headline equality is an artifact of unequal training budgets; also compare Attention Distillation at 100 epochs with λ tuned to see how much of the gain is schedule-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on Table 1, where Attention Distillation equals fine-tuning at 85.7 on ImageNet-1K. But the two methods are not trained under comparable budgets. Appendix C (Table 16) gives the Attention Distillation recipe: 200 epochs, warmup 20, lr 1e-4, weight decay 0.3, drop path 0.2, distillation weight λ=3, and distillation applied to the first 18 layers. Appendix A.2 describes \"100 epochs of regular fine-tuning\" as the baseline; no 200-epoch fine-tuning result is reported. Attention Copy, the cleaner fully-decoupled probe, reaches 85.1 under the 100-epoch recipe in Table 15, closing only 77.8% of the gap. Thus the strongest form of the claim, that attention transfer can match fine-tuning, is currently established only by comparing a 200-epoch, hyperparameter-selected distillation run against a 100-epoch fine-tune. Unless fine-tuning saturates by epoch 100, the equality is a schedule artifact. This is a confound, not a scope limitation: it applies exactly where the paper makes its headline claim, in-distribution on ImageNet-1K. The OOD and cross-dataset failures in Tables 4-6 are acknowledged by the authors and weaken generality, but they do not threaten the in-distribution equality; the epoch mismatch does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two methods, Attention Copy and Attention Distillation, to transfer only the attention maps of a pre-trained Vision Transformer teacher to a randomly initialized student, leaving all other weights to be learned from scratch. On ImageNet-1K with a MAE-pretrained ViT-L, Attention Copy reaches 85.1% top-1 accuracy and Attention Distillation reaches 85.7%, compared with 83.0% for training from scratch and 85.7% for full fine-tuning. The authors use these results to argue that pre-trained features are not essential and that the attention patterns alone largely account for the downstream benefit of pre-training. The paper also analyzes which layers, heads, and attention-related activations matter, shows that attention-transfer students are dissimilar to the fine-tuned teacher by CKA and ensembling, and documents settings where attention transfer underperforms fine-tuning, including cross-dataset transfer and out-of-distribution robustness.","tokens_in":17868,"tokens_out":5634,"duration_ms":53815,"significance":"If the central claim holds, the paper is significant: it challenges the standard feature-centric interpretation of pre-training for vision transformers and identifies inter-token attention routing as a separable, transferable component. The Attention Copy method is a genuinely clean decoupling of inter-token and intra-token computation, and the random-teacher control in Table 7 is a strong falsification check. The paper also honestly reports limitations in Section 5, and the code release will facilitate reproduction. However, the headline quantitative claim that Attention Distillation 'matches fine-tuning' currently rests on an unequal training budget, and one supporting baseline in the appendix is inconsistent with the main text. These issues affect the strength of the central claim and require correction before the paper can be accepted.","major_comments":[{"comment":"The headline equality in Table 1 (Attention Distillation 85.7 = fine-tuning 85.7) is not an equal-budget comparison. The Attention Distillation recipe in Table 16 uses 200 epochs, warmup 20, base learning rate 1e-4, weight decay 0.3, drop path 0.2, distillation weight lambda=3, and distillation applied to the first 18 layers, while Appendix A.2 states that fine-tuning is 100 epochs of regular fine-tuning. No 200-epoch fine-tuning result is reported, so the equality may reflect the doubled training budget and selected hyperparameters rather than the sufficiency of attention maps. Please report fine-tuning under the same 200-epoch recipe, or provide an accuracy-versus-epoch curve showing that fine-tuning saturates by epoch 100, and also report Attention Distillation under the 100-epoch recipe used for the baselines. Without this control, the claim that attention transfer 'completely matches fine-tuning on ImageNet-1K' (Section 7) is not established. Note that the cleaner Attention Copy probe at 100 epochs reaches 85.1, recovering 77.8% of the gap, which remains a strong and interesting result even if the equality with fine-tuning is not confirmed.","section":"Section 3; Appendix A.2; Appendix C, Table 16"},{"comment":"The scratch baseline in Table 13 is reported as 79.7, while the main text and Tables 1, 7, and 8 report scratch ViT-L accuracy as 83.0. Since Table 13 is used to argue that feature distillation (81.3) is not competitive with attention distillation (85.7), this inconsistency matters: if the 79.7 number comes from a different training recipe, the feature-distillation comparison is not controlled against the main scratch baseline. Please clarify which recipe produced the 79.7 result, or rerun the feature-distillation baseline with the same recipe as Table 15 and the main scratch experiments, and reconcile the discrepancy.","section":"Appendix B.2, Table 13"},{"comment":"The abstract states that attention maps are 'sufficient for models to learn high quality features from scratch and achieve comparable downstream performance.' The paper's own Section 5 shows this fails under distribution shift: Attention Distillation is worse than fine-tuning on all four out-of-distribution benchmarks in Table 6, and Attention Copy degrades more than fine-tuning when the pre-training and downstream datasets differ in Tables 4 and 5. Since the authors explicitly acknowledge these limitations, this is not a fatal flaw, but the headline claim should be scoped to in-distribution ImageNet-1K classification or the term 'downstream performance' should be defined to exclude robustness. As written, the abstract overstates the generality of the sufficiency result.","section":"Abstract; Section 5.1 and 5.2, Tables 4-6"}],"minor_comments":[{"comment":"The table header 'layers copied' should read 'layers distilled' for the Attention Distillation recipe; the same wording appears in the surrounding text.","section":"Appendix C, Table 16"},{"comment":"The result that copying queries Q (85.6) outperforms copying the full attention map (85.1) is interesting, but the explanation in Appendix B.3 is speculative and the JSD analysis is acknowledged to have limitations. Consider stating more explicitly that Q transfer still transfers part of the attention computation, and that the result does not contradict the central inter-token-flow interpretation.","section":"Section 4.1, Table 2 and Appendix B.3"},{"comment":"The captions use 'PT' and 'FT' without defining them on first use; please spell out 'pre-trained teacher' and 'fine-tuned model'.","section":"Figure 7 and Appendix B.3"},{"comment":"In the 'none' row, it would be helpful to state explicitly that the teacher is a randomly initialized ViT and that the student is also randomly initialized, to make the control condition unambiguous.","section":"Section 5.3, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The core scientific direction is fresh and the Attention Copy probe is well designed, but the paper's strongest advertised result--matching fine-tuning exactly--depends on an unequal training budget and tuned hyperparameters. The scratch-baseline discrepancy in Appendix B.2 also needs to be resolved. These are fixable with additional experiments or a more carefully scoped claim, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time, but read Table 16 before you believe the headline. The cleanest result — Attention Copy closing 78% of the gap between scratch and fine-tuning on ImageNet, with random-teacher and feature-distillation controls — is solid and genuinely new. The stronger claim, that Attention Distillation matches fine-tuning at 85.7, is built on a 200-epoch run with λ=3 and 18 distilled layers, while the fine-tune baseline is the standard 100-epoch recipe. That's not a fair comparison. Unless fine-tuning saturates by epoch 100, the equality is partly a training-budget artifact. Attention Copy, which is the decoupled probe, gets 85.1 in 100 epochs, so the sufficiency result is 'almost all of the benefit' rather than 'all of it.' The paper would be more honest with an equal-epoch fine-tune baseline and a few seeds.\n\nWhat is genuinely good: the copy experiment is a clean causal isolation of inter-token routing from intra-token features. The random-teacher control (72.7) shows the effect isn't just the distillation mechanism. Feature distillation (81.3) is worse than attention distillation, which supports their interpretation. The paper also reports the negative results — worse OOD robustness, worse cross-dataset transfer — rather than hiding them. The ensembling gain is a nice bonus and makes sense given the CKA and ensemble evidence that the student isn't just relearning the teacher.\n\nThe soft spots beyond the epoch mismatch: no error bars anywhere, so we don't know if 85.7 vs 85.1 is meaningful. The headline claim in the abstract — 'features are not essential' — is too strong given that features are what carry the OOD robustness in Table 6. And the 'attention maps from fine-tuned teacher' variant (85.6) suggests task-specific attention helps, which slightly blunts the 'pre-training maps are enough' story. But these are caveats, not refutations.\n\nWho should read it: anyone working on ViT transfer, representation learning, or knowledge distillation. It deserves a serious referee; the experimental design is thoughtful and the result, once budget-controlled, will either hold up or be an instructive near-miss. I'd ask for the equal-budget comparison and repeated seeds before saying the central equality is established.","headline":"Attention transfer is a real finding with a clean causal design, but the headline match to fine-tuning rests on a 200-epoch distillation run versus a 100-epoch baseline; the phenomenon survives, the exact equality does not.","tokens_in":18355,"tokens_out":2456,"would_cite":true,"duration_ms":22500,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training works because of attention maps, not learned features.","keywords":["attention transfer","vision transformers","pre-training","knowledge distillation","self-supervised learning","fine-tuning","attention maps","representation learning"],"falsifier":"Train a ViT-L from scratch on ImageNet-1K while distilling only the attention maps of a MAE-pretrained teacher, reproducing the paper's recipe; if an independent run cannot reach the reported 85.7 in-distribution accuracy, the sufficiency claim fails. A sharper check: if a fixed, input-independent attention pattern also reaches 85.7, then the learned content of the attention maps is not what carries the benefit.","tokens_in":17379,"feed_emoji":"👁️","tokens_out":7458,"duration_ms":67240,"temperature":0.7,"pith_summary":"This paper challenges the usual story that pre-training helps Vision Transformers by giving them useful features. Its claim is that the inter-token attention maps learned during pre-training are typically sufficient: a student that receives only those maps, and learns all features from scratch, can match full weight fine-tuning on ImageNet-1K. The authors show this with two simple methods, Attention Copy and Attention Distillation, and verify it across pretraining methods, model sizes, and a detection task. The finding matters because it separates what pre-training actually contributes—how information flows between patches—from the features themselves, and points to a way to reuse pretrained models without sharing weights. The paper also marks the boundary of the claim: attention transfer loses to fine-tuning under distribution shift and when the pretraining and downstream datasets are far apart.","feed_headline":"Pretrained attention maps alone match fine-tuning accuracy","feed_subtitle":"Copying only inter-token attention from a pretrained ViT recovers all of the fine-tuning gain on ImageNet.","key_machinery":"The key object is the attention map in each multi-head self-attention block, defined as $\\mathrm{softmax}(QK^\\top)$; it is the only computation in a Vision Transformer that exchanges information between tokens, since value projections, MLPs, and normalization act per token. Attention Copy replaces the student's attention maps with frozen teacher maps, fully decoupling inter-token routing from intra-token feature learning. Attention Distillation replaces that hard constraint with a cross-entropy loss that pushes the student's attention maps toward the teacher's, summed over layers and heads, and the teacher is discarded at inference. This decoupling is what lets the experiments attribute the pre-training gain to information flow rather than to the features that flow through it.","core_discovery":"The central discovery is that for a pre-trained Vision Transformer, the attention map matrix $\\mathrm{softmax}(QK^\\top)$—the only operation that routes information between tokens—carries enough of the pre-training benefit to recover the full fine-tuning accuracy. With a MAE-pretrained ViT-L on ImageNet-1K, training from scratch reaches 83.0, full weight fine-tuning reaches 85.7, Attention Copy reaches 85.1, and Attention Distillation reaches 85.7. Attention Distillation also recovers most of the pre-training gain in COCO object detection, and ensembling a distilled student with a fine-tuned teacher improves ImageNet accuracy to 86.3, showing the student is not merely relearning the teacher. The same pattern holds for MoCo-v3, FLIP, and DINO teachers and across ViT-B/L/H sizes. The paper states the limits explicitly: attention transfer is consistently worse than fine-tuning on ImageNet-A, ImageNet-R, ImageNet-Sketch, and ImageNet-V2, and the gap widens when pretraining and downstream datasets differ, so pretrained features do add robustness and distribution-mismatch tolerance beyond what attention maps provide.","pith_inferences":["Inference: attention-map transfer could make cross-size transfer easier than weight fine-tuning, since attention maps are shaped by sequence length rather than embedding width; the paper raises this but does not test it.","Inference: a hybrid that distills attention maps while also transferring a small amount of feature information might recover the out-of-distribution robustness that pure attention transfer lacks.","Inference: the sufficiency result is naturally read as a statement about the optimization landscape—a randomly initialized student with good routing can fit in-distribution tasks as well as a pretrained network—so the mechanism is likely tied to the training dynamics of attention, not to the semantic content of the features.","Inference: if attention maps are the reusable unit, sharing a small set of maps instead of full weights could address settings where weight sharing is undesirable for security or memory reasons."],"forward_implications":["If attention maps are sufficient, the features learned in pre-training are not what fine-tuning needs, at least for in-distribution classification.","Attention Distillation can replace weight fine-tuning with the same inference-time model and no layerwise learning-rate decay, since the student learns its own features from scratch.","Ensembling an attention-distilled student with a fine-tuned teacher gives a concrete accuracy gain (86.3 on ImageNet-1K), because the two models make different errors.","The transfer benefit scales with model size and is not specific to MAE, holding for contrastive (MoCo-v3) and image-text (FLIP) pretraining as well.","Under distribution shift or large pretrain-to-downstream dataset differences, attention transfer underperforms fine-tuning, so pretrained features still carry robustness value."],"supporting_citations":[{"why":"Supplies the MAE pre-trained teacher and the fine-tuning baseline that attention transfer is compared against.","marker":"[22]"},{"why":"Defines the Vision Transformer architecture and the attention mechanism that the transfer methods operate on.","marker":"[12]"},{"why":"Provides the knowledge-distillation framework that Attention Distillation extends and compares against.","marker":"[26]"},{"why":"Supplies a contrastive self-supervised teacher used to test whether attention transfer generalizes beyond MAE.","marker":"[7]"},{"why":"Supplies an image-text contrastive teacher used to test attention transfer across pretraining methods.","marker":"[36]"},{"why":"Provides the CKA representation-similarity measure used to show the student does not relearn the teacher's features.","marker":"[32]"},{"why":"Provides the ViTDet detection pipeline used to test attention transfer on dense prediction.","marker":"[35]"},{"why":"Establishes the effective-robustness baseline that the out-of-distribution comparison targets.","marker":"[14]"},{"why":"Supplies the ImageNet-1K benchmark and dataset that anchor the main accuracy comparisons.","marker":"[10]"}],"fun_headline_variants":["Attention maps alone carry the pretraining boost","Transfer attention, not features, to match fine-tuning","Copy attention patterns to recover full ImageNet gain","Attention transfer: pretraining without feature reuse","Just attention maps: the key to ViT pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 'downstream performance' is in-distribution accuracy on benchmarks similar to the pre-training data; once robustness to distribution shift or transfer to a different dataset is included, the paper's own experiments show attention maps alone do not suffice.","fun_headline_variants_meta":{"raw":{"variants":["Attention maps alone carry the pretraining boost","Transfer attention, not features, to match fine-tuning","Copy attention patterns to recover full ImageNet gain","Attention transfer: pretraining without feature reuse","Just attention maps: the key to ViT pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1797,"prompt_tokens":971,"completion_tokens":826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":754}},"tokens_in":587,"tokens_out":826,"duration_ms":8544,"temperature":1.0,"reasoning_tokens":754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:22:26.807882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a ViT-L from scratch on ImageNet-1K while distilling only the attention maps of a MAE-pretrained teacher, reproducing the paper's recipe; if an independent run cannot reach the reported 85.7 in-distribution accuracy, the sufficiency claim fails. A sharper check: if a fixed, input-independent attention pattern also reaches 85.7, then the learned content of the attention maps is not what carries the benefit.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a contrastive self-supervised teacher used to test whether attention transfer generalizes beyond MAE."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies an image-text contrastive teacher used to test attention transfer across pretraining methods."},{"cited_title":"Kornblith, M","cited_arxiv_id":null,"evidence_quote":"Provides the CKA representation-similarity measure used to show the student does not relearn the teacher's features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ViTDet detection pipeline used to test attention transfer on dense prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the effective-robustness baseline that the out-of-distribution comparison targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet-1K benchmark and dataset that anchor the main accuracy comparisons."}],"review_version":1}