{"id":"d7545ab7-3580-409b-8d24-0895eff077c6","arxiv_id":"2502.07436","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Squeezing-Heads Distillation mixes several teacher attention maps into one per-sample weighted map, enabling knowledge distillation between transformers with different head counts without extra parameters.","lead":"This paper proposes Squeezing-Heads Distillation (SHD), which lets a smaller transformer learn from a larger one even when the two models have different numbers of attention heads, by blending several teacher attention maps into one map and using it as a training target. The authors report small but consistent gains over baseline distillation on image generation, image classification, and language model fine-tuning tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 8's expansion of the SHD objective is algebraically wrong, so the printed α in Eq. 10 does not minimize Eq. 7; the central derivation is unsupported as written.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: Eq. 8 drops or misplaces the term −A_{2i-1}X_{2i-1}, so Eq. 10 does not solve the stated optimization. My independent expansion confirms this: the constant term in the expansion is A_{2i}X_{2i-1} − A_{2i-1}X_{2i-1}, not A_{2i}X_{2i-1} − A_{2i-1}X_{2i}. This is not a cosmetic typo because the mixing weight α is the core of SHD and is used to construct the teacher supervision in Eq. 12. Without a correct derivation, and with no released code to show what formula was actually implemented, the empirical results cannot verify the method as described. The reader's REJECT verdict therefore remains appropriate, though the issue is potentially fixable in a revision if the authors provide a corrected derivation and confirm the experiments used it.","tokens_in":13014,"tokens_out":10107,"duration_ms":90289,"concrete_test":"Independently re-derive Eq. 8 from Eq. 7 and evaluate the printed formula on a scalar counterexample: set A_{2i-1}=1, A_{2i}=0, X_{2i-1}=1, X_{2i}=0. Eq. 7 becomes (α−1)², whose minimizer is α=1, while the printed Eq. 10 gives α=0. Then run both the printed and the corrected formula in the MiniLLM+SHD fine-tuning setup (Table 3) and compare with the reported metrics; if the reported results match the corrected formula, the paper contains a typo that can be fixed, but if they match the printed formula, the published method is not what was evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The derivation of the mixing weight in §4.2 is incorrect. Expanding Eq. 7 term by term gives E(α) = ‖α(A_{2i-1}−A_{2i})(X_{2i-1}+X_{2i}) + (A_{2i}X_{2i-1} − A_{2i-1}X_{2i-1})‖²_F, not the printed N = A_{2i}X_{2i-1} − A_{2i-1}X_{2i}. The printed N replaces the constant term −A_{2i-1}X_{2i-1} with −A_{2i-1}X_{2i}, which actually belongs to the α-dependent part of the expansion. Consequently, Eq. 10 does not minimize the reconstruction error stated in Eq. 7, and the resulting 'squeezed' teacher attention map is not the optimal linear combination the paper claims. Since SHD is defined by this formula and no code or corrected derivation is provided, the central claim that SHD transfers fine-grained attention knowledge without alignment barriers is not supported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Squeezing-Heads Distillation (SHD), a method to distill attention maps from a teacher transformer with multiple heads to a student with fewer heads. SHD defines a per-sample scalar alpha that linearly combines two teacher attention maps into one squeezed map, minimizing a reconstruction error on the multi-head output, and then applies a KL divergence loss between the squeezed teacher map and the student map. The authors claim this removes the head-alignment barrier without extra parameters, and they report improvements across image generation (MDTv2), image classification (DeiT), and LLM pretraining/fine-tuning (BabyLLaMA, MiniLLM). The central derivation of alpha is in Section 4.2, with experiments in Section 5.","tokens_in":13222,"tokens_out":5894,"duration_ms":52879,"significance":"If the derivation were correct, SHD would be a simple, plug-in, parameter-free distillation loss for transformer variants with different head counts, which is a real gap in current KD methods. The paper is strong in scope: it evaluates on both generative and discriminative tasks, compares against feature-distillation baselines, and includes ablations on loss function, attention temperature, hard selection, and constant merging. The method is not circular: alpha is computed deterministically from the teacher's attention and value projections, not fitted to the student's final metric. However, the central algebraic derivation is incorrect as written, and the rank and complexity arguments used to motivate the approach are also flawed. The paper as written does not support the claim that Eq. (10) produces an optimal linear combination for the stated objective, which undermines the theoretical foundation of SHD despite the breadth of experiments.","major_comments":[{"comment":"The expansion in Eq. (8) is algebraically incorrect. Substituting the linear combination into Eq. (7) gives E(alpha) = || alpha (A_{2i-1} - A_{2i})(X_{2i-1}+X_{2i}) + (A_{2i} - A_{2i-1}) X_{2i-1} ||_F^2, not the printed || alpha M + N ||_F^2 with N = A_{2i} X_{2i-1} - A_{2i-1} X_{2i}. The constant term should involve X_{2i-1} only, not X_{2i}. Consequently, the minimizer in Eq. (10) does not minimize the reconstruction error stated in Eq. (7). Because SHD is defined by this formula and the experiments rely on the resulting alpha, the central claim that SHD is an optimal head-compression method is unsupported as written.","section":"Sec. 4.2, Eq. (8)-(10)"},{"comment":"The rank argument is false for softmax attention. The paper claims that because Q and K are N x d with d < N, the attention map A_i = softmax(Q K^T / sqrt(d)) has rank at most d. This would be true for the pre-softmax score matrix Q K^T, but the softmax nonlinearity can increase the rank; a softmax of a rank-d matrix can easily be full rank. Thus the claimed 'rank limitation' does not provide a mathematical basis for head compression, and the motivating observation in Section 3.3 is incorrect.","section":"Sec. 3.3"},{"comment":"The claimed O(N^6) complexity for the exact optimization in Eq. (5) is overstated. The least-squares problem min_A || A (X_{2i-1}+X_{2i}) - (A_{2i-1} X_{2i-1} + A_{2i} X_{2i}) ||_F^2 has a closed-form solution A = C B^T (B B^T)^+ (or equivalently row-wise via the pseudo-inverse of a d x N matrix), where B = X_{2i-1}+X_{2i} is N x d. Forming and solving the normal equations costs O(N d^2 + N^2 d), not O(N^6). The paper's complexity claim appears to ignore the structure of the problem and should be corrected.","section":"Sec. 4.1"},{"comment":"Even after correcting the algebra, there is no guarantee that the unconstrained minimizer of Eq. (7) lies in the interval [0,1] as required by the reparameterization in Eq. (6). The paper only states that 'in practice alpha often falls within [0,1]' but provides no formal derivation, no bounds, and no description of what is done when alpha falls outside this range. If alpha is projected or clamped, the optimality claim no longer holds; if it is not, the convex-combination interpretation of Eq. (6) is violated. This needs clarification and analysis.","section":"Sec. 4.2, Eq. (6)-(10)"}],"minor_comments":[{"comment":"Eq. (11) is introduced as 'we modify Eq.11 to incorporate attention temperature' but no prior equation numbered 11 is defined; the authors likely mean Eq. (1) or the attention formula in Eq. (4). Please renumber or reference correctly.","section":"Sec. 4.3, Eq. (11)"},{"comment":"The notation for alpha is inconsistent: Eq. (7) writes alpha (without subscript), Eq. (8) writes E(alpha_i), and Eq. (10) uses alpha_i. Please choose one convention and apply it consistently.","section":"Sec. 4.2 and throughout"},{"comment":"The phrase 'state-of-the-art results' in the abstract is stronger than the evidence in Tables 2, 3, and 6, where several metrics are unchanged or slightly worse than baselines; please temper the claim to match the actual gains.","section":"Abstract and Sec. 5"},{"comment":"There are numerous typos and formatting errors, including 'Datatset' in the caption of Figure 1, 'maunal' in the hyperparameters section, and 'V oita' in several citations; a careful proofreading pass is needed.","section":"General"},{"comment":"The comparison with 'FD+SC' and 'FD+Projector' would be clearer if the exact feature-difference loss and the projector architecture were specified in the text or appendix, since the efficiency comparison in Table 8 depends on those choices.","section":"Sec. 5.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The reader's verdict is REJECT, and I agree that the algebraic error in Section 4.2 is load-bearing. However, I recommend major_revision rather than reject because the error is localizable and correctable: replacing the wrong N with (A_{2i} - A_{2i-1}) X_{2i-1} restores a valid (though different) alpha, and the experimental pipeline could have been run with the corrected formula. The rank and complexity claims are also fixable by replacing the relevant paragraphs. The main risk is that the reported experiments were run with the printed (incorrect) formula; without code, the editor should ask the authors to confirm which alpha was actually used and to reconcile the derivation. If they cannot provide code or a corrected derivation, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the head-compression idea is genuinely new and the experimental sweep is wider than most KD papers, but the derivation at the center of the method is algebraically wrong, so as written the paper doesn't support its own central claim.\n\nWhat's actually new: SHD merges two teacher attention heads into a single map using a per-sample scalar α, computed as a least-squares-style combination, and then distills that squeezed map into the student with a KL loss. That's distinct from TinyBERT, MobileBERT, and MiniLM, which either require matched head counts or bolt on projectors. The paper also reports consistent gains across image generation, ImageNet classification, and LLM pretraining/fine-tuning, with ablations that rule out simple hard selection and constant merging. That is real evidence the method does something.\n\nThe soft spots are serious. The derivation in §4.2 is wrong. Expanding Eq. 7 correctly gives the constant term as A_{2i}X_{2i-1} − A_{2i-1}X_{2i-1}, not A_{2i}X_{2i-1} − A_{2i-1}X_{2i} as printed in Eq. 8. The printed N is not the constant term; it contains an X_{2i} that actually belongs inside the α-dependent part. So Eq. 10 does not minimize the reconstruction error stated in Eq. 7. Because SHD is defined by that α formula, the central justification for the method collapses as written. Maybe the empirical results survive with a corrected derivation, but that needs to be shown. The rank argument in §3.3 is also false: softmax is nonlinear, so rank(A_i) is not bounded by d in general. And the O(N^6) complexity claim for the exact solution looks overstated; it's not the main thread, but it adds to the impression that the theory section wasn't checked carefully. No code and no error bars in the reported numbers, which makes verification harder.\n\nAll that said, the idea is worthwhile and the experiments are broad enough that a serious referee should look at it. The errors are fixable in principle. If a corrected derivation and code appear, this could be a solid contribution. Right now, I wouldn't cite it in its current form, but I'd take a look at a revision.","headline":"The head-squeezing idea is genuinely novel and the experimental sweep is broad, but the derivation of α in Eq. 10 is algebraically wrong, so as written the paper doesn't support its central claim.","tokens_in":13755,"tokens_out":3810,"would_cite":false,"duration_ms":28777,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Squeezing-Heads Distillation lets a student transformer learn from a teacher with any number of attention heads, using a per-sample linear mixture of the teacher's attention maps and a KL loss, with no extra parameters.","keywords":["knowledge distillation","multi-head attention","head alignment","linear approximation","attention map compression","transformer","generative models"],"falsifier":"Recompute $\\alpha_i$ by taking the derivative of the full expression $E(\\alpha_i) = \\|(\\alpha_i A_{2i-1} + (1-\\alpha_i)A_{2i})(X_{2i-1}+X_{2i}) - (A_{2i-1}X_{2i-1} + A_{2i}X_{2i})\\|_F^2$ without dropping the term, and compare the resulting minimizer to Eq. 10 on real attention maps from a trained transformer. If the two differ measurably, then Eq. 10 is not the minimizer, and the reported distillation gains must be independent of the claimed optimality of the compression.","tokens_in":12775,"feed_emoji":"🧠","tokens_out":2628,"duration_ms":23072,"temperature":0.7,"pith_summary":"This paper proposes Squeezing-Heads Distillation (SHD), a method for knowledge distillation that removes the need for teacher and student transformers to have the same number of attention heads. Instead of aligning features with projectors or forcing matching head counts, SHD compresses several teacher attention maps into one per-sample linear combination, then supervises a student head with a KL divergence loss. The paper argues this yields fine-grained attention knowledge transfer at the native cost of attention computation, across image generation, image classification, and LLM pretraining and fine-tuning. If correct, SHD is a plug-in distillation loss that makes head-count mismatch a non-issue for transformer distillation.","feed_headline":"Attention-head squeeze lets small models learn from big teachers","feed_subtitle":"A KL loss on compressed teacher attention maps beats projectors and logit KD, with no extra parameters.","key_machinery":"The load-bearing object is the per-sample scalar $\\alpha_i$ computed by Eq. 10 as $\\alpha_i = -\\langle M, N \\rangle / \\|M\\|_F^2$, where $M = (A_{2i-1}-A_{2i})(X_{2i-1}+X_{2i})$ and $N = A_{2i}X_{2i-1} - A_{2i-1}X_{2i}$, with $X_i = V W_i^V W_i^O$ the per-head transformed values. This $\\alpha_i$ is meant to minimize the reconstruction error between the combined two-head output and the squeezed one-head output; the same procedure can merge any number of heads into fewer, aligning teacher and student head counts without architectural changes.","core_discovery":"The central claim is that multi-head attention maps of a teacher can be compressed into a single attention map by a scalar convex combination, $\\tilde{A}_i = \\alpha_i A_{2i-1} + (1-\\alpha_i) A_{2i}$, with $\\alpha_i$ chosen per sample and per head-pair by minimizing the squared reconstruction error of the combined attention output. This compressed map is then used as a soft target for a student head through a KL divergence loss with an attention temperature. The paper reports that this procedure outperforms logit-based and feature-distillation baselines on MDTv2 image generation, DeiT classification, BabyLLaMA pretraining, and MiniLLM fine-tuning, while adding no parameters and running in $O(N^2)$ time.","pith_inferences":["If SHD's per-sample $\\alpha$ truly captures the relative importance of teacher heads, it could serve as a cheap proxy for head importance, potentially informing head pruning or merging decisions at inference time.","The same linear-compression idea might extend to other multi-head structures, such as grouped-query attention or cross-attention in encoder-decoder models, where head-count mismatches also block direct attention distillation.","The reported sensitivity of results to the attention temperature suggests a testable extension: adaptive temperature schedules might improve transfer when teacher and student capacity gaps vary by layer.","SHD's success over feature-correlation methods implies that attention-map distributions contain transferable knowledge that is lost when only output or relation-level signals are used; this could motivate further study of which attention statistics matter most for downstream generalization."],"forward_implications":["Transformer distillation no longer requires matching head counts, so compact models with fewer heads can be trained directly from larger teachers.","SHD can be plugged into existing logit-based distillation frameworks, as an additive loss with a strength weight $\\beta$.","The method scales to long sequences because its per-step cost is $O(N^2)$, matching the attention computation itself.","Applying SHD to language pretraining (BabyLLaMA) and fine-tuning (MiniLLM) improves downstream SuperGLUE and Rouge-L scores over baselines.","On ImageNet classification, SHD alone improves DeiT-Tiny accuracy by 0.95% and adds 0.42% on top of a strong ViTKD+NKD baseline."],"supporting_citations":[{"why":"Provides the per-head output projection decomposition (Eq. 4) that SHD uses to rewrite multi-head attention as a sum over heads.","marker":"[Elhage et al., 2021]"},{"why":"TinyBERT is the example of attention-map distillation that requires matching head counts, motivating the alignment barrier SHD removes.","marker":"[Jiao et al., 2019a]"},{"why":"MobileBERT is cited as a method that needs specially engineered teacher and student to keep head counts equal, defining the problem SHD avoids.","marker":"[Sun et al., 2020]"},{"why":"MiniLLM is the logit-distillation baseline in the LLM fine-tuning experiments; SHD is combined with it and compared against it.","marker":"[Gu et al., 2024]"},{"why":"ViTKD is the feature-distillation baseline in the ImageNet classification experiments that SHD is combined with and compared against.","marker":"[Yang et al., 2022]"},{"why":"NKD provides the normalized-logit baseline that is stacked with ViTKD and then augmented with SHD.","marker":"[Yang et al., 2023]"},{"why":"DeiT introduced the distillation token and attention-based supervision, establishing the teacher-student setup SHD extends.","marker":"[Touvron et al., 2021]"}],"fun_headline_variants":["Squeeze teacher attention heads for parameter-free distillation","No projectors: compress teacher heads for flexible KD","Compress teacher attention heads in linear time for KD","Attention head squeeze: distillation without alignment barriers","Parameter-free attention compression for flexible transformer KD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The formula for the mixing weight $\\alpha_i$ in Eq. 10 is derived from an expansion in Eq. 8 that silently drops the term $-A_{2i-1}X_{2i-1}$, so the printed $\\alpha_i$ does not actually minimize the reconstruction error defined in Eq. 7; the whole method rests on this $\\alpha_i$ being a faithful low-error compression of the teacher's multi-head attention output.","fun_headline_variants_meta":{"raw":{"variants":["Squeeze teacher attention heads for parameter-free distillation","No projectors: compress teacher heads for flexible KD","Compress teacher attention heads in linear time for KD","Attention head squeeze: distillation without alignment barriers","Parameter-free attention compression for flexible transformer KD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001257,"raw_usage":{"total_tokens":5139,"prompt_tokens":922,"completion_tokens":4217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4145}},"tokens_in":538,"tokens_out":4217,"duration_ms":26341,"temperature":1.0,"reasoning_tokens":4145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:46:17.084755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute $\\alpha_i$ by taking the derivative of the full expression $E(\\alpha_i) = \\|(\\alpha_i A_{2i-1} + (1-\\alpha_i)A_{2i})(X_{2i-1}+X_{2i}) - (A_{2i-1}X_{2i-1} + A_{2i}X_{2i})\\|_F^2$ without dropping the term, and compare the resulting minimizer to Eq. 10 on real attention maps from a trained transformer. If the two differ measurably, then Eq. 10 is not the minimizer, and the reported distillation gains must be independent of the claimed optimality of the compression.","supporting_citations":[],"review_version":1}