{"id":"7db49fc1-ae83-484a-931f-7e6b2499e882","arxiv_id":"2608.07921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Spectral outliers in attention weight matrices, identified by the Marchenko-Pastur threshold, contain a dominant share of functionally important learned structure across 11 transformers.","lead":"Applying random matrix theory to transformer attention weights, this paper separates each projection matrix into random-like noise and a small set of spectral outliers, and it shows that zeroing those outliers in Mistral-7B collapses benchmark accuracy. The result suggests outlier singular values carry most learned structure, which could guide pruning and efficient fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central causal claim is not established: outlier removal is run on Mistral-7B while the bulk control is run on other models, and the paper's own LLaMA-3 V-bulk ablation shows below-threshold singular values can be the most critical component.","rationale":"I read the paper in good faith and credit its useful observational contributions: the entry-level heatmaps, persistent residual-stream channels, and the GQA/MHA contrast are interesting and likely reproducible given the released code. The central causal claim, however, is the loading-bearing part of the paper, and the evidence for it is weaker than the abstract implies. The reader's verdict of CONDITIONAL is appropriate. My concern overlaps with the reader's weakest assumption—the MP threshold cleanly separates signal from noise—but I would put the emphasis on experimental control: the headline comparison mixes models and matches count rather than spectral energy, while the paper's own V-bulk ablation on LLaMA-3-8B demonstrates that below-threshold singular values can be functionally critical. This is not an attack on the authors' honesty; the limitations section is candid about the approximation. But the candidness also means the central claim, as stated, is not supported for all components. A same-model, energy-matched bulk ablation, plus explicit treatment of the GQA V regime, would resolve whether the collapse from outlier removal is due to the specific spectral locations or simply due to removing a large share of spectral mass. Until then, the paper should be accepted with re-scoped claims, not with the headline generalization as stated.","tokens_in":10048,"tokens_out":6683,"duration_ms":80777,"concrete_test":"Run the full ablation suite on Mistral-7B, the model used for the headline result: (1) zero all MP-identified outliers as in Table VI; (2) zero a count-matched random subset of below-threshold singular values on the same model; (3) zero an energy-matched subset of below-threshold singular values, chosen as the smallest set whose summed squared singular values equals the removed outlier energy; and (4) repeat the V-only count-matched bulk ablation on Mistral-7B as was done for LLaMA-3-8B, with at least 5 random seeds per condition. If any bulk-control condition, especially the V-only condition, also drives MMLU or HellaSwag close to chance, then the claim that spectral outliers specifically encode a dominant learned structure fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MP-identified spectral outliers encode a dominant learned structure—requires that outlier status track functional importance, with below-threshold singular values serving as a low-importance control. The evidence does not support this. The headline comparison in Section IV is cross-model: zeroing all MP outliers is done on Mistral-7B, while count-matched bulk removal is done only on LLaMA-1-7B and LLaMA-2-7B. No same-model bulk ablation is reported for Mistral-7B, so the observed collapse cannot be attributed specifically to 'outlier' status. Moreover, the control is count-matched but not energy-matched: in Mistral-7B the removed outliers carry roughly 87.5% of Q, 74.7% of K, and 84.6% of O spectral energy (Table II), whereas a random subset of below-threshold singular values removes low-energy tail components. The collapse may therefore reflect deletion of spectral energy rather than deletion of the particular learned structure identified by the MP threshold. The paper's own LLaMA-3-8B V ablation (Table VI, Section IV.C) is a direct internal counterexample to the bulk-as-noise assumption: zeroing only 7,013 randomly selected below-threshold V singular values drops MMLU from 0.660 to 0.254, while zeroing 98,866 V entry outliers leaves MMLU at 0.612. Section VII acknowledges the MP split is approximate and that V under GQA lacks a clean signal/noise separation, but this is not a negligible edge case: it is one of the four core projections in a model central to the paper's comparisons. The claim should be re-scoped to components where the MP separation is empirically validated, and the headline causal comparison needs a same-model, energy-matched bulk control before the 'dominant learned structure' conclusion is warranted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript applies a Marchenko–Pastur (MP) upper-edge threshold to the squared singular values of Transformer attention projection matrices, labeling super-threshold singular values as spectral outliers and below-threshold values as bulk. After documenting outlier counts, energy ratios, entry-level outlier maps, and cross-layer persistent residual-stream dimensions across 11 models, the authors run zeroing ablations: removing all MP outliers in Mistral-7B collapses HellaSwag, MMLU, and PIQA to near chance, while count-matched random bulk removals in LLaMA-1/2 show smaller degradation. The paper concludes that spectral outliers encode a dominant component of learned structure, identifies recurring patterns (Q dominance, V sparsity under GQA, row/column band structure, persistent residual-stream channels), and discusses implications for fine-tuning and pruning.","tokens_in":10396,"tokens_out":9349,"duration_ms":104079,"significance":"The paper is potentially valuable as a weight-space diagnostic: the code is released, the model coverage is broad (six architecture families, 11 checkpoints), and the observational findings—Q-dense/V-sparse spectra under GQA, row-bands in Q, column-bands in O, persistent residual-stream dimensions—are concrete and reproducible. The causal ablation idea is a good way to test the decomposition. However, the central causal claim as stated is not currently supported: the key comparison is confounded by model identity and by spectral energy, and the paper's own V-bulk ablation on LLaMA-3 contradicts the low-importance-bulk assumption. If the authors add a same-model, energy-matched control and restrict the dominance claim to the components and architectures for which it holds, the paper could make a sound contribution.","major_comments":[{"comment":"The headline comparison mixes models and is not energy-matched. The all-outlier ablation is reported only for Mistral-7B (N=112,458), while the random-bulk controls are reported for LLaMA-1-7B and LLaMA-2-7B; Table VI contains no Mistral random-bulk row. The controls are count-matched only in the loose sense of using each model's own outlier count (with f=0.75; N_zero differs across models), and they are not energy-matched. From Table II, the zeroed Mistral outliers carry 87.5% of Q, 74.7% of K, and 84.6% of O spectral energy, whereas random draws from the bulk select low-energy singular values. The observed collapse can therefore be attributed to removing a large fraction of spectral mass rather than specifically to MP-identified 'learned structure.' Please add a same-model Mistral ablation that removes the same number of bulk singular values with matched total spectral energy, and/or report per-unit-energy damage.","section":"IV.B; Table VI"},{"comment":"The below-threshold bulk is not a low-importance control in the paper's own data. The LLaMA-3-8B V random-bulk ablation zeros only 7,013 below-threshold V singular values and drops MMLU from 0.660 to 0.254 and HellaSwag from 0.821 to 0.649, damage comparable to the Mistral all-outlier collapse. This is a core projection of a GQA model, not a negligible edge case. Section VII correctly acknowledges that GQA V lacks a clean signal/noise separation, but the abstract, Section IV.B, and Pattern 5 present the dominance claim without this exception. The manuscript needs to either restrict the claim to Q/K/O in architectures where bulk ablation is benign, or show quantitatively why the V-bulk catastrophe is compatible with the headline 'spectral outliers encode a dominant component of learned structure.'","section":"IV.C; Table VI; Section VII"},{"comment":"The causal design does not isolate the MP threshold from singular-value magnitude. Since the signal/noise split is defined by s_i^2 > λ_+, a comparison of 'large singular values removed' versus 'small singular values removed' may simply reflect a monotone relationship between singular value magnitude and functional importance; a random bulk draw selects low-energy components by construction. The MP threshold would then be a useful ranking heuristic but not a validated discovery of 'learned structure' vs 'noise.' To support the stronger conclusion, compare outlier removal against removal of the largest below-threshold bulk singular values matched on count and on total energy, and add a shuffled-weight or randomly initialized control (the latter is already listed as future work in Section VII).","section":"II.A, II.D, Eq. (1)"}],"minor_comments":[{"comment":"The median-based noise estimator σ̂² = median(s_i²)/(1+γ) is an approximation to the median of the MP law; the Limitation section notes this, but the Method section should state it at the point of definition and cite the exact median correction for finite γ.","section":"II.A / Eq. (1)"},{"comment":"Calling the random-bulk MMLU degradation 'smaller but non-negligible' understates LLaMA-1's f=0.75 row, where MMLU drops from 0.352 to 0.292, leaving only 4.2 points above the 0.250 chance baseline; please report relative degradation and discuss this in the same paragraph as the 'preserves much of the model's capability' sentence.","section":"IV.B / Table VI"},{"comment":"The random-bulk experiments appear to use a single draw with no seed; report at least 5 random draws with mean and standard deviation, as Section VII already says is needed.","section":"II.D / Table VI"},{"comment":"The entry-level threshold |w_ij| > 4σ_W is applied without a null-model baseline; since isolated extremes are expected in large matrices, include a shuffled-weight or random-Gaussian control to support the claim that observed row/column bands are learned, not threshold artifacts.","section":"II.B / Section VII"},{"comment":"Table II leaves Q/K/V blank for Phi-3-mini and Table III omits two models; add a note in each caption or a supplementary table with the full data so the cross-model comparisons are not obscured by page-limit omissions.","section":"Table II / III"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is potentially salvageable but the abstract overclaims relative to the evidence. The most serious problem is the Mistral-vs-LLaMA comparison and the LLaMA-3 V-bulk counterexample; these need to be addressed before publication. The observational sections are solid; I would suggest the authors consider reframing the contribution around 'a diagnostic decomposition with per-component caveats' rather than a universal dominance claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2608.07921. The observational content is genuinely useful: across 11 models the authors find consistent spatial structure in attention weights—row-bands in Q, column-bands in O, a sharp V-projection sparsity under grouped-query attention, and persistent residual-stream dimensions in K and O. That part is well supported and worth reading. The causal claim attached to it—that MP-identified spectral outliers encode a 'dominant learned structure'—is not actually established by the experiments as run.\n\nThe headline comparison mixes models. The outlier-removal collapse is shown for Mistral-7B (zeroing 112,458 outliers takes HellaSwag from .814 to .256, MMLU from .627 to .269, PIQA from .823 to .508), while the count-matched bulk control is only run on LLaMA-1 and LLaMA-2. There is no same-model bulk ablation on Mistral. Worse, the control is count-matched, not energy-matched: Table II shows the Mistral outliers carry roughly 85% of spectral energy in Q and O, so the collapse could simply reflect deleting most of the matrix's energy, not deleting anything special about the outliers.\n\nThe paper's own data contains a sharper counterexample. In Table VI, LLaMA-3-8B: zeroing only 7,013 randomly selected below-threshold V singular values drops MMLU from .660 to .254, while zeroing 98,866 V entry outliers leaves MMLU at .612. That is the opposite of the 'outliers are dominant, bulk is secondary' story. The V-projection under GQA is spectrally diffuse, and the paper's limitation section admits the MP split is approximate—but then the conclusion should be rescaled to components where the split actually holds, not stated as a general pattern across all projections.\n\nWhat the paper does well beyond the observations: the entry-level heatmap methodology is simple but effective, the cross-layer residual-stream analysis is a real addition, and the ablation table, despite the flawed headline comparison, is informative. The authors are transparent about the model coverage gaps (Phi-3 fused QKV, omitted tables) and about the random-bulk control being a diagnostic rather than a full robustness study. Code is released.\n\nBottom line: this is a solid observational paper with an overstated causal conclusion. A serious referee should engage it; with a same-model, energy-matched bulk baseline and a per-component rescoping of the claim, it could be a genuinely useful contribution. For my own work, I'd cite it for the GQA V-sparsity and persistent-channel observations, not for the 'dominant structure' claim. Bring it to reading group—there's a good discussion here about what random matrix theory can and cannot tell you about trained weights.","headline":"Useful observational mapping of attention-weight spectral structure, but the headline causal claim is not supported by the experiments as run; the paper's own V-bulk ablation undercuts it.","tokens_in":10970,"tokens_out":2140,"would_cite":true,"duration_ms":24288,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the functionally important learned structure in transformer attention weights is concentrated in a few spectral outliers above the Marchenko-Pastur threshold, so zeroing them collapses model competence to chance.","keywords":["Marchenko-Pastur","random matrix theory","transformer interpretability","attention mechanisms","spectral outliers","model pruning","LoRA","large language models"],"falsifier":"A single concrete observation would settle the claim: if there exists a count-matched subset of below-threshold singular values in some attention projection of Mistral-7B whose zeroing collapses HellaSwag or MMLU as severely as zeroing the spectral outliers does, then the outliers are not uniquely dominant. The paper's own LLaMA-3-8B V projection already approaches this, with 7,013 bulk singular values driving MMLU from 0.660 to 0.254; a systematic search across projections for similarly catastrophic bulk subsets would either confirm the dominant-outlier claim or falsify it.","tokens_in":9857,"feed_emoji":"📉","tokens_out":4698,"duration_ms":45086,"temperature":0.7,"pith_summary":"This paper tries to establish that the functionally important learned structure in transformer attention weights is concentrated in a small number of spectral outliers—singular values that stand above the Marchenko-Pastur threshold expected from a random matrix. Zeroing all such outliers in Mistral-7B drops HellaSwag from 0.814 to 0.256, MMLU from 0.627 to 0.269, and PIQA from 0.823 to 0.508, close to random guessing, while zeroing a count-matched set of below-threshold singular values does far less damage. A sympathetic reader would care because this offers a principled, parameter-free way to decide which parts of a transformer's weights carry knowledge and which are near-noise, with direct implications for pruning and low-rank fine-tuning. The paper also documents systematic patterns across 11 models: Q projections carry the most outliers, V projections under grouped-query attention largely lose clean signal, and specific residual-stream dimensions persist as outlier \"highways\" across layers.","feed_headline":"Zeroing spectral outliers collapses Mistral-7B to chance","feed_subtitle":"A Marchenko-Pastur split of attention weights shows learned structure lives in a few singular values, not the bulk.","key_machinery":"The central object is the Marchenko-Pastur upper edge estimate λ+ = σ̂²(1+√γ)², with noise variance estimated from the median squared singular value and aspect ratio γ = max(m,n)/min(m,n). Singular values squared above λ+ are classified as spectral outliers (signal); those below form the bulk. The argument is carried by a causal ablation design: zeroing outlier singular values versus zeroing a count-matched random subset of bulk singular values, evaluated with HellaSwag, MMLU, and PIQA, plus entry-level outlier heatmaps and cross-layer residual-stream alignment to show the outliers are spatially organized.","core_discovery":"On the paper's own terms, the central discovery is that attention projection matrices in pretrained transformers can be separated by the Marchenko-Pastur law into a random-like bulk and a sparse set of spectral outliers, and that this split is causally meaningful: deleting the outliers destroys the model's competence, while deleting an equal number of bulk components is comparatively benign. The ablation zeroing roughly 112,000 outlier singular values across all Q, K, V, O projections of Mistral-7B drives three benchmarks to near chance, and per-component ablations on LLaMA-3-8B give a criticality order K > Q ≫ V. A caution the paper itself flags: below-threshold components are not pure noise, because MMLU drops 6–13 points when bulk singular values are removed, and in one LLaMA-3 V-projection ablation zeroing only 7,013 bulk singular values collapses MMLU to 0.254.","pith_inferences":["A testable extension would be to run the same zero-outlier ablation across many more tasks and models; if the collapse is universal, spectral outlier fraction could become a cheap proxy for a model's usable capacity.","The paper's own V-bulk result suggests the MP threshold may systematically misclassify spectrally diffuse but real structure in V under GQA; a heavy-tailed RMT model, as the authors note, might fix this and is a concrete next step.","The cross-component persistent dimensions (e.g., LLaMA-2 dimension 2533 appearing in all four projections) hint at a small set of privileged residual-stream channels; if confirmed by activation-based analysis, these could serve as interpretable 'bus' directions that mediate cross-layer communication.","The entry-level row/column band structure in Q and O suggests that attention heads are not interchangeable: a few heads dominate outlier structure, so head-level pruning based on spectral outlier density may outperform magnitude pruning."],"forward_implications":["If the claim is right, a substantial fraction of a transformer's attention weights are near-random and can be pruned or quantized aggressively without touching the outlier subspace.","Parameter-efficient fine-tuning methods like LoRA should allocate rank budget to outlier-rich projections (Q, and K under GQA) rather than uniformly across layers and matrices.","The persistent residual-stream \"highway\" dimensions in K and O are natural targets for structured pruning or for identifying which directions carry information across layers.","The K > Q ≫ V criticality ordering, if it holds more broadly, means value projections deserve less adaptation budget than query and key projections.","The V-bulk catastrophe under GQA warns that uniform singular-value thresholding across projections is unsafe; each projection type has its own signal/noise regime."],"supporting_citations":[{"why":"Supplies the Marchenko-Pastur law used to set the threshold splitting bulk from outliers.","marker":"[5]"},{"why":"Documents heavy-tailed spectra in trained DNNs, the caveat that makes the MP split approximate.","marker":"[6]"},{"why":"Mistral-7B is the main model whose ablation collapses to chance.","marker":"[14]"},{"why":"LLaMA-3-8B is the model for per-component criticality and the V-bulk catastrophe.","marker":"[13]"},{"why":"GQA is the attention variant whose V projections show the sparse/diffuse regime.","marker":"[17]"},{"why":"HellaSwag is one of the zero-shot benchmarks used to measure ablation damage.","marker":"[18]"},{"why":"MMLU is the benchmark most sensitive to bulk removal and the one that collapses in the V-bulk ablation.","marker":"[19]"},{"why":"PIQA is the third benchmark used in the causal validation.","marker":"[20]"},{"why":"The evaluation harness runs the zero-shot evaluations.","marker":"[21]"}],"fun_headline_variants":["Spectral outliers carry dominant learned structure in attention","Zeroing 112k spectral outliers drops Mistral-7B to chance","Marchenko-Pastur split: outliers drive model, bulk is backup","Attention's learned structure lives in spectral outliers, not bulk","Spectral outliers are the keys: delete them, model fails"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Marchenko-Pastur threshold, computed with a median-based noise estimator, cleanly separates genuine learned signal from random-like bulk, so singular values below the threshold are near-noise and safe to use as a control—an assumption the paper explicitly calls approximate and that its own LLaMA-3 V-bulk ablation shows can fail.","fun_headline_variants_meta":{"raw":{"variants":["Spectral outliers carry dominant learned structure in attention","Zeroing 112k spectral outliers drops Mistral-7B to chance","Marchenko-Pastur split: outliers drive model, bulk is backup","Attention's learned structure lives in spectral outliers, not bulk","Spectral outliers are the keys: delete them, model fails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1786,"prompt_tokens":907,"completion_tokens":879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":792}},"tokens_in":523,"tokens_out":879,"duration_ms":9941,"temperature":1.0,"reasoning_tokens":792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:40:28.460752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single concrete observation would settle the claim: if there exists a count-matched subset of below-threshold singular values in some attention projection of Mistral-7B whose zeroing collapses HellaSwag or MMLU as severely as zeroing the spectral outliers does, then the outliers are not uniquely dominant. The paper's own LLaMA-3-8B V projection already approaches this, with 7,013 bulk singular values driving MMLU from 0.660 to 0.254; a systematic search across projections for similarly catastrophic bulk subsets would either confirm the dominant-outlier claim or falsify it.","supporting_citations":[{"cited_title":"GQA: Training generalized multi-query transformer models from multi-head checkpoints,","cited_arxiv_id":null,"evidence_quote":"GQA is the attention variant whose V projections show the sparse/diffuse regime."},{"cited_title":"HellaSwag: Can a machine really finish your sentence?,","cited_arxiv_id":null,"evidence_quote":"HellaSwag is one of the zero-shot benchmarks used to measure ablation damage."},{"cited_title":"Measuring massive multitask language understanding,","cited_arxiv_id":null,"evidence_quote":"MMLU is the benchmark most sensitive to bulk removal and the one that collapses in the V-bulk ablation."},{"cited_title":"PIQA: Reasoning about physical commonsense in natural language,","cited_arxiv_id":null,"evidence_quote":"PIQA is the third benchmark used in the causal validation."}],"review_version":1}