{"id":"13c5d7e9-9519-40b6-9048-0e6ac4d1d7d1","arxiv_id":"2507.09394","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Applying Marchenko-Pastur spectral diagnostics to LLaMA-130M variants, the paper reports that sharing a single rotary sub-vector across heads in multi-head latent attention suppresses spectral outlier spikes, while standard MHA and pre-RoPE compression develop mid-layer rank collapse.","lead":"This paper trains three small language model variants and uses random matrix theory to look at the eigenvalue patterns of their attention weights. It claims that a design choice in DeepSeek-style latent attention, sharing one rotary embedding across heads, prevents spikes that shrink the model's effective capacity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLA-PreRoPE outlier count (60–65) and stable rank (~45) violate the rank≤32 bound of the Gram matrix defined in Eq. (1); the spectral comparison is not computing what it claims.","rationale":"I read the paper's goal in good faith: if sharing a rotary sub-vector across heads suppresses spectral outlier formation, that would be a useful architectural insight. The load-bearing premise is that the quantities in Figs. 1–3 are derived from G=W_Q W_K^T/d_in as defined in Eq. (1), with d_in=32 for both MLA variants. That premise fails for MLA-PreRoPE under the paper's own reported values. A 768×32 times 32×768 product has rank ≤32, so it cannot have 60–65 eigenvalues above any threshold, and its stable rank cannot exceed 32. The reported numbers are not merely surprising; they are excluded by the stated construction. This is an internally falsified diagnostic, not a disagreement with an external prior. Consequently, the conclusion that 'only the decoupled variant prevents this cascade' is unsupported by the spectral evidence. The reader's REJECT verdict is appropriate; I would keep it unchanged. The aspect-ratio confound (γ=24 for MLA-PreRoPE versus γ=12 for MLA-Decoupled) is also real and would need a matched-dimension control even after the logging is fixed, but the rank-bound violation is the more decisive, directly checkable problem.","tokens_in":7822,"tokens_out":7941,"duration_ms":93402,"concrete_test":"Load or retrain the MLA-PreRoPE checkpoint, take the up-projection matrices W_Q, W_K ∈ R^{768×32}, form G=(1/32)W_Q W_K^T, and diagonalize it. Report (i) the number of nonzero eigenvalues and how many exceed λ_+=(1+√24)^2≈34.8, and (ii) r_+=Σλ_i/λ_1. Both must be ≤32. If they are ≤32, attempt to reproduce the reported 60–65 outlier count and r_+≈45 from these values; if they do not appear, the logger is not evaluating Eq. (1), and Figures 1–3 must be regenerated after fixing the discrepancy. As a control, repeat with MLA-Decoupled (m=384, d_in=32, λ_+=(1+√12)^2≈19.9) under the identical procedure and check that zero outliers and r_+≤32 are obtained.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that only decoupled MLA prevents spectral outlier formation—rests on the MP diagnostics in Figs. 1–3. Under §3, MLA-PreRoPE has W_Q, W_K ∈ R^{768×32} (m=768, d_in=32), so G=(1/32)W_Q W_K^T ∈ R^{768×768} has rank at most 32. Hence it has at most 32 nonzero singular/eigenvalues, and its stable rank r_+ = Σλ_i/λ_1 is at most 32. Yet §4 and Figs. 1–2 report 60–65 outliers per layer and stable rank ≈45 for MLA-PreRoPE, with heatmaps reaching ~90. These values are mathematically impossible for the matrix defined in Eq. (1). Either the logged object is not G (e.g., per-head calculations counted separately, a different d_in, or inclusion of the shared down-projection), or the outlier/stable-rank computations are erroneous. Because this impossible trajectory is used to conclude that MLA-PreRoPE behaves like MHA and unlike MLA-Decoupled, the comparison cannot support the causal attribution to rotary-vector sharing. The perplexity table is internally coherent, but it is not the evidence on which the spectral-suppression claim rests. This flaw is internal to the paper's own definitions and does not require an alternative model to demonstrate; it must be corrected before the central claim can be evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a lightweight random-matrix-theory diagnostic suite (MP-Gap, outlier count and energy, MP-Soft rank, stable rank) applied to the W_Q W_K^T cross-Gram matrix during LLaMA-130M pretraining. Three attention variants are compared: standard MHA, MLA with rotary embeddings applied before compression (MLA-PreRoPE), and MLA with a shared rotary sub-vector across heads (MLA-Decoupled). Based on spectral diagnostics of this Gram matrix, the paper claims that only the decoupled variant suppresses spectral outlier formation and maintains stable rank, attributing this to rotary-vector sharing. A final perplexity table reports that balanced decoupled MLA matches MHA perplexity, while extreme RoPE budgets or no positional encoding degrade performance.","tokens_in":8127,"tokens_out":6395,"duration_ms":71055,"significance":"If the central empirical claim were supported, the paper would offer a practically relevant design heuristic: sharing rotary sub-vectors across heads in MLA avoids mid-layer spectral spikes and rank collapse, and the proposed diagnostics are lightweight enough for routine monitoring. The experiments are nontrivial in scale (2.2B tokens, 12 layers) and the perplexity comparison is internally coherent. However, the spectral evidence as presented is internally inconsistent with the paper's own definition of the analyzed matrix, and the cross-architecture comparison is confounded by different aspect ratios. The causal attribution to rotary sharing is therefore not currently established; the significance is conditional on a corrected and properly controlled analysis.","major_comments":[{"comment":"The reported spectral values for MLA-PreRoPE are impossible for the matrix defined in Eq. (1). With d_in = 32 and m = H·d_k = 768, G = (1/d_in) W_Q W_K^T is a 768×768 matrix of rank at most 32, so it has at most 32 nonzero singular values and its stable rank cannot exceed 32. Yet Figure 1(b) reports 60-65 outliers per layer, Figure 2(b) reports a stable rank of about 45 for MLA-PreRoPE, and Figure 3(f) shows stable-rank heatmap values up to roughly 90. Either the logged object is not the G of Eq. (1) (for example, per-head Gram matrices or a different d_in are used), or the outlier-count and stable-rank computations are erroneous. Because this impossible trajectory is the basis for grouping MLA-PreRoPE with MHA and contrasting it with MLA-Decoupled, the central comparison cannot be evaluated until this is resolved.","section":"Sec. 3, Eq. (1) and Sec. 4, Figs. 1-3"},{"comment":"The cross-architecture comparison is not controlled for aspect ratio. Equation (2) gives lambda_+ = (1 + sqrt(gamma))^2, with gamma = m/d_in: MHA has gamma = 1 and lambda_+ = 4, MLA-PreRoPE has gamma = 24 and lambda_+ ≈ 34.8, and MLA-Decoupled has gamma = 12 and lambda_+ ≈ 19.9. A near-zero MP-Gap for the decoupled variant is therefore partly predetermined by its much larger lambda_+, and the outlier-count and outlier-energy metrics are not directly comparable across the three variants. No experiment isolates the rotary-sharing factor at matched gamma and row dimension, so the headline claim that only decoupled MLA suppresses outliers is confounded by the very different null thresholds.","section":"Sec. 4, Figs. 1-3 and Eq. (2)"},{"comment":"The text and figures are mutually inconsistent about stable rank. The text states that 'Decoupled MLA consistently sustains >60% normalized rank across all layers and training steps,' while Figure 2(b) shows MLA-Decoupled stable rank collapsing to about 5 and the heatmap in Figure 3(e) shows values between 2 and 16. The text also says MLA-PreRoPE retains the highest capacity at about 45, but Figure 3(f) shows values up to about 90. The reader cannot tell whether the reported numbers are raw stable rank, normalized stable rank, or something else, and the discrepancy undermines the capacity argument.","section":"Sec. 4, Figs. 2-3 and text"},{"comment":"The normalization claim that dividing by d_in sets the expected entry variance of G to one is not established for the cross-Gram construction. If W_Q and W_K have i.i.d. entries with variance sigma^2, then Var(G_ij) = sigma^4 / d_in, which equals 1 only under an unusual scaling (sigma^2 = sqrt(d_in)); the paper does not specify the initial weight scaling. In addition, Eq. (2) is the standard Marchenko-Pastur edge for a sample covariance matrix (1/d_in) X X^T with X having i.i.d. entries, but G = (1/d_in) W_Q W_K^T is a product of two independent random matrices, and its singular-value spectrum is not automatically governed by Eq. (2). The theoretical basis for the quoted lambda_+ needs a derivation or a reference to the appropriate product-ensemble result.","section":"Sec. 3, Eqs. (1)-(2) and Table 1"}],"minor_comments":[{"comment":"The training budget is arithmetically inconsistent: 20,000 steps with batch size 512 and context length 256 gives about 2.6B tokens, not the stated 2.2B tokens; if the dataset is 2.2B tokens, the number of steps should be about 16,800.","section":"Sec. 2"},{"comment":"The violin-plot description says the MHA/Pre-RoPE distribution has a 'long tail extending to ~0.60'; since the distribution is centered at ~0.75, this should be described as a lower tail, not a long tail without qualification.","section":"Fig. 4"},{"comment":"The heatmap colorbars in Figure 3 lack labels; the reader must infer the units from the surrounding text, and the captions do not state whether the plotted values are raw or normalized.","section":"Figure 3 captions and text"},{"comment":"The appendix states that MLA-PreRoPE 'nearly disappears' the middle-layer entropy dip and stabilizes faster than MHA, which seems to conflict with the main-text claim that MLA-PreRoPE suffers persistent spikes and rank collapse; the connection between the entropy analysis and the spectral analysis is not explained.","section":"Appendix A"}],"recommendation":"reject","confidential_remarks":"The paper addresses an interesting and timely question, and the diagnostic suite is lightweight. However, the central spectral evidence is mathematically impossible under the paper's own definition of G: rank-32 matrices cannot produce the reported 60-65 outliers or stable ranks above 32. The aspect-ratio confound further prevents any causal attribution to rotary sharing. These are not local presentation issues; the core experiments would need to be redone with a correctly defined and matched object before the claim can be evaluated. I therefore recommend rejection, while noting that a future study that logs per-head Gram matrices or appropriately normalizes across matched aspect ratios could salvage the research direction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper asks a real question—does MLA's latent compression change the spectral dynamics of attention projections—and the perplexity table is clean. But the central spectral analysis is not computing what the authors say it computes. The numbers in Figures 1–3 are incompatible with the matrix defined in Eq. (1) and the stated dimensions. With d_in=32, the cross-Gram G=(1/32)W_Q W_K^T in R^{768×768} has rank at most 32, so it cannot have 60–65 outliers per layer or stable rank ~45. The heatmaps showing stable rank ~90 are equally impossible. Either the logged object is a different matrix, or the outlier/rank computation is wrong. This is a load-bearing flaw because the paper's headline claim—that only decoupled MLA suppresses outlier formation—rests on these numbers.\n\nWhat is genuinely new: the first application of MP diagnostics to MLA variants, and the idea that how RoPE is applied (shared sub-vector vs. pre-compression) might matter for spectral health. The paper is clearly written and honestly lists its limitations (130M model, 20K steps, 2.2B tokens). The perplexity comparison is internally coherent and shows that balanced decoupled MLA matches MHA while NoPE degrades by ~4.7 PPL.\n\nThe other soft spot is the cross-variant comparison. MP bulk edge λ_+ depends on aspect ratio γ. MHA has γ=1 (λ_+≈4), MLA-PreRoPE γ=24 (λ_+≈34.8), MLA-Decoupled γ=12 (λ_+≈19.9). A singular value of, say, 10 would be an outlier in MHA but inside the decoupled bulk. The near-zero MP-Gap for decoupled is partly predetermined by a much larger threshold, not by rotary sharing. No variant isolates sharing at matched dimensions. No code or data is released, so the numbers cannot be audited.\n\nBottom line: the paper is for an audience interested in spectral diagnostics for attention, but in its current form it is not reviewable. The authors need to correct the spectral computation, match aspect ratios, and release artifacts. I would not send this to peer review until that is done. If fixed, it could be a decent workshop-level contribution.","headline":"Good question, clean perplexity table, but the spectral evidence is internally inconsistent and the main comparison is confounded by aspect ratio.","tokens_in":8671,"tokens_out":4770,"would_cite":false,"duration_ms":54195,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that only the MLA variant with a head-shared rotary sub-vector prevents spectral outlier formation and rank collapse during pretraining, while standard MHA and MLA-PreRoPE develop persistent mid-layer spikes.","keywords":["multi-head latent attention","random matrix theory","Marchenko-Pastur distribution","spectral outliers","rank collapse","rotary embeddings","transformers","pretraining dynamics"],"falsifier":"Train two MLA variants with identical latent dimension and identical row dimension, differing only in whether the rotary sub-vector is shared across heads; if the per-head variant also keeps MP-Gap near zero and outlier count at zero, the paper's mechanism is not what suppresses spikes, and the result is a shape artifact.","tokens_in":7596,"feed_emoji":"🧠","tokens_out":15057,"duration_ms":156189,"temperature":0.7,"pith_summary":"Multi-head latent attention (MLA) compresses keys and queries into a low-dimensional latent space to cut memory costs, and this paper asks whether that compression also drains a transformer's internal capacity during pretraining. Tracking the eigenvalue spectrum of the query-key Gram matrix with Marchenko-Pastur diagnostics in a 12-layer LLaMA-130M trained for 20K steps, the paper finds that the answer depends on how rotary embeddings are applied, not just on where compression happens. Standard MHA and MLA with rotary applied before compression develop sharp spectral outliers in middle layers early in training; those spikes persist and propagate, concentrating roughly 70% of spectral energy into a few directions and collapsing stable rank. The decoupled MLA variant, which shares a single rotary sub-vector across all heads, keeps the largest eigenvalue at the Marchenko-Pastur bulk edge, records zero outliers, and sustains over 60% stable rank across layers. If the paper is right, architects get a concrete design rule: sharing the rotary component across heads preserves representational breadth without giving up the memory savings of latent compression.","feed_headline":"Only head-shared rotary embeddings stop rank collapse","feed_subtitle":"Random-matrix diagnostics show decoupled MLA stays inside the noise bulk while MHA and PreRoPE spike.","key_machinery":"The central object is the cross-Gram matrix $G = \\frac{1}{d_{\\mathrm{in}}} W_Q W_K^\\top \\in \\mathbb{R}^{m\\times m}$ formed from each layer's query and key projection weights, whose squared singular values are compared against the Marchenko-Pastur bulk with edges $\\lambda_\\pm = (1\\pm\\sqrt{\\gamma})^2$, where $\\gamma = m/d_{\\mathrm{in}}$. The diagnostics built on that comparison carry the argument: MP-Gap, the distance of the top eigenvalue above the bulk edge; outlier count and outlier energy, how many eigenvalues exceed $\\lambda_+$ and what share of spectral mass they hold; MP-Soft-Rank, the normalized spike distance; and stable rank, the usable dimensionality after spikes are excluded. The Gram object turns 'where spikes form and whether they spread' into a measurable spectral trajectory, and the MP bulk supplies the null model that makes a near-zero gap meaningful.","core_discovery":"On the paper's own terms, the central discovery is a spectral cascade in query-key alignment. In MHA and MLA-PreRoPE, the $W_Q W_K^\\top$ Gram matrix develops eigenvalues above the Marchenko-Pastur upper edge $\\lambda_+ = (1+\\sqrt{\\gamma})^2$, starting around the sixth layer within the first few thousand training steps, and these outliers spread to deeper layers while stable rank falls. MLA-Decoupled keeps the MP-Gap essentially at zero, maintains an outlier count of zero, and returns outlier energy to the bulk, with a median near 0.40 and most mass between 0.20 and 0.55. The paper attributes this difference to the head-shared rotary sub-vector: because every head uses the same rotary component on a 32-dimensional latent, the query-key spectrum stays inside the MP bulk and normalized stable rank stays above 60% at all depths. It also shows the allocation matters: a balanced 50:50 content-to-position split is the stable configuration, imbalanced 0.25/0.75 splits raise outlier energy and perplexity by 0.15 to 0.20, and removing positional encoding entirely collapses the spectrum and costs +4.7 perplexity.","pith_inferences":["The compared variants differ in Gram-matrix shape ($\\gamma=1$ for MHA versus $\\gamma=12$ or $24$ for MLA), so the decisive follow-up is a matched-dimension ablation that holds row and column dimensions fixed while toggling only the shared rotary branch; that would isolate whether rotary sharing causes the flat spectrum.","Since each MLA Gram matrix has rank at most $d_{\\mathrm{in}}=32$, reporting per-layer eigenvalue histograms would show how observed stable-rank values near 45 and outlier counts near 60 relate to the rank bound, and would make the capacity comparison easier to interpret.","If head-shared rotary components are the active ingredient, the same MP diagnostics should predict which other compressed-attention designs with a shared positional branch stay spectrally flat, which is a direct transfer test of the paper's mechanism."],"forward_implications":["A transformer using MLA should apply rotary embeddings through a single head-shared sub-vector after compression, because this is the configuration in which the MP-Gap stays near zero and the outlier count stays zero.","The content-to-position split in the decoupled variant is load-bearing: moving from a 50:50 split to 0.25 or 0.75 raises perplexity by 0.15 to 0.20, and dropping positional encoding entirely costs +4.7 perplexity while pushing outlier energy above 80%.","Spectral health and memory efficiency are separable design axes: MLA-Decoupled obtains both a compressed KV representation and a flat spectrum, whereas MLA-PreRoPE gets the memory saving while retaining an MHA-like spike burden.","Spectral spikes are detectable within the first 5K training steps and localize in a specific mid layer before spreading, so early MP-Gap readings at that layer can forecast later rank collapse."],"supporting_citations":[{"why":"Supplies the Marchenko-Pastur law that defines the bulk edges used as the null spectrum for outlier detection.","marker":"[14]"},{"why":"Supplies the $W_Q W_K^\\top$ cross-Gram object whose spectrum the diagnostics analyze.","marker":"[2]"},{"why":"Provides the 130M-parameter transformer architecture and training recipe used for all runs.","marker":"[10]"},{"why":"Introduces the multi-head latent attention scheme whose variants the paper compares.","marker":"[12]"},{"why":"Documents the production variant of the latent attention scheme that motivates the decoupled rotary treatment.","marker":"[13]"},{"why":"Supplies the soft-rank and stable-rank definitions used to convert spike statistics into capacity metrics.","marker":"[15]"}],"fun_headline_variants":["Decoupled MLA kills spectral spikes that cause rank collapse","Head-shared rotary keeps transformer spectra inside Marchenko-Pastur bulk","MLA Decoupled avoids rank collapse by sharing rotary across heads","Spectral spikes vanish when rotary is shared across MLA heads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing Marchenko-Pastur metrics across architectures with different Gram-matrix shapes and aspect ratios is meaningful, because the decoupled variant's near-zero MP-Gap could be largely fixed by its much wider MP bulk edge rather than produced by sharing the rotary vector.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled MLA kills spectral spikes that cause rank collapse","Head-shared rotary keeps transformer spectra inside Marchenko-Pastur bulk","MLA Decoupled avoids rank collapse by sharing rotary across heads","Spectral spikes vanish when rotary is shared across MLA heads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2897,"prompt_tokens":1029,"completion_tokens":1868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":1799}},"tokens_in":645,"tokens_out":1868,"duration_ms":15049,"temperature":1.0,"reasoning_tokens":1799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:57:34.940167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two MLA variants with identical latent dimension and identical row dimension, differing only in whether the rotary sub-vector is shared across heads; if the per-head variant also keeps MP-Gap near zero and outlier count at zero, the paper's mechanism is not what suppresses spikes, and the result is a shape artifact.","supporting_citations":[{"cited_title":"Distribution of eigenvalues for some sets of random matrices","cited_arxiv_id":null,"evidence_quote":"Supplies the Marchenko-Pastur law that defines the bulk edges used as the null spectrum for outlier detection."},{"cited_title":"Self-attention networks localize when QK- eigenspectrum concentrates","cited_arxiv_id":null,"evidence_quote":"Supplies the $W_Q W_K^\\top$ cross-Gram object whose spectrum the diagnostics analyze."},{"cited_title":"Mix-LN: Unleashing the power of deeper layers by combining pre-LN and post-LN","cited_arxiv_id":null,"evidence_quote":"Provides the 130M-parameter transformer architecture and training recipe used for all runs."},{"cited_title":"Implicit self-regularization in deep neural net- works: Evidence from random matrix theory and implications for learning","cited_arxiv_id":null,"evidence_quote":"Supplies the soft-rank and stable-rank definitions used to convert spike statistics into capacity metrics."}],"review_version":1}