Pith. sign in

REVIEW 4 major objections 4 minor 23 references

A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that only the MLA variant with a head-shared rotary sub-vector prevents spectral outlier formation and rank collapse during pretraining, while standard MHA and MLA-PreRoPE develop persistent mid-layer spikes.

desk verdict Good question, clean perplexity table, but the spectral evidence is internally inconsistent and the main comparison is confounded by aspect ratio. read the letter →

arxiv 2507.09394 v1 pith:JTCELKEJ submitted 2025-07-12 cs.LG

Nandan Kumar Jha , Brandon Reagen This is my paper · ORCID
classification cs.LG
keywords multi-headlatentattentionrandommatrixtheoryMarchenko-Pasturdistributionspectraloutliersrankcollapserotaryembeddingstransformerspretrainingdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-head latent attention (MLA) compresses keys and queries into a low-dimensional latent space to cut memory costs, and this paper asks whether that compression also drains a transformer's internal capacity during pretraining. Tracking the eigenvalue spectrum of the query-key Gram matrix with Marchenko-Pastur diagnostics in a 12-layer LLaMA-130M trained for 20K steps, the paper finds that the answer depends on how rotary embeddings are applied, not just on where compression happens. Standard MHA and MLA with rotary applied before compression develop sharp spectral outliers in middle layers early in training; those spikes persist and propagate, concentrating roughly 70% of spectral energy into a few directions and collapsing stable rank. The decoupled MLA variant, which shares a single rotary sub-vector across all heads, keeps the largest eigenvalue at the Marchenko-Pastur bulk edge, records zero outliers, and sustains over 60% stable rank across layers. If the paper is right, architects get a concrete design rule: sharing the rotary component across heads preserves representational breadth without giving up the memory savings of latent compression.

What carries the argument

The central object is the cross-Gram matrix $G = \frac{1}{d_{\mathrm{in}}} W_Q W_K^\top \in \mathbb{R}^{m\times m}$ formed from each layer's query and key projection weights, whose squared singular values are compared against the Marchenko-Pastur bulk with edges $\lambda_\pm = (1\pm\sqrt{\gamma})^2$, where $\gamma = m/d_{\mathrm{in}}$. The diagnostics built on that comparison carry the argument: MP-Gap, the distance of the top eigenvalue above the bulk edge; outlier count and outlier energy, how many eigenvalues exceed $\lambda_+$ and what share of spectral mass they hold; MP-Soft-Rank, the normalized spike distance; and stable rank, the usable dimensionality after spikes are excluded. The Gram object turns 'where spikes form and whether they spread' into a measurable spectral trajectory, and the MP bulk supplies the null model that makes a near-zero gap meaningful.

What would settle it

Train two MLA variants with identical latent dimension and identical row dimension, differing only in whether the rotary sub-vector is shared across heads; if the per-head variant also keeps MP-Gap near zero and outlier count at zero, the paper's mechanism is not what suppresses spikes, and the result is a shape artifact.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a spectral cascade in query-key alignment. In MHA and MLA-PreRoPE, the $W_Q W_K^\top$ Gram matrix develops eigenvalues above the Marchenko-Pastur upper edge $\lambda_+ = (1+\sqrt{\gamma})^2$, starting around the sixth layer within the first few thousand training steps, and these outliers spread to deeper layers while stable rank falls. MLA-Decoupled keeps the MP-Gap essentially at zero, maintains an outlier count of zero, and returns outlier energy to the bulk, with a median near 0.40 and most mass between 0.20 and 0.55. The paper attributes this difference to the head-shared rotary sub-vector: because every head uses the same rotary component on a 32-dimensional latent, the query-key spectrum stays inside the MP bulk and normalized stable rank stays above 60% at all depths. It also shows the allocation matters: a balanced 50:50 content-to-position split is the stable configuration, imbalanced 0.25/0.75 splits raise outlier energy and perplexity by 0.15 to 0.20, and removing positional encoding entirely collapses the spectrum and costs +4.7 perplexity.

Load-bearing premise

The load-bearing premise is that comparing Marchenko-Pastur metrics across architectures with different Gram-matrix shapes and aspect ratios is meaningful, because the decoupled variant's near-zero MP-Gap could be largely fixed by its much wider MP bulk edge rather than produced by sharing the rotary vector.

Editorial extensions

If this is right

  • A transformer using MLA should apply rotary embeddings through a single head-shared sub-vector after compression, because this is the configuration in which the MP-Gap stays near zero and the outlier count stays zero.
  • The content-to-position split in the decoupled variant is load-bearing: moving from a 50:50 split to 0.25 or 0.75 raises perplexity by 0.15 to 0.20, and dropping positional encoding entirely costs +4.7 perplexity while pushing outlier energy above 80%.
  • Spectral health and memory efficiency are separable design axes: MLA-Decoupled obtains both a compressed KV representation and a flat spectrum, whereas MLA-PreRoPE gets the memory saving while retaining an MHA-like spike burden.
  • Spectral spikes are detectable within the first 5K training steps and localize in a specific mid layer before spreading, so early MP-Gap readings at that layer can forecast later rank collapse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The compared variants differ in Gram-matrix shape ($\gamma=1$ for MHA versus $\gamma=12$ or $24$ for MLA), so the decisive follow-up is a matched-dimension ablation that holds row and column dimensions fixed while toggling only the shared rotary branch; that would isolate whether rotary sharing causes the flat spectrum.
  • Since each MLA Gram matrix has rank at most $d_{\mathrm{in}}=32$, reporting per-layer eigenvalue histograms would show how observed stable-rank values near 45 and outlier counts near 60 relate to the rank bound, and would make the capacity comparison easier to interpret.
  • If head-shared rotary components are the active ingredient, the same MP diagnostics should predict which other compressed-attention designs with a shared positional branch stay spectrally flat, which is a direct transfer test of the paper's mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a lightweight random-matrix-theory diagnostic suite (MP-Gap, outlier count and energy, MP-Soft rank, stable rank) applied to the W_Q W_K^T cross-Gram matrix during LLaMA-130M pretraining. Three attention variants are compared: standard MHA, MLA with rotary embeddings applied before compression (MLA-PreRoPE), and MLA with a shared rotary sub-vector across heads (MLA-Decoupled). Based on spectral diagnostics of this Gram matrix, the paper claims that only the decoupled variant suppresses spectral outlier formation and maintains stable rank, attributing this to rotary-vector sharing. A final perplexity table reports that balanced decoupled MLA matches MHA perplexity, while extreme RoPE budgets or no positional encoding degrade performance.

Significance. If the central empirical claim were supported, the paper would offer a practically relevant design heuristic: sharing rotary sub-vectors across heads in MLA avoids mid-layer spectral spikes and rank collapse, and the proposed diagnostics are lightweight enough for routine monitoring. The experiments are nontrivial in scale (2.2B tokens, 12 layers) and the perplexity comparison is internally coherent. However, the spectral evidence as presented is internally inconsistent with the paper's own definition of the analyzed matrix, and the cross-architecture comparison is confounded by different aspect ratios. The causal attribution to rotary sharing is therefore not currently established; the significance is conditional on a corrected and properly controlled analysis.

major comments (4)
  1. [Sec. 3, Eq. (1) and Sec. 4, Figs. 1-3] The reported spectral values for MLA-PreRoPE are impossible for the matrix defined in Eq. (1). With d_in = 32 and m = H·d_k = 768, G = (1/d_in) W_Q W_K^T is a 768×768 matrix of rank at most 32, so it has at most 32 nonzero singular values and its stable rank cannot exceed 32. Yet Figure 1(b) reports 60-65 outliers per layer, Figure 2(b) reports a stable rank of about 45 for MLA-PreRoPE, and Figure 3(f) shows stable-rank heatmap values up to roughly 90. Either the logged object is not the G of Eq. (1) (for example, per-head Gram matrices or a different d_in are used), or the outlier-count and stable-rank computations are erroneous. Because this impossible trajectory is the basis for grouping MLA-PreRoPE with MHA and contrasting it with MLA-Decoupled, the central comparison cannot be evaluated until this is resolved.
  2. [Sec. 4, Figs. 1-3 and Eq. (2)] The cross-architecture comparison is not controlled for aspect ratio. Equation (2) gives lambda_+ = (1 + sqrt(gamma))^2, with gamma = m/d_in: MHA has gamma = 1 and lambda_+ = 4, MLA-PreRoPE has gamma = 24 and lambda_+ ≈ 34.8, and MLA-Decoupled has gamma = 12 and lambda_+ ≈ 19.9. A near-zero MP-Gap for the decoupled variant is therefore partly predetermined by its much larger lambda_+, and the outlier-count and outlier-energy metrics are not directly comparable across the three variants. No experiment isolates the rotary-sharing factor at matched gamma and row dimension, so the headline claim that only decoupled MLA suppresses outliers is confounded by the very different null thresholds.
  3. [Sec. 4, Figs. 2-3 and text] The text and figures are mutually inconsistent about stable rank. The text states that 'Decoupled MLA consistently sustains >60% normalized rank across all layers and training steps,' while Figure 2(b) shows MLA-Decoupled stable rank collapsing to about 5 and the heatmap in Figure 3(e) shows values between 2 and 16. The text also says MLA-PreRoPE retains the highest capacity at about 45, but Figure 3(f) shows values up to about 90. The reader cannot tell whether the reported numbers are raw stable rank, normalized stable rank, or something else, and the discrepancy undermines the capacity argument.
  4. [Sec. 3, Eqs. (1)-(2) and Table 1] The normalization claim that dividing by d_in sets the expected entry variance of G to one is not established for the cross-Gram construction. If W_Q and W_K have i.i.d. entries with variance sigma^2, then Var(G_ij) = sigma^4 / d_in, which equals 1 only under an unusual scaling (sigma^2 = sqrt(d_in)); the paper does not specify the initial weight scaling. In addition, Eq. (2) is the standard Marchenko-Pastur edge for a sample covariance matrix (1/d_in) X X^T with X having i.i.d. entries, but G = (1/d_in) W_Q W_K^T is a product of two independent random matrices, and its singular-value spectrum is not automatically governed by Eq. (2). The theoretical basis for the quoted lambda_+ needs a derivation or a reference to the appropriate product-ensemble result.
minor comments (4)
  1. [Sec. 2] The training budget is arithmetically inconsistent: 20,000 steps with batch size 512 and context length 256 gives about 2.6B tokens, not the stated 2.2B tokens; if the dataset is 2.2B tokens, the number of steps should be about 16,800.
  2. [Fig. 4] The violin-plot description says the MHA/Pre-RoPE distribution has a 'long tail extending to ~0.60'; since the distribution is centered at ~0.75, this should be described as a lower tail, not a long tail without qualification.
  3. [Figure 3 captions and text] The heatmap colorbars in Figure 3 lack labels; the reader must infer the units from the surrounding text, and the captions do not state whether the plotted values are raw or normalized.
  4. [Appendix A] The appendix states that MLA-PreRoPE 'nearly disappears' the middle-layer entropy dip and stabilizes faster than MHA, which seems to conflict with the main-text claim that MLA-PreRoPE suffers persistent spikes and rank collapse; the connection between the entropy analysis and the spectral analysis is not explained.

Circularity Check

1 steps flagged · score 7.0 of 10

Decoupled MLA 'outlier suppression' is largely a consequence of variant-specific MP thresholds rather than an independent empirical discovery.

  1. renaming known result [Section 3 ('Cross-Gram construction', Eq. (2) and Table 1); Section 4, Fig. 1]
    "In the decoupled setting, we further isolate the RoPE branch, with d_in = 32 and row dimension m = 1/2 H d k ... With aspect ratio γ = m/d_in, the MP bulk edges are therefore λ± = (1 ± √γ)^2. ... MHA and Pre-RoPE both stabilize at 60 to 65 outliers per layer (roughly 5 to 6 per head), while Decoupled MLA consistently exhibits zero, empirically confirming the absence of spectral outliers."

    The 'Outlier Count' metric in Table 1 is #{λ_i > λ_+}, and λ_+ is defined by Eq. (2) as (1+√(m/d_in))^2. The paper chooses m=384,d_in=32 for decoupled (γ=12 ⇒ λ_+≈19.9), m=768,d_in=768 for MHA (γ=1 ⇒ λ_+=4), and m=768,d_in=32 for MLA-Pre (γ=24 ⇒ λ_+≈30.4). Hence 'zero outliers'/'near-zero MP-Gap' for decoupled means only that its top singular value stayed below a much larger threshold; an i.i.d. Gaussian matrix with the same dimensions would also sit at the MP edge. The headline conclusion that shared rotary vectors suppress outliers is therefore a restatement of the metric's definition plus the unequal aspect-ratio choice, not an independently measured effect of rotary sharing. The specific reduction is: outlier count = f(λ_+), λ_+ = f(m,d_in), with m,d_in chosen per variant.

full rationale

The paper contains no self-citations and no fitted-parameter-then-prediction loop; its perplexity table is an independent empirical benchmark. However, the central spectral claim — that only MLA-decoupled suppresses outlier formation — is not supported as an independent discovery. The MP diagnostics are defined against a variant-specific threshold λ_+=(1+√(m/d_in))^2. Because the decoupled branch is analyzed at γ=12 (m=384,d_in=32, λ_+≈19.9) while MHA is at γ=1 (λ_+=4) and MLA-Pre at γ=24 (λ_+≈30.4), 'zero outliers' and 'near-zero MP-Gap' for decoupled are the expected null outcomes for a random matrix of those dimensions; the causal attribution to shared rotary vectors is a renaming of this threshold effect. Separately, the paper's own Eq. (1) with d_in=32 implies rank≤32 for MLA-Pre, yet Section 4 reports 60–65 outliers and stable rank ≈45, so the logged object cannot be the stated G; this internal inconsistency independently undermines the comparison. These are load-bearing flaws in the spectral evidence, although not self-citation. The perplexity results are coherent but do not test the spectral mechanism.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new particles, forces, or theoretical entities. Its central claims rest on three chosen design parameters: the aspect ratio of the Gram matrix, the compression ratio, and the RoPE budget. The aspect ratio choice is especially consequential because it changes the MP threshold and can predetermine the observed 'no outliers' result. The MP null model and cross-architecture comparability are unvalidated assumptions.

free parameters (3)
  • Aspect ratio gamma of the analyzed Gram matrix = MHA: 1 (m=768, d_in=768); MLA-PreRoPE: 24 (m=768, d_in=32); MLA-Decoupled: 12 (m=384, d_in=32)
    The MP bulk edge in Equation (2) depends on gamma. The claim that Decoupled has no outliers is largely a consequence of choosing a larger gamma, not an independent empirical discovery.
  • Latent compression ratio = 2 (latent dimension reduced from 64 to 32)
    All MLA variants use compression ratio 2, which sets the rank of the cross-Gram to at most 32. This conflicts with the reported outlier counts and stable ranks above 32.
  • RoPE budget, content-to-position split = 0.50 default; 0.25 and 0.75 variants
    The paper scans this allocation and finds 50:50 best. This is a chosen design parameter, not an independent estimate, and the variants differ in spectral metrics.
assumptions (2)
  • domain assumption The Marchenko-Pastur bulk formula applies to the cross-Gram of trained query and key projection weights as if they were i.i.d. random matrices.
    Equation (2) uses lambda_plus = (1 + sqrt(gamma))^2. Trained weights are highly structured, so the null model is an approximation that is never validated in the paper.
  • domain assumption Spectral statistics across MHA, MLA-PreRoPE, and MLA-Decoupled are comparable despite different m and d_in.
    Section 3 sets d_in = 32 and m = 768 or 384, while MHA uses d_in = 768 and m = 768. The paper compares MP-Gap and outlier metrics across these shapes without adjustment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention." pith.science (2026). https://pith.science/paper/JTCELKEJ

@misc{pith2026250709394,
  author       = {Pith},
  title        = {Pith review of: A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTCELKEJ}},
  note         = {Machine review of arXiv:2507.09394}
}
abstract

In this work, we study how multi-head latent attention (MLA), a popular strategy for compressing key/value memory, affects a transformer's internal capacity during pretraining. Using a lightweight suite of Marchenko-Pastur (MP) diagnostics, we analyze the spectrum of the $W_{Q}W_{K}^\top$ gram matrix throughout training, comparing three variants: the standard multi-head attention (MHA) baseline, MLA-PreRoPE with rotary applied before compression, and MLA-Decoupled, which shares a single rotary sub-vector across all heads. Our random matrix analysis reveals \textbf{three key findings:} \textbf{ i)} capacity bottlenecks emerge locally: both MHA and MLA-PreRoPE exhibit sharp, early spikes in specific layers that persist and propagate, disrupting the balance between bulk and outlier directions; \textbf{ ii)} these spikes coincide with rank collapse, concentrating the model's expressivity into narrow subspaces; \textbf{ iii)} only the decoupled variant prevents this cascade, maintaining broad spectral support and suppressing outlier formation across layers. These results underscore that \emph{how} rotary embeddings are applied is just as critical as \emph{where} compression occurs. Sharing rotary components across heads mitigates spectral fragmentation and preserves representational capacity.

Figures

Figures reproduced from arXiv: 2507.09394 by the authors.

Figure 1
Figure 1. Spectral-spike dynamics: (a) MP-Gap, (b) outlier count, and (c) outlier energy, for MHA (blue), MLA-Dec (orange), and MLA-Pre (green). Curves show layer-wise means and shaded bands denote ±1 standard deviation in LLaMA-130M. Together, these metrics capture the emergence and strength of spectral outliers in the WQW⊤ K spectrum. Nonetheless, on the the flip side, the Stable-Rank, a proxy for usable dimensionality show… view at source ↗
Figure 2
Figure 2. Spectral-Capacity Dynamics: (a) MP-Soft-Rank, and (b) Stable Rank are shown for LLaMA-130M model with MHA (blue), MLA-Pre (orange), and MLA-Dec (green). Curves show layer means; shaded regions indicate ±1 standard deviation across 12 lay￾ers. Higher MP-Soft-Rank signals sharper spectral spikes; higher Stable Rank indicates better bulk direction usage. MLA-Dec excels at suppressing outliers, while MLA-Pre offers the … view at source ↗
Figure 3
Figure 3. Layerwise spectral dynamics: (Top row)MP-Gap and (bottom row)StableRank heatmaps. MHA exhibits strong mid-layer concentration in MPGap and declining Sta￾bleRank in later layers, while MLA-based methods spread representational changes more evenly, maintaining higher stable ranks across depths The Stable-Rank heat-maps (bottom row) complete the picture. MHA starts with a high rank (∼ 120) in the first few layers but c… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Upper energy violin plot The violin plot in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Upper energy violin plot for various rotatory (RoPE) budget [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Attention entropy patterns in classical MHA and MLA variants (decoupled and Pre￾RoPE) MLA-Decoupled softens these extremes, moderating both overload and starvation. MLA-PreRoPE further improves the entropy distribution: the middle-layer entropy dip nearly disappears, d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 16 canonical work pages

  1. [1]

    A random matrix perspective on mix- tures of nonlinearities in high dimensions

    Ben Adlam, Jake A Levinson, and Jeffrey Pennington. A random matrix perspective on mix- tures of nonlinearities in high dimensions. In International Conference on Artificial Intelli- gence and Statistics, 2022

    work page 2022

  2. [2]

    Self-attention networks localize when QK- eigenspectrum concentrates

    Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when QK- eigenspectrum concentrates. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024

    work page 2024

  3. [3]

    Random matrix theory improved fr ´echet mean of symmetric positive definite matrices

    Florent Bouchard, Ammar Mian, Malik Tiomoko, Guillaume Ginolhac, and Frederic Pascal. Random matrix theory improved fr ´echet mean of symmetric positive definite matrices. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  4. [4]

    A random matrix ap- proach to echo-state neural networks

    Romain Couillet, Gilles Wainrib, Hafiz Tiomoko Ali, and Harry Sevi. A random matrix ap- proach to echo-state neural networks. In International Conference on Machine Learning , 2016

    work page 2016

  5. [5]

    A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities

    Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. In The 28th International Conference on Artificial Intelligence and Statistics , 2025

    work page 2025

  6. [6]

    Random matrix analysis to balance between supervised and unsupervised learning under the low density separation assumption

    Vasilii Feofanov, Malik Tiomoko, and Aladin Virmaux. Random matrix analysis to balance between supervised and unsupervised learning under the low density separation assumption. In International Conference on Machine Learning, 2023

    work page 2023

  7. [7]

    Maximizing the potential of synthetic data: Insights from ran- dom matrix theory

    Aymane El Firdoussi, Mohamed El Amine Seddik, Soufiane Hayou, Reda ALAMI, Ahmed Alzubaidi, and Hakim Hacid. Maximizing the potential of synthetic data: Insights from ran- dom matrix theory. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  8. [8]

    Analysing multi-task regression via random matrix theory with application to time series forecasting

    Romain Ilbert, Malik Tiomoko, Cosme Louart, Ambroise Odonnat, Vasilii Feofanov, Themis Palpanas, and Ievgen Redko. Analysing multi-task regression via random matrix theory with application to time series forecasting. Advances in Neural Information Processing Systems , 2024

    work page 2024

Show all 23 references
  1. [9]

    The underlying scaling laws and universal statistical structure of complex datasets

    Noam Levi and Yaron Oz. The underlying scaling laws and universal statistical structure of complex datasets. arXiv preprint arXiv:2306.14975, 2023

    2023 arXiv

  2. [10]

    Mix-LN: Unleashing the power of deeper layers by combining pre-LN and post-LN

    Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the power of deeper layers by combining pre-LN and post-LN. In The Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  3. [11]

    The dynamics of learning: A random matrix approach

    Zhenyu Liao and Romain Couillet. The dynamics of learning: A random matrix approach. In International Conference on Machine Learning, 2018

    2018

  4. [12]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 7 LEARNING DYNAMICS OF MULTI-...

    2024 arXiv

  5. [13]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  6. [14]

    Distribution of eigenvalues for some sets of random matrices

    V A Marchenko and Leonid A Pastur. Distribution of eigenvalues for some sets of random matrices. Mat. Sb.(NS), 72(114):4, 1967

    1967

  7. [15]

    Implicit self-regularization in deep neural net- works: Evidence from random matrix theory and implications for learning

    Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural net- works: Evidence from random matrix theory and implications for learning. Journal of Ma- chine Learning Research, 2021

    2021

  8. [16]

    Transmla: Multi-head latent attention is all you need

    Fanxu Meng, Zengwei Yao, and Muhan Zhang. Transmla: Multi-head latent attention is all you need. arXiv preprint arXiv:2502.07864, 2025

    2025 arXiv

  9. [17]

    Geometry of neural network loss surfaces via random matrix theory

    Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International conference on machine learning, 2017

    2017

  10. [18]

    Nonlinear random matrix theory for deep learning

    Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. Advances in neural information processing systems, 30, 2017

    2017

  11. [19]

    Locating information in large language models via random matrix theory

    Max Staats, Matthias Thamm, and Bernd Rosenow. Locating information in large language models via random matrix theory. arXiv preprint arXiv:2410.17770, 2024

    2024

  12. [20]

    Random matrix theory analysis of neural network weight matrices

    Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix theory analysis of neural network weight matrices. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024

    2024

  13. [21]

    Random ma- trix improved covariance estimation for a large class of metrics

    Malik Tiomoko, Romain Couillet, Florent Bouchard, and Guillaume Ginolhac. Random ma- trix improved covariance estimation for a large class of metrics. In International Conference on Machine Learning, 2019

    2019

  14. [22]

    More than a toy: Random matrix models pre- dict how real-world neural representations generalize

    Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models pre- dict how real-world neural representations generalize. InInternational conference on machine learning, 2022

    2022

  15. [23]

    Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures

    Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, et al. Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures. arXiv preprint arXiv:2505.09343, 2025. 8 LEA...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.