REVIEW 2 major objections 4 minor 24 references
Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks
T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that the latent dimension at which random low-dimensional reparameterization can reach a low-loss region is not an empirical accident: in a quadratic model it is set by a master formula combining the curvature spectrum…
desk verdict An honest, well-tested paper with a genuinely new orientation-resolved predictor; the unproved ridge-leverage surrogate is a real gap but not fatal—worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The orientation-resolved master formula $\hat\rho_{\mathrm{MF}}(d; \Delta_0) = \sum_i \frac{\kappa(d)\lambda_i}{\lambda_i+\kappa(d)} \Delta_i^2$, with $\kappa(d)$ fixed by $d = \operatorname{tr}[H(H+\kappa I)^{-1}]$, is the central object. It acts as a ridge-leverage surrogate for the expected projection onto the random slice's curvature image, turning an intractable geometric threshold into a computable mean-level predictor. The rigorous conic theorem supplies the transition center as the statistical dimension of the polar cone, $d_{\mathrm{conic}} = \delta(C^\circ)$; the master formula is the operational replacement that uses only the curvature spectrum and the displacement profile.
What would settle it
In a controlled quadratic experiment with $P=1024$ and a slowly decaying spectrum such as $\lambda_i \propto i^{-1}$, take a displacement of fixed norm aligned with a mid-curvature eigenvector, run many random Gaussian slices, and compare the measured $d_{50}$—the dimension at which 50% of slices hit the $\varepsilon$-sublevel set—with the master-formula threshold $\hat d_{\mathrm{MF}}$. If the discrepancy exceeds the sharpened conic window $O(\sqrt{d_{\mathrm{conic}} \log(1/\eta)})$ or a few percent relative error, the ridge-leverage surrogate is falsified for that spectrum.
Extended reading notes
Core claim
For a localized quadratic loss with positive-semidefinite curvature $H$ and displacement $\Delta_0$ from the reference point to the minimizer, the expected residual of a random $d$-dimensional slice is predicted as $\hat\rho_{\mathrm{MF}}(d; \Delta_0) = \sum_i \frac{\kappa(d)\lambda_i}{\lambda_i+\kappa(d)} \Delta_i^2$, where $\kappa(d)$ solves $d = \operatorname{tr}[H(H+\kappa I)^{-1}]$. The predicted critical dimension is the smallest $d$ with $\hat\rho_{\mathrm{MF}}(d; \Delta_0) \le \varepsilon_q = 2\varepsilon$. A key consequence is that two displacements with the same spectrum and same norm can require different latent dimensions when their energy is distributed differently across stiff versus flat curvature directions. The paper reports that this predictor tracks measured quadratic transitions with median relative error about 0.45% in synthetic orientation sweeps and at most 0.91% on neural-curvature probes, while an isotropic-orientation specialization is sharply correct under equal-energy displacement and a radius-only specialization is conservative, recovering the earlier quadratic bound.
Load-bearing premise
The whole quadratic predictor rests on the ridge-leverage surrogate $\mathbb{E}[P_W] \approx H(H+\kappa I)^{-1}$, which is asserted as a finite-dimensional heuristic with no proven accuracy bounds for general spectra; if that surrogate fails for a given spectrum, the predicted critical dimension $\hat d_{\mathrm{MF}}$ is unreliable even though the conic transition theorem itself is rigorous.
Editorial extensions
If this is right
- A latent dimension can be selected without sweeping: given leading curvature eigenvalues and a displacement estimate, $\hat d_{\mathrm{MF}}$ gives an operating dimension for random low-dimensional training.
- Displacement orientation is load-bearing: two tasks with the same Hessian spectrum and the same distance to the solution can differ in required latent dimension, empirically by up to a factor of 37 in the controlled orientation sweeps.
- When only a displacement radius is known, the orientation-uniform predictor $r_{\mathrm{eff}}(\varepsilon, R) = \sum_i \frac{\lambda_i R^2}{\lambda_i R^2 + \varepsilon_q}$ is conservative and reproduces the earlier radius-only quadratic bound.
- Matrix-free implementations with seeded Gaussian maps or structured Hadamard mappings remove $O(dP)$ frozen-map storage, reduce optimizer-state memory to $O(d)$, and still exhibit sharp end-to-end training transitions across MLPs, CNNs, transformers, ResNets, and a pretrained language model.
- The predicted transition is a training-accessibility threshold, not a generalization guarantee: end-to-end midpoints depend on the optimizer, learning-rate protocol, and loss tolerance, and crossing the transition does not by itself ensure full-model predictive performance.
Reading between the lines
- The same master formula could be used to set ranks prospectively in parameter-efficient fine-tuning methods that share frozen random matrices, by treating the rank as the predicted latent dimension for a Fisher or GGN curvature surrogate.
- Because the equal-tail approximation depends on the unresolved spectrum mostly through its sum, coarse spectral estimates with a few leading Ritz pairs plus a trace may suffice for dimension selection; this is testable by comparing predictions made with partial spectra against full-spectrum predictions.
- A natural extension is to predict $d_{\mathrm{gen}}$, the dimension at which generalization approaches the full-parameter model, by applying the same residual analysis to a data-dependent curvature surrogate; the paper separates $d_{\mathrm{train}}$ from $d_{\mathrm{gen}}$ but does not predict the latter.
- For nonconvex losses, the one-sided conic bounds suggest that miss probabilities survive through Gaussian width even when the full transition is not rigorous; a testable extension is whether $\hat d_{\mathrm{MF}}$ still predicts the empirical midpoint when monitored least-squares displacements stay within the localization ball.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies random low-dimensional reparameterizations of neural networks, where a small latent vector z is mapped to a parameter update by a frozen random map. It first recasts the known accessibility transition for compact convex targets in conic form, centering it at the statistical dimension of the polar cone (Theorem 1/2). Its main claimed contribution is an orientation-resolved quadratic master formula bρMF(d; Δ0) = Σ_i κ(d) λ_i / (λ_i + κ(d)) Δ_i^2, with κ(d) fixed by the trace equation tr[H(H+κI)^{-1}] = d, predicting the expected random-slice residual from the curvature spectrum and the displacement profile. Two specializations are derived: an isotropic-orientation predictor and an orientation-uniform predictor that recovers the earlier radius-only bound of Larsen et al. The paper then introduces RaMaN, a framework using structured Hadamard or seed-regenerated Gaussian frozen maps to avoid O(dP) storage, and reports twelve experiments: controlled quadratic transitions, neural-curvature probes, end-to-end training transitions, map-family and optimizer/tolerance ablations, and a ViT-scale GGN–Ritz probe.
Significance. If the master formula is taken as a validated operational predictor, the paper makes a useful contribution: it turns a geometric transition into a computable dimension-selection rule and identifies displacement orientation, not just the Hessian spectrum, as a control parameter. The rigorous conic part (Lemma 1 and Theorem 2) is a correct application of known results, and the paper is commendably explicit about several limitations, notably stating that Experiment 12 is an internal-consistency test rather than independent validation. The controlled experiments (Tables 2, 4, 5) give strong in-model support for the predictor, and the RaMaN framework addresses a real memory bottleneck. However, the central predictor rests on an unproved approximation whose domain of validity is not characterized; this limits the strength of the paper's main claim until that gap is closed or the claim is appropriately narrowed.
major comments (2)
- [Appendix B.2, Approximation 1; Definition 1] The master formula bρMF(d; Δ0) in Definition 1 rests entirely on Approximation 1, E[P_W] ≈ H(H+κI)^{-1} with κ fixed by the trace equation (B.3). The derivation replaces the per-row quantities t_i = a_i^T Q_{-i}^{-1} a_i by a common τ, but this uniform approximation is not controlled. For a spectrum with a dominant outlier, tr(Q_{-i}^{-1}) varies substantially across i, so the stiffest direction can have t_1 far from τ, and the ridge-leverage surrogate can break down. No quantitative error bound is given outside the block-constant model of Section B.5, and the paper's own Experiment 12 is labeled an internal-consistency test. Since Tables 4, 5, and 14 use bdMF as the central predictor, I ask for either (i) a rigorous or quantitative bound on the approximation error under stated conditions (e.g., stable-rank and d/r regime), or (ii) an explicit stress-test experiment with spectra designed to challenge the surrogate, such as a dominant-outlier spectrum, reporting where the predicted critical dimension fails. Without one of these, the domain of validity of the central claim is uncharacterized.
- [Theorem 1, Definition 1, Remark 11] The paper presents the master formula as the main theoretical contribution, but the rigorous conic transition of Theorem 1 applies to the localized target S = Sε ∩ B(θ0, Rloc), whereas the derivation in Appendix B.1 analyzes the unlocalized quadratic residual and ignores Rloc. This gap is acknowledged in Remark 11, but it means that bdMF is not proven to approximate δ(C°), the quantity identified by the theorem. The experiments test the unlocalized least-squares residual rather than the localized conic intersection. I recommend closing the gap by stating explicitly throughout the abstract and introduction that bdMF is a heuristic surrogate for the conic threshold, and by reporting, for at least the quadratic experiments, the quantity ∥Az⋆∥2 relative to Rloc to justify that localization is inactive. As written, the connection between the rigorous theorem and the paper's flagship predictor is not fully closed.
minor comments (4)
- [Figure 3, middle panel] The caption labels the √P curve a 'normalized reference' rather than a prediction, but a reader may misread it as a theoretical envelope; I suggest adding one sentence stating explicitly that no claim of O(√P) width is being made.
- [Table 1] The bert-tiny row lists P = 4,386,178 and then a separate row 'Prep = 413,314' that appears detached; align the reparameterized count with the model row or explain the relationship in the caption.
- [Algorithm 1] The calibration parameters γ and b are introduced without any sensitivity analysis; a sentence on how the selected dimension depends on their choice, or a reference to an ablation if one exists, would help practitioners.
- [Section 4.9, Table 14] The 'resolved fraction' of 0.2–6% and the exact-Hessian-vector spot-check failure are material qualifications of the sub-percent agreement and should be repeated in the table caption or its immediately surrounding text, not only in the prose paragraph.
Circularity Check
No significant circularity: the master-formula predictors are fixed by the trace equation and measured displacement profile, and the central validations use independent random-slice trials; the one shared-surrogate loop (Experiment 12) is explicitly disclosed as an internal-consistency test rather than independent evidence.
full rationale
The derivation chain is self-contained against external benchmarks. Theorem 1 is a rigorous conic phase-transition statement proved via the external kinematic/conic results of Amelunxen, Lotz, McCoy, and Tropp [3] and Gordon [6]. The quadratic master formula in Definition 1 follows from an exact reduction to the least-squares residual rho(A)=||(I-P_W)b||^2 (Appendix B.1) plus a clearly labeled deterministic-equivalent approximation, Approximation 1, in which E[P_W] is replaced by H(H+kappa I)^{-1}. The scalar kappa(d) is not fitted to any measured transition midpoint; it is uniquely determined by the trace identity d=tr[H(H+kappa I)^{-1}], and the displacement profile {Delta_i} is measured rather than tuned. Experiments 1-3 compare bdMF and d*_iso against empirical midpoints d50 obtained from independent Gaussian random slices under the same H and Delta_0; no parameter of the predictors is calibrated on those midpoints. The orientation-sweep experiment (Experiment 2) holds the tolerance and spectrum fixed while varying only the displacement direction, again testing the predictor rather than fitting it. The radius-only specialization reff is explicitly shown to recover the external Larsen et al. bound, which is a consistency check, not a definitional identity. The only place where 'measured' and 'predicted' quantities share a common surrogate is Experiment 12, where both the empirical midpoint and the practical predictors are computed from the same GGN-Ritz-plus-tail approximation. The paper itself states that 'Experiment 12 is an internal consistency test of the surrogate rather than an independent validation of the true network curvature or of the end-to-end training midpoint,' and later warns that the data 'do not establish sub-percent predictor accuracy for the true ViT curvature.' This is a disclosed limitation and a possible correctness risk for large-scale extrapolation, but it is not a circular derivation: the predictor is still a function of the surrogate spectrum and displacement, not of the measured midpoint. Similarly, Approximation 1 lacks proven error bounds for general spectra and is verified only in a block-constant model; that is an unproven heuristic assumption, not a reduction of the prediction to its input. No load-bearing self-citation is present: the cited deterministic-equivalent and random-matrix results [4,13,18] are external, and the authors' own work is not invoked to justify the central formula.
Assumptions & free parameters
free parameters (3)
- Calibration margins γ and b in Algorithm 1 =
unspecified defaults (not stated)
- Layer-wise update scale β_l and nonlinear modulation strength α_l =
β=1 default; α=0.5 in nonlinear SHM ablation
- Loss-excess tolerance ε (and ε_q = 2ε) =
ε = 0.05 in end-to-end experiments; calibrated once in Experiment 2
assumptions (6)
- ad hoc to paper Approximation 1: ridge-leverage surrogate E[P_W] ≈ H(H + κI)^{-1} with trace-matched κ.
- standard math Conic kinematic formula and statistical-dimension phase transition of Amelunxen-Lotz-McCoy-Tropp and Gordon.
- domain assumption Local quadratic model with H ⪰ 0 at a minimizer θ†.
- domain assumption Uniformly random Gaussian subspace for the slice.
- domain assumption Ball localization remains inactive: the least-squares solution stays inside B(θ0, Rloc).
- domain assumption GGN-Ritz surrogate with equal-tail spectral model represents true curvature at scale.
Cite this review
Pith. "Pith review of Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks." pith.science (2026). https://pith.science/paper/3CE3UTLF
@misc{pith2026260812597,
author = {Pith},
title = {Pith review of: Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/3CE3UTLF}},
note = {Machine review of arXiv:2608.12597}
}
read the original abstract
Neural networks can often be trained or fine-tuned through random low-dimensional reparameterization, where a small latent vector is mapped into a full parameter update by a frozen random map. This raises a practical question: how large must the latent search space be to reach a low-loss region? We first express the known accessibility transition in an equivalent conic form, centered for compact convex targets at the statistical dimension of the polar cone. Our main theoretical contribution is an orientation-resolved quadratic master formula that predicts the random-slice residual from both the curvature spectrum and the reference-to-solution displacement profile. It yields a self-consistent isotropic-orientation predictor and, in a conservative radius-only specialization, recovers the earlier Gaussian-width quadratic bound. Building on this analysis, we introduce Random Mapping Networks (RaMaN), which instantiate the predicted latent dimension using structured Hadamard or seed-regenerated Gaussian maps. These constructions avoid the O(dP) storage of dense random maps and reduce optimizer-state memory from O(P) to O(d). We also develop matrix-free curvature approximations and sweep-free dimension selection. Across controlled quadratic and neural-curvature experiments, the orientation-resolved predictor closely tracks measured transition locations and outperforms orientation-agnostic approximations when displacement direction matters. End-to-end experiments further show sharp, protocol-dependent training transitions across image and language models.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning”. In:Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Online: Association for Comp...
work page 2021
-
[2]
The fast Johnson–Lindenstrauss transform and approximate nearest neighbors
Nir Ailon and Bernard Chazelle. “The fast Johnson–Lindenstrauss transform and approximate nearest neighbors”. In:SIAM Journal on computing39.1 (2009), pp. 302–322
work page 2009
-
[3]
Living on the Edge: Phase Transitions in Convex Programs with Random Data
Dennis Amelunxen, Martin Lotz, Michael B. McCoy, and Joel A. Tropp. “Living on the Edge: Phase Transitions in Convex Programs with Random Data”. In:Information and Inference: A Journal of the IMA3.3 (2014), pp. 224–294.doi:10.1093/imaiai/iau005
-
[4]
Exact expressions for double descent and implicit regularization via surrogate random design
Micha l Derezi´ nski, Feynman Liang, and Michael W. Mahoney. “Exact expressions for double descent and implicit regularization via surrogate random design”. In:Advances in Neural Information Processing Systems. Vol. 33. 2020. 47
work page 2020
-
[5]
An investigation into neural net optimization via hessian eigenvalue density
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. “An investigation into neural net optimization via hessian eigenvalue density”. In:International Conference on Machine Learning. PMLR. 2019, pp. 2232–2241
work page 2019
-
[6]
On Milman’s Inequality and Random Subspaces which Escape through a Mesh in Rn
Yehoram Gordon. “On Milman’s Inequality and Random Subspaces which Escape through a Mesh in Rn”. In:Geometric Aspects of Functional Analysis. Vol. 1317. Lecture Notes in Mathematics. Springer, 1988, pp. 84–106
work page 1988
-
[7]
Improving Neural Network Training in Low Dimensional Random Bases
Frithjof Gressmann, Zach Eaton-Rosen, and Carlo Luschi. “Improving Neural Network Training in Low Dimensional Random Bases”. In:Advances in Neural Information Processing Systems. Vol. 33. 2020
work page 2020
-
[8]
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. “LoRA: Low-Rank Adaptation of Large Language Models”. In: International Conference on Learning Representations. 2022
work page 2022
Show all 24 references
-
[9]
NOLA: Compressing LoRA using Linear Combination of Random Basis
Soroush Abbasi Koohpayegani, K. L. Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. “NOLA: Compressing LoRA using Linear Combination of Random Basis”. In:arXiv preprint arXiv:2310.02556(2023).url:https://arxiv.org/abs/2310.02556
2023 arXiv
-
[10]
VeRA: Vector-based Random Matrix Adaptation
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. “VeRA: Vector-based Random Matrix Adaptation”. In:International Conference on Learning Representations. 2024
2024
-
[11]
How Many Degrees of Freedom Do We Need to Train Deep Networks: A Loss Landscape Perspective
Brett W. Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. “How Many Degrees of Freedom Do We Need to Train Deep Networks: A Loss Landscape Perspective”. In: International Conference on Learning Representations. 2022
2022
-
[12]
Fastfood - Computing Hilbert Space Ex- pansions in Loglinear Time
Quoc Le, Tamas Sarlos, and Alexander Smola. “Fastfood - Computing Hilbert Space Ex- pansions in Loglinear Time”. In:Proceedings of the 30th International Conference on Ma- chine Learning. Ed. by Sanjoy Dasgupta and David McAllester. Vol. 28. Proceedings of Machine Learning Res...
2013
-
[13]
Eigenvectors of some large sample covariance matrix ensembles
Olivier Ledoit and Sandrine P´ ech´ e. “Eigenvectors of some large sample covariance matrix ensembles”. In:Probability Theory and Related Fields151 (2011), pp. 233–264.doi: 10.1007/ s00440-010-0298-3
2011
-
[14]
Measuring the intrinsic dimension of objective landscapes
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. “Measuring the intrinsic dimension of objective landscapes”. In:International Conference on Learning Representations. 2018
2018
-
[15]
D´ ecomposition orthogonale d’un espace hilbertien selon deux cˆ ones mutuellement polaires
Jean Jacques Moreau. “D´ ecomposition orthogonale d’un espace hilbertien selon deux cˆ ones mutuellement polaires”. In:Comptes rendus hebdomadaires des s´ eances de l’Acad´ emie des sciences255 (1962), pp. 238–240
1962
-
[16]
PRANC: Pseudo RAndom Networks for Compacting Deep Models
Parsa Nooralinejad, Ali Abbasi, Soroush Abbasi Koohpayegani, Kossar Pourahmadi Meibodi, Rana Muhammad Shahroz Khan, Soheil Kolouri, and Hamed Pirsiavash. “PRANC: Pseudo RAndom Networks for Compacting Deep Models”. In:arXiv preprint arXiv:2206.08464(2022). url:https://arxiv.org...
2022 arXiv
-
[17]
Traces of class/cross-class structure pervade deep learning spectra
Vardan Papyan. “Traces of class/cross-class structure pervade deep learning spectra”. In: Journal of Machine Learning Research21.252 (2020), pp. 1–64
2020
-
[18]
Spectral convergence for a general class of random matrices
Francisco Rubio and Xavier Mestre. “Spectral convergence for a general class of random matrices”. In:Statistics & Probability Letters81.5 (2011), pp. 592–602. 48
2011
-
[19]
Empirical analysis of the hessian of over-parametrized neural networks
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. “Empirical analysis of the hessian of over-parametrized neural networks”. In:arXiv preprint arXiv:1706.04454 (2017)
2017 arXiv
-
[20]
Mapping Networks
Lord Sen and Shyamapada Mukherjee. “Mapping Networks”. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026
2026
-
[21]
Improved analysis of the subsampled randomized Hadamard transform
Joel A Tropp. “Improved analysis of the subsampled randomized Hadamard transform”. In: Advances in Adaptive Data Analysis3.01n02 (2011), pp. 115–126
2011
-
[22]
Cambridge Series in Statistical and Probabilistic Mathematics
Roman Vershynin.High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.doi:10.1017/9781108231596. A General Conic Intersection Theorem Throughout this appe...
2018 doi
-
[23]
Theorem2 (Random affine-slice intersection: conic phase transition).Let S⊂R P be nonempty, compact, and convex, let θ0 /∈S, and let C be the closed convex cone of Lemma 1
Taking expectations over g∼N (0,IP ) gives P =δ(C) +δ(C◦). Theorem2 (Random affine-slice intersection: conic phase transition).Let S⊂R P be nonempty, compact, and convex, let θ0 /∈S, and let C be the closed convex cone of Lemma 1. Let E⊂R P be 50 a uniformly random d-dimension...
-
[24]
Hence E[ρ] = 1− d k λkR2 P = λ(k−d)R 2 P ≤2ε⇐⇒d≥k− 2εP λR2. Deterministic-equivalent route.Equation (B.3) givesd =kλ/(λ +κ), i.e., κ =λ(k−d )/d, and the master formula (B.4) evaluates to E[ρ]≈ κλ λ+κ · kR2 P =λ k−d k · kR2 P = λ(k−d)R 2 P , in exact agreement. Solving (B.6) li...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.