REVIEW 3 major objections 4 minor 12 references
Gradient-free adaptation of language models is governed by the perturbation norm: a frozen frame of 12–16 scalars trails full-weight random search by 1.8 accuracy points, and no other dial—dimension, subspace, or content—survives once scale
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-05 00:00 UTC pith:QUQOIBGB
load-bearing objection A genuinely useful ablation showing that a frozen 12-16 scalar frame nearly matches full-weight gradient-free search, and that random frames tie once per-module Frobenius norm is matched; the strongest 'only the norm matters' claim overreaches the evidence. the 3 major comments →
Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that in gradient-free perturbation adaptation, the perturbation norm is the one factor that determines success or failure, while the search dimension and the choice of subspace are effectively inert. Across a 7×7 grid of models and benchmarks, perturbing only a frozen frame of 12–16 scalars recovers most of full-weight random search, trailing by 1.8 accuracy points on average and by 9.1 at worst. Capacity sweeps from 4 to over 1,100 scalars show no trend, and replacing the SVD frame with a random Gaussian matrix—whose Grassmann overlap with the SVD frame sits at chance level—matches the SVD frame within 0.4 points once a single scalar per module equalizes the Frobenius
What carries the argument
The central object is the frozen low-rank frame: each target weight matrix W is decomposed by a top-r SVD, W = U S V^T, and perturbed in place as W' = W + U S (vP) V^T, where v is a tied vector of about 12–16 scalars, P is a frozen projection, and U, S, V are frozen. The argument's load-bearing quantity is the perturbation norm σ, together with the per-module calibration scalar α_i = ||U_i S_i V_i^T||_F / ||M_i P_i||_F, which equalizes the scale of a random frame to that of the SVD frame. This norm matching is what allows the paper to separate the structural question (which subspace carries the perturbation) from the numeric question (how hard the weights are shaken), and it is the mechanism
Load-bearing premise
The whole conclusion rests on matching perturbation scale with one scalar per module, the ratio of Frobenius norms between the SVD reconstruction and the random lift; if the scale that matters is instead how much the model's outputs change, the frame-indifference result could be an artifact of that norm choice.
What would settle it
Run the frame-ablation comparison using per-module scale matching based on output sensitivity (for example, the KL divergence of layer outputs under unit perturbation) instead of Frobenius weight-norm ratios; if random frames then diverge from SVD frames in accuracy, the norm-only account fails. Alternatively, test the fixed σ=0.05 on a model family outside the seven studied; a collapse would break the transfer claim.
If this is right
- Gradient-free adaptation of language models can be run on a frozen frame of about a dozen scalars with only a small accuracy loss, so full-weight random perturbation is unnecessary.
- Any perturbation frame works once per-layer scale is calibrated; the SVD's only indispensable function is automatic scale bookkeeping, which a single scalar per module can replace.
- A fixed perturbation scale of σ=0.05 transfers across models from 0.5B to 8B and three model families, so per-model scale search is unnecessary within this regime.
- Because candidate scoring consumes 99.7% of wall-clock time, reducing the search dimension does not reduce cost; the practical lever is the number of scored candidates, not the number of perturbed parameters.
- Operating outside the safe σ window is the primary failure mode, and the SVD frame collapses first at large scales, so robust recipes should keep the norm inside the window rather than relying on subspace choice.
Where Pith is reading between the lines
- Editorial inference: If the norm is truly the only live factor, then many gradient-free adaptation failures in practice are likely scale mis-specification, and adaptive step-size methods may be implicitly solving a norm-calibration problem rather than a subspace-selection problem.
- Editorial inference: The Frobenius-norm matching used here may not be the only valid notion of "same perturbation scale"; matching output sensitivity or per-layer function change could either confirm or break the frame-indifference result, and would be a direct test of the paper's central metric choice.
- Editorial inference: The norm-centric account suggests that gradient-free adaptation could be made even simpler by replacing the SVD entirely with a random frame plus a single calibrated scalar, which would remove the one-time SVD initialization cost and simplify deployment on resource-constrained hardware.
- Editorial inference: The same safe-window structure may appear in other black-box perturbation settings, such as prompt or activation perturbation, but the transfer of the specific σ=0.05 value to those settings is untested and should not be assumed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Using a fixed RandOpt-style pipeline, the paper intervenes on four factors in gradient-free adaptation: search dimension, the SVD frame and its content, and the perturbation norm σ. The main empirical claims are that a frozen 12–16 scalar frame trails full-weight random search by an average of 1.8 accuracy points over a 7×7 model–benchmark grid; that dimension and frame/content do not explain the gap; and that the only factor with a failure mode is the perturbation norm, whose safe region transfers across seven models. The evidence includes paired tests on the 49-cell grid, frame ablations on a control cell, Grassmann distance measurements, and σ-collapse curves.
Significance. If substantiated, the paper’s conclusion would usefully redirect gradient-free adaptation from subspace design to scale selection and connect it with the low-dimensional adaptation literature. The experimental design is a genuine strength: one factor is changed at a time, scoring and voting are held fixed, paired statistics are reported with explicit intervals, and the limitations are stated honestly. The norm-collapse evidence in Figure 4 and the demonstration that raw random frames fail without the α correction are direct and valuable. However, the manuscript’s headline conclusions overstate the evidence in two load-bearing places: the dimension null is only an upper bound of about 2.7 points, and the “only live factor” claim is in tension with the frame-dependent collapse shown in the paper’s own Figure 4.
major comments (3)
- [§5–§6, Fig. 4, Eq. (2)] The unqualified claim that “the perturbation norm is therefore the one factor with a failure mode” (abstract, §6) conflicts with the paper’s own Figure 4. In that experiment, per-module Frobenius norms are matched via α_i from Eq. (2), yet the SVD frame collapses at smaller σ than the random frame. This is a frame- and content-dependent failure mode, which the §5 text acknowledges (“the SVD directions collapse first”). The framing also depends on Eq. (2) as the definition of “scale”; because the paper elsewhere adopts singular values as output-sensitivity coefficients (§2), matching Frobenius norms may not equate the function-space effect of different frames. Please either (i) test whether frame equivalence survives matching a data-dependent output-sensitivity quantity (e.g., mean logit/pre-activation change), or (ii) explicitly restrict the conclusion to the moderate-scale safe window a
- [§4, Table 3, paired test text] The sentence “Dimension is a dead dial” and the abstract’s claim that dimension does not explain the full-weight gap are not supported by the displayed statistics. The tiny-vs-large capacity test yields t=-1.19, 95% CI [-2.66,+0.65], which the text itself describes as excluding “capacity effects above about 2.7 points rather than establishing flatness.” A 2.7-point upper bound is of the same size as the 1.8-point main effect the paper is explaining, so the null hypothesis of no dimension effect is not established. Please replace “dead” and its abstract paraphrase with an explicit upper-bound statement, or provide additional capacity conditions that tighten the interval.
- [§5, Tables 4–5, Limitations] The frame/content equivalence is the backbone of the conclusion that “a random frame ... performs identically,” but it is measured on one control cell (Qwen2.5-0.5B/GSM8K, three seeds). The paper acknowledges this in Limitations, but the abstract generalizes to all models and benchmarks. The same caveat applies to Table 4: the tie is reported as point estimates within seed-standard-deviation overlap, with no paired test across seeds. At minimum, please replicate the random-frame condition on a second model/task, or explicitly scope the frame-equivalence claim to the control cell and present the grid results only as full-weight-vs-subspace comparisons.
minor comments (4)
- [Table 3 caption] The statistic “best of five subspace capacities wins or ties on 25 of 49 cells” compares the maximum of five subspace columns with a single full-weight column; this is not a paired comparison. Consider presenting it alongside the tiny-only statistic (13/49), which is already given, and labeling the best-of-five result as descriptive only.
- [Tables 10–11] Rows O11 and O12 are described as σ-grid shifts rather than fixed-σ changes. Please make this explicit in the captions to avoid confusion with the σ-window analysis in Section 6.
- [Figure 4] The “global α” curve is not defined in the caption. A one-sentence definition (the single ratio of module-averaged norms) would help readers who do not go to Appendix D.
- [References] Several references are to 2026 preprints (e.g., Gan and Isola 2026; Hazan et al. 2026). If these are not yet peer-reviewed, include arXiv IDs as is done for other preprints, so that the claims they support can be checked.
Circularity Check
No circularity: the frame ablations and σ-window claims are controlled empirical comparisons; α-matching is not fit to the target, and the paper directly addresses the output-sensitivity confound.
full rationale
The derivation chain is self-contained and empirically controlled. The central elimination (dimension, frame, content do not matter; norm does) rests on (i) the capacity sweep in Table 3, showing no trend from 4 to >1,100 scalars; (ii) the frame ablation in Table 4, where R1–R3 are matched to Base via α_i (Eq. 2), a per-module Frobenius-norm ratio computed from weight norms, not fit to target accuracy; (iii) the content swaps in Table 5; and (iv) the σ-collapse experiments in Figures 4–5. Matching α_i equalizes per-module ΔW Frobenius norms by construction, but the subsequent equality of accuracy is not forced: raw R2/R3 collapse to ~0 without α (Tables 14–15), and at large σ the SVD frame collapses before random ones (Figure 4), so the experiment has power to distinguish frames. The principal threat—that Frobenius norm is the wrong output-sensitivity scale—is explicitly addressed by the early-collapse result, which shows the measure is not merely a stand-in for functional sensitivity. No load-bearing step is justified by self-citation; references to TinyLoRA, RandOpt, LoRA-XS, and LottaLoRA provide baselines or prior frames, but the brunt of the argument is the paper's own ablations. The Limitations section candidly notes the single control cell for causal interventions and single-seed 49-cell grid; those are statistical limitations, not circularity. No prediction is a renamed fit: the fixed σ=0.05 suggestion is a descriptive summary of the measured safe window, not a parameter fitted to the same cells and then reported as independent.
Axiom & Free-Parameter Ledger
free parameters (4)
- sigma (perturbation scale) =
0.05 recommended; per-task optima 0.019-0.100; grid 0.01-1.0
- alpha_i (per-module norm correction) =
168 values, one per module, from Eq. 2
- K (vote size) =
chosen on test set from {1,10,40,50,100}, per cell
- N (population), n_fit (scoring set), r (rank), vdim (frame size) =
N=1000, n_fit=200, r=2, vdim_tiny=12-16
axioms (6)
- domain assumption Random search with score-and-vote on a 200-example train set is a valid adaptation protocol for the tested tasks.
- domain assumption Matching per-module Frobenius norms (Eq. 2) is the correct way to equalize perturbation scale across frames.
- domain assumption The control cell (Qwen2.5-0.5B / GSM8K) is representative of frame and content effects across all 7 models and 7 tasks.
- domain assumption SVD top singular directions of a layer are its highest-gain directions, so equal-norm perturbations shake the function hardest there.
- domain assumption Test-set selection of K is unbiased because it applies the same rule to every arm.
- domain assumption The sigma grid {0.01,0.05,0.1,0.5,1.0} spans the full usable window.
Cite this review
Pith. "Pith review of Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models." pith.science (2026). https://pith.science/paper/QUQOIBGB
@misc{pith2026260801624,
author = {Pith},
title = {Pith review of: Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/QUQOIBGB}},
note = {Machine review of arXiv:2608.01624}
}
read the original abstract
Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars. Gradient-free adaptation, which samples random weight perturbations and keeps the ones that score well, has not followed that trajectory and still perturbs every entry of the weight tensor. It is unknown whether that full-weight search is necessary, and more fundamentally which property of a perturbation makes it work at all, because existing methods vary the search space, the perturbation scale, and the aggregation together. We resolve this by intervening on one factor at a time inside a fixed pipeline, holding candidate scoring and voting constant while we vary the search dimension, the subspace that carries the perturbation, and its norm. Perturbing a frozen frame of 12 to 16 scalars stays 1.8 accuracy points behind full-weight search on average across 49 model-benchmark cells, trailing it in 36 of them. Neither the dimension nor the choice of basis explains that performance. A random frame whose Grassmann overlap with the SVD frame is at chance level performs identically once a single scale factor is matched, and at large scales the SVD directions collapse first. What survives is the perturbation norm, whose usable range closes within a factor of five across seven models and stays flat inside. The perturbation norm is therefore the one factor with a failure mode, and its safe region transfers across scale and family. The design question narrows from which subspace to perturb to how hard to shake.
Figures
Reference graph
Works this paper leans on
-
[1]
Intrin- sic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
Aghajanyan,A.;Gupta,S.;andZettlemoyer,L.2021. Intrin- sic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. InACL. Austin,J.;Odena,A.;Nye,M.;Bosma,M.;Michalewski,H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; and Sutton, C
work page 2021
-
[5]
In NeurIPS Datasets and Benchmarks
Measuring Mathematical Problem Solving with the MATH Dataset. In NeurIPS Datasets and Benchmarks. Hu,E.J.;Shen,Y.;Wallis,P.;Allen-Zhu,Z.;Li,Y.;Wang,S.; Wang,L.;andChen,W.2022. LoRA:Low-RankAdaptation of Large Language Models. InICLR. Kopiczko, D. J.; Blankevoort, T.; and Asano, Y. M
work page 2022
-
[6]
Malladi, S.; Gao, T.; Nichani, E.; Damian, A.; Lee, J
The Blessing of Dimensionality in LLMFine-tuning: AVariance-Curvature Perspective.arXiv preprint arXiv:2602.00170. Malladi, S.; Gao, T.; Nichani, E.; Damian, A.; Lee, J. D.; Chen,D.;andArora,S.2023. Fine-TuningLanguageModels with Just Forward Passes. InNeurIPS. Mania, H.; Guy, A.; and Recht, B
arXiv 2023
-
[7]
Learning to Reason in 13 Parameters.arXiv preprint arXiv:2602.04118. Mostafazadeh, N.; Chambers, N.; He, X.; Parikh, D.; Ba- tra, D.; Vanderwende, L.; Kohli, P.; and Allen, J
-
[10]
Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E
DeepSeekMath: Pushing the Limits of Mathemati- cal Reasoning in Open Language Models.arXiv preprint arXiv:2402.03300. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D
-
[14]
and train scores at noise across two decades ofσ (Table 15). This is the direct evidence for the claim of the subspacesectionofthemainpaperthatscalecalibration,not direction, is what the frame must supply. Variant∆WSVDvPRank / Transform BaseU S(vP)V ⊤ ✓ ✓r=2, top-rSVD R1v·U SV ⊤ ✓–r=2, top-rSVD R2v·M P–✓full-n, rawN(0,1) R3v·M– – full-n, rawN(0,1) Table 1...
work page 2023
-
[2017]
Schwaller,P.;Laino,T.;Gaudin,T.;Bolgar,P.;Hunter,C.A.; Bekas, C.; and Lee, A
Evolution Strategies as a Scalable Alternative to Reinforcement Learning.arXiv preprint arXiv:1703.03864. Schwaller,P.;Laino,T.;Gaudin,T.;Bolgar,P.;Hunter,C.A.; Bekas, C.; and Lee, A. A
-
[2021]
arXiv preprint arXiv:2108.07732
Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732. Bałazy, K.; Banaei, M.; Aberer, K.; and Tabor, J
-
[2023]
Self- Consistency Improves Chain of Thought Reasoning in Lan- guage Models. InICLR. Zhang,Q.;Chen,M.;Bukharin,A.;Karampatziakis,N.;He, P.;Cheng,Y.;Chen,W.;andZhao,T.2023.AdaLoRA:Adap- tive Budget Allocation for Parameter-Efficient Fine-Tuning. InICLR. Appendix A Extended Related Work and Method Definitions Table 8 organizes the methods of the Related Work...
work page 2023
-
[2024]
Salimans, T.; Ho, J.; Chen, X.; Sidor, S.; and Sutskever, I
Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115. Salimans, T.; Ho, J.; Chen, X.; Sidor, S.; and Sutskever, I
-
[2025]
LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. InECAI. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse,C.;andSchulman,J.2021. TrainingVerifierstoSolve Math Word Problems.arXiv preprint arXiv:2110.14168. Gan, Y.; and Isola, P
Pith/arXiv arXiv 2021
-
[2026]
A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need
A Little Rank Goes a Long Way: Random Scaffolds with LoRA Adapters Are All You Need.arXiv preprint arXiv:2604.08749. He,C.;Luo,R.;Bai,Y.;Hu,S.;Thai,Z.L.;Shen,J.;Hu,J.; Han, X.; Huang, Y.; Zhang, Y.; Liu, J.; Qi, L.; Liu, Z.; and Sun, M
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.