REVIEW 4 major objections 5 minor 1 cited by
LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read LENSLLM predicts how each open-source LLM will fine-tune on a task from a few small-subset runs, and claims to pick the best model with up to 91.1% relative accuracy while cutting compute by up to 88.5%.
desk verdict The paper's key theoretical result doesn't survive scrutiny, but the NTK-augmented scaling-law idea is real and the empirical work is substantial enough to send to referees. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the PAC-Bayesian generalization bound for fine-tuned transformers (Theorem 2), which upper-bounds test loss by the empirical loss plus a Hessian-weighted term $\sqrt{C}\sum_i \sqrt{h_i}/\sqrt{n}$ plus $O(n^{-3/4})$. The argument uses Cauchy-Schwarz and a claimed Hessian scaling $h_i \le C_2 n^{-\beta_2}$ to compress this into $C_3 n^{-\beta_3}$, making the phase transition explicit. The practical selector is the NTK-based rectified scaling law $L(D) = B/(F(\Theta,t)+D^\beta)+E$, where $F(\Theta,t) = \|e^{-\eta\Theta t}(f_0(X)-y)\|_2^2$ is the residual error predicted by the neural tangent kernel (the kernel that governs gradient-descent dynamics in the wide-network limit) after $t$ fine-tuning steps; putting $F(\Theta,t)$ in the denominator models the competition between pre-trained knowledge and fine-tuning data. The algorithm fits $B$, $\beta$, $E$, and $t$ by minimizing a log-scale loss on dataset-size/test-loss pairs collected from progressively halved subsets.
What would settle it
For a fixed transformer and task, fine-tune on subsets of sizes $n = 200, 400, \dots, 1{,}638{,}400$ with early stopping, and plot the empirical trace of the Hessian (or gradient variance) against $n$ on log-log axes; if the plot is not a single straight line across the whole range, the $n^{-\beta_2}$ scaling and the phase transition derived from it fail.
Extended reading notes
Core claim
The paper's central claim is that the test loss of a fine-tuned LLM follows a two-phase scaling law whose shape can be derived from a PAC-Bayesian bound. The bound takes the form $L(f_{\hat w}) \le (1+\epsilon)\hat L(f_{\hat w}) + (1+\epsilon)\sqrt{C}\sum_{i=1}^l \sqrt{h_i}/\sqrt{n} + O(n^{-3/4})$, and once the Hessian-related terms are bounded as $h_i \le C_2 n^{-\beta_2}$, the bound compresses to $C_3 n^{-\beta_3}$ with $\beta_3 = (\beta_2+1)/2$. The crossing of this term with the $O(n^{-3/4})$ term marks the transition from the pre-power phase (small data, slow improvement) to the power phase (predictable, nearly linear log-log improvement). On this foundation the paper builds LENSLLM, whose rectified scaling law $L(D) = B/(F(\Theta,t)+D^\beta)+E$ places the NTK-based term $F(\Theta,t)$ in the denominator so that pre-trained initialization competes with dataset size. With the fitted curve, candidate models can be ranked without full fine-tuning; the paper reports up to 91.1% relative accuracy (the normalized performance gap between the selected and best model) and 85.8% Pearson correlation, at up to 88.5% lower computational cost than FullTuning.
Load-bearing premise
The load-bearing premise is that during fine-tuning, the variance of the gradient shrinks as a power of dataset size ($\operatorname{Var}(\nabla L) \propto n^{-\alpha}$), which turns the Hessian trace into $n^{-\beta_1}$; the paper supports this by citing a general result on SGD and asserting empirical agreement, not by a direct proof for fine-tuned transformers.
Editorial extensions
If this is right
- LENSLLM predicts a model's fine-tuned test loss on the full dataset from a few runs on small subsets, so practitioners can rank open-source LLM candidates before spending GPU hours on full fine-tuning.
- The derived bound identifies a transition point: below it, extra data yields slow $O(n^{-3/4})$ improvement (pre-power phase); above it, loss improves predictably as $C_3 n^{-\beta_3}$, so data collection and compute allocation can be planned.
- The fitted exponent $\beta$ and the offset $E$ give each model-task pair an interpretable learning-difficulty profile, replacing heuristic selection scores with parameters tied to the scaling law.
- Because the algorithm halves the training subset and stops when the fitted curve is stable, its total FLOP cost grows with the number of candidates times a small fraction of full data; the paper reports up to 88.5% savings over FullTuning.
Reading between the lines
- If the Hessian-trace scaling $h_i \propto n^{-\beta_2}$ holds across architectures, the phase-transition point should itself be predictable from small-$n$ runs, which could tell practitioners how many labeled examples to gather before fine-tuning; the paper does not test this directly.
- The additive form $F(\Theta,t)+D^\beta$ implies that pre-trained knowledge and fine-tuning data contribute to test loss independently; a testable extension is predicting loss under domain-mixed or augmented data by adjusting $\beta$ and $F$ separately.
- The same NTK-based rectified curve may apply to instruction tuning or multimodal fine-tuning, but only text benchmarks appear in the paper, so that scope is untested.
- A stronger validation would be to fit $B,\beta,E,t$ on one set of tasks and then evaluate ranking on held-out tasks never used in fitting; the paper does not report such a split.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes LENSLLM, a method for selecting which LLM to fine-tune for a downstream task. The authors derive a PAC-Bayesian generalization bound for fine-tuned transformers (Theorem 2), use it to motivate a two-phase (pre-power/power) scaling law (Corollary 1), and introduce an NTK-based rectified scaling model L(D)=B/(F(Theta,t)+D^beta)+E whose fitted parameters are used to rank candidate models. They evaluate on FLAN, Wikitext, and Gigaword across 14 model variants, reporting RMSE for curve fitting, PearCorr/RelAcc for selection, and FLOPs for efficiency.
Significance. If valid, the paper would supply a first-principled explanation for phase transitions in fine-tuning scaling laws and a computationally cheap model-selection score. The paper is clearly written, the model suite is broad, the code and results are released, and the ablation studies are thorough. However, the central theoretical derivation contains a serious gap, and the main empirical numbers are in-sample goodness-of-fit rather than out-of-sample predictions; the selection gains over Rectified Scaling Law are small (1.4-3.2 PearCorr points in Table 3).
major comments (4)
- [Appendix A, Eq. (14)] The O(n^{-3/4}) residual in Theorem 2 is asserted, not derived. The bracketed expression contains C2 sqrt(log(C3 n/δ)/n) Σ||∇_i H_i||_F (order n^{-1/2}√log n unless the Frobenius norms decay), (1+1/β)C1 Σ||∇_i H_i||_F^{3/2} (order 1 unless those norms decay), and C/(2β(1−β)n) log(1/δ) (order n^{-1}). No bound on Σ||∇_i H_i||_F or its powers is supplied. Consequently Eq. (4), Corollary 1 Eq. (7), and Remarks 1-3 rest on an unproved order estimate. Moreover, Theorem 2 is an upper bound; an upper bound containing O(n^{-3/4}) does not establish that the pre-power generalization error decays at rate n^{-3/4}, as Remark 1 claims.
- [Appendix B.2.1, Lemma 3] The claim Var(∇L(θ))∝n^{-α} is the key scaling input, but it is justified only by a citation to Smith & Le (2018) and an appeal to empirical observation. This is a load-bearing assumption that essentially assumes the power-law behavior the paper claims to derive. The chain tr(H)=n^{-1}Var(∇L(θ)) also relies on the stationary condition ∇θ L(θ*)=0 via Eqs. (23)-(24), which is not verified for fine-tuned transformers. Because Lemma 3 drives hi≤C2n^{-β2} and the n^{-β3} crossover in Corollary 1, the theoretical phase-transition result is unsupported as it stands.
- [Sections 4.1-4.2] The evaluation is circular for the headline claims. The parameters B, E, β, t are estimated by minimizing a fitting loss against the observed test losses L(D_i) on the same models and datasets, and Table 2 then reports RMSE between the fitted curve and exactly those observed losses; Table 3 uses the same fitted model to produce selection scores. These numbers are in-sample goodness-of-fit measures, not predictive accuracy on unseen data sizes or tasks. The paper should report held-out dataset sizes, cross-task transfer, or at least a train/validation split over the scaling curve.
- [Table 3] The reported improvements over Rectified Scaling Law are modest (PearCorr gains of 3.2, 2.3, and 2.3 points on FLAN, Wikitext, and Gigaword; RelAcc gains of 1.4-2.6 points). Given the extra NTK computation in Eq. (8) and the additional fitted parameter t, the paper should quantify whether these differences are statistically significant across seeds rather than presenting single-run percentages.
minor comments (5)
- [Table 2] Entries such as '0.21.10' and '0.321.50' appear to concatenate the two compared values with a period; this makes the table hard to read and should be reformatted.
- [Table 4] The symbol s is overloaded: it is used for the SubTuning proportion and also for the LENSLLM stopping proportion in Algorithm 1. Use separate notation for these two quantities.
- [Section 4.1] The heading 'Evaluation Matrics' should be 'Evaluation Metrics'.
- [Appendix B.1, Lemma 1] Lemma 1 uses n both as the dimension of the vector and as the sample size used elsewhere in the paper; in context this is confusing and should be replaced with l.
- [Appendix A, Proposition 2] The proof contains the typo 'wheer', and the step labeled 'after cancellation' is not shown; adding the intermediate algebra would improve readability.
Circularity Check
LENSLLM's headline test-loss 'predictions' and selection scores are in-sample fits: the scaling parameters are fitted to the same L(D_i) values that Table 2 and the selection metrics then evaluate, so the reported accuracy is a goodness-of-fit statistic rather than an out-of-sample prediction.
-
fitted input called prediction
[Section 4.1 (Optimization), Section 4.2 (Test Loss Prediction), and Algorithm 1]
"we estimate B,E,β,t for each model by minimizing the loss function: min_{B,E,β,t} Σ_i [ LSE( log B − log(F(Θ,t)+D_i^β), log E ) − log L(D_i) ] ... Test Loss Prediction: We evaluate the test loss prediction performance of LENSLLM. Table 2 demonstrates our model's superior performance through RMSE comparisons between predicted and actual test losses."
The fitting objective uses the very values log L(D_i) that Table 2 later labels 'actual test losses' and compares against the model's 'predicted' losses; the same fitted parameters also produce the selection scores in Table 3. The RMSE values and the 91.1% RelAcc / 85.8% PearCorr headline numbers therefore measure in-sample fit, not predictive accuracy on unseen dataset sizes. Algorithm 1 makes the construction explicit: it records 'Train m∈M on S ... Add(log|D|, log bL) to C' for the original dataset size D and then evaluates r=exp(ψ(log|D|)) at that same D, so the full-data loss being predicted and selected has already been inserted into the regression set. The reported 'prediction' thus reduces by construction to the fitted inputs.
full rationale
The central circularity is confined to the evaluation protocol. Section 4.1 fits B,E,β,t by minimizing the LSE discrepancy against log L(D_i) over the same dataset sizes whose test losses are then reported as 'predicted versus actual' in Table 2 and which feed the selection scores in Table 3. Algorithm 1 also inserts the full-size (log|D|, log bL) pair into C before halving, so the final score r=exp(ψ(log|D|)) is an interpolation of the full-data loss that the method is supposed to predict. Hence the low RMSE values and the 91.1% RelAcc / 85.8% PearCorr are in-sample residuals forced by the construction, not out-of-sample evidence about fine-tuning dynamics. A score of 6 rather than 8-10 is appropriate because the scaling-law functional form is not itself entailed by the fitted constants: the model could fail to fit other models or tasks, and the comparison against SubTuning/ZeroShot baselines is an external benchmark. The proof gap in Theorem 2 — where the final step declares 'the last part is equal to O(n^{-3/4})' although the displayed terms are O(n^{-1/2}√log n) and O(n^{-1}) — is a genuine mathematical error but not a circularity, so it is noted here without further raising the circularity score. No load-bearing self-citations or imported uniqueness theorems are present, and the cited scaling assumptions (Smith & Le 2018, Dauphin et al. 2024) are external inputs rather than self-referential conclusions.
Assumptions & free parameters
free parameters (6)
- B
- E
- beta
- t
- gamma
- tau
assumptions (5)
- standard math Assumption 1 (Smoothness): activation functions and loss are twice-differentiable with Lipschitz first and second derivatives.
- standard math Assumption 2 (Boundedness): loss and input features are uniformly bounded.
- domain assumption Assumption 3 (Transformer Stability): attention is Lipschitz, softmax scores are bounded, residual connections preserve gradient flow, and layer norm preserves scale.
- ad hoc to paper Lemma 3: during fine-tuning, the variance of the gradient scales as Var(gradient loss) ~ n^{-alpha}.
- ad hoc to paper The empirical scaling law L(D) = B / (F(Theta,t) + D^beta) + E holds for fine-tuned transformers.
Cite this review
Pith. "Pith review of LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection." pith.science (2026). https://pith.science/paper/GHXC5APM
@misc{pith2026250503793,
author = {Pith},
title = {Pith review of: LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHXC5APM}},
note = {Machine review of arXiv:2505.03793}
}
read the original abstract
The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a PAC-Bayesian Generalization Bound that unveils fine-tuning dynamics of LLMs and then introduce LENSLLM, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
Reference graph
Works this paper leans on
-
[1]
The KL divergence betweenPandQis: KL(Q∥P) = 1 2 lX i=1 (Wi−W (s) i )⊤Σ−1 i (Wi−W (s) i )(11)
-
[2]
For isotropic noise distribution at every layer (i.e.,Σi =σ 2 i Id): KL(Q∥P) = lX i=1 ∥Wi−W (s) i ∥2 F 2σ2 i (12) Proof. The proof follows from standard results on multivariate normal distributions with additional attention to transformer components. LetZi be the weight matrix of layeriin the posterior distribution. By definition of KL divergence: KL(Q∥P)...
-
[3]
Calculate the left side|⟨u,v⟩|: ⟨u,v⟩= nX i=1 ( p hi·1) = nX i=1 p hi
-
[4]
(4) wherehi≥max (x,y)∈D vT i H + i [L(f ˆw)]vi. 14 LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection Proof.First, we separate the gap betweenL(f ˆw)and 1 β ˆL(f ˆw)into three parts: L(f ˆw)− 1 β ˆL(f ˆw) =L(f ˆw)−L Q(f ˆw) +LQ(f ˆw)− 1 β ˆLQ(f ˆw) + 1 β ˆLQ(f ˆw)− 1 β ˆL(f ˆw).(13) By Taylor’s expansion, we can bound Equation 13 with respect to th...
-
[5]
Calculate∥v∥: ∥v∥= vuut nX i=1 12 =√n
-
[6]
Apply Cauchy-Schwarz: |⟨u,v⟩|≤∥u∥∥v∥ nX i=1 p hi≤ vuut nX i=1 hi·√n nX i=1 p hi≤ vuutn nX i=1 hi Therefore, we have proven thatPn i=1 √hi≤ p nPn i=1hi. Note:This proof assumes allh i are non-negative real numbers, which is aligned with the property ofhi in our bound. B.2. Property 2: Upper Bound ofh i We first prove that the sum of the trace of the Hessia...
-
[7]
First, recall the Cauchy-Schwarz inequality: For vectorsu,v∈R n,|⟨u,v⟩|≤∥u∥∥v∥
-
[8]
Let’s define our vectors: •u= ( √h1,√h2,..., √hn) •v= (1,1,...,1)(n-dimensional vector of ones)
Show all 19 references
-
[10]
Calculate∥u∥: ∥u∥= vuut nX i=1 ( p hi)2 = vuut nX i=1 hi
-
[13]
• The Hessian matrix for this loss function is defined as: H=∇ 2 θL(θ)(17) For each layerl, letH l be the Hessian of the loss function with respect to the parameters in that layer
Define the Objective Function and Hessian • LetL(θ)be the loss function of the model parameterized byθ. • The Hessian matrix for this loss function is defined as: H=∇ 2 θL(θ)(17) For each layerl, letH l be the Hessian of the loss function with respect to the parameters in that...
-
[14]
Proof.• Consider the empirical loss function: L(θ) = 1 n nX i=1 ℓ(θ;xi)(19) whereℓ(θ;x i)is the loss associated with samplex i
Scaling Behavior of the Hessian with Respect to Dataset Size Lemma 2.If L(θ) is the empirical loss over a dataset of size n, the trace of the Hessian matrix H scales astr(H) = n−1Var(∇L(θ)). Proof.• Consider the empirical loss function: L(θ) = 1 n nX i=1 ℓ(θ;xi)(19) whereℓ(θ;x...
2024
-
[15]
Proof.• Let the variance of the gradient during fine-tuning beσ 2(n)
The behavior of Variance During Fine-Tuning Lemma 3.During fine-tuning, the variance of the gradient scales as Var(∇L(θ))∝n α for some constantα. Proof.• Let the variance of the gradient during fine-tuning beσ 2(n). • Empirical observations and theoretical results from the lit...
2018
-
[16]
Let’s give tr(H) =C 1n−β1 as the conclusion of this statement
Combining Lemmas 1 and 2 • From Lemma 1: tr(H) =n−1σ2(n)(27) • Substituting the result from Lemma 2: tr(H)∝n −1·n−α =n−α−1 (28) The sum of the trace of the Hessian matrix across all layers is proportional to n−β, where β=α+ 1 . Let’s give tr(H) =C 1n−β1 as the conclusion of th...
-
[17]
The dimension of Wi isdi bydi−1, wheredi is the dimension of inputxi
Define the Objective Function and Weight Matrix • Let ˆW (s) be the weight matrices of pre-trained model and Wi be the dimension of layeri. The dimension of Wi isdi bydi−1, wheredi is the dimension of inputxi. • The distance-based regularization is defined as for every layer: ...
-
[18]
Upper Bound for∥W i− ˆW (s) i ∥2 F Lemma 4.There exists an upper bound for∥W i− ˆW (s) i ∥2 F , which is unrelated to the training data size: ∥Wi− ˆW (s) i ∥2 F≤B(33) Proof.• Trauger & Tewari (2023) has proved that for anyt∈N: ∥(W− ˆW)xt∥q q = kX j=1 ((W− ˆW xt)q≤kϵ q (34) • F...
2023
-
[20]
Specifically, the Pearson correlation drops from 78.14 (at average length 20) to 77.39 and 76.89 for lengths 18 and 22, respectively, while relative accuracy similarly declines
As shown in Table 11, both shorter and longer averages lead to slight decreases in Pearson correlation and relative accuracy. Specifically, the Pearson correlation drops from 78.14 (at average length 20) to 77.39 and 76.89 for lengths 18 and 22, respectively, while relative ac...
-
[635]
emnlp-main.635/
URL https://aclanthology.org/2020. emnlp-main.635/. Wang, H., Jing, B., Ding, K., Zhu, Y ., Cheng, W., Zhang, S., Fan, Y ., Zhang, L., and Zhou, D. Mastering long-tail complexity on graphs: Characterization, learning, and generalization, 2024a. URL https://arxiv.org/ abs/2305....
2019 arXiv
-
[2020]
cc/paper_files/paper/2020/file/ c46482dd5d39742f0bfd417b492d0e8e-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ c46482dd5d39742f0bfd417b492d0e8e-Paper. pdf. Lewis, M., Liu, Y ., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V ., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for na...
2020 arXiv
-
[2022]
Kaplan, J., McCandlish, S., Henighan, T., Brown, T
URL https://proceedings.mlr.press/ v162/ju22a.html. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URL https://arxiv.org/abs/2001. 08361. Kaplun, G., G...
2020 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.