{"id":"e833cbef-c83a-46da-b7df-189a521d6cae","arxiv_id":"2505.03793","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LENSLLM fits an NTK-augmented scaling law to small fine-tuning subsets to rank candidate LLMs, reporting up to 85.8% ranking correlation, up to 91.1% relative accuracy, and up to 88.5% compute reduction compared to full tuning.","lead":"This paper introduces LENSLLM, a method that predicts how well a large language model will perform after fine-tuning by training it on small data subsets and fitting a scaling curve. It is worth a look because cheaper, reasonably accurate model selection would save real compute in applied NLP, though the theoretical derivation has gaps and the gains over the closest baseline are modest.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's O(n^{-3/4}) residual is not derived: the proof's final step mislabels O(n^{-1}) and O(n^{-1/2}√log n) terms, so the phase-transition bound and Corollary 1 are unsupported.","rationale":"I read the paper as attempting a first-principled theory of fine-tuning phases and an efficient LLM selection method. The empirical idea (rectified scaling with an NTK-based offset) may be practically useful, and the reported ranking improvements over the Rectified Scaling Law baseline are plausible, but the theoretical derivation is presented as the paper's main novelty and it is not internally consistent. The reader identified Lemma 3 (gradient-variance scaling) as the weakest assumption. My concern is earlier and more fundamental: even if Lemma 3 were accepted, the O(n^{-3/4}) term in Theorem 2 is asserted, not derived. The proof's own Eq. (14) leaves terms of order O(n^{-1}) and O(n^{-1/2}√log n) that dominate n^{-3/4} as n grows, and no argument is given for their cancellation. This invalidates the phase-transition narrative in Remarks 1–3 and the specific functional form of Corollary 1, which the LENSLLM model (Eq. 10) claims to build on. There is also a conceptual gap: Theorem 2 is an upper bound, and the paper treats the bound's exponent as the actual decay rate of the generalization error. I therefore agree with the REJECT verdict, but I would ground it in the unproved O(n^{-3/4}) step rather than only in Lemma 3. The missing LogME comparison and the lack of error bars are additional concerns, but they are secondary to the broken theoretical chain. A rigorous re-derivation of Eq. (4) would settle the issue; if the residual terms cannot be bounded by O(n^{-3/4}), the theorem and its corollaries need substantial revision before the central claims can be accepted.","tokens_in":21936,"tokens_out":8115,"duration_ms":81688,"concrete_test":"Independently re-derive Eq. (4) from the proof in Appendix A, keeping every term in Eq. (14) and tracking the constant factors. In particular, expand the bracket labeled 'the last part' and compute its asymptotic order under Assumptions 1–3. If any term scales as n^{-1/2}√log n or n^{-1} without cancellation, then the theorem's O(n^{-3/4}) is false and Corollary 1/Remarks 1–3 must be restated. A quick numerical check: evaluate n^{-3/4}, n^{-1/2}√log n, and n^{-1} at n=10^4, 10^6, 10^8; the claimed term is orders of magnitude smaller than the unremoved terms, so the mislabeling is directly detectable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LENSLLM is grounded in a PAC-Bayesian bound that 'unveils fine-tuning dynamics' rests on Theorem 2 (Eq. 4) and Corollary 1 (Eq. 7). The proof of Theorem 2 in Appendix A breaks down at the final step. After Eq. (14), the bracketed expression contains (i) C2 sqrt(log(C3 n/δ)/n) times Σ∥∇_i H_i∥_F, which is O(n^{-1/2}√log n) unless Σ∥∇_i H_i∥_F decays as n^{-1/4}, and (ii) C/(2β(1−β)n) log(1/δ), which is O(n^{-1}). The text then states 'the last part is equal to O(n^{-3/4})' with no cancellation or additional assumption. Neither (i) nor (ii) is O(n^{-3/4}); no bound on Σ∥∇_i H_i∥_F is supplied. Thus Eq. (4) is not established. Since Remarks 1–3 and Corollary 1 use exactly the O(n^{-3/4}) term to define the pre-power phase and the n^{-β3} crossover, the theoretical foundation for LENSLLM's rectified scaling model is missing. Even granting Lemma 3's n^{-α} gradient-variance scaling, it appears only later in Appendix B and does not repair this step. Additionally, Theorem 2 is an upper bound; interpreting its O(n^{-3/4}) term as the actual decay rate in the pre-power phase is a further logical gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes LENSLLM, a method for selecting which LLM to fine-tune for a downstream task. The authors derive a PAC-Bayesian generalization bound for fine-tuned transformers (Theorem 2), use it to motivate a two-phase (pre-power/power) scaling law (Corollary 1), and introduce an NTK-based rectified scaling model L(D)=B/(F(Theta,t)+D^beta)+E whose fitted parameters are used to rank candidate models. They evaluate on FLAN, Wikitext, and Gigaword across 14 model variants, reporting RMSE for curve fitting, PearCorr/RelAcc for selection, and FLOPs for efficiency.","tokens_in":22307,"tokens_out":5699,"duration_ms":55403,"significance":"If valid, the paper would supply a first-principled explanation for phase transitions in fine-tuning scaling laws and a computationally cheap model-selection score. The paper is clearly written, the model suite is broad, the code and results are released, and the ablation studies are thorough. However, the central theoretical derivation contains a serious gap, and the main empirical numbers are in-sample goodness-of-fit rather than out-of-sample predictions; the selection gains over Rectified Scaling Law are small (1.4-3.2 PearCorr points in Table 3).","major_comments":[{"comment":"The O(n^{-3/4}) residual in Theorem 2 is asserted, not derived. The bracketed expression contains C2 sqrt(log(C3 n/δ)/n) Σ||∇_i H_i||_F (order n^{-1/2}√log n unless the Frobenius norms decay), (1+1/β)C1 Σ||∇_i H_i||_F^{3/2} (order 1 unless those norms decay), and C/(2β(1−β)n) log(1/δ) (order n^{-1}). No bound on Σ||∇_i H_i||_F or its powers is supplied. Consequently Eq. (4), Corollary 1 Eq. (7), and Remarks 1-3 rest on an unproved order estimate. Moreover, Theorem 2 is an upper bound; an upper bound containing O(n^{-3/4}) does not establish that the pre-power generalization error decays at rate n^{-3/4}, as Remark 1 claims.","section":"Appendix A, Eq. (14)"},{"comment":"The claim Var(∇L(θ))∝n^{-α} is the key scaling input, but it is justified only by a citation to Smith & Le (2018) and an appeal to empirical observation. This is a load-bearing assumption that essentially assumes the power-law behavior the paper claims to derive. The chain tr(H)=n^{-1}Var(∇L(θ)) also relies on the stationary condition ∇θ L(θ*)=0 via Eqs. (23)-(24), which is not verified for fine-tuned transformers. Because Lemma 3 drives hi≤C2n^{-β2} and the n^{-β3} crossover in Corollary 1, the theoretical phase-transition result is unsupported as it stands.","section":"Appendix B.2.1, Lemma 3"},{"comment":"The evaluation is circular for the headline claims. The parameters B, E, β, t are estimated by minimizing a fitting loss against the observed test losses L(D_i) on the same models and datasets, and Table 2 then reports RMSE between the fitted curve and exactly those observed losses; Table 3 uses the same fitted model to produce selection scores. These numbers are in-sample goodness-of-fit measures, not predictive accuracy on unseen data sizes or tasks. The paper should report held-out dataset sizes, cross-task transfer, or at least a train/validation split over the scaling curve.","section":"Sections 4.1-4.2"},{"comment":"The reported improvements over Rectified Scaling Law are modest (PearCorr gains of 3.2, 2.3, and 2.3 points on FLAN, Wikitext, and Gigaword; RelAcc gains of 1.4-2.6 points). Given the extra NTK computation in Eq. (8) and the additional fitted parameter t, the paper should quantify whether these differences are statistically significant across seeds rather than presenting single-run percentages.","section":"Table 3"}],"minor_comments":[{"comment":"Entries such as '0.21.10' and '0.321.50' appear to concatenate the two compared values with a period; this makes the table hard to read and should be reformatted.","section":"Table 2"},{"comment":"The symbol s is overloaded: it is used for the SubTuning proportion and also for the LENSLLM stopping proportion in Algorithm 1. Use separate notation for these two quantities.","section":"Table 4"},{"comment":"The heading 'Evaluation Matrics' should be 'Evaluation Metrics'.","section":"Section 4.1"},{"comment":"Lemma 1 uses n both as the dimension of the vector and as the sample size used elsewhere in the paper; in context this is confusing and should be replaced with l.","section":"Appendix B.1, Lemma 1"},{"comment":"The proof contains the typo 'wheer', and the step labeled 'after cancellation' is not shown; adding the intermediate algebra would improve readability.","section":"Appendix A, Proposition 2"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The central claim—that a PAC-Bayesian bound with an O(n^{-3/4}) residual explains pre-power and power phases in fine-tuning—is not backed by the proof. In Appendix A, the final step of Theorem 2 just asserts that the residual is O(n^{-3/4}). The terms in front of you are O(n^{-1/2}\\sqrt{\\log n}) (from the sum over \\|\\nabla_i H_i\\|_F) and O(n^{-1}) (from the KL term), with no bound on that sum and no cancellation. So Corollary 1 and Remarks 1–3, which all lean on the n^{-3/4} term, are unsupported. Lemma 3, which gives the gradient-variance scaling n^{-\\alpha}, is also asserted rather than proved, and the citation to Smith & Le (2018) does not directly cover fine-tuned transformers. This is a load-bearing flaw, not a minor gap.\n\nWhat the paper does well is the empirical side. The NTK-based rectified scaling law—L(D) = B/(F(\\Theta,t) + D^\\beta) + E—is a sensible modification of the Rectified Scaling Law, and the experiments show that this functional form fits observed test losses far better across 14 architectures and 3 datasets (roughly 4–5x lower RMSE). That is a genuine, reproducible result, even if it is goodness-of-fit rather than prediction.\n\nThe soft spots beyond the proof are the usual scaling-law peril: the parameters B, E, \\beta, t are fitted to the same test losses they then \"predict,\" so the reported RMSE and selection scores are in-sample. The gains over the Rectified Scaling Law baseline are small (1–3 points in PearCorr/RelAcc), no error bars are given, and LogME—a strong baseline they cite—is missing from the comparisons. The claimed 91.1% relative accuracy and 88.5% cost reduction should be read with that caveat. Code and results at LensLLM.io are promised but not in the manuscript, so the reproducibility artifacts cannot be checked.\n\nWho benefits? Researchers working on scaling laws for fine-tuning will find the functional form and the empirical comparisons useful, but not the theory. If the authors can either fix the proof or remove the theoretical framing and present this as an empirical scaling-law study, it would be a decent paper. As written, the overclaiming is too large to accept.\n\nRecommendation: send it to peer review—not desk reject. The empirical idea deserves referee time, and the flaws are potentially fixable. I would not accept it in its current form.","headline":"The paper's key theoretical result doesn't survive scrutiny, but the NTK-augmented scaling-law idea is real and the empirical work is substantial enough to send to referees.","tokens_in":22815,"tokens_out":2453,"would_cite":false,"duration_ms":27561,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LENSLLM predicts how each open-source LLM will fine-tune on a task from a few small-subset runs, and claims to pick the best model with up to 91.1% relative accuracy while cutting compute by up to 88.5%.","keywords":["LLM selection","fine-tuning dynamics","PAC-Bayesian generalization bound","neural tangent kernel","scaling laws","phase transition","test loss prediction","rectified scaling model"],"falsifier":"For a fixed transformer and task, fine-tune on subsets of sizes $n = 200, 400, \\dots, 1{,}638{,}400$ with early stopping, and plot the empirical trace of the Hessian (or gradient variance) against $n$ on log-log axes; if the plot is not a single straight line across the whole range, the $n^{-\\beta_2}$ scaling and the phase transition derived from it fail.","tokens_in":21710,"feed_emoji":"🎯","tokens_out":9836,"duration_ms":87779,"temperature":0.7,"pith_summary":"The paper tries to make LLM selection cheap and principled: instead of fully fine-tuning every candidate model on a downstream task, it predicts each model's fine-tuned test loss from a fitted scaling curve and picks the best. To justify that curve, it derives a PAC-Bayesian generalization bound for fine-tuned transformers that splits test loss into an early pre-power phase (loss decreasing as $O(n^{-3/4})$) and a later power phase (loss decreasing as $C_3 n^{-\\beta_3}$). It then introduces LENSLLM, a Neural Tangent Kernel (NTK)-based rectified scaling model $L(D) = B/(F(\\Theta,t)+D^\\beta)+E$, together with an algorithm that fine-tunes on progressively halved subsets until the fitted curve is stable. The paper reports that LENSLLM selects the best LLM with up to 91.1% ranking accuracy (the normalized performance gap to the best model) and 85.8% Pearson correlation across three benchmarks, while cutting computational cost by up to 88.5% compared to full fine-tuning of every candidate.","feed_headline":"One fitted curve picks the best LLM to fine-tune, 91% accurate","feed_subtitle":"Predicting fine-tuning loss from tiny data subsets cuts selection cost by up to 88.5%.","key_machinery":"The load-bearing object is the PAC-Bayesian generalization bound for fine-tuned transformers (Theorem 2), which upper-bounds test loss by the empirical loss plus a Hessian-weighted term $\\sqrt{C}\\sum_i \\sqrt{h_i}/\\sqrt{n}$ plus $O(n^{-3/4})$. The argument uses Cauchy-Schwarz and a claimed Hessian scaling $h_i \\le C_2 n^{-\\beta_2}$ to compress this into $C_3 n^{-\\beta_3}$, making the phase transition explicit. The practical selector is the NTK-based rectified scaling law $L(D) = B/(F(\\Theta,t)+D^\\beta)+E$, where $F(\\Theta,t) = \\|e^{-\\eta\\Theta t}(f_0(X)-y)\\|_2^2$ is the residual error predicted by the neural tangent kernel (the kernel that governs gradient-descent dynamics in the wide-network limit) after $t$ fine-tuning steps; putting $F(\\Theta,t)$ in the denominator models the competition between pre-trained knowledge and fine-tuning data. The algorithm fits $B$, $\\beta$, $E$, and $t$ by minimizing a log-scale loss on dataset-size/test-loss pairs collected from progressively halved subsets.","core_discovery":"The paper's central claim is that the test loss of a fine-tuned LLM follows a two-phase scaling law whose shape can be derived from a PAC-Bayesian bound. The bound takes the form $L(f_{\\hat w}) \\le (1+\\epsilon)\\hat L(f_{\\hat w}) + (1+\\epsilon)\\sqrt{C}\\sum_{i=1}^l \\sqrt{h_i}/\\sqrt{n} + O(n^{-3/4})$, and once the Hessian-related terms are bounded as $h_i \\le C_2 n^{-\\beta_2}$, the bound compresses to $C_3 n^{-\\beta_3}$ with $\\beta_3 = (\\beta_2+1)/2$. The crossing of this term with the $O(n^{-3/4})$ term marks the transition from the pre-power phase (small data, slow improvement) to the power phase (predictable, nearly linear log-log improvement). On this foundation the paper builds LENSLLM, whose rectified scaling law $L(D) = B/(F(\\Theta,t)+D^\\beta)+E$ places the NTK-based term $F(\\Theta,t)$ in the denominator so that pre-trained initialization competes with dataset size. With the fitted curve, candidate models can be ranked without full fine-tuning; the paper reports up to 91.1% relative accuracy (the normalized performance gap between the selected and best model) and 85.8% Pearson correlation, at up to 88.5% lower computational cost than FullTuning.","pith_inferences":["If the Hessian-trace scaling $h_i \\propto n^{-\\beta_2}$ holds across architectures, the phase-transition point should itself be predictable from small-$n$ runs, which could tell practitioners how many labeled examples to gather before fine-tuning; the paper does not test this directly.","The additive form $F(\\Theta,t)+D^\\beta$ implies that pre-trained knowledge and fine-tuning data contribute to test loss independently; a testable extension is predicting loss under domain-mixed or augmented data by adjusting $\\beta$ and $F$ separately.","The same NTK-based rectified curve may apply to instruction tuning or multimodal fine-tuning, but only text benchmarks appear in the paper, so that scope is untested.","A stronger validation would be to fit $B,\\beta,E,t$ on one set of tasks and then evaluate ranking on held-out tasks never used in fitting; the paper does not report such a split."],"forward_implications":["LENSLLM predicts a model's fine-tuned test loss on the full dataset from a few runs on small subsets, so practitioners can rank open-source LLM candidates before spending GPU hours on full fine-tuning.","The derived bound identifies a transition point: below it, extra data yields slow $O(n^{-3/4})$ improvement (pre-power phase); above it, loss improves predictably as $C_3 n^{-\\beta_3}$, so data collection and compute allocation can be planned.","The fitted exponent $\\beta$ and the offset $E$ give each model-task pair an interpretable learning-difficulty profile, replacing heuristic selection scores with parameters tied to the scaling law.","Because the algorithm halves the training subset and stops when the fitted curve is stable, its total FLOP cost grows with the number of candidates times a small fraction of full data; the paper reports up to 88.5% savings over FullTuning."],"supporting_citations":[{"why":"Supplies the power-law scaling law $L(N,D)$ and the $6ND$ FLOP cost model used to compare selection efficiency.","marker":"Kaplan et al. (2020)"},{"why":"Introduces the rectified scaling law for fine-tuning and the PearCorr/RelAcc metrics that LENSLLM is benchmarked against.","marker":"Haowei et al. (2024)"},{"why":"Gives the Hessian-based generalization bound for feed-forward networks that Theorem 2 extends to transformers.","marker":"Ju et al. (2022)"},{"why":"Defines the neural tangent kernel used to construct $F(\\Theta,t)$ in the LENSLLM scaling law.","marker":"Jacot et al. (2018)"},{"why":"The cited basis for Lemma 3's claim that SGD gradient variance scales as $n^{-\\alpha}$ during fine-tuning.","marker":"Smith & Le (2018)"},{"why":"Provides the relation $\\operatorname{tr}(H)=\\operatorname{Var}(\\nabla L)$ used to tie Hessian trace to gradient variance in Lemma 2.","marker":"Dauphin et al. (2024)"},{"why":"Supplies the norm bound on transformer weight perturbations used to make the $h_i$ upper bound independent of $n$.","marker":"Trauger & Tewari (2023)"}],"fun_headline_variants":["Tiny data subsets reveal which LLM fine-tunes best","One curve replaces full fine-tuning for LLM choice","91% accurate LLM pick from a single scaling curve","Fine-tuning dynamics explained by PAC-Bayesian bound","LLM selection: predict performance from tiny data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that during fine-tuning, the variance of the gradient shrinks as a power of dataset size ($\\operatorname{Var}(\\nabla L) \\propto n^{-\\alpha}$), which turns the Hessian trace into $n^{-\\beta_1}$; the paper supports this by citing a general result on SGD and asserting empirical agreement, not by a direct proof for fine-tuned transformers.","fun_headline_variants_meta":{"raw":{"variants":["Tiny data subsets reveal which LLM fine-tunes best","One curve replaces full fine-tuning for LLM choice","91% accurate LLM pick from a single scaling curve","Fine-tuning dynamics explained by PAC-Bayesian bound","LLM selection: predict performance from tiny data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1848,"prompt_tokens":1073,"completion_tokens":775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":689,"tokens_out":775,"duration_ms":7819,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:38:44.002943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a fixed transformer and task, fine-tune on subsets of sizes $n = 200, 400, \\dots, 1{,}638{,}400$ with early stopping, and plot the empirical trace of the Hessian (or gradient variance) against $n$ on log-log axes; if the plot is not a single straight line across the whole range, the $n^{-\\beta_2}$ scaling and the phase transition derived from it fail.","supporting_citations":[],"review_version":1}