Pith. sign in

REVIEW 73 references

This paper argues that LoRA rank allocation can be reduced to a valid component-wise hypothesis test, and provides central limit theory for adaptive optimizers that makes the p-values asymptotically valid.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:28 UTC pith:WVYNFGHJ

load-bearing objection A real AdamW CLT and a sensible LoRA scoring idea, but the algorithm's data-dependent threshold invalidates the p-values that are the paper's central product.

arxiv 2607.20205 v1 pith:WVYNFGHJ submitted 2026-07-22 stat.ML cs.LGmath.STstat.TH

Statistical Inference for Rank Allocation in Low-Rank Adaptation

Yihang Gao , Vincent Y. F. Tan This is my paper
classification stat.ML cs.LGmath.STstat.TH MSC 62F0362F1262L2068T07
keywords LoRArank allocationhypothesis testingp-valuescentral limit theoremAdamWstochastic approximation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that deciding which rank-one LoRA components to keep or prune can be framed as a one-sided hypothesis test: the null hypothesis says the component's population-level score is at least a threshold Delta, and a small p-value indicates the component is too weak to retain. To make this test operational, the paper proves asymptotic normality for the trajectories of adaptive optimizers such as AdamW, treated as augmented stochastic approximation recursions, and then transfers that normality to averaged component scores via the delta method. A batch-means estimator of the long-run variance turns the theory into a practical p-value per component. Under a fixed rank budget, StatLoRA prunes the components with the smallest p-values. The authors verify empirically that p-value-based pruning is competitive with or better than fixed-rank LoRA and existing adaptive-rank scoring rules on language understanding, generation, and question answering tasks.

Core claim

The paper establishes that LoRA rank allocation can be reduced to a valid component-wise hypothesis testing problem. Each rank-one update b_j a_j^T is assigned a population score s* = ||a*||^2 ||b*||^2; the test is H0: s* >= Delta versus H1: s* < Delta. The empirical score is the product of squared norms of the current LoRA factors, and its averaged value is shown, under a central limit theorem for the augmented AdamW state, to be asymptotically normal. The resulting p-value Phi(sqrt(t)(avg_s - Delta)/sigma_hat) quantifies uncertainty in the empirical importance and drives a budget-constrained pruning rule. The central limit theory is built on representing AdamW as a stochastic approximation

What carries the argument

The load-bearing machinery is the augmented-state stochastic approximation representation of AdamW: the algorithm is written as z_{t+1}=z_t + gamma_t(F(z_t)+xi_t+r_t) with z_t=(x_t,m_t,v_t). Under assumptions of almost-sure convergence to a fixed point whose linearization is attracting, the paper proves a central limit theorem for the iterates and, after averaging, obtains a Gaussian limit with covariance given by a linear matrix equation JSigma+Sigma J^top+Q=0. The delta method then transfers this to the component score h(a,b)=||a||^2 ||b||^2, and a batch-means estimator of the long-run variance yields the p-value Phi(sqrt(t)(avg_s - Delta)/sigma_hat) for the one-sided test H0:s*>=Delta.

Load-bearing premise

The p-values are valid only if the optimizer's augmented state converges almost surely to a fixed point whose linearization is attracting, and the paper only verifies this condition theoretically under strong convexity and small-gradient assumptions, not for the non-convex language-model losses tested.

What would settle it

Run StatLoRA on a synthetic LoRA problem with a known population score s* equal to the threshold Delta; under the null, the reported p-values should be approximately uniform. If they are not, either the central limit theorem, the batch-means variance estimate, or the assumption of a fixed-point attractor is failing in that regime. Equivalently, choose a deliberately non-convex loss with a saddle point near the limit and check whether the averaged-score Q-Q plot breaks down.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Rank allocation under a fixed parameter budget becomes a decision rule with explicit uncertainty quantification: components with the weakest statistical evidence are pruned first, and the p-value is interpretable as the chance of observing a score that low under the null.
  • The central limit theory for AdamW, Adam, and Adafactor means that other statistics derived from optimizer trajectories (e.g., gradient sensitivity, importance measures) can in principle be tested rather than ranked heuristically.
  • Because the method uses only quantities already computed during standard LoRA fine-tuning, it can be dropped into existing pipelines without changing the loss, model parameterization, or optimizer dynamics.
  • Empirically, the p-value-based rule matches or exceeds fixed-rank LoRA and existing adaptive-rank methods across NLU, NLG, and QA benchmarks under matched rank budgets, and the rank allocations are stable across random seeds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same testing template could be applied to any parameter-efficient method whose update decomposes into rank-one components, such as adapter layers or prompt tuning, as long as the component scores are smooth functions of optimizer states.
  • The paper's threshold is chosen adaptively as a quantile of the current score distribution; this creates a data-dependent null that the theory (which assumes a fixed Delta) does not fully cover, so the resulting p-values are diagnostics rather than exact error-control guarantees. A fixed pre-specified Delta would be needed for formal multiple-testing corrections.
  • One direct testable extension: vary the threshold Delta while holding the budget fixed; if the theory is right, the selected modules should change smoothly and the final performance should be insensitive over a range of Deltas.
  • The CLT relies on a step-size schedule gamma_t = gamma_0 t^{-kappa} with kappa in (1/2,1); standard fine-tuning typically uses cosine or constant schedules, so a practical adaptation would be to check whether the asymptotics survive with more common schedules in finite-sample regimes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Circularity Check

2 steps flagged

StatLoRA's p-values are constructed from a data-dependent threshold—the q-quantile of the same scores—so the advertised hypothesis test and its pruning 'evidence' are circular.

specific steps
  1. self definitional [Section 4.3 and Algorithm 1 Step 7; cf. Section 2.2 Eq. (8) and Section 4.2 Eq. (31)]
    "Rather than fixing an absolute threshold, we choose it adaptively from the empirical score distribution. ... ∆(t) = Quantile_q({¯s_ℓ,j(t) : (ℓ, j)∈ A_t})."

    The testing theory is stated for a pre-specified ∆: H0: s*_ℓ,j ≥ ∆, with p-value Φ(√t(¯s_ℓ,j(t)−∆)/σ̂). Algorithm 1 then sets ∆(t) to the q-quantile of the very same averaged scores used in the test statistic. The null hypothesis and the centering are therefore functions of the data under test, so under the null the statistic is not standard normal and Φ(T) is not a valid p-value. The procedure reduces to a data-defined quantile ranking of the scores, not an externally anchored hypothesis test.

  2. fitted input called prediction [Section 5.5.3 and Algorithm 1 Step 9]
    "As shown in Figure 3, the p-values of the pruned components are concentrated near zero. In particular, 76.9% of the removed components have p-values below 0.05. This pattern is consistent with the intended statistical interpretation of StatLoRA."

    Algorithm 1 prunes 'components with the smallest p-values according to the budget.' The low p-values in the pruned set are therefore entailed by the selection rule: components were removed precisely because their p-values were smallest. Reporting that pruned components have small p-values as evidence for the validity of the testing procedure reverses cause and effect; the diagnostic is a tautology and cannot confirm the hypothesis-testing interpretation.

full rationale

The paper's CLT machinery (Theorem 1-3, Proposition 1-2) is a genuine independent derivation: it starts from explicit regularity assumptions and proves asymptotic normality; it is not fitted to produce a target allocation. There is no load-bearing self-citation and no imported uniqueness theorem. However, the central inferential claim is undermined by a self-referential construction: the algorithm replaces the fixed threshold ∆, for which the p-value theory is developed, with the empirical q-quantile of the same scores used to compute the test statistics. This makes the null random and correlated with the data, so the p-values are not valid uncertainty-quantified evidence; they encode relative position in the empirical score distribution. The diagnostic in §5.5.3 then cites the necessarily small p-values of pruned components as validation, which is another circular step. Because the asymptotic theory itself is non-circular but the advertised p-value-based allocation rule reduces to a data-dependent quantile ranking, the appropriate score is 6 rather than higher.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The method assumes strong stochastic-approximation conditions (convergence, Hurwitz Jacobian, moment bounds) and introduces a data-dependent threshold that is not part of the derived theory. No new physical entities are postulated; the population score s* is a conventional target parameter assuming convergence.

free parameters (6)
  • step-size scale γ0 = chosen so learning-rate scale matches AdaLoRA (Section 5.1)
    Constants in γ_t=γ0 t^{-κ}; tuned per task, not derived.
  • step-size exponent κ = 0.51
    Chosen in (1/2,1) to satisfy CLT; not independently motivated.
  • momentum rates α, β = selected to match AdaLoRA effective update magnitudes
    AdamW variant uses α_t=αγ_t, β_t=βγ_t; nonstandard decaying momentum.
  • batch-means batch size b = 120 (NLU), 100 (NLG, QA)
    Hyperparameter; sensitivity reported only on SST-2.
  • threshold quantile q = 0.3
    Adaptive Δ(t) set to q-quantile of current scores; chosen after SST-2 sensitivity analysis.
  • adaptive threshold Δ(t) = q-quantile of empirical score distribution at time t
    Data-dependent null hypothesis; changes every pruning step, invalidating fixed-Δ p-value theory.
axioms (5)
  • domain assumption Almost sure convergence of augmented iterates z_t to z* (Assumption 2 B1)
    Required for all CLTs; not established for non-convex fine-tuning.
  • domain assumption Jacobian of AdamW mean field is Hurwitz (Assumption 2 B5, Prop. 1)
    Needs ∇²f(x*)≻0 and small ∥∇f(x*)∥; unlikely in LLM fine-tuning.
  • domain assumption Batch-means estimator consistency conditions (FCLT, mixing) hold for score process (Prop. 2)
    Not proven for the score process; only stated as standard.
  • ad hoc to paper No zero-score boundary: s*≥Δ>0 (Theorem 3)
    Excludes degenerate delta method case; but adaptive Δ can be near zero.
  • domain assumption Martingale-difference gradient noise with (4+δ) moments (Assumption 2 B3)
    Needed to handle squared gradient noise in the second-moment update.

pith-pipeline@v1.3.0-alltime-deepseek · 49124 in / 16061 out tokens · 142566 ms · 2026-08-01T10:28:18.958271+00:00 · methodology

0 comments
read the original abstract

Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization. Existing adaptive rank methods address this problem mainly through carefully designed importance scores constructed from gradient-derived sensitivity and uncertainty measures, without an explicit statistical interpretation. In this paper, we formulate LoRA rank allocation as a statistical hypothesis testing problem and propose StatLoRA, a statistical inference-based rank allocation method. StatLoRA associates each LoRA component with a test statistic and uses estimated p-values to determine which components should be retained or pruned under a prescribed rank budget. The proposed testing procedure is supported by our central limit theory for stochastic optimizer trajectories. In particular, we establish asymptotic normality for a broad class of commonly used optimizers in deep learning, including AdamW, and derive the corresponding asymptotic distributions for the proposed component scores used in hypothesis testing. We evaluate StatLoRA on LoRA fine-tuning of DeBERTaV3-base, BART-Large, and Qwen2.5-7B across natural language understanding, natural language generation, and question answering tasks. Experiments show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and IGU-LoRA under matched rank budgets. Sensitivity analyses and empirical diagnostics further support the stability of the proposed hypothesis-testing-based allocation rule and provide empirical evidence for the asymptotic theory of component scores.

Figures

Figures reproduced from arXiv: 2607.20205 by Vincent Y. F. Tan, Yihang Gao.

Figure 1
Figure 1. Figure 1: Sensitivity analysis of StatLoRA on the SST-2 dataset in the NLU task. Each setting is [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stability analysis of the selected rank allocation produced by StatLoRA on OpenBookQA [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of p-values for pruned LoRA components. The red dashed line indicates the [PITH_FULL_IMAGE:figures/full_fig_p029_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Q–Q plot for the standardized averaged score statistic of a representative LoRA compo [PITH_FULL_IMAGE:figures/full_fig_p030_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 6 linked inside Pith

  1. [1]

    A polyak-ruppert central limit theorem for SA-Adam with momentum and non-convergent adaptive preconditioning.arXiv preprint arXiv:2606.17364, 2026

    Sunyoung An and Xiaoming Huo. A polyak-ruppert central limit theorem for SA-Adam with momentum and non-convergent adaptive preconditioning.arXiv preprint arXiv:2606.17364, 2026

    Pith/arXiv arXiv 2026

  2. [2]

    Convergence and dynamical behavior of the ADAM algo- rithm for nonconvex stochastic optimization.SIAM Journal on Optimization, 31(1):244–274, 2021

    Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the ADAM algo- rithm for nonconvex stochastic optimization.SIAM Journal on Optimization, 31(1):244–274, 2021

    2021

  3. [3]

    Weight uncer- tainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncer- tainty in neural network. InInternational Conference on Machine Learning, pages 1613–1622. PMLR, 2015

    2015

  4. [4]

    Springer, 2008

    Vivek S Borkar and Vivek S Borkar.Stochastic approximation: a dynamical systems viewpoint, volume 100. Springer, 2008

    2008

  5. [5]

    Language mod- els are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    1901

  6. [6]

    Convergence diagnostics for stochastic gradient descent with constant learning rate

    Jerry Chee and Panos Toulis. Convergence diagnostics for stochastic gradient descent with constant learning rate. In Amos Storkey and Fernando Perez-Cruz, editors,Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84, pages 1476–1485. PMLR, 09–11 Apr 2018

    2018

  7. [7]

    Statistical inference for online decision making via stochastic gradient descent.Journal of the American Statistical Association, 116(534):708– 719, 2021

    Haoyu Chen, Wenbin Lu, and Rui Song. Statistical inference for online decision making via stochastic gradient descent.Journal of the American Statistical Association, 116(534):708– 719, 2021

    2021

  8. [8]

    Online statistical inference for stochastic optimization via kiefer-wolfowitz methods.Journal of the American Statistical Association, 119(548):2972–2982, 2024

    Xi Chen, Zehua Lai, He Li, and Yichen Zhang. Online statistical inference for stochastic optimization via kiefer-wolfowitz methods.Journal of the American Statistical Association, 119(548):2972–2982, 2024

    2024

  9. [9]

    Lee, Xin T

    Xi Chen, Jason D. Lee, Xin T. Tong, and Yichen Zhang. Statistical inference for model param- eters in stochastic gradient descent.The Annals of Statistics, 48(1), February 2020

    2020

  10. [10]

    Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  11. [11]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ...

    2019

  12. [12]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018. 32

    Pith/arXiv arXiv 2018

  13. [13]

    Markov chain Monte Carlo convergence diagnos- tics: a comparative review.Journal of the American Statistical Association, 91(434):883–904, 1996

    Mary Kathryn Cowles and Bradley P Carlin. Markov chain Monte Carlo convergence diagnos- tics: a comparative review.Journal of the American Statistical Association, 91(434):883–904, 1996

    1996

  14. [14]

    IGU-loRA: Adaptive rank allocation via integrated gradients and uncertainty-aware scoring

    Xuan Cui, HuiYue Li, Run Zeng, Yunfei Zhao, Jinrui Qian, Wei Duan, Bo Liu, and Zhanpeng Zhou. IGU-loRA: Adaptive rank allocation via integrated gradients and uncertainty-aware scoring. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  15. [15]

    Asymptotic normality and optimality in nonsmooth stochastic approximation.The Annals of Statistics, 52(4):1485–1508, 2024

    Damek Davis, Dmitriy Drusvyatskiy, and Liwei Jiang. Asymptotic normality and optimality in nonsmooth stochastic approximation.The Annals of Statistics, 52(4):1485–1508, 2024

    2024

  16. [16]

    Central limit theorem for the averaged Adam optimizer

    Steffen Dereich and Arnulf Jentzen. Central limit theorem for the averaged Adam optimizer. arXiv preprint arXiv:2606.21433, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), pages 4171–4186, 2019

    2019

  18. [18]

    Bara: Bayesian adaptive rank allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2606.29184, 2026

    Zhibin Duan, Yuhong Wang, Jiahong Fu, Zongsheng Yue, Bo Chen, and Zongben Xu. Bara: Bayesian adaptive rank allocation for parameter-efficient fine-tuning.arXiv preprint arXiv:2606.29184, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Asymptotic optimality in stochastic optimization.The Annals of Statistics, 49(1):21–48, 2021

    John C Duchi and Feng Ruan. Asymptotic optimality in stochastic optimization.The Annals of Statistics, 49(1):21–48, 2021

    2021

  20. [20]

    Online bootstrap confidence intervals for the stochastic gradient descent estimator.Journal of Machine Learning Research, 19(78):1–21, 2018

    Yixin Fang, Jinfeng Xu, and Lei Yang. Online bootstrap confidence intervals for the stochastic gradient descent estimator.Journal of Machine Learning Research, 19(78):1–21, 2018

    2018

  21. [21]

    Batch means and spectral variance estimators in Markov chain Monte Carlo.The Annals of Statistics, pages 1034–1070, 2010

    James M Flegal and Galin L Jones. Batch means and spectral variance estimators in Markov chain Monte Carlo.The Annals of Statistics, pages 1034–1070, 2010

    2010

  22. [22]

    Estimating the asymptotic variance with batch means.Oper- ations Research Letters, 10(8):431–435, 1991

    Peter W Glynn and Ward Whitt. Estimating the asymptotic variance with batch means.Oper- ations Research Letters, 10(8):431–435, 1991

    1991

  23. [23]

    MIT Press, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning. MIT Press, 2016

    2016

  24. [24]

    DeBERTaV3: Improving DeBERTa using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using electra-style pre-training with gradient-disentangled embedding sharing. InThe Eleventh In- ternational Conference on Learning Representations, 2023

    2023

  25. [25]

    Teaching machines to read and comprehend.Advances in Neural Information Processing Systems, 28, 2015

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend.Advances in Neural Information Processing Systems, 28, 2015

    2015

  26. [26]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InInternational Conference on Machine Learning, pages 2790–2799. PMLR, 2019

    2019

  27. [27]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInter- national Conference on Learning Representations, 2022. 33

    2022

  28. [28]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInterna- tional Conference on Learning Representations, 2015

    2015

  29. [29]

    Online covariance matrix estimation in sketched Newton methods.arXiv preprint arXiv:2502.07114, 2025

    Wei Kuang, Mihai Anitescu, and Sen Na. Online covariance matrix estimation in sketched Newton methods.arXiv preprint arXiv:2502.07114, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Springer, 2003

    Harold J Kushner and G George Yin.Stochastic approximation and recursive algorithms and applications. Springer, 2003

    2003

  31. [31]

    BioBERT: a pre-trained biomedical language representation model for biomed- ical text mining.Bioinformatics, 36(4):1234–1240, 2020

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomed- ical text mining.Bioinformatics, 36(4):1234–1240, 2020

    2020

  32. [32]

    Asymptotic analysis of conditioned stochastic gradient de- scent.Transactions on Machine Learning Research, 2023

    R ´emi Leluc and Franc ¸ois Portier. Asymptotic analysis of conditioned stochastic gradient de- scent.Transactions on Machine Learning Research, 2023

    2023

  33. [33]

    BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. InProceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 7871–7880, 2020

    2020

  34. [34]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, pages 19730–19742. PMLR, 2023

    2023

  35. [35]

    Statistical inference using SGD

    Tianyang Li, Liu Liu, Anastasios Kyrillidis, and Constantine Caramanis. Statistical inference using SGD. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  36. [36]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021

    2021

  37. [37]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  38. [38]

    ALoRA: Allocating low- rank adaptation for fine-tuning large language models

    Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. ALoRA: Allocating low- rank adaptation for fine-tuning large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pages 622–641, 2024

    2024

  39. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  40. [40]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools.Nature Machine Intelli- gence, 6(5):525–535, 2024. 34

    2024

  41. [41]

    A kernel- based view of language model fine-tuning

    Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel- based view of language model fine-tuning. InInternational Conference on Machine Learning, pages 23610–23641. PMLR, 2023

    2023

  42. [42]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018

    2018

  43. [43]

    Derivative-free sequential quadratic programming for equality-constrained stochastic optimization.arXiv preprint arXiv:2510.22458, 2025

    Sen Na. Derivative-free sequential quadratic programming for equality-constrained stochastic optimization.arXiv preprint arXiv:2510.22458, 2025

    arXiv 2025

  44. [44]

    Statistical inference of constrained stochastic optimization via sketched sequential quadratic programming.Journal of Machine Learning Research, 26(33):1– 75, 2025

    Sen Na and Michael Mahoney. Statistical inference of constrained stochastic optimization via sketched sequential quadratic programming.Journal of Machine Learning Research, 26(33):1– 75, 2025

    2025

  45. [45]

    Ab- stractive text summarization using sequence-to-sequence RNNs and beyond

    Ramesh Nallapati, Bowen Zhou, Cicero Dos Santos, C ¸ a˘glar Gulc ¸ehre, and Bing Xiang. Ab- stractive text summarization using sequence-to-sequence RNNs and beyond. InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, 2016

    2016

  46. [46]

    Don’t give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797– 1807, 2018

    2018

  47. [47]

    Springer Science & Business Media, 2012

    Radford M Neal.Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012

    2012

  48. [48]

    The asymptotic variance of semiparametric estimators.Econometrica: Journal of the Econometric Society, pages 1349–1382, 1994

    Whitney K Newey. The asymptotic variance of semiparametric estimators.Econometrica: Journal of the Econometric Society, pages 1349–1382, 1994

    1994

  49. [49]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992

    1992

  50. [50]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763. PMLR, 2021

    2021

  51. [51]

    A stochastic approximation method.The Annals of Math- ematical Statistics, pages 400–407, 1951

    Herbert Robbins and Sutton Monro. A stochastic approximation method.The Annals of Math- ematical Statistics, pages 400–407, 1951

    1951

  52. [52]

    Efficient estimations from a slowly convergent Robbins-Monro process

    David Ruppert. Efficient estimations from a slowly convergent Robbins-Monro process. Tech- nical report, Cornell University Operations Research and Industrial Engineering, 1988

    1988

  53. [53]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. InInternational Conference on Machine Learning, pages 4596–4604. PMLR, 2018

    2018

  54. [54]

    Large language mod- els encode clinical knowledge.Nature, 620(7972):172–180, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language mod- els encode clinical knowledge.Nature, 620(7972):172–180, 2023. 35

    2023

  55. [55]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4149–4158, 2019

    2019

  56. [56]

    Statistical analysis of stochastic gradient methods for generalized linear models

    Panagiotis Toulis, Edoardo Airoldi, and Jason Rennie. Statistical analysis of stochastic gradient methods for generalized linear models. InInternational Conference on Machine Learning, pages 667–675. PMLR, 2014

    2014

  57. [57]

    Panos Toulis and Edoardo M. Airoldi. Asymptotic and finite-sample properties of estimators based on stochastic gradients.The Annals of Statistics, 45(4), August 2017

    2017

  58. [58]

    DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adap- tation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adap- tation. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3274–3287, 2023

    2023

  59. [59]

    Vapnik.Statistical Learning Theory

    Vladimir N. Vapnik.Statistical Learning Theory. Wiley, 1998

    1998

  60. [60]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Infor- mation Processing Systems, volume 30, pages 5998–6008, 2017

    2017

  61. [61]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, 2018

    2018

  62. [62]

    Inference of online newton methods with nesterov’s accelerated sketching

    Haoxuan Wang, Xinchen Du, and Sen Na. Inference of online newton methods with nesterov’s accelerated sketching. InInternational Conference on Machine Learning, 2026

    2026

  63. [63]

    Bayesian low-rank adap- tation for large language models

    Adam Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adap- tation for large language models. InInternational Conference on Learning Representations, volume 2024, pages 1812–1842, 2024

    2024

  64. [64]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    The expressive power of low-rank adaptation

    Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. InInterna- tional Conference on Learning Representations, volume 2024, pages 5078–5123, 2024

    2024

  66. [66]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. InInternational Conference on Learning Representations, 2023

    2023

  67. [67]

    AutoLoRA: Automat- ically tuning matrix ranks in low-rank adaptation based on meta learning

    Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. AutoLoRA: Automat- ically tuning matrix ranks in low-rank adaptation based on meta learning. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 5048–5060, 2024. 36

    2024

  68. [68]

    Asymmetry in low-rank adapters of foundation models

    Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz S ´aez De Oc ´ariz Borde, Rickard Br ¨uel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berk...

    2024

  69. [69]

    Online covariance matrix estimation in stochastic gradient descent.Journal of the American Statistical Association, 118(541):393–404, July 2021

    Wanrong Zhu, Xi Chen, and Wei Biao Wu. Online covariance matrix estimation in stochastic gradient descent.Journal of the American Statistical Association, 118(541):393–404, July 2021. 37 A Proofs of Main Theorems A.1 Preparation Lemmas We first present several auxiliary lemmas that are used in the proofs of the main theorems. Their proofs are deferred to ...

    2021

  70. [70]

    The pointz ⋆ = (x⋆,0,q(x ⋆))satisfies the Adam mean-field equationF(z ⋆) =0, wherex ⋆ is a stationary point of the objective function

  71. [71]

    Then, the Jacobian matrix atz ⋆ admits J=   0−diag 1√q⋆+ε 0 α∇2f(x ⋆)−αI d 0 β∇q(x ⋆)0−βI d   ,(45) where we denoteq(x ⋆)asq ⋆

    Assume thatq(x)is differentiable in a neighborhood ofx ⋆. Then, the Jacobian matrix atz ⋆ admits J=   0−diag 1√q⋆+ε 0 α∇2f(x ⋆)−αI d 0 β∇q(x ⋆)0−βI d   ,(45) where we denoteq(x ⋆)asq ⋆. Moreover,Jis Hurwitz if∇ 2f(x ⋆)≻0. Assumption 3.For the Adam recursion(40), we impose the following conditions. (C1)Almost sure convergence.The augmented iteratez t =...

  72. [72]

    The pointz ⋆ = (x⋆,v ⋆),satisfies the Adafactor mean-field equationF(z ⋆) =0, where ∇f(x ⋆) =0,v ⋆ =q(x ⋆)

  73. [73]

    Assumption 4.For the Adafactor recursion(49), we impose the following conditions

    The Jacobian matrix atz ⋆ is J= −diag 1√ v⋆+ε ∇2f(x ⋆)0 β∇q(x ⋆)−βI d ! .(53) Moreover, the Jacobian matrixJis Hurwitz if∇ 2f(x ⋆)≻0. Assumption 4.For the Adafactor recursion(49), we impose the following conditions. (D1)Almost sure convergence.The augmented iteratez t = (x t,v t)converges almost surely to a point z⋆ = (x⋆,v ⋆), wherex ⋆ is a strict local ...