Pith. sign in

REVIEW 3 major objections 3 minor 45 references

Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LoRA rank allocation can be chosen by a greedy algorithm with a (1 - 1/e) guarantee after projecting the Hessian to enforce submodularity.

desk verdict Clever Hessian-projection idea, but the (1-1/e) guarantee doesn't survive contact with the actual objective; still worth reviewing. read the letter →

arxiv 2507.01841 v1 pith:USQH55FF submitted 2025-07-02 cs.LG cs.ITeess.SPmath.ITmath.OC

Yihang Gao , Vincent Y. F. Tan This is my paper
classification cs.LGcs.ITeess.SPmath.ITmath.OC
keywords Low-RankAdaptationrankdeterminationsubmodularfunctionmaximizationgreedyalgorithmHessianphysics-informedneuralnetworksPDEsparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SubLoRA, a method that decides how many dimensions each low-rank adaptation (LoRA) layer should keep by using second-order (Hessian) information rather than the first-order gradients used by earlier pruning methods. The authors show that once LoRA fine-tuning has nearly converged, gradients vanish and first-order sensitivity scores become unreliable, so they approximate the loss with a quadratic form in the singular values of the update matrices. Because optimizing that quadratic form exactly is NP-hard, they prove a necessary and sufficient condition for the objective to be submodular and construct a closed-form projection of the Hessian that enforces this condition. The projected objective can then be solved by a greedy algorithm with a provable $(1 - 1/e)$ approximation guarantee, and an alternating version interleaves rank selection with parameter updates. Experiments on LoRA fine-tuning of physics-informed neural networks for several PDEs show the method matching or beating first-order and diagonal-Hessian baselines under rank budgets.

What carries the argument

The load-bearing object is the set function $f(S) = \langle[\nabla_{\sigma}\mathcal{L}_{\mathrm{ft}}(\Theta_{\mathrm{LoRA}})]_{S^-}, [\sigma]_{S^-}\rangle - \tfrac{1}{2}[\sigma]_{S^-}^{\top}[G]_{S^-}[\sigma]_{S^-}$, where $S$ is the set of retained singular-value indices and $S^-$ its complement. Theorem 2 gives a necessary and sufficient condition for submodularity: every off-diagonal entry must satisfy $G_{i,j}\sigma_i\sigma_j \geq 0$. The method's key move is the closed-form projection in Eq. (11), which keeps Hessian entries that already satisfy this sign condition, zeros out those that do not, and preserves the diagonal; this converts an NP-hard quadratic selection problem into a monotone submodular maximization problem when the stationarity premise of Lemma 1 holds. The greedy algorithm then serves as the solver, with the approximation guarantee of Theorem 1.

What would settle it

Take a small LoRA model with, say, eight singular values, train it with Algorithm 4's 100-epoch Adam schedule on a problem chosen to keep the Hessian strongly indefinite, brute-force the true optimum of Eq. (12), and compare the greedy solution's objective value; if it falls below the $(1 - 1/e)$ fraction of the optimum, the claimed guarantee is violated. A cheaper direct check is to record every marginal gain $f(S \cup \{i\}) - f(S)$ the greedy algorithm evaluates and count how many are negative, which would contradict the monotonicity that Lemma 1 is meant to provide.

Watch

Extended reading notes

Core claim

The central claim is that rank determination for LoRA can be framed as maximizing a set-valued quadratic function of retained singular values, and that projecting the Hessian with the rule $G_{i,j} = [\nabla^2_{\sigma} \mathcal{L}_{\mathrm{ft}}(\Theta_{\mathrm{LoRA}})]_{i,j}$ when the product $G_{i,j}\sigma_i\sigma_j \geq 0$ and $0$ otherwise, while keeping diagonal entries unchanged, makes this function submodular. Submodularity then licenses a greedy solver whose output is provably within $(1 - 1/e)$ of the optimal rank allocation whenever the fine-tuned parameters satisfy second-order necessary optimality conditions. The paper argues that linearized rank scores degenerate exactly in the converged regime where LoRA is normally used, and reports that the Hessian-guided greedy selection consistently outperforms both first-order pruning and diagonal-only second-order pruning in its experiments on elliptic, Allen–Cahn, and hyperbolic PDEs.

Load-bearing premise

The approximation guarantee rests on the assumption that LoRA fine-tuning drives the gradient to zero and the Hessian to positive semidefinite; Algorithm 4 only runs 100 Adam epochs per round, so this stationarity is not verified, and without it the objective's monotonicity—and the (1 - 1/e) guarantee—can fail.

Editorial extensions

If this is right

  • Rank allocation across LoRA layers can be computed from curvature information alone, in a training-free post-hoc pass, without retraining the adapted model.
  • Whenever the fine-tuned model is near a second-order stationary point, the greedy solution is provably within $1 - 1/e$ of the best possible rank allocation, so no exact combinatorial search is needed.
  • The alternating version of the algorithm lets the rank budget act as a regularizer during fine-tuning, guiding the parameter updates toward low-rank solutions and improving final loss over post-hoc pruning.
  • Because the Hessian is computed only over the singular values rather than all model parameters, the second-order machinery adds only a few seconds of runtime in the reported PINN experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same projection recipe could be applied to other cardinality-constrained quadratic selection problems, such as neuron or channel pruning, whenever a Hessian is available; the paper does not discuss these settings.
  • The paper leaves open whether the $(1 - 1/e)$ guarantee holds empirically; a direct check would compare the greedy objective value to the brute-force optimum on small LoRA problems and record how often monotonicity actually holds after the 100-epoch Adam rounds used in Algorithm 4.
  • Zeroing out negative off-diagonal Hessian terms is a real information loss, so in strongly coupled layers a small perturbation of the projection rule—for example, retaining a fraction of the violated entries—might yield better selections while approximately preserving submodularity.
  • The authors suggest replacing ReLU with smooth activations to apply SubLoRA to large language models; that extension is testable but the Hessian cost would then depend on whether the low-dimensional singular-value Hessian remains accurate in very high-dimensional parameter spaces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes SubLoRA, a method that determines per-layer LoRA ranks after fine-tuning by maximizing a cardinality-constrained second-order surrogate of the negative loss increase. The authors prove a necessary and sufficient condition for a quadratic set function to be submodular, project the Hessian onto the feasible set via the closed-form rule in Eq. (11), and then apply standard greedy or randomized greedy algorithms. An alternating variant (Algorithm 4) interleaves Adam updates with rank pruning. The empirical section evaluates the method on LoRA fine-tuning of PINNs for elliptic, Allen–Cahn, and hyperbolic PDEs.

Significance. The paper contains one elegant and correct structural result: Theorem 2's characterization, together with the closed-form projection in Eq. (11), gives a cheap way to convert a quadratic set function into a submodular one. The experiments are clean and the runtime table (Table 1) shows the overhead is modest. These assets are genuine. However, the advertised theoretical guarantee is not established: the objective in Eq. (12) is non-positive in exactly the stationary regime used to justify monotonicity, so Theorem 1's non-negativity hypothesis fails. The central claim of 'provably near-optimal solutions' therefore rests on an invalid application of a standard result. The empirical contribution may survive as a heuristic, but the paper's stated contribution is not supported.

major comments (3)
  1. [Section 3.3, Eq. (12) and Theorem 1] The stated (1−1/e) approximation guarantee does not follow. Theorem 1 requires the objective f to be non-negative (and, in the standard form used in Algorithm 1, f(∅)=0). Under the conditions of Lemma 1, where ∇L=0 and the Hessian is PSD, the objective reduces to f(S) = −1/2 [σ]_{S−}^T [G]_{S−} [σ]_{S−}. Because the projection rule (11) gives G_{ij}σ_iσ_j ≥ 0 for i≠j and the diagonal entries G_{ii} are non-negative, every such quadratic form is non-negative, so f(S) ≤ 0 for all S; in particular f(∅) < 0 whenever σ≠0 and f([r])=0. Thus Theorem 1 cannot be invoked in the stationary regime, which is precisely the regime used to justify monotonicity. A constant shift does not repair the argument, because the ratio guarantee would then apply to the shifted objective rather than to f. The abstract and Section 3.3 advertise provable guarantees as a main contribution, so this is load-bearing and not a presentation issue.
  2. [Section 3.3, Lemma 1] The proof of Lemma 1 contains a factor-of-two error in the marginal-gain computation. For i∉S, writing A=[r]\(S∪{i}), the correct expression is f(S∪{i})−f(S) = Σ_{a∈A} G_{ai}σ_aσ_i + (1/2)G_{ii}σ_i^2, not 2Σ_{a∈A}G_{ai}σ_aσ_i + G_{ii}σ_i^2 as displayed. The sign conclusion still holds under the stated assumptions, but the algebra should be corrected. More importantly, Lemma 1's premise, that Stage 1 reaches a point satisfying second-order necessary optimality, is not verified in Algorithm 4, where Stage 1 runs only 100 Adam epochs per outer iteration. Consequently, even if Lemma 1 were correct, the paper's justification for preferring deterministic greedy over randomized greedy in the alternating setting would remain unestablished.
  3. [Section 3.3, Theorem 1 statement] The stated theorem also omits the standard normalization f(∅)=0. Non-negativity alone is not sufficient for the (1−1/e) bound in the form claimed; the standard guarantee requires a monotone submodular f with f(∅)=0. Since f(∅)<0 in the stationary regime of the proposed method, the discrepancy is directly relevant to the invalid guarantee identified above, not merely a typo.
minor comments (3)
  1. [Section 5.2, Table 2] The text states that alternating SubLoRA 'consistently achieves the lowest training loss and validation error.' In the Allen–Cahn λ=(1,1) row, SubLoRA-G has training loss 3.10E-3, which is larger than DiagLoRA's 1.40E-3, although its relative error is lower. The claim should be qualified to avoid overstating the experimental result.
  2. [Section 5.1 and 5.2] The setup says a four-layer MLP with three hidden layers, LoRA rank 50 per layer, and 'total initial LoRA rank of 100.' If all three hidden layers are adapted with rank 50, the total would be 150; clarify which layers are adapted or reconcile the arithmetic.
  3. [Section 2.4, Eq. (6)] The linearized surrogate in Eq. (6) is written as a minimization of a sum of inner products of the form ⟨[∇L]_{S−}, −[σ]_{S−}⟩. Since the sign convention here differs from the later maximization formulation, a one-sentence derivation of the equivalence would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the rank-allocation rule is computed from an explicit gradient/Hessian projection and benchmarked against external PDE solutions; the only self-citation is contextual and not load-bearing.

full rationale

The derivation chain in SubLoRA does not reduce to its own inputs. Rank allocations are produced by the explicit formulas in Eqs. (7)-(12): a second-order Taylor expansion of the fine-tuning loss, a Hessian projection Eq. (10) with closed-form solution Eq. (11), and a greedy solver. The resulting subset is then evaluated by training loss and relative error against known PDE solutions, which are external references rather than quantities fitted into the method. The submodularity claim is not a disguised restatement of the objective: Theorem 2 proves an if-and-only-if condition for the set function in Eq. (9), and Eqs. (10)-(11) explicitly construct a projected matrix G satisfying that condition. This is a design choice, not a circular assumption, because the paper never assumes the original unprojected Hessian objective is submodular. The one self-citation, Ref. [11] in the introductory survey of tensorized LoRA, is contextual and plays no role in the main theoretical or experimental claims. The greedy approximation guarantee is imported from standard external references [10, 21], and the monotonicity discussion invokes independent optimizer-convergence results [12, 13]. Even if the non-negativity premise of Theorem 1 is not verified in practice, or if the marginal-gain computation in Lemma 1 contains an algebraic slip, that is a soundness or correctness concern, not evidence of circularity, because the method's output is not forced to equal a fitted parameter or a self-citation chain. Hence the paper is self-contained against external benchmarks and exhibits no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method itself has no fitted constants; the only tunable inputs are the rank budget b and the initial per-layer ranks, which are experimental constraints. The central derivation rests on four unproved premises: smoothness and adequacy of the second-order expansion, convergence to second-order stationarity, validity of fixing the SVD basis while pruning, and satisfaction of the non-negativity hypothesis in Theorem 1. These are flagged in the axioms.

assumptions (4)
  • domain assumption The loss is twice differentiable with respect to the singular values sigma and its second-order Taylor expansion is an adequate surrogate for the pruned-loss objective.
    Used in Section 3.2 to replace Eq. (4) by Eq. (7); fails if activations are not smooth (e.g., ReLU) or if curvature changes rapidly.
  • domain assumption LoRA fine-tuning reaches a second-order stationary point, so the gradient of the loss with respect to sigma is zero and the Hessian is positive semidefinite.
    Assumed in Lemma 1 and in the justification of the greedy algorithm; the experiments use only 100 Adam epochs per round in Algorithm 4, so this is not established.
  • domain assumption The SVD basis U,V obtained from a trained BA is held fixed when pruning singular values; changing sigma alone is a faithful proxy for re-optimizing the LoRA update under a rank budget.
    Invoked implicitly in Section 2.4 and Eq. (1); at non-stationary points the loss also depends on perturbations of U and V, which this analysis ignores.
  • standard math The objective f in Eq. (12) is non-negative, as required by the (1-1/e) guarantee in Theorem 1.
    Theorem 1 states 'Let f be a non-negative objective function'; the paper never shows f is non-negative, and under Lemma 1's assumptions f is non-positive at S = empty set before any constant shift.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization." pith.science (2026). https://pith.science/paper/USQH55FF

@misc{pith2026250701841,
  author       = {Pith},
  title        = {Pith review of: Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USQH55FF}},
  note         = {Machine review of arXiv:2507.01841}
}
read the original abstract

In this paper, we propose SubLoRA, a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function maximization. In contrast to prior approaches, such as AdaLoRA, that rely on first-order (linearized) approximations of the loss function, SubLoRA utilizes second-order information to capture the potentially complex loss landscape by incorporating the Hessian matrix. We show that the linearization becomes inaccurate and ill-conditioned when the LoRA parameters have been well optimized, motivating the need for a more reliable and nuanced second-order formulation. To this end, we reformulate the rank determination problem as a combinatorial optimization problem with a quadratic objective. However, solving this problem exactly is NP-hard in general. To overcome the computational challenge, we introduce a submodular function maximization framework and devise a greedy algorithm with approximation guarantees. We derive a sufficient and necessary condition under which the rank-determination objective becomes submodular, and construct a closed-form projection of the Hessian matrix that satisfies this condition while maintaining computational efficiency. Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency. We further extend SubLoRA to a joint optimization setting, alternating between LoRA parameter updates and rank determination under a rank budget constraint. Extensive experiments on fine-tuning physics-informed neural networks (PINNs) for solving partial differential equations (PDEs) demonstrate the effectiveness of our approach. Results show that SubLoRA outperforms existing methods in both rank determination and joint training performance.

Figures

Figures reproduced from arXiv: 2507.01841 by the authors.

Figure 1
Figure 1. Performance comparison of different rank determination methods on elliptic equations [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of different rank determination methods on Allen–Cahn equa [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of different rank determination methods on hyperbolic equations [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: When the budget is set to b = 20, we observe a slow convergence or even divergence and a noticeably higher training loss. This is because the rank budget underestimates the capacity needed to represent the new PDE solution, leading to limited expressiveness and subopti…
Figure 4
Figure 4. Figure 4: Training trajectories of Algorithm 4 using (first-order) LinearLoRA and (second-order) [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Training trajectories of Algorithm 4 using SubLoRA-G as the rank determination method [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 32 canonical work pages

  1. [1]

    Foundation mod- els defining a new era in vision: a survey and outlook

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation mod- els defining a new era in vision: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    work page 2025

  2. [2]

    LoTR: Low tensor rank weight adaptation

    Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Aleksandr Mikhalev, and Ivan Oseledets. LoTR: Low tensor rank weight adaptation. arXiv preprint arXiv:2402.01376, 2024

    arXiv 2024

  3. [3]

    The challenges of the nonlinear regime for physics-informed neural networks

    Andrea Bonfanti, Giuseppe Bruno, and Cristina Cipriani. The challenges of the nonlinear regime for physics-informed neural networks. Advances in Neural Information Processing Systems, 37:41852–41881, 2024

    work page 2024

  4. [4]

    AdaptFormer: Adapting vision Transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. AdaptFormer: Adapting vision Transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678, 2022

    work page 2022

  5. [5]

    QLoRA: Efficient fine- tuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient fine- tuning of quantized LLMs. Advances in Neural Information Processing Systems , 36:10088– 10115, 2023

    work page 2023

  6. [6]

    LoRA-C: Parameter-efficient fine-tuning of robust CNN for IoT devices

    Chuntao Ding, Xu Cao, Jianhang Xie, Linlin Fan, Shangguang Wang, and Zhichao Lu. LoRA-C: Parameter-efficient fine-tuning of robust CNN for IoT devices. arXiv preprint arXiv:2410.16954, 2024

    arXiv 2024

  7. [7]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235, 2023

    2023

  8. [8]

    KronA: Parameter efficient tuning with Kronecker adapter

    Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. KronA: Parameter efficient tuning with Kronecker adapter. arXiv preprint arXiv:2212.10650, 2022

    arXiv 2022

Show all 45 references
  1. [9]

    Implicit style-content sepa- ration using B-LoRA

    Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content sepa- ration using B-LoRA. In European Conference on Computer Vision, pages 181–198. Springer, 2024

    2024

  2. [10]

    Submodular functions and optimization, volume 58

    Satoru Fujishige. Submodular functions and optimization, volume 58. Elsevier, 2005

    2005

  3. [11]

    Low tensor-rank adaptation of kolmogorov– arnold networks

    Yihang Gao, Michael K Ng, and Vincent YF Tan. Low tensor-rank adaptation of kolmogorov– arnold networks. arXiv preprint arXiv:2502.06153, 2025

    2025 arXiv

  4. [12]

    Escaping from saddle points—online stochastic gradient for tensor decomposition

    Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points—online stochastic gradient for tensor decomposition. In Conference on Learning Theory, pages 797–

  5. [13]

    Matrix completion has no spurious local minimum

    Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. Advances in Neural Information Processing Systems, 29, 2016. 25

    2016

  6. [14]

    Parameter-efficient fine- tuning for large models: A comprehensive survey

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey. Transactions on Machine Learning Re- search, 2024

    2024

  7. [15]

    LoRA+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In International Conference on Machine Learning, pages 17783–17806. PMLR, 2024

    2024

  8. [16]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Inter- national Conference on Learning Representations, 2022

    2022

  9. [17]

    Lee, and Ernest K

    Uijeong Jang, Jason D. Lee, and Ernest K. Ryu. LoRA training in the NTK regime has no spurious local minima. In Forty-first International Conference on Machine Learning, 2024

    2024

  10. [18]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709–727. Springer, 2022

    2022

  11. [19]

    Physics-informed machine learning

    George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021

    2021

  12. [20]

    Junsu Kim, Jaeyeon Kim, and Ernest K. Ryu. LoRA training provably converges to a low-rank global minimum or it fails loudly (but it probably won’t fail). In Forty-second International Conference on Machine Learning, 2025

    2025

  13. [21]

    Submodular function maximization

    Andreas Krause and Daniel Golovin. Submodular function maximization. Tractability, 3(71- 104):3, 2014

    2014

  14. [22]

    Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models

    Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, and Mang Ye. Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 26170– 26180, 2025

    2025

  15. [23]

    ALoRA: Allocating low- rank adaptation for fine-tuning large language models

    Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. ALoRA: Allocating low- rank adaptation for fine-tuning large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage T...

    2024

  16. [24]

    HyperLoRA for PDEs

    Ritam Majumdar, Vishal Jadhav, Anirudh Deodhar, Shirish Karande, Lovekesh Vig, and Venkataramana Runkana. HyperLoRA for PDEs. arXiv preprint arXiv:2308.09290, 2023

    2023 arXiv

  17. [25]

    PIHLoRA: Physics-informed hypernetworks for low-ranked adapta- tion

    Ritam Majumdar, Vishal Jadhav, Anirudh Deodhar, Shirish Karande, Lovekesh Vig, and Venkataramana Runkana. PIHLoRA: Physics-informed hypernetworks for low-ranked adapta- tion. In AI for Accelerated Materials Design-NeurIPS 2023 Workshop, 2023

    2023

  18. [26]

    A kernel- based view of language model fine-tuning

    Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel- based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610–23641. PMLR, 2023. 26

    2023

  19. [27]

    A survey on LoRA of large language models

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on LoRA of large language models. Frontiers of Computer Science , 19(7):197605, 2025

    2025

  20. [28]

    Near-optimal sketchy natu- ral gradients for physics-informed neural networks

    Maricela Best Mckay, Avleen Kaur, Chen Greif, and Brian Wetton. Near-optimal sketchy natu- ral gradients for physics-informed neural networks. In Forty-second International Conference on Machine Learning, 2025

    2025

  21. [29]

    Achieving high accuracy with PINNs via energy nat- ural gradient descent

    Johannes M ¨uller and Marius Zeinhofer. Achieving high accuracy with PINNs via energy nat- ural gradient descent. In International Conference on Machine Learning, pages 25471–25485. PMLR, 2023

    2023

  22. [30]

    Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear par- tial differential equations

    Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear par- tial differential equations. Journal of Computational physics, 378:686–707, 2019

    2019

  23. [31]

    Fine-tuning protein language mod- els boosts predictions across diverse tasks

    Robert Schmirler, Michael Heinzinger, and Burkhard Rost. Fine-tuning protein language mod- els boosts predictions across diverse tasks. Nature Communications, 15(1):7407, 2024

    2024

  24. [32]

    LoRA vs full fine-tuning: An illusion of equivalence

    Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. LoRA vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228, 2024

    2024

  25. [33]

    Tensor decomposition for compressing recurrent neural network

    Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Tensor decomposition for compressing recurrent neural network. In2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018

    2018

  26. [34]

    DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adap- tation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adap- tation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Li...

    2023

  27. [35]

    Parameter-efficient fine-tuning in large language models: A survey of methodologies

    Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter-efficient fine-tuning in large language models: A survey of methodologies. Arti- ficial Intelligence Review, 58(8):227, 2025

    2025

  28. [36]

    Metaxas, and Hao Wang

    Yibin Wang, Haizhou Shi, Ligong Han, Dimitris N. Metaxas, and Hao Wang. BLob: Bayesian low-rank adaptation by backpropagation for large language models. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems, 2024

    2024

  29. [37]

    Transfer learning in physics-informed neurals networks: Full fine-tuning, lightweight fine-tuning, and low-rank adaptation

    Yizheng Wang, Jinshuai Bai, Mohammad Sadegh Eshaghi, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabczuk, and Yinghua Liu. Transfer learning in physics-informed neurals networks: Full fine-tuning, lightweight fine-tuning, and low-rank adaptation. International Journal of Mechanica...

    2025

  30. [38]

    Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison

    Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024. 27

    2024

  31. [39]

    LoRETTA: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models

    Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. LoRETTA: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Li...

    2024

  32. [40]

    Ranking and tuning pre-trained models: A new paradigm for exploiting model hubs

    Kaichao You, Yong Liu, Ziyang Zhang, Jianmin Wang, Michael I Jordan, and Mingsheng Long. Ranking and tuning pre-trained models: A new paradigm for exploiting model hubs. Journal of Machine Learning Research, 23(209):1–47, 2022

    2022

  33. [41]

    The expressive power of low-rank adaptation

    Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  34. [42]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  35. [43]

    Personalized LoRA for human-centered text understanding

    You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, and Xuejie Zhang. Personalized LoRA for human-centered text understanding. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 38, pages 19588–19596, 2024

    2024

  36. [44]

    Neural prompt search

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 28

    2024

  37. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.