Pith. sign in

REVIEW 4 major objections 4 minor 105 references

The paper claims that uniform LoRA rank allocation across transformer layers is provably suboptimal, and that per-layer ranks derived from diagonal Fisher information match or beat existing parameter-efficient fine-tuning methods with fewer

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:56 UTC pith:YKGHABI2

load-bearing objection Useful search-free PEFT recipe with plausible gains, but the theory overreaches and the evaluation skips the closest Fisher-based competitors. the 4 major comments →

arxiv 2607.19391 v1 pith:YKGHABI2 submitted 2026-07-02 cs.LG cs.AI

LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

Ashutosh Tripathi , Surya Deep Singh , Pranab Sahoo , Sriparna Saha This is my paper
classification cs.LG cs.AI
keywords parameter-efficient fine-tuninglow-rank adaptationlayer-wise rank allocationFisher informationdiagonal Fisher approximationLoRAGLUE benchmarkadaptive rank
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to prove that the common practice of giving every transformer layer the same LoRA rank is not merely wasteful but fundamentally suboptimal. It proposes LAARA, which estimates each layer's importance from diagonal Fisher information — approximated by squared LoRA gradient norms already computed during fine-tuning — and maps those scores to per-layer integer ranks. Because deeper layers accumulate larger Fisher traces, LAARA concentrates capacity near the output and compresses earlier layers, producing a rank profile that grows with depth. The paper reports that LAARA consistently matches or outperforms LoRA, AdaLoRA, DyLoRA, and BitFit on GLUE and MathInstruct while using substantially fewer trainable parameters. If right, adaptive rank allocation can be made principled, cheap, and stable without search or auxiliary objectives.

Core claim

LAARA's central claim is that the correct adapter rank for a transformer layer is set by that layer's diagonal Fisher information, measured during ordinary LoRA fine-tuning as an EMA of squared gradients. Formalized: under positive-definite and heterogeneous per-layer Fisher spectra, any uniform rank either starves a sensitive layer or wastes budget on an insensitive one (Theorem 1); under Lipschitz activations, bounded weights, and a contraction condition, expected gradient norms — and therefore optimal ranks — are non-decreasing toward the task loss (Theorem 2). The method converts these scores into stable integer ranks through projection-wise normalization, log-compression, blending of A/

What carries the argument

The load-bearing object is the layer-wise diagonal Fisher information, approximated by the EMA of squared LoRA gradient norms — the expected squared Frobenius norm of the gradient, claimed to be a consistent estimator of the Fisher trace. Each projection type is normalized across layers, log-compressed, blended between A and B adapters, and linearly mapped to an integer rank in [rmin, rmax]; a vote-to-change mechanism requiring τ consecutive identical proposals prevents rank oscillation. Theorems 1 and 2 use this Fisher geometry to argue that uniform rank is suboptimal and that optimal rank is non-decreasing with layer depth.

Load-bearing premise

The method's load-bearing premise is that Fisher importance measured from small rank-4 adapters early in training still predicts which layers deserve the most capacity after the adapters themselves have been resized.

What would settle it

The decisive test is an ablation that fixes the per-layer rank ordering after the early calibration epoch and disables all later resizing; if accuracy does not drop, the dynamic adaptivity is not doing the work. A complementary check: recompute the per-layer Fisher ordering at the end of training and compare it with the ordering at the calibration epoch — a near-zero rank correlation would show the calibration is chasing a moving target.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Uniform-rank LoRA is systematically misallocating capacity; adopting Fisher-guided layer ranks should improve accuracy at equal parameter budgets without any search.
  • A single early calibration pass can set a stable per-layer rank profile for the duration of fine-tuning, since the layer ordering stabilizes after about three epochs.
  • The monotone depth profile provides a principled default: assign deeper layers higher ranks and early layers lower ranks, widening the spread as the total rank budget grows.
  • The allocation mechanism is independent of target projection choice, so the same recipe transfers across Wq/Wv, Wq/Wk/Wv/Wo, and generative reasoning fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A test the paper does not run: freeze ranks after the first calibration epoch and skip all resizing; if accuracy holds, the mid-training dynamics are a robustness feature rather than the source of gains.
  • The same Fisher trace that sets rank could also inform layer-wise quantization or pruning thresholds for the merged adapter, extending the method beyond rank selection.
  • The empirical Fisher ordering should be compared with exact Fisher or K-FAC estimates on a small model; disagreement would weaken the proxy but not necessarily the rank-allocation principle.
  • The paper's own remark that the depth-rank profile reverses when gradients explode invites a direct test: weaken or remove layer normalization and check whether early layers then need the higher ranks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LAARA, a search-free method for allocating per-layer LoRA ranks based on diagonal Fisher information estimates computed during fine-tuning. It claims two theoretical results: Theorem 1 states that uniform rank allocation is provably suboptimal under heterogeneous Fisher spectra, and Theorem 2 claims that gradient norms / Fisher traces induce a monotonic non-decreasing rank profile with layer depth. The method combines EMA-smoothed diagonal Fisher traces, per-projection normalization, log compression, a blended A/B importance score, and a vote-to-change dampening mechanism. Experiments on GLUE with DeBERTa-v3-base and on MathInstruct with Llama-3.2-3B report that LAARA matches or outperforms LoRA, AdaLoRA, DyLoRA, and BitFit while using fewer trainable parameters.

Significance. If the empirical claims hold, LAARA is a practically useful, search-free alternative to fixed-rank LoRA and SVD-based adaptive rank methods. The paper provides a clearly specified algorithm and releases code, and the design choices (EMA smoothing, per-projection normalization, dampening) are reasonable and reproducible from the text. The theoretical machinery, however, is central to the paper's framing: the abstract and introduction claim that uniform rank allocation is 'provably suboptimal' and that the proposed monotone rank schedule is theoretically justified. Those theoretical claims are not established by the proofs as written, and the empirical validation is partly circular because the final rank profile is generated by the allocation rule itself. The contribution is therefore best viewed as a promising empirical recipe needing a corrected or weakened theoretical narrative and additional ablation controls.

major comments (4)
  1. [Appendix A.5, Theorem 2 (Eq. 7)] The proof of Part 3 is invalid. It derives r*_l ≥ c·erank(F_l) ≤ c·erank(F_{l+1}) ≤ r*_{l+1}, which does not imply r*_l ≤ r*_{l+1}; a chain of lower bounds cannot establish an ordering of the optimal ranks. In addition, Eq. (33) gives erank(M) ≤ tr(M)/||M||_2, but this does not imply erank(F_l) ≤ erank(F_{l+1}) from tr(F_l) ≤ tr(F_{l+1}) unless the spectral norms are controlled; Assumption 4 bounds the weight matrices W_l, not the Fisher matrices F_l. The monotonicity claim is therefore unsupported and cannot be used to justify the final rank profile.
  2. [Appendix A.5, Lemma 2] The proof of Lemma 2 relies on a reversed inequality. After assuming erank(F_l) > erank(F_{l'}), the proof writes r = r*_{l'} ≤ c·erank(F_{l'}) < c·erank(F_l) ≤ r*_l, but Lemma 1 is a lower bound r*_{l'} ≥ c·erank(F_{l'}), not an upper bound. A larger lower bound for layer l than for layer l' does not imply r*_l ≠ r*_{l'}; both optimal ranks could be large. Since Theorem 1's Part 1 invokes Lemma 2, the proof that uniform rank always under- or over-parameterizes a layer is not established. This needs a correct argument or the theoretical claim should be downgraded.
  3. [§1 Observation 3 versus §2.3 / Fig. 4] Observation 3 states that the effective rank of the weight updates decreases with depth (from ~6 at layer 0 to ~2 at layer 11), i.e., late layers concentrate their adaptation in few directions. Under the paper's own Lemma 1 logic, concentrated spectra should require smaller ranks, yet Theorem 2 and the LAARA allocation (Eq. 17, Fig. 4) assign larger ranks to deeper layers. The paper never reconciles this tension. The empirical motivation therefore supports either increasing or decreasing rank schedules depending on whether one follows Fisher trace magnitude or update effective rank. This contradiction must be resolved or the theoretical interpretation revised.
  4. [§4.2, Eq. (17), Fig. 4] Figure 4 is presented as evidence that final ranks 'strongly validate' the monotonicity result, but this is partly circular: Eq. (17) defines r_l as a monotone linear function of the compressed Fisher score, so any monotone Fisher profile yields a monotone rank profile by construction. The figure is not an independent test. More importantly, the experimental section never compares LAARA against (i) an ablation with ranks frozen after the first calibration pass, or (ii) a static monotone allocation with the same final parameter budget. Without such controls, it is unclear whether the reported gains come from the dynamic vote-based tracking emphasized in Section 3.4 or simply from any depth-increasing rank profile. Please add these ablations or soften the attribution.
minor comments (4)
  1. [Algorithm 1 and §A.2] Algorithm 1 initializes r_l ← r_max, while Section A.2 says LAARA uses initial rank r_init=4. Please clarify which initialization is actually used; this affects the early-training parameter budget and reproducibility.
  2. [References] Several references are duplicated or near-duplicated: DyLoRA appears as Valipour et al. 2022 and 2023; FLoE appears as Wang et al. 2025a and 2025b; AutoLoRA appears as Zhou et al. 2024b and Zhang et al. 2024. This creates ambiguity about which papers/versions were used as baselines and should be cleaned.
  3. [§4.2, Tables 1, 2, 4] The text says comparisons are made 'at identical parameter budget', but the parameter counts differ: Table 1 has LAARA 851,522 vs LoRA 887,042; Table 2 has 987,042 vs 1,034,498; Table 4 has 6,104,512 vs 8,275,984. Since the selling point is lower parameter use, the comparisons are still informative, but the wording should be corrected or matched-budget runs should be added.
  4. [§A.2, Tables 1–4] Results are reported as averages over three runs, but no standard deviations, confidence intervals, or significance tests are provided. Given the small gaps on some GLUE tasks, adding variance information would strengthen the empirical claims.

Circularity Check

1 steps flagged

One supporting claim is circular: the final monotone rank profile is the deterministic output of Eq. 17, so it cannot independently validate the monotonicity hypothesis; the main benchmark comparisons remain non-circular.

specific steps
  1. self definitional [Section 4.2, Fig. 4 (allocator defined in Eq. 17)]
    "As shown in Fig. 4 on the STS-B dataset, the final layer-wise ranks learned after training strongly validate our motivation. The proposed Fisher-guided LAARA framework consistently assigns lower ranks to early layers and progressively higher ranks to deeper layers, exhibiting a clear monotonic adaptation pattern across the network."

    Eq. 17 defines r_l = clip(round(rmin + (rmax - rmin) * s_l), rmin, rmax), where s_l is the blended, log-compressed, per-projection-normalized Fisher score from Eqs. 14-16. The mapping from s_l to r_l is monotone by construction. Therefore the 'final layer-wise ranks learned after training' are not independent measurements; they are the deterministic image of the Fisher input. Observing that deeper layers get higher ranks is equivalent to observing that deeper layers have higher Fisher scores, which is the same Observation 1/2 used to design the allocator. Fig. 4 cannot 'strongly validate' a monotonic rank schedule because that schedule is imposed by the allocation rule, not confirmed by it.

full rationale

The paper's central empirical evidence against external baselines (LoRA, AdaLoRA, DyLoRA, BitFit on GLUE and MathInstruct) is self-contained and not circular: those tables compare actual fine-tuned performance, and parameter-count reductions follow directly from the rank-allocation formula rather than from any fitted prediction. The main circular step is the supporting claim in Section 4.2 that the final rank profile validates the monotonicity motivation. Because Eq. 17 is a monotone linear (after rounding/clipping) transform of the Fisher-based importance scores, the final per-layer ranks are fully determined by the same Fisher signal that motivated the method; observing monotonic ranks therefore reduces to observing monotonic Fisher scores by construction. The theoretical section also contains non-circular but serious proof defects: Lemma 2 argues from lower bounds to unequal optimal ranks, and Theorem 2 Part 3 combines inequalities in an invalid manner; these are correctness risks, not instances of circularity. No load-bearing self-citations by the authors are used to justify the method, and the benchmark findings are externally grounded. Overall score 4 reflects one concrete self-definitional validation step while the primary performance claims retain independent content.

Axiom & Free-Parameter Ledger

9 free parameters · 7 axioms · 0 invented entities

The ledger shows the method is a heuristic layered on standard approximations. There are no invented entities, but the rank-allocation formula rests on several hand-set hyperparameters and domain assumptions about Fisher spectra and gradient flow. The most consequential is the use of Assumption 4 to bound Fisher matrix norms in the proof of Theorem 2 — that is not established and is a structural gap, not a mere looseness.

free parameters (9)
  • rmin/rmax rank bounds = 2/8 in primary experiments
    Bounded linear mapping in Eq. 17; chosen by hand, not derived; directly determines parameter count.
  • initial rank r_init = 4 in experiments; Algorithm 1 line 1 says r_max
    Starting rank for all layers; inconsistency between algorithm and implementation details.
  • blend coefficient alpha_b = 0.5
    Eq. 16; balances A and B Fisher estimates; fixed across tasks without ablation.
  • log compression gamma = 10
    Eq. 15; attenuates outliers; hand-set.
  • EMA decay beta = 0.97
    Eq. 12; smoothing of Fisher estimates; hand-set.
  • rank update interval DeltaT = 200 steps
    Algorithm 1 line 9; controls how often ranks can change; hand-set.
  • dampening patience tau = 2
    Eq. 18; vote-to-change threshold; hand-set.
  • scaling alpha = 16
    Eq. 8; standard LoRA scaling; chosen by the paper.
  • per-task learning rate = best of {4e-4,5e-4,8e-4,1e-3,1.2e-3,2.2e-3}
    Selected per task for all methods; standard practice but introduces selection when comparing.
axioms (7)
  • domain assumption Assumption 1: each layer FIM is positive definite
    Invoked for Lemma 1 and Theorem 1; plausible but not checked for the actual models/tasks.
  • domain assumption Assumption 2: Fisher spectra differ across layers
    This is exactly the heterogeneity the paper sets out to expose; Theorem 1 shows suboptimality given this assumption. Supported by citations and their own Fig. 2.
  • domain assumption Assumption 3: activations are beta-Lipschitz with beta <= 1
    Used in Lemmas 3-5 and Theorem 2; not verified for real transformer activations (e.g., GELU derivatives exceed 1).
  • domain assumption Assumption 4: bounded weight matrices, misapplied to Fisher matrices
    Bounds W_l only; proof of Theorem 2 Part 3 applies it to F_l, which is not established.
  • domain assumption beta*gamma <= 1 condition
    Needed for monotone gradient/rank schedule; paper does not measure beta*gamma, and Remark 1 admits reversed profile if it fails.
  • domain assumption Empirical Fisher via squared gradients is a valid FIM trace proxy
    Central estimation assumption (Eqs. 10-13); standard but biased for non-MLE losses and mini-batches.
  • domain assumption Pre-LN preserves activation norms in Lemma 5
    Used to claim ||h_{l-2}|| ≈ ||h_{l-1}||; not exact.

pith-pipeline@v1.3.0-alltime-deepseek · 18626 in / 15545 out tokens · 146513 ms · 2026-08-02T08:56:50.885396+00:00 · methodology

0 comments
read the original abstract

Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements. In this work, we show theoretically and empirically that uniform rank allocation is fundamentally suboptimal. Motivated by this observation, we propose LAARA (Layer Aware Adaptive Rank Allocation framework), a search-free framework that dynamically allocates ranks using lightweight diagonal Fisher estimates computed during training. LAARA combines projection-wise normalization, logarithmic compression, blended adapter importance estimation, and a vote-to-change dampening mechanism to produce stable and efficient rank adaptation. Experiments on GLUE and MathInstruct benchmark demonstrate that LAARA consistently matches or outperforms popular state of the art approaches such as LoRA, AdaLoRA, DyLoRA, and Bitfit while using significantly fewer trainable parameters. Our results show that Fisher-guided rank allocation provides a principled and effective foundation for adaptive parameter-efficient fine-tuning. The code is publicly available at: https://anonymous.4open.science/r/LAARA-D305/LAARA.py

Figures

Figures reproduced from arXiv: 2607.19391 by Ashutosh Tripathi, Pranab Sahoo, Sriparna Saha, Surya Deep Singh.

Figure 1
Figure 1. Figure 1: Mean gradient norms vs layer index for Lora A [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Fisher trace vs layer index for Lora A (a) and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effective rank vs layer index on RTE dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Final rank value vs layer index on STS-B [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean gradient norms vs layer index for Lora [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effective rank vs layer index on CoLA dataset. (a) (b) [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

105 extracted references · 3 canonical work pages

  1. [1]

    2026 , eprint=

    TLoRA: Task-aware Low Rank Adaptation of Large Language Models , author=. 2026 , eprint=

    2026

  2. [2]

    Pranab Sahoo and Ashutosh Tripathi and Sriparna Saha and Samrat Mondal , journal=. Fed. 2025 , url=

    2025

  3. [3]

    Transactions on Machine Learning Research , issn=

    Harnessing Heterogeneity: Improving Convergence Through Partial Variance Control in Federated Learning , author=. Transactions on Machine Learning Research , issn=. 2026 , url=

    2026

  4. [4]

    AdaFedProx: A Heterogeneity-Aware Federated Deep Reinforcement Learning for Medical Image Classification , year=

    Sahoo, Pranab and Tripathi, Ashutosh and Saha, Sriparna and Mondal, Samrat and Prakash Singh, Jyoti and Sharma, Bhisham , journal=. AdaFedProx: A Heterogeneity-Aware Federated Deep Reinforcement Learning for Medical Image Classification , year=

  5. [5]

    2024 , eprint=

    FedMRL: Data Heterogeneity Aware Federated Multi-agent Deep Reinforcement Learning for Medical Imaging , author=. 2024 , eprint=

    2024

  6. [6]

    Heterogeneity-Aware Federated Learning for Medical Image Classification With Dynamic Parameter Optimization , year=

    Sahoo, Pranab and Tripathi, Ashutosh and Saha, Sriparna and Mondal, Samrat , journal=. Heterogeneity-Aware Federated Learning for Medical Image Classification With Dynamic Parameter Optimization , year=

  7. [7]

    A uto PEFT : Automatic Configuration Search for Parameter-Efficient Fine-Tuning

    Zhou, Han and Wan, Xingchen and Vuli \'c , Ivan and Korhonen, Anna. A uto PEFT : Automatic Configuration Search for Parameter-Efficient Fine-Tuning. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00662

    doi:10.1162/tacl_a_00662 2024

  8. [8]

    IEEE Signal Processing Letters , year=

    FiLoRA: Parameter-Efficient Fine-Tuning With Fisher Information-Guided Low-Rank Adaptation , author=. IEEE Signal Processing Letters , year=

  9. [9]

    arXiv preprint arXiv:2506.00495 , year=

    FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts , author=. arXiv preprint arXiv:2506.00495 , year=

    Pith/arXiv arXiv

  10. [10]

    2023 , eprint=

    DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing , author=. 2023 , eprint=

    2023

  11. [11]

    Natural Gradient Works Efficiently in Learning , year=

    Amari, Shun-ichi , journal=. Natural Gradient Works Efficiently in Learning , year=

  12. [12]

    AROMA : Autonomous Rank-one Matrix Adaptation

    Sheng, Hao Nan and Wang, Zhi-Yong and So, Hing Cheung and Yang, Mingrui. AROMA : Autonomous Rank-one Matrix Adaptation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.170

    work page doi:10.18653/v1/2025.emnlp-main.170 2025

  13. [13]

    2025 , eprint=

    FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts , author=. 2025 , eprint=

    2025

  14. [14]

    2024 , eprint=

    DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution , author=. 2024 , eprint=

    2024

  15. [15]

    2024 , eprint=

    AROMA: Preserving Spatial Structure for Latent PDE Modeling with Local Neural Fields , author=. 2024 , eprint=

    2024

  16. [16]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    AIRA: Activation-Informed Low-Rank Adaptation for Large Models , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  17. [17]

    2026 , eprint=

    Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models , author=. 2026 , eprint=

    2026

  18. [18]

    Flexora: Flexible Low-Rank Adaptation for Large Language Models

    Wei, Chenxing and Shu, Yao and He, Ying Tiffany and Yu, Fei. Flexora: Flexible Low-Rank Adaptation for Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.713

    work page doi:10.18653/v1/2025.acl-long.713 2025

  19. [19]

    The effective rank: A measure of effective dimensionality , year=

    Roy, Olivier and Vetterli, Martin , booktitle=. The effective rank: A measure of effective dimensionality , year=

  20. [20]

    1998 , publisher=

    Theory of Point Estimation , author=. 1998 , publisher=

    1998

  21. [21]

    2020 , eprint=

    On Layer Normalization in the Transformer Architecture , author=. 2020 , eprint=

    2020

  22. [22]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  23. [23]

    GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

    Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R. GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting Neural Networks for NLP. 2018. doi:10.18653/v1/W18-5446

    doi:10.18653/v1/w18-5446 2018

  24. [24]

    2017 , eprint=

    Spectrally-normalized margin bounds for neural networks , author=. 2017 , eprint=

    2017

  25. [25]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Zhao, Dong and Li, Jinlong and Wang, Shuang and Wu, Mengyao and Zang, Qi and Sebe, Nicu and Zhong, Zhun , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2025 , pages =

    2025

  26. [26]

    Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages =

    Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach , author =. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages =. 2019 , editor =

    2019

  27. [27]

    L 1 RA : Dynamic Rank Assignment in L o RA Fine-Tuning

    Singh, Raul and Brunello, Nicol \`o and Scotti, Vincenzo and Carman, Mark. L 1 RA : Dynamic Rank Assignment in L o RA Fine-Tuning. Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025). 2025

    2025

  28. [28]

    2023 , eprint=

    One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning , author=. 2023 , eprint=

    2023

  29. [29]

    RA - L o RA : Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models

    Kim, Minsoo and Lee, Sihwa and Sung, Wonyong and Choi, Jungwook. RA - L o RA : Rank-Adaptive Parameter-Efficient Fine-Tuning for Accurate 2-bit Quantized Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.933

    work page doi:10.18653/v1/2024.findings-acl.933 2024

  30. [30]

    2026 , eprint=

    DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models , author=. 2026 , eprint=

    2026

  31. [31]

    ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models With Heterogeneous Adaptation Needs , year=

    Shinwari, Haseeb Ullah Khan and Usama, Muhammad , journal=. ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models With Heterogeneous Adaptation Needs , year=

  32. [32]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  33. [33]

    ACM Transactions on Sensor Networks , volume=

    Recent advances in LoRa: A comprehensive survey , author=. ACM Transactions on Sensor Networks , volume=. 2022 , publisher=

    2022

  34. [34]

    arXiv preprint arXiv:2306.12659 , year=

    Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models , author=. arXiv preprint arXiv:2306.12659 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2306.06031 , year=

    Fingpt: Open-source financial large language models , author=. arXiv preprint arXiv:2306.06031 , year=

    arXiv

  36. [36]

    Twitter financial news sentiment , author=

  37. [37]

    arXiv preprint arXiv:2403.19211 , year=

    Dual-Personalizing Adapter for Federated Foundation Models , author=. arXiv preprint arXiv:2403.19211 , year=

    Pith/arXiv arXiv

  38. [38]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  39. [39]

    Bloom: A 176b-parameter open-access multilingual language model , author=

  40. [40]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Heterogeneous lora for federated fine-tuning of on-device foundation models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  41. [41]

    arXiv preprint arXiv:1804.07461 , year=

    Glue: A multi-task benchmark and analysis platform for natural language understanding , author=. arXiv preprint arXiv:1804.07461 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2311.11696 , year=

    Sparse low-rank adaptation of pre-trained language models , author=. arXiv preprint arXiv:2311.11696 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2303.10512 , year=

    AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning , author=. arXiv preprint arXiv:2303.10512 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2101.00190 , year=

    Prefix-tuning: Optimizing continuous prompts for generation , author=. arXiv preprint arXiv:2101.00190 , year=

    Pith/arXiv arXiv

  45. [46]

    Advances in Neural Information Processing Systems , volume=

    Ensemble distillation for robust model fusion in federated learning , author=. Advances in Neural Information Processing Systems , volume=

  46. [47]

    Advances in Neural Information Processing Systems , volume=

    Towards federated foundation models: Scalable dataset pipelines for group-structured learning , author=. Advances in Neural Information Processing Systems , volume=

  47. [48]

    2019 , eprint=

    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , author=. 2019 , eprint=

    2019

  48. [49]

    arXiv preprint arXiv:2404.15381 , year=

    Advances and open challenges in federated learning with foundation models , author=. arXiv preprint arXiv:2404.15381 , year=

    Pith/arXiv arXiv

  49. [50]

    2023 , eprint=

    BLOOM: A 176B-Parameter Open-Access Multilingual Language Model , author=. 2023 , eprint=

    2023

  50. [51]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  51. [52]

    2024 , eprint=

    The Future of Large Language Model Pre-training is Federated , author=. 2024 , eprint=

    2024

  52. [53]

    2024 , eprint=

    Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models , author=. 2024 , eprint=

    2024

  53. [54]

    2024 , eprint=

    When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions , author=. 2024 , eprint=

    2024

  54. [55]

    arXiv preprint arXiv:2211.04325 , year=

    Will we run out of data? an analysis of the limits of scaling datasets in machine learning , author=. arXiv preprint arXiv:2211.04325 , year=

    Pith/arXiv arXiv

  55. [56]

    Advances in Neural Information Processing Systems , volume=

    How far can camels go? exploring the state of instruction tuning on open resources , author=. Advances in Neural Information Processing Systems , volume=

  56. [57]

    arXiv preprint arXiv:2306.16092 , year=

    Chatlaw: Open-source legal large language model with integrated external knowledge bases , author=. arXiv preprint arXiv:2306.16092 , year=

    Pith/arXiv arXiv

  57. [58]

    Nature , volume=

    Large language models encode clinical knowledge , author=. Nature , volume=. 2023 , publisher=

    2023

  58. [59]

    Nature medicine , volume=

    Large language models in medicine , author=. Nature medicine , volume=. 2023 , publisher=

    2023

  59. [60]

    Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    Openfedllm: Training large language models on decentralized private data via federated learning , author=. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

  60. [61]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  61. [62]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Segment anything , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  62. [63]

    arXiv preprint arXiv:2307.10485 , year=

    Fingpt: Democratizing internet-scale data for financial large language models , author=. arXiv preprint arXiv:2307.10485 , year=

    Pith/arXiv arXiv

  63. [64]

    Archives of pathology & laboratory medicine , year=

    Assessment of pathology domain-specific knowledge of ChatGPT and comparison to human performance , author=. Archives of pathology & laboratory medicine , year=

  64. [65]

    arXiv preprint arXiv:2303.17564 , year=

    Bloomberggpt: A large language model for finance , author=. arXiv preprint arXiv:2303.17564 , year=

    Pith/arXiv arXiv

  65. [66]

    arXiv preprint arXiv:2010.02559 , year=

    LEGAL-BERT: The muppets straight out of law school , author=. arXiv preprint arXiv:2010.02559 , year=

    Pith/arXiv arXiv 2010

  66. [67]

    Artificial intelligence and statistics , pages=

    Communication-efficient learning of deep networks from decentralized data , author=. Artificial intelligence and statistics , pages=. 2017 , organization=

    2017

  67. [68]

    International conference on machine learning , pages=

    Scaffold: Stochastic controlled averaging for federated learning , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  68. [69]

    2020 , eprint=

    Federated Optimization in Heterogeneous Networks , author=. 2020 , eprint=

    2020

  69. [70]

    arXiv preprint arXiv:2403.12313 , year=

    Improving loRA in privacy-preserving federated learning , author=. arXiv preprint arXiv:2403.12313 , year=

    Pith/arXiv arXiv

  70. [71]

    International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 , year=

    Heterogeneous lora for federated fine-tuning of on-device foundation models , author=. International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 , year=

    2023

  71. [72]

    arXiv preprint arXiv:2406.17477 , year=

    Towards Federated Low-Rank Adaptation with Rank-Heterogeneous Communication , author=. arXiv preprint arXiv:2406.17477 , year=

    Pith/arXiv arXiv

  72. [73]

    arXiv preprint arXiv:2404.15182 , year=

    Flora: Enhancing vision-language models with parameter-efficient federated learning , author=. arXiv preprint arXiv:2404.15182 , year=

    Pith/arXiv arXiv

  73. [74]

    arXiv preprint arXiv:2409.05976 , year=

    FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations , author=. arXiv preprint arXiv:2409.05976 , year=

    Pith/arXiv arXiv

  74. [75]

    arXiv preprint arXiv:2310.13283 , year=

    Fedlora: Model-heterogeneous personalized federated learning with lora tuning , author=. arXiv preprint arXiv:2310.13283 , year=

    Pith/arXiv arXiv

  75. [76]

    2024 , eprint=

    FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning , author=. 2024 , eprint=

    2024

  76. [77]

    Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

    FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model , author=. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages=

  77. [78]

    Annual Meeting of the Association of Computational Linguistics 2023 , pages=

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models , author=. Annual Meeting of the Association of Computational Linguistics 2023 , pages=. 2023 , organization=

    2023

  78. [79]

    arXiv preprint arXiv:2308.06522 , year=

    SLoRA: Federated parameter efficient fine-tuning of language models , author=. arXiv preprint arXiv:2308.06522 , year=

    Pith/arXiv arXiv

  79. [80]

    Nature Machine Intelligence , volume=

    Parameter-efficient fine-tuning of large-scale pre-trained language models , author=. Nature Machine Intelligence , volume=. 2023 , publisher=

    2023

  80. [81]

    arXiv preprint arXiv:2106.09685 , year=

    Lora: Low-rank adaptation of large language models , author=. arXiv preprint arXiv:2106.09685 , year=

    Pith/arXiv arXiv

Showing first 80 references.