Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that jointly optimizing per-client participation probabilities and LoRA sketching ratios, guided by a new convergence bound that avoids bounded-gradient assumptions, minimizes the wall-clock time of federated LLM…

desk verdict The problem and the combination are real, but the central convergence proof has a load-bearing gap that is not fixed by the stated learning-rate condition. read the letter →

arxiv 2505.23555 v3 pith:TLFTSZCV submitted 2025-05-29 cs.LG

Yanzhao Hou , Jiaxiang Geng , Boyu Li , Xiaofeng Tao , Juncheng Wang , Xiaodong Xu , Bing Luo This is my paper
classification cs.LG
keywords federatedlearningLoRAclientsamplingsketchingmechanismwirelessnetworksconvergenceanalysiswall-clocktimesystemheterogeneity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the wall-clock time of federated fine-tuning of large language models with LoRA can be minimized by making two per-client decisions together: how often each device participates in training rounds, and how much of its LoRA update it computes and transmits. It derives a convergence bound for federated LoRA under independent client sampling that does not rely on the usual uniform bounded-gradient assumption, and uses that bound together with a bandwidth-allocation model to formulate an optimization over participation probabilities and sketching ratios. The proposed algorithm selects these parameters to minimize the expected time to a target accuracy, and experiments on Qwen2.5-1.5B across four commonsense reasoning tasks report 2.8–4.2x wall-clock speedups over the FSLoRA baseline. A careful reader would care because device heterogeneity and stragglers, not just round counts, dominate the real cost of on-device LLM fine-tuning over wireless links.

What carries the argument

The load-bearing object is the randomized sketching matrix $S(\gamma,k_n)$, a diagonal matrix with exactly $k_n$ nonzero entries sampled uniformly, which satisfies $\mathbb{E}[S]=I_\gamma$ and reduces the LoRA update each client computes and transmits from $\gamma^2$ parameters to $k_n^2$ parameters. It is combined with inverse-probability aggregation, where a participating client's update is scaled by $1/q_n$ so the global update stays unbiased under independent Bernoulli sampling. The second piece of machinery is the round-time model: using an equalized-completion-time argument, the expected round time is upper bounded by $\sum_{n=1}^N q_n(t_n/f_{\text{tot}}+\tau_n)$, and both communication time $t_n$ and computation time $\tau_n$ are modeled as $(k_n/\gamma)^2$ times their full-rank values. These pieces feed an alternating-minimization algorithm that estimates four bound constants $A,B,C,D$ from four calibration runs, solves a convex proxy for $q$ with $k$ fixed, and greedily reduces each $k_n$ with $q$ fixed.

What would settle it

Measure, on real mobile hardware and over the intended wireless links, the local computation and communication times for a range of sketching ratios $k_n$, then run the proposed optimization against a scheme that picks $q$ and $k$ directly from the measured time curves. If the actual scaling is not $\tau_n = \tau^\gamma_n (k_n/\gamma)^2$ and $t_n = t^\gamma_n (k_n/\gamma)^2$, the optimized choices will be miscalibrated and the predicted speedup over FSLoRA should fail to materialize, since the optimization is built entirely on that quadratic model.

Watch

Extended reading notes

Core claim

The paper's central claim is that federated LoRA with per-client sketching and independent Bernoulli participation converges for non-convex losses under only smoothness, gradient-variance, and gradient-dissimilarity assumptions, and that the averaged squared gradient norm after $R$ rounds is bounded by $$\frac{1}{R}\sum_{r=0}^{R-1}\mathbb{E}[\|\nabla_X F^S(X_r)\|^2] \le \frac{4(F^S(X_0)-F^*)}{\eta R H} + 6\eta L_s N(\$sigma_g^{2}$+\$sigma_s^{2}$+\$sigma_h^{2}$)\sum_{n=1}^N \frac{$a_n^{2}$}{q_n} + $36L^{2}$\$eta^{2}$$H^{2}$N(\$sigma_g^{2}$+\$sigma_s^{2}$+\$sigma_h^{2}$)\sum_{n=1}^N \frac{$a_n^{2}$}{q_n}\frac{\$gamma^{2}$}{$k_n^{2}$}.$$ The bound separates the effect of participation probabilities $q_n$ from the effect of sketching ratios $k_n$, and it holds for arbitrary independent sampling probabilities and arbitrary sketching ratios, provided every client has positive $q_n$ and $k_n$. The paper further claims that because per-round time grows with participation (through shared bandwidth) and with sketching ratio (through compute and communication cost), the wall-clock-optimal configuration is neither full participation nor full rank, but a joint choice found by minimizing an upper bound on expected round time derived from this convergence bound.

Load-bearing premise

The whole per-round time model assumes each client's computation and communication time scale exactly quadratically with its sketching ratio, $(k_n/\gamma)^2$, and that the constants fitted from four short calibration runs stay valid throughout training to the target accuracy.

Editorial extensions

If this is right

  • If the bound is correct, convergence in rounds is guaranteed only when every client has positive participation probability $q_n$ and positive sketching ratio $k_n$; setting either to zero makes the required number of rounds unbounded.
  • Because round time grows with $q_n$ through bandwidth sharing and with $k_n^2$ through compute and communication, the optimal wall-clock configuration balances small but frequent updates against large but rare ones, rather than simply maximizing participation or rank.
  • The optimization framework produces explicit per-client $q$ and $k$ choices that minimize an upper bound on wall-clock time to reach a target accuracy, and the reported experiments show this reduces time to target accuracy by 2.8–4.2x over FSLoRA on ARC-e, ARC-c, OBQA, and SIQA.
  • The convergence result removes the uniform bounded-gradient assumption that prior independent-sampling analyses relied on, making the adaptive scheme applicable to non-convex LLM fine-tuning where gradient norms naturally decay over training.
  • The bandwidth-allocation scheme equalizes round completion times across sampled clients, so under the model no client idles while waiting for a straggler, and total time is governed by the expected maximum per-client cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the bound's dependence on the worst-case compression factor $K^2=\max_n \gamma^2/k_n^2$ suggests that a single client with a very small sketching ratio can degrade the theoretical convergence rate as much as a slow communication client; equalizing or upper-bounding $k_n$ across clients could be a simpler heuristic worth testing.
  • A testable extension is to measure whether the assumed quadratic $(k_n/\gamma)^2$ scaling of computation and communication time actually holds on real mobile hardware and wireless links; if compute time has a large fixed overhead or communication cost scales with the number of nonzero entries rather than $k_n^2$, the optimization should be rerun with the measured cost curves.
  • The four calibration runs estimate constants $A$–$D$ once and assume they stay valid through the whole trajectory to target accuracy; an online recalibration scheme, refitting the constants periodically from recent rounds, could keep the choices near-optimal if the loss landscape shifts.
  • The same joint participation-and-update-size optimization could transfer to other parameter-efficient fine-tuning methods with tunable per-client update sizes, such as adapter ranks or pruning ratios, whenever per-round cost is a known function of the update size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an adaptive federated LoRA method for heterogeneous wireless networks. It models federated fine-tuning with independent Bernoulli client sampling and per-client sketching matrices, derives a convergence bound (Theorem 1) for non-convex losses without the bounded-gradient assumption, proposes an adaptive bandwidth allocation scheme, and formulates non-convex problems P2/P3 to choose sampling probabilities q and sketching ratios k so as to minimize wall-clock time to a target accuracy. The experimental section reports 2.8-4.2x speedups over FSLoRA on Qwen2.5-1.5B with the Commonsense170K dataset. The main claimed contributions are the new bound, the joint optimization, and the empirical speedups.

Significance. The topic is timely and the framework is well motivated: jointly optimizing independent client sampling and LoRA sketching under computation and communication heterogeneity is a genuine gap, and the paper addresses it in a realistic wireless setting. The unbiased aggregation step in Appendix A is simple and correct, and the sketching setup is a reasonable object of study. If Theorem 1 and the wall-clock model were valid, the paper would be a useful contribution to federated LLM fine-tuning. However, the central convergence proof has an algebraic gap that invalidates Theorem 1 as stated, the wall-clock scaling model is inconsistent with the stated matrix dimensions, and the constants used in the optimization are empirically calibrated rather than derived. These issues are load-bearing, so the claimed theoretical and practical results are not established as written.

major comments (3)
  1. [III-B, Appendix B, Eq. (34)] The proof of Theorem 1 does not establish the required positivity of the coefficient in Eq. (34). The proof needs 36L^2 eta^2 H^2 N(c_h+1) * sum_n (a_n^2/q_n)(gamma^2/k_n^2) <= 1, but the stated condition eta <= 1/(sqrt(18) K H L) only gives L^2 eta^2 H^2 <= 1/(18 K^2), so the left-hand side is bounded by 2N(c_h+1) sum_n (a_n^2/q_n)(gamma^2/k_n^2)/K^2, which is not <= 1. Even using the second condition eta <= 1/(6 K H L Q sqrt(c_h+1)) with Q^2 = sum_n a_n^2/q_n only yields an upper bound of N, not 1. Concretely, for N=2, c_h=0, K=1, eta=1/(6HLQ), and sum_n (a_n^2/q_n)(gamma^2/k_n^2)=Q^2, the coefficient is exactly zero, so the claimed step that it is at least eta H/4 is false; for N>2 it is negative. Thus Theorem 1 is not proven as stated. In addition, the definition of Q^2 in the theorem statement is malformed: it reads as a maximum over an expression that does not depend on the maximized index.
  2. [V-B, Eqs. (18)-(19)] The communication and computation time models assume tn = t^gamma_n (k_n/gamma)^2 and tau_n = tau^gamma_n (k_n/gamma)^2, i.e., quadratic scaling in the sketching ratio. For the LoRA dimensions stated in the paper, B in R^{m x gamma} and A in R^{gamma x n}, the sketching operation BSA with S having k_n nonzero diagonal entries involves k_n(m+n) nonzero parameters, which scales linearly in k_n, not quadratically. Since the objective of P2/P3 and the greedy optimization of k are built entirely on this quadratic surrogate, the optimized q and k will be miscalibrated if the true scaling is linear. The paper provides no empirical validation of the quadratic scaling on the Qwen2.5-1.5B workloads. This is load-bearing because the claimed wall-clock speedups depend on the round-time model.
  3. [V-D1, Algorithm 2, lines 1-4] The constants A, B, C, D are estimated by running Algorithm 1 with four configurations and fitting the relation R = A/(B - CY - DZ) using SVD. With four unknowns and four observations, the fit exactly interpolates the calibration runs, so the 'predicted' R used in P2/P3 is a calibrated empirical model rather than a derivation from Theorem 1. The paper presents the optimization as theory-guided, but it does not show that the fitted constants remain valid for other configurations or along the whole training trajectory down to the target accuracy. The authors should either determine the constants from the system model or reframe the method as an empirical surrogate and validate the fitted R on held-out configurations.
minor comments (5)
  1. [P2, Eq. (21)] The constraint set includes k_n in {0, ..., gamma}, but k_n = 0 makes gamma^2/k_n^2 undefined; the feasible set should be {1, ..., gamma}.
  2. [Eq. (9) and Section V-C] The quantity F* is used but never defined; it should be defined as the infimum of F^S(X) over X, or as the minimum if it is attained.
  3. [Algorithm 1, line 11 and Eq. (8)] Algorithm 1 says 'Compute the total local gradients Delta B_r^n, Delta A_r^n', but Eq. (8) defines these objects as differences between the initial and final local parameters, not as gradients; the terminology should be aligned.
  4. [Table I and Figs. 2-3] The reported wall-clock times in Table I and the figures do not include error bars or the number of seeds; given the stochasticity from independent sampling and sketching, the paper should report variance or multiple runs.
  5. [Appendix B, Eq. (29)] The phrase 'the last inequality follows from Ls-smoothness' is imprecise; the displayed bound uses the per-client smoothness constant L times gamma/k_n, not the global constant Ls, so the sentence should be clarified.

Circularity Check

1 steps flagged · score 6.0 of 10

The convergence-time predictor used in the optimization is calibrated, not derived: constants A–D are fitted with SVD to the algorithm's own observed round counts, making the P2 objective a fitted surrogate rather than a first-principles prediction.

  1. fitted input called prediction [Section V-D1, Eq. (21)-(23), Algorithm 2 lines 1-4]
    "We first propose a method to estimate the unknown parameters A, B, C, and D. This approach leverages the convergence upper bound derived in Theorem 1 and involves running Algorithm 1 with four sets of fixed sampling probabilities qn and fixed sketching ratios kn ... By substituting these into R = A/(B−CY −DZ), we can obtain a following system: M · x = 0 ... This can be solved to estimate the unknown parameters A, B, C, and D using Singular Value Decomposition (SVD)."

    A, B, C, D are not derived from first principles; they are the SVD null-space solution of M·x = 0, where the rows of M are built from the algorithm's own measured round counts R1..R4 needed to reach a predefined loss Fs under four configurations. The P2 objective then minimizes exactly the same calibrated expression A/(B − CΣ a_n²/q_n − DΣ a_n²γ²/(q_n k_n²)) multiplied by the round-time model. Hence the 'predicted' convergence time used to choose q and k is a fit to the system's own observed behavior, not an independent consequence of Theorem 1. The functional form comes from theory, but the numerical predictor is statistically forced by the calibration runs, making the optimization pipeline partially circular.

full rationale

The main convergence bound and the unbiased-aggregation proof are not circular: Theorem 1, the sketching setup, and Appendix A are self-contained and do not rely on the paper's conclusions. No load-bearing self-citation is present; citations to the authors' earlier work (e.g., [28]) support auxiliary round-time/aggregation arguments and are not the basis of the central bound. The substantive circularity is in the parameter-estimation phase of Algorithm 2: A, B, C, D are fitted via SVD to observed round counts of the same algorithm on the same system, and P2/P3 use the fitted R = A/(B−CY−DZ) as the convergence-time predictor. Thus the 'theory-guided' wall-clock optimization is really a calibrated surrogate optimization; the reported speedups are actual measurements, so the circularity is partial rather than total. Separately, the proof of Theorem 1 appears to contain an algebraic gap in the coefficient check around Eq. (34) (the learning-rate condition seems to miss a factor √N), but that is a correctness concern, not circularity, and is outside the scope of this pass.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The optimization P2/P3 depends on four constants A,B,C,D that are fit to four observed training runs rather than derived from known system parameters; the round-time model assumes quadratic scaling of computation and communication with the sketching ratio; and the convergence bound relies on smoothness and variance assumptions standard in FL plus a smoothness-scaling lemma imported from [39]. No new entities are postulated.

free parameters (4)
  • A = estimated via SVD from four calibration runs
    Section V-D1: A = 4(F^S(X0)-F*)/(η N H (σ_g^2+σ_s^2+σ_h^2)) is fit to observed round counts; not derived from first principles.
  • B = estimated via SVD from four calibration runs
    Section V-D1: B = ξ/(N (σ_g^2+σ_s^2+σ_h^2)) is fit to the target system; it encodes the convergence threshold and unknown variances.
  • C = estimated via SVD from four calibration runs
    Section V-D1: C = 6ηL_s is fit to observed R_i values; L_s depends on the loss landscape and is not measured directly.
  • D = estimated via SVD from four calibration runs
    Section V-D1: D = 36L^2η^2H^2 is fit to observed R_i values; L is the smoothness constant and is not measured directly.
assumptions (6)
  • standard math Local loss functions are differentiable and L-smooth (Assumption 1).
    Standard smoothness assumption in FL optimization proofs; used to bound the decrease of F^S between rounds.
  • domain assumption Bounded variance of sketching and stochastic gradients (Assumption 2).
    Assumes σ_s^2 and σ_g^2 are finite for the sketching distribution and data sampling; common in stochastic optimization but not verified on the actual workloads.
  • domain assumption Bounded gradient dissimilarity between local and global losses (Assumption 3).
    Assumes ∥∇F_n^S(X) - ∇F^S(X)∥^2 ≤ c_h∥∇F^S(X)∥^2 + σ_h^2; this is a relaxation of bounded gradients but still an unverified condition on the data distribution.
  • domain assumption Smoothness scaling ∥∇F_n^S(X) - ∇F_n^S(Y)∥ ≤ L(γ/k_n)∥X-Y∥ imported from [39, Appendix D.4].
    Used in Eq. (29) of Appendix B to bound local drift. The paper does not reproduce the proof, so the bound relies entirely on the cited result.
  • domain assumption Local computation and communication time scale as (k_n/γ)^2 (Eqs. (18)-(19)).
    The round-time model and the optimization P2/P3 assume quadratic scaling of τ_n and t_n with the sketching ratio. This is asserted without empirical validation.
  • domain assumption All sampled clients in a round finish at the same time, giving the minimum round time (Eq. (10)).
    Assumes bandwidth can be allocated so that all participating clients complete simultaneously; imported from [28] and used to derive the expected round-time bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling." pith.science (2026). https://pith.science/paper/TLFTSZCV

@misc{pith2026250523555,
  author       = {Pith},
  title        = {Pith review of: Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLFTSZCV}},
  note         = {Machine review of arXiv:2505.23555}
}
read the original abstract

Federated LoRA has emerged as a promising technique for efficiently fine-tuning large language models (LLMs) on distributed devices by reducing the number of trainable parameters. However, existing approaches often inadequately overlook the theoretical and practical implications of system and data heterogeneity, thereby failing to optimize the overall training efficiency, particularly in terms of wall-clock time. In this paper, we propose an adaptive federated LoRA strategy with independent client sampling to minimize the convergence wall-clock time of federated fine-tuning under both computation and communication heterogeneity. We first derive a new convergence bound for federated LoRA with arbitrary and independent client sampling, notably without requiring the stringent bounded gradient assumption. Then, we introduce an adaptive bandwidth allocation scheme that accounts for heterogeneous client resources and system bandwidth constraints. Based on the derived theory, we formulate and solve a non-convex optimization problem to jointly determine the LoRA sketching ratios and sampling probabilities, aiming to minimize wall-clock convergence time. An efficient and low-complexity algorithm is developed to approximate the solution. Finally, extensive experiments demonstrate that our approach significantly reduces wall-clock training time compared to state-of-the-art methods across various models and datasets.

Figures

Figures reproduced from arXiv: 2505.23555 by the authors.

Figure 1
Figure 1. A heterogeneous federated LoRA over wireless networks, where, clients with heterogeneous computation and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance analysis of the optimization of [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Performance analysis of the optimization of [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages

  1. [1]

    Improving language understanding by generative pre-training

    Alec Radford, Narasimhan K, Salimans T, and Sutskever I. Improving language understanding by generative pre-training. Technical report, OpenAI Technical Report, 2018

    2018

  2. [2]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  3. [3]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023

    work page 2023

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024

    arXiv 2024

  6. [6]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024

    arXiv 2024

  7. [7]

    Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18–20, 2019, proceedings 18 , pages 194–206. Springer, 2019

    2019

  8. [8]

    Fog and iot: An overview of research opportunities

    Mung Chiang and Tao Zhang. Fog and iot: An overview of research opportunities. IEEE Internet of things journal , 3(6):854–864, 2016

    work page 2016

Show all 42 references
  1. [9]

    Federated machine learning: Concept and applications

    Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) , 10(2):1–19, 2019

    2019

  2. [10]

    Ten challenging problems in federated foundation models

    Tao Fan, Hanlin Gu, Xuemei Cao, Chee Seng Chan, Qian Chen, Yiqiang Chen, Yihui Feng, Yang Gu, Jiaxiang Geng, Bing Luo, Shuoling Liu, Win Kent Ong, Chao Ren, Jiaqi Shao, Chuan Sun, Xiaoli Tang, Hong Xi Tae, Yongxin Tong, Shuyue Wei, Fan Wu, Wei Xi, Mingcong Xu, He Yang, Xin Yan...

    2025

  3. [12]

    Machine learning with core ml

    Oge Marques and Oge Marques. Machine learning with core ml. Image Processing and Computer Vision in iOS , pages 29–40, 2020

    2020

  4. [13]

    Federated learning of predictive models from federated electronic health records

    Theodora S Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, and Wei Shi. Federated learning of predictive models from federated electronic health records. International journal of medical informatics , 112:59–67, 2018

    2018

  5. [14]

    Demo: Fedcampus: A real-world privacy-preserving mobile application for smart campus via federated learning & analytics

    Jiaxiang Geng, Beilong Tang, Boyan Zhang, Jiaqi Shao, and Bing Luo. Demo: Fedcampus: A real-world privacy-preserving mobile application for smart campus via federated learning & analytics. In Proceedings of the Twenty-Fifth International Symposium on Theory, Algorithmic Founda...

    2024

  6. [15]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p...

    2024

  7. [16]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691, 2021

    2021 arXiv

  8. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021

    2021 arXiv

  9. [18]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional...

    2021

  10. [19]

    Openfedllm: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and ...

    2024

  11. [20]

    On the convergence of fedavg on non-iid data

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019

    1907 arXiv

  12. [21]

    Client selection for federated learning with heterogeneous resources in mobile edge

    Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019- 2019 IEEE international conference on communications (ICC) , pages 1–7. IEEE, 2019

    2019

  13. [22]

    Cost-effective federated learning in mobile edge networks

    Bing Luo, Xiang Li, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Cost-effective federated learning in mobile edge networks. IEEE Journal on Selected Areas in Communications, 39(12):3606–3621, 2021

    2021

  14. [23]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  15. [24]

    On the convergence of local descent methods in federated learning

    Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425, 2019

    1910 arXiv

  16. [25]

    Optimal client sampling for federated learning

    Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13723, 2020

    2010 arXiv

  17. [26]

    Federated learning under im- portance sampling

    Elsa Rizk, Stefan Vlaski, and Ali H Sayed. Federated learning under im- portance sampling. IEEE Transactions on Signal Processing , 70:5381– 5396, 2022

    2022

  18. [27]

    Cost-effective federated learning design

    Bing Luo, Xiang Li, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Cost-effective federated learning design. In IEEE INFOCOM 2021 - IEEE Conference on Computer Communications , pages 1–10, 2021

    2021

  19. [28]

    Tackling system and statistical heterogeneity for federated learning with adaptive client sampling

    Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE conference on computer communications , pages 1739–1748

    2022

  20. [29]

    Probabilistic client sampling and power allocation for wireless federated learning

    Wen Xu, Ben Liang, Gary Boudreau, and Hamza Sokun. Probabilistic client sampling and power allocation for wireless federated learning. In 2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) , pages 1–6. IEEE, 2023

    2023

  21. [30]

    Communication-efficient device scheduling for federated learning using stochastic optimization

    Jake Perazzone, Shiqiang Wang, Mingyue Ji, and Kevin S Chan. Communication-efficient device scheduling for federated learning using stochastic optimization. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 1449–1458. IEEE, 2022

    2022

  22. [31]

    Integration of large language models and federated learning, 2024

    Chaochao Chen, Xiaohua Feng, Yuyuan Li, Lingjuan Lyu, Jun Zhou, Xiaolin Zheng, and Jianwei Yin. Integration of large language models and federated learning, 2024

    2024

  23. [32]

    Improving loRA in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations , 2024

    2024

  24. [33]

    Federated lora with sparse communication, 2024

    Kevin Kuo, Arian Raje, Kousik Rajesh, and Virginia Smith. Federated lora with sparse communication, 2024

    2024

  25. [34]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  26. [35]

    Heterogeneous LoRA for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Lang...

    2024

  27. [36]

    Towards federated low-rank adaptation of language models with rank heterogeneity

    Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation of language models with rank heterogeneity. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingui...

    2025

  28. [37]

    Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Adv...

    2024

  29. [38]

    Towards robust and efficient federated low-rank adaptation with heterogeneous clients, 2024

    Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients, 2024

    2024

  30. [39]

    Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G. Brinton. Federated sketching lora: On-device collaborative fine-tuning of large language models, 2025

    2025

  31. [40]

    Wenzhi Fang, Dong-Jun Han, Evan Chen, Shiqiang Wang, and Christo- pher G. Brinton. Hierarchical federated learning with multi-timescale gradient correction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural In- fo...

    2024

  32. [41]

    Jones, and Yong Zhou

    Wenzhi Fang, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N. Jones, and Yong Zhou. Communication-efficient stochastic zeroth-order optimiza- tion for federated learning. IEEE Transactions on Signal Processing , 70:5058–5073, 2022

    2022

  33. [42]

    Leung, and Leandros Tassiulas

    Yuang Jiang, Shiqiang Wang, V ´ıctor Valls, Bong Jun Ko, Wei-Han Lee, Kin K. Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems , 34(12):10374–10386, 2023. APPENDIX A PROO...

    2023

  34. [43]

    PN n=1 a2 n qn γ2 k2n ≥ ηH 4 . Thus, we have: E[||∇XF S(Xr)||2] ≤ 4(E[F S(Xr) ]−E[F S(Xr+1) ]) ηH +6η LsN (σ2 g +σ2 s +σ2 h) NX n=1 a2 n qn + 36L2η2H 2N (σ2 g + σ2 s + σ2 h) NX n=1 a2 n qn γ2 k2n (35) Then, we can rearrange the terms yields and telescope from r = {1, ..., R} t...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.