Pith. sign in

REVIEW 3 major objections 6 minor 64 references

Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that jointly assigning each device a LoRA depth (number of adapted layers near the output) and an increasing rank distribution, matched to device compute and upload capacity, makes federated fine-tuning of language…

desk verdict Solid systems paper with a real 80-device testbed, but the headline speedups rest on an unvalidated time model that treats LoRA rank as compute-free. read the letter →

arxiv 2412.20004 v1 pith:4NST7F4K submitted 2024-12-28 cs.DC cs.AIcs.NI

Jun Liu , Yunming Liao , Hongli Xu , Yang Xu , Jianchun Liu , Chen Qian This is my paper · ORCID
classification cs.DCcs.AIcs.NI
keywords federatedfine-tuningLoRAlow-rankadaptationsystemheterogeneityparameter-efficientdepthrankdistributioncommunicationefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that federated fine-tuning of large language models on heterogeneous devices can be made much faster by letting each device adapt a different number of LoRA layers, always taken from the output side of the model, and giving deeper layers progressively higher LoRA ranks. The proposed system, LEGEND, estimates each device's compute and upload speed, then assigns a LoRA depth and an increasing rank distribution that fit within that device's memory and communication budget while keeping the slowest device's completion time in check. On a real testbed of 80 commercial edge devices, LEGEND reaches the same target accuracy 1.5–2.8 times faster and uses about 42.3% less communication traffic than existing federated fine-tuning methods such as FedLoRA, HetLoRA, and FedAdapter. The paper's key claim is that depth and rank distribution are coupled, so they must be chosen jointly rather than separately.

What carries the argument

The load-bearing objects are LoRA depth—the number of consecutive transformer layers counted backward from the output that get trainable low-rank bypass matrices—and rank distribution, the sequence of ranks assigned to those layers. LEGEND couples them through an additive per-round time model $t_i^h = \hat{t}_i + k_i^h \mu_i^h + \sum_{l \in L_i^h} r_{i,l} \beta_i^h$, where $\mu_i^h$ is the measured backward-pass time for one transformer layer and $\beta_i^h$ the upload time per unit rank, and a greedy algorithm (LCD) assigns depth $k_i^h$ from the time gap to the slowest device while drawing ranks from a global arithmetic sequence $r_l = r_{l-1} + \lambda$ that increases toward the output. The same rank per layer across devices keeps aggregation well-defined even though depths differ; the parameter server averages each layer only over the devices that actually trained it.

What would settle it

Measure, on a single edge device, the per-batch backward-pass time and peak memory for LoRA ranks 1, 2, 4, 8, 16, 32, 64, and 128 at a fixed depth, and compare the measured completion time of several depth/rank configurations against the prediction of Eq. (12). If doubling the rank more than doubles the per-layer backward time, or if the sum of per-layer times deviates from the measured total by more than a few percent, then the linear model that the LCD algorithm optimizes is false and the claimed speedup would not transfer to that device.

Watch

Extended reading notes

Core claim

The paper's central claim, on its own terms, is that existing federated LoRA methods waste both compute and communication by adapting all transformer layers with uniform ranks, and that a configuration where only the output-side layers are adapted, with ranks growing toward the output, reaches the same target accuracy much faster. LEGEND realizes this by assigning each device a LoRA depth proportional to its speed, drawing ranks from a global arithmetic sequence that increases toward the output, and aggregating each layer only over the devices that actually trained it. The paper reports that this joint configuration yields a 1.5–2.8x wall-clock speedup and about 42.3% communication-traffic reduction at the target accuracy relative to FedLoRA, HetLoRA, and FedAdapter, in experiments spanning three model sizes and three task categories on a physical WiFi testbed.

Load-bearing premise

The whole allocation rests on the assumption that a device's per-round time is an additive linear sum of a fixed forward-pass time, a per-layer backpropagation time that does not depend on rank, and a per-rank upload time; if memory pressure or kernel efficiency makes LoRA rank change compute time nonlinearly, the greedy depth/rank assignments stop minimizing the completion time they are built on.

Editorial extensions

If this is right

  • If LEGEND is right, a deployment can give weak devices a small LoRA depth and strong devices a large one, cutting per-round waiting time without changing the aggregated model's target accuracy.
  • Because only output-side layers are trained and uploaded, total backpropagation and communication traffic shrink roughly with the fraction of layers skipped, which is where the 1.5–2.8x speedup and about 42.3% traffic saving come from.
  • Consistent per-layer ranks across devices make layer-wise averaging well-defined even when depths differ, so LEGEND avoids the rank-mismatch aggregation failure that the paper attributes to HetLoRA.
  • The ablation results show both factors matter: removing the adaptive depth increases completion time substantially, while removing the increasing rank distribution degrades final accuracy, so the paper's claim is that depth and rank distribution must be optimized together.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to replace the linear time model with measured per-layer cost curves: if memory bandwidth makes high-rank layers slower per unit rank, a convex cost model would change which depth/rank trade-off is optimal.
  • The depth-first insight may transfer to larger models with more layers, but the paper does not sweep model depth; one could check whether the optimal adapted fraction shrinks as models grow, since the same target accuracy may be reachable with a smaller output-side block.
  • Layer-wise aggregation over heterogeneous depths implicitly weights shallow layers by the number of capable devices that trained them; tracking per-layer participation and possibly reweighting by staleness would be a natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LEGEND, a federated fine-tuning framework that assigns each device a LoRA configuration consisting of a depth (number of adapted layers near the output) and a gradually increasing rank distribution. It includes a greedy algorithm (LCD) to determine these configurations from estimated per-device compute and communication capabilities, plus adaptive layer-wise aggregation to handle heterogeneous LoRA depths. The method is evaluated on a physical testbed with 80 Jetson devices connected over WiFi, using RoBERTa, DeBERTa, and Llama2 on GLUE, MMLU, and GSM-8K tasks. The headline claims are a 1.5-2.8x speedup and about 42.3% communication savings when reaching a target accuracy, compared with FedLoRA, HetLoRA, and FedAdapter.

Significance. If the claims hold, LEGEND would be a practically useful contribution to federated fine-tuning on heterogeneous edge devices, where both resource constraints and system heterogeneity are real deployment obstacles. The physical testbed with 80 commercial devices is a genuine strength and goes beyond the simulation-only evaluations common in this area. The paper also provides an empirical observation that deep-position LoRA layers with larger ranks are more effective, which is consistent with results from other recent studies. The main weakness is that the analytical model underpinning the configuration algorithm is not validated and appears inconsistent with the paper's own resource-accounting equations, so the central speedup claim is not yet fully explained.

major comments (3)
  1. [Section 4.4, Eqs. (12) and (14)] The completion-time model in Eq. (12), t_i^h = \hat{t}_i + k_i^h * \mu_i^h + \sum_{l \in L_i^h} r_{i,l} * \beta_i^h, treats the per-layer backpropagation time \mu_i^h as independent of LoRA rank. However, Eq. (14) charges a rank-dependent computing resource consumption \sum r_{i,l} * c, and Section 2.4 explicitly demonstrates performance and resource sensitivity to ranks up to 128. If rank affects per-layer backward time through extra LoRA matmuls, memory pressure, or kernel efficiency, then Algorithm 1's greedy depth adjustment optimizes the wrong objective. The paper reports no per-layer timing measurements that would validate the rank-independence assumption, so the speedup claim is not yet supported by the stated model.
  2. [Algorithm 1, Section 4.4] The LoRA configuration algorithm is under-specified. Line 4 defines an arithmetic rank sequence with r_l = r_{l-1} + \lambda but does not specify the starting value r_0, and the total rank budget \psi is never used to bound the sequence. Line 5 says 'adjust LoRA depth' but provides no procedure, making it unclear how the constraints in Eqs. (14) and (15) are enforced, especially because changing depth without changing ranks may not be sufficient to satisfy a total-rank budget. This makes the algorithm non-reproducible and the claimed optimality unverifiable.
  3. [Section 6.1, 'Metrics'] The evaluation sets the target accuracy as 'the minimum accuracy achieved by the four methods.' This biases the completion-time comparison in favor of a method that saturates at the minimum accuracy, and it obscures differences in final performance. Additionally, no error bars or repeated runs are reported, even though device modes are randomized every 20 rounds and WiFi bandwidth fluctuates dynamically. The headline speedup and communication-savings numbers should be accompanied by standard deviations over at least three independent runs and by results for multiple target accuracies.
minor comments (6)
  1. [Section 4.1] There is a typo: 'for arbitary coupled LoRA matrices' should be 'for arbitrary coupled LoRA matrices'.
  2. [Section 6.2 and Figure 8] The text says LEGEND takes 1,479s to achieve 85% accuracy on SST-2, while Figure 8(a) caption says 'Time to reach 95% accuracy'; please reconcile the target accuracy values.
  3. [Section 6.2] The text says 'by Figure 12(c), the average waiting time of LEGEND is 352s for SST-2', but Figure 12(c) corresponds to QQP; the dataset label appears to be incorrect.
  4. [Algorithm 1, line 2] The expression for the depth gap uses t^h and t_{i,min} without defining t_{i,min}; please clarify the notation and the intended formula.
  5. [Sections 2 and 6] The motivating experiments in Section 2 use the same tasks (SST-2, QNLI) and model family (RoBERTa) as the main evaluation in Section 6, so the inferred 'coupled relationship' between depth and rank is validated on the same task distribution from which it was derived; an independent evaluation on held-out tasks would strengthen the claim.
  6. [Section 1] The paper states that this is 'the first study of federated learning based NLP in a real wireless testbed with heterogeneous devices'; please support this strong claim with a specific comparison to prior testbed studies or soften the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the efficiency claims rest on measured wall-clock and traffic outcomes, not on fitting the model inputs to the outputs.

full rationale

LEGEND's derivation chain is empirically motivated rather than axiomatically derived: Section 2 conducts pretests on RoBERTa/SST-2 to observe that deep LoRA placement and increasing rank distributions give favorable accuracy/resource trade-offs; Section 4 converts these observations into a constrained allocation problem whose completion-time model is Equation (12), t_i^h = \hat t_i + k_i^h \mu_i^h + \sum_{l} r_{i,l} \beta_i^h; Section 6 then measures wall-clock completion time and communication traffic to a common target accuracy. The reported 1.5-2.8x speedup and 42.3% traffic saving are measured outcomes, not predictions generated by Equation (12) from fitted parameters. Equation (12) is an additive engineering model used to choose configurations; its simplifying assumption that per-layer backward time \mu_i^h is independent of rank is a correctness or robustness risk, not a circular reduction, since the paper does not claim to predict the speedup from this equation. The reuse of SST-2 in both the motivating experiments and the main evaluation is an in-sample design concern, but LEGEND is also evaluated on QNLI, QQP, MNLI, MMLU, and GSM-8K against external baselines (FedAdapter, HetLoRA, FedLoRA), so the central claim has independent empirical content. Self-citations such as [8,14,17,19] appear only in background citation lists for heterogeneous federated learning and layer importance; the load-bearing design insight is either the paper's own measurements or external citations [26,35,43]. No equation reduces to its input by construction, no fitted parameter is renamed a prediction, and no uniqueness theorem is imported from the authors. Thus no circularity is found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The system is built on several hand-chosen parameters (lambda, rho, r0) and strong domain assumptions about deep-layer dominance and additive time costs. No new physical entities are introduced.

free parameters (5)
  • lambda (rank increment) = 1 (default)
    Rank distribution is fixed to an arithmetic sequence r_l = r_{l-1} + lambda with lambda = 1; no search or sensitivity analysis is reported, it is chosen by hand in Section 4.4.
  • r0 (initial or shallowest rank) = not reported
    The arithmetic sequence in Algorithm 1 line 4 requires a starting rank r0, which is never specified; this determines the total rank budget used by each device.
  • rho (EMA decay) = 0.8
    Moving average coefficient for capacity estimation in Eqs. 8-9, chosen as an experimental default and not varied.
  • epsilon (waiting time threshold) = not specified
    Constraint W_h <= epsilon in Eq. 16; no value or enforcement mechanism is reported in Algorithm 1.
  • c, b (unit resource costs) = not measured
    Eqs. 14-15 assume per-rank compute and communication costs c and b, but the paper never reports how they are obtained or how Algorithm 1 verifies these constraints.
assumptions (5)
  • domain assumption Fine-tuning only a suffix of deep transformer layers with LoRA achieves accuracy comparable to fine-tuning all layers.
    LEGEND assigns each device a LoRA depth k_i^h covering layers [L-k_i^h, L-1]; the entire scheme depends on the premise that shallow layers need no adaptation, supported only by the motivation experiments in Section 2.2.
  • domain assumption Monotonically increasing ranks toward the output are beneficial; an arithmetic sequence with fixed lambda is a sufficient rank schedule.
    The rank distribution is not optimized, it is fixed by construction to be increasing; this assumes the Inc distribution from Section 2.4 generalizes across models, tasks, and data distributions.
  • domain assumption The per-round time on device i decomposes additively as t_i^h = t_hat_i + k_i^h * mu_i^h + sum r_i,l * beta_i^h (Eq. 12).
    This linear model is the basis for the depth/rank allocation; it ignores nonlinear interactions between rank, memory, batch size, and bandwidth contention.
  • domain assumption Layer-wise averaging over the subset of devices that trained layer l yields a valid global LoRA layer (Eq. 17).
    With heterogeneous depths, each layer is aggregated over a different device subset; the paper provides no convergence or bias analysis for this partial aggregation.
  • domain assumption Partial LoRA fine-tuning converges; the paper relies on FlexLoRA [35] for theoretical support.
    Section 2.2 cites Wei et al. for convergence of partial LoRA; this is an external theorem the paper assumes without proof or adaptation to the heterogeneous-depth setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices." pith.science (2026). https://pith.science/paper/4NST7F4K

@misc{pith2026241220004,
  author       = {Pith},
  title        = {Pith review of: Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NST7F4K}},
  note         = {Machine review of arXiv:2412.20004}
}
abstract

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8$\times$ and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.

Figures

Figures reproduced from arXiv: 2412.20004 by the authors.

Figure 1
Figure 1. Illustration of FedNLP, FedLoRA, and LEGEND. FedNLP (left) fine-tunes all parameters of the LM; FedLoRA (mid) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Fine-tuning RoBERTa at different positions. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. The impact of LoRA depth. memory usage) increases almost linearly, resulting in slow convergence rate. For example, by Figures 4(a) and 4(b), with each additional LoRA layer, the per-batch latency increases by approximately 5ms, and the memory usage increases by approximately 107MB. Compared with LoRA depth of 1, fine￾tuning RoBERTa with depth of 12 results in a 252% increase in per-batch latency and 221% growth in … view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: The impact of LoRA rank distribution. rank budget under resource constraints, strategically allocat￾ing higher ranks to task-relevant layers becomes crucial for maximizing model performance [26, 27]. Specifically, the higher ranks of the deep LoRA layers achieve better…
Figure 7
Figure 7. Figure 7: Fine-tuning process of four approaches on general language understanding tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Completion time of four approaches on general language understanding tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Results on mathematical reasoning tasks. tion traffic consumption of these approaches when achiev￾ing the target accuracy. According to the results in [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Communication traffic of four approaches on general language understanding tasks. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Average waiting time of four approaches on general language understanding tasks. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Effect of LoRA depth and rank distribution. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 31 canonical work pages

  1. [1]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Bi- narybert: Pushing the limit of bert quantization

    Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Bi- narybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020

    arXiv 2012

  3. [3]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [4]

    Dynabert: Dynamic bert with adap- tive width and depth

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adap- tive width and depth. Advances in Neural Information Processing Systems, 33:9782–9793, 2020

    work page 2020

  5. [5]

    Language models are few-shot learn- ers

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Communication- efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–

  7. [7]

    Fednlp: Benchmarking federated learning meth- ods for natural language processing tasks.arXiv preprint arXiv:2104.08815, 2021

    Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Aves- timehr. Fednlp: Benchmarking federated learning meth- ods for natural language processing tasks.arXiv preprint arXiv:2104.08815, 2021

    arXiv 2021

  8. [8]

    Accelerating federated learning with data and model parallelism in edge com- puting

    Yunming Liao, Yang Xu, Hongli Xu, Zhiwei Yao, Lun Wang, and Chunming Qiao. Accelerating federated learning with data and model parallelism in edge com- puting. IEEE/ACM Transactions on Networking, 2023

    work page 2023

Show all 64 references
  1. [9]

    Pretraining federated text models for next word prediction

    Joel Stremmel and Arjun Singh. Pretraining federated text models for next word prediction. In Advances in In- formation and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 477–488. Springer, 2021

    2021

  2. [10]

    Fedadapter: Efficient federated learning for modern nlp

    Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xi- aozhu Lin, and Mengwei Xu. Fedadapter: Efficient federated learning for modern nlp. arXiv preprint arXiv:2205.10162, 2022

    2022 arXiv

  3. [11]

    Adaptive control of local up- dating and model compression for efficient federated learning

    Yang Xu, Yunming Liao, Hongli Xu, Zhenguo Ma, Lun Wang, and Jianchun Liu. Adaptive control of local up- dating and model compression for efficient federated learning. IEEE Transactions on Mobile Computing , 22(10):5675–5689, 2022

    2022

  4. [12]

    Pockengine: Sparse and efficient fine-tuning in a pocket

    Ligeng Zhu, Lanxiang Hu, Ji Lin, Wei-Ming Chen, Wei- Chen Wang, Chuang Gan, and Song Han. Pockengine: Sparse and efficient fine-tuning in a pocket. In Proceed- ings of the 56th Annual IEEE/ACM International Sym- posium on Microarchitecture, pages 1381–1394, 2023

    2023

  5. [13]

    A survey of on-device machine learning: An algorithms and learning theory perspective

    Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A survey of on-device machine learning: An algorithms and learning theory perspective. ACM Transactions on Internet of Things, 2(3):1–49, 2021

    2021

  6. [14]

    Mergesfl: Split federated learning with feature merging and batch size regulation

    Yunming Liao, Yang Xu, Hongli Xu, Lun Wang, Zhiwei Yao, and Chunming Qiao. Mergesfl: Split federated learning with feature merging and batch size regulation. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 2054–2067. IEEE, 2024

    2024

  7. [15]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  9. [17]

    Adaptive configuration for heterogeneous participants in decentralized federated learning

    Yunming Liao, Yang Xu, Hongli Xu, Lun Wang, and Chen Qian. Adaptive configuration for heterogeneous participants in decentralized federated learning. InIEEE INFOCOM 2023-IEEE Conference on Computer Com- munications, pages 1–10. IEEE, 2023

    2023

  10. [18]

    Oort: Efficient federated learn- ing via guided participant selection

    Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learn- ing via guided participant selection. In 15th {USENIX} Symposium on Operating Systems Design and Imple- mentation ({OSDI} 21), pages 19–35, 2021

    2021

  11. [19]

    Yoga: Adaptive layer-wise model aggregation for decentralized feder- ated learning

    Jun Liu, Jianchun Liu, Hongli Xu, Yunming Liao, Zhiyuan Wang, and Qianpiao Ma. Yoga: Adaptive layer-wise model aggregation for decentralized feder- ated learning. IEEE/ACM Transactions on Networking, 2023. 13

    2023

  12. [20]

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models

    Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Annual Meeting of the Association of Computational Linguistics 2023, pages 9...

    2023

  13. [21]

    Parameter- efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter- efficient transfer learning for nlp. In International con- ference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  14. [22]

    Parameter- efficient fine-tuning without introducing new latency

    Baohao Liao, Yan Meng, and Christof Monz. Parameter- efficient fine-tuning without introducing new latency. arXiv preprint arXiv:2305.16742, 2023

    2023 arXiv

  15. [23]

    Caraserve: Cpu-assisted and rank- aware lora serving for generative llm inference

    Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, Xusheng Chen, Yizhou Shan, Binhang Yuan, and Wei Wang. Caraserve: Cpu-assisted and rank- aware lora serving for generative llm inference. arXiv preprint arXiv:2401.11240, 2024

    2024 arXiv

  16. [24]

    A survey on sentiment anal- ysis methods, applications, and challenges

    Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. A survey on sentiment anal- ysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731–5780, 2022

    2022

  17. [25]

    Deep learning–based text classification: a comprehen- sive review

    Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Nar- jes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. Deep learning–based text classification: a comprehen- sive review. ACM computing surveys (CSUR), 54(3):1– 40, 2021

    2021

  18. [26]

    Adaptive budget allocation for parameter-efficient fine- tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine- tuning. In The Eleventh International Conference on Learning Representations, 2022

    2022

  19. [27]

    Heterogeneous lora for fed- erated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous lora for fed- erated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

    2023

  20. [28]

    Autofl: Enabling heterogeneity-aware energy efficient federated learning

    Young Geun Kim and Carole-Jean Wu. Autofl: Enabling heterogeneity-aware energy efficient federated learning. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pages 183–198, 2021

    2021

  21. [29]

    Tackling system and statistical heterogeneity for federated learning with adaptive client sampling

    Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE conference on computer communications, pages 1739–1748. IEEE, 2022

    2022

  22. [30]

    Wikimedia downloads

    Wikimedia Foundation. Wikimedia downloads

  23. [31]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  24. [32]

    Federated fine-tuning of billion- sized language models across mobile devices

    Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang. Federated fine-tuning of billion- sized language models across mobile devices. arXiv preprint arXiv:2308.13894, 2023

    2023 arXiv

  25. [33]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  26. [34]

    Challenges and applications of large language models

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Her- bie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023

    2023 arXiv

  27. [35]

    Flexora: Flexible low rank adap- tation for large language models

    Chenxing Wei, Yao Shu, Ying Tiffany He, and Fei Richard Yu. Flexora: Flexible low rank adap- tation for large language models. arXiv preprint arXiv:2408.10774, 2024

    2024

  28. [36]

    Spottune: transfer learning through adaptive fine-tuning

    Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Pro- ceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4805–4814, 2019

    2019

  29. [37]

    Adaptive budget allocation for parameter-efficient fine- tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine- tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  30. [38]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  31. [39]

    Glue: A multi- task benchmark and analysis platform for natural lan- guage understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi- task benchmark and analysis platform for natural lan- guage understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  32. [40]

    Increasing model capacity for free: A simple strat- egy for parameter efficient fine-tuning

    Haobo Song, Hao Zhao, Soumajit Majumder, and Tao Lin. Increasing model capacity for free: A simple strat- egy for parameter efficient fine-tuning. arXiv preprint arXiv:2407.01320, 2024. 14

    2024 arXiv

  33. [41]

    Lora vs full fine-tuning: An illu- sion of equivalence

    Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illu- sion of equivalence. arXiv preprint arXiv:2410.21228, 2024

    2024

  34. [42]

    Layer-wised model aggregation for personalized feder- ated learning

    Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized feder- ated learning. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10092–10101, 2022

    2022

  35. [43]

    Higher layers need more lora experts

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562, 2024

    2024 arXiv

  36. [44]

    Federated learning: Strategies for improving communi- cation efficiency

    Jakub Konecn `y, H Brendan McMahan, Felix X Yu, Pe- ter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communi- cation efficiency. arXiv preprint arXiv:1610.05492, 8, 2016

    2016 arXiv

  37. [45]

    Federated learning for keyword spotting

    David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. Federated learning for keyword spotting. In ICASSP 2019-2019 IEEE in- ternational conference on acoustics, speech and signal processing (ICASSP), pages 6341–6345. IEEE, 2019

    2019

  38. [46]

    Predictable 802.11 packet delivery from wire- less channel measurements

    Daniel Halperin, Wenjun Hu, Anmol Sheth, and David Wetherall. Predictable 802.11 packet delivery from wire- less channel measurements. ACM SIGCOMM computer communication review, 40(4):159–170, 2010

    2010

  39. [47]

    Linkforecast: Cellu- lar link bandwidth prediction in lte networks

    Chaoqun Yue, Ruofan Jin, Kyoungwon Suh, Yanyuan Qin, Bing Wang, and Wei Wei. Linkforecast: Cellu- lar link bandwidth prediction in lte networks. IEEE Transactions on Mobile Computing, 17(7):1582–1594, 2017

    2017

  40. [48]

    When federated learning meets pre-trained language models’ parameter-efficient tuning methods

    Zhuo Zhang, Yuanhang Yang, Yong Dai, Lizhen Qu, and Zenglin Xu. When federated learning meets pre-trained language models’ parameter-efficient tuning methods. arXiv preprint arXiv:2212.10025, 2022

    2022 arXiv

  41. [49]

    A survey on optimized implementation of deep learning models on the nvidia jetson platform

    Sparsh Mittal. A survey on optimized implementation of deep learning models on the nvidia jetson platform. Journal of Systems Architecture, 97:428–442, 2019

    2019

  42. [50]

    Docker: lightweight linux containers for consistent development and deployment

    Dirk Merkel et al. Docker: lightweight linux containers for consistent development and deployment. Linux j, 239(2):2, 2014

    2014

  43. [51]

    Building a virtual system of systems using docker swarm in multiple clouds

    Nitin Naik. Building a virtual system of systems using docker swarm in multiple clouds. In 2016 IEEE Interna- tional Symposium on Systems Engineering (ISSE), pages 1–3. IEEE, 2016

    2016

  44. [52]

    Py- torch: An imperative style, high-performance deep learn- ing library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Py- torch: An imperative style, high-performance deep learn- ing library. Advances in neural information processing systems,...

    2019

  45. [53]

    Open mpi: Goals, concept, and de- sign of a next generation mpi implementation

    Edgar Gabriel, Graham E Fagg, George Bosilca, Thara Angskun, Jack J Dongarra, Jeffrey M Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, et al. Open mpi: Goals, concept, and de- sign of a next generation mpi implementation. InRecent Advances in Para...

    2004

  46. [54]

    http://dast

    Iperf: The tcp/udp bandwidth measurement tool. http://dast. nlanr. net/Projects/Iperf/, 1999

    1999

  47. [55]

    Transformers: State-of-the-art natural language process- ing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language process- ing. In Proceedings of the 2020 conference on empirical methods in ...

    2020

  48. [56]

    Deber- tav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Deber- tav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021

    2021 arXiv

  49. [57]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  50. [58]

    Deep neu- ral solver for math word problems

    Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neu- ral solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 845–854, 2017

    2017

  51. [59]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  52. [60]

    Language mod- els are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language mod- els are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  53. [61]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. 15

    2021 arXiv

  54. [62]

    Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learn- ing

    Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learn- ing. In ICASSP 2023-2023 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. I...

    2023

  55. [63]

    Qlora: Efficient finetuning of quan- tized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quan- tized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  56. [64]

    Longlora: Effi- cient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Effi- cient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023. 16

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.