FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning

Hongli Xu; Jianchun Liu; Liusheng Huang; Rukuo Li

REVIEW 4 major objections 7 minor 40 references

FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FedQuad claims that assigning each device a configuration of LoRA depth and activation quantization layers—chosen by a greedy server-side algorithm—accelerates federated fine-tuning of large language models by 1.4–5.3× over baselines…

desk verdict Real systems contribution coupling output-side LoRA depth with activation quantization via a greedy solver, but Algorithm 1 needs per-layer gradient norms that the stated protocol never sends—fixable, but the paper cannot be reproduced as written. read the letter →

arxiv 2506.01001 v1 pith:MAXYWJTJ submitted 2025-06-01 cs.DC

Rukuo Li , Jianchun Liu , Hongli Xu , Liusheng Huang This is my paper

classification cs.DC

keywords federatedfine-tuningLoRAdepthactivationquantizationresourceheterogeneityon-deviceLLMparameter-efficientconvergenceaccelerationlayer-wiseadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FedQuad, a federated fine-tuning framework for large language models on heterogeneous, memory-limited edge devices. It claims that assigning each device a tailored configuration—a LoRA depth (how many consecutive output-side transformer layers are trainable) and a number of activation-quantization layers—reduces the wall-clock time to reach target accuracy by 1.4–5.3× compared with state-of-the-art baselines, while keeping the full transformer architecture. The design rests on two measurements: output-side layers contribute most of the fine-tuning accuracy gain, and activation storage dominates memory cost, so quantizing activations frees memory that can buy deeper LoRA tuning. A greedy algorithm (ACS) picks the configuration per device by balancing a gradient-norm-based performance gain against estimated completion time. If correct, resource-constrained federations can fine-tune full models rather than dropping layers or shrinking ranks.

What carries the argument

Adaptive Configuration Selection (ACS): a greedy algorithm that, for each device, enumerates feasible pairs of LoRA depth $d$ and quantized activation layers $a$ satisfying the memory inequality $m_f + m_o d - m_q a \le M_i^h$, then selects the pair maximizing $R(d,a)=G(d)/(t_i^h(d,a)-t_{\mathrm{avg}}^{h-1}+c)$, where $G(d)$ sums the global model's layer-wise gradient norms of the last $d$ layers and $t_i^h$ is estimated from the device's compute throughput. The mechanism rests on the reported empirical regularities that output-side LoRA layers give rising then saturating accuracy gains, activation storage dominates memory, and quantization noise slightly improves generalization; the reward converts these into a per-round assignment that keeps strong devices busy without making weak devices stragglers.

What would settle it

Look for the mechanism that supplies $g_l$ in the implementation: if the server never receives or computes per-layer gradient norms, then $G(d)$ in Algorithm 1 cannot be evaluated as stated. A decisive experiment is to rerun FedQuad with $G(d)$ replaced by a constant; if convergence time does not change, the gradient-gain term is inert and the reported speedup, if it persists, comes from memory and compute allocation rather than from output-side layer selection.

Watch

Extended reading notes

Core claim

FedQuad's central claim is that in federated fine-tuning, LoRA depth (the number of consecutive trainable LoRA layers counting from the output) combined with activation quantization is a per-device control that can nearly eliminate synchronization delay without sacrificing model fidelity. Under a memory constraint $m_f + m_o d - m_q a \le M_i^h$, the server enumerates feasible pairs $(d,a)$, estimates the completion time $t_i^h$ from the device's measured compute capability, computes a performance gain $G(d)=\sum_{l=L-d}^{L-1} g_l$ from layer-wise gradient norms of the global model, and selects the pair maximizing the reward $R(d,a)=G(d)/(t_i^h(d,a)-t_{\mathrm{avg}}^{h-1}+c)$. This yields faster time-to-accuracy than layer-dropping methods, layer-selection methods, and rank-adaptation methods, with reported speedups of 1.4–5.3× across four NLP tasks and three large transformer models.

Load-bearing premise

The server's configurator needs per-layer gradient norms of the global model to compute the gain $G(d)$ (Section 3.3, Eq. 16), yet the protocol in Section 3.1 has devices upload only LoRA weight updates and status reports, leaving it unexplained how the server obtains those gradients.

Editorial extensions

If this is right

If FedQuad works as reported, a federation of heterogeneous edge devices can fine-tune a full transformer without any layer dropping, preserving model fidelity and avoiding output-bias artifacts.
Per-round communication remains low because only LoRA updates and status reports are exchanged, so devices with weak bandwidth can participate.
The greedy configuration selection should keep each device's completion time close to the round average, shrinking the average waiting time (reported 48–78% reductions on the four datasets).
Activation quantization more than pays for itself when its memory savings are reinvested in deeper LoRA tuning, giving accuracy gains (e.g., 4.36% absolute on MNLI for (8,5) vs (5,0)).
The speedup grows with device heterogeneity, so the framework matters most precisely when the federation is most imbalanced.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's protocol as described does not explain how the parameter server obtains the per-layer gradient norms $g_l$ needed for $G(d)$, since devices only upload LoRA deltas and status reports; an implementation must either compute these norms from held-out server-side data or approximate them from the LoRA updates themselves.
The reported regularity that quantization noise improves accuracy suggests an independent testable hypothesis: adding controlled noise to activations should reproduce the observed accuracy gain even without memory pressure, which would separate the regularization effect from the memory-saving effect.
The same configuration-search logic could be applied to other PEFT knobs (LoRA rank, dropout, number of frozen layers) under the same memory-latency constraints, potentially extending FedQuad's acceleration claim beyond depth and quantization.
If gradient norms are expensive or unavailable, a proxy such as per-layer validation-loss sensitivity might make ACS deployable in fully privacy-preserving settings where the server never sees gradients.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a genuine systems contribution with a fixable but real algorithm gap. The novel piece is the joint per-device selection of output-side LoRA depth and the number of activation-quantized layers, balanced by a greedy configuration search. LoRA depth selection, activation quantization, and heterogeneous LoRA rank exist separately in the cited literature; the joint decision under memory and synchronization constraints is new, and framing it as the constrained optimization in Eq. (15) is clean.

The motivation study is the strongest part. The measurements on RoBERTa-base/MNLI actually support the design: output-side tuning is both more resource-efficient and more accurate than input-side tuning, and spending quantization-freed memory on deeper LoRA gives clear gains until saturation. The ablation (FedQuad w/o QD vs w/o LD) shows both components earn their keep, and the layer-wise aggregation rule in Eq. (18) sensibly handles devices training different depths.

The stress-test concern lands; I re-read the protocol to check. Section 3.1 steps 5-6 transmit only updated LoRA parameters and status information. Algorithm 1 and Eq. (16) need per-layer gradient norms g_l of the global model to compute G(d). The server has no data and no gradient-norm message is defined, so true gradient norms are unavailable to it. A proxy—say, the norm of the aggregated per-layer LoRA delta—would be a natural substitute, but the paper does not state one. Because R(d,a) drives every configuration decision, the algorithm cannot be reproduced as written. This is fixable, but it needs explicit resolution in revision.

Other weaknesses are minor-to-moderate and mostly conventional for the subfield: experiments are semi-simulated, memory constraints are abstracted as LoRA depth ranges rather than measured, the constants m_o, m_q, m_f, c, and theta are never calibrated, there are no error bars, and no code is released. The target accuracy is defined as the highest achievable across methods, which mildly favors whichever method lands highest. The headline 1.4-5.3x numbers are internally consistent across the tables, and the memory/compute model is at least plausible.

Audience: systems and federated-learning researchers working on fine-tuning LLMs on heterogeneous edge fleets. It deserves a serious referee, conditional on resolving where g_l comes from. My recommendation is to send it to review; I expect major revision, not reject, because the core idea and ablations survive the algorithmic cleanup.

Referee Report

4 major / 7 minor

Summary. The paper proposes FedQuad, a federated fine-tuning framework for LLMs on resource-constrained, heterogeneous devices. Each device is assigned a configuration (d, a), where d is the number of consecutive unfrozen LoRA layers counting back from the output layer and a is the number of activation-quantized layers. A greedy algorithm (ACS, Algorithm 1) selects configurations by maximizing a reward R(d,a)=G(d)/(t_i^h - t_avg^{h-1} + c), where G(d) is a sum of per-layer gradient norms, subject to memory and average-waiting-time constraints. The authors report 1.4–5.3x convergence acceleration over FedRA, InclusiveFL, LayerSel, and HetLoRA on GLUE tasks with BERT-large, RoBERTa-large, and DeBERTaV3-large. The paper also provides motivation experiments showing that activation storage dominates memory and that tuning deeper layers is more efficient, and it describes a layer-wise adaptive aggregation rule.

Significance. If the claimed speedups hold, FedQuad would be a practical contribution to federated fine-tuning of LLMs on heterogeneous edge devices, combining two complementary mechanisms (adaptive LoRA depth and activation quantization) while preserving the full transformer architecture. The manuscript has notable strengths: a concrete system implementation with Triton-based quantization, careful motivation experiments (Figs. 1–4), and comparisons against four relevant baselines across multiple models and datasets. However, the central algorithmic step is not executable from the paper as written because the reward function requires per-layer gradient norms at the parameter server that no stated protocol message provides, and several key constants of the cost and memory models are unspecified. The empirical evaluation is semi-simulated, single-run, and uses an ambiguous target-accuracy definition, so the quantitative speedup claims are not yet reproducible. These issues are load-bearing for the paper's central claim rather than cosmetic.

major comments (4)

[Section 3.3, Eq. (16), Algorithm 1 (lines 15–16)] The reward function R(d,a) in Eq. (17) is defined via G(d) = sum_{l=L-d}^{L-1} g_l, where g_l is the per-layer gradient norm of the global model at the parameter server. However, the protocol in Section 3.1 (steps 5–6) and the status information in Section 3.2 transmit only updated LoRA parameters and memory/compute status from devices to the server; no gradient, gradient-norm, or proxy-data message is defined. Under standard federated assumptions the server cannot compute true per-layer gradient norms without access to local data. This makes the greedy configuration selection unimplementable as described and prevents reproduction of the reported convergence results. Please specify exactly how g_l is obtained, or replace Eq. (17) with a clearly defined proxy (e.g., norms of aggregated LoRA deltas) and validate that proxy against the stated accuracy rationale.
[Section 3.3, Eqs. (10), (13), (17)] The algorithm depends on several constants that are never assigned values or given a calibration procedure: the memory coefficients m_o, m_q, m_f in Eq. (10); the waiting-time threshold θ in Eq. (13); the reward stabilization constant c in Eq. (17); and the linear coefficients of the computational complexity C(d,a) in Eq. (6). These constants determine the feasible set C_i and the ranking of configurations, so the configuration choices and the resulting convergence times cannot be reproduced without them. Please report the values used in all experiments, or describe how they are measured on the Jetson devices, including the per-layer memory and latency measurements behind Figs. 2–4.
[Section 4.1, Metrics; Section 4.2, Table 3] The time-to-accuracy metric is defined by setting 'the target accuracy as the highest achievable accuracy by FedQuad and all baselines.' This is ambiguous: if it means the minimum of the final accuracies across methods, the target depends on the weakest baseline; if it means something else, it should be stated explicitly. Table 3 reports different targets for different models (81%, 88%, 87%) without explaining how they were determined. Because the 1.4–5.3x speedup claim is measured with respect to this target, the target definition is load-bearing. Please define the target independently of the experimental outcome and justify that all methods are compared at the same target.
[Section 4.1, Section 4.2] All numerical results appear to come from a single run with no error bars or multiple seeds, and the evaluation is semi-simulated (eight A6000 GPUs with timing measurements from Jetson devices). Since configuration decisions involve random adjustments of memory depth ranges and random switching of device operating modes every 10 rounds, the reported differences—especially the smaller ones such as 1.29x over HetLoRA—need variance estimates to be convincing. Please report mean and standard deviation over at least three independent runs, or justify why a single run is representative given the stochastic configuration process.

minor comments (7)

[Algorithm 1, line 13] Step 2 references 'µ^{h-1}_i, b^{h-1}_i' but Section 3.2 defines only q_i^h for computational capability and Eq. (6) uses u_i^h; the symbols µ and b are not defined. Please align the notation.
[Algorithm 1, line 5] The variable d_max is used to bound the outer loop over LoRA depth but is never defined; presumably it should be L or derived from the device memory constraint.
[Fig. 7 caption] The caption says 'four approaches' while the figure shows FedQuad and four baselines (five methods total); please correct the caption.
[Section 3.3, Eq. (9)] The term Δw^* in Eq. (9) is not defined; 'the optimal fine-tuning performance' should be made precise, e.g., as the minimizer of F(w0, ·) over the feasible LoRA parameter space.
[Section 3.1, steps 5–6; Section 2.1, Eqs. (4)–(5)] The text alternates between saying devices upload 'updated LoRA parameters' and describing aggregation of 'gradients' Δω. Please clarify whether devices send LoRA parameter deltas or raw gradients, since this affects the interpretation of Eq. (5) and the communication cost claim.
[Fig. 8(d)] The subfigure title has an extra closing parenthesis: 'Time to reach 95% accuracy)'. Please fix.
[Related Work, Section 5] The text cites 'FedAdapter [9]' but reference [9] is the FedPETuning paper; please verify the citation and distinguish the FedAdapter method from the benchmark framework.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: FedQuad's convergence-speedup claim is an empirical comparison against external baselines, not a consequence of its own definitions or fitted parameters.

full rationale

FedQuad's central claim—1.4–5.3x convergence acceleration—is established by wall-clock comparisons against FedRA, InclusiveFL, LayerSel, and HetLoRA (Section 4.2, Figs. 7–8, Table 3). These baselines are external, and the target accuracy is defined independently as the highest achievable accuracy across methods, so the speedup is not built into the method's own definitions or equations. The configuration-selection heuristic (Eqs. 16–17) uses layer-wise gradient norms as a proxy for performance gain; this is an optimization heuristic rather than a fitted parameter or a prediction derived from the paper's own outputs, and its validity is tested empirically against baselines rather than assumed by construction. The main concern raised by a careful reading—Algorithm 1 requires server-side per-layer gradient norms although Section 3.1 transmits only LoRA deltas and status information—is a reproducibility/correctness gap rather than a circular dependency, because nothing in the protocol defines those gradient norms in terms of the claimed speedup. Citations to the authors' prior work (e.g., [12], [18], [29], [38]) appear in experimental setup and efficiency discussions but are not load-bearing for the main convergence claim. No step of the derivation reduces to its own input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central speedup claim rests on a reward function that requires global gradient norms not supplied by the protocol, on uncalibrated memory and compute models, and on a simulation that maps device memory to hand-assigned LoRA depth ranges. These are load-bearing assumptions that should be reported with values and sensitivity analysis before the speedup claim is treated as established.

free parameters (5)

c (reward stabilization constant) = not specified
Appears in the denominator of reward function R(d,a) in Eq. (17). Its value changes the configuration ranking, but no value or calibration is reported.
theta (average waiting time threshold) = not specified
Used in constraint Eq. (13). The chosen value determines how aggressively weak devices are limited, but it is not reported.
memory coefficients m_o, m_q, m_f = not reported
Eq. (10) models memory as m_f + m_o*d - m_q*a. Values would need profiling per model and hardware; the experiments instead represent memory indirectly through LoRA depth ranges.
compute cost slopes for C(d,a) = not reported
Eq. (6) says computational complexity is a linear function of d and a, but no formula or fitted coefficients are given.
LoRA depth ranges by device category = strong 18-24, moderate 11-17, weak 4-10
Section 4.1 uses these hand-assigned ranges to emulate memory capacity; the reported speedups depend on them.

assumptions (5)

domain assumption Increasing LoRA depth, defined as the number of trainable consecutive layers from the output, improves accuracy with diminishing returns.
Section 2.3 and Fig. 3 justify this on one model (RoBERTa-base, MNLI); the tradeoff is central to the method.
domain assumption Activation quantization of GELU, MatMul, Softmax, and LayerNorm activations preserves accuracy while saving memory.
Section 2.4 and 4.1 rely on Jetfire and SLIMFIT; no systematic accuracy and memory tradeoff study is provided across all models and datasets.
ad hoc to paper Per-layer gradient norms computed on the global model measure each layer's marginal contribution to convergence.
Eq. (16) assumes this proxy. It is imported from LayerSel-style intuition and is not validated in FedQuad's quantization and depth setting.
ad hoc to paper Quantizing activations of the final output layer degrades accuracy, so a is restricted to [0, d-1].
Stated in Section 3.3 without supporting data; this constraint shapes the feasible configuration set.
domain assumption The linear cost model C(d,a) and the memory model in Eq. (10) capture real device time and memory consumption.
Sections 3.2 and 3.3 use these models to estimate completion times in Algorithm 1, but no coefficients or validation are given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning." pith.science (2026). https://pith.science/paper/MAXYWJTJ

@misc{pith2026250601001,
  author       = {Pith},
  title        = {Pith review of: FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAXYWJTJ}},
  note         = {Machine review of arXiv:2506.01001}
}

read the original abstract

Federated fine-tuning (FedFT) provides an effective paradigm for fine-tuning large language models (LLMs) in privacy-sensitive scenarios. However, practical deployment remains challenging due to the limited resources on end devices. Existing methods typically utilize parameter-efficient fine-tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), to substantially reduce communication overhead. Nevertheless, significant memory usage for activation storage and computational demands from full backpropagation remain major barriers to efficient deployment on resource-constrained end devices. Moreover, substantial resource heterogeneity across devices results in severe synchronization bottlenecks, diminishing the overall fine-tuning efficiency. To address these issues, we propose FedQuad, a novel LoRA-based FedFT framework that adaptively adjusts the LoRA depth (the number of consecutive tunable LoRA layers from the output) according to device computational capabilities, while employing activation quantization to reduce memory overhead, thereby enabling efficient deployment on resource-constrained devices. Specifically, FedQuad first identifies the feasible and efficient combinations of LoRA depth and the number of activation quantization layers based on device-specific resource constraints. Subsequently, FedQuad employs a greedy strategy to select the optimal configurations for each device, effectively accommodating system heterogeneity. Extensive experiments demonstrate that FedQuad achieves a 1.4-5.3x convergence acceleration compared to state-of-the-art baselines when reaching target accuracy, highlighting its efficiency and deployability in resource-constrained and heterogeneous end-device environments.

Figures

Figures reproduced from arXiv: 2506.01001 by the authors.

**Figure 2.** The impact of LoRA position on fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The impact of LoRA depths on fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: The impact of activation quantization on fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Overview of FedQuad’s workflow. tions to each device. Specifically, in the h-th training round, FedQuad determines a tailored configuration (d h i , ah i ) for each device i according to its resource constrains, where d h i represents the assigned LoRA depth and a h i …

**Figure 6.** Figure 6: An illustrative example of FedQuad. After local model adjustment, device i trains the model on the local dataset Di . 3.5 LoRA Aggregation Upon receipt of the updated LoRA parameters from all devices, the PS performs global aggregation. Because devices may fine-tune d…

**Figure 7.** Figure 7: Test accuracy of four approaches on the four datasets. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: Completion time of four approaches when achieving the target accuracy. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Average waiting time of four approaches on the four datasets. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: Effect of LoRA depth and activation quantization. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

[1]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

Deepseek-v3 technical report,

A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

arXiv 2024
[3]

How can we know what language models know?

Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, “How can we know what language models know?”Transactions of the Association for Computational Linguistics, vol. 8, pp. 423–438, 2020

work page 2020
[4]

Sentiment analysis in the era of large language models: A reality check,

W. Zhang, Y. Deng, B. Liu, S. J. Pan, and L. Bing, “Sentiment analysis in the era of large language models: A reality check,” arXiv preprint arXiv:2305.15005, 2023

arXiv 2023
[5]

Prompting large language model for machine translation: A case study,

B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 41 092–41 110

work page 2023
[6]

Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,

J. Huang, J.-X. Bai, X. Zhang, Z. Liu, Y. Feng, J. Liu, X. Sun, M. Dong, and M. Li, “Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,”IEEE Transactions on Information Forensics and Security, 2024

2024
[7]

The eu general data protec- tion regulation (gdpr),

P . Voigt and A. Von dem Bussche, “The eu general data protec- tion regulation (gdpr),”A practical guide, 1st ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017

work page 2017
[8]

Fednlp: Benchmarking federated learning methods for natural language processing tasks,

B. Y. Lin, C. He, Z. Zeng, H. Wang, Y. Huang, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, and S. Avestimehr, “Fednlp: Benchmarking federated learning methods for natural language processing tasks,”arXiv preprint arXiv:2104.08815, 2021

arXiv 2021

Show all 40 references

[9]

Fedpetuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,

Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “Fedpetuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,” in Annual Meeting of the Association of Computational Linguistics 2023. Association for Computation...

2023
[10]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[11]

A survey on optimized implementation of deep learn- ing models on the nvidia jetson platform,

S. Mittal, “A survey on optimized implementation of deep learn- ing models on the nvidia jetson platform,”Journal of Systems Architecture, vol. 97, pp. 428–442, 2019

2019
[12]

Finch: Enhancing federated learning with hierarchical neural architecture search,

J. Liu, J. Yan, H. Xu, Z. Wang, J. Huang, and Y. Xu, “Finch: Enhancing federated learning with hierarchical neural architecture search,”IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 6012–6026, 2023

2023
[13]

Fedadapter: Efficient federated learning for modern nlp,

D. Cai, Y. Wu, S. Wang, F. X. Lin, and M. Xu, “Fedadapter: Efficient federated learning for modern nlp,”arXiv preprint arXiv:2205.10162, 2022

2022 arXiv
[14]

How much ram does your android phone really need in 2025?

Android Authority, “How much ram does your android phone really need in 2025?” 2025, accessed: Mar. 16, 2025. [Online]. Available: https://www.androidauthority.com

2025
[15]

Parameter- efficient transfer learning for nlp,

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Larous- silhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter- efficient transfer learning for nlp,” inInternational conference on machine learning. PMLR, 2019, pp. 2790–2799

2019
[16]

Lora: Low-rank adaptation of large language models,

E. J. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[17]

Roberta: A ro- bustly optimized bert pretraining approach,

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A ro- bustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

1907 arXiv
[18]

Adaptive local update and neural composition for accelerating federated learning in heterogeneous edge networks,

J. Liu, J. Yan, J. Qi, H. Xu, S. Wang, C. Qiao, and L. Huang, “Adaptive local update and neural composition for accelerating federated learning in heterogeneous edge networks,”IEEE Trans- actions on Networking, 2025

2025
[19]

Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients,

S. Su, B. Li, and X. Xue, “Fedra: A random allocation strategy for federated tuning to unleash the power of heterogeneous clients,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 342–358

2024
[20]

No one left behind: Inclusive federated learning over heterogeneous devices,

R. Liu, F. Wu, C. Wu, Y. Wang, L. Lyu, H. Chen, and X. Xie, “No one left behind: Inclusive federated learning over heterogeneous devices,” inProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 3398–3406

2022
[21]

Dropbp: accelerating fine-tuning of large language models by dropping backward propagation,

S. Woo, B. Park, B. Kim, M. Jo, S. J. Kwon, D. Jeon, and D. Lee, “Dropbp: accelerating fine-tuning of large language models by dropping backward propagation,”arXiv preprint arXiv:2402.17812, 2024

2024 arXiv
[22]

Exploring se- lective layer fine-tuning in federated learning,

Y. Sun, Y. Xie, B. Ding, Y. Li, and J. Zhang, “Exploring se- lective layer fine-tuning in federated learning,”arXiv preprint arXiv:2408.15600, 2024

2024 arXiv
[23]

Het- erogeneous lora for federated fine-tuning of on-device foundation models,

Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, M. Barnes, and G. Joshi, “Het- erogeneous lora for federated fine-tuning of on-device foundation models,” inInternational Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

2023
[24]

A stochastic approximation method,

H. Robbins and S. Monro, “A stochastic approximation method,” The annals of mathematical statistics, pp. 400–407, 1951

1951
[25]

Fedlora: When personalized federated learning meets low-rank adapta- tion,

X. Wu, X. Liu, J. Niu, H. Wang, S. Tang, and G. Zhu, “Fedlora: When personalized federated learning meets low-rank adapta- tion,” 2024

2024
[26]

Glue: A multi-task benchmark and analysis platform for natural language understanding,

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,”arXiv preprint arXiv:1804.07461, 2018

2018 arXiv
[27]

Triton: an intermediate language and compiler for tiled neural network computations,

P . Tillet, H.-T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” inProceed- ings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019, pp. 10–19

2019
[28]

Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization,

H. Xi, Y. Chen, K. Zhao, K. J. Teh, J. Chen, and J. Zhu, “Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization,”arXiv preprint arXiv:2403.12422, 2024

2024 arXiv
[29]

Adaptive control of local updating and model compression for efficient federated learning,

Y. Xu, Y. Liao, H. Xu, Z. Ma, L. Wang, and J. Liu, “Adaptive control of local updating and model compression for efficient federated learning,”IEEE Transactions on Mobile Computing, vol. 22, no. 10, pp. 5675–5689, 2022

2022
[30]

Gora: Gradient-driven adaptive low rank adaptation,

H. He, P . Ye, Y. Ren, Y. Yuan, and L. Chen, “Gora: Gradient-driven adaptive low rank adaptation,”arXiv preprint arXiv:2502.12171, 2025

2025
[31]

Transformers library,

Hugging Face, “Transformers library,” https://github.com/ huggingface/transformers, 2024, accessed: 2025-04-17

2024
[32]

Slimfit: Memory-efficient fine-tuning of transformer-based models using training dynamics,

A. Ardakani, A. Haan, S. Tan, D. T. Popovici, A. Cheung, C. Iancu, and K. Sen, “Slimfit: Memory-efficient fine-tuning of transformer-based models using training dynamics,”arXiv preprint arXiv:2305.18513, 2023

2023 arXiv
[33]

Pytorch: An im- perative style, high-performance deep learning library,

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An im- perative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

2019
[34]

Bert: Pre- training of deep bidirectional transformers for language under- standing,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,”arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[35]

Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing,

P . He, J. Gao, and W. Chen, “Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing,”arXiv preprint arXiv:2111.09543, 2021

2021 arXiv
[36]

Pre-trained weights for llms,

Hugging Face, “Pre-trained weights for llms,” https:// huggingface.co/models, 2025, accessed: 2025-03-30

2025
[37]

{FwdLLM}: Efficient federated finetuning of large language models with perturbed inferences,

M. Xu, D. Cai, Y. Wu, X. Li, and S. Wang, “{FwdLLM}: Efficient federated finetuning of large language models with perturbed inferences,” in2024 USENIX Annual Technical Conference (USENIX ATC 24), 2024, pp. 579–596

2024
[38]

Enhancing semi-supervised federated learning with progressive training in heterogeneous edge computing,

J. Liu, J. Liu, H. Xu, Y. Liao, Z. Yao, M. Chen, and C. Qian, “Enhancing semi-supervised federated learning with progressive training in heterogeneous edge computing,”IEEE Transactions on Mobile Computing, 2024

2024
[39]

Heterogeneous lora for federated fine-tuning of on-device foundation models,

Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” arXiv preprint arXiv:2401.06432, 2024

2024 arXiv
[40]

Adam: A method for stochastic optimiza- tion,

D. P . Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,”arXiv preprint arXiv:1412.6980, 2014

2014 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

Deepseek-v3 technical report,

A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

arXiv 2024

[3] [3]

How can we know what language models know?

Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, “How can we know what language models know?”Transactions of the Association for Computational Linguistics, vol. 8, pp. 423–438, 2020

work page 2020

[4] [4]

Sentiment analysis in the era of large language models: A reality check,

W. Zhang, Y. Deng, B. Liu, S. J. Pan, and L. Bing, “Sentiment analysis in the era of large language models: A reality check,” arXiv preprint arXiv:2305.15005, 2023

arXiv 2023

[5] [5]

Prompting large language model for machine translation: A case study,

B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 41 092–41 110

work page 2023

[6] [6]

Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,

J. Huang, J.-X. Bai, X. Zhang, Z. Liu, Y. Feng, J. Liu, X. Sun, M. Dong, and M. Li, “Keystrokesniffer: An off-the-shelf smart- phone can eavesdrop on your privacy from anywhere,”IEEE Transactions on Information Forensics and Security, 2024

2024

[7] [7]

The eu general data protec- tion regulation (gdpr),

P . Voigt and A. Von dem Bussche, “The eu general data protec- tion regulation (gdpr),”A practical guide, 1st ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017

work page 2017

[8] [8]

Fednlp: Benchmarking federated learning methods for natural language processing tasks,

B. Y. Lin, C. He, Z. Zeng, H. Wang, Y. Huang, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, and S. Avestimehr, “Fednlp: Benchmarking federated learning methods for natural language processing tasks,”arXiv preprint arXiv:2104.08815, 2021

arXiv 2021