REVIEW 4 major objections 5 minor 56 references
LLM-QFL: Distilling Large Language Model for Quantum Federated Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that distilling a locally fine-tuned large language model into quantum federated learning—using the LLM as both teacher and controller—yields an O(1/T) convergence rate while cutting communication rounds and gradient…
desk verdict The LLM-as-controller idea is novel and worth a look, but the convergence proof is a restatement of FedAvg with no handling of the adaptive steps or biased client selection, and the experiments are too weak to support the claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the locally fine-tuned LLM used as both a distillation teacher and a controller. The local update combines gradient descent on the client's quantum model with a distillation term $\lambda K(\theta_g, \theta_i^{t+1})$ that pulls the local model toward the global teacher. The controller uses the loss ratio $L_i^t / L_{\text{LLM}}^t$ to set the number of local optimizer steps, $K_i^t = \text{iter} \cdot L_i^t / L_{\text{LLM}}^t$, so a client whose quantum loss exceeds the LLM benchmark runs more COBYLA iterations; the selection rule is $S_t = \{i : d_i^t \text{ among the smallest } k\%\}$ with $d_i^t = |L_i^t - L_g^t|$. The proof's mechanism is the standard FedAvg error decomposition of equation (9), $\mathbb{E}\|(\theta^{t+1}, \phi^{t+1}) - (\theta^*, \phi^*)\|^2 \leq (1 - \eta_t \mu)\mathbb{E}\|(\theta^t, \phi^t) - (\theta^*, \phi^*)\|^2 + \eta_t^2 (B + C)$, where $B$ regroups client variance, the non-IID gap, and local-step drift and $C$ captures the variance of the selected subset; the paper assumes this decomposition carries over from fixed-step FedAvg to the adaptive-step, alignment-selected setting.
What would settle it
Derive the drift and variance terms in inequality (9) for $K_i^t = \text{iter} \cdot L_i^t / L_{\text{LLM}}^t$ and for $S_t$ selected by smallest $|L_i^t - L_g^t|$; if the combined term $B+C$ grows with the spread of local losses or with the selection threshold, the claimed $O(1/T)$ rate does not follow. As an empirical check, run LLM-QFL and standard QFL with the same total optimizer iterations and compare communication rounds to a fixed accuracy: the adaptive method must need no more rounds.
Extended reading notes
Core claim
The central claim is that LLM-guided regulation of quantum federated learning improves both convergence speed and communication efficiency. Concretely, each client first fine-tunes a small LLM on its private data with LoRA or QLoRA, and in later rounds the LLM acts as a benchmark: the optimizer's maximum iteration count is scaled by the loss ratio $L_i^t / L_{\text{LLM}}^t$, so lagging clients take more local steps; the server averages only the $k\%$ of clients whose losses are closest to the server loss; and rounds stop when the relative server-loss improvement falls below $\epsilon$. Theorem VI.4 asserts that under L-smoothness, bounded local variance, and $\mu$-strong convexity, the global objective satisfies $\mathbb{E}[F(\theta^T, \phi^T)] - F^* \leq \frac{2L}{\mu}\frac{\Psi}{T + \gamma}$, and Corollary VI.8 translates this into a reduction in communication rounds by $\mathbb{E}[K^t]/K$ and in global-update variance by $(1 - k/N)$ compared with standard QFL. The experiments on genomic and TweetEval sentiment classification, run on simulator backends and on a real IBM quantum processor, are offered as evidence that these mechanisms converge in fewer communication rounds and reduce idle computation.
Load-bearing premise
The argument rests on a standard convergence bound for federated averaging still holding after the paper lets each client run a different number of local steps and averages only the best-aligned clients, without re-deriving the error terms for those changes.
Editorial extensions
If this is right
- Under the paper's assumptions, the global model reaches $\epsilon$-accuracy in $O\!\left(\frac{L}{\mu}\log\frac{1}{\epsilon} + \frac{B+C}{\mu\epsilon}\right)$ communication rounds, preserving the FedAvg-style rate while adding adaptivity.
- Adaptive local steps reduce the number of communication rounds by the factor $\mathbb{E}[K^t]/K$ compared with fixed-step QFL, because lagging clients take more optimizer iterations per round.
- Selecting only the $k\%$ of clients closest to the server loss reduces the variance of the global update by at most $1 - k/N$ relative to random selection, which should make convergence less sensitive to outlier clients.
- Early stopping when the relative server-loss improvement drops below $\epsilon$ removes rounds that contribute little, lowering total communication cost.
- LoRA and QLoRA fine-tuning keep the LLM controller small enough to run on quantum-classical devices with limited memory, making the scheme scalable to resource-constrained settings.
Reading between the lines
- The controller pattern—loss-ratio-scaled local steps plus alignment-based client selection—does not depend on the local model being quantum; the same design could be tested in classical federated LLM fine-tuning, where communication cost is also dominated by rounds and uploads.
- The claimed variance reduction assumes the server can rank clients by a loss that reflects true generalization; a testable extension would compare selection by training loss, validation loss, and LLM-consistency scores to see which proxy preserves the $(1 - k/N)$ benefit.
- Because the real-hardware experiments were limited to a few communication rounds by usage quotas, a natural follow-up is to run the adaptive controller for many rounds under a fixed noise model; this would show whether early-round gains persist after quantum noise accumulates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LLM-QFL, a framework that integrates large language models into quantum federated learning (QFL) by locally fine-tuning an LLM on each client, distilling it into quantum models, and using it as a controller to adjust optimizer iterations, select clients by alignment with the global model, and stop training early. The authors claim a convergence rate of O(1/T) under L-smoothness, bounded variance, and strong convexity, as well as communication and variance reductions. Experiments are reported on genomic (DemoHumanOrWorm) and TweetEval-sentiment datasets using VQC and QCNN models, with simulations and real IBM quantum hardware.
Significance. If the theoretical claims were valid, the paper would offer a useful bridge between LLM-based adaptive control and quantum federated learning, with practical implications for communication efficiency and client selection. The authors deserve credit for running experiments on real IBM quantum hardware and for releasing code, which is uncommon and valuable. However, the central theoretical contribution is not established: the convergence theorem is imported from standard FedAvg analysis without addressing the adaptive local steps and biased client selection that define the algorithm, and a key remark assumes the very inequality the paper aims to prove, while an appendix figure contradicts it. The experimental evidence, while suggestive, is also limited by near-chance accuracies on the genomic task and a small number of communication rounds on real hardware.
major comments (4)
- [Appendix A, Theorem VI.4] The proof of Theorem VI.4 transfers the FedAvg convergence bound of Li et al. [45] to Algorithm 1, but the transfer is not justified: equation (8) defines local updates with an adaptive number of local steps K_i^t = iter * L_i^t / L_LLM and aggregation over the alignment-selected set S_t = argmin_{|S|=K} sum_{i in S} |L_i^t - L_g^t|, whereas the bound in [45] requires a fixed number E of local steps and uniform random (or full) participation to keep the aggregated update unbiased and to control client drift. Equation (9) is asserted without deriving the drift and variance terms for these modified choices, so the claimed O(1/T) rate is unsupported.
- [Appendix A, Theorem VI.4, constants B and C] The constants B and C in Theorem VI.4 contain terms (E-1)^2 G^2 and E^2 G^2, but the adaptive step count K_i^t = iter * L_i^t / L_LLM is not bounded above unless L_LLM is uniformly bounded away from zero; no such control is stated or proved. Consequently, inequality (9) cannot be obtained from the fixed-E derivation in [45], and the theorem's statement is not supported by the proof.
- [Section II.B, Remark II.1] Remark II.1 assumes the inequality l(·,·) < l_LLM(·,·) as a consequence of distillation, but this inequality is the central performance claim the paper aims to establish, so the remark is circular. Moreover, Figure 23 in Appendix H reports that the LLM's loss is lower than the quantum model's loss in all three settings, which directly contradicts the assumed inequality and further undermines the remark's validity.
- [Appendix A, Corollary VI.8] The efficiency corollary is not established. Claim 1 reduces the communication-round improvement to the definitional identity E[K_i^t] = E[iter * L_i^t / L_LLM] >= K, which holds only when L_i^t >= L_LLM; it does not follow from Theorem VI.4 because the theorem did not prove a convergence rate for the adaptive-selection algorithm. Claim 2 bounds Var(∇F_LLM-QFL) by (1 - k/N) Var(∇F_QFL) using a Markov-type argument on sorted deviations d_i^t, but the variance term in the FedAvg bound is not this population quantity, and the selection rule makes the aggregated gradient biased, so the comparison with random selection is not valid.
minor comments (5)
- [Algorithm 1, line 22] The notation N' = n/N is unclear; this appears intended to denote a count of selected devices, but as written it is a fraction of N, and the subsequent averaging over N' devices is ambiguous.
- [Equation (3)] Equation (3) uses the same index i on both sides of the update, so the expression theta^{t+1}_i = sum_i w_i theta^t_i is not well defined; the aggregation should use distinct client indices.
- [Table I] The reported train and test accuracies for the genomic task are close to 0.5 for several simulator and real-hardware runs, which is near chance level; the authors should discuss whether the method actually learns the task under those settings.
- [Section IV.C and Appendix C] There is an inconsistency in the spelling of the IBM hardware name ('IBM Brisbane' in the main text and 'IBM Bribane' in Appendix C), and the paper should use a consistent citation for the LLaMA model, since reference [35] is the original LLaMA paper rather than the Meta-LLAMA 3.2-1B model described in the experiments.
- [Figures 4–10] Several convergence and performance figures (e.g., Figures 4, 5, 8, 9) report single runs without error bars or statistical significance, which makes it difficult to assess the robustness of the claimed efficiency gains.
Circularity Check
The central convergence theorem imports the FedAvg bound with renamed adaptive-step and client-selection terms, and the efficiency corollaries read the claimed gains off their own definitions.
-
self definitional
[Section II.B, Remark II.1]
"By acting as a knowledge distiller, the LLM guides QFL training, improving convergence and efficiency while reducing computational overhead. Therefore, we have an inequality l(·,·)< l LLM(·,·) which implies that the QFL-trained quantum model achieves a lower loss than an LLM used in isolation."
The inequality l < l_LLM is the central benefit the paper claims to establish, but it is asserted as an immediate consequence of the phrase 'acting as a knowledge distiller' rather than derived from the algorithm, the convergence theorem, or the measurements. The paper's own Figure 23 states that 'the performance of LLM is unmatched,' i.e., l_LLM < l in the experiments, so the assumed inequality is exactly the contested conclusion, not a premise supplied by the method.
-
renaming known result
[Appendix A, Theorem VI.4, proof equations (8)-(9)]
"With error decomposed, we can have E∥(θt+1, ϕt+1)−(θ ∗, ϕ∗)∥2 ≤(1−η tµ)E∥ (θt, ϕt)−(θ ∗, ϕ∗)∥2 +η 2 t (B+C)(9) ... B= PM i=1 w2 i σ2 i + 6LΓ + 8(E−1) 2G2, and C= 4 St E2G2 ... Et i =iter∗ Lt i LLLM for i∈S t and St = arg min S⊆{1,...,N},|S|=K P i∈S |Lt i − Lt g|."
Equation (9) is the FedAvg recursion of Li et al. [45] with the fixed local-step count E replaced by K_i^t = iter * L_i^t / L_LLM and the client set replaced by S_t = argmin sum |L_i^t - L_g^t|. The proof states the decomposition without re-deriving the drift and variance terms B and C for loss-dependent step counts or for selection that is correlated with current client losses; such selection biases the aggregated gradient estimator, and the (E-1)^2 and E^2 terms do not follow. The O(1/T) rate is therefore the imported FedAvg result with renamed symbols, not a bound proved for Algorithm 1.
1 more flagged steps
-
self definitional
[Appendix A, Corollary VI.8.1]
"TQFL TLLM-QFL ≥ E[K t i ] K ... The adaptive mechanism ensures: E[K (t) i ] =E iter· Lt i Lt LLM ≥K whenever L t i ≥L t LLM . Thus, TLLM-QFL =O 1 E[K t i ] ≤ K E[K t i ] TQFL"
The claimed communication-round reduction is read directly off the definition K_i^t = iter * L_i^t / L_LLM. The proof assumes the round count scales as the reciprocal of the average number of local steps, but the very bound imported in Theorem VI.4 contains 8(E-1)^2 G^2 and E^2 G^2 drift terms, which grow with E; more adaptive local steps can increase drift and slow convergence in the non-iid setting. The ratio E[K]/K is the designed input of the algorithm, not a derived or measured efficiency gain.
full rationale
The paper's theoretical section is largely a relabeling exercise. Theorem VI.4 cites [45] and writes the FedAvg error recursion as equation (9) after replacing E with the adaptive K_i^t and the participation rule with alignment-based S_t; no proof shows that the FedAvg drift and variance bounds survive these replacements. Corollary VI.8.1 then converts the definition of K_i^t into a claimed communication-round reduction, and Corollary VI.8.2 asserts a (1 - k/N) variance reduction via a Markov step that is not justified and is false in general when the d_i^2 are equal. Separately, Remark II.1 assumes the inequality l < l_LLM that the framework is supposed to deliver, and Figure 23 contradicts it. These are not harmless self-citations: the load-bearing theoretical claims reduce to an imported theorem with renamed terms or to definitions of the adaptive mechanism. The paper does include genuine experiments on simulators and IBM hardware with external datasets and baselines, so the experimental portion is not circular and can stand on its own; the external FedAvg bound is also real evidence, though not for this modified algorithm. The circularity score is therefore intermediate: partial, because the empirical comparisons and the external theorem are real inputs, but the central convergence and efficiency claims as stated are not established by the derivation chain.
Assumptions & free parameters
free parameters (5)
- lambda (distillation weight) =
not specified
- mu (regularization weight) =
not specified
- epsilon (termination threshold) =
not specified
- client selection fraction k/N =
10% in experiments
- initial optimizer maxiter =
10 (Exp. I) and 5 (Exp. 2 on real hardware)
assumptions (5)
- standard math Assumption VI.1: Each local loss function is L-smooth.
- standard math Assumption VI.2: Bounded gradient variance and bounded expected squared norm of gradients.
- domain assumption Assumption VI.3: The global loss function is mu-strongly convex.
- ad hoc to paper Remark II.1: The optimized LLM-QFL loss satisfies l(.,.) < l_LLM(.,.).
- ad hoc to paper The FedAvg convergence bound of Li et al. [45] remains valid when local steps are adaptive and clients are selected by alignment.
Cite this review
Pith. "Pith review of LLM-QFL: Distilling Large Language Model for Quantum Federated Learning." pith.science (2026). https://pith.science/paper/RKUOSF5B
@misc{pith2026250518656,
author = {Pith},
title = {Pith review of: LLM-QFL: Distilling Large Language Model for Quantum Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/RKUOSF5B}},
note = {Machine review of arXiv:2505.18656}
}
read the original abstract
Inspired by the power of large language models (LLMs), our research adapts them to quantum federated learning (QFL) to boost efficiency and performance. We propose a federated fine-tuning method that distills an LLM within QFL, allowing each client to locally adapt the model to its own data while preserving privacy and reducing unnecessary global updates. The fine-tuned LLM also acts as a reinforcement agent, optimizing QFL by adjusting optimizer steps, cutting down communication rounds, and intelligently selecting clients. Experiments show significant efficiency gains. We pioneer a synergy between LLM and QFL, offering: i) practical efficiency: Reduced communication costs and faster convergence. ii) theoretical rigor: Provable guarantees for adaptive federated optimization. iii) scalability: PEFT methods (LoRA, QLoRA) enable deployment on resource-constrained quantum devices. Code implementation is available here 1.
Figures
Figures from the paper (18 more)
Reference graph
Works this paper leans on
-
[45]
ON THE CONVERGENCE OF FEDA VG ON NON-IID,
X. Li, W. Yang, K. Huang, S. Wang, and Z. Zhang, “ON THE CONVERGENCE OF FEDA VG ON NON-IID,”ICLR, 2020. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2025 11 APPENDIX A. CONVERGENCE& COMPLEXITY OFLLM-QFL We made the following assumptions. Assumption VI.1(Lipschitz Smoothness).Each local loss functionF i(θ, ϕ)is L-smooth: ∥∇Fi(θ, ϕ)− ∇Fi(θ′, ϕ′)∥ ≤L...
work page 2020
-
[1]
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning,
R. Yeet al., “OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning,” Feb. 2024, arXiv:2402.06954 [cs]
arXiv 2024
-
[2]
Towards Quantum Machine Learn- ing for Constrained Combinatorial Optimization: a Quantum QAP Solver,
X. Ye, G. Yan, and J. Yan, “Towards Quantum Machine Learn- ing for Constrained Combinatorial Optimization: a Quantum QAP Solver,”ICML, 2023
work page 2023
-
[3]
Quantum Support Vector Machine for Classifying Noisy Data,
J. Li, Y . Li, J. Song, J. Zhang, and S. Zhang, “Quantum Support Vector Machine for Classifying Noisy Data,”IEEE Transactions on Computers, vol. 73, no. 9, pp. 2233–2247, Sep. 2024
work page 2024
-
[4]
Quantum federated learning experiments in the cloud with data encoding,
S. R. Pokhrel, N. Yash, J. Kua, G. Li, and L. Pan, “Quantum federated learning experiments in the cloud with data encoding,” arXiv preprint arXiv:2405.00909, 2024
arXiv 2024
-
[5]
A Survey on Multimodal Large Language Models,
S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A Survey on Multimodal Large Language Models,” Apr. 2024, arXiv:2306.13549 [cs]
arXiv 2024
-
[6]
Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning,
D. Guoet al., “Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[7]
NetLLM: Adapting Large Language Models for Networking
D. Wu, X. Wang, Y . Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “NetLLM: Adapting Large Language Models for Networking,” inProceedings of the ACM SIGCOMM 2024 Conference, Aug. 2024, pp. 661–678, arXiv:2402.02338 [cs]
work page Pith review arXiv 2024
Show all 56 references
-
[8]
FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model,
F. Wu, Z. Li, Y . Li, B. Ding, and J. Gao, “FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’24. New York, NY , USA: Association for Computing Machinery, ...
2024
-
[9]
A data- encoding approach to quantum federated learning: Experiment- ing with cloud challenges,
S. R. Pokhrel, N. Yash, J. Kua, G. Li, and L. Pan, “A data- encoding approach to quantum federated learning: Experiment- ing with cloud challenges,” inProceedings of the 8th Asia- Pacific Workshop on Networking, 2024, pp. 179–180
2024
-
[10]
QuantumFed: A Federated Learning Framework for Collaborative Quantum Training,
Q. Xia and Q. Li, “QuantumFed: A Federated Learning Framework for Collaborative Quantum Training,” in2021 IEEE Global Communications Conference (GLOBECOM), Dec. 2021, pp. 1–6
2021
-
[11]
Quantum Federated Learning with Quantum Data,
M. Chehimi and W. Saad, “Quantum Federated Learning with Quantum Data,” inICASSP 2022 - 2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2022, pp. 8617–8621
2022
-
[12]
Quantum federated learning through blind quantum computing,
W. Li, S. Lu, and D.-L. Deng, “Quantum federated learning through blind quantum computing,”Science China Physics, Mechanics & Astronomy, vol. 64, no. 10, p. 100312, Oct. 2021
2021
-
[13]
Federated quantum natural gradient descent for quantum federated learning,
J. Qi and M.-H. Hsieh, “Federated quantum natural gradient descent for quantum federated learning,” pp. 329–341, 2024
2024
-
[14]
Slimmable quantum federated learning,
W. J. Yun, J. P. Kim, S. Jung, J. Park, M. Bennis, and J. Kim, “Slimmable quantum federated learning,”arXiv preprint arXiv:2207.10221, 2022
2022 arXiv
-
[15]
Federated Quantum Machine Learning,
S. Y .-C. Chen and S. Yoo, “Federated Quantum Machine Learning,”Entropy, vol. 23, no. 4, p. 460, Apr. 2021
2021
-
[16]
Exact Decomposition of Quantum Channels for Non-IID Quantum Federated Learning,
H. Zhao, “Exact Decomposition of Quantum Channels for Non-IID Quantum Federated Learning,” Sep. 2022, arXiv:2209.00768
2022 arXiv
-
[17]
Quantum Federated Learning With Decentralized Data,
R. Huang, X. Tan, and Q. Xu, “Quantum Federated Learning With Decentralized Data,”IEEE Journal of Selected Topics in Quantum Electronics, vol. 28, no. 4: Mach. Learn. in Photon. Commun. and Meas. Syst., pp. 1–10, Jul. 2022
2022
-
[18]
A personalized quantum federated learning,
D. Gurung and S. R. Pokhrel, “A personalized quantum federated learning,” inProceedings of the 8th Asia-Pacific Workshop on Networking, 2024, pp. 175–176
2024
-
[19]
Chained continuous quantum federated learning frame- work,
——, “Chained continuous quantum federated learning frame- work,”Future Generation Computer Systems, p. 107800, Mar. 2025
2025
-
[20]
Quantum federated learning for metaverse: Analysis, design and implementation,
D. Gurung, S. R. Pokhrel, and G. Li, “Quantum federated learning for metaverse: Analysis, design and implementation,” IEEE Transactions on Network and Service Management, pp. 1–1, 2025
2025
-
[21]
Performance Analysis and De- sign of a Weighted Personalized Quantum Federated Learning,
D. Gurung and S. R. Pokhrel, “Performance Analysis and De- sign of a Weighted Personalized Quantum Federated Learning,” IEEE Transactions on Artificial Intelligence, pp. 1–12, 2025
2025
-
[22]
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs,
C. Hou, A. Shrivastava, H. Zhan, R. Conway, T. Le, A. Sagar, G. Fanti, and D. Lazar, “PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs,” inProceedings of the 41st International Conference on Machine Learning. PMLR, Jul. 2024, pp. 19 043–19 061
2024
-
[23]
FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models,
J. Sun, Z. Xu, H. Yin, D. Yang, D. Xu, Y . Liu, Z. Du, Y . Chen, and H. R. Roth, “FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models,” inProceedings of the 41st International Conference on Machine Learning. PMLR, Jul. 2024, pp. 47 159–47 173
2024
-
[24]
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes,
Z. Qin, D. Chen, B. Qian, B. Ding, Y . Li, and S. Deng, “Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes,” in Proceedings of the 41st International Conference on Machine Learning. PMLR, Jul. 2024, pp. 41 473–41 497
2024
-
[25]
FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning,
J. QI, Z. Luan, S. Huang, C. Fung, H. Yang, and D. Qian, “FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning,” Jun. 2024, arXiv:2406.07925 [cs]
2024 arXiv
-
[26]
FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning,
H. Chen, Y . Zhang, D. Krompass, J. Gu, and V . Tresp, “FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 10, pp. 11 285– 11 293, Mar. 2024, number: 10
2024
-
[27]
FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations,
Z. Wang, Z. Shen, Y . He, G. Sun, H. Wang, L. Lyu, and A. Li, “FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations,” Sep. 2024, arXiv:2409.05976 [cs]
2024 arXiv
-
[28]
FedPETuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,
Z. Zhang, Y . Yang, Y . Dai, Q. Wang, Y . Yu, L. Qu, and Z. Xu, “FedPETuning: When federated learning meets the parameter- efficient tuning methods of pre-trained language models,” in Findings of the Association for Computational Linguistics: ACL
-
[29]
pFedPrompt: Learning Per- sonalized Prompt for Vision-Language Models in Federated Learning,
T. Guo, S. Guo, and J. Wang, “pFedPrompt: Learning Per- sonalized Prompt for Vision-Language Models in Federated Learning,” inProceedings of the ACM Web Conference 2023, ser. WWW ’23. New York, NY , USA: Association for Computing Machinery, Apr. 2023, pp. 1364–1374
2023
-
[30]
Scaling Federated Learning for Fine-Tuning of Large Language Models,
A. Hilmkil, S. Callh, M. Barbieri, L. R. S ¨utfeld, E. L. Zec, and O. Mogren, “Scaling Federated Learning for Fine-Tuning of Large Language Models,” inNatural Language Processing and Information Systems, E. M ´etais, F. Meziane, H. Horacek, and E. Kapetanios, Eds. Cham: Spring...
2021
-
[31]
Titanic: Towards Production Federated Learning with Large Language Models,
N. Su, C. Hu, B. Li, and B. Li, “Titanic: Towards Production Federated Learning with Large Language Models,” inIEEE INFOCOM 2024 - IEEE Conference on Computer Commu- nications. Vancouver, BC, Canada: IEEE, May 2024, pp. 611–620
2024
-
[32]
Recovering Private Text in Federated Learning of Language Models,
S. Gupta, Y . Huang, Z. Zhong, T. Gao, K. Li, and D. Chen, “Recovering Private Text in Federated Learning of Language Models,”Advances in Neural Information Processing Systems, vol. 35, pp. 8130–8143, Dec. 2022
2022
-
[33]
Genomic benchmarks: a collection of datasets for genomic sequence classification,
K. Greˇsov´a, V . Martinek, D.ˇCech´ak, P. ˇSimeˇcek, and P. Alexiou, “Genomic benchmarks: a collection of datasets for genomic sequence classification,”BMC Genomic Data, vol. 24, no. 1, p. 25, 2023
2023
-
[34]
Semeval-2017 task 4: Sentiment analysis in twitter,
S. Rosenthal, N. Farra, and P. Nakov, “Semeval-2017 task 4: Sentiment analysis in twitter,” inProceedings of the 11th international workshop on semantic evaluation (SemEval-2017), 2017, pp. 502–518
2017
-
[35]
Llama: Open and efficient foundation language models,
H. Touvronet al., “Llama: Open and efficient foundation language models,” 2023
2023
-
[36]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
-
[37]
Attention is All you Need,
A. Vaswaniet al., “Attention is All you Need,” inAdvances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017
2017
-
[38]
A Survey of Large Language Models,
W. X. Zhaoet al., “A Survey of Large Language Models,” Nov. 2023, arXiv:2303.18223 [cs]
2023 arXiv
-
[39]
Parameter- Efficient Fine-Tuning for Large Models: A Comprehensive Survey,
Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang, “Parameter- Efficient Fine-Tuning for Large Models: A Comprehensive Survey,” Sep. 2024, arXiv:2403.14608
2024 arXiv
-
[40]
LoRA: Low-Rank Adaptation of Large Language Models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” Oct. 2021, arXiv:2106.09685
2021 arXiv
-
[41]
Distilling the Knowledge in a Neural Network,
G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” Mar. 2015
2015
-
[42]
Model compression,
C. Bucilu ˇa, R. Caruana, and A. Niculescu-Mizil, “Model compression,” inProceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’06. New York, NY , USA: Association for Computing Machinery, Aug. 2006, pp. 535–541
2006
-
[43]
A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimen- tal Design,
M. Morafah, W. Wang, and B. Lin, “A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimen- tal Design,”IEEE Transactions on Artificial Intelligence, vol. 5, no. 4, pp. 1708–1717, Apr. 2024
2024
-
[44]
Communication-efficient federated learning via knowledge distillation,
C. Wuet al., “Communication-efficient federated learning via knowledge distillation,”Nature Communications, vol. 13, no. 1, p. 2032, Apr. 2022
2022
-
[47]
(Adaptive Step Size Efficiency) TQFL TLLM-QFL ≥ E[K t i ] K where, K is the fixed number of local steps in QFL and E[K (t) i ]is the average adaptive steps in LLM-QFL
-
[48]
(Variance Reduction) Var(∇FLLM-QFL)≤ 1− k N Var(∇FQFL) where,k/Nis the fraction of selected clients. Proof. 1. From the convergence rate in Theorem 1, the effective progress per round scales with the number of local steps. The adaptive mechanism ensures: E[K (t) i ] =E iter· L...
2025
-
[49]
For client selection variance, let St rand be random selection, and St align be our alignment-based selection. The variance decomposes as, Var(∇F) =E[∥∇F i − ∇F∥2] = 1 N NX i=1 d(t)2 i Our selection criterion S(t) ={i|d t i smallestk%} minimizes Var(∇FLLM-QFL) = 1 k X i∈S(t) a...
-
[50]
For thesentencetask, there are three classesnegative, positive,andneutral with varying sample as shown in Figure 12
TweetEval, Sentiment:TweetEval consists of het- erogeneous tasks in Twitter which is used as multiclass dataset with tasks likeirony, hate, offensive, stance, emoji, emotionandsentiment. For thesentencetask, there are three classesnegative, positive,andneutral with varying sam...
2000
-
[51]
The encoded and PCA (n=4) applied dataset is shown in Figure 13
Genomic-DemoHumanOrWorm Dataset:Demo- HumanOrWorm is a benchmark genomic sequence dataset for classification purposes which consists of 75,000 train samples and 25,000 test set samples with data instance as, (’TCAACTGACTTCCGAGGGAATAAGTGTTTCGCCAT CTCGAACTGTATACTCTGCTATCAA GACCG...
-
[52]
LLM is based on the Transformer architecture, while VQC is based on the Quantum Neural Network architecture
Data Preparation of LLM fine-tuning and VQC training:Both the architecture of LLM and VQC is different. LLM is based on the Transformer architecture, while VQC is based on the Quantum Neural Network architecture. We cannot feed the same data format to both models. Thus, there ...
-
[53]
Based on the required data sample size, we deduct small samples
First, we load and shuffle dataset. Based on the required data sample size, we deduct small samples
-
[54]
dset” for train or “test
Then, we convert dataset into dataframe with columns “dset” for train or “test” set, cat to store labels (Human or Worm), “seq” for nucleotide sequences
-
[55]
For LLaMA model, first we convert dataframe to HuggingFace Dataset format. For tokenization, we first define k-mer tokenization (substrings of length k=6), load LLaMA tokenizer (meta-llama/Llama- 3.2-1B), ensure padding, load pretrained model for sequence classification (2 lab...
-
[56]
Job create exceeds open plan job usage limits
For VQC model, one-hot encoding is applied to nucleotide sequences as A=[1,0,0,0], C=[0,1,0,0], G=[0,0,1,0], T=[0,0,0,1] and are converted to one- hot vectors. Also, the Principal component analysis is applied for dimensionality reduction with PCA (n components=4) which reduce...
2025
-
[2023]
Association for Computational Linguistics (ACL), 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.