A Survey on Efficient Federated Learning Methods for Foundation Model Training

Alexander Isenko; Hans-Arno Jacobsen; Herbert Woisetschl\"ager; Ruben Mayer; Shiqiang Wang

arxiv: 2401.04472 · v3 · submitted 2024-01-09 · 💻 cs.LG · cs.AI· cs.DC

A Survey on Efficient Federated Learning Methods for Foundation Model Training

Herbert Woisetschl\"ager , Alexander Isenko , Shiqiang Wang , Ruben Mayer , Hans-Arno Jacobsen This is my paper

Pith reviewed 2026-05-24 04:41 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.DC

keywords federated learningfoundation modelsparameter-efficient fine-tuningcomputational efficiencycommunication efficiencyprivacyfine-tuningsurvey

0 comments

The pith

A survey proposes a taxonomy of efficiency methods to enable foundation model training in federated learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Federated learning allows collaborative model training without sharing raw data, preserving privacy. Foundation models are typically pre-trained on broad data and then fine-tuned on smaller task-specific sets, but accessing those sets can be difficult due to data silos. The paper introduces a taxonomy centered on computational and communication efficiency to address the challenges of applying federated learning to these large models. It reviews parameter-efficient fine-tuning approaches, evaluates existing federated learning frameworks for foundation model support, and outlines future directions including generative model evaluation and privacy interactions.

Core claim

With this survey, the authors introduce a novel taxonomy focused on computational and communication efficiency as the vital elements to make use of foundation models in federated learning systems. They discuss the benefits and drawbacks of parameter-efficient fine-tuning for federated applications, elaborate on the readiness of federated learning frameworks to work with foundation models, and provide future research opportunities on evaluating generative models in federated learning as well as the interplay of privacy and parameter-efficient fine-tuning.

What carries the argument

The novel taxonomy focused on computational and communication efficiency for using foundation models in federated learning systems.

If this is right

Parameter-efficient fine-tuning methods have identifiable benefits and drawbacks in federated learning contexts.
Federated learning frameworks differ in their preparedness to support foundation models.
New research is needed to develop methods for evaluating generative models trained via federated learning.
The relationship between privacy preservation and parameter-efficient fine-tuning requires dedicated investigation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Applying the taxonomy to emerging methods could help identify underexplored efficiency trade-offs at larger model scales.
The efficiency dimensions might usefully extend to other privacy-preserving distributed training scenarios involving large models.
Systematic categorization using this taxonomy could accelerate progress by highlighting gaps in current approaches.

Load-bearing premise

The collection of prior work reviewed is representative of the field and the taxonomy adequately covers the essential computational and communication efficiency dimensions without major omissions or overlaps.

What would settle it

A follow-up analysis that identifies a large set of relevant efficiency techniques for foundation models in federated learning that cannot be classified under the proposed taxonomy categories.

Figures

Figures reproduced from arXiv: 2401.04472 by Alexander Isenko, Hans-Arno Jacobsen, Herbert Woisetschl\"ager, Ruben Mayer, Shiqiang Wang.

read the original abstract

Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

A survey that organizes existing work on efficient FL for foundation models around a taxonomy of compute and comms efficiency but introduces no new methods or results.

read the letter

The one thing to know is that this is a survey paper on efficient federated learning for training or adapting foundation models. It introduces what the authors call a novel taxonomy based on computational and communication efficiency aspects. What the paper does well is to structure the discussion around parameter-efficient fine-tuning techniques, explaining their benefits and drawbacks when used in a federated setting. It also looks at whether existing FL frameworks are ready to handle foundation models and suggests some directions for future research, including how to evaluate generative models under FL and the interaction between privacy mechanisms and PEFT methods. This kind of organized summary can be practical for readers who need to understand the state of the field without diving into every paper. On the soft spots, the central issue for any survey is coverage and whether the taxonomy actually captures the important distinctions without significant gaps or redundant categories. The abstract presents a coherent outline, but the real test is in the full list of cited works and how they are grouped. Since no new method or theorem is proposed, there are no equations or experiments to verify for correctness. The representativeness of the surveyed literature is the only real question mark, and that's inherent to survey papers rather than a flaw in the argument itself. This paper would be most useful for researchers or engineers who are starting to work at the intersection of federated learning and foundation models and want a quick way to get oriented. Experts already familiar with the area might not find much new. It deserves to go through peer review because the topic is relevant and a well-executed survey can provide value even without original technical contributions, as long as the collection and categorization are done carefully.

Referee Report

0 major / 2 minor

Summary. The manuscript is a survey on efficient federated learning (FL) methods for foundation model (FM) training. It introduces a novel taxonomy organized around computational and communication efficiency, reviews the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) techniques when applied in FL, assesses the current readiness of FL frameworks to support FMs, and outlines future research opportunities on evaluating generative models in FL settings as well as the interplay between privacy mechanisms and PEFT.

Significance. If the proposed taxonomy is comprehensive and the surveyed literature representative, the work could provide a useful organizing lens for an emerging intersection of FL and large-scale models. The explicit focus on efficiency dimensions, combined with practical discussion of framework readiness and open questions around generative-model evaluation and privacy-PEFT interactions, may help researchers identify actionable gaps. The purely descriptive nature of the paper means its value rests on coverage and balance rather than novel technical results.

minor comments (2)

The abstract states that the taxonomy is 'novel,' but the manuscript should explicitly contrast the new taxonomy against prior FL or FM taxonomies (e.g., those based on client heterogeneity or model compression) to substantiate the claim of novelty.
Section headings and subsection numbering should be checked for consistency; the transition between the taxonomy presentation and the PEFT discussion would benefit from an explicit mapping of which taxonomy branches correspond to which PEFT methods.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their thorough review and positive recommendation to accept the manuscript. We are pleased that the taxonomy, coverage of PEFT techniques, framework readiness assessment, and outlined research opportunities were viewed as providing a useful organizing lens for the intersection of federated learning and foundation models.

Circularity Check

0 steps flagged

No significant circularity: purely descriptive survey with no derivations or predictions

full rationale

The paper is explicitly a survey that organizes existing literature on efficient FL for foundation models. It introduces a taxonomy focused on computational and communication efficiency but does not advance any deductive chain, equations, fitted parameters, or empirical predictions that could reduce to its own inputs. No self-citation load-bearing steps, uniqueness theorems, or ansatzes are invoked in a manner that creates circularity. The central contribution is organizational and descriptive, with the representativeness of surveyed work being a standard survey limitation rather than a circularity issue. This matches the default expectation for non-circular papers.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

As a survey the paper introduces no free parameters, new axioms, or invented entities; it relies on standard background assumptions from the federated learning and foundation-model literature.

pith-pipeline@v0.9.0 · 5727 in / 957 out tokens · 20237 ms · 2026-05-24T04:41:57.313436+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

A Survey on Foundation Models for Personalized Federated Intelligence
cs.AI 2025-05 unverdicted novelty 3.0

The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.

Reference graph

Works this paper leans on

73 extracted references · 73 canonical work pages · cited by 1 Pith paper

[1]

Parizi, and Fahad Saeed

Mohammed Aledhari, Rehma Razzak, Reza M. Parizi, and Fahad Saeed. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access , 8:140699–140725, 2020

work page 2020
[2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, et al. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems , volume 30, 2017

work page 2017
[3]

Elkordy, et al

Sara Babakniya, Ahmed R. Elkordy, et al. Slora: Federated parameter efficient fine-tuning of language models, 2023

work page 2023
[4]

Federated learning review: Fundamentals, enabling technologies, and future applications

Syreen Banabilah, Moayad Aloqaily, et al. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Management , 59(6), 2022

work page 2022
[5]

Beutel, Taner Topal, et al

Daniel J. Beutel, Taner Topal, et al. Flower: A friendly federated learning research framework, 2020

work page 2020
[6]

Hudson, et al

Rishi Bommasani, Drew A. Hudson, et al. On the opportunities and risks of foundation models, 2021

work page 2021
[7]

Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning

Yuanyuan Chen, Zichen Chen, et al. Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 . IJCAI Org., 2023

work page 2023
[8]

Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2020

Enmao Diao, Jie Ding, et al. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2020

work page 2020
[9]

Parameter-efficient fine-tuning of large-scale pre-trained language models

Ning Ding, Yujia Qin, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , 5(3):220–235, March 2023

work page 2023
[10]

An image is worth 16x16 words: Transformers for image recognition at scale, 2020

Alexey Dosovitskiy, Lucas Beyer, et al. An image is worth 16x16 words: Transformers for image recognition at scale, 2020

work page 2020
[11]

How can we train deep learning models across clouds and continents? an experimental study, 2023

Alexander Erben, Ruben Mayer, and Hans-Arno Jacobsen. How can we train deep learning models across clouds and continents? an experimental study, 2023

work page 2023
[12]

Fate-llm: A industrial grade federated learning framework for large language models, 2023

Tao Fan, Yan Kang, et al. Fate-llm: A industrial grade federated learning framework for large language models, 2023

work page 2023
[13]

Openfl: the open federated learning library

Patrick Foley, Micah J Sheller, et al. Openfl: the open federated learning library. Physics in Medicine & Biology , 67(21), 2022

work page 2022
[14]

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR , 2019

work page 2019
[15]

Substra: a framework for privacy-preserving, traceable and collaborative machine learning, 2019

Mathieu N Galtier and Camille Marini. Substra: a framework for privacy-preserving, traceable and collaborative machine learning, 2019

work page 2019
[16]

Inc. Google. Tensorflow federated: Machine learning on decentralized data. https://www.tensorflow.org/federated, 2019. Accessed: 2023-12-01

work page 2019
[17]

Knowledge distillation in vision transformers: A critical review, 2023

Gousia Habib, Tausifa Jan Saleem, and Brejesh Lall. Knowledge distillation in vision transformers: A critical review, 2023

work page 2023
[18]

Fedml: A research library and benchmark for federated machine learning, 2020

Chaoyang He, Songze Li, et al. Fedml: A research library and benchmark for federated machine learning, 2020

work page 2020
[19]

Distiller: A systematic study of model distillation methods in natural language processing, 2021

Haoyu He, Xingjian Shi, et al. Distiller: A systematic study of model distillation methods in natural language processing, 2021

work page 2021
[20]

Fj ORD : Fair and accurate federated learning under heterogeneous targets with ordered dropout

Samuel Horv \'a th, Stefanos Laskaridis, et al. Fj ORD : Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Advances in Neural Information Processing Systems , 2021

work page 2021
[21]

Parameter-efficient transfer learning for nlp, 2019

Neil Houlsby, Andrei Giurgiu, et al. Parameter-efficient transfer learning for nlp, 2019

work page 2019
[22]

Hu, Yelong Shen, et al

Edward J. Hu, Yelong Shen, et al. Lora: Low-rank adaptation of large language models, 2021

work page 2021
[23]

Distributed pruning towards tiny neural networks in federated learning, 2022

Hong Huang, Lan Zhang, et al. Distributed pruning towards tiny neural networks in federated learning, 2022

work page 2022
[24]

Sparse random networks for communication-efficient federated learning, 2022

Berivan Isik, Francesco Pase, et al. Sparse random networks for communication-efficient federated learning, 2022

work page 2022
[25]

Visual prompt tuning, 2022

Menglin Jia, Luming Tang, et al. Visual prompt tuning, 2022

work page 2022
[26]

Model pruning enables efficient federated learning on edge devices

Yuang Jiang, Shiqiang Wang, et al. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems , 2022

work page 2022
[27]

Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

Weirui Kuang, Bingchen Qian, et al. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

work page 2023
[28]

Block pruning for faster transformers

Francois Lagunas, Ella Charlaix, et al. Block pruning for faster transformers. In Proceedings of the 2021 Conference on EMNLP , Online and Punta Cana, Dominican Republic, November 2021. ACL

work page 2021
[29]

The power of scale for parameter-efficient prompt tuning, 2021

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021

work page 2021
[30]

Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020

Ang Li, Jingwei Sun, et al. Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020

work page 2020
[31]

Soteriafl: A unified framework for private federated learning with communication compression

Zhize Li, Haoyu Zhao, et al. Soteriafl: A unified framework for private federated learning with communication compression. In Advances in Neural Information Processing Systems , volume 35, 2022

work page 2022
[32]

A survey on federated learning systems: Vision, hype and reality for data privacy and protection

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering , 35(4):3347–3366, April 2023

work page 2023
[33]

Fate: An industrial grade platform for collaborative learning with data protection

Yang Liu, Tao Fan, et al. Fate: An industrial grade platform for collaborative learning with data protection. JMLR , 22(1), 2021

work page 2021
[34]

From distributed machine learning to federated learning: a survey

Ji Liu, Jizhou Huang, et al. From distributed machine learning to federated learning: a survey. Knowledge and Information Systems , 64(4), 2022

work page 2022
[35]

The flan collection: Designing data and methods for effective instruction tuning, 2023

Shayne Longpre, Le Hou, et al. The flan collection: Designing data and methods for effective instruction tuning, 2023

work page 2023
[36]

Fedclip: Fast generalization and personalization for clip in federated learning, 2023

Wang Lu, Xixu Hu, et al. Fedclip: Fast generalization and personalization for clip in federated learning, 2023

work page 2023
[37]

Ibm federated learning: an enterprise framework white paper v0.1, 2020

Heiko Ludwig, Nathalie Baracaldo, et al. Ibm federated learning: an enterprise framework white paper v0.1, 2020

work page 2020
[38]

Communication-Efficient Learning of Deep Networks from Decentralized Data

Brendan McMahan, Eider Moore, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data . In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research . PMLR, 2017

work page 2017
[39]

Distributed learning with compressed gradient differences, 2019

Konstantin Mishchenko, Eduard Gorbunov, et al. Distributed learning with compressed gradient differences, 2019

work page 2019
[40]

Pappas, and Hamed Hassani

Aritra Mitra, Rayana Jaafar, George J. Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In Advances in Neural Information Processing Systems , 2021

work page 2021
[41]

Nguyen, Ming Ding, et al

Dinh C. Nguyen, Ming Ding, et al. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials , 23(3), 2021

work page 2021
[42]

GPT-4 technical report, 2023

OpenAI. GPT-4 technical report, 2023

work page 2023
[43]

The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

Guilherme Penedo, Quentin Malartic, et al. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

work page 2023
[44]

Federated adversarial domain adaptation

Xingchao Peng, Zijun Huang, et al. Federated adversarial domain adaptation. In ICLR , 2020

work page 2020
[45]

Audio-visual model distillation using acoustic images

Andres Perez, Valentina Sanguineti, et al. Audio-visual model distillation using acoustic images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2020

work page 2020
[46]

Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, 2023

Zhen Qin, Daoyuan Chen, et al. Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, 2023

work page 2023
[47]

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, et al. Language models are unsupervised multitask learners. 2019

work page 2019
[48]

Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization

Amirhossein Reisizadeh, Aryan Mokhtari, et al. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108. PMLR, 2020

work page 2020
[49]

Roth, Yan Cheng, et al

Holger R. Roth, Yan Cheng, et al. Nvidia flare: Federated learning from simulation to real-world. 2022

work page 2022
[50]

Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

Max Ryabinin, Tim Dettmers, et al. Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

work page 2023
[51]

Movement pruning: Adaptive sparsity by fine-tuning

Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020

work page 2020
[52]

Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023

Guangyu Sun, Matias Mendieta, et al. Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023

work page 2023
[53]

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

work page 2023
[54]

Fedbert: When federated learning meets pre-training

Yuanyishu Tian, Yao Wan, et al. Fedbert: When federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology , 13(4), 2022

work page 2022
[55]

Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023

Herbert Woisetschl\" a ger, Alexander Erben, et al. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023

work page 2023
[56]

Federatedscope: A flexible federated learning platform for heterogeneity

Yuexiang Xie, Zhen Wang, et al. Federatedscope: A flexible federated learning platform for heterogeneity. Proceedings of the VLDB Endowment , 16(5), 2023

work page 2023
[57]

Federated learning of gboard language models with differential privacy

Zheng Xu, Yanxiang Zhang, et al. Federated learning of gboard language models with differential privacy. 2023

work page 2023
[58]

Uniaudio: An audio foundation model toward universal audio generation, 2023

Dongchao Yang, Jinchuan Tian, et al. Uniaudio: An audio foundation model toward universal audio generation, 2023

work page 2023
[59]

Dual-personalizing adapter for federated foundation models

Yiyuan Yang, Guodong Long, Taoshu Shen, Jing Jiang, and Michael Blumenstein. Dual-personalizing adapter for federated foundation models. ArXiv , abs/2403.19211, 2024

work page arXiv 2024
[60]

Green federated learning, 2023

Ashkan Yousefpour, Shen Guo, et al. Green federated learning, 2023

work page 2023
[61]

Pablo Muñoz, and Ali Jannesari

Sixing Yu, J. Pablo Muñoz, and Ali Jannesari. Federated foundation models: Privacy-preserving and collaborative learning for large models, 2023

work page 2023
[62]

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021

work page 2021
[63]

A survey on federated learning

Chen Zhang, Yu Xie, et al. A survey on federated learning. Knowledge-Based Systems , 216, 2021

work page 2021
[64]

Towards building the federated gpt: Federated instruction tuning, 2023

Jianyi Zhang, Saeed Vahidian, et al. Towards building the federated gpt: Federated instruction tuning, 2023

work page 2023
[65]

FedPETuning : When federated learning meets the parameter-efficient tuning methods of pre-trained language models

Zhuo Zhang, Yuanhang Yang, et al. FedPETuning : When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023 . ACL, 2023

work page 2023
[66]

Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022

Haodong Zhao, Wei Du, et al. Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022

work page 2022
[67]

Secrets of rlhf in large language models part i: Ppo, 2023

Rui Zheng, Shihan Dou, et al. Secrets of rlhf in large language models part i: Ppo, 2023

work page 2023
[68]

Adaptive quantization for deep neural network

Yiren Zhou, Seyed Moosavi-Dezfooli, et al. Adaptive quantization for deep neural network. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018

work page 2018
[69]

To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017

work page 2017
[70]

Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus

Maohua Zhu, Tao Zhang, et al. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , MICRO ’52. ACM, 2019

work page 2019
[71]

When foundation model meets federated learning: Motivations, challenges, and future directions, 2023

Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions, 2023

work page 2023
[72]

PySyft: A Library for Easy Federated Learning

Alexander Ziller, Andrew Trask, et al. PySyft: A Library for Easy Federated Learning . Springer International Publishing, 2021

work page 2021
[73]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

work page

[1] [1]

Parizi, and Fahad Saeed

Mohammed Aledhari, Rehma Razzak, Reza M. Parizi, and Fahad Saeed. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access , 8:140699–140725, 2020

work page 2020

[2] [2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, et al. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems , volume 30, 2017

work page 2017

[3] [3]

Elkordy, et al

Sara Babakniya, Ahmed R. Elkordy, et al. Slora: Federated parameter efficient fine-tuning of language models, 2023

work page 2023

[4] [4]

Federated learning review: Fundamentals, enabling technologies, and future applications

Syreen Banabilah, Moayad Aloqaily, et al. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Management , 59(6), 2022

work page 2022

[5] [5]

Beutel, Taner Topal, et al

Daniel J. Beutel, Taner Topal, et al. Flower: A friendly federated learning research framework, 2020

work page 2020

[6] [6]

Hudson, et al

Rishi Bommasani, Drew A. Hudson, et al. On the opportunities and risks of foundation models, 2021

work page 2021

[7] [7]

Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning

Yuanyuan Chen, Zichen Chen, et al. Fedobd: Opportunistic block dropout for efficiently training large-scale neural networks through federated learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 . IJCAI Org., 2023

work page 2023

[8] [8]

Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2020

Enmao Diao, Jie Ding, et al. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2020

work page 2020

[9] [9]

Parameter-efficient fine-tuning of large-scale pre-trained language models

Ning Ding, Yujia Qin, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , 5(3):220–235, March 2023

work page 2023

[10] [10]

An image is worth 16x16 words: Transformers for image recognition at scale, 2020

Alexey Dosovitskiy, Lucas Beyer, et al. An image is worth 16x16 words: Transformers for image recognition at scale, 2020

work page 2020

[11] [11]

How can we train deep learning models across clouds and continents? an experimental study, 2023

Alexander Erben, Ruben Mayer, and Hans-Arno Jacobsen. How can we train deep learning models across clouds and continents? an experimental study, 2023

work page 2023

[12] [12]

Fate-llm: A industrial grade federated learning framework for large language models, 2023

Tao Fan, Yan Kang, et al. Fate-llm: A industrial grade federated learning framework for large language models, 2023

work page 2023

[13] [13]

Openfl: the open federated learning library

Patrick Foley, Micah J Sheller, et al. Openfl: the open federated learning library. Physics in Medicine & Biology , 67(21), 2022

work page 2022

[14] [14]

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR , 2019

work page 2019

[15] [15]

Substra: a framework for privacy-preserving, traceable and collaborative machine learning, 2019

Mathieu N Galtier and Camille Marini. Substra: a framework for privacy-preserving, traceable and collaborative machine learning, 2019

work page 2019

[16] [16]

Inc. Google. Tensorflow federated: Machine learning on decentralized data. https://www.tensorflow.org/federated, 2019. Accessed: 2023-12-01

work page 2019

[17] [17]

Knowledge distillation in vision transformers: A critical review, 2023

Gousia Habib, Tausifa Jan Saleem, and Brejesh Lall. Knowledge distillation in vision transformers: A critical review, 2023

work page 2023

[18] [18]

Fedml: A research library and benchmark for federated machine learning, 2020

Chaoyang He, Songze Li, et al. Fedml: A research library and benchmark for federated machine learning, 2020

work page 2020

[19] [19]

Distiller: A systematic study of model distillation methods in natural language processing, 2021

Haoyu He, Xingjian Shi, et al. Distiller: A systematic study of model distillation methods in natural language processing, 2021

work page 2021

[20] [20]

Fj ORD : Fair and accurate federated learning under heterogeneous targets with ordered dropout

Samuel Horv \'a th, Stefanos Laskaridis, et al. Fj ORD : Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Advances in Neural Information Processing Systems , 2021

work page 2021

[21] [21]

Parameter-efficient transfer learning for nlp, 2019

Neil Houlsby, Andrei Giurgiu, et al. Parameter-efficient transfer learning for nlp, 2019

work page 2019

[22] [22]

Hu, Yelong Shen, et al

Edward J. Hu, Yelong Shen, et al. Lora: Low-rank adaptation of large language models, 2021

work page 2021

[23] [23]

Distributed pruning towards tiny neural networks in federated learning, 2022

Hong Huang, Lan Zhang, et al. Distributed pruning towards tiny neural networks in federated learning, 2022

work page 2022

[24] [24]

Sparse random networks for communication-efficient federated learning, 2022

Berivan Isik, Francesco Pase, et al. Sparse random networks for communication-efficient federated learning, 2022

work page 2022

[25] [25]

Visual prompt tuning, 2022

Menglin Jia, Luming Tang, et al. Visual prompt tuning, 2022

work page 2022

[26] [26]

Model pruning enables efficient federated learning on edge devices

Yuang Jiang, Shiqiang Wang, et al. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems , 2022

work page 2022

[27] [27]

Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

Weirui Kuang, Bingchen Qian, et al. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning, 2023

work page 2023

[28] [28]

Block pruning for faster transformers

Francois Lagunas, Ella Charlaix, et al. Block pruning for faster transformers. In Proceedings of the 2021 Conference on EMNLP , Online and Punta Cana, Dominican Republic, November 2021. ACL

work page 2021

[29] [29]

The power of scale for parameter-efficient prompt tuning, 2021

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021

work page 2021

[30] [30]

Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020

Ang Li, Jingwei Sun, et al. Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets, 2020

work page 2020

[31] [31]

Soteriafl: A unified framework for private federated learning with communication compression

Zhize Li, Haoyu Zhao, et al. Soteriafl: A unified framework for private federated learning with communication compression. In Advances in Neural Information Processing Systems , volume 35, 2022

work page 2022

[32] [32]

A survey on federated learning systems: Vision, hype and reality for data privacy and protection

Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering , 35(4):3347–3366, April 2023

work page 2023

[33] [33]

Fate: An industrial grade platform for collaborative learning with data protection

Yang Liu, Tao Fan, et al. Fate: An industrial grade platform for collaborative learning with data protection. JMLR , 22(1), 2021

work page 2021

[34] [34]

From distributed machine learning to federated learning: a survey

Ji Liu, Jizhou Huang, et al. From distributed machine learning to federated learning: a survey. Knowledge and Information Systems , 64(4), 2022

work page 2022

[35] [35]

The flan collection: Designing data and methods for effective instruction tuning, 2023

Shayne Longpre, Le Hou, et al. The flan collection: Designing data and methods for effective instruction tuning, 2023

work page 2023

[36] [36]

Fedclip: Fast generalization and personalization for clip in federated learning, 2023

Wang Lu, Xixu Hu, et al. Fedclip: Fast generalization and personalization for clip in federated learning, 2023

work page 2023

[37] [37]

Ibm federated learning: an enterprise framework white paper v0.1, 2020

Heiko Ludwig, Nathalie Baracaldo, et al. Ibm federated learning: an enterprise framework white paper v0.1, 2020

work page 2020

[38] [38]

Communication-Efficient Learning of Deep Networks from Decentralized Data

Brendan McMahan, Eider Moore, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data . In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research . PMLR, 2017

work page 2017

[39] [39]

Distributed learning with compressed gradient differences, 2019

Konstantin Mishchenko, Eduard Gorbunov, et al. Distributed learning with compressed gradient differences, 2019

work page 2019

[40] [40]

Pappas, and Hamed Hassani

Aritra Mitra, Rayana Jaafar, George J. Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In Advances in Neural Information Processing Systems , 2021

work page 2021

[41] [41]

Nguyen, Ming Ding, et al

Dinh C. Nguyen, Ming Ding, et al. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials , 23(3), 2021

work page 2021

[42] [42]

GPT-4 technical report, 2023

OpenAI. GPT-4 technical report, 2023

work page 2023

[43] [43]

The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

Guilherme Penedo, Quentin Malartic, et al. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

work page 2023

[44] [44]

Federated adversarial domain adaptation

Xingchao Peng, Zijun Huang, et al. Federated adversarial domain adaptation. In ICLR , 2020

work page 2020

[45] [45]

Audio-visual model distillation using acoustic images

Andres Perez, Valentina Sanguineti, et al. Audio-visual model distillation using acoustic images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2020

work page 2020

[46] [46]

Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, 2023

Zhen Qin, Daoyuan Chen, et al. Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, 2023

work page 2023

[47] [47]

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, et al. Language models are unsupervised multitask learners. 2019

work page 2019

[48] [48]

Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization

Amirhossein Reisizadeh, Aryan Mokhtari, et al. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108. PMLR, 2020

work page 2020

[49] [49]

Roth, Yan Cheng, et al

Holger R. Roth, Yan Cheng, et al. Nvidia flare: Federated learning from simulation to real-world. 2022

work page 2022

[50] [50]

Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

Max Ryabinin, Tim Dettmers, et al. Swarm parallelism: Training large models can be surprisingly communication-efficient, 2023

work page 2023

[51] [51]

Movement pruning: Adaptive sparsity by fine-tuning

Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020

work page 2020

[52] [52]

Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023

Guangyu Sun, Matias Mendieta, et al. Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning, 2023

work page 2023

[53] [53]

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

work page 2023

[54] [54]

Fedbert: When federated learning meets pre-training

Yuanyishu Tian, Yao Wan, et al. Fedbert: When federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology , 13(4), 2022

work page 2022

[55] [55]

Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023

Herbert Woisetschl\" a ger, Alexander Erben, et al. Federated fine-tuning of llms on the very edge: The good, the bad, the ugly, 2023

work page 2023

[56] [56]

Federatedscope: A flexible federated learning platform for heterogeneity

Yuexiang Xie, Zhen Wang, et al. Federatedscope: A flexible federated learning platform for heterogeneity. Proceedings of the VLDB Endowment , 16(5), 2023

work page 2023

[57] [57]

Federated learning of gboard language models with differential privacy

Zheng Xu, Yanxiang Zhang, et al. Federated learning of gboard language models with differential privacy. 2023

work page 2023

[58] [58]

Uniaudio: An audio foundation model toward universal audio generation, 2023

Dongchao Yang, Jinchuan Tian, et al. Uniaudio: An audio foundation model toward universal audio generation, 2023

work page 2023

[59] [59]

Dual-personalizing adapter for federated foundation models

Yiyuan Yang, Guodong Long, Taoshu Shen, Jing Jiang, and Michael Blumenstein. Dual-personalizing adapter for federated foundation models. ArXiv , abs/2403.19211, 2024

work page arXiv 2024

[60] [60]

Green federated learning, 2023

Ashkan Yousefpour, Shen Guo, et al. Green federated learning, 2023

work page 2023

[61] [61]

Pablo Muñoz, and Ali Jannesari

Sixing Yu, J. Pablo Muñoz, and Ali Jannesari. Federated foundation models: Privacy-preserving and collaborative learning for large models, 2023

work page 2023

[62] [62]

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021

work page 2021

[63] [63]

A survey on federated learning

Chen Zhang, Yu Xie, et al. A survey on federated learning. Knowledge-Based Systems , 216, 2021

work page 2021

[64] [64]

Towards building the federated gpt: Federated instruction tuning, 2023

Jianyi Zhang, Saeed Vahidian, et al. Towards building the federated gpt: Federated instruction tuning, 2023

work page 2023

[65] [65]

FedPETuning : When federated learning meets the parameter-efficient tuning methods of pre-trained language models

Zhuo Zhang, Yuanhang Yang, et al. FedPETuning : When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023 . ACL, 2023

work page 2023

[66] [66]

Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022

Haodong Zhao, Wei Du, et al. Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning, 2022

work page 2022

[67] [67]

Secrets of rlhf in large language models part i: Ppo, 2023

Rui Zheng, Shihan Dou, et al. Secrets of rlhf in large language models part i: Ppo, 2023

work page 2023

[68] [68]

Adaptive quantization for deep neural network

Yiren Zhou, Seyed Moosavi-Dezfooli, et al. Adaptive quantization for deep neural network. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), 2018

work page 2018

[69] [69]

To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017

work page 2017

[70] [70]

Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus

Maohua Zhu, Tao Zhang, et al. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture , MICRO ’52. ACM, 2019

work page 2019

[71] [71]

When foundation model meets federated learning: Motivations, challenges, and future directions, 2023

Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions, 2023

work page 2023

[72] [72]

PySyft: A Library for Easy Federated Learning

Alexander Ziller, Andrew Trask, et al. PySyft: A Library for Easy Federated Learning . Springer International Publishing, 2021

work page 2021

[73] [73]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

work page