Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

FFT-MoE claims that sparse mixture-of-experts adapters, with per-client top-K activation and a thresholded balancing loss, outperform LoRA-based federated fine-tuning under heterogeneous edge conditions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Sparse MoE adapters with per-client expert selection and a thresholded load-balancing loss improve federated fine-tuning accuracy under non-IID data compared with LoRA baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid incremental idea; the headline claims outrun the experiments, which never test the paper's own adaptive-activation mechanism. the 4 major comments →

arxiv 2508.18663 v1 pith:DDEPCF2U submitted 2025-08-26 cs.LG cs.AI

FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge

Gang Hu , Yinglei Teng , Pengfei Wu , Nan Wang This is my paper
classification cs.LG cs.AI
keywords federated fine-tuningmixture of expertsparameter-efficient fine-tuningLoRAnon-IID datadevice heterogeneityexpert load balancingsparse routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FFT-MoE argues that the right adapter for federated fine-tuning is not a low-rank matrix but a sparse mixture of small expert networks. Each client keeps the foundation model frozen and trains only a router plus a shared pool of experts, activating a client-chosen subset of experts per token. Because the experts are shared and only the router changes locally, clients with different compute budgets can use different numbers of experts and still upload compatible parameters for aggregation. A thresholded KL-divergence auxiliary loss counteracts the expert-load imbalance that data skew creates. Across BERT and ViT fine-tuning on text and image benchmarks, the paper reports that this design beats LoRA-based federated baselines in accuracy, convergence speed, and efficiency, with the largest gains in extreme non-IID settings.

Core claim

The paper's central claim is that sparse MoE adapters provide a better structural unit than LoRA for federated fine-tuning under heterogeneity. The paper shows LoRA is a special case of MoE with uniform activation (K = M and routing weight 1), frames FFT-MoE as generalizing LoRA by letting a router select a subset of experts, and builds a framework where each client uses top-K sparsity with per-client K. The server aggregates by weighted averaging of expert weights and router weights. To prevent routers collapsing onto a few experts, a heterogeneity-aware auxiliary loss applies KL divergence toward a uniform target only when the router's max activation exceeds a threshold, preserving special

What carries the argument

The load-bearing object is the trainable MoE adapter inserted into each transformer FFN: M two-layer experts E_m and a routing matrix W_R, with output x' = W_FM x + sum_m R(x)_m E_m(x) and top-K sparsity on the routing logits. The LoRA-as-special-case identity shows that uniform expert activation recovers LoRA, establishing that the MoE parameterization contains LoRA within it. Per-client top-K activation lets each client pick its own sparsity level while keeping the shared expert and router shapes aggregation-compatible. The thresholded KL auxiliary loss activates only when the router is overly biased toward a single expert, dynamically trading exploration against specialization. These piec

Load-bearing premise

The load-bearing premise is that averaging the clients' locally trained routers and expert weights at the server produces one coherent global mixture-of-experts model, even though clients may have used different numbers of activated experts and routed their local data through specialized subsets.

What would settle it

Run a federation where clients deliberately use different per-client top-K values (for example K=1, 2, and 4) on the same non-IID split and check whether the server-aggregated router still assigns sensible expert distributions. If expert-load divergence on the server fails to decrease or accuracy drops to the LoRA baseline, the aggregation-compatibility claim fails. A simpler check is to compare the federated router to a router trained on pooled data under the same parameter budget; strong divergence would indicate that weighted averaging is not preserving routing behavior.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Federated fine-tuning can support heterogeneous clients without forcing equal rank or structure: each client chooses its own top-K and still contributes to one global adapter.
  • Because LoRA is a special case of MoE under uniform activation, a federation could in principle mix LoRA-style and MoE-style clients and aggregate them into the same parameter space.
  • The thresholded auxiliary loss gives a practical recipe for load balancing: apply regularization only when the router's confidence in one expert exceeds a threshold, rather than always penalizing specialization.
  • Under a fixed parameter budget, the evidence in the paper points to spending the budget on more, lower-rank experts rather than on higher rank, especially when data is non-IID.
  • Faster convergence over communication rounds means fewer rounds are needed to reach a target accuracy, directly reducing communication cost in federated deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: actually vary K_n per client within one federation and measure whether weighted averaging of routers with different sparsity levels preserves routing specialization; the paper asserts aggregation compatibility but only reports configurations where rank and expert count vary, not per-client top-K differences.
  • If the aggregation assumption holds, the same MoE-adapter construction should transfer to federated instruction tuning and multimodal fine-tuning, since the mechanism is modality-agnostic and only the frozen backbone changes.
  • The thresholded KL loss suggests a simple server-side stopping rule the paper does not derive: monitor average expert-load divergence across layers and stop increasing the balancing coefficient once imbalance stops decreasing, yielding an adaptive schedule.
  • A comparison against personalized federated methods, rather than only global PEFT baselines, would clarify how much of the reported gain comes from MoE specialization versus the shared global aggregation itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes FFT-MoE, a federated fine-tuning framework in which LoRA adapters are replaced by sparse Mixture-of-Experts (MoE) adapters. Each client trains a local gating network to activate a top-K subset of a shared expert pool, and the server aggregates the local MoE adapters by weighted averaging (Eq. 3). To counter expert load imbalance, the authors introduce a thresholded KL-divergence auxiliary loss (Eqs. 11-14). Experiments on AgNews/BERT and CIFAR-10/ViT compare against FedAdapter, FedPrompt, and FLoRA under IID and non-IID partitions, reporting accuracy, convergence curves, and ablations on the auxiliary-loss hyperparameters. The paper also shows algebraically that LoRA is a special case of MoE with uniform expert activation.

Significance. The direction is timely: replacing fixed low-rank adapters with sparse MoE adapters in federated fine-tuning could provide a principled way to adapt per-client compute while preserving a common aggregation structure. The thresholded auxiliary loss and the LoRA-as-MoE equivalence are conceptually interesting. If validated, the framework would be a useful contribution to heterogeneous federated fine-tuning. However, the current evidence does not establish the headline claims: the closest MoE-based federated baselines are absent, the per-client top-K heterogeneity mechanism is never actually varied in experiments, and no training-efficiency metrics are reported. The paper provides code and a self-contained derivation, which are strengths, but the experimental support needs substantial strengthening.

major comments (4)
  1. [Experiments, Table 1] The comparison omits MoE-based federated fine-tuning baselines. The claim of outperforming 'state-of-the-art FFT baselines' is not established against the closest competitors, such as FedMoE (Meii et al., 2024), FedMoE-DA (Zhan et al., 2024), or other sparse-MoE FL methods. Without these, the observed gains could be due to the MoE architecture in general rather than FFT-MoE's specific design. Please add at least one MoE-based FFT baseline and discuss how FFT-MoE differs from it.
  2. [Experimental Setup / Adaptive activation (Eq. 5)] Device heterogeneity is described in the setup as 'randomly assigning each client to one of two device types,' but no experiment actually varies the per-client sparsity K_n. Tables 1 and 2 report only R and M; no K_n assignments or efficiency metrics (FLOPs, communication volume, latency) are given. Consequently, the central 'adaptive activation for heterogeneous computing' mechanism is not empirically supported. The reported gains could come entirely from a fixed-top-K MoE adapter. Add experiments with heterogeneous K_n across clients and report per-client computational/communication budgets.
  3. [Aggregation, Eq. (3)] The server aggregates local MoE adapters by weighted averaging the routers and experts even when clients have used different K_n, trained on different data, and specialized their routers differently. The paper asserts structural compatibility but provides no convergence or consistency argument for this averaging, and no experiment varies K_n to test whether the averaged global router remains coherent. This is load-bearing for the heterogeneity claim. Either provide an analysis of when Eq. (3) yields a sensible global MoE, or directly test it by varying K_n across clients and checking global-model behavior and accuracy.
  4. [Ablation Study, Figure 6] The ablation for the auxiliary loss varies λ and θth and reports test accuracy. If these hyperparameters were selected based on the test split, the benefit of the auxiliary loss is partly a fitted outcome. The paper does not state whether selection was done on a separate validation split. Please clarify the selection procedure or report the sensitivity on validation data. This is important because Table 1 shows a very large gain from the auxiliary loss in the 1-Label AgNews setting (69.53% to 86.25%), and the reader needs to know the loss is not being tuned directly on the test set.
minor comments (4)
  1. [Table 2] The table formatting is difficult to parse: the header appears duplicated ('FFT-MoE Aux-Loss FFT-MoE Aux-Loss') and the row/column correspondence for R and M is unclear. Please restructure the table so each configuration and its accuracy are unambiguous.
  2. [Figures and terminology] The baseline name is inconsistent: 'FedAdapter' in the text and Table 1, 'FedAdaptor' in Figure 5, and 'FFT-Adapter' in Table 1's 'FFT-Adapter' column header. Please standardize.
  3. [Eq. (12)-(14)] The notation P_l(x)_m is described as 'logits output' but is used in a KL divergence, which requires probability distributions. Clarify whether P_l is normalized (e.g., softmax of routing logits) and specify that the KL is over experts m.
  4. [Title and terminology] The paper calls the MoE 'large-scale' and 'Large-scale Sparse MoE,' but the experiments use M=2,4,8,16 experts with small per-expert ranks. The wording is overstated; consider a more modest description unless larger-scale experiments are provided.

Circularity Check

0 steps flagged

No significant circularity: FFT-MoE's derivation is self-contained, and its benchmark comparisons are not forced by construction.

full rationale

I examined the paper's derivation chain for the defined circularity patterns. The central algebraic claim, that LoRA is a special case of MoE with uniform expert activation, is derived explicitly in Eqs. (6)-(10) from block-matrix multiplication; it is an identity given the definitions of E_m = E^2_m E^1_m, and does not assume the paper's target result. The aggregation rule in Eq. (3) is stated as a design choice rather than derived from any fitted parameter, and the paper makes no convergence claim that would depend on a self-citation. The auxiliary loss in Eqs. (11)-(14) is a standard load-balancing regularizer; the hyperparameters lambda and theta_th are examined in an ablation study, not fitted to the benchmark and then renamed as a prediction. The paper's references to prior MoE and federated learning work are external, and I found no load-bearing self-citation chain. The fact that the paper does not empirically vary per-client K_n is a gap between claim and experiment, but that is a correctness/evidence limitation, not circularity. The reported MoE-versus-LoRA accuracy differences are not entailed by the auxiliary loss definition or by the LoRA-as-MoE equivalence, so the derivation does not reduce to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The method's novelty rests on two hyperparameterized mechanisms (thresholded KL loss and per-client K_n) plus the standard MoE adapter. The algebraic equivalence to LoRA is standard block-matrix decomposition. No new physical or model entities are introduced.

free parameters (4)
  • auxiliary loss weight lambda = ~1e-5 to 1e-4
    Selected based on test accuracy in Fig. 6(a); no validation split reported.
  • activation threshold theta_th = not stated for main runs; Fig. 6(b) shows 0.0-0.5 range
    Selected based on test accuracy in Fig. 6(b); exact value used in Table 1 not reported.
  • number of experts M and per-expert rank R = R=2,M=8 and R=4,M=4 in Table 1
    Configuration chosen by authors, potentially by test performance; not determined by resource constraints in the reported experiments.
  • per-client top-K K_n = not reported
    The device-heterogeneity mechanism depends on K_n per client, but the paper never states how K_n is assigned to high vs low capacity clients or which values were used.
axioms (6)
  • standard math Block matrix multiplication identity BA = sum_m E^2_m E^1_m (Eq. 10)
    Used to claim LoRA is a special case of MoE; this is basic algebra.
  • domain assumption Frozen pretrained BERT/ViT weights provide transferable features for downstream classification under lightweight adaptation
    Standard assumption in PEFT; not tested here.
  • domain assumption Dirichlet partitioning with alpha captures realistic non-IID FL data
    Common practice in FL research but no real-world validation.
  • ad hoc to paper Uniform target distribution P_g is the correct global fairness objective for routing
    Eq. (13) sets P_g as uniform; the paper does not justify why uniformity maximizes generalization in federated settings.
  • ad hoc to paper Thresholding the auxiliary loss on theta = max_m P(x)_m is sufficient to detect harmful load imbalance
    Eq. (14) and Eq. (12); no derivation or empirical validation of this detection rule.
  • domain assumption Weighted averaging of heterogeneous local adapters produces a usable global model
    Eq. (3); standard FedAvg assumption, unexamined for MoE with different top-K.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge." pith.science (2026). https://pith.science/paper/DDEPCF2U

@misc{pith2026250818663,
  author       = {Pith},
  title        = {Pith review of: FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDEPCF2U}},
  note         = {Machine review of arXiv:2508.18663}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As FMs drive progress toward Artificial General Intelligence (AGI), fine-tuning them under privacy and resource constraints has become increasingly critical particularly when highquality training data resides on distributed edge devices. Federated Learning (FL) offers a compelling solution through Federated Fine-Tuning (FFT), which enables collaborative model adaptation without sharing raw data. Recent approaches incorporate Parameter-Efficient Fine-Tuning (PEFT) techniques such as Low Rank Adaptation (LoRA) to reduce computational overhead. However, LoRA-based FFT faces two major limitations in heterogeneous FL environments: structural incompatibility across clients with varying LoRA configurations and limited adaptability to non-IID data distributions, which hinders convergence and generalization. To address these challenges, we propose FFT MoE, a novel FFT framework that replaces LoRA with sparse Mixture of Experts (MoE) adapters. Each client trains a lightweight gating network to selectively activate a personalized subset of experts, enabling fine-grained adaptation to local resource budgets while preserving aggregation compatibility. To further combat the expert load imbalance caused by device and data heterogeneity, we introduce a heterogeneity-aware auxiliary loss that dynamically regularizes the routing distribution to ensure expert diversity and balanced utilization. Extensive experiments spanning both IID and non-IID conditions demonstrate that FFT MoE consistently outperforms state of the art FFT baselines in generalization performance and training efficiency.

Figures

Figures reproduced from arXiv: 2508.18663 by Gang Hu, Nan Wang, Pengfei Wu, Yinglei Teng.

Figure 1
Figure 1. Figure 1: LoRA-Based Tuning vs. MoE-based Tuning. services (Myakala, Jonnalagadda, and Bura 2024). To ad￾dress these challenges,FL decentralizes model training by orchestrating gradient or model updates across clients, thus sidestepping the need for centralized data aggregation and preserving user privacy (McMahan et al. 2017). While FL offers privacy advantages, directly training large-scale FMs in federated settin… view at source ↗
Figure 2
Figure 2. Figure 2: Expert load of local and global models on FL datasets. We apply the Sparse MoE mechanism to the BERT model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left (a) is the pipeline of FFT-MoE, and right (b) is the illustration of auxiliary loss for expert load imbalance. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration: LoRA as a special case of MoE. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The accuracy and convergence performance with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of heterogeneity-aware auxiliary loss [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Enhancing Cross-Problem Vehicle Routing via Federated Learning

    cs.AI 2026-04 unverdicted novelty 6.0

    MPSF-FL combines multi-problem pre-training with federated learning to improve performance and generalization of neural solvers across diverse vehicle routing problems.

  2. Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

    cs.LG 2026-06 unverdicted novelty 5.0

    A priority-aware learning-unlearning framework with orthogonal LoRA enables robust correction for device join/leave events in dynamic decentralized federated LLM fine-tuning.

Reference graph

Works this paper leans on

41 extracted references · 32 canonical work pages · cited by 2 Pith papers · 2 internal anchors

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, J.; Chen, D.; Qian, B.; Yao, L.; and Li, Y. 2024. Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 14457--14483. Vancouver, Canada: Curran Associates, Inc

    work page 2024

  4. [4]

    Brewer, W.; Kashi, A.; Dash, S.; Tsaris, A.; Yin, J.; Shankar, M.; and Wang, F. 2024. Scalable artificial intelligence for science: Perspectives, methods and exemplars. arXiv:2406.17812

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    X.; and Xu, M

    Cai, D.; Wu, Y.; Wang, S.; Lin, F. X.; and Xu, M. 2023. Efficient Federated Learning for Modern NLP. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 1--16. New York, NY, USA: ACM

    work page 2023

  6. [6]

    Cheng, Y.; Zhang, W.; Zhang, Z.; Zhang, C.; Wang, S.; and Mao, S. 2025. Towards Federated Large Language Models: Motivations, Methods, and Future Directions. IEEE Communications Surveys & Tutorials, 1--1

    work page 2025

  7. [7]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 4171--4186. Minneapolis, Minneso...

    work page 2019

  8. [8]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR). Virtual: OpenReview.net

    work page 2021

  9. [9]

    Fang, Z.; Lin, Z.; Chen, Z.; Chen, X.; Gao, Y.; and Fang, Y. 2024. Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models. arXiv:2404.06448

    Pith/arXiv arXiv 2024

  10. [10]

    Fedus, W.; Zoph, B.; and Shazeer, N. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120): 1--39

    2022

  11. [11]

    G.; and Goldstein, T

    Goldblum, M.; Souri, H.; Ni, R.; Shu, M.; Prabhu, V.; Somepalli, G.; Chattopadhyay, P.; Ibrahim, M.; Bardes, A.; Hoffman, J.; Chellappa, R.; Wilson, A. G.; and Goldstein, T. 2023. Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S....

    work page 2023

  12. [12]

    Guo, T.; Guo, S.; Wang, J.; Tang, X.; and Xu, W. 2024. PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models – Federated Learning in Age of Foundation Model. IEEE Transactions on Mobile Computing, 23(5): 5179--5194

    work page 2024

  13. [13]

    J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), volume 1, 3. Virtual: OpenReview.net

    work page 2022

  14. [14]

    Jia, C.; Hu, M.; Chen, Z.; Yang, Y.; Xie, X.; Liu, Y.; and Chen, M. 2024. AdaptiveFL: Adaptive Heterogeneous Federated Learning for Resource-Constrained AIoT Systems. In Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC), 84, 1--6. New York, NY, USA: Association for Computing Machinery

    work page 2024

  15. [15]

    B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A

    Kairouz, P.; McMahan, H. B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A. N.; Bonawitz, K.; Charles, Z.; Cormode, G.; Cummings, R.; et al. 2021. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1--2): 1--210

    work page 2021

  16. [16]

    Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Technical report, Univ. Toronto, Toronto, ON, Canada

    work page 2009

  17. [17]

    Li, J.; Sun, Z.; He, X.; Zeng, L.; Lin, Y.; Li, E.; Zheng, B.; Zhao, R.; and Chen, X. 2024. LocMoE: a low-overhead MoE for large language model training. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (ICJAI), volume 37, 1--11. Jeju, Korea: ICJAI

    work page 2024

  18. [18]

    Liu, H.; Zhai, Y.; Xu, K.; Feng, D.; and Li, Y. 2024. Nuclear-Norm Maximization for Low-Rank Updates. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 11101--11105. Seoul, Korea: IEEE

    work page 2024

  19. [19]

    Liu, J.; Liao, Y.; Xu, H.; and Xu, Y. 2025. A Survey of Large Language Models. arXiv:2503.21213

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017. Communication efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (AISTATS), 1273--1282. Fort Lauderdale, Florida, USA.: PMLR

    work page 2017

  21. [21]

    Meii, H.; Cai, D.; Zhou, A.; Wang, S.; and Xu, M. 2024. FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts. arXiv:2408.11304

    Pith/arXiv arXiv 2024

  22. [22]

    K.; Jonnalagadda, A

    Myakala, P. K.; Jonnalagadda, A. K.; and Bura, C. 2024. Federated learning and data privacy: A review of challenges and opportunities. International Journal of Research Publication and Reviews, 5(12): 10--55248

    work page 2024

  23. [23]

    Nie, X.; Liu, Q.; Fu, F.; Zhu, S.; Miao, X.; Li, X.; Zhang, Y.; Liu, S.; and Cui, B. 2024. LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 54161--54182. Vancouver, Canada: Curran Associates, Inc

    work page 2024

  24. [24]

    Reisizadeh, A.; Tziotis, I.; Hassani, H.; Mokhtari, A.; and Pedarsani, R. 2022. Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity. IEEE Journal on Selected Areas in Information Theory, 3(2): 197--205

    work page 2022

  25. [25]

    Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q.; Hinton, G.; and Dean, J. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations (ICLR)

    2017

  26. [26]

    Shen, L.; Sun, Y.; Yu, Z.; Ding, L.; Tian, X.; and Tao, D. 2024. On efficient training of large-scale deep learning models. ACM Computing Surveys, 57(3): 1--36

    work page 2024

  27. [27]

    Tamirisa, R.; Xie, C.; Bao, W.; Zhou, A.; Arel, R.; and Shamsian, A. 2024. FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 23985--23994. Seattle, WA, USA: IEEE

    work page 2024

  28. [28]

    H.; and Pham, V

    Tran, V.-T.; Khiem, L. H.; and Pham, V. Q. 2025. Revisiting Sparse Mixture of Experts for Resource-adaptive Federated Fine-tuning Foundation Models. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning

    work page 2025

  29. [29]

    Wang, X.; Qiao, Y.; Wu, D.; Wu, C.; and Wang, F. 2025. Cluster-Based Heterogeneous Federated Foundation Model Adaptation and Fine-Tuning. In AAAI Conference on Artificial Intelligence (AAAI), volume 39, 21269--21277. Philadelphia, Pennsylvania, USA: OpenReview.net

    work page 2025

  30. [30]

    Wang, Z.; Shen, Z.; He, Y.; Sun, G.; Wang, H.; Lyu, L.; and Li, A. 2024. FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. In Advances in Neural Information Processing Systems (NeurIPS), 22513--22533. Red Hook, NY, USA.: Curran Associates, Inc

    work page 2024

  31. [31]

    Xie, Z.; and Song, s. 2023. FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence. IEEE Journal on Selected Areas in Communications, 41(4): 1227--1242

    work page 2023

  32. [32]

    Yan, Y.; Yang, Q.; Tang, S.; and Shi, Z. 2024. FeDeRA: Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition. arXiv:2404.18848

    Pith/arXiv arXiv 2024

  33. [33]

    Zeng, Y.; Huang, C.; Mei, Y.; Zhang, L.; Su, T.; Ye, W.; Shi, W.; and Wang, S. 2025. EfficientMoE: Optimizing Mixture-of-Experts Model Training With Adaptive Load Balance. IEEE Transactions on Parallel and Distributed Systems, 36(4): 677--688

    work page 2025

  34. [34]

    W.; Chuan, W.; Guo, D.; and Chen, X

    Zhan, Z.; Zhao, W.; Li, W., Yuanqing abd Liu; Zhang, X.; Tan, C. W.; Chuan, W.; Guo, D.; and Chen, X. 2024. FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation. arXiv:2411.02115

    Pith/arXiv arXiv 2024

  35. [35]

    Zhang, J.; Vahidian, S.; Kuo, M.; Li, C.; Zhang, R.; Yu, T.; Wang, G.; and Chen, Y. 2024 a . Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education . In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6915--6919. Seoul, Korea.: IEEE

    work page 2024

  36. [36]

    Zhang, W.; Wang, Q.; Kong, X.; Xiong, J.; Ni, S.; Cao, D.; Niu, B.; Chen, M.; Li, Y.; Zhang, R.; et al. 2024 b . Fine-tuning large language models for chemical text mining. Chemical science, 15(27): 10600--10611

    work page 2024

  37. [37]

    Zhang, X.; Zhao, J.; and LeCun, Y. 2015. Character-level Convolutional Networks for Text Classification. In Cortes, C.; Lawrence, N.; Lee, D.; Sugiyama, M.; and Garnett, R., eds., Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc

    work page 2015

  38. [38]

    Zhang, Z.; Yang, Y.; Dai, Y.; Wang, Q.; Yu, Y.; Qu, L.; and Xu, Z. 2023. F ed PET uning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, 9963--9977. Toronto, Canada: Association for Computational Linguistics

    work page 2023

  39. [39]

    Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; Du, Y.; Yang, C.; Chen, Y.; Chen, Z.; Jiang, J.; Ren, R.; Li, Y.; Tang, X.; Liu, Z.; Liu, P.; Nie, J.-Y.; and Wen, J.-R. 2023. A Survey of Large Language Models. arXiv:2303.18223

    Pith/arXiv arXiv 2023

  40. [40]

    M.; Chen, z.; Le, Q

    Zhou, Y.; Lei, T.; Liu, H.; Du, N.; Huang, Y.; Zhao, V.; Dai, A. M.; Chen, z.; Le, Q. V.; and Laudon, J. 2022. Mixture-of-Experts with Expert Choice Routing. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 7103--7114. New Orleans, Louisiana, USA: Curran Associates, Inc

    work page 2022

  41. [41]

    Zhu, Z.; Shi, Y.; Fan, P.; Peng, C.; and Letaief, K. B. 2024. ISFL: Federated Learning for Non-i.i.d. Data With Local Importance Sampling. IEEE Internet of Things Journal, 11(16): 27448--27462

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.