Pith. sign in

REVIEW 4 major objections 4 minor 12 references

This survey proposes the first unified, non-overlapping taxonomy of methods that combine foundation models with federated learning, organized by the model development life cycle of training, customization, and deployment, and covering 42 di

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:32 UTC pith:ILVJXIUM

load-bearing objection Useful lifecycle taxonomy of FL-FM methods, but inconsistent counts and a narrow search protocol mean the 'comprehensive' claim is not yet earned. the 4 major comments →

arxiv 2509.05142 v1 pith:ILVJXIUM submitted 2025-09-05 cs.LG

Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights

Cosmin-Andrei Hatfaludi , Alex Serban This is my paper
classification cs.LG
keywords federated learningfoundation modelstaxonomyliterature surveyparameter-efficient fine-tuningLoRAknowledge distillationhealthcare AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing surveys of the intersection between foundation models (FMs) and federated learning (FL) either lack a structured taxonomy, use overlapping categories, or cover only one development stage. To fill that gap, the authors reviewed more than 4,200 articles, narrowed them to 260 full-text reads, and identified 42 unique methods, which they organized into a taxonomy built around the life cycle of a model: train, customize, and deploy. They further rate each method by implementation complexity, communication efficiency, and scalability, and apply the map to healthcare as a case study. If the taxonomy holds, researchers and practitioners gain a way to position new methods uniquely and see which combinations of techniques are still unexplored. The paper also identifies concrete gaps, such as the near-absence of deployment-stage methods and the limited use of privacy-preserving aggregation in the reviewed literature.

Core claim

The central claim is that the intersection of foundation models and federated learning can be classified without overlap by asking where in the development life cycle the FM is used: pre-training the entire model, pre-training only part of it, adapting an existing model to a task, or running inference. The customize stage is the richest, splitting into selective fine-tuning, additive fine-tuning (adapters and LoRA variants), knowledge distillation, compression/pruning/quantization, and hybrids. The deploy stage currently contains a single method, which the authors use to mark a research gap. The taxonomy is validated internally by two expert reviewers, and each method is additionally scored

What carries the argument

The load-bearing object is the life-cycle taxonomy, whose top-level classes are train, customize, and deploy. Within each class, subclasses are derived from the core algorithmic technique: whole-model versus partial pre-training, additive versus selective fine-tuning, distillation versus compression, and hybrid combinations. The taxonomy is designed so that any new method fits into exactly one class, which is the property the paper contrasts with earlier overlapping taxonomies.

Load-bearing premise

The paper's coverage and taxonomy claims rest on the reviewed article set being representative and complete; the search was limited to the first five pages of results and 50 API results per query, with title/abstract screening and grey literature excluded, so an important missing method family would make the taxonomy and gap analysis incomplete.

What would settle it

A check that a recent federated-foundation-model method (for example, a split-learning or personalized FL approach) cannot be placed in exactly one class of the proposed train/customize/deploy taxonomy, or a demonstration that an earlier taxonomy already classifies all 42 methods without overlap, would refute the paper's non-overlap and uniqueness claims.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Researchers can position a new federated-foundation-model method in exactly one class of the taxonomy, avoiding the overlap problem in prior surveys.
  • The field's center of gravity has shifted from pre-training to customization over time, a shift the paper attributes to rising training costs and the popularity of low-rank adaptation (LoRA).
  • The deploy class containing only one method identifies a concrete open area: federated inference with heterogeneous client resources is nearly untouched.
  • Healthcare is the canonical FL use case but remains thinly populated, with few methods tested on clinical data, signaling an application gap.
  • Most reviewed methods build on Federated Averaging (FedAVG), so adopting more adaptive aggregation mechanisms is a clear, unexplored direction the paper highlights.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's three-scale ratings for complexity, efficiency, and scalability could be turned into a quantitative benchmark if the 42 methods were evaluated on a shared dataset; the authors note that no such standardized benchmark currently exists.
  • The single deploy-stage method suggests that inference over heterogeneous devices, such as extracting tokens from different transformer layers depending on client resources, may be a growth area for edge AI.
  • Because grey literature and non-first-page search results were excluded, practitioner-oriented techniques such as system-level communication optimizations may be underrepresented, so the taxonomy may need revision as more methods appear.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript surveys technical methods at the intersection of foundational models (FMs) and federated learning (FL). The authors propose a taxonomy organized by development lifecycle stage — train, customize, deploy — with subclasses based on algorithmic technique, and they compare the identified methods along complexity, efficiency, and scalability. The paper also presents a healthcare case study, practical implementation guidance, answers to five research questions, and a discussion of open challenges. The central claim is that this is the first unified, self-contained survey of the FL-FM intersection and that the proposed taxonomy is non-overlapping, improving on prior taxonomies.

Significance. A reliable map of the FL-FM intersection would be genuinely useful, and this manuscript has real strengths: it documents a PRISMA-style study protocol, provides detailed descriptions of a substantial number of methods, positions itself against prior surveys in Table 1, and includes a self-contained appendix on FL, contrastive pre-training, LoRA, and knowledge distillation. The healthcare and practical-perspectives sections give practitioners a starting point. However, the survey's principal added value — completeness of coverage and a non-overlapping taxonomy — is not yet established by the reported evidence. The count inconsistencies, the page-capped search, the internal-only validation, and the informal scoring of Table 3 all bear on the load-bearing claims and require attention.

major comments (4)
  1. [Abstract; Fig. 1; §Discussion (RQ1); §Conclusions] The corpus counts are mutually inconsistent. The abstract says 'over 250 thoroughly reviewed articles' and '42 unique methods'; the PRISMA flow (Fig. 1) reports 260 full-text assessments, 42 articles after full-text filtering, 6 added by snowballing, and 48 'relevant articles'; the RQ1 answer in the Discussion says 'approximately 48 distinct methods'; the Conclusions say 'over 260 articles were manually inspected, with more than 40 identified as highly relevant.' If '42 unique methods' refers to the 42 articles before snowballing, then the snowballed articles are not reflected; if methods can exceed one per article, the mapping is unexplained. Because the paper's contribution is a complete map of methods, the exact corpus underlying the taxonomy must be unambiguous.
  2. [Survey Methodology; Study Limitations] The search strategy does not support the claim that the survey covers 'the majority of articles' at this intersection. Google and Google Scholar were limited to the first five pages, Scopus/ScienceDirect to 50 API answers per query, screening was title/abstract only, and grey literature was excluded. Snowballing started only from the already-included set, so it cannot recover a method family absent from the initial queries. A single omitted family — for example, a distinct PEFT strategy or aggregation mechanism — could change the leaves of the taxonomy and the gap analysis. The Limitations section candidly acknowledges selection bias, but the abstract's 'majority of articles' claim remains stronger than the methodology can support. The authors should either substantially expand the search or soften the coverage claim to match the evidence.
  3. [Survey Methodology — Taxonomy; Fig. 2] The 'non-overlapping' property of the taxonomy is not validated. The taxonomy is derived from the methods it classifies, and the described internal validation — 'Two experts independently assigned each method to one of the taxonomy classes, followed by a review to resolve disagreements' — only checks assignment to predefined classes. It does not test completeness, nor does it test mutual exclusivity against a fresh set of methods outside those used to build the taxonomy. Some boundaries are delicate; for instance, prompt-based methods (Guo et al., 2023b; Su et al., 2022; Zhao et al., 2023) are placed under 'Train / test-time adaptation' because the pre-training phase adjusts inputs, but the same techniques could naturally be described as customization. A formal decision rule for each class, together with an external or held-out validation of both completeness and non-overlap, is needed t
  4. [Practical perspectives; Table 3] Table 3 is load-bearing for RQ3 and for the practical insights, but the scoring procedure is not reproducible. The text says complexity is 'estimated based on effort', efficiency by 'amount of information transmitted', and scalability by 'resources required', with two experts assigning scores and resolving disagreements. No concrete rubric is given: there is no operational definition of 'low', 'moderate', or 'high', no inter-rater agreement measure, and no calculation or reference supporting particular placements. The table also mixes non-FL background work with FL methods: Houlsby et al. (2019) is not a federated learning method, yet it is rated in the FL-FM comparison. The guidance 'best to begin with partial model training...' is based on these ratings, so the table should either be accompanied by a detailed scoring protocol and agreement analysis, or be explicitly framed as an inform
minor comments (4)
  1. [Fig. 5 caption] Typo: 'from the clients tosss the server' should be 'to the server'.
  2. [Study Limitations; Discussion] Typos: 'othre independent researchers' should be 'other'; 'complexiry' should be 'complexity'; 'encription' should be 'encryption'.
  3. [Methods — Distillation] Missing cross-reference: 'as described in Section .' appears with no section number.
  4. [Table 3] The layout is hard to parse: the star columns are separated by criterion but the reader must reconstruct the mapping from the caption. Consider giving each criterion an explicit column header (e.g., 'Complexity: Low / Med / High') and using a single cell per method with its rating, or a checkmark matrix.

Circularity Check

0 steps flagged

No significant circularity: the survey's taxonomy is descriptive and no central claim reduces to its inputs by construction.

full rationale

This paper is a literature survey and taxonomy construction, not a derivation of predictive results. Its central outputs are (i) a curated corpus of methods at the FL-FM intersection, (ii) a taxonomy organized by lifecycle stage (train/customize/deploy) with subclasses, and (iii) qualitative expert-assigned ratings of complexity, efficiency, and scalability. None of these are predictions fitted to data, and there is no equation in which an output is equivalent to an input by construction. The taxonomy was built from the reviewed methods, as the Abstract states: 'The methods were used to construct the taxonomy.' This is circular only in the trivial descriptive sense that a classification scheme is induced from the items it classifies; the paper does not claim to use the taxonomy to predict an external quantity or to verify an independent mathematical result. The internal validation with two experts checks consistency of assignment to the already-defined taxonomy, not the taxonomy's external validity, so it does not constitute a self-referential derivation. There are no self-citations by the authors: Hatfaludi and Serban appear in no reference in the bibliography, so no self-citation chain is load-bearing. The stated limitations — five-page search caps, 50-API-result caps, title/abstract screening, and exclusion of grey literature — undermine the coverage claim ('aims to cover the majority of articles'), but an incomplete or non-representative corpus is a correctness/completeness threat, not a circularity. Similarly, the inconsistent counts ('42 unique methods', '48 relevant articles', 'approximately 48 distinct methods') are reporting inconsistencies, not evidence that any result is defined in terms of itself. By the rules of this pass, which require quoting a specific reduction or self-citation chain, no circular step can be exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The survey rests on assumptions about literature coverage, screening validity, taxonomy non-overlap, and the reliability of subjective ratings. No free parameters or invented entities are present.

axioms (5)
  • domain assumption The search engines and queries (first five pages, 50 API answers per query) retrieve a representative sample of the FL+FM literature.
    Used to justify coverage in Survey Methodology; limited page/API limits may exclude relevant work.
  • domain assumption The inclusion criteria and manual screening correctly identify all methods relevant to the research questions.
    Survey Methodology; screening is subjective and not fully documented.
  • ad hoc to paper The lifecycle taxonomy classes (train, customize, deploy and subclasses) are non-overlapping and cover all possible methods.
    Taxonomy section; validated only by two internal experts, not against external benchmarks.
  • ad hoc to paper The complexity, efficiency, and scalability ratings in Table 3 are consistent and meaningful proxies for practical properties.
    Practical perspectives; star ratings are author-assigned and lack formal definitions or measurements.
  • ad hoc to paper The expert consensus process eliminates subjective bias in taxonomy assignment and ratings.
    Taxonomy and Practical perspectives; only two experts from the same team, no inter-rater reliability measure.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 28018 in / 11682 out tokens · 109099 ms · 2026-08-05T05:32:25.407396+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights." pith.science (2026). https://pith.science/paper/ILVJXIUM

@misc{pith2026250905142,
  author       = {Pith},
  title        = {Pith review of: Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILVJXIUM}},
  note         = {Machine review of arXiv:2509.05142}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning.

Figures

Figures reproduced from arXiv: 2509.05142 by Alex Serban, Cosmin-Andrei Hatfaludi.

Figure 1
Figure 1. Figure 1: PRISMA flow chart illustrating the article selection process at each stage of the filtering criteria, as defined [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of methods for training, customizing or deploying foundational models using federating learning. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Temporal trends in the evolution of methods over the years following the taxonomy illustrated in Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of pre-training methods for foundational models using federated learning. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of additive fine-tuning methods using federated learning, where for each type of methods the [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Overview of knowledge distillation methods using federated learning. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Illustration of pre-training FMs using contrastive learning. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Illustration of LoRA. aims to decorrelate embeddings of distinct objects while preserving the variance of embeddings for similar concepts, and (iii) masked input modelling (e.g., (He et al., 2022; Baevski et al., 2023)), where the task involves reconstructing original inputs from masked versions of it (e.g., masking words in sentences or patches of images). Recent studies sug￾gest that, despite their diffe… view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of knowledge distillation. for minimizing the distance between the teacher’s and student’s outputs is the Kullback-Leibler (KL) divergence (Gou et al., 2021). In the context of FL, KD is used to transfer knowledge from local models to a global model by minimizing the divergence between their outputs. This approach allows the global model to benefit from the collective knowledge of the local mo… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages · 2 internal anchors

  1. [1]

    Agarwal, A., Rezagholizadeh, M., and Parthasarathi, P. (2023). Practical takes on federated learning with pretrained language models. InFindings of the Association for Computational Linguistics: EACL

    work page 2023

  2. [3]

    Sun, G., Mendieta, M., Yang, T., and Chen, C. (2022). Exploring parameter-efficient fine-tuning for improving communication efficiency in federated learning. Tan, Y ., Long, G., Ma, J., Liu, L., Zhou, T., and Jiang, J. (2022). Federated learning from pre-trained models: A contrastive learning approach.Advances in neural information processing systems. Tan...

    Pith/arXiv arXiv 2022

  3. [4]

    Deng, Y ., Qiao, Z., Ren, J., Liu, Y ., and Zhang, Y . (2023). Mutual enhancement of large and small language models with cross-silo knowledge transfer.International Conference on Learning Representations (ICLR). Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2024). Qlora: Efficient finetuning of quantized llms. Advances in Neural Informati...

    work page 2023

  4. [5]

    Garcia Bernal, D., Giaretta, L., Girdzijauskas, S., and Sahlgren, M. (2021). Federated word2vec: Leveraging federated learning to encourage collaborative representation learning. Garrido, Q., Chen, Y ., Bardes, A., Najman, L., and LeCun, Y . (2023). On the duality between contrastive and non- contrastive self-supervised learning. InThe 11th Int. Conf. on ...

    work page 2021

  5. [6]

    Han, P., Wang, S., Jiao, Y ., and Huang, J. (2024). Federated learning while providing model as a service: Joint training and inference optimization. InIEEE INFOCOM 2024-IEEE Conference on Computer Communications, pages 631–640. IEEE. He, K., Chen, X., Xie, S., Li, Y ., Doll´ar, P., and Girshick, R. (2022). Masked autoencoders are scalable vision learners...

    work page 2024

  6. [7]

    J., Lee, W.-H., Leung, K

    Jiang, Y ., Wang, S., Valls, V ., Ko, B. J., Lee, W.-H., Leung, K. K., and Tassiulas, L. (2022). Model pruning en- ables efficient federated learning on edge devices.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10374–10386. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormo...

    Pith/arXiv arXiv 2022

  7. [8]

    C., Lo, W.-Y ., Dollar, P., and Girshick, R

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y ., Dollar, P., and Girshick, R. (2023). Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision. Kuo, K., Raje, A., Rajesh, K., and Smith, V . (2024). Federated LoRA with sparse communication.arXiv:240...

    Pith/arXiv arXiv 2023

  8. [11]

    Rehman, Y . A. U., Gao, Y ., De Gusm˜ao, P. P. B., Alibeigi, M., Shen, J., and Lane, N. D. (2023). L-dawa: Layer-wise divergence aware weight aggregation in federated self-supervised visual representation learning. InProceedings of the IEEE/CVF international conference on computer vision. Ren, C., Yu, H., Peng, H., Tang, X., Zhao, B., Yi, L., Tan, A. Z., ...

    Pith/arXiv arXiv 2023

  9. [36]

    Li, H., Zhao, X., Guo, D., Gu, H., Zeng, Z., Han, Y ., Song, Y ., Fan, L., and Yang, Q. (2024a). Federated domain- specific knowledge transfer on large language models using synthetic data.arXiv:2405.14212. Li, Q., He, B., and Song, D. (2021). Model-contrastive federated learning. InProceedings of the IEEE/CVF conference on computer vision and pattern rec...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [37]

    Parulekar, A., Collins, L., Shanmugam, K., Mokhtari, A., and Shakkottai, S. (2023). Infonce loss provably learns cluster-preserving representations. InThe Thirty Sixth Annual Conference on Learning Theory, pages 1914–1961. PMLR. Pfeiffer, J., Kamath, A., R ¨uckl´e, A., Cho, K., and Gurevych, I. (2021). Adapterfusion: Non-destructive task compo- sition for...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [2023]

    M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F

    Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. (2025). Foundation models defining a new era in vision: a survey and outlook.IEEE Transactions on Pattern Analysis and Machine Intelligence. Azad, B., Azad, R., Eskandari, S., Bozorgpour, A., Kazerouni, A., Rekik, I., and Merhof, D. (2023). Foundational mo...

    Pith/arXiv arXiv 2025

  12. [2024]

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. (2025). A survey on mixture of experts in large language models.IEEE Transactions on Knowledge and Data Engineering. Cai, Z., Ravichandran, A., Maji, S., Fowlkes, C., Tu, Z., and Soatto, S. (2021). Exponential moving average normal- ization for self-supervised and semi-supervised learning. InP...

    Pith/arXiv arXiv 2025