Pith. sign in

REVIEW 3 major objections 5 minor 62 references

This paper claims that a federated framework with dual mixture-of-experts and caption-enhanced learning lets hospitals with disjoint imaging modalities jointly train a medical foundation model, outperforming existing federated methods on cl

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:19 UTC pith:3GKNOMLV

load-bearing objection A genuinely useful federated medical classification system, but the advertised generalization guarantee is false and the caption learning is likely label leakage — still worth a careful referee, not a desk reject. the 3 major comments →

arxiv 2607.13386 v1 pith:3GKNOMLV submitted 2026-07-15 cs.CV

FM²: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen , Ting Shu This is my paper
classification cs.CV
keywords Federated LearningMedical ImagingData HeterogeneityFoundation ModelsMixture-of-ExpertsMedical VQAModality HeterogeneityCaption-Enhanced Learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that imaging-modality heterogeneity—hospitals operating in overlapping versus disjoint modality regimes—can be handled by one unified federated framework. It proposes FM2, which trains a visual backbone from scratch, pairs each client with a personalized class-wise expert and a globally shared domain-wise expert, and regularizes the domain experts toward consensus. The claim is that this disentanglement, plus locally retained language captions, yields a single system that consistently beats federated baselines on three tasks and transfers to unseen modalities. If true, it would let medical institutions with different scanner types collaboratively build a multimodal foundation model without sharing raw images or labels.

Core claim

The central discovery is that the two kinds of heterogeneity in federated medical imaging—label skew within a shared modality and completely disjoint modalities across clients—can be addressed by a dual-expert decomposition. FM2 splits each client's model into a Class-wise Mixture-of-Experts (one expert per class, kept local) and a Domain-wise Mixture-of-Experts (one expert per imaging modality, globally aggregated), and adds a Heterogeneous Modality Alignment regularizer that pulls the domain experts toward a global FedAvg consensus. The paper proves an O(1/sqrt(T)) convergence rate and a generalization bound for this objective. To transfer knowledge when no visual modality is shared, it ad

What carries the argument

The central object is the dual MoE decomposition: a Class-wise MoE with per-class experts that stay on the client, and a Domain-wise MoE with per-modality experts that are federated and globally aggregated. The Heterogeneous Modality Alignment penalty λ||M_d^i - Mhat_d||² anchors the domain experts to the global consensus, and the Caption-Enhanced Learning loss aligns visual features with caption text embeddings via a contrastive objective. Together they carry the argument: the local expert handles label heterogeneity, the global expert handles modality heterogeneity, and text acts as a bridge across disjoint visual domains.

Load-bearing premise

The framework's cross-modality transfer rests on the assumption that GPT-4o-generated captions describe image content without embedding the class label; the example caption in Section 3.4 explicitly states the diagnosis ("consistent with moderate non-proliferative diabetic retinopathy"), so this assumption is untested and may be violated.

What would settle it

Generate a second set of captions that describe only morphology and visual findings, with all diagnosis-bearing words removed, then rerun the MIMH-SA and cross-modality CEL experiments; if the accuracy gains over image-only baselines disappear or shrink sharply, the CEL contribution is explained by label-derived text rather than by a shared semantic space.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Institutions with entirely disjoint imaging equipment can jointly train a shared visual backbone while each site retains its own class-level personalization.
  • The HMA regularization gives a provable O(1/sqrt(T)) convergence rate and an explicit generalization bound, independent of how heterogeneous the modalities are.
  • With Caption-Enhanced Learning, even clients that share no visual modality can transfer knowledge through a common textual vocabulary, improving all federated baselines tested.
  • The same MoE-enhanced visual features extend to medical visual question answering without task-specific re-engineering.
  • A model trained on five modalities generalizes to held-out modalities such as chest X-ray, OCT, and abdominal CT with only 5% fine-tuning data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline cross-modal gains, especially in the non-overlapped setting, may stem in part from label leakage: diagnostic captions that explicitly state the diagnosis could make the contrastive objective a softer form of label supervision rather than a pure modality-invariant bridge.
  • A direct test of this would be to generate morphology-only captions that never name the diagnosis and rerun the MIMH-SA experiments; if the gains vanish, the semantic-bridge story is weakened.
  • The dual MoE recipe—keep task-specific experts local, share domain experts globally, regularize only the shared part—may generalize beyond imaging to other federated modalities such as time series or sensor data.
  • The Observer Top-K sampling step is under-analyzed: in extreme non-IID settings it might oversample the most confident examples and bias expert training; comparing it against random sampling at the same budget would clarify its role.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FM², a federated learning framework for medical imaging under 'imaging modality heterogeneity,' covering overlapped clients (shared modalities, non-IID labels) and non-overlapped clients (disjoint modalities). The architecture couples a locally retained Class-wise MoE with a globally aggregated Domain-wise MoE, and adds a Heterogeneous Modality Alignment regularizer with claimed O(1/√T) convergence and generalization guarantees. It also introduces Caption-Enhanced Learning using locally retained GPT-4o captions as a cross-modality textual bridge, and extends the same backbone to federated medical VQA. A new MIMH benchmark (five MedMNIST modalities, four configurations) is used for classification and CEL, and three VQA task settings are evaluated on SLAKE/VQA-RAD/VQA-Med. The empirical results show large and consistent margins over FedAvg/FedProx/FedProto/FedRep and specialized VQA baselines.

Significance. If the results hold, FM² would be a useful step toward federated multimodal medical foundation models: the dual MoE decomposition is a plausible way to separate class-specific personalization from modality-shared representations, and the MIMH benchmark plus the VQA evaluation provide a broad testbed. The experiments are extensive—multiple non-IID severities, participation ratios, ablations, hyperparameter sensitivity, communication cost, and convergence curves—and the classification margins over strong FL baselines are substantial. The CEL and VQA extensions broaden the framework's scope. However, two load-bearing pieces are not currently established: the advertised generalization guarantee (Proposition 2) is invalid as written, and the CEL evaluation is confounded by class labels appearing in the GPT-4o captions. The proof of Theorem 1 also does not match the executed federated update. The paper's empirical contribution is therefore stronger than its theoretical and CEL-specific claims.

major comments (3)
  1. [Sec. 3.5, Proposition 2 / Eq. (10)] Proposition 2's generalization guarantee is not derived. After Jensen, the proof asserts each expert's excess risk is controlled by d_TV to its Bayes-optimal law, then replaces the two individual distances by d_JS(P_d, P_c) with no connecting step. The final ε_V = inf_{V'} E||V(X)-V'(X)||^2 is an Observer reconstruction error, not an approximation error of h, and is not linked to the risk. The bound is false as stated: with squared loss and deterministic labels, f_d = f_c ≡ 0.5 gives h = 0.5, d_JS = 0, R*_d = R*_c = ε_V = 0, yet excess risk 0.25, contradicting Eq. (10). Eq. (10) has no sample-size term, so it is not a finite-sample generalization bound. Since 'provable ... generalization guarantees' is an advertised contribution (abstract; Sec. 1 item 2), this must be corrected or removed.
  2. [Sec. 3.4, Eq. (4); Tables 3 and Fig. 3(c-d)] The CEL evaluation is confounded by label leakage through the GPT-4o captions. The example in Sec. 3.4 — 'consistent with moderate non-proliferative diabetic retinopathy' — states the class label explicitly. Because Eq. (4) aligns visual features with such caption embeddings, the reported gains over image-only baselines (e.g., FM2-Large cross-modality average 95.04 vs. 90.48 in Table 3) may reflect access to class names rather than a modality-invariant 'textual semantic bridge.' No control removing diagnostic tokens (e.g., morphology-only paraphrases) is reported, and the 'Template' condition in Fig. 3(c) is not described enough to know whether it also contains labels. This is load-bearing for the CEL contribution and the out-of-modality generalization claim; it should be tested explicitly.
  3. [Sec. 3.5, Theorem 1 proof / Eq. (6)] The proof of Theorem 1 analyzes an aggregated update θ^{t+1}=θ^t−ηḡ_t in which ḡ_t contains the alignment term 2λ(M_i^d−M̂_d). In the actual FedAvg procedure of Sec. 3.3, the server aggregates local DMoE updates with weights p_i, and Σ_i p_i (M_i^d−M̂_d)=0, so the alignment term cancels in the aggregate; the global update is only the averaged data gradient. Thus the 'align. drift' term 2λLΔM/T in Eq. (6) does not follow from the executed dynamics, and the convergence claim for FM²'s federated update is not established. The proof also assumes a single local step and full participation, whereas experiments use 1 local epoch and join ratios r=30/50/100%. Clarify or re-derive.
minor comments (5)
  1. [General] Appendices A–D are referenced in Secs. 3.1–3.2 and 4.1–4.2 but are not included in the manuscript. Appendix D (full CEL prompts and quality-filtering criteria) is essential for assessing the label-leakage concern.
  2. [Sec. 4.1] 'ResNet-4 [22]' is not a standard architecture and is not defined; please specify the exact network used.
  3. [Sec. 4.2] The CEL experiments upgrade to CLIP ViT-B/16, a natural-image pretrained model. This appears to conflict with the paper's motivation to train from scratch for medical-domain fidelity; clarify how this fits with the framework described in Sec. 3.
  4. [Fig. 3(c-d)] The 'Template' and 'LLM Expert' conditions are not precisely defined, and 'expert annotations' is ambiguous because no set of human expert captions was introduced. Also specify whether the held-out datasets in Table 3's Cross-Mod. column receive GPT-4o captions during 5% fine-tuning.
  5. [Sec. 3.5 / Eq. (9)] The proof text says the second term from the alignment drift recursion 'enters as O(Lηδ²)', while the displayed bound contains 4Lηδ²; the constants should be reconciled.

Circularity Check

2 steps flagged

The advertised 'provable generalization guarantee' is asserted rather than derived, and CEL's caption bridge carries the target labels, so two central claims reduce to unproven assumptions / label-derived supervision.

specific steps
  1. other [Section 3.5, Proposition 2 (Eq. 10 and proof)]
    "The cross-term in the convex combination carries a factor 2√γ_dγ_c (by Cauchy–Schwarz on the weighted excess risks), giving 2√(2γ_dγ_c d_JS(P_d∥P_c))."

    The proof only established |E[ℓ(f_d,y)]−R*_d| ≤ d_TV(P_{f_d},P_{f*_d}) and the analogous bound for f_c. Cauchy–Schwarz on those two per-expert bounds cannot produce d_JS(P_d∥P_c), the divergence between the two experts' own output laws; no inequality connects distance-to-Bayes to distance-between-experts. Eq. (10) therefore inserts the JS term it claims to control, making the advertised generalization guarantee an assumed conclusion rather than a derived one. The final ε_V term is also defined as a reconstruction error, not the composed predictor's approximation error. This is a load-bearing omitted proof, not a classical circular fit.

  2. self definitional [Section 3.4 (CEL) and Eq. (4); Table 3 in Section 4.2]
    "we generate diagnostic captions for each image ... in a zero-shot setting (e.g., "Fundus image showing microaneurysms and hard exudates in the macular region, consistent with moderate non-proliferative diabetic retinopathy") ... LCEL = −1/B Σ_j log exp(sim(v_j,t_j)/τ) / Σ_k exp(sim(v_j,t_k)/τ)"

    The caption t_j aligned to visual feature v_j in Eq. (4) explicitly restates the class label of the image. The 'textual semantic bridge' is therefore constructed from label-carrying text, so CEL's reported cross-modality and out-of-modality gains (e.g., FM2-Large+CEL 95.04 vs 90.48 image-only in Table 3) are attributable to label-derived supervision rather than to a label-independent shared vocabulary. The claimed bridge reduces by construction to the label information already in the data.

full rationale

Theorem 1 is a self-contained FedProx-style convergence proof and does not exhibit circularity. The paper's self-citations (e.g., [8], [13]) are background support and are not load-bearing for the core derivation; no uniqueness theorem or ansatz is imported from the authors' prior work. The circularity-relevant issues are concentrated in two advertised claims. First, Proposition 2: the final generalization bound uses d_JS(P_d∥P_c) even though the derivation only bounds each expert's risk by its total variation distance to its Bayes predictor; the Cauchy–Schwarz sentence is not a derivation, so the provable-generalization contribution reduces to an assumed smallness of cross-expert divergence. Second, CEL: the GPT-4o captions used in Eq. (4) state the diagnosis explicitly, so the caption-enhanced gains are confounded by label leakage and the semantic-bridge claim is label-defined by construction. These are partial, load-bearing reductions; they do not affect the self-contained empirical benchmark comparisons in Table 1, which are not circular. Score 6 reflects partial circularity of the central guarantee and the CEL extension.

Axiom & Free-Parameter Ledger

8 free parameters · 9 axioms · 4 invented entities

The framework rests almost entirely on new architectural components whose behavior is demonstrated only in the paper's own experiments; no independent benchmarks, formalized guarantees, or external predictions support them. The most consequential implicit assumption — that GPT-4o captions are label-free — is contradicted by the paper's own example caption.

free parameters (8)
  • gamma_c, gamma_d = 0.5, 0.5
    Balancing weights in the inference combination (Eq. 2); hand-chosen defaults. Fig. 3b shows ±2% impact for other values.
  • lambda = 0.1
    HMA alignment strength in Eq. (3); chosen by hand; sensitivity explored in Fig. 3b.
  • alpha
    Weight of the CEL loss in the total local objective (Section 3.4); no default value is reported.
  • tau
    Temperature in the InfoNCE contrastive loss (Eq. 4); not specified in the main text.
  • K (Top-K samples per class)
    Number of representative samples selected per class by the Observer (Section 3.2); sensitivity analysis deferred to Appendix B.
  • Number of DMoE experts
    Pre-assigned via the global modality vocabulary at federation setup (Section 3.2); a design choice, not fitted.
  • Fine-tuning fraction for cross-modality adaptation = 5%
    Arbitrary fraction of held-out data used in Tables 2 and 3; no sensitivity analysis is provided.
  • LoRA rank and alpha (VQA) = r=8, alpha=32
    Common defaults for low-rank adaptation; fixed across all VQA experiments.
axioms (9)
  • standard math Assumption 1: Each local loss L_i is L-smooth
    Required for the convergence proof in Section 3.5; standard in non-convex optimization analysis.
  • standard math Assumption 2: Unbiased stochastic gradients with bounded variance sigma^2
    Standard stochastic gradient assumption used in Theorem 1's proof.
  • domain assumption Assumption 3: Bounded gradient heterogeneity, (1/Z) sum ||grad L_i - grad F||^2 <= delta^2
    Bounds the dissimilarity between client objectives; standard in federated learning convergence analysis.
  • domain assumption Global modality vocabulary declared at federation setup
    Section 3.2 assumes 'each clinical site knows its imaging modality' and experts are pre-assigned via a shared vocabulary; not all real-world sites may disclose this.
  • domain assumption The loss in Proposition 2 is bounded in [0,1] and convex in its first argument
    Needed for the total-variation bound and Jensen inequality in the generalization proof; typical classification losses are bounded only with clipping.
  • standard math d_TV(P_f, P_f*) <= |E[l(f)] - R*| for losses in [0,1]
    A standard coupling inequality, but it requires the joint law of (f(X), y) and holds for bounded losses; invoked without proof in Proposition 2.
  • standard math Pinsker's inequality relating total variation and JS divergence
    Used in Proposition 2's proof; correct as stated, but the transition from the convex-combination bound to the JS term is not derived.
  • ad hoc to paper GPT-4o captions are faithful, detailed, and do not leak the class label
    The CEL method assumes the captions provide a modality-invariant semantic description without encoding the label; the example caption 'consistent with moderate non-proliferative diabetic retinopathy' contradicts this, making the assumption load-bearing and likely false.
  • domain assumption Pretrained CLIP/LLaVA backbones are appropriate for medical CEL/VQA despite the paper's own claim that natural-image representations transfer poorly
    CEL uses CLIP ViT-B/16 and VQA uses LLaVA-1.5/CLIP; this contradicts the from-scratch motivation and assumes the pretrained features are helpful.
invented entities (4)
  • Class-wise MoE (CMoE) no independent evidence
    purpose: Per-class personalized expert module retained locally to handle label heterogeneity
    Introduced by this paper; its effectiveness is only shown in the paper's own ablation; no external falsifiable prediction.
  • Domain-wise MoE (DMoE) no independent evidence
    purpose: Per-modality shared expert module aggregated across clients to handle modality heterogeneity
    No independent evidence outside the paper's experiments; the pre-assigned modality vocabulary is an unverified assumption.
  • Heterogeneous Modality Alignment (HMA) regularizer no independent evidence
    purpose: Proximal penalty pulling local DMoE parameters toward the global FedAvg aggregate
    A method component; its claimed benefit is based on the paper's t-SNE and ablations, not on an external benchmark.
  • Observer with Top-K Sampling no independent evidence
    purpose: Backbone-guided selection of representative samples for training the MoE modules in stage two
    A mechanism specific to this paper; sensitivity analysis is deferred to an appendix and not independently validated.

pith-pipeline@v1.3.0-alltime-deepseek · 20788 in / 21078 out tokens · 184016 ms · 2026-08-02T05:19:46.191999+00:00 · methodology

0 comments
read the original abstract

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM$^2$, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM$^2$ equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable $O(1/\sqrt{T})$ convergence and generalization guarantees. FM$^2$ further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Figures

Figures reproduced from arXiv: 2607.13386 by Shengchao Chen, Ting Shu.

Figure 1
Figure 1. Figure 1: FM2 (single-client view). Observer extracts representative samples for training dual MoE modules. Domain-wise MoE is globally aggregated; Class-wise MoE remains local. Text inputs (diagnostic captions for CEL; questions for VQA) are processed by a local text encoder and fused with MoE-enhanced visual features, enabling all downstream tasks within a single protocol. where 𝑿mid and 𝑿ˆ denote latent and outpu… view at source ↗
Figure 2
Figure 2. Figure 2: MIMH with four configs: MIMH-5/4/3-Domain (over [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation and analysis across all three tasks. (a) Component ablation. (b) Hyperparameter sensitivity across different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Convergence and feature analysis. (a) Classifica [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 12 linked inside Pith

  1. [1]

    Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. 2021. Federated learning based on dynamic regularization.arXiv preprint arXiv:2111.04263(2021)

    Pith/arXiv arXiv 2021

  2. [2]

    Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. 2019. Federated learning with personalization layers.arXiv preprint arXiv:1912.00818(2019)

    Pith/arXiv arXiv 2019

  3. [3]

    Samyadeep Basu, Shell Hu, Daniela Massiceti, and Soheil Feizi. 2024. Strong baselines for parameter-efficient few-shot fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 11024–11031

    2024

  4. [4]

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning Müller. 2019. Vqa-med: Overview of the medical visual question an- swering task at imageclef 2019. InProceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019

    2019

  5. [5]

    Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman, Sadid A Hasan, and Henning Müller. 2021. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. InProceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-working notes. 21-24 September 2021

    2021

  6. [6]

    Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. Tinytl: Reduce memory, not parameters for efficient on-device learning.Advances in Neural Information Processing Systems33 (2020), 11285–11297

    2020

  7. [7]

    Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. 2024. Feddat: An approach for foundation model finetuning in multi-modal hetero- geneous federated learning. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 11285–11293

    2024

  8. [8]

    Shengchao Chen, Guodong Long, Michael Blumenstein, and Jing Jiang. 2026. FeDaL: Federated Dataset Learning for General Time Series Foundation Models. InThe Fourteenth International Conference on Learning Representations

    2026

  9. [9]

    Shengchao Chen, Guodong Long, Jing Jiang, and Chengqi Zhang. 2024. Per- sonalized adapter for large meteorology model on devices: Towards weather foundation models.Advances in Neural Information Processing Systems37 (2024), 84897–84943

    2024

  10. [10]

    Shengchao Chen, Guodong Long, Jing Jiang, and Chengqi Zhang. 2025. Federated foundation models on heterogeneous time series. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 15839–15847

    2025

  11. [11]

    Shengchao Chen, Guodong Long, Tao Shen, and Jing Jiang. 2023. Prompt feder- ated learning for weather forecasting: Toward foundation models on meteoro- logical data.arXiv preprint arXiv:2301.09152(2023)

    Pith/arXiv arXiv 2023

  12. [12]

    Shengchao Chen, Sufen Ren, Guanjun Wang, Mengxing Huang, and Chenyang Xue. 2023. Interpretable CNN-multilevel attention transformer for rapid recog- nition of pneumonia from chest X-ray images.IEEE Journal of Biomedical and Health Informatics28, 2 (2023), 753–764

    2023

  13. [13]

    Shengchao Chen and Ting Shu. 2025. Restyled, Tuning, and Alignment: Taming VLMs for Federated Non-IID Medical Image Analysis. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 530– 540

    2025

  14. [14]

    I Glenn Cohen and Michelle M Mello. 2018. HIPAA and protecting health infor- mation in the 21st century.Jama320, 3 (2018), 231–232

    2018

  15. [15]

    Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. InInter- national conference on machine learning. PMLR, 2089–2099

    2021

  16. [16]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020)

    Pith/arXiv arXiv 2020

  17. [17]

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized feder- ated learning: A meta-learning approach.arXiv preprint arXiv:2002.07948(2020)

    Pith/arXiv arXiv 2020

  18. [18]

    Lulu Feng and Shengchao Chen. 2026. Visual and textual spaces both matter: Taming CLIP for non-IID federated medical image classification.Knowledge-Based Systems338 (2026), 115524

    2026

  19. [19]

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing.ACM Transactions on Computing for Healthcare3, 1 (2021), 1–23

    2021

  20. [20]

    Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. 2023. Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.IEEE Transactions on Mobile Computing23, 5 (2023), 5179–5194

    2023

  21. [21]

    Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. 2020. Lower bounds and optimal algorithms for personalized federated learning.Ad- vances in Neural Information Processing Systems33 (2020), 2304–2315

    2020

  22. [22]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  23. [23]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. InInternational conference on machine learning. PMLR, 2790–2799

    2019

  24. [24]

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification.arXiv preprint arXiv:1909.06335(2019)

    Pith/arXiv arXiv 2019

  25. [25]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685(2021)

    Pith/arXiv arXiv 2021

  26. [26]

    Joana Palés Huix, Adithya Raju Ganeshan, Johan Fredin Haslum, Magnus Söder- berg, Christos Matsoukas, and Kevin Smith. 2024. Are Natural Domain Foun- dation Models Useful for Medical Image Classification?. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 7634–7643

    2024

  27. [27]

    Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning.cell172, 5 (2018), 1122–1131

    2018

  28. [28]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al

  29. [29]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images.Scientific data5, 1 (2018), 1–10

    2018

  30. [30]

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and robust federated learning through personalization. InInternational conference on machine learning. PMLR, 6357–6368

    2021

  31. [31]

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems2 (2020), 429–450

    2020

  32. [32]

    Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623(2021)

    Pith/arXiv arXiv 2021

  33. [33]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th international symposium on biomedical imaging (ISBI). IEEE, 1650–1654

    2021

  34. [34]

    Chang Liu, Yong Luo, Yongchao Xu, and Bo Du. 2024. Foundation models matter: federated learning for multi-center tuberculosis diagnosis via adaptive regularization and model-contrastive learning.World Wide Web27, 3 (2024), MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Shengchao Chen and Ting Shu 1–17

    2024

  35. [35]

    Meilin Liu, Jiaying Wang, and Jing Shan. 2026. OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21099–21109

    2026

  36. [36]

    Yuxi Liu, Guibo Luo, and Yuesheng Zhu. 2024. FedFMS: Exploring Federated Foundation Models for Medical Image Segmentation. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 283– 293

    2024

  37. [37]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. InInternational Conference on Learning Representations

    2019

  38. [38]

    Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. 2021. Federated Multi-Task Learning under a Mixture of Distributions. In Advances in Neural Information Processing Systems, Vol. 34. 15434–15447

    2021

  39. [39]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep net- works from decentralized data. InArtificial intelligence and statistics. PMLR, 1273–1282

    2017

  40. [40]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  41. [41]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learning. PMLR, 8748–8763

    2021

  42. [42]

    Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečn`y, Sanjiv Kumar, and H Brendan McMahan. 2020. Adaptive feder- ated optimization.arXiv preprint arXiv:2003.00295(2020)

    Pith/arXiv arXiv 2020

  43. [43]

    Sufen Ren, Yule Hu, Shengchao Chen, and Guanjun Wang. 2024. Federated distillation for medical image classification: Towards trustworthy computer- aided diagnosis.arXiv preprint arXiv:2407.02261(2024)

    Pith/arXiv arXiv 2024

  44. [44]

    Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The annals of mathematical statistics(1951), 400–407

    1951

  45. [45]

    Pramit Saha, Felix Wagner, Divyanshu Mishra, Can Peng, Anshul Thakur, David A Clifton, Konstantinos Kamnitsas, and J Alison Noble. 2025. Fˆ 3OCUS-Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics. InProceedings of the Computer Vision and Pattern Recognition Conference....

    2025

  46. [46]

    Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022. Fedproto: Federated prototype learning across hetero- geneous clients. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8432–8440

    2022

  47. [47]

    Malte Tölle, Philipp Garthe, Clemens Scherer, Jan Moritz Seliger, Andreas Leha, Nina Krüger, Stefan Simm, Simon Martin, Sebastian Eble, Halvar Kelm, et al

  48. [48]

    Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection reg- ulation (gdpr).A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10–5555

    2017

  49. [49]

    Jiahao Wang, Hong Peng, Shengchao Chen, and Sufen Ren. 2024. Ensemble learning for retinal disease recognition under limited resources.Medical & Biological Engineering & Computing62, 9 (2024), 2839–2852

    2024

  50. [50]

    Xinghao Wu, Xuefeng Liu, Jianwei Niu, Guogang Zhu, and Shaojie Tang. 2023. Bold but cautious: Unlocking the potential of personalized federated learning through cautiously aggressive collaboration. InProceedings of the IEEE/CVF inter- national conference on computer vision. 19375–19384

    2023

  51. [51]

    Yihang Wu, Christian Desrosiers, and Ahmad Chaddad. 2024. FACMIC: Federated Adaptative CLIP Model for Medical Image Classification. InInternational Confer- ence on Medical Image Computing and Computer-Assisted Intervention. Springer, 531–541

    2024

  52. [52]

    Luyuan Xie, Tianyu Luan, Wenyuan Cai, Guochen Yan, Zhaoyu Chen, Nan Xi, Yuejian Fang, Qingni Shen, Zhonghai Wu, and Junsong Yuan. 2025. dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Anal- ysis. InProceedings of the Computer Vision and Pattern Recognition Conference. 10203–10213

    2025

  53. [53]

    Peng Xu, Xiatian Zhu, and David A Clifton. 2023. Multimodal learning with transformers: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence45, 10 (2023), 12113–12132

    2023

  54. [54]

    Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. 2019. Efficient multiple organ localization in CT image using 3D region proposal network.IEEE transactions on medical imaging38, 8 (2019), 1885–1898

    2019

  55. [55]

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. 2023. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific Data10, 1 (2023), 41

    2023

  56. [56]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Ilit Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations

    2023

  57. [57]

    Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez

  58. [58]

    Shaoting Zhang and Dimitris Metaxas. 2024. On the challenges and perspectives of foundation models for medical image analysis.Medical image analysis91 (2024), 102996

    2024

  59. [59]

    Weiming Zhuang, Chen Chen, and Lingjuan Lyu. 2023. When foundation model meets federated learning: Motivations, challenges, and future directions.arXiv preprint arXiv:2306.15546(2023)

    Pith/arXiv arXiv 2023

  60. [2020]

    Personalized federated learning with first order model optimization.arXiv preprint arXiv:2012.08565(2020)

    Pith/arXiv arXiv 2012

  61. [2023]

    InProceedings of the IEEE/CVF International Conference on Computer Vision

    Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026

  62. [2024]

    Federated Foundation Model for Cardiac CT Imaging.arXiv preprint arXiv:2407.07557(2024)

    arXiv 2024