Pith. sign in

REVIEW 2 major objections 5 minor 35 references

A prototype-conditioned mixture of experts synthesizes missing modalities in federated medical learning without public data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 00:53 UTC pith:AWHYVVJC

load-bearing objection Solid empirical FL paper: client-aware prototype bank + modality-routed MoE for missing CXR modalities beats FeatImp/PmcmFL without public data, especially under hetero partitions. the 2 major comments →

arxiv 2607.06633 v1 pith:AWHYVVJC submitted 2026-07-07 cs.CV cs.AI

ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities

Aavash Chhetri , Bibek Niroula , Eduard Vazquez , Yash Raj Shrestha , Prashnna Gyawali , Loris Bazzani , Binod Bhattarai This is my paper
classification cs.CV cs.AI
keywords multimodal federated learningmissing modalitiesprototype learningmixture of expertschest X-rayfeature synthesisnon-IID clients
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hospitals hold complementary imaging and report data but cannot pool it, and many sites only have one modality. Prior federated methods either demand a public multimodal dataset or invent the missing features from the available modality alone, which fails when institutions differ. ProMoE-FL instead keeps a global bank of client-aware class prototypes that summarize how each modality looks across sites, then routes those prototypes through a shared mixture of experts so a unimodal client can reconstruct the missing representation. On four chest X-ray collections under both matched and mismatched client partitions, the method raises diagnostic AUC over feature-imputation and pure-prototype baselines, and it preserves separation of rare pathologies that simpler synthesis collapses.

Core claim

Conditioning a shared mixture-of-experts decoder on an accumulated, client-aware global prototype bank of the missing modality produces more accurate cross-modal features than either public-data augmentation or synthesis that uses only the observed modality, and the gain is largest under realistic non-IID client mixtures.

What carries the argument

Prototype-Conditioned Mixture of Experts (ProMoE): a Transformer decoder bank whose experts attend over the global prototype set of the target modality, gated by a modality-aware router that also receives the source feature and the (from, to) modality indices.

Load-bearing premise

The paper assumes that cross-modal maps learned by reconstruction on multimodal clients, when steered by the accumulated prototype bank, still work for unimodal clients whose data distributions differ from those multimodal sites.

What would settle it

Train the same architecture on the heterogeneous 8:0:2 or 4:4:2 partitions but replace the accumulated client-aware prototype bank with either averaged global prototypes or pure source-only imputation; if macro AUC and rare-pathology centroid separation then match or exceed ProMoE-FL, the claimed value of the bank collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. ProMoE-FL addresses multimodal federated learning with missing modalities by synthesizing missing latent features via a prototype-conditioned Mixture-of-Experts. Clients maintain class-conditioned modality prototypes that are accumulated (not averaged) into a global client-aware prototype bank; a Transformer-decoder PCD attends over the target-modality bank, and a modality-index-aware router mixes E such experts (Eq. 4). Multimodal clients train the MoE by symmetric MSE reconstruction (Eq. 6); unimodal clients use the synthesized features for the task loss. Experiments on MIMIC-CXR, NIH Open-I, PadChest, and CheXpert cover nine I:T:M partitions under homogeneous and heterogeneous regimes, reporting multi-seed macro AUC against Zero/Unif filling, FeatImp, PmcmFL, and (with a virtual public client) CAR-MFL, plus t-SNE/UMAP qualitative checks and a single MoE-vs-single-PCD ablation.

Significance. Missing-modality multimodal FL is a realistic clinical constraint, and the paper targets it without requiring public multimodal data. The combination of an accumulated client-aware prototype bank with direction-aware MoE routing is a concrete architectural contribution relative to pure feature imputation (FeatImp) and class-prototype substitution (PmcmFL). Strengths include multi-seed Table 1 results across nine partitions in both homogeneous and heterogeneous regimes, explicit comparison to a public-data baseline when adapted, qualitative alignment evidence (Fig. 2), and a public code link. If the heterogeneous gains hold under broader scrutiny, the method is a useful practical advance for multi-center chest X-ray federations with incomplete modality availability.

major comments (2)
  1. [Abstract; Table 1] Abstract and §3 claim that ProMoE-FL 'consistently outperforms' SOTA in both homogeneous and heterogeneous settings. Table 1 supports this fully for heterogeneous partitions, but under homogeneous data ProMoE-FL is best in only 6/9 configurations and trails FeatImp in 0:4:6, 0:6:4, and 0:8:2 (text-heavy). The abstract should be tightened to match the table (e.g., 'best or competitive under homogeneous; consistently best under heterogeneous').
  2. [§2 Eqs. 4–6; §3 MoE ablation; Fig. 2] The load-bearing transfer claim—that MSE reconstruction on multimodal clients (Eq. 6) conditioned on an accumulated prototype bank and modality-aware MoE (Eq. 4) transfers to unimodal clients from different institutions—is supported mainly by end-task AUC and t-SNE/UMAP. The only MoE ablation is a single 8:0:2 heterogeneous comparison (79.68 vs 78.98). A short ablation isolating accumulation vs averaging of the prototype bank, and at least one more MoE-vs-single-PCD setting (e.g., 4:4:2 or 0:8:2), would make the central design choices more load-bearing rather than confounded with the full pipeline.
minor comments (5)
  1. [§2 Missing Modality Feature Synthesis; Implementation Details] Number of experts E is never stated in the main text (only that E PCDs are used); report E and any sensitivity.
  2. [Eq. (4)] Eq. (4) normalizes the weighted expert sum by its L2 norm; a one-sentence justification relative to unnormalized gating would help.
  3. [Fig. 1] Fig. 1 caption and panel labels are dense; clarify which blocks are frozen vs trainable on unimodal clients.
  4. [§1] Typo: 'Toaddresstheselimitations' (missing spaces) near the end of the introduction.
  5. [§3 Clinical Relevancy] PadChest Spanish reports are a nice cross-lingual stress; a brief note on whether the BERT encoder was multilingual or English-only would help interpret that setting.

Circularity Check

0 steps flagged

No significant circularity: empirical FL method whose AUC gains are measured against external public benchmarks and independent baselines, not forced by definition or self-citation.

full rationale

ProMoE-FL is an engineering/method paper. Its central claims are comparative macro-AUC numbers on held-out MIMIC-CXR splits under nine client partitions (homogeneous and heterogeneous mixes of MIMIC-CXR, Open-I, PadChest, CheXpert). The prototype alignment loss (Eq. 3), the MoE reconstruction objective on multimodal clients (Eq. 6), and the gated synthesis for unimodal clients (Eqs. 4–5) are ordinary training objectives; none of them algebraically equals the reported AUC. Self-citations (CAR-MFL, FeatImp, Med-MMFL) supply baselines, datasets, and related prior work; they do not supply a uniqueness theorem or an ansatz that forces the measured gains. Qualitative t-SNE/UMAP figures are post-hoc visualizations of the same trained features, not tautological rewrites of the inputs. The paper is therefore self-contained against external benchmarks; circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

Load-bearing content is architectural and empirical, not axiomatic physics. Free parameters are standard ML knobs (λ, optimizer, rounds, latent dim). Domain assumptions are the usual FL and multi-label CXR setup. Invented entities are the method modules themselves; they have no evidence outside this paper's experiments.

free parameters (4)
  • prototype loss weight λ
    Explicitly tuned on the grid {0.1, 0.5, 1, 5, 10}; relative weighting of L_proto vs L_task affects prototype quality and thus synthesis.
  • number of MoE experts E
    Architecture width of Φ_MoE; ablation contrasts MoE vs single PCD but main-table E is not fixed in the text, so performance depends on this design choice.
  • latent feature dimension d=256
    Chosen projection size for encoders and prototypes; shapes capacity of synthesis and prototype bank.
  • local epochs N=3, communication rounds=30, lr=1e-4
    Training schedule and Adam step size chosen by authors; affect convergence under non-IID partitions.
axioms (4)
  • domain assumption FedAvg aggregation of encoders, classifier, projection heads, and MoE parameters is sufficient for stable multimodal FL under the studied non-IID partitions.
    Stated in §2 Federated Training; no alternative aggregators tested.
  • ad hoc to paper Class-conditioned client prototypes accumulated into a global bank supply complementary target-modality priors that improve synthesis under client heterogeneity.
    Core design claim in §2 Prototype Construction; accumulation (not averaging) is a paper-specific choice.
  • domain assumption MSE reconstruction of latent features on multimodal clients is an adequate training signal for clinically useful missing-modality features.
    Eq. 6; standard feature-imputation assumption shared with FeatImp-style work.
  • domain assumption Macro AUC on multi-label CXR findings is a valid proxy for method quality under missing modalities.
    Evaluation protocol §3 following CAR-MFL splits.
invented entities (3)
  • client-aware global prototype bank (accumulated, not averaged) no independent evidence
    purpose: Provide federation-wide, client-specific modality priors that the decoder attends to when synthesizing missing features.
    Defined in §2; validated only via this paper's ablations/results, not independent external measurement.
  • Prototype-Conditioned Decoder (PCD) as Transformer decoder expert no independent evidence
    purpose: Let available-modality features attend over target-modality prototypes to reconstruct missing latent vectors.
    Fig. 1(c); architectural invention of the paper.
  • Modality-Aware Router R(h_i, P_j, i, j) for direction-aware MoE gating no independent evidence
    purpose: Route synthesis through shared experts conditioned on instance context and (from,to) modality indices without combinatorial parameter growth.
    Eq. 4 and Fig. 1(d); paper-specific routing design.

pith-pipeline@v1.1.0-grok45 · 14850 in / 3199 out tokens · 47103 ms · 2026-07-11T00:53:40.979468+00:00 · methodology

0 comments
read the original abstract

In this paper, we address the problem of multimodal federated learning with missing modality. Existing methods utilize an additional public dataset or perform naive feature synthesis that is based solely on the available modality. To address these limitations, we propose ProMoE-FL, a Prototype-conditioned Mixture-of-Experts framework for robust missing-modality feature synthesis in multimodal federated learning. ProMoE-FL builds a global client-aware prototype bank that captures clinically meaningful modality priors across institutions. Our Mixture of Experts is conditioned on these prototypes and modality indices to enable direction-aware expert routing for dynamically synthesizing missing features. We perform extensive quantitative and qualitative evaluations on four public chest X-ray datasets (MIMIC-CXR, NIH Open-I, PadChest, and CheXpert) and demonstrate that ProMoE-FL consistently outperforms state-of-the-art methods in both homogeneous as well as the more challenging heterogeneous settings.

Figures

Figures reproduced from arXiv: 2607.06633 by Aavash Chhetri, Bibek Niroula, Binod Bhattarai, Eduard Vazquez, Loris Bazzani, Prashnna Gyawali, Yash Raj Shrestha.

Figure 1
Figure 1. Figure 1: Overview of the proposed ProMoE-FL framework. from multimodal clients can degrade performance under client heterogeneity, where cross-modal relationships vary across sites and available modalities pro￾vide incomplete priors. To mitigate this, we introduce a client-aware global pro￾totype bank encoding federation-wide modality priors, conditioning the synthe￾sis process on target-modality prototypes. Furthe… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative Analysis of the methods. (a–b) t-SNE of learnt features with class centroids. (c) Gating distributions across experts. (d–e) UMAP of rare pathological classes. semantic mapping. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages · 6 internal anchors

  1. [1]

    Nature Medicine28(9), 1773–1784 (2022)

    Acosta, J., Falcone, G., Rajpurkar, P., Topol, E.: Multimodal biomedical ai. Nature Medicine28(9), 1773–1784 (2022)

    work page 2022

  2. [2]

    J Med Internet Res26, e59505 (Sep 2024)

    AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.A., Damseh, R., Sheikh, J.: Multimodal large language models in health care: Applications, challenges, and future outlook. J Med Internet Res26, e59505 (Sep 2024)

    work page 2024

  3. [3]

    In: ICML (2024)

    Bao, G., Zhang, Q., Miao, D., Gong, Z., Hu, L., Liu, K., Liu, Y., Shi, C.: Multi- modal federated learning with missing modality via prototype mask and contrast. In: ICML (2024)

    work page 2024

  4. [4]

    Medical Image Anal- ysis66, 101797 (2020)

    Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Anal- ysis66, 101797 (2020)

    work page 2020

  5. [5]

    Computerized medical imaging and graphics : the official journal of the Com- puterized Medical Imaging Society113, 102342 (2024)

    Chen, J., Pan, R.: Medical report generation based on multimodal federated learn- ing. Computerized medical imaging and graphics : the official journal of the Com- puterized Medical Imaging Society113, 102342 (2024)

    work page 2024

  6. [6]

    Chen, Y., Liu, C., Huang, W., Cheng, S., Arcucci, R., Xiong, Z.: Generative text-guided 3d vision-language pretraining for unified medical image segmentation (2023)

    work page 2023

  7. [7]

    arXiv preprint arXiv:2602.04416 (2026)

    Chhetri, A., Niroula, B., Shrestha, P., Shrestha, Y.R., Anderson, L.A., Gyawali, P.K., Bazzani, L., Bhattarai, B.: Med-mmfl: A multimodal federated learning benchmark in healthcare. arXiv preprint arXiv:2602.04416 (2026)

    work page arXiv 2026

  8. [8]

    In: ICCV (2025)

    Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., He, T.: Unbiased missing-modality mul- timodal learning. In: ICCV (2025)

    work page 2025

  9. [9]

    Journal of the American Medical Informatics Association23(2), 304–310 (2016)

    Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016)

    work page 2016

  10. [10]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 10 A. Chhetri et al

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  11. [11]

    Circulation [Online]101(23), e215–e220 (2000)

    Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P.C., Mark, R., Stan- ley, H.E.: Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation [Online]101(23), e215–e220 (2000)

    work page 2000

  12. [12]

    In: CVPR

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)

    work page 2016

  13. [13]

    In: AAAI (2019)

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI (2019)

    work page 2019

  14. [14]

    Scientific data6(1), 317 (2019)

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data6(1), 317 (2019)

    work page 2019

  15. [15]

    Adam: A Method for Stochastic Optimization

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  16. [16]

    A unified representation network for segmentation with missing modalities

    Lau, K., Adler, J., Sjölund, J.: A unified representation network for segmentation with missing modalities. arXiv preprint arXiv:1908.06683 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  17. [17]

    Information Fusion122, 103219 (2025)

    Le, H.Q., Thwal, C.M., Qiao, Y., Tun, Y.L., Nguyen, M.N., Huh, E.N., Hong, C.S.: Cross-modal prototype based multimodal federated learning under severely missing modality. Information Fusion122, 103219 (2025)

    work page 2025

  18. [18]

    In: CHIL (2024)

    Lee, H., Lee, D.Y., Kim, W., Kim, J.H., Kim, T., Kim, J., Sunwoo, L., Choi, E.: Vision-language generative model for view-specific chest x-ray generation. In: CHIL (2024)

    work page 2024

  19. [19]

    In: CVPR (2021)

    Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: CVPR (2021)

    work page 2021

  20. [20]

    In: AAAI (2021)

    Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: Smil: Multimodal learning with severely missing modality. In: AAAI (2021)

    work page 2021

  21. [21]

    In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)

    McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.: Communication- efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 1273–1282. PMLR (2017)

    work page 2017

  22. [22]

    In: MIUA (2025)

    Poudel, P., Chhetri, A., Gyawali, P., Leontidis, G., Bhattarai, B.: Multimodal federated learning with missing modalities through feature imputation network. In: MIUA (2025)

    work page 2025

  23. [23]

    In: MICCAI (2024)

    Poudel, P., Shrestha, P., Amgain, S., Shrestha, Y.R., Gyawali, P., Bhattarai, B.: CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities . In: MICCAI (2024)

    work page 2024

  24. [24]

    Qayyum, A., Ahmad, K., Ahsan, M.A., Al-Fuqaha, A., Qadir, J.: Collaborative federatedlearningforhealthcare:Multi-modalcovid-19diagnosisattheedge.IEEE Open Journal of the Computer Society3, 172–184 (2022)

    work page 2022

  25. [25]

    SN Computer Science4, 1–12 (2023)

    Sachin, D.N., Annappa, B., Ambasange, S., Tony, A.E.: A multimodal contrastive federated learning for digital healthcare. SN Computer Science4, 1–12 (2023)

    work page 2023

  26. [26]

    Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

    Saha, P., Mishra, D., Wagner, F., Kamnitsas, K., Noble, J.A.: Examining modality incongruityinmultimodalfederatedlearningformedicalvisionandlanguage-based disease detection. arXiv preprint arXiv:2402.05294 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    AAAI39(27), 28331–28339 (Apr 2025)

    Saha, P., Mishra, D., Wagner, F., Kamnitsas, K., Noble, J.A.: Incongruent multi- modal federated learning for medical vision and language-based multi-label disease detection. AAAI39(27), 28331–28339 (Apr 2025)

    work page 2025

  28. [28]

    Seibold, C., Reiß, S., Sarfraz, M.S., Stiefelhagen, R., Kleesiek, J.: Breaking with Fixed Set Pathology Recognition Through Report-Guided Contrastive Training, p. 690–700. Springer Nature Switzerland (2022) ProMoE-FL 11

    work page 2022

  29. [29]

    Medical Vision Language Pretraining: A survey

    Shrestha, P., Amgain, S., Khanal, B., Linte, C., Bhattarai, B.: Medical vision language pretraining: A survey. arXiv preprint arXiv:2312.06224 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    In: ECCV

    Sun, G., Mendieta, M., Dutta, A., Li, X., Chen, C.: Towards multi-modal trans- formers in federated learning. In: ECCV. p. 229–246. Springer-Verlag, Berlin, Hei- delberg (2024)

    work page 2024

  31. [31]

    Multimodal Federated Learning in Healthcare: a Review

    Thrasher, J., Devkota, A., Siwakotai, P., Chivukula, R., Poudel, P., Hu, C., Bhat- tarai, B., Gyawali, P.: Multimodal federated learning in healthcare: A review. arXiv preprint arXiv:2310.09650 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    In: CVPR (2024)

    Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., Carneiro, G.: Multi-modal learning with missing modality via shared-specific feature modelling. In: CVPR (2024)

    work page 2024

  33. [33]

    You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E.K., Baek, W., Roh, B.: CXR- CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training, p. 101–111. Springer Nature Switzerland (2023)

    work page 2023

  34. [34]

    In: ICLR (2023)

    Yu, Q., Liu, Y., Wang, Y., Xu, K., Liu, J.: Multimodal federated learning via contrastive representation ensemble. In: ICLR (2023)

    work page 2023

  35. [35]

    In: Zong, C., Xia, F., Li, W., Navigli, R

    Zhao, J., Li, R., Jin, Q.: Missing modality imagination network for emotion recog- nition with uncertain missing modalities. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) ACL. pp. 2608–2618 (Aug 2021)

    work page 2021