Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read ClusMFL keeps multimodal federated learning accurate when hospitals or patients lack MRI or PET.

desk verdict A sensible prototype-clustering extension for missing-modality federated learning, but the evaluation's fixed test mix undercuts the 'varying incompleteness' claim. read the letter →

arxiv 2502.12180 v1 pith:HWHRADHM submitted 2025-02-14 eess.IV cs.AIcs.CVcs.LG

classification eess.IVcs.AIcs.CVcs.LG
keywords multimodalfederatedlearningmodalityincompletenessbrainimaginganalysisfeatureclusteringsupervisedcontrastivemissingMRIandPETAlzheimer'sdiseaseclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal federated learning usually assumes every participating hospital can supply the same imaging modalities, but real institutions may have only PET, only MRI, or a mixture of complete and incomplete patient records. This paper simulates that two-level incompleteness and proposes ClusMFL, a training procedure that groups each modality's feature embeddings into many small clusters per disease label instead of collapsing them to one class average. The resulting cluster centers pull same-label embeddings together through supervised contrastive learning and serve as stand-ins for a missing modality at classification time, so single-modality sites still contribute to both encoders. On a public Alzheimer's imaging dataset with structural MRI and PET, the authors report that ClusMFL outperforms prototype-based, generative, and standard federated baselines across rising levels of modality missingness, and remains stable where baselines degrade.

What carries the argument

The load-bearing object is the global pool of cluster centers per modality-label pair. Each client runs a parameter-free first-neighbor clustering algorithm (FINCH) on its local feature embeddings for each modality and label, producing multiple cluster centers and sizes, which are uploaded, concatenated server-side, and redistributed as global pools. These centers have two jobs in local training: they act as positive anchors in a supervised contrastive loss that aligns local embeddings with global structure, and as weighted proxies for the absent modality so that a PET-only instance can still be classified through the MRI encoder and vice versa. The final piece is modality-aware aggregation, which assigns each encoder an aggregation weight proportional to the number of instances of that modality across clients, rather than a single uniform weight for the whole model.

What would settle it

Run ClusMFL on a naturally incomplete multimodal clinical cohort where the reasons for missing scans are recorded, or simulate missingness whose probability depends on the diagnosis and disease severity. If the accuracy edge over prototype-based and generative baselines disappears or reverses under such label-correlated missingness, the paper's central claim about realistic settings fails.

Watch

Extended reading notes

Core claim

The central claim is that a cluster-enhanced multimodal federated learning framework can maintain classification accuracy under realistic modality incompleteness that operates at two levels: whole clients missing one modality and individual instances missing one modality. ClusMFL builds a global pool of cluster centers from local feature embeddings for each modality-label pair using a parameter-free clustering algorithm, uses those centers in a supervised contrastive loss for feature alignment, and treats weighted cluster centers as proxy embeddings for the missing modality in a completion loss. A modality-aware aggregation step weights each encoder by the number of modality-specific instances contributed across clients. The authors argue this finer-grained representation beats class-level prototypes and generative reconstruction, with the largest margin in severe incompleteness settings.

Load-bearing premise

The evaluation creates missingness by deleting MRI or PET from originally complete scans, and the reported gains depend on that simulated incompleteness matching real clinical absence; if missingness in practice tracks disease stage, scanner type, or institutional practice, the advantage may shrink.

Editorial extensions

If this is right

  • Institutions missing an entire imaging modality can still contribute to training both modality encoders, because cluster centers from other sites proxy the absent modality during local updates.
  • Model performance stays comparatively stable as the fraction of single-modality instances rises, whereas prototype-based and generative baselines decline.
  • The framework converges in fewer communication rounds than the baselines, reducing the coordination burden of federated training.
  • Because no generative model is used to synthesize missing scans, training avoids adversarial instability and the comparatively high time cost of GAN-based completion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because only cluster centers, sizes, and model parameters are shared, the same cluster-proxy recipe should transfer to other modality pairs, such as CT with pathology images, without changing the privacy posture.
  • Editorial inference: Using several cluster centers rather than one prototype could be exploited at inference time to output a distribution over candidate predictions, and the spread of that distribution might flag low-confidence cases for human review.
  • Editorial inference: The modality-aware aggregation weights could be reused as a lightweight indicator of a client's modality mix at test time, enabling models to be selected or calibrated by expected missingness without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ClusMFL, a cluster-enhanced multimodal federated learning framework for brain imaging under modality incompleteness. The method uses FINCH clustering to build a pool of modalitity-label cluster centers, aligns local embeddings with these centers via supervised contrastive learning, approximates missing modalities with cluster centers in a completion loss, and aggregates encoder modules with modality-aware weights. Experiments on ADNI MRI/PET data with simulated client-level and instance-level missingness compare ClusMFL against FedAvg, FedProx, FedMed-GAN, FedMI, MFCPL, and PmcmFL, reporting consistent improvements across several alpha/beta settings, an ablation study, and convergence curves.

Significance. If the reported results hold, ClusMFL would be a useful practical contribution to healthcare federated learning: it addresses a realistic form of missingness that most prior work oversimplifies, avoids generative models that are unstable under scarce complete data, and leverages a parameter-free clustering algorithm (FINCH) plus a modality-aware aggregation rule. The paper studies a public dataset, reports five metrics, includes an ablation, and compares against several relevant baselines. However, the evaluation protocol has a load-bearing mismatch between training and test incompleteness, the empirical claims lack significance testing, and key implementation details are unreported, so the current evidence does not firmly establish the headline claim.

major comments (3)
  1. [Section IV.A and Table I] The evaluation does not measure performance at the claimed varying levels of test-time modality incompleteness. Section IV.A fixes the test set so that multimodal, PET-only, and MRI-only instances each constitute 1/3 regardless of alpha and beta, while the training distribution follows Eq. (1). For example, with alpha=0.4 and beta=0.4 the training mix is roughly 48% PET-only, 48% MRI-only, and only 4% multimodal, but the test set remains 33% multimodal. Thus Table I varies only the training-side incompleteness, and the conclusion that ClusMFL outperforms baselines 'across varying levels of modality incompleteness' conflates training-time and test-time incompleteness. Please re-evaluate with test sets whose incompleteness distribution matches each alpha/beta setting, or explicitly reframe the claims as being about robustness to training-side incompleteness.
  2. [Section III.E, Section IV.A, and Table I] The central empirical claim is not reproducible because several load-bearing experimental details are missing. The encoder architectures f_P and f_M are never specified, the input representation is only partially described (90 AAL-90 ROI features per modality, without dimensions or preprocessing of those features), and the values of the regularization coefficients lambda1 and lambda2 in Eq. (14) and the temperature tau in the contrastive loss are not reported. Since these coefficients directly control the balance between the three loss terms, their omission makes it impossible to assess sensitivity or to reproduce the reported numbers. Please report all hyperparameters, architectures, and preferably release code.
  3. [Table I and Section IV.B] The claim that ClusMFL 'consistently outperforms' all baselines is not supported by statistical evidence. Many of the reported mean differences are smaller than the standard deviations; for instance, under alpha=0.4, beta=0.2, ClusMFL achieves accuracy 56.37 +/- 4.10 versus FedMI's 54.95 +/- 2.69, and several other comparisons show overlapping intervals. No significance tests or effect sizes are reported for the five-fold results. Please add paired tests across folds (or equivalent) and discuss which differences are statistically meaningful, or soften the claim to the observed mean improvements.
minor comments (6)
  1. [Section IV.A] The sentence 'we first split the dataset into a training set and a test set with a ratio of 1:4' is ambiguous: it is unclear whether this means 1 part training to 4 parts test or the reverse, and the subsequent 5-fold cross-validation is not reconciled with this split. Please clarify the data partitioning.
  2. [Section III.B] There is a typo in the text after Eq. (4): 'cancatenated' should be 'concatenated'.
  3. [Section III.E] In the sentence defining the prediction vector, 'the m instances' should be 'the n_i instances' to match the notation introduced earlier.
  4. [Section III.C and Eq. (6)] The symbol 'direct sum' is used for concatenation of feature embeddings with multiple cluster centers, but the notation is not defined, and the extension of the label vector in Eq. (7) is described verbally rather than with clear set notation. Please define these operations explicitly.
  5. [Table II] The ablation table uses checkmarks to indicate which components are enabled, but the rows are hard to read and the baseline (no checkmarks) row is missing. Consider reformatting with explicit 'yes/no' entries and include the no-component baseline.
  6. [Section IV.D and Fig. 3] The text describes convergence in terms of 'test loss', but the figure panel labels are not described; please state what is plotted in each panel and on which axis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central SOTA claim is an external test-set comparison; no prediction is constructed from its own definition.

full rationale

ClusMFL's central claim is an empirical accuracy/F1/AUC comparison against FedAvg, FedProx, FedMed-GAN, FedMI, MFCPL, and PmcmFL on the ADNI test split (Table I). The model's components—FINCH cluster centers (Eq. 2), supervised contrastive loss (Eq. 8), modality-completion loss (Eqs. 10-12), and modality-aware aggregation (Eqs. 16-18)—are training objectives, not fitted parameters renamed as predictions. The reported metrics are measured on held-out test instances, and the ADNI data and baseline methods are external to the framework. The only related concern, that the test set is fixed at 1/3 multimodal, 1/3 PET-only, and 1/3 MRI-only while training incompleteness varies with alpha and beta (Section IV.A), is an evaluation-protocol mismatch about external validity, not a circularity: varying training availability is not the same as varying test-time incompleteness, but no tested number is forced by construction. No load-bearing self-citation appears; references [18] and [19] are external algorithm and objective citations. Hence the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on the stability of FINCH clusters, the validity of sharing cluster centers, and the faithfulness of synthetic modality dropping; none of these are independently verified in the paper. Hyperparameters lambda1, lambda2, and tau are not reported, so post-hoc tuning cannot be ruled out.

free parameters (4)
  • lambda1 (contrastive regularization coefficient)
    Balances supervised contrastive loss in Eq. 14; value chosen by hand and not reported, so central results depend on unstated tuning.
  • lambda2 (modality completion coefficient)
    Balances completion loss in Eq. 14; not reported.
  • tau (contrastive temperature)
    Temperature in Eq. 8 affects alignment strength; value not reported.
  • encoder architecture hyperparameters
    Number of layers, hidden dimensions, and input feature dimensions are not specified in Section III or IV; needed to reproduce the experiments.
assumptions (4)
  • domain assumption FINCH clustering produces stable, informative cluster centers from local feature embeddings under non-IID federated data.
    Section III-B relies on FINCH to summarize each modality-label distribution; if cluster structure is unstable across rounds, the global pool and contrastive alignment degrade.
  • domain assumption Shared cluster centers do not leak private patient information and are safe to exchange.
    Section III-B transmits local cluster centers and sizes to the server; the paper motivates privacy but gives no differential-privacy or leakage analysis.
  • domain assumption Simulated modality dropping from complete ADNI instances is a faithful proxy for real clinical modality incompleteness.
    Section IV.A assigns instance types and drops modalities synthetically; if real missingness correlates with disease or acquisition, conclusions may not transfer.
  • domain assumption Zero-padding the missing modality embedding before classification is a valid inference strategy.
    Section III.A replaces missing embeddings with zero tensors; no analysis of bias from zeros is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis." pith.science (2026). https://pith.science/paper/HWHRADHM

@misc{pith2026250212180,
  author       = {Pith},
  title        = {Pith review of: ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HWHRADHM}},
  note         = {Machine review of arXiv:2502.12180}
}
read the original abstract

Multimodal Federated Learning (MFL) has emerged as a promising approach for collaboratively training multimodal models across distributed clients, particularly in healthcare domains. In the context of brain imaging analysis, modality incompleteness presents a significant challenge, where some institutions may lack specific imaging modalities (e.g., PET, MRI, or CT) due to privacy concerns, device limitations, or data availability issues. While existing work typically assumes modality completeness or oversimplifies missing-modality scenarios, we simulate a more realistic setting by considering both client-level and instance-level modality incompleteness in this study. Building on this realistic simulation, we propose ClusMFL, a novel MFL framework that leverages feature clustering for cross-institutional brain imaging analysis under modality incompleteness. Specifically, ClusMFL utilizes the FINCH algorithm to construct a pool of cluster centers for the feature embeddings of each modality-label pair, effectively capturing fine-grained data distributions. These cluster centers are then used for feature alignment within each modality through supervised contrastive learning, while also acting as proxies for missing modalities, allowing cross-modal knowledge transfer. Furthermore, ClusMFL employs a modality-aware aggregation strategy, further enhancing the model's performance in scenarios with severe modality incompleteness. We evaluate the proposed framework on the ADNI dataset, utilizing structural MRI and PET scans. Extensive experimental results demonstrate that ClusMFL achieves state-of-the-art performance compared to various baseline methods across varying levels of modality incompleteness, providing a scalable solution for cross-institutional brain imaging analysis.

Figures

Figures reproduced from arXiv: 2502.12180 by the authors.

Figure 1
Figure 1. Illustration of Modality Incompleteness Setting. Dash boxes denote [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of ClusMFL. In this figure, PET-only instances are used as examples of single-modality instances in local training. Different patterns [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training curves of different methods. In our experiment, we set the number of clients to N = 10, with the number of communication rounds fixed at 30 and each client performing 10 local training epochs. For opti￾mization, we employ the Adam [28] optimizer with an initial learning rate of 0.01. To dynamically adjust the learning rate during training, we use cosine annealing strategy. We conduct 5-fold cross-validation… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Online Federated Learning with Modality Missing in Internet of Things

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Introduces MMO-FL, an online federated multimodal learning framework with a prototype-based algorithm, PMM, for compensating missing sensor modalities.

Reference graph

Works this paper leans on

28 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [21]

    Stage-wise training: An improved feature learning strategy for deep models,

    E. Barshan and P. Fieguth, “Stage-wise training: An improved feature learning strategy for deep models,” in Feature extraction: modern questions and challenges . PMLR, 2015, pp. 49–59

  2. [22]

    Graphene and graphene oxide: synthesis, properties, and applications,

    Y . Zhu, S. Murali, W. Cai, X. Li, J. W. Suk, J. R. Potts, and R. S. Ruoff, “Graphene and graphene oxide: synthesis, properties, and applications,” Advanced materials , vol. 22, no. 35, pp. 3906–3924, 2010

  3. [1]

    Multimodal federated learning in healthcare: a review,

    J. Thrasher, A. Devkota, P. Siwakotai, R. Chivukula, P. Poudel, C. Hu, B. Bhattarai, and P. Gyawali, “Multimodal federated learning in healthcare: a review,” arXiv preprint arXiv:2310.09650 , 2023

  4. [2]

    Multimodal federated learning: A survey,

    L. Che, J. Wang, Y . Zhou, and F. Ma, “Multimodal federated learning: A survey,” Sensors, vol. 23, no. 15, p. 6986, 2023

  5. [3]

    A survey of multimodal federated learning: background, applications, and perspectives,

    H. Pan, X. Zhao, L. He, Y . Shi, and X. Lin, “A survey of multimodal federated learning: background, applications, and perspectives,” Mul- timedia Systems, vol. 30, no. 4, p. 222, 2024

  6. [4]

    A multimodal federated learning framework for modality incomplete scenarios in healthcare,

    Y . An, Y . Bai, Y . Liu, L. Guo, and X. Chen, “A multimodal federated learning framework for modality incomplete scenarios in healthcare,” in International Symposium on Bioinformatics Research and Applica- tions. Springer, 2024, pp. 245–256

  7. [5]

    Cross-modal prototype based multimodal federated learning under severely missing modality,

    H. Q. Le, C. M. Thwal, Y . Qiao, Y . L. Tun, M. N. Nguyen, and C. S. Hong, “Cross-modal prototype based multimodal federated learning under severely missing modality,” arXiv preprint arXiv:2401.13898 , 2024

  8. [6]

    Multimodal federated learning with missing modality via prototype mask and contrast,

    G. Bao, Q. Zhang, D. Miao, Z. Gong, L. Hu, K. Liu, Y . Liu, and C. Shi, “Multimodal federated learning with missing modality via prototype mask and contrast,” arXiv preprint arXiv:2312.13508 , 2023

Show all 28 references
  1. [7]

    Meta-baseline: Exploring simple meta-learning for few-shot learning,

    Y . Chen, Z. Liu, H. Xu, T. Darrell, and X. Wang, “Meta-baseline: Exploring simple meta-learning for few-shot learning,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 9062–9071

  2. [8]

    Free lunch for few-shot learning: Distribution calibration,

    S. Yang, L. Liu, and M. Xu, “Free lunch for few-shot learning: Distribution calibration,” in International Conference on Learning Representations, 2021

  3. [9]

    Epl: Empirical prototype learning for deep face recognition,

    W. Fan, J. Wen, X. Jia, L. Shen, J. Zhou, and Q. Li, “Epl: Empirical prototype learning for deep face recognition,” 2024. [Online]. Available: https://arxiv.org/abs/2405.12447

  4. [10]

    Fedmed-gan: Federated domain translation on unsupervised cross- modality brain image synthesis,

    J. Wang, G. Xie, Y . Huang, J. Lyu, F. Zheng, Y . Zheng, and Y . Jin, “Fedmed-gan: Federated domain translation on unsupervised cross- modality brain image synthesis,” Neurocomputing, vol. 546, p. 126282, 2023

  5. [11]

    Client-adaptive cross-model reconstruction network for modality-incomplete multi- modal federated learning,

    B. Xiong, X. Yang, Y . Song, Y . Wang, and C. Xu, “Client-adaptive cross-model reconstruction network for modality-incomplete multi- modal federated learning,” in Proceedings of the 31st ACM Interna- tional Conference on Multimedia , 2023, pp. 1241–1249

  6. [12]

    Deep multi- modal learning with missing modality: A survey,

    R. Wu, H. Wang, H.-T. Chen, and G. Carneiro, “Deep multi- modal learning with missing modality: A survey,” arXiv preprint arXiv:2409.07825, 2024

  7. [13]

    Generative adversarial net- works,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial net- works,” Communications of the ACM , vol. 63, no. 11, pp. 139–144, 2020

  8. [14]

    Ht-fed-gan: Federated generative model for decentralized tabular data synthesis,

    S. Duan, C. Liu, P. Han, X. Jin, X. Zhang, T. He, H. Pan, and X. Xiang, “Ht-fed-gan: Federated generative model for decentralized tabular data synthesis,” Entropy, vol. 25, no. 1, p. 88, 2022

  9. [15]

    Fligan: Enhancing federated learning with incomplete data using gan,

    P. J. Maliakel, S. Ilager, and I. Brandic, “Fligan: Enhancing federated learning with incomplete data using gan,” 2024. [Online]. Available: https://arxiv.org/abs/2403.16930

  10. [16]

    Fedmac: Tackling partial-modality missing in federated learning with cross-modal aggregation and contrastive regularization,

    M. D. Nguyen, T. T. Nguyen, H. H. Pham, T. N. Hoang, P. L. Nguyen, and T. T. Huynh, “Fedmac: Tackling partial-modality missing in federated learning with cross-modal aggregation and contrastive regularization,” 2024. [Online]. Available: https://arxiv.org/abs/2410.03070

  11. [17]

    Adaptive hyper-graph aggregation for modality- agnostic federated learning,

    Q. Fan and L. Shuai, “Adaptive hyper-graph aggregation for modality- agnostic federated learning,” in 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2024, pp. 12 312– 12 321

  12. [18]

    Efficient parameter- free clustering using first neighbor relations,

    S. Sarfraz, V . Sharma, and R. Stiefelhagen, “Efficient parameter- free clustering using first neighbor relations,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 8934–8943

  13. [19]

    Supervised contrastive learning,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” 2021. [Online]. Available: https://arxiv.org/abs/2004.11362

  14. [20]

    The alzheimer’s disease neuroimaging initiative,

    S. G. Mueller, M. W. Weiner, L. J. Thal, R. C. Petersen, C. Jack, W. Jagust, J. Q. Trojanowski, A. W. Toga, and L. Beckett, “The alzheimer’s disease neuroimaging initiative,” Neuroimaging Clinics , vol. 15, no. 4, pp. 869–877, 2005

  15. [23]

    V oxel-based morphometry—the methods,

    J. Ashburner and K. J. Friston, “V oxel-based morphometry—the methods,” Neuroimage, vol. 11, no. 6, pp. 805–821, 2000

  16. [24]

    Automated anatom- ical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain,

    N. Tzourio-Mazoyer, B. Landeau, D. Papathanassiou, F. Crivello, O. Etard, N. Delcroix, B. Mazoyer, and M. Joliot, “Automated anatom- ical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain,” Neuroimage, vol. 15, no. 1...

  17. [25]

    Communication-efficient learning of deep networks from decentral- ized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentral- ized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273–1282

  18. [26]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems , vol. 2, pp. 429–450, 2020

  19. [27]

    Unpaired image- to-image translation using cycle-consistent adversarial networks,

    J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image- to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2223–2232

  20. [28]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.