Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

Divyanshu Mishra; Felix Wagner; J. Alison Noble; Konstantinos Kamnitsas; Pramit Saha

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05294 v1 pith:TD7XKPZ6 submitted 2024-02-07 cs.LG cs.AIcs.CLcs.CV

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

Pramit Saha , Divyanshu Mishra , Felix Wagner , Konstantinos Kamnitsas , J. Alison Noble This is my paper

classification cs.LG cs.AIcs.CLcs.CV

keywords modalityclientsincongruitymmflmultimodalunimodalfederatedlearning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

Multimodal Federated Learning (MMFL) utilizes multiple modalities in each client to build a more powerful Federated Learning (FL) model than its unimodal counterpart. However, the impact of missing modality in different clients, also called modality incongruity, has been greatly overlooked. This paper, for the first time, analyses the impact of modality incongruity and reveals its connection with data heterogeneity across participating clients. We particularly inspect whether incongruent MMFL with unimodal and multimodal clients is more beneficial than unimodal FL. Furthermore, we examine three potential routes of addressing this issue. Firstly, we study the effectiveness of various self-attention mechanisms towards incongruity-agnostic information fusion in MMFL. Secondly, we introduce a modality imputation network (MIN) pre-trained in a multimodal client for modality translation in unimodal clients and investigate its potential towards mitigating the missing modality problem. Thirdly, we assess the capability of client-level and server-level regularization techniques towards mitigating modality incongruity effects. Experiments are conducted under several MMFL settings on two publicly available real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology reports.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Deep Multimodal Learning with Missing Modality: A Survey
cs.CV 2024-09 unverdicted novelty 7.0

This survey provides the first comprehensive overview of deep multimodal learning methods designed to remain robust when some input modalities are absent.
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
cs.CV 2026-07 conditional novelty 6.0

Prototype-conditioned Mixture-of-Experts synthesizes missing modalities in federated learning and beats prior methods on heterogeneous chest X-ray clients without public data.