{"id":"8d3ff767-6b12-48df-b8b4-fc42671b7308","arxiv_id":"2607.13386","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"FM² uses dual mixture-of-experts (per-class local, per-modality shared) with a proximal alignment regularizer to train federated medical imaging models across overlapped and disjoint modality settings, reporting consistent gains over FedAvg-style baselines.","lead":"Federated learning lets hospitals train models without sharing patient images, but it struggles when different hospitals have different imaging types and label mixes. This paper proposes FM², which splits the model into per-class and per-modality experts and adds a text-caption signal, reporting large gains over standard federated baselines across classification, caption learning, and visual question answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 2's advertised generalization guarantee is not derived: the TV/JS step bounds divergence between the two experts' output laws, not either expert's excess risk. The central 'provable guarantees' claim is therefore unsupported.","rationale":"The reader's weakest_assumption is CEL label leakage; that is a real confound and worth testing. But the single most load-bearing issue for the stated central claim is the advertised theoretical guarantee: Proposition 2's proof is internally inconsistent, not merely outside consensus. The bound's RHS is a function of divergence between expert output laws, yet the argument needs bounds on each expert's distance to Bayes optimality; no step supplies that. The convergence theorem has a related unresolved issue: Eq. (3) defines \\hat M_d as an argmin FedAvg aggregate, and the proof's gradient of the alignment penalty vanishes in aggregate if \\hat M_d is the current client average, so the claimed alignment-drift term is not established either. These are analytical, checkable failures rather than interpretive disagreements. The CEL issue could be settled by a caption-ablation experiment, but even if CEL were clean, the paper would still advertise a guarantee it does not prove. Hence the reader's REJECT verdict stands; I would not alter it. I mark partial agreement because the reader's rationale included the theory, even though their stated weakest assumption was CEL.","tokens_in":21209,"tokens_out":12231,"duration_ms":113086,"concrete_test":"Re-derive Proposition 2 under a minimal counterexample: binary labels y = 1_{X∈A}, squared loss, and f_d = f_c ≡ 0.5 with V the identity. Then d_JS(P_d∥P_c) = 0; if R*_d, R*_c are Bayes-optimal risks, the right-hand side of Eq. (10) is γ_d R*_d + γ_c R*_c, while the actual excess risk is 0.25, so the inequality is false. If the authors intend restricted expert classes, specify them and supply the missing derivation of the TV/JS step; without that, Eq. (10) cannot be used as a guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and contributions advertise 'provable O(1/√T) convergence and generalization guarantees (Theorem 1–Proposition 2)'. Proposition 2 is not a proof. Starting from h = γ_d f_d + γ_c f_c (Eq. 2), Jensen gives R(h) ≤ γ_d E[ℓ(f_d,y)] + γ_c E[ℓ(f_c,y)]. The proof then asserts that each excess risk satisfies |E[ℓ(f_d,y)] − R*_d| ≤ d_TV(P_{f_d}, P_{f*_d}). Even if this held, the next step replaces these two individual distances to the Bayes predictors by d_TV(P_d, P_c), the distance between the two experts' output laws. There is no derivation connecting distance-to-Bayes to distance-between-experts; the claimed 'cross-term ... by Cauchy–Schwarz on the weighted excess risks' is not shown and does not follow. The final ε_V term is defined as inf_{V'} E||V(X)−V'(X)||², a label-free reconstruction error, so it cannot represent the composed predictor's approximation error. Thus Eq. (10) is unsupported: with squared loss and deterministic labels, taking f_d = f_c ≡ 0.5 gives d_JS = 0 yet excess risk 0.25, directly contradicting the bound. Since 'provable guarantees' is an advertised contribution, this is load-bearing. The CEL label-leakage issue in Sec. 3.4 is an additional confound but not the point I would rest the rejection on.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FM², a federated learning framework for medical imaging under 'imaging modality heterogeneity,' covering overlapped clients (shared modalities, non-IID labels) and non-overlapped clients (disjoint modalities). The architecture couples a locally retained Class-wise MoE with a globally aggregated Domain-wise MoE, and adds a Heterogeneous Modality Alignment regularizer with claimed O(1/√T) convergence and generalization guarantees. It also introduces Caption-Enhanced Learning using locally retained GPT-4o captions as a cross-modality textual bridge, and extends the same backbone to federated medical VQA. A new MIMH benchmark (five MedMNIST modalities, four configurations) is used for classification and CEL, and three VQA task settings are evaluated on SLAKE/VQA-RAD/VQA-Med. The empirical results show large and consistent margins over FedAvg/FedProx/FedProto/FedRep and specialized VQA baselines.","tokens_in":21623,"tokens_out":17230,"duration_ms":160226,"significance":"If the results hold, FM² would be a useful step toward federated multimodal medical foundation models: the dual MoE decomposition is a plausible way to separate class-specific personalization from modality-shared representations, and the MIMH benchmark plus the VQA evaluation provide a broad testbed. The experiments are extensive—multiple non-IID severities, participation ratios, ablations, hyperparameter sensitivity, communication cost, and convergence curves—and the classification margins over strong FL baselines are substantial. The CEL and VQA extensions broaden the framework's scope. However, two load-bearing pieces are not currently established: the advertised generalization guarantee (Proposition 2) is invalid as written, and the CEL evaluation is confounded by class labels appearing in the GPT-4o captions. The proof of Theorem 1 also does not match the executed federated update. The paper's empirical contribution is therefore stronger than its theoretical and CEL-specific claims.","major_comments":[{"comment":"Proposition 2's generalization guarantee is not derived. After Jensen, the proof asserts each expert's excess risk is controlled by d_TV to its Bayes-optimal law, then replaces the two individual distances by d_JS(P_d, P_c) with no connecting step. The final ε_V = inf_{V'} E||V(X)-V'(X)||^2 is an Observer reconstruction error, not an approximation error of h, and is not linked to the risk. The bound is false as stated: with squared loss and deterministic labels, f_d = f_c ≡ 0.5 gives h = 0.5, d_JS = 0, R*_d = R*_c = ε_V = 0, yet excess risk 0.25, contradicting Eq. (10). Eq. (10) has no sample-size term, so it is not a finite-sample generalization bound. Since 'provable ... generalization guarantees' is an advertised contribution (abstract; Sec. 1 item 2), this must be corrected or removed.","section":"Sec. 3.5, Proposition 2 / Eq. (10)"},{"comment":"The CEL evaluation is confounded by label leakage through the GPT-4o captions. The example in Sec. 3.4 — 'consistent with moderate non-proliferative diabetic retinopathy' — states the class label explicitly. Because Eq. (4) aligns visual features with such caption embeddings, the reported gains over image-only baselines (e.g., FM2-Large cross-modality average 95.04 vs. 90.48 in Table 3) may reflect access to class names rather than a modality-invariant 'textual semantic bridge.' No control removing diagnostic tokens (e.g., morphology-only paraphrases) is reported, and the 'Template' condition in Fig. 3(c) is not described enough to know whether it also contains labels. This is load-bearing for the CEL contribution and the out-of-modality generalization claim; it should be tested explicitly.","section":"Sec. 3.4, Eq. (4); Tables 3 and Fig. 3(c-d)"},{"comment":"The proof of Theorem 1 analyzes an aggregated update θ^{t+1}=θ^t−ηḡ_t in which ḡ_t contains the alignment term 2λ(M_i^d−M̂_d). In the actual FedAvg procedure of Sec. 3.3, the server aggregates local DMoE updates with weights p_i, and Σ_i p_i (M_i^d−M̂_d)=0, so the alignment term cancels in the aggregate; the global update is only the averaged data gradient. Thus the 'align. drift' term 2λLΔM/T in Eq. (6) does not follow from the executed dynamics, and the convergence claim for FM²'s federated update is not established. The proof also assumes a single local step and full participation, whereas experiments use 1 local epoch and join ratios r=30/50/100%. Clarify or re-derive.","section":"Sec. 3.5, Theorem 1 proof / Eq. (6)"}],"minor_comments":[{"comment":"Appendices A–D are referenced in Secs. 3.1–3.2 and 4.1–4.2 but are not included in the manuscript. Appendix D (full CEL prompts and quality-filtering criteria) is essential for assessing the label-leakage concern.","section":"General"},{"comment":"'ResNet-4 [22]' is not a standard architecture and is not defined; please specify the exact network used.","section":"Sec. 4.1"},{"comment":"The CEL experiments upgrade to CLIP ViT-B/16, a natural-image pretrained model. This appears to conflict with the paper's motivation to train from scratch for medical-domain fidelity; clarify how this fits with the framework described in Sec. 3.","section":"Sec. 4.2"},{"comment":"The 'Template' and 'LLM Expert' conditions are not precisely defined, and 'expert annotations' is ambiguous because no set of human expert captions was introduced. Also specify whether the held-out datasets in Table 3's Cross-Mod. column receive GPT-4o captions during 5% fine-tuning.","section":"Fig. 3(c-d)"},{"comment":"The proof text says the second term from the alignment drift recursion 'enters as O(Lηδ²)', while the displayed bound contains 4Lηδ²; the constants should be reconciled.","section":"Sec. 3.5 / Eq. (9)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical component, but the theory section and CEL evaluation both need substantial repair. In my view, Proposition 2 is the more serious issue: it is not merely a gap but a false bound, and the abstract advertises it. If the authors cannot supply a corrected statement or remove the guarantee claim, I would not support acceptance. The CEL label-leakage concern is addressable with a redaction control, but it must be run."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: FM2 is a solid engineering contribution to federated medical classification across modality-heterogeneous clients, and the classification-only results are worth taking seriously. But the two headline selling points — the generalization bound and the caption-based bridge — don't hold up as written. I'd send it to peer review rather than desk reject, but with a strong expectation of major revision.\n\nWhat's new and good: The dual MoE decomposition (class-wise local, domain-wise global) with the HMA proximal regularizer is a real combination, and the MIMH benchmark (5 modalities, 4 configs) is a useful construction. The classification experiments are extensive: large, consistent margins over FedAvg/FedProx/FedProto/FedRep, sensible ablations, and the two-stage observer/top-K sampling is a reasonable idea. The VQA extension shows modest but consistent gains. The paper is well-organized and the empirical work looks carefully done.\n\nSoft spots, in order of severity:\n\n1. Proposition 2 is not a proof and, as stated, is false. The proof never derives the step from each expert's distance-to-Bayes to the JS divergence between the two experts' output laws; the 'Cauchy–Schwarz on weighted excess risks' is asserted. The stress-test counterexample is convincing: with squared loss and deterministic labels, f_d=f_c≡0.5 gives d_JS=0 and excess risk 0.25, contradicting Eq. (10). The abstract and contributions advertise 'provable generalization guarantees,' so this is load-bearing. Theorem 1's convergence proof is more plausible, but the ΔM recursion is hand-wavy.\n\n2. CEL is confounded. The example caption in Section 3.4 states the diagnosis ('consistent with moderate non-proliferative diabetic retinopathy'), and the MIMH labels are exactly those diagnoses. Aligning visual features to such captions via Eq. (4) is effectively training with label-derived text. The gains in Tables 3 and 5 therefore cannot cleanly support the 'textual semantic bridge / out-of-modality generalization' claim. The paper never tests this.\n\n3. Missing comparisons: OmniFM is cited as the VQA setup source but not included in Tables 4–6. No code/data released. These are minor-to-moderate, but they weaken the claims.\n\nWho this is for: the classification recipe and MIMH benchmark will be useful to people working in federated medical imaging. The theory and CEL sections should be read skeptically.\n\nRecommendation: send to serious peer review. The authors should be asked to either prove a correct generalization bound (or remove the claim) and to re-run CEL with captions that do not contain diagnosis words, plus add OmniFM and release code/data.","headline":"A genuinely useful federated medical classification system, but the advertised generalization guarantee is false and the caption learning is likely label leakage — still worth a careful referee, not a desk reject.","tokens_in":22118,"tokens_out":5592,"would_cite":false,"duration_ms":62123,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a federated framework with dual mixture-of-experts and caption-enhanced learning lets hospitals with disjoint imaging modalities jointly train a medical foundation model, outperforming existing federated methods on cl","keywords":["Federated Learning","Medical Imaging","Data Heterogeneity","Foundation Models","Mixture-of-Experts","Medical VQA","Modality Heterogeneity","Caption-Enhanced Learning"],"falsifier":"Generate a second set of captions that describe only morphology and visual findings, with all diagnosis-bearing words removed, then rerun the MIMH-SA and cross-modality CEL experiments; if the accuracy gains over image-only baselines disappear or shrink sharply, the CEL contribution is explained by label-derived text rather than by a shared semantic space.","tokens_in":21042,"feed_emoji":"🏥","tokens_out":3887,"duration_ms":37111,"temperature":0.7,"pith_summary":"The paper tries to establish that imaging-modality heterogeneity—hospitals operating in overlapping versus disjoint modality regimes—can be handled by one unified federated framework. It proposes FM2, which trains a visual backbone from scratch, pairs each client with a personalized class-wise expert and a globally shared domain-wise expert, and regularizes the domain experts toward consensus. The claim is that this disentanglement, plus locally retained language captions, yields a single system that consistently beats federated baselines on three tasks and transfers to unseen modalities. If true, it would let medical institutions with different scanner types collaboratively build a multimodal foundation model without sharing raw images or labels.","feed_headline":"One federated model spans five medical imaging modalities","feed_subtitle":"Dual expert modules and caption learning let hospitals train together without sharing patient data.","key_machinery":"The central object is the dual MoE decomposition: a Class-wise MoE with per-class experts that stay on the client, and a Domain-wise MoE with per-modality experts that are federated and globally aggregated. The Heterogeneous Modality Alignment penalty λ||M_d^i - Mhat_d||² anchors the domain experts to the global consensus, and the Caption-Enhanced Learning loss aligns visual features with caption text embeddings via a contrastive objective. Together they carry the argument: the local expert handles label heterogeneity, the global expert handles modality heterogeneity, and text acts as a bridge across disjoint visual domains.","core_discovery":"The central discovery is that the two kinds of heterogeneity in federated medical imaging—label skew within a shared modality and completely disjoint modalities across clients—can be addressed by a dual-expert decomposition. FM2 splits each client's model into a Class-wise Mixture-of-Experts (one expert per class, kept local) and a Domain-wise Mixture-of-Experts (one expert per imaging modality, globally aggregated), and adds a Heterogeneous Modality Alignment regularizer that pulls the domain experts toward a global FedAvg consensus. The paper proves an O(1/sqrt(T)) convergence rate and a generalization bound for this objective. To transfer knowledge when no visual modality is shared, it ad","pith_inferences":["The headline cross-modal gains, especially in the non-overlapped setting, may stem in part from label leakage: diagnostic captions that explicitly state the diagnosis could make the contrastive objective a softer form of label supervision rather than a pure modality-invariant bridge.","A direct test of this would be to generate morphology-only captions that never name the diagnosis and rerun the MIMH-SA experiments; if the gains vanish, the semantic-bridge story is weakened.","The dual MoE recipe—keep task-specific experts local, share domain experts globally, regularize only the shared part—may generalize beyond imaging to other federated modalities such as time series or sensor data.","The Observer Top-K sampling step is under-analyzed: in extreme non-IID settings it might oversample the most confident examples and bias expert training; comparing it against random sampling at the same budget would clarify its role."],"forward_implications":["Institutions with entirely disjoint imaging equipment can jointly train a shared visual backbone while each site retains its own class-level personalization.","The HMA regularization gives a provable O(1/sqrt(T)) convergence rate and an explicit generalization bound, independent of how heterogeneous the modalities are.","With Caption-Enhanced Learning, even clients that share no visual modality can transfer knowledge through a common textual vocabulary, improving all federated baselines tested.","The same MoE-enhanced visual features extend to medical visual question answering without task-specific re-engineering.","A model trained on five modalities generalizes to held-out modalities such as chest X-ray, OCT, and abdominal CT with only 5% fine-tuning data."],"fun_headline_variants":["Federated model handles both shared and disjoint medical scans","Dual experts let hospitals train AI on any mix of imaging modalities","Caption bridge transfers knowledge between clients with zero shared modalities","Federated framework learns from heterogeneous imaging with privacy","Hospital collaboration without data sharing: one model for all imaging types"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework's cross-modality transfer rests on the assumption that GPT-4o-generated captions describe image content without embedding the class label; the example caption in Section 3.4 explicitly states the diagnosis (\"consistent with moderate non-proliferative diabetic retinopathy\"), so this assumption is untested and may be violated.","fun_headline_variants_meta":{"raw":{"variants":["Federated model handles both shared and disjoint medical scans","Dual experts let hospitals train AI on any mix of imaging modalities","Caption bridge transfers knowledge between clients with zero shared modalities","Federated framework learns from heterogeneous imaging with privacy","Hospital collaboration without data sharing: one model for all imaging types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2792,"prompt_tokens":800,"completion_tokens":1992,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1910}},"tokens_in":544,"tokens_out":1992,"duration_ms":14806,"temperature":1.0,"reasoning_tokens":1910,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:19:46.191999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a second set of captions that describe only morphology and visual findings, with all diagnosis-bearing words removed, then rerun the MIMH-SA and cross-modality CEL experiments; if the accuracy gains over image-only baselines disappear or shrink sharply, the CEL contribution is explained by label-derived text rather than by a shared semantic space.","supporting_citations":[],"review_version":1}