{"id":"af525713-9094-444e-a76f-6d46cf0f7cb5","arxiv_id":"2505.06911","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MMiC combines one-layer parameter substitution, Banzhaf-inspired client selection, and Markowitz-inspired global aggregation to report top global and personalized results on multiple multimodal federated learning benchmarks with missing modalities.","lead":"MMiC is a clustered federated learning framework that handles clients with missing data modalities by replacing a model layer with cluster-mates' parameters, biasing client selection toward high-contribution clients, and down-weighting risky clusters during global aggregation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage in BPI/MPO: if a_{i,t} is computed on the visible test set, the central outperformance claim is invalid.","rationale":"The paper's contribution is an engineering framework whose two adaptive mechanisms (BPI and MPO) are driven by client return signals. The manuscript provides code, which is positive, but the experimental setup contains a sentence that directly threatens the validity of the central claim: test sets are accessible to clients and server. Since the paper never states the split on which a_{i,t} is computed, the most natural reading, given the immediately preceding sentence, is that local model performance is measured on that test set. Under that reading, Eq. 9's core-member test and Eq. 11's selection probabilities, as well as Eq. 12's covariance and Eq. 15's dynamic beta, are optimized against the evaluation labels. This would make Table 1 a comparison between a method that tunes its selection and aggregation to the test metric and baselines that do not. The concern is load-bearing because the ablation shows removing the modules changes results (Table 2), and the same a_{i,t} signal is used in both modules. The appropriate remedy is not to delete the modules but to fix the data source: a_{i,t} must be computed on a validation split disjoint from the final test set, or the claim must be restricted to settings where such feedback is legitimately available. Secondary issues (the suspicious Eq. 6, Banzhaf/Markowitz naming, Table 1 vs Table 2 metric mismatch, missing error bars in Table 3 and Fig. 2) are real but secondary; they do not change the primary disposition. I therefore keep the reader's conditional verdict: the paper is acceptable only if the provenance of a_{i,t} is clarified and the experiments are re-run without test-set feedback.","tokens_in":20433,"tokens_out":5896,"duration_ms":59705,"concrete_test":"Inspect the released code at https://github.com/gotobcn8/MMiC and trace the computation of a_{i,t} (Eq. 7), cluster return A (Eq. 8), and MPO covariance (Eq. 12): identify the exact dataset split used in the performance evaluation feeding these quantities. If that split is the test set, or overlaps the split used to produce Table 1, the concern is confirmed. Then rerun the Table 1 protocol with a_{i,t} computed on a held-out validation split that is disjoint from the final test set; if MMiC no longer secures top positions across all settings, the claimed consistent outperformance depends on test-set leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MMiC consistently outperforms baselines in global and personalized performance under missing modalities. The mechanism leans on per-round client performance a_{i,t} (Eq. 7) and its history: core-member identification (Eq. 9), selection probability (Eq. 11), cluster covariance (Eq. 12), and RARC-based aggregation weights (Eqs. 13-15). Section 5.1.3 states that 'Each cluster had a test set visible to all clients within the cluster. Additionally, the global server provided accessibility to the test set for all clients.' The paper never says what data a_{i,t} is computed on. If a_{i,t} is evaluated on that test set, then BPI client selection and MPO cluster weighting are functions of the exact labels and metric used for the final evaluation. The comparison in Table 1 would then be circular: MMiC is given access to test-set outcomes during training that baselines do not use, so the reported margins do not establish a fair superiority. The concern is load-bearing because removing BPI/MPO (ablation 'w/o b,p,s', Table 2) drops performance substantially; both modules depend on a_{i,t}. This is not resolved by the ablation itself, since every MMiC variant in Table 1 uses the same test-visible returns. A statement of the evaluation split for a_{i,t} and confirmation that it is disjoint from the final test set is required before the headline claim can be assessed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMiC, a clustered federated learning framework for multimodal data with missing modalities. MMiC combines three modules: poverty parameter substitution (PPS), which replaces the most affected layer of a client model with a weighted combination of complete-modality clients' parameters; a Banzhaf Power Index (BPI) based client selection mechanism; and a Markowitz Portfolio Optimization (MPO) based dynamic global aggregation scheme. The authors evaluate MMiC and two variants on CrisisMMD, UPMC-Food101, Flickr30k, and two HAR datasets, reporting consistent improvements over FedAvg, FedOpt, FedSoft, PACFL, CreamFL, and PmcmFL in both global and personalized performance.","tokens_in":20806,"tokens_out":6744,"duration_ms":70643,"significance":"If the experimental evaluation is free of data leakage, the paper addresses a relevant and under-explored problem: missing modalities in clustered multimodal federated learning. The proposed modules are model-independent and modality-agnostic in principle, the code is released, and the experimental coverage spans classification, retrieval, and sensor-based activity recognition. However, the central outperformance claim depends on an unresolved question about what data the per-round performance signal a_{i,t} is computed on, and there are additional reproducibility gaps in the method description. The contribution is therefore potentially useful but currently not fully assessed.","major_comments":[{"comment":"The paper never states what data the per-client per-round performance a_{i,t} in Eq. (7) is computed on. Section 5.1.3 says 'Each cluster had a test set visible to all clients within the cluster. Additionally, the global server provided accessibility to the test set for all clients.' If a_{i,t} is evaluated on this visible test set, then BPI core-member identification (Eqs. 9-11) and MPO aggregation weights (Eqs. 12-15) are functions of the exact labels and metric used for the final evaluation in Table 1. Baselines do not use such per-round outcomes in their selection or aggregation procedures, so the reported comparison would be circular and the headline outperformance claim would be invalid. Please state the exact evaluation split used to compute a_{i,t}, confirm that it is disjoint from the final test set, and explain how all baselines are given the same information.","section":"Section 5.1.3 and Eqs. (7)-(15)"},{"comment":"The core-member identification condition in Eq. (9) uses a threshold \\alpha_m^t that is not defined anywhere in the paper. The authors define A_{S_m,t} and A_{S'_m,t}, but the threshold \\alpha_m^t is introduced without explanation, making the BPI selection step not reproducible as written.","section":"Section 4.2, Eq. (9)"},{"comment":"Equation (6) is presented as an Adam-based calculation of parameter variation, but it does not follow from the Adam update rule. For an Adam step, \\theta_b = \\theta_{b-1} - \\eta \\hat{m}_b / (\\sqrt{\\hat{\\nu}_b} + \\epsilon), so the relative parameter change involves \\theta_{b-1} in the denominator, not \\theta_b as written in Eq. (6). Since the poverty-layer identification in Eqs. (2)-(4) depends on this quantity, the equation should be corrected or explicitly justified as a rough approximation, and the text should state whether saved parameters or the approximation is used in the experiments.","section":"Section 4.1, Eq. (6)"},{"comment":"The main results and the ablation results are inconsistent. For Flickr30k under (mm@0.2, mc@0.5), Table 1 reports MMiC global RSum 8.322 and personalized RSum 75.814, whereas Table 2 reports an MMiC RSum of 11.02 for the same setting. In addition, CrisisMMD is evaluated with F1 in Table 1 but with accuracy in Table 2. These discrepancies need to be reconciled, because the ablation conclusions about the contribution of each module are not fully supported by the reported numbers.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The claim that MMiC is modality-agnostic is supported only by comparing MMiC with FedOpt on two sensor datasets; a comparison with the full baseline set would make the generalization claim stronger.","section":"Section 5.4"},{"comment":"The covariance formula in Eq. (12) reuses t both as the current global round and as the summation index, and \\bar{\\alpha}_{i,t-1} is described as the historical average return for round t-1 without a clear definition of the averaging window. Please rewrite with distinct indices and define all quantities precisely.","section":"Eq. (12)"},{"comment":"The missing-modality simulation is described ambiguously: clients are assigned a fixed number of missing samples by the mm rate, but the mc rate is applied per round to selected clients. The precise relationship between the global mc/mm parameters and the actual per-client missing status in each round should be stated.","section":"Section 5.1.4"},{"comment":"The time-complexity derivation would benefit from defining M and K explicitly at the start and clarifying whether prefix sums are maintained per cluster or globally; as written, the index conventions are easy to misread.","section":"Section 5.8"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the test-set visibility statement in Section 5.1.3. If the authors can confirm that a_{i,t} is computed on a disjoint validation split and that the final test set is used only for reporting, the comparison may be valid; otherwise the headline claim is not supported. I would be willing to re-evaluate a revision that resolves this point and reconciles the table inconsistencies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper proposes a genuinely modular way to handle missing modalities in clustered federated learning, and the authors evaluate it broadly. The problem is that the evaluation has a leak-shaped hole in it, and until that is closed the headline \"consistently outperforms\" cannot be taken at face value.\n\nWhat's new: the combination of cluster-level layer substitution (PPS), a per-round contribution score for client selection, and a return-risk term for cluster weighting. The layer substitution idea is simple and model-agnostic, and the paper does real work on three datasets, multiple missing-modality rates, ablations, extra sensor-modality experiments, and communication-round comparisons. The code link is a plus.\n\nThe leak: Section 5.1.3 says every cluster has a test set visible to all clients in the cluster, and the server also gives all clients access to the test set. Equation 7 defines the per-round client return a_{i,t} as the difference in local model performance between rounds. The paper never says what data a_{i,t} is computed on. If it is computed on that visible test set, then BPI selection and MPO weighting are being optimized on the exact labels and metric used for the final evaluation. The ablation shows that removing BPI and MPO ('w/o b,p,s') costs a lot, so these modules are load-bearing; you can't wave the leak away. This is the first thing I would ask the authors.\n\nSecond-order issues: the 'Banzhaf' client selection is actually a thresholded leave-one-out rule (Eq. 9), not a Banzhaf power index, and 'Markowitz' aggregation is a covariance-weighted heuristic, not portfolio optimization. Those names overclaim. Eq. 6 says it derives from Adam but doesn't follow from the Adam update. The ablation in Table 2 uses accuracy for CrisisMMD while Table 1 uses F1, so the two tables aren't comparable. Table 3 shows no error bars and only compares against FedOpt.\n\nIf the a_{i,t} problem is resolved with a proper validation split, this is a solid incremental contribution to FL and worth citing. As written, the comparison may be circular. I would still send it to peer review — a good referee can pin down the data split, and the framework itself deserves a fair test. But my own verdict would be conditional at best.","headline":"Plausible modular framework for missing modalities in clustered FL, but the reported gains are uninterpretable until the authors disclose whether the per-round returns that drive BPI and MPO were computed on the test set.","tokens_in":21289,"tokens_out":3524,"would_cite":false,"duration_ms":38294,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMiC claims a three-part fix for missing modalities in clustered federated learning and reports top global and personalized results on three multimodal benchmarks.","keywords":["federated learning","multimodal federated learning","missing modalities","clustered federated learning","client selection","Banzhaf Power Index","Markowitz portfolio optimization","personalized federated learning"],"falsifier":"Inspect the released code for the data source behind $a_{i,t}$ in Eq. (7). If $a_{i,t}$ is the accuracy, F1, or RSum computed on the cluster test set visible to all clients or on the global test set, rerun the comparison with $a_{i,t}$ computed only from local training batches or a held-out per-client validation split; the central claim collapses if MMiC's edge over FedAvg, FedOpt, and PACFL shrinks or vanishes under that correction.","tokens_in":20246,"feed_emoji":"🧩","tokens_out":9085,"duration_ms":85383,"temperature":0.7,"pith_summary":"Federated learning over multimodal data usually assumes every client has every modality; in practice some clients lack text, images, or sensor channels, and existing fixes either reconstruct the missing modality or require task-specific architectures. MMiC tries to make the training process itself robust instead: within each cluster it detects the single network layer most disturbed by missing-modality batches, replaces that poverty layer with a weighted average of parameters from complete-modality clients in the same cluster, then uses the Banzhaf Power Index to bias client selection toward contributors, and a portfolio-optimization rule to down-weight risky clusters during global aggregation. The paper claims that this three-part recipe outperforms standard federated, clustered-federated, and missing-modality baselines in both global and personalized evaluation across classification and retrieval datasets. A sympathetic reading is that the authors are trying to show that missing modalities can be handled by editing and selection inside an existing clustered federated loop, without rebuilding the base model.","feed_headline":"MMiC beats federated baselines when data modalities are missing","feed_subtitle":"A poverty-layer repair, power-index selection, and portfolio aggregation take top global and personalized scores on three multimodal…","key_machinery":"Three coupled mechanisms carry the argument. The poverty layer is identified by $\\rho_{k,t}^{e,b}(p)$, the mean relative parameter change of layer $p$ between a batch with complete modalities and the following batch with a missing modality, and is edited via $\\boldsymbol{\\theta}_{k,t}(P_{k,t}) = \\sum_j w_j \\boldsymbol{\\theta}_{j,t}(P_{k,t})$ using parameters of complete-modality clients in the cluster. The Banzhaf Power Index appears through a per-round return $a_{i,t}$ and a core-member test: client $i$ is core if excluding it pushes the cluster's return below its threshold, and selection probabilities are $\\mathrm{prob}(i) = e^{\\tau \\phi(i)/T(i)} / \\sum_{i \\in \\mathcal{M}_m} e^{\\tau \\phi(i)/T(i)}$. The portfolio module computes intra-cluster covariance $\\bar{\\sigma}_{m,t}$, a risk-adjusted return coefficient $\\gamma_{m,t} = \\lambda \\bar{\\sigma}_{m,t} - (1-\\lambda) A_{m,t}$, and a dynamic $\\beta^*_{m,t}$ that retains more historical updates when a cluster is risky. The poverty substitution is the model-agnostic repair, BPI biases which clients participate, and MPO gates how much each cluster's update counts in global aggregation.","core_discovery":"The central claim is that modality incompleteness can be mitigated at the level of the federated optimization loop rather than at the data or architecture level. For each client, MMiC identifies the layer whose parameters change most sharply when a missing-modality batch follows a complete-modality batch, calls it the poverty layer, and substitutes a weighted average of corresponding parameters from cluster peers that trained on complete data. Client selection then replaces uniform randomness with a probability based on the Banzhaf Power Index: a client is a core member when removing it from the selected set drops the cluster's return below a threshold, and historical core counts are balanced against selection counts to set future probabilities. Global aggregation extends adaptive federated optimization with a risk-return score per cluster, adjusting how much of the current cluster update is retained. The paper reports that MMiC and its two variants, one with modality reconstruction and one with geometric multimodal contrastive fusion, achieve the top global and personalized results across CrisisMMD, UPMC-Food101, and Flickr30k under three missing-modality settings.","pith_inferences":["Replacing only the single most-affected layer is a communication-saving choice; the paper's own conclusion flags substituting more poverty parameters, so a direct extension is to make the number and identity of replaced layers adaptive.","Since the poverty-layer signal is defined by a batch-to-batch parameter change, the same detector could be used to schedule when to activate reconstruction modules or when to pause a client's contribution, not just where to edit.","The Banzhaf and portfolio modules both depend on a scalar per-client return; one could test cheaper proxies such as gradient norm or loss change, which would make the mechanism applicable when per-client evaluation labels are unavailable.","The paper fixes missing-modality rates per client; a stronger stress test would vary the missing rates round by round, exercising the selection and aggregation modules under the non-stationary conditions they are designed for."],"forward_implications":["If the poverty-layer signal is reliable, missing modalities can be repaired by editing one parameter layer per client, so a cluster model can keep training without reconstructing the absent data.","Because BPI changes only which clients join, the framework can cut communication rounds: on Flickr30k, random selection needed 12, 24, or 16 extra rounds to match BPI selection across the three missing-modality settings.","MPO-based dynamic $\\beta^*$ slows performance decline after convergence, so the same mechanism that hedges missing modalities also mitigates client overfitting.","The recipe transfers across modalities: on accelerometer-gyroscope activity data (UCI-HAR and KU-HAR), MMiC beats FedOpt by about 6 and 2 percent respectively, supporting the modality-agnostic claim.","Because MMiC is model-independent, the three modules can be layered onto different multimodal base models, including a lightweight concatenation model, CLIP, SCAN, and GMC fusion, without changing the core training objective."],"supporting_citations":[{"why":"Introduces the base federated-averaging objective, serves as the primary baseline, and supplies the global test-set evaluation protocol.","marker":"[29]"},{"why":"Provides the adaptive federated aggregation mechanism that MMiC extends with portfolio-style cluster weighting.","marker":"[38]"},{"why":"Supplies the one-shot clustered federated learning setting and clustering baseline that MMiC builds on.","marker":"[47]"},{"why":"Supplies the Locality-Sensitive-Hashing clustering used in the experiments for client grouping.","marker":"[52]"},{"why":"Defines a missing-modality MFL baseline with server-side prototype compensation that MMiC is compared against and contrasts with.","marker":"[1]"},{"why":"Provides the FedMultimodal benchmark protocol, base models, and evaluation settings for the multimodal datasets.","marker":"[10]"},{"why":"Supplies the SCAN retrieval model that carries the Flickr30k evaluation.","marker":"[22]"},{"why":"Supplies the CLIP backbone that carries the UPMC-Food101 classification evaluation.","marker":"[36]"},{"why":"Establishes the Banzhaf Power Index properties used to score client contribution in the selection module.","marker":"[7]"},{"why":"Establishes the portfolio-selection theory that underlies the risk-return coefficient used in global aggregation.","marker":"[30]"}],"fun_headline_variants":["MMiC swaps poverty layers to repair missing modalities in clusters","Banzhaf power index drives client choice in missing-modality MFL","Portfolio-style aggregation beats baselines when modalities are missing","Cluster peer substitution fixes missing-modality federated learning","MMiC: poverty-layer repair, power-index selection, portfolio aggregation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-client performance returns $a_{i,t}$ used by both the Banzhaf selection and the portfolio aggregation are honest signals, but the paper does not say whether they are computed on the cluster or global test sets described in Section 5.1.3; if they are, the method is being selected and weighted with the evaluation answer key in hand.","fun_headline_variants_meta":{"raw":{"variants":["MMiC swaps poverty layers to repair missing modalities in clusters","Banzhaf power index drives client choice in missing-modality MFL","Portfolio-style aggregation beats baselines when modalities are missing","Cluster peer substitution fixes missing-modality federated learning","MMiC: poverty-layer repair, power-index selection, portfolio aggregation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000865,"raw_usage":{"total_tokens":3761,"prompt_tokens":970,"completion_tokens":2791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":2703}},"tokens_in":586,"tokens_out":2791,"duration_ms":21358,"temperature":1.0,"reasoning_tokens":2703,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:30:32.560795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released code for the data source behind $a_{i,t}$ in Eq. (7). If $a_{i,t}$ is the accuracy, F1, or RSum computed on the cluster test set visible to all clients or on the global test set, rerun the comparison with $a_{i,t}$ computed only from local training batches or a held-out per-client validation split; the central claim collapses if MMiC's edge over FedAvg, FedOpt, and PACFL shrinks or vanishes under that correction.","supporting_citations":[{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Introduces the base federated-averaging objective, serves as the primary baseline, and supplies the global test-set evaluation protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the adaptive federated aggregation mechanism that MMiC extends with portfolio-style cluster weighting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the one-shot clustered federated learning setting and clustering baseline that MMiC builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Locality-Sensitive-Hashing clustering used in the experiments for client grouping."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FedMultimodal benchmark protocol, base models, and evaluation settings for the multimodal datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SCAN retrieval model that carries the Flickr30k evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP backbone that carries the UPMC-Food101 classification evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the Banzhaf Power Index properties used to score client contribution in the selection module."}],"review_version":1}