{"id":"072a0776-8d3c-4f7f-8c9c-ce9489de9673","arxiv_id":"2506.22036","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A masked diffusion imputer plus dual distillation (MMFeD3-HidE) improves link prediction on a new federated multimodal knowledge graph benchmark with 50% missing visual/textual modalities.","lead":"This paper introduces a federated setting for multimodal knowledge graph completion, where institutions keep separate images, texts, and triples, and proposes a diffusion imputer with server-client distillation to handle missing modalities and client heterogeneity. It also builds an FB15K-237 benchmark with three client partitions and reports gains over its own baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The imputation sampler is unconditional: reverse generation starts from noise and Eq. (6) overwrites observed slots after sampling, so §IV.C's claim that missing modalities are drawn conditional on observed ones has no mechanism.","rationale":"The reader correctly identified the masked-diffusion objective as the weakest assumption, but the stress-test reveals a sharper, internal inconsistency: even if Eq. (7) trained the observed-slot reconstruction perfectly, the reverse-sampling procedure described in §IV.B and Fig. 3 never conditions the generated missing slots on observed modalities. The final imputation in Eq. (6) is a post-hoc overwrite, not a conditional draw. This breaks the paper's central claim that HidE recovers complete multimodal distributions from available modalities, because the mechanism asserted in §IV.C does not exist in the described pipeline. The gains in Table III could therefore be produced by the distillation losses alone, which would make the paper's headline contribution an artifact of its training objectives rather than genuine imputation. The Fed10 numeric contradiction independently undercuts the text's claim of beating the centralized upper bound on all three datasets. These are addressable in principle—one could adopt RePaint-style conditioning at each reverse step or direct guidance—but as written the central argument is not sound. No code, data, or error bars are provided, so the experimental results cannot be independently checked. For these reasons, the current manuscript should not be accepted as-is; the verdict should move from conditional to reject, unless the authors can demonstrate a conditional sampling mechanism and released reproducible code showing genuine imputation fidelity. This is not an ad hominem or a disagreement with the federated benchmark idea; it is a specific technical defect in the claimed mechanism.","tokens_in":25567,"tokens_out":6278,"duration_ms":73046,"concrete_test":"Reproduce the HidE inference path on an entity with a held-out modality: mask the text modality for an entity, record the ground-truth text embedding before masking, then run Algorithm 2's imputation exactly as specified (reverse from x_T plus Eq. (6)). Measure cosine similarity and L2 distance between the imputed text slot and the ground-truth text embedding. Compare against (a) sampling x_mis unconditionally from the diffusion prior with observed slots never shown, and (b) the same MMFeD3 pipeline with λ=0 so that L_DI is removed. If (a) matches the imputation quality, or if (b) retains the Table III gains, the claim that HidE recovers missing modalities is unsupported. Also re-run the FB15K-237-Fed10 row to check the upper-bound comparison.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing problem is not just an unverified distributional assumption; the described inference procedure cannot condition on observed modalities. In §IV.B, the reverse chain is generative: x_T is drawn from N(0,I) and pθ(x_{t-1}|x_t) is applied without conditioning on x0 to produce x̂_0. Eq. (6) then overwrites observed slots with the ground-truth x0 after sampling. The final missing slots are therefore drawn from the marginal pθ(x_mis), not from pθ(x_mis|x_obs). The sentence in §IV.C claiming \"x̂_0 ∼ pθ(x|x̂_obs=x_obs)\" does not describe this procedure. Because the masked objective in Eq. (7) supervises only observed slots and the sampler never conditions on x_obs, there is no mechanism by which \"available modalities lead the missing ones to learn from the same reconstruction distributions.\" Consequently, the Table III gains can be explained entirely by the feature-distillation term L_FD (Eq. 11), which pulls imputed embeddings toward server embeddings, and by logit distillation, without any genuine modality recovery. In addition, Table III itself contradicts the text: on FB15K-237-Fed10, MMFeD3-HidE/MMFeD3 obtains MRR 0.372 versus MMCen-weighted 0.373, so the claim of outperforming the upper bound on all three datasets is not supported by the paper's own numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, Federated Multimodal Knowledge Graph Completion (FedMKGC), where multiple clients hold multimodal knowledge graphs with potentially missing visual/textual modalities and aim to train a global model without sharing raw data. The proposed framework MMFeD3-HidE combines a hyper-modal diffusion imputation model (HidE) that is intended to recover missing modalities from incomplete entity embeddings, with a federated dual-distillation mechanism (MMFeD3) transferring knowledge between clients and server via logit and feature distillation. The authors construct a benchmark based on FB15K-237 with non-IID partitions and random modality missingness, and compare against several FL and incomplete-multimodal baselines, reporting improvements on MRR and Hits@K. The paper also presents ablations, convergence analyses, and visualizations.","tokens_in":25916,"tokens_out":6149,"duration_ms":61504,"significance":"The task is timely and practical, and the paper is, to my knowledge, the first to formulate FedMKGC with uncertain missing modalities. The benchmark construction with Dirichlet-based multimodal partition and random availability masks is a useful contribution. The empirical results show consistent improvements over the included baselines. However, the central mechanism claimed for HidE—that missing modalities are imputed conditionally from available ones—is not supported by the described inference procedure, which weakens the interpretation of the gains. Provided the authors can fix the inference procedure or provide evidence that the imputation genuinely recovers missing modalities (rather than benefiting from distillation), the contribution could be valuable.","major_comments":[{"comment":"The reverse process is described as sampling from p_θ(x_{t-1}|x_t) starting from x_T ~ N(0,I), with no conditioning on the observed slots M^c ⊙ x0. Equation (6) overwrites the observed slots only after the full reverse chain, so the missing slots are drawn from the marginal p_θ(x_mis), not from p_θ(x_mis | x_obs). Consequently, the sentence in §IV.C asserting that 'x̂_0 ∼ pθ(x|x̂_obs^0 = x_obs^0)' does not describe the implemented procedure. Since Eq. (7) supervises only observed slots, there is no training signal that forces the missing slots to be consistent with observed modalities. The improvements in Table III could therefore be driven by the feature distillation loss L_FD (Eq. 11) and logit distillation rather than by genuine modality recovery. Please either modify the reverse process to condition on observed data at every step (e.g., by replacing observed slots with forward-noised observed values during sampling) or provide a controlled experiment that isolates the imputation contribution.","section":"§IV.B and §IV.C, Eq. (6)"},{"comment":"The text states that 'MMFeD3-HidE also outperforms MMCen-weighted on three datasets,' but Table III shows on FB15K-237-Fed10 an MRR of 0.372 for MMFeD3-HidE versus 0.373 for MMCen-weighted. The paper's own numbers do not support the claim of beating the upper bound on all three datasets. Please correct the claim and discuss the Fed10 result in the text.","section":"§V.C, Table III"},{"comment":"All reported numbers are single runs without variance estimates or significance tests. Several differences are very small (e.g., Fed5 MRR 0.382 vs. 0.380 for MMCen-weighted; Fed10 0.372 vs. 0.373), so without error bars or multiple seeds the reported improvements may be within noise. Please report means and standard deviations over at least three random seeds for all main tables.","section":"§V.A and Tables III–IV"},{"comment":"The derivation calls Eq. (7) a 'masked variational bound,' but masking the per-dimension squared error means the objective does not lower-bound the log-marginal likelihood of the full observation (only of the observed dimensions, and even that is not shown). The step from Eq. (18) to Eq. (19) drops the missing dimensions from the loss without providing any guarantee about their distribution. This is not merely a presentation issue: it is the basis for the claim that 'the completed three modalities are from the same distributions.' Please either provide a formal statement of what the masked objective optimizes, or soften the claims to say that the objective encourages consistency only among observed slots.","section":"§IV.C and App. B.B"}],"minor_comments":[{"comment":"The phrase 'It is non-trivial to propose address the FedMKGC task' should be 'It is non-trivial to address the FedMKGC task.'","section":"§I"},{"comment":"The phrase 'MKG multimodal information partition following Dirichlet distribution' is grammatically awkward; please rephrase for clarity.","section":"§III.B"},{"comment":"The average entity and triple counts are non-integer; state explicitly whether these are means rounded to one decimal place.","section":"Table II"},{"comment":"The hyperparameter sensitivity plots are difficult to read; the weight labels (e.g., '2 2', '2 1' at the top of the figure) appear corrupted and should be fixed.","section":"Fig. 7"},{"comment":"Variables S^{s,c}_{ro+1}, W^{s,c}_{v,ro+1}, W^{s,c}_{d,ro+1} are updated by the arg min but are not defined in the notation table; please clarify the update targets.","section":"Algorithm 2"},{"comment":"The notation \\x^{c}_0 appears before it is explicitly defined; consider introducing it as the original hyper-modal feature H^c.","section":"Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and task formulation are the strongest parts of this submission. However, the core imputation mechanism is not correctly described, and one of the three headline results is contradicted by the table. I would encourage the editor to seek a major revision that fixes the sampler or presents controlled experiments, and to require error bars. If the imputation claim cannot be substantiated, the paper should be reframed as a distillation-based method with a diffusion regularizer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read arXiv:2506.22036. The paper's real contribution is the FedMKGC task and benchmark: partitioning FB15K-237 into non-IID clients by relation IDs and Dirichlet-split multimodal features, with missing modalities generated by Bernoulli masks. That is a useful resource, and the baseline construction (three groups: MKGC fusion, FL, IMML) is thorough. The dual distillation (logit + feature) is a sensible adaptation of FedLU-style ideas to the multimodal setting and seems to help convergence.\n\nThe soft spot is the diffusion imputation story, and it is load-bearing. I checked the equations: the reverse process in Section IV.B is unconditional. x_T is drawn from N(0,I), pθ(x_{t-1}|x_t) is applied without conditioning on x_0, and Eq. (6) overwrites the observed slots only after sampling. So the missing slots are drawn from the marginal pθ(x_mis), not from pθ(x_mis|x_obs). The sentence in Section IV.C claiming x̂0 ∼ pθ(x|x̂obs=xobs) does not describe the procedure. The masked objective Eq. (7) supervises only observed slots, which trains the model to denoise observed coordinates, but it never injects observed coordinates into the reverse chain. As a result, the claim that \"available modalities lead the missing ones\" has no mechanism. Any gains attributed to modality recovery could equally come from the feature distillation term L_FD, which pulls imputed embeddings toward server embeddings, and from logit distillation.\n\nThere are smaller issues: no error bars anywhere; no code or data release; and Table III itself contradicts the text on FB15K-237-Fed10 (MMFeD3-HidE MRR 0.372 vs MMCen-weighted 0.373, so the \"outperforms upper bound on all three datasets\" claim is false for that row).\n\nThat said, the benchmark and task formulation remain useful, and the dual distillation is a reasonable baseline contribution. The diffusion flaw is addressable: use a conditional diffusion model (e.g., concatenate mask and observed values as input, or use a guided reverse process) and then re-run the ablation to see if the gains persist. As written, the central mechanism is unverified and the paper's own numbers undercut one headline claim. I would send it to peer review, because the benchmark and task are worth referee time, but the authors need to fix the imputation story or substantially weaken the claims.","headline":"Useful new FedMKGC benchmark and distillation framework, but the diffusion imputation's conditioning claim does not match the algorithm.","tokens_in":26447,"tokens_out":3300,"would_cite":false,"duration_ms":34394,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that MMFeD3-HidE, a federated multimodal knowledge graph completion framework, recovers randomly missing image and text modalities so effectively that it outperforms federated, incomplete-multimodal, and even…","keywords":["federated learning","multimodal knowledge graph completion","uncertain missing modalities","diffusion model imputation","knowledge distillation","non-IID multimodal data","link prediction"],"falsifier":"Replace the diffusion imputer in MMFeD3-HidE on FB15K-237-Fed3 at a 50% availability rate with constant or random padding of missing slots while keeping both distillation losses and all other hyperparameters; if the MRR stays near the reported 0.387 instead of collapsing toward the MMFedE range, the gains attributed to diffusion-based distribution recovery are not load-bearing. A complementary check is to hold out known image and text features, impute them from the remaining modalities, and measure each entity's distance to the true held-out features.","tokens_in":1731,"feed_emoji":"🧩","tokens_out":2655,"duration_ms":74658,"temperature":0.7,"pith_summary":"The paper introduces and evaluates Federated Multimodal Knowledge Graph Completion (FedMKGC), the task of completing missing links in knowledge graphs that are split across institutions and contain images and text that are partly and unpredictably unavailable in each client. To solve it, the paper proposes MMFeD3-HidE: a per-client diffusion model imputes the missing visual and textual embeddings, while a server-side dual distillation of logits and features coordinates the clients without transferring raw multimodal data. On its own benchmark built from FB15K-237 with three, five, and ten federated clients, the framework reports the best Hits@1 and MRR among federated and incomplete-multimodal baselines, and it edges out the centralized upper bound MMCen-weighted (e.g., MRR 0.387 vs 0.373 on FB15K-237-Fed3). A sympathetic reading is that the paper establishes the first working recipe for federated multimodal link prediction when modalities are missing with no reconstruction supervision.","feed_headline":"Federated knowledge graphs with missing data beat centralized training","feed_subtitle":"A diffusion-based imputer recovers lost images and text, improving link prediction past the centralized upper bound.","key_machinery":"The load-bearing mechanism is Hyper-modal Imputation Diffusion Embedding (HidE). An entity's multimodal features are concatenated into $H^c=[S^c||V^c||D^c]$, a hyper-modal vector in which the structural block $S^c$ is always available while visual and textual blocks are partly masked by $M^c$. HidE runs a standard DDPM forward process that adds Gaussian noise to $H^c$, then a reverse process in which a Cascade Residual Autoencoder (CRA) predicts the complete $\\hat{x}_0$ at each diffusion step. The training objective, Equation (7) (equivalently Equation (19)), supervises only the observed slots: $\\mathbb{E}\\,\\lVert M^c\\odot \\hat{x}_\\theta(x_t,t)-M^c\\odot x_0\\rVert_2^2$, and the final imputation fills the missing slots with the predicted values while keeping the observed slots intact. Around this, MMFeD3 adds a KL-divergence logit distillation in both directions between client and server, plus an L2 feature distillation between the client's imputed embeddings and the server's available-modality embeddings; the diffusion parameters stay local and are never transmitted.","core_discovery":"The paper's central claim is that uncertain modality unavailability in federated multimodal knowledge graphs can be handled by treating each entity's structural, visual, and textual embeddings as one incomplete hyper-modal vector and learning to complete that vector with a diffusion model whose training signal touches only the observed slots. On the paper's own terms, the discovery is that this masked diffusion imputation, combined with logit and feature distillation between clients and the server, yields better link prediction than federated graph baselines, incomplete-multimodal baselines, and even a centralized model trained on the same data with weighted fusion. The key experimental comparison is Table III at a 50% modality availability rate, where MMFeD3-HidE reports the best MRR and Hits@1 on all three Fed settings and improves over the centralized upper bound, which the paper attributes to genuine recovery of the unavailable modalities rather than to the extra training signal alone.","pith_inferences":["An untested but direct extension is that the same masked-diffusion objective could impute missing modality features in other federated graph tasks, provided each client keeps its own availability mask.","The comparison against MMCen-weighted may understate the true centralized upper bound, because that bound also trains on the randomly zero-padded incomplete features; a centralized model trained on genuinely complete modalities would be the stricter test.","A practical follow-up would feed the imputed visual and textual features back into the server aggregation so later rounds refine the imputation with global knowledge, whereas the current design keeps the diffusion model strictly local.","Because the diffusion parameters are never shared, HidE offers a natural starting point for differentially private federated multimodal reasoning, where noise added to gradients could be absorbed by the already noisy diffusion process."],"forward_implications":["Federated MKGC is feasible: training over decentralized multimodal knowledge graphs without sharing triples, images, or descriptions improves local link prediction beyond independent local training.","Imputing uncertain missing modalities with a masked diffusion objective improves MRR and Hits@1 beyond GAN-, autoencoder-, and cycle-consistency-based incomplete multimodal baselines.","Dual logit and feature distillation gives faster and more stable convergence than logit-only federated distillation baselines such as MMFedLU.","The full framework outperforms the centralized weighted-fusion upper bound on FB15K-237-Fed3, Fed5, and Fed10, suggesting that imputation can compensate for not having all data in one place.","At a 50% modality availability rate, performance stays close to the fully observed setting, so partial image and text coverage does not necessarily degrade reasoning quality."],"supporting_citations":[{"why":"Supplies the federated aggregation and permutation-mapping scheme for structural entity embeddings that MMFedE and MMFeD3 build on.","marker":"[22]"},{"why":"Supplies the non-IID data partition and Bernoulli availability-mask construction used to create the FedMKGC benchmark.","marker":"[27]"},{"why":"Supplies the DDPM forward and reverse processes and the ELBO simplification that HidE adapts into a masked imputation objective.","marker":"[82]"},{"why":"Provides the Cascade Residual Autoencoder used as HidE's reconstruction network for iterative modality recovery.","marker":"[46]"},{"why":"Provides the uncertain-missing-modalities baseline and cycle-consistency imputation objective adapted for the IMML comparison group.","marker":"[32]"},{"why":"Provides the RotatE score function used as the KGE decoder in all client and server objectives.","marker":"[71]"},{"why":"Provides the logit-distillation federated KGC baseline that MMFeD3 extends with a second feature-distillation channel.","marker":"[24]"},{"why":"Provides the KL-divergence logit distillation formulation that MMFeD3 uses for mutual server-client knowledge transfer.","marker":"[85]"}],"fun_headline_variants":["Federated graphs with missing modalities beat centralized training","Diffusion imputation plus distillation beats centralized in federated KG","Federated multimodal completion surpasses centralized via masked diffusion","Masked diffusion imputation beats centralized in federated KG"],"cache_read_input_tokens":28544,"weakest_assumption_plain":"The load-bearing premise is that supervising the diffusion model only on the observed modality slots of an entity is enough to force the imputed missing slots to come from the same distribution, so HidE's recovered visual and textual embeddings are genuinely meaningful rather than arbitrary fill-ins.","fun_headline_variants_meta":{"raw":{"variants":["Federated graphs with missing modalities beat centralized training","Diffusion imputation plus distillation beats centralized in federated KG","Federated multimodal completion surpasses centralized via masked diffusion","Masked diffusion imputation beats centralized in federated KG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000972,"raw_usage":{"total_tokens":4138,"prompt_tokens":954,"completion_tokens":3184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":3117}},"tokens_in":570,"tokens_out":3184,"duration_ms":23483,"temperature":1.0,"reasoning_tokens":3117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:13:05.869256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the diffusion imputer in MMFeD3-HidE on FB15K-237-Fed3 at a 50% availability rate with constant or random padding of missing slots while keeping both distillation losses and all other hyperparameters; if the MRR stays near the reported 0.387 instead of collapsing toward the MMFedE range, the gains attributed to diffusion-based distribution recovery are not load-bearing. A complementary check is to hold out known image and text features, impute them from the remaining modalities, and measure each entity's distance to the true held-out features.","supporting_citations":[{"cited_title":"Fede: Embedding knowledge graphs in federated setting,","cited_arxiv_id":null,"evidence_quote":"Supplies the federated aggregation and permutation-mapping scheme for structural entity embeddings that MMFedE and MMFeD3 build on."},{"cited_title":"Fedmultimodal: A bench- mark for multimodal federated learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the non-IID data partition and Bernoulli availability-mask construction used to create the FedMKGC benchmark."},{"cited_title":"Missing modalities imputation via cascaded residual autoencoder,","cited_arxiv_id":null,"evidence_quote":"Provides the Cascade Residual Autoencoder used as HidE's reconstruction network for iterative modality recovery."},{"cited_title":"Missing modality imagination network for emotion recognition with uncertain missing modalities,","cited_arxiv_id":null,"evidence_quote":"Provides the uncertain-missing-modalities baseline and cycle-consistency imputation objective adapted for the IMML comparison group."},{"cited_title":"Rotate: Knowledge graph embedding by relational rotation in complex space,","cited_arxiv_id":null,"evidence_quote":"Provides the RotatE score function used as the KGE decoder in all client and server objectives."},{"cited_title":"Heterogeneous federated knowledge graph embedding learning and unlearning,","cited_arxiv_id":null,"evidence_quote":"Provides the logit-distillation federated KGC baseline that MMFeD3 extends with a second feature-distillation channel."}],"review_version":1}