Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that MMFeD3-HidE, a federated multimodal knowledge graph completion framework, recovers randomly missing image and text modalities so effectively that it outperforms federated, incomplete-multimodal, and even…

desk verdict Useful new FedMKGC benchmark and distillation framework, but the diffusion imputation's conditioning claim does not match the algorithm. read the letter →

arxiv 2506.22036 v2 pith:ZYGZYDFF submitted 2025-06-27 cs.LG cs.MM

Ying Zhang , Yu Zhao , Xuhui Sui , Baohang Zhou , Xiangrui Cai , Li Shen , Xiaojie Yuan , Dacheng Tao This is my paper
classification cs.LGcs.MM
keywords federatedlearningmultimodalknowledgegraphcompletionuncertainmissingmodalitiesdiffusionmodelimputationdistillationnon-IIDdatalinkprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces and evaluates Federated Multimodal Knowledge Graph Completion (FedMKGC), the task of completing missing links in knowledge graphs that are split across institutions and contain images and text that are partly and unpredictably unavailable in each client. To solve it, the paper proposes MMFeD3-HidE: a per-client diffusion model imputes the missing visual and textual embeddings, while a server-side dual distillation of logits and features coordinates the clients without transferring raw multimodal data. On its own benchmark built from FB15K-237 with three, five, and ten federated clients, the framework reports the best Hits@1 and MRR among federated and incomplete-multimodal baselines, and it edges out the centralized upper bound MMCen-weighted (e.g., MRR 0.387 vs 0.373 on FB15K-237-Fed3). A sympathetic reading is that the paper establishes the first working recipe for federated multimodal link prediction when modalities are missing with no reconstruction supervision.

What carries the argument

The load-bearing mechanism is Hyper-modal Imputation Diffusion Embedding (HidE). An entity's multimodal features are concatenated into $H^c=[S^c||V^c||D^c]$, a hyper-modal vector in which the structural block $S^c$ is always available while visual and textual blocks are partly masked by $M^c$. HidE runs a standard DDPM forward process that adds Gaussian noise to $H^c$, then a reverse process in which a Cascade Residual Autoencoder (CRA) predicts the complete $\hat{x}_0$ at each diffusion step. The training objective, Equation (7) (equivalently Equation (19)), supervises only the observed slots: $\mathbb{E}\,\lVert M^c\odot \hat{x}_\theta(x_t,t)-M^c\odot x_0\rVert_2^2$, and the final imputation fills the missing slots with the predicted values while keeping the observed slots intact. Around this, MMFeD3 adds a KL-divergence logit distillation in both directions between client and server, plus an L2 feature distillation between the client's imputed embeddings and the server's available-modality embeddings; the diffusion parameters stay local and are never transmitted.

What would settle it

Replace the diffusion imputer in MMFeD3-HidE on FB15K-237-Fed3 at a 50% availability rate with constant or random padding of missing slots while keeping both distillation losses and all other hyperparameters; if the MRR stays near the reported 0.387 instead of collapsing toward the MMFedE range, the gains attributed to diffusion-based distribution recovery are not load-bearing. A complementary check is to hold out known image and text features, impute them from the remaining modalities, and measure each entity's distance to the true held-out features.

Watch

Extended reading notes

Core claim

The paper's central claim is that uncertain modality unavailability in federated multimodal knowledge graphs can be handled by treating each entity's structural, visual, and textual embeddings as one incomplete hyper-modal vector and learning to complete that vector with a diffusion model whose training signal touches only the observed slots. On the paper's own terms, the discovery is that this masked diffusion imputation, combined with logit and feature distillation between clients and the server, yields better link prediction than federated graph baselines, incomplete-multimodal baselines, and even a centralized model trained on the same data with weighted fusion. The key experimental comparison is Table III at a 50% modality availability rate, where MMFeD3-HidE reports the best MRR and Hits@1 on all three Fed settings and improves over the centralized upper bound, which the paper attributes to genuine recovery of the unavailable modalities rather than to the extra training signal alone.

Load-bearing premise

The load-bearing premise is that supervising the diffusion model only on the observed modality slots of an entity is enough to force the imputed missing slots to come from the same distribution, so HidE's recovered visual and textual embeddings are genuinely meaningful rather than arbitrary fill-ins.

Editorial extensions

If this is right

  • Federated MKGC is feasible: training over decentralized multimodal knowledge graphs without sharing triples, images, or descriptions improves local link prediction beyond independent local training.
  • Imputing uncertain missing modalities with a masked diffusion objective improves MRR and Hits@1 beyond GAN-, autoencoder-, and cycle-consistency-based incomplete multimodal baselines.
  • Dual logit and feature distillation gives faster and more stable convergence than logit-only federated distillation baselines such as MMFedLU.
  • The full framework outperforms the centralized weighted-fusion upper bound on FB15K-237-Fed3, Fed5, and Fed10, suggesting that imputation can compensate for not having all data in one place.
  • At a 50% modality availability rate, performance stays close to the fully observed setting, so partial image and text coverage does not necessarily degrade reasoning quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested but direct extension is that the same masked-diffusion objective could impute missing modality features in other federated graph tasks, provided each client keeps its own availability mask.
  • The comparison against MMCen-weighted may understate the true centralized upper bound, because that bound also trains on the randomly zero-padded incomplete features; a centralized model trained on genuinely complete modalities would be the stricter test.
  • A practical follow-up would feed the imputed visual and textual features back into the server aggregation so later rounds refine the imputation with global knowledge, whereas the current design keeps the diffusion model strictly local.
  • Because the diffusion parameters are never shared, HidE offers a natural starting point for differentially private federated multimodal reasoning, where noise added to gradients could be absorbed by the already noisy diffusion process.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a new task, Federated Multimodal Knowledge Graph Completion (FedMKGC), where multiple clients hold multimodal knowledge graphs with potentially missing visual/textual modalities and aim to train a global model without sharing raw data. The proposed framework MMFeD3-HidE combines a hyper-modal diffusion imputation model (HidE) that is intended to recover missing modalities from incomplete entity embeddings, with a federated dual-distillation mechanism (MMFeD3) transferring knowledge between clients and server via logit and feature distillation. The authors construct a benchmark based on FB15K-237 with non-IID partitions and random modality missingness, and compare against several FL and incomplete-multimodal baselines, reporting improvements on MRR and Hits@K. The paper also presents ablations, convergence analyses, and visualizations.

Significance. The task is timely and practical, and the paper is, to my knowledge, the first to formulate FedMKGC with uncertain missing modalities. The benchmark construction with Dirichlet-based multimodal partition and random availability masks is a useful contribution. The empirical results show consistent improvements over the included baselines. However, the central mechanism claimed for HidE—that missing modalities are imputed conditionally from available ones—is not supported by the described inference procedure, which weakens the interpretation of the gains. Provided the authors can fix the inference procedure or provide evidence that the imputation genuinely recovers missing modalities (rather than benefiting from distillation), the contribution could be valuable.

major comments (4)
  1. [§IV.B and §IV.C, Eq. (6)] The reverse process is described as sampling from p_θ(x_{t-1}|x_t) starting from x_T ~ N(0,I), with no conditioning on the observed slots M^c ⊙ x0. Equation (6) overwrites the observed slots only after the full reverse chain, so the missing slots are drawn from the marginal p_θ(x_mis), not from p_θ(x_mis | x_obs). Consequently, the sentence in §IV.C asserting that 'x̂_0 ∼ pθ(x|x̂_obs^0 = x_obs^0)' does not describe the implemented procedure. Since Eq. (7) supervises only observed slots, there is no training signal that forces the missing slots to be consistent with observed modalities. The improvements in Table III could therefore be driven by the feature distillation loss L_FD (Eq. 11) and logit distillation rather than by genuine modality recovery. Please either modify the reverse process to condition on observed data at every step (e.g., by replacing observed slots with forward-noised observed values during sampling) or provide a controlled experiment that isolates the imputation contribution.
  2. [§V.C, Table III] The text states that 'MMFeD3-HidE also outperforms MMCen-weighted on three datasets,' but Table III shows on FB15K-237-Fed10 an MRR of 0.372 for MMFeD3-HidE versus 0.373 for MMCen-weighted. The paper's own numbers do not support the claim of beating the upper bound on all three datasets. Please correct the claim and discuss the Fed10 result in the text.
  3. [§V.A and Tables III–IV] All reported numbers are single runs without variance estimates or significance tests. Several differences are very small (e.g., Fed5 MRR 0.382 vs. 0.380 for MMCen-weighted; Fed10 0.372 vs. 0.373), so without error bars or multiple seeds the reported improvements may be within noise. Please report means and standard deviations over at least three random seeds for all main tables.
  4. [§IV.C and App. B.B] The derivation calls Eq. (7) a 'masked variational bound,' but masking the per-dimension squared error means the objective does not lower-bound the log-marginal likelihood of the full observation (only of the observed dimensions, and even that is not shown). The step from Eq. (18) to Eq. (19) drops the missing dimensions from the loss without providing any guarantee about their distribution. This is not merely a presentation issue: it is the basis for the claim that 'the completed three modalities are from the same distributions.' Please either provide a formal statement of what the masked objective optimizes, or soften the claims to say that the objective encourages consistency only among observed slots.
minor comments (6)
  1. [§I] The phrase 'It is non-trivial to propose address the FedMKGC task' should be 'It is non-trivial to address the FedMKGC task.'
  2. [§III.B] The phrase 'MKG multimodal information partition following Dirichlet distribution' is grammatically awkward; please rephrase for clarity.
  3. [Table II] The average entity and triple counts are non-integer; state explicitly whether these are means rounded to one decimal place.
  4. [Fig. 7] The hyperparameter sensitivity plots are difficult to read; the weight labels (e.g., '2 2', '2 1' at the top of the figure) appear corrupted and should be fixed.
  5. [Algorithm 2] Variables S^{s,c}_{ro+1}, W^{s,c}_{v,ro+1}, W^{s,c}_{d,ro+1} are updated by the arg min but are not defined in the notation table; please clarify the update targets.
  6. [Eq. (6)] The notation \x^{c}_0 appears before it is explicitly defined; consider introducing it as the original hyper-modal feature H^c.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity or load-bearing self-citation; one explanatory claim in Section IV.C restates its own distributional assumption as a conclusion.

  1. other [Section IV-C, Eq. (7) and the following paragraph]
    "Since after the optimization, the observed modalities and their reconstructed features are the same ˆxobs0 = xobs0, and the reconstructed hyper-modal ˆx0 ∼ pθ(x|ˆxobs0 = xobs0), thus estimated missing features ˆxmis0 are also from the same distributions... which made sure the completed three modalities are from the same distributions and maintain the semantic consistency."

    The 'same distributions' conclusion is the same assumption used to motivate Eq. (7): the text first says the missing modalities xmis_t are 'from the same distributions' so that constraining observed slots suffices, and after optimizing Eq. (7) it presents that assumption as a consequence. Eq. (7) is ∥M^c⊙x̂θ(xt,t)−M^c⊙x0∥², which contains no term for the (1−M^c) missing slots; Eq. (6) copies the observed slots after an unconditional reverse chain. Thus the conditional statement pθ(x|ˆxobs0=xobs0) is not implemented or implied by the objective; it is the input assumption restated as a result.

full rationale

No circularity was found in the main predictive chain: the KGC loss, logit/feature distillation, and masked diffusion loss are not fitted to MRR or Hits@K, no parameter is tuned on the target metric, and the cited prior work (FedE, DDPM, CRA, etc.) is used as a component rather than as self-referential authority. The benchmark is author-constructed, but it uses standard FB15K-237 and standard filtered ranking metrics, so the empirical comparison is not definitionally forced. The one genuinely circular passage is Section IV.C, where the claim that imputed missing modalities come from the same distribution as observed modalities is the motivating assumption restated after Eq. (7); because Eq. (7) only supervises observed slots and Eq. (6) overwrites them after an unconditional reverse sampling process, the conditional-imputation explanation has no mechanism. This is partly a correctness/omitted-proof concern rather than equation-level circularity, and it does not by itself invalidate the empirical comparison. Separately, the text's claim that MMFeD3-HidE outperforms MMCen-weighted on three datasets is contradicted by Table III on FB15K-237-Fed10 (MRR 0.372 vs 0.373); that is an internal inconsistency, not circularity. Overall circularity is limited to the one explanatory restatement, so the score is low.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'hyper-modal vector' Hc is a concatenation of existing structural, visual, and textual embeddings, not an entity with independent evidence requirements.

free parameters (6)
  • lambda, mu, eta (loss weights) = not stated in main text; swept in Fig. 7
    Balance L_KGC, L_DI, L_LD, and L_FD in Eq. (12); tuned on validation and affects all reported numbers.
  • diffusion schedule beta_low, beta_up, scale s = 5e-4, 5e-2, 1e-4
    Controls forward noise in Eq. (3)-(4); taken from Diffusion Recommender Model and not independently justified.
  • diffusion training steps T and inference steps = not stated; Fig. 7 sweeps training steps 0..5 and inference steps 2..50
    Number of denoising steps changes imputation quality; default configuration is not explicitly reported.
  • modality availability rate r = 50% in main experiments
    Defines the missing-modality regime; results and conclusions depend on this chosen rate.
  • partition heterogeneity alpha_Dir = 0.1
    Controls non-IID split of images and texts in Section III.B; high heterogeneity chosen by hand.
  • entity and relation embedding sizes, negative samples, batch size, local epochs = 512, 256, 256, 1024, 3
    Implementation choices from [71] and [22]; not fitted but affect capacity and optimization.
assumptions (6)
  • standard math DDPM ELBO simplification and Gaussian posterior assumptions (Eq. 13-21)
    Relies on standard DDPM derivation with fixed noise schedule and Gaussian transition kernels.
  • domain assumption Missing visual/textual modalities are missing completely at random, independent of entity semantics
    Section III.C constructs masks with Bernoulli draw; real missingness could be informative, which would break the imputation training signal.
  • domain assumption Enforcing equality on observed slots forces unobserved slots to come from the same distribution
    Section IV.C Eq. (7)-(8); the key mechanism is asserted, not proven.
  • domain assumption Partition by relation IDs and Dirichlet-split images/texts yields realistic cross-silo MKGs
    Section III.B; realism of the benchmark is assumed and not validated against real federated MKGs.
  • domain assumption Fixed pretrained BERT/ViT features are sufficient representations of textual and visual modalities
    Section III.A; no end-to-end fine-tuning or feature quality analysis is provided.
  • domain assumption Not sharing raw features, relation embeddings, or diffusion parameters preserves privacy
    Appendix C claims privacy from non-sharing; the paper admits no provable guarantee and suggests future differential privacy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion." pith.science (2026). https://pith.science/paper/ZYGZYDFF

@misc{pith2026250622036,
  author       = {Pith},
  title        = {Pith review of: Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYGZYDFF}},
  note         = {Machine review of arXiv:2506.22036}
}
read the original abstract

With the increasing multimodal knowledge privatization requirements, multimodal knowledge graphs in different institutes are usually decentralized, lacking of effective collaboration system with both stronger reasoning ability and transmission safety guarantees. In this paper, we propose the Federated Multimodal Knowledge Graph Completion (FedMKGC) task, aiming at training over federated MKGs for better predicting the missing links in clients without sharing sensitive knowledge. We propose a framework named MMFeD3-HidE for addressing multimodal uncertain unavailability and multimodal client heterogeneity challenges of FedMKGC. (1) Inside the clients, our proposed Hyper-modal Imputation Diffusion Embedding model (HidE) recovers the complete multimodal distributions from incomplete entity embeddings constrained by available modalities. (2) Among clients, our proposed Multimodal FeDerated Dual Distillation (MMFeD3) transfers knowledge mutually between clients and the server with logit and feature distillation to improve both global convergence and semantic consistency. We propose a FedMKGC benchmark for a comprehensive evaluation, consisting of a general FedMKGC backbone named MMFedE, datasets with heterogeneous multimodal information, and three groups of constructed baselines. Experiments conducted on our benchmark validate the effectiveness, semantic consistency, and convergence robustness of MMFeD3-HidE.

Figures

Figures reproduced from arXiv: 2506.22036 by the authors.

Figure 1
Figure 1. Toy example of the FedMKGC task for decentralized MKGs. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our MMFeD3-HidE. S c , Vc , Dc represent structural, visual, and textual modalities in client MKGs, where shadowed blocks represent missing. The HidE imputes the missing modalities of entities with diffusion imputation. The MMFeD3 optimizes the federated MKGC with dual distillation objectives and KGC objectives of clients and server. C. Missing Modalities Construction We randomly generate the modalit… view at source ↗
Figure 3
Figure 3. The HidE model details, including hyper-modal construction, forward [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: FL baseline performance on valid set during iteration. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: MMFeD3-HidE performance with different reconstruction networks. The rankings of each reconstruction network are labeled above the bars. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance of MMFeD3-HidE with different hyper-parameters. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison between MMFedE and MMFeD3-HidE with [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dark Side of Modalities: Reinforced Multimodal Distillation for Multimodal Knowledge Graph Reasoning

    cs.MM 2025-07 conditional novelty 6.0 of 10

    A unimodal student model, taught by reinforcement-selected combinations of multimodal teachers via neighbor-decoupled knowledge distillation, sets new state-of-the-art results on five multimodal knowledge graph reason...

Reference graph

Works this paper leans on

91 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Multi-modal knowledge graph construction and application: A survey,

    X. Zhu, Z. Li, X. Wang, X. Jiang, P. Sun, X. Wang, Y . Xiao, and N. J. Yuan, “Multi-modal knowledge graph construction and application: A survey,” IEEE Transactions on Knowledge and Data Engineering , vol. 36, no. 2, pp. 715–735, 2024

    2024

  2. [2]

    A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal,

    K. Liang, L. Meng, M. Liu, Y . Liu, W. Tu, S. Wang, S. Zhou, X. Liu, F. Sun, and K. He, “A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal,” IEEE TPAMI, 2024

    2024

  3. [3]

    Ok-vqa: A visual question answering benchmark requiring external knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in CVPR, 2019, pp. 3195–3204

    2019

  4. [4]

    Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering,

    Y . Ding, J. Yu, B. Liu, Y . Hu, M. Cui, and Q. Wu, “Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5089–5098

    2022

  5. [5]

    Multi-modal knowledge graphs for recommender systems,

    R. Sun, X. Cao, Y . Zhao, J. Wan, K. Zhou, F. Zhang, Z. Wang, and K. Zheng, “Multi-modal knowledge graphs for recommender systems,” in CIKM, 2020, pp. 1405–1414

    2020

  6. [6]

    Context matters: Distilling knowledge graph for enhanced object detection,

    A. Yang, S. Lin, C. Yeh, M. Shu, Y . Yang, and X. Chang, “Context matters: Distilling knowledge graph for enhanced object detection,” IEEE Trans. Multim. , vol. 26, pp. 487–500, 2024

    2024

  7. [7]

    Image-embodied knowledge representation learning,

    R. Xie, Z. Liu, H. Luan, and M. Sun, “Image-embodied knowledge representation learning,” in IJCAI, 2017, pp. 3140–3146

    2017

  8. [8]

    Representation learning of knowledge graphs with entity descriptions,

    R. Xie, Z. Liu, J. Jia, H. Luan, and M. Sun, “Representation learning of knowledge graphs with entity descriptions,” in AAAI, vol. 30, no. 1, 2016

    2016

Show all 91 references
  1. [9]

    Is visual context really helpful for knowledge graph? a representation learning perspective,

    M. Wang, S. Wang, H. Yang, Z. Zhang, X. Chen, and G. Qi, “Is visual context really helpful for knowledge graph? a representation learning perspective,” in ACM MM, 2021, pp. 2735–2743

    2021

  2. [10]

    Mose: Modality split and ensemble for multimodal knowledge graph completion,

    Y . Zhao, X. Cai, Y . Wu, H. Zhang, Y . Zhang, G. Zhao, and N. Jiang, “Mose: Modality split and ensemble for multimodal knowledge graph completion,” in EMNLP, 2022, pp. 10 527–10 536

    2022

  3. [11]

    Hybrid transformer with multi-level fusion for multimodal knowledge graph completion,

    X. Chen, N. Zhang, L. Li, S. Deng, C. Tan, C. Xu, F. Huang, L. Si, and H. Chen, “Hybrid transformer with multi-level fusion for multimodal knowledge graph completion,” in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retr...

    2022

  4. [12]

    Lafa: Multimodal knowledge graph completion with link aware fusion and aggregation,

    B. Shang, Y . Zhao, J. Liu, and D. Wang, “Lafa: Multimodal knowledge graph completion with link aware fusion and aggregation,” in Proceed- ings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 8, 2024, pp. 8957–8965

    2024

  5. [13]

    Dbpedia: A nucleus for a web of open data,

    S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, “Dbpedia: A nucleus for a web of open data,” in international semantic web conference. Springer, 2007, pp. 722–735

    2007

  6. [14]

    Wikidata: a free collaborative knowl- edgebase,

    D. Vrande ˇci´c and M. Kr ¨otzsch, “Wikidata: a free collaborative knowl- edgebase,” Communications of the ACM , vol. 57, no. 10, pp. 78–85, 2014

    2014

  7. [15]

    Free- base: a collaboratively created graph database for structuring human knowledge,

    K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Free- base: a collaboratively created graph database for structuring human knowledge,” in Proceedings of the 2008 ACM SIGMOD international conference on Management of data , 2008, pp. 1247–1250. 13

    2008

  8. [16]

    Amazon-kg: A knowledge graph enhanced cross-domain recommendation dataset,

    Y . Wang, Q. Xie, M. Tang, L. Li, J. Yuan, and Y . Liu, “Amazon-kg: A knowledge graph enhanced cross-domain recommendation dataset,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2024, pp. 123– 130

    2024

  9. [17]

    Alime mkg: A multi-modal knowledge graph for live- streaming e-commerce,

    G. Xu, H. Chen, F.-L. Li, F. Sun, Y . Shi, Z. Zeng, W. Zhou, Z. Zhao, and J. Zhang, “Alime mkg: A multi-modal knowledge graph for live- streaming e-commerce,” in Proceedings of the 30th ACM CIKM , 2021, pp. 4808–4812

    2021

  10. [18]

    Learning a health knowledge graph from electronic medical records,

    M. Rotmensch, Y . Halpern, A. Tlimat, S. Horng, and D. Sontag, “Learning a health knowledge graph from electronic medical records,” Scientific reports, vol. 7, no. 1, p. 5994, 2017

    2017

  11. [19]

    Federated machine learning: Concept and applications,

    Q. Yang, Y . Liu, T. Chen, and Y . Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019

    2019

  12. [20]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273– 1282

    2017

  13. [21]

    Advances and open problems in federated learning,

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al. , “Advances and open problems in federated learning,” Foundations and trends® in machine learning , vol. 14, no. 1–2, pp. 1–210, 2021

    2021

  14. [22]

    Fede: Embedding knowledge graphs in federated setting,

    M. Chen, W. Zhang, Z. Yuan, Y . Jia, and H. Chen, “Fede: Embedding knowledge graphs in federated setting,” in Proceedings of the 10th International Joint Conference on Knowledge Graphs , 2021, pp. 80– 88

    2021

  15. [23]

    Federated knowledge graph completion via embedding- contrastive learning,

    ——, “Federated knowledge graph completion via embedding- contrastive learning,” Knowledge-Based Systems, 2022

    2022

  16. [24]

    Heterogeneous federated knowledge graph embedding learning and unlearning,

    X. Zhu, G. Li, and W. Hu, “Heterogeneous federated knowledge graph embedding learning and unlearning,” in Proceedings of the ACM Web Conference 2023, 2023, pp. 2444–2454

    2023

  17. [25]

    Fedean: Entity-aware adversarial negative sampling for federated knowledge graph reasoning,

    L. Meng, K. Liang, H. Yu, Y . Liu, S. Zhou, M. Liu, and X. Liu, “Fedean: Entity-aware adversarial negative sampling for federated knowledge graph reasoning,” IEEE Transactions on Knowledge and Data Engi- neering, 2024

    2024

  18. [26]

    Imf: Interactive multimodal fusion model for link prediction,

    X. Li, X. Zhao, J. Xu, Y . Zhang, and C. Xing, “Imf: Interactive multimodal fusion model for link prediction,” in ACM Web Conference, 2023, pp. 2572–2580

    2023

  19. [27]

    Fedmultimodal: A bench- mark for multimodal federated learning,

    T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang, S. Avestimehr, and S. Narayanan, “Fedmultimodal: A bench- mark for multimodal federated learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 4035–4045

    2023

  20. [28]

    Multimodal federated learning via contrastive representation ensemble,

    Q. Yu, Y . Liu, Y . Wang, K. Xu, and J. Liu, “Multimodal federated learning via contrastive representation ensemble,” in The Eleventh In- ternational Conference on Learning Representations , 2023

    2023

  21. [29]

    A unified framework for multi- modal federated learning,

    B. Xiong, X. Yang, F. Qi, and C. Xu, “A unified framework for multi- modal federated learning,” Neurocomputing, vol. 480, pp. 110–118, 2022

    2022

  22. [30]

    Maco: A modality adversarial and contrastive framework for modality-missing multi-modal knowledge graph completion,

    Y . Zhang, Z. Chen, and W. Zhang, “Maco: A modality adversarial and contrastive framework for modality-missing multi-modal knowledge graph completion,” in CCF International Conference on Natural Lan- guage Processing and Chinese Computing . Springer, 2023, pp. 123– 134

    2023

  23. [31]

    Unleashing the power of imbalanced modality information for multi-modal knowledge graph completion,

    Y . Zhang, Z. Chen, L. Liang, H. Chen, and W. Zhang, “Unleashing the power of imbalanced modality information for multi-modal knowledge graph completion,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ...

    2024

  24. [32]

    Missing modality imagination network for emotion recognition with uncertain missing modalities,

    J. Zhao, R. Li, and Q. Jin, “Missing modality imagination network for emotion recognition with uncertain missing modalities,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Langu...

    2021

  25. [33]

    Transformer-based feature re- construction network for robust multimodal sentiment analysis,

    Z. Yuan, W. Li, H. Xu, and W. Yu, “Transformer-based feature re- construction network for robust multimodal sentiment analysis,” in Proceedings of the 29th ACM International Conference on Multimedia , 2021, pp. 4400–4407

    2021

  26. [34]

    Robust multimodal sentiment analysis via tag encoding of uncertain missing modalities,

    J. Zeng, J. Zhou, and T. Liu, “Robust multimodal sentiment analysis via tag encoding of uncertain missing modalities,” IEEE Transactions on Multimedia, vol. 25, pp. 6301–6314, 2022

    2022

  27. [35]

    Unimf: a unified multimodal framework for multimodal sentiment analysis in missing modalities and unaligned multimodal sequences,

    R. Huan, G. Zhong, P. Chen, and R. Liang, “Unimf: a unified multimodal framework for multimodal sentiment analysis in missing modalities and unaligned multimodal sequences,” IEEE Transactions on Multimedia , 2023

    2023

  28. [36]

    Incomplete multimodality-diffused emo- tion recognition,

    Y . Wang, Y . Li, and Z. Cui, “Incomplete multimodality-diffused emo- tion recognition,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  29. [37]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems , vol. 2, pp. 429–450, 2020

    2020

  30. [38]

    Federated learning on non-iid data silos: An experimental study,

    Q. Li, Y . Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th international conference on data engineering (ICDE) . IEEE, 2022, pp. 965–978

    2022

  31. [39]

    Fedsea: Federated learning via selective feature alignment for non-iid multimodal data,

    M. Tan, Y . Feng, L. Chu, J. Shi, R. Xiao, H. Tang, and J. Yu, “Fedsea: Federated learning via selective feature alignment for non-iid multimodal data,” IEEE Trans. Multim. , vol. 26, pp. 5807–5822, 2024

    2024

  32. [40]

    Gain: Missing data imputation using generative adversarial nets,

    J. Yoon, J. Jordon, and M. Schaar, “Gain: Missing data imputation using generative adversarial nets,” in International conference on machine learning. PMLR, 2018, pp. 5689–5698

    2018

  33. [41]

    Multivariate time series imputation with generative adversarial networks,

    Y . Luo, X. Cai, Y . Zhang, J. Xu et al. , “Multivariate time series imputation with generative adversarial networks,” NeurIPS, vol. 31, 2018

    2018

  34. [42]

    Generative adversarial networks,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,” Communications of the ACM , vol. 63, no. 11, pp. 139–144, 2020

    2020

  35. [43]

    Vigan: Missing view imputation with generative adversarial networks,

    C. Shang, A. Palmer, J. Sun, K.-S. Chen, J. Lu, and J. Bi, “Vigan: Missing view imputation with generative adversarial networks,” in 2017 IEEE International conference on big data (Big Data) . IEEE, 2017, pp. 766–775

    2017

  36. [44]

    Extract- ing and composing robust features with denoising autoencoders,

    P. Vincent, H. Larochelle, Y . Bengio, and P.-A. Manzagol, “Extract- ing and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on Machine learning , 2008, pp. 1096–1103

    2008

  37. [45]

    Autoencoders, unsupervised learning, and deep architectures,

    P. Baldi, “Autoencoders, unsupervised learning, and deep architectures,” in Proceedings of ICML workshop on unsupervised and transfer learn- ing. JMLR Workshop and Conference Proceedings, 2012, pp. 37–49

    2012

  38. [46]

    Missing modalities imputation via cascaded residual autoencoder,

    L. Tran, X. Liu, J. Zhou, and R. Jin, “Missing modalities imputation via cascaded residual autoencoder,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 1405–1414

    2017

  39. [47]

    Distribution-consistent modal recovering for incomplete multimodal learning,

    Y . Wang, Z. Cui, and Y . Li, “Distribution-consistent modal recovering for incomplete multimodal learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 025–22 034

    2023

  40. [48]

    Gcnet: Graph completion network for incomplete multimodal learning in conversation,

    Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao, “Gcnet: Graph completion network for incomplete multimodal learning in conversation,” IEEE Transactions on pattern analysis and machine intelligence, vol. 45, no. 7, pp. 8419–8432, 2023

    2023

  41. [49]

    Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,

    J. Zeng, J. Zhou, and T. Liu, “Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2924–2934

    2022

  42. [50]

    Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities,

    M. Li, D. Yang, X. Zhao, S. Wang, Y . Wang, K. Yang, M. Sun, D. Kou, Z. Qian, and L. Zhang, “Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  43. [51]

    Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning,

    S. Wei, C. Luo, and Y . Luo, “Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 039–20 049

    2023

  44. [52]

    Federated learning for vision- and-language grounding problems,

    F. Liu, X. Wu, S. Ge, W. Fan, and Y . Zou, “Federated learning for vision- and-language grounding problems,” in AAAI, vol. 34, no. 07, 2020, pp. 11 572–11 579

    2020

  45. [53]

    Pfedprompt: Learning personalized prompt for vision-language models in federated learning,

    T. Guo, S. Guo, and J. Wang, “Pfedprompt: Learning personalized prompt for vision-language models in federated learning,” in Proceed- ings of the ACM Web Conference 2023 , 2023, pp. 1364–1374

    2023

  46. [54]

    Federated modality-specific encoders and multimodal anchors for personalized brain tumor segmentation,

    Q. Dai, D. Wei, H. Liu, J. Sun, L. Wang, and Y . Zheng, “Federated modality-specific encoders and multimodal anchors for personalized brain tumor segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 2, 2024, pp. 1445–1453

    2024

  47. [55]

    Multi- modal federated learning for cancer staging over non-iid datasets with unbalanced modalities,

    K. Borazjani, N. Khosravan, L. Ying, and S. Hosseinalipour, “Multi- modal federated learning for cancer staging over non-iid datasets with unbalanced modalities,” IEEE Transactions on Medical Imaging , 2024

    2024

  48. [56]

    Multimodal federated learning on iot data,

    Y . Zhao, P. Barnaghi, and H. Haddadi, “Multimodal federated learning on iot data,” in 2022 IEEE/ACM IoTDI . IEEE, 2022, pp. 43–54

    2022

  49. [57]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihal- cea, “Meld: A multimodal multi-party dataset for emotion recognition in conversations,” arXiv preprint arXiv:1810.02508 , 2018

    2018 arXiv

  50. [58]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,” IEEE transactions on affective computing , vol. 5, no. 4, pp. 377–390, 2014. 14

    2014

  51. [59]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv:1212.0402, 2012

    2012 arXiv

  52. [60]

    Moments in time dataset: one million videos for event understanding,

    M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, S. A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfreund, C. V ondrick et al., “Moments in time dataset: one million videos for event understanding,” IEEE transactions on pattern analysis and machine intelligence, vol. 42, no. 2, ...

    2019

  53. [61]

    The hateful memes challenge: Detecting hate speech in multimodal memes,

    D. Kiela, H. Firooz, A. Mohan, V . Goswami, A. Singh, P. Ringshia, and D. Testuggine, “The hateful memes challenge: Detecting hate speech in multimodal memes,” Advances in neural information processing systems, vol. 33, pp. 2611–2624, 2020

    2020

  54. [62]

    Crisismmd: Multimodal twitter datasets from natural disasters,

    F. Alam, F. Ofli, and M. Imran, “Crisismmd: Multimodal twitter datasets from natural disasters,” in Proceedings of the international AAAI con- ference on web and social media , vol. 12, no. 1, 2018

    2018

  55. [63]

    Reducing training time in cross-silo federated learning us- ing multigraph topology,

    T. Do, B. X. Nguyen, V . Pham, T. Tran, E. Tjiputra, Q. D. Tran, and A. Nguyen, “Reducing training time in cross-silo federated learning us- ing multigraph topology,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 19 409–19 419

    2023

  56. [64]

    Robust heterogeneous federated learning under data corruption,

    X. Fang, M. Ye, and X. Yang, “Robust heterogeneous federated learning under data corruption,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 5020–5030

    2023

  57. [65]

    Client-adaptive cross-model reconstruction network for modality-incomplete multimodal federated learning,

    B. Xiong, X. Yang, Y . Song, Y . Wang, and C. Xu, “Client-adaptive cross-model reconstruction network for modality-incomplete multimodal federated learning,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 1241–1249

    2023

  58. [66]

    Fedmsplit: Correlation-adaptive federated multi- task learning across multimodal split networks,

    J. Chen and A. Zhang, “Fedmsplit: Correlation-adaptive federated multi- task learning across multimodal split networks,” in Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, 2022, pp. 87–96

    2022

  59. [67]

    Con- trast then memorize: Semantic neighbor retrieval-enhanced inductive multimodal knowledge graph completion,

    Y . Zhao, Y . Zhang, B. Zhou, X. Qian, K. Song, and X. Cai, “Con- trast then memorize: Semantic neighbor retrieval-enhanced inductive multimodal knowledge graph completion,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Informatio...

    2024

  60. [68]

    Tokenization, fusion, and augmentation: Towards fine- grained multi-modal entity representation,

    Y . Zhang, Z. Chen, L. Guo, Y . Xu, B. Hu, Z. Liu, W. Zhang, and H. Chen, “Tokenization, fusion, and augmentation: Towards fine- grained multi-modal entity representation,” 2024. [Online]. Available: https://arxiv.org/abs/2404.09468

    2024 arXiv

  61. [69]

    Translating embeddings for modeling multi-relational data,

    A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” Advances in neural information processing systems , vol. 26, 2013

    2013

  62. [70]

    Com- plex embeddings for simple link prediction,

    T. Trouillon, J. Welbl, S. Riedel, ´E. Gaussier, and G. Bouchard, “Com- plex embeddings for simple link prediction,” in International conference on machine learning . PMLR, 2016, pp. 2071–2080

    2016

  63. [71]

    Rotate: Knowledge graph embedding by relational rotation in complex space,

    Z. Sun, Z.-H. Deng, J.-Y . Nie, and J. Tang, “Rotate: Knowledge graph embedding by relational rotation in complex space,” in ICLR, 2018

    2018

  64. [72]

    Differentially private fed- erated knowledge graphs embedding,

    H. Peng, H. Li, Y . Song, V . Zheng, and J. Li, “Differentially private fed- erated knowledge graphs embedding,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management , 2021, pp. 1416–1425

    2021

  65. [73]

    Fedcke: Cross-domain knowledge graph embedding in federated learning,

    W. Huang, J. Liu, T. Li, S. Ji, D. Wang, and T. Huang, “Fedcke: Cross-domain knowledge graph embedding in federated learning,” IEEE Transactions on Big Data , 2022

    2022

  66. [74]

    Efficient federated learning on knowledge graphs via privacy-preserving relation embedding aggregation,

    K. Zhang, Y . Wang, H. Wang, L. Huang, C. Yang, X. Chen, and L. Sun, “Efficient federated learning on knowledge graphs via privacy-preserving relation embedding aggregation,” in Findings of the Association for Computational Linguistics: EMNLP 2022 , 2022, pp. 613–621

    2022

  67. [75]

    Exploiting pre- trained language models for black-box attack against knowledge graph embeddings,

    G. Yang, L. Zhang, Y . Liu, H. Xie, and Z. Mao, “Exploiting pre- trained language models for black-box attack against knowledge graph embeddings,” ACM Transactions on Knowledge Discovery from Data , 2024

    2024

  68. [76]

    Poisoning attack on federated knowledge graph embedding,

    E. Zhou, S. Guo, Z. Ma, Z. Hong, T. Guo, and P. Dong, “Poisoning attack on federated knowledge graph embedding,” in Proceedings of the ACM on Web Conference 2024 , 2024, pp. 1998–2008

    2024

  69. [77]

    Quantifying and defending against privacy threats on federated knowl- edge graph embedding,

    Y . Hu, W. Liang, R. Wu, K. Xiao, W. Wang, X. Li, J. Liu, and Z. Qin, “Quantifying and defending against privacy threats on federated knowl- edge graph embedding,” in Proceedings of the ACM Web Conference 2023, 2023, pp. 2306–2317

    2023

  70. [78]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019, pp. 4171–4186

    2019

  71. [79]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations , 2020

    2020

  72. [80]

    Measuring the effects of non- identical data distribution for federated visual classification,

    T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non- identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335, 2019

    1909 arXiv

  73. [81]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML. PMLR, 2015, pp. 2256–2265

    2015

  74. [82]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  75. [83]

    Understanding diffusion models: A unified perspective,

    C. Luo, “Understanding diffusion models: A unified perspective,” arXiv preprint arXiv:2208.11970, 2022

    2022 arXiv

  76. [84]

    Think locally, act globally: Federated learning with local and global representations,

    P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Think locally, act globally: Federated learning with local and global representations,” arXiv preprint arXiv:2001.01523, 2020

    2001 arXiv

  77. [85]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  78. [86]

    Mmkg: multi-modal knowledge graphs,

    Y . Liu, H. Li, A. Garcia-Duran, M. Niepert, D. Onoro-Rubio, and D. S. Rosenblum, “Mmkg: multi-modal knowledge graphs,” in The Semantic Web: 16th International Conference, ESWC 2019, Portoro ˇz, Slovenia, June 2–6, 2019, Proceedings 16 . Springer, 2019, pp. 459–474

    2019

  79. [87]

    Kg-bert: Bert for knowledge graph completion,

    L. Yao, C. Mao, and Y . Luo, “Kg-bert: Bert for knowledge graph completion,” arXiv preprint arXiv:1909.03193 , 2019

    1909 arXiv

  80. [88]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  81. [89]

    Diffusion recommender model,

    W. Wang, Y . Xu, F. Feng, X. Lin, X. He, and T.-S. Chua, “Diffusion recommender model,” in Proceedings of the 46th International ACM SI- GIR Conference on Research and Development in Information Retrieval, 2023, pp. 832–841

    2023

  82. [90]

    Efficient large-scale multi-modal classification,

    D. Kiela, E. Grave, A. Joulin, and T. Mikolov, “Efficient large-scale multi-modal classification,” in AAAI, vol. 32, no. 1, 2018

    2018

  83. [91]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008

    2008

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.