HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs

Rui Zhang; Yongkang Xiao

REVIEW 3 major objections 4 minor 51 references

HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that multimodal knowledge graph completion improves when a multimodal retriever narrows candidates and an LLM chooses the final entity, and reports consistent wins over prior methods on three benchmarks.

desk verdict A plausible generative pipeline for multimodal KGC, but the central Hits@1 gains are uninterpretable until the authors specify how the LLM's single-entity answer becomes a ranked list. read the letter →

arxiv 2506.00826 v2 pith:BAFB5YGU submitted 2025-06-01 cs.CL cs.AIcs.LG

Yongkang Xiao , Rui Zhang This is my paper

classification cs.CLcs.AIcs.LG

keywords multimodalknowledgegraphcompletionretrieval-augmentedgenerationlargelanguagemodelsmixtureofexpertsfusionrelation-awaregatingLoRAfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal knowledge graph completion (MMKGC) has mostly been a discriminative problem: models fuse image, text, and graph signals to score every candidate fact. The paper argues that this leaves reasoning capacity unused, and that a generative stage should make the final call. HERGC is that two-stage design: a retriever builds a compact candidate list from relation-aware multimodal embeddings, and a large language model picks the answer from the list using an instruction prompt, optionally with injected fused embeddings and LoRA fine-tuning. On MKG-W, MKG-Y, and DB15K, the paper reports the best overall MRR and Hits@k among the compared methods, with the largest gains concentrated in Hits@1. If the results hold, the practical lesson is that open-source LLMs can carry the disambiguation burden in MMKGC once retrieval recall is high enough.

What carries the argument

The load-bearing mechanism is the two-stage HERR-to-GLP pipeline. HERR's Mixture of Heterogeneous Experts (MoHE) applies gated top-k selection over two kinds of experts—simple linear whitening experts and complex block-hypercomplex (PHM) experts—so each modality is represented from multiple perspectives; the Relation-aware Gated Multimodal Unit (RaGMU) then computes a relation-modulated gate vector over the projected modality embeddings and produces a fused embedding per entity. TuckER scores candidate triples from these fused embeddings and returns a compact candidate list. GLP consumes that list as an instruction prompt with entity descriptions and, for fine-tuned open-source LLMs, the fused embeddings themselves, turning final completion into a constrained multiple-choice-style generation.

What would settle it

Re-run the three benchmark evaluations with two explicit ranking protocols—the LLM's chosen entity placed at rank 1 with all other candidates keeping their retriever order, versus the chosen entity inserted at the position implied by the retriever score—and compare MRR, Hits@1, Hits@3, and Hits@10. If the numbers move materially, the claimed superiority is an artifact of the placement rule; if they are stable, the ranking-protocol concern is settled.

Watch

Extended reading notes

Core claim

HERGC's central claim is that the generative paradigm can be brought to multimodal knowledge graph completion and outperforms closed-world discriminative models. The Heterogeneous Experts Representation Retriever (HERR) enriches each entity's visual, textual, and structural embeddings with simple and block-hypercomplex experts, fuses them with a relation-aware gated unit, and scores triples with TuckER to produce a ranked candidate list. The Generative LLM Predictor (GLP) then reformulates the incomplete triple as a natural-language question and asks the LLM to choose one entity from the top-k candidates; open-source models are adapted by LoRA fine-tuning with fused embeddings injected through an adapter. The paper reports that this pipeline beats all compared baselines on three standard benchmarks, and that the LLaMA-3 variant with fine-tuning outperforms the GPT-4 in-context variant on most metrics, which the authors attribute to the injected multimodal embeddings supplementing the LLM's pretrained knowledge.

Load-bearing premise

The headline ranking metrics assume a conversion from the LLM's single-entity answer to a ranked candidate list that the paper never describes, so the reported MRR and Hits@k gains depend on that unstated placement rule.

Editorial extensions

If this is right

On all three benchmarks, HERGC's best configuration (LLaMA-3 with LoRA and injected fused embeddings) reports MRR of 39.12, 39.82, and 40.95 on MKG-W, MKG-Y, and DB15K, above every compared baseline.
Removing any single modality—image, text, or structure—lowers performance, so the multimodal fusion and relation-aware gating carry part of the gain rather than the LLM alone.
Replacing TuckER scoring with TransE, RotatE, or ComplEx in the retriever degrades results markedly, so the scoring function inside the retriever is a load-bearing choice.
Graph sparsity hurts more than noise or missing modality inputs: removing 30% of training triples drops MRR by 15.4%, 25.1%, and 11.8% across the three datasets, suggesting retrieval quality is the fragile link.
Candidate-set size k=20 captures most of the benefit; larger candidate sets add inference time with marginal metric gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

This design implies that future gains in MMKGC will likely come from raising retriever recall rather than from stronger LLMs; an LLM cannot pick an answer that the candidate list omits, so recall of HERR caps the whole pipeline.
The small gap between LLaMA-3 and LLaMA-3-Vision suggests that images in the current benchmarks carry little relation-discriminative signal; a testable follow-up would build or select an MMKG where images are directly tied to relation semantics (flags, logos, species photos) and check whether a vision-capable predictor then pulls ahead.
Because the ranking protocol that converts the LLM's single-entity output into MRR and Hits@k is not specified, reproducing the headline numbers requires fixing that protocol; future work should state whether the chosen entity is placed at rank 1 with the retriever order preserved for the rest, since Hits@3 and Hits@10 can shift under different placement rules.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is sound and the paper is worth engaging: this is a retriever-plus-generative-LLM design for multimodal knowledge graph completion, and the combination of a heterogeneous-expert multimodal retriever with a flexible LLM predictor (in-context or LoRA-tuned) is genuinely new in this subfield. The ablation studies are thorough, the robustness experiments are useful, and the appendix's candid discussion of why vision-language models did not help shows honest engagement. The 'first generative MMKGC' claim is plausible, though I would not bet on it without a broader literature check.

The soft spot is real and load-bearing: the evaluation protocol for the generative predictor is never specified. The prompt in Table 5 asks the LLM to answer exactly one entity chosen from the top-20 candidates. Appendix A.4 defines MRR and Hits@k as if the model scores every candidate and produces a ranked list, but no mapping from the LLM's single output to that ranked list is given. Is the selected entity placed at rank 1 with the remaining candidates in retriever order? Are non-retrieved entities assigned rank 21 or excluded? If the correct entity is not in the retrieved set, what rank does it get? This matters because the headline wins over MCKGC are concentrated in Hits@1 (e.g., 33.65 vs. 31.32 on MKG-W). A rule that always places the LLM's pick at rank 1 would turn pick accuracy into Hits@1 gains by construction. The authors need to write down the exact conversion, and ideally report a separate "generative accuracy within top-20" metric so the two are not conflated.

Two other weaknesses are less severe but still limit confidence: baselines are copied from other papers without re-running under a unified protocol, and no error bars or significance tests are reported. These are common in this literature, but they matter more here because the evaluation protocol is opaque. No code or data is released, which further blocks independent verification.

There is no circularity problem—the test sets are genuinely held out—and the use of the validation set for LLM fine-tuning is explained honestly. This is a fixable paper. The evaluation gap can be closed in a revision, and if the mapping is what I suspect (selected entity at rank 1), the results may still hold but should be presented as a different metric.

My take: send it to peer review, but flag the evaluation protocol as a major-revision issue. The paper deserves referee time because the idea is timely and the empirical evidence, once interpretable, could be a solid state-of-the-art result for the MMKGC community. I would not cite the numbers in my own work until the protocol is clarified.

Referee Report

3 major / 4 minor

Summary. The paper proposes HERGC, a retrieval-augmented generative framework for multimodal knowledge graph completion (MMKGC). It consists of a Heterogeneous Experts Representation Retriever (HERR), which fuses visual, textual, and structural embeddings via a mixture of heterogeneous experts and a relation-aware gated fusion unit, and a Generative LLM Predictor (GLP) that selects the correct entity from a retrieved candidate set using either in-context learning with closed-source LLMs or LoRA fine-tuning of open-source LLMs. Experiments on MKG-W, MKG-Y, and DB15K report state-of-the-art MRR and Hits@k, with ablations showing contributions of each modality, retriever component, and score function, as well as robustness to noisy and missing inputs.

Significance. If the empirical claims hold, HERGC is a meaningful contribution: it is, to the authors' knowledge, the first generative MMKGC framework, and it demonstrates that LLM-based reranking over a multimodal retriever can outperform discriminative MMKGC models on standard benchmarks. The paper provides detailed ablation studies, a candidate-size analysis, and a complexity breakdown, and it supports both API-based and locally fine-tuned LLM predictors, which is practically useful. However, the central performance claim depends on an incompletely specified evaluation protocol for converting the LLM's single-entity output into ranking metrics, and the baseline numbers are taken from prior papers without re-running under a unified protocol. These gaps currently limit the interpretability and reproducibility of the headline results.

major comments (3)

[§4.3, §5.2, Appendix A.4] The paper must specify the exact ranking protocol, including the treatment of correct entities outside the retrieved top-k, and justify that the reported MRR/Hits@k values follow from that protocol.
[§5.1, Baseline Methods] The paper should provide error bars or significance tests, and either re-run baselines or provide a detailed comparability argument.
[Appendix A.3] The text should be corrected and the authors should explicitly state which splits were used for retriever training, GLP fine-tuning, and model selection.

minor comments (4)

[§3, Preliminary] In the definition of T, the condition "t∈ R" should be "r∈ R"; the current text is a typo that could confuse readers.
[Figure 2] The figure label "Knowledge Graph (Structure Infrmation)" contains a typo ("Infrmation").
[§5.5, Table 3] The table header reads "Llama-3.2-3B" while the surrounding text and Table 1 refer to "LLaMA-3-8B" and "Llama-3.2-3B" inconsistently; unify the model naming throughout the paper (e.g., use LLaMA consistently).
[Appendix A.3] The description of the GLP fine-tuning data construction says the correct entity "often receives a high score and is consistently ranked first" in the retriever output, and therefore the validation set is used to prevent shortcut learning; this is an important detail and should be stated earlier in the main text, since it affects how the LLM predictor is trained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central results are empirical comparisons on held-out test sets; the one alarming A.3 sentence about training on test sets is contradicted within the paper and is a protocol risk, not a circular derivation.

full rationale

The paper's central claim (Section 5.2, Table 1) is an empirical comparison on the original test splits of MKG-W, MKG-Y, and DB15K. The retriever HERR is trained with the binary cross-entropy objective in Eq. (10) on training triples, and the GLP is LoRA fine-tuned only on a split of the validation set, as stated in Appendix A.3: 'The test sets remain identical to the original benchmarks.' No parameter is fitted to the test targets, so there is no fitted-input-called-prediction circularity at the equation level. The only self-citation, Xiao et al. (2024), is a related biomedical link-prediction reference and is not load-bearing. One sentence in Appendix A.3 says 'We train the retriever HERR using the training and test sets,' which would be a fatal test-set leak if literal; however, Section 5.1 says the TuckER structural embeddings are trained 'on the training split,' and the same appendix immediately says 'since the retriever is trained on the training set, the correct entity often receives a high score,' indicating that the earlier phrase is a typo. Separately, Appendix A.4 defines MRR and Hits@k as if 'the model scores every candidate entity, producing a ranked list,' while the GLP prompt in Table 5 instructs the model to 'only need to answer one entity' from the top-20 candidates; the paper never specifies how the single output is converted into a ranked list. This is a genuine evaluation-protocol ambiguity that threatens reproducibility and comparability with baselines, but it is not circular: even under the most favorable protocol, the reported numbers are empirical accuracies on unseen test triples rather than quantities forced to equal the model's own inputs. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical performance claim, so the ledger mostly captures the modeling choices and evaluation assumptions that the numbers depend on. The most fragile items are the use of reported baselines, the validation-set fine-tuning, and the undefined re-ranking protocol.

free parameters (6)

candidate set size k = 20
Selected from {10, 20, 30, 40} based on validation performance; directly controls the ceiling for LLM re-ranking and time cost.
LoRA rank r = 64
Hyperparameter for LoRA fine-tuning of LLaMA-3-8B; not central to the claim but affects fine-tuning behavior.
LoRA alpha = 16
LoRA scaling hyperparameter used in the fine-tuning setup.
retriever embedding dimension = selected from {200, 300, 400}
Tunes the representation capacity of the retriever; no sensitivity analysis is shown.
retriever learning rate = selected from {0.005, 0.001, 0.0005}
Standard training hyperparameter for the retriever.
number of experts = 2 simple + 2 complex
Design choice for the MoHE module; the paper does not vary this configuration.

assumptions (5)

domain assumption TuckER tensor factorization is an effective scoring function for multimodal triple plausibility.
The retriever uses TuckER as the scoring function (Eq. 9) and the paper only justifies it empirically by comparing against TransE, RotatE, and ComplEx in the ablation.
domain assumption The three benchmark datasets and the filtered ranking protocol are standard and fair for MMKGC evaluation.
Used in Sections 5.1 and A.4; the paper relies on these to make claims of superiority without re-running baselines.
domain assumption LLMs can reason about relational facts from prompts better than discriminative models when given a constrained candidate set.
The entire GLP component rests on this; the paper tests it empirically but does not provide a formal or mechanistic justification.
domain assumption Fine-tuning the LLM on the validation set does not bias test performance.
Section A.3 describes using validation triples to fine-tune the LLM; this assumes the validation set is an unbiased source of training signal, which is not standard practice for hyperparameter tuning.
ad hoc to paper The unspecified rule for converting the LLM's single answer into ranking metrics yields correct MRR and Hits@k values.
No description is given for how the single-entity LLM output becomes a ranked list; this is a hidden assumption in Section 5.2 that the reported metrics depend on.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs." pith.science (2026). https://pith.science/paper/BAFB5YGU

@misc{pith2026250600826,
  author       = {Pith},
  title        = {Pith review of: HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BAFB5YGU}},
  note         = {Machine review of arXiv:2506.00826}
}

read the original abstract

Multimodal knowledge graphs (MMKGs) enrich traditional knowledge graphs (KGs) by incorporating diverse modalities such as images and text. multimodal knowledge graph completion (MMKGC) seeks to exploit these heterogeneous signals to infer missing facts, thereby mitigating the intrinsic incompleteness of MMKGs. Existing MMKGC methods typically leverage only the information contained in the MMKGs under the closed-world assumption and adopt discriminative training objectives, which limits their reasoning capacity during completion. Recent large language models (LLMs), empowered by massive parameter scales and pretraining on vast corpora, have demonstrated strong reasoning abilities across various tasks. However, their potential in MMKGC remains largely unexplored. To bridge this gap, we propose HERGC, a flexible Heterogeneous Experts Representation and Generative Completion framework for MMKGs. HERGC first deploys a Heterogeneous Experts Representation Retriever that enriches and fuses multimodal information and retrieves a compact candidate set for each incomplete triple. It then uses a Generative LLM Predictor, implemented via either in-context learning or lightweight fine-tuning, to accurately identify the correct answer from these candidates. Extensive experiments on three standard MMKG benchmarks demonstrate HERGC's effectiveness and robustness, achieving superior performance over existing methods.

Figures

Figures reproduced from arXiv: 2506.00826 by the authors.

**Figure 1.** Comparison between (a) HERGC and (b) existing methods. Unlike prior methods, HERGC leverages the knowledge and reasoning capabilities of LLMs to generate predictions. Like traditional KGs, MMKGs also suffer from incompleteness, often due to missing facts in the underlying data sources or facts that have yet to be discovered by humans. Unlike unimodal knowledge graph completion (KGC), which primarily leverages the … view at source ↗

**Figure 2.** Overview of the HERGC framework BERT and VGG, respectively, yielding the initial modality-specific embeddings eD and eV. Structure Embedding. To encode structural information from the MMKG, we adopt TuckER (Balaževic et al. ´ , 2019), a representative KGE model that learns entity and relation embeddings via tensor factorization. TuckER is also employed as the scoring function in the retrieval module. The resulti… view at source ↗

**Figure 3.** t-SNE data visualization of entity representa [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: Changes in MRR metrics of HERGC on three [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 4.** Figure 4: shows the trends in time consumption and ranking-based metrics as k varies. As expected, inference time increases approximately linearly with larger k values due to longer prompts constructed from larger candidate sets, which has more tokens in the prompt. However, th…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 27 canonical work pages

[1]

John Arevalo, Thamar Solorio, Manuel Montes-y G \'o mez, and Fabio A Gonz \'a lez. 2017. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992

arXiv 2017
[2]

Ivana Bala z evi \'c , Carl Allen, and Timothy M Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590

arXiv 2019
[3]

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26

2013
[4]

Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. 2022. Otkge: Multi-modal knowledge graph embeddings via optimal transport. Advances in neural information processing systems, 35:39090--39102

work page 2022
[5]

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and 1 others. 2024 a . Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42--61

work page 2024
[6]

Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, and 1 others. 2024 b . Knowledge graphs meet multi-modal learning: A comprehensive survey. arXiv preprint arXiv:2402.05391

arXiv 2024
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

2019
[8]

Yuxiao Gao, Fuwei Zhang, Zhao Zhang, Xiaoshuang Min, and Fuzhen Zhuang. 2025. Mixed-curvature multi-modal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 11699--11707

work page 2025

Show all 51 references

[9]

Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. 2021. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33(2):494--514

2021
[10]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. 2017. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . Preprint, arXiv:1412.6980

2017 arXiv
[11]

Jaejun Lee, Chanyoung Chung, Hochang Lee, Sungho Jo, and Joyce Whang. 2023. Vista: Visual-textual knowledge graph representation learning. In Findings of the association for computational linguistics: EMNLP 2023, pages 7314--7328

2023
[12]

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, S \"o ren Auer, and 1 others. 2015. Dbpedia--a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(...

2015
[13]

u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Informati...

2020
[14]

Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, and Chunxiao Xing. 2023. Imf: Interactive multimodal fusion model for link prediction. In Proceedings of the ACM Web Conference 2023, pages 2572--2580

2023
[15]

Xuan Lin, Zhe Quan, Zhi-Jie Wang, Tengfei Ma, and Xiangxiang Zeng. 2020. Kgnn: Knowledge graph neural network for drug-drug interaction prediction. In IJCAI, volume 380, pages 2739--2745

2020
[16]

Yang Liu, Xiaobin Tian, Zequn Sun, and Wei Hu. 2024. Finetuning generative large language models with discrimination instructions for knowledge graph completion. In International Semantic Web Conference, pages 199--217. Springer

2024
[17]

Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. 2019. Mmkg: multi-modal knowledge graphs. In The semantic web: 16th international conference, ESWC 2019, portoro z , Slovenia, June 2--6, 2019, proceedings 16 , pages 459--474. Springer

2019
[18]

Xinyu Lu, Lifang Wang, Zejun Jiang, Shichang He, and Shizhong Liu. 2022. Mmkrl: A robust embedding approach for multi-modal knowledge graph representation learning. Applied Intelligence, pages 1--18

2022
[19]

Hatem Mousselly-Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018. A multimodal translation-based approach for knowledge graph representation learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 225--234

2018
[20]

Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11--33

2015
[21]

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7):3580--3599

2024
[22]

Jay Pujara, Eriq Augustine, and Lise Getoor. 2017. Sparsity and noise: Where knowledge graph embeddings fall short. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 1751--1756

2017
[23]

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3--7, 2018, proc...

2018
[24]

Bin Shang, Yinliang Zhao, Jun Liu, and Di Wang. 2024. Lafa: Multimodal knowledge graph completion with link aware fusion and aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8957--8965

2024
[25]

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

2017 arXiv
[26]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

2014 arXiv
[27]

Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697--706

2007
[28]

Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. 2020. Multi-modal knowledge graphs for recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 1405--1414

2020
[29]

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197

2019 arXiv
[30]

Yun Tang, Jing Huang, Guangtao Wang, Xiaodong He, and Bowen Zhou. 2019. Orthogonal relation transforms with graph context modeling for knowledge graph embedding. arXiv preprint arXiv:1911.04910

2019 arXiv
[31]

Th \'e o Trouillon, Johannes Welbl, Sebastian Riedel, \'E ric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071--2080. PMLR

2016
[32]

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019. Composition-based multi-relational graph convolutional networks. arXiv preprint arXiv:1911.03082

2019 arXiv
[33]

Denny Vrande c i \'c and Markus Kr \"o tzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78--85

2014
[34]

Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. arXiv preprint arXiv:2203.02167

2022 arXiv
[35]

Meng Wang, Sen Wang, Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. 2021. Is visual context really helpful for knowledge graph? a representation learning perspective. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2735--2743

2021
[36]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019 a . Kgat: Knowledge graph attention network for recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 950--958

2019
[37]

Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. 2019 b . Multimodal data enhanced representation learning for knowledge graphs. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE

2019
[38]

Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.580 KICGPT : Large language model with knowledge in context for knowledge graph completion . In Findings of the Association for Computational Linguistics: EMNLP 2023, page...

2023 doi
[39]

Yongkang Xiao, Sinian Zhang, Huixue Zhou, Mingchen Li, Han Yang, and Rui Zhang. 2024. Fuselinker: Leveraging llm’s pre-trained text embeddings and domain knowledge to enhance gnn-based link prediction on biomedical knowledge graphs. Journal of Biomedical Informatics, 158:104730

2024
[40]

Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Image-embodied knowledge representation learning. arXiv preprint arXiv:1609.07028

2016 arXiv
[41]

Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. 2022. Relation-enhanced negative sampling for multimodal knowledge graph completion. In Proceedings of the 30th ACM international conference on multimedia, pages 3857--3866

2022
[42]

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575

2014 arXiv
[43]

Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193

2019 arXiv
[44]

Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. 2021. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. arXiv preprint arXiv:2102.08597

2021 arXiv
[45]

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. 2025 a . Tokenization, fusion, and augmentation: Towards fine-grained multi-modal entity representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 3...

2025
[46]

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Wen Zhang, and Huajun Chen. 2024. Making large language models perform better in knowledge graph completion. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 233--242

2024
[47]

Yichi Zhang, Zhuo Chen, Lingbing Guo, yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. 2025 b . https://openreview.net/forum?id=ue1Tt3h1VC Multiple heads are better than one: Mixture of modality knowledge experts for entity representation learning . In The Thirteent...

2025
[48]

Yu Zhao, Xiangrui Cai, Yike Wu, Haiwei Zhang, Ying Zhang, Guoqing Zhao, and Ning Jiang. 2022. Mose: Modality split and ensemble for multimodal knowledge graph completion. arXiv preprint arXiv:2210.08821

2022 arXiv
[49]

Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, and Nicholas Jing Yuan. 2022. Multi-modal knowledge graph construction and application: A survey. IEEE Transactions on Knowledge and Data Engineering, 36(2):715--735

2022
[50]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[51]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

John Arevalo, Thamar Solorio, Manuel Montes-y G \'o mez, and Fabio A Gonz \'a lez. 2017. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992

arXiv 2017

[2] [2]

Ivana Bala z evi \'c , Carl Allen, and Timothy M Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590

arXiv 2019

[3] [3]

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26

2013

[4] [4]

Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. 2022. Otkge: Multi-modal knowledge graph embeddings via optimal transport. Advances in neural information processing systems, 35:39090--39102

work page 2022

[5] [5]

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and 1 others. 2024 a . Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42--61

work page 2024

[6] [6]

Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, and 1 others. 2024 b . Knowledge graphs meet multi-modal learning: A comprehensive survey. arXiv preprint arXiv:2402.05391

arXiv 2024

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186

2019

[8] [8]

Yuxiao Gao, Fuwei Zhang, Zhao Zhang, Xiaoshuang Min, and Fuzhen Zhuang. 2025. Mixed-curvature multi-modal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 11699--11707

work page 2025