Pith. sign in

REVIEW 3 major objections 10 minor 21 references

A Survey on Knowledge Graph Structure and Knowledge Graph Embeddings

T0 review · 3 major / 10 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey synthesizes evidence that a knowledge graph's structure — especially node degree — systematically shapes how well embedding models predict links and which hyperparameters work best.

desk verdict Useful first survey of KG structure–KGEM performance relationships, but the evidence tables overstate the authors' own studies and need reconciling with the text before I'd trust them. read the letter →

arxiv 2412.10092 v1 pith:4QONHAPY submitted 2024-12-13 cs.LG

Jeffrey Sardina , John D. Kelleher , Declan O'Sullivan This is my paper · ORCID
classification cs.LG
keywords KnowledgeGraphsGraphEmbeddingsLinkPredictionStructureNodeDegreeRelationshipFrequencyHyperparameterOptimizationSurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper assembles the published literature on how Knowledge Graph Embedding Models (KGEMs) behave on the link prediction task as a function of graph structure. Its central message is that node degree and relationship frequency are recurring, documented drivers of KGEM performance: high-degree nodes and frequent relations are learned reliably, while low-degree nodes and rare relations are predicted poorly. It also argues that benchmark datasets such as FB15k and WN18 exaggerate KGEM quality because their test sets overrepresent high-degree entities. The survey further collates evidence that optimal hyperparameters — negative samplers, loss functions, optimizers — depend on both the KG and the KGEM. If this aggregated picture is correct, structural metrics deserve a standard place in KGEM evaluation and benchmark design.

What carries the argument

The central organizing device is a taxonomy of four frequency-based structural metrics: node degree, relationship frequency, node-relationship co-frequency, and node-node co-frequency. These metrics are the lens through which the reviewed studies connect graph structure to link prediction performance and hyperparameter choice. The survey also relies on two synthesis tables (Tables I and II) that map each study to the structural metrics and hyperparameters it evaluated, making cross-study patterns explicit.

What would settle it

A controlled experiment that retrains TransE and ComplEx on FB15k-237 with degree-stratified test sets and finds no material performance gap between high- and low-degree entities would falsify the survey's central conclusion that node degree drives link prediction performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that it is the first comprehensive survey to aggregate established relationships between Knowledge Graph structure and Knowledge Graph Embedding Models. On the evidence it reviews, the four frequency-based metrics — node degree, relationship frequency, node-relationship co-frequency, and node-node co-frequency — are the most consistently documented structural drivers of link prediction performance. Higher-degree nodes are learned and predicted better, lower-frequency relations are learned less reliably, and these effects persist across models, datasets, and domains including biomedical KGs. The survey also aggregates findings that hyperparameter preference is dependent on both the KG and the KGEM, with negative sampler choice and loss function showing clear structural sensitivity. A further aggregated finding is that common benchmark KGs exhibit extreme degree and frequency skew, so reported KGEM performance can be inflated by models that learn only a few high-degree entities well.

Load-bearing premise

The survey's conclusions depend on the reviewed studies being a complete and unbiased sample of the relevant literature, but the paper provides no systematic search protocol and includes two self-authored studies.

Editorial extensions

If this is right

  • KGEM evaluation should report and control for node-degree distribution, since high-degree test entities can mask poor performance on the long tail.
  • Benchmark design should move toward structurally diverse or structurally controlled KGs, so results are not artifacts of degree skew.
  • Hyperparameter optimization should be conducted per KG-KGEM pair, as optimal samplers and losses vary with graph connectivity.
  • Low-frequency relations and low-degree nodes are the frontier where KGEM performance is weakest; improvements there show up more clearly under stratified evaluation.
  • Existing published KGEM comparisons may need to be re-read as partially measuring degree utilization rather than reasoning quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the survey's central claim would be to retrain a standard KGEM on degree-stratified splits of FB15k-237: if MRR on low-degree entities is not markedly worse, the degree-bias effect is overstated.
  • The same degree-bias lens could be applied to graph neural network link predictors or language-model-based KG completion, which the survey does not cover, to see whether the structural effect generalizes beyond the KGEM families reviewed.
  • The pattern the survey documents — high-frequency items dominate predictions — resembles popularity bias in recommender systems, suggesting that debiasing strategies from that field (e.g., inverse propensity weighting) could transfer to KGEM evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 10 minor

Summary. This paper surveys the literature on how knowledge graph structure affects knowledge graph embedding models (KGEMs) and on how hyperparameter choices interact with structure and performance. It identifies four frequency-based structural metrics (node degree, relation frequency, node-relation co-frequency, node-node co-frequency), reviews ten studies in Section II and seven in Section III, and presents two evidence tables (Tables I and II) summarizing the structural metrics used and the hyperparameters evaluated. The paper concludes with open research directions. The authors claim this is the first comprehensive survey of established relationships between KG structure, KGEM hyperparameters, and link prediction performance.

Significance. The survey addresses a real gap: prior work is scattered across system-oriented papers and benchmarks. The in-text summaries of Rossi et al. (2020), Bonner et al. (2022), Kotnis et al. (2017), and Mohamed et al. (2020) are faithful and helpful. The paper's explicit acknowledgment of limitations in its own earlier studies is commendable. If the evidence tables are corrected, the survey will be a valuable resource for researchers selecting models and hyperparameters and for developers of structure-controlled benchmarks. However, the current inconsistency between Table II's annotations and the text's disclaimers for the authors' own studies undermines confidence in the survey's central claim of documenting established relationships.

major comments (3)
  1. [Table II and Section III.5] The table key defines only '**', '*', and '.' as annotations, but the row for Sardina et al. (2024) contains '***' and '****' in every column. These symbols are undefined, and the row is the most prominent in the table. More importantly, Section III.5 explicitly states that this work 'does not result as yet in direct evidence for any specific hyperparameter influence on KGEM performance,' yet the table's '***' presumably means something stronger than 'directly evaluated' (which is '**'). This is an internal contradiction that directly affects the survey's evidence inventory. Please define all symbols and re-annotate the row to match the text.
  2. [Section II.8] The text states that the Lasso model in Sardina et al. (2022) was trained on all data without a train-test split, which means the reported accuracy is an in-sample correlation. The authors then write that they 'were able to provide evidence for a direct link between their chosen structural features and link prediction performance.' This overstates the evidential weight. In-sample fits provide correlational evidence, not predictive or causal evidence. Please rephrase and adjust Table I's annotation for this row accordingly.
  3. [Sections I and IV] The claim to be 'the first comprehensive survey' is not supported by any description of the search methodology (e.g., databases, query strings, inclusion/exclusion criteria) or of the screening process. Given that two of the eleven entries in Table I and one of the seven entries in Table II are the authors' own prior works, the absence of a systematic protocol makes the comprehensiveness claim unverifiable and raises concerns about selection bias. Please either add a methodology section or soften the claim to 'a survey of selected studies.'
minor comments (10)
  1. [Section III.2] The heading 'Sameh et al. (2019)' is incorrect; the cited work is Mohamed et al. (2019) (S.K. Mohamed, V. Nováček, P.Y. Vandenbussche, E. Muñoz). Please correct the heading and Table II row.
  2. [Section III.2] The second category of losses is labeled 'Pointwise losses' but the text describes pairwise losses; it should be 'Pairwise losses.'
  3. [Section III.2] The sentence 'They con Binary Cross Entropy Loss and Negative Log Softmax Loss' is missing the word 'consider'; please fix.
  4. [Section III.2] The phrase 'KGEM scouring functions' should be 'KGEM scoring functions.'
  5. [Section III.2] The years for the Jain and Kadlec references are swapped: the text refers to 'Jain et al. (2017)' and 'Kadlec et al. (2020),' but reference [8] is Jain et al. (2020) and reference [9] is Kadlec et al. (2017). Please correct these citations.
  6. [Section I] The phrase 'out fo the scope' should be 'out of the scope.'
  7. [Section II.8] The phrases 'the ration of sink nodes to triple' and 'the ration of repeat nodes to triples' should be 'the ratio of sink nodes to triples' and 'the ratio of repeat nodes to triples.'
  8. [Section IV] The phrase 'we to conclude' should be 'we wish to conclude' or 'we conclude.'
  9. [Figure 1] Consider increasing the font size of the node labels, which are difficult to read in the current rendering.
  10. [Table I] The key uses periods for 'not mentioned'; consider using an explicit dash or 'N/A' for clarity, and ensure the abbreviations (Rel, Freq) are expanded in the caption.

Circularity Check

2 steps flagged · score 4.0 of 10

In-sample Lasso 'prediction' and self-cited table annotations overstate evidence, but the survey's central conclusion is independently supported.

  1. fitted input called prediction [Section II.8 (Sardina et al. 2022) and Table I]
    "It is important to note, however, that the Lasso model was trained on all of the data; in other words, we did not train-test split the Lasso model... Our results indicated we could predict KGEM performance given global KG structure in all cases with generally high, but variable, accuracy [19]."

    The 2022 study fits a Lasso regression to the full dataset and then reports that it can 'predict' KGEM performance and that the fitted coefficients 'provide evidence for a direct link' between structure and performance. Because the same data are used both to fit the Lasso and to evaluate its fit, the reported accuracy is an in-sample measure; the 'prediction' is forced by construction and cannot independently establish a predictive relationship. The survey nevertheless enters this study in Table I as evidence that structural features characterize KGs and link prediction, giving the in-sample correlation more evidentiary weight than it supports.

  2. other [Section II.9, Section III.5, Tables I and II]
    "we did not perform an ablation study to explore which structural features are important to KGEM performance, which means we cannot yet make claims about which (if any) structural features were relevant to KGEM performance... it does not result as yet in direct evidence for any specific hyperparameter influence on KGEM performance."

    The paper's own text disclaims direct evidence for specific structural and hyperparameter effects, yet Tables I and II assign the authors' self-cited 2024 paper the fullest annotations, including stars that are not even defined by the table key. This is not a formal derivation reduction, but it is a self-citation over-crediting pattern: the strongest-looking rows in the evidence tables come from the authors' own work and directly contradict the in-text limitations. The central survey claim does not collapse because independent studies such as Rossi, Bonner, Kotnis, and Mohamed carry the structural conclusion, but a reader cannot tell from the tables which entries are direct evidence and which are in-sample correlations or simulations.

full rationale

The paper is a literature survey, so there is no formal derivation chain to collapse. Its central claim—that KG structure, including node degree, relationship frequency, and co-frequency metrics, is a documented determinant of link prediction performance and hyperparameter choice—rests primarily on independent studies such as Rossi et al. (2020), Bonner et al. (2022), Kotnis et al. (2017), Mohamed et al. (2020), Sadeghi et al. (2021), Ruffinelli et al. (2020), and Ali et al. (2022). Those are external controlled evaluations and therefore break any circularity. Two issues lower the score below a clean non-finding. First, the survey reports the authors' 2022 Lasso study as 'predicting' KGEM performance while explicitly stating the model was trained on all data with no train-test split; that accuracy is an in-sample fit, not a prediction, so Table I uses statistically forced evidence. Second, Tables I and II annotate the authors' 2024 TWIG paper with the fullest stars even though the same text says no ablation was done and no direct claim about specific structural or hyperparameter effects can yet be made. This is self-citation over-crediting and an internal inconsistency in the evidence tables. It does not, however, force the survey's conclusion, which would survive removal of both self-cited rows; hence the score is 4 rather than higher.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey rests on the selection and faithful summarization of roughly 20 external papers. No code, data, or formal methods are used. The main implicit input is the assumption that the chosen papers are sufficient and representative.

assumptions (2)
  • domain assumption The set of studies in Tables I and II is representative of the relevant literature.
    The survey claims comprehensiveness but provides no systematic search strategy, so the selection may be incomplete or biased.
  • domain assumption The results of the cited studies are accurately summarized.
    The survey's conclusions depend on the fidelity of its summaries; several typographical and attribution errors suggest this assumption is imperfect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey on Knowledge Graph Structure and Knowledge Graph Embeddings." pith.science (2026). https://pith.science/paper/4QONHAPY

@misc{pith2026241210092,
  author       = {Pith},
  title        = {Pith review of: A Survey on Knowledge Graph Structure and Knowledge Graph Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4QONHAPY}},
  note         = {Machine review of arXiv:2412.10092}
}
read the original abstract

Knowledge Graphs (KGs) and their machine learning counterpart, Knowledge Graph Embedding Models (KGEMs), have seen ever-increasing use in a wide variety of academic and applied settings. In particular, KGEMs are typically applied to KGs to solve the link prediction task; i.e. to predict new facts in the domain of a KG based on existing, observed facts. While this approach has been shown substantial power in many end-use cases, it remains incompletely characterised in terms of how KGEMs react differently to KG structure. This is of particular concern in light of recent studies showing that KG structure can be a significant source of bias as well as partially determinant of overall KGEM performance. This paper seeks to address this gap in the state-of-the-art. This paper provides, to the authors' knowledge, the first comprehensive survey exploring established relationships of Knowledge Graph Embedding Models and Graph structure in the literature. It is the hope of the authors that this work will inspire further studies in this area, and contribute to a more holistic understanding of KGs, KGEMs, and the link prediction task.

Figures

Figures reproduced from arXiv: 2412.10092 by the authors.

Figure 1
Figure 1. An overview of hyperparameter and graph structure influence on KGEM performance on the link prediction task, annotated by evidence in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages

  1. [1]

    IEEE Transactions on Pat- tern Analysis and Machine Intelligence 44(12), 8825–8845 (2022)

    Ali, M., Berrendorf, M., Hoyt, C.T., Vermue, L., Galkin, M., Shar- ifzadeh, S., Fischer, A., Tresp, V ., Lehmann, J.: Bringing light into the dark: A large-scale evaluation of knowledge graph embed- ding models under a unified framework. IEEE Transactions on Pat- tern Analysis and Machine Intelligence 44(12), 8825–8845 (2022). https://doi.org/10.1109/TPAM...

    arXiv 2022

  2. [2]

    Journal of Machine Learning Research 22(82), 1–6 (2021), http://jmlr.org/papers/v22/20-825.html

    Ali, M., Berrendorf, M., Hoyt, C.T., Vermue, L., Sharifzadeh, S., Tresp, V ., Lehmann, J.: PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings. Journal of Machine Learning Research 22(82), 1–6 (2021), http://jmlr.org/papers/v22/20-825.html

    work page 2021

  3. [3]

    Briefings in Bioinformatics 23(5) (07 2022)

    Bonner, S., Kirik, U., Engkvist, O., Tang, J., Barrett, I.P.: Im- plications of topological imbalance for representation learning on biomedical knowledge graphs. Briefings in Bioinformatics 23(5) (07 2022). https://doi.org/10.1093/bib/bbac279, https://doi.org/10.1093/bib/ bbac279, bbac279

    doi:10.1093/bib/bbac279 2022

  4. [4]

    In: International Conference on Neural-Symbolic Learning and Reasoning

    Dave, B., Christou, A., Shimizu, C.: Towards understanding the impact of graph structure on knowledge graph embeddings. In: International Conference on Neural-Symbolic Learning and Reasoning. pp. 41–50. Springer (2024)

    work page 2024

  5. [5]

    arXiv preprint arXiv:2203.09219 (2022)

    D ¨orpinghaus, J., Weil, V ., D ¨uing, C., Sommer, M.W.: Centrality mea- sures in multi-layer knowledge graphs. arXiv preprint arXiv:2203.09219 (2022)

    arXiv 2022

  6. [6]

    ACM Comput

    Hogan, A., Blomqvist, E., Cochez, M., D’amato, C., Melo, G.D., Gutierrez, C., Kirrane, S., Gayo, J.E.L., Navigli, R., Neumaier, S., Ngomo, A.C.N., Polleres, A., Rashid, S.M., Rula, A., Schmelzeisen, L., Sequeda, J., Staab, S., Zimmermann, A.: Knowledge graphs. ACM Comput. Surv. 54(4) (7 2021). https://doi.org/10.1145/3447772, https: //doi.org/10.1145/3447772

    doi:10.1145/3447772 2021

  7. [7]

    In: European Semantic Web Conference

    Hubert, N., Monnin, P., d’Aquin, M., Monticolo, D., Brun, A.: Pygraft: Configurable generation of synthetic schemas and knowledge graphs at your fingertips. In: European Semantic Web Conference. pp. 3–20. Springer (2024)

    work page 2024

  8. [8]

    arXiv preprint arXiv:2005.00804 (2020)

    Jain, P., Rathi, S., Chakrabarti, S., et al.: Knowledge base completion: Baseline strikes back (again). arXiv preprint arXiv:2005.00804 (2020)

    arXiv 2020

Show all 21 references
  1. [9]

    arXiv preprint arXiv:1705.10744 (2017)

    Kadlec, R., Bajgar, O., Kleindienst, J.: Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744 (2017)

    2017 arXiv

  2. [10]

    Kotnis, B., Nastase, V .: Analysis of the impact of negative sampling on link prediction in knowledge graphs (08 2017)

    2017

  3. [11]

    In: Conference on Uncertainty in Artificial Intelligence

    Mohamed, A., Parambath, S., Kaoudi, Z., Aboulnaga, A.: Popularity agnostic evaluation of knowledge graph embeddings. In: Conference on Uncertainty in Artificial Intelligence. pp. 1059–1068. PMLR (2020)

    2020

  4. [12]

    In: DL4KG@ESWC (2019)

    Mohamed, S.K., Nov ´acek, V ., Vandenbussche, P.Y ., Mu˜noz, E.: Loss functions in knowledge graph embedding models. In: DL4KG@ESWC (2019)

    2019

  5. [13]

    Proceedings of the IEEE104(1), 11–33 (2016)

    Nickel, M., Murphy, K., Tresp, V ., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE104(1), 11–33 (2016). https://doi.org/10.1109/JPROC.2015.2483592

    2016

  6. [14]

    ACM Transactions on Knowledge Discovery from Data 15, 1–49 (01 2021)

    Rossi, A., Barbosa, D., Firmani, D., Matinata, A., Merialdo, P.: Knowl- edge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data 15, 1–49 (01 2021). https://doi.org/10.1145/3424672

    2021 doi

  7. [15]

    Rossi, A., Matinata, A.: Knowledge graph embeddings: Are relation- learning models learning relations? In: EDBT/ICDT Workshops. vol. 2578 (2020)

    2020

  8. [16]

    In: ICLR (2020)

    Ruffinelli, D., Broscheit, S., Gemulla, R.: You can teach an old dog new tricks! on training knowledge graph embeddings. In: ICLR (2020)

    2020

  9. [17]

    In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases

    Sadeghi, A., Collarana, D., Graux, D., Lehmann, J.: Embedding knowl- edge graphs attentive to positional and centrality qualities. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 548–564. Springer (2021)

    2021

  10. [18]

    In: 2024 IEEE 18th International Conference on Semantic Computing (ICSC)

    Sardina, J., Kelleher, J.D., O’Sullivan, D.: Twig: Towards pre-hoc hy- perparameter optimisation and cross-graph generalisation via simulated kge models. In: 2024 IEEE 18th International Conference on Semantic Computing (ICSC). pp. 122–129. IEEE (2024)

    2024

  11. [19]

    In: SeWeBMeDA-2022@ESWC (2022)

    Sardina, J., O’Sullivan, D.: Structural characteristics of knowledge graphs determine the quality of knowledge graph embeddings across model and hyperparameter choices. In: SeWeBMeDA-2022@ESWC (2022)

    2022

  12. [20]

    IEEE Transac- tions on Knowledge and Data Engineering 29(12), 2724–2743 (2017)

    Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge graph em- bedding: A survey of approaches and applications. IEEE Transac- tions on Knowledge and Data Engineering 29(12), 2724–2743 (2017). https://doi.org/10.1109/TKDE.2017.2754499

    2017

  13. [21]

    GigaScience 13, giae001 (2024)

    Zietz, M., Himmelstein, D.S., Kloster, K., Williams, C., Nagle, M.W., Greene, C.S.: The probability of edge existence due to node degree: a baseline for network-based predictions. GigaScience 13, giae001 (2024)

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.