Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Extending TWIG: Zero-Shot Predictive Hyperparameter Selection for KGEs based on Graph Structure

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that TWIG can predict ComplEx hyperparameter performance on knowledge graphs it has never seen, with zero-shot R2 up to 0.73.

desk verdict A genuinely new zero-shot evaluation of TWIG for KGE hyperparameter prediction, with positive but narrow evidence; the paper's own limitations and the raw-count feature scaling concern mean the headline claim is not yet secure for larger graphs. read the letter →

arxiv 2412.14801 v1 pith:PXNWIRDQ submitted 2024-12-19 cs.LG

Jeffrey Sardina , John D. Kelleher , Declan O'Sullivan This is my paper · ORCID
classification cs.LG
keywords knowledgegraphsgraphembeddingslinkpredictionhyperparameterselectionzero-shotlearningMRRstructureComplEx
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the hyperparameter settings of a knowledge-graph embedding model can be predicted before the model is ever trained on a new graph, using only the graph's structure. It extends the TWIG simulation approach to multiple knowledge graphs and tests it on the KGE model ComplEx. The central result is that TWIG trained on four graphs predicts ComplEx's mean reciprocal rank (MRR) on a held-out fifth graph for unseen hyperparameter settings, with zero-shot $R^2$ values between 0.54 and 0.73. If this holds generally, hyperparameter selection could become a pre-hoc step based on graph structure instead of an expensive grid search.

What carries the argument

TWIG (Topologically-Weighted Intelligence Generation) is the central object: a neural network with a hyperparameter-learning block, a graph-structure-learning block, and an integration block. It takes hyperparameters (negative sampler type, negatives count, loss function, margin, learning rate, embedding dimension, regularisation coefficient) and per-triple structural features (subject/object degree, predicate frequency, co-occurrence frequencies, and neighbourhood degree summaries) and outputs predicted ranks for link prediction queries; the predicted ranks are then aggregated into an MRR value. The machine-learning machinery carries the argument because it is the mechanism that lets structure alone predict performance.

What would settle it

Evaluate TWIG in the same zero-shot protocol but with FB15k-237 or WN18RR as the held-out graph; if the predicted-versus-true ComplEx MRR $R^2$ collapses toward zero or negative, the representativeness assumption fails. Alternatively, keep the same five graphs but simulate TransE instead of ComplEx; a near-zero $R^2$ would show the transfer effect is specific to ComplEx rather than a general property of KGEs.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that one TWIG model can simulate ComplEx's rank-based output across several knowledge graphs at once, and that this simulation transfers to graphs whose ComplEx outputs were never seen in training. With four graphs as training data, TWIG predicts ComplEx's MRR on the held-out fifth graph across the full hyperparameter grid at $R^2 = 0.54$-$0.73$; finetuning on 5% of the target graph's hyperparameter grid raises the range to $0.77$-$0.96$, and finetuning on 25% gives $0.86$-$0.99$. The paper takes this as evidence that hyperparameter preference is largely a function of KG structure, that the structural signal is domain-agnostic, and that pre-hoc hyperparameter optimisation is feasible.

Load-bearing premise

The load-bearing premise is that the five small knowledge graphs used for training are representative of graph structure broadly, and that ComplEx's response to structure is typical enough that what TWIG learns transfers to other graphs and models.

Editorial extensions

If this is right

  • Hyperparameter preference for KGEs is predictable from graph structure before training, so practitioners can pre-filter hyperparameter grids without running the KGE.
  • TWIG's zero-shot transfer across biology, family-tree, and general-knowledge graphs suggests the structural signal is domain-agnostic.
  • Finetuning on a small slice of a target graph's hyperparameter grid yields near-supervised accuracy, making adaptation cheap.
  • TWIG-style simulation could replace full hyperparameter searches for model selection on new knowledge graphs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same structural features could be used to predict which KGE family (e.g. TransE vs DistMult vs ComplEx) suits a graph, not just which hyperparameters; the paper leaves model choice untested.
  • Because all five graphs are small, the zero-shot $R^2$ range may shrink on much larger graphs; testing the protocol on FB15k-237 or WN18RR would map the regime where pre-hoc selection is reliable.
  • TWIG's predicted MRR could serve as a cheap acquisition function inside a Bayesian hyperparameter search, using simulated performance to decide which settings to train for real.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper extends the TWIG simulation model to the multi-KG setting, with the goal of predicting the MRR that the ComplEx KGE model would achieve under various hyperparameter settings on knowledge graphs not seen during training. The authors train TWIG on structural and hyperparameter features from four KGs and evaluate zero-shot on a held-out fifth KG, reporting R² values between 0.54 and 0.73; they also report few-shot finetuning results and within-KG held-out-hyperparameter results. The paper concludes that TWIG generalizes across KGs and that this could enable pre-hoc hyperparameter selection for KGE models.

Significance. If the zero-shot results are robust, this is a meaningful step toward cheap, structure-based hyperparameter selection for KGE models, and it would support the hypothesis that hyperparameter preference is predictable from graph structure. The paper's strengths include a genuinely held-out-KG evaluation, public code and data links, and an explicit limitations paragraph. However, the current evidence is limited to five small KGs and one KGE model, and the absence of error bars and scale-variation tests weakens the general claim made in the abstract.

major comments (3)
  1. [§3.2, Table 4; §4] The zero-shot claim is load-bearing and is currently supported only by five small KGs. The paper's own limitations paragraph (§4) states that 'all KGs examined in this work are relatively small' and that 'it is unclear how TWIG reacts to KGs of different size.' Since the structural features in Table 1 are raw counts (s_deg, p_freq, co-occurrence counts) with no normalization or standardization described, a larger KG such as FB15k-237 or WN18RR would produce feature values outside the training range. Please either add transfer experiments on larger or size-varied KGs, or explicitly restrict the abstract and conclusion claims to the small-KG regime and to a size-agnostic-within-tested-range claim.
  2. [§2.2, Tables 3 and 4] The paper states in §2.2 that four replicates are produced, but all reported R² values are single point estimates with no confidence intervals or error bars. Given that the zero-shot R² for OpenEA is 0.54 (Table 4), the reader cannot judge whether the zero-shot result is robust or whether the differences across KGs are meaningful. Please report per-replicate R² values or means with standard deviations or confidence intervals, and ideally also rank-based agreement metrics.
  3. [§2.3, §3.2] The paper motivates TWIG as enabling 'pre-hoc hyperparameter selection,' but the evaluation uses R² between predicted and ground-truth MRR, which does not directly measure whether the best hyperparameter setting is identified. Please add a selection-oriented evaluation, such as Kendall's tau between predicted and true MRR rankings or top-k overlap of recommended hyperparameter configurations, at least for the zero-shot and 25%-shot settings.
minor comments (6)
  1. [§3.2, Table 4] There is a typo: 'zero-shot and few-show prediction' should be 'few-shot,' and 'CodExSmall' in Table 4 should be 'CoDExSmall' for consistency.
  2. [§3.2] The claim that zero-shot and few-shot prediction 'work regardless of the domain of the unseen KG' is based on a single biological KG (UMLS) and a single family-tree KG (Kinships); please soften this to 'across the domains tested' or add more domains.
  3. [§2.3.1] The sentence 'TWIG is then trained to simulate the output of all hyperparameter combinations on all five knowledge graphs' is imprecise because 10% of hyperparameter combinations are held out; please clarify that training uses the remaining 90%.
  4. [§1.1, §2.2] The MRR computation is not specified as filtered or unfiltered; please state the evaluation protocol used by ComplEx (e.g., filtered vs. raw ranks), since this materially affects absolute MRR values and comparability with other work.
  5. [Figure 2 caption] The caption contains 'An schematic overview'; this should be 'A schematic overview.'
  6. [§1.2] The text refers to 'Sameh et al.' for reference [15], but the reference list gives 'Mohamed, Nováček, Vandenbussche, and Muñoz'; please use the standard surname-based citation form.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the zero-shot evaluation uses held-out KGs and held-out hyperparameters, with self-citations only for architecture details.

full rationale

The central claim is that TWIG, trained on ground-truth ComplEx outputs for 90% of hyperparameter settings on four KGs, can predict ComplEx MRR for settings and for a held-out KG it never saw. This is a supervised meta-model evaluated against externally generated ComplEx MRR; the test targets are not used in fitting the tested KG, so no prediction reduces by construction to an input. The paper's self-citations to [18] import the TWIG architecture, feature set, and two-phase training procedure, but they do not establish the zero-shot result; the current paper supplies new held-out-KG experiments (Table 4) and explicitly acknowledges limitations in Section 4: only ComplEx is tested, all KGs are relatively small compared to FB15k-237 and WN18RR, and it is unclear how TWIG reacts to KGs of different size. Those limitations constrain external validity but are not circularity. I find no step where an equation equals its input, a fitted parameter is renamed a prediction, or a self-citation is load-bearing for the main claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new free parameters or entities; it reuses the TWIG architecture from prior work and evaluates it on a new multi-KG zero-shot task. The load-bearing assumptions are the sufficiency of the structural feature set and the representativeness of the five small datasets.

assumptions (4)
  • standard math MRR and R² are appropriate metrics for evaluating link prediction and simulation accuracy.
    The paper uses MRR (Eq. 1) to summarize rank lists and R² to compare predicted and true MRR; both are standard metrics, and the paper's claims are framed in terms of them.
  • domain assumption The structural features in Table 1 are sufficient to predict KGE performance across KGs.
    TWIG's only graph input is this feature set, so if any important structural driver is missing, zero-shot generalization would fail; the paper does not ablate features.
  • domain assumption The five selected KGs are representative of diverse KG structures and domains.
    The paper relies on these five small graphs to demonstrate cross-domain generalization (Section 2.2), and its limitations note the absence of large benchmarks.
  • domain assumption ComplEx is representative of KGE models for the purpose of hyperparameter prediction.
    Only ComplEx is simulated, and the introduction motivates hyperparameter prediction for KGEs generally; the authors state generalization to other models is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extending TWIG: Zero-Shot Predictive Hyperparameter Selection for KGEs based on Graph Structure." pith.science (2026). https://pith.science/paper/PXNWIRDQ

@misc{pith2026241214801,
  author       = {Pith},
  title        = {Pith review of: Extending TWIG: Zero-Shot Predictive Hyperparameter Selection for KGEs based on Graph Structure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXNWIRDQ}},
  note         = {Machine review of arXiv:2412.14801}
}
read the original abstract

Knowledge Graphs (KGs) have seen increasing use across various domains -- from biomedicine and linguistics to general knowledge modelling. In order to facilitate the analysis of knowledge graphs, Knowledge Graph Embeddings (KGEs) have been developed to automatically analyse KGs and predict new facts based on the information in a KG, a task called "link prediction". Many existing studies have documented that the structure of a KG, KGE model components, and KGE hyperparameters can significantly change how well KGEs perform and what relationships they are able to learn. Recently, the Topologically-Weighted Intelligence Generation (TWIG) model has been proposed as a solution to modelling how each of these elements relate. In this work, we extend the previous research on TWIG and evaluate its ability to simulate the output of the KGE model ComplEx in the cross-KG setting. Our results are twofold. First, TWIG is able to summarise KGE performance on a wide range of hyperparameter settings and KGs being learned, suggesting that it represents a general knowledge of how to predict KGE performance from KG structure. Second, we show that TWIG can successfully predict hyperparameter performance on unseen KGs in the zero-shot setting. This second observation leads us to propose that, with additional research, optimal hyperparameter selection for KGE models could be determined in a pre-hoc manner using TWIG-like methods, rather than by using a full hyperparameter search.

Figures

Figures reproduced from arXiv: 2412.14801 by the authors.

Figure 1
Figure 1. A sample Knowledge graph (left) and an example of link prediction (right) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An schematic overview of how rank-based evaluation of KGE models is performed. (MRR) metric [10, 11]. Specifically, MRR calculates the mean of the reciprocal of the ranks assigned to the correct answers of all link prediction queries. Mathematically, this is expressed as: 𝑀𝑅𝑅 = ∑︀𝑛 𝑖=1 1 𝑟𝑎𝑛𝑘𝑖 𝑛 where 𝑛 is the total number of link prediction queries posed and 𝑟𝑎𝑛𝑘𝑖 is the rank of the correct answer to the 𝑖 𝑡ℎ link … view at source ↗
Figure 3
Figure 3. An overview of link prediction (using KGEs) and of KGE simulation (using TWIG). 2.1. The TWIG Model The job of TWIG is to simulate a single knowledge graph embedding model (such as ComplEx [25]) by predicting the ranks that the KGE model would assign to link prediction queries in the form (𝑠, 𝑝, ?) or (?, 𝑝, 𝑜). To do this, we use the TWIG neural network as published in the original TWIG paper [18]. The TWIG neural … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A visualisation of the TWIG neural network architecture. KG structure (in magenta) and hyperparam￾eter influence (in yellow) are first learned in separate blocks, then combined in an integration component (in red) before predicted ranks are output. are relatively small…
Figure 5
Figure 5. Figure 5: A visualisation of the region around a triple that TWIG uses to calculate structural features. Details statistics are gathered for the main triple (shown in red), and aggregate statistics are gathered for all triples connecting to that triple (shown in blue). Other tri…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [1]

    Hogan, E

    A. Hogan, E. Blomqvist, M. Cochez, C. D’amato, G. D. Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier, A.-C. N. Ngomo, A. Polleres, S. M. Rashid, A. Rula, L. Schmelzeisen, J. Sequeda, S. Staab, A. Zimmermann, Knowledge graphs, ACM Comput. Surv. 54 (2021). URL: https://doi.org/10.1145/3447772. doi:10.1145/3447772

    doi:10.1145/3447772 2021

  2. [2]

    Q. Wang, Z. Mao, B. Wang, L. Guo, Knowledge graph embedding: A survey of approaches and applications, IEEE Transactions on Knowledge and Data Engineering 29 (2017) 2724–2743. doi:10.1109/TKDE.2017.2754499

    arXiv 2017

  3. [3]

    Nickel, K

    M. Nickel, K. Murphy, V. Tresp, E. Gabrilovich, A review of relational machine learning for knowl- edge graphs, Proceedings of the IEEE 104 (2016) 11–33. doi: 10.1109/JPROC.2015.2483592

    arXiv 2016

  4. [4]

    Rossi, D

    A. Rossi, D. Barbosa, D. Firmani, A. Matinata, P. Merialdo, Knowledge graph embedding for link prediction: A comparative analysis, ACM Transactions on Knowledge Discovery from Data 15 (2021) 1–49. doi:10.1145/3424672

    doi:10.1145/3424672 2021

  5. [5]

    Bonner, U

    S. Bonner, U. Kirik, O. Engkvist, J. Tang, I. P. Barrett, Implications of topological imbalance for representation learning on biomedical knowledge graphs, Briefings in bioinformatics 23 (2022) bbac279

    work page 2022

  6. [6]

    S. K. Mohamed, V. Nováček, A. Nounu, Discovering protein drug targets using knowledge graph embeddings, Bioinformatics 36 (2020) 603–610

    work page 2020

  7. [7]

    Zhang, D

    R. Zhang, D. Hristovski, D. Schutte, A. Kastrin, M. Fiszman, H. Kilicoglu, Drug repurposing for covid-19 via knowledge graph completion, Journal of biomedical informatics 115 (2021) 103696

    work page 2021

  8. [8]

    S. K. Mohamed, A. Nounu, V. Nováček, Drug target discovery using knowledge graph embeddings, in: Proceedings of the 34th ACM/SIGAPP symposium on applied computing, 2019, pp. 11–18

    work page 2019

Show all 30 references
  1. [9]

    Celebi, H

    R. Celebi, H. Uyar, E. Yasar, O. Gumus, O. Dikenelli, M. Dumontier, Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings, BMC bioinformatics 20 (2019) 1–14

    2019

  2. [10]

    M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, M. Galkin, S. Sharifzadeh, A. Fischer, V. Tresp, J. Lehmann, Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework, IEEE Transactions on Pattern Analysis and Machine In...

    2021

  3. [11]

    Ruffinelli, S

    D. Ruffinelli, S. Broscheit, R. Gemulla, You can teach an old dog new tricks! on training knowledge graph embeddings, in: ICLR, 2020

    2020

  4. [12]

    Kadlec, O

    R. Kadlec, O. Bajgar, J. Kleindienst, Knowledge base completion: Baselines strike back, arXiv preprint arXiv:1705.10744 (2017)

    2017 arXiv

  5. [13]

    P. Jain, S. Rathi, S. Chakrabarti, et al., Knowledge base completion: Baseline strikes back (again), arXiv preprint arXiv:2005.00804 (2020)

    2020 arXiv

  6. [14]

    Kotnis, V

    B. Kotnis, V. Nastase, Analysis of the impact of negative sampling on link prediction in knowledge graphs, arXiv preprint arXiv:1708.06816 (2017)

    2017 arXiv

  7. [15]

    S. K. Mohamed, V. Novácek, P.-Y. Vandenbussche, E. Muñoz, Loss functions in knowledge graph embedding models, in: DL4KG@ESWC, 2019

    2019

  8. [16]

    Zhang, T

    H. Zhang, T. Zheng, J. Gao, C. Miao, L. Su, Y. Li, K. Ren, Data poisoning attack against knowledge graph embedding, arXiv preprint arXiv:1904.12052 (2019)

    2019 arXiv

  9. [17]

    Bhardwaj, J

    P. Bhardwaj, J. Kelleher, L. Costabello, D. O’Sullivan, Adversarial attacks on knowledge graph embeddings via instance attribution methods, arXiv preprint arXiv:2111.03120 (2021)

    2021 arXiv

  10. [18]

    Sardina, J

    J. Sardina, J. D. Kelleher, D. O’Sullivan, Twig: Towards pre-hoc hyperparameter optimisation and cross-graph generalisation via simulated kge models, in: 2024 IEEE 18th International Conference on Semantic Computing (ICSC), 2024, pp. 122–129. doi:10.1109/ICSC59802.2024.00025

    2024

  11. [19]

    A. T. McCray, A. Burgun, O. Bodenreider, Aggregating umls semantic types for reducing conceptual complexity, Studies in health technology and informatics 84 (2001) 216

    2001

  12. [20]

    Chandak, K

    P. Chandak, K. Huang, M. Zitnik, Building a knowledge graph to enable precision medicine, Scientific Data 10 (2023) 67

    2023

  13. [21]

    Dumontier, A

    M. Dumontier, A. Callahan, J. Cruz-Toledo, P. Ansell, V. Emonet, F. Belleau, A. Droit, Bio2rdf release 3: a larger connected network of linked data for the life sciences, in: Proceedings of the 2014 international conference on posters & demonstrations track, volume 1272, Cites...

    2014

  14. [22]

    Toutanova, D

    K. Toutanova, D. Chen, Observed versus latent features for knowledge base and text inference, in: A. Allauzen, E. Grefenstette, K. M. Hermann, H. Larochelle, S. W.-t. Yih (Eds.), Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, Asso...

    2015 doi

  15. [23]

    Mahdisoltani, J

    F. Mahdisoltani, J. Biega, F. M. Suchanek, A knowledge base from multilingual wikipedias–yago3, Technical report, Telecom ParisTech (2014)

    2014

  16. [24]

    Sardina, D

    J. Sardina, D. O’Sullivan, Structural characteristics of knowledge graphs determine the quality of knowledge graph embeddings across model and hyperparameter choices (2020)

    2020

  17. [25]

    Lacroix, N

    T. Lacroix, N. Usunier, G. Obozinski, Canonical tensor decomposition for knowledge base comple- tion, in: International Conference on Machine Learning, PMLR, 2018, pp. 2863–2872

    2018

  18. [26]

    Safavi, D

    T. Safavi, D. Koutra, Codex: A comprehensive knowledge graph completion benchmark, arXiv preprint arXiv:2009.07810 (2020)

    2020 arXiv

  19. [27]

    B. Shi, T. Weninger, Open-world knowledge graph completion, in: Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  20. [28]

    C. Kemp, J. B. Tenenbaum, T. L. Griffiths, T. Yamada, N. Ueda, Learning systems of concepts with an infinite relational model, in: AAAI, volume 3, 2006, p. 5

    2006

  21. [29]

    Z. Sun, Q. Zhang, W. Hu, C. Wang, M. Chen, F. Akrami, C. Li, A benchmarking study of embedding- based entity alignment for knowledge graphs, arXiv preprint arXiv:2003.07743 (2020)

    2020 arXiv

  22. [30]

    M. Ali, M. Berrendorf, C. T. Hoyt, L. Vermue, S. Sharifzadeh, V. Tresp, J. Lehmann, PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings, Journal of Machine Learning Research 22 (2021) 1–6. URL: http://jmlr.org/papers/v22/20-825.html

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.