REVIEW 2 major objections 3 minor
KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models
T0 review · 2 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper proposes EDAS, a meta-metric that combines multiple evaluation metrics across datasets into a single normalized score, to rank knowledge graph completion models consistently.
desk verdict A modest but sensible application of a known MCDM method to KGC evaluation; the abstract alone doesn't support the strong claims, but the idea is worth a look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
EDAS (Evaluation based on Distance from Average Solution) is a multi-criteria decision-making method. It first computes the average solution across all candidate models for each evaluation metric, then scores each model by its distances from the average-good and average-bad solutions, and finally normalizes these distances into a single score. In this paper it is used as a meta-layer on top of existing KGC metrics, taking a matrix of model-by-metric results and producing one comparable number per model.
What would settle it
On a fixed benchmark set, compute EDAS scores for all models, then remove the top-ranked model and recompute scores for the remaining ones. If their relative order changes, the claimed consistency and generalizability of the ranking would be contradicted.
Extended reading notes
Core claim
The central claim is that EDAS, when applied to KGC evaluation, produces a single score M_i in [0,1] for each model that reflects its distance from the average best and worst hypothetical solutions across all metrics and datasets. A model that is close to the ideal performance profile and far from the anti-ideal profile gets a high score, and the resulting ranking is claimed to be robust, interpretable, and generalizable. This addresses the problem that a model may lead on one dataset or metric but lag on another, and it gives a direct answer to 'which model should I pick?'
Load-bearing premise
The ranking produced by EDAS depends on which models are included in the comparison; the average solution is not an absolute reference, so adding or removing a model can change the relative order of the others.
Editorial extensions
If this is right
- Model selection for KGC becomes a single-number comparison instead of weighing conflicting MRR and Hit@k results.
- Cross-dataset evaluation is unified, so a model that does well across several benchmarks can be identified directly.
- The normalized score gives an interpretable bound (0 to 1) that lets practitioners see how far a model is from the ideal.
- If adopted, EDAS could serve as a standard reporting metric alongside traditional per-metric results.
Reading between the lines
- Because EDAS uses the average of the compared models as its reference point, the ranking can shift when the set of models changes; the paper does not discuss this sensitivity, so the model pool should be disclosed in any reported EDAS scores.
- The same average-distance logic could be imported into other evaluation contexts where multiple conflicting metrics exist, such as question answering or retrieval, although the paper only demonstrates it for KGC.
- A natural test the paper does not run is whether choosing a model by EDAS score leads to better downstream task performance than choosing by MRR alone; that would validate the meta-metric's practical value.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KG-EDAS, a meta-metric for Knowledge Graph Completion (KGC) evaluation that applies the EDAS (Evaluation based on Distance from Average Solution) method to aggregate multiple existing metrics (MRR, MR, Hit@k) across datasets such as FB15k-237 and WN18RR into a single normalized score Mi in [0,1]. The abstract claims this yields a consistent, robust, and generalizable framework for model selection, resolving conflicts among metrics and datasets. The submission is reviewed from the abstract only; no full text, aggregation formulas, experimental tables, or comparison baselines are available.
Significance. If the claimed results hold, KG-EDAS could provide a practical decision-support tool for KGC model selection by offering a single aggregate score over heterogeneous evaluation criteria. However, the methodological novelty is incremental: EDAS is a well-known multi-criteria decision-making method, and its application to KGC is the main contribution. The abstract provides no quantitative evidence, no error analysis, and no comparison to alternative aggregation schemes, so the significance cannot be assessed from the submitted material. The paper would be more valuable if it substantiated the 'consistent, robust, and generalizable' claims with stability analysis and open, reproducible experiments.
major comments (2)
- [Abstract] The central claim that KG-EDAS provides a 'consistent, robust, and generalizable' ranking is undermined by a known property of EDAS: the 'average solution' is computed from the cohort of models under evaluation. Consequently, the normalized scores Mi and their relative ordering can change when models are added or removed. The abstract does not state any invariance property or cohort-relative caveat. This is load-bearing for the claimed generalizability and should be addressed explicitly, either by proving a stability result or by qualifying the score as cohort-relative and reporting sensitivity analyses.
- [Abstract (Experimental results)] The abstract asserts that 'experimental results ... demonstrate' the effectiveness of KG-EDAS, but it reports no aggregation formula, no numerical results, no error bars, and no comparison to alternative meta-metric approaches. The reader cannot verify whether the proposed method outperforms simple baselines such as rank aggregation, normalized weighted sums, or Pareto dominance. At minimum, the paper must specify the exact EDAS variant used, the normalization procedure, and a table with per-metric, per-dataset scores and the resulting Mi ranking.
minor comments (3)
- [Abstract] The notation 'Mi' is introduced but the index i is not defined in the abstract; it should be stated that i indexes KGC models.
- [Abstract] Terms such as 'Mean Rank' and 'Hit@k' are used without formal definition; a brief parenthetical definition would improve clarity.
- [Abstract] The phrase 'KG Evaluation based on Distance from Average Solution' is slightly awkward in English; consider 'A KGC Evaluation Meta-Metric based on Distance from Average Solution'.
Circularity Check
No significant circularity: EDAS aggregates existing metrics and does not reduce to its inputs by construction.
full rationale
The abstract describes a meta-metric (EDAS) that synthesizes existing KGC evaluation metrics (MRR, MR, Hit@k) across datasets into a normalized score. Its output is a function of those input metrics; this is an aggregation, not a circular derivation. There are no fitted parameters that are later 'predicted,' no self-citations, and no uniqueness theorem invoked. The property that the average solution depends on the set of models is a known property of relative ranking methods such as EDAS, but it does not make the ranking equivalent to its inputs by construction—it is an inherent property of any comparative evaluation, not a logical circularity. The abstract's claim of 'consistent, robust, and generalizable' may be overstated given cohort dependence, but that is a correctness/limitation concern, not a circularity concern. With no equations or full text available, and with the abstract itself not exhibiting any step where X is defined in terms of Y, the appropriate finding is no circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Aggregating existing metrics (MRR, MR, Hit@k) via distance to the average solution preserves the ranking information needed for model selection.
- domain assumption The benchmark datasets FB15k-237 and WN18RR and the chosen metric set are representative of KGC evaluation.
Cite this review
Pith. "Pith review of KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models." pith.science (2026). https://pith.science/paper/GGFO2BWQ
@misc{pith2026250815357,
author = {Pith},
title = {Pith review of: KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/GGFO2BWQ}},
note = {Machine review of arXiv:2508.15357}
}
abstract
Knowledge Graphs (KGs) enable applications in various domains such as semantic search, recommendation systems, and natural language processing. KGs are often incomplete, missing entities and relations, an issue addressed by Knowledge Graph Completion (KGC) methods that predict missing elements. Different evaluation metrics, such as Mean Reciprocal Rank (MRR), Mean Rank (MR), and Hit@k, are commonly used to assess the performance of such KGC models. A major challenge in evaluating KGC models, however, lies in comparing their performance across multiple datasets and metrics. A model may outperform others on one dataset but underperform on another, making it difficult to determine overall superiority. Moreover, even within a single dataset, different metrics such as MRR and Hit@1 can yield conflicting rankings, where one model excels in MRR while another performs better in Hit@1, further complicating model selection for downstream tasks. These inconsistencies hinder holistic comparisons and highlight the need for a unified meta-metric that integrates performance across all metrics and datasets to enable a more reliable and interpretable evaluation framework. To address this need, we propose KG Evaluation based on Distance from Average Solution (EDAS), a robust and interpretable meta-metric that synthesizes model performance across multiple datasets and diverse evaluation criteria into a single normalized score ($M_i \in [0,1]$). Unlike traditional metrics that focus on isolated aspects of performance, EDAS offers a global perspective that supports more informed model selection and promotes fairness in cross-dataset evaluation. Experimental results on benchmark datasets such as FB15k-237 and WN18RR demonstrate that EDAS effectively integrates multi-metric, multi-dataset performance into a unified ranking, offering a consistent, robust, and generalizable framework for evaluating KGC models.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.