{"id":"4eaafd54-360f-43e1-a0c5-0d47442c9c24","arxiv_id":"2508.15357","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces EDAS, a meta-metric that averages multiple KGC evaluation metrics across datasets into a single normalized score.","lead":"This paper proposes a new way to score knowledge graph completion models by combining their results on several existing metrics into one number. The goal is to make it easier to pick the best model when different metrics disagree.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EDAS ranking depends on the model set, yet the abstract claims consistency and generalizability without caveat","rationale":"The reader's weakest assumption identified the average-solution reference as a potential stability issue. My concern is the same: the central claim of a consistent, robust, generalizable meta-metric is undermined if the ranking can flip when the model set changes. The abstract provides no discussion or caveat about this property. However, since only the abstract was available, it is impossible to determine whether the full paper addresses this limitation. Therefore the verdict remains UNVERDICTED—not because the concern is absent, but because the evidence needed to resolve it is not present. My proposed test would settle the issue if the full data were available. The reader and I agree on the core concern.","tokens_in":744,"tokens_out":2644,"duration_ms":31068,"concrete_test":"Take a fixed set of KGC models (e.g., three representative models: TransE, RotatE, and a GNN-based model) evaluated on FB15k-237 and WN18RR with a fixed set of metrics (MRR, Hit@1, Hit@10). Compute EDAS scores and rankings. Then perform two perturbations: (a) add a deliberately weak model (e.g., a random predictor), and (b) remove a mid-tier model. Recompute EDAS scores and rankings. If the relative ordering of the original models changes in either perturbation, the ranking is not model-set independent, directly falsifying the abstract's claim of a consistent and generalizable framework. If the ordering remains identical, the concern is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EDAS provides a 'consistent, robust, and generalizable framework' for KGC model selection, synthesizing performance into a single normalized score. The method is based on distance from the average solution (as the name 'EDAS' implies), so the average—and hence the scores—depends critically on the set of models being compared. This is a known property of relative multiple-criteria decision-making methods. If the paper does not explicitly address this dependence, then the ranking of models is not an intrinsic property but a function of the evaluation cohort. Adding, removing, or replacing a single model can alter the relative ordering of other models, undermining the claimed 'consistent' and 'generalizable' framework. The abstract states 'unified ranking' and 'global perspective', which are only true within a fixed model set. Without a stated invariance property or a caveat that the score is cohort-relative, the central claim as worded is misleading. This is the weakest load-bearing assumption because the entire utility of the meta-metric for model selection depends on whether its rankings are stable across reasonable changes to the model set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KG-EDAS, a meta-metric for Knowledge Graph Completion (KGC) evaluation that applies the EDAS (Evaluation based on Distance from Average Solution) method to aggregate multiple existing metrics (MRR, MR, Hit@k) across datasets such as FB15k-237 and WN18RR into a single normalized score Mi in [0,1]. The abstract claims this yields a consistent, robust, and generalizable framework for model selection, resolving conflicts among metrics and datasets. The submission is reviewed from the abstract only; no full text, aggregation formulas, experimental tables, or comparison baselines are available.","tokens_in":1017,"tokens_out":1920,"duration_ms":20944,"significance":"If the claimed results hold, KG-EDAS could provide a practical decision-support tool for KGC model selection by offering a single aggregate score over heterogeneous evaluation criteria. However, the methodological novelty is incremental: EDAS is a well-known multi-criteria decision-making method, and its application to KGC is the main contribution. The abstract provides no quantitative evidence, no error analysis, and no comparison to alternative aggregation schemes, so the significance cannot be assessed from the submitted material. The paper would be more valuable if it substantiated the 'consistent, robust, and generalizable' claims with stability analysis and open, reproducible experiments.","major_comments":[{"comment":"The central claim that KG-EDAS provides a 'consistent, robust, and generalizable' ranking is undermined by a known property of EDAS: the 'average solution' is computed from the cohort of models under evaluation. Consequently, the normalized scores Mi and their relative ordering can change when models are added or removed. The abstract does not state any invariance property or cohort-relative caveat. This is load-bearing for the claimed generalizability and should be addressed explicitly, either by proving a stability result or by qualifying the score as cohort-relative and reporting sensitivity analyses.","section":"Abstract"},{"comment":"The abstract asserts that 'experimental results ... demonstrate' the effectiveness of KG-EDAS, but it reports no aggregation formula, no numerical results, no error bars, and no comparison to alternative meta-metric approaches. The reader cannot verify whether the proposed method outperforms simple baselines such as rank aggregation, normalized weighted sums, or Pareto dominance. At minimum, the paper must specify the exact EDAS variant used, the normalization procedure, and a table with per-metric, per-dataset scores and the resulting Mi ranking.","section":"Abstract (Experimental results)"}],"minor_comments":[{"comment":"The notation 'Mi' is introduced but the index i is not defined in the abstract; it should be stated that i indexes KGC models.","section":"Abstract"},{"comment":"Terms such as 'Mean Rank' and 'Hit@k' are used without formal definition; a brief parenthetical definition would improve clarity.","section":"Abstract"},{"comment":"The phrase 'KG Evaluation based on Distance from Average Solution' is slightly awkward in English; consider 'A KGC Evaluation Meta-Metric based on Distance from Average Solution'.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review. I could not access the full text, so I cannot determine whether the manuscript meets the journal's standards. The main risk is that the abstract overclaims generalizability for a cohort-dependent method. I recommend requesting the full manuscript and a stability analysis before making a final decision. The paper appears to be a straightforward application of an existing MCDM method; the editor should weigh whether this provides sufficient novelty for the target journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is EDAS (a standard multi-criteria decision-making method) applied to the KGC model-selection problem. That's not a bad idea — the field genuinely lacks a principled way to reconcile conflicting metrics like MRR and Hit@k across datasets. Framing the mess as a multi-criteria decision problem is a clean move, and the paper seems to be the first to do it for KGC. Credit where due: the problem is real, and the proposed solution is not arbitrary.\n\nWhat the abstract actually delivers is thin: no equations, no tables, no error bars, no comparisons to obvious baselines like averaging ranks or Pareto frontiers. So the central claims — “robust,” “generalizable,” “consistent” — are currently unsupported. I can’t tell if the authors address the known property that EDAS scores are cohort-relative: the average solution depends on which models are in the comparison set. If that dependence isn't discussed and bounded, the word “consistent” in the abstract is misleading. That's the main soft spot, and it's load-bearing for an evaluation tool meant for model selection.\n\nThe stress-test note flags the cohort dependence. I think it's a legitimate concern, not a manufactured one. But it isn't fatal: many practical evaluation frameworks are relative, and the paper may well include a discussion or sensitivity analysis that we can't see from the abstract alone. The reader's low confidence and no-verdict stance is fair.\n\nOne more slight reservation: applying a known method to a new domain is a real but modest contribution. The significance is bounded — it's an evaluation tool, not a new model or theory. That's fine, but the authors should not oversell it.\n\nWho is this for? KGC practitioners who need a single number for model selection, and meta-science people interested in evaluation methodology. It deserves a serious referee: the idea is coherent, the domain is appropriate, and the missing details (aggregation formula, invariance checks, comparison to simpler meta-metrics) are exactly what peer review is for. I'd read the full paper, not desk-reject it.\n\nMy recommendation: send it to review, but the reviewers should push hard on the cohort-dependence question and on whether EDAS adds anything over a well-chosen weighted average.\n\nBest.","headline":"A modest but sensible application of a known MCDM method to KGC evaluation; the abstract alone doesn't support the strong claims, but the idea is worth a look.","tokens_in":1382,"tokens_out":910,"would_cite":false,"duration_ms":10976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes EDAS, a meta-metric that combines multiple evaluation metrics across datasets into a single normalized score, to rank knowledge graph completion models consistently.","keywords":["Knowledge Graph Completion","EDAS","meta-metric","model evaluation","multi-criteria decision analysis","MRR","Hit@k","FB15k-237"],"falsifier":"On a fixed benchmark set, compute EDAS scores for all models, then remove the top-ranked model and recompute scores for the remaining ones. If their relative order changes, the claimed consistency and generalizability of the ranking would be contradicted.","tokens_in":728,"feed_emoji":"📊","tokens_out":2222,"duration_ms":23846,"temperature":0.7,"pith_summary":"Knowledge graph completion (KGC) models are usually compared with metrics like MRR, Mean Rank, and Hit@k, which can disagree and make it hard to decide which model is best overall. The paper argues that a unified meta-metric is needed and introduces KG-EDAS, which adapts the multi-criteria decision method EDAS to combine all metric scores into one number between 0 and 1 per model. On benchmarks FB15k-237 and WN18RR, the paper reports that this single score gives a stable, consistent ranking of models across metrics and datasets. If correct, this offers a practical way to compare KGC models holistically rather than piecemeal.","feed_headline":"One score ranks knowledge graph models across every metric","feed_subtitle":"EDAS merges MRR, Mean Rank, and Hit@k into a single 0-to-1 score for fair model selection.","key_machinery":"EDAS (Evaluation based on Distance from Average Solution) is a multi-criteria decision-making method. It first computes the average solution across all candidate models for each evaluation metric, then scores each model by its distances from the average-good and average-bad solutions, and finally normalizes these distances into a single score. In this paper it is used as a meta-layer on top of existing KGC metrics, taking a matrix of model-by-metric results and producing one comparable number per model.","core_discovery":"The central claim is that EDAS, when applied to KGC evaluation, produces a single score M_i in [0,1] for each model that reflects its distance from the average best and worst hypothetical solutions across all metrics and datasets. A model that is close to the ideal performance profile and far from the anti-ideal profile gets a high score, and the resulting ranking is claimed to be robust, interpretable, and generalizable. This addresses the problem that a model may lead on one dataset or metric but lag on another, and it gives a direct answer to 'which model should I pick?'","pith_inferences":["Because EDAS uses the average of the compared models as its reference point, the ranking can shift when the set of models changes; the paper does not discuss this sensitivity, so the model pool should be disclosed in any reported EDAS scores.","The same average-distance logic could be imported into other evaluation contexts where multiple conflicting metrics exist, such as question answering or retrieval, although the paper only demonstrates it for KGC.","A natural test the paper does not run is whether choosing a model by EDAS score leads to better downstream task performance than choosing by MRR alone; that would validate the meta-metric's practical value."],"forward_implications":["Model selection for KGC becomes a single-number comparison instead of weighing conflicting MRR and Hit@k results.","Cross-dataset evaluation is unified, so a model that does well across several benchmarks can be identified directly.","The normalized score gives an interpretable bound (0 to 1) that lets practitioners see how far a model is from the ideal.","If adopted, EDAS could serve as a standard reporting metric alongside traditional per-metric results."],"supporting_citations":[],"fun_headline_variants":["EDAS: one score to rank all KG completion models","Single score unifies KG model ranking across metrics and datasets","KG-EDAS: distance-based meta-metric for holistic model selection","Stop comparing MRR vs Hit@k: EDAS gives one number","Meta-metric EDAS turns conflicting KG scores into one ranking"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The ranking produced by EDAS depends on which models are included in the comparison; the average solution is not an absolute reference, so adding or removing a model can change the relative order of the others.","fun_headline_variants_meta":{"raw":{"variants":["EDAS: one score to rank all KG completion models","Single score unifies KG model ranking across metrics and datasets","KG-EDAS: distance-based meta-metric for holistic model selection","Stop comparing MRR vs Hit@k: EDAS gives one number","Meta-metric EDAS turns conflicting KG scores into one ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2668,"prompt_tokens":838,"completion_tokens":1830,"prompt_tokens_details":{"cached_tokens":768},"prompt_cache_hit_tokens":768,"prompt_cache_miss_tokens":70,"completion_tokens_details":{"reasoning_tokens":1752}},"tokens_in":70,"tokens_out":1830,"duration_ms":292937,"temperature":1.0,"reasoning_tokens":1752,"cache_read_input_tokens":768,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:56:06.426873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed benchmark set, compute EDAS scores for all models, then remove the top-ranked model and recompute scores for the remaining ones. If their relative order changes, the claimed consistency and generalizability of the ranking would be contradicted.","supporting_citations":[],"review_version":1}