{"id":"e007da9a-c610-47a1-87dc-6d8c9f40d136","arxiv_id":"2411.14858","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Mixing type-constrained and random negative sampling improves KGE link prediction on FB15k-237 and Hetionet for all four tested models, but gives near-zero or negative gains on WN18RR.","lead":"A knowledge-graph embedding paper tests a negative sampling strategy that mixes two existing techniques: corrupting triples with entities from a relation's domain or range, and corrupting with any random entity. The authors report link prediction gains on some benchmarks, with the largest gains on a biological graph that has a real ontology.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline +150% Hetionet gain rests on a random split that the authors concede is problematic; a non-random split must be run before the central claim is trusted.","rationale":"The reader's weakest assumption identified the Hetionet random split as the key threat to the central claim, and I agree. The central claim has two pillars: a modest +10% MRR gain on FB15k-237 and a dramatic +150% MRR gain on Hetionet. The FB15k-237 result is plausible and relatively standard, but the Hetionet result is the one that makes the paper's headline impressive, and it depends on an evaluation protocol the authors themselves flag as problematic. A random split makes the link-prediction task artificially homogeneous: test triples are missing uniformly at random, so the relation-type constraints used during negative sampling are perfectly matched to the distribution of test positives. In a realistic drug-repurposing setting, missing facts are concentrated around unseen drug-disease pairs or other structured non-random gaps, and the benefits of ontology-constrained negatives may shrink or reverse. The absence of multiple seeds compounds this, as a single run cannot separate signal from noise, especially for gains as large as those reported on Hetionet. I also note the Appendix A.5 versus Table 1 discrepancy for TransE on WN18RR. That is a serious reporting error and must be fixed, but it affects a peripheral row of the results, not the main positive claims; the Hetionet split is more load-bearing for the abstract's central assertion. Because the reader already conditioned acceptance on a more realistic Hetionet split, seeded runs, and corrected reporting, my stress-test does not change the verdict; it reinforces the need for those conditions. The concrete test above directly settles whether the random split is the source of the Hetionet gains.","tokens_in":12686,"tokens_out":11273,"duration_ms":114285,"concrete_test":"Re-run TransE, DistMult, ComplEx-N3, and RotatE on Hetionet using a non-random split, e.g., holding out all triples involving a held-out set of drugs or diseases as recommended in [31], with the same AmpliGraph grid (k=100/200, eta=10/20, nu stepped 0.0-1.0) and at least 3 seeds per configuration. If the proposed mixed strategy no longer beats both random and type-constrained baselines on MRR by a clear margin, the +150% headline is an artifact of the random split; if it does, the central claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's largest quantitative claim, over +150% MRR on Hetionet, rests entirely on a random 10,000-triple test split that the authors themselves concede \"poses multiple challenges\" (Section 4.1, footnote 1). Under a random split, test triples are missing uniformly at random from the same graph used for training, so relation-specific domain/range constraints align tightly with the training-time corruption distribution; this can systematically inflate the apparent benefit of type-constrained negatives over uniform-random negatives. The authors argue that a random split is adequate because their goal is relative comparison across negative-sampling strategies, but that assumption is exactly what needs testing: the relative ranking of strategies under random missingness need not transfer to realistic settings where missing edges are not missing at random. Because no multiple-seed runs or variance estimates are reported, it is also unclear whether the large margins on Hetionet (e.g., RotatE 0.26 vs 0.10 MRR) are stable. If the ordering does not survive a realistic split, the central claim that the proposed sampling materially improves KGE link prediction on ontology-backed graphs is not established; the smaller FB15k-237 gain (+10% for ComplEx-N3) would likely remain, but the abstract's headline would not. A compounding internal inconsistency, noted by the reader, is that Appendix A.5 reports TransE-on-WN18RR MRR as 0.03-0.04 while Table 1 lists 0.19-0.25; this must be resolved before any final verdict, but the Hetionet split is the more load-bearing threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a negative-sampling strategy for knowledge graph embedding (KGE) models that mixes type-constrained negatives (drawn from the domain or range of the relation) with uniform-random negatives, with the mixture ratio controlled by a hyperparameter ν. The method is evaluated on FB15k-237, WN18RR, and Hetionet with TransE, DistMult, ComplEx-N3, and RotatE, comparing against uniform-random and pure type-constrained baselines. The authors report consistent improvements on FB15k-237 across all four models, mixed results on WN18RR, and very large relative gains on Hetionet, and claim that the strategy is simple, scalable, and leads to state-of-the-art-competitive performance.","tokens_in":12885,"tokens_out":4733,"duration_ms":42916,"significance":"If the empirical claims hold, the paper provides a deceptively simple modification to negative sampling that could improve KGE link prediction on ontology-rich graphs, with little computational overhead. The strengths include a systematic grid search over hyperparameters, evaluation across four model families and three datasets, and a clear pseudo-code description of the method. However, the headline gains rest on a single evaluation protocol for Hetionet that the authors themselves flag as problematic, and the abstract's '+10% MRR' is a best-case number from one model on one dataset rather than a representative result. The work is potentially useful to practitioners, but the current evidence does not fully support the broad claims.","major_comments":[{"comment":"The central claim of over +150% MRR on Hetionet rests entirely on a random 10,000-triple test split that the authors concede 'poses multiple challenges.' Under a random split, test triples are missing uniformly at random from the same distribution used to train, which can systematically inflate the apparent value of type-constrained negatives because the domain/range constraints used at test time align tightly with the training-time corruption distribution. The authors state that their purpose is a relative comparison of negative-sampling strategies, but this is precisely the assumption that needs to be verified: the relative ranking under random missingness does not necessarily transfer to realistic settings where missing edges are not missing at random. Since no variance estimates or multiple-seed runs are reported, it is also unclear whether the large margins on Hetionet (e.g., RotatE 0.26 vs 0.10 MRR) are stable. I ask the authors to evaluate at least one non-random split, such as the disease-centric split of [31], and to report standard deviations over multiple seeds before the Hetionet claim can be accepted.","section":"Section 4.1, footnote 1; Table 1 (Hetionet rows)"},{"comment":"There is a direct internal inconsistency that undermines the credibility of the WN18RR results. Appendix A.5 states that 'the MRR of TransE on WN18RR ... was in the range of [0.03, 0.04] for all νs, including the baseline,' but Table 1 lists TransE with random sampling at 0.25 MRR, type-constrained at 0.19, and the proposed method at 0.24. These two numbers cannot both be correct. Since the paper's own narrative relies on TransE (and DistMult) being the exceptions on WN18RR, this discrepancy must be resolved—either the table or the appendix text is wrong, and the correct values must be reported.","section":"Appendix A.5 vs Table 1 (TransE on WN18RR)"},{"comment":"The pseudo-code in Algorithm 1 contains a typo that makes the algorithm ill-defined as written: the line 'St.c. ← sample(T − t.c., ηrand)' uses ηrand instead of ηt.c. for the number of type-constrained negatives. With this line, when ν=0 (pure random baseline), ηrand=0 and no negatives would be generated at all, contradicting the experiments where ν=0 is the random baseline. The implementation presumably uses ηt.c., but the paper must correct the pseudo-code, as it is the formal specification of the proposed method.","section":"Section 3.2, Algorithm 1"},{"comment":"The abstract's claim of 'substantial improvement (+10% MRR) for standard benchmark datasets' is cherry-picked: the +10% figure is the largest gain on FB15k-237 (ComplEx-N3), while WN18RR shows no improvement for DistMult and ComplEx-N3 and a regression for TransE under the proposed method. The abstract should either present the full range of results or state the average improvement, rather than highlighting only the best case. This overstates the strength and consistency of the empirical evidence.","section":"Abstract and Section 4.2"}],"minor_comments":[{"comment":"The text states that 'the optimal value of η for TransE, DistMult and ComplEx-N3 on WN18RR is 30,' but Table 3 shows that ComplEx-N3 achieves best results with η=20 on WN18RR for all three negative-sampling strategies. This should be corrected or clarified.","section":"Section 4.2"},{"comment":"The caption says 'Best results for each datasets are reported in bold, while across models are underlined,' but the table as rendered does not contain any bold or underlined entries. The formatting should be made consistent or the caption removed.","section":"Table 1 caption"},{"comment":"Figure 1 is referenced and captioned, but the actual plot is not visible in the provided text. Please ensure the figure is included in the final version.","section":"Appendix A.5, Figure 1"},{"comment":"The definition of domainp and rangep in Section 3.1 is given in terms of the graph G, but Section 3.2 says the type constraints are 'extracted from the ontology (if given) or from the facts in the training data.' For Hetionet, where an ontology is available, it is unclear whether the domain/range sets are taken from the ontology or inferred from the training facts. This distinction matters for the interpretation of the Hetionet results and should be made explicit.","section":"Section 3.1-3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extended abstract (LoG 2024) with appendices; if this is intended for a full journal version, the authors should be encouraged to expand the evaluation to include multiple seeds and a more realistic Hetionet split, and to resolve the TransE/WN18RR inconsistency. The self-citation to AmpliGraph [38] is appropriate as the implementation vehicle, but no standalone code is provided; consider asking for a code release for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. The core idea is sensible: mix type-constrained negative sampling with random uniform sampling via a tunable ratio ν, which is a small and natural extension of Krompaß et al. The FB15k-237 results are plausibly real, with consistent relative gains across four models and a modest +10% MRR for ComplEx-N3. But the paper has an internal contradiction that needs resolving before you trust any of its numbers, and the abstract oversells the result.\n\nThe new bit is the specific combination and the tunable ratio; the individual components are known, but the mixture isn't, and the paper is transparent about hyperparameters and uses a public library, so the experiments are checkable. The overhead is small (about 5ms per step), which matters for practical use.\n\nNow the soft spots, in proportion. The most serious is Appendix A.5 vs Table 1: the appendix says TransE MRR on WN18RR is 0.03–0.04 for all ν, while Table 1 lists 0.19–0.25 for the same entries. That is a four- to six-fold discrepancy, not a rounding error. It might be a typo in the appendix, but it poisons well confidence in the reported numbers, and the authors could have caught it. The abstract claims \"+10% MRR for standard benchmark datasets,\" but WN18RR is flat or negative for TransE and DistMult; the +10% is a cherry-picked best case on FB15k-237. The Hetionet +150% gain is the headline, but it rests on a random 10,000-triple split that the authors themselves admit \"poses multiple challenges.\" For a graph with strong schema, a random split can systematically favor type-constrained negatives because the test distribution matches the training-time corruption distribution. The authors argue they only want relative comparison, but that is exactly the assumption that needs testing with a more realistic split. Also, no multiple-seed runs or variance estimates are reported, so we don't know if the large margins on Hetionet are stable.\n\nWho gets value from this? Someone working on negative sampling for KGEs, or on ontology-backed graphs like Hetionet, would want to read it as a baseline. It is an extended abstract, so the depth is limited. I think it deserves a serious referee, but with a clear request for major revision: fix the TransE/WN18RR contradiction, report seeded runs with variance, run a non-random Hetionet split, and rewrite the abstract to match what the experiments actually show.","headline":"A simple and plausible negative-sampling interpolation, but the paper's own numbers are internally contradictory and the headline Hetionet gain rests on a split the authors concede is problematic.","tokens_in":13558,"tokens_out":2189,"would_cite":false,"duration_ms":22237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-aware negative sampling lifts knowledge-graph link prediction by up to 150%.","keywords":["knowledge graph embedding","negative sampling","link prediction","type constraints","domain and range","ontology","Hetionet","FB15k-237"],"falsifier":"Rerun the Hetionet experiments with a non-random held-out split (e.g., withholding triples by relation type or by connected component) and with several random seeds, then check whether the mixture still beats random sampling by a large margin; if the >150% MRR advantage does not persist, the paper's strongest empirical claim is not supported.","tokens_in":12363,"feed_emoji":"🔗","tokens_out":7995,"duration_ms":64965,"temperature":0.7,"pith_summary":"This paper argues that synthetic negative triples for training knowledge graph embedding models should be drawn neither purely at random nor purely from a relation's type constraints, but from a mixture of the two. The authors extend an existing type-constrained strategy that corrupts triples by replacing the head or tail with entities from the relation's domain or range, and blend it with uniform random sampling through a hyperparameter $\\nu$ that sets the proportion of each. On FB15k-237 the mixture outperforms both baselines across all four models tested, with up to +10% mean reciprocal rank (MRR) over random and +33% over type-constrained negative sampling; on the ontology-backed Hetionet dataset the relative gain over random reaches more than +150% MRR. The only exceptions are TransE and DistMult on WN18RR, where the mixture does not beat random sampling. If the result holds, a nearly free change to how negatives are generated can materially improve link prediction, and it can do so with fewer negatives per triple.","feed_headline":"Mixing type-aware negatives lifts link prediction 150%","feed_subtitle":"A simple blend of domain/range and random corruptions beats both baselines on most KGE models.","key_machinery":"The load-bearing object is the negative set constructed for each training triple $t=(s,p,o)$. Type-constrained negatives are drawn from $T^-_{t.c.} = \\{(s',p,o) \\mid s' \\in \\mathrm{domain}_p\\} \\cup \\{(s,p,o') \\mid o' \\in \\mathrm{range}_p\\}$; random negatives come from replacing either entity with any entity in the graph. The method partitions the $\\eta$ negatives per triple into $\\lfloor\\eta\\nu\\rfloor$ random and $\\lceil\\eta(1-\\nu)\\rceil$ type-constrained samples, so $\\nu$ controls the trade-off between semantic relevance and diversity. This addresses the two failure modes of the pure strategies: pure random sampling is dominated by over-represented entity types, while pure type-constrained sampling repeats the same entities and produces false negatives when a class has very few instances.","core_discovery":"The central claim is that domain- and range-aware negative sampling, when combined with random uniform sampling, improves link prediction for knowledge graph embedding models relative to either strategy alone. The paper defines, for each relation $p$, the domain as all entities that appear as a subject in some $(s,p,o)$ triple and the range as all entities that appear as an object; type-constrained negatives corrupt a triple only within those sets. The proposed method samples a fraction $\\nu$ of the $\\eta$ negatives from this constrained set and the remaining fraction from the full entity set. In experiments with TransE, DistMult, ComplEx-N3, and RotatE, this mixture beats the random and type-constrained baselines in every dataset except WN18RR with TransE and DistMult, where it falls slightly short of the random baseline. On Hetionet, where classes come from a real ontology and some types such as Gene dominate, the gains are largest: well over +150% MRR relative to random sampling.","pith_inferences":["The large Hetionet improvement may not survive a more realistic evaluation: the paper uses a random split of triples, whereas in practice missing edges are not missing at random; a systematic split by relation or by entity subset would test this directly.","The method's value should depend on the cardinality distribution of domain and range classes; a diagnostic that varies class sizes while holding the graph fixed could predict when the mixture helps and by how much.","Because the strategy is scoring-function agnostic in the paper's experiments, it is plausible to combine it with harder-negative or self-adversarial schemes, but that extension is untested and goes beyond what the paper shows."],"forward_implications":["On FB15k-237, the mixture gives a 10% MRR improvement over random sampling for ComplEx-N3 and a 33% improvement over type-constrained negative sampling, without changing the model architecture.","On ontology-backed graphs with skewed class sizes, such as Hetionet, the strategy can multiply MRR by roughly 2.5 relative to random sampling while adding only about 5 ms per training step.","The benefit is most pronounced when the number of negatives per triple is small ($\\eta=10$); with larger $\\eta$, random sampling alone supplies enough informative negatives, so the strategy can train with fewer samples.","The optimal $\\nu$ is easy to tune: results are consistent across neighbouring values and across all four scoring functions on a given dataset."],"supporting_citations":[{"why":"Introduces random uniform negative sampling, the baseline the paper compares against.","marker":"[3]"},{"why":"Introduces type-constrained negative sampling, the strategy the paper extends.","marker":"[7]"},{"why":"Provides the FB15k-237 benchmark dataset used for main experiments.","marker":"[29]"},{"why":"Provides the WN18RR benchmark and the filtered evaluation protocol.","marker":"[9]"},{"why":"Provides the Hetionet ontology-backed dataset where the largest gains appear.","marker":"[30]"},{"why":"Documents the challenges of splitting Hetionet, supporting the authors' caveat about the random test split.","marker":"[31]"},{"why":"Defines the RotatE scoring function used in the experiments.","marker":"[23]"},{"why":"Software library used to run all experiments.","marker":"[38]"}],"fun_headline_variants":["Combining type-aware and random negatives improves KGE link prediction","Blend of domain/range and random negatives lifts KGE accuracy 150%","Hybrid negative sampling with domain/range constraints improves KGE embeddings","Domain- and range-aware negatives beat pure random in most KGE models","Mix of type-aware and random negatives yields 150% gain on Hetionet"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The crux is the assumption that the relative ordering of negative-sampling strategies measured on a random 10,000-triple test split of Hetionet carries over to the real setting in which missing edges are not missing at random; on top of that, all results come from a single run, so the size of the reported gains has no measured variance.","fun_headline_variants_meta":{"raw":{"variants":["Combining type-aware and random negatives improves KGE link prediction","Blend of domain/range and random negatives lifts KGE accuracy 150%","Hybrid negative sampling with domain/range constraints improves KGE embeddings","Domain- and range-aware negatives beat pure random in most KGE models","Mix of type-aware and random negatives yields 150% gain on Hetionet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001129,"raw_usage":{"total_tokens":4667,"prompt_tokens":896,"completion_tokens":3771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":3674}},"tokens_in":512,"tokens_out":3771,"duration_ms":27201,"temperature":1.0,"reasoning_tokens":3674,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:47:35.348309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Hetionet experiments with a non-random held-out split (e.g., withholding triples by relation type or by connected component) and with several random seeds, then check whether the mixture still beats random sampling by a large margin; if the >150% MRR advantage does not persist, the paper's strongest empirical claim is not supported.","supporting_citations":[{"cited_title":"Translating embeddings for modeling multi-relational data","cited_arxiv_id":null,"evidence_quote":"Introduces random uniform negative sampling, the baseline the paper compares against."},{"cited_title":"Type-constrained representation learning in knowledge graphs","cited_arxiv_id":null,"evidence_quote":"Introduces type-constrained negative sampling, the strategy the paper extends."},{"cited_title":"Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery","cited_arxiv_id":"2105.10488","evidence_quote":"Documents the challenges of splitting Hetionet, supporting the authors' caveat about the random test split."},{"cited_title":"Rotate: Knowledge graph em- bedding by relational rotation in complex space","cited_arxiv_id":null,"evidence_quote":"Defines the RotatE scoring function used in the experiments."}],"review_version":1}