{"id":"cda7b3f5-eaa8-4678-aaa3-e3ce459ee6c3","arxiv_id":"2508.05587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A PyKEEN extension integrates seven known negative sampling strategies; experiments on FB15K and WN18 show pool sizes shrink sharply and random fallback often dominates at high sample counts.","lead":"This paper packages several advanced negative-sampling methods into PyKEEN, a widely used library for learning embeddings of fact triples. It also reports experiments on four datasets, showing that strict samplers often run out of candidates and fall back on random sampling.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical claim of negligible impact of extra negatives is confounded by the fallback rule in §4.1 that substitutes random negatives when pools are small; the flat curves may be an artifact.","rationale":"The reader's weakest_assumption identifies exactly the same issue: the fallback to random sampling when pools are too small makes the 'negligible impact' result reflect the supplementation rule rather than the samplers. I agree with this assessment. The paper's own text (§5.2) confirms the mechanism. This is the single most load-bearing concern because the empirical study is the proof-of-concept for the extension's utility; if the conclusion is confounded, the paper's contribution rests almost entirely on the software itself. I considered other concerns: the code example in §3.4 registers RelationalNegativeSampler but invokes 'typed', and §3.1 says 'Five new negative samplers' but lists only three, but these are inconsistencies that do not affect the central argument. The absence of a repository URL is a serious verifiability issue, but the reader already conditioned the verdict on code release, and the fallback confound is more specific and scientifically substantive. Since the reader's CONDITIONAL verdict already accounts for this concern, no adjustment is needed; the verdict remains CONDITIONAL (or UNCHANGED in the terms of this stress-test).","tokens_in":10479,"tokens_out":4003,"duration_ms":41090,"concrete_test":"Rerun the FB15K and WN18 experiments with the extension configured to disable random supplementation (set the supplement parameter to false, or cap num_negs_per_pos at the per-triple pool size). For each sampler and each N in {1,2,5,20,50,100} where feasible, compute filtered Hits@10. Also compute the fraction of negatives generated by fallback for each sampler and N, and analyze results only for triples whose pool size is at least N. If the Hits@10 curves remain flat when no fallback is used and for triples with sufficient pools, the 'negligible impact' claim is robust; if they rise or differ across samplers, the original result is an artifact of the supplementation rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core empirical claim (§5.2) is that increasing the number of negatives per positive has negligible impact on Hits@10. However, the experimental design in §4.1 supplements any negative pool smaller than the requested N with random entities. Table 3 shows this fallback dominates for non-random samplers at high N: e.g., Relational sampling on WN18 has an average pool of 2, and >99% of triples have fewer than 100 negatives; Typed on YAGO4-20 has no pool for 16% of triples. Consequently, for N=20,50,100, the majority of negatives under Corrupt, Typed, and Relational are actually random, so the performance curves become essentially random sampling with different N. The paper itself acknowledges this in §5.2: 'the sampling behavior becoming increasingly similar to the random corruption scheme as the number of negatives increases.' This makes the 'negligible impact' conclusion a property of the supplementation rule rather than a meaningful comparison of samplers. If the fallback were disabled (e.g., by skipping triples with insufficient pools or capping N at the pool size), the performance curves at high N could diverge across samplers, and the headline empirical insight might change. Since the empirical proof-of-concept is part of the central claim, this confound is load-bearing; it does not invalidate the software contribution but undermines the paper's scientific conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a modular extension to the PyKEEN knowledge-graph embedding framework that implements several negative sampling strategies (Corrupt, Typed, Relational, Nearest Neighbor, Adversarial) on top of PyKEEN's existing random and Bernoulli samplers. The extension provides a common abstraction with two methods for samplers, a fallback mechanism for negative pools that are too small, a custom filterer for invalid triples, and a data loader for semantic metadata. As a proof of concept, the authors report negative-pool statistics on four datasets and filtered Hits@10 link prediction results on FB15K and WN18, with hyperparameter optimization and a sweep over the number of negatives per positive (1, 2, 5, 20, 50, 100). The main empirical observation is that increasing the number of negatives has negligible impact on performance, which the paper attributes to the random-supplementation rule kicking in when a sampler's pool is too small.","tokens_in":10675,"tokens_out":4613,"duration_ms":48837,"significance":"The software contribution is genuinely useful: a clean, documented, and extensible implementation of advanced negative samplers inside the popular PyKEEN ecosystem, with examples for training, hyperparameter optimization, and standalone use, would lower the barrier for practitioners and researchers to test a variety of sampling strategies. The architecture (two abstract methods, a fallback flag, a filter for invalid triples, and an external-metadata loader) is well conceived. The empirical value is currently limited because the headline 'negligible impact of more negatives' conclusion is confounded by the paper's own random-fallback rule, as the paper itself acknowledges. The resource paper itself is sound and worth publishing after experimental revision.","major_comments":[{"comment":"The claim that increasing the number of negatives per positive has negligible impact on Hits@10 is an artifact of the supplementation rule. For Relational sampling on WN18, the average pool is 2 and 99.9% of triples have fewer than 100 negatives; for Typed on YAGO4-20, 16% of triples have no pool at all. Thus for N=20, 50, 100, most negatives under Corrupt, Typed, and Relational are random fallback negatives, so the curves essentially compare random sampling at different N. The paper admits this in §5.2: 'the sampling behavior becoming increasingly similar to the random corruption scheme as the number of negatives increases.' To support the 'negligible impact' conclusion, the authors should disable supplementation (either by skipping triples with insufficient pools or by capping N at the 95th percentile of pool sizes) and rerun; it is quite possible that samplers diverge at high N when t","section":"§5.2, Table 3, §4.1"},{"comment":"The comparative link prediction results lack statistical support: no repeated seeds, no confidence intervals, no significance tests are reported. Figures 2 and 3 are unreadable in the provided text, making it impossible to verify the claimed differences such as 'Bernoulli strong, adversarial least effective.' Please report numeric Hits@10 values with standard deviations across several seeds (e.g., 5-10 runs), provide the full results in a table, and regenerate the figures with larger fonts and higher resolution or as vector graphics.","section":"§4.1 and §5.2, Figs. 2-3"},{"comment":"The adversarial sampler is evaluated only with RESCAL as the auxiliary model, and RESCAL is also one of the baseline models and one of the worst performers. The paper's own explanation that 'RESCAL was one of the lowest performing models, which significantly affected the performance of the adversarial sampler' shows that the experiment conflates the quality of the adversarial sampler with the quality of its auxiliary model. To make any claim about adversarial sampling itself, the authors should evaluate with at least one stronger auxiliary model (e.g., TransE or ComplEx) or explicitly restrict their conclusions to the specific auxiliary model used.","section":"§5.2"}],"minor_comments":[{"comment":"The abstract method names appear as blank placeholders (empty boxes in the PDF). Please replace them with the actual method names (e.g., `_prepare`, `_create_negative_pool`). Also, the number of new samplers is inconsistent: §3.1 lists three ('Corrupt', 'Typed', 'Relational'), while §6 says five; clarify which five are meant and note that Nearest Neighbor and Adversarial are also implemented.","section":"§3.1"},{"comment":"Typo: 'extesion' should be 'extension'.","section":"§1"},{"comment":"Typo: 'statical analysis' should be 'statistical analysis'.","section":"§3.4"},{"comment":"The repository URL and version/commit identifier are missing from the manuscript; please include them for reproducibility.","section":"§4"},{"comment":"Link prediction experiments are reported only for FB15K and WN18, while YAGO4-20 and DBpedia50 are used only for pool statistics. Please either report link prediction results for the latter two datasets or explicitly state that the proof-of-concept scope is limited to two datasets.","section":"§5.1 and §5.2"},{"comment":"The header percentages are garbled in the provided text (e.g., '� ���'); please ensure the thresholds are clearly labeled.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The reader's take is fair and aligns with my assessment: the software contribution is solid and well-motivated, but the key empirical claim is confounded by the paper's own fallback rule. The paper is suitable as a resource-oriented contribution after a careful experimental revision that isolates the effect of sampler choice from the effect of pool supplementation, adds statistical rigor, and addresses the adversarial-sampler auxiliary-model issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the software: a modular PyKEEN extension that packages several negative samplers (Corrupt, Typed, Relational, plus two dynamic ones) under a common interface, with a custom metadata loader, a -1 placeholder for uncorruptable triples, and a filterer. That genuinely fills a gap—PyKEEN only ships basic samplers—and the design looks sensible. The pool-size statistics in Table 3 are also worth having; showing that Relational sampling on WN18 has an average pool of 2, or that Typed has missing metadata for a chunk of YAGO4-20, is the kind of concrete information practitioners need. The authors re-implement published samplers rather than inventing new ones, and they cite the originals. Fine.\n\nThe weak part is the empirical proof-of-concept, and the stress-test note is right: the \"negligible impact of more negatives\" conclusion is an artifact of their own fallback rule. When a sampler's pool is smaller than the requested N, they fill the rest with random entities. Because pools are small for most non-random samplers, at N=20, 50, 100 the majority of negatives are actually random, so the curves just asymptote to random sampling. The authors even say this in Section 5.2. That means the headline scientific claim is not really about the samplers at all. It's a property of the supplementation rule. If they disabled the fallback (skip triples with insufficient pools, or cap N at pool size), the curves might diverge, and the claim would be meaningful. As it stands, it's a non-result dressed as an insight. This is the load-bearing issue for the empirical section, and it should be fixed or reframed, not just acknowledged.\n\nOther soft spots are smaller but real: the text says \"five new samplers\" and then lists three; the code example registers RelationalNegativeSampler but invokes negative_sampler=\"typed\"; the comparison figures are unreadable in the provided text; there are no error bars. None of these are fatal for a software paper, but they add friction.\n\nThe citation pattern is fine—the self-citation to Barile et al. merely supplies a dataset subset, and the sampler equations are properly attributed. No circularity worth worrying about.\n\nWho is this for? Anyone using PyKEEN and wanting to test non-random negative sampling without writing their own glue code. The tool is likely useful. But the empirical wrapping needs serious revision. I'd send it to peer review, with a clear request to either disable the fallback and report those results, or narrow the claim to what the design actually supports. The software contribution deserves referee time; the current empirical claim does not.","headline":"Useful PyKEEN extension for negative samplers, but the headline empirical claim about extra negatives is an artifact of the random-fallback rule, and the paper should be revised to separate the tool contribution from the confounded experiment.","tokens_in":11317,"tokens_out":1821,"would_cite":false,"duration_ms":20568,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a modular PyKEEN extension can host five advanced negative samplers, and that link prediction performance stays nearly flat as the number of negatives per positive grows because small negative pools are topped up with","keywords":["knowledge graph embedding","negative sampling","PyKEEN","link prediction","corruption strategies","typed sampling","relational sampling","knowledge graphs"],"falsifier":"Re-run the FB15K and WN18 link prediction experiments at negatives per positive 1, 2, 5, 20, 50, 100 with the supplementation parameter disabled, or with duplicates resampled from the restricted pool instead of random entities. If the Hits@10 curves separate by sampler or slope noticeably with the negative count, the paper's 'negligible impact' finding reflects the fallback rather than the samplers. The paper's own pool statistics indicate where to look: relational sampling has average pools of 2 to 7 entities, far below 100.","tokens_in":10196,"feed_emoji":"🧩","tokens_out":6146,"duration_ms":63339,"temperature":0.7,"pith_summary":"The paper tries to establish that a modular extension to PyKEEN can make advanced negative sampling strategies—typed, relational, corrupt, Bernoulli, and dynamic variants—available to any knowledge graph embedding model through the standard pipeline. It also argues empirically that for link prediction, increasing the number of negative samples per positive triple has a negligible effect on Hits@10. The authors explain this by showing that many advanced samplers have small negative pools, so beyond a modest count the extension's supplementation rule fills most negatives with random entities. If this holds, sampler choice and dataset compatibility matter more than negative count, and pushing negative counts higher mainly makes training approximate ordinary random corruption.","feed_headline":"More negative samples barely change link prediction","feed_subtitle":"A PyKEEN extension adds five samplers, but its own fallback rule explains the flat performance curves.","key_machinery":"The central mechanism is the negative pool, $\\mathcal{P}_{\\text{head}}(\\langle h,r,t\\rangle)$ and $\\mathcal{P}_{\\text{tail}}(\\langle h,r,t\\rangle)$, the set of entities that can replace the head or tail without producing a triple already in the graph. Each sampler is one rule for building these pools; the extension's base class reduces implementing a sampler to a precompute method and a candidate-negatives method, while a supplementation parameter fills short pools with random entities. That last mechanism does most of the explanatory work in the empirical study.","core_discovery":"The central claim is that a PyKEEN extension can provide a consistent, reusable home for advanced negative samplers, and that the experiments reveal a boundary condition for those samplers. Each sampler is defined by how it constructs the negative pools $\\mathcal{P}_{\\text{head}}(\\langle h,r,t\\rangle)$ and $\\mathcal{P}_{\\text{tail}}(\\langle h,r,t\\rangle)$ for a positive triple; static strategies precompute candidate sets from relational structure, types, or co-occurrence assumptions, while dynamic strategies use an auxiliary model's embeddings to pick hard negatives. The proof-of-concept study on FB15K and WN18 shows that raising the number of negatives per positive from 1 to 100 leaves Hits","pith_inferences":["A natural next experiment, not run in the paper: disable supplementation and observe whether advanced samplers outperform random at high negative counts; the pool statistics suggest they may saturate early instead.","The extension's modular design invites combining schema constraints with a tunable random mixing fraction, which could let practitioners control the precision-coverage trade-off the paper exposes.","The paper measures Hits@10 only; negative-sample quality metrics such as false-negative rates or pool purity would be needed to separate sampler quality from pool-size effects in these results."],"forward_implications":["Practitioners can compare five negative samplers inside the standard PyKEEN pipeline without forking model code, and hyperparameter search can treat the sampler as just another configuration choice.","On the tested datasets, sampler identity matters more than negative count: at 100 negatives per positive the behavior converges toward random corruption, so gains from 'more negatives' should not be expected beyond a small threshold.","Typed and relational samplers are viable mainly on datasets with rich type metadata and simple relational patterns; their average pool sizes, as low as 2 entities for relational sampling, quantify this constraint.","The pool-size statistics supply a reusable diagnostic for deciding in advance whether a dataset can support a given sampling strategy.","Using RESCAL as the auxiliary model can drag down dynamic and adversarial samplers, so dynamic sampler comparisons are coupled to the choice of auxiliary model.","Vanilla PyKEEN's basic negative sampler set is a real gap: the extension's standardized interface makes advanced strategies as easy to invoke as random corruption."],"supporting_citations":[{"why":"The PyKEEN library whose pipeline interface the extension is built into and must remain compatible with.","marker":"[3]"},{"why":"Introduces TransE and the Bernoulli-style corruption strategy used as a static baseline.","marker":"[6]"},{"why":"Defines the adversarial negative sampling approach underlying the dynamic adversarial variant.","marker":"[9]"},{"why":"Supplies the relational, nearest-neighbor, and adversarial sampler designs and the use of RESCAL as auxiliary model in the experiments.","marker":"[18]"},{"why":"Defines typed/type-constrained sampling using domain, range, and class membership pools.","marker":"[19]"},{"why":"Provides the relation-aware corrupt sampling pool definition used by the Corrupt sampler.","marker":"[31]"}],"fun_headline_variants":["Negative sampling: more isn't better in link prediction","PyKEEN's new samplers: flat gains on link prediction","Advanced negative samplers fail to boost KGE performance","Five new samplers, same link prediction curve","Negative samples: diminishing returns for KGE"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The headline empirical result rests on the rule that fills undersized negative pools with random entities; if that rule were changed, the measured flatness could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Negative sampling: more isn't better in link prediction","PyKEEN's new samplers: flat gains on link prediction","Advanced negative samplers fail to boost KGE performance","Five new samplers, same link prediction curve","Negative samples: diminishing returns for KGE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1014,"prompt_tokens":734,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":201}},"tokens_in":478,"tokens_out":280,"duration_ms":3300,"temperature":1.0,"reasoning_tokens":201,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:13:42.587469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the FB15K and WN18 link prediction experiments at negatives per positive 1, 2, 5, 20, 50, 100 with the supplementation parameter disabled, or with duplicates resampled from the restricted pool instead of random entities. If the Hits@10 curves separate by sampler or slope noticeably with the negative count, the paper's 'negligible impact' finding reflects the fallback rather than the samplers. The paper's own pool statistics indicate where to look: relational sampling has average pools of 2 to 7 entities, far below 100.","supporting_citations":[{"cited_title":"Advances in neural information processing systems 26 (2013)","cited_arxiv_id":null,"evidence_quote":"Introduces TransE and the Bernoulli-style corruption strategy used as a static baseline."},{"cited_title":"In: Walker, M., Ji, H., Stent, A","cited_arxiv_id":null,"evidence_quote":"Defines the adversarial negative sampling approach underlying the dynamic adversarial variant."},{"cited_title":"In: The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I","cited_arxiv_id":null,"evidence_quote":"Defines typed/type-constrained sampling using domain, range, and class membership pools."},{"cited_title":"In: European Semantic Web Conference","cited_arxiv_id":null,"evidence_quote":"Provides the relation-aware corrupt sampling pool definition used by the Corrupt sampler."}],"review_version":1}