{"id":"ed084d6f-5d24-4421-b2d0-16a38e89c5da","arxiv_id":"2607.20163","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Community-based negative sampling plus competition-aware plausibility scores give better prioritization of candidate biomedical annotations than classifier confidence alone.","lead":"This paper proposes a machine-learning pipeline that scores how plausible a candidate biomedical annotation is by combining classifier confidence with competing relationships in knowledge graphs. The authors report that community-based negative sampling and competition-aware scores help prioritize annotations for expert review across five biomedical knowledge graphs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Blind test set not blind: TransE embeddings are computed on the full KG before the 10% split, so held-out positive triples can leak into classifier features, inflating the reported 5.8% and plausibility gains.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the blind test set is likely contaminated because node embeddings are computed on the entire KG before the train/test split. My independent reading of the manuscript confirms this: the pipeline description and Figure 2 place embedding before partitioning, and the Experimental Protocol does not mention recomputing embeddings after reserving the 10% test set. This is a classic transductive leakage scenario for node-embedding-based classifiers. The impact is severe because the main empirical contributions—the 5.8% balanced accuracy improvement and the superiority of plausibility measures over confidence alone—are all measured on this contaminated test set. If the leak is real, the measured gains may be largely an artifact, and the framework's ability to prioritize unseen annotations is not established. The proposed concrete test directly addresses this by recomputing embeddings on training edges only; if results are unchanged, the concern is refuted, but if they degrade, the paper's central quantitative claims fail. The reader's verdict of REJECT is therefore appropriate, and my stress-test does not move it. I considered secondary issues (e.g., test negatives generated by the same sampling strategy, missing error bars in Table 5, and the small case study) but those are less load-bearing: even if fixed, the leakage question would dominate the credibility of the evaluation. I found no other single concern that could overturn the verdict if the leakage were resolved.","tokens_in":19994,"tokens_out":3923,"duration_ms":41777,"concrete_test":"Re-run the protocol with masked embeddings: compute TransE embeddings on the subgraph containing only the 90% training edges (and generated negatives), then train the RF classifiers and evaluate on the same held-out BlP_i. Compare balanced accuracy and the Table 5 metrics against the published full-embedding results. If the average improvement over random sampling drops materially (e.g., below the reported 5.8%) or absolute BAcc falls, the reported gains are inflated by transductive leakage; if metrics are unchanged, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative claims rest on evaluating classifiers on truly unseen triples. The protocol is not blind. The 'KG Embedding and Binary Classifiers' section states that node and edge embeddings are 'first computed' on the whole KG (Figure 2), and the 'Experimental Protocol' section then reserves 10% of observed triples as the blind positive test set BlP_i. No step re-embeds the KG after removing those triples; the statement that 'Neiter BlPi nor Ni is used during training' refers only to classifier training, not embedding. Because TransE node embeddings are optimized on all edges, the existence of each test triple is encoded in the embeddings of its subject and object nodes. A relation-specific classifier trained on the remaining 90% can thus exploit features that already contain the label of the test triple, inflating both absolute balanced accuracy and the improvement attributed to community-based negatives. Since every plausibility measure in Table 5 (Base, Gain, Combo, SoftMax) is built on these classifier scores, the entire quantitative evaluation is suspect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for prioritizing candidate biomedical annotations for expert curation. The pipeline first computes knowledge-graph embeddings (default TransE) for each of five bioKGs, partitions edges by schema facts, trains a relation-specific random-forest classifier for each schema fact using either random or community-based negative sampling, and then derives plausibility scores (Base, Gain, Combo, SoftMax) that combine classifier confidence, classifier reliability, and competing predicates. The authors report that community-based negative sampling improves balanced accuracy by 5.8% on average, that Pcombo provides a balanced trade-off between calibration and competition-aware separation, and that these scores can support curator workflows. An open-source implementation and a small expert-annotated case study are included.","tokens_in":20218,"tokens_out":8221,"duration_ms":82202,"significance":"The framework addresses a real bottleneck in biocuration, and the idea of making plausibility estimates competition-aware is potentially valuable. The paper is clearly written, and the authors provide open code/data and evaluate on five large, heterogeneous bioKGs. The formal definitions and the negative-sampling procedure are concrete and reproducible. However, as described, the main experimental protocol does not support the central quantitative claims: the 'blind' positive test set is likely contaminated by transductive embedding leakage, and the negative test set is generated from the same distribution used to train the classifiers. These issues affect the 5.8% accuracy gain and the plausibility comparisons in Table 5, so the contribution is not yet established.","major_comments":[{"comment":"The blind test set is not blind. The manuscript states that 'A KG G is first embedded into a latent space using a KGE model' (Section 'KG Embedding and Binary Classifiers', Figure 2), and only afterwards the Experimental Protocol holds out 10% of observed triples as BlP_i. Nowhere do the authors re-embed the KG after removing BlP_i. TransE node embeddings are optimized on all edges of the full KG, so the existence of each held-out test triple is encoded in the embeddings of its subject and object nodes. The relation-specific classifiers are trained on the remaining 90%, but their input features for a test triple are computed from these contaminated embeddings. Consequently, the test labels can influence the classifier features, inflating balanced accuracy and every downstream plausibility metric in Table 5, Figure 5, and Supp. Tables S1–S5 (including the 5.8% average improvement). The au","section":"Experimental Protocol / KG Embedding and Binary Classifiers"},{"comment":"The negative test set shares the generation mechanism with the training negatives. The protocol creates N_i via 'the adopted negative sampling strategy' and, for each experiment, the training negatives are generated by the same strategy (community-based or random). The NRR and Cal metrics in Table 5 therefore measure rejection of synthetic negatives drawn from the same distribution that the classifier was trained to reject. This is not an external gold standard for 'implausible' candidate annotations, and the improvement attributed to community-based negatives is expected under this setup. Likewise, the positive test set consists of existing KG edges rather than newly extracted candidate annotations. The only external evaluation, the 29-annotation case study (Section 'Case Study'), has just 3 disagreements and 13 uncertain cases and is too small to establish prioritization quality. A sep","section":"Experimental Protocol / Metrics for evaluating plausibility"},{"comment":"The high Sepbest value for Pgain (99.3% in Table 5) is largely a by-product of the definition of Pgain. Pgain is defined as σ(λ·(weighted target score − max_j weighted competitor score)), so for the triple with the highest weighted score, the target and the strongest competitor receive scores that are mirror images (one above 0.5, one below 0.5 if the same set of competitors is used). Any threshold between those values will put them on opposite sides. Thus Sepbest≈100% does not provide independent evidence that Pgain 'better separates' biologically correct predicates; it mainly reflects the construction of the score. The comparison between Pbase and Pgain on competition metrics should be reported with a metric that is not defined from the same pairwise difference being measured.","section":"Metrics for evaluating plausibility / Quantitative Evaluation of Plausibility Scores"}],"minor_comments":[{"comment":"The values in parentheses are not defined in the table caption. The main text implies they are improvements from community-based negatives, but the caption should state this explicitly.","section":"Table 5"},{"comment":"Condition (ii) uses an unquantified q. Clarify the relationship between q, p, and the edge set E, and explain why a triple that exists with a different predicate is treated as a negative for the target predicate.","section":"Definition 3"},{"comment":"The text says 'Only three annotations disagree' and then 'Two disagreements arise from weak structural support'. This is likely a typo for 'Two of the three disagreements arise...'.","section":"Case Study"},{"comment":"Because condition (ii) can generate as a negative a triple that exists in the KG with a different predicate, some training labels may be biologically true under an alternative relation. The authors should discuss this potential label noise or provide a robustness check.","section":"The Negative Selection Strategy"}],"recommendation":"major_revision","confidential_remarks":"The main quantitative claims rest on a fixable but fundamental protocol error: embeddings are computed on the full KG before the test split. If the authors re-run with embeddings computed after the split and provide a larger external gold standard for negative annotations, the paper could become publishable. Without such changes, the central claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it in depth. First, the central quantitative claim — community-based negative sampling improving balanced accuracy by 5.8% and the plausibility measures improving prioritization — is compromised by transductive leakage. The authors compute TransE embeddings on the entire KG, then hold out 10% of triples for testing. The test triples are therefore already encoded in the node and edge features the classifiers see. The statement that the blind test set is \"not used during training\" is true only for classifier training, not embedding. A classifier can easily pick up the trace of a held-out edge inside its subject and object embeddings. That inflates absolute accuracy, and it likely inflates the difference between community-based and random negatives. This is not a minor footnote; it is the load-bearing part of the evaluation.\n\nSecond, the paper does have genuine new content. The three plausibility formulations (Gain, Combo, SoftMax) and the competition metrics (separation rates) are new, and they address a real problem: ranking candidate annotations for expert review in a way that accounts for alternative predicates between the same entities. The writing is clear, the experiments cover five large bioKGs, and the code and data are public. The qualitative examples, such as competing predicates falling into the uncertain region, show the framework can behave sensibly. The community-based negative sampling strategy itself comes from their own workshop paper, so the novelty is in the packaging, not the sampling algorithm.\n\nWhere the paper is soft, beyond the leakage issue: Table 5 reports plausibility metrics without error bars, so we do not know whether the differences between Combo and Gain are noise. The case study has 29 annotations and no baseline — no randomized prioritization, no classifier-confidence-only ranking — so it cannot support the claim of practical utility. And the test negatives are generated with the same community-based procedure used for training negatives, so the \"generated negatives\" are partly circular; real false annotations from literature mining will not look exactly like cross-community pairs.\n\nIf the authors re-run the experiments with embeddings computed only on the training split, and then re-evaluate the plausibility scores, the paper could be salvageable. The framework is worth keeping, the evaluation needs to be redone. I would not cite this version, but I would send it to peer review with a clear request for the re-embedding experiment and an external gold standard. The flaw is fixable, and the problem is real.","headline":"The plausibility framework is reasonable and clearly presented, but the headline numbers are not trustworthy because embeddings are learned on the full KG before the test split, so the held-out triples are not truly unseen.","tokens_in":20733,"tokens_out":1982,"would_cite":false,"duration_ms":24776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A framework that scores the plausibility of candidate biomedical annotations, not just classifier confidence, can better prioritize which ones deserve expert review.","keywords":["plausibility","biomedical annotations","knowledge graph","negative sampling","community detection","classifier confidence","annotation prioritization","expert curation"],"falsifier":"Rerun the pipeline with embeddings computed only on the training 90% of edges (or with test edges explicitly removed before embedding). If the average balanced-accuracy gain over random negatives falls substantially below 5.8%, or if P_combo's separation advantage disappears, the central claim about community-based negatives and competition-aware plausibility would not hold.","tokens_in":19854,"feed_emoji":"🧬","tokens_out":2568,"duration_ms":29470,"temperature":0.7,"pith_summary":"The paper argues that the bottleneck in biomedical curation is no longer generating candidate annotations but deciding which ones to check first. It proposes replacing raw classifier confidence with plausibility scores that also weigh competing predicates linking the same entity pair, and trains classifiers on harder negatives drawn from different graph communities. On five large biomedical knowledge graphs, this community-based negative sampling raises balanced accuracy by an average of 5.8%, and the proposed plausibility measures separate true annotations from alternatives better than confidence alone. If the claim holds, curators could use these scores to triage large candidate lists, focusing human effort on genuinely uncertain cases.","feed_headline":"Plausibility scores outrank confidence for prioritizing annotations","feed_subtitle":"A knowledge-graph framework lifts balanced accuracy by 5.8% and separates true annotations from competing predicates.","key_machinery":"The central object is the schema-fact partition of a bioKG: the graph is split into homogeneous subgraphs by schema fact (s, p, t), and for each partition a binary classifier is trained on edge embeddings (the Hadamard product of TransE node embeddings). Negative examples are generated by sampling from nodes in different communities detected by Louvain. The load-bearing identity is the relative gain Δ_fi = Model_fi(s,p,t)·w_i − max_{j≠i}{Model_fj(s,p_j,t)·w_j, 0}, which expresses how much the target predicate outscores its strongest competitor after weighting by classifier reliability and partition size. Gain, Combo, and SoftMax plausibilities are then defined by sigmoid or softmax transform","core_discovery":"The central claim is that plausibility—defined as classifier confidence adjusted for classifier reliability and for competition among alternative predicates that connect the same two biological entities—ranks candidate annotations more usefully than confidence alone. The paper constructs three competition-aware formulations (Gain, Combo, SoftMax) and shows that the Combo score, a weighted mix of base confidence and the gain over the strongest competitor, offers the best balance between calibration and predicate separation. Community-based negative sampling, which builds negatives from nodes in different Louvain communities, yields a 5.8% average improvement in balanced accuracy and assigns h","pith_inferences":["A direct testable extension: run the same plausibility scoring on a much larger, independently annotated set of candidate annotations and measure agreement with expert binary decisions; the 29-annotation case study is too small to establish practical utility.","The competition-aware idea should transfer to any multi-label classification task with overlapping relations—drug–drug interactions, regulatory networks, or general knowledge graphs—where the strongest competitor provides a natural reference point.","A cautionary inference: since node and edge embeddings are computed on the whole graph before the 10% split, some of the reported gains could reflect information leakage from test edges into the embeddings; a cleaner protocol would fit embeddings on the training subgraph only.","The community-based negatives are likely 'harder' in that they connect structurally plausible distant nodes; this may force the classifier to learn more discriminating features, but it could also over-penalize true but cross-community biological associations."],"forward_implications":["Curators could use plausibility regions—plausible, uncertain, implausible—to triage incoming annotation candidates, spending expert time on the uncertain band.","The framework is embedding-agnostic and KG-agnostic, so the same pipeline could be applied to other heterogeneous knowledge graphs beyond biomedicine.","Community-based negatives improve classifier robustness without increasing runtime, making the gain scalable to very large partitions.","The case study suggests plausibility scores may also expose schema-level mismatches, such as relations that the KG structure does not support.","Because the negatives differ sharply between community-based and random sampling (≤3.5% overlap), the performance difference is attributable to negative quality, not trivial overlap."],"fun_headline_variants":["Plausibility beats confidence for ranking biomedical annotations","Knowledge-graph plausibility improves annotation prioritization","Competition-aware plausibility outperforms raw confidence","Plausibility scoring lifts annotation ranking accuracy by 5.8%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes the 10% held-out positive triples are truly unseen, but embeddings are computed on the full graph before the split; if those embeddings encode the presence of the held-out edges, the reported accuracy and plausibility improvements are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Plausibility beats confidence for ranking biomedical annotations","Knowledge-graph plausibility improves annotation prioritization","Competition-aware plausibility outperforms raw confidence","Plausibility scoring lifts annotation ranking accuracy by 5.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000931,"raw_usage":{"total_tokens":3827,"prompt_tokens":753,"completion_tokens":3074,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":3010}},"tokens_in":497,"tokens_out":3074,"duration_ms":21259,"temperature":1.0,"reasoning_tokens":3010,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:34:48.204408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline with embeddings computed only on the training 90% of edges (or with test edges explicitly removed before embedding). If the average balanced-accuracy gain over random negatives falls substantially below 5.8%, or if P_combo's separation advantage disappears, the central claim about community-based negatives and competition-aware plausibility would not hold.","supporting_citations":[],"review_version":1}