Plausibility-Driven Prioritization of Candidate Biomedical Annotations

Dario Malchiodi; Emanuele Cavalleri; Marco Mesiti; Miad Alavinezhad

Plausibility-Driven Prioritization of Candidate Biomedical Annotations

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A framework that scores the plausibility of candidate biomedical annotations, not just classifier confidence, can better prioritize which ones deserve expert review.

desk verdict The plausibility framework is reasonable and clearly presented, but the headline numbers are not trustworthy because embeddings are learned on the full KG before the test split, so the held-out triples are not truly unseen. read the letter →

arxiv 2607.20163 v2 pith:4GOFU6CG submitted 2026-07-22 q-bio.QM cs.DBcs.LG

Emanuele Cavalleri , Miad Alavinezhad , Dario Malchiodi , Marco Mesiti This is my paper · ORCID

classification q-bio.QMcs.DBcs.LG

keywords plausibilitybiomedicalannotationsknowledgegraphnegativesamplingcommunitydetectionclassifierconfidenceannotationprioritizationexpertcuration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the bottleneck in biomedical curation is no longer generating candidate annotations but deciding which ones to check first. It proposes replacing raw classifier confidence with plausibility scores that also weigh competing predicates linking the same entity pair, and trains classifiers on harder negatives drawn from different graph communities. On five large biomedical knowledge graphs, this community-based negative sampling raises balanced accuracy by an average of 5.8%, and the proposed plausibility measures separate true annotations from alternatives better than confidence alone. If the claim holds, curators could use these scores to triage large candidate lists, focusing human effort on genuinely uncertain cases.

What carries the argument

The central object is the schema-fact partition of a bioKG: the graph is split into homogeneous subgraphs by schema fact (s, p, t), and for each partition a binary classifier is trained on edge embeddings (the Hadamard product of TransE node embeddings). Negative examples are generated by sampling from nodes in different communities detected by Louvain. The load-bearing identity is the relative gain Δ_fi = Model_fi(s,p,t)·w_i − max_{j≠i}{Model_fj(s,p_j,t)·w_j, 0}, which expresses how much the target predicate outscores its strongest competitor after weighting by classifier reliability and partition size. Gain, Combo, and SoftMax plausibilities are then defined by sigmoid or softmax transform

What would settle it

Rerun the pipeline with embeddings computed only on the training 90% of edges (or with test edges explicitly removed before embedding). If the average balanced-accuracy gain over random negatives falls substantially below 5.8%, or if P_combo's separation advantage disappears, the central claim about community-based negatives and competition-aware plausibility would not hold.

Watch

Extended reading notes

Core claim

The central claim is that plausibility—defined as classifier confidence adjusted for classifier reliability and for competition among alternative predicates that connect the same two biological entities—ranks candidate annotations more usefully than confidence alone. The paper constructs three competition-aware formulations (Gain, Combo, SoftMax) and shows that the Combo score, a weighted mix of base confidence and the gain over the strongest competitor, offers the best balance between calibration and predicate separation. Community-based negative sampling, which builds negatives from nodes in different Louvain communities, yields a 5.8% average improvement in balanced accuracy and assigns h

Load-bearing premise

The evaluation assumes the 10% held-out positive triples are truly unseen, but embeddings are computed on the full graph before the split; if those embeddings encode the presence of the held-out edges, the reported accuracy and plausibility improvements are inflated.

Editorial extensions

If this is right

Curators could use plausibility regions—plausible, uncertain, implausible—to triage incoming annotation candidates, spending expert time on the uncertain band.
The framework is embedding-agnostic and KG-agnostic, so the same pipeline could be applied to other heterogeneous knowledge graphs beyond biomedicine.
Community-based negatives improve classifier robustness without increasing runtime, making the gain scalable to very large partitions.
The case study suggests plausibility scores may also expose schema-level mismatches, such as relations that the KG structure does not support.
Because the negatives differ sharply between community-based and random sampling (≤3.5% overlap), the performance difference is attributable to negative quality, not trivial overlap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct testable extension: run the same plausibility scoring on a much larger, independently annotated set of candidate annotations and measure agreement with expert binary decisions; the 29-annotation case study is too small to establish practical utility.
The competition-aware idea should transfer to any multi-label classification task with overlapping relations—drug–drug interactions, regulatory networks, or general knowledge graphs—where the strongest competitor provides a natural reference point.
A cautionary inference: since node and edge embeddings are computed on the whole graph before the 10% split, some of the reported gains could reflect information leakage from test edges into the embeddings; a cleaner protocol would fit embeddings on the training subgraph only.
The community-based negatives are likely 'harder' in that they connect structurally plausible distant nodes; this may force the classifier to learn more discriminating features, but it could also over-penalize true but cross-community biological associations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper before reading it in depth. First, the central quantitative claim — community-based negative sampling improving balanced accuracy by 5.8% and the plausibility measures improving prioritization — is compromised by transductive leakage. The authors compute TransE embeddings on the entire KG, then hold out 10% of triples for testing. The test triples are therefore already encoded in the node and edge features the classifiers see. The statement that the blind test set is "not used during training" is true only for classifier training, not embedding. A classifier can easily pick up the trace of a held-out edge inside its subject and object embeddings. That inflates absolute accuracy, and it likely inflates the difference between community-based and random negatives. This is not a minor footnote; it is the load-bearing part of the evaluation.

Second, the paper does have genuine new content. The three plausibility formulations (Gain, Combo, SoftMax) and the competition metrics (separation rates) are new, and they address a real problem: ranking candidate annotations for expert review in a way that accounts for alternative predicates between the same entities. The writing is clear, the experiments cover five large bioKGs, and the code and data are public. The qualitative examples, such as competing predicates falling into the uncertain region, show the framework can behave sensibly. The community-based negative sampling strategy itself comes from their own workshop paper, so the novelty is in the packaging, not the sampling algorithm.

Where the paper is soft, beyond the leakage issue: Table 5 reports plausibility metrics without error bars, so we do not know whether the differences between Combo and Gain are noise. The case study has 29 annotations and no baseline — no randomized prioritization, no classifier-confidence-only ranking — so it cannot support the claim of practical utility. And the test negatives are generated with the same community-based procedure used for training negatives, so the "generated negatives" are partly circular; real false annotations from literature mining will not look exactly like cross-community pairs.

If the authors re-run the experiments with embeddings computed only on the training split, and then re-evaluate the plausibility scores, the paper could be salvageable. The framework is worth keeping, the evaluation needs to be redone. I would not cite this version, but I would send it to peer review with a clear request for the re-embedding experiment and an external gold standard. The flaw is fixable, and the problem is real.

Referee Report

3 major / 4 minor

Summary. The paper proposes a framework for prioritizing candidate biomedical annotations for expert curation. The pipeline first computes knowledge-graph embeddings (default TransE) for each of five bioKGs, partitions edges by schema facts, trains a relation-specific random-forest classifier for each schema fact using either random or community-based negative sampling, and then derives plausibility scores (Base, Gain, Combo, SoftMax) that combine classifier confidence, classifier reliability, and competing predicates. The authors report that community-based negative sampling improves balanced accuracy by 5.8% on average, that Pcombo provides a balanced trade-off between calibration and competition-aware separation, and that these scores can support curator workflows. An open-source implementation and a small expert-annotated case study are included.

Significance. The framework addresses a real bottleneck in biocuration, and the idea of making plausibility estimates competition-aware is potentially valuable. The paper is clearly written, and the authors provide open code/data and evaluate on five large, heterogeneous bioKGs. The formal definitions and the negative-sampling procedure are concrete and reproducible. However, as described, the main experimental protocol does not support the central quantitative claims: the 'blind' positive test set is likely contaminated by transductive embedding leakage, and the negative test set is generated from the same distribution used to train the classifiers. These issues affect the 5.8% accuracy gain and the plausibility comparisons in Table 5, so the contribution is not yet established.

major comments (3)

[Experimental Protocol / KG Embedding and Binary Classifiers] The blind test set is not blind. The manuscript states that 'A KG G is first embedded into a latent space using a KGE model' (Section 'KG Embedding and Binary Classifiers', Figure 2), and only afterwards the Experimental Protocol holds out 10% of observed triples as BlP_i. Nowhere do the authors re-embed the KG after removing BlP_i. TransE node embeddings are optimized on all edges of the full KG, so the existence of each held-out test triple is encoded in the embeddings of its subject and object nodes. The relation-specific classifiers are trained on the remaining 90%, but their input features for a test triple are computed from these contaminated embeddings. Consequently, the test labels can influence the classifier features, inflating balanced accuracy and every downstream plausibility metric in Table 5, Figure 5, and Supp. Tables S1–S5 (including the 5.8% average improvement). The au
[Experimental Protocol / Metrics for evaluating plausibility] The negative test set shares the generation mechanism with the training negatives. The protocol creates N_i via 'the adopted negative sampling strategy' and, for each experiment, the training negatives are generated by the same strategy (community-based or random). The NRR and Cal metrics in Table 5 therefore measure rejection of synthetic negatives drawn from the same distribution that the classifier was trained to reject. This is not an external gold standard for 'implausible' candidate annotations, and the improvement attributed to community-based negatives is expected under this setup. Likewise, the positive test set consists of existing KG edges rather than newly extracted candidate annotations. The only external evaluation, the 29-annotation case study (Section 'Case Study'), has just 3 disagreements and 13 uncertain cases and is too small to establish prioritization quality. A sep
[Metrics for evaluating plausibility / Quantitative Evaluation of Plausibility Scores] The high Sepbest value for Pgain (99.3% in Table 5) is largely a by-product of the definition of Pgain. Pgain is defined as σ(λ·(weighted target score − max_j weighted competitor score)), so for the triple with the highest weighted score, the target and the strongest competitor receive scores that are mirror images (one above 0.5, one below 0.5 if the same set of competitors is used). Any threshold between those values will put them on opposite sides. Thus Sepbest≈100% does not provide independent evidence that Pgain 'better separates' biologically correct predicates; it mainly reflects the construction of the score. The comparison between Pbase and Pgain on competition metrics should be reported with a metric that is not defined from the same pairwise difference being measured.

minor comments (4)

[Table 5] The values in parentheses are not defined in the table caption. The main text implies they are improvements from community-based negatives, but the caption should state this explicitly.
[Definition 3] Condition (ii) uses an unquantified q. Clarify the relationship between q, p, and the edge set E, and explain why a triple that exists with a different predicate is treated as a negative for the target predicate.
[Case Study] The text says 'Only three annotations disagree' and then 'Two disagreements arise from weak structural support'. This is likely a typo for 'Two of the three disagreements arise...'.
[The Negative Selection Strategy] Because condition (ii) can generate as a negative a triple that exists in the KG with a different predicate, some training labels may be biologically true under an alternative relation. The authors should discuss this potential label noise or provide a robustness check.

Circularity Check

2 steps flagged · score 6.0 of 10

Blind test set not blind: embeddings computed on the full KG encode the held-out triples, and negative test sets are generated by the strategy being evaluated.

fitted input called prediction [Section 'KG Embedding and Binary Classifiers' (p. 3, Fig. 2); 'Experimental Protocol' (p. 5)]
"A KG G is first embedded into a latent space using a KGE model... Node and edge embeddings are first computed and then partitioned according to the schema facts. ... For each schema fact f_i, the observed triples were first divided into two disjoint subsets. A random 10% was held out before training to build the blind positive test set BlP_i ... Neither BlP_i nor N_i is used during training, validation, or model selection."

The embedding step is performed on the full KG before the 10% split, so the 'blind' positive test triples are included in the TransE training edges. The classifier input for a test triple is a function (Hadamard product) of the node embeddings of its subject and object, and those embeddings were optimized using the very triple being predicted. The test label is therefore already encoded in the features, making the reported balanced-accuracy and confidence gains an in-sample (leaked) evaluation rather than a prediction of unseen triples. No re-embedding step after removing BlP_i is described, so the headline results are not independent of the labels they purport to predict.
other [Experimental Protocol (p. 5); Evaluating the Classification Approach (p. 7)]
"Since bioKGs do not contain explicit negative facts, an equally sized set of implausible triples N_i was generated through the adopted negative sampling strategy. Neither BlP_i nor N_i is used during training, validation, or model selection. ... Negative training examples were generated either through random sampling or through the proposed community-based strategy, depending on the experiment."

When comparing the community-based strategy with the random baseline, each classifier is tested on negatives generated by the same strategy used to train it. The community-based model is thus evaluated on negatives drawn from the exact distribution it was trained to reject, while the random model faces a different, uniform-negative distribution. The reported +5.8% balanced-accuracy improvement is therefore not a paired comparison on a fixed, independent test set; it partly measures how well each model matches its own negative generator rather than a general robustness gain.

full rationale

The paper's central quantitative claims are derived from an evaluation protocol that is circular in two ways. First, the KG embeddings are computed on the full graph before the 10% held-out split, so the supposedly blind positive test triples are already present in the features used to classify them; every plausibility measure inherits this leak. Second, the negative test set N_i is generated by the same strategy being evaluated, so the community-versus-random comparison is confounded by different test distributions. The self-citation to the authors' prior workshop paper [9] is not load-bearing because the present work independently re-runs the comparison, and the plausibility equations themselves are internally coherent. However, the headline numbers—5.8% balanced accuracy, PAR/NRR values, and the case-study agreement—all reduce to evaluating the model on its own inputs, so the main empirical claims are partially circular.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new biological entities. Its free parameters are standard ML hyperparameters and heuristic constants selected on validation data. The most important assumptions are the validity of generated negatives as a proxy for implausibility and the absence of transductive leakage in the embedding step.

free parameters (6)

lambda (λ) = 10
Controls sharpness of the sigmoid in Gain Plausibility and the softmax temperature in SoftMax Plausibility. Selected as best compromise between calibration and separation on validation data.
alpha (α) = 0.5
Mixing parameter in Combo Plausibility, balancing classifier confidence and competition-aware gain. Tuned on validation metrics.
k (top-k competitors) = 2
Number of competing schema facts used in SoftMax Plausibility. Selected because limited improvements were observed beyond k=2.
beta (β) = 1
Used in the weighted harmonic mean for classifier relevance weight w_i and in the F-hat metric, emphasizing balanced accuracy and false-positive/false-negative trade-off equally.
Embedding dimension = 32
Node and edge embedding dimensionality, chosen without reported justification or sensitivity analysis.
Random 1.5M edge cap = 1.5M
For partitions with more than 1.5M edges, triples were randomly sampled to cap computational cost; this affects large schema facts like Disease-ASSOCIATED WITH-Gene in OptimusKG.

assumptions (4)

domain assumption A triple not present in the KG and connecting different communities is a valid negative example of an annotation.
The entire evaluation treats generated negatives as implausible annotations. This is an assumption about biological validity that is never tested against external labels.
domain assumption TransE embeddings computed on the full KG can be used as features for held-out triples without leaking label information.
The protocol computes embeddings before splitting; this assumes no information about held-out edges leaks into the classifier features. This is likely false in a transductive setting.
domain assumption The Louvain community structure of each induced subgraph is biologically meaningful for negative sample generation.
The community-based strategy assumes that edges across communities are less likely to be true than edges within communities; this biological interpretation is not validated.
domain assumption Classifier calibration on KG edges transfers to candidate annotations extracted from literature by LLMs.
The case study applies classifiers trained on KG edges to LLM-extracted candidates, assuming the distribution is similar enough for prioritization to work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Plausibility-Driven Prioritization of Candidate Biomedical Annotations." pith.science (2026). https://pith.science/paper/4GOFU6CG

@misc{pith2026260720163,
  author       = {Pith},
  title        = {Pith review of: Plausibility-Driven Prioritization of Candidate Biomedical Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GOFU6CG}},
  note         = {Machine review of arXiv:2607.20163}
}

read the original abstract

The rapid growth of biomedical knowledge has made the validation of automatically generated biological annotations a major bottleneck in biomedical curation. While computational methods can rapidly produce large numbers of candidate annotations, determining which are biologically valid still requires costly expert review. Prioritizing these candidates before manual curation has therefore become a fundamental challenge. Machine learning techniques can support this process by exploiting biomedical knowledge graphs (bioKGs), which capture biological entities and their functional associations. In this work, we propose a framework that leverages bioKGs to estimate the plausibility of candidate annotations and guide expert curation. Starting from knowledge graph embeddings, we train relation-specific binary classifiers using a community-based negative sampling strategy to obtain reliable confidence estimates. We then introduce a family of plausibility measures that combine classifier confidence, classifier reliability, and the semantic context provided by alternative relationships involving the same pair of biological entities. Unlike conventional confidence estimation, the proposed approach explicitly accounts for multiple biologically meaningful relations that may coexist between the same entities. Experimental results on five large bioKGs demonstrate that the proposed negative sampling strategy consistently improves classifier robustness, increasing balanced accuracy by an average of 5.8%. Moreover, the plausibility measures outperform classifier confidence alone, enabling more effective prioritization of candidate annotations for expert review. Overall, our results show that the use of bioKGs improves the efficiency of AI-assisted biomedical curation while preserving expert control over the final annotation assessment.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.