Pith. sign in

REVIEW 4 major objections 6 minor 9 references

Enhancing Biomedical Relation Extraction with Directionality

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper enriches the BioRED corpus with 10,864 subject/object directionality annotations and demonstrates a soft-prompt multi-task model that jointly predicts relation type, novelty, and directionality, outperforming fine-tuned GPT-3.5…

desk verdict Genuinely useful directionality annotation for BioRED, but unmeasured single-annotator reliability and an overclaimed BC5CDR significance keep it from being a solid benchmark as-is. read the letter →

arxiv 2501.14079 v1 pith:MZSI5PKK submitted 2025-01-23 cs.CL

Po-Ting Lai , Chih-Hsuan Wei , Shubo Tian , Robert Leaman , Zhiyong Lu This is my paper · ORCID
classification cs.CL
keywords relationextractiondirectionalityBioREDmulti-tasklearningsoftpromptdocument-levelbiomedicaltextminingentityroles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to add the one thing missing from the BioRED relation corpus: the direction of each relationship, that is, which entity is the subject and which is the object. Across 1,000 abstracts the authors manually assigned these entity-role labels, producing 10,864 directionality annotations. They then train a BERT-based multi-task model that jointly predicts relation type, novelty, and directionality, using soft prompts and a prefix/suffix/infix chunking scheme to cope with documents longer than 512 tokens. On the BioRED-BC8 test set the model reaches 75.34 entity-pair F1, 56.06 relation-type F1, 48.62 with directionality, and 37.02 for all labels together; on BC5CDR it reaches 71.4 F1, ahead of fine-tuned GPT-3.5 and Llama 3.2. The intended payoff is a corpus and model that support constructing directed biological networks rather than undirected association graphs.

What carries the argument

The load-bearing machinery is the directionality annotation itself plus a multi-task transformer architecture. Directionality is recorded per relation pair as one of Rightward (first entity subject), Leftward (second entity subject), Undirected, or None, and this label is predicted alongside relation type and novelty. The model represents each document as a hard prompt with special tokens [REL], [DIR], and [NOV], a learned soft prompt, and three overlapping context chunks (prefix, suffix, infix) to stay within BERT's 512-token limit; the hidden states of the task tokens are pooled with a hidden-token pooler initialized from BERT's pooler, max-pooled across chunks, and scored with separate classifiers. A weighted binary cross-entropy loss combines the three tasks.

What would settle it

Re-annotate a random sample of the 950 single-annotated abstracts with a second curator and compute pairwise agreement; if it falls well below the reported 89.96%, the directionality-dependent F1 scores in Tables 2 and 3 partly measure label noise. A second check is to have domain experts review the model's predicted directionality on a held-out set and see whether the Undirected label is being overused for hard gene-gene and chemical-gene pairs.

Watch

Extended reading notes

Core claim

The central claim is that directionality of biomedical relations can be annotated reliably on top of BioRED and predicted jointly with relation type and novelty without sacrificing performance on the original tasks. The authors define directionality as a four-way label per entity pair: Rightward, Leftward, Undirected, or None, and they provide 10,864 such labels. Their model, built on a biomedical language model with task-specific hard prompts and learned soft prompts, treats the three prediction targets as jointly trained multi-label classification problems, with context chunking supplying local and document-level evidence. In their experiments it outperforms fine-tuned GPT-3.5 and Llama 3.2 on both benchmarks, and it improves over the previous BioREx and PubMedBERT systems on BioRED-BC8 while matching or exceeding them on BC5CDR. The paper's message is that directed relation extraction is both annotatable and learnable, so downstream network construction no longer has to flatten causal or activating relationships into symmetric associations.

Load-bearing premise

The directionality labels are trustworthy enough to benchmark against, which relies on the high agreement measured on 50 practice abstracts carrying over to the 950 abstracts that each received only a single annotator's judgment.

Editorial extensions

If this is right

  • The enriched BioRED corpus with 10,864 directionality annotations can serve as a standard benchmark for directed biomedical relation extraction.
  • If the reported scores hold, researchers can build directed networks of gene, chemical, disease, and variant relationships directly from abstracts without a separate subject/object assignment pipeline.
  • The combination of multi-task learning, soft prompts, and prefix/suffix/infix chunking offers a reusable recipe for other long-document language model tasks that carry multiple label sets.
  • Soft prompting at length 8 gives consistent gains over no soft prompt, making it a low-cost enhancement for biomedical relation extraction.
  • Despite parameter-efficient fine-tuning, general-purpose LLMs remain below the proposed smaller task-specific model, indicating that a domain-tuned model remains competitive for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: reliable directionality labels would let BioRED support causal or pathway-oriented network construction, not just undirected association graphs; the paper motivates this but does not build such a network itself.
  • Inference: because only BioRED carries these labels, a natural extension is to use them as a secondary training signal for other document-level relation extraction datasets, where subject/object roles are often implied but not annotated.
  • Inference: the gap between relation-type F1 and relation-type-plus-directionality F1 suggests directionality is the hardest part of the task, so future work could target explicit causal-verb cues or curated signaling-pathway knowledge, neither of which the current model uses.
  • Inference: the quickest empirical check of the benchmark's trustworthiness is re-annotating a random sample of the 950 single-annotator abstracts, since the reported 89.96% agreement was measured on only 50 practice abstracts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper enriches the BioRED corpus with directionality annotations (subject/object roles) for 10,864 relation instances and proposes a multi-task BERT-style model with soft-prompt learning and context chunking to jointly predict relation type, novelty, and directionality. The model is evaluated on the BioCreative VIII BioRED test set and on BC5CDR, with comparisons to fine-tuned LLMs (GPT-3.5, Llama 3.2) and to prior BERT-based systems. The authors report improved F1 scores on most BioRED metrics and a numerical but explicitly non-significant improvement on BC5CDR over BioREx. The paper also includes ablations for context chunks and soft-prompt length, an error analysis, and releases source code and the annotated corpus.

Significance. If the directionality annotations are reliable, the resource is a valuable extension of BioRED that enables directed biomedical network construction, and the multi-task model with soft prompts provides a practical approach for joint directionality, novelty, and relation extraction. The paper is transparent about some limitations, such as the non-significant BC5CDR gain and the partial significance of soft-prompt improvements. The release of code and data supports reproducibility. However, the benchmark value of the corpus hinges on annotation reliability beyond the 50-abstract pilot, and several significance claims are not fully supported by the reported statistics.

major comments (4)
  1. [§2.1.2] The annotation reliability evidence covers only the 50-abstract practice set: the authors report 89.96% pairwise agreement and then randomly assign the remaining 950 abstracts to single annotators, with no adjudication, second pass, or post-hoc inter-annotator agreement described. Because directionality is context-dependent and the paper's own §5.3 examples show genuine ambiguity (e.g., IL-17A and NF-kappaB), the 10,864 released directionality labels are not yet demonstrated to be a reliable gold standard. Please report chance-corrected agreement (e.g., Cohen's kappa) on a held-out sample or an additional annotation pass for the full corpus; otherwise every directionality-dependent result in Tables 2 and 5 partly reflects label noise.
  2. [§3.5 / Table 3] The text in §3.5 states that the BC5CDR improvement over BioREx 'is not statistically significant,' yet the Conclusion says the method 'significantly surpasses its predecessor, BioREx, particularly on the BC5CDR test set.' This is a direct contradiction. Please either report the significance test with its statistic and p-value, or revise the conclusion to say that the gain is numerical but not statistically significant.
  3. [§3.3 / Table 2] The significance markers ('*' and '**') in Table 2 are not supported by any description of the statistical testing procedure, standard deviations, confidence intervals, or p-values, even though §3.3 says experiments were run ten times with different random seeds. Please specify the test used (e.g., paired bootstrap or approximate randomization), report the variance or confidence intervals, and state whether any correction for multiple comparisons was applied.
  4. [§3.4 / Table 2] The GPT-4 comparison is zero-shot only, and the zero-shot rows do not include directionality or novelty labels (as noted in §3.3), so the abstract's claim of outperforming 'state-of-the-art GPT-4 and Llama-3' overstates the evidence. Please restrict the claim to the evaluated settings or provide fine-tuned GPT-4 results, especially for the directionality metrics.
minor comments (6)
  1. [Section numbering] The Discussion section is Section 4, but its subsections are numbered 5.1, 5.2, and 5.3; please renumber them as 4.1, 4.2, and 4.3.
  2. [§3.5] The sentence 'Interestingly, GPT-3.5, with fine-tuning, outperforms most state-of-the-art models except ours.' is duplicated verbatim; please remove the duplicate.
  3. [Table 2] The column header 'RT+DI. All' contains an extra period; it should be 'RT+DI' and 'All' as separate columns.
  4. [Abstract and Table 2] The abstract says 'state-of-the-art GPT-4 and Llama-3,' but the experiments use Llama3.2-11B; please specify the exact model versions to avoid ambiguity.
  5. [§2.3] The LLM prompt uses the first mentioned named entity for each pair, while BioRED relation instances may involve mentions that are not the first occurrence; please clarify how this affects the comparability of LLM results with the token-boundary-based BERT models.
  6. [§3.4] The BioREx row in Table 2 includes novelty prediction scores, but BioREx is described in the Introduction as focusing solely on relation type classification; please specify in the methods that the novelty scores were obtained by combining BioREx relation predictions with a PubMedBERT novelty classifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical annotation and benchmark study whose central claims rest on manual annotations and held-out test evaluations, not on a derivation that reduces to its inputs.

full rationale

This paper's central contributions are (1) a manual directionality annotation layer added to BioRED and (2) a multi-task model evaluated on held-out test sets (BioRED-BC8 and BC5CDR). No 'prediction' in the paper is derived by construction from a fitted parameter: the 10,864 directionality annotations are human labels, and the reported F1 scores are computed against held-out gold annotations. The only self-citation that plays a role in the method is the decision to initialize from BioREx, the authors' own prior system ('The pre-trained model of BioREx was selected as the pre-trained language model due to its specialized biomedical knowledge'), and BioREx is also used as a baseline. Using one's own system as a pretrained initialization is a legitimate empirical choice, not a circular reduction: the model's outputs are not constrained to equal BioREx's outputs, and the evaluation compares against BioREx and other systems on external benchmarks. The annotation-quality concern raised by the reader — single-annotator labels on 950 of the 1,000 abstracts with no post-hoc IAA — is a data-reliability limitation, not circularity, because the annotation guidelines and labels are not derived from the model's predictions or from a fitted constant. The paper is self-contained against external benchmarks and does not invoke a uniqueness theorem or an ansatz smuggled in via self-citation. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no invented scientific entities. The ledger captures two features of the empirical design: the released directionality corpus rests on an unadjudicated single-annotator assumption for 95% of abstracts, and the method builds on a same-group model (BioREx) as initialization. Free parameters are standard training choices; the paper reports ranges but not all final values.

free parameters (3)
  • Soft prompt length q = 8
    Table 5: q=8 yields the best trade-off on both test sets and is selected as the final setting in Ours; without it performance drops (EP 74.54 vs 75.34).
  • Training hyperparameters (learning rate, epochs, batch size) = Not fully reported
    §3.3 searches lr in {1e-5,5e-6,1e-6}, epochs in {5,10,20}, batch in {4,8,16} on development sets; the specific chosen combination is not stated in the text.
  • Learnable task weights w_k = Learned during training
    Equation (7) weights the REL, NOV, and DIR losses with w_k learned by the model, rather than fixed constants; these values are not reported.
assumptions (4)
  • domain assumption Gold relation and novelty labels in BioRED are correct and complete.
    The model is trained and evaluated against BioRED labels throughout §3 without questioning label noise or coverage; any incompleteness would change F1 comparability.
  • domain assumption Directionality labels on the 950 single-annotated abstracts are as reliable as the 89.96% IAA measured on 50 practice abstracts.
    §2.1.2 reports IAA only for the initial 50-abstract phase and then splits the remaining 950 abstracts between two annotators without adjudication; corpus quality is assumed to match the practice phase.
  • domain assumption BioREx can be reused as a biomedical pre-trained initialization without label leakage that biases the comparison.
    The paper initializes from the BioREx model developed by overlapping authors (§3.3) but does not analyze whether its prior training on multiple RE datasets, possibly including BioRED, gives an unfair advantage in the baseline comparison.
  • standard math Standard transformer, BCE loss, and max-pooling operations are valid.
    Equations (1)-(7) rely on conventional BERT and deep-learning machinery; no new mathematical claims are made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Biomedical Relation Extraction with Directionality." pith.science (2026). https://pith.science/paper/MZSI5PKK

@misc{pith2026250114079,
  author       = {Pith},
  title        = {Pith review of: Enhancing Biomedical Relation Extraction with Directionality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZSI5PKK}},
  note         = {Machine review of arXiv:2501.14079}
}
read the original abstract

Biological relation networks contain rich information for understanding the biological mechanisms behind the relationship of entities such as genes, proteins, diseases, and chemicals. The vast growth of biomedical literature poses significant challenges updating the network knowledge. The recent Biomedical Relation Extraction Dataset (BioRED) provides valuable manual annotations, facilitating the develop-ment of machine-learning and pre-trained language model approaches for automatically identifying novel document-level (inter-sentence context) relationships. Nonetheless, its annotations lack directionality (subject/object) for the entity roles, essential for studying complex biological networks. Herein we annotate the entity roles of the relationships in the BioRED corpus and subsequently propose a novel multi-task language model with soft-prompt learning to jointly identify the relationship, novel findings, and entity roles. Our results in-clude an enriched BioRED corpus with 10,864 directionality annotations. Moreover, our proposed method outperforms existing large language models such as the state-of-the-art GPT-4 and Llama-3 on two benchmarking tasks. Our source code and dataset are available at https://github.com/ncbi-nlp/BioREDirect.

Figures

Figures reproduced from arXiv: 2501.14079 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. An Example of BioRED prompt [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 6 canonical work pages

  1. [2]

    ChemProt-3.0: a global chemical biology diseases mapping

    Kringelum, J., et al. ChemProt-3.0: a global chemical biology diseases mapping. Database 2016;2016. Lai, P.-T. and Lu, Z. BERT-GT: cross-sentence n-ary relation extraction with BERT and Graph Transformer. Bioinformatics 2021;36(24):5678-5685. Lai, P.-T., et al. BioREx: Improving Biomedical Relation Extraction by Leveraging Heterogeneous Datasets. Journal ...

    arXiv 2016

  2. [6]

    and Kilicoglu, H

    Ming, S., Zhang, R. and Kilicoglu, H. Enhancing the coverage of SemRep using a relation classification approach. Journal of Biomedical Informatics 2024;155:104658. Miranda, A., et al. Overview of DrugProt BioCreative VII track: quality evaluation and large scale text mining of drug -gene/protein relations. In, Proceedings of the seventh BioCreative challe...

    work page 2024

  3. [8]

    https://openai.com Peng, C

    ChatGPT: GPT-3.5 Model. https://openai.com Peng, C. , et al. Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction. Journal of Biomedical Informatics 2024;153:104630. Wei, C.-H., et al. Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical -disease re...

    arXiv 2024

  4. [441]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., et al. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 2024;36. Dirkson, A., et al. How do others cope? Extracting coping strategies for adverse drug events from social media. Journal of Biomedical Informatics 2023;139:104228. Gu, Y., et al. Domain-specific language model pretraining for biome...

    work page 2024

  5. [2019]

    4925 -4936

    p. 4925 -4936. Christopoulou, F., et al. Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods. Journal of the American Medical Informatics Association 2019;27(1):39-46. Dettmers, T., et al. QLORA: efficient finetuning of quantized LLMs. In, Proceedings of the 37th International Conference ...

    work page 2019

  6. [2021]

    ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing

    Neumann, M., et al. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. ArXiv 2019;abs/1902.07669. OpenAi

    arXiv 2019

  7. [2022]

    p. 10-20. Huang, M. -S., et al. Surveying biomedical relation extraction: a critical examination of current datasets and the proposal of a new resource. Briefings in Bioinformatics 2024;25(3):bbae132. Islamaj, R., et al. TeamTat: a collaborative text annotation tool. Nucleic Acids Research 2020;48(W1):W5-W11. Islamaj, R., et al. The overview of the BioRED...

    work page 2024

  8. [2023]

    The biomedical relationship corpus of the BioRED track at the BioCreative VIII challenge and workshop

    Islamaj, R., et al. The biomedical relationship corpus of the BioRED track at the BioCreative VIII challenge and workshop. Database 2024;2024. Kenton, J.D.M.-W.C. and Toutanova, L.K. Bert: Pre-training of deep bidirectional transformers for language understanding. In, Proceedings of naacL-HLT

    work page 2024

Show all 9 references
  1. [2024]

    CD-REST: a system for extracting chemical-induced disease relation in literature

    Xu, J., et al. CD-REST: a system for extracting chemical-induced disease relation in literature. Database 2016;2016. Yuan, J., et al. Document-level biomedical relation extraction via hierarchical tree graph and relation segmentation module. Bioinformatics 2024;40(7). Zhou, W....

    2016

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.