{"id":"3f20977e-ea6a-4cca-83f0-f1aa959e499d","arxiv_id":"2501.14079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors release BioRED with 10,864 new directionality annotations and show a BERT-based multi-task soft-prompt system outperforms BioREx and several LLMs on relation extraction benchmarks.","lead":"This paper adds subject/object labels to 10,864 relationships in the BioRED biomedical abstracts and trains a model that reads full abstracts to predict the relationship, whether it is new, and which entity acts on which. A generalist might care because knowing which gene activates or inhibits another is what lets researchers build meaningful disease and drug networks from millions of papers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 10,864 directionality labels rest on single-annotator decisions for 950 abstracts with no adjudication or post-hoc IAA; the 89.96% agreement on 50 practice abstracts does not, by itself, establish corpus-level label reliability.","rationale":"The central claim is two-part: the released directionality annotations are reliable enough to serve as a benchmark, and the proposed model achieves strong performance on that benchmark. The first part is the more load-bearing because every directionality-dependent F1 score in the paper is computed against these gold labels; if the labels are noisy, the benchmark itself is compromised, and model comparisons become uninterpretable. The reader's weakest assumption identifies exactly this point, and the manuscript text supports it: the only reported IAA is from 50 practice abstracts, while the remaining 950 were singly annotated with no described adjudication or post-hoc consistency check. The pilot agreement, 89.96% pairwise accuracy, does not establish corpus-level reliability, especially for rare or ambiguous relation types that the paper itself discusses in §5.3. This is a missing measurement, not a claim of misconduct. One secondary flag is that Table 1 reports 12,536 relation/novelty pairs but only 10,864 directionality annotations; the paper does not explain whether 1,672 pairs were intentionally excluded or are 'None' pairs, which also bears on the completeness of the released corpus. Because the reader's conditional verdict already requires adjudication or uncertainty quantification for the single-annotator labels, my stress-test does not move the verdict; it reinforces the condition.","tokens_in":11334,"tokens_out":7987,"duration_ms":74502,"concrete_test":"Randomly sample approximately 100 abstracts from the 950 single-annotated abstracts; have both original annotators independently re-annotate directionality for all relation pairs, and have a third expert adjudicate disagreements. Compute pairwise agreement and Cohen's kappa, overall and per relation type, between the original labels and the adjudicated labels. If agreement on this held-out sample is not at or above the pilot's 89.96% (or is uneven across relation types), the corpus should be adjudicated and the directionality F1 scores in Tables 2 and 5 recomputed on corrected labels; if agreement is comparable, the single-annotator assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1.2 reports that two annotators reached 89.96% pairwise agreement on 50 practice abstracts, then 'randomly assigned the remaining 950 abstracts to the two annotators, with each abstract being annotated by one individual.' No adjudication, second pass, or reliability measurement for those 950 abstracts is described. The 10,864 directionality annotations are therefore the product of one annotator's per-abstract judgment, and the only published evidence that this judgment matches the guideline is from the 50-abstract pilot. That evidence is not sufficient to certify a benchmark, because directionality is inherently context-dependent: the paper's own error analysis (§5.3) shows that cases such as 'Association' between genes and the direction of 'TNFalpha/IL-17A-induced... NF-kappaB' are difficult enough that trained annotators may reasonably disagree. The reported 89.96% is also raw pairwise agreement rather than a chance-corrected measure such as Cohen's kappa, so it may overstate reliability. If single-annotator labels on the 950 abstracts are materially noisier than the pilot, then every directionality-dependent result in Tables 2 and 5 (RT+DI 48.62, All 37.02) is partly a measure of label noise, and the released corpus would need adjudication before being used as a gold standard for directed network construction. This is a missing measurement rather than an accusation of error; the paper should quantify post-annotation consistency and, if it is not high, provide adjudicated labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper enriches the BioRED corpus with directionality annotations (subject/object roles) for 10,864 relation instances and proposes a multi-task BERT-style model with soft-prompt learning and context chunking to jointly predict relation type, novelty, and directionality. The model is evaluated on the BioCreative VIII BioRED test set and on BC5CDR, with comparisons to fine-tuned LLMs (GPT-3.5, Llama 3.2) and to prior BERT-based systems. The authors report improved F1 scores on most BioRED metrics and a numerical but explicitly non-significant improvement on BC5CDR over BioREx. The paper also includes ablations for context chunks and soft-prompt length, an error analysis, and releases source code and the annotated corpus.","tokens_in":11617,"tokens_out":3365,"duration_ms":29604,"significance":"If the directionality annotations are reliable, the resource is a valuable extension of BioRED that enables directed biomedical network construction, and the multi-task model with soft prompts provides a practical approach for joint directionality, novelty, and relation extraction. The paper is transparent about some limitations, such as the non-significant BC5CDR gain and the partial significance of soft-prompt improvements. The release of code and data supports reproducibility. However, the benchmark value of the corpus hinges on annotation reliability beyond the 50-abstract pilot, and several significance claims are not fully supported by the reported statistics.","major_comments":[{"comment":"The annotation reliability evidence covers only the 50-abstract practice set: the authors report 89.96% pairwise agreement and then randomly assign the remaining 950 abstracts to single annotators, with no adjudication, second pass, or post-hoc inter-annotator agreement described. Because directionality is context-dependent and the paper's own §5.3 examples show genuine ambiguity (e.g., IL-17A and NF-kappaB), the 10,864 released directionality labels are not yet demonstrated to be a reliable gold standard. Please report chance-corrected agreement (e.g., Cohen's kappa) on a held-out sample or an additional annotation pass for the full corpus; otherwise every directionality-dependent result in Tables 2 and 5 partly reflects label noise.","section":"§2.1.2"},{"comment":"The text in §3.5 states that the BC5CDR improvement over BioREx 'is not statistically significant,' yet the Conclusion says the method 'significantly surpasses its predecessor, BioREx, particularly on the BC5CDR test set.' This is a direct contradiction. Please either report the significance test with its statistic and p-value, or revise the conclusion to say that the gain is numerical but not statistically significant.","section":"§3.5 / Table 3"},{"comment":"The significance markers ('*' and '**') in Table 2 are not supported by any description of the statistical testing procedure, standard deviations, confidence intervals, or p-values, even though §3.3 says experiments were run ten times with different random seeds. Please specify the test used (e.g., paired bootstrap or approximate randomization), report the variance or confidence intervals, and state whether any correction for multiple comparisons was applied.","section":"§3.3 / Table 2"},{"comment":"The GPT-4 comparison is zero-shot only, and the zero-shot rows do not include directionality or novelty labels (as noted in §3.3), so the abstract's claim of outperforming 'state-of-the-art GPT-4 and Llama-3' overstates the evidence. Please restrict the claim to the evaluated settings or provide fine-tuned GPT-4 results, especially for the directionality metrics.","section":"§3.4 / Table 2"}],"minor_comments":[{"comment":"The Discussion section is Section 4, but its subsections are numbered 5.1, 5.2, and 5.3; please renumber them as 4.1, 4.2, and 4.3.","section":"Section numbering"},{"comment":"The sentence 'Interestingly, GPT-3.5, with fine-tuning, outperforms most state-of-the-art models except ours.' is duplicated verbatim; please remove the duplicate.","section":"§3.5"},{"comment":"The column header 'RT+DI. All' contains an extra period; it should be 'RT+DI' and 'All' as separate columns.","section":"Table 2"},{"comment":"The abstract says 'state-of-the-art GPT-4 and Llama-3,' but the experiments use Llama3.2-11B; please specify the exact model versions to avoid ambiguity.","section":"Abstract and Table 2"},{"comment":"The LLM prompt uses the first mentioned named entity for each pair, while BioRED relation instances may involve mentions that are not the first occurrence; please clarify how this affects the comparability of LLM results with the token-boundary-based BERT models.","section":"§2.3"},{"comment":"The BioREx row in Table 2 includes novelty prediction scores, but BioREx is described in the Introduction as focusing solely on relation type classification; please specify in the methods that the novelty scores were obtained by combining BioREx relation predictions with a PubMedBERT novelty classifier.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claims rest on two load-bearing points: the reliability of the single-annotator directionality labels and the statistical significance of the reported improvements. The former is a missing measurement rather than an error, but it affects the released corpus as a benchmark. The latter includes a direct contradiction between §3.5 and the Conclusion. Also, the model is initialized from BioREx, a system developed by overlapping authors that also serves as a baseline; the paper should disclose this relationship explicitly, and the editor may wish to verify that the baseline comparison is not inadvertently advantageous to the proposed method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is the annotation resource: 10,864 directionality labels on BioRED. That is a genuinely new, useful thing. The model is a capable combination of known pieces—soft prompts, three-chunk context, multi-task heads—and it beats the listed BioRED baselines on most metrics. The paper is clearly written, the ablation is honest (infix-only struggles), and the error analysis admits the hard cases explicitly.\n\nThe stress-test concern lands. Only 50 abstracts were double-annotated; the other 950 were split between two annotators, one pass each, with no adjudication or post-hoc IAA. The 89.96% raw pairwise agreement on the pilot is not enough to certify a benchmark, and it is raw agreement, not kappa. Every RT+DI / All score in Tables 2 and 5 is measured against those single-annotator labels. The paper should either adjudicate the 950 or report a chance-corrected reliability measure on a sample and give the reader a sense of the ceiling.\n\nTwo smaller issues. First, the BC5CDR improvement over BioREx is 71.4 vs 70.6 and the paper itself says it is not significant, yet the conclusion claims it \"significantly surpasses\" BioREx on BC5CDR. That is not what the numbers say. Second, the abstract's \"outperforms GPT-4 and Llama-3\" is true only against zero-shot GPT-4 and fine-tuned Llama-3, with no directionality for GPT-4; worth toning down. Also Table 2 gives no variance or significance method for the new directionality metrics, though the paper says ten seeds were run.\n\nCode and data are promised, not yet verified. If the released data is clean, this becomes a useful directed-network testbed for the subfield.\n\nWho should read it: anyone working on document-level biomedical RE, and anyone building direction-aware networks from literature. I would send to peer review, with the expectation that the authors add a reliability analysis and fix the overclaims. The core idea and resource deserve a serious look.","headline":"Genuinely useful directionality annotation for BioRED, but unmeasured single-annotator reliability and an overclaimed BC5CDR significance keep it from being a solid benchmark as-is.","tokens_in":12203,"tokens_out":3374,"would_cite":true,"duration_ms":28978,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper enriches the BioRED corpus with 10,864 subject/object directionality annotations and demonstrates a soft-prompt multi-task model that jointly predicts relation type, novelty, and directionality, outperforming fine-tuned GPT-3.5…","keywords":["relation extraction","directionality","BioRED","multi-task learning","soft prompt","document-level relation extraction","biomedical text mining","entity roles"],"falsifier":"Re-annotate a random sample of the 950 single-annotated abstracts with a second curator and compute pairwise agreement; if it falls well below the reported 89.96%, the directionality-dependent F1 scores in Tables 2 and 3 partly measure label noise. A second check is to have domain experts review the model's predicted directionality on a held-out set and see whether the Undirected label is being overused for hard gene-gene and chemical-gene pairs.","tokens_in":11106,"feed_emoji":"🧬","tokens_out":6422,"duration_ms":52881,"temperature":0.7,"pith_summary":"The paper sets out to add the one thing missing from the BioRED relation corpus: the direction of each relationship, that is, which entity is the subject and which is the object. Across 1,000 abstracts the authors manually assigned these entity-role labels, producing 10,864 directionality annotations. They then train a BERT-based multi-task model that jointly predicts relation type, novelty, and directionality, using soft prompts and a prefix/suffix/infix chunking scheme to cope with documents longer than 512 tokens. On the BioRED-BC8 test set the model reaches 75.34 entity-pair F1, 56.06 relation-type F1, 48.62 with directionality, and 37.02 for all labels together; on BC5CDR it reaches 71.4 F1, ahead of fine-tuned GPT-3.5 and Llama 3.2. The intended payoff is a corpus and model that support constructing directed biological networks rather than undirected association graphs.","feed_headline":"10,864 direction labels turn BioRED into a directed-network resource","feed_subtitle":"A soft-prompt multi-task model jointly predicts relation, novelty, and direction, beating fine-tuned GPT-3.5 and Llama.","key_machinery":"The load-bearing machinery is the directionality annotation itself plus a multi-task transformer architecture. Directionality is recorded per relation pair as one of Rightward (first entity subject), Leftward (second entity subject), Undirected, or None, and this label is predicted alongside relation type and novelty. The model represents each document as a hard prompt with special tokens [REL], [DIR], and [NOV], a learned soft prompt, and three overlapping context chunks (prefix, suffix, infix) to stay within BERT's 512-token limit; the hidden states of the task tokens are pooled with a hidden-token pooler initialized from BERT's pooler, max-pooled across chunks, and scored with separate classifiers. A weighted binary cross-entropy loss combines the three tasks.","core_discovery":"The central claim is that directionality of biomedical relations can be annotated reliably on top of BioRED and predicted jointly with relation type and novelty without sacrificing performance on the original tasks. The authors define directionality as a four-way label per entity pair: Rightward, Leftward, Undirected, or None, and they provide 10,864 such labels. Their model, built on a biomedical language model with task-specific hard prompts and learned soft prompts, treats the three prediction targets as jointly trained multi-label classification problems, with context chunking supplying local and document-level evidence. In their experiments it outperforms fine-tuned GPT-3.5 and Llama 3.2 on both benchmarks, and it improves over the previous BioREx and PubMedBERT systems on BioRED-BC8 while matching or exceeding them on BC5CDR. The paper's message is that directed relation extraction is both annotatable and learnable, so downstream network construction no longer has to flatten causal or activating relationships into symmetric associations.","pith_inferences":["Inference: reliable directionality labels would let BioRED support causal or pathway-oriented network construction, not just undirected association graphs; the paper motivates this but does not build such a network itself.","Inference: because only BioRED carries these labels, a natural extension is to use them as a secondary training signal for other document-level relation extraction datasets, where subject/object roles are often implied but not annotated.","Inference: the gap between relation-type F1 and relation-type-plus-directionality F1 suggests directionality is the hardest part of the task, so future work could target explicit causal-verb cues or curated signaling-pathway knowledge, neither of which the current model uses.","Inference: the quickest empirical check of the benchmark's trustworthiness is re-annotating a random sample of the 950 single-annotator abstracts, since the reported 89.96% agreement was measured on only 50 practice abstracts."],"forward_implications":["The enriched BioRED corpus with 10,864 directionality annotations can serve as a standard benchmark for directed biomedical relation extraction.","If the reported scores hold, researchers can build directed networks of gene, chemical, disease, and variant relationships directly from abstracts without a separate subject/object assignment pipeline.","The combination of multi-task learning, soft prompts, and prefix/suffix/infix chunking offers a reusable recipe for other long-document language model tasks that carry multiple label sets.","Soft prompting at length 8 gives consistent gains over no soft prompt, making it a low-cost enhancement for biomedical relation extraction.","Despite parameter-efficient fine-tuning, general-purpose LLMs remain below the proposed smaller task-specific model, indicating that a domain-tuned model remains competitive for this task."],"supporting_citations":[{"why":"Supplies the original BioRED corpus with relation and novelty annotations that the directionality labels extend.","marker":"Luo, et al. (2022)"},{"why":"Provides the 1,000-abstract BioRED track corpus that serves as the annotation substrate.","marker":"Islamaj, et al. (2023)"},{"why":"Source of the zero-shot GPT-3.5 and GPT-4 comparison numbers and of the BioRED-BC8 evaluation framing.","marker":"Islamaj, et al. (2024)"},{"why":"Defines PubMedBERT, the domain-specific pretrained model used as the base and as a prior baseline.","marker":"Gu, et al. (2021)"},{"why":"Defines BioREx, the strongest prior BERT-based system and the source of the pretrained checkpoint used here.","marker":"Lai, et al. (2023)"},{"why":"Establishes soft prompts as learnable parameter-efficient conditioning, the technique the model adopts.","marker":"Lester, et al. (2021)"},{"why":"Introduces the segmentation strategy for long text that the prefix/suffix/infix chunking adapts to documents over 512 tokens.","marker":"Huang, et al. (2022)"},{"why":"Defines the BC5CDR benchmark and its official document-level evaluation protocol.","marker":"Wei, et al. (2016)"}],"fun_headline_variants":["10,864 direction labels make BioRED a directed network","Soft-prompt model beats GPT-4 on directed relation extraction","BioRED now directed: 10,864 new labels and a multi-task model","Joint direction, relation, and novelty prediction outperforms LLMs","Directionality added to BioRED: 10,864 annotations, one model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The directionality labels are trustworthy enough to benchmark against, which relies on the high agreement measured on 50 practice abstracts carrying over to the 950 abstracts that each received only a single annotator's judgment.","fun_headline_variants_meta":{"raw":{"variants":["10,864 direction labels make BioRED a directed network","Soft-prompt model beats GPT-4 on directed relation extraction","BioRED now directed: 10,864 new labels and a multi-task model","Joint direction, relation, and novelty prediction outperforms LLMs","Directionality added to BioRED: 10,864 annotations, one model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1433,"prompt_tokens":939,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":402}},"tokens_in":555,"tokens_out":494,"duration_ms":4996,"temperature":1.0,"reasoning_tokens":402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:23:36.821863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 950 single-annotated abstracts with a second curator and compute pairwise agreement; if it falls well below the reported 89.96%, the directionality-dependent F1 scores in Tables 2 and 3 partly measure label noise. A second check is to have domain experts review the model's predicted directionality on a held-out set and see whether the Undirected label is being overused for hard gene-gene and chemical-gene pairs.","supporting_citations":[{"cited_title":"The biomedical relationship corpus of the BioRED track at the BioCreative VIII challenge and workshop","cited_arxiv_id":null,"evidence_quote":"Source of the zero-shot GPT-3.5 and GPT-4 comparison numbers and of the BioRED-BC8 evaluation framing."}],"review_version":1}