Pith. sign in

REVIEW 3 major objections 5 minor 29 references

BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BioHopR, a new benchmark for multi-hop, multi-answer biomedical question answering, shows that current large language models struggle to reason across even two knowledge-graph steps.

desk verdict BioHopR is a useful new public dataset for multi-hop multi-answer biomedical QA, but the reported model rankings are unreliable until the evaluation pipeline is fixed. read the letter →

arxiv 2505.22240 v2 pith:OPSE2ZOT submitted 2025-05-28 cs.CL

Yunsoo Kim , Yusuf Abdulle , Honghan Wu This is my paper
classification cs.CL
keywords BioHopRmulti-hopreasoningbiomedicalquestionansweringknowledgegraphlargelanguagemodelsmulti-answerbenchmarkPrimeKGprecisionevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents BioHopR, a question-answering benchmark built from a large biomedical knowledge graph in which every question can have many correct answers and answering may require one or two relational hops. The authors construct thousands of 1-hop and 2-hop queries spanning drugs, diseases, proteins, and phenotypes, then test ten large language models on them. They find that even the best model answers only 37.93% of 1-hop questions correctly, and all models fall to about 14.57% on 2-hop questions, with open-source biomedical models often scoring near zero. The benchmark is meant to expose how current models fail at step-by-step reasoning over interconnected biomedical knowledge.

What carries the argument

The load-bearing object is the graph-walking question: each query starts at a query node, optionally crosses a bridge node, and must end at the full set of target nodes. The construction pipeline samples these paths from a biomedical knowledge graph, keeps only one-to-many-to-many patterns, and turns each path into a natural-language template question. Scoring then compares model outputs against the graph-derived answer list using cosine similarity to biomedical sentence embeddings with a fixed 0.9 threshold, so a predicted term counts as correct only if it is very close to some answer embedding.

What would settle it

Take a random sample of one-hop and two-hop model outputs and ask two clinicians to judge each as correct, incorrect, or plausible-but-unlisted; if a substantial share of answers scored wrong by cosine similarity are judged clinically valid, the benchmark's precision numbers misstate reasoning ability.

Watch

Extended reading notes

Core claim

BioHopR is, the authors argue, the first publicly available benchmark that evaluates multi-hop, multi-answer reasoning inside a structured biomedical knowledge graph. The dataset is deliberately restricted to one-to-many-to-many relations, so a single query such as naming a disease treated by a drug that has a given side effect can have dozens of valid answers. On this benchmark no model is close to reliable: the strongest system reaches 37.93% precision on one-hop questions and 14.57% on two-hop questions, while open-source biomedical models trained for medical exam reasoning score below 1%. The paper reads the consistent drop from one-hop to two-hop performance as evidence that identifying the intermediate bridge entity is the core unsolved step.

Load-bearing premise

The whole evaluation rests on treating the graph's answer lists as complete truth and on a fixed 0.9 embedding-similarity cutoff as the definition of a correct answer, so any valid answer missing from the lists, or expressed differently than the embeddings capture, is scored as a model error.

Editorial extensions

If this is right

  • BioHopR gives the field a public, reproducible way to compare biomedical language models on graph-structured multi-hop reasoning, so future models can be measured against the reported leaderboard.
  • Because even the best model is right on fewer than 15% of two-hop questions, current large language models should not be trusted for multi-step biomedical inferences without external verification.
  • Open-source models trained on medical exam questions do not transfer to graph-traversal reasoning, suggesting that exam-style training alone is not enough for relational multi-hop competence.
  • The near-identical two-hop precision across the proprietary models points to a shared ceiling or failure mode in bridge-entity inference rather than model-specific weaknesses.
  • Demanding exhaustive answer lists lowers precision far below single-answer prompting, so multi-answer recall remains an unsolved evaluation target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The fixed 0.9 similarity threshold and graph-derived gold lists probably understate true model accuracy, since the paper itself shows models giving clinically sound answers such as hepatotoxicity that are marked wrong because they are absent from the gold list.
  • A natural next test is to give models access to the graph or a retrieval tool; if scores jump when bridge entities can be looked up, the bottleneck is memory and identification rather than logical traversal.
  • The benchmark could be extended to report recall as well as precision, because multi-answer questions with dozens of gold targets make exhaustive generation a distinct capability from choosing one correct answer.
  • Re-scoring a sample with clinician judgments would reveal how much of the reported gap is missing knowledge versus incomplete gold standards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BioHopR, a question-answering benchmark built from the PrimeKG knowledge graph, designed to evaluate multi-hop and multi-answer biomedical reasoning. The dataset contains 2,494 one-hop questions and 7,633 two-hop questions over drug, disease, protein, and phenotype entities, with questions constructed from one-to-many-to-many graph paths. The authors evaluate nine large language models in a zero-shot setting, scoring predictions by cosine similarity between BioLORD embeddings of the prediction and the gold answer set, using a threshold of tau=0.9. They report that O3-mini achieves the highest one-hop precision (37.93%) and that all proprietary models tie at 14.57% on two-hop precision, while open-source biomedical models perform substantially worse. The central claims are that BioHopR is a useful public benchmark and that the reported model rankings reflect current reasoning capabilities.

Significance. If the evaluation is made reliable, BioHopR would be a valuable resource: it is publicly released, built from a comprehensive external knowledge graph, and it explicitly targets a genuinely underexplored setting, namely multi-answer and multi-hop biomedical QA. The dataset construction pipeline is described in enough detail to be reproducible, and the statistics on relation types and answer-set sizes are informative. The paper also includes an ablation of prompting strategies, which is a useful practical contribution. However, the empirical ranking claims are the main advertised result, and they currently rest on an evaluation design whose correctness signal is not validated: the gold answer set is treated as complete despite the authors' own qualitative evidence that clinically valid answers are marked wrong, the similarity threshold is tuned on the benchmark itself, and the identical 14.57% two-hop precision across three proprietary models is unexplained. These issues must be resolved before the model comparison can be taken as evidence about reasoning capability.

major comments (3)
  1. [§4.3, Eq. (9); Appendix Figure 4] The cosine similarity threshold tau=0.9 is selected by grid search on the benchmark's own evaluation data, using 'Both Correct' accuracy as the optimization target, while the reported metric is precision as defined in Eq. (10). This is a post-hoc threshold choice on the test set, and the optimized objective is not the reported metric. The authors should either pre-register the threshold, tune it on a held-out validation split constructed from PrimeKG, or report results across a range of thresholds to show that the rankings are not an artifact of the 0.9 cutoff.
  2. [§5.4 and Appendix A; §4.3] The gold answer set, derived solely from PrimeKG, is treated as complete for scoring, but the paper's own qualitative analysis shows models producing clinically valid answers that are not in the gold set and are therefore scored as false positives under Eq. (9). Examples include 'hepatotoxicity' for Troglitazone and 'clozapine' for schizophrenia. Because precision is the only ranking metric, gold-set incompleteness can systematically depress scores and, more importantly, reorder models if proprietary models produce more out-of-gold but valid answers. The authors need a human-validated sample of predictions or an additional expert-judged metric to establish that the reported rankings are robust to gold-set incompleteness.
  3. [§5.1, Table 5] All three proprietary models, GPT4O, GPT4O-mini, and O3-mini, are reported with exactly 14.57% precision on two-hop tasks, despite having materially different one-hop precision (32.88%, 28.11%, and 37.93%). Identical two-decimal scores across independently sampled models with different reasoning behavior is highly implausible and suggests either an evaluation pipeline artifact or a metric that is saturating at a degenerate value. The authors should report per-model counts of predicted and matched answers, run repeated evaluations with temperature variation, and verify the pipeline, especially whether the same bridge or target entities are being matched for all three models.
minor comments (5)
  1. [§3.3] The template strings, such as 'Name a Type({Query}) that is Label({Relationship}) by {Query}', are difficult to parse; please provide a concrete instantiation for each of the question templates used in the final dataset.
  2. [§5.5.2, Table 6] The table contains formatting and notation inconsistencies, such as '20.00 /44.00' without a space and the use of 'GPT4O' and 'GPT-4' interchangeably; the entry showing GPT4O-mini at 44.00% for Drug:Protein:Disease one-hop multi-answer prompting appears to be an outlier relative to the overall 6.11% and needs verification.
  3. [Appendix A] The sentence 'none of the models generated questions for Alzheimer's Disease' is unclear; presumably the authors mean the models did not produce correct answers or did not address the disease in their responses, and this should be reworded.
  4. [§4.3, Eq. (10)] There is a stray period after the fraction in Eq. (10), and the sentence describing the threshold grid search ('which led an optimal setting with 0.9') should be rewritten for clarity.
  5. [References] The citation 'Wolf, 2019' should be updated to the full HuggingFace Transformers citation with all authors, and the reference for the dataset release should include the HuggingFace URL and version information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BioHopR's construction and evaluation rely on external resources (PrimeKG and BioLORD), and the only self-citation is contextual, not load-bearing.

full rationale

BioHopR is built from an external knowledge graph (PrimeKG) and scored with an external embedding model (BioLORD-2023-C), so the benchmark's construction and the reported model rankings do not reduce to the paper's own outputs. The only self-citation is MedExQA (Kim et al., 2024), which appears in Related Work and Table 1 as a comparison; no load-bearing claim depends on it, and it is not invoked as a uniqueness theorem or to forbid alternatives. The cosine threshold tau=0.9 is tuned by grid search on the benchmark itself, which is a validity concern that can bias the reported metric, but it does not make the precision numbers equivalent to the input by construction: model predictions are not fitted to the gold answers, and the threshold is a scoring parameter rather than a derived prediction. Likewise, the incompleteness of the PrimeKG-derived gold answer set, illustrated by the paper's own qualitative examples where clinically correct answers such as hepatotoxicity are marked wrong, is a soundness and completeness issue, not a circular derivation. Overall, the central empirical claim is an externally grounded evaluation result with no circular step identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper is a benchmark construction and evaluation; it introduces no new theoretical entities. The main free parameter is the scoring threshold. The axioms are the external data sources (PrimeKG, BioLORD) and setup assumptions that make the benchmark meaningful.

free parameters (1)
  • Cosine similarity threshold tau = 0.9
    Chosen via grid search over 0.5 to 0.9 on the BioHopR evaluation set itself to prioritize precision (Section 4.3, Appendix Figure 4). This post-hoc selection affects all reported precision numbers.
assumptions (4)
  • domain assumption PrimeKG is a comprehensive and up-to-date biomedical knowledge graph, so BioHopR built from it reflects real-world biomedical knowledge.
    Section 3 selects PrimeKG over Hetionet for broader coverage and up-to-date knowledge (citing Chandak et al., 2023); the benchmark's validity as a biomedical testbed depends on PrimeKG quality.
  • domain assumption Template-generated questions preserve one-to-many-to-many biomedical relations and are a faithful proxy for real human queries.
    Sections 3.1-3.3 define the templates and answer extraction; if templates do not reflect realistic query formulations, benchmark results may not generalize to actual usage.
  • domain assumption BioLORD-2023-C embeddings encode biomedical entity names such that a cosine threshold of 0.9 reliably separates correct from incorrect answers.
    Section 4.3 uses this embedding model and threshold for all precision calculations; every reported precision number depends on this mapping.
  • domain assumption Zero-shot evaluation with temperature 0 produces deterministic and representative model outputs with a single sample.
    Section 4.2 sets temperature to 0 except for O3-mini; no multiple sampling or variance estimates are reported, so precision values are assumed stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain." pith.science (2026). https://pith.science/paper/OPSE2ZOT

@misc{pith2026250522240,
  author       = {Pith},
  title        = {Pith review of: BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPSE2ZOT}},
  note         = {Machine review of arXiv:2505.22240}
}
read the original abstract

Biomedical reasoning often requires traversing interconnected relationships across entities such as drugs, diseases, and proteins. Despite the increasing prominence of large language models (LLMs), existing benchmarks lack the ability to evaluate multi-hop reasoning in the biomedical domain, particularly for queries involving one-to-many and many-to-many relationships. This gap leaves the critical challenges of biomedical multi-hop reasoning underexplored. To address this, we introduce BioHopR, a novel benchmark designed to evaluate multi-hop, multi-answer reasoning in structured biomedical knowledge graphs. Built from the comprehensive PrimeKG, BioHopR includes 1-hop and 2-hop reasoning tasks that reflect real-world biomedical complexities. Evaluations of state-of-the-art models reveal that O3-mini, a proprietary reasoning-focused model, achieves 37.93% precision on 1-hop tasks and 14.57% on 2-hop tasks, outperforming proprietary models such as GPT4O and open-source biomedical models including HuatuoGPT-o1-70B and Llama-3.3-70B. However, all models exhibit significant declines in multi-hop performance, underscoring the challenges of resolving implicit reasoning steps in the biomedical domain. By addressing the lack of benchmarks for multi-hop reasoning in biomedical domain, BioHopR sets a new standard for evaluating reasoning capabilities and highlights critical gaps between proprietary and open-source models while paving the way for future advancements in biomedical LLMs.

Figures

Figures reproduced from arXiv: 2505.22240 by the authors.

Figure 1
Figure 1. Illustration of the relationships observed in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Qualitative analysis of model responses to [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Common target and bridge entities for each node type in BioHopR. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Grid search results showing the relationship between cosine similarity threshold and accuracy for "Both [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages

  1. [1]

    Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67

    work page 2023

  2. [2]

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925

    arXiv 2024

  3. [3]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  4. [4]

    Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31

    work page 2018

  5. [5]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    arXiv 2020

  6. [6]

    Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, and Sergio E Baranzini. 2017. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. Elife, 6:e26726

    work page 2017

  7. [7]

    Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403

    arXiv 2022

  8. [8]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    arXiv 2024

Show all 29 references
  1. [9]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  2. [10]

    Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024. Medexqa: Medical question answering benchmark with multiple explanations. arXiv preprint arXiv:2406.06331

    2024 arXiv

  3. [11]

    Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

    2016 arXiv

  4. [12]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234--1240

    2020

  5. [13]

    Xiaomin Liang, Daifeng Li, Min Song, Andrew Madden, Ying Ding, and Yi Bu. 2019. Predicting biomedical relationships using the knowledge and graph embedding cascade model. PLoS One, 14(6):e0218264

    2019

  6. [14]

    Nicholas Matsumoto, Hyunjun Choi, Jay Moran, Miguel E Hernandez, Mythreye Venkatesan, Xi Li, Jui-Hsuan Chang, Paul Wang, and Jason H Moore. 2025. Escargot: an ai agent leveraging large language models, dynamic graph of thoughts, and biomedical knowledge graphs for enhanced rea...

    2025

  7. [15]

    Kanishka Misra, Cicero Nogueira dos Santos, and Siamak Shakeri. 2023. Triggering multi-hop reasoning for question answering in language models using soft prompts and random walks. arXiv preprint arXiv:2306.04009

    2023 arXiv

  8. [16]

    OpenAI. 2025. Openai o3-mini. https://openai.com/index/openai-o3-mini/. Accessed: 2025-02-01

    2025

  9. [17]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248--260. PMLR

    2022

  10. [18]

    Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511

    2024

  11. [19]

    Dattaraj J Rao, Shraddha S Mane, and Mukta A Paliwal. 2022. Biomedical multi-hop question answering using knowledge graph embeddings and language models. arXiv preprint arXiv:2211.05351

    2022 arXiv

  12. [20]

    Fran c ois Remy, Kris Demuynck, and Thomas Demeester. 2023. Biolord-2023: Semantic textual representations fusing llm and clinical knowledge graph insights. arXiv preprint arXiv:2311.16075

    2023 arXiv

  13. [21]

    Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. 2024. Morehopqa: More than multi-hop reasoning. arXiv preprint arXiv:2406.13397

    2024 arXiv

  14. [22]

    Jay S Skyler, George L Bakris, Ezio Bonifacio, Tamara Darsow, Robert H Eckel, Leif Groop, Per-Henrik Groop, Yehuda Handelsman, Richard A Insel, Chantal Mathieu, et al. 2017. Differentiation of diabetes by pathophysiology, natural history, and prognosis. Diabetes, 66(2):241--255

    2017

  15. [23]

    Xiaorui Su, Yibo Wang, Shanghua Gao, Xiaolong Liu, Valentina Giunchiglia, Djork-Arn \'e Clevert, and Marinka Zitnik. 2024. Knowledge graph based agent for complex, knowledge-intensive qa in medicine. arXiv preprint arXiv:2410.04660

    2024 arXiv

  16. [24]

    Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? arXiv preprint arXiv:2109.07154

    2021 arXiv

  17. [25]

    T Wolf. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  18. [26]

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402.16837

    2024 arXiv

  19. [27]

    Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. 2024. Ultramedical: Building specialized generalists in biomedicine. arXiv preprint arXiv:2406.03949

    2024 arXiv

  20. [28]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.