Pith. sign in

REVIEW 5 major objections 4 minor 22 references

CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read A fine-tuned 8B biomedical QA model reaches 0.8 concept accuracy but 0.0–0.2 exact match, exposing a gap between semantic understanding and strict answer formatting.

desk verdict A routine system paper with new MedHopQA numbers, but the central claim about 0.8 concept-level accuracy is undercut by the official test scores it reports. read the letter →

arxiv 2509.00806 v1 pith:N567ZOTG submitted 2025-08-31 cs.CL cs.AIcs.LG

classification cs.CLcs.AIcs.LG
keywords biomedicalquestionansweringmulti-hopQAsupervisedfine-tuningLLaMA38Blow-rankadaptersexactmatchconcept-levelaccuracyanswerextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a supervised fine-tuning system for a multi-hop biomedical question-answering track centered on diseases, genes, and chemicals. By training a general-purpose 8-billion-parameter LLM on a 10,000-question mixture assembled from public biomedical QA sources, the authors find that their models reach about 0.8 concept-level accuracy on the development set—evidence the model has internalized biomedical knowledge—while exact-match scores stay around 0.5 on validation and drop to 0.0–0.2 on the official test set. The paper's central move is to identify the gap as a formatting problem, not a knowledge problem, and to attack it with a two-stage inference pipeline that extracts concise phrases from verbose first-stage outputs. The pipeline helps in some cases but does not reliably close the gap, and the authors argue that tight output control under strict evaluation metrics remains an open limitation for biomedical LLM question answering.

What carries the argument

The load-bearing mechanism is a two-stage inference pipeline on top of a supervised fine-tuned model. In the first stage, the model generates a free-form answer to the question; in the second, a follow-up prompt instructs it to extract the exact answer phrase or entity from that response, and if three extraction attempts fail, the system falls back to the first-stage longer output. The model itself is a LLaMA 3 8B adapted with low-rank adapters (LoRA), a parameter-efficient fine-tuning method that tunes small adapter weights instead of all model weights. The pipeline is what attempts to translate the model's semantic competence into the strict short-answer format the evaluation demands.

What would settle it

Re-score the official test submissions after normalizing answer variants (e.g., '2' to 'Chromosome 2', 'Chr.2' to the canonical chromosome label, plus synonym resolution). If normalized EM rises toward concept-level accuracy, the claimed gap is mostly formatting; if normalized EM stays near zero on rare-disease questions, the gap is a content or distribution failure.

Watch

Extended reading notes

Core claim

The paper's central claim is that a general-purpose 8-billion-parameter LLM, fine-tuned with low-rank adapters on a mixture of external biomedical QA pairs, learns enough biomedical knowledge to score up to 0.8 on concept-level evaluation of multi-hop questions about diseases, genes, and chemicals—yet fails exact-match evaluation because its outputs are verbose, paraphrased, or slightly misformatted. The authors demonstrate this with three fine-tuning setups, reporting validation EM near 0.5 and test EM of 0.2, 0.0, and 0.0 for combined, short-only, and long-only training. Zero-shot general-purpose models score near zero on both metrics, indicating that fine-tuning is what recovers semantic

Load-bearing premise

The load-bearing premise is that the external training mixture and the small development set represent the same distribution as the hidden test set; if that distribution shift is large, the reported gap between concept accuracy and exact match could be a benchmark artifact rather than a general LLM limitation.

Editorial extensions

If this is right

  • Fine-tuned models can reach roughly 0.8 concept-level accuracy on multi-hop biomedical questions, so specialized domain knowledge is achievable from a small curated training mixture.
  • Exact-match evaluation penalizes surface form as much as content: correct-concept variants like '2', 'Chr.2', and 'Chromosome 2p13' all count as misses, so the metric shapes the measured gap.
  • Training on combined short and long answers was the only setup with a nonzero official test EM (0.2), suggesting answer-format diversity during training supports generalization.
  • Prompt refinement and two-stage extraction can lift test EM from near zero to about 0.49 without new training data, so output-control techniques are a promising, low-cost lever.
  • Zero-shot general-purpose models fail on both concept and exact-match metrics, so instruction-following alone is insufficient for specialized biomedical multi-hop reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Re-scoring the official predictions with a canonical-name normalizer (gene symbols, chromosome labels, synonym expansion) would quantify how much of the EM gap is pure formatting; the authors' own error examples suggest the share is large.
  • The development-set concept accuracy (~0.8) and the official leaderboard concept scores (0.11–0.31) diverge sharply; a per-topic breakdown by rare-disease category could reveal whether the model fails on rare entities or on multi-hop composition, and whether the development set is a trustworthy predictor.
  • The two-stage pipeline is a heuristic form of constrained decoding; comparing it against grammar-guided or constrained decoding would show how much of the remaining format gap is addressable by generation-time control, which the authors leave for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper describes a supervised fine-tuning system (LLaMA 3 8B with LoRA) for the BioCreative IX MedHopQA multi-hop biomedical QA track. Three models are trained on a self-curated 10,000-question dataset assembled from BioASQ, MedQuAD, TREC, and other public biomedical QA sources, using combined short/long, short-only, and long-only answer formats. A two-stage inference pipeline is introduced to extract concise answers from verbose model outputs. Reported results: validation EM around 0.5 and concept-level accuracy around 0.8; official test EM scores of 0.2, 0.0, and 0.0 for the three setups, with leaderboard concept scores of 0.3120, 0.1140, and 0.2250. The paper concludes that the model has strong semantic understanding but fails to produce exact-match answers.

Significance. If the validation concept-level accuracy of ~0.8 held on the official test, the paper would provide useful evidence on the gap between semantic understanding and exact-match evaluation in biomedical QA. The paper's strengths are its transparent reporting of official leaderboard scores and its clear description of a practical extraction pipeline. However, the central empirical claim is weakened by contradictory development-set descriptions, the absence of per-setup validation tables, a large unexplained drop from validation to test, and no quantitative evaluation of the proposed post-processing pipeline. As a system description, the contribution is modest and primarily of interest to the BioCreative community.

major comments (5)
  1. [§3.1 and §4.2] The development set is described twice inconsistently: Section 3.1 first says it includes '45 question instances' and later says the 'official 1,000-example development set' was used as validation. If the ~0.8 concept accuracy and ~0.5 EM are computed on 45 questions, the scores correspond to roughly 36/45 and 23/45, with wide confidence intervals; if computed on the 1,000-example set, the text must say so explicitly. Please state the exact validation denominator and report per-setup validation scores with confidence intervals or at least exact numerators.
  2. [§4.2, abstract, §5] The paper's central claim of 'strong domain understanding' relies on the validation concept accuracy of ~0.8, but the official leaderboard concept scores are 0.3120, 0.1140, and 0.2250 for the three setups. The paper attributes the drop to verbosity and formatting issues, yet this conflates formatting errors with a clear train/validation/test distribution shift. As written, the claim that the model 'understands biomedical concepts well' is supported only on the validation distribution and is contradicted on the official benchmark. The authors should either reframe the conclusion to acknowledge the distribution shift or provide an explicit error analysis (e.g., concept scores after answer extraction, human evaluation of a test sample) that separates formatting failures from reasoning failures.
  3. [§3.3 and §4.2] The two-stage inference pipeline is a central contribution, but the paper provides no quantitative ablation. It reports only examples of verbose outputs and a qualitative statement that the pipeline 'improved output quality'. The 'unofficial test evaluation' that achieved an EM of 0.49 is not described: which test data, which gold labels, how many examples, and what post-processing was applied? Without before/after EM or concept scores for the same model and same evaluation set, the claimed improvement is not verifiable.
  4. [§3.1] The training-data construction is underspecified. The paper lists source datasets with a total of roughly 70,000 QA pairs, states that 10,000 pairs were used, and mentions 'cleaned and filtered' without describing the filtering criteria, per-source counts, duplicate removal, or checks for overlap with the MedHopQA validation/test sets. Given that the paper's premise is that this mixture transfers to MedHopQA, the lack of data-composition details is a reproducibility gap that should be addressed.
  5. [§4.2] The zero-shot baseline results are reported only qualitatively ('near-zero EM', 'approximately 80% accuracy' for medical-specialty questions). No numerical scores, sample sizes, or error counts are provided. This makes the claimed benefit of supervised fine-tuning relative to zero-shot inference not quantitatively established. This is secondary to the main claims but should be fixed with a small table.
minor comments (4)
  1. [§3.1 / §1] The test-set size is inconsistent: Section 1 and Section 3.1 say the test set contains 10,000 questions, while Section 4.2 says the testing phase was evaluated on 1,000 examples only. Please clarify the relationship between the 10,000-question set and the 1,000-example evaluation.
  2. [§3.3] Figure 1 is referenced but does not appear in the provided text. Please ensure the figure is included and referenced correctly.
  3. [Throughout] There are numerous typos and formatting issues: 'traning', 'envel⌢pe-⌢pen', 'optimiser', 'emphasise', 'generalizing', and an unresolved author token 'L. N. 0001' in reference [13]. A careful proofread is needed.
  4. [References] Reference [4] contains a duplicated phrase 'Overview of overview of the medhopqa track'. Also, several arXiv identifiers or URLs are not consistently formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical shared-task system report with no self-citation, no fitted-parameter-as-prediction, and no definitional reduction.

full rationale

This paper is an empirical system description for a shared task, not a theoretical derivation. There are no equations or first-principles claims whose outputs reduce to their inputs. The central quantitative claims—validation concept accuracy ~0.8, EM ~0.5, official test EM 0.0–0.2, and leaderboard concept scores 0.3120/0.1140/0.2250—are reported measurements, not predictions derived from fitted parameters. The two-stage inference pipeline is a post-processing method; its unofficial 0.49 EM on the test set is explicitly labeled unofficial and was obtained by prompt refinement on the test data, which is a methodological disclosure rather than a circular derivation. The paper contains no load-bearing self-citations: references to MedHopQA and external datasets are organizational, and the author list does not overlap with the cited task-definition papers. The inconsistency in the development-set description (45 vs. 1,000 examples) and the transfer gap between validation and leaderboard scores raise correctness and generalization concerns, but these are not circularity. No step in the paper's reasoning defines X in terms of Y, fits a parameter and relabels it as a prediction, or imports an ansatz via a self-citation. Hence there is no significant circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. The central results rest on hand-chosen hyperparameters, the transferability assumption of external QA data, and the organizer-defined concept-level metric. No derivation or first-principles claim is made, so the free-parameter count is limited to standard training choices.

free parameters (6)
  • LoRA rank r = 64
    Hand-chosen LoRA rank for fine-tuning; standard choice, not fitted to the evaluation metric.
  • LoRA alpha = 16
    Hand-chosen LoRA scaling parameter.
  • learning rate = 1e-4
    Hand-chosen learning rate for LoRA fine-tuning.
  • epochs = 5
    Hand-chosen number of training epochs.
  • temperature = 0.01
    Decoding temperature chosen for near-greedy generation.
  • top_p = 0.95
    Decoding nucleus sampling parameter.
assumptions (3)
  • domain assumption Supervised fine-tuning on external biomedical QA datasets transfers to MedHopQA
    Section 3.2: the entire approach assumes SFT on BioASQ, MedQuAD, TREC, etc. improves performance on the target task, with no ablation or evidence shown for this transfer.
  • domain assumption The concept-level accuracy metric is a valid measure of semantic equivalence
    Section 4.1: the paper relies on the organizer-defined concept-level evaluation without defining or verifying how it is computed, which weakens the interpretation of the 0.8 versus 0.31 numbers.
  • domain assumption The 10,000 QA-pair curated training set is representative of MedHopQA's distribution
    Section 3.1: the cleaning and filtering of external pairs is described only vaguely, and MedHopQA emphasizes rare diseases, so representativeness is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA." pith.science (2026). https://pith.science/paper/N567ZOTG

@misc{pith2026250900806,
  author       = {Pith},
  title        = {Pith review of: CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N567ZOTG}},
  note         = {Machine review of arXiv:2509.00806}
}
read the original abstract

Large language models (LLMs) are increasingly evident for accurate question answering across various domains. However, rigorous evaluation of their performance on complex question-answering (QA) capabilities is essential before deployment in real-world biomedical and healthcare applications. This paper presents our approach to the MedHopQA track of the BioCreative IX shared task, which focuses on multi-hop biomedical question answering involving diseases, genes, and chemicals. We adopt a supervised fine-tuning strategy leveraging LLaMA 3 8B, enhanced with a curated biomedical question-answer dataset compiled from external sources including BioASQ, MedQuAD, and TREC. Three experimental setups are explored: fine-tuning on combined short and long answers, short answers only, and long answers only. While our models demonstrate strong domain understanding, achieving concept-level accuracy scores of up to 0.8, their Exact Match (EM) scores remain significantly lower, particularly in the test phase. We introduce a two-stage inference pipeline for precise short-answer extraction to mitigate verbosity and improve alignment with evaluation metrics. Despite partial improvements, challenges persist in generating strictly formatted outputs. Our findings highlight the gap between semantic understanding and exact answer evaluation in biomedical LLM applications, motivating further research in output control and post-processing strategies.

Figures

Figures reproduced from arXiv: 2509.00806 by the authors.

Figure 1
Figure 1. Two-stage inference pipeline for extracting precise short answers. Exact Match (EM) A prediction is considered correct under the EM metric if it exactly matches the gold standard answer after applying normalisation techniques. These include lowercasing, removal of punctuation and articles, and basic synonym resolution. This metric emphasises the precision of the model in generating strictly correct answer forms. Con… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages

  1. [1]

    X. Lin, Z. Huang, Z. Zhang, J. Zhou, E. Chen, Explore what llm does not know in complex question answering, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025, pp. 24585–24594

  2. [2]

    Ashish, Attention is all you need, Advances in neural information processing systems 30 (2017) I

    V. Ashish, Attention is all you need, Advances in neural information processing systems 30 (2017) I

  3. [3]

    Singhal, T

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al., Toward expert-level medical question answering with large language models, Nature Medicine (2025) 1–8

  4. [4]

    Islamaj, J

    R. Islamaj, J. Chan, R. Leaman, Z. Lu, Overview of overview of the medhopqa track at biocreative ix: track description, participation and evaluation of systems for multi-hop medical question answering, in: BioCreative IX Challenge and Workshop (BC9): Large Language Models for Clinical and Biomedical NLP at the International Joint Conference on Artificial ...

  5. [5]

    Lu, Medhopqa, 2025

    Z. Lu, Medhopqa, 2025. URL: https://www.ncbi.nlm.nih.gov/research/bionlp/medhopqa#dataset

  6. [6]

    Pearce, T

    K. Pearce, T. Zhan, A. Komanduri, J. Zhan, A comparative study of transformer-based language models on extractive question answering, CoRR abs/2110.03142 (2021). URL: https://arxiv.org/abs/ 2110.03142. arXiv:2110.03142

  7. [7]

    M. V. Koroteev, Bert: a review of applications in natural language processing and understanding, arXiv preprint arXiv:2103.11943 (2021)

  8. [8]

    Roberts, D

    K. Roberts, D. Demner-Fushman, E. M. Voorhees, S. Bedrick, W. R. Hersh, Overview of the trec 2022 clinical trials track., in: TREC, 2022

Show all 22 references
  1. [9]

    P. He, X. Liu, J. Gao, W. Chen, Deberta: Decoding-enhanced bert with disentangled attention, arXiv preprint arXiv:2006.03654 (2020)

  2. [10]

    Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, Albert: A lite bert for self-supervised learning of language representations, arXiv preprint arXiv:1909.11942 (2019)

  3. [11]

    V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019)

  4. [12]

    A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, M. Auli, Eli5: Long form question answering, arXiv preprint arXiv:1907.09190 (2019)

  5. [13]

    Yuntao, N

    K. Yuntao, N. M. Phuong, T. Racharak, T. Le, L. M. N. 0001, An effective method to answer multi-hop questions by single-hop qa system., in: ICAART (2), 2022, pp. 244–253

  6. [14]

    R. S. Roy, A. Anand, Multi-hop question answering, in: Question Answering for the Curated Web: Tasks and Methods in QA over Knowledge Bases and Text Collections, Springer, 2022, pp. 121–128

  7. [15]

    Tsatsaronis, M

    G. Tsatsaronis, M. Schroeder, G. Paliouras, Y. Almirantis, I. Androutsopoulos, E. Gaussier, P. Galli- nari, T. Artieres, M. R. Alvers, M. Zschunke, et al., Bioasq: A challenge on large-scale biomedical semantic indexing and question answering., in: AAAI fall symposium: Informa...

  8. [16]

    H. Yang, H. Chen, H. Guo, Y. Chen, C.-S. Lin, S. Hu, J. Hu, X. Wu, X. Wang, Llm-medqa: Enhancing medical question answering through case studies in large language models, arXiv preprint arXiv:2501.05464 (2024)

  9. [17]

    R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, T.-Y. Liu, Biogpt: Gener- ative pre-trained transformer for biomedical text generation and mining, Briefings in Bioinformatics 23 (2022). URL: https://www.microsoft.com/en-us/research/publication/ biogpt-generative-pre-trained...

  10. [18]

    Ben Abacha, D

    A. Ben Abacha, D. Demner-Fushman, A question-entailment approach to question answering, BMC Bioinform. 20 (2019) 511:1–511:23. URL: https://bmcbioinformatics.biomedcentral.com/articles/10. 1186/s12859-019-3119-4

  11. [19]

    Unger, C

    C. Unger, C. Forascu, V. Lopez, A.-C. N. Ngomo, E. Cabrio, P. Cimiano, S. Walter, Question answering over linked data (qald-4), in: Working notes for CLEF 2014 conference, 2014

  12. [20]

    M. Zhu, A. Ahuja, D.-C. Juan, W. Wei, C. K. Reddy, Question answering with long multiple-span answers, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 3840–38...

  13. [21]

    Krithara, A

    A. Krithara, A. Nentidis, K. Bougiatiotis, G. Paliouras, Bioasq-qa: A manually curated corpus for biomedical question answering, Scientific Data 10 (2023) 170

  14. [22]

    Nentidis, G

    A. Nentidis, G. Katsimpras, A. Krithara, S. Lima López, E. Farré-Maduell, L. Gasco, M. Krallinger, G. Paliouras, Overview of bioasq 2023: The eleventh bioasq challenge on large-scale biomedical semantic indexing and question answering, in: International Conference of the Cross...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.