Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HealthBranches is a benchmark of 4,063 medical Q&A pairs, each generated from an explicit clinical decision pathway, and the paper shows that supplying the pathway improves LLM answers while text alone does not.

desk verdict Useful new medical Q&A benchmark with a real reasoning-path niche, but the 'clinically validated' claim needs quantified human review before it holds. read the letter →

arxiv 2508.07308 v1 pith:76LRHN3W submitted 2025-08-10 cs.CL cs.AIcs.IRcs.LG

Cristian Cosentino , Annamaria Defilippo , Marco Dossena , Christopher Irwin , Sara Joubbi , Pietro Liò This is my paper
classification cs.CLcs.AIcs.IRcs.LG
keywords medicalquestionansweringclinicaldecisiontreesreasoningpathsLLMevaluationretrieval-augmentedgenerationsyntheticdatasethealthcareNLPbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HealthBranches is a benchmark of 4,063 medical question-answer pairs built by converting the decision trees found in medical textbooks into realistic patient scenarios. The paper's central claim is that these explicit decision pathways can anchor both the questions and their answers, so every item carries a full step-by-step reasoning path instead of just a final label. The authors report that LLMs perform markedly better in the quiz setting when given the reasoning path, an improvement they do not see when given only the textbook text. If the dataset holds up, it gives the field a reusable way to test multi-step clinical reasoning, RAG retrieval, and answer explainability in one resource.

What carries the argument

The central object is the decision pathway: a root-to-leaf traversal of a clinical decision tree, where each node is a diagnostic or therapeutic step. The pipeline enumerates these paths from graph representations extracted from textbooks (up to two paths per leaf), refines them with Gemini-flash 2.0 to normalize terminology without changing clinical semantics, and uses the same paths to generate a patient scenario, a correct answer, and four plausible-but-wrong distractors. The reasoning path is then attached to each Q&A pair, so it serves simultaneously as the source of question content, the ground-truth justification, and the retrieval unit for RAG evaluation.

What would settle it

Take a random sample of the unreviewed Q&A pairs, have clinicians trace each answer back to the source textbook's decision tree, and count mismatches. If a substantial fraction of 'correct' answers does not follow from the published path—or if the path itself misstates the textbook—the benchmark's ground truth fails regardless of how well LLMs score on it.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that clinically validated algorithmic frameworks—the decision graphs in textbooks such as those for dyspnea—can be semi-automatically transformed into a large Q&A benchmark. Each of the 4,063 pairs is linked to a refined root-to-leaf reasoning path, appears in both open-ended and multiple-choice form, and has distractors generated to be plausible but invalid on that same path. Validation combined LLM-based refinement (Llama 3.1 405B to flag flawed questions, GPT-4o with web search to audit them) and human review of a subset, with an expert panel finding only 19 questions scoring 3 or below on any aspect and 6 below 9/15 overall. The evaluation result t

Load-bearing premise

The load-bearing premise is that the LLM extraction and path-refinement steps preserve the clinical meaning of the source decision trees; the paper verifies this by human review on only a subset of the 4,063 questions, leaving the fidelity of the unreviewed majority unverified.

Editorial extensions

If this is right

  • Evaluators can score not just whether an LLM picks the right answer, but whether its stated reasoning follows the clinical path.
  • The RAG setting can retrieve the path itself, making it possible to test whether retrieval of structured knowledge improves multi-step answers.
  • The open-ended plus multiple-choice pairing lets the same clinical case be used for generation and selection, supporting apples-to-apples comparisons.
  • Distractor generation from the same path produces harder, more realistic wrong options than random alternatives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the path-extraction pipeline generalizes, the same recipe could produce grounded Q&A benchmarks for other guideline-driven fields beyond the 17 clinical categories covered here.
  • The reported gap between path-augmented and text-only performance hints that models may be using the path as a scaffold; a follow-up could test whether they still follow it when the path is contradicted by the patient vignette.
  • The refinement step filters questions by multi-model failure, so the benchmark may be tuned toward questions that are hard for current LLMs rather than questions that are clinically most consequential; clinician-rated difficulty and error-severity labels would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces HealthBranches, a medical Q&A benchmark of 4,063 items generated semi-automatically from decision trees in medical textbooks. The pipeline parses textual and graphical streams using Gemini-flash 2.0, enumerates root-to-leaf paths, refines the paths, generates open-ended and multiple-choice Q&A pairs, and then filters/refines low-quality items using Llama models and GPT-4o with human review of a subset. The paper reports model evaluations under zero-shot, RAG, and path-provided settings. The central claim is that the dataset provides clinically grounded Q&A pairs with explicit, clinically valid reasoning paths.

Significance. If the validation claim holds, HealthBranches is a useful resource: it combines multiple-choice and open-ended formats with explicit reasoning paths, enabling step-by-step reasoning evaluation and RAG benchmarking. The paper is transparent about the pipeline and releases code and data. Strengths include a detailed, prompt-level description of the generation process, multi-model evaluation, and the use of several model families in refinement. However, the 'clinically validated' claim currently rests on unquantified human review coverage and a refinement filter that cannot detect systematic errors shared by the LLM families used. The paper would be significantly strengthened by reporting complete validation denominators and adding an independent provenance check.

major comments (4)
  1. [§4.2 Expert Evaluations] The abstract and §1 claim 'clinically validated reasoning chains,' but the only expert-validation evidence reported is: 'Only 19 questions received a score of 3 or below in at least one category, and 6 fell below 9/15 overall.' No denominator, number of reviewed items, sampling protocol, or inter-reviewer agreement is given. It is therefore impossible to know whether the 4,063 final items are clinically sound beyond the small reviewed subset. Please report the total number of expert-reviewed questions, how they were sampled (random? only flagged items?), and item-level or aggregate scores for that full review set.
  2. [§3.1 Q&A Refinement] The refinement filter only inspects questions misanswered by both Llama3.3-70B and Llama3.1-405B and by at least half of the benchmark models. This design cannot detect systematic errors shared by those models—for example, a clinically incorrect answer that Gemini generates and that Llama/GPT-4o accept because they were trained on similar text. The paper acknowledges hybridization but does not address shared-bias failure. Please add an independent validation signal, e.g., a per-item provenance mapping from each answer to the specific path nodes that entail it, or a random sample of un-flagged items reviewed by experts.
  3. [§3.1 and Figure 4] There is no automated check that the generated answer is actually entailed by the reasoning path. The prompt asks for an answer 'derived via the reasoning path,' but no mapping or verification is reported. Since the benchmark includes the full ground-truth path and the path-conditioned setting in Figure 4-top likely provides that path to the model, the observed improvement may partly reflect answer leakage (the path contains the answer almost verbatim) rather than reasoning ability. Please add a control condition, e.g., a corrupted or answer-redacted path, or quantify the lexical overlap between path and answer.
  4. [§4.2 Figure 3a] The text says 'Figure 3a shows the accuracy gains after applying the refinement protocol,' but it is not stated whether the reported accuracy is measured on the 1,203 refined questions, the final 4,063-item benchmark, or another subset. If the evaluation is only on the flagged subset, the improvement is not representative of the final dataset. Please specify the evaluation set, sample size, and confidence intervals for this figure, and report the intermediate yields (initial generated, after path sampling, after filtering) so readers can interpret the pipeline's effect.
minor comments (6)
  1. [§4.1] The text says 'As shown in Table 1, some macro-categories...' but the category distribution is in Table 2; the reference should be corrected.
  2. [Figure 4 caption] The y-axis truncation is acknowledged, but the visual exaggeration is strong. Please add a note explaining the truncation more prominently and consider showing the full axis or a zoom inset.
  3. [§4.3] The sentence '...this improvement is not observed in the topline with only textual' appears incomplete; please finish the sentence and explain what the textual-only condition contains.
  4. [References] References [38] and [42] are the same Llama 3 Herd technical report; consolidate to avoid duplicate citation.
  5. [§3.1] The text refers to 'Gemini-flash 2.0' but the cited Gemini reference [37] is the general Gemini paper; please clarify the exact model version and prompt used for each pipeline stage.
  6. [§3.1 Q&A Refinement] The phrase 'i.e. 5 models, excluding Llama3.3 (70 B)' is unclear about which models constitute the benchmark set. List the models explicitly so the reader can verify the 'at least half' criterion.

Circularity Check

1 steps flagged · score 5.0 of 10

Refinement uses the evaluated models' errors to rewrite/remove benchmark items, making reported accuracy gains and benchmark scores partly self-fulfilling; the textbook-based clinical grounding is not itself circular.

  1. fitted input called prediction [Section 3.1 (Q&A Refinement) and Section 4.2 (Q&A validation, Figure 3a)]
    "First, we identified all generated questions that were incorrectly answered by both Llama3.3 (70 B) and Llama3.1 (405 B). From this subset, we retained only the questions that were misanswered by at least half of the models in our benchmark (i.e. 5 models, excluding Llama3.3 (70 B)), resulting in a pool of 1,203 questions for further examination. ... Figure 3a shows the accuracy gains after applying the refinement protocol, with clear improvements in both zero-shot and RAG-based settings particularly for larger models."

    The refinement stage selects questions for rewriting or elimination based on the errors of the very models that are later benchmarked (Llama3.3-70B is used in the first filter and is also an evaluated model). The reported accuracy gains in Figure 3a and the benchmark scores in Figure 4 therefore are not independent measurements: the dataset items were changed or removed precisely where those models failed. The improvement is partly a mechanical consequence of adjusting the ground truth to the models' mistakes, so the evaluation does not provide fully independent evidence of model capability or dataset quality. The textbook-derived clinical content is external, so the circularity is partial rather than total.

full rationale

The paper's central construction chains — parsing textbook decision graphs with Gemini-flash, extracting paths, generating Q&A pairs from those paths, and auditing a model-selected subset with GPT-4o and human reviewers — are not self-referential in the sense of defining the target in terms of the result. The clinical grounding is imported from external medical textbooks, and the final ground truth is not mathematically entailed by the pipeline's inputs. No load-bearing self-citations or imported uniqueness theorems appear. The main circularity is the model-in-the-loop refinement: questions that Llama3.3-70B and a majority of the benchmark models answer incorrectly are preferentially rewritten or eliminated, and then the same models are evaluated on the refined dataset. The accuracy improvements shown in Figure 3a are therefore partly forced by the selection rule, and the Figure 4 benchmark scores inherit that construction dependence. The expert evaluation is reported only as '19 questions' and '6 below 9/15' with no denominator or inter-reviewer agreement, which leaves the coverage of human validation unquantified; that is an evidence gap rather than a circularity. Overall, the dataset may still be useful and largely externally grounded, but the evaluation component contains a partial circularity that prevents treating the measured gains and model rankings as fully independent.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

HealthBranches introduces no new medical entities, forces, or mechanisms. Its axioms are all about trust in the LLM extraction pipeline and the source textbooks.

free parameters (3)
  • Expert review subset size = unspecified
    The number of questions reviewed by physicians and students is not stated, so the reported validation statistics cannot be interpreted.
  • Number of retained paths per leaf = 2
    Up to two paths per leaf is a hand-chosen balance parameter; it directly controls dataset size and diversity.
  • Misanswered-by-5-models filter = 1,203 candidate questions
    The threshold for sending questions to GPT-4o refinement is a hand-chosen rule based on model disagreement, not on clinical ground truth.
assumptions (4)
  • domain assumption Gemini-flash 2.0 extracts decision graphs and clinical text from PDFs without introducing errors.
    The whole pipeline starts with LLM-generated JSON trees from three textbooks (Section 3.1, Appendix A.1); no independent check of extraction fidelity is reported.
  • domain assumption The three medical textbooks are authoritative and their decision trees are clinically valid.
    The authors do not audit the source guidelines; they inherit any textbook errors, as the limitations section acknowledges.
  • domain assumption LLM refinement preserves intended clinical semantics.
    Path refinement is performed by Gemini-flash 2.0 with an instruction not to change meaning (Appendix A.1), but semantic drift is not measured.
  • domain assumption Human expert review of a subset is representative of the full dataset.
    The paper reports only summary statistics for a subset, making the quality of the remaining items an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways." pith.science (2026). https://pith.science/paper/76LRHN3W

@misc{pith2026250807308,
  author       = {Pith},
  title        = {Pith review of: HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76LRHN3W}},
  note         = {Machine review of arXiv:2508.07308}
}
read the original abstract

HealthBranches is a novel benchmark dataset for medical Question-Answering (Q&A), specifically designed to evaluate complex reasoning in Large Language Models (LLMs). This dataset is generated through a semi-automated pipeline that transforms explicit decision pathways from medical source into realistic patient cases with associated questions and answers. Covering 4,063 case studies across 17 healthcare topics, each data point is based on clinically validated reasoning chains. HealthBranches supports both open-ended and multiple-choice question formats and uniquely includes the full reasoning path for each Q&A. Its structured design enables robust evaluation of LLMs' multi-step inference capabilities, including their performance in structured Retrieval-Augmented Generation (RAG) contexts. HealthBranches establishes a foundation for the development of more trustworthy, interpretable, and clinically reliable LLMs in high-stakes domains while also serving as a valuable resource for educational purposes.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    An LLM-orchestrated framework enables conformance checking in stroke care from unstructured texts, achieving over 86% conformance in hospital data.

Reference graph

Works this paper leans on

52 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    A comprehensive overview of large language models.arXiv preprint arXiv:2307.06435, 2023

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muham- mad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models.arXiv preprint arXiv:2307.06435, 2023

    arXiv 2023

  3. [3]

    Unifying large language models and knowledge graphs: A roadmap.arXiv preprint arXiv:2306.08302, 2023

    Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap.arXiv preprint arXiv:2306.08302, 2023

    arXiv 2023

  4. [4]

    Role of chat gpt in public health.Annals of biomedical engineering, 51(5):868–869, 2023

    Som S Biswas. Role of chat gpt in public health.Annals of biomedical engineering, 51(5):868–869, 2023

    work page 2023

  5. [5]

    Benchmarking retrieval- augmented generation for medicine.arXiv preprint arXiv:2402.13178, 2024

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval- augmented generation for medicine.arXiv preprint arXiv:2402.13178, 2024

    arXiv 2024

  6. [6]

    A survey on rag with llms.Procedia Computer Science, 246:3781–3790, 2024

    Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. A survey on rag with llms.Procedia Computer Science, 246:3781–3790, 2024

    work page 2024

  7. [7]

    Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges

    Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, and Shaoting Zhang. Medical dialogue: A survey of categories, methods, evaluation and challenges.arXiv preprint arXiv:2405.10630, 2024

    work page Pith review arXiv 2024

  8. [8]

    Extract- ing training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extract- ing training data from large language models. In30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021

    work page 2021

Show all 52 references
  1. [9]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems,...

    2020

  3. [11]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM Computing Surveys, 55(9):1–35, 2023

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM Computing Surveys, 55(9):1–35, 2023

    2023

  4. [12]

    Retrieval augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. InInternational conference on machine learning, pages 3929–3938. PMLR, 2020

    2020

  5. [13]

    Atlas: Few-shot learning with retrieval augmented language models.Journal of Machine Learning Research, 24(251):1–43, 2023

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models.Journal of Machine Learning Research, 24(251):1–43, 2023

    2023

  6. [14]

    Retrieving supporting evidence for llms generated answers.arXiv preprint arXiv:2306.13781, 2023

    Siqing Huo, Negar Arabzadeh, and Charles LA Clarke. Retrieving supporting evidence for llms generated answers.arXiv preprint arXiv:2306.13781, 2023

    2023 arXiv

  7. [15]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023

    2023 arXiv

  8. [16]

    Health-llm: Personalized retrieval-augmented disease prediction system.arXiv preprint arXiv:2402.00746, 2024

    Mingyu Jin, Qinkai Yu, Dong Shu, Chong Zhang, Lizhou Fan, Wenyue Hua, Suiyuan Zhu, Yanda Meng, Zhenting Wang, Mengnan Du, et al. Health-llm: Personalized retrieval-augmented disease prediction system.arXiv preprint arXiv:2402.00746, 2024

    2024 arXiv

  9. [17]

    Healthq: Unveiling questioning capabilities of llm chains in healthcare conversations.arXiv preprint arXiv:2409.19487, 2024

    Ziyu Wang, Hao Li, Di Huang, and Amir M Rahmani. Healthq: Unveiling questioning capabilities of llm chains in healthcare conversations.arXiv preprint arXiv:2409.19487, 2024. 11

    2024 arXiv

  10. [18]

    Aidan Gilson, Xuguang Ai, Thilaka Arunachalam, Ziyou Chen, Ki Xiong Cheong, Amisha Dave, Cameron Duic, Mercy Kibe, Annette Kaminaka, Minali Prasad, et al. Enhancing large language models with domain-specific retrieval augment generation: A case study on long-form consumer heal...

    2024 arXiv

  11. [19]

    Graph retrieval-augmented generation for large language models: A survey

    Tyler Thomas Procko and Omar Ochoa. Graph retrieval-augmented generation for large language models: A survey. In2024 Conference on AI, Science, Engineering, and Technology (AIxSET), pages 166–169. IEEE, 2024

    2024

  12. [20]

    medikal: Integrating knowledge graphs as assistants of llms for enhanced clinical diagnosis on emrs.arXiv preprint arXiv:2406.14326, 2024

    Mingyi Jia, Junwen Duan, Yan Song, and Jianxin Wang. medikal: Integrating knowledge graphs as assistants of llms for enhanced clinical diagnosis on emrs.arXiv preprint arXiv:2406.14326, 2024

    2024 arXiv

  13. [21]

    Medical graph rag: Towards safe medical large language model via graph retrieval-augmented generation.arXiv preprint arXiv:2408.04187, 2024

    Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, and Vicente Grau. Medical graph rag: Towards safe medical large language model via graph retrieval-augmented generation.arXiv preprint arXiv:2408.04187, 2024

    2024 arXiv

  14. [22]

    Leveraging retrieval-augmented generation for reliable medical question answering using large language models

    Ksenia Kharitonova, David Pérez-Fernández, Javier Gutiérrez-Hernando, Asier Gutiérrez-Fandiño, Zoraida Callejas, and David Griol. Leveraging retrieval-augmented generation for reliable medical question answering using large language models. InInter- national Conference on Hybr...

    2024

  15. [23]

    Head-qa: A healthcare dataset for complex reasoning

    David Vilares and Carlos Gómez-Rodríguez. Head-qa: A healthcare dataset for complex reasoning. arXiv preprint arXiv:1906.04701, 2019

    1906 arXiv

  16. [24]

    Mediaqa: A question answering dataset on medical dialogues.arXiv preprint arXiv:2108.08074, 2021

    Huqun Suri, Qi Zhang, Wenhua Huo, Yan Liu, and Chunsheng Guan. Mediaqa: A question answering dataset on medical dialogues.arXiv preprint arXiv:2108.08074, 2021

    2021 arXiv

  17. [25]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    2021

  18. [26]

    Pub- medqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pub- medqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu...

    2019

  19. [27]

    Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing Systems, 37:84730–84745, 2024

    Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al. Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing System...

    2024

  20. [28]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  21. [29]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  22. [30]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025

    2025 arXiv

  23. [31]

    Knowledge graph-based question answering with electronic health records

    Junwoo Park, Youngwoo Cho, Haneol Lee, Jaegul Choo, and Edward Choi. Knowledge graph-based question answering with electronic health records. InMachine Learning for Healthcare Conference, pages 36–53. PMLR, 2021

    2021

  24. [32]

    Ultramedical: Building specialized generalistsinbiomedicine

    Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, et al. Ultramedical: Building specialized generalistsinbiomedicine. Advances in Neural Information Processing Systems, 37:26045– 26081, 2024

    2024

  25. [33]

    Sm3-text-to-query: Synthetic multi-model medical text-to-query benchmark

    Sithursan Sivasubramaniam, Cedric E Osei-Akoto, Yi Zhang, Kurt Stockinger, and Jonathan Fürst. Sm3-text-to-query: Synthetic multi-model medical text-to-query benchmark. Advances in Neural Information Processing Systems, 37:88627–88663, 2024. 12

    2024

  26. [34]

    Elsevier Health Sciences, 2009

    Stuart B Mushlin and Harry L Greene.Decision making in medicine: an algorithmic approach. Elsevier Health Sciences, 2009

    2009

  27. [35]

    Algorithms for emergency medicine

    Mark Harrison and Ala Mohammed. Algorithms for emergency medicine. Oxford University Press, 2023

    2023

  28. [36]

    Elsevier Health Sciences, 2021

    Alexander Goldfarb-Rumyantzev.Critical Care Medicine: An Algorithmic Approach E-Book. Elsevier Health Sciences, 2021

    2021

  29. [37]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  30. [40]

    Identifying and mitigating vulnerabilities in llm-integrated applications

    Fengqing Jiang. Identifying and mitigating vulnerabilities in llm-integrated applications. Master’s thesis, University of Washington, 2024

    2024

  31. [41]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [42]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  33. [43]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  34. [44]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  35. [45]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ra- mona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  36. [46]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  37. [47]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  38. [48]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  39. [49]

    https://www.together.ai/

  40. [50]

    G-eval: NLG evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023

  41. [51]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024. 13

    2024 arXiv

  42. [52]

    Lessons learned from the chameleon testbed

    Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S Gunawi, Cody Hammock, et al. Lessons learned from the chameleon testbed. In2020 USENIX annual technical conference (USENIX ATC 20), pages 219–233, 2020. 14 A ...

    2020

  43. [53]

    It has to be structured as a decision tree

    Put the content of the decision tree/graph in a valid JSON format. It has to be structured as a decision tree. If the branch of a decision is unclear or missing, get it from the text. The JSON must contain these keys: a) node: containing the node name b) content: a brief descr...

  44. [54]

    Make sure that you don’t get ’Invalid control character at line’ errors

    Make sure that the JSON is valid and well structured. Make sure that you don’t get ’Invalid control character at line’ errors. The system prompt used to polish raw text extracted from the PDF (textual stream) is: Prompt used to refine raw text extracted from a PDF The uploaded...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.