Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TAGS claims test-time reasoning structure alone can lift hard medical QA accuracy by up to 17 points.

desk verdict Plausible retrieval+reasoning wrapper for medical QA, but the retrieval corpus overlaps the test sources and the paper never shows disjointness, so the headline gains are not yet interpretable. read the letter →

arxiv 2505.18283 v1 pith:7VRCZZCX submitted 2025-05-23 cs.CL cs.AIcs.MA

Jianghao Wu , Feilong Tang , Yulong Li , Ming Hu , Haochen Xue , Shoaib Jameel , Yutong Xie , Imran Razzak This is my paper
classification cs.CLcs.AIcs.MA
keywords medicalquestionansweringtest-timeinferenceretrieval-augmentedreasoningmulti-agentchain-of-thoughtanswerverificationzero-shotpromptingLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TAGS is a test-time wrapper that tries to make medical question answering more accurate without any model training: it pairs a general practitioner-style agent with a specialty-tagged specialist agent, feeds both retrieved medical chain-of-thought examples at two retrieval depths, and lets a zero-shot verifier pick the most internally consistent answer. The paper reports average Pass@1 accuracy gains of 13.8 percentage points on GPT-4o, 16.8 percentage points on DeepSeek-R1, and 9.8 percentage points on a 7B model on a 862-question hard medical QA set, beating several fine-tuned medical LLMs along the way. If correct, these results matter because they suggest that structure at inference time can substitute for expensive fine-tuning and can be dropped onto any frozen model. The paper also claims the framework gains little from adding more specialists or larger retrieval pools, pointing to the two-round design itself as the source of the benefit rather than sheer scale.

What carries the argument

The load-bearing mechanism is the two-stage Hierarchical Retrieval Augmentation (HRA) combined with a four-candidate generation and scoring loop. HRA first retrieves top-K exemplars by cosine similarity of a frozen text encoder's embeddings over question-plus-options, then re-retrieves after the first reasoning round using similarity between the agent's generated rationale and stored rationales, so exemplars are aligned at the reasoning level as well as the surface level. The Generalist-Specialist Reasoning Collaboration (GSRC) uses an auxiliary LLM call to infer a medical specialty label, then runs a general practitioner agent and that specialist agent through two reasoning rounds, yielding four (rationale, answer) candidates. Uncertainty-Aware Answer Aggregation (UAAA) scores each candidate with a zero-shot Reasoning Consistency Evaluator on a 1-5 rubric for logical and clinical coherence, then selects the top-scoring candidate with a deterministic tie-break order. The default configuration, one specialist and K=2 retrieval, was chosen after a hyperparameter sweep showing that accuracy peaks at one specialist and two retrieved exemplars.

What would settle it

Check exact and near-duplicate overlap between the 32,682 retrieval entries and the 862 hard test questions. If any test question or answer appears in a retrieved exemplar, rerun TAGS without those entries and compare average Pass@1; if the gap closes, the gains come from answer leakage rather than from reasoning structure.

Watch

Extended reading notes

Core claim

The paper's central claim is that a structured, inference-only pipeline can make a frozen LLM reason about hard medical multiple-choice questions substantially better, even compared with models fine-tuned for medicine. On the 862-question hard set, the authors report average Pass@1 accuracy rising from 18.0% to 31.8% with GPT-4o, from 27.2% to 44.0% with DeepSeek-R1, and from 14.1% to 23.9% with Qwen2.5-7B. They attribute the gains to the interaction of three components: hierarchical retrieval of medical chain-of-thought exemplars, a generalist/specialist agent pair, and consistency-based answer verification. The same pattern holds across the nine medical QA benchmarks, and removing the ten closest retrieval matches does not erase the improvement, which the authors take as evidence that the method benefits from reasoning patterns rather than from copying near-identical answers.

Load-bearing premise

The measured gains rest on the assumption that the 32,682-example retrieval corpus does not contain the test questions or their answers; the paper gives no overlap analysis, so retrieved material could be leaking ground-truth answers.

Editorial extensions

If this is right

  • A frozen LLM can gain more from structured two-round reasoning with one specialist and two retrieved exemplars than from adding more specialists or longer retrieval lists, since accuracy peaks at K=2 and one specialist.
  • The recipe transfers across model families and scales, including a 7B model, so the same wrapper can be dropped onto new or updated models without retraining.
  • On these hard QA benchmarks, inference-time structure can close the gap with medically fine-tuned models, offering a path to domain improvement when tuning data or compute is scarce.
  • The w/o-top10 ablation, if it holds, implies the method improves through exposure to varied reasoning patterns rather than by copying nearest-neighbor answers.
  • The accuracy gain comes at a latency cost: TAGS takes about 72 seconds per MedQA question with GPT-4o versus 27.7 seconds for CoT-SC, so deployment must trade speed for accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct next test would be to apply the same wrapper to open-ended or non-multiple-choice clinical questions; the verifier's 1-5 rubric and tie-break order are built for a small option set, so transfer is not guaranteed.
  • The paper's w/o-top10 ablation removes only the ten nearest semantic matches and is run on two datasets with a 7B model, so it does not rule out leakage through rationale-level retrieval or lower-ranked examples; an overlap analysis between the retrieval corpus and the test split would settle whether part of the gain is answer recall rather than reasoning.
  • Because the verifier is the same zero-shot LLM that generates the candidates, it could systematically over-score plausible-sounding but wrong rationales; calibration or human sample auditing would be needed before clinical use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces TAGS, a test-time, training-free framework for multiple-choice medical question answering. TAGS combines a generalist agent and a specialty-inferred specialist agent, a two-stage hierarchical retriever that draws top-K exemplars from the external MedReason corpus, and a reasoning-consistency scorer that selects among four candidate (rationale, answer) pairs. The authors evaluate on a curated 862-question hard set across nine medical QA benchmarks, reporting average Pass@1 gains of +13.8 points for GPT-4o, +16.8 for DeepSeek-R1, and +9.8 for Qwen2.5-7B over their zero-shot baselines, and comparing favorably with several fine-tuned medical LLMs.

Significance. If the reported gains are not an artifact of retrieval leakage, TAGS would be a practically valuable drop-in wrapper: it requires no parameter updates, works across different base models, and the paper includes module ablations and a sensitivity analysis. The use of an external retrieval corpus and exact reporting of per-dataset numbers are strengths. However, the evaluation currently lacks a disjointness check between the retrieval corpus and the test set, and the hyperparameters are selected on the test data, so the central claim of reasoning-driven improvement is not yet supported.

major comments (4)
  1. [§4.1, §3.1, Appendix D] The MedReason corpus is built from MedQA, MedMCQA, PubMedQA, MMLU, MedXpert, Huatuo, and HLE, while the test set is drawn from MedQA, PubMedQA, MedMCQA, MedBullets, MedExQA, MMLU, MMLU-Pro, and MedXpertQA. Because HRA retrieves from this corpus in both Stage 1 (Eq. 1) and Stage 2 (Eqs. 2-3) before every answer, any overlapping or near-duplicate question can directly provide the ground-truth answer or a reasoning chain that reveals it. The paper provides no overlap analysis. The w/o-top10 ablation in §4.4/Table 4 removes only the ten nearest semantic matches, is run on two datasets with Qwen2.5-7B, and does not control Stage-2 rationale retrieval or exact/near duplicates beyond the top-10. Its results are also inconsistent (MMLU 35.6 to 37.0; MedXpert-U 29.0 to 24.0), contradicting the text's claim that performance "only marginally decreases." An exact and near-duplicate overlap analysis is required, and the main experiments should be re-run after removing any overlapping corpus entries.
  2. [§4.5, Figure 2] The default configuration (one specialist, K=2) is chosen by maximizing accuracy on MMLU and MedXpert-U, which are themselves part of the nine-dataset test set and appear in Tables 1-3 and Table 4. This is a form of test-set selection; the reported gains on these two datasets are in-sample. The authors should either introduce a held-out validation split for hyperparameter selection or report results for a range of configurations across all nine datasets.
  3. [Tables 1-3] No error bars, confidence intervals, or repeated runs are reported. Since each dataset contains only 73-100 questions, a 5-10 point difference is a small number of questions, and per-dataset method ordering is noisy. For example, on MedExQA with GPT-4o, TAGS (17) is well below CoT (24) and Self-Refine (22) in Table 1. The average improvements may be robust, but the paper should provide bootstrap confidence intervals or multiple runs and avoid claiming "consistent superiority" (Section 5) without such support.
  4. [Section 5 vs. Table 1] The claim that TAGS exhibits "consistent superiority" over baselines is not supported by the GPT-4o results. On MedMCQA, few-shot (34) and CoT-SC (35) outperform TAGS (32); on MedExQA, CoT (24) outperforms TAGS (17); and on PubMedQA, MedAgents and Multi-Persona both achieve 15 vs. TAGS's 13. The conclusion should be revised to describe the pattern accurately, for example by stating that gains are concentrated in certain datasets, and to specify which comparisons support the claim.
minor comments (6)
  1. [Abstract, Section 1] "By 13.8%" should read "by 13.8 percentage points" to avoid confusion with relative improvement (the relative gain is 31.8/18.0 - 1 = 76.7%).
  2. [§3.3 vs. Appendix C] Section 3.3 states c_k ∈ [0,5] while Appendix C states the score ranges from 1 to 5; unify these definitions.
  3. [Table 2, Section 4.2] The text references "MDAgents consistently failed under this setting due to format inconsistencies," but Table 2 has no MDAgents row; include the row with an appropriate footnote or remove the reference.
  4. [Figure 2] The figure lacks axis legends and dataset labels; the reader cannot tell which curve corresponds to MMLU versus MedXpert-U.
  5. [Table 4] Several rows use identical checkmark patterns, making it impossible to identify which modules are ablated; label each row or use distinct symbols to indicate the combination tested.
  6. [Section 4.1] "Nine MedQA benchmarks" should be "nine medical QA benchmarks" (or "nine datasets") for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: TAGS's reported gains are measurements against external benchmarks using an external retrieval corpus; no equation or fitted parameter reduces the result to its own inputs.

full rationale

The derivation chain is a test-time pipeline rather than a fitted model. HRA (Eqs. 1-3) retrieves external MedReason exemplars by cosine similarity, GSRC (Eqs. 4-5) generates four candidate (rationale, answer) pairs from a frozen LLM conditioned on those exemplars, and UAAA (Eq. 7) selects the final answer by an argmax over RCE scores. No prediction is defined in terms of the target answer, no parameter is fitted to the test set by construction, and no load-bearing uniqueness or self-citation argument is used; the improvement claims are Pass@1 measurements on the MEDAGENTSBENCH hard sets. The Limitations section itself flags the legitimate concern that the RCE is a zero-shot prompt of the same LLM that generated the candidates, which weakens independent verification but does not make answer selection a definitional reduction. Similarly, the absence of an overlap analysis between the 32,682 MedReason entries and the 862 test questions is a data-contamination/correctness risk (the w/o-top10 ablation in §4.4 only removes the ten nearest semantic matches), but actual overlap would have to be demonstrated before it could count as a reduction of the result to retrieved answers. Test-set hyperparameter selection (e.g., K=2 in Figure 2b) is an overfitting concern, not a circular step. Accordingly, no circularity is established, and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method contributes no fitted model parameters and no new physical or ontological entities. Its load-bearing assumptions are the quality and non-overlap of the external MedReason corpus, the reliability of the zero-shot RCE verifier, the accuracy of the specialty classifier, and the representativeness of the failure-rate-selected hard subset. Hyperparameters K=2 and one specialist are the only hand-tuned numbers and were selected from test-set curves.

free parameters (2)
  • K (retrieval size) = 2
    Chosen because accuracy peaks at K=2 in Figure 2(b) on MMLU and MedXpert-U; no held-out validation split is reported.
  • number of specialist agents = 1
    Chosen because adding more specialists gives no further gains in Figure 2(a); tuned on the same test benchmarks.
assumptions (4)
  • domain assumption MedReason retrieval corpus is factually reliable and disjoint from the test questions.
    Used by both HRA stages in §3.1; no overlap analysis between the 32,682 retrieval entries and the 862 test questions is reported.
  • domain assumption Zero-shot Reasoning Consistency Evaluator (RCE) scores are meaningful.
    Eq. 7 in §3.3 selects the final answer by argmax over RCE scores; Limitations admits the verifier may inherit the generator's hallucination tendencies.
  • domain assumption Specialty classifier identifies a useful medical specialty.
    §3.2 injects the inferred specialty into the specialist prompt; the S-w-3rd robustness check in §4.4 only partially probes classifier errors.
  • domain assumption The failure-rate-selected hard subset is a representative stress test.
    §4.1 selects questions with less than 50% accuracy across strong models; this selection can amplify differences between inference-time methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification." pith.science (2026). https://pith.science/paper/7VRCZZCX

@misc{pith2026250518283,
  author       = {Pith},
  title        = {Pith review of: TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VRCZZCX}},
  note         = {Machine review of arXiv:2505.18283}
}
read the original abstract

Recent advances such as Chain-of-Thought prompting have significantly improved large language models (LLMs) in zero-shot medical reasoning. However, prompting-based methods often remain shallow and unstable, while fine-tuned medical LLMs suffer from poor generalization under distribution shifts and limited adaptability to unseen clinical scenarios. To address these limitations, we present TAGS, a test-time framework that combines a broadly capable generalist with a domain-specific specialist to offer complementary perspectives without any model fine-tuning or parameter updates. To support this generalist-specialist reasoning process, we introduce two auxiliary modules: a hierarchical retrieval mechanism that provides multi-scale exemplars by selecting examples based on both semantic and rationale-level similarity, and a reliability scorer that evaluates reasoning consistency to guide final answer aggregation. TAGS achieves strong performance across nine MedQA benchmarks, boosting GPT-4o accuracy by 13.8%, DeepSeek-R1 by 16.8%, and improving a vanilla 7B model from 14.1% to 23.9%. These results surpass several fine-tuned medical LLMs, without any parameter updates. The code will be available at https://github.com/JianghaoWu/TAGS.

Figures

Figures reproduced from arXiv: 2505.18283 by the authors.

Figure 1
Figure 1. Overview of the proposed TAGS framework. The architecture consists of three modules: (A) HRA (Hierarchical Retrieval Augmentation), a two-stage retrieval process that selects semantically relevant exemplars (T1) and refines them based on rationale alignment (TG,2, TS,2). (B) GSRC (Generalist-Specialist Reasoning Collaboration) employs dual-agent reasoning across two rounds, generating four candidate (Rationale, Answ… view at source ↗
Figure 2
Figure 2. Hyper-parameter sensitivity analysis of spe [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning

    cs.AI 2026-07 accept novelty 6.0 of 10

    A dual clinical-computational taxonomy for medical LLM reasoning plus a five-level 5k-sample benchmark showing specialists excel at diagnosis and general models at decision support/dialogue.

Reference graph

Works this paper leans on

46 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. 2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657

    arXiv 2025

  4. [4]

    Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67

    2023

  5. [5]

    Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2024 a . Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060

    arXiv 2024

  6. [6]

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024 b . M 3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318--2335

    work page 2024

  7. [7]

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024 c . Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925

    arXiv 2024

  8. [8]

    Kai Chen, Xinfeng Li, Tianpei Yang, Hewei Wang, Wei Dong, and Yang Gao. 2025 a . Mdteamgpt: A self-evolving llm-based multi-agent framework for multi-disciplinary team medical consultation. arXiv preprint arXiv:2503.13856

    arXiv 2025

Show all 46 references
  1. [9]

    Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023 a . Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv e-prints, pages arXiv--2305

    2023

  2. [10]

    Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, et al. 2023 b . A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommenda...

    2023 arXiv

  3. [11]

    Xi Chen, Huahui Yi, Mingke You, WeiZhi Liu, Li Wang, Hairui Li, Xue Zhang, Yingman Guo, Lei Fan, Gang Chen, et al. 2025 b . Enhancing diagnostic capability with multi-agents conversational large language models. NPJ digital medicine, 8(1):159

    2025

  4. [12]

    Xuhang Chen, Shenghong Luo, Chi-Man Pun, and Shuqiang Wang. 2024 d . MedPrompt : Cross-modal prompting for multi-task medical image translation. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 61--75. Springer

    2024

  5. [13]

    Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Wang Siyuan, Zhongyu Wei, and Fei Huang. 2025. Ai hospital: Benchmarking large language models in a multi-agent medical interaction simulator. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1...

    2025

  6. [14]

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. 2024. Empowering biomedical discovery with ai agents. Cell, 187(22):6125--6151

    2024

  7. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Qiangqiang He, Shuwei Qian, Jie Zhang, and Chongjun Wang. 2025. Inference retrieval-augmented multi-modal chain-of-thoughts reasoning for language models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2025

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  10. [18]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1--43

    2023

  11. [19]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  12. [20]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. PubMedQA : A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146

    2019 arXiv

  13. [21]

    Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022. Biomedical question answering: a survey of approaches and challenges. ACM Computing Surveys (CSUR), 55(2):1--36

    2022

  14. [22]

    Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. 2024 a . MDAgents : An adaptive collaboration of llms in medical decision making. arXiv preprint arXiv:2404.15155

    2024 arXiv

  15. [23]

    Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. 2024 b . MedExQA : Medical question answering benchmark with multiple explanations. arXiv preprint arXiv:2406.06331

    2024 arXiv

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information proc...

    2020

  17. [25]

    Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, and Zuozhu Liu. 2024. Medcot: Medical chain of thought via hierarchical expert. arXiv preprint arXiv:2412.13736

    2024 arXiv

  18. [26]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-Refine : Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA : A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

    2022

  20. [28]

    Malaikannan Sankarasubbu Ankit Pal and Malaikannan Sankarasubbu. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. Hugging Face repository

    2024

  21. [29]

    Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384

    2024

  22. [30]

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652

    2023 arXiv

  23. [31]

    Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Haotian Sun, Hang Wu, Carl Yang, and May D Wang. 2024. Medadapter: Efficient test-time adaptation of large language models towards medical reasoning. arXiv preprint arXiv:2405.03000

    2024 arXiv

  24. [32]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1--8

    2025

  25. [33]

    Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, et al. 2025. Medagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv preprint arXiv:2503.07459

    2025 arXiv

  26. [34]

    Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. 2023. MedAgents : Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.10537

    2023 arXiv

  27. [35]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

    2022

  28. [36]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024. MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    2024 arXiv

  29. [37]

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300

    2023 arXiv

  30. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  31. [39]

    Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. 2025. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993

    2025 arXiv

  32. [40]

    Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. 2022. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pages 24631--24645. PMLR

    2022

  33. [41]

    Shaochen Xu, Yifan Zhou, Zhengliang Liu, Zihao Wu, Tianyang Zhong, Huaqin Zhao, Yiwei Li, Hanqi Jiang, Yi Pan, Junhao Chen, et al. 2024. Towards next-generation medical agent: How o1 is reshaping decision-making in medical scenarios. arXiv preprint arXiv:2411.14461

    2024 arXiv

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  35. [43]

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024 b . Self-distillation bridges distribution gap in language model fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1028--1043

    2024

  36. [44]

    Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. 2024. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:9...

    2024

  37. [45]

    Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112

    2023 arXiv

  38. [46]

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. MedXpertQA : Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.