Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that test-time scaling in medical AI only helps reasoning models on hard tasks, and that the best strategy depends on the model and the task.

desk verdict Useful applied study with a solid core result—longer reasoning isn't universally better in medicine—but its strategy tables outrun the measurements. read the letter →

arxiv 2506.13102 v1 pith:IG6KS3YP submitted 2025-06-16 cs.CL cs.AIcs.CY

Gyutaek Oh , Seoyeon Kim , Sangjoon Park , Byung-Hoon Kim This is my paper · ORCID
classification cs.CLcs.AIcs.CY
keywords test-timescalingmedicalquestionansweringlargelanguagemodelsvision-languagechain-of-thoughtreasoningsequentialandparallelmodelrobustnesstokenbudgets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that giving medical AI models more time to think during inference is not always helpful. Across five text-based and two vision-based medical benchmarks, it finds that longer chain-of-thought reasoning improves accuracy mainly for models that naturally produce long reasoning traces, and mainly on difficult tasks; on easy tasks extra tokens either do nothing or hurt. It then compares sequential self-revision with parallel sampling, arguing that the best choice depends on the model's token usage and on task difficulty. If this holds, choosing a scaling strategy per model and task could deliver most of the benefit of test-time scaling without wasting compute, and could guide how to avoid misleading prompts in high-stakes medical question answering.

What carries the argument

The central object is the test-time scaling strategy itself, operationalized along two axes: how many tokens the model is allowed to emit, controlled by the maximum sequence length with forced boxed-answer completion, and how reasoning is extended, either by iterative 'Wait.' self-revision in sequential scaling or by generating multiple independent responses aggregated with a shortest majority vote in parallel scaling. The argument is carried by comparing accuracy and coverage across token budgets, and by classifying models as reasoning versus non-reasoning and medical versus general through their measured token usage.

What would settle it

Run a medical reasoning model on the hard subset of MedXpertQA with token budgets 512, 2048, and 8192: if accuracy is flat or falls with budget, the core difficulty-based claim fails. Conversely, if a non-reasoning general model such as Llama 3-8B shows a large accuracy jump on PubMedQA when the budget grows, the claim that easy tasks gain nothing from more tokens would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that longer reasoning is not universally beneficial across medical tasks. On easy benchmarks such as PubMedQA and OmniMedVQA, increasing the token budget fails to improve accuracy, and sequential revision can drive models away from originally correct answers. On difficult benchmarks such as MedXpertQA and MedCalc-Bench, models that use extended reasoning traces gain from more tokens, with the paper reporting clear scalability for m1 among LLMs and QVQ among VLMs. The paper further claims that medical fine-tuned models underperform general models on medical calculation because their training data lacks numerical reasoning, and that a hybrid sequential-parallel strategy suits intermediate difficulty, while fully parallel scaling suits models that produce short traces and easy tasks. It concludes with strategy tables: parallel scaling for easy tasks and short-reasoning models, sequential scaling for difficult QA, token-budget expansion for calculation tasks, and avoidance of all user-driven misleading cues on hard tasks.

Load-bearing premise

Everything rests on the assumption that a handful of open-source models and the authors' ordering of datasets by difficulty are representative enough that the observed patterns generalize to other medical LLMs and VLMs.

Editorial extensions

If this is right

  • Easy medical QA tasks should be handled with parallel sampling rather than longer reasoning, because sequential revision can reduce accuracy.
  • For difficult QA, iterative sequential scaling should be used for reasoning models, while calculation-heavy tasks favor a single extended reasoning path with a larger token budget.
  • Models that naturally produce short answers, such as HuatuoGPT-o1, gain little from token-budget scaling, whereas models like m1 that naturally reason long benefit strongly.
  • Users should avoid injecting physician-opinion hints, especially definitive or expert-toned ones, because perturbed prompts hurt more on hard tasks and scaling does not fully restore accuracy.
  • Medical fine-tuned models may need calculation-oriented training data to match general models on numeric tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If the difficulty-based pattern transfers to clinical deployment, easy triage questions could run with short parallel samples while long sequential reasoning is reserved for complex cases, a workflow the paper does not itself test.
  • Editorial: The observed gap between coverage and final accuracy suggests that adding a trained verifier to select among parallel candidates, which the paper lists as unexplored, could convert correct intermediate reasoning into correct final answers.
  • Editorial: For VLMs, the evidence implies that current medical vision benchmarks may not separate visual reasoning from textual clue-following, so a benchmark that removes text-side hints could change the measured benefits of scaling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper empirically studies test-time scaling strategies for medical AI across LLMs and VLMs. The authors vary token budgets for a broad set of open-source models on five text and two multimodal medical benchmarks; compare sequential revision ('Wait.'-style), parallel sampling with shortest-majority-vote, and hybrid configurations on two medical reasoning LLMs (HuatuoGPT-o1-7B and m1-7B); and then test robustness to misleading user-provided physician opinions. The central claim is that longer reasoning is not universally beneficial in medicine: effectiveness depends on model type and task difficulty, with reasoning models benefiting more on difficult tasks, VLMs benefiting only marginally, and parallel scaling being preferable for models that naturally produce short reasoning traces. The paper distills the findings into recommended strategies in Tables III and IV.

Significance. The question addressed is timely and practically important, and the paper has several genuine strengths: it covers a wide range of open-source LLMs and VLMs, separates token-budget scaling from sequential/parallel scaling, distinguishes easy and difficult medical tasks, and explicitly studies robustness to user-driven factors, which is rarely examined in test-time-scaling work. The main qualitative finding that longer reasoning is not universally beneficial is plausible and consistent with recent literature. The paper is also commendably explicit about its limitations, including the exclusion of proprietary models and the limited VLM analysis. If the recommendations in Tables III and IV were fully supported, the paper would provide useful guidance for practitioners deploying medical QA systems. However, as detailed below, the central model-aware recommendation rests on a much narrower experimental base than the tables claim, and the absence of uncertainty quantification weakens several quantitative comparisons.

major comments (4)
  1. [Section IV-B, Fig. 5, Tables III-IV] The sequential-versus-parallel comparison is run on only two medical-reasoning 7B LLMs (HuatuoGPT-7B and m1-7B), yet Tables III and IV issue strategy recommendations for all four model-type categories, including General Non-Reasoning, General Reasoning, Medical Non-Reasoning, and VLMs. No non-reasoning model, general reasoning model, or VLM is subjected to the 'Wait.' revision or shortest-majority-vote protocol, so the entries 'Prefer parallel scaling' and 'Apply iterative sequential scaling' are inferred from token-budget saturation in Section IV-A rather than measured. Observing that a model does not consume a larger token budget does not establish how it behaves under parallel majority voting or iterative self-revision. Either the sequential/parallel protocol should be run on representative models from each category, or the tables should be explicitly restricted to the measured model types.
  2. [Section IV-C] The robustness experiments use 'the optimal configuration identified in Section IV-B' as the Seq-Para condition, but that configuration was selected on the same two models and on the same benchmarks (MedQA, MedBullets, MedXpertQA) using the results of Fig. 5. The subsequent robustness evaluation therefore reuses the selection data and does not provide an independent test of whether the recommended hybrid strategy generalizes. The authors should report the exact pair counts and token budgets used, justify the absence of any held-out model or dataset split, and ideally compare against a small predefined set of configurations rather than a data-selected optimum.
  3. [Section IV-A, Figs. 4-5] No error bars, repeated runs, or statistical significance tests are reported anywhere. Several load-bearing conclusions rest on small or visually noisy differences; for example, the claim that QVQ shows a 'consistent upward trend' in accuracy with increased token budgets on OmniMedVQA in Fig. 4(b), and the differences between sequential, parallel, and hybrid arms in Fig. 5. Given that the paper offers practical guidance for choosing among scaling strategies, the authors should provide variance estimates over at least a few independent runs, or confidence intervals, for the headline comparisons.
  4. [Section III-B and Section IV-A] Models with 70B or more parameters are run in 4-bit quantization (indicated by '*' in the legends), and this choice is not validated against full-precision baselines. Since quantization can affect both token usage and reasoning behavior, the comparisons between large and small models, such as the DeepSeek-R1 size trends, may be confounded. A small validation set comparing quantized and unquantized outputs for at least one large model would substantiate the assumption that 4-bit quantization does not change relative behavior.
minor comments (5)
  1. [General] There are several typos: 'Followng' in Section II-A, 'experiments wit VLMs' in Section IV-A2, 'OmniMedQV A' in Table I, 'Whle' in Section IV-A1, and 'gain' in Section IV-A2.
  2. [Figure 4] The caption of Fig. 4(b) says 'Accuracy of LLMs' but the panel reports VLM accuracy; the caption should be corrected.
  3. [Table II and Section III-B] The notation for HuatuoGPT-o1 variants (7B, 8B, 70B, 72B) and m1-7B-1k is not fully defined; Table II lists HuatuoGPT-o1 with base models Llama or Qwen but does not specify which variant corresponds to which base model.
  4. [Table IV] The rows for General Reasoning and Medical Reasoning list both 'High performance on reasoning-intensive tasks' and 'Relatively low performance on reasoning-intensive medical question answering tasks'; the apparent contradiction should be resolved, since the latter appears to be intended for calculation tasks.
  5. [Section III-A] The difficulty ordering of datasets is asserted from prior work and the number of answer choices, but since Table III depends directly on the easy/intermediate/difficult categorization, providing baseline accuracies or a validation of the ordering would make the recommendations easier to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the recommendations are empirical summaries of direct measurements, with no fitted parameter renamed as a prediction.

full rationale

The paper's central claims are direct empirical observations rather than derivations from fitted inputs. Token-budget scaling, sequential versus parallel scaling, and robustness under misleading prompts are all measured in Figs. 3-6 and then summarized in Tables III-IV; no model parameter is fitted to a subset of data and then reported as a prediction of a closely related quantity. The difficulty ordering of datasets in Section III-A is an explicit input assumption, not an output of the experiments, so the task-difficulty recommendations are inductive summaries rather than circular deductions. Section IV-C reuses the "optimal configuration identified in Section IV-B" on the same benchmark families, but the user-driven perturbation conditions are new measurements, so the robustness claim is not forced by construction; at most this is post-hoc selection, which is a methodological limitation, not circularity. The paper cites its own prior work [50] to justify the misleading-prompt scenario design, but that citation is methodological and non-load-bearing: the perturbation results rest on the paper's own runs involving GPT-4o-selected misleading hints, tone, and expertise level. The skeptical concern that sequential/parallel recommendations are extrapolated from only two 7B medical-reasoning LLMs is a valid external-validity limitation, and the paper itself acknowledges the open-source-only scope in Section V, but this does not make any claim equivalent to its input by definition or by self-citation. No circular step meeting the required evidentiary standard was identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical constructs or fitted scientific parameters. The free parameters listed are experimental design choices. The axioms are the assumptions about benchmark validity and model representativeness that the recommendations rely on. No invented entities.

free parameters (4)
  • Token budget levels = 512, 1024, 2048, 4096, 8192
    Hand-chosen range for the max generation length. Results vary with these, but the qualitative finding that most non-reasoning models saturate at about 500 tokens is robust across budgets.
  • Sequential/parallel scaling grid = Pairs such as (0,4), (1,2), (2,1), (4,0)
    Hand-chosen combinations of iterative revisions and parallel samples. The optimal combination is selected empirically in Section IV-B.
  • Optimal configuration for user-driven experiments = Varies by model and task, selected from Section IV-B
    Data-dependent selection. This adds a mild circularity because the same benchmarks are used to choose and then evaluate the configuration.
  • 4-bit quantization threshold = 70B and above
    Chosen for computational feasibility; assumed not to change relative model behavior.
assumptions (5)
  • domain assumption Benchmark datasets and their difficulty ordering accurately reflect medical task complexity.
    The paper defines difficulty order in Section III-A and builds recommendations on it (Table III).
  • domain assumption Accuracy via exact match or 5% error margin is a valid metric for medical QA and calculation tasks.
    Evaluation in Section III-C2; the metric choice is standard but assumes no meaningful partial credit.
  • domain assumption 4-bit quantization does not materially change model reasoning behavior.
    Used for 70B+ models in Section III-B; no validation provided.
  • domain assumption The selected open-source models are representative of their categories (general, medical, reasoning, non-reasoning).
    Recommendations in Table IV generalize from a limited set of models.
  • domain assumption The 'Wait.' sequential scaling and shortest majority vote parallel scaling are faithful implementations of these strategies.
    These are adapted from prior work [17], [20]; results depend on these implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs." pith.science (2026). https://pith.science/paper/IG6KS3YP

@misc{pith2026250613102,
  author       = {Pith},
  title        = {Pith review of: Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IG6KS3YP}},
  note         = {Machine review of arXiv:2506.13102}
}
read the original abstract

Test-time scaling has recently emerged as a promising approach for enhancing the reasoning capabilities of large language models or vision-language models during inference. Although a variety of test-time scaling strategies have been proposed, and interest in their application to the medical domain is growing, many critical aspects remain underexplored, including their effectiveness for vision-language models and the identification of optimal strategies for different settings. In this paper, we conduct a comprehensive investigation of test-time scaling in the medical domain. We evaluate its impact on both large language models and vision-language models, considering factors such as model size, inherent model characteristics, and task complexity. Finally, we assess the robustness of these strategies under user-driven factors, such as misleading information embedded in prompts. Our findings offer practical guidelines for the effective use of test-time scaling in medical applications and provide insights into how these strategies can be further refined to meet the reliability and interpretability demands of the medical domain.

Figures

Figures reproduced from arXiv: 2506.13102 by the authors.

Figure 1
Figure 1. Overview of our study. We present a comprehensive investigation of test-time scaling in the medical domain across diverse medical benchmarks and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Test-time scaling strategies used in our experiments: (a) No test-time [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Test-time scaling of various LLMs across different token budgets on multiple medical benchmark datasets: (a) Average number of reasoning tokens [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Test-time scaling of various VLMs across different token budgets on medical VQA benchmark datasets: (a) Average number of reasoning tokens used [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of sequential and parallel test-time scaling of LLMs on multiple medical benchmark datasets. The dotted lines indicate the accuracy of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Test-time scaling under different user-driven factor types on medical QA benchmark datasets. (a) HuatuoGPT-o1-7B and (b) m1-7B-1k. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 16 canonical work pages

  1. [1]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  2. [2]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al. , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , vol. 2, no. 3, p. 6, 2023

    2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  5. [5]

    GPT-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “GPT-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    arXiv 2024

  6. [6]

    OpenAI o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney et al. , “OpenAI o1 system card,” arXiv preprint arXiv:2412.16720 , 2024

    arXiv 2024

  7. [7]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The Llama 3 herd of models,” arXiv e-prints , pp. arXiv–2407, 2024

    work page 2024

  8. [8]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

Show all 55 references
  1. [9]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X...

    2025 arXiv

  2. [10]

    Flamingo: A visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: A visual language model for few-shot learning,” Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  3. [11]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning . PMLR, 2022, pp. 12 888–12 900

    2022

  4. [12]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  5. [13]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, pp. 34 892– 34 916, 2023

    2023

  6. [14]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  7. [15]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al., “Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  8. [16]

    xGen-MM (BLIP- 3): A family of open large multimodal models,

    L. Xue, M. Shu, A. Awadalla, J. Wang, A. Yan, S. Purushwalkam, H. Zhou, V . Prabhu, Y . Dai, M. S. Ryoo et al. , “xGen-MM (BLIP- 3): A family of open large multimodal models,” arXiv preprint arXiv:2408.08872, 2024

    2024

  9. [17]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. Hashimoto, “s1: Simple test-time scaling,” arXiv preprint arXiv:2501.19393 , 2025

    2025 arXiv

  10. [18]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,

    C. V . Snell, J. Lee, K. Xu, and A. Kumar, “Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,” in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https: //openreview.net/forum?i...

    2025

  11. [19]

    Towards thinking-optimal scaling of test-time compute for LLM reasoning,

    W. Yang, S. Ma, Y . Lin, and F. Wei, “Towards thinking-optimal scaling of test-time compute for LLM reasoning,” URL https://arxiv. org/abs/2502.18080, 2025

    2025

  12. [20]

    Revisiting the test- time scaling of o1-like models: Do they truly possess test-time scaling capabilities?

    Z. Zeng, Q. Cheng, Z. Yin, Y . Zhou, and X. Qiu, “Revisiting the test- time scaling of o1-like models: Do they truly possess test-time scaling capabilities?” arXiv preprint arXiv:2502.12215 , 2025

    2025 arXiv

  13. [21]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  14. [22]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  15. [23]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “DeepSeekMath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300 , 2024

    2024 arXiv

  16. [24]

    Multi- modal understanding and generation for medical images and text via vision-language pre-training,

    J. H. Moon, H. Lee, W. Shin, Y .-H. Kim, and E. Choi, “Multi- modal understanding and generation for medical images and text via vision-language pre-training,” IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 12, pp. 6070–6080, 2022

    2022

  17. [25]

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “LLaV A-Med: Training a large language-and-vision 11 TABLE IV RECOMMENDED TEST -TIME SCALING STRATEGIES BASED ON MODEL TYPES IN THE MEDICAL DOMAIN . Model Type Strength Weakness Recommendat...

    2023

  18. [26]

    Self- supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology,

    S. Park, E. S. Lee, K. S. Shin, J. E. Lee, and J. C. Ye, “Self- supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology,” Medical Image Analysis , vol. 91, p. 103021, 2024

    2024

  19. [27]

    HuatuoGPT-o1, towards medical complex reasoning with LLMs,

    J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang, “HuatuoGPT-o1, towards medical complex reasoning with LLMs,”arXiv preprint arXiv:2412.18925, 2024

    2024 arXiv

  20. [28]

    A generalist vision–language foundation model for diverse biomedical tasks,

    K. Zhang, R. Zhou, E. Adhikarla, Z. Yan, Y . Liu, J. Yu, Z. Liu, X. Chen, B. D. Davison, H. Ren et al. , “A generalist vision–language foundation model for diverse biomedical tasks,” Nature Medicine, pp. 1–13, 2024

    2024

  21. [29]

    UltraMedical: Building specialized generalists in biomedicine,

    K. Zhang, S. Zeng, E. Hua, N. Ding, Z.-R. Chen, Z. Ma, H. Li, G. Cui, B. Qi, X. Zhu et al. , “UltraMedical: Building specialized generalists in biomedicine,” Advances in Neural Information Processing Systems , vol. 37, pp. 26 045–26 081, 2024

    2024

  22. [30]

    Meds3: Towards medical small language models with self-evolved slow think- ing,

    S. Jiang, Y . Liao, Z. Chen, Y . Zhang, Y . Wang, and Y . Wang, “Meds3: Towards medical small language models with self-evolved slow think- ing,” arXiv preprint arXiv:2501.12051 , 2025

    2025

  23. [31]

    QoQ-Med: Building mul- timodal clinical foundation models with domain-aware GRPO training,

    W. Dai, P. Chen, C. Ekbote, and P. P. Liang, “QoQ-Med: Building mul- timodal clinical foundation models with domain-aware GRPO training,” arXiv preprint arXiv:2506.00711 , 2025

    2025

  24. [32]

    Med-R1: Reinforcement learning for generalizable medical reasoning in vision-language models,

    Y . Lai, J. Zhong, M. Li, S. Zhao, and X. Yang, “Med-R1: Reinforcement learning for generalizable medical reasoning in vision-language models,” arXiv preprint arXiv:2503.13939 , 2025

    2025

  25. [33]

    MedVLM-R1: Incentivizing medical reasoning capability of vision-language models (VLMs) via reinforcement learning,

    J. Pan, C. Liu, J. Wu, F. Liu, J. Zhu, H. B. Li, C. Chen, C. Ouyang, and D. Rueckert, “MedVLM-R1: Incentivizing medical reasoning capability of vision-language models (VLMs) via reinforcement learning,” arXiv preprint arXiv:2502.19634, 2025

    2025 arXiv

  26. [34]

    m1: Unleash the potential of test-time scaling for medical reasoning with large language models,

    X. Huang, J. Wu, H. Liu, X. Tang, and Y . Zhou, “m1: Unleash the potential of test-time scaling for medical reasoning with large language models,” arXiv preprint arXiv:2504.00869 , 2025

    2025

  27. [35]

    O1 replication journey–Part 3: Inference-time scaling for medical reasoning,

    Z. Huang, G. Geng, S. Hua, Z. Huang, H. Zou, S. Zhang, P. Liu, and X. Zhang, “O1 replication journey–Part 3: Inference-time scaling for medical reasoning,” arXiv preprint arXiv:2501.06458 , 2025

    2025 arXiv

  28. [36]

    Inference-time scaling for complex tasks: Where we stand and what lies ahead,

    V . Balachandran, J. Chen, L. Chen, S. Garg, N. Joshi, Y . Lara, J. Lang- ford, B. Nushi, V . Vineet, Y . Wu et al. , “Inference-time scaling for complex tasks: Where we stand and what lies ahead,” arXiv preprint arXiv:2504.00294, 2025

    2025 arXiv

  29. [37]

    HuatuoGPT-Vision, towards injecting medical visual knowledge into multimodal LLMs at scale,

    J. Chen, C. Gui, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, R. Zhang, Z. Cai, K. Ji et al. , “HuatuoGPT-Vision, towards injecting medical visual knowledge into multimodal LLMs at scale,” arXiv preprint arXiv:2406.19280, 2024

    2024 arXiv

  30. [38]

    MedGemma: Advanced AI models for medical text and image analysis,

    G. Health, “MedGemma: Advanced AI models for medical text and image analysis,” May 2025. [Online]. Available: https://medgemma.org/

    2025

  31. [39]

    Qwen2.5-VL technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang et al. , “Qwen2.5-VL technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  32. [40]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ere et al. , “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786 , 2025

    2025 arXiv

  33. [41]

    LLaV A- o1: Let vision language models reason step-by-step,

    G. Xu, P. Jin, L. Hao, Y . Song, L. Sun, and L. Yuan, “LLaV A- o1: Let vision language models reason step-by-step,” arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  34. [42]

    QVQ: To see the world with wisdom,

    Q. Team, “QVQ: To see the world with wisdom,” December 2024. [Online]. Available: https://qwenlm.github.io/blog/qvq-72b-preview/

    2024

  35. [43]

    Yi: Open foundation models by 01. ai,

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chen et al. , “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652 , 2024

    2024 arXiv

  36. [44]

    CLIMB: Data foundations for large scale multimodal clinical foundation models,

    W. Dai, P. Chen, M. Lu, D. Li, H. Wei, H. Cui, and P. P. Liang, “CLIMB: Data foundations for large scale multimodal clinical foundation models,” arXiv preprint arXiv:2503.07667 , 2025

    2025 arXiv

  37. [45]

    PubMedQA: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu, “PubMedQA: A dataset for biomedical research question answering,” arXiv preprint arXiv:1909.06146, 2019

    1909 arXiv

  38. [46]

    What disease does this patient have? A large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? A large-scale open domain question answering dataset from medical exams,” Applied Sciences , vol. 11, no. 14, p. 6421, 2021

    2021

  39. [47]

    Benchmarking large language models on answering and explaining challenging medical questions,

    H. Chen, Z. Fang, Y . Singla, and M. Dredze, “Benchmarking large language models on answering and explaining challenging medical questions,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan...

    2025

  40. [48]

    MedXpertQA: Benchmarking expert-level medical reasoning and understanding,

    Y . Zuo, S. Qu, Y . Li, Z. Chen, X. Zhu, E. Hua, K. Zhang, N. Ding, and B. Zhou, “MedXpertQA: Benchmarking expert-level medical reasoning and understanding,” arXiv preprint arXiv:2501.18362 , 2025

    2025 arXiv

  41. [49]

    MedCalc- Bench: Evaluating large language models for medical calculations,

    N. Khandekar, Q. Jin, G. Xiong, S. Dunn, S. Applebaum, Z. Anwar, M. Sarfo-Gyamfi, C. Safranek, A. Anwar, A. Zhang et al. , “MedCalc- Bench: Evaluating large language models for medical calculations,” Ad- vances in Neural Information Processing Systems , vol. 37, pp. 84 730– 84...

    2024

  42. [50]

    Susceptibility of large language models to user-driven factors in medical queries,

    K. H. Lim, U. Kang, X. Li, J. S. Kim, Y .-C. Jung, S. Park, and B.-H. Kim, “Susceptibility of large language models to user-driven factors in medical queries,” arXiv preprint arXiv:2503.22746 , 2025

    2025 arXiv

  43. [51]

    Om- niMedVQA: A new large-scale comprehensive evaluation benchmark for medical LVLM,

    Y . Hu, T. Li, Q. Lu, W. Shao, J. He, Y . Qiao, and P. Luo, “Om- niMedVQA: A new large-scale comprehensive evaluation benchmark for medical LVLM,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 22 170–22 183

    2024

  44. [52]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  45. [53]

    KTO: Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “KTO: Model alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  46. [54]

    Reasoning models don’t always say what they think,

    Y . Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schul- man, A. Somani, P. Hase, M. Wagner, F. Rogeret al., “Reasoning models don’t always say what they think,” arXiv preprint arXiv:2505.05410 , 2025

    2025 arXiv

  47. [55]

    Disentangling reasoning and knowledge in medical large language models,

    R. Thapa, Q. Wu, K. Wu, H. Zhang, A. Zhang, E. Wu, H. Ye, S. Bedi, N. Aresh, J. Boen et al. , “Disentangling reasoning and knowledge in medical large language models,” arXiv preprint arXiv:2505.11462, 2025. 12 Supplementary Figure 1. Prompts for text-only medical benchmark dat...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.