Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Direct decoder-only answer SFT is the strongest reliable adaptation family on the MedFrameQA medical VQA benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:32 UTC pith:V2X2ZW6J

load-bearing objection Plausible but under-supported: the direct-SFT win over complex families rests on missing baseline tables and inconsistent numbers; needs major revision. the 4 major comments →

arxiv 2607.27566 v1 pith:V2X2ZW6J submitted 2026-07-30 cs.CV

Objective-Aligned Direct Answer SFT for Robust Multi-Frame Medical VQA

Site Li , Jianyi Hao , Xiaofeng Liu This is my paper
classification cs.CV
keywords multi-frame medical VQAobjective alignmentdirect answer SFTparameter-efficient fine-tuningLoRAMedFrameQAmodel calibrationseed variance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks which adaptation strategy actually holds up on the multi-frame medical VQA benchmark MedFrameQA once evaluation is controlled for fixed splits, matched compute budgets, repeated seeds, and calibration. It argues that the simplest strategy—directly fine-tuning a multimodal model's decoder to predict only the correct answer, with no controller heads, reranking, or auxiliary objectives—is the strongest stable family. The tuned answer-only model raises held-out report accuracy from 45.21% to 51.52% on MedGemma-1.5-4B, and matched variants with slightly different budgets or a shallow vision adapter are statistically inseparable. More elaborate families (hard-negative mixing, continuation training, controllers, scaffold evolution) either match the direct family with higher variance or fall well short. The paper also shows post-hoc calibration cuts expected calibration error from about 33% to near 1% without changing accuracy, and the same recipe transfers to a second backbone.

Core claim

The central claim is that objective alignment dominates architectural complexity on MedFrameQA. The benchmark scores only the final multiple-choice answer, so adaptation families that optimize exactly that answer loss should win once variance is penalized. Empirically, direct decoder-only answer SFT on MedGemma-1.5-4B achieves 51.52% report accuracy, a +6.31-point gain over the frozen baseline, with seed standard deviation 0.50; paired bootstrap shows nearby direct variants are not decisively separable. Answer continuation is competitive (51.48%, std 0.63), but hard-negative variants degrade stability (std up to 1.72) and static 2:1 hard-negative mixing nearly collapses to baseline (46.29%,

What carries the argument

The load-bearing object is the direct answer-only supervised fine-tuning (SFT) family: a decoder-only LoRA fine-tune on the last eight decoder blocks of MedGemma-1.5-4B, supervised solely by the negative log-likelihood of the correct answer given the multi-frame input. The comparison is governed by a robustness-aware criterion J(F) = E[R(θ̂_F)] + τ√Var(R(θ̂_F)), which explicitly penalizes seed variance, and a narrow proposition stating that any auxiliary objective must reduce global report risk enough to offset its extra bias and variance. The variants text30, text35, and vision35 are matched perturbations of the same family, differing only in training budget or a shallow vision adapter.

Load-bearing premise

The conclusion stands only if the complex baselines—controllers, scaffold evolution, rerankers, and hard-negative variants—were given genuinely matched tuning budgets and their reported numbers (notably the uniform 46.88% on a 256-example slice) are representative of their true performance.

What would settle it

Train a controller or scaffold-based variant with the same LoRA budget and hyperparameter search as text35, evaluate on the full 1,024-example report split across at least five seeds, and show it exceeds 51.52% report accuracy with a seed standard deviation at or below 0.50. A single such result would falsify the family-level claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On MedFrameQA, new adaptation methods should be compared against a direct answer-only SFT baseline under matched budget and repeated seeds; beating it requires lowering held-out report risk, not just improving slices.
  • Post-hoc calibration (temperature scaling or histogram binning) should be treated as a deployment repair, not a source of task accuracy; it cuts ECE from ~33% to ≤5% with no accuracy change.
  • The direct answer-only recipe transfers to Qwen2.5-VL-3B with a consistent +2.58-point average gain, suggesting the result is not backbone-specific.
  • High seed variance is disqualifying for a benchmark method even if a single checkpoint peaks; the static hard-negative mixing family's 6.86-point standard deviation is treated as evidence against it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same protocol were applied to other multi-frame VQA benchmarks, the direct answer SFT family might similarly dominate, because the benchmark objective is the answer and any auxiliary head pays an alignment tax. This is an extrapolation beyond MedFrameQA.
  • The paper's Proposition 1 is essentially a bias-variance argument; one testable extension is to measure the auxiliary-bias term B(θ) directly by comparing the answer loss of complex families on held-out data, which the paper does not report.
  • The controller/scaffold comparison rests on a single 256-example slice with a uniform 46.88% report; a fair full-split comparison with matched LoRA budgets and seed statistics would either confirm or overturn the family-level ranking. This is the natural next experiment.
  • The 6.31-point gain over frozen baseline is large relative to the ~1-point gaps among direct variants; if the benchmark's report split is small or noisy, the family-level claim may be more defensible than any single-variant ranking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a controlled, family-level comparison of adaptation methods for multi-frame medical VQA on the MedFrameQA benchmark. It argues that direct decoder-only answer SFT on MedGemma-1.5-4B is the strongest robust adaptation family, reporting a +6.31 point gain over the frozen baseline on the 1,024-example report split with a 0.50-point seed standard deviation, while more complex mechanisms (controllers, scaffold evolution, hard-negative continuation, static mixing) are claimed to be weaker or less robust. The paper also reports post-hoc calibration repairs ECE without hurting accuracy, and that the approach transfers to Qwen2.5-VL-3B. A formal objective-alignment framework is introduced in Proposition 1 to justify the family-level preference.

Significance. If the central claim holds, the paper provides a valuable counterpoint to the trend of increasingly complex architectural adaptation in medical VQA, and it demonstrates a rigorous evaluation protocol (fixed splits, repeated seeds, paired bootstrap, calibration, transfer). The direct answer-only family is well characterized internally, and the repeated-seed methodology is a real strength. However, the headline claim that direct SFT is the strongest robust family depends on a comparison with controller/scaffold baselines that is currently reported in a single sentence on a different evaluation slice, without budgets, seeds, or per-method statistics. The paper also contains an unexplained inconsistency in the reported accuracy of the 'final MedGemma run.' These gaps prevent the current evidence from fully supporting the strongest-family claim.

major comments (4)
  1. [§4, 'Controller and scaffold variants are weaker still'] The only evidence against controller/scaffold families is the sentence: 'on a common 256-example report slice, Full-Shinka, evolved rerankers, narrow-policy control, and random-search control all remain at 46.88% while incurring up to 4.06× runtime.' This is not a valid comparison with the rest of Table 1, which uses the 1,024-example report_test split. A 256-example slice has a binomial standard error of roughly 3 percentage points, so 46.88% is statistically indistinguishable from the direct family's ~51.5% on the larger split. Moreover, no optimization budgets, hyperparameter ranges, LoRA ranks, number of seeds, or per-method standard deviations are provided, making the claimed 'matched budgets' unverifiable. The family-level conclusion requires a full table on the same 1,024 split with per-seed results and an explicit budget description.
  2. [§4, 'Calibration and transfer' (Figure 2)] Table 1 reports the tuned text35 model at 51.52% report accuracy, but the text and Figure 2 state that the 'final MedGemma run' achieves 52.25%. This is an unexplained discrepancy. If 52.25% comes from a different run, seed, or evaluation subset, that must be stated; if it is a typo, it must be corrected. As written, the paper gives two conflicting accuracy numbers for what appears to be the same final model.
  3. [Table 2 and §4] The direct family is claimed to be 'the strongest robust family,' but the closest competitor, answer continuation, reaches 51.48% versus 51.52% for text35 — a 0.04-point difference. No paired bootstrap or variance comparison is reported between text35 and answer continuation on the 1,024 split, even though the robustness criterion J includes a variance penalty τ. Without such a comparison, the paper does not establish that the direct family is statistically stronger than answer continuation; the family-level claim is supported against the weaker hard-negative and static-mix families only. Also, the row labeled 'text35_vs_answer_old' in Table 2 is not defined in the text.
  4. [§3, Proposition 1] Proposition 1 assumes E[B(θ_complex)] ≥ 0 and Var(R(θ_complex)) ≥ Var(R(θ_direct)). These assumptions are effectively the empirical facts that the experiments must establish in order to prefer the direct family. The proposition therefore largely restates the intended conclusion and cannot serve as independent theoretical evidence. The empirical comparison is the load-bearing part; if the proposition is kept, it should be presented as a framing device, and the assumed inequalities should be explicitly tested or at least tied to the measured seed variances.
minor comments (5)
  1. [§3 / Table 3] The notation 'text30/text35/vision35' is admittedly unusual; the paper defines it clearly, but it would help to add a small table summarizing the LoRA rank, number of epochs, and vision adapter settings for each variant. Currently the LoRA rank is not stated, which limits reproducibility.
  2. [Table 2] The label 'answer_old' is cryptically undefined. If it refers to answer continuation or an earlier checkpoint, the table should say so explicitly.
  3. [§4] For the Qwen2.5-VL-3B transfer result, only the average accuracy and the +2.58-point improvement are given; per-seed results and the number of runs are mentioned ('five runs') but not tabulated. A small table or error bar would make this secondary claim more transparent.
  4. [§4 'Static answer+hardneg mix (2:1)'] The static mixing ratio is a free parameter. The paper reports only the 2:1 case; a sentence on whether other ratios were explored (and if not, why) would preempt concerns that the comparison is sensitive to this hyperparameter.
  5. [Figure 2] The calibration accuracy annotations (52.25 and 52.54 for the two backbones) appear inside the plot but are not described in the caption; please clarify which run/split these accuracies refer to, especially given the discrepancy with Table 1.

Circularity Check

0 steps flagged

No significant circularity: the central family-level claim is grounded in held-out MedFrameQA evaluations, not derived from Proposition 1 or from self-citations.

full rationale

The paper's central claim—that objective-aligned direct answer-only SFT is the strongest robust adaptation family—is established empirically on the held-out report_test split (1,024 examples) in Table 1, with repeated seeds, paired bootstrap, and explicit standard deviations. This is a self-contained experimental comparison, not a derivation from a fitted input or a renamed known result. The main theoretical statement, Proposition 1 in §3, is a conditional formalization: given explicit assumptions that complex families have non-negative auxiliary bias and no smaller risk variance, the direct family is preferred under the robustness-aware criterion J. Those assumptions are stated as premises, not fitted from data or derived from the conclusion; the proposition does not by itself assert that the direct family wins, and the paper does not use it as proof of the empirical result. The experiments are said to 'match Proposition 1 closely,' but the match is an interpretation of held-out measurements, not a reduction. The controller/scaffold baselines are reported only on a separate 256-example slice in §4 ('Controller and scaffold variants are weaker still: on a common 256-example report slice, Full-Shinka, evolved rerankers, narrow-policy control, and random-search control all remain at 46.88%'), which raises a comparability and reporting concern, not a circularity concern: the numbers are not fitted to predict the main result, and the issue belongs to evidence quality/correctness rather than to the paper's derivation chain. There are no load-bearing self-citations; references to MedFrameQA, MedGemma, Qwen, and baseline methods are external. Direct answer SFT optimizes the benchmark's answer loss (Eq. 2), but the empirical finding that this yields the best held-out accuracy is nontrivial—optimizing the same objective on training data does not by construction guarantee report-test performance. Overall, the paper's derivation chain is not circular; the main claim stands on independent held-out experiments.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claim depends on the fairness of the family-level comparison, the validity of report accuracy on the 1,024-example split as the criterion, and Proposition 1's assumptions about auxiliary bias and variance. No new entities are introduced; tau in J is a hand-chosen variance penalty.

free parameters (4)
  • Variance penalty tau = unspecified, tau > 0
    Introduced in §3 Eq (4) to define the robustness criterion J(F). The amount of variance penalty directly affects which family is 'most robust', so the main claim rests on this hand-chosen weighting.
  • Training budget (epochs) = 3.0 and 3.5
    The matched-budget comparison for the direct family uses 3.0 vs 3.5 epochs; the claim that nearby variants are statistically inseparable depends on these specific budgets.
  • Input frame count = 5
    The direct family uses five input frames; this is a design choice not swept in the paper, so family-level robustness under frame-count perturbation is untested.
  • Static mixing ratio = 2:1
    The static answer+hard-negative mix baseline uses a 2:1 ratio. This is a hyperparameter of a compared family, not the central method, but it affects the headline instability result.
axioms (3)
  • ad hoc to paper Complex families introduce non-negative expected auxiliary bias and no smaller risk variance (E[B]>=0, Var(R(complex))>=Var(R(direct))).
    Assumed in Proposition 1 (§3) and used to conclude that the direct family is preferred by J. It is not derived and effectively encodes the conclusion rather than proving it.
  • domain assumption The MedFrameQA report split is a valid and sufficient measure of multi-frame clinical reasoning.
    The whole comparison uses report accuracy on 1,024 held-out examples as the primary criterion. If this metric is noisy or not representative, the family ranking is ungrounded (§4).
  • domain assumption Compared families were matched in training budget and tuning effort.
    The family-level conclusion depends on fair comparison. The paper asserts matched budgets but does not show per-family compute, hyperparameter search budgets, or selection procedures for all families (§4 protocol paragraph).

pith-pipeline@v1.3.0-daily-deepseek · 8192 in / 13332 out tokens · 129427 ms · 2026-08-01T05:32:37.942129+00:00 · methodology

0 comments
read the original abstract

Multi-frame medical VQA appears to reward increasingly complex adaptation: controller-style inference, localization-aware reranking, static hard-negative mixing, and staged continuation all appear plausible from first principles. We test a simpler competing hypothesis on MedFrameQA: methods that remain tightly aligned with the benchmark's final answer objective should be the strongest \emph{robust} adaptation family once evaluation is controlled across fixed splits, matched budgets, repeated seeds, and calibration. We compare controller-based methods, scaffold evolution, static mixed supervision, continuation-heavy variants, and direct answer-only supervised fine-tuning (SFT). The strongest robust family is direct decoder-only answer SFT on MedGemma-1.5-4B. Empirically, this family yields substantial improvements in held-out report accuracy over frozen baselines while remaining remarkably stable across repeated seeds and matched controls, ensuring our claims reflect true family-level robustness rather than an isolated hyperparameter peak. Furthermore, post-hoc calibration effectively repairs confidence estimation without compromising accuracy, and the core approach transfers consistently to secondary backbones like Qwen2.5-VL-3B. The main result is therefore not that a complex auxiliary mechanism wins, but that objective-aligned direct answer SFT is the strongest robust adaptation family we found for MedFrameQA. By establishing this strong, minimalist baseline, we hope to redirect community focus toward fundamentally robust optimization rather than architectural complexity.

Figures

Figures reproduced from arXiv: 2607.27566 by Jianyi Hao, Site Li, Xiaofeng Liu.

Figure 1
Figure 1. Figure 1: Family-level comparison on MedGemma. Left: report [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Post-hoc calibration of the final MedGemma model and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 17 linked inside Pith

  1. [1]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering.Proceedings of the IEEE International Conference on Computer Vision, 2015. 2

    2015

  2. [2]

    Meng, and Bo Zhao

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q.-H. Meng, and Bo Zhao. M3D: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024. 2

    Pith/arXiv arXiv 2024

  3. [3]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, et al. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025. 2, 3

    Pith/arXiv arXiv 2025

  4. [4]

    Hasan, Viji Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller

    Asma Ben Abacha, Sadid A. Hasan, Viji Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. VQA-Med: Overview of the medical visual question answering task at ImageCLEF 2019. InCLEF Working Notes, 2019. 2

    2019

  5. [5]

    Hasan, Dina Demner- Fushman, and Henning Müller

    Asma Ben Abacha, Viji Datla, Sadid A. Hasan, Dina Demner- Fushman, and Henning Müller. Overview of the VQA-Med task at ImageCLEF 2021. InCLEF Working Notes, 2021

    2021

  6. [6]

    GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical AI.arXiv preprint arXiv:2408.03361, 2024

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical AI.arXiv preprint arXiv:2408.03361, 2024. 2

    Pith/arXiv arXiv 2024

  7. [7]

    QLoRA: Efficient finetuning of quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 2023. 2

    2023

  8. [8]

    Making the V in VQA matter: El- evating the role of image understanding in visual question answering

    Yash Goyal, Tanmay Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: El- evating the role of image understanding in visual question answering. InProceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2017. 2

    2017

  9. [9]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InInternational Conference on Machine Learning, 2017. 2

    2017

  10. [10]

    PathVQA: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. PathVQA: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020. 2

    Pith/arXiv arXiv 2003

  11. [11]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Ben Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mo- hammad Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InInternational Conference on Machine Learning, 2019. 2

    2019

  12. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.International Conference on Learning Representations, 2022. 2

    2022

  13. [13]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical LVLM

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical LVLM. arXiv preprint arXiv:2402.09181, 2024. 2

    Pith/arXiv arXiv 2024

  14. [14]

    Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C

    Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Mercy Ranjit, Anton Schwaighofer, Fer- nando Pérez-García, Valentina Salvatelli, et al. MAIRA-1: A specialised large multimodal model for radiology report generation.arXiv preprint arXiv:2311.13668, 2023. 2

    Pith/arXiv arXiv 2023

  15. [15]

    Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

    Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. InAdvances in Neural Information Processing Systems, 2019. 2

    2019

  16. [16]

    Med-R1: Reinforcement learning for general- izable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xi- aofeng Yang. Med-R1: Reinforcement learning for general- izable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025. 2

    arXiv 2025

  17. [17]

    Shinkaevolve: Towards open-ended and sample-efficient pro- gram evolution.arXiv preprint arXiv:2509.19349, 2025

    Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. Shinkaevolve: Towards open-ended and sample-efficient pro- gram evolution.arXiv preprint arXiv:2509.19349, 2025. 2

    Pith/arXiv arXiv 2025

  18. [18]

    Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific Data, 5:180251, 2018. 2

    2018

  19. [19]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021. 2

    Pith/arXiv arXiv 2021

  20. [20]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems Workshop, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems Workshop, 2023. 2

    2023

  21. [21]

    SLAKE: A semantically-labeled knowledge- enhanced dataset for medical visual question answering.IEEE International Symposium on Biomedical Imaging, 2021

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: A semantically-labeled knowledge- enhanced dataset for medical visual question answering.IEEE International Symposium on Biomedical Imaging, 2021. 2

    2021

  22. [22]

    MedVLM-R1: Incentivizing medical reasoning ca- pability of vision-language models (VLMs) via reinforcement learning.arXiv preprint arXiv:2502.19634, 2025

    Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. MedVLM-R1: Incentivizing medical reasoning ca- pability of vision-language models (VLMs) via reinforcement learning.arXiv preprint arXiv:2502.19634, 2025. 2

    Pith/arXiv arXiv 2025

  23. [23]

    Capabilities of gemini models in medicine.arXiv preprint arXiv:2404.18416, 2024

    Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine.arXiv preprint arXiv:2404.18416, 2024. 2

    Pith/arXiv arXiv 2024

  24. [24]

    Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, et al. Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Medgemma 1.5 technical report.arXiv preprint arXiv:2604.05081, 2026

    Andrew Sellergren, Chufan Gao, Fereshteh Mahvar, Timo Kohlberger, Fayaz Jamil, Madeleine Traverse, Alberto Tono, Bashir Sadjad, et al. Medgemma 1.5 technical report.arXiv preprint arXiv:2604.05081, 2026. 3

    Pith/arXiv arXiv 2026

  26. [26]

    Towards generalist biomedical ai.arXiv preprint arXiv:2307.14334, 2023

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, et al. Towards generalist biomedical ai.arXiv preprint arXiv:2307.14334, 2023. 2

    Pith/arXiv arXiv 2023

  27. [27]

    Medframeqa: A multi-image medical vqa benchmark for clinical reasoning

    Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, Yannan Yu, and Yuyin Zhou. Medframeqa: A multi-image medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505.16964, 2025. 1, 2, 3

    arXiv 2025

  28. [28]

    MMMU: A massive multi-discipline mul- timodal understanding and reasoning benchmark for expert AGI.arXiv preprint arXiv:2311.16502, 2023

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline mul- timodal understanding and reasoning benchmark for expert AGI.arXiv preprint arXiv:2311.16502, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2024

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark.arXiv preprint arXiv:2409.02813, 2024. 2

    Pith/arXiv arXiv 2024

  30. [30]

    Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers

    Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. InInternational Conference on Machine Learning,

  31. [31]

    Transforming classifier scores into accurate multiclass probability estimates

    Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. InACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002. 2

    2002

  32. [32]

    PMC-VQA: Visual in- struction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: Visual in- struction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023. 2

    Pith/arXiv arXiv 2023

  33. [33]

    MedXpertQA: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025. 2

    Pith/arXiv arXiv 2025