Pith. sign in

REVIEW 3 major objections 4 minor 69 references

MIRA: Medical Image Reflection for Agentic Diagnosis

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage training procedure that makes a medical vision-language model verify its tool use improves its average diagnostic score from 57.29 to 64.73 across nine benchmarks.

desk verdict MIRA is a genuinely new training recipe for medical VLMs—validation-gated reflection memory plus failure-driven SFT data—with an internally consistent +7.4-point gain, but the unspecified validation set for memory acceptance is the one thing a referee must pin down. read the letter →

arxiv 2608.10827 v1 pith:D6Y5GV5Y submitted 2026-08-11 cs.CV cs.AI

Shengzhi Wang , Jun Yang , Kai Wu , Xiaozhong Ji , Yiwen Ye , Ziyang Chen , Mingliang Xiong , Wen Fang
show 6 more authors
Mingqing Liu Mengyuan Xu Miaoxuan Shan Caiyan Liu Bin He Qingwen Liu
This is my paper · ORCID
classification cs.CVcs.AI
keywords medicalvisualquestionansweringagenticdiagnosistooluseverificationMonteCarloTreeSearchreinforcementlearningreflectionmemoryevidencegroundingmultimodalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MIRA claims that medical image diagnosis can be made more reliable by turning a static question-answering model into an agent that actively inspects images and checks whether each tool result supports its current diagnostic hypothesis. Its central quantitative assertion is that a two-stage pipeline—Monte Carlo Tree Search–generated tool trajectories for supervised fine-tuning, then reinforcement learning with a reflection memory—converts an 8B vision-language backbone into a stronger diagnostician, improving the average score from 57.29 to 64.73 across nine benchmarks. If that holds, selective and evidence-grounded tool use, not simply more tool calls, is what drives medical visual reasoning gains, and a compact open model can narrow the gap to much larger systems. The paper also reports useful tool use rising to 73.8% and harmful tool use falling to 1.6%, and its case studies show the agent re-examining evidence and correcting early conclusions.

What carries the argument

The load-bearing machinery is a two-stage training loop built on reflection. A tool-augmented Monte Carlo Tree Search data engine, offering six lightweight tools (zoom, grounding, point, rotate, measure, and web search), turns static VQA instances into verified evidence-seeking trajectories, with a unified verifier rendering spatial tool calls on the image and scoring each node for visual plausibility and semantic usefulness. Supervised fine-tuning on those trajectories plus failure-driven and intention-level reflection data provides a cold-start policy. GRPO reinforcement learning on fresh rollouts then applies a composite reward—format, answer correctness, and a consistency score paid only when the answer is correct—alongside a validation-gated reflection memory that distills repeated failures into candidate principles, injects them into the system prompt, and accepts them only when they improve mean rollout reward on a held-out validation set under a frozen policy. This memory is the distinctive mechanism: it makes failures reusable across cases while guarding against memorizing single outliers.

What would settle it

Check whether any accepted reflection-memory rule, or the validation set used in the memory-selection rule, contains examples drawn from the test splits of SLAKE, PMC-VQA, VQA-RAD, PathVQA, MedLesionVQA, MedLesionMCQ, OmniMedVQA, MedXpertQA-MM, or MMMU-Medical. Concretely, recompute the headline average after re-running evaluation with memory rules that were rejected by the validation gate; if rejected memories produce larger test gains than accepted ones, the selection signal is not predictive. A simpler observation is to inspect the released memory-update logs for dataset names or test-set identifiers—if any appear, the separation claim fails.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a medical visual agent learns to decide when to use tools, verify spatial claims, and revise premature conclusions through a carefully staged curriculum. MIRA-SFT first builds a cold-start policy from MCTS-searched trajectories, where search nodes are jointly scored by visual grounding plausibility and semantic consistency with the ground-truth answer, then adds failure-driven correction reflections and intention-level thoughts. MIRA-RL then optimizes this policy online with GRPO on fresh multi-turn rollouts, using a composite reward that gates a consistency bonus behind answer correctness, and a validation-gated reflection memory that converts repeated failures into reusable diagnostic principles before later rollouts. The paper's central quantitative claim is the +7.44 average improvement over the backbone, with the largest gains on lesion-level tasks (MedLesionVQA +14.5, MedLesionMCQ +14.4). Fine-grained analysis locates the gains in basic perception and understanding/diagnosis/suggestion rather than in content recognition, which is consistent with the intended mechanism of evidence-grounded verification rather than memorization.

Load-bearing premise

The whole comparison rests on the claim that the validation set used to accept reflection-memory rules never overlaps the nine test sets used for the final scores, and the paper supplies no dataset documentation to verify that separation.

Editorial extensions

If this is right

  • If the central claim is correct, training medical vision-language models on evidence-verifying tool trajectories rather than static QA data becomes a practical recipe for improving medical visual reasoning on a fixed backbone.
  • Verification-gated tool use becomes a measurable objective: useful tool use rises, harmful tool use falls, and completed reasoning shortens, suggesting that reinforcement learning can make agents more selective rather than more verbose.
  • Because the gains concentrate in basic perception and understanding/diagnosis rather than content recognition, tool-augmented reasoning should be prioritized for interpretation and decision tasks rather than for factual recall.
  • The validation-gated reflection-memory mechanism implies that a small, editable, textual memory can guide policy during RL, potentially reducing the number of rollouts needed to learn from repeated failures.
  • A compact 8B medical agent may approach or surpass much larger open and medical-specific systems, narrowing the scale gap at a fixed inference budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the reflection-memory claim would be to ablate the validation gate while keeping the same candidate edits: if rejected edits sometimes improve test reward, the gate's conservatism may be hiding real gains.
  • The paper treats the validation set used in the memory-acceptance rule as unspecified; absent documentation that it is disjoint from the nine evaluation test sets, the headline improvement should be read with the possibility of validation-guided selection in mind.
  • The framework suggests a general principle for agentic systems beyond medicine: any tool-using model can benefit from a separate 'should I use this tool, and does the output confirm my hypothesis?' verification stage followed by corrective loops.
  • A testable extension is to measure tool-use necessity on unseen disease types or imaging modalities, since the paper's reflection memory is distilled from on-policy failures and may generalize less far under distribution shift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MIRA, a two-stage medical visual diagnostic agent built on Qwen3-VL-8B. Stage one uses a tool-augmented Monte Carlo Tree Search engine to synthesize golden tool-use and reflection trajectories for SFT. Stage two applies GRPO with a composite reward (format, accuracy, consistency) and an online reflection memory that is updated only when a candidate memory improves rollout reward on a held-out validation set. The central empirical claim is that MIRA-VL-8B raises the average accuracy across nine medical VQA benchmarks from 57.29 to 64.73 (Table 1), with especially large gains on lesion-level benchmarks, while also increasing useful tool use from 56.2% to 73.8% and reducing harmful tool use from 8.9% to 1.6% (Section 4.4). The paper also provides ablations isolating SFT data components and RL rewards, fine-grained task analyses, training curves, and qualitative case studies.

Significance. If the headline evaluation result is unbiased, the contribution is significant for medical visual agent research: it demonstrates that a compact 8B model can approach or exceed much larger generalist and medical-specific systems by combining active tool use with explicit verification and memory-guided reflection. The paper has concrete strengths: the central numbers in Table 1 are internally consistent (I re-computed the averages and deltas), the two-stage training recipe is described in enough detail to be reproducible, and the authors state that trajectory data, tool environment, training recipes, and checkpoints will be released. The core improvement is measured on external public benchmarks, so the result is not circular by construction. However, the validity of the headline gain depends on whether the validation set used for reflection-memory selection is genuinely disjoint from the nine evaluation test splits; that condition is asserted but not documented, and one benchmark appears under two different names in the paper.

major comments (3)
  1. [§3.3, Eq. (14); §1; §2.2; Appendix E] The validation-gated reflection memory is a load-bearing selection step: Eq. (14) accepts a candidate memory only when S_t(˜m) > S_t(m) on a held-out validation set D_val, and the accepted memory is injected into the system prompt at inference (Section 3.3 and Appendix E). The paper never specifies how D_val is constructed, its size, or which data sources it draws from, and it does not provide documentation showing D_val is disjoint from the test splits of the nine benchmarks in Table 1. Sections 1 and 2.2 assert that test splits are never used for validation or checkpoint selection, but an assertion is not an auditable dataset statement. Because every memory acceptance is a form of model selection, any overlap between D_val and the evaluation splits would inflate the reported +7.44 average gain and the per-benchmark gains. The Table 4 ablation cannot resolve this issue because it evaluates the memory component on the same benchmarks. The authors should specify D_val's composition and size, release it or describe its construction from training-only sources, and verify disjointness from all nine test splits; otherwise the headline improvement is not independently verifiable.
  2. [§4.4, Figure 4, Table A2 vs. §4.2, Table 1] The tool-use necessity analysis labels two benchmarks as MedImgRU-QA and MedImgRU-MCQ, while Table 1 and Section 4.2 report the same two benchmarks as MedLesionVQA and MedLesionMCQ. The paper should clarify whether these are the same datasets with inconsistent naming or two different benchmark families. If they are the same, the names should be unified throughout the paper and the appendix; if they are different, the claim of evaluating 'all nine benchmarks' and the aggregated useful/harmful tool-use percentages in Section 4.4 need to be recomputed and restated. As written, the inconsistency undermines the claim that the tool-use necessity improvement is consistent across the same nine benchmarks used for the main accuracy result.
  3. [§3.2, Eq. (13); §3.3, Eq. (14); Appendix G] The consistency reward used in Eq. (13) is computed by a judge that receives the current reflection memory as the 'injected skill' and is asked to penalize trajectories that contradict that skill. Consequently, the validation score S_t(m) in Eq. (14) is not a pure measure of diagnostic improvement: it includes a self-referential component in which the memory being evaluated is also the standard against which trajectory consistency is judged. A candidate memory that is more prescriptive can mechanically raise the consistency reward even if it does not improve answer correctness, although the result-reward gating in Eq. (13) limits the impact on the final benchmark accuracies. The paper should either adjust the validation score to use a memory-independent measure of trajectory quality or explicitly acknowledge and quantify this coupling when presenting the memory acceptance criterion as a held-out reward improvement.
minor comments (4)
  1. [Appendix G] The evaluation section states that all benchmarks are evaluated with the 'MedEvalKit testing suite [46]', but reference [46] is the Lingshu paper, not a MedEvalKit citation; please provide the correct reference for MedEvalKit or cite the appropriate software/benchmark resource.
  2. [Table 1; Table 3; Table 4; §4.2] The benchmark names MedLesionVQA and MedLesionMCQ appear with inconsistent capitalization (e.g., 'Medlesion VQA', 'Medlesion MCQ') across tables and prose; unify the spelling to match Section 4.2.
  3. [Table 4] The row label 'MIRA-RL w/o Tools' refers to evaluation with tool execution disabled, while the model was trained with tools; consider renaming the row to 'MIRA-RL, tools disabled at inference' to avoid ambiguity about training versus inference-time tool use.
  4. [§4.4 and Figure 5] The RL training curves report accuracy reward and consistency reward values but do not state whether these curves are computed on a held-out validation set or on the on-policy rollout batches; please specify the data source for Figure 5 so readers can interpret the curves as training diagnostics rather than evaluation results.

Circularity Check

0 steps flagged · score 0.0 of 10

No constructed circularity: the headline gain is an external-benchmark comparison; D_val overlap and self-authored MedLesion benchmarks are audit/validity risks, not definitional reductions.

full rationale

The central claim—MIRA-VL-8B improving the nine-benchmark average from 57.29 to 64.73 (Section 4.3, Table 1)—is an empirical comparison against public or external benchmarks and the Qwen3-VL-8B backbone, not a quantity defined in terms of the training objective. The training signal is standard: MCTS trajectories are filtered by correctness judged against ground-truth answers (Appendix C, Algorithm 1), reflection data is answer-conditioned (Eqs. 4–7), and RL rewards combine result, format, and consistency terms (Eq. 13). These fit the model to labeled training data; none of them by construction determines the held-out test scores. The reflection-memory gate (Eq. 14) is the only step that could in principle select on evaluation data, but the paper asserts that held-out test splits are never used for validation, hyperparameter selection, or checkpoint selection (Sections 1 and 2.2), and no evidence shows D_val overlaps the test splits; this is an auditability/leakage risk, not a demonstrated circular reduction. Two benchmarks, MedLesionVQA and MedLesionMCQ, come from a paper with overlapping authors (ref [48]: Deli Yu, Shengzhi Wang, Xiaozhong Ji, et al.) and show the largest per-benchmark gains; however, the remaining seven external benchmarks still improve, so the central result does not reduce to a self-authored benchmark. Table 4 also shows MIRA-RL w/o Tools gains +4.1 average without tool execution, further separating the method from the tool-use evaluation. No uniqueness theorem, ansatz-via-self-citation, or renamed-known-result step is load-bearing. The derivation is therefore self-contained in the sense required for a circularity finding, with the noted validity caveats falling under correctness/auditability rather than circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the training data construction, the MCTS and RL hyperparameters, the judge models, and the assumption that evaluation benchmarks are not contaminated. The reflection memory is a tuned textual artifact with no external evidence beyond the paper's own ablations.

free parameters (5)
  • MCTS exploration constant cpuct = 1.4
    Hand-chosen in Section 2.2 (Eq. 1); controls exploration-exploitation balance in trajectory search and thus the quality of the SFT data.
  • MCTS search budget = 20 simulations, depth 4, branching 3
    Hand-chosen search configuration (Appendix C); determines the diversity and correctness of synthesized trajectories.
  • RL reward coefficients lambda_c, lambda_f = 0.5, 0.5
    Fixed in Eq. 13 (Section 3.2) across all experiments; balance consistency and format rewards against accuracy.
  • Failure buffer threshold delta and minimum unique cases Nmin = 0.3 and 4
    Hand-chosen in Section 3.3 (Eq. 19) to decide when a failure triggers a memory update; affects which principles enter the reflection memory.
  • Reflection memory content (initial prompt and accepted edits) = Text prompt in Appendix E, evolving by validation-gated acceptance
    The default memory and all accepted rule edits are selected by the authors or by the validation-reward gate (Eq. 14). This is part of the final inference-time system prompt and is the most load-bearing tuned artifact.
assumptions (5)
  • domain assumption Test splits of the evaluation benchmarks are disjoint from all training, validation, and memory-selection data
    Section 1 and Section 2.2 assert this, and the RL stage's validation-gated memory acceptance (Section 3.3, Eq. 14) requires a D_val whose composition is never specified. The claim's validity depends on this separation.
  • domain assumption The in-house Internet-sourced dataset does not overlap with evaluation test images
    Section 2.2 mentions an 'in-house high-quality medical image dataset collected from Internet sources' but provides no overlap check against the nine evaluation benchmarks.
  • domain assumption The MCTS verifier and terminal judge, given access to ground-truth answers, can reliably identify tool calls that efficiently lead to the correct answer
    Appendix C states the verifier scores nodes 'with access to the ground-truth answer' and prunes branches; trajectory quality and the SFT signal depend on this verification being sound.
  • domain assumption The LLM judges used for answer grading and tool-use necessity agree with human expert judgments
    Appendix G describes the grading pipeline and Section 4.4 the necessity analysis; no human agreement study is reported, so the measured accuracies and tool-use rates depend on these judge models.
  • standard math Qwen3-VL-8B and the tool execution environment behave as documented
    The backbone's perception and the rendering, cropping, and measurement operations are taken as given; no verification is provided that tool outputs preserve all diagnostically relevant information.
invented entities (1)
  • Reflection memory
    purpose: A textual set of diagnostic principles injected into the system prompt during RL rollouts and at inference; it is updated by validation-gated edits.
    The memory is an internal artifact of the proposed pipeline. Its only support is the authors' own ablations (Table 4 and Appendix E); no external falsifiable handle or formal analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIRA: Medical Image Reflection for Agentic Diagnosis." pith.science (2026). https://pith.science/paper/D6Y5GV5Y

@misc{pith2026260810827,
  author       = {Pith},
  title        = {Pith review of: MIRA: Medical Image Reflection for Agentic Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6Y5GV5Y}},
  note         = {Machine review of arXiv:2608.10827}
}
read the original abstract

Medical visual agents can use tools to inspect images and retrieve external knowledge, but indiscriminate tool use may introduce noisy or misleading evidence. Reliable diagnosis therefore requires not only acquiring additional observations, but also verifying whether tool actions are necessary and whether the resulting evidence supports the current hypothesis. We introduce MIRA (Medical Image Reflection for Agentic Diagnosis), a medical visual diagnostic framework for autonomous evidence search and reflective verification. MIRA dynamically invokes image-processing operations, including zooming, grounding, pointing, rotation, and measurement, as well as web search, while evaluating the relevance and consistency of the acquired evidence. We develop MIRA through a two-stage training strategy. First, a tool-augmented Monte Carlo Tree Search data engine explores diverse diagnostic hypotheses and jointly verifies visual grounding accuracy and semantic consistency to construct supervised fine-tuning trajectories. Second, reinforcement learning further improves decision-making through online reflective principle evolution: failure cases are distilled into candidate principles, and only principles that improve held-out rollout rewards are retained. Across nine medical visual reasoning benchmarks, MIRA achieves an average score of 64.73, improving its Qwen3-VL-8B backbone by 7.44 points. It also increases useful tool-use judgments from 56.2% to 73.8% and reduces harmful judgments from 8.9% to 1.6%. Qualitative analyses show that MIRA can re-examine evidence, correct premature conclusions, and adapt its tool-use strategy. Project page: https://MIRA-VL.github.io/

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 31 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    arXiv 2025

  3. [3]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024. URLhttps://arxiv.org/abs/2406.19280

    arXiv 2024

  4. [4]

    Meissa: Multi-modal medical agentic intelligence

    Yixiong Chen, Xinyi Bai, Yue Pan, Zongwei Zhou, and Alan Yuille. Meissa: Multi-modal medical agentic intelligence. arXiv preprint arXiv:2603.09018, 2026

    arXiv 2026

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  6. [6]

    Thinking with generated images

    Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with generated images. arXiv preprint arXiv:2505.22525, 2025

    arXiv 2025

  7. [7]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    2023

  8. [8]

    Medmo: Grounding and understanding multimodal large language model for medical images.arXiv preprint arXiv:2602.06965, 2026

    Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, and Imran Razzak. Medmo: Grounding and understanding multimodal large language model for medical images.arXiv preprint arXiv:2602.06965, 2026

    arXiv 2026

Show all 69 references
  1. [9]

    Gemini 2.5 Pro.https://deepmind.google/models/gemini/pro/, 2025

    Google. Gemini 2.5 Pro.https://deepmind.google/models/gemini/pro/, 2025. Accessed: 2026-07-14

    2025

  2. [10]

    Medvh: Toward systematic evaluation of hallucination for large vision language models in the medical context.Advanced Intelligent Systems, 8(1): 2500255, 2026

    Zishan Gu, Jiayuan Chen, Fenglin Liu, Changchang Yin, and Ping Zhang. Medvh: Toward systematic evaluation of hallucination for large vision language models in the medical context.Advanced Intelligent Systems, 8(1): 2500255, 2026

    2026

  3. [11]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  4. [12]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  5. [13]

    Deepeyesv2: Toward agentic multimodal model

    Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, and Xing Yu. Deepeyesv2: Toward agentic multimodal model. arXiv preprint arXiv:2511.05271, 2025

    2025 arXiv

  6. [14]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  7. [15]

    Incentivizing tool-augmented thinking with images for medical image analysis.arXiv preprint arXiv:2512.14157, 2025

    Yankai Jiang, Yujie Zhang, Peng Zhang, Yichen Li, Jintai Chen, Xiaoming Shi, and Shihui Zhen. Incentivizing tool-augmented thinking with images for medical image analysis.arXiv preprint arXiv:2512.14157, 2025

    2025 arXiv

  8. [16]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. InEuropean conference on machine learning, pages 282–293. Springer, 2006

    2006

  9. [17]

    Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025

    Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025

    2025 arXiv

  10. [18]

    Med- r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, Yuheng Li, Konstantinos Psounis, and Xiaofeng Yang. Med- r1: Reinforcement learning for generalizable medical reasoning in vision-language models, 2025. URLhttps: //arxiv.org/abs/2503.13939

    2025

  11. [19]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018. 15

    2018

  12. [20]

    Latent visual reasoning.arXiv preprint arXiv:2509.24251, 2025

    Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, and Zicheng Liu. Latent visual reasoning.arXiv preprint arXiv:2509.24251, 2025

    2025 arXiv

  13. [21]

    Llava-med: Training a large language-and-vision assistant for biomedicine.arXiv preprint arXiv:2306.00890, 2023

    Chunyuan Li et al. Llava-med: Training a large language-and-vision assistant for biomedicine.arXiv preprint arXiv:2306.00890, 2023

    2023 arXiv

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  15. [23]

    think with videos

    Wenjie Li, Yujie Zhang, Haoran Sun, Xingqi He, Hongcheng Gao, Chenglong Ma, Ming Hu, Guankun Wang, Shiyi Yao, Renhao Yang, et al. Medscope: Incentivizing" think with videos" for clinical reasoning via coarse-to-fine tool calling. In Forty-thirdInternational Conference on Machi...

    2026

  16. [24]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In2021IEEE18thinternationalsymposium onbiomedical imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  17. [25]

    Pmc-vqa: Visual question answering for medical images

    Bo Liu et al. Pmc-vqa: Visual question answering for medical images. InProceedings of the ACM International Conference on Multimedia, 2021

    2021

  18. [26]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  19. [27]

    Medsam-agent: Empowering interactive medical image segmentation with multi-turn agentic reinforcement learning

    Shengyuan Liu, Liuxin Bao, Qi Yang, Wanting Geng, Boyun Zheng, Chenxin Li, Wenting Chen, Houwen Peng, and Yixuan Yuan. Medsam-agent: Empowering interactive medical image segmentation with multi-turn agentic reinforcement learning. arXiv preprint arXiv:2602.03320, 2026

    2026

  20. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  21. [29]

    Medvistagym: A scalable training environment for thinking with medical images via tool-integrated reinforcement learning.arXiv preprint arXiv:2601.07107, 2026

    Meng Lu, Yuxing Lu, Yuchen Zhuang, Megan Mullins, Yang Xie, Guanghua Xiao, Charles Fleming, Wenqi Shi, and Xuan Wang. Medvistagym: A scalable training environment for thinking with medical images via tool-integrated reinforcement learning.arXiv preprint arXiv:2601.07107, 2026

    2026

  22. [30]

    A multimodal generative ai copilot for human pathology.Nature, 634(8033):466–473, 2024

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology.Nature, 634(8033):466–473, 2024

    2024

  23. [31]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Ed- uardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. InMachine Learning for Health (ML4H), pages 353–367. PMLR, 2023

    2023

  24. [32]

    Medix-r1: Open ended medical reinforcement learning.arXiv preprint arXiv:2602.23363, 2026

    Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed, Mohamed Zidan, Fahad Khan, Salman Khan, Rao Anwer, and Hisham Cholakkal. Medix-r1: Open ended medical reinforcement learning.arXiv preprint arXiv:2602.23363, 2026

    2026

  25. [33]

    Hello gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024

    OpenAI. Hello gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024. Accessed: 2026-07-27

    2024

  26. [34]

    Introducing gpt-5.2

    OpenAI. Introducing gpt-5.2. https://openai.com/index/introducing-gpt-5-2/, December 2025. Accessed: 2026-07-27

    2025

  27. [35]

    Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning

    Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. InInternational Conference on Medical Ima...

    2025

  28. [36]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations, volume 2025, pages 28...

    2025

  29. [37]

    Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Pal...

    2024 arXiv

  30. [38]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  31. [39]

    Medxiaohe: A comprehensive recipe for building medical mllms

    Baorong Shi, Bo Cui, Boyuan Jiang, Deli Yu, Fang Qian, Haihua Yang, Huichao Wang, Jiale Chen, Jianfei Pan, Jieqiong Cao, et al. Medxiaohe: A comprehensive recipe for building medical mllms. arXiv preprint arXiv:2602.12705, 2026

    2026 arXiv

  32. [40]

    Large language models encode clinical knowledge.Nature, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, et al. Large language models encode clinical knowledge.Nature, 2023

    2023

  33. [41]

    Openthinkimg: Learning to think with images via visual tool reinforcement learning

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617, 2025

    2025 arXiv

  34. [42]

    Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025

    Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025

    2025 arXiv

  35. [43]

    Beyond textual rationales: Anatomy-grounded chain-of-thought for traceable radiology reasoning

    Shengzhi Wang, Kai Wu, Jun Yang, Mengyuan Xu, Mingliang Xiong, Wen Fang, Mingqing Liu, Hao Deng, Bin He, Gang Li, et al. Beyond textual rationales: Anatomy-grounded chain-of-thought for traceable radiology reasoning. Knowledge-Based Systems, page 116475, 2026

    2026

  36. [44]

    Vtool-r1: Vlms learn to think with images via reinforcement learning on multimodal tool use

    Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, and Klara Nahrstedt. Vtool-r1: Vlms learn to think with images via reinforcement learning on multimodal tool use. arXiv preprint arXiv:2505.19255, 2025

    2025

  37. [45]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024

    2024

  38. [46]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    2025 arXiv

  39. [47]

    Medreasoner: Reinforcement learning drives reasoning grounding from clinical thought to pixel-level precision

    Zhonghao Yan, Muxi Diao, Yuxuan Yang, Ruoyan Jing, Jiayuan Xu, Kaizhou Zhang, Lele Yang, Yanxi Liu, Kongming Liang, and Zhanyu Ma. Medreasoner: Reinforcement learning drives reasoning grounding from clinical thought to pixel-level precision. InProceedings of the AAAI Conferenc...

    2026

  40. [48]

    Medlesionvqa: A multimodal benchmark emulating clinical visual diagnosis for body surface health

    Deli Yu, Shengzhi Wang, Xiaozhong Ji, Bo Cui, Jieqiong Cao, Huichao Wang, Boyuan Jiang, Xu Wang, Qian Xu, Yi Zhao, et al. Medlesionvqa: A multimodal benchmark emulating clinical visual diagnosis for body surface health. In The FourteenthInternational Conference on Learning Rep...

    2026

  41. [49]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark

    Xiang Yue et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  42. [50]

    Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025

    Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025

    2025 arXiv

  43. [51]

    thinking with images

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025

    2025 arXiv

  44. [52]

    Medical sam 2: Segment medical images as video via segment anything model 2.arXiv preprint arXiv:2408.00874, 2024

    Jiayuan Zhu, Abdullah Hamdi, Yunli Qi, Yueming Jin, and Junde Wu. Medical sam 2: Segment medical images as video via segment anything model 2.arXiv preprint arXiv:2408.00874, 2024

    2024 arXiv

  45. [53]

    name": "SEARCH

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025. URLhttps://arxiv.org/abs/2501.18362. 17 Appendix Append...

    2025 arXiv

  46. [54]

    impossible

    Selection 2. Expansion 3. Simulation 4. Backpropagation Figure A4Overview of the tool-augmented MCTS procedure. The search iteratively expands candidate diagnostic actions, verifies tool-use validity and reasoning consistency, and backpropagates node rewards to guide subsequen...

  47. [55]

    If more than one, they must represent genuinely different directions

    Exact count: Output exactly {spark_count} thought sparks, no more, no fewer. If more than one, they must represent genuinely different directions

  48. [56]

    * If the context suggests a highly relevant path, at least one thought spark should lead directly toward forming a concise conclusion

    Context-aware diversity: * If the context suggests a dead end, your thought sparks must pivot to a completely different observation angle. * If the context suggests a highly relevant path, at least one thought spark should lead directly toward forming a concise conclusion

  49. [57]

    Natural language: Use fluent statements or questions, like a clinician's genuine inner monologue

  50. [58]

    {failed_thought}

    Strict prohibitions: A8 * Do not use imperative or directive phrasing. * Do not output tags like ['<{'}'> or <spark>. * Do not mention scores, evaluations, or external feedback. # Output Format Output a JSON list of {spark_count} strings. Each string must be a thought spark in...

  51. [61]

    Options: {choices} Response: {response} A18 Output: The extracted option string is uppercased and compared deterministically with the ground-truth option set

    You should output one or more uppercase option letters, e.g., ABCDEFGHIJ or +. Options: {choices} Response: {response} A18 Output: The extracted option string is uppercased and compared deterministically with the ground-truth option set. The symbol+is treated as incorrect. For...

  52. [65]

    Especially for attributes such as size, length, color, area, and extent, approximate but reasonable descriptions should be accepted, and exact matching is not required. The question is: {question} The standard answer: {answer} The user's answer: {response} Please strictly foll...

  53. [66]

    The final answer should be supported by the final observation/reasoning rather than ignoring it

    Final trajectory consistency: the last action, last observation, reasoning process, and final answer should not contradict each other. The final answer should be supported by the final observation/reasoning rather than ignoring it

  54. [67]

    Judge whether the model's actual action logic and reasoning path are consistent with that skill logic

    Skill-instruction/action-logic consistency: treat the injected skill A19 as a system-prompt instruction that describes how the model should think, choose tools/actions, inspect evidence, eliminate options, and decide the answer. Judge whether the model's actual action logic an...

  55. [68]

    First extract the core conclusion from the model response, and judge only the final answer

  56. [69]

    Ignore verbose reasoning, explanations, suggestions, <think> content, and extra modifiers in parentheses, as long as they do not contradict the core conclusion

  57. [70]

    If the prediction is close to the ground truth in meaning, or is a visually and medically reasonable approximation, it should be judged as correct

    Do not rely on the ground-truth answer too strictly or mechanically. If the prediction is close to the ground truth in meaning, or is a visually and medically reasonable approximation, it should be judged as correct

  58. [71]

    Especially for attributes such as size, length, color, area, and extent, approximate but reasonable descriptions should be accepted, and exact matching is not required. A20 The question is: {question} The standard answer: {answer} The user's answer: {response} Please strictly ...

  59. [72]

    Only output the option letter(s), nothing else

  60. [73]

    If all options are significantly different from the response, output the + symbol

  61. [74]

    erythematous finger joint region

    You should output one or more uppercase option letters, e.g., ABCDEFGHIJ or +. Options: {options} Response: {response} Output: Evaluation Prompts.In each template below, {question} is filled with the actual sample’s question, {options} is replaced with sample’s multiple-choice...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.