Pith. sign in

REVIEW 3 major objections 7 minor 44 references

Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a two-stage SFT-plus-RFT recipe turns a 3B multimodal LLM into the first reasoning model for Surgical-VQLA, beating state-of-the-art baselines on EndoVis-18 and EndoVis-17.

desk verdict A genuinely useful dataset and a sensible coherence reward, but the EndoVis-17 'external validation' claim is compromised by an unstated possible train/test leak that the authors must fix. read the letter →

arxiv 2506.19469 v1 pith:IEU5YPWX submitted 2025-06-24 cs.CV cs.AI

Pengfei Hao , Shuaibo Li , Hongqiu Wang , Zhizhuo Kou , Junhang Zhang , Guang Yang , Lei Zhu This is my paper
classification cs.CVcs.AI
keywords Surgical-VQLAreasoningmultimodalLLMreinforcementfine-tuningGRPOchain-of-thoughtvisualgroundingcoherencerewardsurgicalsceneunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that explicit reasoning and interpretability can be added to visual question localized-answering in robotic surgery. It does so by introducing the Surgery-R1-54k dataset, which pairs surgical images with chain-of-thought traces, visual QA pairs, and grounding QA pairs, and by training a 3B multimodal LLM in two stages: supervised fine-tuning followed by reinforcement fine-tuning with GRPO. The central wager is that a rule-based reward system, including a new Multimodal Coherence reward, will push the model to answer questions and localize instruments correctly while reasoning about them in coherent spatial language. If true, surgical scene-understanding systems could explain their own answers instead of returning black-box predictions, which matters for surgeon trust and for tracing the source of errors. The paper reports state-of-the-art accuracy, F-score, and mIoU on EndoVis-18 and improved accuracy and F-score over prior models on EndoVis-17.

What carries the argument

The load-bearing mechanism is the two-stage training recipe wrapped around a 3B Qwen2.5-VL backbone. Stage one is supervised fine-tuning on the Surgery-R1-54k dataset, whose chain-of-thought traces decompose each Surgical-VQLA question into planning, principle, visual analysis, comparison, contact analysis, and conclusion. Stage two is reinforcement fine-tuning with GRPO, where a rule-based reward system replaces human preference labels. Three rewards carry the optimization: the Linguistic Answer reward checks exact answer match; the Visual Grounding reward gives the IoU between predicted and ground-truth boxes above a threshold; and the Multimodal Coherence reward — the paper's new mechanism — divides the image into four quadrants and pays 1 only when the quadrant of the predicted box's center matches the locative term used in the reasoning text. That last reward is what specifically targets the positional hallucinations the authors observed after SFT.

What would settle it

Inspect the released Surgery-R1-54k dataset and its SFT/RFT split: if any image, question, chain-of-thought trace, or grounding pair originating from EndoVis-17-VQLA appears in training, the EndoVis-17 accuracy of 0.5672 measures memorization rather than generalization.

Watch

Extended reading notes

Core claim

The paper's central claim is that a general-purpose multimodal LLM can be turned into the first Reasoning MLLM for Surgical-VQLA by combining chain-of-thought supervision with reinforcement learning. On top of the Qwen2.5-VL-3B base, Surgery-R1 is first SFT-trained on surgeon-reviewed CoT, Visual-QA, and Grounding-QA data, then RFT-trained with GRPO and three rule-based rewards: Linguistic Answer, Visual Grounding (IoU), and the new Multimodal Coherence reward. On EndoVis-18 it reports accuracy 0.7356, F-score 0.4576, and mIoU 0.8721, all above the previous state of the art EndoChat; on EndoVis-17 it reports accuracy 0.5672, F-score 0.4422, and mIoU 0.8422, beating EndoChat on accuracy and F-score. The authors take these results to show that reasoning ability improves both answer accuracy and spatial grounding in surgical scenes, and that the Multimodal Coherence reward specifically suppresses the model's habit of describing the wrong quadrant while placing the box correctly.

Load-bearing premise

The EndoVis-17 results are presented as external-validation gains, which only holds if the Surgery-R1-54k training data excludes every sample derived from EndoVis-17-VQLA; the paper does not state that it does.

Editorial extensions

If this is right

  • Surgical-VQLA becomes a reasoning task: answer accuracy, grounding quality, and the coherence between text and box can all be optimized together with rule-based rewards, so models can be improved without human preference labels.
  • The same SFT-then-GRPO recipe should transfer to other surgical multimodal tasks that have ground-truth answers and boxes, such as instrument-state classification, tissue-interaction detection, and surgical report generation.
  • A 3B-parameter backbone is enough to reach state-of-the-art Surgical-VQLA performance, lowering the compute barrier for deploying reasoning models in surgical settings.
  • If the EndoVis-17 results are genuine external-validation results, the recipe generalizes across challenge years and unseen surgical scenes — a basic precondition for clinical trust.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the Multimodal Coherence reward measures only coarse quadrant agreement; replacing the four-quadrant grid with finer spatial cells or with directional expressions would likely sharpen the anti-hallucination effect and could be tested without changing the training pipeline.
  • Because the ablation shows SFT alone delivers most of the improvement, a natural extension is to scale only the RFT stage — more groups per question, more epochs, or harder question mixes — to see whether reasoning gains continue to accumulate as they do for language-only RL.
  • The dataset-generation path (MLLM API sub-answers plus surgeon review) is reusable for other surgical procedures and could be extended from single frames to video clips, where temporally grounded questions about tool motion and tissue change are still open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Surgery-R1, claimed to be the first Reasoning Multimodal Large Language Model for Surgical-VQLA. The authors construct a 54k-sample dataset (Surgery-R1-54k) containing Chain-of-Thought, Visual-QA, and Grounding-QA data derived from EndoVis-18-VQLA and EndoVis-17-VQLA, and train a Qwen2.5-VL-3B base model in two stages: supervised fine-tuning followed by reinforcement fine-tuning with GRPO. The reward system combines a linguistic answer reward, a visual grounding reward based on IoU, and a multimodal coherence reward designed to reduce positional hallucination. Experiments on EndoVis-18 and EndoVis-17 report higher accuracy and F-score than existing Surgical-VQLA models and MLLM baselines, with ablations supporting the contributions of SFT, RFT, and CoT.

Significance. If the results hold, the paper makes a useful contribution by demonstrating that a two-stage SFT+RFT recipe with rule-based rewards can be transferred from general reasoning MLLMs to a surgical scene-understanding task. The Surgery-R1-54k dataset, with surgeon-reviewed CoTs, is a concrete resource for the community, and the specific design of a coherence reward to align spatial terms with bounding boxes is a sensible and potentially reusable idea. The paper also includes quantitative ablations that are internally consistent in the tables. However, the central generalization claim is currently at risk because the paper does not establish that EndoVis-17-derived data were excluded from the SFT/RFT training splits, and several technical details of the reward system are underspecified. These issues must be resolved before the reported external-validation numbers can be trusted.

major comments (3)
  1. [IV.A.1, III.B, IV.A.3] The paper's own wording leaves a possible direct data leak in the EndoVis-17 evaluation. Section IV.A.1 states that Surgery-R1-54k 'is an extension of the EndoVis-18-VQLA and EndoVis-17-VQLA datasets,' and Section III.B says CoT, Visual-QA, and Grounding-QA are generated for 'each category of questions in the EndoVis-18-VQLA and EndoVis-17-VQLA datasets.' Section IV.A.1 then says 80% of these data go to SFT and 20% of COTs go to RFT, with no statement that EndoVis-17-derived samples were excluded. If any EndoVis-17-derived samples enter SFT or RFT, the EndoVis-17 results in Tables I-III measure memorization rather than generalization, which directly undermines the claim in IV.A.3 that EndoVis-17-VQLA is 'an external validation dataset to test our model's generalization on unseen data.' The authors must either explicitly confirm that all EndoVis-17-derived samples were held out from both training stages, or re-run the experiments under a clean split and report the corrected numbers.
  2. [III.D.2, Eq. (4), IV.C] The Visual Grounding reward in Eq. (4) depends on the IoU threshold τ, but τ is never specified anywhere in the manuscript, including the Implementation Details section. This is a core hyperparameter of the RFT stage: if τ is set too high, the reward is mostly zero; if too low, the reward loses discriminative power. Without reporting τ, the reported RFT gains are not reproducible and the sensitivity of the results to this threshold is unknown. Please provide the value used and, ideally, a short sensitivity analysis.
  3. [III.D.2, Eq. (7)] The Multimodal Coherence reward in Eq. (7) uses Q_inferred, described as 'the inferred location during reasoning,' but the paper never defines how Q_inferred is extracted from the model's free-form output. It is unclear whether the authors parse the CoT text for a spatial term, require a fixed format such as <answer>left-bottom</answer>, or use a heuristic based on the final answer. Since this reward directly feeds the GRPO advantage computation, the implementation must be specified precisely for the method to be reproducible and for the reported gains to be interpretable.
minor comments (7)
  1. [IV.E.3] The text says 'the M2 model underwent SFT on the Surgery-R1 dataset without CoT data,' which contradicts the definitions in IV.D and the comparison in IV.E.5, where M2 is the model trained with CoT. Please correct this label so the ablation story is internally consistent.
  2. [IV.E.3] The reported 'improvements of 63.05% in accuracy, 29.75% in F-score, and 43.82% in mIoU' are absolute percentage-point differences, not relative improvements. Please state this explicitly to avoid misleading readers.
  3. [IV.C] Several training hyperparameters are missing: the GRPO KL coefficient β from Eq. (2), the RFT learning rate, the number of RFT optimization steps or epochs, and the LoRA rank/alpha. These are needed to reproduce the experiments.
  4. [IV.A.1 and Fig. 1] The number of Visual-QA pairs is given as 33,342 in Section IV.A.1 but as 33,324 in the Fig. 1 caption. Please reconcile these numbers.
  5. [Table III and Table II] There are typographical issues in the tables: 'EndosVis-17' in Table III and 'LLaV A-1.5' in Table II should be 'EndoVis-17' and 'LLaVA-1.5', respectively.
  6. [IV.A.3] The sentence 'It concludes 97 frames' should be 'It contains 97 frames'.
  7. [Eqs. (1)-(3)] The notation in Eq. (2) would benefit from a brief explanation that this is the standard KL estimator used in GRPO, and from a definition of ϵ, which appears in the text but not in the equations.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; minor unresolved risk that EndoVis-17 training data overlaps the 'external' test set.

full rationale

The derivation chain is not circular: Surgery-R1-54k is constructed from EndoVis-18/17-VQLA questions; SFT then RFT with GRPO uses rule-based rewards (LA, VG, MC, Eqs. 4-7) that compare model outputs to ground-truth labels and are not fitted to the test split. Reported Acc/F-score/mIoU are external metrics, not re-expressions of the rewards. Self-citations such as EnVR-LPKG [7] serve as comparison baselines or related work, not as load-bearing justification for the main claim. The only substantive concern is data hygiene: Section IV.A.1 says 'The Surgery-R1-54k dataset is an extension of the EndoVis-18-VQLA and EndoVis-17-VQLA datasets' and the 80/20 SFT/RFT split is described globally over COTs/Visual-QA/Grounding-QA, while Section IV.A.3 calls EndoVis-17-VQLA 'an external validation dataset to test our model's generalization on unseen data.' The paper never explicitly states that EndoVis-17-derived samples are excluded from SFT/RFT. If any were included, the EndoVis-17 numbers in Tables I-III would measure memorization, not generalization. This is a correctness/leakage risk and a missing explicit statement, but it is not a circular derivation: no equation defines a prediction in terms of its own input, and no fitted parameter is renamed as a prediction. Hence the low circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper rests on data quality and evaluation integrity rather than on a mathematical derivation. There are no fitted constants inside a derived formula; the main numeric tunables are the unreported IoU threshold tau and the KL coefficient beta. The largest burden is that the dataset and its split must not leak the evaluation sets, especially EndoVis-17.

free parameters (2)
  • Visual Grounding reward IoU threshold tau = not reported
    Used in Eq. (4) to decide when a predicted bounding box receives the IoU-based reward; the value is never stated, and the reward's behavior depends on it.
  • GRPO KL regularization coefficient beta = not reported
    Appears in the GRPO objective in Eq. (1) to penalize deviation from the reference policy; its value is not given in the implementation details.
assumptions (4)
  • domain assumption Ground-truth labels in EndoVis-18-VQLA and EndoVis-17-VQLA are correct and complete.
    The paper relies on public annotations as the reference for both supervised fine-tuning and reward computation; Section IV.A.
  • domain assumption Qwen2.5-VL-72B-generated Chains of Thought, after surgeon review, are valid reasoning targets for a 3B model.
    The CoT data is synthesized by a proprietary API and then checked by surgeons; the paper does not quantify inter-surgeon agreement or residual errors. Section III.B.
  • domain assumption Rule-based rewards (exact answer match, IoU threshold, and quadrant coherence) are sufficient proxies for clinically meaningful reasoning and localization.
    The Multimodal Coherence reward reduces reasoning quality to a quadrant match, and the Linguistic Answer reward requires exact string equality. Section III.D.2.
  • domain assumption The constructed Surgery-R1-54k dataset does not leak EndoVis-17 or EndoVis-18 test data into training.
    The dataset is built from both EndoVis benchmarks, yet the paper does not explicitly state that EndoVis-17-derived samples are excluded from SFT and RFT while EndoVis-17 is used as external validation. Section IV.A.1 and IV.A.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning." pith.science (2026). https://pith.science/paper/IEU5YPWX

@misc{pith2026250619469,
  author       = {Pith},
  title        = {Pith review of: Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IEU5YPWX}},
  note         = {Machine review of arXiv:2506.19469}
}
read the original abstract

In recent years, significant progress has been made in the field of surgical scene understanding, particularly in the task of Visual Question Localized-Answering in robotic surgery (Surgical-VQLA). However, existing Surgical-VQLA models lack deep reasoning capabilities and interpretability in surgical scenes, which limits their reliability and potential for development in clinical applications. To address this issue, inspired by the development of Reasoning Multimodal Large Language Models (MLLMs), we first build the Surgery-R1-54k dataset, including paired data for Visual-QA, Grounding-QA, and Chain-of-Thought (CoT). Then, we propose the first Reasoning MLLM for Surgical-VQLA (Surgery-R1). In our Surgery-R1, we design a two-stage fine-tuning mechanism to enable the basic MLLM with complex reasoning abilities by utilizing supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Furthermore, for an efficient and high-quality rule-based reward system in our RFT, we design a Multimodal Coherence reward mechanism to mitigate positional illusions that may arise in surgical scenarios. Experiment results demonstrate that Surgery-R1 outperforms other existing state-of-the-art (SOTA) models in the Surgical-VQLA task and widely-used MLLMs, while also validating its reasoning capabilities and the effectiveness of our approach. The code and dataset will be organized in https://github.com/FiFi-HAO467/Surgery-R1.

Figures

Figures reproduced from arXiv: 2506.19469 by the authors.

Figure 1
Figure 1. The left part displays the original Surgical-VQLA, where the input consists of surgical images and related questions, and the output includes answers [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Details of data construction. The upper left part illustrates the workflow of data construction. The right part shows an example of constructing a chain [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Qualitative analysis of the reasoning process of MLLM after SFT and after SFT+RFT. The figure shows significant improvements in spatial positional [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages

  1. [1]

    Concepts and trends in autonomy for robot-assisted surgery,

    P. Fiorini, K. Y . Goldberg, Y . Liu, and R. H. Taylor, “Concepts and trends in autonomy for robot-assisted surgery,”Proceedings of the IEEE, vol. 110, no. 7, pp. 993–1011, 2022

    work page 2022

  2. [2]

    Video-instrument synergistic network for referring video instrument segmentation in robotic surgery,

    H. Wang, G. Yang, S. Zhang, J. Qin, Y . Guo, B. Xu, Y . Jin, and L. Zhu, “Video-instrument synergistic network for referring video instrument segmentation in robotic surgery,”IEEE Transactions on Medical Imag- ing, 2024

    2024

  3. [3]

    Surgical scene understanding in the era of foundation ai models: A comprehensive review,

    U. Khan, U. Nawaz, A. Qayyum, S. Ashraf, M. Bilal, and J. Qadir, “Surgical scene understanding in the era of foundation ai models: A comprehensive review,”arXiv preprint arXiv:2502.14886, 2025

    work page arXiv 2025

  4. [4]

    Dynamic interactive relation capturing via scene graph learning for robotic surgical report generation,

    H. Wang, Y . Jin, and L. Zhu, “Dynamic interactive relation capturing via scene graph learning for robotic surgical report generation,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 2702–2709

    work page 2023

  5. [5]

    Surgical-vqla: Trans- former with gated vision-language embedding for visual question localized-answering in robotic surgery,

    L. Bai, M. Islam, L. Seenivasan, and H. Ren, “Surgical-vqla: Trans- former with gated vision-language embedding for visual question localized-answering in robotic surgery,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 6859–6865

    work page 2023

  6. [6]

    Cat-vil: co-attention gated vision- language embedding for visual question localized-answering in robotic surgery,

    L. Bai, M. Islam, and H. Ren, “Cat-vil: co-attention gated vision- language embedding for visual question localized-answering in robotic surgery,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 397–407

    work page 2023

  7. [7]

    Enhancing visual reasoning with llm-powered knowledge graphs for visual question localized-answering in robotic surgery,

    P. Hao, H. Wang, G. Yang, and L. Zhu, “Enhancing visual reasoning with llm-powered knowledge graphs for visual question localized-answering in robotic surgery,”IEEE Journal of Biomedical and Health Informatics, 2025

    work page 2025

  8. [8]

    Gpt-4 technical report. arxiv 2303.08774,

    R. OpenAI, “Gpt-4 technical report. arxiv 2303.08774,”View in Article, vol. 2, no. 5, 2023

    arXiv 2023

Show all 44 references
  1. [9]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  2. [10]

    Llm-assisted multi-teacher continual learning for visual question answering in robotic surgery,

    K. Chen, Y . Du, T. You, M. Islam, Z. Guo, Y . Jin, G. Chen, and P.-A. Heng, “Llm-assisted multi-teacher continual learning for visual question answering in robotic surgery,”arXiv preprint arXiv:2402.16664, 2024

    2024 arXiv

  3. [11]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  4. [12]

    Otter: A multi-modal model with in-context instruction tun- ing,

    B. Li, Y . Zhang, L. Chen, J. Wang, F. Pu, J. A. Cahyono, J. Yang, C. Li, and Z. Liu, “Otter: A multi-modal model with in-context instruction tun- ing,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  5. [13]

    Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,

    G. Wang, L. Bai, W. J. Nah, J. Wang, Z. Zhang, Z. Chen, J. Wu, M. Islam, H. Liu, and H. Ren, “Surgical-lvlm: Learning to adapt large vision-language model for grounded visual question answering in robotic surgery,”arXiv preprint arXiv:2405.10948, 2024

    2024 arXiv

  6. [14]

    Endochat: Grounded multimodal large language model for endoscopic surgery,

    G. Wang, L. Bai, J. Wang, K. Yuan, Z. Li, T. Jiang, X. He, J. Wu, Z. Chen, Z. Leiet al., “Endochat: Grounded multimodal large language model for endoscopic surgery,”arXiv preprint arXiv:2501.11347, 2025

    2025 arXiv

  7. [15]

    A survey on hallucination in large vision-language models,

    H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,” arXiv preprint arXiv:2402.00253, 2024

    2024 arXiv

  8. [16]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carneyet al., “Openai o1 system card,”arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025. 11

    2025 arXiv

  10. [18]

    Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective,

    Z. Zeng, Q. Cheng, Z. Yin, B. Wang, S. Li, Y . Zhou, Q. Guo, X. Huang, and X. Qiu, “Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective,”arXiv preprint arXiv:2412.14135, 2024

    2024 arXiv

  11. [19]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,”arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  12. [20]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl,

    Y . Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang, “Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl,”arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  13. [21]

    Vision-r1: Evolving human-free alignment in large vision- language models via vision-guided reinforcement learning,

    Y . Zhan, Y . Zhu, S. Zheng, H. Zhao, F. Yang, M. Tang, and J. Wang, “Vision-r1: Evolving human-free alignment in large vision- language models via vision-guided reinforcement learning,”arXiv preprint arXiv:2503.18013, 2025

    2025 arXiv

  14. [22]

    Video-r1: Reinforcing video reasoning in mllms,

    K. Feng, K. Gong, B. Li, Z. Guo, Y . Wang, T. Peng, B. Wang, and X. Yue, “Video-r1: Reinforcing video reasoning in mllms,”arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  15. [23]

    Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models,

    Y . Lai, J. Zhong, M. Li, S. Zhao, and X. Yang, “Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models,” arXiv preprint arXiv:2503.13939, 2025

    2025

  16. [24]

    Huatuogpt-o1, towards medical complex reasoning with llms,

    J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang, “Huatuogpt-o1, towards medical complex reasoning with llms,”arXiv preprint arXiv:2412.18925, 2024

    2024 arXiv

  17. [25]

    How abilities in large language models are affected by supervised fine-tuning data composition,

    G. Dong, H. Yuan, K. Lu, C. Li, M. Xue, D. Liu, W. Wang, Z. Yuan, C. Zhou, and J. Zhou, “How abilities in large language models are affected by supervised fine-tuning data composition,”arXiv preprint arXiv:2310.05492, 2023

    2023 arXiv

  18. [26]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  19. [27]

    Aligning large multimodal models with factually augmented rlhf,

    Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y . Shen, C. Gan, L.-Y . Gui, Y .-X. Wang, Y . Yanget al., “Aligning large multimodal models with factually augmented rlhf,”arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  20. [28]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, pp. 53 728–53 741, 2023

    2023

  21. [29]

    Spatialrgpt: Grounded spatial reasoning in vision language models,

    A.-C. Cheng, H. Yin, Y . Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu, “Spatialrgpt: Grounded spatial reasoning in vision language models,”arXiv preprint arXiv:2406.01584, 2024

    2024 arXiv

  22. [30]

    Thinking in space: How multimodal large language models see, remember, and recall spaces,

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in space: How multimodal large language models see, remember, and recall spaces,”arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  23. [31]

    Langloc: Language-driven localization via formatted spatial description genera- tion,

    W. Shi, C. Chen, K. Li, Y . Xiong, X. Cao, and Z. Zhou, “Langloc: Language-driven localization via formatted spatial description genera- tion,”IEEE Transactions on Image Processing, 2025

    2025

  24. [32]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  25. [34]

    Surgical-vqa: Visual question answering in surgical scenes using transformer,

    L. Seenivasan, M. Islam, A. K. Krishna, and H. Ren, “Surgical-vqa: Visual question answering in surgical scenes using transformer,” in International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2022, pp. 33–43

    2022

  26. [35]

    Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering,

    Z. Yu, J. Yu, C. Xiang, J. Fan, and D. Tao, “Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering,” IEEE transactions on neural networks and learning systems, vol. 29, no. 12, pp. 5947–5959, 2018

    2018

  27. [36]

    Block: Bilinear superdiagonal fusion for visual question answering and visual relation- ship detection,

    H. Ben-Younes, R. Cadene, N. Thome, and M. Cord, “Block: Bilinear superdiagonal fusion for visual question answering and visual relation- ship detection,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 8102–8109

    2019

  28. [37]

    Mutan: Multi- modal tucker fusion for visual question answering,

    H. Ben-Younes, R. Cadene, M. Cord, and N. Thome, “Mutan: Multi- modal tucker fusion for visual question answering,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 2612– 2620

    2017

  29. [38]

    Surgicalgpt: end- to-end language-vision gpt for visual question answering in surgery,

    L. Seenivasan, M. Islam, G. Kannan, and H. Ren, “Surgicalgpt: end- to-end language-vision gpt for visual question answering in surgery,” inInternational conference on medical image computing and computer- assisted intervention. Springer, 2023, pp. 281–290

    2023

  30. [39]

    2018 robotic scene segmentation challenge,

    M. Allan, S. Kondo, S. Bodenstedt, S. Leger, R. Kadkhodamoham- madi, I. Luengo, F. Fuentes, E. Flouty, A. Mohammed, M. Pedersen et al., “2018 robotic scene segmentation challenge,”arXiv preprint arXiv:2001.11190, 2020

    2018 arXiv

  31. [40]

    2017 robotic instrument segmentation challenge,

    M. Allan, A. Shvets, T. Kurmann, Z. Zhang, R. Duggal, Y .-H. Su, N. Rieke, I. Laina, N. Kalavakonda, S. Bodenstedtet al., “2017 robotic instrument segmentation challenge,”arXiv preprint arXiv:1902.06426, 2019

    2017 arXiv

  32. [41]

    A review on evaluation metrics for data classification evaluations,

    M. Hossin and M. N. Sulaiman, “A review on evaluation metrics for data classification evaluations,”International journal of data mining & knowledge management process, vol. 5, no. 2, p. 1, 2015

    2015

  33. [42]

    Generalized intersection over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” inProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2019, pp. 658–666

    2019

  34. [43]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  35. [44]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,”Advances in Neural Information Processing Systems, vol. 36, pp. 28 541–28 564, 2023

    2023

  36. [45]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.