Pith. sign in

REVIEW 4 major objections 4 minor 33 references

An 11.2k dataset that breaks CT reading into slice-by-slice reasoning steps can unlock 3D spatial awareness in 2D-pretrained medical vision-language models, rivaling native 3D models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:28 UTC pith:HLS6BVGT

load-bearing objection The Hounsfield-CoT dataset and the dual-agent synthesis pipeline are genuinely new and plausible, but the central empirical claim is not supported because training and evaluation share the same CT-RATE source with no disclosed split. the 4 major comments →

arxiv 2607.13860 v1 pith:HLS6BVGT submitted 2026-07-15 cs.CV

Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

Zhuoyuan Fu , Zeshang Li , Yiqiong Zhang , Hangui Lin , Yan Shu , Yan Li , Binyang Li , Yaru Zhao This is my paper
classification cs.CV
keywords 3D medical imagingchain-of-thought reasoningmultimodal large language modelsCT-RATEspatial reasoninginstruction tuningdata synthesisinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the main obstacle to 3D medical reasoning in multimodal language models is not the architecture but the format of supervision. Existing 3D datasets give models final answers or whole reports, never the slice-by-slice reasoning radiologists actually perform. To fix this, the authors build Hounsfield-CoT, an 11.2k-instance dataset that decomposes CT volumes into per-slice observations and re-synthesizes them into explicit chain-of-thought rationales. Instruction-tuning a standard 2D-pretrained vision-language model on this data produces large accuracy gains on 3D-RadVQA (anomaly detection jumps from 7.2% to 33.5%) and approaches native 3D performance with far less computation. If correct, 3D spatial reasoning can be taught through data rather than through expensive 3D-specific pretraining.

Core claim

The central claim is that explicit, slice-grounded reasoning traces are the missing supervision for volumetric understanding. Using an Observer-Synthesizer dual-agent pipeline, the authors take volume-level CT-RATE reports, expand them into fine-grained per-slice findings, and integrate those findings into chain-of-thought trajectories that must include sequential spatial tracking, true 3D spatial awareness (distinguishing real abnormalities from partial-volume artifacts), and differential exclusion. Fine-tuning a 2D-pretrained multimodal LLM with low-rank adaptation on 11.2k such instances yields sharp improvements on 3D-RadVQA—T1 accuracy rises from 7.2% to 33.5%, T2 from 7.7% to 24.1%—whi

What carries the argument

The central object is Hounsfield-CoT, a dataset synthesized by a dual-agent (Observer-Synthesizer) framework. The Observer extracts per-slice findings from global radiology reports using a large language model; the Synthesizer fuses them into a chain-of-thought constrained to three clinical principles: sequential spatial tracking, true 3D spatial awareness, and differential exclusion. Its role is to convert opaque volume-level QA pairs into transparent, step-by-step spatial reasoning trajectories that can serve as instruction-tuning supervision for a 2D vision-language model, transferring 3D spatial capability through data rather than architecture.

Load-bearing premise

The entire result depends on the 11.2k training instances being built from CT-RATE scans and reports that are disjoint from the 3D-RadVQA evaluation cases; if those sets overlap, the accuracy gains may reflect memorized report content rather than learned 3D spatial reasoning.

What would settle it

Compare scan and report identifiers between the Hounsfield-CoT training set and the 3D-RadVQA test set; if any test case derives from a report used in training, the finding is confounded. A cleaner check is to evaluate the fine-tuned model on a held-out CT benchmark not sourced from CT-RATE and see whether the T1-level gains persist.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuning on Hounsfield-CoT lifts 3D-RadVQA accuracy decisively (T1: 7.2% to 33.5%; T2: 7.7% to 24.1%; T3: 3.2% to 10.3%) while BERT-F1 stays flat, indicating the model learns new reasoning rather than memorized phrasing.
  • The same 11.2k dataset improves a second 2D-pretrained baseline (Lingshu, T1 from 9.9% to 19.4%), so the paradigm is model-agnostic.
  • The adapted model rivals or approaches native 3D competitors (e.g., M3D at 46.0% T1) while using a 2D backbone and low-rank adaptation, offering a much cheaper path to volumetric understanding.
  • Data-scaling experiments show spatial tasks improve steadily with more CoT examples, and binary detection recovers from an initial drop—consistent with replacing prior-based guessing with verified reasoning.
  • Generated rationales explicitly track lesions across slices and rule out partial-volume artifacts, making the diagnostic process interpretable rather than a black box.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit: the same slice-wise CoT synthesis could be applied to other volumetric modalities such as MRI or ultrasound; if the gains reproduce, the paradigm is a general recipe for teaching 3D reasoning rather than a CT-specific fix.
  • Because synthesis relies on already-written radiology reports, it can only teach reasoning the reports contain; adding per-slice annotations or pixel-level grounding could push the model further and would directly address the subtle spatial hallucinations the authors acknowledge in their limitations.
  • A natural next experiment is to fine-tune a native 3D model on Hounsfield-CoT; if the gains add to those of 3D pretraining, it would show CoT supervision and volumetric encoders solve complementary problems rather than the same one.
  • The accuracy metric parses semantic coordinates but does not certify clinical safety of the intermediate steps; a human reader study of the think blocks would tell whether the chains are genuinely verifiable or merely plausible.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a slice-wise data synthesis paradigm, Hounsfield-CoT, that converts CT-RATE volume-level clinical reports into 11.2k structured chain-of-thought reasoning instances via a dual-agent (Observer-Synthesizer) pipeline. The authors instruction-tune a 2D-pretrained MLLM (Hounsfield2D) on this dataset using LoRA, and report substantial accuracy improvements on the CT-RATE-derived 3D-RadVQA benchmark (e.g., T1 accuracy from 7.2% to 33.5%), as well as modest gains on DeepchestVQA. The central claim is that such a data-centric approach can unlock volumetric spatial reasoning in 2D models without 3D-specific pre-training.

Significance. If the empirical claim were sound, the paper would offer a practical, low-cost route to 3D medical reasoning and an interpretable reasoning format that aligns with clinical workflow. The public release of the dataset and code is a positive step. However, the core evidence is compromised by an undisclosed train/eval split: the synthetic training data is built exclusively from CT-RATE reports (Sec. III.A), and the primary evaluation benchmark, 3D-RadVQA, is also CT-RATE-derived (Sec. V.A). With no patient-, scan-, or report-level split described, the reported gains may reflect memorization of the answer tokens associated with specific reports rather than learned spatial reasoning. The absence of a control fine-tuned on the same data without CoT, and the lack of error bars or multiple seeds, further weaken the causal attribution. The current results are therefore internally consistent but not externally valid as evidence for the paper's central claim.

major comments (4)
  1. [Sec. III.A and Sec. V.A] The training data (Hounsfield-CoT) is derived 'exclusively from the CT-RATE dataset' and the task formulation is 'restricted to the core competencies evaluated in the 3D-RadVQA benchmark.' The evaluation is then performed on 3D-RadVQA T1–T4, which is itself built from CT-RATE. The manuscript never specifies a patient-, scan-, or report-level split between the 11.2k training instances and the evaluation set. If any scan or report in 3D-RadVQA appears in the training data, the reported accuracy jumps (e.g., T1 from 7.2% to 33.5%) could be a memorization artifact, not spatial reasoning. This must be resolved by disclosing the exact split or by evaluating on a held-out dataset with no provenance overlap.
  2. [Sec. IV.B / Table II] No control experiment fine-tunes the same baseline on the same 11.2k instances with the CoT chains removed (e.g., answer-only or flat QA format). Without such a control, the observed accuracy gains cannot be attributed to the CoT structure or the three clinical principles; they could equally result from additional in-domain training on CT-RATE-derived data. A matched control is essential to support the paper's causal claim that 'structured CoT supervision... transforms standard 2D backbones into robust 3D volumetric reasoners.'
  3. [Table II and Sec. V.B] The paper interprets stagnant BERT-F1 scores alongside large accuracy gains as evidence that the model shifts from 'surface-level textual mimicry' to 'genuine multi-step spatial reasoning.' However, this pattern is also fully consistent with memorization of sparse answer tokens: exact-match accuracy can increase while semantic overlap with a reference sentence remains unchanged. The claim requires additional evidence, such as human evaluation of reasoning steps or error analysis on cases where the report prior is misleading.
  4. [Fig. 5 (Data Scaling Ablation)] The scaling ablation shows monotonic improvement for T1/T2 but does not include a non-CoT control at matched data volumes, nor does it report variance across seeds. The T4 drop-and-recover pattern is interpreted as 'shortcut disruption,' but without a control or statistical confidence intervals this interpretation is speculative. The authors should report means over at least three random seeds and include an answer-only data scaling curve to enable a meaningful comparison.
minor comments (4)
  1. [Sec. III.B] The pipeline in Fig. 2 mentions 'opacity dimming' to maintain a bounding-box-free constraint, but the caption does not explain how this prevents the model from using visual shortcuts. A clear methodological note would help.
  2. [Abstract / Sec. III.A] The paper states the dataset comprises 11.2k 'instances' but does not report the number of unique CT volumes or patients underlying those instances. This is important for assessing potential scan-level leakage.
  3. [Sec. V.A] The use of 'LLM-as-a-judge' for open-ended T1–T3 accuracy is not described in sufficient detail: which LLM was used, what prompt, and how were partial matches scored? This needs specification for reproducibility.
  4. [References] Some references are to future-dated preprints (e.g., GPT-5.4, arXiv:2603.13800, arXiv:2606.11740). While not disqualifying, the authors should verify these citations are publicly available and correctly dated.

Circularity Check

1 steps flagged

Central evaluation is circular: Hounsfield-CoT is built from CT-RATE to target 3D-RadVQA, then 3D-RadVQA is used as the test set with no disclosed train/test split.

specific steps
  1. fitted input called prediction [Sec. III (Dataset Construction) and Sec. V.A (Experimental Settings)]
    "Moving away from static multi-source aggregation, we focused exclusively on the large-scale CT-RATE dataset to construct high-quality, task-specific reasoning trajectories for 3D-RadVQA."

    The training set Hounsfield-CoT is synthesized exclusively from CT-RATE volume-level clinical reports, with task formulation restricted to the T1–T4 competencies of the 3D-RadVQA benchmark. Section V.A then evaluates on 3D-RadVQA T1–T4. The paper never discloses a patient-, scan-, or report-level split between the 11.2k synthesized training instances and the 3D-RadVQA test instances. Because the <answer> tokens in the training CoT are instantiated from the same CT-RATE metadata that defines 3D-RadVQA ground truth, the reported accuracy jumps (e.g., T1 7.2% to 33.5%) may reflect memorization of report-derived answers rather than learned spatial reasoning. The zero-shot DeepchestVQA results provide some independent signal, but the central claims of spatial reasoning gains, cross-architecture

full rationale

The paper's derivation chain is otherwise self-contained: standard 2D-pretrained MLLMs are instruction-tuned on a synthetic CoT dataset, and the proposed data synthesis pipeline is described in detail. Self-citations to works such as UniReason-Med, Dyco-RL, and Terrascope appear only as related-work motivation or analogy and are not load-bearing for the main derivation; they do not, by themselves, raise the circularity score. The dominant circularity concern is evaluative: Hounsfield-CoT is constructed from CT-RATE specifically to match 3D-RadVQA, and 3D-RadVQA is then used as the primary benchmark without any disclosed split. Since the benchmark and the training set share the same provenance, the reported improvements on 3D-RadVQA cannot be unambiguously attributed to genuine 3D spatial reasoning. The paper's own limitation statement acknowledges possible spatial hallucinations in synthesis but does not address the missing train/test separation. The DeepchestVQA zero-shot results (e.g., Medical Reasoning 39.7% to 45.1%) offer partial independent validation and prevent the score from being higher, but the main quantitative claims remain vulnerable to benchmark contamination. Score 7 reflects a central result whose primary evaluation reduces, by construction, to the same data distribution the model was trained to imitate.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities. The main hidden load is the shared CT-RATE provenance of training and evaluation, plus several domain assumptions about the validity of LLM-generated reasoning chains. The only explicit hand-chosen numeric parameter is the eight-slice serialization.

free parameters (1)
  • Slice count = 8
    The 2.5D spatial serialization uses exactly eight contiguous z-slices. This hand-chosen number is central to the method, and no ablation varying the slice count is reported.
axioms (5)
  • domain assumption CT-RATE clinical reports are accurate and complete ground truth for all pathological findings.
    Invoked in Sec. III.A when reports are described as 'the ground-truth pathological findings of the entire 3D volume' and used to instantiate QA queries.
  • domain assumption The GPT-5.4 Observer agent can correctly translate volume-level report findings into per-slice visual observations without seeing slice-level labels.
    Sec. III.B uses this to generate fine-grained per-slice findings; the paper acknowledges in Sec. VI that this may produce spatial hallucinations.
  • ad hoc to paper Sequential spatial tracking, 3D spatial awareness, and differential exclusion are sufficient and necessary components for clinical 3D reasoning.
    These three principles are introduced in Sec. III.B as mandatory; no independent validation shows that omitting one invalidates the approach.
  • domain assumption Eight contiguous slices adequately represent the volumetric structure for reasoning tasks.
    Sec. IV.A fixes the serialization to eight slices, which is a strong reduction of a full CT volume; the paper does not analyze sensitivity to this choice.
  • domain assumption 3D-RadVQA benchmark answers are not contaminated by the CT-RATE reports used to construct the training data.
    Sec. V.A evaluates on 3D-RadVQA while Sec. III.A constructs training data from CT-RATE; no explicit train/test split is provided, so this assumption is load-bearing.

pith-pipeline@v1.3.0-alltime-deepseek · 11090 in / 9808 out tokens · 89953 ms · 2026-08-02T03:28:10.412332+00:00 · methodology

0 comments
read the original abstract

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.

Figures

Figures reproduced from arXiv: 2607.13860 by Binyang Li, Hangui Lin, Yan Li, Yan Shu, Yaru Zhao, Yiqiong Zhang, Zeshang Li, Zhuoyuan Fu.

Figure 1
Figure 1. Figure 1: Overview of Hounsfield-CoT and model performance refinement. (a) Our paradigm integrates sequential slice-by-slice tracking, 3D spatial awareness, and differential exclusion as core competencies. (b) We validate this framework by instruction-tuning the Hounsfield2D model on our curated CoT dataset via LoRA to acquire advanced volumetric reasoning capabilities, resulting in the Hounsfield-3D model. (c) Qual… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Hounsfield-CoT data synthesis pipeline. The workflow explicitly models the slice-by-slice reading process of radiologists to transform raw 3D CT volumes into a high-quality CoT dataset, entirely bypassing the need for bounding boxes or image cropping. Top: Dual-Agent Pipeline. A raw CT volume is sampled into 8 slices and processed sequentially by a GPT-5.4 Observer and a GPT-5.4 Synthesizer… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative example of the generated Hounsfield-CoT reasoning path. The presented case strictly adheres to our proposed three required clinical elements. In this consolidation localization task, the model provides a Sequential slice-by-slice description of the abnormal opacity, demonstrates True 3D spatial awareness by integrating the lingular lesion across contiguous slices, and performs Differential excl… view at source ↗
Figure 4
Figure 4. Figure 4: Case study of Hounsfield-CoT demonstrating explicit 3D volumetric reasoning. Given an 8-slice CT sequence for abnormality localization, the model performs explicit slice-by-slice tracking from apex to base (blue). It successfully identifies linear streaky opacities in the lower lobes and leverages 3D spatial continuity (green) to logically rule out partial-volume artifacts and pleural interfaces (orange), … view at source ↗
Figure 5
Figure 5. Figure 5: Data Scaling Ablation on Spatial and Complex Reasoning Tasks. We evaluate the impact of training data volume on tasks that heavily demand spatial grounding (Tasks 1 and 2) and com￾prehensive multi-step reasoning (Task 4). Pure computation (Task 3) is omitted here to prevent the semantic priors of the base LLM from confounding the spatial scaling trends. Spatial tasks (T1, T2) show consistent scaling improv… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 14 linked inside Pith

  1. [1]

    Umit: Unifying medical imaging tasks via vision-language models.arXiv preprint arXiv:2503.15892, 2025

    Haiyang Yu, Siyang Yi, Ke Niu, Minghan Zhuo, and Bin Li. Umit: Unifying medical imaging tasks via vision-language models.arXiv preprint arXiv:2503.15892, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Multimodal large language models in health care: applications, challenges, and future outlook.Journal of medical Internet research, 26:e59505, 2024

    Rawan AlSaad, Alaa Abd-Alrazaq, Sabri Boughorbel, Arfan Ahmed, Max-Antoine Renault, Rafat Damseh, and Javaid Sheikh. Multimodal large language models in health care: applications, challenges, and future outlook.Journal of medical Internet research, 26:e59505, 2024

    2024

  3. [3]

    Towards generalist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mo- hamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024

    2024

  4. [4]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Fleming-vl: Towards universal medical visual reasoning with multimodal llms.arXiv preprint arXiv:2511.00916, 2025

    Yan Shu, Chi Liu, Robin Chen, Derek Li, and Bryan Dai. Fleming-vl: Towards universal medical visual reasoning with multimodal llms.arXiv preprint arXiv:2511.00916, 2025

    arXiv 2025

  6. [6]

    Hulu- med: A transparent generalist model towards holistic medical vision- language understanding.arXiv preprint arXiv:2510.08668, 2025

    Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu- med: A transparent generalist model towards holistic medical vision- language understanding.arXiv preprint arXiv:2510.08668, 2025

    arXiv 2025

  7. [7]

    Developing generalist foundation models from a multimodal dataset for 3d computed tomog- raphy.arXiv preprint arXiv:2403.17834, 2024

    Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomog- raphy.arXiv preprint arXiv:2403.17834, 2024

    arXiv 2024

  8. [8]

    Towards a holistic framework for multimodal llm in 3d brain ct radiology report generation.Nature Communications, 16(1):2258, 2025

    Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, Hritik Bansal, Ling Chen, Yi-Ping Yang, Yu-Chun Chen, Shih-Pin Chen, et al. Towards a holistic framework for multimodal llm in 3d brain ct radiology report generation.Nature Communications, 16(1):2258, 2025

    2025

  9. [9]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36:28541–28564, 2023

    2023

  10. [10]

    Biomedclip: a multimodal biomedical foundation model pre- trained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pre- trained from fifteen million scientific image-text pairs.arXiv preprint arXiv:2303.00915, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Merlin: A vision language foundation model for 3d computed tomography.Research Square, pages rs–3, 2024

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean- Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography.Research Square, pages rs–3, 2024

    2024

  12. [12]

    Vista3d: A unified segmentation foundation model for 3d medical imaging

    Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Benjamin Simon, Mason Belue, et al. Vista3d: A unified segmentation foundation model for 3d medical imaging. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 20863–20873, 2025

    2025

  13. [13]

    M3d: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Segment anything model for medical image analysis: an experimental study.Medical Image Analysis, 89:102918, 2023

    Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study.Medical Image Analysis, 89:102918, 2023

    2023

  15. [15]

    Med-2e3: A 2d-enhanced 3d medical multimodal large language model

    Yiming Shi, Xun Zhu, Kaiwen Wang, Ying Hu, Chenyi Guo, Miao Li, and Ji Wu. Med-2e3: A 2d-enhanced 3d medical multimodal large language model. In2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2754–2759. IEEE, 2025

    2025

  16. [16]

    Beyond medical diagnostics: How medical multimodal large language models think in space.arXiv preprint arXiv:2603.13800, 2026

    Zhanwei Zhou, Junde Wu, Jiayuan Zhu, Yuhao Huang, and Min Xu. Beyond medical diagnostics: How medical multimodal large language models think in space.arXiv preprint arXiv:2603.13800, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    3drea- sonknee: Advancing grounded reasoning in medical vision language models

    Sraavya Sambara, Sung Eun Kim, Xiaoman Zhang, Luyang Luo, Shreya Johri, Mohammed Baharoon, Du Hyun Ro, and Pranav Rajpurkar. 3drea- sonknee: Advancing grounded reasoning in medical vision language models. InBiocomputing 2026: Proceedings of the Pacific Symposium, pages 99–113. World Scientific, 2025

    2026

  18. [18]

    Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Clincot: Clinical-aware visual chain-of-thought for medical vision language models.arXiv preprint arXiv:2603.01124, 2026

    Xiwei Liu, Yulong Li, Xinlin Zhuang, Xuhui Li, Jianxu Chen, Haolin Yang, Imran Razzak, and Yutong Xie. Clincot: Clinical-aware visual chain-of-thought for medical vision language models.arXiv preprint arXiv:2603.01124, 2026

    arXiv 2026

  20. [20]

    Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with llm and mllm integration

    Lai Wei, Wenkai Wang, Xiaoyu Shen, Yu Xie, Zhihao Fan, Xiaojin Zhang, Zhongyu Wei, and Wei Chen. Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with llm and mllm integration. arXiv preprint arXiv:2410.04521, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Fleming-r1: Toward expert-level medical reasoning via reinforcement learning.arXiv preprint arXiv:2509.15279, 2025

    Chi Liu, Derek Li, Yan Shu, Robin Chen, Derek Duan, Teng Fang, and Bryan Dai. Fleming-r1: Toward expert-level medical reasoning via reinforcement learning.arXiv preprint arXiv:2509.15279, 2025

    arXiv 2025

  22. [22]

    M3cotbench: Benchmark chain-of-thought of mllms in medical image understanding.arXiv preprint arXiv:2601.08758, 2026

    Juntao Jiang, Jiangning Zhang, Yali Bi, Jinsheng Bai, Weixuan Liu, Weiwei Jin, Zhucun Xue, Yong Liu, Xiaobin Hu, and Shuicheng Yan. M3cotbench: Benchmark chain-of-thought of mllms in medical image understanding.arXiv preprint arXiv:2601.08758, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Unireason-med: A shared grounded rea- soning interface for 2d-to-3d transfer in medical vqa.arXiv preprint arXiv:2606.11740, 2026

    Mengzhuo Chen, Yan Shu, Chi Liu, Hongming Piao, Xidong Wang, Derek Li, and Bryan Dai. Unireason-med: A shared grounded rea- soning interface for 2d-to-3d transfer in medical vqa.arXiv preprint arXiv:2606.11740, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Med3dvlm: An efficient vision-language model for 3d medical image analysis.IEEE Journal of Biomedical and Health Informatics, 2025

    Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. Med3dvlm: An efficient vision-language model for 3d medical image analysis.IEEE Journal of Biomedical and Health Informatics, 2025

    2025

  26. [26]

    Medcot: Medical chain of thought via hierarchical expert

    Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, and Zuozhu Liu. Medcot: Medical chain of thought via hierarchical expert. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17371–17389, 2024

    2024

  27. [27]

    Dyco-rl: Dynamic cross-modal coordination for visual reasoning.arXiv preprint arXiv:2606.08035, 2026

    Hangui Lin, Yan Shu, Zhengyang Liang, Chi Liu, Xiangrui Liu, Minghao Qin, Teng Long, Zheng Liu, and Nicu Sebe. Dyco-rl: Dynamic cross-modal coordination for visual reasoning.arXiv preprint arXiv:2606.08035, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Pmc-vqa: Visual instruction tun- ing for medical visual question answering, 2024.URL https://arxiv

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tun- ing for medical visual question answering, 2024.URL https://arxiv. org/abs/2305.10415, 40, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Medtrinity- 25m: A large-scale multimodal dataset with multigranular annotations for medicine.arXiv preprint arXiv:2408.02900, 2024

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity- 25m: A large-scale multimodal dataset with multigranular annotations for medicine.arXiv preprint arXiv:2408.02900, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Biomedical visual instruction tuning with clinician preference alignment.Advances in neural information processing systems, 37:96449–96467, 2024

    Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment.Advances in neural information processing systems, 37:96449–96467, 2024

    2024

  31. [31]

    Towards injecting medical visual knowledge into multimodal llms at scale

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et al. Towards injecting medical visual knowledge into multimodal llms at scale. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 7346–7370, 2024

    2024

  32. [32]

    Radgenome-chest ct: A grounded vision- language dataset for chest ct analysis.arXiv preprint arXiv:2404.16754, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision- language dataset for chest ct analysis.arXiv preprint arXiv:2404.16754, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Terrascope: Pixel-grounded visual reasoning for earth observation

    Yan Shu, Bin Ren, Zhitong Xiong, Xiao Xiang Zhu, Beg ¨um Demir, Nicu Sebe, and Paolo Rota. Terrascope: Pixel-grounded visual reasoning for earth observation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16712–16722, 2026

    2026