Pith. sign in

REVIEW 4 major objections 5 minor 52 references

HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3D CT model that combines two complementary encoders and a spatial-preserving projector tops prior benchmarks on report generation, retrieval, and VQA.

desk verdict The retrieval half is probably solid; the generative SOTA claims are underdetermined because the baselines use different LLMs, and the paper's own ablations show the visual modules contribute only a few points. read the letter →

arxiv 2506.09634 v1 pith:SPDL2TKL submitted 2025-06-11 cs.CV cs.AI

Yanzhao Shi , Xiaodan Zhang , Junzhong Ji , Haoning Jiang , Chengxin Zheng , Yinong Wang , Liangqiong Qu This is my paper
classification cs.CVcs.AI
keywords HSENet3Dmedicalvision-languageCTreportgenerationcontrastivepretrainingmultimodalprojectorspatialencodingvisualquestionansweringCT-RATE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a 3D medical vision-language model can understand chest CT volumes substantially better than prior 2D-based or single-encoder 3D systems by separating visual perception from projection. It claims that two complementary 3D encoders—one trained for global volume-report alignment, one refined with 2D slice knowledge to emphasize anatomy—provide the visual cues an LLM needs, and that a Spatial Packer projector preserves spatial detail while compressing the visual input to few tokens. On the CT-RATE benchmark it reports 39.85% R@100 for report-to-volume retrieval, 24.01% BLEU-4 for report generation, and 73.60% Major Class Accuracy on RadGenome-ChestCT VQA, each above reproduced baselines. These gains, if they hold, would suggest a practical way to reduce diagnostic hallucinations in automated CT interpretation.

What carries the argument

The Spatial Packer is the projector that turns high-resolution 3D features into a compact token set: it partitions the 3D feature map into local voxels, pools each to a centroid point, and then runs Voxel2Point Cross-Attention from each centroid query over its own voxel keys and values, so spatial detail is absorbed rather than discarded. It works alongside the dual-stage pretraining, where stage two's 2E3 encoder uses 2D slices to score 3D patches and a semantic consistency loss keeps the local refinement aligned with global stage-one relations.

What would settle it

Run each configuration (HSENet, M3D-CLIP, Med3DVLM, Med-2E3) with at least five random seeds on CT-RATE and RadGenome-ChestCT, and report mean and standard deviation of R@100, BLEU-4, and Major Class Accuracy; if the confidence intervals overlap for the reported deltas, the central performance claim fails.

Watch

Extended reading notes

Core claim

The central claim is that hybrid spatial encoding—dual 3D vision encoders pretrained in two stages and projected through the Spatial Packer—delivers state-of-the-art 3D medical vision-language understanding. Stage one aligns a 3D Vision Transformer with diagnostic reports using contrastive learning. Stage two trains a second, 2D-enhanced 3D encoder (2E3 Vision Encoder) that uses BioMedCLIP features from 2D slices to score and weight 3D patches, emphasizing diagnostically relevant anatomy, while a semantic consistency loss anchors the local refinement to the global alignment learned in stage one. The Spatial Packer reshapes patch features back into 3D voxel layout, pools each local voxel to a centroid, and runs Voxel2Point Cross-Attention so each centroid query absorbs its high-resolution voxel details into compact tokens for the LLM. The paper reports that this combination outperforms M3D-CLIP, Med3DVLM, Med-2E3, and other baselines on retrieval, report generation, and VQA, and shows ablations where each component contributes to the final numbers.

Load-bearing premise

The load-bearing premise is that the reported performance gaps are statistically real, but the paper runs each experiment once and reports no error bars or significance tests.

Editorial extensions

If this is right

  • If the gains replicate, the dual-encoder plus Spatial Packer design offers a template for 3D medical MLLMs that preserves depth information without inflating token counts.
  • The 2D-slice-guided patch scoring introduces a cheap supervision source: slicing existing 3D data, without external patient records or LLM-generated text.
  • The Spatial Packer's token compression maintains clinical relevance even at 32 tokens, which could lower LLM compute costs in 3D inference.
  • The BIMCV-R transfer results indicate the pretrained encoders generalize to a new CT dataset without retraining, pointing toward data-efficient adaptation.
  • RaTE-Score improvements suggest the model's reports are not only linguistically better but clinically more aligned with ground-truth findings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper runs each experiment once and reports no error bars or significance tests, so whether the +5.96% R@100 and +1.99% VQA margins are robust to seed or split variation is an open question the paper does not settle.
  • The same dual-encoder/V2P-CA recipe could be tested on other volumetric modalities such as MRI or PET, or on non-medical 3D data, to separate the effect of the architecture from the effect of 2D-slice supervision.
  • The 2D-slice scoring mechanism may be sensitive to the number of slices and the choice of 2D encoder; the paper fixes $N_s=32$ and BioMedCLIP, so that sensitivity is unexplored.
  • The paper's claim of reduced hallucinations is supported mainly by qualitative examples; a systematic entity-level factual consistency metric would provide a direct quantitative test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces HSENet, a 3D medical vision-language framework for CT data. HSENet uses two pretrained 3D vision encoders: a standard 3D ViT for global structure and a '2E3' encoder that uses 2D slice features to weight low-level 3D patches for fine-grained anatomy. A new projector, the Spatial Packer with Voxel2Point Cross-Attention, compresses 3D features into compact visual tokens. The authors report state-of-the-art results on CT-RATE retrieval (R@100 39.85%, +5.96% over M3D-CLIP), CT-RATE report generation (BLEU-4 24.01%, +8.01% over Med3DVLM), and RadGenome-ChestCT VQA (Major Class Accuracy 73.60%, +1.99%/3.21% over prior methods), with ablations for the projector, encoders, token count, and pretraining losses, plus a cross-dataset evaluation on BIMCV-R.

Significance. If the claims hold, the paper makes a useful architectural contribution: the dual-encoder design with 2D-slice-guided patch scoring is a plausible route to capturing both global and local 3D anatomical cues, and the V2P-CA projector is a principled way to compress 3D features while preserving spatial structure. The paper is empirically extensive, with ablations for the projector, encoder combinations, token counts, and loss terms, and it includes an external dataset evaluation, efficiency measurements, and a public code URL. The contributions are clearly positioned relative to existing 3D medical MLLMs. However, the headline generative SOTA claims are currently underdetermined because the cross-system comparisons do not control the base LLM, and the absence of error bars or significance tests leaves small margins unsupported.

major comments (4)
  1. [Section 4.1, Tables 2(a) and 4] The cross-system report-generation and VQA comparisons do not control the base LLM. HSENet uses Phi-4-4B-Instruct, while the reproduced Med-2E3 and Med3DVLM baselines are run from official code, but the paper never states which LLM those baselines use. Because the language model dominates text generation quality, the headline gains (+8.01 BLEU-4 in Table 2(a) and +3.21 Major Class Acc in Table 4) cannot be attributed to the dual encoders or Spatial Packer. Please add matched-LLM controls, for example by fixing Phi-4-4B-Instruct across all visual encoders/projectors, or by clearly reporting each baseline LLM and performing a swap experiment that isolates the visual components.
  2. [Section 4.1, NeurIPS checklist item 7] No error bars, confidence intervals, or significance tests are reported, and the checklist explicitly answers 'No' to statistical significance. The flagship margins are small relative to likely run-to-run variability: +5.96% R@100 in Table 1(a), +1.99% Major Class Acc over M3D-LaMed in Table 4, and +8.01 BLEU-4 in Table 2(a). With single runs, these deltas are not statistically supported. Please report at least a few seeds for the main comparisons, or a paired significance test over the test set, and tone down the 'state-of-the-art' wording accordingly.
  3. [Abstract vs. Section 4.4] The abstract states '73.60% of Major Class Accuracy, +1.99% gain', which matches the gap to M3D-LaMed (71.61 in Table 4), while Section 4.4 states '+3.21% over Med3DVLM' (70.39 in Table 4). The comparison baseline is ambiguous. Please specify in the abstract which prior method the gain is measured against.
  4. [Table 1(a)] In the VocabFine row, R@100 (2.00) is lower than R@50 (2.30), which is impossible for a recall-at-k metric. This appears to be a typographical error, but it should be corrected and the surrounding numbers verified, since the table supports the retrieval SOTA claim.
minor comments (5)
  1. [Equation (5)] The displayed equation contains a stray comma and spacing artifact: 'LSCL =L 2e3 CL +λ sLSA, ,'. Please fix the typesetting.
  2. [Table 5] In the 256-token row, the BLEU-1 and BLEU-2 values are concatenated as '63.4138.31'. Please insert a separator.
  3. [Section 4.1 and reference [1]] The paper says 'Phi4-4B-Instruct' but the cited reference is the Phi-4-Mini technical report. Please align the model name and citation.
  4. [Throughout] Several typos and spacing issues remain, including 'V oxel2Point' in Section 3.3, 'V ocabFine' in Table 1, and 'diffrernt' in the Table 1(b) caption. A careful proofread is recommended.
  5. [Supplemental material, code availability] The supplemental checklist says the code 'is included in the supplemental material and will be open-sourced upon acceptance', while the abstract gives a public GitHub URL. Please state clearly whether the code is currently available and under what license.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: HSENet's main results are measured on held-out splits and its ablations fix the LLM; the only self-citation is non-load-bearing.

full rationale

HSENet is an empirical systems paper, not a derivation, and its claimed gains are grounded in held-out test sets rather than in fitted parameters renamed as predictions. The dual encoders are pretrained on the CT-RATE training split (20,000 unique patients) and evaluated on a test split with 1,304 distinct patients, so the retrieval results in Table 1 are not reconstructed from the training objective. Report-generation and VQA results are likewise measured on separate test sets, and Appendix B adds an out-of-distribution BIMCV-R evaluation. The internal ablations (Tables 2b and 3) hold the LLM fixed and vary only the projector or the visual encoders, providing a matched control for the architectural claims. The cross-system SOTA comparisons in Tables 2(a) and 4 mix different base LLMs (HSENet uses Phi-4-4B-Instruct; the reproduced baselines use their official LLMs), but that is a comparison confound, not circularity, and the paper's own ablations provide partial independent support for the visual components. No equation reduces to its own input: the Stage-2 semantic-consistency loss anchors to Stage-1 features as a regularizer, not as a definition of the test metric, and the Spatial Packer is an architectural component validated by ablations rather than an identity. The only self-citation is reference [49] (MEPNet), used in the introduction as an example of prior slice-by-slice brain CT analysis; it is not load-bearing for any of the paper's claims. The absence of error bars affects statistical robustness, not circularity. Score 2 reflects the presence of one minor, non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The ledger is clean in that no new physical entities or assumptions outside standard ML practice are introduced. The main items are the hand-chosen compression stride and loss weight, plus three background assumptions about dataset label validity, baseline reproduction fidelity, and BioMedCLIP test-set independence, none of which the paper verifies with external evidence.

free parameters (3)
  • Spatial packer strides (Sd, Sw, Sh) = (8, 4, 4) = (8, 4, 4)
    Chosen by hand as the default token count (128). The appendix sensitivity study shows performance varies with this choice, confirming that results depend on a tuning decision rather than a derived optimum.
  • Semantic consistency loss weight lambda_s = 0.1
    Chosen without a sensitivity study or derivation; the paper reports only that this value is used (Section 4.1).
  • Pretraining epochs (50), fine-tuning epochs (6 and 4), learning rates (1e-4 and 5e-5), temperature tau, number of 2D… = Various
    Standard hyperparameters selected to make training converge; no sensitivity analysis is provided, and reported gains could shift under different settings.
assumptions (3)
  • domain assumption CT-RATE and RadGenome-ChestCT test labels are accurate and reflect the clinical ground truth used for evaluation.
    All reported gains (R@100, BLEU-4, RaTE-Score, VQA accuracy) are computed against these dataset labels; if labels contain annotation noise, all comparisons inherit that noise.
  • domain assumption The authors' reimplementations of M3D-CLIP, Med3DVLM, and Med-2E3 faithfully reproduce the original methods' intended performance.
    Baselines are marked with a dagger, meaning the authors reproduced them using official code. Cross-model deltas (e.g., +5.96% R@100) are only as trustworthy as these reproductions, and the paper does not validate them against official checkpoints.
  • domain assumption No test-set contamination: the 2D BioMedCLIP slice features and pretraining were obtained without using the test reports of CT-RATE or RadGenome-ChestCT.
    The paper does not state how BioMedCLIP was trained relative to these datasets. If BioMedCLIP saw CT-RATE or RadGenome reports during its pretraining, the retrieval and report metrics could be inflated.
invented entities (2)
  • 2E3 Vision Encoder, a 3D encoder that uses 2D slice features to score and weight low-level 3D patches independent evidence
    purpose: To emphasize diagnostically relevant local 3D patches and improve fine-grained anatomical understanding.
    The entity is testable in the sense that its patch scores are visualized and its contribution is measured by ablation (Table 1b vs 1a, Table 3); it is a new learned module rather than an untestable postulate.
  • Spatial Packer with Voxel2Point Cross-Attention (V2P-CA) independent evidence
    purpose: To compress high-resolution 3D features into compact tokens while preserving spatial structure via centroid queries.
    The module is compared against Q-Former and pooling baselines (Table 2b, Table 5), so its benefit is falsifiable within the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understanding." pith.science (2026). https://pith.science/paper/SPDL2TKL

@misc{pith2026250609634,
  author       = {Pith},
  title        = {Pith review of: HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPDL2TKL}},
  note         = {Machine review of arXiv:2506.09634}
}
read the original abstract

Automated 3D CT diagnosis empowers clinicians to make timely, evidence-based decisions by enhancing diagnostic accuracy and workflow efficiency. While multimodal large language models (MLLMs) exhibit promising performance in visual-language understanding, existing methods mainly focus on 2D medical images, which fundamentally limits their ability to capture complex 3D anatomical structures. This limitation often leads to misinterpretation of subtle pathologies and causes diagnostic hallucinations. In this paper, we present Hybrid Spatial Encoding Network (HSENet), a framework that exploits enriched 3D medical visual cues by effective visual perception and projection for accurate and robust vision-language understanding. Specifically, HSENet employs dual-3D vision encoders to perceive both global volumetric contexts and fine-grained anatomical details, which are pre-trained by dual-stage alignment with diagnostic reports. Furthermore, we propose Spatial Packer, an efficient multimodal projector that condenses high-resolution 3D spatial regions into a compact set of informative visual tokens via centroid-based compression. By assigning spatial packers with dual-3D vision encoders, HSENet can seamlessly perceive and transfer hybrid visual representations to LLM's semantic space, facilitating accurate diagnostic text generation. Experimental results demonstrate that our method achieves state-of-the-art performance in 3D language-visual retrieval (39.85% of R@100, +5.96% gain), 3D medical report generation (24.01% of BLEU-4, +8.01% gain), and 3D visual question answering (73.60% of Major Class Accuracy, +1.99% gain), confirming its effectiveness. Our code is available at https://github.com/YanzhaoShi/HSENet.

Figures

Figures reproduced from arXiv: 2506.09634 by the authors.

Figure 1
Figure 1. Architecture of the proposed HSENet. The input 3D CT volume is processed in parallel by [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the dual-stage pretraining framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of 3D patch scores in 2E3 Vision [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of 3D CT report generation and medical VQA. Different colors in reports [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Experiments on the clinical effectiveness of VQA across different body locations. Colored [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Additional visualizations of 3D patch scores generated by the 2E3 Visual Encoder [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Visualization of medical volume-to-report retrieval. The 2E3 visual encoder [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Textual prompts for medical report generation follow the format of Bai et al. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Textual prompts for medical VQA follow the format of Zhang et al. [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 30 canonical work pages

  1. [1]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin...

    arXiv 2025

  2. [2]

    DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions

    Gorkem Can Ates, Kuang Gong, und Wei Shao. DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions. CoRR, abs/2502.05091, 2025

    arXiv 2025

  3. [3]

    M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

    Fan Bai, Yuxin Du, Tiejun Huang, Max Qinghu Meng, und Bo Zhao. M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models. CoRR, abs/2404.00578, 2024

    arXiv 2024

  4. [4]

    Shah, Andrew Johnston, Robert D

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Pontes Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hu...

    arXiv 2024

  5. [5]

    Bruno, Eric A

    Michael A. Bruno, Eric A. Walker, und Hani H. Abujudeh. Understanding and Confronting Our Mistakes: The Epidemiology of Error in Radiology and Strategies for Error Reduction. Radiographics : a review publication of the Radiological Society of North America, Inc, 35 6 S. 1668–76, 2015

    work page 2015

  6. [6]

    3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models

    Hao Chen, Wei Zhao, Yingli Li, Tianyang Zhong, Yisong Wang, Youlan Shang, Lei Guo, Junwei Han, Tianming Liu, Jun Liu, und Tuo Zhang. 3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models. CoRR, abs/2409.19330, 2024

    arXiv 2024

  7. [7]

    HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, und Benyou Wang. HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale. CoRR, abs/2406.19280, 2024

    arXiv 2024

  8. [8]

    Dragonfly: Multi-Resolution Zoom Supercharges Large Visual-Language Model

    Kezhen Chen, Rahul Thapa, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, und James Zou. Dragonfly: Multi-Resolution Zoom Supercharges Large Visual-Language Model. CoRR, abs/2406.00977, 2024

    arXiv 2024

Show all 52 references
  1. [9]

    MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts

    Qiuhui Chen, Xinyue Hu, Zirui Wang, und Yi Hong. MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts. CoRR, abs/2305.10799, 2023

    2023 arXiv

  2. [10]

    BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval

    Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, und Zhiwei Xiong. BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval. InMedical Image Computing and Computer Assisted Intervention, MICCAI 2024, volume 15011 ofLecture Notes in Computer Science, S. 124–134, 2024. 10

    2024

  3. [11]

    Dia-LLaMA: Towards Large Language Model-driven CT Report Generation

    Zhixuan Chen, Luyang Luo, Yequan Bie, und Hao Chen. Dia-LLaMA: Towards Large Language Model-driven CT Report Generation. CoRR, abs/2403.16386, 2024

    2024 arXiv

  4. [12]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, und Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2019

  5. [13]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, und Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at...

    2021

  6. [14]

    Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography

    Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esir- gun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Simsar, Mehmet Simsar, Emine Bensu Erdemir, Abdullah Alanbay, Anjany Kumar Sekuboyina, Berkan Lafci, Mehmet Kemal Ozdemir, u...

    2024

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, und Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    Ball, Norah Borus, Andrew Huang, Bhavik N

    Shih-Cheng Huang, Tanay Kothari, Imon Banerjee, Christopher Chute, Robyn L. Ball, Norah Borus, Andrew Huang, Bhavik N. Patel, Pranav Rajpurkar, Jeremy Irvin, Jared Dunnmon, Joseph Bledsoe, Katie S. Shpanskaya, Abhay Dhaliwal, Roham Zamanian, Andrew Y . Ng, und Matthew P. Lungr...

    2020

  9. [17]

    Lungren, und Serena Yeung

    Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, und Serena Yeung. GLoRIA: A Multi- modal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, S. 3922–3931, 2021

    2021

  10. [18]

    Baoyu Jing, Pengtao Xie, und Eric P. Xing. On the Automatic Generation of Medical Imaging Reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, S. 2577–2586, 2018

    2018

  11. [19]

    MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making

    Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeon- hoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, und Hae Won Park. MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making. In Advances in Neural Information Processing Systems 38...

    2024

  12. [20]

    Kevin Zhou

    Haoran Lai, Zihang Jiang, Qingsong Yao, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Wei Wei, Weifu Lv, und S. Kevin Zhou. E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model. CoRR, abs/2410.14200, 2024

    2024 arXiv

  13. [21]

    Kevin Zhou

    Haoran Lai, Zihang Jiang, Qingsong Yao, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Wei Wei, Weifu Lv, und S. Kevin Zhou. Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis. CoRR, abs/2501.03565, 2025

    2025

  14. [22]

    METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

    Alon Lavie und Abhaya Agarwal. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, WMT@ACL 2007, S. 228–231, 2007

    2007

  15. [23]

    Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation

    Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, Hritik Bansal, Ling Chen, Yi-Ping Yang, Yu-Chun Chen, Shih-Pin Chen, Jiing-Feng Lirng, Kai-Wei Chang, und Shih-Hwa Chiou. Towards a holistic framework for multimodal LLM in 3D brain CT radiology report gener...

    2024

  16. [24]

    LLaV A-Med: Training a Large Language-and- Vision Assistant for Biomedicine in One Day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, und Jianfeng Gao. LLaV A-Med: Training a Large Language-and- Vision Assistant for Biomedicine in One Day. In Annual Conference on Neural Information Processing System...

    2023

  17. [25]

    Junnan Li, Dongxu Li, Silvio Savarese, und Steven C. H. Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models. In International Conference on Machine Learning, ICML 2023, volume 202, S. 19730–19742, 2023

    2023

  18. [26]

    Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation

    Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, und Xiaojun Chang. Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation. In IEEE Conference on Computer Vision and Pattern Recognition, S. 3334–3343, 2023

    2023

  19. [27]

    TokenPacker: Efficient Visual Projector for Multimodal LLM

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, und Lei Zhang. TokenPacker: Efficient Visual Projector for Multimodal LLM. CoRR, abs/2407.02392, 2024

    2024 arXiv

  20. [28]

    Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, und Jiaya Jia. Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models. CoRR, abs/2403.18814, 2024

    2024 arXiv

  21. [29]

    Macro-and micro- anatomical, histological and computed tomography scan characterization of the nasopalatine canal

    Martens W Hu Y Adriaensens P Quirynen M Lambrichts I Liang X, Jacobs R. Macro-and micro- anatomical, histological and computed tomography scan characterization of the nasopalatine canal. Journal of clinical periodontology, 36(7) S. 598–603, 2009

    2009

  22. [30]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, S. 74–81, 2004

    2004

  23. [31]

    MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training

    Xuefeng Ni, Linshan Wu, Jiaxin Zhuang, Qiong Wang, Mingxiang Wu, Varut Vardhanabhuti, Lihai Zhang, Hanyu Gao, und Hao Chen. MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training. CoRR, abs/2412.05876, 2024

    2024 arXiv

  24. [32]

    Bleu: a Method for Auto- matic Evaluation of Machine Translation

    Kishore Papineni, Salim Roukos, Todd Ward, und Wei-Jing Zhu. Bleu: a Method for Auto- matic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL 2002, S. 311–318, 2002

    2002

  25. [33]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, und Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of th...

    2021

  26. [34]

    Salvolini, E

    L. Salvolini, E. B. Secchi, L. Costarelli, et al. Clinical applications of 2D and 3D CT imaging of the airways—a review. European journal of radiology, 34(1) S. 9–25, 2000

    2000

  27. [35]

    Time Is Money: Considerations for Measuring the Radiological Reading Time

    Raphael Sexauer und Caroline Bestler. Time Is Money: Considerations for Measuring the Radiological Reading Time. Journal of Imaging, 8, 2022

    2022

  28. [36]

    Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model

    Yiming Shi, Xun Zhu, Ying Hu, Chenyi Guo, Miao Li, und Ji Wu. Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model. CoRR, abs/2411.12783, 2024

    2024

  29. [37]

    Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, Karan Singhal, Mike Schaekermann, Rhys May, Roy Lee, SiWai Man, S. Sara Mahdavi, Zahra Ahmed, Yossi Matias, Joelle Barra...

    2024

  30. [38]

    Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, und Alexey Dosovitskiy. MLP-Mixer: An all-MLP Architecture for Vision. In Advances in Neural Info...

    2021

  31. [39]

    Fleet, Philip Andrew Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David J. Fleet, Philip Andrew Mansfield, Sushant Prakash, Renee Wong, Sunn...

    2023 arXiv

  32. [40]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, und Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  33. [41]

    Cross-modal prototype driven network for radiology report generation

    Jun Wang, Abhir Bhalerao, und Yulan He. Cross-modal prototype driven network for radiology report generation. In Computer Vision - ECCV 2022 - 17th European Conference, S. 563–579. Springer, 2022

    2022

  34. [42]

    XLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training

    Biao Wu, Yutong Xie, Zeyu Zhang, Minh Hieu Phan, Qi Chen, Ling Chen, und Qi Wu. XLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training. CoRR, abs/2407.19546, 2024

    2024 arXiv

  35. [43]

    Towards Generalist Foundation Model for Radiology

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, und Weidi Xie. Towards Generalist Foundation Model for Radiology. CoRR, abs/2308.02463, 2023

    2023 arXiv

  36. [44]

    Zou, und Huaxiu Yao

    Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James ...

    2024

  37. [45]

    Med3DVLM: An Efficient Vision- Language Model for 3D Medical Image Analysis

    Yu Xin, Gorkem Can Ates, Kuang Gong, und Wei Shao. Med3DVLM: An Efficient Vision- Language Model for 3D Medical Image Analysis. volume arXiv:2503.20047, 2025

    2025

  38. [46]

    Sigmoid Loss for Language Image Pre-Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, und Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, S. 11941–11952, 2023

    2023

  39. [47]

    Lungren, Tristan Naumann, Sheng Wang, und Hoifung Poon

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P....

    2024

  40. [48]

    Weinberger, und Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, und Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, 2020

    2020

  41. [49]

    MEPNet: Medical Entity-Balanced Prompting Network for Brain CT Report Generation

    Xiaodan Zhang, Yanzhao Shi, Junzhong Ji, Chengxin Zheng, und Liangqiong Qu. MEPNet: Medical Entity-Balanced Prompting Network for Brain CT Report Generation. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, S. 25940–25948, 2025

    2025

  42. [50]

    RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, und Weidi Xie. RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis. CoRR, abs/2404.16754, 2024

    2024 arXiv

  43. [51]

    There are emphysematous changes in both lungs

    Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, und Weidi Xie. RaTEScore: A Metric for Radiology Report Generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, S. 15004–15019, 2024. 13 A Visual Token Compr...

    2024

  44. [52]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.