Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A medical vision-language pre-training method adds disease-level contrastive learning and a VQ-VAE normality model to CT scan-report alignment, and reports state-of-the-art zero-shot diagnosis with an average AUC of 84.9% across 54…

desk verdict A real empirical advance in CT vision-language pre-training, with clearly ablated components and a genuine gap in validating the LLM-derived normality labels. read the letter →

arxiv 2508.03742 v1 pith:JZA6KMN5 submitted 2025-08-01 eess.IV cs.AIcs.CVcs.LG

Weiwei Cao , Jianpeng Zhang , Zhongyi Shui , Sinuo Wang , Zeli Chen , Xi Li , Le Lu , Xianghua Ye
show 3 more authors
Tingbo Liang Qi Zhang Ling Zhang
This is my paper
classification eess.IVcs.AIcs.CVcs.LG
keywords vision-languagepre-trainingcomputedtomographysemanticdensitydisease-levelcontrastivelearningnormalitymodelinganomalydetectionzero-shotdiagnosisradiologyreportgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that medical vision-language pre-training fails to zero-shot diagnose CT scans because the images carry far fewer diagnostic signals per token than the paired reports, a gap called low vision semantic density. It introduces ViSD-Boost, which first contrasts normal and abnormal anatomy embeddings and then models the distribution of healthy anatomies with a VQ-VAE so that reconstruction error amplifies disease cues. On three CT benchmarks, CT-RATE, Rad-ChestCT, and MedVL-CT69K, ViSD-Boost reports higher zero-shot AUC than existing VLP methods, including an average AUC of 84.9% across 54 diseases and 15 organs on the abdominal benchmark. A sympathetic reader would care because the result suggests that report-supervised pre-training can reach diagnostic accuracy without manually labeled lesion annotations.

What carries the argument

The load-bearing component is the anatomical normality model, a VQ-VAE that operates on anatomy-conditioned visual tokens in latent space. A Transformer encoder maps the vision tokens of each organ to a discrete codebook of 100 prototype vectors per anatomy, updated with exponential moving average; a decoder reconstructs the tokens from the nearest codebook vectors. Because only samples labeled normal by the LLM update the codebook, abnormal anatomy tokens reconstruct poorly and their reconstruction error marks the disease-relevant signal. An MLP 'abnormality perceiver' concatenates the original embedding with the reconstruction and emits the final vision token used in the contrastive alignment of Eq. 1. The disease-level contrastive loss of Eq. 2, applied before pre-training, supplies the normality/abnormality separation that makes the reconstruction error meaningful.

What would settle it

Take a random sample of MedVL-CT69K or CT-RATE reports, have a radiologist label each of the 15 organs as normal or abnormal, and compare those labels with Qwen's; if agreement falls below roughly 90% (or if a variant trained on the radiologist labels does not beat ViSD-Boost), the LLM-label premise fails. A second concrete check: remove all samples where Qwen says 'normal' but the report text names a disease, retrain, and see whether the 84.9% AUC drops significantly.

Watch

Extended reading notes

Core claim

The central claim is that a semantic density gap, not raw alignment difficulty, explains why general vision-language models underperform on CT: a tiny disease region sits inside an overwhelmingly large normal anatomy, and global alignment washes it out. The paper shows that two additions to a fine-grained anatomy-wise alignment framework close most of that gap: (1) disease-level contrastive learning which pulls normal samples of the same organ together while keeping abnormal samples distinct, and (2) anatomical normality modeling, a VQ-VAE trained on normal anatomy tokens that reconstructs a healthy reference embedding so the residual between the original and the reconstruction concentrates on the abnormal region. An abnormality perceiver MLP fuses the original and reconstructed tokens before vision-language alignment, and the resulting model reaches 79.0% AUC on CT-RATE, 69.4% on Rad-ChestCT, and 84.9% on MedVL-CT69K, ahead of the previous fVLM baseline by 1.2, 1.4, and 3.6 percentage points respectively.

Load-bearing premise

Everything downstream rests on the LLM's organ-level normal/abnormal labels being accurate; if the LLM frequently labels a diseased organ as normal, those abnormal samples enter the normality model, and the reconstruction-error signal that drives the method's gains is diluted or corrupted.

Editorial extensions

If this is right

  • Zero-shot diagnosis becomes viable for organs and diseases that never appear in supervised training labels, because the normality prior is learned from healthy anatomy alone.
  • The pre-trained vision encoder transfers to radiology report generation, raising clinical F1 and natural-language metrics, without any task-specific visual supervision.
  • Disease-level contrastive learning separates normal from abnormal anatomy while preserving inter-abnormality differences, which should make the representation useful for fine-grained disease classification.
  • The method is portable to any CT dataset where organ segmentation and report parsing are available, including the chest CT benchmarks where it improves on the strong fVLM baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never measures the accuracy of the LLM-generated organ-level normal/abnormal labels; if the LLM's labels are noisy in a biased way, the reported gains may partly reflect label noise acting as regularization rather than true normality modeling. A radiologist-annotated subset would settle this.
  • The normality codebook is trained only on samples the LLM calls normal, so any mislabeled diseased organ contaminates the healthy distribution; a test could deliberately inject known abnormal samples and measure how AUC degrades.
  • The same normality model could run as a standalone organ-level anomaly detector, offering a testable bridge from report-supervised pre-training to unsupervised screening.
  • The sparse-activation effect reported after VSDB suggests the model's attention is more localized; one could verify this by lesion-localization or region-prompted diagnosis, not just zero-shot AUC.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes ViSD-Boost, a vision-language pre-training method for CT images that builds on the fVLM anatomy-wise alignment framework and adds two components: (1) disease-level contrastive learning (Eq. 2) that pulls normal samples of the same organ together while separating abnormal samples, using per-organ normal/abnormal labels extracted from radiology reports by the LLM Qwen; and (2) anatomical normality modeling with a VQ-VAE (Eq. 3) trained only on normal samples, where an MLP perceiver amplifies reconstruction errors as abnormal signals. The model is evaluated on zero-shot diagnosis on CT-RATE, Rad-ChestCT, and MedVL-CT69K, reporting state-of-the-art AUCs (e.g., 84.9% over 54 diseases in 15 organs on MedVL-CT69K), as well as on radiology report generation and linear-probing classification.

Significance. If the results hold, the paper makes a useful empirical contribution by showing that explicitly modeling the normal distribution of each anatomy with a VQ-VAE and adding disease-level contrastive learning can improve zero-shot CT diagnosis over a strong fVLM baseline. The experiments are extensive, spanning two chest CT datasets and one large abdominal CT dataset, and the code is publicly released. The ablations in Table 5 support the contribution of each module, with a 6% AUC gain over the fVLM-style baseline on the MedVL-CT69K validation set. However, the significance is currently tempered by the lack of validation of the LLM-generated abnormality labels and the absence of statistical confidence measures for the headline gaps; these issues are central to the claimed mechanisms and are addressed in the major comments.

major comments (3)
  1. [§3.2, Eq. (2) and Eq. (3)] The per-organ normal/abnormal labels y from Qwen are the gating signal for both novel losses, but the paper provides no validation of these labels, no prompt details, and no statement on how organs not mentioned in the report are labeled. This is load-bearing: if label noise is high (e.g., unmentioned organs defaulting to normal), the 'normal' codebook in Eq. (3) contains abnormal samples, undermining the reconstruction-error signal that VSDB is claimed to amplify. The paper should report label agreement (e.g., on a sample against radiologist annotation or against the text-classifier-based disease labels), specify the prompt and the default rule for unmentioned organs, and ideally ablate with a label-noise model.
  2. [§4.3, Tables 1–2, abstract] The claim 'significantly surpassing existing methods' is not supported by any statistical analysis. All results appear to be from a single run with no error bars or significance tests. The 1.2–1.4% gaps on the chest datasets are small; without variance estimates or multiple seeds, we cannot rule out random variation. Provide means ± std over at least three seeds or conduct a significance test for the key comparisons.
  3. [Table 1 footnote] The exclusion of 'lymphadenopathy' and 'medical material' from the chest comparison is not justified; if these entities are present in the evaluation protocol of the baselines but excluded for our method, the comparison is biased. Please report results with and without these entities, or justify the exclusion by a pre-registered criterion.
minor comments (7)
  1. [Throughout] 'VQ-V AE' appears with irregular spacing throughout; please use the standard 'VQ-VAE' notation.
  2. [§4.3] 'VisD-Boost' is a typo; it should be 'ViSD-Boost'.
  3. [References] References [37] and [38] are duplicates (both are Van Den Oord et al., Neural Discrete Representation Learning); consolidate them.
  4. [Eq. (2)] The notation Iyi,j in Eq. (2) is missing a space and the denominator is ambiguous: please clarify that the inner sum over k runs over all samples in the batch, and define the indices precisely.
  5. [§3.3.2] The sentence 'qI i,j represents the reconstructed embedding of the original embedding f I i,j' is confusing; it should state that qI i,j is the VQ-VAE reconstruction of f I i,j.
  6. [Table 5 caption] The abbreviations 'AA V' and 'AAC' in the Table 5 caption are not defined; please expand them in the caption.
  7. [Figure 2] Figure 2 is too small to read; the text and arrows are difficult to follow. Consider enlarging or splitting the figure.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the contrastive and VQ-VAE losses are trained on training-set labels and evaluated on held-out test sets; self-citations to fVLM/MedVL-CT69K are baseline infrastructure, not forced evidence.

full rationale

The paper's derivation chain is an empirical training recipe rather than a closed-form mathematical derivation. The two novel losses (Eq. 2 disease-level contrastive learning and Eq. 3 VQ-VAE normality modeling) use organ-level normal/abnormal labels y obtained from Qwen on training reports; these labels are a supervision source, not a quantity derived from the final zero-shot AUC. The final evaluation is on held-out test sets (CT-RATE, Rad-ChestCT, MedVL-CT69K test split), so the reported numbers are not fitted by construction. The self-citations to fVLM [35] provide the anatomy-wise alignment backbone and the MedVL-CT69K benchmark, and the paper compares against fVLM as a baseline rather than invoking it as proof of the improvement; external chest CT benchmarks give independent support. The unvalidated Qwen labels are a correctness/robustness risk (mislabeled 'normal' organs could contaminate the normality codebook), but this is not circularity because the reconstruction-error signal is not fitted to the disease labels. No equation reduces to its own inputs, and no fitted parameter is renamed as a prediction. Hence no significant circularity; score 2 reflects the presence of self-citation for the base pipeline and benchmark without load-bearing circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on pseudo-labels from a proprietary LLM (Qwen), segmentation quality, and the assumption that abnormal samples produce detectable reconstruction errors. These are external dependencies that the paper treats as ground truth without validation.

free parameters (5)
  • Temperature tau in contrastive losses = not reported
    Appears in Eq. 1 and Eq. 2; controls the sharpness of similarity distributions. Not specified whether fixed or learned.
  • VQ-VAE loss weight beta = 0.25
    Default balancing factor in Eq. 3; affects codebook learning versus reconstruction.
  • Codebook size K and dimension C = K=100, C=1024
    Hyperparameters of the discrete codebook, chosen by the authors.
  • LLM prompt for normal/abnormal labeling = not shown
    The exact prompt given to Qwen determines the pseudo-labels y, which drive the contrastive and VQ-VAE losses. This is a hidden design choice.
  • MLP perceiver architecture = not specified
    The abnormality perceiver that fuses original and reconstructed embeddings is described only as an MLP; layer sizes are omitted.
assumptions (6)
  • domain assumption Diagnostic reports accurately describe per-organ normality and abnormality.
    Invoked in Sec 3.2 when Qwen labels each organ as normal or abnormal. If reports are incomplete or inaccurate, the pseudo-labels are corrupted.
  • domain assumption TotalSegmentator accurately segments the 104 anatomical structures, and grouping into 36 anatomies preserves diagnostic information.
    Used in Sec 4.2 for anatomy parsing. Segmentation errors would misalign image and text tokens.
  • domain assumption Qwen accurately decomposes reports into anatomy-specific descriptions.
    Used in Sec 3.1-3.2 to generate structured reports and labels; no validation of the LLM outputs is provided.
  • domain assumption Abnormal samples show detectable distribution shifts in the latent space used by the VQ-VAE.
    Core premise of Sec 3.3: reconstruction error on abnormal inputs is assumed to correlate with disease presence.
  • domain assumption The evaluation text classifier accurately identifies the 54 disease entities.
    Sec 4.2 relies on this classifier to measure report generation and multi-disease classification; its accuracy is not reported.
  • ad hoc to paper Contrastive loss formulation in Eq. 2 behaves as intended (normal samples cluster, abnormal samples remain distinct).
    The unusual sum-of-logs objective for normal samples is not theoretically justified and may be unstable; no analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training." pith.science (2026). https://pith.science/paper/JZA6KMN5

@misc{pith2026250803742,
  author       = {Pith},
  title        = {Pith review of: Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JZA6KMN5}},
  note         = {Machine review of arXiv:2508.03742}
}
read the original abstract

Vision-language pre-training (VLP) has great potential for developing multifunctional and general medical diagnostic capabilities. However, aligning medical images with a low signal-to-noise ratio (SNR) to reports with a high SNR presents a semantic density gap, leading to visual alignment bias. In this paper, we propose boosting vision semantic density to improve alignment effectiveness. On one hand, we enhance visual semantics through disease-level vision contrastive learning, which strengthens the model's ability to differentiate between normal and abnormal samples for each anatomical structure. On the other hand, we introduce an anatomical normality modeling method to model the distribution of normal samples for each anatomy, leveraging VQ-VAE for reconstructing normal vision embeddings in the latent space. This process amplifies abnormal signals by leveraging distribution shifts in abnormal samples, enhancing the model's perception and discrimination of abnormal attributes. The enhanced visual representation effectively captures the diagnostic-relevant semantics, facilitating more efficient and accurate alignment with the diagnostic report. We conduct extensive experiments on two chest CT datasets, CT-RATE and Rad-ChestCT, and an abdominal CT dataset, MedVL-CT69K, and comprehensively evaluate the diagnosis performance across multiple tasks in the chest and abdominal CT scenarios, achieving state-of-the-art zero-shot performance. Notably, our method achieved an average AUC of 84.9% across 54 diseases in 15 organs, significantly surpassing existing methods. Additionally, we demonstrate the superior transfer learning capabilities of our pre-trained model. Code is available at https://github.com/alibaba-damo-academy/ViSD-Boost.

Figures

Figures reproduced from arXiv: 2508.03742 by the authors.

Figure 1
Figure 1. (a) Illustration of semantic density gap between vision [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of the proposed ViSD-Boost. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. T-SNE visualization of normal (blue) and abnormal (red) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visual activation maps generated by the proposed method for diagnosing six different diseases. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Vision semantic density comparison between models w/ [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Vision semantic density comparison between models w/ and w/o VSDB. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ORCA: ORgan-Centroid Aggregation for Training-Free 3D CT Visual Token Compression

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ORCA compresses 3D CT tokens into organ-guided connected regions with sinusoidal centroid encoding, outperforming grid average and other compressors at matched budgets.

  2. Cheap Probes Predict Expensive Training in 3D-CT Vision--Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Disease-probe AUROC on frozen 3D-CT tokens predicts report-generation clinical micro-F1 across encoder×compression cells at r=0.95, ρ=0.89 (six cells, preliminary, one dataset).

Reference graph

Works this paper leans on

51 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    Medical image segmentation review: The suc- cess of u-net

    Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof. Medical image segmentation review: The suc- cess of u-net. IEEE TPAMI, 2024. 1

    work page 2024

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 3, 4

    arXiv 2023

  3. [3]

    Merlin: A vision language foun- dation model for 3d computed tomography

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foun- dation model for 3d computed tomography. arXiv preprint arXiv:2406.06512, 2024. 2, 5, 6, 7

    arXiv 2024

  4. [4]

    A review of the appli- cation of deep learning in medical image classification and segmentation

    Lei Cai, Jingyang Gao, and Di Zhao. A review of the appli- cation of deep learning in medical image classification and segmentation. Annals of translational medicine, 8(11), 2020. 1

    work page 2020

  5. [5]

    Bootstrapping chest ct image understanding by distilling knowledge from x-ray expert models

    Weiwei Cao, Jianpeng Zhang, Yingda Xia, Tony CW Mok, Zi Li, Xianghua Ye, Le Lu, Jian Zheng, Yuxing Tang, and Ling Zhang. Bootstrapping chest ct image understanding by distilling knowledge from x-ray expert models. In CVPR, pages 11238–11247, 2024. 2, 3, 5, 6, 7

    work page 2024

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 3

    work page 2020

  7. [7]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pages 104–120. Springer, 2020. 2

    work page 2020

  8. [8]

    Align, rea- son and learn: Enhancing medical vision-and-language pre- training with knowledge

    Zhihong Chen, Guanbin Li, and Xiang Wan. Align, rea- son and learn: Enhancing medical vision-and-language pre- training with knowledge. In ACM MM, pages 5152–5161,

    work page

Show all 51 references
  1. [9]

    Masked image modeling advances 3d medical image analysis

    Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Mariann Micsinai Balan, and Kevin Brown. Masked image modeling advances 3d medical image analysis. InPro- ceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 1970–1980, 2023. 6, 7

    1970

  2. [10]

    Prior: Prototype representation joint learning from medical images and reports

    Pujin Cheng, Li Lin, Junyan Lyu, Yijin Huang, Wenhan Luo, and Xiaoying Tang. Prior: Prototype representation joint learning from medical images and reports. In ICCV, pages 21361–21371, 2023. 2

    2023

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  4. [12]

    Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes

    Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnor- mality prediction with large-scale chest computed tomogra- phy volumes. Medical image analysis, 67:101857, 2021. 2, 5

    2021

  5. [13]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, pages 12175–12185, 2022. 4

    2022

  6. [14]

    A foundation model utiliz- ing chest ct volumes and radiology reports for supervised- level zero-shot detection of abnormalities

    Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Sim- sar, Mehmet Simsar, et al. A foundation model utiliz- ing chest ct volumes and radiology reports for supervised- level ze...

    2024

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 3

    2016

  8. [16]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In CVPR, pages 9729–9738, 2020. 4

    2020

  9. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 3

    2022

  10. [18]

    Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition

    Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local represen- tation learning framework for label-efficient medical image recognition. In ICCV, pages 3942–3951, 2021. 2

    2021

  11. [19]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Pe- tersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation. Nature methods, 18(2):203–211, 2021. 1

    2021

  12. [20]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. NeurIPS, 34:9694–9705,

  13. [21]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 5, 6

    2022

  14. [22]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, pages 10965–10975,

  15. [23]

    Dynamic graph enhanced contrastive learning for chest x-ray report generation

    Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiao- dan Liang, and Xiaojun Chang. Dynamic graph enhanced contrastive learning for chest x-ray report generation. In CVPR, pages 3334–3343, 2023. 2

    2023

  16. [24]

    Anatomical structure-guided medical vision- language pre-training

    Qingqiu Li, Xiaohan Yan, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, and Shu- jun Wang. Anatomical structure-guided medical vision- language pre-training. arXiv preprint arXiv:2403.09294 ,

    arXiv

  17. [25]

    An organ-aware diagnosis framework for radiology report generation

    Shiyu Li, Pengchong Qiao, Lin Wang, Munan Ning, Li Yuan, Yefeng Zheng, and Jie Chen. An organ-aware diagnosis framework for radiology report generation. IEEE Transac- tions on Medical Imaging, 2024. 3

    2024

  18. [26]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR, pages 23390–23400, 2023. 1, 2, 3

    2023

  19. [27]

    Ct-glip: 3d grounded language- image pretraining with ct scans and radiology reports for full-body scenarios

    Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Le Lu, Jiebo Luo, and Ling Zhang. Ct-glip: 3d grounded language- image pretraining with ct scans and radiology reports for full-body scenarios. arXiv preprint arXiv:2404.15272, 2024. 2

    2024

  20. [28]

    Imitate: Clinical prior guided hierarchical vision-language pre-training

    Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, and Rossella Arcucci. Imitate: Clinical prior guided hierarchical vision-language pre-training. arXiv preprint arXiv:2310.07355, 2023. 6

    2023 arXiv

  21. [29]

    Bootstrapping large language models for radiology report generation

    Chang Liu, Yuanhe Tian, Weidong Chen, Yan Song, and Yongdong Zhang. Bootstrapping large language models for radiology report generation. In AAAI, pages 18635–18643,

  22. [30]

    Joint learning of localized representations from medical images and reports

    Philip M ¨uller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. Joint learning of localized representations from medical images and reports. In ECCV, pages 685–701. Springer, 2022. 2, 3, 5, 6

    2022

  23. [31]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 4

    2018 arXiv

  24. [32]

    Green: Generative radiology report eval- uation and error notation

    Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report eval- uation and error notation. arXiv preprint arXiv:2405.03595,

    arXiv

  25. [33]

    Deep learning for anomaly detection: A review

    Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1–38, 2021. 4

    2021

  26. [34]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. pages 8748–8763. PMLR, 2021. 1, 2, 5, 6, 7

    2021

  27. [35]

    Large-scale and fine-grained vision-language pre-training for enhanced ct image under- standing

    Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, and Ling Zhang. Large-scale and fine-grained vision-language pre-training for enhanced ct image under- standing. In ICLR, 2025. 2, 3, 5, 6, 7, 1

    2025

  28. [36]

    Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning

    Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, An- drew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self- supervised learning. Nature Biomedical Engineering, 6(12): 1399–1406, 2022. 2

    2022

  29. [37]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 4

    2017

  30. [38]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017. 2

    2017

  31. [39]

    Attention is all you need

    A Vaswani. Attention is all you need. NeurIPS, 2017. 4

    2017

  32. [40]

    Multi-granularity cross-modal align- ment for generalized medical visual representation learning

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanab- huti, and Lequan Yu. Multi-granularity cross-modal align- ment for generalized medical visual representation learning. NeurIPS, 35:33536–33549, 2022. 2, 3, 6

    2022

  33. [41]

    To- talsegmentator: robust segmentation of 104 anatomic struc- tures in ct images

    Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. To- talsegmentator: robust segmentation of 104 anatomic struc- tures in ct images. Radiology: Artificial In...

  34. [42]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In CVPR, pages 21372–21383, 2023. 2

    2023

  35. [43]

    Multimodal chatgpt for medical appli- cations: an experimental study of gpt-4v

    Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun. Multimodal chatgpt for medical appli- cations: an experimental study of gpt-4v. arXiv preprint arXiv:2310.19061, 2023. 1

    2023 arXiv

  36. [44]

    Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre- training on chest radiology images.Nature Communications, 14(1):4542, 2023. 2

    2023

  37. [45]

    When radiology report genera- tion meets knowledge graph

    Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan Yuille, and Daguang Xu. When radiology report genera- tion meets knowledge graph. In AAAI, pages 12910–12917,

  38. [46]

    Advancing radiograph representation learning with masked record modeling

    Hong-Yu Zhou, Chenyu Lian, Liansheng Wang, and Yizhou Yu. Advancing radiograph representation learning with masked record modeling. arXiv preprint arXiv:2301.13155,

    arXiv

  39. [47]

    Large model driven radiology report generation with clinical quality reinforcement learn- ing

    Zijian Zhou, Miaojing Shi, Meng Wei, Oluwatosin Alabi, Zi- jie Yue, and Tom Vercauteren. Large model driven radiology report generation with clinical quality reinforcement learn- ing. arXiv preprint arXiv:2403.06728, 2024. 3 Boosting Vision Semantic Density with Anatomy Normal...

    2024 arXiv

  40. [48]

    Variety in visual encoder selection In the 3D CT VLP task, we discover that the CNN visual encoder outperforms the ViT

    More ablation studies 6.1. Variety in visual encoder selection In the 3D CT VLP task, we discover that the CNN visual encoder outperforms the ViT. Consequently, we explore the impact of various CNN backbones on model performance. As illustrated in Table 6, both ResNet34 and Re...

  41. [49]

    More implementation details For the MedVL-CT69K dataset, we utilize the pre-trained BERT-base [11] as the text encoder. Our ViSD-boost is trained with the Adam optimizer, where the learning rate in- creases linearly to 1e-4 in the first epoch and then decreases gradually to 1e...

  42. [50]

    More visualizations of semantic density We present the distributions of visual tokens across addi- tional anatomical structures, as illustrated in Figure 6. The figure clearly demonstrates that, for all organs, the visual tokens of the model exhibit increased sparsity after th...

  43. [51]

    Details about zero-shot performance Table 10 displays the zero-shot performance of the pro- posed method across 54 abnormalities spanning 15 distinct anatomies. Type Diseases Local kidney cyst, kidney stone, adrenal gland nodule, stomach cancer, gallstone, pancreatic cancer, s...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.