Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adapting a masked-modeling vision encoder with 8% trainable parameters outperforms full contrastive medical vision-language pretraining.

desk verdict Solid, credible paper showing a frozen masked radiograph encoder can be cheaply aligned to text; the main fix is an explicit inference protocol for absent temporal/multiview slots. read the letter →

arxiv 2506.08990 v1 pith:2AUURWZE submitted 2025-06-10 cs.CV cs.AIcs.LG

Chenyu Lian , Hong-Yu Zhou , Dongyun Liang , Jing Qin , Liansheng Wang This is my paper
classification cs.CVcs.AIcs.LG
keywords medicalvision-languagealignmentparameter-efficientadaptationmaskedrecordmodelingradiographrepresentationlearningtemporal-multiviewchestX-rayscontrastivezero-shotclassificationmultimodalretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ALTA (ALign Through Adapting) is a parameter-efficient method for aligning chest-radiograph images with radiology reports. It takes a vision encoder that was pretrained by masked record modeling -- restoring masked image patches and masked report tokens -- and leaves it frozen, inserting small trainable adapter modules into both the vision and language transformers. Training then applies global and local contrastive losses between radiograph and report embeddings, plus masked-language and masked-image losses to preserve the encoder's representational strengths, using only about 8% of the parameters and less than one-fifth of the compute of the original masked-modeling pretraining. The paper reports that this adapted model outperforms the best baseline by over 4 absolute points in text-to-image retrieval and about 6 absolute points in image-to-text retrieval, and that it also improves zero-shot classification and downstream vision and language understanding. A second ingredient is restructuring training records as temporal-multiview quaternions (current and prior studies, frontal and lateral views), which the paper argues makes radiographs and reports informationally consistent.

What carries the argument

The machinery is a frozen transformer pair plus trainable adapters. Each adapter is a down-projection to a low dimension, a GELU nonlinearity, and an up-projection, inserted after the multi-head self-attention and in parallel with the feed-forward network in every block of both the vision encoder and the language processor. The input record is a quaternion of four radiographs -- current frontal, current lateral, prior frontal, prior lateral -- with missing views zero-filled; learnable temporal-view embeddings and fixed positional embeddings are added, and random masking of 75% of image patches and 50% of report tokens provides efficiency and supervision. Global and local alignment use InfoNCE; the total objective is $L = L_{\mathrm{GLOBAL}} + \lambda_1 L_{\mathrm{LOCAL}} + \lambda_2 L_{\mathrm{MLM}} + \lambda_3 L_{\mathrm{MIM}}$, where the MLM and MIM terms keep the frozen masked-modeling behavior alive during alignment. The vision encoder and image decoder stay frozen throughout, so the only trained parameters are the adapters and projection heads.

What would settle it

Run ALTA on a single-image benchmark in two evaluation modes: zero-filling the three missing quaternion slots versus feeding only the available image by excluding absent slots from the encoder. If the zero-filled mode loses several accuracy points relative to the alternative, or if a model trained with only current-frontal images matches the full quaternion model when both are evaluated on single images, then part of the reported gain is attributable to the zero-filling shortcut rather than to temporal-multiview alignment. A second check is to train ALTA without the masked-language and masked-image losses but with the same contrastive losses and quaternion inputs; if retrieval performance does not drop, those maintenance losses are not load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that mask-based pretraining and cross-modal alignment do not have to be fused into one objective. Starting from a frozen masked-record-modeling vision encoder and a frozen biomedical language encoder, the authors add trainable bottleneck adapters inside each transformer block and optimize a sum of four losses: a global InfoNCE contrastive loss between a pooled vision embedding and the report's [CLS] embedding; a local contrastive loss between view-local image patch embeddings and report token embeddings; a masked language modeling loss in which hybrid vision-language embeddings reconstruct masked report tokens; and a masked image modeling loss that restores masked patches of the current frontal radiograph. Records are reorganized into a temporal-multiview quaternion $(I_{\mathrm{cf}}, I_{\mathrm{c\ell}}, I_{\mathrm{pf}}, I_{\mathrm{p\ell}})$ so that prior studies and lateral views contribute to the aligned representations. If the paper is right, the contradiction between contrastive image-text pretraining (good matching, weaker visual features) and masked modeling (strong visual features, no matching) is resolved by keeping the masked-modeling weights intact and aligning through adapters alone.

Load-bearing premise

The load-bearing premise is that a radiograph study can always be represented as a fixed quaternion of current, prior, frontal, and lateral images, with absent views filled by zero tensors, and that at inference the same zero-filling works for benchmarks that supply only a single image; the paper defines the zero-filling rule for training but never states how single-image evaluation is converted into the quaternion, and the effect of injecting zero patches into the shared encoder is not analyzed.

Editorial extensions

If this is right

  • Vision-language alignment can be decoupled from pretraining: masked-modeling encoders that have no cross-modal matching ability can be turned into strong retrieval and zero-shot models by training only adapter modules.
  • Temporal-multiview consistency matters for medical vision-language: including prior and lateral radiographs in the same record as the report improves not only cross-modal retrieval but also temporal and nontemporal language understanding.
  • The aligned encoder remains a strong visual encoder: after alignment, the adapted vision model roughly matches or exceeds the original masked-modeling encoder on downstream radiograph classification, so alignment does not destroy visual representations.
  • Parameter-efficient alignment is competitive with full fine-tuning: adapter training and low-rank adaptation both outperform full-model training in the reported retrieval tasks, suggesting that catastrophic forgetting in the frozen foundation models is reduced.
  • The method scales: applying the same adapter-based alignment to a larger masked vision encoder gives retrieval results comparable to the base model, so the efficiency gains are not tied to one architecture size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would separate the contribution of the temporal-multiview quaternion from the contribution of adapter-based adaptation by training the same adapter pipeline on records with only a single current frontal image and on records with four images but no temporal-view embeddings; the current ablations remove inputs but keep the rest fixed, so an independent factorial design could qua
  • The zero-filling rule implies that the encoder sees zero patches for absent views; if zeros behave like a learned 'missing' token after adaptation, the same trick could transfer to other multimodal settings with optional views, but that transfer is not studied in the paper.
  • Because the language model is also frozen and adapted, the alignment stage could in principle be re-run at much lower cost when a better text encoder becomes available, rather than repeating masked pretraining.
  • The reported robustness to time-interval variation suggests the method may be ready for longitudinal screening scenarios, but the restructured training records cover only a subset of the full chest radiograph database; whether the gains persist with more diverse or incomplete input regimes is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ALTA, a parameter-efficient method that aligns an MRM-pretrained, frozen vision encoder with a frozen biomedical language encoder using trainable adapters. The training objective combines global and local contrastive losses with MLM and MIM maintenance losses, and the input representation is a temporal-multiview radiograph quaternion (current frontal, current lateral, prior frontal, prior lateral) with zero-filling for missing views. On CheXpert 8x200, CheXpert 5x200, and RSNA benchmarks, the authors report consistent improvements over CLIP-based baselines in retrieval and zero-shot classification, with ablation studies on input modalities, loss functions, masking ratios, and parameter-efficient modules. The central efficiency claim is about 8% trainable parameters and less than one-fifth of MRM pretraining compute.

Significance. If the results hold, ALTA is a practically useful contribution: it shows that a masked-model vision encoder can be aligned to language without full multimodal pretraining, and the temporal-multiview record restructuring is a reasonable and well-motivated design choice. The experimental breadth is a strength: three retrieval tasks, two zero-shot classification tasks, two language-understanding tasks, and extensive ablations, with public code provided. The main reservations concern an unspecified inference protocol for single-image benchmarks, unreported loss weights in the multi-task objective, and an unsubstantiated compute comparison; these issues currently prevent full confidence in the headline numbers.

major comments (3)
  1. [III-B, IV-K] The evaluation protocol for absent temporal/multiview slots is not specified. Training zero-fills missing slots (Section III-B), and Eq. (4a) averages the [CLS] embeddings over all four slots while Eq. (4c) concatenates local tokens from all four slots, so zero-filled radiographs contribute a zero-image CLS embedding and zero-patch tokens to both global and local representations. Section IV-K only states that all image patches are kept for retrieval and classification; it does not state whether a single CheXpert or RSNA image is mapped to the quaternion by zero-filling, by duplication, or by using only the current frontal slot. Because the headline 4-6% retrieval gains are measured on these single-image benchmarks, an undocumented mapping choice can materially affect P@k values. Please specify the inference mapping and include a sensitivity analysis over alternative mappings.
  2. [IV-K, Abstract] The claim that ALTA requires 'less than 1/5 of the computational consumption' of MRM pretraining is not substantiated. Section IV-K reports that ALTA training takes about 10 hours on four GPUs, but no comparable measurement for MRM pretraining is provided, and the comparison basis (number of epochs, data size, hardware, and whether the same masking ratio is used) is not defined. Please provide a direct GPU-hour comparison or clearly qualify the claim as relative to the authors' own MRM setup with the missing measurement supplied.
  3. [III-D, Eq. (8)] The loss weights lambda_1, lambda_2, and lambda_3 in Eq. (8) are never reported. The ablation study in Table VI varies each loss on or off, but the actual numerical weights in the combined objective are absent, and multi-task weighting can be consequential for the balance between global contrastive alignment, local alignment, MLM, and MIM. The values should be reported, and a brief sensitivity analysis over at least a small range of weights would strengthen reproducibility and the claim that the chosen objective is robust.
minor comments (5)
  1. [Table I] In the Random row, the text-to-image P@5 value is printed as '12.5.' with a stray period; also, the BioViL row shows identical numbers for text-to-image P@5/P@10 and image-to-text P@5/P@10, which looks like a copy-paste artifact and should be checked.
  2. [Section IV-D] The dataset name is written as 'RNSA Pneumonia' instead of 'RSNA Pneumonia'; the same section uses 'pertaining' where 'pretraining' is intended.
  3. [Section IV-K] The text contains a typo 'tine-tuning' instead of 'fine-tuning', and 'Biovil' capitalization is inconsistent in Table II.
  4. [Table V] The row for GLoRIA and the row for GLoRIA-ViT are run together as '67.73GLoRIA-ViT' with no line break, making the table hard to read.
  5. [IV-B, IV-D] Retrieval results in Tables I and VI are reported without standard deviations or seed information; a paired permutation test is mentioned in the text, but the details of what is permuted and how many independent runs are used are not given. Reporting at least the permutation-test setup would improve the statistical grounding.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: ALTA's retrieval and classification gains are measured on held-out CheXpert and RSNA benchmarks, and the MRM backbone is re-evaluated in Table III rather than taken on faith.

full rationale

The paper's central claims (4-6% retrieval gains and zero-shot classification improvements) are empirical results on held-out CheXpert 8x200, CheXpert 5x200, and RSNA test sets; no parameter fitted to those benchmarks is renamed as a prediction, and no equation reduces to its own input by construction. The method builds on the authors' own MRM backbone [10] and LoRA fine-tuning recipe [44], and these self-citations are present but not load-bearing in a circular sense: MRM's visual-representation advantage is independently re-measured in Table III (ALTA 80.7/84.3/86.0 vs MRM* 80.6/84.0/85.8 on NIH ChestX-ray) and is additionally supported by external works [11]-[15]. The temporal-multiview contribution is validated by ablations (Tables V-VI) that show degradation when inputs are removed, rather than being assumed. The inference-time handling of absent temporal/multiview slots on single-image benchmarks is not specified (Section IV-K only states all patches are kept), which is a reproducibility and protocol gap, but it is not a circularity because it does not make the evaluation equal to the training objective or to a fitted quantity. Overall, the derivation chain is self-contained against external benchmarks, so no circular step is identified.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The empirical claims rest on the assumption that MRM features are strong and transferable, that zero-filled missing views are a valid proxy for absence, and that the combined losses are balanced correctly. These are domain assumptions, not derived facts.

free parameters (4)
  • loss weighting coefficients lambda_1, lambda_2, lambda_3 = not reported
    Equation (8) defines the total objective as a weighted sum of local, MLM, and MIM losses, but the weights are never given in Section IV-K, so the exact objective cannot be reconstructed.
  • image masking ratio = 75%
    Selected via ablation in Table VII; controls the information available to the encoder during alignment training.
  • report masking ratio = 50%
    Selected via ablation in Table VII for the MLM loss; affects the language-modeling objective.
  • adapter bottleneck ratio = 0.25
    Given in Section IV-K; controls the parameter count and capacity of the adapters, which is central to the 8% trainable-parameter claim.
assumptions (3)
  • domain assumption The MRM-pretrained vision encoder has superior radiograph representations compared with CLIP-based encoders, making it a suitable base for alignment.
    Invoked in the Introduction and Section II-A; supported by prior work including self-citations. If false, starting from MRM gives no advantage over CLIP-style encoders.
  • ad hoc to paper Zero-filled missing views behave like absent views and do not distort the learned representations.
    Section III-B states that missing lateral and prior images are filled with zeros; the paper does not analyze how zero patches interact with the transformer encoder or how single-image inference should be performed.
  • domain assumption The global and local contrastive losses, combined with MLM and MIM losses, jointly optimize alignment without conflicting objectives.
    Equations (5)-(8); ablations support the contribution of each loss, but the text also notes that removing some losses can improve individual metrics, indicating partial conflicts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models." pith.science (2026). https://pith.science/paper/2AUURWZE

@misc{pith2026250608990,
  author       = {Pith},
  title        = {Pith review of: Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AUURWZE}},
  note         = {Machine review of arXiv:2506.08990}
}
read the original abstract

Medical vision-language alignment through cross-modal contrastive learning shows promising performance in image-text matching tasks, such as retrieval and zero-shot classification. However, conventional cross-modal contrastive learning (CLIP-based) methods suffer from suboptimal visual representation capabilities, which also limits their effectiveness in vision-language alignment. In contrast, although the models pretrained via multimodal masked modeling struggle with direct cross-modal matching, they excel in visual representation. To address this contradiction, we propose ALTA (ALign Through Adapting), an efficient medical vision-language alignment method that utilizes only about 8% of the trainable parameters and less than 1/5 of the computational consumption required for masked record modeling. ALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling. Additionally, we integrate temporal-multiview radiograph inputs to enhance the information consistency between radiographs and their corresponding descriptions in reports, further improving the vision-language alignment. Experimental evaluations show that ALTA outperforms the best-performing counterpart by over 4% absolute points in text-to-image accuracy and approximately 6% absolute points in image-to-text retrieval accuracy. The adaptation of vision-language models during efficient alignment also promotes better vision and language understanding. Code is publicly available at https://github.com/DopamineLcy/ALTA.

Figures

Figures reproduced from arXiv: 2506.08990 by the authors.

Figure 1
Figure 1. Comparison of CLIP-based methods, MRM, and our ALTA. We integrate temporal-multiview inputs and execute efficient vision-language alignment by introducing trainable adapters. Radiographs from the prior study and lateral view are integrated to match radiology reports better. adapts the pretrained vision model from masked record model￾ing for medical vision-language alignment with only 8% train￾able parameters and 1/5… view at source ↗
Figure 2
Figure 2. Overview of ALTA and adapter integration. The illustrated adapter-integrated block is employed as the transformer block in both the vision encoder and language processor. Only 8% of the trainable parameters are included in the framework during efficient vision-language alignment. Global and local alignment losses are adopted to align vision-language representations while MLM and MIM losses contribute to the maintena… view at source ↗
Figure 3
Figure 3. Qualitative results of text-to-image and image-to-text retrieval tasks. We represent the top five predicted images and the top three texts for ALTA and GLoRIA-ViT, where prediction scores are labeled above the examples. We note the categories below wrongly retrieved samples. after implementing the proposed efficient alignment. Since LoRA can enhance fine-tuning for downstream tasks [44], we conduct LoRA fine-tuning … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Log-scale histogram visualization of the time interval distribution between prior and current studies in the restructured temporal-multiview MIMIC-CXR dataset. H. Time Interval Distribution Analysis To better understand the time interval distribution be￾tween prior and…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 27 canonical work pages

  1. [1]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  2. [2]

    Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,

    A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y . Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng, “Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs,”arXiv preprint arXiv:1901.07042, 2019

    arXiv 1901

  3. [3]

    Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,

    X. Wang, Y . Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2097–2106

    work page 2017

  4. [4]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,

    J. Irvin, P. Rajpurkar, M. Ko, Y . Yu, S. Ciurea-Ilcus, C. Chute, H. Mark- lund, B. Haghgoo, R. Ball, K. Shpanskayaet al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” inProceedings of the AAAI conference on artificial intelligence, vol. 33, 2019, pp. 590–597

    work page 2019

  5. [5]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  6. [6]

    Contrastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” inMachine Learning for Healthcare Conference. PMLR, 2022, pp. 2–25

    work page 2022

  7. [7]

    Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,

    S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Gloria: A multimodal global-local representation learning framework for label- efficient medical image recognition,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3942–3951

    work page 2021

  8. [8]

    Making the most of text semantics to improve biomedical vision– language processing,

    B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valleet al., “Making the most of text semantics to improve biomedical vision– language processing,” inEuropean conference on computer vision. Springer, 2022, pp. 1–21

    2022

Show all 56 references
  1. [9]

    A multimodal biomedical foundation model trained from fifteen million image–text pairs,

    S. Zhang, Y . Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluriet al., “A multimodal biomedical foundation model trained from fifteen million image–text pairs,”NEJM AI, vol. 2, no. 1, p. AIoa2400640, 2025

    2025

  2. [10]

    Advancing radiograph representation learning with masked record modeling,

    H.-Y . Zhou, C. Lian, L. Wang, and Y . Yu, “Advancing radiograph representation learning with masked record modeling,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  3. [11]

    Rad-dino: Exploring scalable medical image encoders beyond text supervision,

    F. P ´erez-Garc´ıa, H. Sharma, S. Bond-Taylor, K. Bouzid, V . Salvatelli, M. Ilse, S. Bannur, D. C. Castro, A. Schwaighofer, M. P. Lungren et al., “Rad-dino: Exploring scalable medical image encoders beyond text supervision,”arXiv preprint arXiv:2401.10815, 2024

    2024 arXiv

  4. [12]

    Medical vision language pretraining: A survey,

    P. Shrestha, S. Amgain, B. Khanal, C. A. Linte, and B. Bhat- tarai, “Medical vision language pretraining: A survey,”arXiv preprint arXiv:2312.06224, 2023

    2023 arXiv

  5. [13]

    Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classifica- tion,

    T. Dai, R. Zhang, F. Hong, J. Yao, Y . Zhang, and Y . Wang, “Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classifica- tion,”IEEE Transactions on Medical Imaging, vol. 43, no. 8, pp. 2901– 2912, 2024

    2024

  6. [14]

    Im- proving medical vision-language contrastive pretraining with semantics- aware triage,

    B. Liu, D. Lu, D. Wei, X. Wu, Y . Wang, Y . Zhang, and Y . Zheng, “Im- proving medical vision-language contrastive pretraining with semantics- aware triage,”IEEE Transactions on Medical Imaging, vol. 42, no. 12, pp. 3579–3589, 2023

    2023

  7. [15]

    Exploring scalable medical image encoders beyond text supervision,

    F. P ´erez-Garc´ıa, H. Sharma, S. Bond-Taylor, K. Bouzid, V . Salvatelli, M. Ilse, S. Bannur, D. C. Castro, A. Schwaighofer, M. P. Lungrenet al., “Exploring scalable medical image encoders beyond text supervision,” Nature Machine Intelligence, pp. 1–12, 2025

    2025

  8. [16]

    Learning to exploit temporal structure for biomedical vision-language processing,

    S. Bannur, S. Hyland, Q. Liu, F. Perez-Garcia, M. Ilse, D. C. Castro, B. Boecking, H. Sharma, K. Bouzid, A. Thiemeet al., “Learning to exploit temporal structure for biomedical vision-language processing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  9. [17]

    Gener- alized radiograph representation learning via cross-supervision between images and free-text radiology reports,

    H.-Y . Zhou, X. Chen, Y . Zhang, R. Luo, L. Wang, and Y . Yu, “Gener- alized radiograph representation learning via cross-supervision between images and free-text radiology reports,”Nature Machine Intelligence, vol. 4, no. 1, pp. 32–40, 2022

    2022

  10. [18]

    Chexrelnet: An anatomy-aware model for tracking longi- tudinal relationships between chest x-rays,

    G. Karwande, A. B. Mbakwe, J. T. Wu, L. A. Celi, M. Moradi, and I. Lourentzou, “Chexrelnet: An anatomy-aware model for tracking longi- tudinal relationships between chest x-rays,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springe...

    2022

  11. [19]

    Chexfusion: Effective fusion of multi-view features using transformers for long-tailed chest x-ray classification,

    D. Kim, “Chexfusion: Effective fusion of multi-view features using transformers for long-tailed chest x-ray classification,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2702–2710

    2023

  12. [20]

    Act like a radiologist: towards reliable multi-view correspondence reasoning for mammogram mass detection,

    Y . Liu, F. Zhang, C. Chen, S. Wang, Y . Wang, and Y . Yu, “Act like a radiologist: towards reliable multi-view correspondence reasoning for mammogram mass detection,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5947–5961, 2021

    2021

  13. [21]

    Deltanet: Conditional medical report generation for covid- 19 diagnosis,

    X. Wu, S. Yang, Z. Qiu, S. Ge, Y . Yan, X. Wu, Y . Zheng, S. K. Zhou, and L. Xiao, “Deltanet: Conditional medical report generation for covid- 19 diagnosis,” inProceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 2952–2961

    2022

  14. [22]

    Self-supervised learning for medical image analysis using image context restoration,

    L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueck- ert, “Self-supervised learning for medical image analysis using image context restoration,”Medical image analysis, vol. 58, p. 101539, 2019

    2019

  15. [23]

    Models genesis,

    Z. Zhou, V . Sodha, J. Pang, M. B. Gotway, and J. Liang, “Models genesis,”Medical image analysis, vol. 67, p. 101840, 2021

    2021

  16. [24]

    Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations,

    H.-Y . Zhou, S. Yu, C. Bian, Y . Hu, K. Ma, and Y . Zheng, “Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations,” inInternational Conference on Medical Image 12 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2020 C...

    2020

  17. [25]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 12 888–12 900

    2022

  18. [26]

    Vilt: Vision-and-language transformer without convolution or region supervision,

    W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 5583–5594

    2021

  19. [27]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,”Advances in neural information processing systems, vol. 34, pp. 9694–9705, 2021

    2021

  20. [28]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks,

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Somet al., “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” arXiv preprint arXiv:2208.10442, 2022

    2022 arXiv

  21. [29]

    Knowledge- enhanced visual-language pre-training on chest radiology images,

    X. Zhang, C. Wu, Y . Zhang, W. Xie, and Y . Wang, “Knowledge- enhanced visual-language pre-training on chest radiology images,”Na- ture Communications, vol. 14, no. 1, p. 4542, 2023

    2023

  22. [30]

    Carzero: Cross-attention alignment for radiology zero-shot classifica- tion,

    H. Lai, Q. Yao, Z. Jiang, R. Wang, Z. He, X. Tao, and S. K. Zhou, “Carzero: Cross-attention alignment for radiology zero-shot classifica- tion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 11 137–11 146

    2024

  23. [31]

    Enhancing representation in radiography- reports foundation model: A granular alignment algorithm using masked contrastive learning,

    W. Huang, C. Li, H.-Y . Zhou, H. Yang, J. Liu, Y . Liang, H. Zheng, S. Zhang, and S. Wang, “Enhancing representation in radiography- reports foundation model: A granular alignment algorithm using masked contrastive learning,”Nature Communications, vol. 15, no. 1, p. 7620, 2024

    2024

  24. [32]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis,

    C. Wu, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 21 372–21 383

    2023

  25. [33]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational Conference on Machine Learning. PMLR, 2019, pp. 2790–2799

    2019

  26. [34]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  27. [35]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022

    1950

  28. [36]

    Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, Y . Goldberg, and S. Ravfogel, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 1–9

    2022

  29. [37]

    Towards a unified view of parameter-efficient transfer learning,

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,”arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  30. [38]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 709–727

    2022

  31. [39]

    Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks,

    Y .-L. Sung, J. Cho, and M. Bansal, “Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5227–5237

    2022

  32. [40]

    Aim: Adapting image models for efficient video action recognition,

    T. Yang, Y . Zhu, Y . Xie, A. Zhang, C. Chen, and M. Li, “Aim: Adapting image models for efficient video action recognition,”arXiv preprint arXiv:2302.03024, 2023

    2023 arXiv

  33. [41]

    Vision transformer adapter for dense predictions,

    Z. Chen, Y . Duan, W. Wang, J. He, T. Lu, J. Dai, and Y . Qiao, “Vision transformer adapter for dense predictions,”arXiv preprint arXiv:2205.08534, 2022

    2022 arXiv

  34. [42]

    Parameter-efficient fine-tuning for medical image analysis: The missed opportunity,

    R. Dutt, L. Ericsson, P. Sanchez, S. A. Tsaftaris, and T. Hospedales, “Parameter-efficient fine-tuning for medical image analysis: The missed opportunity,”arXiv preprint arXiv:2305.08252, 2023

    2023 arXiv

  35. [43]

    Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,

    Y . Zhu, Z. Shen, Z. Zhao, S. Wang, X. Wang, X. Zhao, D. Shen, and Q. Wang, “Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,”arXiv preprint arXiv:2311.08236, 2023

    2023 arXiv

  36. [44]

    Less could be better: Parameter-efficient fine-tuning advances medical vision foundation mod- els,

    C. Lian, H.-Y . Zhou, Y . Yu, and L. Wang, “Less could be better: Parameter-efficient fine-tuning advances medical vision foundation mod- els,”arXiv preprint arXiv:2401.12215, 2024

    2024 arXiv

  37. [45]

    Prompt tuning for parameter- efficient medical image segmentation,

    M. Fischer, A. Bartler, and B. Yang, “Prompt tuning for parameter- efficient medical image segmentation,”Medical Image Analysis, vol. 91, p. 103024, 2024

    2024

  38. [46]

    Towards foundation models and few-shot parameter-efficient fine-tuning for volumetric organ seg- mentation,

    J. Silva-Rodr ´ıguez, J. Dolz, and I. B. Ayed, “Towards foundation models and few-shot parameter-efficient fine-tuning for volumetric organ seg- mentation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 213–224

    2023

  39. [47]

    Contrastive alignment of vision to language through parameter-efficient transfer learning,

    Z. Khan and Y . Fu, “Contrastive alignment of vision to language through parameter-efficient transfer learning,”arXiv preprint arXiv:2303.11866, 2023

    2023 arXiv

  40. [48]

    Scaling language- image pre-training via masking,

    Y . Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He, “Scaling language- image pre-training via masking,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23 390–23 400

    2023

  41. [49]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in neural information processing systems, 2017, pp. 5998–6008

    2017

  42. [50]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),”arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  43. [51]

    Mvco- dot: Multi-view contrastive domain transfer network for medical report generation,

    R. Wang, X. Wang, Z. Xu, W. Xu, J. Chen, and T. Lukasiewicz, “Mvco- dot: Multi-view contrastive domain transfer network for medical report generation,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  44. [52]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  45. [53]

    Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia,

    G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, L. M. Prevedello, T. S. Cook, A. Sharma, J. K. Amorosa, V . Arteaga, M. Galperin-Aizenberg et al., “Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia,”Radiology. Art...

    2019

  46. [54]

    Im- proving factual completeness and consistency of image-to-text radiology report generation,

    Y . Miura, Y . Zhang, E. B. Tsai, C. P. Langlotz, and D. Jurafsky, “Im- proving factual completeness and consistency of image-to-text radiology report generation,”arXiv preprint arXiv:2010.10042, 2020

    2010 arXiv

  47. [55]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  48. [56]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.