REVIEW 3 major objections 6 minor 30 references
Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiology
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Zero-shot BiomedCLIP over-predicts every label on imbalanced chest X-rays, while linear probing recovers most fine-tuning gains at a fraction of the training cost.
desk verdict A useful but under-specified empirical comparison: the zero-shot 'over-predicts all labels' headline is not interpretable because the decision threshold is never stated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by the contrastive embedding space of BiomedCLIP and by three measurement tools applied to it. The inter-class versus intra-class Euclidean distance ratio quantifies separability, multi-label metrics (macro-F1, exact-match, LRAP, coverage error) quantify ranking and detection quality, and Grad-CAM heatmaps compare model attention against radiologist-annotated regions. Together they let the paper attribute differences in behavior to the adaptation regime rather than to the dataset or the pretrained weights alone.
What would settle it
Recompute the per-class F1 scores and LRAP on a test set where a radiologist manually labels every image, especially Pneumothorax and Consolidation; if fine-tuning then matches or beats linear probing on mid-frequency classes, the paper's claim that linear probing generalizes better on scarce classes is refuted.
Extended reading notes
Core claim
The paper's central claim is that BiomedCLIP's zero-shot behavior is systematically over-predictive on imbalanced out-of-distribution radiographs: it assigns labels too liberally, so macro-F1 is 0.105, exact-match accuracy is 0.0%, LRAP is 0.250, and coverage error is 7.70, and the inter/intra-class distance ratio stays at 1.506. Full fine-tuning increases the separation ratio to 1.784 and raises macro-F1 to 0.235, exact-match to 13.4%, and LRAP to 0.779, while linear probing on the frozen encoder reaches 0.183 macro-F1, 4.3% exact-match, and 0.741 LRAP at 6.10 minutes of training versus 15.47 minutes. Per-class, fine-tuning wins on abundant classes and fails on several rare ones, whereas linear probing does better on mid-frequency diseases like Pneumonia, Atelectasis, and Fracture. Grad-CAM comparisons against 15 radiologist annotations show zero-shot heatmaps tightly localized to pathology, fine-tuning heatmaps broad and non-specific, and linear probing retaining near-zero-shot spatial fidelity while adding accuracy.
Load-bearing premise
The comparison assumes the CheXbert-generated labels on IU-Xray reports are accurate enough ground truth for all 14 disease classes, even though CheXbert has roughly 80% validated accuracy and the rarest classes have only 20 to 31 samples.
Editorial extensions
If this is right
- Zero-shot inference with a biomedical VLM should not be trusted as a clinical classifier on imbalanced out-of-distribution data; its high recall comes at the cost of overwhelming false positives.
- Linear probing is a viable low-cost adaptation: it recovers most of full fine-tuning's ranking performance and takes less than half the training time.
- Full fine-tuning is the best choice when abundant classes matter most, but it can erase performance on scarce classes and degrade spatial interpretability.
- Grad-CAM fidelity and classification accuracy do not move together; zero-shot can localize well while ranking poorly, so interpretability evaluations must be separate from accuracy evaluations.
Reading between the lines
- The same evaluation could be run on a manually labeled out-of-distribution set to test whether the rare-class advantage of linear probing is real or an artifact of label noise.
- A natural extension is to test prompt-based and adapter-based adaptations, which the paper lists as future work, against the same three heads to see whether the fine-tuning/linear-probing trade-off is specific to full weight updates.
- The shallow-layer localization result suggests that clinicians may get more useful saliency maps by reading earlier blocks of the encoder, even when the final layer is used for classification.
- If the zero-shot over-prediction pattern holds on other imbalanced medical datasets, it would argue for defaulting to lightweight probing before deploying any pretrained biomedical VLM in a low-data clinic.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates BiomedCLIP on the IU-Xray chest radiograph dataset, which is treated as a highly imbalanced, out-of-distribution, multi-label benchmark. Labels are obtained automatically from radiology reports using CheXbert. The authors compare three adaptation regimes: zero-shot inference, full fine-tuning, and linear probing, reporting embedding-space separability, multi-label classification metrics (macro-F1, exact-match, LRAP, coverage error), and per-label/domain F1. They also present Grad-CAM visualizations compared against 15 radiologist-annotated radiographs. The central claims are that zero-shot BiomedCLIP 'over-predicts all labels' with poor precision and separability, full fine-tuning improves classification of distinct diseases, and linear probing recovers much of the fine-tuning gain at lower cost while preserving spatial fidelity.
Significance. If the quantitative results are taken at face value, the paper provides a useful practical message for users of biomedical VLMs: zero-shot inference on imbalanced OOD radiology data is unreliable, and linear probing is a cost-effective alternative to full fine-tuning. The authors release code and include 15 radiologist-annotated samples for qualitative comparison, which are valuable assets. The embedding-space analysis and the comparison of three adaptation regimes on a realistic imbalanced dataset address a question of genuine interest to the medical imaging community. However, the significance is currently conditional: the headline zero-shot result is threshold-dependent and the manuscript does not specify the decision rule, so the quantitative contrast could be an artifact. The CheXbert labeling and the single train/val/test split further weaken the reliability of per-class comparisons.
major comments (3)
- [Section 4.2 and Section 5.1] The paper never specifies the decision rule that converts BiomedCLIP's similarity scores or logits into the binary predictions used in Tables 1 and 2. Section 4.2 (1) states only that zero-shot yields image and text embeddings, and Section 4.3 gives implementation details without any threshold. The claim in the abstract and Section 5.1 that zero-shot 'over-predicts all labels' is precisely a statement about where the positive/negative boundary was placed; a sufficiently low threshold would mechanically predict all 14 labels for every image, producing macro-F1 near 0.105 and LRAP near 0.250 regardless of model quality. The same issue applies to the BCE-trained heads in fine-tuning and linear probing. The authors must specify the threshold for each setting, state how it was chosen (e.g., fixed at 0.5, tuned on validation, or based on a percentile), and ideally supplement the threshold-dependent metrics with threshold-independent ones such as average precision or precision-recall curves. Without this, the central quantitative contrast in Table 1 is uninterpretable.
- [Table 1 vs. Section 5.1] There is a direct numerical inconsistency in the reported exact-match accuracy for linear probing. Table 1 lists exact-match as 0.143 (14.3%), while Section 5.1 and the Conclusion state 'exact-match to 4.3%' and 'exact-match = 4.3%', respectively. Since exact-match is one of the headline metrics used to compare the three settings, the authors must correct this inconsistency and verify that all numbers in the text, tables, and abstract are mutually consistent.
- [Section 4.3 and Section 6] The quantitative comparison rests on a single train/val/test split (70/10/20) with no repeated runs or confidence intervals, and the ground-truth labels are produced by CheXbert rather than manual review, as the authors acknowledge in the limitations. This is particularly concerning for rare classes such as Pneumothorax (n=20) and Consolidation (n=31), where a handful of mislabeled reports could change per-class F1 values substantially. Claims such as 'linear probing generalizes better on mid-frequency diseases' (Section 5.2) are based on small per-class differences that could be within label-noise or split-induced variation. The authors should report results across multiple seeds or cross-validation folds, provide error bars, and discuss the sensitivity of the reported per-class gaps to CheXbert label noise; a patient-level split should also be used if patients contribute multiple images.
minor comments (6)
- [Abstract and Section 5.1] The phrase 'over-predicts all labels' is used as a definitive finding, but without the decision-threshold details it is better framed as a conditional observation; please qualify it until the threshold is specified.
- [Introduction] There are several typos: 'Morever' should be 'Moreover'; 'Standford' should be 'Stanford'; 'onewarm-up' should be 'one warm-up'; 'multiperceptron' should be 'multilayer perceptron'.
- [References] References [4] and [5] appear to refer to the same Boecking et al. ECCV 2022 paper and are duplicated; please merge or disambiguate.
- [Section 5.3] The Grad-CAM discussion would benefit from clearer quantification of 'aligns very closely' and 'co-localized'; consider reporting a quantitative overlap metric (e.g., Dice or IoU) against the radiologist annotations rather than only qualitative descriptions.
- [Section 3] The dataset description should mention that IU-Xray contains frontal and lateral views and clarify whether the model is evaluated on both views or only one, since this affects the interpretation of the OOD claim.
- [Table 2] The zero-shot F1 scores in Table 2 are all positive even for rare classes; this is surprising given the claim of 'over-predicting all labels' and suggests the threshold may not be extremely low. Clarifying this would help the reader reconcile Table 2 with the headline narrative.
Circularity Check
No circular reasoning: all reported results are empirical evaluations against externally generated labels, not derivations from the paper's own assumptions.
full rationale
The paper's central claims are empirical observations about BiomedCLIP's behavior on IU-Xray: that zero-shot inference over-predicts labels, that full fine-tuning improves macro-F1 and LRAP, and that linear probing recovers most of those gains. These claims are supported by measured metrics (Tables 1-3) computed on model outputs against CheXbert-generated labels, which are external to BiomedCLIP and to the authors' own framework. No parameter is fitted and then renamed as a prediction; no equation is defined in terms of the result it is supposed to explain; and no load-bearing step relies on a self-citation or an imported uniqueness theorem. The authors' adoption of standard adaptation regimes (zero-shot, linear probing, full fine-tuning) and standard multi-label metrics are not circular: they are evaluation protocols, and the paper does not claim to derive model behavior from those protocols. The acknowledged limitation about CheXbert label fidelity and the unspecified threshold or decision rule for converting similarity scores to binary predictions are methodological or reporting concerns, not circularity. The paper is self-contained as an empirical benchmark study, and none of its conclusions reduces to its inputs by construction. Therefore, no circular step is present, and the appropriate score is 0.
Assumptions & free parameters
free parameters (3)
- zero-shot multi-label threshold =
unspecified
- fine-tuning hyperparameters =
not fully reported
- CheXbert labeler threshold and version =
unspecified
assumptions (4)
- domain assumption CheXbert labels are a sufficiently accurate proxy for true disease labels in IU-Xray.
- domain assumption IU-Xray is out-of-distribution for BiomedCLIP because BiomedCLIP was not pretrained on it.
- domain assumption Euclidean distance in the embedding space reflects semantic class separability.
- domain assumption Grad-CAM activations accurately reflect the model's visual evidence for its predictions.
Cite this review
Pith. "Pith review of Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiology." pith.science (2026). https://pith.science/paper/6N5JBJMX
@misc{pith2026250614136,
author = {Pith},
title = {Pith review of: Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiology},
year = {2026},
howpublished = {\url{https://pith.science/paper/6N5JBJMX}},
note = {Machine review of arXiv:2506.14136}
}
read the original abstract
In this paper, we construct two research objectives: i) explore the learned embedding space of BiomedCLIP, an open-source large vision language model, to analyse meaningful class separations, and ii) quantify the limitations of BiomedCLIP when applied to a highly imbalanced, out-of-distribution multi-label medical dataset. We experiment on IU-xray dataset, which exhibits the aforementioned criteria, and evaluate BiomedCLIP in classifying images (radiographs) in three contexts: zero-shot inference, full finetuning, and linear probing. The results show that the model under zero-shot settings over-predicts all labels, leading to poor precision and inter-class separability. Full fine-tuning improves classification of distinct diseases, while linear probing detects overlapping features. We demonstrate visual understanding of the model using Grad-CAM heatmaps and compare with 15 annotations by a radiologist. We highlight the need for careful adaptations of the models to foster reliability and applicability in a real-world setting. The code for the experiments in this work is available and maintained on GitHub.
Figures
Reference graph
Works this paper leans on
-
[1]
Atmakuru, A., Chakraborty, S., Faust, O., Salvi, M., Barua, P. D., Molinari, F., Acharya, U., and Homaira, N.Deep learning in radiology for lung cancer diagnostics: A systematic review of classification, segmentation, and predictive modeling techniques.Expert Systems with Applications(2024), 124665. BIOKDD’25, August 03–07, 2025, Ontario, Canada Nafiz Sad...
work page 2024
-
[2]
K., Rauland, A., Jia, Y., Avval, A
Azad, R., Aghdam, E. K., Rauland, A., Jia, Y., Avval, A. H., Bozorgpour, A., Karimijafarbigloo, S., Cohen, J. P., Adeli, E., and Merhof, D.Medical image segmentation review: The success of u-net.IEEE Transactions on Pattern Analysis and Machine Intelligence 46(2024), 10076–10095
work page 2024
-
[3]
C., Boeck- ing, B., Sharma, H., Bouzid, K., Thieme, A., Schwaighofer, A., Wetscherek, M., Lungren, M
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boeck- ing, B., Sharma, H., Bouzid, K., Thieme, A., Schwaighofer, A., Wetscherek, M., Lungren, M. P., Nori, A., Alvarez-Valle, J., and Oktay, O.Learning to exploit temporal structure for biomedical vision-language processing. InPro- ceedings of the IEEE/CVF Conference on Computer ...
work page 2023
-
[4]
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-V alle, J., Poon, H., and Oktay, O.Making the most of text semantics to improve biomedical vision–language processing. InComputer Vision – ECCV 2022(Cham, 2022), S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner...
work page 2022
-
[5]
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hy- land, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-V alle, J., Poon, H., and Oktay, O.Making the most of text semantics to improve biomedical vision– language processing. InProceedings of the European Conference on Computer Vision (ECCV)(2022), vol. 13696, Springer, pp. 1–21
work page 2022
-
[6]
Chambon, P. J. M., Bluethgen, C., Langlotz, C., and Chaudhari, A.Adapting pretrained vision-language foundational models to medical imaging domains. In NeurIPS 2022 Foundation Models for Decision Making Workshop(2022)
work page 2022
-
[7]
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Ro- driguez, L., Antani, S., Thoma, G. R., and McDonald, C. J.Preparing a col- lection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Association 23, 2 (2015), 304–310
work page 2015
-
[8]
Dosovitskiy, A.An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020)
arXiv 2020
Show all 30 references
-
[9]
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2023), pp
Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A.Finetune like you pretrain: Improved finetuning of zero-shot vision models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2023), pp. 19338–19347
2023
-
[10]
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H.Domain-specific language model pretraining for biomedical natural language processing.ACM Trans. Comput. Healthcare 3, 1 (2021)
2021
-
[11]
P., and Yeung, S.Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition
Huang, S.-C., Shen, L., Lungren, M. P., and Yeung, S.Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)(October 2021), pp. 3942–3951
2021
-
[12]
Jing, B., Xie, P., and Xing, E.On the automatic generation of medical imaging reports.arXiv preprint arXiv:1711.08195(2017)
2017 arXiv
-
[13]
Koleilat, T., Asgariandehkordi, H., Rivaz, H., and Xiao, Y.Biomedcoop: Learning to prompt for biomedical vision-language models.arXiv preprint arXiv:2411.15232(2024)
2024 arXiv
-
[14]
J., Young, G., van de Meent, J.-W., and W allace, B
McInerney, D. J., Young, G., van de Meent, J.-W., and W allace, B. C.That’s the wrong lung! evaluating and improving the interpretability of unsupervised multimodal encoders for medical data. InProceedings of the Conference on Empir- ical Methods in Natural Language Processing...
2022
-
[15]
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Ma...
2021
-
[16]
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Ma...
2021
-
[17]
Schäfer, R., Nicke, T., Höfener, H., Lange, A., Merhof, D., Feuerhake, F., Schulz, V., Lotz, J., and Kiessling, F.Overcoming data scarcity in biomedical imaging with a foundational multi-task model.Nature Computational Science 4 (2024), 495–509
2024
-
[18]
R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.Grad-cam: Visual explanations from deep networks via gradient-based local- ization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D.Grad-cam: Visual explanations from deep networks via gradient-based local- ization. InProceedings of the IEEE International Conference on Computer Vision (ICCV)(Oct 2017)
2017
-
[19]
R., Gao, M., Lu, L., Xu, Z., Nogues, I., Y ao, J., Mollura, D., and Summers, R
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Y ao, J., Mollura, D., and Summers, R. M.Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning.IEEE transactions on medical imaging 35, 5 ...
2016
-
[20]
Sloan, P., Clatworthy, P., Simpson, E., and Mirmehdi, M.Automated radiology report generation: A review of recent advances.IEEE Reviews in Biomedical Engineering 18(2025), 368–387
2025
-
[21]
Y., and Lungren, M
Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A. Y., and Lungren, M. P. Chexbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert, 2020
2020
-
[22]
Varoqaux, G., and Cheplygina, V.Machine learning for medical imaging: methodological failures and recommendations for the future.NPJ digital medicine 5, 1 (2022), 48
2022
-
[23]
InProceedings of the Conference on Empirical Methods in Natural Language Processing
W ang, Z., Wu, Z., Agarwal, D., and Sun, J.Medclip: Contrastive learning from unpaired medical images and text. InProceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing(2022), vol. 2022, p. 3876
2022
-
[24]
Wojtara, M., Rana, E., Rahman, T., Khanna, P., and Singh, H.Artificial intelligence in rare disease diagnosis and treatment.Clinical and Translational Science 16, 11 (2023), 2106–2111
2023
-
[25]
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al.A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings. Nature Communications 16, 1 (2025), 3108
2025
-
[26]
R., Badgeley, M
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K.Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.PLoS medicine 15(2018), e1002683
2018
-
[27]
IEEE Transactions on Knowledge and Data Engineering 26, 8 (2014), 1819–1837
Zhang, M.-L., and Zhou, Z.-H.A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering 26, 8 (2014), 1819–1837
2014
-
[28]
P., Nau- mann, T., W ang, S., and Poon, H.A multimodal biomedical foundation model trained from fifteen million image–text pairs.NEJM AI 2, 1 (2025), AIoa2400640
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., V alluri, N., Wong, C., Tupini, A., W ang, Y., Mazzola, M., Shukla, S., Liden, L., Gao, J., Crabtree, A., Piening, B., Bifulco, C., Lungren, M. P., Nau- mann, T., W ang, S., and Poon, H...
2025
-
[29]
Zhang, X., Wu, C., Zhang, Y., Xie, W., and Wang, Y.Knowledge-enhanced visual-language pre-training on chest radiology images.Nature Communications 14, 1 (2023), 4542
2023
-
[30]
D., and Langlotz, C
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P.Contrastive learning of medical visual representations from paired images and text. In Proceedings of the 7th Machine Learning for Healthcare Conference(2022), vol. 182, pp. 2–25. Received 20 February 2007; re...
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.