Pith. sign in

REVIEW 4 major objections 5 minor 81 references

Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Healthcare vision-language model research needs its own four-category reporting guidelines, because existing AI checklists are built for single-phase models and miss reproducibility-critical details of pretraining, finetuning, and…

desk verdict A useful, practical position paper on VLM reporting standards whose four-category taxonomy is the right kind of proposal but needs validation before journals adopt it. read the letter →

arxiv 2505.08818 v1 pith:V3XYXD5R submitted 2025-05-12 cs.CY cs.AIcs.LG

Amara Tariq , Rimita Lahiri , Charles Kahn , Imon Banerjee This is my paper · ORCID
classification cs.CYcs.AIcs.LG
keywords vision-languagemodelshealthcareAIreportingguidelinesreproducibilitychecklistpretrainingfinetuningprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper tries to establish that the standard reporting and evaluation checklists used for machine-learning studies in healthcare—such as TRIPOD+AI and CLAIM—cannot properly capture vision-language model (VLM) research, because VLM work happens in multiple phases and takes several distinct forms. It proposes sorting every healthcare VLM study into one of four categories: pretraining a new VLM, domain-specific finetuning, task-specific finetuning, and prompting-based use of an off-the-shelf model. For each category it specifies what must be reported about model design, training objectives, dataset splits, performance metrics, and bias analysis, and it condenses these requirements into a peer-review checklist. A sympathetic reader would care because reproducible reporting is the precondition for clinical adoption of VLMs; if the categories are adopted, journals and reviewers would hold VLM papers to a different, phase-aware standard than current checklists allow.

What carries the argument

The central mechanism is the four-category taxonomy of VLM studies, defined by where in the training-and-deployment pipeline a study acts. Category assignments determine which reporting responsibilities apply: pretraining and domain-finetuning studies must report training objectives and run ablations when objectives are combined, task-finetuning studies inherit standard machine-learning reporting plus finetuning specifics, and prompting studies must document prompt type, optimization strategy, and model versioning. The taxonomy does the work of converting a vague demand for reproducibility into concrete, per-phase items.

What would settle it

A concrete way to test the framework: take a published healthcare VLM study that combines continuous learning or an agentic pipeline with prompting and try to assign it a single category and complete the checklist. If the classification is ambiguous or a reporting responsibility cannot be mapped, the central mapping breaks. Alternatively, show that a paper satisfying every checklist item still fails to reproduce when another group re-runs it, which would indicate the required items are not sufficient.

Watch

Extended reading notes

Core claim

The paper's central claim is that VLM studies in healthcare cannot be judged by reporting rules designed for end-to-end trained, single-modality prediction models, because a VLM's behavior and risks are spread across a pretraining phase, one or more finetuning phases, and a deployment phase that may be prompt-based. It therefore proposes a four-way categorization—VLM pretraining, domain-specific finetuning, task-specific finetuning, and prompting-based studies—and derives from that categorization a set of reporting requirements organized by manuscript section, with separate data split classes (pretraining, domain finetuning, task finetuning, domain testing, task testing), a two-pronged evaluation rule for pretraining and domain studies (benchmark the training objective and benchmark transferability to downstream tasks), and a consolidated checklist for authors and peer reviewers.

Load-bearing premise

The load-bearing premise is that every VLM study falls cleanly into one of the four fixed categories and that the categories remain stable as methods evolve; the paper itself notes the taxonomy is based on the 2020–2024 literature and may not fit upcoming innovations.

Editorial extensions

If this is right

  • If adopted, the checklist would replace or supplement TRIPOD+AI and CLAIM for VLM submissions, so peer reviewers would evaluate pretraining objectives, dataset split overlap, and prompt optimization rather than only final-task metrics.
  • Pretraining and domain-finetuning studies would be expected to benchmark both the training objective itself (for example, image-to-text and text-to-image retrieval) and transferability to multiple downstream tasks, changing what counts as sufficient evaluation.
  • Dataset reporting would separate five splitting classes and require explicit overlap checks, making data leakage between pretraining corpora and test sets a standard disclosure item.
  • Prompting-based studies would need to report model version, prompt template, few-shot context, and whether prompts were manual or algorithmically optimized, closing a common reproducibility gap.
  • Legal restrictions on releasing weights trained on private medical data would remain a documented barrier, and the guidelines would force authors to state that barrier explicitly rather than omit it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the four-category taxonomy could reasonably be adopted beyond healthcare wherever VLMs are used in regulated settings, such as autonomous driving or clinical documentation, because the phase structure and data-overlap risks are the same.
  • Editorial inference: the categories are likely to be stressed by agentic and continuously learning VLM pipelines, which update weights during deployment; such systems would fall between task-specific finetuning and prompting, so the taxonomy would need a fifth category or a rule for assigning studies with mixed phases.
  • Editorial inference: because the checklist items are mostly declarative, the framework lends itself to automated compliance screening of manuscripts, though the paper itself does not propose such a tool.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This position paper argues that existing machine learning reporting standards such as TRIPOD+AI and CLAIM are inadequate for vision-language model (VLM) studies in healthcare, because VLMs involve multiphase training (pretraining, domain-specific finetuning, task-specific finetuning) and flexible modes of use (prompting, linear probing). The authors propose a four-category taxonomy of VLM studies — VLM pretraining, domain-specific finetuning, task-specific finetuning, and prompting-based studies — and derive category-specific reporting requirements for model design, data description, performance evaluation, bias analysis, and manuscript structure. They provide a consolidated checklist (Table 2) and a table of recommended evaluation metrics (Table 1). The paper also includes a limitations section and an alternate-position section suggesting that extending existing guidelines could be a viable alternative to creating a new taxonomy.

Significance. If the proposed taxonomy is reliable, the paper addresses a genuine gap: VLM papers in healthcare are currently reviewed with checklists designed for single-phase, single-modality prediction models, and the resulting selective reporting is a real reproducibility problem. The manuscript is clearly grounded in the 2020-2024 VLM literature and offers concrete, itemized guidance, including a peer-review checklist that could be adopted by journals. The explicit alternate position in Section 8 is a fair-minded framing. However, because the entire framework rests on the ability of authors and reviewers to classify a study into exactly one of four categories, the taxonomy's boundary ambiguities and lack of validation are load-bearing concerns.

major comments (4)
  1. [Section 3(ii)] The definition of domain-specific finetuning is internally inconsistent with its own example. The text states that domain-specific finetuning aligns image and text modalities 'targeting more than one downstream task,' but the cited example PubMedCLIP (Eslami et al., 2023) is described as reporting evaluations only on VQA tasks, which is a single downstream task. This ambiguity is consequential because Section 6 assigns different evaluation obligations: domain-specific finetuning requires two-pronged pretraining-style benchmarking, whereas task-specific finetuning can rely on TRIPOD+AI/CLAIM. A reviewer could assign different reporting burdens to the same study depending on how this definition is read, so the categorization criterion needs to be stated precisely and reconciled with the example.
  2. [Section 3(iii)] The classification of Vo et al. (2024) as task-specific finetuning is questionable under the paper's own definition. That work uses frozen CLIP encoders and updates only a fusion classifier, which is closer to linear probing or a lightweight head on top of a VLM backbone than to finetuning a VLM's weights. Since Section 6 treats task-specific finetuning as adequately covered by TRIPOD+AI/CLAIM while prompting-based and pretraining-style studies receive different reporting requirements, the paper should either explicitly state whether linear probing belongs to this category or create a separate subcategory. Otherwise, category assignment becomes a judgment call that undermines the reproducibility the framework is meant to ensure.
  3. [Section 7 (Limitations); Section 3] The taxonomy is asserted but not validated. The limitations section concedes that the categorization is based on the 2020-2024 literature and may not cover future innovations, but there is no inter-rater reliability study, coverage analysis, or worked application to a sample of recent VLM papers. Given that category membership determines which reporting requirements are mandatory (e.g., two-pronged benchmarking for domain-specific finetuning vs. standard checklists for task-specific finetuning), the absence of any empirical audit leaves the central mechanism unsupported. The authors should provide at least a small set of worked examples, preferably including ambiguous cases, and discuss objective decision rules for boundary assignments.
  4. [Section 4 (Reporting of model design and training)] The requirement that pretraining and domain-specific finetuning studies 'must report ... the comparative analysis with other objectives on a hold-out test set' is difficult to reconcile with the same section's acknowledgment that 'VLM pretraining is not systematically optimized' due to computational and time requirements. As stated, the checklist would demand an experiment that the authors themselves recognize is often infeasible, which risks making the guideline unenforceable and encouraging pro-forma noncompliance. The paper should separate mandatory reporting of the objectives actually used from a more clearly conditioned recommendation to report comparisons when computational resources permit.
minor comments (5)
  1. [Figure 1] The caption says the arrows represent 'unidirectional interdependencies between the VLM categories,' but the figure itself is not shown in the accessible text and no explanation of what each arrow means is provided. Please either include the figure or describe the dependencies in the caption.
  2. [Appendix Table 2] The row for 'Downstream task(s)' in the Performance section says 'recommended metrics (Table XX),' but no table number is resolved. This should refer to Table 1.
  3. [Section 4, paragraph on finetuning] The citation '(Zheng et al., 2024; Lai et al.)' is incomplete: the Lai et al. entry lacks a year in both the text and the reference list (the reference list gives only '2023' in the URL line without a formal year).
  4. [Section 3(iii)] The name 'V o et al.' is rendered with a stray space; it should be 'Vo et al.' throughout.
  5. [Section 5, privacy paragraph] The phrase 'governing it's use' contains a typo; it should be 'governing its use.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proposal is a self-contained normative framework with no fitted parameters, derived predictions, or load-bearing self-citations.

full rationale

This is a position paper that proposes a taxonomy of VLM studies and derives reporting guidelines from that taxonomy. The causal chain is definitional in the benign sense: categories are defined by study characteristics (pretraining, domain-specific finetuning, task-specific finetuning, prompting), and the paper then assigns reporting duties to each category. There is no quantity fitted to data and then re-predicted, no equation that reduces to an input, and no 'uniqueness theorem' invoked to rule out alternatives. The paper's central claim—that traditional reporting standards must be restructured for multiphase VLM studies—is argued from the stated complexity of multi-phase training and multimodal evaluation, not from a self-referential derivation. Self-citations to Tariq et al. 2021, Banerjee et al. 2023, and Huang et al. 2020 appear only as background on fusion techniques and bias, and they do not carry the argument for the proposed categorization. The critique of TRIPOD+AI and CLAIM cites CLAIM-related work that includes co-author Kahn, but that citation is used as an external benchmark to be critiqued, not as evidence for the paper's own guidelines. Section 7's limitation that the taxonomy is based on 2020–2024 literature is an honest boundary condition, not a circular dependence: the taxonomy is inductive rather than derived from a theorem. The internal inconsistencies noted in reviewer skepticism (e.g., the PubMedCLIP example spanning category definitions) may affect validity or usability, but they are not circularity. Under the required evidentiary standard—quoting a specific reduction of a derived result to its inputs—no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper contributes a normative framework, so the 'axioms' are its domain assumptions about how VLM studies are structured and what makes reports reproducible. There are no fitted parameters or invented physical entities. The central assumptions are that the four categories cover the field, that TRIPOD/CLAIM fail for VLMs, and that checklist adherence improves reproducibility.

assumptions (3)
  • domain assumption VLM studies can be cleanly partitioned into four categories: pretraining, domain-specific finetuning, task-specific finetuning, and prompting-based studies.
    Section 3 introduces the four-way categorization as foundational; Section 7 limits it to the 2020-2024 literature, so future paradigms may not fit.
  • domain assumption Traditional checklists (TRIPOD+AI, CLAIM) are unsuitable for VLMs.
    Section 2 argues incompatibility due to multi-phase training and variable model adoption strategies, but this is asserted using examples rather than demonstrated empirically.
  • domain assumption Reporting the specified items (training objectives, data splits, bias metrics, prompt details, etc.) will improve reproducibility and clinical adoption.
    Implicit throughout Sections 4 to 6 and the checklist; the paper does not validate that checklist adherence produces better reproducibility or performance reporting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare." pith.science (2026). https://pith.science/paper/V3XYXD5R

@misc{pith2026250508818,
  author       = {Pith},
  title        = {Pith review of: Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3XYXD5R}},
  note         = {Machine review of arXiv:2505.08818}
}
read the original abstract

The intricate and multifaceted nature of vision language model (VLM) development, adaptation, and application necessitates the establishment of clear and standardized reporting protocols, particularly within the high-stakes context of healthcare. Defining these reporting standards is inherently challenging due to the diverse nature of studies involving VLMs, which vary significantly from the development of all new VLMs or finetuning for domain alignment to off-the-shelf use of VLM for targeted diagnosis and prediction tasks. In this position paper, we argue that traditional machine learning reporting standards and evaluation guidelines must be restructured to accommodate multiphase VLM studies; it also has to be organized for intuitive understanding of developers while maintaining rigorous standards for reproducibility. To facilitate community adoption, we propose a categorization framework for VLM studies and outline corresponding reporting standards that comprehensively address performance evaluation, data reporting protocols, and recommendations for manuscript composition. These guidelines are organized according to the proposed categorization scheme. Lastly, we present a checklist that consolidates reporting standards, offering a standardized tool to ensure consistency and quality in the publication of VLM-related research.

Figures

Figures reproduced from arXiv: 2505.08818 by the authors.

Figure 1
Figure 1. shows this categorization and interdependencies between VLM development strategies. (i) VLM pretraining: In this category, we classify studies that propose a new VLM model with self-supervised pretraining where the objective is to model relationship between image and text modalities to learn a shared representation space. VLM pretraining allows encoding of both visual and tex￾tual information, and performs multiple … view at source ↗
Figure 2
Figure 2. Conceptual categorization of VLM datasets based on our proposed study categorization. Generic and large scale – The image-text paired dataset should be large enough to capture a wide variety of visual and textual patterns during VLM pretraining, enabling the model to generalize well across different domains. Com￾mon generic large-scale datasets for pretraining VLMs in￾clude COCO (2.5M) (Lin et al., 2014), Conceptual… view at source ↗
Figure 3
Figure 3. Conceptual diagram for VLM performance reporting. modality reasoning. We posit that benchmarking must be balanced in terms of two modalities, for example, both image-to-text and text-to-image retrieval and matching per￾formance should be reported. For example, (Wang et al., 2022) proposed contrastive loss based alignment of vision and language representation in a joint embedding space, and evaluated on image-to-text… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 49 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Shortcuts

    Banerjee, I., Bhattacharjee, K., Burns, J. L., Trivedi, H., Purkayastha, S., Seyyed-Kalantari, L., Patel, B. N., Shiradkar, R., and Gichoya, J. “Shortcuts” causing bias in radiology artificial intelligence: causes, evaluation and mitigation. Journal of the American College of Radiology, 2023

    work page 2023

  3. [3]

    K., Aggarwal, K., Som, S., Piao, S., and Wei, F

    Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35: 0 32897--32912, 2022

    work page 2022

  4. [4]

    U., and Kahembwe, E

    Birhane, A., Prabhu, V. U., and Kahembwe, E. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963, 2021

    arXiv 2021

  5. [5]

    P., Kumar, A., Van Veen, D., Gardezi, S

    Blankemeier, L., Cohen, J. P., Kumar, A., Van Veen, D., Gardezi, S. J. S., Paschali, M., Chen, Z., Delbrouck, J.-B., Reis, E., Truyts, C., et al. Merlin: A vision language foundation model for 3d computed tomography. arXiv preprint arXiv:2406.06512, 2024

    arXiv 2024

  6. [6]

    C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al

    Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. Making the most of text semantics to improve biomedical vision--language processing. In European conference on computer vision, pp.\ 1--21. Springer, 2022

    2022

  7. [7]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  8. [8]

    Multi-modal masked autoencoders for medical vision-and-language pre-training

    Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., and Chang, T.-H. Multi-modal masked autoencoders for medical vision-and-language pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 679--689. Springer, 2022

    work page 2022

Show all 81 references
  1. [9]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  2. [10]

    C., Hobbs, S

    Colak, E., Kitamura, F. C., Hobbs, S. B., Wu, C. C., Lungren, M. P., Prevedello, L. M., Kalpathy-Cramer, J., Ball, R. L., Shih, G., Stein, A., et al. The rsna pulmonary embolism ct dataset. Radiology: Artificial Intelligence, 3 0 (2): 0 e200254, 2021

    2021

  3. [11]

    S., Reitsma, J

    Collins, G. S., Reitsma, J. B., Altman, D. G., and Moons, K. G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod) the tripod statement. Circulation, 131 0 (2): 0 211--219, 2015

    2015

  4. [12]

    S., Moons, K

    Collins, G. S., Moons, K. G., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., Van Smeden, M., et al. TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. ...

    2024

  5. [13]

    Embedding arithmetic of multimodal queries for image retrieval

    Couairon, G., Douze, M., Cord, M., and Schwenk, H. Embedding arithmetic of multimodal queries for image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4950--4958, 2022

    2022

  6. [14]

    Redcaps: Web-curated image-text data created by the people, for the people

    Desai, K., Kaul, G., Aysola, Z., and Johnson, J. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431, 2021

    2021 arXiv

  7. [15]

    Coarse-to-fine vision-language pre-training with fusion in the backbone

    Dou, Z.-Y., Kamath, A., Gan, Z., Zhang, P., Wang, J., Li, L., Liu, Z., Liu, C., LeCun, Y., Peng, N., et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. Advances in neural information processing systems, 35: 0 32942--32956, 2022

    2022

  8. [16]

    Multi-modal alignment using representation codebook

    Duan, J., Chen, L., Tran, S., Yang, J., Xu, Y., Zeng, B., and Chilimbi, T. Multi-modal alignment using representation codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15651--15660, 2022

    2022

  9. [17]

    Eslami, S., Meinel, C., and De Melo, G. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pp.\ 1181--1193, 2023

    2023

  10. [18]

    Simcse: Simple contrastive learning of sentence embeddings

    Gao, T., Yao, X., and Chen, D. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021

    2021 arXiv

  11. [19]

    Uncurated image-text datasets: Shedding light on demographic bias

    Garcia, N., Hirota, Y., Wu, Y., and Nakashima, Y. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 6957--6966, 2023

    2023

  12. [20]

    B., Visweswaran, S., and Batmanghelich, K

    Ghosh, S., Poynton, C. B., Visweswaran, S., and Batmanghelich, K. Mammo-clip: A vision language foundation model to enhance data efficiency and robustness in mammography. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 632--642. ...

    2024

  13. [21]

    Radalign: Advancing radiology report generation with vision-language concept alignment

    Gu, D., Gao, Y., Zhou, Y., Zhou, M., and Metaxas, D. Radalign: Advancing radiology report generation with vision-language concept alignment. arXiv preprint arXiv:2501.07525, 2025

    2025 arXiv

  14. [22]

    Vector quantized diffusion model for text-to-image synthesis

    Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10696--10706, 2022

    2022

  15. [23]

    Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., and Hoi, S. C. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846, 2022

    2022 arXiv

  16. [24]

    Integrating medclip and cross-modal fusion for automatic radiology report generation

    Han, Q., Liu, J., Qin, Z., and Zheng, Z. Integrating medclip and cross-modal fusion for automatic radiology report generation. In 2024 IEEE International Conference on Big Data (BigData), pp.\ 7313--7317. IEEE, 2024

    2024

  17. [25]

    M., Griffith, B., and Klochko, C

    Hayden, N., Gilbert, S., Poisson, L. M., Griffith, B., and Klochko, C. Performance of gpt-4 with vision on text-and image-based acr diagnostic radiology in-training examination questions. Radiology, 312 0 (3): 0 e240153, 2024

    2024

  18. [26]

    D., Xu, D., and Popescu, M

    He, F., Liu, K., Yang, Z., Chen, Y., Hammer, R. D., Xu, D., and Popescu, M. pathclip: Detection of genes and gene relations from biological pathway figures through image-text contrastive learning. IEEE Journal of Biomedical and Health Informatics, 2024

    2024

  19. [27]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  20. [28]

    Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., and Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, 3 0 (1): 0 136, 2020

    2020

  21. [29]

    P., and Yeung, S

    Huang, S.-C., Shen, L., Lungren, M. P., and Yeung, S. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3942--3951, 2021

    2021

  22. [30]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial inte...

    2019

  23. [31]

    Gpt-4 vision: Multi-modal evolution of chatgpt and potential role in radiology

    Javan, R., Kim, T., and Mostaghni, N. Gpt-4 vision: Multi-modal evolution of chatgpt and potential role in radiology. Cureus, 16 0 (8): 0 e68298, 2024

    2024

  24. [32]

    Multimodal fusion with recurrent neural networks for rumor detection on microblogs

    Jin, Z., Cao, J., Guo, H., Zhang, Y., and Luo, J. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM international conference on Multimedia, pp.\ 795--816, 2017

    2017

  25. [33]

    D., and Jawahar, C

    Khare, Y., Bagal, V., Mathew, M., Devi, A., Priyakumar, U. D., and Jawahar, C. Mmbert: Multimodal bert pretraining for improved medical vqa. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp.\ 1033--1036. IEEE, 2021

    2021

  26. [34]

    H., Park, C

    Kim, H., Kim, P., Joo, I., Kim, J. H., Park, C. M., and Yoon, S. H. Chatgpt vision for radiological interpretation: an investigation using medical school radiology examinations. Korean Journal of Radiology, 25 0 (4): 0 403, 2024

    2024

  27. [35]

    C., Lo, W.-Y., et al

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023

    2023

  28. [36]

    Medclip-sam: Bridging text and image towards universal medical image segmentation

    Koleilat, T., Asgariandehkordi, H., Rivaz, H., and Xiao, Y. Medclip-sam: Bridging text and image towards universal medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 643--653. Springer, 2024

    2024

  29. [37]

    A., et al

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123: 0 32--73, 2017

    2017

  30. [38]

    Veclip: Improving clip training via visual-enriched captions, 2023

    Lai, Z., Zhang, H., Zhang, B., Wu, W., Bai, H., Timofeev, A., Du, X., Gan, Z., Shan, J., Chuah, C.-N., et al. Veclip: Improving clip training via visual-enriched captions, 2023. URL https://arxiv. org/abs/2310.07699

    2023 arXiv

  31. [39]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Lauren c on, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A., Kiela, D., et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Deep learning based approach for emotion recognition using image-text fusion

    Li, H., Liu, H., Yu, P., Zhao, J., Wan, B., and Li, W. Deep learning based approach for emotion recognition using image-text fusion. In 2023 2nd International Joint Conference on Information and Communication Engineering (JCICE), pp.\ 11--15. IEEE, 2023

    2023

  33. [41]

    Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 0 9694--9705, 2021

    2021

  34. [42]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  35. [43]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents

    Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., and Xie, W. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 525--536. Springer, 2023

    2023

  36. [44]

    V., Ravazio, R., Kupssinsk \"u , L., Parraga, O., Delucis, M

    Mattjie, C., De Moura, L. V., Ravazio, R., Kupssinsk \"u , L., Parraga, O., Delucis, M. M., and Barros, R. C. Zero-shot performance of the segment anything model (sam) in 2d medical imaging: A comprehensive evaluation and practical guidelines. In 2023 IEEE 23rd International C...

    2023

  37. [45]

    Improving zero-shot detection of low prevalence chest pathologies using domain pre-trained language models

    Mishra, A., Mittal, R., Jestin, C., Tingos, K., and Rajpurkar, P. Improving zero-shot detection of low prevalence chest pathologies using domain pre-trained language models. arXiv preprint arXiv:2306.08000, 2023

    2023 arXiv

  38. [46]

    and Bagdanov, A

    Mistretta, M. and Bagdanov, A. D. Re-tune: Incremental fine tuning of biomedical vision-language models for multi-label chest x-ray classification. arXiv preprint arXiv:2410.17827, 2024

    2024 arXiv

  39. [47]

    Mongan, J., Moy, L., and Kahn Jr, C. E. Checklist for artificial intelligence in medical imaging (claim): a guide for authors and reviewers, 2020

    2020

  40. [48]

    H., Lee, H., Shin, W., Kim, Y.-H., and Choi, E

    Moon, J. H., Lee, H., Shin, W., Kim, Y.-H., and Choi, E. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Journal of Biomedical and Health Informatics, 26 0 (12): 0 6070--6080, 2022

    2022

  41. [49]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  42. [50]

    Pelka, O., Koitka, S., R \"u ckert, J., Nensa, F., and Friedrich, C. M. Radiology objects in context (roco): a multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint ...

    2018

  43. [51]

    P., Senadeera, M., Jacobs, S., Coghlan, S., and Le, V

    Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., and Le, V. Trust and medical AI : the challenges we face and the expertise needed to overcome them. Journal of the American Medical Informatics Association, 28 0 (4): 0 890--894, 2021

    2021

  44. [52]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  45. [53]

    A., Tavakoli, N., Kim, G

    Rahsepar, A. A., Tavakoli, N., Kim, G. H. J., Hassani, C., Abtin, F., and Bedayat, A. How AI responds to common lung cancer questions: ChatGPT versus Google Bard . Radiology, 307 0 (5): 0 e230922, 2023

    2023

  46. [54]

    Rajpurkar, P., Chen, E., Banerjee, O., and Topol, E. J. AI in health and medicine. Nature medicine, 28 0 (1): 0 31--38, 2022

    2022

  47. [55]

    Zero-shot text-to-image generation

    Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International conference on machine learning, pp.\ 8821--8831. Pmlr, 2021

    2021

  48. [56]

    R., Disch, N., Holzschuh, J., Zimmerer, D., and Maier-Hein, K

    Roy, S., Wald, T., Koehler, G., Rokuss, M. R., Disch, N., Holzschuh, J., Zimmerer, D., and Maier-Hein, K. H. Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model. arXiv preprint arXiv:2304.05396, 2023

    2023 arXiv

  49. [57]

    Improved techniques for training gans

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016

    2016

  50. [58]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 2556--...

    2018

  51. [59]

    Flava: A foundational language and vision alignment model

    Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15638--15650, 2022

    2022

  52. [60]

    A., Newsome, J

    Tariq, A., Celi, L. A., Newsome, J. M., Purkayastha, S., Bhatia, N. K., Trivedi, H., Gichoya, J. W., and Banerjee, I. Patient-specific COVID-19 resource utilization prediction using fusion AI model. NPJ digital medicine, 4 0 (1): 0 94, 2021

    2021

  53. [61]

    S., Klontzas, M

    Tejani, A. S., Klontzas, M. E., Gatti, A. A., Mongan, J., Moy, L., Park, S. H., and Kahn Jr, C. E. Updating the checklist for artificial intelligence in medical imaging (claim) for reporting AI research. Nature Machine Intelligence, 5 0 (9): 0 950--951, 2023

    2023

  54. [62]

    S., Klontzas, M

    Tejani, A. S., Klontzas, M. E., Gatti, A. A., Mongan, J. T., Moy, L., Park, S. H., Kahn Jr, C. E., and Panel, C. . U. Checklist for artificial intelligence in medical imaging (claim): 2024 update. Radiology: Artificial Intelligence, pp.\ e240300, 2024

    2024

  55. [63]

    P., Ng, A

    Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., and Rajpurkar, P. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6 0 (12): 0 1399--1406, 2022

    2022

  56. [64]

    On large visual language models for medical imaging analysis: An empirical study

    Van, M.-H., Verma, P., and Wu, X. On large visual language models for medical imaging analysis: An empirical study. In 2024 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pp.\ 172--176. IEEE, 2024

    2024

  57. [65]

    Q., Wang, L., Wong, K

    Vo, H. Q., Wang, L., Wong, K. K., Ezeana, C. F., Yu, X., Yang, W., Chang, J., Nguyen, H. V., and Wong, S. T. Frozen large-scale pretrained vision-language models are the effective foundational backbone for multimodal breast cancer prediction. IEEE Journal of Biomedical and Hea...

    2024

  58. [66]

    Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias

    Wan, Z., Liu, C., Zhang, M., Fu, J., Wang, B., Cheng, S., Ma, L., Quilodr \'a n-Casas, C., and Arcucci, R. Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias. Advances in Neural Information Processing Systems, 36, 2024

    2024

  59. [67]

    L., and Boker, H

    Wang, F., Mizrachi, S., Beladev, M., Nadav, G., Amsalem, G., Assaraf, K. L., and Boker, H. H. Mumic--multimodal embedding for multi-label image classification with tempered sigmoid. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 15603--15611, 2023

    2023

  60. [68]

    Medclip: Contrastive learning from unpaired medical images and text

    Wang, Z., Wu, Z., Agarwal, D., and Sun, J. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022

    2022 arXiv

  61. [69]

    Medim: Boost medical image representation via radiology report-guided masking

    Xie, Y., Gu, L., Harada, T., Zhang, J., Xia, Y., and Wu, Q. Medim: Boost medical image representation via radiology report-guided masking. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 13--23. Springer, 2023

    2023

  62. [70]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., and Feichtenhofer, C. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021

    2021 arXiv

  63. [71]

    Versatile diffusion: Text, images and variations all in one diffusion model

    Xu, X., Wang, Z., Zhang, G., Wang, K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 7754--7765, 2023

    2023

  64. [72]

    T., Bajwa, J., Jacob, J., Pinnock, M

    Yildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. Multimodal healthcare ai: identifying and designing clinically relevant vision-language applications for radiology. In Procee...

    2024

  65. [73]

    K., Baek, W., and Roh, B

    You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E. K., Baek, W., and Roh, B. Cxr-clip: Toward large scale chest x-ray language-image pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.\ 101--111. Springer, 2023

    2023

  66. [74]

    Attention prompting on image for large vision-language models

    Yu, R., Yu, W., and Wang, X. Attention prompting on image for large vision-language models. In European Conference on Computer Vision, pp.\ 251--268. Springer, 2025

    2025

  67. [75]

    and Ma \'n dziuk, J

    \.Z elaszczyk, M. and Ma \'n dziuk, J. Cross-modal text and visual generation: A systematic review. part 1: Image to text. Information Fusion, 93: 0 302--329, 2023

    2023

  68. [76]

    Multi-task paired masking with alignment modeling for medical vision-language pre-training

    Zhang, K., Yang, Y., Yu, J., Jiang, H., Fan, J., Huang, Q., and Han, W. Multi-task paired masking with alignment modeling for medical vision-language pre-training. IEEE Transactions on Multimedia, 2023 a

    2023

  69. [77]

    Large-scale domain-specific pretraining for biomedical vision-language processing

    Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2 0 (3): 0 6, 2023 b

    2023 arXiv

  70. [78]

    D., and Langlotz, C

    Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp.\ 2--25. PMLR, 2022

    2022

  71. [79]

    A multimodal fusion approach for image captioning

    Zhao, D., Chang, Z., and Guo, S. A multimodal fusion approach for image captioning. Neurocomputing, 329: 0 476--485, 2019

    2019

  72. [80]

    Fine-tuning large language models for domain-specific machine translation

    Zheng, J., Hong, H., Wang, X., Su, J., Liang, Y., and Wu, S. Fine-tuning large language models for domain-specific machine translation. arXiv preprint arXiv:2402.15061, 2024

    2024 arXiv

  73. [81]

    Advancing radiograph representation learning with masked record modeling

    Zhou, H.-Y., Lian, C., Wang, L., and Yu, Y. Advancing radiograph representation learning with masked record modeling. arXiv preprint arXiv:2301.13155, 2023

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.