Pith. sign in

REVIEW 4 major objections 5 minor 39 references

Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An 8-billion-parameter fundus model beats a 40-billion-parameter medical rival on retinal question answering by 26.6 percentage points.

desk verdict A genuinely useful data engine and model, but the report-generation headline rests on a judge that is also the baseline; needs human validation before that 77% vs 47.6% claim can be trusted. read the letter →

arxiv 2507.17539 v1 pith:356JH2DH submitted 2025-07-23 cs.AI cs.CVeess.IV

Xinyao Liu , Diping Song This is my paper
classification cs.AIcs.CVeess.IV
keywords fundusimagingmultimodallargelanguagemodelophthalmologyclinicalreasoningregionlocalizationmedicalreportgenerationinstructiontuningdatascalinglaw
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FundusExpert is an attempt to show that an ophthalmology-specific multimodal language model can beat much larger general medical models at reading retinal photographs, and that the decisive ingredient is not model scale but training data shaped like clinical reasoning. The paper builds FundusGen, roughly 200,000 fundus images annotated at three levels (disease, lesion boxes, feature text) and organized into multi-turn cognitive-chain dialogues that move from localizing a region to analyzing its features to concluding a diagnosis. On out-of-distribution fundus questions from a general medical benchmark, the 8B FundusExpert scores 66.7% versus 40.1% for the 40B MedRegA, and its zero-shot reports reach 77.0% clinical consistency versus GPT-4o's 47.6%. The authors also report a data-quality scaling law, $L \propto N^{0.068}$, which they interpret as evidence that cognitive-chain annotations make each training sample more useful.

What carries the argument

The clinical cognitive chain is the load-bearing mechanism. It is a multi-turn instruction format that forces the model to first identify and localize abnormal regions (often with explicit bounding-box references), then describe their features, then infer a diagnosis, optionally followed by a confirmation turn that deepens the evidence chain for a specific suspected disease. The second machinery is Fundus-Engine, the data-generation pipeline: nnU-Net segmentation models produce lesion and structure masks, DBSCAN clustering converts those masks into bounding boxes, and MLLM-driven semantic expansion converts the structured labels into natural-language reports and reasoning dialogues. Together these turn fragmented ophthalmic annotations into unified, clinically ordered training examples.

What would settle it

A concrete test: take FundusExpert's zero-shot grounding reports on a held-out fundus set, extract the bounding-box coordinates for hard exudates and microaneurysms, and compare them to expert-annotated lesion regions with Intersection-over-Union. If lesion IoU stays near the 0.1-0.2 range implied by the paper's segmentation overlap scores, while report-quality gains remain large, then precise lesion boxes are not what carries the diagnostic improvement; if lesion IoU is high, the pseudo-label pipeline is validated.

Watch

Extended reading notes

Core claim

FundusExpert establishes the central claim that region-level localization and diagnostic reasoning can be trained together in one ophthalmic multimodal language model, and that this joint training transfers to unseen diseases. Fundus-Engine automatically produces bounding boxes by clustering nnU-Net segmentations, then uses an MLLM to turn discrete labels into clinically worded descriptions and multi-turn reasoning dialogues. Fine-tuning an 8B generalist MLLM on this data yields the top accuracy on both the in-domain Fundus-MMBench (69.7%) and the out-of-distribution GMAI-MMBench fundus subset (66.7%), surpassing the 40B MedRegA by 26.6 percentage points on the latter. In zero-shot report generation, FundusExpert's reports match the ground-truth labels with 77.0% clinical consistency under a GPT-4o-based semantic scorer, compared to GPT-4o's own 47.6%.

Load-bearing premise

The load-bearing premise is that automatically generated lesion boxes are accurate enough to teach correct spatial associations, even though on an out-of-domain test the segmentation masks behind them overlap true lesions by only about a fifth to a third for hard exudates, microaneurysms, and cotton-wool spots.

Editorial extensions

If this is right

  • An 8B domain-specific model can outperform a 40B generalist medical model on fundus question answering, suggesting model scale is not the limiting factor once reasoning-aligned data exists.
  • Region-level supervision is required for out-of-distribution generalization: removing region data lowers GMAI-MMBench accuracy by 5.4 percentage points.
  • Unbroken multi-turn cognitive chains matter: splitting them into single-turn tasks costs 3.5 points on average and 4.8 points on 21 complex or rare diseases.
  • Cognitive-chain annotations make data more efficient: 10% of FundusGen matches 100% of classification-annotation-only data, with a fitted scaling law $L \propto N^{0.068}$.
  • The trained model can replace commercial MLLMs as the data generator for the next round of dataset construction, since fine-tuning on FundusExpert-generated text outperforms GPT-4o-generated text on both tested base models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same cognitive-chain recipe (localize, analyze features, then diagnose) could be adapted to other imaging specialties, such as chest X-ray or pathology, where localization evidence and diagnostic decisions are tightly coupled.
  • Beyond the paper: the reported lesion-localization IoUs (about 0.12-0.19 for microaneurysms and hard exudates) are far below the disc and cup IoUs, so the report-quality gains may come mostly from global reasoning patterns rather than from precise lesion boxes; a lesion-level ablation would separate these.
  • Beyond the paper: a direct test would be to have ophthalmologists independently score the same generated reports instead of a VLM judge; if human scorers do not reproduce the 77% versus 47.6% gap, the evaluation itself is the main uncertainty.
  • Beyond the paper: the measured scaling-law exponent implies a testable claim: adding cognitive-chain annotations to a fixed set of images should improve performance more than adding the same number of images with flat labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Fundus-Engine, an automated pipeline that converts fundus images plus segmentation and disease labels into the FundusGen instruction dataset with a 'region localization → feature analysis → diagnostic reasoning' cognitive chain. The authors fine-tune InternVL2.5-8B on this dataset to obtain FundusExpert, an 8B ophthalmic MLLM. The main claims are: (1) FundusExpert achieves state-of-the-art accuracy on ophthalmic question answering, outperforming the 40B MedRegA by 26.6% on the GMAI-MMBench fundus subset and reaching 69.7% on the new Fundus-MMBench; (2) it outperforms GPT-4o in zero-shot report generation with a clinical consistency of 77.0% versus 47.6%; (3) an empirical scaling law L ∝ N^0.068 is observed between data volume and model accuracy. The paper also reports ablation studies showing the contributions of cognitive-chain, region-aware, and startup data, plus pseudo-label quality results for the automated bounding-box annotation.

Significance. If the results hold, the work is significant: it demonstrates that a compact 8B model can outperform much larger generalist medical MLLMs on ophthalmic tasks when fine-tuned on a carefully structured, region-grounded instruction dataset with explicit clinical reasoning chains. The FundusGen dataset and Fundus-MMBench evaluation framework are potentially reusable resources, and the paper's ablations attempt to isolate the contributions of different annotation components. The scaling-law observation, while empirical, points to an interesting relationship between annotation granularity and data efficiency. However, the headline report-generation comparison relies on a judge that is also the baseline competitor, and the GMAI-MMBench results are based on very small per-category samples, which currently limits the certainty of the findings.

major comments (4)
  1. [Section 5.1.2, Clinical Consistency metric, and Figure S5] The report-generation claim (77.0% vs. 47.6%) is evaluated by GPT-4o, which is simultaneously the baseline model being compared. The metric lets the judge define both S and the union denominator |L ∪ S|, and the baseline prompt shown in Figure S5 instructs GPT-4o to 'give a detailed description' rather than to emit structured diagnostic labels; if the judge counts missing labels as misses, this can systematically disadvantage GPT-4o and inflate the reported gap. The authors should add a blinded human clinician evaluation with inter-rater agreement statistics, or use an independent judge model that is not also a baseline, and should ensure the same output instruction format for all compared models.
  2. [Section 4.3 and Figure 4] The paper explicitly acknowledges that over 90% of GMAI-MMBench fundus categories contain around five test samples each. The headline 26.6% accuracy margin over MedRegA and the 66.7% GMAI-MMBench accuracy therefore carry very high sampling variance, and no confidence intervals or significance tests are reported. Please provide per-category sample counts, bootstrap or exact confidence intervals, or a statistical test, and state how the headline margin changes if the analysis is restricted to categories with at least ten samples.
  3. [Section 3, Stage 2 and Section 5.5, Table 4] The pseudo-label bounding boxes are generated from nnU-Net segmentations whose out-of-domain Dice on Messidor is 29.1% for hard exudates and 21.1% for microaneurysms. Because these pseudo-labels are the only grounding supervision in FundusGen, the positioning-diagnosis claims depend on their quality; Table 4 demonstrates cross-domain feasibility only on Messidor and does not validate the in-house pseudo-labels used for training. The authors should add human validation on a random sample of generated boxes, evaluate FundusExpert's detection on a ground-truth-labeled fundus set, and discuss how label noise affects the Table 2 IoU numbers.
  4. [Section 5.2 and Figure 5] The scaling law L ∝ N^0.068 is an empirical fit over a small number of data points (the number and identity of sampled data percentages are not reported), with adjusted R² = 0.930 on Fundus-MMBench. The comparison with Classification Annotation-Guided Data is also confounded because the control condition differs in two ways simultaneously (no localization information and no explicit reasoning instructions). Please report the exact data fractions and points, provide parameter uncertainties, and add an ablation that varies only one annotation component at a time to support the claim that cognitive alignment annotations enhance data utilization efficiency.
minor comments (5)
  1. [Appendix V.1, Table S2] The model name 'InternerVL2.5' appears to be a typo for 'InternVL2.5'; please correct it.
  2. [Section 5.1.1 and Table 1] The spacing in 'LLaV A-v1.5' is inconsistent; use 'LLaVA-v1.5' throughout.
  3. [Section 3.1] The phrase '20w fundus images' should be written as '200K fundus images' or '200,000 fundus images' for clarity.
  4. [References and Appendix I.1] The dataset name is written as both 'MESSIDOR2' and 'Messidor' at different points; please standardize.
  5. [Section 5.2, Figure 5 caption] The figure reports R², adjusted R², and MSE but the number of fitted data points is not given; a reader cannot judge the stability of the fit without this information.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation is present: QA gains are measured on external/isolated benchmarks, and the scaling law is an empirical fit; the GPT-4o-as-judge report metric is an evaluation confound, not a circular step.

full rationale

I traced the claimed derivation chain: FundusGen is built from nnU-Net pseudo-labels, GPT-4o semantic expansion, and human review; FundusExpert is fine-tuned on FundusGen; and the headline QA results are measured on Fundus-MMBench (strictly isolated from training data) and GMAI-MMBench (an external benchmark). No parameter is fitted to these benchmarks and then renamed a prediction, and the 26.6% improvement over MedRegA is an externally verified comparison rather than a construction. The scaling law L ~ N^0.068 is presented as a fitted power-law curve over accuracy at different data percentages, with R^2 and MSE reported; it is an empirical summary of measured points, not a result that is equivalent to its own input by definition. The only self-citation is VisionUnite in Related Works, and it is descriptive background, not load-bearing. No uniqueness theorem or prior-work ansatz is imported to force the model choice. The one legitimate concern is the Clinical Consistency metric in Section 5.1.2: GPT-4o is both the VLM judge and the baseline being compared (77.0% vs 47.6%), and Figure S5 shows the GPT-4o baseline prompt suppressing diagnostic labels. This is an evaluation-reliability confound that could bias the comparison, but it does not make any claimed result equal to its own input by construction: the metric is applied identically to both models, and the judge is not fitted to produce the reported gap. Therefore no circular step is established, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several fitted hyperparameters, the most important being the scaling exponent. They also rely on assumptions about the reliability of GPT-4o-generated text, the accuracy of nnU-Net pseudo-labels, the appropriateness of a power-law fit, and the impartiality of a GPT-4o judge that is also a competing model. These assumptions are stated or implicit in the pipeline and evaluation sections.

free parameters (5)
  • scaling exponent alpha = 0.068
    Fit to model accuracy vs data percentage on GMAI-MMBench, Section 5.2, Figure 5.
  • DBSCAN epsilon = 160
    Set for clustering segmentation pixels into boxes, Appendix IV.1.
  • DBSCAN min_samples = 10
    Set for clustering segmentation pixels into boxes, Appendix IV.1.
  • box area threshold = >100 pixels
    Boxes smaller than threshold are discarded, Appendix IV.1.
  • top-3 boxes retained = 3
    Only the three largest boxes are kept after sorting by area, Appendix IV.1.
assumptions (4)
  • domain assumption GPT-4o-generated semantic descriptions are clinically valid and traceable to image features
    Stage 3 of Fundus-Engine relies on GPT-4o to expand structured labels into natural language; if these texts contain hallucinations or misalignments, FundusGen inherits the noise.
  • domain assumption nnU-Net pseudo-labels provide adequate localization supervision
    Section 5.5 reports Dice around 29% for lesions on Messidor; these pseudo-labels are used to create bounding boxes for FundusGen.
  • ad hoc to paper A power-law function L = c N^alpha accurately models the accuracy-data relationship
    Section 5.2 fits L ∝ N^0.068 without testing alternative functional forms or providing out-of-sample validation.
  • domain assumption GPT-4o as a VLM judge gives an unbiased clinical consistency score for both models' reports
    Section 5.1.2 defines the metric and uses GPT-4o to extract and match semantic features; this assumption is untested against human annotations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoning." pith.science (2026). https://pith.science/paper/356JH2DH

@misc{pith2026250717539,
  author       = {Pith},
  title        = {Pith review of: Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/356JH2DH}},
  note         = {Machine review of arXiv:2507.17539}
}
abstract

Multimodal large language models (MLLMs) demonstrate significant potential in the field of medical diagnosis. However, they face critical challenges in specialized domains such as ophthalmology, particularly the fragmentation of annotation granularity and inconsistencies in clinical reasoning logic, which hinder precise cross-modal understanding. This paper introduces FundusExpert, an ophthalmology-specific MLLM with integrated positioning-diagnosis reasoning capabilities, along with FundusGen, a dataset constructed through the intelligent Fundus-Engine system. Fundus-Engine automates localization and leverages MLLM-based semantic expansion to integrate global disease classification, local object detection, and fine-grained feature analysis within a single fundus image. Additionally, by constructing a clinically aligned cognitive chain, it guides the model to generate interpretable reasoning paths. FundusExpert, fine-tuned with instruction data from FundusGen, achieves the best performance in ophthalmic question-answering tasks, surpassing the average accuracy of the 40B MedRegA by 26.6%. It also excels in zero-shot report generation tasks, achieving a clinical consistency of 77.0%, significantly outperforming GPT-4o's 47.6%. Furthermore, we reveal a scaling law between data quality and model capability ($L \propto N^{0.068}$), demonstrating that the cognitive alignment annotations in FundusGen enhance data utilization efficiency. By integrating region-level localization with diagnostic reasoning chains, our work develops a scalable, clinically-aligned MLLM and explores a pathway toward bridging the visual-language gap in specific MLLMs. Our project can be found at https://github.com/MeteorElf/FundusExpert.

Figures

Figures reproduced from arXiv: 2507.17539 by the authors.

Figure 1
Figure 1. Comparison of FundusExpert with other models. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Collaborative Annotation of Fundus-Engine. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Two Typical Interaction Patterns of Clinically Aligned Cognitive Chains and Curated Instruction Fine-tuning Data Scheme. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Zero-shot Performance Improvement for Out-of-domain Testing. (Each category has five samples) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The change of model accuracy on GMAI-MMBench(left) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

  1. [1]

    https://www.kaggle

    Aptos 2019 blindness detection. https://www.kaggle. com/c/aptos2019- blindness- detection . Ac- cessed: 2025-03-08. 3, 1 5

    work page 2019

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 6

    arXiv 2023

  3. [3]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 1

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 6

    arXiv 2025

  5. [5]

    Biomedlm: A 2.7 b parameter language model trained on biomedical text

    Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. Biomedlm: A 2.7 b parameter language model trained on biomedical text. arXiv preprint arXiv:2403.18421, 2024. 1

    arXiv 2024

  6. [6]

    Gmai-mmbench: A comprehensive mul- timodal evaluation benchmark towards general medical ai

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongy- ing Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. Gmai-mmbench: A comprehensive mul- timodal evaluation benchmark towards general medical ai. arXiv preprint arXiv:2408.03361, 2024. 2, 5, 6

    arXiv 2024

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 5, 6, 7, 8, 3

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 1

    work page 2024

Show all 39 references
  1. [9]

    Feedback on a publicly distributed image database: the mes- sidor database

    Etienne Decenci `ere, Xiwei Zhang, Guy Cazuguel, Bruno Lay, B´eatrice Cochener, Caroline Trone, Philippe Gain, John- Richard Ord´o˜nez-Varela, Pascale Massin, Ali Erginay, et al. Feedback on a publicly distributed image database: the mes- sidor database. Image Analysis & Stere...

  2. [10]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia,...

    2024

  3. [11]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, J¨org Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, pages 226–231,

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 8

    2025 arXiv

  5. [13]

    Meddr: Diagnosis- guided bootstrapping for large-scale medical vision-language learning

    Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hong- mei Wang, Shu Yang, and Hao Chen. Meddr: Diagnosis- guided bootstrapping for large-scale medical vision-language learning. arXiv preprint arXiv:2404.15127, 2024. 1, 3, 6

    2024 arXiv

  6. [14]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Na- ture methods, 18(2):203–211, 2021. 4

    2021

  7. [15]

    Cataract dataset

    jr2ngb. Cataract dataset. https://www.kaggle.com/ datasets/jr2ngb/cataractdataset . Accessed: 2025-03-08. 1

    2025

  8. [16]

    Machine learn for glaucoma, 2018

    Ungsoo Kim. Machine learn for glaucoma, 2018. 1

    2018

  9. [17]

    Oleksandr Kovalyk, Juan Morales-S ´anchez, Rafael Verd ´u- Monedero, Inmaculada Sell ´es-Navarro, Ana Palaz ´on- Cabanes, and Jos ´e-Luis Sancho-G ´omez. PAPILA. 2022. 3, 1

    2022

  10. [18]

    Inte- grated image-based deep learning and language models for primary diabetes care

    Jiajia Li, Zhouyu Guan, Jing Wang, Carol Y Cheung, Yingfeng Zheng, Lee-Ling Lim, Cynthia Ciwei Lim, Paisan Ruamviboonsuk, Rajiv Raman, Leonor Corsino, et al. Inte- grated image-based deep learning and language models for primary diabetes care. Nature medicine, 30(10):2886–2896,

  11. [19]

    Applications of deep learning in fundus images: A review

    Tao Li, Wang Bo, Chunyu Hu, Hong Kang, Hanruo Liu, Kai Wang, and Huazhu Fu. Applications of deep learning in fundus images: A review. Medical Image Analysis, 69: 101971, 2021. 1

    2021

  12. [20]

    Gmai-vl & gmai-vl-5.5 m: A large vision-language model and a comprehensive multimodal dataset towards gen- eral medical ai

    Tianbin Li, Yanzhou Su, Wei Li, Bin Fu, Zhe Chen, Ziyan Huang, Guoan Wang, Chenglong Ma, Ying Chen, Ming Hu, et al. Gmai-vl & gmai-vl-5.5 m: A large vision-language model and a comprehensive multimodal dataset towards gen- eral medical ai. arXiv preprint arXiv:2411.14522, 2024. 6

    2024 arXiv

  13. [21]

    Visio- nunite: A vision-language foundation model for ophthal- mology enhanced with clinical knowledge

    Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E Kinahan, and Yu Qiao. Visio- nunite: A vision-language foundation model for ophthal- mology enhanced with clinical knowledge. arXiv preprint arXiv:2408.02865, 2024. 3

    2024 arXiv

  14. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 6, 7, 3

    2024

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1

    2024

  16. [24]

    Brset: a brazilian multilabel oph- thalmological dataset of retina fundus photos

    Luis Filipe Nakayama, David Restrepo, Jo ˜ao Matos, Lu- cas Zago Ribeiro, Fernando Korn Malerbi, Leo Anthony Celi, and Caio Saito Regatieri. Brset: a brazilian multilabel oph- thalmological dataset of retina fundus photos. PLOS Digital Health, 3(7):e0000454, 2024. 3, 1

    2024

  17. [25]

    OpenAI. Chatgpt. https://openai.com/blog/ chatgpt, 2023. Accessed: 2023-12-22. 1

    2023

  18. [26]

    Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research

    Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data, 3(3):25, 2018. 3, 1

    2018

  19. [27]

    Development and validation of a multimodal multitask vision foundation model for generalist ophthalmic artificial intelligence

    Jianing Qiu, Jian Wu, Hao Wei, Peilun Shi, Minqing Zhang, Yunyun Sun, Lin Li, Hanruo Liu, Hongyi Liu, Simeng Hou, et al. Development and validation of a multimodal multitask vision foundation model for generalist ophthalmic artificial intelligence. NEJM AI, 1(12):AIoa2300221, ...

    2024

  20. [28]

    Eyeclip: A visual-language founda- tion model for multi-modal ophthalmic image analysis

    Danli Shi, Weiyi Zhang, Jiancheng Yang, Siyu Huang, Xi- aolan Chen, Mayinuer Yusufu, Kai Jin, Shan Lin, Shunming Liu, Qing Zhang, et al. Eyeclip: A visual-language founda- tion model for multi-modal ophthalmic image analysis. arXiv preprint arXiv:2409.06644, 2024. 1

    2024 arXiv

  21. [29]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6

    2023 arXiv

  22. [30]

    Towards generalist biomedical ai

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138, 2024. 1

    2024

  23. [31]

    Interpretable bilin- gual multimodal large language model for diverse biomedical tasks

    Lehan Wang, Haonan Wang, Honglong Yang, Jiaji Mao, Ze- hong Yang, Jun Shen, and Xiaomeng Li. Interpretable bilin- gual multimodal large language model for diverse biomedical tasks. arXiv preprint arXiv:2410.18387, 2024. 1, 6, 7

    2024 arXiv

  24. [32]

    Common and rare fundus diseases identification using vision-language foundation model with knowledge of over 400 diseases

    Meng Wang, Tian Lin, Aidi Lin, Kai Yu, Yuanyuan Peng, Lianyu Wang, Cheng Chen, Ke Zou, Huiyu Liang, Man Chen, et al. Common and rare fundus diseases identification using vision-language foundation model with knowledge of over 400 diseases. arXiv preprint arXiv:2406.09317, 2024. 2, 6

    2024 arXiv

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 6, 8, 3

    2024 arXiv

  26. [34]

    Mm-retinal: Knowledge-enhanced foun- dational pretraining with fundus image-text expertise

    Ruiqi Wu, Chenran Zhang, Jianle Zhang, Yi Zhou, Tao Zhou, and Huazhu Fu. Mm-retinal: Knowledge-enhanced foun- dational pretraining with fundus image-text expertise. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 722–732. Springer,

  27. [35]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900, 2024. 3

    2024 arXiv

  28. [36]

    Vilref: An expert knowledge enabled vision-language retinal foundation model

    Shengzhu Yang, Jiawei Du, Jia Guo, Weihang Zhang, Hanruo Liu, Huiqi Li, and Ningli Wang. Vilref: An expert knowledge enabled vision-language retinal foundation model. arXiv preprint arXiv:2408.10894, 2024. 3

    2024

  29. [37]

    Automorph: automated retinal vascu- lar morphology quantification via a deep learning pipeline

    Yukun Zhou, Siegfried K Wagner, Mark A Chia, An Zhao, Moucheng Xu, Robbert Struyven, Daniel C Alexander, Pearse A Keane, et al. Automorph: automated retinal vascu- lar morphology quantification via a deep learning pipeline. Translational vision science & technology, 11(7):12–1...

    2022

  30. [38]

    A foundation model for generalizable disease detection from retinal images

    Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981):156–163, 2023. 3

    2023

  31. [39]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1 7

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.