Pith. sign in

REVIEW 4 major objections 6 minor 27 references

A 7B multimodal model turns retinal biomarker measurements into evidence-grounded diagnostic reports, outperforming larger open-source baselines and OpenAI-o3 on biomarker-grounded metrics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

RetiBridge/GROK is a 7B multimodal LLM trained to turn quantitative retinal biomarkers into qualitative diagnoses, outperforming larger open models and OpenAI-o3 on its self-built, LLM-judged benchmark.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection GROK is a plausible training recipe with a genuinely new benchmark, but the headline evaluation is circular—o3 writes the targets, o3 judges the outputs, and no clinician ever checks—so the central 'clinician-grade' claim is not established. the 4 major comments →

arxiv 2510.04281 v2 pith:XDF3TLXI submitted 2025-10-05 cs.AI

RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model

Zhuangzhi Gao , Hongyi Qin , He Zhao , Qinkai Yu , Feixiang Zhou , Fu Wang , Jinru Ding , Eduard Shantsila
show 5 more authors
Uazman Alam Alena Shantsila Wahbi El-Bouri Gregory Y. H. Lip Yalin Zheng
This is my paper
classification cs.AI
keywords multimodal large language modelretinal biomarkerscolor fundus photographyoptical coherence tomographychain-of-thought reasoningknowledge-guided instructionbiomarker groundingLoRA fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces GROK, a multimodal large language model that jointly reads color fundus photographs and OCT scans and writes a diagnostic report that first quantifies abnormal retinal biomarkers, then derives qualitative clinical sub-inferences from those numbers, and finally aligns the sub-inferences with a coherent final diagnosis. The authors claim that, despite using only LoRA fine-tuning of a 7B-parameter Qwen2 backbone, GROK exceeds all evaluated open-source 7B and 32B models—and OpenAI-o3—on quantitative accuracy, evidence grounding, coverage completeness, and semantic report similarity. The reason the claim matters is that most existing ophthalmic AI models are black-box classifiers or unimodal, whereas a biomarker-grounded chain of thought is auditable by clinicians and could support safer deployment. The paper builds a benchmark of paired CFP-OCT images with 37 biomarkers and uses a knowledge-guided instruction pipeline, where a strong closed-source model generates reasoning targets from a doctor-verified prompt template.

Core claim

The central discovery is that an explicit quantitative-to-qualitative reasoning chain can be instilled into a small multimodal LLM through a three-stage recipe: (1) generate fine-grained chain-of-thought training targets by prompting OpenAI-o3 with an expert-crafted Eye-Guideline template that embeds measured biomarkers and diagnostic labels as soft constraints; (2) pre-train an OCT encoder by contrastively aligning 2D central-foveal B-scan embeddings with their corresponding 3D biomarker vectors, so the visual representation is grounded in clinically meaningful numeric features; and (3) supervised instruction fine-tuning with LoRA on a Qwen2-7B backbone, fusing CFP and OCT embeddings throug

What carries the argument

The load-bearing mechanism is the quantitative-to-qualitative chain-of-thought, implemented by three interacting modules: knowledge-guided instruction generation (Eye-Guideline prompts that turn 37 extracted biomarkers into o3-generated, biomarker-anchored reasoning reports), CLIP-style OCT-biomarker alignment (a contrastive loss that ties 2D OCT B-scan embeddings to 3D biomarker vectors), and cross-modal fusion (projection layers that map CFP and OCT embeddings into the LLM's text space, followed by LoRA-tuned autoregressive training on Qwen2-7B). The alignment step is what lets the model estimate numbers like retinal thickness and vessel calibre from images, and the instruction step is wha

Load-bearing premise

The results rest on the assumption that the OpenAI-o3-generated reports, produced by a doctor-verified prompt template that includes diagnostic labels as soft constraints, are clinically correct enough to serve as both training targets and evaluation references; the authors provide no clinician validation of these generated reports, and several key metrics are scored by OpenAI-o3 itself.

What would settle it

Have a panel of ophthalmologists independently score a random sample of GROK's reports for quantitative accuracy and diagnostic correctness against the original images and patient records, without seeing the o3-generated references; if clinician ratings diverge substantially from the o3-based scores, the claim of clinician-grade grounded reasoning fails. Additionally, train GROK on human-written reports for the same images instead of o3-generated targets; a large performance drop would indicate the model is matching a teacher's style rather than learning clinically grounded reasoning.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A 7B model with LoRA fine-tuning can match or outscore 32B open-source models and a frontier closed-source model on biomarker-grounded report metrics.
  • Replacing the domain-aligned OCT encoder with a generic CLIP or RETFound encoder sharply reduces quantitative accuracy (78.2 to 61.5 or 58.8), showing that image-biomarker alignment is essential.
  • Removing the CFP branch causes quantitative accuracy to fall from 78.2 to 41.1 and evidence grounding from 68.7 to 36.3, confirming that joint CFP-OCT reasoning is necessary.
  • Contrastive OCT-biomarker alignment improves regression of 31 biomarkers (R² 0.71 vs 0.66, lower MAE/RMSE), so the encoder learns clinically relevant numeric features.
  • The framework produces an end-to-end, biomarker-anchored rationale that is transparent and verifiable, unlike black-box classifiers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because both the training targets and several headline evaluation metrics are produced by OpenAI-o3, the reported gains may partly reflect imitation of a particular reasoning style; a clinician-annotated reference set would be the real test of clinical accuracy.
  • The recipe (align image encoders to quantitative clinical measurements, then instruction-tune on LLM-generated chain-of-thought) could transfer to other imaging domains where quantitative biomarkers or lab values exist, such as chest CT with lung-function measurements.
  • If the OCT-biomarker alignment genuinely improves quantitative estimation, the model could serve as an automated biomarker extractor, potentially enabling large-scale retinal screening without manual segmentation.
  • The low macro F1 (≤0.25 across all models, including o3) suggests that classification accuracy is far from diagnostic; the paper's contribution is interpretable report quality rather than a deployable classifier.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents GROK (also named RetiBridge in the abstract), a 7B multimodal LLM that jointly analyzes CFP and OCT images, converts quantitative retinal biomarkers into qualitative sub-inferences, and generates a coherent diagnostic conclusion. The method rests on three stages: (1) knowledge-guided instruction generation, where OpenAI-o3 produces training reports from a doctor-verified Eye-Guideline prompt that incorporates 37 biomarkers and the diagnostic label as a soft constraint; (2) CLIP-style alignment of a 2D OCT B-scan encoder with 3D biomarker vectors; and (3) LoRA fine-tuning of Qwen2-7B with cross-modal fusion projectors. Evaluation on a UK Biobank-derived benchmark covers macro-level classification, BERTScore report quality, and an OpenAI-o3-judged rubric of fine-grained clinical quality. The paper claims GROK outperforms open-source 7B/32B baselines and even surpasses OpenAI-o3 on biomarker-grounded metrics.

Significance. If the claims were supported, this would be a meaningful contribution: an interpretable, biomarker-anchored MLLM that jointly uses CFP and OCT, with released code and data, and a reproducible instruction-generation pipeline. The OCT-biomarker regression result (Table III) independently shows that the contrastively aligned encoder predicts numeric biomarkers with lower MAE/RMSE and higher R² than RETFound, which is a concrete, non-circular positive result. However, the central diagnostic claims rest on an unvalidated circular setup: OpenAI-o3 writes the training targets, OpenAI-o3 judges the headline clinical-quality metrics, and clinician evaluation is deferred to future work. The claimed superiority over OpenAI-o3 is therefore not established.

major comments (4)
  1. [§III-B, Eq. (1) and §IV-C] The instruction targets are generated as y_n = OpenAI-o3(EyeGuideline(I_o, I_c, B, g)), and the fine-grained clinical quality metrics (Quantitative Accuracy, Evidence Grounding, Coverage Completeness, etc.) are scored by an OpenAI-o3 LLM judge. Section V explicitly states that clinician evaluation is future work. This is a self-referential loop: GROK is fine-tuned to imitate o3's output style and then evaluated by the same model family against o3-written references. The headline claim that GROK 'surpasses OpenAI-o3' on these metrics is therefore not a valid measure of clinical grounding. The authors should provide human expert scoring on a held-out test set, or at minimum validate a representative sample of the o3-generated reference reports.
  2. [§III-B, Eq. (1)] The diagnostic label g is supplied to OpenAI-o3 as a 'soft constraint' when generating the reference report. At test time, GROK must infer the label from the images, while the reference and the judge already know it. This asymmetry inflates apparent evidence-grounding and reasoning-consistency scores. To support the claimed advantage, the evaluation should be repeated with references generated without conditioning on g, or with clinician-authored references that do not use the answer key.
  3. [§IV-D, Table II] The paper acknowledges that BERTScore may partly reflect stylistic alignment with the training data but does not apply the same caution to the OpenAI-o3-judged fine-grained metrics. Because the training reports are produced by the same model family that performs the judging, the higher Quantitative Accuracy and Evidence Grounding scores may also be artifacts of style imitation rather than clinical superiority. The independent regression result in Table III validates the OCT encoder but not the diagnostic chain-of-thought. The 'clinician-grade' claim in the abstract and introduction is therefore unsupported.
  4. [§IV-A, Table I] The benchmark exhibits severe class imbalance (e.g., AMD 0.86%, Alzheimer 0.18%), and all models achieve Macro F1 ≤ 0.25. The paper describes GROK's Macro F1 of 0.23 as 'competitive,' but this level is far below clinically deployable accuracy. This is not necessarily a flaw of the method, but it contradicts the 'clinician-grade diagnosis' claim and should be tempered or supported with additional evidence such as per-class precision/recall or a clinically meaningful operating threshold.
minor comments (6)
  1. [Title/Abstract vs. Body] The paper title and abstract use 'RetiBridge,' while the body, figures, and the repository name consistently use 'GROK.' This naming inconsistency should be resolved.
  2. [§IV-A] The dataset size is given as 15,663 in the full text but 15,611 in the abstract. Please reconcile the discrepancy.
  3. [§III-C] There are typographical errors, e.g., 'iht heir' in the description of OCT-Biomarker Alignment. Please proofread.
  4. [Table II] The table formatting is hard to read; several entries lack clear separators (e.g., '47.7960.13' appears as one number). Please align columns and add spaces.
  5. [Figure 1] OpenAI-o3 is described as an 'existing MLLM' alongside open-source baselines, but it is a proprietary API model. Clarify the distinction in the text.
  6. [§IV-C] The LLM-as-judge rubric is cited to [27], but no reliability analysis (e.g., agreement with human raters, variance across prompts) is reported. This is especially important given the circularity concern above.

Circularity Check

2 steps flagged

OpenAI-o3 writes the training targets and also scores the headline clinical-quality metrics, so GROK's reported edge over o3 is largely a self-referential evaluation; BERTScore is measured against o3-generated references the model was trained to imitate.

specific steps
  1. fitted input called prediction [Section III-B, Eq. (1); Section IV-C]
    "yn = OpenAI-o3(EyeGuideline(Io_n, Ic_n, Bn, gn)) (1) ... we design a Rubric-based Scoring Framework, implemented with OpenAI-o3 as LLM-judge [27] to score reports against templates, ground truth, and biomarkers."

    The supervision reports that GROK is fine-tuned to reproduce are generated by OpenAI-o3, and the same model family (OpenAI-o3) is then used as the LLM judge for the headline fine-grained clinical metrics (Quantitative Accuracy, Evidence Grounding, Coverage Completeness, Reasoning Consistency). GROK is therefore rewarded for matching the output style and reasoning format of the model that wrote its training targets. The reported 'superiority over OpenAI-o3' on these metrics is assessed by OpenAI-o3 itself, with no clinician validation provided; the paper's Limitations section defers clinician evaluation to future work. This makes the central benchmark claim self-referential rather than independently grounded.

  2. fitted input called prediction [Section IV-D, BERTScore discussion]
    "While this suggests stronger semantic alignment with ground-truth references, it is worth noting that the evaluation is conducted on reports generated in a consistent prompting style with GROK’s training data. Therefore, the BERT-score may partially reflect stylistic alignment rather than general semantic superiority."

    The paper explicitly concedes that the BERTScore comparison (GROK 0.88 vs OpenAI-o3 0.87) is computed against ground-truth reports generated with the same Eye-Guideline prompting style used to create GROK's training targets. Since GROK was supervised on that exact distribution, its higher BERTScore is partly forced by construction. The paper labels this a possible stylistic artifact, yet still presents the BERTScore advantage as evidence of report-generation quality.

full rationale

The main circularity is in the evaluation loop, not in the model architecture. Eq. (1) makes every training target an OpenAI-o3 output conditioned on an Eye-Guideline prompt and the true diagnostic label g; Section IV-C then uses OpenAI-o3 as the LLM judge for the six fine-grained clinical metrics that support the headline claim that GROK 'surpasses OpenAI-o3 on key biomarker-grounded metrics.' Because the judge and the target generator are the same model family, and because the gold reference reports were produced with the answer label in hand, the fine-grained metrics partly measure style imitation and answer-conditioned reference matching rather than independent clinical correctness. The paper itself flags this for BERTScore, and Section V states that clinician evaluation is future work, so the 'clinician-grade' framing is not externally validated. This is not a case of self-citation circularity: the cited encoders (RetiZero, RETFound), the UK Biobank labels, and the OCT-biomarker regression in Table III are independent evidence, and the ablations are informative. However, the paper's central quantitative claim of superiority over OpenAI-o3 rests substantially on an o3-generated, o3-judged benchmark, which is partially circular and inflates the apparent strength of the result. Score 6 reflects that one or more of the headline 'predictions' reduce by construction while meaningful independent components remain.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical entities are postulated. The load-bearing assumptions are the clinical validity of o3-generated instruction data, the accuracy of UK Biobank labels, the sufficiency of the chosen biomarkers/disease categories, and the validity of using the same model family as teacher and judge. These assumptions, especially the first and last, drive the central 'grounded' claim and the reported performance gaps.

free parameters (4)
  • temperature tau = 0.5
    InfoNCE temperature in the OCT-biomarker contrastive loss (Eq. 2), set by hand.
  • biomarker missing-value threshold = 20%
    Used to filter OCT biomarkers in Section III-B; determines which 31 of the available biomarkers are retained and thus shapes the instruction data.
  • training epochs = 50 (alignment stage), 2 (SFT stage)
    Chosen in Section IV-B; part of the experimental setup that affects final performance.
  • LoRA rank and alpha = not reported
    LoRA hyperparameters in Section III-D are not specified in the paper; they are selections that affect the fine-tuned model and are not derived from any principle.
axioms (5)
  • ad hoc to paper OpenAI-o3-generated reports from the Eye-Guideline prompt are clinically accurate and can serve as ground truth for training and evaluation.
    Section III-B defines y_n = OpenAI-o3(EyeGuideline(...)) with no clinician verification; the paper's Limitations (Section V) states clinician evaluation is future work. The entire benchmark labels rest on this premise.
  • domain assumption UK Biobank biomarker values and diagnostic labels are correct ground truth for the images.
    Section IV-A aggregates self-reported illness codes, hospital ICD-10 records, and clinical fields; the labels are not independently confirmed by image-based grading, and 'Normal' means only absence of six listed conditions.
  • domain assumption CLIP-style contrastive alignment between 2D central-foveal OCT B-scans and 3D biomarker vectors produces clinically grounded OCT representations.
    Section III-C assumes the InfoNCE loss transfers quantitative biomarker meaning into the image embedding; the paper provides regression evidence that this helps biomarker prediction, but the clinical usefulness of the representations is not independently established.
  • ad hoc to paper OpenAI-o3 as an LLM judge with the rubric-based scoring framework yields valid clinical quality scores.
    Section IV-C uses OpenAI-o3 to score reports on quantitative accuracy, evidence grounding, reasoning consistency, etc. No comparison to human clinician scoring is provided, and the judge is also the model that generated the training targets.
  • domain assumption Six disease categories and 37 biomarkers are sufficient to represent clinically meaningful ophthalmic/systemic diagnosis in this benchmark.
    Section IV-A selects six classes with a long-tail distribution; the benchmark may not reflect the breadth of real ophthalmic presentations, and the paper notes class imbalance limits all models.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/XDF3TLXI

@misc{pith2026251004281,
  author       = {Pith},
  title        = {Pith review of: RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDF3TLXI}},
  note         = {Machine review of arXiv:2510.04281}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Retinal biomarkers captured by color fundus photography and optical coherence tomography provide clinically valuable evidence for both ocular and systemic diseases. Multimodal large language models (MLLMs) have shown promise for retinal image interpretation, yet existing ophthalmic models rarely quantify these clinically relevant biomarkers or explicitly translate their measurements into qualitative, evidence-grounded diagnostic conclusions. To address this gap, we introduce RetiBridge, a knowledge-guided multimodal large language model that jointly analyzes color fundus photography (CFP), optical coherence tomography (OCT), and text, explicitly bridging quantitative retinal biomarkers to qualitative clinical sub-inferences and coherent diagnostic conclusions. RetiBridge combines knowledge-guided instruction generation, OCT-biomarker alignment, and supervised multimodal instruction tuning to learn a biomarker-grounded quantitative-to-qualitative diagnostic pathway. Using 15,611 paired CFP-OCT samples from UK Biobank with 31 OCT and 6 CFP biomarkers, we construct the Grounded Ophthalmic Understanding benchmark to evaluate diagnostic classification, report generation quality, and fine-grained clinical quality. Despite using only LoRA-based fine-tuning of a 7B-parameter Qwen2 backbone, RetiBridge outperforms all evaluated open-source 7B and 32B baselines, achieving the highest quantitative accuracy, evidence grounding, coverage completeness, and BERTScore, while surpassing OpenAI o3 on these key biomarker-grounded metrics. Our code and data are released in the RetiBridge repository.

Figures

Figures reproduced from arXiv: 2510.04281 by Alena Shantsila, Eduard Shantsila, Feixiang Zhou, Fu Wang, Gregory Y. H. Lip, He Zhao, Hongyi Qin, Jinru Ding, Qinkai Yu, Uazman Alam, Wahbi El-Bouri, Yalin Zheng, Zhuangzhi Gao.

Figure 1
Figure 1. Figure 1: Motivating example. On paired CFP & OCT images, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of GROK’s model architecture. It consists of two clip-style vision encoders (for CFP and OCT), followed by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Knowledge-Guided Instruction Generation: Eye-Guideline prompts and OpenAI-o3 convert CFP/OCT biomarkers into a grounded chain-of-thought report. CLIP-Style OCT-biomarker alignment: A contrastive loss aligns 2-D central-foveal OCT slices with their 3-D biomarker vectors, yielding clinically grounded OCT features. Cross-Modal Fusion: Modality-specific projectors embed CFP and OCT features into a shared langu… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of diagnostic reports generated by GROK and OpenAI-o3 on retinal fundus images from a diabetic [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 5 linked inside Pith

  1. [1]

    Retinal vascular caliber, cardiovascular risk factors, and inflammation: the multi-ethnic study of atherosclerosis (mesa),

    T. Y . Wong, F. A. Islam, R. Klein, B. E. Klein, M. F. Cotch, C. Castro, A. R. Sharrett, and E. Shahar, “Retinal vascular caliber, cardiovascular risk factors, and inflammation: the multi-ethnic study of atherosclerosis (mesa),”Investigative ophthalmology & visual science, vol. 47, no. 6, pp. 2341–2350, 2006

    2006

  2. [2]

    Quantification of nerve fiber layer thickness in normal and glaucomatous eyes using optical coherence tomography: a pilot study,

    J. S. Schuman, M. R. Hee, C. A. Puliafito, C. Wong, T. Pedut-Kloizman, C. P. Lin, E. Hertzmark, J. A. Izatt, E. A. Swanson, and J. G. Fujimoto, “Quantification of nerve fiber layer thickness in normal and glaucomatous eyes using optical coherence tomography: a pilot study,”Archives of ophthalmology, vol. 113, no. 5, pp. 586–596, 1995

    1995

  3. [3]

    Retinal arteriolar diameter and risk for hypertension,

    T. Y . Wong, R. Klein, A. R. Sharrett, B. B. Duncan, D. J. Couper, B. E. Klein, L. D. Hubbard, F. J. Nieto, and A. R. in Communities Study, “Retinal arteriolar diameter and risk for hypertension,”Annals of internal medicine, vol. 140, no. 4, pp. 248–255, 2004

    2004

  4. [4]

    Retinal nerve fiber layer thickness in patients with alzheimer disease,

    S. Kirbas, K. Turkyilmaz, O. Anlar, A. Tufekci, and M. Durmus, “Retinal nerve fiber layer thickness in patients with alzheimer disease,”Journal of Neuro-Ophthalmology, vol. 33, no. 1, pp. 58–61, 2013

    2013

  5. [5]

    Retinal imaging in the twenty-first century: state of the art and future directions,

    P. A. Keane and S. R. Sadda, “Retinal imaging in the twenty-first century: state of the art and future directions,”Ophthalmology, vol. 121, no. 12, pp. 2489–2500, 2014

    2014

  6. [6]

    Automorph: automated retinal vascular morphology quantification via a deep learning pipeline,

    Y . Zhou, S. K. Wagner, M. A. Chia, A. Zhao, M. Xu, R. Struyven, D. C. Alexander, P. A. Keaneet al., “Automorph: automated retinal vascular morphology quantification via a deep learning pipeline,”Translational vision science & technology, vol. 11, no. 7, pp. 12–12, 2022

    2022

  7. [7]

    A foundation model for generalizable disease detection from retinal images,

    Y . Zhou, M. A. Chia, S. K. Wagner, M. S. Ayhan, D. J. Williamson, R. R. Struyven, T. Liu, M. Xu, M. G. Lozano, P. Woodward-Courtet al., “A foundation model for generalizable disease detection from retinal images,”Nature, vol. 622, no. 7981, pp. 156–163, 2023

    2023

  8. [8]

    Common and rare fundus diseases identification using vision-language foundation model with knowledge of over 400 diseases,

    M. Wang, T. Lin, K. Yu, A. Lin, Y . Peng, L. Wang, C. Chen, K. Zou, H. Liang, M. Chenet al., “Common and rare fundus diseases identification using vision-language foundation model with knowledge of over 400 diseases,”arXiv e-prints, pp. arXiv–2406, 2024

    2024

  9. [9]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,”Advances in Neural Information Processing Systems, vol. 36, pp. 28 541–28 564, 2023

    2023

  10. [10]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning,

    W. Xu, H. P. Chan, L. Li, M. Aljunied, R. Yuan, J. Wang, C. Xiao, G. Chen, C. Liu, Z. Liet al., “Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning,”arXiv preprint arXiv:2506.07044, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Ophglm: Training an ophthalmology large language- and-vision assistant based on instructions and dialogue,

    W. Gao, Z. Deng, Z. Niu, F. Rong, C. Chen, Z. Gong, W. Zhang, D. Xiao, F. Li, Z. Caoet al., “Ophglm: Training an ophthalmology large language- and-vision assistant based on instructions and dialogue,”arXiv preprint arXiv:2306.12174, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Specialized curricula for training vision-language models in retinal image analysis,

    R. Holland, T. R. Taylor, C. Holmes, S. Riedl, J. Mai, M. Patsiamanidi, D. Mitsopoulou, P. Hager, P. M ¨uller, H. P. Schollet al., “Specialized curricula for training vision-language models in retinal image analysis,” arXiv preprint arXiv:2407.08410, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Bilinear attention networks,

    J.-H. Kim, J. Jun, and B.-T. Zhang, “Bilinear attention networks,” Advances in neural information processing systems, vol. 31, 2018

    2018

  14. [14]

    Deep modular co-attention networks for visual question answering,

    Z. Yu, J. Yu, Y . Cui, D. Tao, and Q. Tian, “Deep modular co-attention networks for visual question answering,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 6281– 6290

    2019

  15. [15]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,”Advances in neural information processing systems, vol. 32, 2019

    2019

  16. [16]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in European conference on computer vision. Springer, 2020, pp. 104–120

    2020

  17. [17]

    Multi- modal understanding and generation for medical images and text via vision-language pre-training,

    J. H. Moon, H. Lee, W. Shin, Y .-H. Kim, and E. Choi, “Multi- modal understanding and generation for medical images and text via vision-language pre-training,”IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 12, pp. 6070–6080, 2022

    2022

  18. [18]

    Learning to exploit temporal structure for biomedical vision-language processing,

    S. Bannur, S. Hyland, Q. Liu, F. Perez-Garcia, M. Ilse, D. C. Castro, B. Boecking, H. Sharma, K. Bouzid, A. Thiemeet al., “Learning to exploit temporal structure for biomedical vision-language processing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 016–15 027

    2023

  19. [19]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  20. [20]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    2023

  21. [21]

    Gem: Empowering mllm for grounded ecg understanding with time series and images,

    X. Lan, F. Wu, K. He, Q. Zhao, S. Hong, and M. Feng, “Gem: Empowering mllm for grounded ecg understanding with time series and images,”arXiv preprint arXiv:2503.06073, 2025

    arXiv 2025

  22. [22]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13923

    Pith/arXiv arXiv 2025

  23. [23]

    Evaluation of retinal image quality assessment networks in different color-spaces,

    H. Fu, B. Wang, J. Shen, S. Cui, Y . Xu, J. Liu, and L. Shao, “Evaluation of retinal image quality assessment networks in different color-spaces,” inInternational conference on medical image computing and computer- assisted intervention. Springer, 2019, pp. 48–56

    2019

  24. [24]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  25. [25]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,

    F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,”Nature methods, vol. 18, no. 2, pp. 203–211, 2021

    2021

  26. [26]

    Graph-based region and boundary aggregation for biomedical image segmentation,

    Y . Meng, H. Zhang, Y . Zhao, X. Yang, Y . Qiao, I. J. MacCormick, X. Huang, and Y . Zheng, “Graph-based region and boundary aggregation for biomedical image segmentation,”IEEE transactions on medical imaging, vol. 41, no. 3, pp. 690–701, 2021

    2021

  27. [27]

    Is chatgpt a good nlg evaluator? a preliminary study,

    J. Wang, Y . Liang, F. Meng, Z. Sun, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou, “Is chatgpt a good nlg evaluator? a preliminary study,”arXiv preprint arXiv:2303.04048, 2023

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.