REVIEW 3 major objections 5 minor 28 references
A 4B vision-language model trained on a structured ocular-surface VQA corpus beats medical VLMs up to 34B.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 19:55 UTC pith:OJSKM6DS
load-bearing objection Solid data-engine paper for external-eye OSD VLMs; SOTA numbers are real on their synthetic split but the clinical leap is still unproven. the 3 major comments →
IRIS: An Intelligent Vision-Language System for Ocular Surface Diseases via Topic Tree and Scene-Driven VQA Generation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Aligning a 4B-parameter vision-language model on IRIS-120K, a corpus generated by the Topic Finding Tree plus scene-driven engine, produces state-of-the-art performance on ocular-surface VQA and comprehensively surpasses generalist and specialized medical VLMs with up to 34B parameters, demonstrating that structured clinical knowledge injection prevails over sheer parameter scaling.
What carries the argument
The dual-branch Clinically-driven Data Engine: Topic Finding Tree (TFT) that hierarchically maps each image onto ten anatomical regions and a four-stage think chain (observation, correlation, deduction, conclusion), paired with scene-driven generation of role-adaptive clinical dialogues.
Load-bearing premise
That LLM-synthesized VQA pairs, filtered only by another model and without independent ophthalmologist annotation of the gold test set, are a valid proxy for real clinical diagnostic competence.
What would settle it
An independent, ophthalmologist-annotated external test set of smartphone external-eye photographs on which IRIS-4B fails to retain its claimed accuracy margin over larger medical VLMs, or a prospective clinical trial showing that the model’s diagnoses do not match specialist consensus at the reported rates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents IRIS, a compact vision-language system for ocular surface disease (OSD) understanding from external eye photographs. To address the scarcity of high-fidelity multimodal instruction data, the authors curate IRIS-120K via a dual-branch Clinically-driven Data Engine: (1) a Topic Finding Tree (TFT) that hierarchically maps visual features to 10 anatomical regions and region-specific findings, generating VQA pairs with explicit 4-stage reasoning chains; and (2) Scene-driven generation that produces role-adaptive dialogues for Patient/Doctor/Student personas across clinical scenarios. A 4B-parameter Qwen3-VL model fine-tuned on this corpus (IRIS-4B) is reported to achieve SOTA on the authors’ held-out test split across closed-ended (Judge/Single/Multi-choice) and open-ended (Tree/Scene-VQA) metrics, outperforming generalist and medical VLMs up to 34B parameters. Ablations attribute gains to the TFT–Scene synergy and identify 4B as the capacity sweet-spot. The work claims that structured clinical knowledge injection enables resource-efficient, interpretable edge deployment for scalable OSD screening.
Significance. If the reported gains transfer beyond the synthetic distribution, the contribution is substantial: it fills a clear modality gap (external-eye vs. fundus/OCT bias in existing ophthalmic foundation models), supplies the largest dedicated OSD VQA corpus to date, and demonstrates that carefully engineered anatomical priors can let a 4B model dominate much larger generalist and medical VLMs. The dual-branch engine, quality-aware sampling hierarchy, explicit <think> chains, and attention-based visual grounding are concrete, reusable design patterns. Planned public release of code, IRIS-120K, and model weights further raises the work’s value for reproducible MeHealth research. The parameter-efficiency result is especially relevant for offline smartphone screening in resource-limited settings.
major comments (3)
- [Sec. 2.2 / Experimental Setup] Sec. 2.2 and Experimental Setup: The 117.7K training pairs and the 8.2K “gold-standard” test set are generated by Qwen3-VL-32B from image-caption surrogates (TFT 4-stage Think chains + scene templates) and then filtered by GPT-5-mini. No independent ophthalmologist annotation or adjudication of either the VQA labels or the reasoning chains is reported. Consequently the central claim that IRIS-4B reaches “expert-level” clinical competence (and the leap to “scalable OSD screening”) rests on a self-referential synthetic distribution rather than clinically validated ground truth. This is load-bearing for every SOTA statement in Table 1 and the abstract.
- [Table 1] Table 1: Because the same LLM family that synthesizes the structured answers is also among the baselines being compared, the large absolute gains (IRIS-4B overall avg. 74.26 vs. Lingshu-32B 55.00 / Qwen3-VL-32B 50.61) may largely reflect format and stylistic alignment to the synthetic distribution rather than superior anatomical grounding or diagnostic validity. The TFT-only vs. Scene-only ablations confirm internal synergy on this distribution but do not break the circularity. An external clinician-rated evaluation set or an independent real-world OSD cohort is required to substantiate the clinical SOTA claim.
- [Sec. 2.1–2.2] Sec. 2.1–2.2: The quality hierarchy (WeChat > Books > Paper-Single > …) that drives dynamic sampling densities is derived from “sampling-based human evaluations,” yet no inter-rater statistics, sample sizes, or criteria definitions are supplied. Because this hierarchy directly controls how many TFT paths and scenes are generated per image, its opacity undermines reproducibility of the claimed data-engine advantage.
minor comments (5)
- [Experimental Setup] Experimental Setup: “First, we We fine-tune Qwen3-VL” contains a duplicated “We”.
- [Fig. 2] Fig. 2 caption and body: the train/test source and VQA-type distributions are useful but the absolute counts (especially the 8.2K test split after GPT-5-mini filtering) should be stated numerically in the text for clarity.
- [Fig. 3] Fig. 3: attention contours are described as “>25% yellow, >50% red” but the precise aggregation method (token-level, layer, head) is not specified; a short methods note would aid interpretability claims.
- [References] References: several arXiv preprints (e.g., Qwen3-VL, MedGemma, Lingshu) lack final venue or version dates; consistent citation style would improve polish.
- [Abstract / Conclusion] Abstract and Conclusion: the phrase “profoundly prevails over sheer parameter scaling” is repeated almost verbatim; slight rephrasing would reduce redundancy.
Circularity Check
SOTA claim and 'expert-level' leap rest on train/test VQA pairs both synthesized by the same Qwen-family + GPT-5-mini pipeline (TFT/Scene), so fine-tuned IRIS-4B scores largely measure synthetic-distribution matching rather than independent clinical prediction.
specific steps
-
fitted input called prediction
[Sec. 2.2 (Quality-Aware Dynamic Sampling & Dataset Finalization) + Experimental Setup + Table 1]
"we first employ Qwen3-VL-32B [2] to generate a detailed visual description Tdesc for each image. Combined with the original caption/label Tcap, this forms a rich textual surrogate x=(Tcap,Tdesc)... For each image, 1–3 prompts are generated using either the TFT or Scene-Driven method. This balanced protocol generates approximately 130K initial VQA pairs... An initial test pool of 9K VQA pairs undergoes a rigorous final quality control pass using GPT-5-mini to filter out factual hallucinations... yields the IRIS-120K, comprising 117.7K training pairs and an 8.2K gold-standard test set... IRIS-4B"
Both train and held-out test VQA (including the exact 4-stage Think chains and role-adaptive dialogues that define Tree-VQA/Scene-VQA metrics) are synthesized by the identical LLM pipeline from the same image-caption surrogates. Fine-tuning the Qwen3-VL-4B student on the train split then scoring BLEU/ROUGE/accuracy on the test split is statistically forced distribution matching; the large margins over zero-shot larger models (including the generator) do not constitute an independent clinical prediction.
-
other
[Abstract + Sec. 4 Conclusion + Fig. 3]
"By explicitly aligning a compact 4B-parameter VLM on this structurally enriched corpus, IRIS achieves state-of-the-art performance, comprehensively outperforming both generalist and specialized medical VLMs with up to 34B parameters. Our findings underscore that structured knowledge injection profoundly prevails over sheer parameter scaling, unlocking the potential for resource-efficient, expert-level AI deployment on mobile edge devices for scalable OSD screening."
The leap from 'beats larger VLMs on our synthetic IRIS-120K benchmark' to 'expert-level' clinical competence and scalable real-world OSD screening is unsupported by any external clinician-annotated or real-patient cohort. Qualitative <think> blocks and attention maps remain self-referential to the same synthetic generation process, so the central applied claim reduces in part to the paper's own data-construction choices.
full rationale
This is an empirical VLM paper, not a first-principles derivation, so classic mathematical self-definition is absent. The load-bearing claim (compact IRIS-4B achieves overall avg 74.26, beating 32B/34B medical and generalist VLMs by large margins via structured knowledge injection) is evaluated exclusively on the IRIS-120K 8.2K test split. That split, like the 117.7K train set, is produced by the dual-branch engine: Qwen3-VL-32B first writes Tdesc, then LLM generation of TFT 4-stage Think chains and Scene-driven role dialogues, followed by GPT-5-mini filtering of a 9K pool (Sec. 2.2). pHash only removes image duplicates; label style, format, and reasoning chains remain matched by construction. Zero-shot baselines (including the generator Qwen3-VL-32B itself at 50.61) therefore underperform for distributional reasons, while ablations merely confirm internal TFT/Scene synergy on the same synthetic distribution. No clinician-adjudicated external cohort or real-world MeHealth benchmark is reported. The result therefore contains partial circularity: high generative metrics and the leap to 'expert-level ... scalable OSD screening' reduce in part to matching the paper's own generation process. Independent content remains in the data-engine design and parameter-efficiency observation, keeping the score moderate rather than 8-10.
Axiom & Free-Parameter Ledger
free parameters (4)
- quality hierarchy sampling densities =
empirical ranking; denser sampling for high-quality sources
- prompts per image (1–3) =
1–3
- lesion-area classification heuristic (50%) =
50%
- LoRA / fine-tuning hyperparameters
axioms (5)
- domain assumption Ten predefined ocular anatomical regions (Pupil, Cornea, Iris, Sclera-Conjunctiva, Eyelashes, Upper/Lower Eyelid, Superior/Inferior Palpebral Conjunctiva, Others) exhaustively cover the visual features needed for OSD diagnosis.
- domain assumption A detailed visual description T_desc produced by Qwen3-VL-32B plus the original caption forms a faithful semantic surrogate for clinical image content.
- ad hoc to paper GPT-5-mini filtering of the 9K test pool removes factual hallucinations and privacy issues sufficiently to yield a gold-standard clinical test set.
- domain assumption BLEU-1 and ROUGE-1/L-F are adequate metrics for open-ended clinical dialogue quality.
- standard math Standard LoRA fine-tuning of Qwen3-VL preserves enough capacity to absorb the injected clinical structure without catastrophic forgetting of general vision-language skill.
invented entities (4)
-
Topic Finding Tree (TFT)
no independent evidence
-
Scene-driven role-adaptive generation
no independent evidence
-
IRIS-120K corpus
no independent evidence
-
IRIS-2B/4B/8B models
no independent evidence
read the original abstract
While Large Vision-Language Models (VLMs) demonstrate remarkable generic capabilities, their clinical reasoning in specialized domains like ocular surface diseases (OSDs) is severely hindered by a paucity of high-fidelity, multimodal instruction-tuning data. To dismantle this data bottleneck, we introduce IRIS, an Intelligent Recognition and Interaction System tailored for fine-grained OSD understanding via external eye photography. First, we curate IRIS-120K, the largest and most comprehensive OSD visual question-answering (VQA) dataset to date. Crucially, to overcome the semantic shallowness of conventional image-caption pairs, we propose a synergistic data generation paradigm to explicitly inject clinical priors. Our data engine operates via a dual-branch framework: 1) a Topic Finding Tree (TFT) that hierarchically anchors visual features to precise anatomical and pathological concepts, enforcing rigorous medical deduction logic; and 2) a Scene-driven strategy that synthesizes role-adaptive clinical dialogues to ensure pragmatic generalization. By explicitly aligning a compact 4B-parameter VLM on this structurally enriched corpus, IRIS achieves state-of-the-art performance, comprehensively outperforming both generalist and specialized medical VLMs with up to 34B parameters. Our findings underscore that structured knowledge injection profoundly prevails over sheer parameter scaling, unlocking the potential for resource-efficient, expert-level AI deployment on mobile edge devices for scalable OSD screening. Code, datasets, and model weights will be publicly released by this repo.
Figures
Reference graph
Works this paper leans on
-
[1]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Baghbanzadeh, N., Fallahpour, A., Parhizkar, Y., Ogidi, F., Roy, S., Ashkezari, S., Khazaie, V.R., Colacci, M., Etemad, A., Afkanpour, A., et al.: Advancing medical representation learning through high-quality data. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 24–33. Springer (2025)
2025
-
[2]
arXiv preprint arXiv:2511.21631 (2025)
Bai, S., Cai, Y., Chen, R., Chen, K., et.al, X.C.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)
Pith/arXiv arXiv 2025
-
[3]
arXiv preprint arXiv:2502.13923 (2025)
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)
Pith/arXiv arXiv 2025
-
[4]
arXiv preprint arXiv:2406.19280 (2024)
Chen, J., Gui, C., Ouyang, R., Gao, A., Chen, S., Chen, G.H., Wang, X., Zhang, R., Cai, Z., Ji, K., et al.: Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280 (2024)
Pith/arXiv arXiv 2024
-
[5]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198 (2024)
2024
-
[6]
Heliyon10(17) (2024)
Graham, A.D., Kothapalli, T., Wang, J., Ding, J., Tse, V., Asbell, P.A., Yu, S.X., Lin, M.C.: A machine learning approach to predicting dry eye-related signs, symp- toms and diagnoses from meibography images. Heliyon10(17) (2024)
2024
-
[7]
arXiv preprint arXiv:2510.08668 (2025)
Jiang, S., Wang, Y., Song, S., Hu, T., Zhou, C., Pu, B., Zhang, Y., Yang, Z., Feng, Y., Zhou, J.T., et al.: Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668 (2025)
arXiv 2025
-
[8]
Advances in ophthalmology practice and research (2025)
Jin, K., Yu, T., Ying, G.s., Ge, Z., Li, K.Z., Zhou, Y., Shi, D., Wang, M., Goktas, P., Grzybowski, A.: A systematic review of vision and vision-language foundation models in ophthalmology. Advances in ophthalmology practice and research (2025)
2025
-
[9]
In: Proceedings of the 33rd ACM International Conference on Multimedia
Li, S., Lin, T., Lin, L., Zhang, W., Liu, J., Yang, X., Li, J., He, Y., Song, X., Xiao, J., et al.: Eyecaregpt: Boosting comprehensive ophthalmology understanding with tailored dataset, benchmark and model. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 3893–3902 (2025)
2025
-
[10]
Frontiers in Cell and Devel- opmental Biology13, 1600202 (2025)
Li, Z., Wang, Z., Xiu, L., Zhang, P., Wang, W., Wang, Y., Chen, G., Yang, W., Chen, W.: Large language model-based multimodal system for detecting and grad- ing ocular surface diseases from smartphone images. Frontiers in Cell and Devel- opmental Biology13, 1600202 (2025)
2025
-
[11]
In: Text sum- marization branches out
Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text sum- marization branches out. pp. 74–81 (2004)
2004
-
[12]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Lozano, A., Sun, M.W., Burgess, J., Chen, L., Nirschl, J.J., Gu, J., Lopez, I., Aklilu, J., Rau, A., Katzer, A.W., et al.: Biomedica: An open biomedical image- caption archive, dataset, and vision-language models derived from scientific litera- ture. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 19724–19735 (2025) 10 H. Wei et al
2025
-
[13]
Diagnostics15(24), 3199 (2025)
Lu, T.C., Huang, C.H., Lin, I.C.: Artificial intelligence application in cornea and external diseases. Diagnostics15(24), 3199 (2025)
2025
-
[14]
Scientific Reports15(1), 5117 (2025)
Maehara, H., Ueno, Y., Yamaguchi, T., Kitaguchi, Y., Miyazaki, D., Nejima, R., Inomata, T., Kato, N., Chikama, T.i., Ominato, J., et al.: Artificial intelligence support improves diagnosis accuracy in anterior segment eye diseases. Scientific Reports15(1), 5117 (2025)
2025
-
[15]
Scientific Reports15(1), 1462 (2025)
Maehara, H., Ueno, Y., Yamaguchi, T., Kitaguchi, Y., Miyazaki, D., Nejima, R., Inomata, T., Kato, N., Chikama, T.i., Ominato, J., et al.: The importance of clinical experience in ai-assisted corneal diagnosis: verification using intentional ai misleading. Scientific Reports15(1), 1462 (2025)
2025
-
[16]
In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)
2002
-
[17]
arXiv preprint arXiv:2310.04992 (2023)
Qiu, J., Wu, J., Wei, H., Shi, P., Zhang, M., Sun, Y., Li, L., Liu, H., Liu, H., Hou, S., et al.: Visionfm: a multi-modal multi-task vision foundation model for generalist ophthalmic artificial intelligence. arXiv preprint arXiv:2310.04992 (2023)
Pith/arXiv arXiv 2023
-
[18]
arXiv preprint arXiv:2507.05201 (2025)
Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al.: Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025)
Pith/arXiv arXiv 2025
-
[19]
npj Digital Medicine8(1), 381 (2025)
Shi, D., Zhang, W., Yang, J., Huang, S., Chen, X., Xu, P., Jin, K., Lin, S., Wei, J., Yusufu, M., et al.: A multimodal visual–language foundation model for compu- tational ophthalmology. npj Digital Medicine8(1), 381 (2025)
2025
-
[20]
Singh, A., Fry, A., Perelman, A., et al.: Openai gpt-5 system card (Dec 2025), https://arxiv.org/abs/2601.03267, accessed: 2026-02-21
Pith/arXiv arXiv 2025
-
[21]
Srinivasan, S., Ji, H., Chen, D.Z., Wong, W., Da Soh, Z., Goh, J.H.L., Push- panathan, K., Wang, X., Ma, W., Wong, T.Y., et al.: Can off-the-shelf visual large language models detect and diagnose ocular diseases from retinal photographs? BMJ Open Ophthalmology10(1) (2025)
2025
-
[22]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Sun,Y.,Zhu,C.,Zheng,S.,Zhang,K.,Sun,L.,Shui,Z.,Zhang,Y.,Li,H.,Yang,L.: Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5034–5042 (2024)
2024
-
[23]
iLABMED3(1), 64–85 (2025)
Wang, M.H., Pan, Y., Jiang, X., Lin, Z., Liu, H., Liu, Y., Cui, J., Tan, J., Gong, C., Hou, G., et al.: Leveraging artificial intelligence and clinical laboratory evidence to advance mobile health applications in ophthalmology: Taking the ocular surface disease as a case study. iLABMED3(1), 64–85 (2025)
2025
-
[24]
arXiv preprint arXiv:2506.07044 (2025)
Xu, W., Chan, H.P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu,C.,Li,Z.,etal.:Lingshu:Ageneralistfoundationmodelforunifiedmultimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044 (2025)
Pith/arXiv arXiv 2025
-
[25]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Yeh, C.H., Wang, J., Graham, A.D., Liu, A.J., Tan, B., Chen, Y., Ma, Y., Lin, M.C.: Insight: A multi-modal diagnostic pipeline using llms for ocular surface dis- ease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 711–721. Springer (2024)
2024
-
[26]
Allergology International74(1), 86–96 (2025)
Yonehara, M., Nakagawa, Y., Ayatsuka, Y., Hara, Y., Shoji, J., Ebihara, N., In- omata, T., Huang, T., Nagino, K., Fukuda, K., et al.: Use of explainable ai on slit-lamp images of anterior surface of eyes to diagnose allergic conjunctival dis- eases. Allergology International74(1), 86–96 (2025)
2025
-
[27]
Master’s thesis, University of Applied Sciences Hagenberg, Hagenberg, Aus- tria (Jul 2010),https://www.phash.org/docs/pubs/thesis_zauner.pdf, mas- ter’s thesis
Zauner, C.: Implementation and Benchmarking of Perceptual Image Hash Func- tions. Master’s thesis, University of Applied Sciences Hagenberg, Hagenberg, Aus- tria (Jul 2010),https://www.phash.org/docs/pubs/thesis_zauner.pdf, mas- ter’s thesis. pHash library implementation IRIS: An Intelligent Vision-Language System 11
2010
-
[28]
Nature622(7981), 156–163 (2023)
Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P., et al.: A foundation model for generalizable disease detection from retinal images. Nature622(7981), 156–163 (2023)
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.