REVIEW 4 major objections 5 minor 23 references
An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read An agentic workflow—LLM plus four specialized tools plus a reflection step—raises glaucoma classification accuracy by 16–47 percentage points over LLM alone and comes within 6 points of a specialist.
desk verdict Real and consistent gains, but the improvement over LLM-alone is driven mainly by the injected SwinV2-Tiny classifier, not the agentic orchestration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the three-step agentic workflow, which turns the LLM from direct predictor into orchestrator: the LLM first reads the image and gives an initial impression, then native function calling invokes four deterministic open-source tools—QAModel and FundaQ-8 for image quality, SwinV2-Tiny for glaucoma probability, and SegFormer-B0 for optic disc/cup segmentation with vertical cup-to-disc ratio—and finally the LLM reflects, integrating its initial impression with tool outputs into a single structured report. This anchors predictions in reproducible measurements while preserving the LLM's clinical reasoning.
What would settle it
Run the same 16 conditions on a third, held-out fundus dataset with three masked glaucoma specialists and compare agentic versus LLM-alone accuracy, specialist agreement, and run-to-run $\kappa$; the central claim is contradicted if the agentic accuracy advantage over LLM-alone falls below the reported 16-point margin or if run-to-run $\kappa$ does not rise above fair agreement.
Extended reading notes
Core claim
The paper's central claim is that an agentic AI workflow—structured as LLM initial assessment, function calling to four specialized open-source deep learning tools (QAModel and FundaQ-8 for quality, SwinV2-Tiny for glaucoma classification, and SegFormer-B0 for disc/cup segmentation), and LLM reflection—overcomes the three LLM limitations of weak ophthalmic grounding, non-determinism, and hallucinated confidence. Across both LLMs, both datasets, and both field-of-view conditions, the workflow improved glaucoma classification accuracy by 16 to 47 percentage points over LLM-alone, corrected GPT-5.4 mini's systematic positive bias (sensitivity 95–100%, specificity 0–5%) and Gemini 2.5 Flash's stochastic variability, and raised run-to-run $\kappa$ from as low as $-0.01$ to as high as $0.96$. It reached within 6 points of a masked fellowship-trained glaucoma specialist on ORIGA (73% vs. 79%) and matched the specialist's 88% accuracy on RIM-ONE-v3. The authors take this as evidence that a modular orchestrated system, rather than a larger monolithic LLM, can make LLM-based image analysis clinically reproducible.
Load-bearing premise
The framework's gains assume the four open-source deep learning tools give trustworthy outputs on ORIGA and RIM-ONE-v3 images, even though SwinV2-Tiny and SegFormer-B0 were trained on REFUGE and QAModel's training set is not publicly documented; if those tools transfer poorly or carry dataset-specific biases, the agentic grounding inherits or amplifies them.
Editorial extensions
If this is right
- If the reported gains hold, glaucoma screening systems can approach specialist-level accuracy using low-cost LLM tiers plus a small suite of open-source tools, instead of relying on flagship models.
- The jump in run-to-run agreement (from $\kappa$ near $-0.01$ to up to $0.96$) means an LLM-based reader could satisfy the reproducibility requirement for clinical workflows.
- Because the specialized tools always receive the full fundus image, agentic accuracy stays stable when the LLM sees only a disc-centered crop, removing a preprocessing dependency that hurt LLM-alone reading.
- The reflection step can outperform the DL classifier alone (69–73% vs. 68% on ORIGA) by correcting classifier false positives, so the system acts as an intelligent integrator rather than a pass-through.
Reading between the lines
- A natural extension the paper does not test is that the same orchestration pattern could correct LLM bias in other image-based screening tasks (diabetic retinopathy, AMD, skin lesions) wherever a deterministic classifier and a segmentation/quality tool already exist.
- The paper's arbitration analysis implies morphological evidence (segmentation-derived CDR) may be more reliable than classifier probability scores when tools conflict; a targeted ablation could test this by withholding either tool output from the reflection step.
- The RIM-ONE quality over-classification suggests the quality tools carry dataset-specific illumination biases; an implication is that the framework would need per-site calibration of its quality thresholds before deployment on new cameras or populations.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an agentic AI framework for glaucoma detection from fundus photographs, in which an LLM first gives an initial assessment, then invokes four specialized deep learning tools (QAModel, FundaQ-8, SwinV2-Tiny, SegFormer-B0) via function calling, and finally reflects on the combined evidence to produce a structured output. The authors evaluate two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) on 100-image subsets of ORIGA and RIM-ONE-v3 under cropped and uncropped conditions, with each condition run twice. They report that the agentic workflow improves classification accuracy by 16-47 percentage points over LLM-alone, brings accuracy within 6 points of a fellowship-trained specialist on ORIGA and matches 88% on RIM-ONE-v3, reduces CDR estimation error, and raises run-to-run kappa from near-random to near-perfect. The paper argues that the agentic architecture, not model scale, drives the gains.
Significance. If the central claim holds, the paper provides a practical template for grounding LLM-based medical image interpretation in deterministic, auditable tool outputs, directly addressing reproducibility and hallucination concerns that are currently central to clinical LLM deployment. The use of open-source DL tools, explicit reporting of per-condition accuracy/sensitivity/specificity/F1, and the inclusion of run-to-run consistency as a safety metric are strengths; the paper also honestly discusses several limitations, including the RIM-ONE quality overclassification issue and the single-specialist reference standard. The main significance hinges on whether the agentic orchestration itself adds value over simply using the DL tools, which is not yet established.
major comments (4)
- [Results, 'Agentic AI vs. DL tools'] The standalone DL classifier SwinV2-Tiny achieves 68% accuracy on ORIGA and 88% on RIM-ONE-v3, which is essentially identical to the agentic pipeline's 69-73% and 85-88%. Because the 16-47 percentage-point gains over LLM-alone are thus almost entirely explained by the deterministic DL tool outputs, the claim that agentic orchestration is the driver of improvement is not supported without a static-fusion baseline. The authors should add an ablation in which the LLM reflection is replaced by a fixed rule over the same tool outputs (e.g., majority vote, or priority to segmentation-derived CDR), and compare its accuracy and consistency to the agentic workflow. If the agentic version does not exceed that baseline, the abstract's attribution of gains to the agentic architecture must be revised.
- [Results and Table 3] No confidence intervals, significance tests, or effect-size measures are provided for any accuracy, sensitivity, specificity, or kappa difference. With n=100 per dataset and only two runs per condition, the 1-5 percentage-point differences among agentic conditions (e.g., ORIGA 69% vs 73%) are within plausible sampling variability. Report bootstrap confidence intervals for proportions and kappas, use paired McNemar tests for accuracy comparisons against the same images, and report Run-2 accuracy values rather than only Run-1 accuracy, so that readers can assess the stability of the headline improvements.
- [Results, 'Glaucoma Classification Performance' and Table 3] The statement that agentic AI performs 'within 6 points of the specialist' is based on comparing both AI and the specialist to dataset labels. On ORIGA, however, the specialist agrees with dataset labels in only 79% of cases, and direct AI-specialist agreement on ORIGA is only kappa = 0.38-0.55 (fair to moderate by Landis-Koch). The near-specialist framing in the abstract and conclusion should be tempered by explicitly presenting the direct AI-specialist kappa values alongside the accuracy comparison, especially for ORIGA, and by acknowledging that a single specialist grading with 3 excluded 'cannot determine' images per dataset is a weak reference standard for the specialist-comparison claim.
- [Methods, 'Evaluations and Repeatability Analysis'] The exclusion of images graded 'cannot determine' (n=3 per dataset) is stated, but the paper does not report how those exclusions affect the dataset-label-based accuracy comparisons or the specialist-agreement analyses. Because the excluded images are exactly those where the specialist was uncertain, their removal could inflate both AI-specialist agreement and the accuracy of any predictor that tends to avoid committing on hard images. The authors should report the number of excluded images in each analysis and provide a sensitivity analysis (e.g., treating excluded images as incorrect for both AI and specialist) to show that the main conclusions are robust to this exclusion.
minor comments (5)
- [Table 5 footnote] The footnote explaining the RIM-ONE-v3 Cropped poor-recall denominators (17 vs 18) is confusing: the denominator for recall should be the number of specialist-graded poor images, which is fixed at 18 for RIM-ONE-v3; clarify that one image was excluded because the predictor labeled it ungradable, and state how this affects the reported recall values.
- [Supplemental Figures] Two different figures are both labeled 'Supplemental Figures S2' (one showing classification performance, one showing predicted vs specialist CDR). Renumber them sequentially to avoid ambiguity.
- [Figure 1] Figure 1A refers to 'ChatGPT-5' while the study evaluates Gemini 2.5 Flash and GPT-5.4 mini; clarify whether the illustrative panels are from the current models or from prior work, and ensure the caption matches the models actually evaluated.
- [Tables 2 and 3] The column header 'ChatGPT' should be changed to 'GPT-5.4 mini' for consistency with the text, since the paper evaluates GPT-5.4 mini rather than the base ChatGPT model.
- [Discussion] The phrase 'paradigm shift' in the conclusion is editorializing and goes beyond the evidence presented; a more measured statement about the potential of orchestrated systems would be appropriate given the limitations acknowledged in the same section.
Circularity Check
No significant circularity: the agentic gains are measured against same-study LLM-alone baselines using externally trained tools, not fitted to the test labels.
full rationale
The paper's central claim is that an agentic workflow improves glaucoma classification over LLM-alone approaches. This comparison is made within the same study: identical prompts, same LLMs, same images, and default inference settings for both the LLM-alone and agentic conditions. The specialized tools (QAModel, FundaQ-8, SwinV2-Tiny, SegFormer-B0) are external, open-source models trained on REFUGE or other public/undocumented datasets; none is fit to the ORIGA or RIM-ONE-v3 labels used for evaluation. No model parameter, prompt weight, or threshold in the paper is tuned to the test data. The abstract's 16-47 percentage point gains are simply the observed difference between the measured agentic outputs and the measured LLM-alone outputs in Tables 2 and 3, not a quantity derived from itself. The 'Agentic AI vs. DL tools' section provides an important internal control: agentic accuracy on ORIGA (69-73%) is close to SwinV2-Tiny alone (68%), and on RIM-ONE-v3 it is 85-88% versus 88% for the classifier alone. This could undermine the attribution of the gains to orchestration rather than to the injected classifier, but that is an ablation and soundness concern, not circularity: the paper does not define the agentic result as the classifier's output, and it reports correcting classifier false positives and arbitrating tool disagreements. Run-to-run consistency improvements are likewise measured directly and follow from the deterministic tool outputs, which is an architectural explanation rather than a circular assumption. Self-citations such as [16] for prior GPT-4V performance and [3] for preprocessing are contextual and non-load-bearing; the paper re-measures LLM-alone behavior on the current datasets rather than relying on those citations for its conclusions. The acknowledged over-classification of RIM-ONE-v3 images as poor quality is presented as a limitation, not hidden. Overall, the derivation chain is self-contained against the external benchmarks and does not reduce any central prediction to its own inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption Dataset labels and the masked fellowship-trained specialist's grades are a valid reference standard
- domain assumption The four pretrained DL tools generalize to ORIGA and RIM-ONE-v3 despite training on other datasets
- domain assumption Two API runs with default settings in April-May 2026 represent model behavior
- ad hoc to paper Excluding images graded 'cannot determine' by the specialist does not bias comparisons
Cite this review
Pith. "Pith review of An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography." pith.science (2026). https://pith.science/paper/HJVRI2PU
@misc{pith2026260807651,
author = {Pith},
title = {Pith review of: An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography},
year = {2026},
howpublished = {\url{https://pith.science/paper/HJVRI2PU}},
note = {Machine review of arXiv:2608.07651}
}
read the original abstract
Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency. We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography. The workflow had three steps: (1) LLM initial assessment; (2) function calling to invoke specialized tools for image quality (QAModel, FundaQ-8), glaucoma classification (SwinV2-Tiny), and optic disc/cup segmentation (SegFormer-B0); and (3) LLM reflection integrating the initial impression with tool outputs. Two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) were evaluated on two public datasets (ORIGA, n=100; RIM-ONE-v3, n=100) under uncropped and cropped fields of view; all images were independently graded by a masked fellowship-trained glaucoma specialist. The agentic workflow improved classification accuracy by 16 to 47 percentage points across all conditions, reaching within 6 points of the specialist; on RIM-ONE-v3 the best configurations matched the specialist accuracy of 88%. LLM-alone approaches failed in two ways: GPT-5.4 mini showed positive bias (sensitivity 95-100%, specificity 0-5%), while Gemini 2.5 Flash varied stochastically between runs; the agentic workflow corrected both. Cup-to-disc ratio error fell 15-50% (MAE 0.156-0.228 to 0.104-0.132), and correlation with specialist grading rose from weak (r=0.12-0.39) to moderate-strong (r=0.59-0.84). Run-to-run consistency rose from near-random (kappa as low as -0.01) to near-perfect (kappa up to 0.96). Integrating LLMs with specialized tools addressed key limitations of LLM-alone approaches, including over-diagnosis and run-to-run variability. Gains held for both LLMs, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.
Reference graph
Works this paper leans on
-
[1]
Tham, Y.-C., et al., Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology,
-
[2]
Flaxman, S.R., et al., Global causes of blindness and distance vision impairment 1990–2020: a systematic review and meta-analysis. The Lancet Global Health,
work page 1990
-
[3]
Bounded quality score (0.0–1.0); categorized into Low, Average, or good thresholds Custom model; base architecture via Szegedy et al., 2016 (arXiv:1512.00567)
arXiv 2016
-
[7]
Chen, S.F., et al., LLM-assisted systematic review of large language models in clinical medicine. Nature medicine, 2026: p. 1-8
work page 2026
-
[10]
Ophthalmology Science, 2026: p
Jalili, J., et al., Image-Quality–Aware Multimodal AI for Automated Structured OCT Report Generation in Glaucoma Evaluation. Ophthalmology Science, 2026: p. 101254
work page 2026
-
[15]
arXiv preprint arXiv:2503.10647,
Subedi, K., The reliability of llms for medical diagnosis: An examination of consistency, manipulation, and contextual awareness. arXiv preprint arXiv:2503.10647,
-
[19]
arXiv preprint arXiv:2503.05777,
Kim, Y., et al., Medical hallucinations in foundation models and their impact on healthcare. arXiv preprint arXiv:2503.05777,
-
[20]
Deep learning models integrated within the agentic AI framework for image quality verification and structural glaucoma assessment. Tool Architecture Task Training Dataset Input Output Reference QAModel Inception V3 with custom regression head (2048→1024→512→1, sigmoid activation) Continuous fundus image quality regression and categorical triage Custom-lab...
Show all 23 references
-
[21]
arXiv preprint arXiv:2510.09404,
Bluethgen, C., et al., Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges. arXiv preprint arXiv:2510.09404,
-
[22]
Bounded continuous quality score (0.0–1.0) Lee et al., 2025 IEEE [36]; Source code and model: https://github.com/qmed-asia/FundaQ-8 SwinV2-Tiny Swin Transformer V2 (Tiny variant) Binary classification of glaucomatous vs. normal optic discs REFUGE Challenge dataset (n = 1,200) ...
2025
-
[25]
arXiv preprint arXiv:2312.04511,
Kim, S., et al., An llm compiler for parallel function calling. arXiv preprint arXiv:2312.04511,
-
[26]
arXiv preprint arXiv:2601.02749,
Sibai, N., et al., The Path Ahead for Agentic AI: Challenges and Opportunities. arXiv preprint arXiv:2601.02749,
-
[27]
Renze, M. and E. Guven, Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682,
-
[28]
arXiv preprint arXiv:2507.06261,
Comanici, G., et al., Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261,
-
[29]
March 17, 2026; Available from: https://openai.com/index/introducing-gpt-5-4-mini-and-nano/
Introducing GPT‑5.4 mini and nano. March 17, 2026; Available from: https://openai.com/index/introducing-gpt-5-4-mini-and-nano/
2026
-
[30]
Origa-light: An online retinal fundus image database for glaucoma analysis and research
Zhang, Z., et al. Origa-light: An online retinal fundus image database for glaucoma analysis and research. in 2010 Annual international conference of the IEEE engineering in medicine and biology
2010
-
[31]
RIM-ONE: An open retinal image database for optic nerve evaluation
Fumero, F., et al. RIM-ONE: An open retinal image database for optic nerve evaluation. in 2011 24th international symposium on computer-based medical systems (CBMS)
2011
-
[36]
FundaQ-8: A Clinically-Inspired Scoring Framework for Automated Fundus Image Quality Assessment
Zun, L.Q., et al. FundaQ-8: A Clinically-Inspired Scoring Framework for Automated Fundus Image Quality Assessment. in 2025 IEEE 7th Symposium on Computers & Informatics (ISCI)
2025
-
[39]
Source code and model: https://huggingface.co/pamixsun/swinv2_tiny_for_glaucoma_classification SegFormer-B0 SegFormer (Lightweight hierarchical encoder with MLP decoder) Multi-class semantic segmentation of the optic disc and cup REFUGE Challenge dataset (n = 1,200$) Color fun...
2021
-
[42]
Landis, J.R. and G.G. Koch, The measurement of observer agreement for categorical data. biometrics, 1977: p. 159-174
1977
-
[44]
medRxiv, 2024: p
Agbareia, R., et al., The role of prompt engineering for multimodal LLM glaucoma diagnosis. medRxiv, 2024: p. 2024.10. 30.24316434
2024
-
[45]
arXiv preprint arXiv:2507.01278,
Tabuse, C.L., et al., Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening. arXiv preprint arXiv:2507.01278,
-
[2014]
2081-2090
121(11): p. 2081-2090
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.