Pith. sign in

REVIEW 5 major objections 5 minor 14 references

Evaluation of an AI System for the Detection of Diabetic Retinopathy from Images Captured with a Handheld Portable Fundus Camera: the MAILOR AI study

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An AI diabetic-retinopathy system kept proliferative-disease accuracy on handheld-camera images but lost significant ground on referable disease, from 98.5% to 89.4% AUROC.

desk verdict Useful real-world evaluation of AI on handheld fundus images, but the headline RDR drop is confounded by the grading-scheme mismatch the authors themselves identify, so the causal attribution to the device does not hold. read the letter →

arxiv 1908.06399 v1 pith:VAANVNC3 submitted 2019-08-18 eess.IV cs.CV

T W Rogers , J Gonzalez-Bueno , R Garcia Franco , E Lopez Star , D Méndez Marín , J Vassallo , V C Lansingh , S Trikha
show 1 more author
N Jaccard
This is my paper · ORCID
classification eess.IVcs.CV
keywords diabeticretinopathydeeplearninghandheldfunduscamerareferableproliferativereal-worldvalidationgradingscalemismatchAUROC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether an AI diabetic-retinopathy detector trained and validated on high-quality desktop fundus images keeps its accuracy when fed images from a lightweight handheld fundus camera used in real-world screening. It reports that the system's ability to detect proliferative diabetic retinopathy transferred well, with AUROC statistically unchanged (94.3% versus 92.2% on a curated benchmark). But for referable diabetic retinopathy, the AUROC dropped significantly from 98.5% to 89.4%, with sensitivity and specificity both near 82%. The result matters because handheld cameras are the practical way to reach remote populations, and AI screening is only useful if it works on the images those devices produce. The paper's own explanation points mainly to a mismatch between the grading scheme the software uses and the scheme used to label the real-world images, not necessarily to a failure of the AI itself.

What carries the argument

The argument is carried by a paired, out-of-the-box evaluation design. The same frozen AI system—a deep-learning tool (Pegasus) that outputs a diabetic-retinopathy grade on the International Clinical Diabetic Retinopathy (ICDR) scale—is run without any adaptation on two image sets: a curated public benchmark of 516 desktop-camera photographs, and a real-world cohort of 5,752 patients imaged with a handheld portable non-mydriatic camera, yielding 22,180 images. The real-world reference standard is the Scottish DR grading scheme, and the paper compares AUROC at the referable (RDR) and proliferative (PDR) thresholds, using bootstrap confidence intervals and permutation tests for significance. The grading-scheme mismatch is a deliberate part of the machinery: it is the paper's main candidate explanation for the RDR gap, because PDR definitions coincide across schemes while RDR definitions do not.

What would settle it

Regrade the handheld-camera images using the same ICDR scale the AI outputs and re-run the frozen system; if the referable-DR AUC returns to near the 98.5% desktop benchmark, the grading-system mismatch is the driver. If the gap persists after regrading, image quality or device differences are responsible.

Watch

Extended reading notes

Core claim

The central claim is that transferability from curated desktop-camera images to real-world handheld-camera images is disease-severity dependent. For proliferative diabetic retinopathy, the system's AUROC on the handheld cohort was 94.3% (95% CI 91.0-96.9), statistically indistinguishable from the 92.2% (95% CI 89.4-94.8) on the desktop benchmark (p=0.172). For referable diabetic retinopathy, the AUROC fell to 89.4% (95% CI 88.0-90.7) from 98.5% (95% CI 97.8-99.2) on the benchmark (p<0.001), with sensitivity and specificity of about 82% at the equal-error operating point. The paper concludes that the system transfers well for PDR but that RDR performance drops substantially, and attributes the RDR gap mainly to the mismatch between the software's ICDR output and the Scottish reference standard, with image quality and field type as contributing factors.

Load-bearing premise

The load-bearing premise is that the observed difference in referable-DR performance between the two datasets is attributable to the handheld camera and real-world conditions, rather than to systematic differences between the datasets—above all, the mismatch between the Scottish grading scheme used for the real-world reference standard and the ICDR scheme the AI outputs.

Editorial extensions

If this is right

  • Proliferative diabetic retinopathy screening with a handheld camera and an unmodified desktop-trained AI is plausible in this type of real-world cohort, since PDR accuracy did not degrade.
  • Referable diabetic retinopathy screening should not be assumed to transfer; a validation against the actual device and grading protocol is needed before deployment.
  • Aligning the AI's grading scale with the local screening program's referral definitions could remove a major source of apparent false positives, per the paper's own analysis.
  • Curated public benchmark results for referable DR should be treated as optimistic upper bounds, not expected field performance.
  • Using macula-centred fields rather than disc-centred fields improved referable-DR AUC by 2.3%, so image field selection affects screening performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because PDR lesions are coarse while RDR hinges on small haemorrhages and microaneurysms, the paper's pattern suggests that the transferability gap is concentrated in precisely the features most sensitive to resolution, focus, and grading definitions; the paper does not disentangle these, but its false-positive figures point that way.
  • A direct test of the paper's main explanation would be to regrade the handheld images with the ICDR scale and rerun the frozen AI; narrowing of the RDR gap would implicate the grading mismatch, while a persistent gap would implicate image quality or device differences.
  • If the grading-scheme mismatch is the dominant factor, the practical fix is a mapping layer that translates the AI output into the local referral rules, which could recover much of the lost RDR performance without retraining the network.
  • The comparison also implies that high benchmark AUCs in the literature are weak evidence of field readiness for referable diabetic retinopathy; local device-specific validation is the deciding test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript reports a retrospective evaluation of a proprietary deep learning system (Pegasus, Visulytix) for diabetic retinopathy (DR) detection on images from a handheld portable fundus camera (Volk Pictor Plus) used in a Mexican screening cohort (MAILOR, ~5,752 patients, 22,180 images). The authors compare Pegasus's performance on MAILOR against its performance on the public IDRiD benchmark (516 desktop-camera images) for referable DR (RDR) and proliferative DR (PDR). They find a statistically significant drop in AUROC for RDR between IDRiD (98.5%) and MAILOR (89.4%), while PDR AUROC is not significantly different (92.2% vs 94.3%). The abstract concludes that Pegasus generalizes well to the handheld device for PDR but shows a substantial decrease for RDR when using the handheld device. The Discussion lists four possible explanatory factors, of which the authors identify the mismatch between the Scottish grading scheme used as the MAILOR reference standard and the ICDR scheme output by Pegasus as the 'biggest contributing factor.'

Significance. If the reported performance figures are reliable, this is a practically relevant evaluation: real-world clinical deployment of AI DR screening on low-cost handheld cameras is an important use case for telemedicine, and independent evidence on such transferability is scarce. The study uses a large, naturalistic cohort, a clearly described protocol, and standard AUROC analysis with bootstrap CIs; the external IDRiD benchmark provides a common reference. The measured performance numbers are empirical and the paper does not fit parameters to the test data. However, the study is sponsored and analyzed by the system's developer, and the central causal interpretation—that the RDR drop is due to the handheld device—is not supported by the study design, as the comparison is confounded by multiple simultaneous differences between the two datasets. The manuscript itself acknowledges the largest confounder (grading-scheme mismatch), which makes the abstract's device-specific conclusion overreach.

major comments (5)
  1. [Abstract and Discussion] The abstract conclusion attributes the RDR performance decrease to 'using the handheld device,' but the Discussion (factor 2) explicitly states that 'the biggest contributing factor is the mismatch in grading systems' between the Scottish scheme used for the MAILOR reference standard and the ICDR scale output by Pegasus. Because the two datasets also differ in camera, population, mydriasis, fields of view, and image quality, the measured AUROC difference cannot be attributed to the device. The abstract should be revised to state that the RDR drop is an observed difference between two unadjusted cohorts, with the grading-system mismatch as a leading potential cause, rather than a device effect.
  2. [Table 1] The disease severity distribution in Table 1 is internally inconsistent. For MAILOR, 5,017 + 595 + 60 = 5,672, not 5,752, and the percentages sum to 98.6%. For IDRiD, 168 + 323 + 62 = 553, not 516, and the listed percentages (32.6%, 52.6%, 12.0%) do not match the counts (323/516 = 62.6%, not 52.6%). The authors should clarify whether the categories are mutually exclusive (and if RDR includes PDR, state that explicitly) and correct the table so that counts and percentages are consistent and sum to the stated totals.
  3. [Discussion, comparison to Rajalakshmi et al.] The sensitivity and specificity confidence intervals reported for Pegasus in this paragraph do not match those reported in the Results (Table 2). For RDR on MAILOR, the text quotes '81.6% (95% CI: 83.9-90.2)' for sensitivity and '81.7% (95% CI: 85.7-89.7)' for specificity, whereas Table 2 lists 81.6% (79.0-84.2) and 81.7% (80.9-82.6). The subsequent sentence 'In terms of RDR prediction, Rajalakshmi et al. report ... compared to the 86.6% ... sensitivity and 87.7% ... specificity obtained by Pegasus' uses PDR values. These appear to be copy-paste errors that must be corrected for the manuscript to be considered reliable.
  4. [Discussion, factor 2] The paper identifies the grading-scheme mismatch as the 'biggest contributing factor' to the RDR drop, yet provides no quantitative estimate of its effect. The authors should either (a) re-grade the MAILOR images according to the ICDR scheme (or have a subset re-graded) and recompute the comparison, or (b) at minimum quantify the fraction of Pegasus false positives that fall into the discordant region (e.g., cases with haemorrhages in fewer than four hemifields, or exudates farther than one disc diameter from the fovea). Without such a quantitative assessment, the claim that the device is responsible for the RDR drop is unsubstantiated.
  5. [Statistical Analysis / Results] The comparison of AUROC between the MAILOR and IDRiD cohorts is unadjusted for any confounders (e.g., image quality, field type, patient demographics, and grading scheme). Given the authors themselves list four differing factors, the permutation test only establishes that the two independent samples have different AUROC values; it does not isolate the device contribution. The authors should temper the causal language and either perform a stratified or matched analysis (e.g., restricting both datasets to macula-centered, good-quality images) or explicitly state that the comparison is descriptive and hypothesis-generating.
minor comments (5)
  1. [References] Reference numbering is disturbed in the Discussion: the 'One of the systems evaluated could not handle disc-centred images... 10' citation appears to refer to Tufail et al. (ref. 12), and the ICDR scale is also ref. 10. Please renumber and re-check all in-text citations for accuracy.
  2. [Conclusion] The claim that this is 'the first to evaluate ... a fully portable, handheld device' is questionable because the Discussion cites Rajalakshmi et al. using a smartphone-based fundus camera, which is also a portable handheld imaging approach. Please qualify the claim (e.g., a dedicated handheld fundus camera) or adjust the literature statement.
  3. [Discussion] There is a typo in the final paragraph: 'clinical practise' should be 'clinical practice'.
  4. [Table 1] The abbreviation 'CRS' in the Table 1 header is not defined in the table footnote; define it (clinical reference standard) or spell out fully.
  5. [Results] The sentence 'There was approximately a 12-13% disparity in the RDR sensitivity/specificity performances between the handheld and benchmark desktop devices' is slightly imprecise; the sensitivity drop is 11.8 percentage points and the specificity drop is 12.5 percentage points. Using exact values would be more rigorous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark comparison with measured AUROCs; the acknowledged grading-scheme mismatch is a confound affecting attribution, not a derivation.

full rationale

This is an empirical device-transfer evaluation, not a derivation chain. Pegasus's RDR and PDR AUROC values on IDRiD and the MAILOR cohort are measured outcomes on independent datasets; no parameter is fitted to produce them, and no equation in the paper equates the predicted labels with the reference labels by construction. The main quantitative claim, that RDR AUROC falls from 98.5% on IDRiD to 89.4% on MAILOR, is an observed comparison, not an algebraic consequence of the inputs. The Discussion explicitly identifies the principal confound: 'the biggest contributing factor is the mismatch in grading systems' (ICDR software output versus Scottish reference standard), and the paper lists grading-mismatch false positives and other factors. That is an attribution/validity concern, not circularity: the measured numbers remain externally determined and were not derived from the conclusion. The only self-citation found is the reference to Ting et al. (JAMA 2017), which includes one of the present authors as a co-author; it is used as an external literature comparison and is not load-bearing for any derived quantity. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in via citation. Internal reporting inconsistencies, such as the Discussion's sensitivity confidence intervals not matching Table 2, are quality issues but do not constitute circular reasoning. Therefore the paper is self-contained as an empirical evaluation and no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rely on assumptions that the IDRiD dataset represents upper-bound desktop performance, that the RDR grade mapping between Scottish and ICDR schemes is meaningful for comparison, that manual image grading is an adequate reference standard, and that excluding ungradable images does not bias the comparison. These are domain assumptions stated or implied in the Methods and Discussion.

assumptions (4)
  • domain assumption The IDRiD benchmark dataset is representative of upper-bound desktop camera performance.
    The authors assume IDRiD provides a good benchmark for the upper bound on diagnostic performance, as stated in Methods, Data Collection.
  • domain assumption The Scottish and ICDR grading schemes can be mapped for RDR in a way that supports cross-dataset comparison.
    The paper maps RDR between schemes but acknowledges the mismatch in Discussion factor 2; the comparison of RDR AUROC between datasets assumes the mapping is meaningful.
  • domain assumption Manual grading by certified experts is an adequate reference standard.
    The CRS is based on image grading, not final clinical diagnosis; the authors note this limitation in Discussion.
  • domain assumption Excluding ungradable and unlabelled images does not bias the comparison between cohorts.
    Images identified as ungradable by graders were excluded for MAILOR, while IDRiD images were pre-verified as adequate quality; this differential exclusion may affect the comparability of AUROC estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation of an AI System for the Detection of Diabetic Retinopathy from Images Captured with a Handheld Portable Fundus Camera: the MAILOR AI study." pith.science (2026). https://pith.science/paper/VAANVNC3

@misc{pith2026190806399,
  author       = {Pith},
  title        = {Pith review of: Evaluation of an AI System for the Detection of Diabetic Retinopathy from Images Captured with a Handheld Portable Fundus Camera: the MAILOR AI study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VAANVNC3}},
  note         = {Machine review of arXiv:1908.06399}
}
read the original abstract

Objectives: To evaluate the performance of an Artificial Intelligence (AI) system (Pegasus, Visulytix Ltd., UK), at the detection of Diabetic Retinopathy (DR) from images captured by a handheld portable fundus camera. Methods: A cohort of 6,404 patients (~80% with diabetes mellitus) was screened for retinal diseases using a handheld portable fundus camera (Pictor Plus, Volk Optical Inc., USA) at the Mexican Advanced Imaging Laboratory for Ocular Research. The images were graded for DR by specialists according to the Scottish DR grading scheme. The performance of the AI system was evaluated, retrospectively, in assessing Referable DR (RDR) and Proliferative DR (PDR) and compared to the performance on a publicly available desktop camera benchmark dataset. Results: For RDR detection, Pegasus performed with an 89.4% (95% CI: 88.0-90.7) Area Under the Receiver Operating Characteristic (AUROC) curve for the MAILOR cohort, compared to an AUROC of 98.5% (95% CI: 97.8-99.2) on the benchmark dataset. This difference was statistically significant. Moreover, no statistically significant difference was found in performance for PDR detection with Pegasus achieving an AUROC of 94.3% (95% CI: 91.0-96.9) on the MAILOR cohort and 92.2% (95% CI: 89.4-94.8) on the benchmark dataset. Conclusions: Pegasus showed good transferability for the detection of PDR from a curated desktop fundus camera dataset to real-world clinical practice with a handheld portable fundus camera. However, there was a substantial, and statistically significant, decrease in the diagnostic performance for RDR when using the handheld device.

Figures

Figures reproduced from arXiv: 1908.06399 by the authors.

Figure 1
Figure 1. Examples of images used in this study. Left: From the benchmark dataset (IDRiD), captured using a desktop (VX-10 alpha, Kowa American Corp., USA) fundus camera. Right: From MAILOR, captured using a portable handheld (Pictor Plus, Volk Optical Inc., USA) fundus camera [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
Figure 2
Figure 2. CONSORT-style diagram for the MAILOR cohort [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 4
Figure 4. Selected examples of false positives for RDR detection from the MAILOR study. Exudate (left) and haemorrhage (right) detections by Pegasus are overlaid on the images as white bounding boxes and pink and blue heatmaps, respectively. Image zooms are inset for features detected by Pegasus that are difficult to see [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Receiver Operating Characteristic (ROC) curves for Pegasus measured on the IDRiD patient cohort for RDR and PDR. The shaded regions indicate the estimated 95% confidence intervals on the ROC curves [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 1
Figure 1. Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p022_1.png]
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p022_2.png]
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p023_3.png]
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages

  1. [1]

    Diabetes prevention: global health policy and perspectives from the ground

    Bergman M, Buysschaert M, Schwarz PE, Albright A, Narayan KV, Yach D. Diabetes prevention: global health policy and perspectives from the ground. Diabetes management (London, England). 2012;2(4):309

    work page 2012

  2. [2]

    Workforce Census, November 2016 {Available at https://www.rcophth.ac.uk/wp-content/uploads/2017/03/RCOphth-Workforce-Census -2016.pdf ​ , accessed 17th October 2018}

    Royal College of Ophthalmologists. Workforce Census, November 2016 {Available at https://www.rcophth.ac.uk/wp-content/uploads/2017/03/RCOphth-Workforce-Census -2016.pdf ​ , accessed 17th October 2018}

    work page 2016

  3. [3]

    The number of ophthalmologists in practice and training worldwide: a growing gap despite more than 200 000 practitioners

    Resnikoff S, Felch W, Gauthier TM, Spivey B. The number of ophthalmologists in practice and training worldwide: a growing gap despite more than 200 000 practitioners. British Journal of Ophthalmology. 2012 Jun 1;96(6):783-7

    work page 2012

  4. [4]

    Dermatologist-level classification of skin cancer with deep neural networks

    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017 Feb;542(7639):115

    work page 2017

  5. [5]

    Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer

    Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Investigative radiology. 2017 Jul 1;52(7):434-40

    work page 2017

  6. [6]

    Retinal imaging and image analysis

    Abràmoff MD, Garvin MK, Sonka M. Retinal imaging and image analysis. IEEE reviews in biomedical engineering. 2010 Dec 10;3:169-208

    work page 2010

  7. [7]

    Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research

    Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, Meriaudeau F. Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research. Data. 2018 Sep;3(3):25. ​ Available from: http://dx.doi.org/10.21227/H25W98

    doi:10.21227/h25w98 2018

  8. [8]

    Quality and learning curve of handheld versus stand-alone non-mydriatic cameras

    Gosheva M, Klameth C, Norrenberg L, Clin L, Dietter J, Haq W, Ivanov IV, Ziemssen F, Leitritz MA. Quality and learning curve of handheld versus stand-alone non-mydriatic cameras. Clinical ophthalmology (Auckland, NZ). 2017;11:1601

    work page 2017

Show all 14 references
  1. [9]

    Grading diabetic retinopathy (DR) using the Scottish grading protocol

    Zachariah S, Wykes W, Yorston D. Grading diabetic retinopathy (DR) using the Scottish grading protocol. Community eye health. 2015;28(92):72

    2015

  2. [11]

    Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence

    Rajalakshmi ​ R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye. 2018 Jun;32(6):1138

    2018

  3. [12]

    Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders

    Tufail A, Rudisill C, Egan C, Kapetanakis VV, Salas-Vega S, Owen CG, Lee A, Louw V, Anderson J, Liew G, Bolter L. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders. Ophthalmology. 2017 Mar 1;124(3):343-51

    2017

  4. [13]

    Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes

    Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, Hamzah H, Garcia-Franco R, San Yeo IY, Lee SY, Wong EY. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. J...

    2017

  5. [14]

    Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 2016 Dec 13;316(22):2402-10. Fi...

    2016

  6. [2016]

    Available from: http://www.icoph.org/downloads/Diabetic-Retinopathy-Scale.pdf ​ 17th October 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.