{"id":"5f82a785-168d-4f94-88de-b0f3b0b14517","arxiv_id":"2504.15928","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GlobeReady, a training-free retrieval-based platform, reports 87.2-98.5% Top-1 accuracy on 11 fundus and 15 OCT disease classes across four countries.","lead":"The paper presents GlobeReady, an AI platform that diagnoses eye diseases from retina photos and OCT scans without retraining or coding. It combines pretrained visual models with a labeled image library, local image augmentation, and confidence scoring, and reports high accuracy across hospitals in China, Vietnam, Singapore, and the UK.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-flagging accuracy and OOD detection rely on a confidence threshold selected using test-set labels (Eq. 3), so the reported 99.4%/96.2% and OOD rates are likely optimistic; a held-out threshold is needed.","rationale":"The reader's weakest_assumption identifies exactly the same circular step: Equation 3 selects θ* using test-set labels, and the paper's headline confidence and OOD numbers are then measured on that same test set. This is a genuine correctness risk rather than a mere reproducibility gap, because it directly inflates the most safety-relevant numbers in the paper. The readers' CONDITIONAL verdict is appropriate: the main diagnostic and retrieval evidence can still be salvaged if the threshold is chosen on a separate validation fold or pre-registered, and if the authors release code or feature libraries for independent checks. I do not see a reason to move the verdict to REJECT: the threshold issue is localized and fixable, and the base retrieval accuracies, while also depending on locally labeled reference data, are internally consistent with a retrieval-augmented approach. My stress-test therefore agrees with the reader and recommends no change to the CONDITIONAL verdict.","tokens_in":19431,"tokens_out":4890,"duration_ms":50885,"concrete_test":"Split each of JLHW11, JSIEC-OCT15, CFPOOD49, and OCT-OOD13 once into a validation fold and a held-out test fold before any threshold computation. Select θ* by maximizing Youden's index on the validation fold only, then compute post-flagging accuracy, recall, and OOD detection on the held-out test fold. If the held-out numbers fall materially below 99.4%/96.2% (e.g., by more than 2–3 points) or the OOD detection rates drop, the Section 2.3–2.4 claims are not deployment-realistic. Report also the fraction of samples flagged so the workload cost of the threshold is transparent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing soft spot is the test-set-selected confidence threshold. Section 4.12 defines θ* as the maximizer of Youden's index over confidence scores computed on the test dataset (Eq. 3: 'based on the distribution of confidence scores in the test dataset'), and Sections 2.3–2.4 then report post-flagging accuracy (99.4% for CFPs, 96.2% for OCT) and OOD detection rates (86.3% and 90.6%) on that same set. This is label-guided threshold selection on the evaluation distribution, not a deployment protocol. At a new site, clinicians will not know the confidence score that separates correct from incorrect predictions, and a threshold tuned to local test labels will be optimistic. It also invalidates the reported P values for thresholded comparisons, since the same data determine both the threshold and the outcome. The central 'confidence-quantifiable' and OOD claims therefore rest on a circular step. The retrieval evidence, including the cross-center improvements, is plausible but depends on labeled local reference libraries; that is a secondary caveat, not the binding weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GlobeReady, a retrieval-based ophthalmic image analysis platform that performs disease diagnosis without retraining or fine-tuning. The system is pretrained with DINOv2 on 38 million synthetic ophthalmic images and with CLIP on 475,845 image-text pairs, then used to extract features for k-nearest-neighbor diagnosis against labeled reference libraries. The authors report Top-1 accuracies of 93.9% on an 11-class CFP dataset and 87.2% on a 15-class OCT dataset, improved cross-center accuracy after local feature augmentation (e.g., 73.4–91.0% in Singapore, 86.3–96.9% in Vietnam, 90.2–98.9% in the UK), and enhanced post-threshold accuracy (99.4% CFP, 96.2% OCT) using a Bayesian confidence mechanism with Monte Carlo dropout. The paper also describes OOD detection for 49 CFP and 13 OCT disease categories, feature-based case retrieval evaluated by three ophthalmologists, and a usability survey of seven clinicians. The central methodological claim is that training-free retrieval, augmented with locally extracted features, is sufficient for deployment across centers and populations.","tokens_in":19645,"tokens_out":2572,"duration_ms":24765,"significance":"If the results hold, GlobeReady would be a practically valuable contribution: it combines large-scale pretraining, a code-free retrieval interface, and local feature augmentation to address domain shift without model retraining. The base retrieval accuracies are internally plausible for a k-NN classifier, and the prospective retrieval evaluation by three ophthalmologists is a genuine strength. However, the headline 'confidence-quantifiable' and OOD results rest on a threshold-selection procedure that uses test-set labels, making those specific numbers optimistic estimates of deployment performance. The cross-center adaptation results are also dependent on labeled local reference libraries, which should be clearly positioned as a supervised adaptation cost. Overall, the paper's core idea is sound and likely publishable, but the confidence-threshold claims need a methodological fix before acceptance.","major_comments":[{"comment":"The confidence threshold θ* is defined in Eq. (3) as the maximizer of Youden's index computed on the test dataset, and Sec. 2.3 then reports post-flagging accuracy of 99.4% (CFP) and 96.2% (OCT) on that same test set. Because the threshold is fitted to the test labels, these numbers are in-sample optimized values, not predictions of deployment performance. The improvement over the unthresholded baseline (93.9% and 87.2%) is therefore partly an artifact of test-set adaptation, and the reported P values comparing thresholded and unthresholded results are invalid for inferring a real gain. Please re-evaluate with a threshold chosen on a separate validation set or fixed a priori, and report the corresponding performance and uncertainty.","section":"Sec. 4.12, Eq. (3); Sec. 2.3"},{"comment":"The OOD detection rates (86.3% for CFPOOD49, 90.6% for OCT-OOD13, and the low-quality image detection rates) are reported using 'confidence-threshold filtering,' but the manuscript does not specify how the threshold was selected for these OOD evaluations. If θ* is again tuned on the OOD test distribution, the same circularity as in Sec. 4.12 applies. Please state explicitly the threshold-selection protocol for all OOD results, including whether the threshold was held out from the OOD test set, and provide detection rates along with confidence intervals.","section":"Sec. 2.4"},{"comment":"The comparison with RETFound and VisionFM reports that GlobeReady showed 'comparable or superior Top-1' accuracy and superior thresholded accuracy, but the threshold for the fine-tuned baselines is not described. For a fair comparison, the confidence-thresholding protocol must be identical across methods: either a fixed threshold for all methods or a separately tuned threshold per method, with the tuning set disjoint from the test set. Otherwise, the superiority claim for thresholded accuracy may reflect test-set tuning rather than a genuine advantage.","section":"Sec. 2.3, Supplementary Fig. 3"}],"minor_comments":[{"comment":"The heading reads 'Ocular disease diagnosis by GlobaFree' and should be 'GlobeReady'; this typo appears to be a simple misspelling.","section":"Sec. 2.1 heading"},{"comment":"The phrase 'P> 0.001' following the OCT post-flagging recall likely should be 'P < 0.001' to match the direction of the reported improvement; please verify and correct.","section":"Sec. 2.3, JSIEC-OCT15 result"},{"comment":"The sentence 'We then curated 475,845 real image-text pairs from these global datasets.' is duplicated verbatim; one occurrence should be removed.","section":"Introduction, paragraph 1"},{"comment":"The abbreviation for color fundus photographs is inconsistent: the abstract and Section 2.1 use 'CPFs' while most of the text uses 'CFPs' (e.g., 'CFPOOD49'). Please standardize to a single abbreviation.","section":"Throughout"},{"comment":"The definition of Sensitivity and Specificity in Eq. (2) is unclear: it should state explicitly that these are computed with respect to the binary event 'prediction is correct' as the positive class, not with respect to a disease label. Please clarify the notation.","section":"Sec. 4.12, Eq. (2)"},{"comment":"The number of stochastic forward passes (100) and the retrieval neighborhood size k are presented as fixed choices, but no sensitivity analysis is provided; a brief statement on how these were chosen would help reproducibility.","section":"Sec. 4.11"}],"recommendation":"major_revision","confidential_remarks":"The paper's core retrieval-based diagnostic approach appears sound and the dataset scale is impressive, but the confidence-threshold and OOD results are the most prominent quantitative claims and they currently rest on test-set-fitted thresholds. This is a fixable methodological issue, but it is load-bearing for the 'confidence-quantifiable' and OOD sections. I would also suggest the editors ask the authors to clarify whether the local feature augmentation requires labeled local data at each deployment site, since that is a practical constraint that should be prominent in the discussion. The paper seems within scope for a computer vision or medical imaging venue, though the clinical claims would benefit from external validation by an independent group."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — GlobeReady is a serious attempt at a training-free retinal diagnosis platform. The headline numbers are probably optimistic, but the core retrieval-based diagnostic results are worth engaging with. The main flaw is the confidence threshold selected on the test set.\n\nWhat's actually new: new datasets (JLHW11, JSIEC-OCT15, CFPOOD49, OCT-OOD13) assembled from multiple centers, and the local feature augmentation idea — basically retrieval-augmented diagnosis. The multi-center evaluation across China, Vietnam, Singapore, and the UK is a strong selling point and gives real signal about domain shift. The platform does what it says without per-site retraining, though it does need locally labeled data to build the reference library.\n\nSoft spots: the post-flagging and OOD numbers (99.4% CFP, 96.2% OCT, 86.3%/90.6% OOD detection) rest on picking θ* via Youden's index on the test set (Eq. 3). That's label-guided threshold selection on the evaluation distribution — a deployment protocol you can't reproduce at a new site. The P values for the thresholded comparisons are invalid because the same data determine both the threshold and the outcome. This is the load-bearing weakness; it needs a genuinely held-out threshold. Secondary: no code or model weights, so the platform can't be checked independently. And the 'training-free' framing overstates the need for local labeled feature libraries.\n\nBase retrieval performance (93.9% top-1 on CFP, 87.2% on OCT) is plausible for k-NN with strong pretrained features, and the cross-center gains after local augmentation are dramatic and believable. The paper would be much stronger with a separate validation set for threshold selection, confidence intervals, and release of the reference libraries or code.\n\nWho's this for? Medical imaging researchers working on domain adaptation or clinician-facing AI platforms. It deserves a serious referee — the core idea is sound, the data collection is substantial, and the threshold flaw is fixable in revision. I'd send it out, but the revision should include a proper validation split and artifact release.","headline":"GlobeReady has a solid retrieval-based core and substantial new datasets, but its headline confidence-threshold and OOD numbers are inflated by choosing θ* on the test set.","tokens_in":20348,"tokens_out":1896,"would_cite":false,"duration_ms":18117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GlobeReady's central claim is that training-free retrieval from a labeled image library, augmented with local features, can diagnose eye diseases across countries and modalities without retraining.","keywords":["ophthalmic foundation model","retrieval-augmented diagnosis","domain shift","fundus photography","optical coherence tomography","confidence quantification","out-of-distribution detection","training-free deployment"],"falsifier":"Re-run the JLHW11 and JSIEC-OCT15 flagging experiments with the confidence cutoff chosen on an independent validation split instead of on the test set; if Top-1 accuracy after flagging falls materially below the reported 99.4% (CFP) and 96.2% (OCT), the deployment-time benefit of the confidence mechanism is not supported.","tokens_in":19226,"feed_emoji":"👁️","tokens_out":6893,"duration_ms":56862,"temperature":0.7,"pith_summary":"GlobeReady is an attempt to remove the main obstacle to using AI in eye clinics: the need to retrain or fine-tune a model for each new hospital, device, or patient population. The paper claims that a single pretrained feature extractor, combined with retrieval from a labeled library of fundus and OCT images, can diagnose 11 disease categories from color fundus photographs (93.9-98.5% Top-1 accuracy) and 15 categories from OCT scans (87.2-92.7% Top-1 accuracy) with no task-specific retraining. In external centers, adding locally extracted features to the library raised accuracy substantially, and a Bayesian confidence mechanism with threshold-based flagging further improved accuracy while detecting diseases never seen in the reference set. If the claim holds, deploying ophthalmic AI would become a matter of uploading images and reference data, not a machine-learning project.","feed_headline":"Eye-disease AI deploys with zero retraining","feed_subtitle":"GlobeReady matches fundus and OCT images against a labeled library, lifting new-clinic accuracy from 33-56% to 90-99%.","key_machinery":"The load-bearing mechanism is a feature-matching diagnostic engine. Its backbone is a ViT-L/16 vision transformer pretrained with DINOv2 self-supervision on synthetic ophthalmic images, then aligned to clinical text with a CLIP-style contrastive objective on over 475,000 image-text pairs. Diagnosis happens by retrieval: the query embedding is compared with a labeled reference library, and the labels of the top matches supply the prediction. Domain adaptation is achieved without weight updates by appending locally extracted features to the library. Confidence comes from 100 stochastic forward passes with Monte Carlo dropout: the consistency of predictions across passes yields a confidence score, and Youden's index selects a threshold for flagging low-confidence cases for clinician review.","core_discovery":"The paper's central claim is that ophthalmic disease diagnosis can be cast as a nearest-neighbor retrieval problem over a fixed feature library rather than as a trained classifier. GlobeReady builds the library by extracting features with a vision transformer pretrained first on 38 million synthetic fundus and OCT images via DINOv2-style self-supervision, then on 475,845 real image-text pairs via CLIP-style contrastive learning. At inference, a query image is matched against the library and the labels of the top matches determine the diagnosis. The authors report Top-1 accuracies of 93.9-98.5% on 11 CFP categories and 87.2-92.7% on 15 OCT categories, and show that augmenting the library with local features lifts cross-center accuracy in Vietnam from 51.0-72.7% to 86.3-96.9% and in the UK from 33.6-55.9% to 90.2-98.9%. A Bayesian variant using Monte Carlo dropout produces confidence scores, and thresholding on those scores raises Top-1 accuracy to 99.4% on CFPs and 96.2% on OCT while flagging out-of-distribution diseases with 86.3% and 90.6% detection rates respectively.","pith_inferences":["If replicated, the retrieval-based design shifts the deployment bottleneck from model training to data curation and library maintenance; a natural next test is how accuracy scales with library size, label noise, and the mix of local versus global reference images.","Because local feature augmentation works without retraining, the same mechanism could be tested on severity grading or on other imaging modalities such as ultrasound or pathology, not just disease classification.","The reported post-flagging accuracies depend on the threshold being chosen from the test distribution, so a practical extension would be per-site threshold selection on a small labeled sample, which the paper does not evaluate.","Retrieval-based diagnosis exposes the evidence behind each prediction, so a follow-up study could measure whether showing clinicians the retrieved similar cases changes their trust in or agreement with the system."],"forward_implications":["A single pretrained feature extractor can serve both fundus photography and OCT diagnosis across 11 and 15 disease categories without task-specific retraining.","Hospitals in new regions can adapt the system by adding their own images to the reference library; the largest accuracy gains appear where baseline performance was lowest.","Confidence scoring with a threshold allows low-confidence diagnoses to be deferred to clinicians, raising Top-1 accuracy to 99.4% on CFPs and 96.2% on OCT after flagging.","The same feature library supports case retrieval, letting clinicians find similar images in unlabeled local databases without retraining.","Clinician usability ratings (SUS 86.4, helpfulness 4.6/5) indicate the platform is operable without programming expertise."],"supporting_citations":[{"why":"Supplies the DINOv2 self-supervised pretraining framework used to initialize the visual backbone on synthetic ophthalmic images.","marker":"[13]"},{"why":"Supplies the CLIP-style contrastive image-text pretraining that aligns visual features with clinical semantic context.","marker":"[14]"},{"why":"FundusGAN generates the 38 million synthetic fundus and OCT images used in the first pretraining stage.","marker":"[33]"},{"why":"Provides the 475,845 ophthalmic image-text pairs used for the contrastive pretraining stage.","marker":"[34]"},{"why":"Supplies the Singapore SEED dataset used to test cross-center, multi-ethnic generalization in local retrieval augmentation.","marker":"[35]"},{"why":"Supplies RETFound, the fine-tuned ophthalmic foundation model that GlobeReady is compared against on the same datasets.","marker":"[8]"},{"why":"Supplies VisionFM, the other fine-tuned foundation model baseline in the comparison.","marker":"[9]"},{"why":"Provides the retrieval-augmented generation concept that inspired the local feature augmentation approach.","marker":"[15,16]"},{"why":"Supplies the Monte Carlo dropout method behind the Bayesian confidence quantification.","marker":"[40]"},{"why":"Supplies Youden's index, used to select the confidence threshold for flagging low-confidence diagnoses.","marker":"[41]"}],"fun_headline_variants":["Zero-retraining eye AI matches scans to diagnose 15 diseases","No-retrain eye AI generalizes across 4 nations, up to 99% accurate","Eye diagnosis by retrieval: no fine-tuning, works in new clinics","GlobeReady: eye AI that skips retraining, hits 99% accuracy","Eye AI that matches, not trains: new clinics hit 90-99%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The post-flagging accuracy and OOD detection numbers assume the confidence cutoff is chosen using the test set answers themselves (Youden's index on the test distribution), so those numbers may not hold for a cutoff fixed before deployment.","fun_headline_variants_meta":{"raw":{"variants":["Zero-retraining eye AI matches scans to diagnose 15 diseases","No-retrain eye AI generalizes across 4 nations, up to 99% accurate","Eye diagnosis by retrieval: no fine-tuning, works in new clinics","GlobeReady: eye AI that skips retraining, hits 99% accuracy","Eye AI that matches, not trains: new clinics hit 90-99%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001587,"raw_usage":{"total_tokens":6421,"prompt_tokens":1127,"completion_tokens":5294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":5192}},"tokens_in":743,"tokens_out":5294,"duration_ms":38069,"temperature":1.0,"reasoning_tokens":5192,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:14:12.942821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the JLHW11 and JSIEC-OCT15 flagging experiments with the confidence cutoff chosen on an independent validation split instead of on the test set; if Top-1 accuracy after flagging falls materially below the reported 99.4% (CFP) and 96.2% (OCT), the deployment-time benefit of the confidence mechanism is not supported.","supporting_citations":[{"cited_title":"Cohort profile: the singapore epidemiology of eye diseases study (seed),","cited_arxiv_id":null,"evidence_quote":"Supplies the Singapore SEED dataset used to test cross-center, multi-ethnic generalization in local retrieval augmentation."},{"cited_title":"A foundation model for generalizable disease detection from retinal images,","cited_arxiv_id":null,"evidence_quote":"Supplies RETFound, the fine-tuned ophthalmic foundation model that GlobeReady is compared against on the same datasets."},{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo dropout method behind the Bayesian confidence quantification."},{"cited_title":"Youden index and associated cut-points for three ordinal diagnostic groups,","cited_arxiv_id":null,"evidence_quote":"Supplies Youden's index, used to select the confidence threshold for flagging low-confidence diagnoses."}],"review_version":1}