{"id":"8139bb50-3eec-46f9-b96f-7c90f87ac7dc","arxiv_id":"2412.06624","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A model predicts visual acuity from fundus images with PAC-guaranteed prediction intervals, achieving 71.49% coverage at a 70% target on a 0 to 10 scale.","lead":"This paper trains computer models to estimate eyesight from photos of the back of the eye, then wraps each estimate in a range that is statistically guaranteed to contain the true eyesight most of the time. It is the first application of such guarantees to eyesight prediction, with accuracy comparable to earlier models that offered no guarantees.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PAC calibration is standard, but the iid assumption in Eq. (1) may fail at the sampling-unit level: the paper never states whether the 54,781 images are split by patient, so clustered images could invalidate the empirical coverage claim.","rationale":"The paper applies a well-established PAC interval construction, and the finite-sample calibration logic is mathematically sound if the exchangeability assumption holds. The reader identified dataset shift as the weakest assumption; I agree that is a valid limitation, but it is explicitly acknowledged in Section 4.6 and Appendix B.2. A more immediate and possibly more damaging issue is whether the experiment itself satisfies the iid precondition of Eq. (1). If the 54,781 images are split at image level rather than patient level, within-patient correlation makes the calibration indicators dependent, and the Clopper-Pearson confidence bound used to select c in Section 3.3 is no longer exact. This would not be a failure of the PAC theorem but a failure of the empirical claim that the guarantee is upheld on this dataset. The reader's conditional verdict already requires additional data details; I would keep CONDITIONAL and add the explicit condition that patient-level independence be verified. If the patient-level re-split preserves coverage, the central claim is credible and the concern is resolved.","tokens_in":13254,"tokens_out":7816,"duration_ms":95987,"concrete_test":"Obtain de-identified patient/eye identifiers from the dataset owners and re-run the full Section 3.3 pipeline with patient-level splitting, so no patient appears in more than one of train, validation, and test, using the same models and delta=0.001%. If the patient-level split coverage for EfficientNetV2-S at epsilon=0.3 falls below 70% or the average width changes materially, the empirical central claim is not supported under Eq. (1). If the dataset contains exactly one image per patient/eye and the existing splits are already at the patient level, report that evidence and the concern dissolves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the PAC guarantee of Eq. (1), which is valid only if the validation set Zn and the test point are iid draws from D. Section 4.1 says the dataset is 'randomly divided' into 6:2:2 splits, but it does not state the sampling unit. Fundus datasets of this size typically contain multiple images per patient and per eye, and if image-level random splits are used, images from the same patient can appear in training, calibration, and test. Then Dn = D x ... x D is not an accurate model: images within a patient are correlated, and the calibration indicators W_i are not independent Bernoulli draws. The Clopper-Pearson lower bound used to choose c in Section 3.3 assumes a binomial count; with positively correlated indicators it becomes anti-conservative, so the reported 71.49% coverage for epsilon=0.3 could reflect patient leakage rather than a genuine PAC guarantee. Appendix B.2 already shows coverage dropping when the distribution shifts, confirming sensitivity to the exchangeability assumption; patient-level dependence is a violation of that assumption inside the claimed experimental setup, not merely an external limitation. The paper does not report unique patient counts, patient-level splitting, or any clustering adjustment, so the empirical demonstration of Eq. (1) is not yet established on the actual data-generating process.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for constructing prediction intervals for visual acuity (VA) estimates from fundus images, using a regressor that outputs both a mean and a standard deviation, and then calibrating a width multiplier c on a validation set via a Clopper-Pearson-based PAC bound. The central claim, expressed in Eq. (1), is that the interval C_c(x) = [f_mu(x) - c f_sigma(x), f_mu(x) + c f_sigma(x)] satisfies a PAC coverage guarantee: with probability at least 1-delta over the validation set, the true coverage on future data from the same distribution is at least 1-epsilon. Experiments on a 54,781-image fundus dataset with four base models and five random splits report that the coverage bound is satisfied; for epsilon=0.3 and delta=0.001%, EfficientNetV2-S achieves 71.49% coverage with an average width of 3.04. The paper also compares point- and interval-level performance with Bayesian neural networks, vanilla conformal prediction, and two prior VA prediction studies.","tokens_in":13541,"tokens_out":5328,"duration_ms":56138,"significance":"If the central claim is accepted, the paper would provide a clinically relevant application of PAC-style uncertainty quantification to visual acuity prediction, a task where prior work offers only point predictions without guarantees. The positive aspects are that the method uses a finite-sample exact calibration tool (Clopper-Pearson), the experiments are repeated over multiple splits, the code is made available, and the authors are transparent about the degradation of coverage under severe distribution shift in Appendix B.2. The methodological novelty is limited because the PAC calibration machinery is imported from Park et al. (2019), and the paper's contribution is mainly in the application and in the empirical demonstration on a real medical dataset. However, a load-bearing detail is missing: the paper does not state whether the data split is at the image level or the patient level, and this determines whether the i.i.d. assumption in Eq. (1) holds. The significance of the empirical result therefore cannot be fully assessed as written.","major_comments":[{"comment":"The paper states that the dataset is 'randomly divided' into 6:2:2 training/validation/test splits but never specifies the sampling unit. Fundus image datasets of this size routinely contain multiple images per patient and per eye. If the split is image-level, images from the same patient can appear in all three sets, and the calibration indicators W_i = 1{y_i in C_c(x_i)} are not independent Bernoulli draws as required by the Clopper-Pearson bound used in Section 3.3. Positively correlated indicators make the binomial-based lower confidence bound anti-conservative, so the reported coverage rates, e.g., 71.49% for epsilon=0.3 in Table 2, could reflect patient leakage rather than a genuine PAC guarantee. The paper must report the number of unique patients, the number of images per patient, and results with a patient-disjoint split, or justify explicitly why image-level independence is an appropriate model for this dataset.","section":"Section 4.1, Eq. (1)"},{"comment":"The optimization problem defining c* is not coherent as written. The text reads 'c* = arg min_c c subject to c >= 1 - epsilon, where [c, c] is the Clopper-Pearson interval for W = {1(y_i in C_c(x_i)) | (x_i, y_i) in Z} with significance level delta.' This mixes the width multiplier c with a coverage threshold: the constraint 'c >= 1 - epsilon' does not make sense dimensionally, and the notation '[c, c]' for the Clopper-Pearson interval is undefined. The intended condition is presumably that the lower Clopper-Pearson confidence bound for the coverage of C_c on the validation set is at least 1-epsilon. The authors should restate the optimization correctly, specify the search procedure for c (e.g., grid range and step), and report the validation-set size used in the calibration. As written, Section 3.3 is not reproducible.","section":"Section 3.3"},{"comment":"The abstract and conclusion claim that the proposed method is 'comparable to or better than' the two prior works (Kim et al., 2022; Paul et al., 2023). The comparison in Section 4.5 is based on different datasets, different label schemes, different evaluation metrics (macro-accuracy on a balanced test set vs. coverage on the full imbalanced test set), and the authors themselves note that 'a fair comparison is challenging.' These limitations mean the headline claim is not supported by the reported numbers. The claim should be softened to 'comparable in reported point-prediction metrics on separate datasets' or the comparisons should be made under matched conditions, such as applying the prior methods to the same data.","section":"Section 4.5, Abstract"}],"minor_comments":[{"comment":"The notation '[c, c]' should be replaced with explicit lower and upper confidence bounds, e.g., [L(c), U(c)], to avoid confusion with the width multiplier c.","section":"Section 3.3"},{"comment":"Please report the number of unique patients and the per-patient image count; this information is essential for assessing the validity of the i.i.d. assumption and for the reproducibility of the dataset split.","section":"Section 4.1"},{"comment":"The red dotted line indicating the coverage bound is only drawn in the left-column plots; the right-column width plots should either include a note that no bound applies or use a different labeling to avoid implying a width bound.","section":"Figure 4"},{"comment":"The sentence 'However, for practical usage with this 70% coverage, we require a slightly narrower width, around 2, which aligns with the variability in VA measurement by humans' is a value judgment that mixes clinical heuristics with the reported results; it could be better phrased as a clinical requirement that should be validated with domain experts.","section":"Section 4.3.2"},{"comment":"The code repository URL in the footnote contains a space ('va pred pac') and is not a clickable link in the provided text; please provide the correct and complete URL.","section":"Footnote 1"},{"comment":"The explanation of why MA-ACC can be lower than the guaranteed coverage appears as a footnote; this caveat is important and should be moved into the main text so that readers do not misinterpret the comparison.","section":"Section 4.5.1, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The PAC guarantee is sourced from Park et al. (2019), which shares an author with the present paper (Insup Lee). This is not circular, but the novelty should be framed as an application of an existing calibration procedure to a new clinical task, and the authors should make the provenance of the bound explicit in the introduction. The missing patient-level split information is the most serious technical gap; without it, the empirical evidence for the PAC guarantee is inconclusive. The manuscript is otherwise a reasonable short-paper contribution if the method section is corrected and the claims are recalibrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core of this paper is simple: train a regression model with Gaussian outputs, calibrate a constant c on a validation set using a Clopper-Pearson lower bound, and report coverage with a PAC guarantee. That algorithm is from Park et al. (2019), and the authors say so. The genuinely new piece is the application to visual acuity prediction from fundus images, with a careful empirical study on 54,781 images and five repeated splits. They show that coverage stays above the target across epsilon values, and that the intervals are adaptive via the per-example standard deviation. They also compare against BNNs and vanilla conformal prediction, and they are explicit that their method does not fix dataset shift. On those terms, the paper is a solid, honest application of known machinery.\n\nThe main soft spot is the sampling unit. The paper says the dataset is 'randomly divided' 6:2:2, but never says whether the split is by image or by patient. With 54k images from what is presumably a few thousand patients, image-level splitting puts images from the same patient in training, calibration, and test. That violates the exchangeability assumption behind the Clopper-Pearson bound, and the resulting coverage estimate can be anticonservative. This is not a hypothetical concern; the PAC guarantee in Eq. (1) depends on Dn = D × ... × D, which requires independent calibration points. The paper needs to state the number of unique patients and perform either patient-level splits or report cluster-adjusted coverage. Until that is done, the empirical claim in Section 4.3.2 is not fully established.\n\nThere is also a practical caveat the paper acknowledges: the best average width is 3.04 on a 0-10 scale, wider than the ~2 units of human measurement variability they cite. That limits clinical utility somewhat, but it is not a flaw in the method—it is a realistic statement of where the field stands.\n\nOverall, the central argument holds up if the split unit is corrected. This deserves a serious referee: the application is new, the experiments are transparent, and the missing detail is fixable in revision. I would ask the authors to report patient counts and re-run the splits at the patient level before accepting.\n\nBest.","headline":"A clean, modest first application of PAC prediction intervals to VA screening, but the missing patient-level split detail leaves the central empirical claim under-specified.","tokens_in":14072,"tokens_out":2498,"would_cite":false,"duration_ms":24686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fundus-image AI can give visual-acuity estimates a PAC coverage guarantee.","keywords":["Visual Acuity Prediction","Fundus Images","Prediction Intervals","PAC Guarantees","Uncertainty Quantification","Clopper-Pearson","Gaussian Regression"],"falsifier":"Draw a fresh iid validation and test split from the same source and repeat the calibration 1,000 times; if the empirical coverage is below $1-\\epsilon$ in a fraction of repetitions much larger than $\\delta$, the Clopper-Pearson calibration is not delivering the stated guarantee. Alternatively, take the exact pipeline and evaluate on fundus images from a different device or with Gaussian blur kernel size 9 as in the paper; the coverage falling below the bound in that setting would falsify any claim that the guarantee survives distribution shift.","tokens_in":13080,"feed_emoji":"👁️","tokens_out":7993,"duration_ms":75025,"temperature":0.7,"pith_summary":"The paper sets out to give visual acuity (VA) predictions from fundus images a formal reliability certificate: instead of a single point estimate, the model outputs an interval that contains the true VA with probability at least $1-\\epsilon$, with high confidence $1-\\delta$. The proposed method trains a regressor whose output is a Gaussian distribution (mean and standard deviation), then calibrates one width multiplier $c$ on a validation set so that the interval $[f_\\mu(x) - c f_\\sigma(x), f_\\mu(x) + c f_\\sigma(x)]$ satisfies the PAC guarantee of Eq. (1). On a 54,781-image fundus dataset with VA labels 0–10, the method empirically meets the coverage bound for $\\epsilon=0.2$, $0.3$, $0.4$ across four architectures, with the narrowest average width 3.04 at $\\epsilon=0.3$ (EfficientNetV2-S). The practical payoff is that screening tools could hand clinicians intervals with a stated long-run coverage rate, something prior VA prediction models did not offer.","feed_headline":"PAC-guaranteed visual-acuity intervals from fundus images","feed_subtitle":"Calibrated intervals meet the 70% coverage target with average width 3.04 on 54k+ fundus images.","key_machinery":"The load-bearing object is the Gaussian-parameterized regressor: $f(x) = (f_\\mu(x), f_\\sigma(x))$, trained with negative log-likelihood loss. The standard deviation output $f_\\sigma(x)$ sets the per-example interval half-width, so the interval is $C_c(x) = f_\\mu(x) \\pm c f_\\sigma(x)$. The calibration constant $c$ is chosen by solving $\\min c$ subject to the condition that the Clopper-Pearson lower confidence bound for the coverage of $C_c$ on the validation set is at least $1-\\epsilon$ at significance $\\delta$; this one step is what confers the PAC guarantee.","core_discovery":"The paper's central claim is that a regression model for visual acuity, trained to output both a mean and a standard deviation, can be converted into a PAC prediction interval $C_c(x) = [f_\\mu(x) - c f_\\sigma(x), f_\\mu(x) + c f_\\sigma(x)]$ by choosing the scalar $c$ through a Clopper-Pearson binomial confidence bound on a validation set (at significance $\\delta$). The authors report that on the fundus dataset, the empirical coverage stays above the target bound $1-\\epsilon$ for all tested architectures and values of $\\epsilon$, with EfficientNetV2-S achieving average width 3.04 at a 70% coverage target. They further report performance comparable to or better than two earlier VA prediction studies that provide no guarantees, and they show that interval width adapts to the model's estimated per-example uncertainty.","pith_inferences":["The calibration step is agnostic to the base architecture, so the same pipeline could be dropped onto stronger backbones (e.g., EfficientNetV2-L or retinal foundation models) that the paper names as future work, potentially shrinking widths while preserving the guarantee.","Because the interval width scales linearly with the predicted standard deviation, the method implicitly treats 'hard' images (blurry, or from underrepresented acuity classes) as higher-uncertainty; a direct test would be to check whether the coverage gap between the guaranteed bound and empirical coverage widens for the minority classes.","The paper's robustness result under mild blur (kernel size up to 5) suggests that the method may tolerate mild distribution shift, but the sharp failure at kernel sizes 7–9 indicates a natural safety threshold that deployment systems could detect and act on."],"forward_implications":["With a PAC guarantee in place, a clinician can state that the true VA lies inside the reported interval with a known minimum long-run probability, rather than relying on an unguaranteed point estimate.","Interval widths are example-dependent: images the model is unsure about receive wider intervals, which is a step toward flagging low-quality or atypical images.","The method reaches comparable or better point-estimate performance than prior VA prediction models while adding the coverage guarantee.","Because the guarantee is conditional on exchangeability, deployment requires monitoring for dataset shift; the paper's blur experiment shows coverage can drop below the bound when that condition is violated.","The same PAC calibration procedure is proposed by the authors for downstream classification tasks such as diabetic retinopathy and glaucoma detection."],"supporting_citations":[{"why":"Supplies the PAC prediction-interval calibration: choosing c via the Clopper-Pearson bound so the constructed interval satisfies Eq. (1).","marker":"Park et al., 2019"},{"why":"Cited as the guarantee that the interval C_c* satisfies the PAC coverage condition.","marker":"Park et al., 2022a"},{"why":"Source of the 54,781 fundus images and VA labels used to train and evaluate, and the four-level classification baseline.","marker":"Kim et al., 2022"},{"why":"Regression baseline for VA prediction; used to compare error distributions and establish comparable accuracy.","marker":"Paul et al., 2023"},{"why":"Defines the PAC framework that gives meaning to the coverage guarantee.","marker":"Valiant, 1984"},{"why":"Provides the ~2-letter repeatability of human VA measurements used to judge whether the achieved interval width is clinically useful.","marker":"Vesel` y and Synek, 2012"}],"fun_headline_variants":["Fundus images yield PAC-guaranteed visual acuity intervals","Visual acuity from fundus images with PAC coverage guarantees","PAC-calibrated VA intervals from fundus images match or beat prior works","VA intervals from fundus images: PAC guarantee meets 70% target","EfficientNetV2-S PAC intervals: 70% coverage from fundus images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee is only as good as the assumption that the validation set used to pick the width multiplier and the test images are exchangeable draws from the same distribution; if deployment data is shifted (different camera, different population, degraded images), coverage can fall below the bound, as the paper's own blur experiment with kernel sizes 7 and 9 shows.","fun_headline_variants_meta":{"raw":{"variants":["Fundus images yield PAC-guaranteed visual acuity intervals","Visual acuity from fundus images with PAC coverage guarantees","PAC-calibrated VA intervals from fundus images match or beat prior works","VA intervals from fundus images: PAC guarantee meets 70% target","EfficientNetV2-S PAC intervals: 70% coverage from fundus images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002082,"raw_usage":{"total_tokens":8075,"prompt_tokens":901,"completion_tokens":7174,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":7081}},"tokens_in":517,"tokens_out":7174,"duration_ms":45804,"temperature":1.0,"reasoning_tokens":7081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:28:00.551964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Draw a fresh iid validation and test split from the same source and repeat the calibration 1,000 times; if the empirical coverage is below $1-\\epsilon$ in a fraction of repetitions much larger than $\\delta$, the Clopper-Pearson calibration is not delivering the stated guarantee. Alternatively, take the exact pipeline and evaluate on fundus images from a different device or with Gaussian blur kernel size 9 as in the paper; the coverage falling below the bound in that setting would falsify any claim that the guarantee survives distribution shift.","supporting_citations":[{"cited_title":"A deep learning ensemble method to visual acuity measurement using fundus images","cited_arxiv_id":null,"evidence_quote":"Source of the 54,781 fundus images and VA labels used to train and evaluate, and the four-level classification baseline."},{"cited_title":"Accuracy of artificial intelligence in estimating best-corrected visual acuity from fundus photographs in eyes with diabetic macular edema","cited_arxiv_id":null,"evidence_quote":"Regression baseline for VA prediction; used to compare error distributions and establish comparable accuracy."},{"cited_title":"Repeatability and reliability of the visual acuity examination on logmar etdrs and snellen chart","cited_arxiv_id":null,"evidence_quote":"Provides the ~2-letter repeatability of human VA measurements used to judge whether the achieved interval width is clinically useful."}],"review_version":1}