{"id":"8274c1a6-afa3-4a70-b3ba-33ff7ca11924","arxiv_id":"1908.02300","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A transfer learning pupil detector, RAPDNet, is shown to detect relative afferent pupillary defect from headset videos with 90.6% sensitivity and specificity over 64 cases, outperforming three handcrafted algorithms.","lead":"This paper introduces RAPDNet, a transfer learning system that detects a vision-threatening pupil reflex disorder called RAPD from headset camera videos. It reports higher accuracy than three handcrafted algorithms on a new clinical dataset of 64 cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported sensitivity/specificity are optimized on the same 64 test cases used for evaluation, making the headline point estimates in-sample and likely optimistic.","rationale":"The reader's concern about ground-truth label reliability is legitimate: the clinical diagnosis was made via subjective swinging flashlight and neutral density filter tests with no inter-rater reliability measure, so label noise could inflate or deflate the reported metrics. However, the most load-bearing weakness is the test-set threshold and configuration selection. The paper explicitly states that the threshold is selected to maximize sensitivity and specificity on the same 64 cases, and that the patch size and smoothing are chosen by highest AUC on the same set. This makes the reported 90.6% sensitivity/specificity and 0.929 AUC in-sample optima. The central claim in the abstract presents these as achievable performance levels, but a clinician adopting RAPDNet would likely see lower performance. The comparison between RAPDNet and the three handcrafted algorithms is still informative because all methods undergo the same selection procedure, and the paper's contribution of a new dataset and a transfer-learning pupil detector remains useful. Therefore the reader's CONDITIONAL verdict is appropriate: the work is promising but the evaluation must be redone with a properly held-out test set or cross-validation before the numerical claims can be trusted. I agree with the reader's direction but identify a different, more directly load-bearing weak point.","tokens_in":13634,"tokens_out":2986,"duration_ms":33994,"concrete_test":"Re-run the RAPD classification with a nested evaluation: split the 64 cases into a development set (e.g., 44 cases) and a held-out test set (20 cases). Select the threshold and the configuration (patch size and smoothing) using only the development set, then evaluate on the held-out set. Repeat with random splits or use leave-one-out cross-validation with threshold selection inside each training fold. If the held-out AUC or sensitivity/specificity falls materially below 0.929 and 90.6% (e.g., by more than 5 percentage points), the reported headline performance is not a reliable estimate of real-world screening accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, sensitivity/specificity of 90.6% and AUC 0.929, rests on an evaluation protocol that selects both the classification threshold and the algorithm configuration on the test set itself. In Section IV-D, the threshold is chosen by measuring sensitivity and specificity for all possible values and selecting the one that maximizes both, using the 64 test cases. In Section V-B, the patch size and smoothing configuration are also selected on the same test set by highest AUC. Consequently, the reported operating point is the best achievable on this cohort, not an unbiased estimate of screening performance on new patients. The comparison between RAPDNet and handcrafted algorithms may still be fair because all go through the same selection procedure, but the abstract's claim that RAPDNet 'can achieve' these numbers is not supported as a prospective accuracy. There is no external validation, no confidence intervals, and no nested cross-validation; the uncertainty of the point estimate is unknown. This is the most load-bearing concern because if a held-out evaluation yields substantially lower numbers, the central contribution is weakened. The ground-truth reliability issue noted by the reader is also relevant, but this test-set selection is directly and immediately tied to the headline metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an automated framework for screening relative afferent pupillary defect (RAPD) using video sequences recorded by a custom headset. Pupils are localized either by handcrafted algorithms (Starburst, ExCuSe, ElSe) or by a proposed transfer-learned pupil detector based on AlexNet with a single new fully connected layer, trained on ElSe dataset patches. Pupil diameters are estimated via Circular Hough Transform, and a dissimilarity index (RAPD index) is computed from left/right pupil size changes. On 64 balanced clinical test cases, the proposed RAPDNet achieves 90.6% sensitivity and specificity and AUC 0.929, outperforming the handcrafted algorithms. The authors conclude that transfer learning from generic object recognition provides a practical objective RAPD screening method.","tokens_in":13839,"tokens_out":5418,"duration_ms":48073,"significance":"If the reported performance transfers to independent clinical data, this would be a practically valuable contribution: a low-cost, portable, automated RAPD screening tool. The paper's strengths include the development of a new clinical dataset (RAPD-HD), a careful comparison of existing pupil datasets, a benchmark of three established pupil localization algorithms, and a transfer-learning scheme that requires training only one fully connected layer. The pupil detector itself is evaluated on a separate (ElSe) test set, which is a sound use of external data. However, the headline RAPD detection metrics are compromised by test-set selection of the threshold and configuration, and the claimed dominance across all metrics relies on different configurations per metric. These issues need to be addressed before the quantitative claims can be accepted.","major_comments":[{"comment":"The classification threshold is selected on the same 64 test cases by measuring sensitivity and specificity for all possible threshold values and choosing the one that maximizes both (Section IV-D), and the patch size and smoothing configuration are selected per algorithm by the highest AUC on the same test set (Section V-B). Consequently, the reported sensitivity of 90.6%, specificity of 90.6%, and AUC of 0.929 are in-sample optimized values rather than unbiased estimates of performance on new patients. The abstract's claim that RAPDNet 'can achieve' these numbers is not supported as a prospective accuracy. The authors should provide a held-out validation cohort, nested cross-validation, or at least bootstrap confidence intervals, and should explicitly state that the current numbers are the result of test-set selection.","section":"V-B, IV-D"},{"comment":"The claim that 'RAPDNet outperforms all other algorithms in every performance category' is based on highlighting the best value per metric across rows, and different RAPDNet configurations attain the best value for different metrics: for example, the highest AUC ROC (0.929) uses image size/2 with moving average, while the highest accuracy (92.2%) and specificity (93.8%) use 60x60 without smoothing. Thus no single RAPDNet configuration simultaneously achieves all of the best values, and the comparison does not establish that a fixed RAPDNet pipeline dominates all handcrafted algorithms on all metrics. The authors should either compare configurations on a fixed basis (e.g., a single chosen configuration per algorithm, selected a priori) or clearly state that different configurations are used for different metrics.","section":"Table VI"},{"comment":"The ground-truth RAPD labels were obtained from clinical diagnoses based on the swinging flashlight test and neutral density filter test, which are subjective. The paper reports no inter-rater agreement, no repeatability analysis, and no description of how many practitioners performed the diagnoses or whether they were masked to the algorithm results. If the reference standard is unreliable, the reported sensitivity and specificity are not anchored to a trustworthy ground truth. The authors should add a reliability analysis (e.g., repeated examinations or agreement between clinicians) or at minimum discuss this limitation explicitly.","section":"III"},{"comment":"The definition of the RAPD index is not fully specified. The equation uses ΔR and ΔL as the 'pupil size change in the right eye' and 'pupil size change in the left eye,' but the manuscript does not define over which temporal interval or stimulus condition these changes are measured, nor how the multiple direct and consensual light reflex instances are aggregated. Without this information, the framework is not reproducible and the index could be computed in multiple inequivalent ways. Please specify the exact signal-processing steps that produce the scalar ΔR and ΔL for each test case.","section":"IV-D"}],"minor_comments":[{"comment":"The word 'propse' should be 'propose'.","section":"I"},{"comment":"The word 'hancrafted' should be 'handcrafted'.","section":"V"},{"comment":"The training images are reported as '3,620 images from datasets XII, X, XIII, XIV, XVI, XVII, and XVIII in the ElSe study'; please clarify whether these are individual frames and how the dataset labels (e.g., 'XII') map to the ElSe publication.","section":"IV-B"},{"comment":"The phrase 'select the values that correspond to highest sensitivity and specificity' is ambiguous; it should state how ties between sensitivity and specificity are resolved.","section":"IV-D"}],"recommendation":"major_revision","confidential_remarks":"The paper's main quantitative claims require a revision of the evaluation protocol. The test-set selection issue is likely to be raised by any statistically informed reviewer. Additionally, the authors hold patents and royalty interests in the technology; while this is properly disclosed, the editor may want to ensure the clinical validation is independent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing in this paper is the RAPD-HD dataset: 64 test cases, balanced, captured with a controlled headset, each with video sequences, clinical labels, and medical history. That fills a real gap—existing pupil datasets are mostly static, uncontrolled, and lack RAPD annotations. The benchmark is systematic: three handcrafted pupil detectors (Starburst, ExCuSe, ElSe) against a transfer-learned AlexNet detector, scored on nine metrics with ROC analysis. The RAPD dissimilarity index in Eq. 1 is simple and reasonable.\n\nThe transfer learning approach itself is not novel, but applying it to pupil detection within this framework is sensible, and training only a single fully connected layer is a practical choice for a small clinical dataset. I believe the comparative claim—that RAPDNet outperforms the handcrafted baselines on this data—is plausible, because all methods go through the same evaluation pipeline.\n\nThe soft spot is the evaluation protocol, and it is load-bearing. Section IV-D selects the classification threshold to maximize sensitivity/specificity on the same 64 test cases. Section V-B then selects patch size and smoothing configuration by AUC on those same cases. So the headline 90.6% sensitivity/specificity and AUC 0.929 are the best achievable points on this cohort, not unbiased estimates for new patients. There is no held-out validation, no confidence intervals, no nested cross-validation. The reader's concern about ground-truth reliability is also fair: the reference diagnosis uses subjective swinging flashlights and neutral density filters, with no inter-rater agreement reported.\n\nOne minor inconsistency: the text says 64 test cases with two sequences each, while the dataset table lists 64 videos and 24 subjects. That needs a clarifying sentence.\n\nBottom line: this is a solid clinical engineering contribution, but the reported performance numbers need re-estimation with a proper held-out threshold. I would send it to a serious reviewer, with the expectation that the authors revise the evaluation, report uncertainty, and ideally release the data and code. If the numbers survive honest validation, this would be a useful reference for automated RAPD screening.\n\nRecommendation: engage with it critically; the dataset and benchmark are worth building on, but cite the numbers with caution.","headline":"A useful new RAPD dataset and a plausible comparative benchmark, but the headline accuracy numbers are in-sample and should not be taken as prospective performance.","tokens_in":14416,"tokens_out":3753,"would_cite":false,"duration_ms":36992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A headset-based automated swinging flashlight test using a transfer-learned pupil detector screens for relative afferent pupillary defect at 90.6% sensitivity and specificity.","keywords":["relative afferent pupillary defect","pupillary light reflex","transfer learning","pupil detection","RAPDNet","swinging flashlight test","pupillography","clinical video dataset"],"falsifier":"Run the full pipeline on a new cohort in which each subject's RAPD status is established by masked agreement of at least two experienced examiners using standardized neutral-density-filter quantification, or by an independent automated pupillograph, and compare RAPDNet's thresholded scores against those labels; a large drop from 90.6% sensitivity/specificity, or substantial disagreement with the original clinical labels, would weaken the central claim.","tokens_in":13430,"feed_emoji":"👁️","tokens_out":8906,"duration_ms":83202,"temperature":0.7,"pith_summary":"This paper tries to establish that an automated, objective screening test for relative afferent pupillary defect—an asymmetry in how the two pupils react to light that can signal optic nerve damage—can be built from a headset that runs a swinging flashlight test and a pupil detector obtained by transfer learning. The authors introduce RAPDNet, which reuses the visual feature layers of a generic object-recognition network and trains only one final layer to recognize pupil patches. On 64 balanced clinical test cases, RAPDNet achieves 90.6% sensitivity and specificity (AUC 0.929), and it outperforms three handcrafted pupil-localization benchmarks on all nine performance metrics they report. If this holds, portable and consistent RAPD screening could complement subjective in-clinic testing.","feed_headline":"Headset eye test catches optic-nerve defect at 90.6%","feed_subtitle":"A transfer-learned pupil detector outperforms handcrafted trackers on every reported metric in 64 clinical cases.","key_machinery":"The central object is RAPDNet, a transfer-learned pupil detector. A convolutional feature extractor pretrained for generic object recognition is kept fixed; its final classifier is replaced with one fully connected layer trained from scratch on 50x50 pupil/no-pupil patches from existing eye-tracking datasets, converging in a few epochs. During screening, overlapping patches are classified, the median of the top-5 confident pupil patches gives the pupil center, and a circular Hough transform measures pupil radius in a cropped patch. The RAPD index, $1 - \\min(|\\Delta R|, |\\Delta L|) / \\max(|\\Delta R|, |\\Delta L|)$, converts the left and right absolute pupil-size changes into a 0-to-1 dissimilarity score, and a per-algorithm threshold turns that score into a binary RAPD call. The transfer-learning machinery does the work of avoiding handcrafted feature engineering: the pretrained features already encode curved and intensity edges that distinguish the dark pupil from the surrounding iris.","core_discovery":"RAPDNet—a pupil detector made by taking a pretrained AlexNet convolutional network, discarding its original classifier, and training a single fully connected layer to label overlapping image patches as pupil or no pupil—can drive an automated swinging-flashlight RAPD test. Combined with circular Hough transform pupil-size measurement and a proposed RAPD index, the system classifies RAPD-positive versus no-RAPD cases with 90.6% sensitivity and 90.6% specificity and an AUC of 0.929 over 64 balanced test cases from a headset-collected HD video dataset. The paper reports that RAPDNet beats the Starburst, ExCuSe, and ElSe pupil-localization baselines on every one of the nine metrics considered, with the best handcrafted AUC at 0.848. It also reports that the handcrafted algorithms need per-dataset parameter tuning, whereas RAPDNet inherits generic visual features and learns only the final mapping.","pith_inferences":["Editorial inference: the reported 90.6% figures are likely optimistic because the detection threshold was selected on the same 64 test cases that produced the ROC curve; an independent test set would probably show lower accuracy.","Editorial inference: if the transfer-learning recipe generalizes, the same frozen-features-plus-one-layer approach could be applied to other pupillary measurements, such as constriction latency, amplitude, or velocity, and to anisocoria screening, since all share the same pupil-localization front end.","Editorial inference: the headset isolates each eye and controls the light stimulus, so the validation covers this specific geometry; performance on video from off-axis or handheld cameras remains untested."],"forward_implications":["If replicated, a headset-based automated swinging flashlight test with RAPDNet can produce objective RAPD screening without relying on a practitioner's subjective judgment, which matters where ophthalmology specialists are scarce.","Because only a single fully connected layer is trained, adapting the pupil detector to a new headset or camera geometry may require modest new labeled data rather than retraining a full network.","The RAPD index is a continuous 0-to-1 score, not just a binary label, so the same framework could support repeated monitoring of a patient's pupillary asymmetry over time.","The benchmark suggests that choosing the detection threshold separately per algorithm is necessary; a deployment-ready system would need to fix that threshold before use on new patients."],"supporting_citations":[{"why":"supplies the large-scale object-recognition image database whose learned features are reused for pupil detection.","marker":"[29]"},{"why":"provides the AlexNet convolutional architecture whose pretrained layers are kept fixed in RAPDNet.","marker":"[30]"},{"why":"contributes the ElSe pupil datasets used to train and validate the pupil/no-pupil classifier, and the ElSe algorithm serves as a benchmark.","marker":"[16]"},{"why":"provides the ExCuSe handcrafted pupil-localization algorithm benchmarked against RAPDNet.","marker":"[15]"},{"why":"provides the Starburst pupil-localization algorithm benchmarked against RAPDNet.","marker":"[22]"},{"why":"the authors' earlier RAPD detection system on 32 cases, which the current benchmark extends to 64 cases.","marker":"[23]"}],"fun_headline_variants":["Transfer-learning pupil detector hits 90.6% in RAPD screening","Headset-based RAPDNet outperforms handcrafted pupil trackers","AI headset test for optic-nerve defect: AUC 0.929","RAPDNet: AlexNet transfer learning for automated pupil test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The clinical RAPD labels that define ground truth come from subjective swinging-flashlight and neutral-density-filter examinations by practitioners, with no reported inter-rater reliability or repeatability; if those labels are partly wrong, the measured sensitivity and specificity are not meaningful.","fun_headline_variants_meta":{"raw":{"variants":["Transfer-learning pupil detector hits 90.6% in RAPD screening","Headset-based RAPDNet outperforms handcrafted pupil trackers","AI headset test for optic-nerve defect: AUC 0.929","RAPDNet: AlexNet transfer learning for automated pupil test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3080,"prompt_tokens":954,"completion_tokens":2126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2044}},"tokens_in":570,"tokens_out":2126,"duration_ms":18177,"temperature":1.0,"reasoning_tokens":2044,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:47:36.970875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a new cohort in which each subject's RAPD status is established by masked agreement of at least two experienced examiners using standardized neutral-density-filter quantification, or by an independent automated pupillograph, and compare RAPDNet's thresholded scores against those labels; a large drop from 90.6% sensitivity/specificity, or substantial disagreement with the original clinical labels, would weaken the central claim.","supporting_citations":[{"cited_title":"Imagenet: A large-scale hierarchical image database,","cited_arxiv_id":null,"evidence_quote":"supplies the large-scale object-recognition image database whose learned features are reused for pupil detection."},{"cited_title":"Imagenet classification with deep convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"provides the AlexNet convolutional architecture whose pretrained layers are kept fixed in RAPDNet."},{"cited_title":"Else: Ellipse selection for robust pupil detection in real-world environments,","cited_arxiv_id":null,"evidence_quote":"contributes the ElSe pupil datasets used to train and validate the pupil/no-pupil classifier, and the ElSe algorithm serves as a benchmark."},{"cited_title":"Excuse: Robust pupil detection in real-world scenarios,","cited_arxiv_id":null,"evidence_quote":"provides the ExCuSe handcrafted pupil-localization algorithm benchmarked against RAPDNet."},{"cited_title":"Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches,","cited_arxiv_id":null,"evidence_quote":"provides the Starburst pupil-localization algorithm benchmarked against RAPDNet."},{"cited_title":"Automated pupillary light reflex assessment on a portable platform,","cited_arxiv_id":null,"evidence_quote":"the authors' earlier RAPD detection system on 32 cases, which the current benchmark extends to 64 cases."}],"review_version":1}