{"id":"55a41eff-6368-450c-9f35-550c3fe1755b","arxiv_id":"2501.02800","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"COph100 is a new public benchmark of 491 infant retinal image pairs with manual ground truth, showing that existing registration algorithms often fail on pediatric fundus images.","lead":"The authors introduce COph100, a public dataset of 491 retinal image pairs from 100 infants with retinopathy of prematurity, with manual registration ground truth and vessel masks. It is built by selecting and annotating images from the existing RIDIRP database, and baseline tests show that current registration methods struggle on it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth control-point accuracy is unvalidated, and reported method differences are small enough that label noise could drive the benchmark conclusions.","rationale":"I agree with the reader that the manual ground-truth point correspondences are the single most load-bearing premise. Unlike the acceptable-subset reporting issue in Table 4, which mainly affects fairness of cross-method comparison, ground-truth accuracy affects every quantitative conclusion in the paper. The central claim is that COph100 is a challenging, first-of-its-kind infant registration benchmark; that challenge is demonstrated by measured registration performance, and all measurements are computed against the unvalidated manual points. The paper itself flags difficult image conditions, but no evidence is provided that markers placed around vessel intersections in blurred or obstructed images are accurate enough to support the reported RMSE and mAUC values. The reader's CONDITIONAL verdict already captures this concern, and my stress-test does not identify a new issue that would move the verdict further. The recommended action remains to require independent label validation before the dataset is adopted as a standard benchmark. The vessel segmentation masks and image-quality statistics are supporting contributions but do not substitute for ground-truth validation; the Table 4 acceptable-subset reporting is a secondary weakness, not the most load-bearing one.","tokens_in":10661,"tokens_out":4051,"duration_ms":44800,"concrete_test":"Select a stratified random subset of roughly 50 image pairs from COph100 covering all blur/obstruction levels, and have two ophthalmologists independently re-annotate 10 control-point correspondences per pair following the paper's protocol. Compute per-pair median inter-observer point distance and repeatability, then recompute Table 4's mAUC and acceptable-RMSE using each annotator's labels. If the inter-observer error is below about 1 pixel and method rankings and mAUC differences remain unchanged, the concern is resolved. If label differences reach or exceed the reported method gaps (around 4-5 pixels RMSE), the benchmark conclusions need to be revised, and the dataset should report confidence intervals or a consensus-labeling protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that COph100 is a useful, challenging infant retinal registration benchmark rests on the manual control-point correspondences described in the 'Registration Groundtruth' section. Ten point pairs per image pair are marked around vessel intersections, yet no inter-observer variability, repeat-labeling, or independent verification is reported. This is load-bearing because every metric in Table 4 (mAUC, RMSE, failed/inaccurate rates) is computed against these points, and the dataset is deliberately hard: 213/325 images are obstructed and 100/325 are blurred (Figures 5-6). At 640x480, the best acceptable RMSE values are around 4.8 pixels, and the differences among top methods are sub-pixel to a few pixels; human click error on blurred or obstructed vessel intersections can plausibly be of that magnitude. The author contributions list multiple people as doing 'data labelling, GT and image quality check', but no quantitative agreement measure is given. The source ROP dataset is cited, but the paper does not describe how points were marked across sessions with large appearance changes or whether one annotator's labels were independently checked. Without a label-error estimate, the quantitative benchmark is not independently anchored: the 'challenging' conclusion is only as strong as unmeasured label accuracy, and the central claim is not yet supported at the precision claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COph100, a retinal image registration dataset built from 100 infant eyes (491 image pairs) selected from the public Timkovic ROP dataset. For each image pair the authors provide 10 manually marked control-point correspondences, automatic vessel segmentation masks, and vessel overlay images. They evaluate 12 existing traditional and deep-learning registration methods on the dataset, reporting failure/inaccurate rates, RMSE, and mAUC, and show that current methods perform substantially worse than on the adult FIRE dataset. The paper claims this is the first retinal registration dataset specifically focused on disease progression in infants.","tokens_in":10912,"tokens_out":7350,"duration_ms":67961,"significance":"If the annotations are reliable, COph100 is a valuable new public benchmark: it is larger than most existing retinal registration datasets (491 pairs, 100 eyes), targets a pediatric population, and contains realistic image-quality challenges such as blur, obstruction, and illumination change. The public release of images, ground-truth points, vessel masks, and evaluation code supports reproducible benchmarking. The evaluation is largely non-circular because it uses pre-trained models without fine-tuning to the target dataset, avoiding parameter fitting on the benchmark itself.","major_comments":[{"comment":"The reliability of the 10 manual control-point pairs per image pair is not assessed. No inter-observer variability, repeated labeling, or independent verification is reported, even though the author contributions list several people involved in labelling and ground-truth generation. Because every quantitative metric in Table 4 (mAUC, RMSE, failed/inaccurate rates) is computed against these points, unknown label noise could be comparable to the sub-pixel to few-pixel differences among top methods (for example, SuperGlue RMSE 5.079 vs SuperPoint RMSE 5.075). Please add a reproducibility study, such as having two or more annotators re-mark a random subset of pairs and reporting the mean and standard deviation of point distances, and ideally analyze how metric rankings change under realistic label noise.","section":"Registration Groundtruth"},{"comment":"The RMSE metric is computed only on the subset of registrations classified as 'acceptable' (MEE < 20 and MAE < 50), so different methods are compared on different subsets of image pairs. For example, LK-SuperRetina has 40.53% failed registrations without segmentation and an RMSE of 4.841, lower than SuperGlue's 5.079, but this likely reflects an easier subset rather than superior accuracy. This biases cross-method comparisons. Please report RMSE on a common subset of pairs (for example, pairs that all methods register acceptably) or use an error measure that accounts for all pairs, and clarify how failed registrations are incorporated into the mAUC.","section":"Registration evaluation, Table 4"},{"comment":"The discussion of segmentation-based results is internally inconsistent. The text states that 'methods trained in natural image environments have demonstrated improved accuracy with the addition of segmentation, except for SuperGlue', but Table 4 shows that SuperJunction's failure rate jumps from 0% to 100% with segmentation, and that SuperRetina, Swin U-SuperRetina, and LK-SuperRetina also degrade. Please correct the narrative to match the table and provide a concrete explanation for why segmentation information can be harmful for these methods, particularly the complete failure of SuperJunction.","section":"Registration evaluation, Table 4 and surrounding text"},{"comment":"The annotation protocol is underspecified. It is unclear whether the 10 control points are the same set tracked across all examinations of an eye, as Figure 2 suggests for the 9-examination case, or whether they are selected independently for each image pair. The paper also does not state how points in obstructed or blurred regions, which affect 213/325 and 100/325 images respectively, are handled. Without this description, users cannot assess the consistency and validity of the ground-truth correspondences.","section":"Registration Groundtruth and Figure 2"}],"minor_comments":[{"comment":"The phrase 'Minimal apprarance variablity' contains typos and should read 'Minimal appearance variability'.","section":"Introduction"},{"comment":"Please correct 'opthalmologists' to 'ophthalmologists' and 'Root Square Error' to 'Root Mean Square Error (RMSE)'.","section":"Throughout"},{"comment":"The publish time for COph100 is listed as 2024, but the paper is dated 2025; please update for consistency.","section":"Table 1"},{"comment":"The acronym RIDIRP is used but never defined; please spell out the name and explain its relationship to the Timkovic et al. dataset.","section":"Title and Abstract"},{"comment":"The GitHub link is truncated in the printed text; please ensure the full URL is provided.","section":"Data Records"},{"comment":"The claim that COph100 includes a 'diverse patient population' is not supported by any demographic data about the 100 eyes; please either provide such data or temper the claim.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself appears to be a real and useful contribution, and the pair count is consistent with the examination distribution in Table 3. The main obstacle is the unvalidated manual ground truth, which the authors can address with a modest inter-observer study. The RMSE-on-acceptable-subset issue is also important for the benchmark's validity and should be fixed in the revision. If these points are addressed, the paper would likely be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely useful dataset paper, not a methods paper. The new contribution is COph100, a curated subset of 325 low-res infant ROP fundus images from the public RIDIRP dataset, with 491 registration pairs across 100 eyes, manual control points, and vessel masks. That fills a real gap: existing retinal registration benchmarks are small, adult, and high-quality. The authors are honest that they build on Timkovic et al. and that they introduce no new code; the value is in the curation and the baseline evaluation.\n\nThe reader's conditional verdict is fair, and the stress-test note lands. The manual ground truth is load-bearing, and the paper reports no inter-observer or repeat-labeling experiment. Ten points per pair around vessel intersections on images that are often blurred or obstructed (213/325 obstructed, 100/325 blurred) is exactly the situation where human click error can reach a few pixels, and the top RMSEs in Table 4 differ by fractions of a pixel (4.84 vs 5.08). Without a label-error estimate, the precise ranking of methods is not trustworthy. That said, the broader conclusion that these images are hard for existing methods does not depend on small differences: best mAUC is 0.809, and many methods fail or are inaccurate on a large fraction of pairs. So the central claim survives, but the quantitative benchmark needs a health warning.\n\nA secondary issue: RMSE is computed only on the 'acceptable' registrations. The paper discloses this, but it means cross-method RMSE comparisons are conditional on different subsets, which weakens direct comparison. The image-quality labels (blur, obstruction) are also subjective and unvalidated, though they are presented as descriptive statistics rather than as a benchmark target. The vessel masks come from a model trained and evaluated on FIVES, not on infant images; the authors only say an ophthalmologist judged them sufficient for disease progression analysis. That is a minor concern for a registration benchmark, but worth flagging.\n\nWhat is solid: the dataset is public, the pair counts track Table 3, the inclusion/exclusion criteria are clearly described, and the baseline set is broad (traditional, deep, natural-image, fundus-specific). The paper is well organized and readable. The 'first pediatric registration dataset' claim is reasonable, and the scale (491 pairs, 100 eyes) beats the existing small adult datasets.\n\nWho is this for? Anyone working on retinal registration, especially longitudinal or pediatric applications. It deserves a serious referee, not a desk reject. The referee should ask for inter-observer agreement on a subset of control points, ideally with a repeat-labeling study, and a discussion of how label noise affects the reported differences. Even with those caveats, this will be a useful community resource.\n\nRecommendation: engage with it, but treat the ground truth as provisional until validated.","headline":"A useful pediatric retinal registration benchmark whose manual ground truth needs an error bar before the numbers can be taken at face value.","tokens_in":11447,"tokens_out":2076,"would_cite":true,"duration_ms":20856,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COph100, the first infant-focused retinal registration dataset, shows current algorithms still misalign key vessels in retinopathy-of-prematurity follow-up exams.","keywords":["retinal image registration","fundus image dataset","retinopathy of prematurity","infant ophthalmology","disease progression","vessel segmentation","registration benchmark","COph100"],"falsifier":"Have two independent graders annotate the 10 corresponding points on a random sample of 50 image pairs and measure the mean distance between graders; if that distance is comparable to or larger than the roughly 4.8-pixel acceptable root-mean-square error, the benchmark's accuracy rankings become unreliable.","tokens_in":10462,"feed_emoji":"👁️","tokens_out":7521,"duration_ms":67786,"temperature":0.7,"pith_summary":"This paper introduces COph100, a public dataset for retinal image registration in infants, built from follow-up examinations of eyes with retinopathy of prematurity (ROP), a disease of premature infants' retinas. The central claim is that COph100 is the first registration dataset specifically aimed at disease progression in infants, providing 100 eyes with 491 image pairs, each pair annotated with 10 manual corresponding points and automatic vessel segmentation masks. The authors benchmark twelve classical and learned registration algorithms and find that the best method reaches a mean area under the curve of 0.809, while acceptable registrations still show about 4.8 pixels of root-mean-square error at 640 by 480 pixels. The dataset's purpose is to make the difficulty of infant fundus registration, caused by blur, obstruction, and illumination shifts, directly measurable and to support algorithms that track ROP over time.","feed_headline":"First infant retina registration dataset stumps current AI","feed_subtitle":"100 infant eyes, 491 follow-up image pairs; best current model still misses key vessels by ~4.8 pixels.","key_machinery":"The central object is the dataset itself, COph100, which functions as a controlled benchmark by pairing each query image with a later-examination reference image and supplying 10 manual control-point correspondences as ground truth. The evaluation machinery follows the established fundus registration protocol, classifying registrations as failed, inaccurate, or acceptable using median and maximum point-error thresholds, and then summarizing performance with RMSE and area under the curve. The vessel segmentation masks serve a second role, as an alternate input modality to test whether registration methods improve when vascular structure is the only guide. This combination of manual points, masks, and a fixed protocol is what lets the authors turn a collection of messy clinical images into a measurable registration challenge.","core_discovery":"On the paper's own terms, the contribution is a new resource: COph100, a subset of 325 images drawn from a public ROP dataset of 6,004 images, organized into 491 pairs from 100 eyes with 2 to 9 examination sessions each. For every pair, the authors provide 10 manually placed control points at vessel intersections and a vessel segmentation mask produced by a model trained on a public fundus vessel dataset. They benchmark twelve registration methods, including general-purpose feature matchers and retina-specific ones, both with and without segmentation input. The headline numbers are that SuperGlue achieves the best mean area under the curve (mAUC) of 0.809 and the best acceptable registrations have a root-mean-square error near 4.8 pixels, while the traditional method GDB-ICP improves to a 2.24 percent inaccurate rate when given vessel masks; several retina-trained methods fail or degrade on this data. The paper concludes that current registration algorithms do not yet align infant ROP images reliably, and that COph100 provides a challenging benchmark for closing that gap.","pith_inferences":["Beyond the paper's claims, the same selection pipeline could be applied to the higher-resolution sections of the source dataset, letting researchers test whether current algorithms improve with more image detail.","The absence of repeated or multi-observer ground-truth labeling suggests that adding an inter-observer variability study would sharpen the reliability of the benchmark.","Because the source dataset also contains lesion segmentations, a natural extension is to use COph100 pairs for joint registration and disease-progression measurement rather than registration alone.","The finding that segmentation masks hurt some retina-trained models but help natural-image-trained models hints that domain shift, not segmentation per se, drives much of the performance gap."],"forward_implications":["If COph100 becomes a standard benchmark, registration algorithms will have to handle the blur, obstruction, illumination shifts, and limited overlap that are typical of infant examinations.","The longitudinal structure of 2 to 9 sessions per eye makes it possible to evaluate registration as a tool for tracking lesion and vessel changes over time.","Because vessel masks are provided, researchers can test whether feeding segmentation maps to feature matchers improves accuracy, which the paper reports for most methods.","The gap between the best current result and perfect vessel alignment quantifies the headroom for future registration work on pediatric images."],"supporting_citations":[{"why":"Supplies all source fundus images and the 2 to 9 examination structure from which COph100 is selected.","marker":"[1]"},{"why":"Provides the reference fundus registration benchmark and its evaluation style that COph100 compares against.","marker":"[18]"},{"why":"Gives the architecture for the automatic vessel segmentation masks included in the dataset.","marker":"[21]"},{"why":"Supplies the training data for the vessel segmentation model that produced the masks.","marker":"[23]"},{"why":"Is the Figshare record where the COph100 images, point pairs, and masks are released.","marker":"[24]"},{"why":"Defines the failed/inaccurate/acceptable registration classification used to score the benchmark.","marker":"[28]"},{"why":"Represents the best deep-learning baseline; its 0.809 mAUC states the remaining challenge.","marker":"[34]"}],"fun_headline_variants":["Infant retina dataset exposes AI registration limits","Best model off by 4.8 px on infant retinal pairs","100 infant eyes, 491 pairs: new retinal registration benchmark","New infant retinal dataset challenges current registration algorithms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manual 10-point ground-truth correspondences are accurate enough to serve as the evaluation target, yet the paper reports no inter-observer variability, no repeat labeling, and no independent verification of those points.","fun_headline_variants_meta":{"raw":{"variants":["Infant retina dataset exposes AI registration limits","Best model off by 4.8 px on infant retinal pairs","100 infant eyes, 491 pairs: new retinal registration benchmark","New infant retinal dataset challenges current registration algorithms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3932,"prompt_tokens":943,"completion_tokens":2989,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2925}},"tokens_in":559,"tokens_out":2989,"duration_ms":21565,"temperature":1.0,"reasoning_tokens":2925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:04.338179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent graders annotate the 10 corresponding points on a random sample of 50 image pairs and measure the mean distance between graders; if that distance is comparable to or larger than the roughly 4.8-pixel acceptable root-mean-square error, the benchmark's accuracy rankings become unreliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies all source fundus images and the 2 to 9 examination structure from which COph100 is selected."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the reference fundus registration benchmark and its evaluation style that COph100 compares against."},{"cited_title":"Coph100: A comprehensive fundus image registration dataset from infants constituting the \"ridirp\" database","cited_arxiv_id":null,"evidence_quote":"Is the Figshare record where the COph100 images, point pairs, and masks are released."}],"review_version":1}