{"id":"cdbfd75e-efe9-44e3-9280-4d5404ae83cc","arxiv_id":"2606.03774","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AmbientEye is a new 2.6M-image dataset for pupil segmentation under ambient sunlight IR, showing existing algorithms drop from 0.928 to 0.767 IoU compared to controlled IR settings.","lead":"The paper introduces AmbientEye, a dataset of over 2.6 million outdoor eye images captured under natural sunlight with passive IR cameras. This could support development of power-efficient eye tracking for smart glasses usable all day outdoors.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"SAM2 + human refinement may not yield reliable ground truth for ambient IR pupil segmentation","rationale":"The reader's weakest_assumption matches the load-bearing point exactly: annotation quality is the prerequisite for trusting the performance numbers that support the central claim. No other internal inconsistency appears from the provided material.","tokens_in":1713,"tokens_out":302,"duration_ms":21375,"concrete_test":"Sample 200 images from AmbientEye; obtain independent pupil segmentations from three expert annotators (blind to SAM2 output); compute mean pairwise IoU and compare against the same protocol on a controlled-IR dataset. If AmbientEye agreement falls below 0.90 while controlled-IR stays above 0.95, the annotation reliability concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result is the performance drop from 0.928 (controlled IR) to 0.767 (AmbientEye), used to argue that ambient illumination poses a distinct challenge. This comparison is only meaningful if the AmbientEye labels constitute accurate, consistent ground truth. The annotation process relies on SAM2 (a general-purpose model) followed by unspecified human refinement; no inter-annotator agreement, expert validation, or domain-specific error analysis is referenced. In the presence of variable sunlight, reflections, and off-axis views, SAM2 errors may be systematic and incompletely corrected, making the measured gap partly an artifact of label noise rather than true algorithmic difficulty.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces AmbientEye, a dataset of 2,606,225 eye images captured outdoors from 35 participants across 19 countries under natural sunlight (two off-axis camera setups and two sun-orientation conditions). Pupil annotations are produced via SAM2 automatic segmentation followed by human refinement. A state-of-the-art pupil segmentation algorithm is benchmarked, yielding 0.767 performance on AmbientEye versus 0.928 on prior controlled-IR datasets; the gap is presented as evidence that ambient illumination constitutes a distinct practical challenge for eye tracking.","tokens_in":1849,"tokens_out":519,"duration_ms":19103,"significance":"If the ground-truth annotations are shown to be reliable and consistent, the work supplies the first large-scale benchmark for passive-IR pupil segmentation in unconstrained outdoor settings. This directly addresses power-consumption barriers for all-day smart-glasses eye tracking and supplies a falsifiable testbed for future algorithms under variable sunlight and reflections.","major_comments":[{"comment":"§3 (Dataset Creation / Annotation): The central empirical claim rests on the 0.767 figure being a valid measure of algorithmic difficulty rather than annotation noise. The description states only that annotations result from 'SAM2 automatic segmentation, followed by refinement by human annotators' with no inter-annotator agreement statistics, expert validation against ophthalmologists, or quantitative analysis of residual SAM2 errors under off-axis sunlight and corneal reflections. This omission directly affects whether the reported performance gap can be interpreted as domain difficulty.","section":"§3 (Dataset Creation / Annotation)"},{"comment":"§4 (Benchmarking and Results): The comparison to the 0.928 controlled-IR baseline is load-bearing for the 'distinct challenge' conclusion, yet no statistical test of the gap, participant-level variance, or exclusion-criteria justification is supplied. Without these, the headline drop cannot be assessed for robustness.","section":"§4 (Benchmarking and Results)"}],"minor_comments":[{"comment":"Abstract: The phrase 'high-quality pupil annotation' is used without supporting metrics; this should be qualified or moved to the methods section.","section":"Abstract"},{"comment":"Table/Figure captions: Ensure all reported metrics (e.g., IoU or Dice) are explicitly defined and that the exact evaluation protocol on prior datasets is stated for reproducibility.","section":"Tables/Figures"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive comments. We address each major comment below and indicate the revisions that will be incorporated into the next manuscript version.","responses":[{"response":"We agree that quantitative validation of the annotations is important for interpreting the performance numbers. In the revised manuscript we will add (i) inter-annotator agreement (mean Dice score) computed on a random subset of 10 000 images that were independently refined by two human annotators and (ii) a breakdown of the fraction of images in which the SAM2 initialization was substantially edited by humans, stratified by sun-orientation condition. We will not add ophthalmologist validation because the annotations concern geometric pupil boundaries in infrared imagery rather than clinical diagnosis; we will explicitly state this scope limitation. These additions directly address the concern about annotation noise versus domain difficulty.","revision_made":"partial","referee_comment":"[§3 (Dataset Creation / Annotation)] §3 (Dataset Creation / Annotation): The central empirical claim rests on the 0.767 figure being a valid measure of algorithmic difficulty rather than annotation noise. The description states only that annotations result from 'SAM2 automatic segmentation, followed by refinement by human annotators' with no inter-annotator agreement statistics, expert validation against ophthalmologists, or quantitative analysis of residual SAM2 errors under off-axis sunlight and corneal reflections. This omission directly affects whether the reported performance gap can be interpreted as domain difficulty."},{"response":"We concur that additional statistical reporting is needed. The revised manuscript will include (i) a Wilcoxon signed-rank test comparing per-image IoU on AmbientEye versus the controlled-IR datasets, (ii) participant-level mean IoU and standard deviation across the 35 subjects to quantify variance, and (iii) an explicit description of the exclusion criteria (images removed for severe motion blur, extreme head pose, or hardware failure) together with the number of frames excluded per condition. These changes will allow readers to evaluate the robustness of the reported performance gap.","revision_made":"yes","referee_comment":"[§4 (Benchmarking and Results)] §4 (Benchmarking and Results): The comparison to the 0.928 controlled-IR baseline is load-bearing for the 'distinct challenge' conclusion, yet no statistical test of the gap, participant-level variance, or exclusion-criteria justification is supplied. Without these, the headline drop cannot be assessed for robustness."}],"tokens_in":1422,"tokens_out":516,"duration_ms":29659,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper releases a large new dataset of over 2.6 million outdoor eye images under natural sunlight IR, captured with two camera setups and sun orientations from 35 participants across 19 countries. It benchmarks a pupil segmentation model and reports a drop from 0.928 on controlled IR data to 0.767 here, which flags a real domain shift for passive eye tracking in smart glasses.\n\nWhat stands out is the scale and the deliberate focus on unconstrained ambient conditions without active illumination. That matches a practical need for all-day outdoor use where power matters. The diversity in participants and the explicit variations in sun position and camera angle are clear strengths for a benchmark.\n\nThe soft spot is the ground truth. The paper describes SAM2 followed by human refinement but supplies no inter-annotator agreement numbers, no expert validation, and no breakdown of where the automatic step might fail systematically under reflections or variable lighting. Without those, the 0.767 figure could partly trace to label noise rather than pure algorithmic difficulty. The comparison to prior datasets also lacks detail on whether the same model and evaluation protocol were used exactly.\n\nThis is for researchers working on eye tracking for AR or ambient intelligence who need outdoor data. Anyone building or testing segmentation methods in passive IR would find the raw images and splits useful if the labels prove reliable.\n\nIt deserves peer review. The dataset itself addresses an underexplored setting, and referees can push for the missing validation metrics without rejecting the contribution outright.","headline":"AmbientEye is a solid dataset release for ambient IR pupil segmentation with real practical value, but the annotation quality lacks the checks needed to fully trust the reported performance drop.","tokens_in":2318,"tokens_out":385,"would_cite":false,"duration_ms":16274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Pupil segmentation accuracy drops from 0.928 to 0.767 when moving from controlled active IR to natural ambient sunlight alone.","keywords":["pupil segmentation","ambient infrared","eye tracking","outdoor dataset","passive illumination","smart glasses","natural sunlight","SAM2 annotation"],"falsifier":"Re-annotating a random subset of AmbientEye images with an entirely independent annotation protocol and then re-running the same segmentation algorithm to check whether the 0.767 score changes by more than a few points.","tokens_in":2635,"feed_emoji":"👁","tokens_out":693,"duration_ms":20372,"temperature":0.7,"pith_summary":"The paper tests whether passive infrared cameras without any active light source can support reliable pupil detection in real outdoor settings where sunlight is the only illumination. It introduces the AmbientEye dataset of 2,606,225 eye images from 35 participants across 19 countries, captured with off-axis cameras under two sun-orientation conditions. High-quality pupil labels are created by running SAM2 automatic segmentation and then refining the results with human annotators. Benchmarking a current leading segmentation algorithm on this data shows a clear performance decline compared with existing controlled-IR datasets. The gap demonstrates that the ambient-light case introduces practical difficulties for eye-tracking uses such as smart glasses.","feed_headline":"Pupil segmentation accuracy falls to 0.767 outdoors without active IR","feed_subtitle":"New dataset of 2.6 million images shows why active illumination remains essential for reliable eye tracking.","key_machinery":"The AmbientEye dataset of 2,606,225 eye images captured outdoors under passive natural sunlight with off-axis IR cameras and SAM2-plus-human annotations.","core_discovery":"AmbientEye supplies the first large-scale benchmark for pupil segmentation under natural ambient infrared illumination from sunlight alone. Images were gathered outdoors from a diverse participant pool using two camera configurations and two sun positions. Annotations combine SAM2 output with human refinement. When a state-of-the-art pupil segmentation algorithm is evaluated on AmbientEye, its score falls from 0.928 on prior controlled-IR collections to 0.767, establishing the dataset as a reference point for this unconstrained outdoor scenario.","pith_inferences":["Smart-glasses battery life could increase if methods close the performance gap, because active IR sources consume significant power.","Outdoor augmented-reality applications that rely on gaze may need hybrid active-plus-passive systems until ambient-only solutions mature.","The dataset's scale and participant diversity suggest it can also support training new models that generalize across skin tones and lighting angles."],"forward_implications":["Existing pupil segmentation methods developed for controlled active-IR settings do not transfer directly to outdoor ambient conditions.","Reliable pupil detection for all-day outdoor use will require algorithms explicitly designed for variable natural sunlight.","AmbientEye provides the first public reference for measuring progress on passive-IR eye tracking.","Power savings from removing active IR illuminators remain out of reach until segmentation robustness improves."],"fun_headline_variants":["Pupil segmentation falls to 0.767 on AmbientEye in natural sunlight","AmbientEye dataset shows pupil segmentation at 0.767 under sunlight","Accuracy drops from 0.928 to 0.767 for pupil segmentation outdoors","AmbientEye benchmarks pupil detection at 0.767 without active IR"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The combination of SAM2 automatic segmentation followed by human annotator refinement produces annotations of sufficient quality and consistency to serve as a reliable benchmark standard for the new ambient illumination domain.","fun_headline_variants_meta":{"raw":{"variants":["Pupil segmentation falls to 0.767 on AmbientEye in natural sunlight","AmbientEye dataset shows pupil segmentation at 0.767 under sunlight","Accuracy drops from 0.928 to 0.767 for pupil segmentation outdoors","AmbientEye benchmarks pupil detection at 0.767 without active IR"]},"model":"grok-4.3","cost_usd":0.005125,"raw_usage":{"total_tokens":2502,"prompt_tokens":689,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":51249500,"prompt_tokens_details":{"text_tokens":689,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1735,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":689,"tokens_out":78,"duration_ms":19741,"temperature":1.0,"reasoning_tokens":1735,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T10:51:11.348777+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-annotating a random subset of AmbientEye images with an entirely independent annotation protocol and then re-running the same segmentation algorithm to check whether the 0.767 score changes by more than a few points.","supporting_citations":[],"review_version":1}