{"id":"04c99329-0078-43e1-99b2-7202bdfc93d1","arxiv_id":"2504.18249","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Four challenge teams achieved pixel errors between 1.14 and 1.61 on event-based pupil-center prediction, and the survey documents their architectures and post-processing tricks.","lead":"This paper summarizes the 2025 CVPR event-based eye tracking challenge, where four teams reached under 1.7 pixel error in predicting pupil centers from event camera data. It is a survey of winning methods and hardware considerations, useful as a benchmark snapshot for low-power eye tracking in AR/VR and healthcare.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Year-over-year improvement claim is not supported because the 2025 pixel-error metric and 100 Hz labels are not compared against a 2024 baseline under the same protocol.","rationale":"The reader's weakest assumption correctly identifies the metric change as the core issue: the paper claims improvement over last year while simultaneously switching from p-accuracy at 20 Hz to pixel error at 100 Hz, with no like-for-like baseline. My independent reading of Sections 2.2, 2.5, and the team result tables confirms that no 2024 model is re-evaluated under the 2025 protocol, and no variance or label-noise information is provided. The absolute leaderboard scores are still credible as a snapshot of the 2025 challenge, so the paper remains useful; however, the headline improvement claim is not supported by the evidence presented. The reviewer's CONDITIONAL verdict is appropriate, and my concern aligns with theirs. I found no additional load-bearing issue that would justify a stronger rejection. The proposed test directly targets the missing baseline and would settle the question empirically.","tokens_in":13652,"tokens_out":6377,"duration_ms":60448,"concrete_test":"Obtain the 2024 challenge winner's model and code from the AIS 2024 survey [47], apply it to the 2025 3ET+ test recordings, and evaluate it with the 2025 evaluation script at 100 Hz labels (if the model outputs at 20 Hz, upsample its predictions by nearest-neighbor or interpolation to match label timestamps). Compare its pixel error to the 1.7 threshold. Additionally, compute p@10 accuracy for the four 2025 teams from their per-sample errors; if the 2024 model's pixel error is below 1.7, or if any 2025 team's p@10 is below the 2024 best 99.37%, the claimed year-over-year improvement is not established. This isolates the metric/frequency change from genuine algorithmic progress.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5 states that both the label frequency (20 Hz to 100 Hz) and the evaluation metric (p-accuracy to pixel error) changed, then asserts that 'four participating teams achieved a pixel error below 1.7, outperforming the best results from the previous year.' No evidence is provided that the previous year's best result, reported as p-accuracy (e.g., p@10 = 99.37% in CherryChums' Table 7), would correspond to a pixel error above 1.7 under the 2025 protocol. p-accuracy and mean pixel error are not interchangeable: a high p@10 score is consistent with a wide range of mean errors, and the 5x increase in label frequency changes task difficulty and per-label displacement. The paper reports no baseline re-evaluation of 2024 models on the 2025 benchmark, no error bars or variance on the 1.14-1.61 pixel errors, and no estimate of manual label noise. If ground-truth annotation noise is on the order of 1-2 pixels, the differences among top teams (1.14, 1.42, 1.50, 1.61) may be within that noise. Therefore the central 'notable improvement' claim is unsupported as stated, although the absolute leaderboard numbers remain useful descriptive facts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a workshop survey of the 2025 Event-Based Eye Tracking Challenge held at CVPR 2025. It introduces the 3ET+ dataset and the pupil-center regression task, describes the evaluation metric (pixel error) and challenge organization, summarizes the top four team solutions (BRAT, EyeTracking@SMU post-processing, TDTracker, and CherryChums data augmentation), and closes with a hardware-oriented discussion. The central claim is that four teams achieved pixel errors below 1.7, outperforming the best results from the previous year's challenge.","tokens_in":13896,"tokens_out":4683,"duration_ms":43242,"significance":"If taken as reported, the challenge results are a useful descriptive record: compact models (0.8–7.1M parameters) reportedly achieve pixel errors of 1.14–1.61 at 100 Hz on 3ET+. The survey has concrete strengths: it documents the challenge protocol, reports parameter counts and MACs, and mandates source-code submission for reproducibility. However, the headline claim of year-over-year improvement is not supported because the 2025 and 2024 evaluation protocols are not comparable. The survey therefore is a useful archive of the 2025 competition but does not, as written, establish that event-based eye tracking has advanced beyond the previous edition.","major_comments":[{"comment":"The claim that 'four participating teams achieved a pixel error below 1.7, outperforming the best results from the previous year' is not supported by the evidence presented. The manuscript itself states that both the label frequency (20 Hz to 100 Hz) and the evaluation metric (p-accuracy to pixel error) changed. A high p@10 score from 2024 (e.g., 99.37% for CherryChums in Table 7) is not equivalent to a pixel error below or above 1.7, and the 5x increase in label frequency changes the task. No 2024 model was re-evaluated under the 2025 protocol, so the improvement claim is not a like-for-like comparison. This claim is load-bearing for the paper's narrative and should be either removed or supported by a re-evaluation of 2024 baselines under the 2025 metric.","section":"Section 2.5"},{"comment":"The top-team pixel errors (1.14, 1.42, 1.50, 1.61) are reported as single point values without error bars, standard deviations, or multiple runs. The differences between adjacent ranks are less than half a pixel, which may be within the noise of manual pupil-center annotation or run-to-run variation. Without variance estimates, the ranking itself and the 'notable improvement' claim are not statistically grounded. The survey should report repeated evaluations or at least discuss the expected annotation noise level.","section":"Table 1 and Section 3"}],"minor_comments":[{"comment":"The update for the y-coordinate uses dx in the numerator: 'y_(R,f,pred) <- y_(f,pred) + dx / ||dx,dy||'. This appears to be a typographical error; it should use dy. As written, the pseudocode does not implement the described optical-flow refinement.","section":"Algorithm 2, line 26"},{"comment":"The loss formula is written as (1/T) * sqrt( sum_{t=1}^T (pred - label)^2 ). This is not the standard RMSE, which would be sqrt( (1/T) * sum ). The current expression scales the error down by an additional factor of sqrt(T), which may unintentionally alter the training objective. Please clarify.","section":"Section 5.1, Eq. (4)"},{"comment":"It is not stated whether the 'pixel error' column in Table 1 refers to the private or public leaderboard scores. Teams report both in their own tables; the survey should specify which value is used for the final ranking.","section":"Table 1 and Tables 4, 6"},{"comment":"The sentence 'After post-processing, the MSE is optimized to 1.4936 on the interpolation ground truth from 3ET+ 2024' is confusing, as the paper elsewhere uses pixel error, and the relationship between this value and the private/public scores is not explained.","section":"Section 5.3, Results"},{"comment":"The phrase 'methods from teams rank the top' is ungrammatical; it should be 'methods from the top-ranked teams.' Also, 'In each method, accuracy, model size, and number of operations are reported' is awkward and could be rephrased.","section":"Abstract"},{"comment":"The statement 'There were, in total, 22 user accounts registered and participated the challenge' is ambiguous: does it mean 22 unique teams or 22 accounts? It should be clarified.","section":"Section 3.2"},{"comment":"The hardware discussion is general and not tightly connected to the challenge results presented earlier; consider linking it to the specific efficiency figures reported by the teams.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2504.18249. This is the 2025 event-based eye-tracking challenge survey, not an original research paper. What's actually new is the leaderboard snapshot: four teams with pixel errors 1.14–1.61 on 3ET+ at 100 Hz, with model sizes 0.8–7.1M parameters. If those numbers hold, they are a useful community data point for low-power gaze tracking. The individual methods are reported more fully in the teams' own papers, so don't read this for algorithmic depth.\n\nWhat the paper does well: it documents the challenge protocol, justifies the metric change (p-accuracy saturated), requires code submission for reproducibility, and reports parameter counts and, for some teams, MACs/FLOPs. The absolute leaderboard numbers are the main takeaway and they look plausible. I don't see a circularity problem: this is a report on an externally run Kaggle competition, even if the organizers are also dataset authors.\n\nThe soft spot is exactly where the reader's stress test lands. Section 2.5 says the 2025 results 'outperform' last year's best, but both the label frequency (20→100 Hz) and the metric (p-accuracy→pixel error) changed. No 2024 model was re-evaluated on the 2025 protocol, so the improvement claim is not supported as stated. The absolute numbers stand on their own; the year-over-year sentence should be softened or backed by a rerun. Also, no error bars or variance are reported anywhere. If annotation noise is in the 1–2 pixel range, the 1.14 vs 1.61 spread among top teams could be partly noise. That doesn't invalidate the snapshot, but it should be noted. Finally, Algorithm 2 contains an apparent typo: line 26 updates the y coordinate using dx in the numerator. That matters because the whole point of that algorithm is optical-flow-based refinement.\n\nThe hardware discussion is qualitative and mostly generic, but it's short and doesn't pretend to be more than it is. The citation pattern is fine; the self-citations are to dataset and baseline papers, which is normal for this kind of survey.\n\nVerdict: the paper is a legitimate workshop survey and the leaderboard table has real value. I'd send it to review, not desk reject, with requests to add a like-for-like baseline comparison (or remove the comparison claim), report variance, and fix the algorithm typo. For a workshop proceedings, light revision suffices.","headline":"The 2025 leaderboard numbers are a useful descriptive snapshot, but the year-over-year improvement claim is not supported because the metric and label frequency changed; the paper is worth publishing with modest revisions.","tokens_in":14567,"tokens_out":2337,"would_cite":true,"duration_ms":22177,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Event-based pupil tracking reaches 1.14-pixel error on a public benchmark, the paper reports.","keywords":["event-based vision","eye tracking","pupil center localization","dynamic vision sensor","challenge survey","pixel error","temporal modeling","augmented reality"],"falsifier":"Rerun this year's top models on the 2024 evaluation protocol (p-accuracy at a 10-pixel threshold with 20 Hz labels) and compute their p-10 scores; if they do not exceed last year's best p-10 on the same protocol, the claimed improvement is an artifact of the metric and label-rate change.","tokens_in":13448,"feed_emoji":"👁️","tokens_out":4930,"duration_ms":47722,"temperature":0.7,"pith_summary":"This paper is the survey of the 2025 event-based eye-tracking challenge, in which teams predicted pupil-center coordinates from event-camera streams on the 3ET+ benchmark. Its central claim is that four compact models, ranging from 0.8 to 7.1 million parameters, all achieve a pixel error below 1.7, with the best at 1.14 pixels, at labels sampled at 100 Hz. The paper argues this improves on the previous year's best result, and it attributes the gains to short- and long-term temporal modeling, data augmentation, and model-agnostic post-processing. A sympathetic reader would care because accurate, low-cost pupil tracking at high temporal resolution is a building block for eye-based interaction in augmented and virtual reality and for non-invasive health monitoring.","feed_headline":"Pupil tracking from event cameras hits 1.14-pixel error","feed_subtitle":"Four compact models stay under 1.7 pixels at 100 Hz, a step toward low-power AR/VR eye tracking.","key_machinery":"The central objects are the event stream as spatiotemporal tuples $(x, y, t, p)$, the binned event-frame representation, the pixel-error metric defined as Euclidean distance between predicted and ground-truth pupil coordinates, and a set of network designs that combine spatial encoders with temporal sequence models. The winning architecture uses a bidirectional relative-positional attention bias that linearly down-weights attention with temporal distance, separately for forward and backward directions. The post-processing mechanism computes local motion variance to set an adaptive median-filter window and aggregates event flow in a region of interest to shift predictions. These components together carry the argument that sparse event data plus explicit temporal modeling can reach sub-1.7-pixel accuracy at 100 Hz.","core_discovery":"The survey reports that event-based pupil tracking has crossed a practical accuracy threshold: four independent methods localize the pupil center with 1.14 to 1.61 pixel error at 100 Hz on the 3ET+ benchmark while keeping model sizes between 0.8 and 7.1 million parameters. The winning approach combines a convolutional spatial encoder with a bidirectional gated recurrent unit and a self-attention module whose relative position bias is split into forward and backward temporal components. Other top solutions use a 3D CNN cascaded with GRU and Mamba modules to capture implicit short-term and explicit long-term dynamics, or apply inference-time post-processing—motion-aware median filtering and optical-flow-based local refinement—to an existing lightweight spatiotemporal network. The paper further reports that data augmentation such as temporal shifting, spatial flipping, and random event deletion improves accuracy from 1.70 to 1.61 pixel error on the same network, and it frames these results as evidence that event-based eye tracking is viable for low-power, high-speed wearable systems.","pith_inferences":["Editorial inference: the year-over-year comparison is not like-for-like because both the label rate (20 Hz to 100 Hz) and the metric (p-accuracy to pixel error) changed; the real gain may be in temporal resolution rather than in spatial accuracy.","Editorial inference: reporting only point estimates of pixel error without variance, confidence intervals, or per-subject breakdowns makes differences among the four top teams (1.14 vs 1.61) hard to interpret; a repeated-seed evaluation could establish whether the ordering is stable.","Editorial inference: a natural follow-up is an end-to-end event-driven implementation, since several components (attention, GRU, Mamba) are sequential and the paper's own hardware section names latency, sparsity, and burst handling as the open constraints.","Editorial inference: the augmentation recipe (temporal shift, spatial flip, event deletion) is tested on only one base network; applying it across the other top architectures would tell whether the gain is general or architecture-specific."],"forward_implications":["At 1.14 to 1.61 pixel error, event-based trackers are within the range needed for gaze-based interaction in head-mounted displays, where sub-2-pixel errors are considered usable.","Model-agnostic post-processing that adds only about 512 floating-point operations per prediction can improve any base tracker without retraining, which is a practical recipe for edge deployment.","Sub-million-parameter models reach 1.42 to 1.61 pixel error, suggesting that event-based eye tracking is feasible for the power and memory budgets of wearable devices.","The shift from p-accuracy to raw pixel error exposes headroom that near-perfect p-10 scores hid, making further algorithmic improvement visible and measurable.","Explicitly modeling short-term motion (with recurrent or 3D convolutional layers) and long-term dependencies (with attention or state-space models) is a workable template for event-sequence regression."],"supporting_citations":[{"why":"Defines the previous challenge edition and the p-accuracy metric at 20 Hz labels that this year's results are compared against.","marker":"[47]"},{"why":"Introduces the 3ET dataset and the CNN-GRU baseline used for pretraining and as a base model for post-processing tests.","marker":"[10]"},{"why":"Provides the lightweight spatiotemporal network that two top teams adopted as their base architecture.","marker":"[37]"},{"why":"Describes the model-agnostic inference-time post-processing with motion-aware median filtering and optical-flow refinement used by the runner-up team.","marker":"[5]"},{"why":"Documents the event camera used to capture the 3ET+ benchmark recordings.","marker":"[2]"}],"fun_headline_variants":["Event eye tracking hits 1.14-pixel pupil error","Pupil tracking from events: 1.14-pixel accuracy at 100 Hz","Event cameras track pupils at 1.14-pixel error","1.14-pixel pupil error: event eye tracking wins","Top event eye trackers: 1.14-pixel accuracy, tiny models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 'better than last year' assumes that this year's pixel-error scores at 100 Hz labels are directly comparable with last year's p-accuracy scores at 20 Hz labels, even though the survey states both changed.","fun_headline_variants_meta":{"raw":{"variants":["Event eye tracking hits 1.14-pixel pupil error","Pupil tracking from events: 1.14-pixel accuracy at 100 Hz","Event cameras track pupils at 1.14-pixel error","1.14-pixel pupil error: event eye tracking wins","Top event eye trackers: 1.14-pixel accuracy, tiny models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000109,"raw_usage":{"total_tokens":996,"prompt_tokens":839,"completion_tokens":157,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":61}},"tokens_in":455,"tokens_out":157,"duration_ms":2227,"temperature":1.0,"reasoning_tokens":61,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:20:39.989844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun this year's top models on the 2024 evaluation protocol (p-accuracy at a 10-pixel threshold with 20 Hz labels) and compute their p-10 scores; if they do not exceed last year's best p-10 on the same protocol, the claimed improvement is an artifact of the metric and label-rate change.","supporting_citations":[{"cited_title":"Conde, Radu Timofte, Shih-Chii Liu, Qinyu Chen, et al","cited_arxiv_id":null,"evidence_quote":"Defines the previous challenge edition and the p-accuracy metric at 20 Hz labels that this year's results are compared against."},{"cited_title":"3et: Efficient event-based eye tracking using a change-based convlstm network","cited_arxiv_id":null,"evidence_quote":"Introduces the 3ET dataset and the CNN-GRU baseline used for pretraining and as a base model for post-processing tests."},{"cited_title":"A Lightweight Spatiotem- poral Network for Online Eye Tracking with Event Camera","cited_arxiv_id":null,"evidence_quote":"Provides the lightweight spatiotemporal network that two top teams adopted as their base architecture."},{"cited_title":"Model-agnostic inference-time post-processing and local re- finement for enhanced event-based eye tracking","cited_arxiv_id":null,"evidence_quote":"Describes the model-agnostic inference-time post-processing with motion-aware median filtering and optical-flow refinement used by the runner-up team."},{"cited_title":"https://inivation","cited_arxiv_id":null,"evidence_quote":"Documents the event camera used to capture the 3ET+ benchmark recordings."}],"review_version":1}