REVIEW 3 major objections 6 minor 27 references
High-frequency near-eye ground truth for event-based eye tracking
T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims a semi-automatic pipeline can add 200Hz pupil-center annotations to the Angelopoulos event-based eye-tracking dataset, along with blink and saccade labels.
desk verdict The paper's own Table 1 sinks the headline claim: only 7% of frames carry pupil-center labels, so this is not continuous 200 Hz pupil ground truth, but the saccade/blink annotations may still be a useful artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pipeline has three automatic stages followed by a human validation stage. First, events are accumulated into RGB frames at 200Hz (5ms windows). Second, an eye-movement detector flags a saccade when a frame contains more than 150 events, marking saccade start and end. Third, during saccades, the pupil center is found by convolving the polarity frame with 8 directional templates, picking the highest-scoring template, and then running RANSAC with 1000 iterations to fit an ellipse to events inside the resulting region of interest. Human annotators then correct any mislabels, and an iterative quality-check step plots inter-frame pupil displacement to catch anomalies.
What would settle it
Re-annotate a random sample of frames from each of the six users with multiple independent human annotators and compute inter-annotator agreement, or compare the released labels against a high-speed camera-based ground truth; if the agreement is poor or the labels deviate systematically, the claim that these annotations constitute valid ground truth at 200Hz would be refuted.
Extended reading notes
Core claim
On the paper's terms, the central claim is that a high-frequency (200Hz) pupil-center annotation, together with blink and saccade labels, can be produced for the Angelopoulos event-based eye-tracking dataset using a semi-automatic pipeline that combines automatic detection with human correction. The paper reports that 114,222 frames were reviewed and annotated, yielding 22,305 distinct eye center positions, 2,101 saccades, and 120 blinks across six users. This density of labels, the authors argue, is sufficient to capture all documented eye movements, including microsaccades, and improves on the original dataset's screen-space gaze labels.
Load-bearing premise
The released labels are only as good as the human annotators' corrections, but the paper provides no measure of agreement between annotators or against an external gold standard to confirm those corrections are accurate.
Editorial extensions
If this is right
- Researchers can train and evaluate event-based pupil detectors against a 200Hz supervision signal rather than the original screen-space gaze labels.
- The released blink and saccade annotations enable supervised learning for eye-state classification, not just pupil localization.
- The pipeline itself can be reapplied to other event-based eye datasets or future recordings to produce dense labels.
- Because 200Hz exceeds the documented duration of microsaccades (10-30ms), the ground truth should capture transient eye movements that slower annotations miss.
Reading between the lines
- The paper does not quantify agreement between the automatic annotations and the human-corrected labels, so the actual accuracy gain over fully automatic methods is unknown; a natural next step would be to report such statistics.
- If the human review step is genuinely reliable, the same semi-automatic approach could be used to leverage larger unlabeled event-camera recordings in a human-in-the-loop labeling effort.
- A testable extension would be to benchmark downstream eye-tracking models trained on these 200Hz labels versus the original dataset's labels, to see whether the denser supervision improves saccade or blink detection in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a semi-automatic annotation pipeline for event-based eye-tracking data. The authors accumulate events from the Angelopoulos dataset into 5 ms frames, detect saccades with a threshold, match motion templates, and fit an ellipse with RANSAC to obtain pupil-center candidates; human annotators then review and correct the automatic labels. The pipeline is applied to the left eye of six users, and the released annotations include pupil-center positions, saccade states, and blink states. The central claim is that these annotations constitute high-frequency ground truth for pupil detection at 200 Hz.
Significance. Pupil-level annotations for event-based eye tracking are scarce, and a public, carefully documented dataset would be a valuable resource for training and validation. The paper offers a public release and a reproducible pipeline, which are strengths. However, as presented, the 200 Hz ground-truth claim is not supported by the reported statistics, and the annotation quality is not quantitatively validated. The actual contribution therefore depends on whether the authors can either provide dense annotations or accurately characterize the sparse coverage and demonstrate label quality.
major comments (3)
- [Abstract, §2, Table 1] The abstract and Section 2 state that the authors provide 'annotations for pupil detection at 200Hz' and 'retrieve the center of the pupil ... at a rate of 200 Hz', but Table 1 shows only 22,305 eye-center positions across 315,213 analyzed frames (7.1%). Since Section 3.1 computes the pupil center only when a saccade is detected and Section 3.2 excludes frames with few events, the released pupil positions are sparse and do not constitute a continuous 200 Hz ground truth. The claim should be revised to reflect the actual coverage (e.g., pupil centers only in saccade frames), and the authors should report the temporal coverage of pupil positions per user.
- [§3.2] The manual review stage is described as the final validation of the ground truth, but no inter-annotator agreement, comparison to an independent gold standard, or uncertainty measure is reported. Without such statistics, readers cannot assess label accuracy. The authors should add a double-annotation study on a subset (e.g., at least two annotators on a few thousand frames) and report agreement metrics such as per-coordinate error and intraclass correlation, and also compare the automatic labels against the corrected labels to quantify the automatic stage's error.
- [§3.1] The pipeline's behavior is governed by several ad-hoc thresholds and hyperparameters: the saccade event-count threshold of 150, the per-user event thresholds in §3.2, the number of templates (8), and the number of RANSAC iterations (1000). No sensitivity analysis is provided. Since the saccade threshold directly determines which frames receive pupil labels, a small change in this threshold could substantially alter the coverage and quality of the released annotations. The authors should justify these values empirically or provide a sensitivity analysis.
minor comments (6)
- [§2] In the description of the Ini-30 dataset, the phrase 'with labels dire' appears to be a typo; the intended word is likely 'direct'.
- [Fig. 1] The caption does not explain the color coding (green/red polarities) or specify which user and acquisition are shown; please add these details.
- [§3.1] The sentence 'we employ RANSAC to fit an ellipse described using the equation of a generic conic with the f parameter fixed to -1' does not give the conic equation; please specify the equation and the ellipse constraint explicitly.
- [§4] The text says 'We reviewed and annotated 114222 frames' and then 'the total number of frames reviewed was 315213'; please clarify the relationship between reviewed and annotated frames and use consistent terminology.
- [§4] The release covers only six of the 27 users in the original Angelopoulos dataset; please state this prominently and explain the selection criteria for these users.
- [§5] The conclusion repeats the overclaim that the annotations are 'improved annotations at 200Hz'; please align this sentence with the corrected claim from Major Comment 1.
Circularity Check
No significant circularity: the annotation pipeline inherits the authors' prior method but the released annotations stand as an independent dataset contribution.
full rationale
The paper's contribution is a semi-automatic annotation pipeline and a public release of pupil, saccade, and blink labels for the Angelopoulos event-based eye-tracking dataset. There is no predictive derivation whose output reduces to a fitted input. The automatic step uses template matching and RANSAC ellipse fitting "inspired by the method described in [16]" (Section 3.1), and [16] is a prior paper by overlapping authors (Mentasti and Matteucci). This is a self-citation, but it is not load-bearing: the central claim is the availability of the computed annotations, whose value can be checked directly against the raw event data and by downstream users; no uniqueness theorem or forbidden-alternative argument rests on [16]. The manual review stage (Section 3.2) treats annotator-corrected positions as ground truth without inter-annotator agreement or an external gold standard. That is a validity limitation, not circularity, because the labels are the output, not a "prediction" of an independently defined quantity. The paper itself flags sparsity: Section 3.2 excludes frames with few events, and Section 4 reports only 22,305 eye-center positions among 315,213 frames, so the advertised continuous 200Hz pupil-center ground truth is internally inconsistent. This is a correctness/consistency concern, not a circular derivation. No equation or fitted parameter is renamed as a prediction, and no load-bearing claim reduces by construction to its own inputs. Score 2 reflects the one minor, non-load-bearing self-citation to [16].
Assumptions & free parameters
free parameters (5)
- Saccade event-count threshold =
150 events
- Per-user review event threshold =
not disclosed
- Anomaly displacement threshold =
not disclosed
- Number of motion templates =
8
- RANSAC iterations =
1000
assumptions (6)
- domain assumption 200Hz event accumulation captures all relevant eye movements, including saccades and microsaccades.
- ad hoc to paper A fixed event-count threshold of 150 separates saccade frames from non-saccade frames.
- ad hoc to paper Frames with fewer than a per-user threshold of events contain no usable eye information and can be excluded from annotation.
- domain assumption Human annotator corrections provide accurate ground truth.
- domain assumption Pupil center displacement between consecutive 5ms frames is bounded by the chosen anomaly threshold.
- domain assumption RANSAC ellipse fitting on events inside the detected ROI yields the pupil center.
Cite this review
Pith. "Pith review of High-frequency near-eye ground truth for event-based eye tracking." pith.science (2026). https://pith.science/paper/XY7FPMOS
@misc{pith2026250203057,
author = {Pith},
title = {Pith review of: High-frequency near-eye ground truth for event-based eye tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/XY7FPMOS}},
note = {Machine review of arXiv:2502.03057}
}
read the original abstract
Event-based eye tracking is a promising solution for efficient and low-power eye tracking in smart eyewear technologies. However, the novelty of event-based sensors has resulted in a limited number of available datasets, particularly those with eye-level annotations, crucial for algorithm validation and deep-learning training. This paper addresses this gap by presenting an improved version of a popular event-based eye-tracking dataset. We introduce a semi-automatic annotation pipeline specifically designed for event-based data annotation. Additionally, we provide the scientific community with the computed annotations for pupil detection at 200Hz.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Ap- plied Linguistics Papers (25/2), 123–132 (2018)
Andrychowicz-Trojanowska, A.: Basic terminology of eye-tracking research. Ap- plied Linguistics Papers (25/2), 123–132 (2018)
work page 2018
-
[2]
arXiv preprint arXiv:2004.03577 (2020)
Angelopoulos, A.N., Martel, J.N., Kohli, A.P., Conradt, J., Wetzstein, G.: Event based, near eye gaze tracking beyond 10,000 hz. arXiv preprint arXiv:2004.03577 (2020)
arXiv 2020
-
[3]
Angelov, V., Petkov, E., Shipkovenski, G., Kalushkov, T.: Modern virtual reality headsets. In: 2020 International congress on human-computer interaction, opti- mization and robotic applications (HORA). pp. 1–5. IEEE (2020)
work page 2020
-
[4]
Arena, F., Collotta, M., Pau, G., Termine, F.: An overview of augmented reality. Computers 11(2), 28 (2022)
work page 2022
-
[5]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Bonazzi, P., Bian, S., Lippolis, G., Li, Y., Sheik, S., Magno, M.: Retina: Low- power eye tracking with event camera and spiking hardware. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5684– 5692 (2024)
work page 2024
-
[6]
Bulling, A., Gellersen, H.W.: Toward mobile eye-based human-computer interac- tion.IEEEPervasiveComputing 9,8–12(2010). https://doi.org/10.1109/MPRV. 2010.86
doi:10.1109/mprv 2010
-
[7]
Sensors (Basel, Switzerland) 18 (2018)
Burger, G., Guna, J., Pogacnik, M.: Suitability of inexpensive eye-tracking device for user experience evaluations. Sensors (Basel, Switzerland) 18 (2018). https: //doi.org/10.3390/s18061822
-
[8]
In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications
Crafa, D.M., Di Giacomo, S., Natali, D., Fiorini, C.E., Carminati, M.: Towards invisible eye tracking with lens-coupled lateral photodetectors. In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications. pp. 1–7 (2024)
work page 2024
Show all 27 references
-
[9]
Communi- cations of the ACM24(6), 381–395 (1981)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communi- cations of the ACM24(6), 381–395 (1981)
1981
-
[10]
arXiv preprint arXiv:1601.04902 (2016)
Fuhl, W., Santini, T., Kasneci, G., Kasneci, E.: Pupilnet: Convolutional neural networks for robust pupil detection. arXiv preprint arXiv:1601.04902 (2016)
2016 arXiv
-
[11]
IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)
Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A.J., Conradt, J., Daniilidis, K., et al.: Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)
2020
-
[12]
Holmqvist, K., Andersson, R.: Eye-tracking: A comprehensive guide to methods, paradigms and measures (11 2017)
2017
-
[13]
Kassner, M., Patera, W., Bulling, A.: Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction (April 2014),http://arxiv.org/ abs/1405.0006
2014 arXiv
-
[14]
Scientific Reports 7(1), 16823 (Dec 2017)
Khazali, M.F., Pomper, J.K., Thier, P.: Blink associated resetting eye movements (barms) are functionally complementary to microsaccades in correcting for fixation errors. Scientific Reports 7(1), 16823 (Dec 2017). https://doi.org/10.1038/ s41598-017-17229-w, https://doi.org/1...
2017 doi
-
[15]
PLoS ONE 16 (2021)
Liu, Z., Yang, Z., Gu, Y., Liu, H., Wang, P.: The effectiveness of eye tracking in the diagnosis of cognitive disorders: A systematic review and meta-analysis. PLoS ONE 16 (2021). https://doi.org/10.1371/journal.pone.0254059
2021 doi
-
[16]
In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications
Mentasti, S., Lattari, F., Santambrogio, R., Careddu, G., Matteucci, M.: Event- based eye tracking for smart eyewear. In: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications. pp. 1–7 (2024) High-frequency near-eye ground truth for event-based eye tracking 11
2024
-
[17]
Neurological Sciences36, 683– 688 (2015)
Popa, L., Selejan, O., Scott, A., Muresanu, D., Balea, M., Rafila, A.: Reading beyond the glance: eye tracking in neurosciences. Neurological Sciences36, 683– 688 (2015). https://doi.org/10.1007/s10072-015-2076-6
2015 doi
-
[18]
Computer Vision and Image Understanding170, 40–50 (2018)
Santini, T., Fuhl, W., Kasneci, E.: Pure: Robust pupil detection for real-time per- vasive eye tracking. Computer Vision and Image Understanding170, 40–50 (2018)
2018
-
[19]
IEEE Reviews in Biomedical Engineering 16, 260–277 (2021)
Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N., Tripoliti, E., Marias, K., Fotiadis, D., Tsiknakis, M.: Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Reviews in Biomedical Engineering 16, 260–277 (...
2021
-
[20]
Spil, T.A., Kijl, B., Romijnders, V.: The adoption and diffusion of wearables. In: ICT Unbounded, Social Impact of Bright ICT Adoption: IFIP WG 8.6 Interna- tional Conference on Transfer and Diffusion of IT, TDIT 2019, Accra, Ghana, June 21–22, 2019, Proceedings. pp. 31–47. Sp...
2019
-
[21]
Computers & Electrical Engineering93, 107193 (2021)
Wan, Z.H., Xiong, C.H., Chen, W.B., Zhang, H.Y.: Robust and accurate pupil detection for head-mounted eye tracking. Computers & Electrical Engineering93, 107193 (2021)
2021
-
[22]
ais 2024 challenge survey (2024),https://arxiv.org/ abs/2404.11770
Wang, Z., Gao, C., Wu, Z., Conde, M.V., Timofte, R., Liu, S.C., Chen, Q., jun Zha, Z., Zhai, W., Han, H., Liao, B., Wu, Y., Wan, Z., Wang, Z., Cao, Y., Tan, G., Chen, J., Pei, Y.R., Brüers, S., Crouzet, S., McLelland, D., Coenen, O., Zhang, B., Gao, Y., Li, J., So, H.K.H., Bic...
2024 arXiv
-
[23]
In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX)
Winkler, S., Subramanian, R.: Overview of eye tracking datasets. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX). pp. 212–
2013
-
[24]
Sensors23(18), 7753 (2023)
Zafar, A., Martin Calderon, C., Yeboah, A.M., Dalton, K., Irving, E., Niechwiej- Szwedo, E.: Investigation of camera-free eye-tracking glasses compared to a video- based system. Sensors23(18), 7753 (2023)
2023
-
[25]
Computational Intelligence and Neuroscience2017 (2017)
Zhang, X., Liu, X., Yuan, S., Lin, S.: Eye tracking based control system for natural human-computer interaction. Computational Intelligence and Neuroscience2017 (2017). https://doi.org/10.1155/2017/5739301
2017 doi
-
[26]
Advances in Neural Information Processing Systems36 (2024)
Zhao, G., Yang, Y., Liu, J., Chen, N., Shen, Y., Wen, H., Lan, G.: Ev-eye: Rethink- ing high-frequency eye tracking through the lenses of event cameras. Advances in Neural Information Processing Systems36 (2024)
2024
-
[27]
IEEE Transactions on biomedical engineering54(12), 2246–2260 (2007)
Zhu, Z., Ji, Q.: Novel eye gaze tracking techniques under natural head movement. IEEE Transactions on biomedical engineering54(12), 2246–2260 (2007)
2007
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.