A deep learning approach to track eye movements based on events

Chirag Seth; Divya Naiken; Keyan Lin

REVIEW 3 major objections 6 minor 11 references

A deep learning approach to track eye movements based on events

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A CNN-LSTM model predicts pupil center coordinates from event-camera frames with about 81% accuracy, beating CNN-GRU and CNN-BiLSTM.

desk verdict A clearly written student project report whose headline architecture comparison is confounded by recurrent depth and underdocumented evaluation. read the letter →

arxiv 2508.04827 v1 pith:ULN6J2S6 submitted 2025-08-06 cs.CV

Chirag Seth , Divya Naiken , Keyan Lin This is my paper

classification cs.CV

keywords event-basedeyetrackingCNN-LSTMpupilcenterpredictioneventcameragazeestimationrecurrentneuralnetworksexplainableAILayer-wiseRelevancePropagation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a hybrid CNN-LSTM network can locate the pupil center (x, y) directly from sparse event-camera frames, a setting where fast eye movements (up to 300°/s) would normally demand expensive high-speed cameras. It compares three CNN-plus-recurrent designs on a public event-based eye-tracking dataset and reports that CNN-LSTM is the most accurate, reaching about 81% test accuracy versus 77% for CNN-BiLSTM and 72% for CNN-GRU. The authors interpret this as evidence that convolutional layers for spatial features plus multi-layer LSTM layers for temporal dependencies form a strong combination for gaze prediction. They position the result as a step toward cost-effective, interpretable eye tracking for VR/AR and attention research, with Layer-wise Relevance Propagation proposed as the next stage.

What carries the argument

The load-bearing mechanism is the CNN-LSTM hybrid: a convolutional network turns raw event frames into spatial feature maps, and a multi-layer LSTM reads the resulting sequence to model temporal gaze dynamics. Training uses a weighted mean squared error loss with batch normalization and dropout; evaluation uses pixel accuracy at tolerances of 5, 10, and 15 pixels. The event-frame representation is what makes the sparse, high-speed eye-movement signal tractable for a standard deep network.

What would settle it

Retrain the CNN-LSTM on a session-held-out split of the same public event-based eye-tracking data—every frame from a recording session in exactly one of training or test—and compare pixel accuracy at the same tolerances. A large drop would show the reported 81% relies on temporal leakage; a similar value would support the architecture claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an architecture ranking: on the public event-based eye-tracking dataset recorded with the DVXplorer Mini event camera, CNN-LSTM predicts pupil center coordinates with approximately 81% accuracy, ahead of CNN-BiLSTM at 77% and CNN-GRU at 72%. The claim is that combining convolutional feature extraction with a multi-layer LSTM is the best of the three tested ways to fuse spatial and temporal information for this task. The authors take this as evidence that the hybrid CNN-LSTM framework is a viable, comparatively cheap foundation for real-time gaze estimation, and they identify Layer-wise Relevance Propagation as the route to making such predic

Load-bearing premise

The 81% accuracy figure assumes frames in the public dataset were split so that no recording session contributes to both training and test; the paper never specifies the split, and temporally adjacent frames are highly similar.

Editorial extensions

If this is right

The reported ranking makes CNN-LSTM the natural starting architecture for event-based pupil tracking on this type of data.
A single hybrid network can predict gaze at 20 Hz from labels originally recorded at 100 Hz, which is consistent with low-latency consumer devices.
The results imply that temporal modeling matters for eye tracking: recurrent layers add value beyond convolutional feature extraction.
Because the winning architecture is standard, it provides a concrete baseline for future interpretability work such as LRP heatmaps of gaze decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not describe the train/test split; a strict session-held-out evaluation could change the 81% figure, since temporally adjacent frames are highly correlated.
The margin between winners is 4 points, so the claim is safest as evidence for the CNN-plus-recurrent family rather than for LSTM specifically.
A natural next experiment is to feed continuous event streams directly to the recurrent layers instead of frame accumulations, exploiting the event camera's temporal resolution.
If LRP succeeds on the hybrid model, gaze predictions could be traced to specific pixels, which would make the accuracy result actionable for AR/VR interface design.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a project report, not a research contribution. It applies CNN-LSTM, CNN-GRU, and CNN-BiLSTM to the public AIS2024 event-based eye-tracking Kaggle dataset and reports 81% accuracy for CNN-LSTM. I don't think that claim survives contact with the methods section.

What's good: the authors used a public dataset, listed their hyperparameters (learning rate, batch size, spatial/temporal factors, epochs, tolerances), and were honest that LRP is future work rather than pretending they implemented it. The writing is plain and the pipeline figures are clear. That is real but modest merit.

The soft spots are load-bearing. First, the comparison is confounded. Sections 3.3.1 and 3.3.2 specify single-layer GRU and single-layer BiLSTM, while 3.3.3 uses a multi-layer LSTM. So the comparison changes both cell type and recurrent depth/parameter count. LSTMs already have roughly 4/3 the parameters of GRUs at equal depth; added layers widen that gap. The 72/77/81 ordering in Table 1 could just be a capacity effect. There are no parameter counts, no controlled-depth runs, no seeds, and no error bars. Second, the evaluation is underdescribed. Pixel accuracy depends on 'specified tolerance ranges,' and the hyperparameter list gives tolerances [5, 10, 15], but the paper never says which tolerance produced the 81% figure or whether it is one tolerance or some average. The train/test split is not described, which matters because temporally adjacent event frames are highly correlated; if sessions overlap the split, the number could reflect memorization rather than generalization. Third, the weighted MSE includes weights w_i but the weights are never specified. None of these issues are fatal to the idea that CNN-LSTM can work on this task; they are fatal to the paper's central claim as written.

The reader's take and the stress-test note both land. The stress-test point about multi-layer LSTM versus single-layer GRU/BiLSTM is exactly right, and I confirmed it in the model summaries. The circularity concern is minimal; this is supervised empirical evaluation, not a derivation.

Who gets value: someone wanting a quick baseline overview for this dataset, or a cautionary example of how to write an unreproducible comparison. It could be a fine undergraduate or graduate project deliverable. For a serious refereed venue, I would desk-reject it as-is: the empirical claim cannot be checked, and the architecture comparison is not controlled. If the authors add code, split/seeds/tolerances, error bars, and a matched-depth ablation, it becomes a reasonable workshop data point.

Referee Report

3 major / 6 minor

Summary. The manuscript compares three hybrid neural architectures—CNN-GRU, CNN-BiLSTM, and CNN-LSTM—for predicting pupil center coordinates from event-camera frames in an eye-tracking task. The data come from the public Event-based Eye Tracking AIS2024 Kaggle dataset (§3.1). The authors report that CNN-LSTM achieves the highest test accuracy, approximately 81%, versus 72% for CNN-GRU and 77% for CNN-BiLSTM (Table 1). Sections 4–5 describe the loss function, two evaluation metrics, hyperparameters, and a literature-review-level discussion of Layer-wise Relevance Propagation as planned future work. The central claim is empirical: CNN-LSTM outperforms the other two architectures on this dataset under the reported evaluation protocol.

Significance. If the reported result is reproducible under a leakage-free split and a controlled architecture comparison, it would provide modest evidence that LSTM recurrence is preferable to GRU or BiLSTM for event-based eye tracking. The use of a public dataset is a strength, as is the explicit listing of some training hyperparameters in §4.5. The paper is not circular: the accuracy figures are empirical measurements after supervised training, not analytical deductions. However, the contribution is essentially a single empirical comparison; no code, data split description, error bars, or multiple-seed statistics are provided, and the LRP material is a literature review rather than an implementation. The value of the work therefore depends entirely on the completeness and validity of the evaluation, which currently has several load-bearing gaps.

major comments (3)

[§3.3.1–3.3.3 and Table 1] The reported comparison changes both the recurrent cell type and the recurrent capacity. Sections 3.3.1 and 3.3.2 specify a single-layer GRU and a single-layer BiLSTM, while §3.3.3 specifies a multi-layer LSTM. LSTMs already have roughly 4/3 the parameters of GRUs at equal hidden size, and adding layers further increases capacity. The 81/77/72 ordering in Table 1 may therefore reflect model capacity or training dynamics rather than an intrinsic advantage of LSTM recurrence. No parameter counts, controlled-depth experiments, or multiple seeds are reported. The claim that 'CNN-LSTM outperforms' the other architectures is thus not supported as a statement about architecture choice. Please include a depth/parameter-matched comparison (e.g., single-layer LSTM and multi-layer GRU/BiLSTM) and report seeds/error bars.
[§3.1–3.2 and Table 1] The train/test split is not described. Section 3.1 states the dataset contains multiple recording sessions per subject, and Section 3.2 describes label downsampling to 20Hz, but no criterion is given for partitioning sessions or subjects into training and test sets. Because temporally adjacent event frames are highly correlated, a split that places frames from the same session in both training and test sets would inflate accuracy via near-duplicate inputs. The reported 81% accuracy can only be interpreted as generalization if the split is leakage-free. Please specify the split (e.g., by session, by subject), report the number of sessions/subjects in each split, and verify that no frames from the same recording appear in both training and test.
[§4.1, §4.3, §4.5] The quantitative claim rests on an incompletely defined accuracy metric. Section 4.1 defines pixel accuracy as 'correctly predicted pixels' over total pixels but does not state how the tolerance enters the calculation; Section 4.5 lists pixel tolerances [5,10,15] but Table 1 reports a single accuracy value without identifying which tolerance was used. Additionally, no error bars, confidence intervals, or number of runs are given, so the differences 72/77/81 cannot be distinguished from run-to-run or tolerance-choice variation. Report accuracy at each tolerance, specify the tolerance used for the headline number, and include multiple-seed statistics.

minor comments (6)

[§4] The weighted MSE formula defines weights w_i but never states what the weights are or how they were chosen. This affects reproducibility of the training objective.
[§4.2] The 'px euclidean dist' formula sums over N samples, but the text calls it a 'total' distance. State whether the reported metric is a sum or a mean, and report its values for the three models since the paper says two metrics are used but Table 1 only shows accuracy.
[§3.1 and Figures 1–2] Figures 1 and 2 are referenced but their content is not described in the text. Please add captions that explain the data-processing flow and the model architecture, including layer dimensions.
[§4.5] The hyperparameter list omits key architecture dimensions: number of convolutional layers, kernel sizes, number of filters, recurrent hidden size, and number of LSTM layers. These are necessary to reproduce the models.
[Throughout] Model naming is inconsistent: the abstract and §3.3.3 use 'CNN_LSTM', while §3.3 and Table 1 use 'CNN-LSTM'. Please unify.
[§3.1] The dataset volume and format are given, but not the spatial resolution of event frames, the number of frames per session, or the event-frame generation method. Add these details for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central 81% accuracy claim is an empirical test-set measurement, not a self-referential construction.

full rationale

This paper makes no first-principles derivation that could reduce to its own inputs. The central claim (CNN-LSTM reaches ~81% accuracy and outperforms CNN-GRU/CNN-BiLSTM, Table 1) is a supervised-learning measurement on a test set after training with MSE; the evaluation metric (pixel accuracy within tolerances [5,10,15]) is an independent threshold on held-out predictions. There are no self-citations: references [1]-[10] are external datasets, methods, and LRP literature, and no uniqueness theorem or prior result by the same authors is invoked to force the architecture choice. The LRP discussion is explicitly future work and self-identifies implementation challenges rather than claiming a derived result. Concerns that the multi-layer LSTM vs. single-layer GRU/BiLSTM comparison is confounded by depth/parameter count, and that the data split is not described, are internal-validity/reproducibility issues, not circularity: the accuracy number is not constructed from the tolerance or from the compared architectures by definition. Accordingly no circular step can be quoted or exhibited, and the score is 0.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central claim, an 81% test accuracy for CNN-LSTM, rests entirely on a training setup with hand-chosen hyperparameters, unspecified architecture details, and trust in the public competition dataset's labels and the authors' split. No parameter-free derivation or external benchmark is involved.

free parameters (7)

learning_rate = 0.001
Hand-chosen training hyperparameter; its effect on the reported accuracy is not analyzed.
batch_size = 20
Hand-chosen; no tuning or sensitivity analysis.
spatial_factor = 0.125
Hand-chosen preprocessing factor that downscales event frames; affects what the CNN sees.
temporal_subsample_factor = 0.2
Downsampling of 100 Hz labels to 20 Hz; chosen by the authors.
epochs = 200
Fixed training length; no early stopping or convergence check reported.
pixel_tolerances = [5, 10, 15]
Accuracy threshold; the paper does not state which tolerance produced the reported 81%.
CNN/LSTM architecture dimensions = unspecified
Number of convolutional layers, filter sizes, LSTM layers and hidden units are not given, so the model is underdetermined.

assumptions (3)

domain assumption Ground truth pupil labels from the Kaggle competition dataset are accurate and time-synchronized with the event data.
Section 3.2 states labels are provided at 100 Hz with coordinates (x,y) and a blink indicator; the paper provides no verification of label quality or synchronization.
domain assumption Event frames converted from raw events capture sufficient spatial and temporal information for pupil localization.
Section 3.1 describes transforming raw events into frames, but no analysis of information loss or frame timing is given.
domain assumption Pixel accuracy within a tolerance correctly measures eye tracking quality.
Section 4.1 defines the metric but does not justify the tolerance values or how they map to practical gaze error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A deep learning approach to track eye movements based on events." pith.science (2026). https://pith.science/paper/ULN6J2S6

@misc{pith2026250804827,
  author       = {Pith},
  title        = {Pith review of: A deep learning approach to track eye movements based on events},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULN6J2S6}},
  note         = {Machine review of arXiv:2508.04827}
}

read the original abstract

This research project addresses the challenge of accurately tracking eye movements during specific events by leveraging previous research. Given the rapid movements of human eyes, which can reach speeds of 300{\deg}/s, precise eye tracking typically requires expensive and high-speed cameras. Our primary objective is to locate the eye center position (x, y) using inputs from an event camera. Eye movement analysis has extensive applications in consumer electronics, especially in VR and AR product development. Therefore, our ultimate goal is to develop an interpretable and cost-effective algorithm using deep learning methods to predict human attention, thereby improving device comfort and enhancing overall user experience. To achieve this goal, we explored various approaches, with the CNN\_LSTM model proving most effective, achieving approximately 81\% accuracy. Additionally, we propose future work focusing on Layer-wise Relevance Propagation (LRP) to further enhance the model's interpretability and predictive performance.

Figures

Figures reproduced from arXiv: 2508.04827 by the authors.

**Figure 2.** Figure Showing the high level architecture of our model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages

[1]

Stuart, Eye Tracking: Background, Methods, and Applications, vol

S. Stuart, Eye Tracking: Background, Methods, and Applications, vol. 183. New York, NY: Springer US,

work page
[2]

Ahmed, Z. A. T., Albalawi, E., Aldhyani, T. H. H., Jadhav, M. E., Janrao, P., & Obeidat, M. R. M. (2023). Applying Eye Tracking with Deep Learning Techniques for Early-Stage Detection of Autism Spectrum Disorders. Data (Basel), 8(11), 168-. https://doi.org/10.3390/data8110168

work page doi:10.3390/data8110168 2023
[3]

Ayush Kumar, Prantik Howlader, Rafael Garcia, Daniel Weiskopf, and Klaus Mueller. 2020. Challenges in Interpretability of Neural Networks for Eye Movement Data. https://doi.org/10.1145/3379156.3391361

arXiv 2020
[4]

ChrisJudy, Nanashi, Zuowen Wang. (2024). Event-based Eye Tracking - AIS2024 CVPR Workshop. Kaggle. https://kaggle.com/competitions/event-based-eye-tracking-ais2024

work page 2024
[5]

(2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Bach S, Binder A, Montavon G, Klauschen F, M ¨uller KR, et al. (2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE 10(7): e0130140. https://doi.org/10.1371/journal.pone.0130140

doi:10.1371/journal.pone.0130140 2015
[6]

Holzinger, A., Saranti, A., Molnar, C., Biecek, P., Samek, W. (2022). Explainable AI Methods - A Brief Overview. In: Holzinger, A., Goebel, R., Fong, R., Moon, T., M ¨uller, KR., Samek, W. (eds) xxAI - Beyond Explainable AI. xxAI 2020. Lecture Notes in Computer Science(), vol 13200. Springer, Cham. https://doi.org/10.1007/978-3-031-04083-2 2

doi:10.1007/978-3-031-04083-2 2022
[7]

Zang, B., Ding, L., Feng, Z., Zhu, M., Lei, T., Xing, M., & Zhou, X. (2021). CNN-LRP: Understanding Con- volutional Neural Networks Performance for Target Recognition in SAR Images. Sensors (Basel, Switzerland), 21(13), 4536. https://doi.org/10.3390/s21134536

doi:10.3390/s21134536 2021
[8]

Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020). Towards Best Practice in Explaining Neural Network Decisions with LRP. CoRR, 1910.09840. https://doi.org/10.1109/ijcnn48605.2020.9206975

work page Pith review arXiv 2020

Show all 11 references

[9]

Arras, L., Montavon, G., M¨uller, K., & Samek, W. (2017). Explaining Recurrent Neural Network Predictions in Sentiment Analysis. CoRR, 1706.07206. https://doi.org/10.18653/v1/w17-5221

2017 arXiv
[10]

Warnecke, A., Arp, D., Wressnegger, C., & Rieck, K. (2020). Evaluating Explanation Methods for Deep Learning in Security. 158-174. 10.1109/EuroSP48549.2020.00018. 7

2020
[2022]

doi: 10.1007/978-1-0716-2391-6

doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Stuart, Eye Tracking: Background, Methods, and Applications, vol

S. Stuart, Eye Tracking: Background, Methods, and Applications, vol. 183. New York, NY: Springer US,

work page

[2] [2]

Ahmed, Z. A. T., Albalawi, E., Aldhyani, T. H. H., Jadhav, M. E., Janrao, P., & Obeidat, M. R. M. (2023). Applying Eye Tracking with Deep Learning Techniques for Early-Stage Detection of Autism Spectrum Disorders. Data (Basel), 8(11), 168-. https://doi.org/10.3390/data8110168

work page doi:10.3390/data8110168 2023

[3] [3]

Ayush Kumar, Prantik Howlader, Rafael Garcia, Daniel Weiskopf, and Klaus Mueller. 2020. Challenges in Interpretability of Neural Networks for Eye Movement Data. https://doi.org/10.1145/3379156.3391361

arXiv 2020

[4] [4]

ChrisJudy, Nanashi, Zuowen Wang. (2024). Event-based Eye Tracking - AIS2024 CVPR Workshop. Kaggle. https://kaggle.com/competitions/event-based-eye-tracking-ais2024

work page 2024

[5] [5]

(2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Bach S, Binder A, Montavon G, Klauschen F, M ¨uller KR, et al. (2015) On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE 10(7): e0130140. https://doi.org/10.1371/journal.pone.0130140

doi:10.1371/journal.pone.0130140 2015

[6] [6]

Holzinger, A., Saranti, A., Molnar, C., Biecek, P., Samek, W. (2022). Explainable AI Methods - A Brief Overview. In: Holzinger, A., Goebel, R., Fong, R., Moon, T., M ¨uller, KR., Samek, W. (eds) xxAI - Beyond Explainable AI. xxAI 2020. Lecture Notes in Computer Science(), vol 13200. Springer, Cham. https://doi.org/10.1007/978-3-031-04083-2 2

doi:10.1007/978-3-031-04083-2 2022

[7] [7]

Zang, B., Ding, L., Feng, Z., Zhu, M., Lei, T., Xing, M., & Zhou, X. (2021). CNN-LRP: Understanding Con- volutional Neural Networks Performance for Target Recognition in SAR Images. Sensors (Basel, Switzerland), 21(13), 4536. https://doi.org/10.3390/s21134536

doi:10.3390/s21134536 2021

[8] [8]

Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., & Lapuschkin, S. (2020). Towards Best Practice in Explaining Neural Network Decisions with LRP. CoRR, 1910.09840. https://doi.org/10.1109/ijcnn48605.2020.9206975

work page Pith review arXiv 2020