{"id":"d3ac76cd-244a-43f4-8a05-44283ba72ef8","arxiv_id":"2508.04827","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A CNN-LSTM model predicts pupil center coordinates from event camera frames with roughly 81% pixel accuracy on a public eye-tracking dataset, outperforming CNN-GRU and CNN-BiLSTM in this comparison.","lead":"This paper compares three standard deep learning architectures for predicting pupil center coordinates from event camera data, reporting that a CNN-LSTM model achieves about 81% accuracy. It proposes future interpretability work using Layer-wise Relevance Propagation, but does not implement it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CNN-LSTM's edge may reflect multi-layer depth, not LSTM recurrence; GRU/BiLSTM are single-layer.","rationale":"The reader's verdict correctly flags the missing split description and other documentation gaps. I agree those are important for reproducibility. However, the claim that matters most is the comparative one: CNN-LSTM is the 'most effective' architecture. That claim is internally invalidated by the architecture descriptions in §3.3. The LSTM model is explicitly multi-layer while the GRU and BiLSTM are single-layer. Because LSTM cells are parameter-heavy and stacking adds capacity, the observed ranking is confounded. It is possible that a single-layer LSTM would not beat the others. This is not a matter of external generalization (leakage), but a direct un-controlled variable in the experiment. The test I propose (single-layer LSTM vs single-layer GRU/BiLSTM with matched size) would settle it. If the authors can show the advantage persists under matched depth, the claim survives; if not, the conclusion should be weakened. Thus the reader's CONDITIONAL verdict remains appropriate, but for a reason not emphasized in the reader's weakest-assumption field.","tokens_in":5884,"tokens_out":8463,"duration_ms":98306,"concrete_test":"Retrain the three models with a single recurrent layer each (LSTM-1, GRU-1, BiLSTM-1), matched hidden size and CNN backbone, identical hyperparameters and seeds, and report accuracy at pixel tolerances 5, 10, 15. If LSTM-1 does not outperform GRU-1 and BiLSTM-1, the original Table 1 ranking is attributable to recurrent depth/capacity rather than to the LSTM architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sections 3.3.1–3.3.3 compare a CNN followed by a single-layer GRU and a CNN followed by a single-layer BiLSTM against a CNN followed by a multi-layer LSTM. The comparison therefore changes both the recurrent cell type and the recurrent depth/parameter count. LSTMs already have roughly 4/3 the parameters of GRUs at equal depth, and the multi-layer stack further increases capacity. The reported 72/77/81% ranking (Table 1) could reflect capacity or training dynamics rather than any intrinsic advantage of LSTM recurrence. The paper provides no parameter counts, no controlled-depth experiments, and no error bars or seeds, so the central claim that 'CNN-LSTM outperforms' is not supported as a statement about architecture choice. This is a load-bearing internal-validity issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares three hybrid neural architectures—CNN-GRU, CNN-BiLSTM, and CNN-LSTM—for predicting pupil center coordinates from event-camera frames in an eye-tracking task. The data come from the public Event-based Eye Tracking AIS2024 Kaggle dataset (§3.1). The authors report that CNN-LSTM achieves the highest test accuracy, approximately 81%, versus 72% for CNN-GRU and 77% for CNN-BiLSTM (Table 1). Sections 4–5 describe the loss function, two evaluation metrics, hyperparameters, and a literature-review-level discussion of Layer-wise Relevance Propagation as planned future work. The central claim is empirical: CNN-LSTM outperforms the other two architectures on this dataset under the reported evaluation protocol.","tokens_in":6081,"tokens_out":4023,"duration_ms":43674,"significance":"If the reported result is reproducible under a leakage-free split and a controlled architecture comparison, it would provide modest evidence that LSTM recurrence is preferable to GRU or BiLSTM for event-based eye tracking. The use of a public dataset is a strength, as is the explicit listing of some training hyperparameters in §4.5. The paper is not circular: the accuracy figures are empirical measurements after supervised training, not analytical deductions. However, the contribution is essentially a single empirical comparison; no code, data split description, error bars, or multiple-seed statistics are provided, and the LRP material is a literature review rather than an implementation. The value of the work therefore depends entirely on the completeness and validity of the evaluation, which currently has several load-bearing gaps.","major_comments":[{"comment":"The reported comparison changes both the recurrent cell type and the recurrent capacity. Sections 3.3.1 and 3.3.2 specify a single-layer GRU and a single-layer BiLSTM, while §3.3.3 specifies a multi-layer LSTM. LSTMs already have roughly 4/3 the parameters of GRUs at equal hidden size, and adding layers further increases capacity. The 81/77/72 ordering in Table 1 may therefore reflect model capacity or training dynamics rather than an intrinsic advantage of LSTM recurrence. No parameter counts, controlled-depth experiments, or multiple seeds are reported. The claim that 'CNN-LSTM outperforms' the other architectures is thus not supported as a statement about architecture choice. Please include a depth/parameter-matched comparison (e.g., single-layer LSTM and multi-layer GRU/BiLSTM) and report seeds/error bars.","section":"§3.3.1–3.3.3 and Table 1"},{"comment":"The train/test split is not described. Section 3.1 states the dataset contains multiple recording sessions per subject, and Section 3.2 describes label downsampling to 20Hz, but no criterion is given for partitioning sessions or subjects into training and test sets. Because temporally adjacent event frames are highly correlated, a split that places frames from the same session in both training and test sets would inflate accuracy via near-duplicate inputs. The reported 81% accuracy can only be interpreted as generalization if the split is leakage-free. Please specify the split (e.g., by session, by subject), report the number of sessions/subjects in each split, and verify that no frames from the same recording appear in both training and test.","section":"§3.1–3.2 and Table 1"},{"comment":"The quantitative claim rests on an incompletely defined accuracy metric. Section 4.1 defines pixel accuracy as 'correctly predicted pixels' over total pixels but does not state how the tolerance enters the calculation; Section 4.5 lists pixel tolerances [5,10,15] but Table 1 reports a single accuracy value without identifying which tolerance was used. Additionally, no error bars, confidence intervals, or number of runs are given, so the differences 72/77/81 cannot be distinguished from run-to-run or tolerance-choice variation. Report accuracy at each tolerance, specify the tolerance used for the headline number, and include multiple-seed statistics.","section":"§4.1, §4.3, §4.5"}],"minor_comments":[{"comment":"The weighted MSE formula defines weights w_i but never states what the weights are or how they were chosen. This affects reproducibility of the training objective.","section":"§4"},{"comment":"The 'px euclidean dist' formula sums over N samples, but the text calls it a 'total' distance. State whether the reported metric is a sum or a mean, and report its values for the three models since the paper says two metrics are used but Table 1 only shows accuracy.","section":"§4.2"},{"comment":"Figures 1 and 2 are referenced but their content is not described in the text. Please add captions that explain the data-processing flow and the model architecture, including layer dimensions.","section":"§3.1 and Figures 1–2"},{"comment":"The hyperparameter list omits key architecture dimensions: number of convolutional layers, kernel sizes, number of filters, recurrent hidden size, and number of LSTM layers. These are necessary to reproduce the models.","section":"§4.5"},{"comment":"Model naming is inconsistent: the abstract and §3.3.3 use 'CNN_LSTM', while §3.3 and Table 1 use 'CNN-LSTM'. Please unify.","section":"Throughout"},{"comment":"The dataset volume and format are given, but not the spatial resolution of event frames, the number of frames per session, or the event-frame generation method. Add these details for reproducibility.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a course project report rather than a full journal article. The central empirical claim is plausible but currently under-specified: the architecture comparison is confounded with recurrent depth, the data split is not described, and the accuracy metric is incompletely defined. These are fixable with additional experiments and reporting, so I recommend major revision rather than rejection. There is no evidence of circular reasoning or misconduct; the issue is under-specification and internal validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a project report, not a research contribution. It applies CNN-LSTM, CNN-GRU, and CNN-BiLSTM to the public AIS2024 event-based eye-tracking Kaggle dataset and reports 81% accuracy for CNN-LSTM. I don't think that claim survives contact with the methods section.\n\nWhat's good: the authors used a public dataset, listed their hyperparameters (learning rate, batch size, spatial/temporal factors, epochs, tolerances), and were honest that LRP is future work rather than pretending they implemented it. The writing is plain and the pipeline figures are clear. That is real but modest merit.\n\nThe soft spots are load-bearing. First, the comparison is confounded. Sections 3.3.1 and 3.3.2 specify single-layer GRU and single-layer BiLSTM, while 3.3.3 uses a multi-layer LSTM. So the comparison changes both cell type and recurrent depth/parameter count. LSTMs already have roughly 4/3 the parameters of GRUs at equal depth; added layers widen that gap. The 72/77/81 ordering in Table 1 could just be a capacity effect. There are no parameter counts, no controlled-depth runs, no seeds, and no error bars. Second, the evaluation is underdescribed. Pixel accuracy depends on 'specified tolerance ranges,' and the hyperparameter list gives tolerances [5, 10, 15], but the paper never says which tolerance produced the 81% figure or whether it is one tolerance or some average. The train/test split is not described, which matters because temporally adjacent event frames are highly correlated; if sessions overlap the split, the number could reflect memorization rather than generalization. Third, the weighted MSE includes weights w_i but the weights are never specified. None of these issues are fatal to the idea that CNN-LSTM can work on this task; they are fatal to the paper's central claim as written.\n\nThe reader's take and the stress-test note both land. The stress-test point about multi-layer LSTM versus single-layer GRU/BiLSTM is exactly right, and I confirmed it in the model summaries. The circularity concern is minimal; this is supervised empirical evaluation, not a derivation.\n\nWho gets value: someone wanting a quick baseline overview for this dataset, or a cautionary example of how to write an unreproducible comparison. It could be a fine undergraduate or graduate project deliverable. For a serious refereed venue, I would desk-reject it as-is: the empirical claim cannot be checked, and the architecture comparison is not controlled. If the authors add code, split/seeds/tolerances, error bars, and a matched-depth ablation, it becomes a reasonable workshop data point.","headline":"A clearly written student project report whose headline architecture comparison is confounded by recurrent depth and underdocumented evaluation.","tokens_in":6562,"tokens_out":2164,"would_cite":false,"duration_ms":26416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CNN-LSTM model predicts pupil center coordinates from event-camera frames with about 81% accuracy, beating CNN-GRU and CNN-BiLSTM.","keywords":["event-based eye tracking","CNN-LSTM","pupil center prediction","event camera","gaze estimation","recurrent neural networks","explainable AI","Layer-wise Relevance Propagation"],"falsifier":"Retrain the CNN-LSTM on a session-held-out split of the same public event-based eye-tracking data—every frame from a recording session in exactly one of training or test—and compare pixel accuracy at the same tolerances. A large drop would show the reported 81% relies on temporal leakage; a similar value would support the architecture claim.","tokens_in":5776,"feed_emoji":"👁️","tokens_out":6692,"duration_ms":72172,"temperature":0.7,"pith_summary":"The paper tries to establish that a hybrid CNN-LSTM network can locate the pupil center (x, y) directly from sparse event-camera frames, a setting where fast eye movements (up to 300°/s) would normally demand expensive high-speed cameras. It compares three CNN-plus-recurrent designs on a public event-based eye-tracking dataset and reports that CNN-LSTM is the most accurate, reaching about 81% test accuracy versus 77% for CNN-BiLSTM and 72% for CNN-GRU. The authors interpret this as evidence that convolutional layers for spatial features plus multi-layer LSTM layers for temporal dependencies form a strong combination for gaze prediction. They position the result as a step toward cost-effective, interpretable eye tracking for VR/AR and attention research, with Layer-wise Relevance Propagation proposed as the next stage.","feed_headline":"CNN-LSTM beats GRU and BiLSTM for event-camera pupil tracking","feed_subtitle":"Hybrid network predicts pupil-center coordinates from sparse event frames, topping two recurrent alternatives.","key_machinery":"The load-bearing mechanism is the CNN-LSTM hybrid: a convolutional network turns raw event frames into spatial feature maps, and a multi-layer LSTM reads the resulting sequence to model temporal gaze dynamics. Training uses a weighted mean squared error loss with batch normalization and dropout; evaluation uses pixel accuracy at tolerances of 5, 10, and 15 pixels. The event-frame representation is what makes the sparse, high-speed eye-movement signal tractable for a standard deep network.","core_discovery":"On the paper's own terms, the central discovery is an architecture ranking: on the public event-based eye-tracking dataset recorded with the DVXplorer Mini event camera, CNN-LSTM predicts pupil center coordinates with approximately 81% accuracy, ahead of CNN-BiLSTM at 77% and CNN-GRU at 72%. The claim is that combining convolutional feature extraction with a multi-layer LSTM is the best of the three tested ways to fuse spatial and temporal information for this task. The authors take this as evidence that the hybrid CNN-LSTM framework is a viable, comparatively cheap foundation for real-time gaze estimation, and they identify Layer-wise Relevance Propagation as the route to making such predic","pith_inferences":["The paper does not describe the train/test split; a strict session-held-out evaluation could change the 81% figure, since temporally adjacent frames are highly correlated.","The margin between winners is 4 points, so the claim is safest as evidence for the CNN-plus-recurrent family rather than for LSTM specifically.","A natural next experiment is to feed continuous event streams directly to the recurrent layers instead of frame accumulations, exploiting the event camera's temporal resolution.","If LRP succeeds on the hybrid model, gaze predictions could be traced to specific pixels, which would make the accuracy result actionable for AR/VR interface design."],"forward_implications":["The reported ranking makes CNN-LSTM the natural starting architecture for event-based pupil tracking on this type of data.","A single hybrid network can predict gaze at 20 Hz from labels originally recorded at 100 Hz, which is consistent with low-latency consumer devices.","The results imply that temporal modeling matters for eye tracking: recurrent layers add value beyond convolutional feature extraction.","Because the winning architecture is standard, it provides a concrete baseline for future interpretability work such as LRP heatmaps of gaze decisions."],"supporting_citations":[{"why":"Supplies the public event-camera dataset with 100 Hz pupil labels; it is the training and test ground for all three compared models.","marker":"[4]"},{"why":"Motivates combining CNNs with recurrent layers for eye-tracking tasks, the architecture family under comparison.","marker":"[2]"},{"why":"Provides the application context that makes high-speed, low-cost eye tracking relevant.","marker":"[1]"}],"fun_headline_variants":["Event-camera gaze: CNN-LSTM scores 81%, BiLSTM 77%, GRU 72%","CNN-LSTM beats BiLSTM and GRU on event-camera pupil tracking","Deep hybrid CNN-LSTM leads event-based eye tracking with 81% accuracy","Event-based pupil tracking: CNN-LSTM outperforms recurrent rivals"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The 81% accuracy figure assumes frames in the public dataset were split so that no recording session contributes to both training and test; the paper never specifies the split, and temporally adjacent frames are highly similar.","fun_headline_variants_meta":{"raw":{"variants":["Event-camera gaze: CNN-LSTM scores 81%, BiLSTM 77%, GRU 72%","CNN-LSTM beats BiLSTM and GRU on event-camera pupil tracking","Deep hybrid CNN-LSTM leads event-based eye tracking with 81% accuracy","Event-based pupil tracking: CNN-LSTM outperforms recurrent rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2299,"prompt_tokens":675,"completion_tokens":1624,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":1535}},"tokens_in":419,"tokens_out":1624,"duration_ms":13459,"temperature":1.0,"reasoning_tokens":1535,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:43:57.776798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the CNN-LSTM on a session-held-out split of the same public event-based eye-tracking data—every frame from a recording session in exactly one of training or test—and compare pixel accuracy at the same tolerances. A large drop would show the reported 81% relies on temporal leakage; a similar value would support the architecture claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the public event-camera dataset with 100 Hz pupil labels; it is the training and test ground for all three compared models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates combining CNNs with recurrent layers for eye-tracking tasks, the architecture family under comparison."},{"cited_title":"Stuart, Eye Tracking: Background, Methods, and Applications, vol","cited_arxiv_id":null,"evidence_quote":"Provides the application context that makes high-speed, low-cost eye tracking relevant."}],"review_version":1}