REVIEW 5 major objections 6 minor 30 references
Evaluating Sensitivity Parameters in Smartphone-Based Gaze Estimation: A Comparative Study of Appearance-Based and Infrared Eye Trackers
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that a calibration-free, smartphone-based deep-learning gaze estimator achieves statistically comparable accuracy to a commercial infrared eye tracker, while remaining more sensitive to lighting, glasses, and age.
desk verdict Useful empirical comparison undermined by a wrong Wilcoxon statistic that invalidates the parity claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is MobileEYE, an appearance-based gaze estimator that predicts screen coordinates from a single 128×128 grayscale face image using a MobileNet-V3 convolutional backbone and LSTM layers for temporal context, optimized for on-device inference. The comparison machinery is the evaluation protocol: a moving-dot stimulus serves as dynamic ground truth, the Tobii Pro Nano provides an infrared reference, Euclidean distance in millimetres is the error metric, and Wilcoxon signed-rank tests plus ANOVA compare distributions across age, gender, vision correction, lighting, device, and head position. MobileEYE's design—face-only input, no calibration, and screen-boundary-constrained output—is what makes the accuracy comparison meaningful.
What would settle it
A controlled session in which participants are instructed to fixate a second point 30 mm away from the moving dot for randomly timed intervals would settle the ground-truth assumption: if a tracker reports gaze near the second point, the measured 'error' reflects human instruction-following, not tracker inaccuracy.
Extended reading notes
Core claim
The paper's central claim is that MobileEYE, a deep-learning model combining MobileNet-V3 with LSTM layers on 128×128 grayscale face images, produces gaze estimates whose median error is not statistically distinguishable from the Tobii Pro Nano's, at 17.76 mm versus 16.53 mm mean Euclidean distance with W=65518.0 and p=0.278. The comparison used 582 recordings from 50 participants following a moving dot on two smartphones across varied lighting, head position, and vision-correction conditions. The paper further reports that MobileEYE had fewer null frames than the Tobii Pro Nano (12.04% versus 28.38%), but larger error increases under low light, with glasses, on the Samsung S22, and for older participants. On this evidence, the authors argue that calibration-free appearance-based gaze estimation is a feasible alternative for realistic mobile conditions, with known sensitivity trade-offs.
Load-bearing premise
The results treat the moving dot's screen position as the true gaze point, assuming participants continuously and accurately fixate or pursue it, so any lag or lapse is counted as gaze-estimation error.
Editorial extensions
If this is right
- If the claim holds, calibration-free smartphone gaze tracking is accurate enough for applications that tolerate roughly 1.5 cm average error, removing the need for dedicated infrared hardware.
- Deployment decisions should weight the known failure modes: low light, prescription glasses, older users, and specific phone cameras degrade the appearance-based model more than the infrared baseline.
- Because MobileEYE produced fewer null frames (12.04% versus 28.38%), a gaze-based system may return more usable data overall even where its per-frame accuracy is slightly worse.
- The reported sensitivity parameters give a reusable benchmark template for comparing future mobile gaze estimators against a commercial reference.
- Device-specific differences suggest that training or adapting appearance models to camera characteristics could reduce error without abandoning the calibration-free design.
Reading between the lines
- The Tobii's high null rate means the two error distributions are computed over partly different frame sets; a complete-case or imputed comparison could shift the headline gap.
- Because the moving dot is treated as ground truth, the absolute errors include any human pursuit or fixation lag; the comparison between systems is more defensible than the absolute accuracy values.
- A stronger test would measure gaze-contingent behaviour or use a second high-frequency reference to subtract oculomotor latency, yielding a cleaner estimate of each tracker's true error.
- A screen-boundary-constrained model cannot report errors beyond the screen, so tail comparisons between the two systems are not apples-to-apples; reporting angular error in degrees of visual angle would improve comparability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript evaluates the authors' MobileEYE appearance-based gaze estimator (a MobileNet-V3 + LSTM model) against a commercial infrared tracker, the Tobii Pro Nano, using 582 recordings from 50 participants who viewed dynamic moving-dot stimuli on two smartphones. Accuracy is measured as Euclidean distance between each tracker's output and the stimulus dot position. The paper reports an overall mean error of 17.76 mm for MobileEYE versus 16.53 mm for the Tobii Pro Nano, and a Wilcoxon signed-rank test (W=65518.0, p=0.278) that is claimed to show no statistically significant difference between the two methods. It then reports sensitivity analyses across age, gender, vision correction, head position, device type, and lighting, with several ANOVA results, and concludes that the deep learning method is more sensitive to lighting, age, and glasses, while also noting lower null-value rates and calibration-free operation of MobileEYE.
Significance. The question addressed is timely and practically relevant: can a calibration-free appearance-based smartphone gaze estimator achieve accuracy comparable to a commercial infrared tracker under realistic conditions? Strengths of the paper include a real user study with 582 recordings across two phone models, three head positions, and two lighting conditions, an a priori power analysis, and unusually transparent reporting of null-value rates and calibration failures. However, the central parity claim rests on a Wilcoxon statistic that is internally inconsistent with the reported sample size, and the ground-truth assumption and handling of missing data are not adequately described. If the statistical issues are corrected and the analysis clarified, the study could provide a useful reference benchmark for smartphone-based gaze estimation. As it stands, the reported evidence does not support the main conclusion that the two methods are statistically indistinguishable.
major comments (5)
- [Section 5.1] The Wilcoxon signed-rank result W=65518.0, p=0.278 is internally inconsistent with n=582. Under the null hypothesis, the expected value of the signed-rank statistic is n(n+1)/4 = 84,826.5 and its standard deviation is sqrt(n(n+1)(2n+1)/24) ≈ 4,058.4; the reported W is about 4.76 standard deviations below the mean, which would give a p-value on the order of 10⁻⁶, not 0.278. Even if W is interpreted as the smaller of the two rank sums (the common convention for some software), the discrepancy remains. The authors must state the exact statistic computed (sum of positive ranks or smaller rank sum), the handling of ties, and the software used, and report a corrected p-value. Without this, the statement that there is 'no statistically significant difference in the median error values' is unsupported.
- [Sections 4.3 and 5.1] The ground-truth gaze point is taken to be the position of the moving dot on the smartphone screen, and error is the Euclidean distance from each tracker's output to that dot position. This assumes participants continuously and accurately fixate or pursue the dot; any smooth-pursuit lag, saccade latency, or momentary inattention is therefore attributed to the eye tracker. The authors should describe how the stimulus was synchronized with the recording, whether data from frames in which the participant was in saccade (e.g., dot jumps) were excluded, and how the half-RTT timestamp adjustment in Section 4.2.1 affects gaze alignment. The validity of the absolute error comparison depends on this assumption, which is not acknowledged in the limitations.
- [Sections 5.2 and 5.3] The paper does not specify how null frames are handled when computing the mean errors in Table 2 and the subgroup statistics. The Tobii Pro Nano had 28.38% null values and MobileEYE 12.04%, and 10.05% of recordings used default calibration after failed calibration attempts. If null frames are discarded separately for each system, the comparison is biased because each system's errors are computed over different subsets of frames, and both systems are most likely to fail in difficult conditions. The authors should report the computation pipeline (per-frame or per-recording aggregation, thresholds for excluding recordings, and whether the same frames are used for both methods) and provide a sensitivity analysis that excludes the 10.05% default-calibration recordings, which the manuscript itself identifies as a potential source of baseline inaccuracy.
- [Section 3 and Section 5.1] The manuscript never specifies how MobileEYE's predicted gaze coordinates are converted from the 128×128 grayscale face image input to physical screen positions in millimeters. This mapping is essential to interpret the Euclidean error values in Table 2 and all subgroup comparisons. The authors should state whether the model outputs normalized screen coordinates, how those are mapped to the screen dimensions of the Samsung S22 and iPhone 14, and whether any per-device adjustment is used.
- [Sections 5.4 to 5.9] The sensitivity analyses use standard ANOVA on 582 recordings from only 50 participants, with six recordings per participant per device and repeated measures across positions and lighting. This violates the independence assumption of ordinary ANOVA and can inflate significance levels. A mixed-effects model with participant as a random effect (and device, position, and lighting as repeated factors) is needed. Additionally, Section 5.5 contains an internal contradiction: it states that 'the effect of gender is not significant, with an F-value of 6.377 and a p-value of 0.012,' yet p=0.012 is significant at α=0.05. This error, together with the unstated treatment of repeated measures, undermines confidence in the reported F and p values for age, vision correction, eye position, device, and lighting.
minor comments (6)
- [Section 5.4] The text says the lowest mean error for MobileEYE was in the '39-38 age group'; this should be the 39-48 age group (Table 3).
- [Abstract and Conclusion] The abstract and conclusion state that data came from 51 participants, but one participant was removed, leaving 50 (Table 1 and Section 4.3). Please use consistent participant counts throughout.
- [Section 5.1 and Table 2] The Wilcoxon test is described as comparing medians, but Table 2 reports means and standard deviations; the median values are not shown. Please include medians for both methods.
- [Section 4.2.1] The half-RTT synchronization method is described briefly; please clarify whether the same half-RTT adjustment was applied to both smartphone platforms and whether any time drift during the 21-minute sessions was assessed.
- [Figure 6 caption] The caption 'The segments indicate the number of recordings with calibration failures and had to proceed with the default calibration' is grammatically incomplete; please rephrase to clarify what the segments represent.
- [Section 5.8] The phrase 'these two decide types' should be 'these two device types'.
Circularity Check
No material circularity: the comparison is an external benchmark against the Tobii Pro Nano, and the authors' self-citations only supply architectural context. A reported Wilcoxon inconsistency is a statistical correctness issue, not a circular derivation.
full rationale
The paper is an empirical evaluation rather than a derivation: MobileEYE is taken from the authors' prior work [20], but the current study's central claim rests on newly collected gaze data from 50 participants compared against an external commercial infrared tracker (Tobii Pro Nano). The reported means (17.76 mm vs 16.53 mm) and the Wilcoxon result are measured outputs, not quantities obtained by construction from the model definition or from fitted constants. Self-citations ([16], [17], [20], [28]) motivate the architecture and algorithm, but they do not supply the comparison result; the Tobii benchmark is independent and externally defined. No equation in the paper reduces a predicted quantity to an input, and no fitted parameter is renamed as a prediction. The moving-dot ground-truth assumption is a shared measurement limitation affecting both systems equally and is not a circular step. The paper's own limitations (Sections 5.3 and 6: default calibration in 10.05% of recordings, uneven age representation) are data-quality concerns, not circularity. One correctness caveat: with n=582, the reported W=65518 is far below the null expectation of 84826.5 (about 4.76 SD), making p=0.278 internally implausible; this affects the 'no statistically significant difference in the median error values between the two methods' conclusion but is not evidence of circular derivation.
Assumptions & free parameters
free parameters (3)
- Lux thresholds for lighting categories =
15 and 150 lux
- Age group bins =
18-28, 29-38, 39-48, 49-58, 59-68
- Assumed effect size d for power analysis =
0.39
assumptions (4)
- domain assumption Tobii Pro Nano provides a valid reference gaze signal despite calibration failures and missing data.
- domain assumption The moving dot position is the ground-truth gaze target.
- ad hoc to paper Half-RTT timestamp adjustment synchronizes devices sufficiently.
- domain assumption Standard ANOVA assumptions apply to repeated-measures data.
Cite this review
Pith. "Pith review of Evaluating Sensitivity Parameters in Smartphone-Based Gaze Estimation: A Comparative Study of Appearance-Based and Infrared Eye Trackers." pith.science (2026). https://pith.science/paper/PMWYWKXE
@misc{pith2026250611932,
author = {Pith},
title = {Pith review of: Evaluating Sensitivity Parameters in Smartphone-Based Gaze Estimation: A Comparative Study of Appearance-Based and Infrared Eye Trackers},
year = {2026},
howpublished = {\url{https://pith.science/paper/PMWYWKXE}},
note = {Machine review of arXiv:2506.11932}
}
read the original abstract
This study evaluates a smartphone-based, deep-learning eye-tracking algorithm by comparing its performance against a commercial infrared-based eye tracker, the Tobii Pro Nano. The aim is to investigate the feasibility of appearance-based gaze estimation under realistic mobile usage conditions. Key sensitivity factors, including age, gender, vision correction, lighting conditions, device type, and head position, were systematically analysed. The appearance-based algorithm integrates a lightweight convolutional neural network (MobileNet-V3) with a recurrent structure (Long Short-Term Memory) to predict gaze coordinates from grayscale facial images. Gaze data were collected from 51 participants using dynamic visual stimuli, and accuracy was measured using Euclidean distance. The deep learning model produced a mean error of 17.76 mm, compared to 16.53 mm for the Tobii Pro Nano. While overall accuracy differences were small, the deep learning-based method was more sensitive to factors such as lighting, vision correction, and age, with higher failure rates observed under low-light conditions among participants using glasses and in older age groups. Device-specific and positional factors also influenced tracking performance. These results highlight the potential of appearance-based approaches for mobile eye tracking and offer a reference framework for evaluating gaze estimation systems across varied usage conditions.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
- [20]
-
[1]
Garcia De La Santa Ramos, A., Cabeza, R. & Villanueva, A. Calibration free eye tracking solution for mobile and embedded devices.Proceedings Of The 2023 Symposium On Eye Tracking Research And Applications. pp. 1-3 (2023)
work page 2023
-
[2]
(2023).Eye Tracking Market Size, Growth Trends and F orecast, 2024–2032
Global Market Insights. (2023).Eye Tracking Market Size, Growth Trends and F orecast, 2024–2032. Retrieved fromhttps://www.gminsights.com/industry-analysis/eye-tracking-market 17
work page 2023
-
[3]
Mokatren, M., Kuflik, T. & Shimshoni, I. Calibration-Free Mobile Eye-Tracking Using Corneal Imaging.Sen- sors.24, 1237 (2024)
work page 2024
-
[4]
Bansal, R., Raj, G. & Choudhury, T. Blur image detection using Laplacian operator and Open-CV .2016 Inter- national Conference System Modeling&Advancement In Research Trends (SMART). pp. 63-67 (2016)
work page 2016
-
[5]
Cheng, Y ., Wang, H., Bao, Y . & Lu, F. Appearance-based gaze estimation with deep learning: A review and benchmark.IEEE Transactions On Pattern Analysis And Machine Intelligence. (2024)
work page 2024
-
[6]
Valliappan, N., Dai, N., Steinberg, E., He, J., Rogers, K., Ramachandran, V ., Xu, P., Shojaeizadeh, M., Guo, L., Kohlhoff, K. & Others Accelerating eye movement research via accurate and affordable smartphone eye tracking.Nature Communications.11, 4553 (2020)
work page 2020
-
[7]
Parker, C., Scott, S. & Geddes, A. Snowball sampling.SAGE Research Methods F oundations. (2019)
work page 2019
Show all 30 references
-
[8]
& Bulling, A
Zhang, X., Sugano, Y . & Bulling, A. Evaluation of appearance-based methods and implications for gaze-based applications.Proceedings Of The 2019 CHI Conference On Human Factors In Computing Systems. pp. 1-13 (2019)
2019
-
[9]
& Hilliges, O
Park, S., Zhang, X., Bulling, A. & Hilliges, O. Learning to find eye region landmarks for remote gaze estimation in unconstrained settings.Proceedings Of The 2018 ACM Symposium On Eye Tracking Research&Applications. pp. 1-10 (2018)
2018
-
[10]
& Bulling, A
Zhang, X., Sugano, Y ., Fritz, M. & Bulling, A. It’s written all over your face: Full-face appearance-based gaze estimation.Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition Workshops. pp. 51-60 (2017)
2017
-
[11]
(2024), https://help.tobii.com/hc/en-us/sections/360001805718-Tobii-EyeX
Tobii Tobii EyeX Eye Tracker. (2024), https://help.tobii.com/hc/en-us/sections/360001805718-Tobii-EyeX
2024
-
[12]
& De V os, M
Strobl, M., Lipsmeier, F., Demenescu, L., Gossens, C., Lindemann, M. & De V os, M. Look me in the eye: evaluating the accuracy of smartphone-based eye tracking for potential application in autism spectrum disorder research.Biomedical Engineering Online.18pp. 1-12 (2019)
2019
-
[13]
& Torralba, A
Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W. & Torralba, A. Eye tracking for everyone.Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 2176-2184 (2016)
2016
-
[14]
& Bulling, A
Bâce, M., Staal, S. & Bulling, A. Accurate and Robust Eye Contact Detection During Everyday Mobile Device Interactions.ArXiv Preprint ArXiv:1907.11115. (2019)
2019 arXiv
-
[15]
Lei, Y ., He, S., Khamis, M. & Ye, J. An end-to-end review of gaze estimation and its interactive applications on handheld mobile devices.ACM Computing Surveys.56, 1-38 (2023)
2023
-
[16]
& Javadi, B
Gunawardena, N., Ginige, J. & Javadi, B. Eye-Tracking Technologies in Mobile Devices Using Edge Comput- ing: A Systematic Review.ACM Computing Surveys (CSUR). (2022)
2022
-
[17]
& Lui, G
Gunawardena, N., Ginige, J., Javadi, B. & Lui, G. Performance analysis of CNN models for mobile device eye tracking with edge computing.Procedia Computer Science.207pp. 2291-2300 (2022)
2022
-
[18]
& Adam, H
Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M. & Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications.ArXiv Preprint ArXiv:1704.04861. (2017)
2017 arXiv
-
[19]
& Sun, J
Zhang, X., Zhou, X., Lin, M. & Sun, J. Shufflenet: An extremely efficient convolutional neural network for mobile devices.Proceedings Of The IEEE Conference On Computer Vision And Pattern Recognition. pp. 6848- 6856 (2018) 18
2018
-
[21]
(2023), https://www.tobii.com/products/discontinued/tobii-pro-nano
Tobii Tobii Pro Nano. (2023), https://www.tobii.com/products/discontinued/tobii-pro-nano
2023
-
[22]
inside out
Swift, A., Liew, S., Weinkle, S., Garcia, J. & Silberberg, M. The facial aging process from the “inside out”. Aesthetic Surgery Journal.41, 1107-1119 (2021)
2021
-
[23]
& Nazri, A
Agbolade, O. & Nazri, A. A two dimensional facial features analysis for gender-based comparison using mor- phometrics approach.International Journal Of Engineering&Technology.7, 214-219 (2018)
2018
-
[24]
& Hiramatsu, C
Paulus, Y ., Remijn, G., Syn, Y . & Hiramatsu, C. The use of glasses during registration into a low-cost eye tracking device under different lighting conditions.2017 2nd International Conference On Automation, Cogni- tive Science, Optics, Micro Electro-Mechanical System, And I...
2017
-
[25]
& Cattin, P
Wyder, S. & Cattin, P. Eye tracker accuracy: quantitative evaluation of the invisible eye center location.Interna- tional Journal Of Computer Assisted Radiology And Surgery.13pp. 1651-1660 (2018)
2018
-
[26]
& Kruijff, E
Marquardt, A., Steininger, M., Trepkowski, C., Weier, M. & Kruijff, E. Selection Performance and Reliability of Eye and Head Gaze Tracking Under Varying Light Conditions.2024 IEEE Conference Virtual Reality And 3D User Interfaces (VR). pp. 546-556 (2024)
2024
-
[27]
& Hinton, G
Krizhevsky, A., Sutskever, I. & Hinton, G. Imagenet classification with deep convolutional neural networks. Advances In Neural Information Processing Systems.25(2012)
2012
-
[28]
& Javadi, B
Gunawardena, N., Lui, G., Ginige, J. & Javadi, B. Smartphone-based eye tracking system using edge intelligence and model optimisation.Internet Of Things.29pp. 101481 (2025)
2025
-
[29]
& Alt, F
Le, T., Dietz, F., Pfeuffer, K. & Alt, F. A Practical Method to Eye-tracking on the Phone: Toolkit, Accuracy and Precision.Proceedings Of The 21st International Conference On Mobile And Ubiquitous Multimedia. pp. 182-188 (2022)
2022
-
[30]
& Cui, H
Zhang, L. & Cui, H. Reliability of MUSE 2 and Tobii Pro Nano at capturing mobile application users’ real-time cognitive workload changes.Frontiers In Neuroscience.16pp. 1011475 (2022) 19
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.