REVIEW 4 major objections 5 minor 25 references
FlatTrack: Eye-tracking with ultra-thin lensless cameras
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A flat, mask-based lensless camera tracks gaze on par with lensed trackers, at over 125 fps.
desk verdict First real lensless near-eye gaze dataset and a sound real-data evaluation; the 'parity with lensed' claim rests on simulation and should be tempered. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the PhlatCam phase-mask camera: a thin phase mask placed slightly under 1.5 mm from the sensor, designed for 700 nm near-infrared light, produces a point-spread function $P$ such that the measurement is $Y = P \ast X + N$, with the scene $X$ globally multiplexed. Because the mask-to-sensor distance is tiny, the camera has a very large depth of field, so it can image the eye from roughly 4 cm away without a focusing element. The argument is carried by a two-stage pipeline: FlatNet, a learned reconstruction network trained on simulated measurements of natural images, reconstructs a visible eye image from the lensless capture; then ResNet-18 regresses a 3D gaze unit vector, projected to 2D monitor pixels where an $\ell^1$ loss is applied. Subject-specific fine-tuning of the last two layers of the regressor handles the user calibration that real AR/VR deployments need.
What would settle it
Set up a rig with a lensed camera and the PhlatCam imaging the same eye from the same 4 cm distance on the same 15 by 15 gaze grid with the same illumination, and compare per-subject angular errors. If the real lensless error is substantially larger than the simulated 1.62 to 1.84 degrees and no longer sits within the lensed error band of roughly 1.6 to 1.8 degrees, the paper's central on-par claim fails. A cheaper check is to capture FlatTrack-style images with two NIR illuminators placed to cover the dark corners identified in Section 6.5 and see whether the average error drops below 1.92 degrees.
Extended reading notes
Core claim
The central claim is that a thin phase-mask lensless camera can replace a conventional lensed camera for near-eye gaze estimation without a meaningful accuracy penalty. The paper reports that on its own FlatTrack dataset, the FlatNet plus ResNet-18 pipeline achieves 1.92 degrees average angular error (best subject 0.91 degrees) with inference at 7.81 ms, i.e., over 125 fps, and that a 5 percent accuracy gain from the EyeCOD method is not worth its 3x inference cost. To support the parity claim, the paper simulates lensless captures from the lensed Davis-GS dataset by convolving with the PhlatCam PSF and adding noise, then reconstructs with FlatNet; across three test subjects the lensed error is 1.79, 1.72, and 1.67 degrees while the lensless error is 1.84, 1.81, and 1.62 degrees. The paper interprets this as minimal difference, indicating that lensless imaging does not introduce significant performance loss while enabling a compact form factor.
Load-bearing premise
The load-bearing premise is that the simulated lensless images used for the lensed-versus-lensless comparison, Davis-GS lens images convolved with the PhlatCam PSF plus noise, faithfully represent what a real PhlatCam would record; real captures include uneven illumination, sensor noise correlations, head motion, and calibration drift that the simulation omits.
Editorial extensions
If this is right
- AR/VR headsets can place the eye-tracking camera inside the eyeglasses frame rather than at a lens's focusing distance, cutting thickness and weight.
- At over 125 fps with 7.81 ms inference, the pipeline is fast enough for interactive gaze-based interfaces and foveated rendering.
- The 20,475-capture dataset gives the lensless-gaze community a real-data training and benchmarking target instead of relying only on simulated lensless images.
- User-specific fine-tuning with a small calibration set keeps accuracy near 1 to 2 degrees without per-subject retraining from scratch.
- The parity result predicts that progress on lensed gaze trackers can transfer to lensless hardware through the same reconstruction-plus-regression pipeline.
Reading between the lines
- The paper does not collect a real lensed-versus-lensless comparison on the same rig; its parity numbers come from simulated lensless Davis-GS images, so a same-setup real comparison could confirm or narrow the claimed parity.
- Section 6.5 attributes high-error grid points to dark reconstructions with poor pupil contrast, so adding NIR illuminators aimed at those corners is a direct, testable fix that could push the 1.92 degree average lower.
- The globally multiplexed raw measurements suggest a privacy property beyond the form-factor win, but the paper does not quantify how much harder identity or screen-content reconstruction becomes from FlatTrack data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FlatTrack, a near-eye gaze tracking system built from an NIR PhlatCam lensless camera and a two-stage pipeline: lensless reconstruction (Wiener deconvolution or FlatNet) followed by a gaze regression network (ResNet-18, EyeCOD, MobileNetV2). The authors collect a 20,475-capture dataset from 13 subjects with calibrated gaze labels, evaluate per-subject held-out rounds, report a best average angular error of 1.92 degrees for FlatNet plus ResNet-18 at over 125 fps, and present a simulated lensed-versus-lensless comparison on the Davis-GS dataset that shows roughly a 0.1 degree difference. The central claim is that lensless eye tracking performs on par with conventional lens-based trackers while enabling a flatter and more compact form factor.
Significance. If the parity claim holds, the contribution is significant: it demonstrates that a mask-based lensless camera placed about 4 cm from the eye can support usable gaze estimation, potentially removing the focusing-distance constraint in AR/VR headsets. The paper's concrete strengths include a new real lensless gaze dataset (the first of its kind, to the authors' knowledge), a sound per-subject held-out evaluation protocol for the real-data experiments, comparisons of two reconstruction methods and three regressors, and clearly reported inference-speed measurements. The significance is tempered by the fact that the headline parity result rests on a simulation whose fidelity to real captures is not established, and the dataset and code are promised only upon acceptance.
major comments (4)
- [Sec. 6.4 / Abstract] The abstract's statement that the proposed system 'performs on par with conventional lens-based trackers' is supported only by the simulated Davis-GS comparison in Sec. 6.4, where lensless measurements are generated via Y = P*X + N (Eq. 1) with a single PSF and added noise. The real FlatTrack experiments (Secs. 6.2-6.3) achieve 1.92 degrees average angular error but include no lensed baseline on the same setup, subjects, and protocol. The simulation omits spatially varying illumination, correlated sensor noise, head motion, and calibration drift, which Sec. 6.5 identifies as error sources in the real captures. The parity claim therefore needs a same-setup real lensed-versus-lensless comparison, or the claim should be restricted to the simulated setting.
- [Table 3 / Sec. 6.4] The lensed-versus-lensless comparison is reported for only three subjects, with differences of 0.02-0.09 degrees and no confidence intervals or per-subject variance. With n=3, the claim of 'minimal difference' is statistically fragile, especially because Subject 27 is slightly better in the lensless condition. The text also says 21 of the 27 Davis-GS subjects are used for pre-training and 'the remaining 3' for fine-tuning and evaluation, leaving 3 subjects unaccounted for. Please report variability over held-out rounds, include more test subjects, and clarify the subject allocation.
- [Sec. 6.5 / Sec. 6.4] Sec. 6.5 shows that the largest real-data gaze errors occur at grid points where reconstructed images are degraded by 'harsher illumination' and 'lack of proper lighting.' These effects are not represented in the Sec. 6.4 simulation, which adds only synthetic noise to PSF-convolved images. Consequently, the simulated lensed-versus-lensless gap of about 0.1 degrees may understate the real gap. The paper should quantify how many of the 225 grid points fall into the high-error category and, ideally, evaluate the system under controlled illumination to separate lighting artifacts from lensless-specific degradation.
- [Sec. 6.1 / Sec. 6.4] The validity of the Sec. 6.4 simulation depends on the PSF and noise model used to synthesize lensless inputs, but the manuscript does not state whether the PSF is measured from the physical PhlatCam prototype or simulated, nor how the added noise level was set to match real captures. Since the same frozen FlatNet pretrained on simulated MIRFLICKR measurements is used for reconstruction, a PSF or noise mismatch could bias the simulated comparison in favor of the proposed pipeline. Please provide the PSF calibration procedure, the noise-level justification, and a quantitative comparison of simulated and real lensless captures (e.g., residual statistics or paired examples).
minor comments (5)
- [Sec. 6.2 / Table 1] The manuscript uses both 'Wiener' and 'Weiner' spellings; please standardize to 'Wiener' throughout.
- [Table 2] The 'best-case error' metric reports only the best subject's held-out error, which is not a standard evaluation statistic; please report per-subject errors or a distribution instead of, or in addition to, the best-case value.
- [Sec. 6.4] The paper should clarify that the 'lensed' condition comes from the Davis-GS dataset and describe the exact capture device for those grayscale frames; the current text reads as if a conventional lens camera and the PhlatCam were compared directly, which is not the case.
- [Abstract / Sec. 5] The abstract describes the algorithm as 'co-designed,' but Sec. 5 freezes the reconstruction stage and trains only the gaze regressor; please either demonstrate co-design (e.g., end-to-end training or joint optimization) or reword the claim.
- [Sec. 4 / Fig. 3] The text gives the vertical stimulus spacing as 66.3 pixels and the vertical field of view as 29.6 degrees, while the Fig. 3 caption says 65.3 pixels and 29.80 degrees; please reconcile these numbers.
Circularity Check
No circularity: the gaze regressor is trained on held-out real labels, and the lensed-vs-lensless simulation is a benchmark rather than a derivation that reduces to its own inputs.
full rationale
The paper's derivation chain is self-contained. Lensless captures are modeled as Y = P*X + N (Eq. 1), reconstructions are obtained via a frozen pretrained FlatNet or Wiener deconvolution (Eqs. 2-4), and gaze is trained with an L1 loss against calibrated held-out labels (Eq. 5). The reported 1.92-degree average angular error is measured on a held-out test round per subject after subject-specific fine-tuning; it is not a fitted parameter renamed as a prediction. The lensed-vs-lensless comparison in Sec 6.4 does use the authors' own PSF and FlatNet to synthesize lensless images from Davis-GS, but this is an external benchmark simulation, not a reduction of the conclusion to its inputs: the gaze regressor is trained and evaluated on these synthesized data, and the comparison could in principle have favored either imaging system. The use of FlatNet [13], a prior publication by some of the same authors, is an invocation of an externally published, frozen reconstruction tool rather than a load-bearing self-citation; the paper does not claim to derive FlatNet from the present dataset. Limitations documented in Sec 6.5, such as uneven illumination and dark images lacking pupil resolution, are correctness concerns about real-data generalization of the simulated parity claim, not circularity. No exhibited step makes the output equivalent to its input by construction, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Wiener regularization coefficient gamma =
1e-5
- Training hyperparameters =
LR 0.0001, weight decay 0.0005, 50 epochs, StepLR decay 0.5 every 5 epochs
- Affine augmentation ranges =
not specified
assumptions (4)
- domain assumption Lensless forward model Y = P * X + N with an accurate PSF P (Eq. 1).
- domain assumption FlatNet pretrained on MIRFLICKR natural images transfers to near-eye eye images.
- domain assumption Ground-truth gaze vectors from a 15x15 monitor grid at 50 cm are accurate.
- ad hoc to paper Simulated lensless Davis-GS images represent real lensless captures.
Cite this review
Pith. "Pith review of FlatTrack: Eye-tracking with ultra-thin lensless cameras." pith.science (2026). https://pith.science/paper/VTKS35YE
@misc{pith2026250115450,
author = {Pith},
title = {Pith review of: FlatTrack: Eye-tracking with ultra-thin lensless cameras},
year = {2026},
howpublished = {\url{https://pith.science/paper/VTKS35YE}},
note = {Machine review of arXiv:2501.15450}
}
read the original abstract
Existing eye trackers use cameras based on thick compound optical elements, necessitating the cameras to be placed at focusing distance from the eyes. This results in the overall bulk of wearable eye trackers, especially for augmented and virtual reality (AR/VR) headsets. We overcome this limitation by building a compact flat eye gaze tracker using mask-based lensless cameras. These cameras, in combination with co-designed lightweight deep neural network algorithm, can be placed in extreme close proximity to the eye, within the eyeglasses frame, resulting in ultra-flat and lightweight eye gaze tracker system. We collect a large dataset of near-eye lensless camera measurements along with their calibrated gaze directions for training the gaze tracking network. Through real and simulation experiments, we show that the proposed gaze tracking system performs on par with conventional lens-based trackers while maintaining a significantly flatter and more compact form-factor. Moreover, our gaze regressor boasts real-time (>125 fps) performance for gaze tracking.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Manufacturing application-driven foveated near-eye displays
Kaan Aks ¸it, Praneeth Chakravarthula, Kishore Rathinavel, Youngmo Jeong, Rachel Albert, Henry Fuchs, and David Luebke. Manufacturing application-driven foveated near-eye displays. IEEE transactions on visualization and computer graphics, 25(5):1928–1939, 2019. 1
work page 1928
-
[2]
Hadi Alzayer, Kevin Zhang, Brandon Feng, Christopher A. Metzler, and Jia-Bin Huang. Seeing the world through your eyes. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 4864– 4873, June 2024. 2
work page 2024
-
[3]
Anastasios N. Angelopoulos, Julien N.P. Martel, Amit P. Kohli, Jorg Conradt, and Gordon Wetzstein. Event-based near-eye gaze tracking beyond 10,000 hz.IEEE Transactions on Visualization and Computer Graphics, 27(5):2577–2586, May 2021. 3, 6
work page 2021
-
[4]
Diffusercam: lensless single-exposure 3d imaging
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. Diffusercam: lensless single-exposure 3d imaging. Optica, 5(1):1, Dec
-
[5]
Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G
M. Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G. Baraniuk. Flat- cam: Thin, lensless cameras using coded aperture and com- putation. IEEE Transactions on Computational Imaging , 3(3):384–397, 2017. 2, 3
work page 2017
-
[6]
Flatnet3d: intensity and ab- solute depth from single-shot lensless capture
Dhruvjyoti Bagadthey, Sanjana Prabhu, Salman S Khan, D Tony Fredrick, Vivek Boominathan, Ashok Veeraragha- van, and Kaushik Mitra. Flatnet3d: intensity and ab- solute depth from single-shot lensless capture. JOSA A , 39(10):1903–1912, 2022. 2
work page 1903
-
[7]
Phlatcam: Designed phase-mask based thin lensless camera
Vivek Boominathan, Jesse K Adams, Jacob T Robinson, and Ashok Veeraraghavan. Phlatcam: Designed phase-mask based thin lensless camera. IEEE transactions on pattern analysis and machine intelligence , 42(7):1618–1629, 2020. 2, 3
work page 2020
-
[8]
Aayush K. Chaudhary, Rakshit Kothari, Manoj Acharya, Shusil Dangi, Nitinraj Nair, Reynold Bailey, Christopher Kanan, Gabriel Diaz, and Jeff B. Pelz. Ritnet: Real-time semantic segmentation of the eye for gaze tracking. In 2019 IEEE/CVF International Conference on Computer Vi- sion Workshop (ICCVW). IEEE, Oct. 2019. 6
work page 2019
Show all 25 references
-
[9]
Foveated 3d graphics.ACM Trans
Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. Foveated 3d graphics.ACM Trans. Graph., 31(6), nov 2012. 1
2012
-
[10]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2016. 5, 6
2016
-
[11]
The mir flickr retrieval evaluation
Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In Proceedings of the 1st ACM international con- ference on Multimedia information retrieval , pages 39–43,
-
[12]
Foveated rendering: Motivation, taxon- omy, and research directions, 2022
Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. Foveated rendering: Motivation, taxon- omy, and research directions, 2022. 1
2022
-
[13]
Flatnet: Towards photorealistic scene reconstruction from lensless measure- ments
Salman Siddique Khan, Varun Sundar, Vivek Boominathan, Ashok Veeraraghavan, and Kaushik Mitra. Flatnet: Towards photorealistic scene reconstruction from lensless measure- ments. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–1, 2020. 3, 4, 7
2020
-
[14]
Opencam: Lensless optical encryption camera
Salman S Khan, Xiang Yu, Kaushik Mitra, Manmo- han Chandraker, and Francesco Pittaluga. Opencam: Lensless optical encryption camera. arXiv preprint arXiv:2312.01077, 2023. 2
2023 arXiv
-
[15]
Nvgaze: An anatomically-informed dataset for low-latency, near-eye gaze estimation
Joohwan Kim, Michael Stengel, Alexander Majercik, Shalini De Mello, David Dunn, Samuli Laine, Morgan McGuire, and David Luebke. Nvgaze: An anatomically-informed dataset for low-latency, near-eye gaze estimation. In Proceedings of the 2019 CHI conference on human factors in com...
2019
-
[16]
Eye tracking for everyone
Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kan- nan, Suchendra Bhandarkar, Wojciech Matusik, and Anto- nio Torralba. Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 2176–2184, 2016. 3
2016
-
[17]
Spectral diffusercam: lensless snapshot hy- perspectral imaging with a spectral filter array
Kristina Monakhova, Kyrollos Yanny, Neerja Aggarwal, and Laura Waller. Spectral diffusercam: lensless snapshot hy- perspectral imaging with a spectral filter array. Optica, 7(10):1298–1307, Oct 2020. 2
2020
-
[18]
Openeds2020: Open eyes dataset
Cristina Palmero, Abhishek Sharma, Karsten Behrendt, Kapil Krishnakumar, Oleg V Komogortsev, and Sachin S Talathi. Openeds2020: Open eyes dataset. arXiv preprint arXiv:2005.03876, 2020. 3, 6
2005 arXiv
-
[19]
Towards foveated rendering for gaze-tracked virtual reality
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG), 35(6):1–12, 2016. 1
2016
-
[20]
Lindeman, and Mark Billinghurst
Thammathip Piumsomboon, Gun Lee, Robert W. Lindeman, and Mark Billinghurst. Exploring natural eye-gaze-based in- teraction for immersive virtual reality. In 2017 IEEE Sym- posium on 3D User Interfaces (3DUI) , pages 36–39, 2017. 1
2017
-
[21]
Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. IEEE, June 2018. 5, 6
2018
-
[22]
Gaze locking: passive eye contact detection for human- object interaction
Brian A Smith, Qi Yin, Steven K Feiner, and Shree K Na- yar. Gaze locking: passive eye contact detection for human- object interaction. In Proceedings of the 26th annual ACM symposium on User interface software and technology, pages 271–280, 2013. 3
2013
-
[23]
Eyecod: eye tracking system acceleration via flatcam-based algorithm and accel- erator co-design
Haoran You, Cheng Wan, Yang Zhao, Zhongzhi Yu, Yong- gan Fu, Jiayi Yuan, Shang Wu, Shunyao Zhang, Yon- gan Zhang, Chaojian Li, Vivek Boominathan, Ashok Veer- araghavan, Ziyun Li, and Yingyan Lin. Eyecod: eye tracking system acceleration via flatcam-based algorithm and accel- e...
2022
-
[24]
Eye tracking based control system for natural human-computer interaction
Xuebai Zhang, Xiaolong Liu, Shyan-Ming Yuan, and Shu- Fan Lin. Eye tracking based control system for natural human-computer interaction. Computational intelligence and neuroscience, 2017, 2017. 1
2017
-
[25]
Appearance-based gaze estimation in the wild
Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520, 2015. 3
2015
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.