Pith. sign in

REVIEW 4 major objections 3 minor 20 references

Gaze2AOI: Open Source Deep-learning Based System for Automatic Area of Interest Annotation with Eye Tracking Data

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents Gaze2AOI, an open-source tool that uses YOLOv8 object tracking to automatically detect and label areas of interest in video and pairs them with eye-tracking data to compute dwell time, time to first fixation, and…

desk verdict A plausible tool description with no code and no validation; the core claims are currently unverifiable but worth testing through review. read the letter →

arxiv 2411.13346 v1 pith:HYE6SOXV submitted 2024-11-20 cs.SE

Karolina Trajkovska , Matjaž Kljun , Klen Čopič Pucihar This is my paper · ORCID
classification cs.SE
keywords eyetrackingareaofinterestAOIannotationYOLOv8objectgazeanalysisdwelltimeopen-sourcetool
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Gaze2AOI claims to remove the most labor-intensive step in video-based eye-tracking analysis: manually drawing and labeling areas of interest (AOIs) frame by frame. The paper describes an open-source system that runs YOLOv8 object detection and tracking over a video, automatically labels detected objects as AOIs, and aligns them with gaze and fixation data by frame number. From that alignment it computes time to first fixation, dwell time, and AOI revisit frequency. The authors state that, to their knowledge, no existing open-source tool automates this AOI annotation and metric pipeline, which is the paper's central contribution. A sympathetic reader would care because accurate AOI metrics are the standard way to say what people actually looked at, and making the pipeline automatic would let attention studies scale to long, dynamic videos.

What carries the argument

The load-bearing mechanism is the coupling of YOLOv8's per-frame object detection with eye-tracking data through a shared frame number. YOLOv8 provides bounding boxes and class labels for around 600 object types; the tool's tracker maintains object identity across frames; and the frame-number join lets the system decide, for any fixation, which AOI bounding box contains it. Around this core sit two supporting mechanisms: a CLI option to precompute tracking results as a CSV so the analysis can be rerun without re-detecting, and a key-frame selection routine that presents only frames with a changed set of detected objects to speed up manual labeling.

What would settle it

Take a video-and-eye-tracking dataset with ground-truth AOI labels and object classes outside or on the boundary of the ~600 YOLOv8 classes, run Gaze2AOI, and compare its TTFF, dwell time, and revisit counts with the ground truth; systematic misses or label errors on those objects would refute the claim that automatic annotation is sufficient for arbitrary study videos.

Watch

Extended reading notes

Core claim

The central claim is that a single open-source tool can replace manual AOI annotation in post-hoc eye-tracking video analysis. The system detects objects with a pre-trained YOLOv8 model supporting roughly 600 object classes, tracks them across frames with the ByteTrack tracker, and writes per-frame predictions that record, for every detected object, whether it is present, gazed upon, or contains the fixation point. Because predictions are keyed to frame numbers, they can be joined directly to eye-tracking samples. From this join the tool derives the standard AOI metrics: time to first fixation, dwell duration, and number of revisits. The paper further claims that this makes it the first open-source tool to offer automated AOI annotation and eye-tracking integration, and it supports custom labeling so users can correct false detections or add semantic names (for example, a person's name) to detected objects.

Load-bearing premise

Everything the tool reports depends on the pre-trained object detector recognizing, in the researcher's video, every object that is semantically an AOI; the paper does not validate detection accuracy on any eye-tracking video, so a missed detection silently makes gaze at that object invisible to all computed metrics.

Editorial extensions

If this is right

  • Attention studies on video can bypass manual frame-by-frame ROI drawing for any AOI that is one of the ~600 detectable object classes.
  • Standard metrics -- time to first fixation, dwell time, and revisit count -- can be computed directly from the frame-joined detections without bespoke analysis scripts.
  • Because the tool accepts interchangeable YOLO models, research teams can swap in a model trained on their own object classes and retain the same AOI-metric pipeline.
  • The custom labeling step gives users a correction path, so imperfect automatic detections can be refined rather than discarded.
  • Precomputing predictions and skipping frames without gaze data makes long recordings practical to process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own claims imply that any gaze fixated on an object the pre-trained model cannot detect will be silently excluded from every AOI metric; users should audit detection recall before trusting TTFF and dwell values.
  • The frame-number join assumes the eye tracker's timestamps map cleanly onto video frames; a one-frame misalignment would shift TTFF and dwell estimates by one frame period, which matters for fast-cut content.
  • Extending the same pipeline to streaming, real-time analysis would let the tool move from post-hoc behavioral analysis to live adaptive content, a direction the paper lists but does not implement.
  • Comparing Gaze2AOI's output against manually annotated ground truth on a benchmark of dynamic videos would quantify how much accuracy the automation costs, which the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The manuscript describes Gaze2AOI, a proposed open-source system that automatically detects and labels areas of interest (AOIs) in video using an Ultralytics YOLOv8 model pre-trained on OpenImagesv7, integrates the detections with eye-tracking data, and computes metrics such as time to first fixation, dwell duration, and revisit frequency. The paper presents the system architecture, user interface, tracking workflow, and a custom labelling feature for correcting or refining YOLO outputs. The central claim is that Gaze2AOI is the first open-source tool to automate AOI annotation and gaze-metric computation for arbitrary eye-tracking videos, but the manuscript provides no repository link, no evaluation dataset, no accuracy measurements, and no comparison with manual AOI annotation.

Significance. If the system worked as described and were actually released, it would address a real bottleneck in eye-tracking research, where manual AOI annotation is labour-intensive. The paper has some useful design elements: the use of a public pre-trained detector, the ByteTrack tracker, modular model swapping, pre-computed predictions stored as CSV, and a key-frame extraction scheme for efficient manual correction. However, the scientific contribution hinges on two unverified premises: that the tool is genuinely available as open-source software, and that its automatic detections are reliable enough that the derived gaze metrics are meaningful. Neither premise is supported by evidence in the manuscript. Because the central claim is a working, validated tool and the paper supplies neither validation nor a verifiable artifact, the significance cannot be assessed beyond the proposal level.

major comments (4)
  1. [Section 3.2] The manuscript does not report any evaluation of the object detection and tracking component. There is no dataset, no accuracy or precision/recall measurement, no comparison with manual AOI annotation, and no error analysis. Since every derived metric (TTFF, dwell time, revisit frequency) is computed from the YOLOv8 bounding boxes, the correctness of the entire tool rests on detection reliability, which is never demonstrated.
  2. [Section 3.2] The connection between eye-tracking data and video frames is described only as "associated with the eye-tracking data based on the frame number." The paper does not explain how gaze samples are synchronized to video timecodes, how gaze coordinates are mapped to pixel coordinates, how fixations are detected, or how a gaze point is determined to fall inside a bounding box. Without this information, the reported metrics cannot be reproduced or audited, and the system's validity is unverifiable.
  3. [Abstract and Section 2] The paper repeatedly states that the tool will be made available as open-source software and claims it is the first such open-source tool, but no repository, URL, or archival identifier is provided anywhere in the manuscript. The central novelty claim is therefore unverifiable, and the claimed contribution to the community cannot be assessed.
  4. [Section 3.3 and Figure 2(b)] The paper itself demonstrates a false positive detection in which an oven is recognized as a window, and states that users must manually relabel such cases. This is not inherently disqualifying, but the manuscript does not quantify how often such corrections are needed, how many classes are affected, or whether the key-frame selection algorithm reliably surfaces all misdetections. Without this information, the 'automatic annotation' claim and the implied labour savings are not substantiated.
minor comments (3)
  1. [Introduction] The text contains typographical errors, including "predominantlyperformedmanually" and "where new a new object is detected." These should be corrected.
  2. [Figure 1] The caption refers to part (b) as the system architecture, but the body text does not describe the architecture diagram; a short explanation of the components and data flow would help readers understand the system.
  3. [Section 3.2] The phrase "YOLOv8 object tracking algorithm" is imprecise; YOLOv8 is a detection model, and tracking is performed separately by ByteTrack or BoT-SORT. Clarifying this boundary would improve the technical accuracy of the description.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the system's core detection and metrics are external to the paper's own claims, and the only self-citations are context-setting rather than load-bearing.

full rationale

The paper derives no mathematical results and fits no parameters to its own outputs. The central pipeline uses an externally pre-trained YOLOv8 model sourced from Ultralytics and trained on OpenImagesv7, so the object detections come from an independent, published model rather than from anything defined by the paper. The AOI metrics (TTFF, dwell time, revisits) are computed by associating those detections with eye-tracking data based on frame number; these are standard operational definitions, not predictions that reduce to fitted inputs. The self-citations [10,9] simply state that the tool was initially developed for analysing an interactive documentary; they do not justify the tool's validity or the novelty claim. The novelty claim about absence of open-source tools is a literature statement with no circular derivation. The paper's weaknesses (missing repository, no accuracy evaluation, admitted false positives in Figure 2b) are correctness risks, not circularity. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters, and the central claims rest on domain assumptions about the YOLO pre-trained model, frame synchronization, and novelty of the tool, none of which are empirically validated.

assumptions (3)
  • domain assumption Pre-trained YOLOv8 model on OpenImagesv7 detects and tracks objects suitable for AOI annotation across eye-tracking videos.
    The system relies entirely on this pre-trained model for AOI generation (Section 3.2); no validation or accuracy measurements are provided for eye-tracking video content.
  • domain assumption Frame numbers provide a correct synchronization between eye-tracking data and video frames.
    The paper states 'The outcomes are associated with the eye-tracking data based on the frame number' (Section 3.2). Any timing or frame mismatch would invalidate the computed metrics.
  • domain assumption No open-source tool currently automates AOI annotation, as claimed in the introduction.
    The novelty claim rests on a limited review of related work (Section 2), not a comprehensive survey; if an existing open-source tool is missed, the main contribution is weakened.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gaze2AOI: Open Source Deep-learning Based System for Automatic Area of Interest Annotation with Eye Tracking Data." pith.science (2026). https://pith.science/paper/HYE6SOXV

@misc{pith2026241113346,
  author       = {Pith},
  title        = {Pith review of: Gaze2AOI: Open Source Deep-learning Based System for Automatic Area of Interest Annotation with Eye Tracking Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HYE6SOXV}},
  note         = {Machine review of arXiv:2411.13346}
}
read the original abstract

Eye gaze is considered an important indicator for understanding and predicting user behaviour, as well as directing their attention across various domains including advertisement design, human-computer interaction and film viewing. In this paper, we present a novel method to enhance the analysis of user behaviour and attention by (i) augmenting video streams with automatically annotating and labelling areas of interest (AOIs), and (ii) integrating AOIs with collected eye gaze and fixation data. The tool provides key features such as time to first fixation, dwell time, and frequency of AOI revisits. By incorporating the YOLOv8 object tracking algorithm, the tool supports over 600 different object classes, providing a comprehensive set for a variety of video streams. This tool will be made available as open-source software, thereby contributing to broader research and development efforts in the field.

Figures

Figures reproduced from arXiv: 2411.13346 by the authors.

Figure 1
Figure 1. Gaze2AOI: (a) User interface for selecting object classes and performing object tracking; (b) System architecture. arXiv:2411.13346v1 [cs.SE] 20 Nov 2024 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Customised Labelling: (a) depicts a true positive AOI detection with an asso￾ciated user-defined label; (b) illustrates a false positive AOI detection with a corrected user-defined label. Gaze2AOI offers capability to complement the automatically generated AOI by providing additional labels to every AOI. This feature is useful when YOLO prediction is inaccurate or where a more specific semantic of AOIs is required. … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages

  1. [1]

    In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces

    Barz, M., Bhatti, O.S., Alam, H.M.T., Nguyen, D.M.H., Sonntag, D.: Interac- tive fixation-to-aoi mapping for mobile eye tracking data based on few-shot image classification. In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. p. 175–178. IUI ’23 Companion, Association for Computing Machinery, New York, NY, USA (2...

    arXiv 2023

  2. [2]

    In: ACM Symposium on Eye Tracking Research and Applications

    Barz, M., Kapp, S., Kuhn, J., Sonntag, D.: Automatic recognition and augmen- tation of attended objects in real-time using eye tracking and a head-mounted display. In: ACM Symposium on Eye Tracking Research and Applications. pp. 1–4 (2021)

    work page 2021

  3. [3]

    on the motions of the eye, in illustration of the uses of the muscles and nerves of the orbit

    Bell, C.: Xv. on the motions of the eye, in illustration of the uses of the muscles and nerves of the orbit. Philosophical Transactions of the Royal Society of London (113), 166–186 (1823)

    work page

  4. [4]

    In: Eurovis (stars)

    Blascheck, T., Kurzhals, K., Raschke, M., Burch, M., Weiskopf, D., Ertl, T.: State- of-the-art of visualization for eye tracking data. In: Eurovis (stars). p. 29 (2014)

    work page 2014

  5. [5]

    International Journal of Psychophysiology155, 49–62 (2020)

    Carter, B.T., Luke, S.G.: Best practices in eye tracking research. International Journal of Psychophysiology155, 49–62 (2020)

    work page 2020

  6. [6]

    arXiv preprint arXiv:2401.17270 (2024)

    Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., Shan, Y.: Yolo-world: Real-time open-vocabulary object detection. arXiv preprint arXiv:2401.17270 (2024)

    arXiv 2024

  7. [7]

    In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recog- nition (FG 2018)

    Datta, S., Sharma, G., Jawahar, C.: Unsupervised learning of face representations. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recog- nition (FG 2018). pp. 135–142. IEEE (2018) 8 K. Trajkovska et al

    work page 2018

  8. [8]

    In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

    De Beugher, S., Ichiche, Y., Brône, G., Goedemé, T.: Automatic analysis of eye- tracking data using object detection algorithms. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing. p. 677–680. UbiComp ’12, Association for Computing Machinery, New York, NY, USA (2012).https://doi.org/10.1145/ 2370216.2370363, https://doi.org/10.1145/2370216.2370363

    arXiv 2012

Show all 20 references
  1. [9]

    International Journal of Human–Computer Interaction 38(10), 949–972 (2022)

    Ducasse, J., Kljun, M., Attygalle, N.T., Pucihar, K.Č.: Interactive web documen- taries: A case study of video viewing behaviour on iotok. International Journal of Human–Computer Interaction 38(10), 949–972 (2022)

    2022

  2. [10]

    International Journal of Human–Computer Interaction36(16), 1558–1584 (2020)

    Ducasse, J., Kljun, M., Čopič Pucihar, K.: Interactive web documentaries: A case study of audience reception and user engagement on iotok. International Journal of Human–Computer Interaction36(16), 1558–1584 (2020)

    2020

  3. [11]

    In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Ap- plications

    Duchowski, A.T., Gehrer, N.A., Schönenberg, M., Krejtz, K.: Art facing science: Artistic heuristics for face detection: tracking gaze when looking at faces. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Ap- plications. ETRA ’19, Association for Computing...

    2019

  4. [12]

    In: Proceedings of the Symposium on Eye Tracking Research and Applications

    Fehringer, B.C.O.F.: Eye tracking gaze visualiser: eye tracker and experimental software independent visualisation of gaze data. In: Proceedings of the Symposium on Eye Tracking Research and Applications. p. 259–262. ETRA ’14, Association for Computing Machinery, New York, NY,...

    2014

  5. [13]

    IJCV (2020)

    Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Ka- mali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., Ferrari, V.: The open images dataset v4: Unified image classification, object detection, and visual rela- tionship detection at sca...

    2020

  6. [14]

    https://storage

    Open Images: Open Images V7 — storage.googleapis.com. https://storage. googleapis.com/openimages/web/index.html (2022)

    2022

  7. [15]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)

    2016

  8. [16]

    Ultralytics: Home — docs.ultralytics.com.https://docs.ultralytics.com/, [Ac- cessed 15-04-2024]

    2024

  9. [17]

    Oxford University Press (2005)

    Wade, N., Tatler, B.W.: The moving tablet of the eye: The origins of modern eye movement research. Oxford University Press (2005)

    2005

  10. [18]

    Journal of Eye Movement Research11(6) (2018)

    Wolf, J., Hess, S., Bachmann, D., Lohmeyer, Q., Meboldt, M.: Automating areas of interest analysis in mobile eye tracking experiments based on machine learning. Journal of Eye Movement Research11(6) (2018)

    2018

  11. [19]

    Multimedia Tools and Applications79, 23729– 23791 (2020)

    Xiao, Y., Tian, Z., Yu, J., Zhang, Y., Liu, S., Du, S., Lan, X.: A review of object detection based on deep learning. Multimedia Tools and Applications79, 23729– 23791 (2020)

    2020

  12. [20]

    Ieee Access6, 49056–49066 (2018)

    Zhang, X., Yuan, S.M., Chen, M.D., Liu, X.: A complete system for analysis of video lecture based on eye tracking. Ieee Access6, 49056–49066 (2018)

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.