{"id":"20bcc82c-c828-42db-9d90-403f8db295dc","arxiv_id":"2411.13346","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Gaze2AOI is a proposed open-source system that automatically labels objects in video frames with YOLOv8 and merges the labels with eye tracking data to compute dwell time, revisits, and time to first fixation.","lead":"The paper describes a planned open-source tool that automatically marks areas of interest in videos using YOLOv8 object detection and connects them with eye tracking data to compute attention metrics. A generalist might read it to see whether a practical, automatic alternative to manual AOI annotation has actually been built and verified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'first open-source tool' claim is unsupported: the paper neither links any repository nor evaluates the YOLOv8-based AOI detection and gaze metrics against manual annotation or any baseline.","rationale":"The reader's verdict REJECT is justified. The paper presents a system without the two pieces of evidence that a tool paper needs: a working publicly accessible artifact and an evaluation showing the tool's outputs are valid. I specifically stress that the absence of any source link is not merely a packaging issue; it makes the central 'open-source' contribution unverifiable. The absence of evaluation is decisive because the metric computations (TTFF, dwell, revisits) depend on the correctness of frame-by-frame object detection and on accurate association between gaze samples and bounding boxes. The manuscript does not even describe this association algorithm, so there is no way to assess whether the metrics are computed correctly. The internal admission of misclassification in Section 3.3 supports the concern that automatic detection cannot be assumed reliable. The paper's novelty claim may turn out to be true, but it is not evidenced by the text. I do not see a need to adjust the reader's verdict; the rejection stands, and the concrete test above would provide the missing validation if the authors later supply the tool and an evaluation.","tokens_in":5854,"tokens_out":2310,"duration_ms":28145,"concrete_test":"Obtain the tool's source code from the authors or a public repository; if no downloadable, freely licensed repository exists, the 'open-source' claim fails. With the code in hand, run Gaze2AOI on a short video (e.g., 30–60 seconds) with recorded gaze data, and compare its automatically generated AOI labels and computed TTFF/dwell/revisit metrics against manual AOI annotation by two or more human annotators. Report per-class detection agreement (e.g., IoU-based mAP or label accuracy) and metric-level correlation. If no such evaluation is provided and the code cannot be obtained, the central claim remains unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two load-bearing premises: (1) that Gaze2AOI is actually available as open-source software, and (2) that a pretrained YOLOv8 detector on OpenImagesv7 can serve as a reliable automatic AOI detector for arbitrary eye-tracking videos, so that TTFF, dwell time, and revisit-frequency metrics are meaningful. Neither premise is supported by evidence in the manuscript. No repository or URL is provided despite repeated statements that the tool 'will be made available as open-source software' (Abstract, Section 2, Section 4). More importantly, Section 3.2 describes only that YOLOv8 predictions are acquired for every frame, but gives no accuracy, no benchmark, no comparison with manual AOI annotation, and no description of how gaze samples are synchronized to video frames or mapped to bounding-box coordinates. Section 3.3 itself admits that detections are frequently wrong (Figure 2b: an oven is mistaken for a window) and that users must manually relabel such cases. This directly weakens the 'automatic annotation' part of the claim: if the automatic output usually needs manual correction, then the claimed labour savings and metric validity are questionable. The tool's open-source status also cannot be verified without a repository, so the novelty claim of being the first open-source tool of its kind is unsubstantiated. These are not internal contradictions, but they are correctness risks: the paper's positive assertions go beyond what is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes Gaze2AOI, a proposed open-source system that automatically detects and labels areas of interest (AOIs) in video using an Ultralytics YOLOv8 model pre-trained on OpenImagesv7, integrates the detections with eye-tracking data, and computes metrics such as time to first fixation, dwell duration, and revisit frequency. The paper presents the system architecture, user interface, tracking workflow, and a custom labelling feature for correcting or refining YOLO outputs. The central claim is that Gaze2AOI is the first open-source tool to automate AOI annotation and gaze-metric computation for arbitrary eye-tracking videos, but the manuscript provides no repository link, no evaluation dataset, no accuracy measurements, and no comparison with manual AOI annotation.","tokens_in":6033,"tokens_out":2647,"duration_ms":32089,"significance":"If the system worked as described and were actually released, it would address a real bottleneck in eye-tracking research, where manual AOI annotation is labour-intensive. The paper has some useful design elements: the use of a public pre-trained detector, the ByteTrack tracker, modular model swapping, pre-computed predictions stored as CSV, and a key-frame extraction scheme for efficient manual correction. However, the scientific contribution hinges on two unverified premises: that the tool is genuinely available as open-source software, and that its automatic detections are reliable enough that the derived gaze metrics are meaningful. Neither premise is supported by evidence in the manuscript. Because the central claim is a working, validated tool and the paper supplies neither validation nor a verifiable artifact, the significance cannot be assessed beyond the proposal level.","major_comments":[{"comment":"The manuscript does not report any evaluation of the object detection and tracking component. There is no dataset, no accuracy or precision/recall measurement, no comparison with manual AOI annotation, and no error analysis. Since every derived metric (TTFF, dwell time, revisit frequency) is computed from the YOLOv8 bounding boxes, the correctness of the entire tool rests on detection reliability, which is never demonstrated.","section":"Section 3.2"},{"comment":"The connection between eye-tracking data and video frames is described only as \"associated with the eye-tracking data based on the frame number.\" The paper does not explain how gaze samples are synchronized to video timecodes, how gaze coordinates are mapped to pixel coordinates, how fixations are detected, or how a gaze point is determined to fall inside a bounding box. Without this information, the reported metrics cannot be reproduced or audited, and the system's validity is unverifiable.","section":"Section 3.2"},{"comment":"The paper repeatedly states that the tool will be made available as open-source software and claims it is the first such open-source tool, but no repository, URL, or archival identifier is provided anywhere in the manuscript. The central novelty claim is therefore unverifiable, and the claimed contribution to the community cannot be assessed.","section":"Abstract and Section 2"},{"comment":"The paper itself demonstrates a false positive detection in which an oven is recognized as a window, and states that users must manually relabel such cases. This is not inherently disqualifying, but the manuscript does not quantify how often such corrections are needed, how many classes are affected, or whether the key-frame selection algorithm reliably surfaces all misdetections. Without this information, the 'automatic annotation' claim and the implied labour savings are not substantiated.","section":"Section 3.3 and Figure 2(b)"}],"minor_comments":[{"comment":"The text contains typographical errors, including \"predominantlyperformedmanually\" and \"where new a new object is detected.\" These should be corrected.","section":"Introduction"},{"comment":"The caption refers to part (b) as the system architecture, but the body text does not describe the architecture diagram; a short explanation of the components and data flow would help readers understand the system.","section":"Figure 1"},{"comment":"The phrase \"YOLOv8 object tracking algorithm\" is imprecise; YOLOv8 is a detection model, and tracking is performed separately by ByteTrack or BoT-SORT. Clarifying this boundary would improve the technical accuracy of the description.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"The manuscript is a system description without empirical validation. The absence of a repository link and the complete lack of an accuracy evaluation prevent the central claims from being checked. If the authors add a benchmark against manual annotation, make the code available, and describe the gaze-to-frame mapping in detail, a substantially revised submission could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a tool description for an eye-tracking AOI annotation pipeline built on YOLOv8. The integrated system is plausible and the open-source intention is good, but the paper gives no way to verify the tool exists and no evidence that the metrics it computes are accurate. As submitted, the main claims are unsupported.\n\nWhat is genuinely here: the authors combine a pre-trained YOLOv8 detector with gaze data mapped to frame numbers, and compute standard AOI metrics (TTFF, dwell, revisits). The UI with class check-boxes and custom labelling is sensible, and the acknowledgment that YOLO produces false positives (Figure 2b shows an oven misdetected as a window) is honest. The related work is reasonable and correctly notes that eyeNotate is not open source.\n\nThe soft spots are large. There is no repository or URL anywhere in the manuscript, despite the abstract and intro repeatedly calling the tool open source. That makes the 'first open-source tool' claim unverifiable. More importantly, there is no validation: no accuracy on detection, no comparison with manual AOI annotation, no description of how gaze samples are synchronized to bounding boxes beyond 'based on the frame number.' If the detector misses an object, the dwell and TTFF metrics for that AOI are silently zero or undefined. The paper itself demonstrates the detection errors, so 'automatic annotation' is really 'automatic suggestion with manual correction,' which may still save time but needs to be measured.\n\nThe math and data are not the issue; there simply is no data. The citation pattern looks fine. The paper is honest about its limitations but does not provide the evidence needed to support its central claims.\n\nWho this is for: eye-tracking researchers who might want to try such a tool. As a publication, it needs major revision. I would send it to peer review rather than desk reject, because the tool is testable and the topic is relevant, but I would expect a referee to demand a public repository and an evaluation against manual annotation or a baseline. As is, I would not cite it.","headline":"A plausible tool description with no code and no validation; the core claims are currently unverifiable but worth testing through review.","tokens_in":6642,"tokens_out":2047,"would_cite":false,"duration_ms":23874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents Gaze2AOI, an open-source tool that uses YOLOv8 object tracking to automatically detect and label areas of interest in video and pairs them with eye-tracking data to compute dwell time, time to first fixation, and…","keywords":["eye tracking","area of interest","AOI annotation","YOLOv8","object tracking","gaze analysis","dwell time","open-source tool"],"falsifier":"Take a video-and-eye-tracking dataset with ground-truth AOI labels and object classes outside or on the boundary of the ~600 YOLOv8 classes, run Gaze2AOI, and compare its TTFF, dwell time, and revisit counts with the ground truth; systematic misses or label errors on those objects would refute the claim that automatic annotation is sufficient for arbitrary study videos.","tokens_in":5610,"feed_emoji":"👁️","tokens_out":5156,"duration_ms":51030,"temperature":0.7,"pith_summary":"Gaze2AOI claims to remove the most labor-intensive step in video-based eye-tracking analysis: manually drawing and labeling areas of interest (AOIs) frame by frame. The paper describes an open-source system that runs YOLOv8 object detection and tracking over a video, automatically labels detected objects as AOIs, and aligns them with gaze and fixation data by frame number. From that alignment it computes time to first fixation, dwell time, and AOI revisit frequency. The authors state that, to their knowledge, no existing open-source tool automates this AOI annotation and metric pipeline, which is the paper's central contribution. A sympathetic reader would care because accurate AOI metrics are the standard way to say what people actually looked at, and making the pipeline automatic would let attention studies scale to long, dynamic videos.","feed_headline":"Gaze2AOI automates AOI annotation in eye-tracking video","feed_subtitle":"Detects ~600 object classes and computes time to first fixation, dwell, and revisits without manual ROI drawing.","key_machinery":"The load-bearing mechanism is the coupling of YOLOv8's per-frame object detection with eye-tracking data through a shared frame number. YOLOv8 provides bounding boxes and class labels for around 600 object types; the tool's tracker maintains object identity across frames; and the frame-number join lets the system decide, for any fixation, which AOI bounding box contains it. Around this core sit two supporting mechanisms: a CLI option to precompute tracking results as a CSV so the analysis can be rerun without re-detecting, and a key-frame selection routine that presents only frames with a changed set of detected objects to speed up manual labeling.","core_discovery":"The central claim is that a single open-source tool can replace manual AOI annotation in post-hoc eye-tracking video analysis. The system detects objects with a pre-trained YOLOv8 model supporting roughly 600 object classes, tracks them across frames with the ByteTrack tracker, and writes per-frame predictions that record, for every detected object, whether it is present, gazed upon, or contains the fixation point. Because predictions are keyed to frame numbers, they can be joined directly to eye-tracking samples. From this join the tool derives the standard AOI metrics: time to first fixation, dwell duration, and number of revisits. The paper further claims that this makes it the first open-source tool to offer automated AOI annotation and eye-tracking integration, and it supports custom labeling so users can correct false detections or add semantic names (for example, a person's name) to detected objects.","pith_inferences":["The paper's own claims imply that any gaze fixated on an object the pre-trained model cannot detect will be silently excluded from every AOI metric; users should audit detection recall before trusting TTFF and dwell values.","The frame-number join assumes the eye tracker's timestamps map cleanly onto video frames; a one-frame misalignment would shift TTFF and dwell estimates by one frame period, which matters for fast-cut content.","Extending the same pipeline to streaming, real-time analysis would let the tool move from post-hoc behavioral analysis to live adaptive content, a direction the paper lists but does not implement.","Comparing Gaze2AOI's output against manually annotated ground truth on a benchmark of dynamic videos would quantify how much accuracy the automation costs, which the paper does not report."],"forward_implications":["Attention studies on video can bypass manual frame-by-frame ROI drawing for any AOI that is one of the ~600 detectable object classes.","Standard metrics -- time to first fixation, dwell time, and revisit count -- can be computed directly from the frame-joined detections without bespoke analysis scripts.","Because the tool accepts interchangeable YOLO models, research teams can swap in a model trained on their own object classes and retain the same AOI-metric pipeline.","The custom labeling step gives users a correction path, so imperfect automatic detections can be refined rather than discarded.","Precomputing predictions and skipping frames without gaze data makes long recordings practical to process."],"supporting_citations":[{"why":"Supplies the YOLOv8 implementation, tracker variants, and model zoo that the tool wraps.","marker":"[16]"},{"why":"Provides the OpenImagesv7 dataset used to pre-train the object detection model.","marker":"[14,13]"},{"why":"Introduces the YOLO algorithm on which the detection and tracking pipeline is based.","marker":"[15]"},{"why":"The closest existing AOI annotation tool, contrasted for its fixation-patch approach and lack of open-source availability.","marker":"[1]"},{"why":"Prior machine-learning gaze-to-AOI mapping work that Gaze2AOI extends and claims to automate further.","marker":"[18]"},{"why":"Earlier system limited to face detection, used to support the novelty of handling many object classes.","marker":"[20]"}],"fun_headline_variants":["Open-source tool auto-annotates areas of interest in eye-tracking video","Gaze2AOI: auto AOI annotation for eye tracking, 600+ classes, no manual ROIs","Automate eye-tracking AOI annotation with Gaze2AOI's open-source tool","First open-source solution to auto-annotate AOIs in eye-tracking videos","Auto-annotate eye-tracking AOIs with 600-class YOLOv8 tracker"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything the tool reports depends on the pre-trained object detector recognizing, in the researcher's video, every object that is semantically an AOI; the paper does not validate detection accuracy on any eye-tracking video, so a missed detection silently makes gaze at that object invisible to all computed metrics.","fun_headline_variants_meta":{"raw":{"variants":["Open-source tool auto-annotates areas of interest in eye-tracking video","Gaze2AOI: auto AOI annotation for eye tracking, 600+ classes, no manual ROIs","Automate eye-tracking AOI annotation with Gaze2AOI's open-source tool","First open-source solution to auto-annotate AOIs in eye-tracking videos","Auto-annotate eye-tracking AOIs with 600-class YOLOv8 tracker"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2811,"prompt_tokens":873,"completion_tokens":1938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1822}},"tokens_in":489,"tokens_out":1938,"duration_ms":11980,"temperature":1.0,"reasoning_tokens":1822,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:30:45.883780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video-and-eye-tracking dataset with ground-truth AOI labels and object classes outside or on the boundary of the ~600 YOLOv8 classes, run Gaze2AOI, and compare its TTFF, dwell time, and revisit counts with the ground truth; systematic misses or label errors on those objects would refute the claim that automatic annotation is sufficient for arbitrary study videos.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv8 implementation, tracker variants, and model zoo that the tool wraps."},{"cited_title":"Journal of Eye Movement Research11(6) (2018)","cited_arxiv_id":null,"evidence_quote":"Prior machine-learning gaze-to-AOI mapping work that Gaze2AOI extends and claims to automate further."},{"cited_title":"Ieee Access6, 49056–49066 (2018)","cited_arxiv_id":null,"evidence_quote":"Earlier system limited to face detection, used to support the novelty of handling many object classes."}],"review_version":1}