{"id":"81c0cc35-0b41-4bbe-b6dc-3b8872a95694","arxiv_id":"2505.01451","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"AdSight predicts per-slot fixation time, fixation count, and noticeability on search engine results pages from mouse cursor trajectories alone.","lead":"This paper introduces AdSight, a Transformer model that converts mouse cursor movements into per-slot attention estimates for search engine results pages. It predicts how long users looked at each ad or organic result and whether they noticed it, with eye-tracking data as ground truth.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy is not shown to generalize: hyperparameters are tuned with 3-fold CV (Sec. 5.1/5.2), no held-out test set is specified (Sec. 5.4), and trial-level folds can leak the 47 participants across train/test; the headline MSE/AUC claims may be optimistic.","rationale":"The reader's weakest assumption was that the evaluation protocol lacks a separate held-out test set, so the reported metrics may be optimistically biased. I agree with that concern, and it is part of my analysis. However, I think the more load-bearing issue is that the paper never specifies whether cross-validation folds are grouped by participant. Given only 47 participants and ~2,776 trials, trial-level splitting would create strong user-identity leakage: cursor trajectories are known to vary systematically across users, and the model could memorize user-specific movement and fixation patterns instead of learning a general mapping from cursor behavior to attention. This directly threatens the central claim of scalable, accurate attention quantification for new users, independent of the hyperparameter-selection issue.\n\nBoth problems fall under 'the reported metrics are not unbiased estimates of generalization,' so my verdict remains conditional rather than reject: the architecture and empirical gains over the included baselines are plausible, but the accuracy numbers need re-estimation under a valid protocol. The concrete test of participant-grouped nested CV would settle whether the concern lands. I did not find evidence of fraud or internal inconsistency; the issue is missing methodological detail plus a realistic leakage path. The 'unprecedented precision' wording is also overstated relative to the comparisons shown, but the evaluation-protocol concern is the substantive one.","tokens_in":20308,"tokens_out":4710,"duration_ms":55303,"concrete_test":"Re-run the main experiments with participant-grouped nested cross-validation: assign all trials from each of the 47 participants to the same outer fold, run Optuna hyperparameter search on the training folds only, and report mean and standard deviation over the outer folds. Compare the Seq2Seq Transformer TFT MSE (currently 2.86), NDCG, and average classification AUC (currently 81.24) against the published values, and also report the gap to the best MLP baseline under this protocol. If the metrics degrade substantially or the advantage over baselines shrinks, the generalization claim for new users is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that mouse cursor trajectories plus slot metadata predict per-slot fixation time, fixation count, and noticeability with 'unprecedented precision' (abstract; Sec. 6.1 reports TFT MSE 2.86, Sec. 6.2 reports average AUC 81.24). For this claim to be meaningful, the reported numbers must estimate generalization to new data. The paper does not establish that.\n\nSection 5.1 and 5.2 state that hyperparameters are optimized with Optuna using 3-fold cross-validation. Section 5.4 says results are 'the average predictive and ranking performance across test instances,' but no separate held-out test set or nested cross-validation is described. If the same folds used for hyperparameter selection are also used for the reported metrics, those metrics are optimistically biased by selection.\n\nA second, arguably more serious independence problem is participant leakage. The data come from 47 participants (Sec. 3.1) contributing 2,776 trials (Sec. 3.3), roughly 60 analyzed trials per participant. The paper never states that trials from the same participant are kept in the same fold. If folds are split at trial level, the same user's cursor trajectories and eye fixations appear in both training and test partitions. Cursor movement style is user-specific, so the model can exploit participant-identifying patterns rather than learning a general cursor-to-attention mapping. That would inflate all reported accuracies, including the comparisons against MLP, BiLSTM, and ResNet50 baselines, and it directly undermines the scalable, deployable interpretation of the method for new users.\n\nAdditionally, the phrase 'average error of 1.69 seconds' equates sqrt(MSE) with average absolute error; for non-constant errors, RMSE exceeds MAE, so this wording may overstate precision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdSight, a Transformer-based sequence-to-sequence model that predicts per-slot attention metrics (total fixation time and fixation count) and slot-noticeability labels from mouse cursor trajectories and slot metadata on multi-slot Google SERPs. The evaluation uses an eye-tracking dataset of 2,776 trials from 47 participants, comparing AdSight against MLP-readout baselines and the methods of Arapakis and Leiva [6]. The authors report consistent improvements, including a TFT MSE of 2.86 and an average classification AUC of 81.24, together with ablations on slot coordinates, slot metadata, cursor features, and auxiliary slots.","tokens_in":20647,"tokens_out":4310,"duration_ms":46090,"significance":"If the reported accuracy estimates are unbiased, the paper makes a useful contribution: it proposes a scalable mouse-based proxy for eye-tracking attention in multi-slot layouts, handles a variable number of slots with a Seq2Seq architecture, and validates the approach against objective eye-tracking ground truth and external baselines. The strengths include the use of a realistic SERP dataset, statistically tested comparisons with Wilcoxon signed-rank tests, and systematic ablation studies. However, the evaluation protocol is under-specified in a way that directly affects the headline claims; the central accuracy numbers cannot currently be taken as unbiased estimates of generalization to new users or new SERPs.","major_comments":[{"comment":"The evaluation protocol is not fully specified. Hyperparameters are optimized with Optuna using 3-fold cross-validation (§5.1 and §5.2), and §5.4 says results are 'the average predictive and ranking performance across test instances,' but the manuscript never states whether a separate held-out test set or nested cross-validation is used. If the same folds used for hyperparameter selection are also used for the reported metrics, the headline numbers in Table 1 and Table 2 are optimistically biased by model selection. Please clarify the exact protocol and, if no independent test set exists, re-evaluate with a proper held-out split.","section":"§5.1, §5.2, §5.4"},{"comment":"The data come from 47 participants contributing 2,776 trials, roughly 60 analyzed trials per participant, but the paper does not state whether trials from the same participant are kept in the same cross-validation fold. If folds are split at the trial level, the same user's cursor trajectories and eye fixations can appear in both training and test partitions. Because cursor movement style is user-specific, the model could exploit participant-identifying patterns rather than learning a general cursor-to-attention mapping. This would inflate all reported accuracies, including the comparisons against the MLP and [6] baselines. Please report results computed with participant-disjoint folds or with leave-one-participant-out evaluation.","section":"§3.1, §3.3, §5.1"}],"minor_comments":[{"comment":"The statement that a TFT MSE of 2.86 'corresponds to an average error of 1.69 seconds' is a square-root transformation, i.e., an RMSE, not an average absolute error. Please state this explicitly and report mean absolute error if that is the intended interpretation.","section":"§6.1, Table 1"},{"comment":"The classification labels depend on median-based TFT and TFC thresholds, but the actual threshold values are not reported. Reporting them is necessary for reproducibility of the label construction and for interpreting the observed fixation rates.","section":"§4.3.2"},{"comment":"The procedure says the experiment was divided into eight blocks of ten trials each, while §3.4 says the SERPs were randomly organized into ten blocks. These numbers should be reconciled.","section":"§3.3, §3.4"},{"comment":"The sentence 'we incorporates slot-specific features, which significantly improves the model's performance' contains a grammatical error ('we incorporates' should be 'we incorporate') and should be corrected.","section":"§5.1.1"}],"recommendation":"major_revision","confidential_remarks":"The primary risk is the evaluation protocol: without a held-out test set and participant-disjoint folds, the reported gains may be optimistic. If the authors can supply a clean participant-disjoint evaluation, the paper is likely acceptable. The use of a co-authored dataset paper [43] and a co-authored baseline [6] is not itself problematic given that the baselines are appropriate and the comparison is against an independent eye-tracking ground truth."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real method paper. The Seq2Seq Transformer with slot metadata is a sensible extension of prior single-slot work, and the gains over the MLP baseline are consistent across metrics, loss functions, and cursor representations. The main weakness is the evaluation protocol: they tune with 3-fold CV and never show that the reported numbers come from a held-out test set or participant-disjoint folds. That needs to be addressed before I'd trust the headline precision claims.\n\nWhat's new: prior work by Arapakis and Leiva [6] looked at single ads; this handles variable-slot SERPs with an encoder-decoder Transformer where the decoder consumes slot-specific features. The auxiliary-slot idea is useful, and the ablations on slot coordinates, cursor features, and auxiliary loss weight are genuinely informative. The comparison against [6]'s BiLSTM/ResNet50 with eye-tracking ground truth is fair, and the Wilcoxon tests are appropriate. Credit where due: the consistent advantage across regression and classification is a real empirical signal, not noise.\n\nSoft spots, in proportion. First and most important: evaluation. Section 5.1/5.2 say Optuna with 3-fold CV; Section 5.4 says results are averages over test instances, but no separate test set or nested CV is described. If the same folds select hyperparameters and produce the tables, the metrics are selection-biased. Second: participant leakage. There are 47 participants and roughly 2,776 trials; nothing says folds are grouped by participant. Cursor dynamics are user-specific, so trial-level folds could let the model recognize participants rather than learn a general cursor-to-attention mapping. That's not established as fact from the text, but the burden is on the authors to rule it out. Third, minor: 'average error of 1.69 seconds' is sqrt(MSE), not mean absolute error—that wording overstates precision, and 'unprecedented precision' in the abstract should be toned down regardless.\n\nNone of this kills the contribution. The architecture, the metadata encoding, and the empirical pattern are solid enough to justify a serious referee. I'd send it out and ask the authors to (1) specify or add a proper held-out split, (2) confirm participant-disjoint folds, and (3) report MAE alongside RMSE. If they do that, the numbers become credible.\n\nWho's this for: people in IR and computational advertising who care about cursor-based attention estimation on modern SERPs. I wouldn't cite it until the evaluation is clarified, but it's a worthwhile read for that community and deserves peer review rather than a desk reject.","headline":"Solid Seq2Seq architecture with consistent gains, but the evaluation protocol is under-specified and the headline accuracy numbers may be optimistic until the authors clarify or fix the train/test split and participant grouping.","tokens_in":21257,"tokens_out":1890,"would_cite":false,"duration_ms":20441,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdSight claims that mouse cursor trails alone can predict per-slot user attention on search ads to within about 1.7 seconds of true fixation time.","keywords":["SERP","sponsored search","user attention","mouse cursor tracking","eye tracking","Transformer","sequence-to-sequence","attention prediction"],"falsifier":"Run the same hyperparameter search inside an outer cross-validation loop or on a held-out set of SERPs and participants, and compare the total fixation time mean squared error; if the average error on truly unseen pages is substantially above 1.69 seconds, the generalization claim fails.","tokens_in":20106,"feed_emoji":"🖱️","tokens_out":5270,"duration_ms":52457,"temperature":0.7,"pith_summary":"AdSight predicts how much a user attends to each ad slot on a search results page using only mouse cursor movements and slot metadata. The core claim is that a Transformer-based sequence-to-sequence model can estimate each slot's total fixation time to within an average error of 1.69 seconds, match the true attention ranking with NDCG 96.07, and classify whether a slot category was noticed with an average AUC of 81.24. Because the decoder emits one output per slot, the method adapts to pages with any number of ads. If these accuracies hold beyond the lab, attention measurement could move out of eye-tracker laboratories and into real-world search and advertising systems.","feed_headline":"Cursor trails gauge ad attention to within ~1.7s","feed_subtitle":"A Transformer reads mouse movements plus slot positions to estimate fixation time, count, and which ads were noticed.","key_machinery":"The central mechanism is a Transformer-based sequence-to-sequence (Seq2Seq) network. The encoder reads a multivariate time series of cursor events, including normalized coordinates, dwell time, slot category, and sequence index. The decoder ingests slot metadata embeddings, each slot's normalized center coordinates and category, and emits one prediction per slot through a shared MLP readout. Auxiliary slots inserted between the real slots act as additional areas of interest, refining the cursor-position categorisation and contributing an auxiliary loss term that improves the main predictions.","core_discovery":"The paper claims that mouse cursor trajectories, when combined with slot-specific metadata such as normalized center coordinates and slot category, carry enough signal to predict per-slot gaze behavior on multi-slot search engine result pages. The best model predicts total fixation time per slot with an average mean squared error of 2.86, corresponding to an average error of 1.69 seconds, and matches the true slot ranking with NDCG 96.07. For classification, the architecture determines whether a user noticed each slot category with an average AUC of 81.24. The evidence comes from a lab study with 47 participants, 2,776 trials, and eye-tracking ground truth, where the Seq2Seq model consistently beat MLP baselines across all cursor representations, target metrics, and loss functions.","pith_inferences":["If the accuracy survives deployment outside the lab, per-impression attention estimates could be produced at the scale of real ad auctions on live search traffic.","The authors' observation that slot order does not affect predictions suggests the decoder learns about slot properties rather than a fixed layout, which could transfer to layouts never seen in training.","A concrete extension is applying the same encoder-decoder to other multi-element interfaces, such as social-media feeds or product shelves, using the same slot-metadata representation.","A stress test is low-cursor activity: users who navigate by keyboard or scroll without moving the mouse may break the input assumption, and the paper does not report such cases."],"forward_implications":["The method can estimate per-slot attention without eye-tracking hardware, making attention measurement scalable to large user populations.","Per-slot fixation time predictions can directly feed attention-weighted ad pricing schemes, where advertisers pay according to how likely their ad is noticed.","Slot noticeability classification with average AUC above 80 provides a practical signal for pay-per-attention auctions.","The Seq2Seq readout handles a variable number of slots, so the model can be applied to SERPs with different ad counts without retraining per layout."],"supporting_citations":[{"why":"Supplies the BiLSTM and ResNet50 baselines that the proposed model must beat, forming the core comparison for both regression and classification tasks.","marker":"[6]"},{"why":"Provides the dataset of mouse and eye movements on 2,776 SERP trials that is used to train and evaluate AdSight.","marker":"[43]"},{"why":"Introduces the Transformer encoder-decoder architecture that the Seq2Seq model is built upon.","marker":"[65]"},{"why":"Establishes earlier mouse-cursor attention prediction on single direct displays, which this work extends to multiple slots simultaneously.","marker":"[5]"},{"why":"Motivates the pay-per-attention auction scheme that makes slot-level attention prediction valuable for computational advertising.","marker":"[8]"},{"why":"Provides the Vision Transformer architecture used as an alternative cursor embedding from visual representations.","marker":"[23]"},{"why":"Defines the listwise rank loss used as an alternative training objective alongside mean squared error.","marker":"[15]"}],"fun_headline_variants":["Cursor paths predict ad gaze within 1.7 seconds","Transformer reads mouse moves to gauge ad attention","AdSight: cursor trails forecast ad fixation time","Mouse motion maps ad attention to ~1.7s error","Cursor trajectories reveal which ads users notice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume the validation data used for hyperparameter selection is separate from the data used for final performance reporting, but the paper never specifies a held-out test set.","fun_headline_variants_meta":{"raw":{"variants":["Cursor paths predict ad gaze within 1.7 seconds","Transformer reads mouse moves to gauge ad attention","AdSight: cursor trails forecast ad fixation time","Mouse motion maps ad attention to ~1.7s error","Cursor trajectories reveal which ads users notice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1285,"prompt_tokens":850,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":466,"tokens_out":435,"duration_ms":4771,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:03:15.046590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same hyperparameter search inside an outer cross-validation loop or on a held-out set of SERPs and participants, and compare the total fixation time mean squared error; if the average error on truly unseen pages is substantially above 1.69 seconds, the generalization claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Transformer encoder-decoder architecture that the Seq2Seq model is built upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the pay-per-attention auction scheme that makes slot-level attention prediction valuable for computational advertising."}],"review_version":1}