{"id":"6ca1c7c6-d14d-402b-b930-8f1127d593bb","arxiv_id":"2607.14535","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Transformer fine-tuned with a loss that matches its internal attention to human eye-tracking gaze time predicts programmer visual attention better than the compared baselines on three datasets.","lead":"A team at Notre Dame fine-tuned a large language model so that its internal attention patterns predict where programmers look when reading code, testing it on three eye-tracking datasets and a scanpath-prediction task. The model beats the cited software-engineering and vision baselines on all three datasets, though one headline gain is large only relative to a very weak baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation only covers word tokens that were fixated; non-fixated words are excluded, so the headline claim about predicting attention over source-code words is untested for the attended-vs-unattended distinction.","rationale":"The paper's own limitation section notes that ptgt aggregates attention into a scalar per token and collapses reading order, but it does not acknowledge that the evaluation set W' excludes all non-fixated tokens. This is not merely a missing error bar or an unablated design choice; it changes the prediction problem from 'which words will be read' to 'among words known to be read, how much attention will each get'. The reader's weakest assumption focuses on the hand-selected attention heads (10–16) and on hyperparameter selection. Those are secondary: the supervised loss can push any readout toward human alignment, and the hyperparameter concern can be addressed by reporting validation-based selection. The truncation to fixated tokens is the least secure condition for the central claim because no amount of significance testing or head ablation can fix a target definition that excludes the attended/unattended decision. The concrete check—recomputing correlations with zero-attention words included—can settle whether the reported advantage survives the more complete evaluation. I therefore keep the reader's CONDITIONAL verdict unchanged; the paper should either scope its central claim to 'attention among fixated words' or, preferably, evaluate on all words and show the margins persist.","tokens_in":20257,"tokens_out":15106,"duration_ms":174457,"concrete_test":"Re-run the Table IV evaluation, but for each function include every source-code word token in W, assigning human ptgt=0 for tokens never fixated in the evaluation session/participant, and compute the same four correlations for all models (Bansal, Tafasca, ours-nocorr, ours), keeping all training hyperparameters fixed as in Table II. If 'ours' no longer exceeds both baselines on Pearson r, or if the margins drop below the reported 64%/16%/467%, the headline result is an artifact of evaluating only on fixated tokens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that 'ours' predicts human visual attention over source-code words better than baselines—is evaluated in §IV-E over the set of canonical fixation tokens only. §IV-E1 states: 'We extract a single attention score for each canonical fixation word token from the model', and W' is explicitly the subset of word tokens that were fixated. Non-fixated words, for which ptgt = 0 by the definition in §III-A, are never included in the correlation vectors a and \\hat a. Thus the reported Pearson correlations in Table IV measure how well the model ranks words that are already known to have been looked at by at least one participant; they do not measure whether the model can tell attended from non-attended words. That latter ability is the core of the Introduction's definition: 'a forecast of which words in a program’s source code will be read most often, reread most often, and/or to what degree and in what order'. Because the same truncation is applied to all models, the 64%/16%/467% margins are internally consistent, but they do not support the abstract's unqualified claim. If non-fixated words are included with ptgt=0, a model that assigns positive mass to every token will be penalized, and the claimed advantage could shrink or reverse.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Transformer-based model for predicting human visual attention over source-code words, operationalized as percent total gaze time (ptgt). The model is built by fine-tuning a GPT-2-style code language model (jamm) with a custom loss that combines standard cross-entropy with MSE and Pearson-correlation terms, where the prediction is read out by averaging self-attention heads 10-16 of the final layer. The evaluation compares this model against a software-engineering GNN baseline, a computer-vision baseline, and an ablation without the correlation loss, on three eye-tracking datasets (Wallace, Smith, Rodeghero) using four correlation metrics. A second experiment applies the model to scanpath prediction on the Bansal dataset, comparing against the Bansal scanpath baseline, GPT-5, and Claude. The paper claims consistent improvements over baselines, with the largest gains on the Smith dataset, and a statistically significant improvement in normalized Levenshtein distance on the reading scanpath task.","tokens_in":20535,"tokens_out":6390,"duration_ms":61144,"significance":"If the claims hold, the work would provide a practical way to predict where programmers look while reading code, with applications to code-summarization tools, IDE design, and cognitive studies of software engineering. The paper has several concrete strengths: it releases the source code, evaluates on three independent eye-tracking datasets spanning two languages and two task types, uses leave-one-participant-out evaluation, and includes a transfer task (scanpath prediction) that provides some independent evidence beyond direct ptgt fitting. The novel loss formulation is an interesting and clearly stated idea, and the comparison against the most relevant prior models is appropriate. However, the headline claim is currently broader than what the evaluation actually measures, and several methodological gaps need to be addressed before the quantitative conclusions can be taken at face value.","major_comments":[{"comment":"The correlation vectors are built exclusively from W', the set of canonical fixation tokens; non-fixated words, whose ptgt is 0 by the definition in §III-A, are never included in a or a-hat. Thus the Pearson correlations in Table IV measure how well the model ranks words already known to have been fixated by at least one participant, not whether the model can distinguish attended from non-attended words. The Introduction defines the task as forecasting 'which words in a program’s source code will be read most often, reread most often, and/or to what degree and in what order,' and the abstract's unqualified claim is not supported by the reported evaluation. Because the same truncation is applied to all baselines, the relative margins are internally consistent, but the absolute claim is overstated. Please include non-fixated tokens with ptgt=0 in the vectors (or clearly reframe the claim t","section":"§IV-E and Table IV"},{"comment":"The attention readout is a hand-selected slice: heads 10-16 of the final Transformer layer, with layers 14-24 fine-tuned, adopted from Wang et al. No ablation of alternative head/layer choices is provided, and the training hyperparameters in Table II are described as 'based on our experiments' on the same three datasets used for evaluation. Leave-one-participant-out protects against participant-level overfitting but does not protect against dataset-level selection of the readout and hyperparameters. Please add ablations over readout choices and/or a nested validation scheme, or explicitly state which design choices were fixed a priori before inspecting the evaluation datasets.","section":"§III-C, §IV-B, Table II"},{"comment":"The main correlation results are reported without confidence intervals, bootstrap estimates, or significance tests. For example, in the Smith study the Bansal baseline has Pearson r=0.0438, ours has r=0.2484, and the human reference is 0.2134; with a small number of functions and participants, these differences could easily be within sampling noise. The scanpath experiment uses Wilcoxon tests for Table V, but Table IV has no such quantification. Please add per-participant or per-function confidence intervals (or equivalent uncertainty measures) for each metric before claiming consistent improvement over baselines.","section":"§V-A, Table IV"},{"comment":"The human-reference formula as written appears self-referential. For a held-out participant p*, the text defines a-hat_{f,t}=a_{p*,f,t} and a_{s,f,t}=a_{s,f,t}-a_{p*,f,t}, where a_s ranges over other participants' observations. This computes the correlation between one participant's attention and a residualized version of the other participants' attention, not the similarity of that participant to the group. If implemented literally, the human-reference rows in Table IV are uninterpretable. If this is a typo, please correct the definition and recompute; the comparison to inter-human agreement is load-bearing for the claim that the model approaches human-level consistency.","section":"§IV-E6"}],"minor_comments":[{"comment":"The attention-readout formula has an arithmetic inconsistency: with H=16, the sum over h=10 to H includes 7 heads, but the denominator H-10 = 6. The denominator should be H-9 (or the sum should be h=10 to H-1).","section":"§III-C, Eq. (5)"},{"comment":"The scaling factors α_MSE and α_corr are never defined. Please state how they are computed from the ratio between cross-entropy loss and attention loss; without this, the final loss is not fully reproducible.","section":"§III-C, Eq. (8)"},{"comment":"The statement that the Rodeghero model is initialized from a model already fine-tuned on Wallace data should be described more carefully; as written, Rodeghero is not evaluated from a fresh training run, which should be acknowledged in the dataset-description section.","section":"§IV-B"},{"comment":"Several typos and formatting issues: 'stat-of-the-art' in the Conclusion, 'Disounted' in §VI-C, 'jamasM pre' in §IV-B, and the formula in §IV-E6 (as noted above). Please proofread carefully.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: this is a solid method paper on a useful, incremental question — can you fine-tune a code LM so its internal self-attention tracks programmer gaze? The authors build a custom loss (CCE + MSE + Pearson on the attention readout), evaluate on three eye-tracking datasets, and release the code. The scanpath transfer experiment is a nice extra. It is a legitimate extension of the Bansal/EyeTrans line, with the right baselines.\n\nWhat actually works well: leave-one-participant-out is a reasonable protocol, results are consistent across three datasets and four metrics, and the ablation shows the Pearson component helps. The significance test on the reading-task NLD is a step up from most papers in this area.\n\nSoft spots, in order of size. The biggest is the evaluation scope. Section IV-E builds the correlation vectors over W'_s — the set of canonical fixation tokens — so non-fixated words, which have ptgt=0, never enter. Reported Pearson correlations therefore measure how well the model ranks words that are already known to have been looked at. They do not test whether the model can tell attended from unattended words, which is exactly the promise of the introduction and the abstract. The margins over baselines may be internally consistent, but they support a narrower claim than advertised.\n\nSecond, the headline Table IV numbers come without confidence intervals or significance tests. The scanpath section is better because it uses Wilcoxon, but the main result is point estimates only.\n\nThird, the free parameters — layer range 14–24, heads 10–16, loss weights, epochs — are selected on the same datasets used for evaluation. Leave-one-participant-out does not protect against readout selection on test data. An ablation over readout choices would help.\n\nFourth, the human-comparison formula in §IV-C is confusing: the vector definitions look off (the subtraction of p* from the 'a' vector is odd) and the resulting human-reference values deserve a clearer derivation. This is minor.\n\nOn balance, this is a serious paper. The central idea is plausible, the implementation is clean, and the problem is in the evaluation scope, not the architecture. It deserves a real referee. If I were the editor, I'd send it out with a request for major revision: include non-fixated tokens in the evaluation (or explicitly limit the claim and abstract), add uncertainty quantification, and document hyperparameter selection. With those changes, it becomes a positive contribution.","headline":"Solid method paper, but the headline claim is over-sold: the evaluation only covers fixated tokens, so attended-vs-unattended discrimination is never tested.","tokens_in":21062,"tokens_out":5692,"would_cite":true,"duration_ms":56890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a language model's internal attention with eye-tracking data predicts which words in source code a programmer will look at, outperforming prior software-engineering and computer-vision baselines on three datasets.","keywords":["human visual attention","eye tracking","source code","Transformer language model","percent total gaze time","scanpath prediction","attention readout","software engineering"],"falsifier":"Run an ablation that replaces the fixed heads-10–16 readout with a learned linear readout trained to predict ptgt from the same final-layer attention; if the learned readout performs as well or better than the fixed slice, then the specific head choice is not load-bearing, and the reported gains cannot be attributed to that design decision.","tokens_in":20095,"feed_emoji":"👀","tokens_out":3988,"duration_ms":39428,"temperature":0.7,"pith_summary":"The paper claims that a Transformer language model can predict aggregate human visual attention over words in source code by fine-tuning its internal self-attention mechanism with eye-tracking data. The authors introduce a custom loss that aligns the model's attention scores with the percent total gaze time (ptgt) measured from programmers, using both mean squared error and Pearson correlation. Across three eye-tracking datasets (two Java, one C), the model outperforms the graph neural network baseline and a computer vision baseline, with 64%, 16%, and 467% improvements in Pearson correlation. The model also improves scanpath prediction over the previous baseline on a reading task, suggesting it captures order as well as distribution of attention. A sympathetic reader would care because predicting visual attention has practical value for interface design, tooling, and understanding what information programmers need while reading code.","feed_headline":"Fine-tuned LLM predicts which source-code words programmers read","feed_subtitle":"Outperforms prior models on three eye-tracking datasets; improves scanpath prediction on reading tasks.","key_machinery":"The key mechanism is the attention readout: the model extracts the average of self-attention probability distributions from attention heads 10 through 16 in the final Transformer layer. The custom loss function then computes a weighted combination of the standard cross-entropy loss, a mean squared error term, and a Pearson correlation term between this readout and the human gaze data (ptgt). This readout is what converts the model's internal attention into a prediction of where a human will look, and the loss is what shapes that readout during fine-tuning.","core_discovery":"The central claim is that human visual attention over source code can be predicted by averaging the self-attention scores from a specific subset of heads (heads 10 through 16) in the final Transformer layer of a fine-tuned language model. The fine-tuning loss combines standard cross-entropy with mean squared error and Pearson correlation between this attention readout and the human gaze metric ptgt, reshaping the model's internal attention to be more human-like. The paper reports that the resulting model predicts which words programmers fixate on more accurately than the compared baselines on all three datasets, and that its predicted scanpaths on a reading task are closer to human reading o","pith_inferences":["A direct next step would be to ablate the choice of heads 10–16 and the final-layer readout; the paper does not test whether other head subsets or layers would yield similar or better predictions, so the specificity of the readout remains an open question.","The custom loss may implicitly regularize the language model to attend in a human-like way, which could improve downstream tasks that benefit from human-like focus, such as code summarization; the paper hints at but does not test this transfer.","The strikingly large improvement on the bug-localization dataset (467%) suggests the method may be especially valuable for tasks where human attention is scattered and hard to model, but this needs replication across other bug-localization datasets.","One could test whether the attentional prior transfers across tasks: fine-tune on one task (e.g., Java code summarization) and evaluate on another (e.g., C bug localization) without further tuning, which would indicate a task-independent human attentional prior."],"forward_implications":["Code editors and development environments could rank or highlight tokens by predicted human attention to improve tool layouts and documentation generation.","The approach can be extended to other programming languages and tasks, provided eye-tracking data exists, to build task-specific attentional priors.","Because the fine-tuned model remains a standard autoregressive language model, it could serve as a drop-in backbone for code summarization or navigation that implicitly uses human-like attention.","The success of the attention readout suggests that certain internal layers of Transformer models encode behaviorally plausible attention, which may inform interpretability research.","The improvement in scanpath prediction indicates the model captures fixation order, not just aggregate distribution, opening the door to predicting reading order in code."],"fun_headline_variants":["LLM self-attention predicts programmers' gaze on code","Transformer heads reveal which code words coders read","Predict human attention in code: fine-tuned LLM maps gaze","Gaze prediction in source code: transformer attention beats baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole method depends on the unverified premise that the average of attention heads 10–16 in the final Transformer layer is the right readout for human attention; if that choice is wrong, the custom loss may distort the language model without producing trustworthy attention predictions.","fun_headline_variants_meta":{"raw":{"variants":["LLM self-attention predicts programmers' gaze on code","Transformer heads reveal which code words coders read","Predict human attention in code: fine-tuned LLM maps gaze","Gaze prediction in source code: transformer attention beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3210,"prompt_tokens":717,"completion_tokens":2493,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2434}},"tokens_in":461,"tokens_out":2493,"duration_ms":16634,"temperature":1.0,"reasoning_tokens":2434,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:47:19.121133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an ablation that replaces the fixed heads-10–16 readout with a learned linear readout trained to predict ptgt from the same final-layer attention; if the learned readout performs as well or better than the fixed slice, then the specific head choice is not load-bearing, and the reported gains cannot be attributed to that design decision.","supporting_citations":[],"review_version":1}