{"id":"18fc9fb3-7093-4474-a5a5-ae284cc7e661","arxiv_id":"2506.00693","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Programmers spend most of their gaze on a few functions, and successful memory-bug localizers show higher rereading rates and shorter gaze jumps than unsuccessful ones.","lead":"This paper tracked the eye movements of 21 programmers as they tried to locate memory bugs in C source code, logging 31 hours of effort. It found that successful bug finders reread code and move their eyes in tighter patterns, while unsuccessful readers often looked at the buggy line but did not recognize it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Key Findings 3-4 rely on pooled Mann-Whitney tests that ignore nesting of tasks within participants and bugs, leaving the reported p-values potentially confounded.","rationale":"The reader's weakest-assumption analysis identifies exactly the statistical non-independence that I consider load-bearing: pooling participant-task observations in Mann-Whitney U tests while participants and bugs are clustered. My stress-test adds a concrete mechanism: Table 7 already shows bug-level variation in regression rate, and Figure 11 shows participant-level imbalance in accuracy groups, so the observed differences could reflect who attempted which bug rather than a general property of successful versus failed localization. A mixed-effects model with random intercepts for participant and bug is a direct, feasible check that would settle whether the Table 11 p-values survive. The paper is transparent, provides a new dataset, and its descriptive findings (e.g., 75% of fixations on 25% of functions) are useful; the central inferential claim about success versus failure, however, requires the more conservative analysis. Since the reader already conditioned acceptance on this issue, my recommendation is UNCHANGED relative to the reader's verdict.","tokens_in":22940,"tokens_out":3559,"duration_ms":38551,"concrete_test":"Re-run the Table 11 comparison for regression rate and mean Euclidean distance using mixed-effects models with a fixed effect for high/low group and random intercepts for participant ID and bug codename, using the same data and exclusions. If the group coefficient remains significant at alpha=0.05 after this adjustment, the pooled Mann-Whitney result survives the nesting concern; if not, the reported p-values are inflated by non-independence and the attention-difference claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 11 reports statistically significant differences in regression rate (p=0.003) and mean Euclidean distance (p=0.0004) between high and low where-accuracy groups. These tests treat each participant-task as an independent observation. But each of the 21 participants contributes up to eight tasks, and tasks are drawn from only eight bugs. Figure 11 explicitly shows that the high and low outcome groups are imbalanced by participant, and Table 7 shows that regression rate varies strongly by bug (e.g., weevil 67% versus silverfish 43%). Under the reported analysis, two explanations are indistinguishable: (a) successful localizers genuinely regress more and scan less, or (b) participants who are less skilled, or bugs that are more difficult, produce both lower accuracy and a different eye-movement signature. The authors acknowledge the imbalance in the Figure 11 discussion but do not model it. Without accounting for random intercepts for participant and bug, the reported p-values overstate the evidence, and the central claim that visual attention differs measurably between success and failure is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an eye-tracking study of 21 programmers attempting to localize eight memory bugs (leaks, overflows, double frees) in three C programs, with 31 hours of observed effort. The paper addresses three research questions: task outcomes, overall patterns of visual attention, and differences in attention between successful and failed bug localization attempts. The principal findings are that participants concentrate about 75% of fixations on 25% of viewed functions; that even unsuccessful participants often fixate on the correct buggy line; that higher 'where accuracy' is associated with higher regression rate and lower mean Euclidean fixation distance (p=0.003 and p=0.0004 in Table 11); and that successful cases show more time on or regressions to the buggy line in illustrative examples. The authors release their data and provide transparent descriptions of the procedure, grading rubrics, and threats to validity.","tokens_in":23214,"tokens_out":2267,"duration_ms":23464,"significance":"If the results hold, this is a useful first step in characterizing human visual attention specifically for memory-bug localization in C, a task type that has received less eye-tracking attention than Java program comprehension. The study's strengths include a clearly described experimental protocol, a reproducibility package with raw and processed data, explicit rubrics for grading 'what' and 'where' answers, and appropriately cautious language about the preliminary nature of the findings. The observation that unsuccessful participants often fixate on the correct line is an interesting and falsifiable result that could motivate future work on why programmers dismiss correct locations. However, the central quantitative claim linking attention metrics to success is currently supported by pooled tests that ignore the nested structure of the data, and one of the four key findings rests on anecdotal examples rather than a statistical comparison.","major_comments":[{"comment":"The Mann-Whitney U tests for regression rate and mean Euclidean distance treat each participant-task as an independent observation, but the 21 participants contribute up to eight tasks each and the tasks are drawn from only eight bug reports. Figure 11 shows that high and low outcome groups are imbalanced by participant, and Table 7 shows strong bug-level variation in regression rate (e.g., weevil 67% versus silverfish 43%). Under this analysis, the reported p-values (0.003 and 0.0004) cannot distinguish the claim that successful localizers genuinely regress more and scan less from the confound that less skilled participants, or more difficult bugs, produce both lower accuracy and a different eye-movement signature. The text acknowledges the imbalance in the Figure 11 discussion but does not model it; without random intercepts for participant and bug, or a participant-level aggregation analysis, the evidence for Key Finding 3 is overstated.","section":"Section 4.3, Table 11"},{"comment":"Key Finding 4 states that in success cases participants 'often spend more time on the buggy line or regress back to the buggy line several times,' but this claim is supported only by four selected fixation timelines. No quantitative definition of 'time on the buggy line' or 'regressions to the buggy line' is given, and no test compares these quantities across the high and low accuracy groups. If this is intended as a key finding, it needs a metric and a statistical comparison; if it is intended as anecdotal motivation, it should be labeled as such rather than presented as a finding alongside Key Findings 1-3.","section":"Section 4.3, Key Finding 4 and Figures 10a-10d"}],"minor_comments":[{"comment":"The phrase 'statically significant results' should be 'statistically significant results'.","section":"Section 4.3, paragraph 1"},{"comment":"The sentence 'To our knowledge, this first study to examine human visual attention during memory bug localization in C programs' is missing a verb; it should read 'this is the first study.'","section":"Section 5.2"},{"comment":"The metric name 'mean euclidean distance' should be capitalized as 'mean Euclidean distance' for consistency with standard terminology.","section":"Table 11 and text throughout"},{"comment":"The y-axis tick labels (100, 101, 102) appear to be a logarithmic scale, but the caption does not state this; please clarify the scale in the caption or use linear ticks.","section":"Figures 5a and 5b"},{"comment":"Several entries are marked 'UNK' without a footnote explaining how to interpret them; the caption mentions that participants changed regions, but the current explanation is too terse to allow readers to interpret the table.","section":"Table 10"},{"comment":"Section 3.2 says 'Three bugs were the same for all participants' while Section 3.5 says 'The first two bug reports were the same for everyone'; these statements should be reconciled.","section":"Sections 3.2 and 3.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of TOSEM and the data release is valuable. The main statistical concern is fixable: a mixed-effects analysis with participant and bug as random effects, or a participant-level aggregation, would directly address the nesting problem. The anecdotal support for Key Finding 4 should be either upgraded to a quantitative analysis or explicitly de-emphasized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First eye-tracking study of C memory-bug localization, with real data and a transparent protocol. The 75/25 function-fixation measure is a clean new result. The success/failure gaze differences are promising but statistically under-supported as reported.\n\nWhat the paper does well: 21 programmers, 31 hours, eight real bugs from OpenSSL, redis, and sway; graded \"what\"/\"where\" answers; raw eye-tracking data released on GitHub. The protocol is detailed and the threats-to-validity section is unusually honest. The fixation concentration result — 75% of fixations land on 25% of fixated functions — is a useful quantitative anchor for anyone designing context-aware code tools. The exploratory finding that regression rate is higher and mean fixation distance is shorter in successful attempts is worth pursuing.\n\nThe soft spots are in RQ3. Six eye metrics are tested without multiple-comparison correction, and the two significant results come from Mann-Whitney tests that treat each participant-task as independent. Participants contribute up to eight tasks; there are only eight bugs. Figure 11 shows the high/low groups are imbalanced by participant, and Table 7 shows regression rate varies by bug (weevil 67%, silverfish 43%). So the reported p-values likely overstate the evidence; the gaze differences could be tracking participant skill or bug difficulty rather than a general success signature. The authors acknowledge the imbalance but do not model it. A mixed model with random intercepts for participant and bug, or cluster-robust inference, would be the natural fix. Also, Key Finding 2 overreaches: saying participants who fixated on the buggy line \"fail to recognize it\" assumes fixation equals recognition. They might have considered the line and rejected it. That language needs softening.\n\nBottom line: the dataset and descriptive findings are solid enough for peer review, and I would cite the paper for the 75/25 result. The success/failure attention signature should be treated as preliminary until the nesting is handled. Worth a serious referee, but expect revision.","headline":"First eye-tracking dataset for C memory-bug localization, with a solid descriptive result; the success/failure claims need nested statistics before they land.","tokens_in":23632,"tokens_out":2431,"would_cite":true,"duration_ms":22141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Eye-tracking study of C memory bugs ties success to rereading and tight gaze.","keywords":["eye tracking","visual attention","bug localization","memory bugs","C programs","program comprehension","regression rate","fixation analysis"],"falsifier":"Re-analyze the released eye-tracking data with a mixed-effects model that includes random intercepts for participant and bug; if the regression-rate and fixation-distance differences disappear or drop below significance once individual skill and bug difficulty are accounted for, the central claim would be refuted.","tokens_in":22715,"feed_emoji":"👀","tokens_out":5430,"duration_ms":52382,"temperature":0.7,"pith_summary":"This paper reports an eye-tracking study of 21 programmers who spent up to two hours each trying to locate memory bugs (leaks, overflows, double frees) in three real C programs. It tries to establish that success and failure in bug localization are visibly different in the eyes: successful localizers re-fixate already-read tokens at a higher rate and keep successive fixations closer together, while unsuccessful ones often look directly at the correct line but do not register it as the answer. The study also documents that visual attention is highly concentrated, with 75% of fixations landing on 25% of the functions viewed. If these patterns hold, gaze data could become a real-time signal for when a programmer is about to miss a bug they have already seen.","feed_headline":"Eye tracking shows bug finders reread code and scan less","feed_subtitle":"Rereading and tighter gaze separated success from failure; many who failed had already looked at the right line.","key_machinery":"The argument runs on gaze-based metrics computed from eye-tracker fixations. A fixation is a brief pause of the eyes that lets the brain process what is seen; the study detects fixations with an IVT filter using a velocity threshold of 50 and a minimum duration of 80 ms. Two metrics carry the central comparison: regression rate, the percentage of fixations whose token was fixated more than once, and mean Euclidean distance, the average pixel distance from each fixation to the next. These are compared between tasks graded high (accuracy 4 or 5) and low (accuracy 1 to 3) on a rubric for locating the bug, with the high and low groups defined by the authors' graded 'where accuracy' scores. The same fixation data also yield the concentration result that about 75% of visual effort falls on 25% of the functions viewed.","core_discovery":"The paper's central claim is that success and failure in locating C memory bugs are associated with distinguishable patterns of visual attention. On tasks where participants pinpointed the correct location, their eyes returned to already-read tokens more often (higher regression rate) and traveled shorter distances between successive fixations than on low-accuracy tasks; those two differences were statistically significant in Mann-Whitney U tests. At the same time, participants in failing tasks often did fixate the line that would have been the correct answer, so failure was not simply failure to look. The authors interpret the combination as evidence that what separates success from failure is not finding the buggy line but registering it: successful attempts involve rereading and dwelling, while unsuccessful attempts glance at the right line and move on.","pith_inferences":["The paper's between-group comparison mixes participants and bugs, so a natural next step is a within-subject analysis: does the same programmer show a higher regression rate on bugs they solve than on bugs they miss, holding the bug fixed? The released data would allow this check.","If rereading causes recognition, then training or user-interface nudges that encourage deliberate revisits of candidate lines would be a testable intervention; the paper does not test that.","The 'looked but did not see' pattern hints that failed localization is partly a recognition or confirmation failure rather than a pure search failure, which could be explored with retrospective interviews or think-aloud protocols.","The attention metrics might transfer to other bug types or programming languages, but the paper only studies three C programs, so that transfer is an open question."],"forward_implications":["If success is marked by rereading and tight scanning, then exposing programmers to the relevant code is not enough; interventions should target recognition and confirmation of a suspicious line.","Gaze metrics like regression rate and fixation distance can be computed in real time, so they could be used to flag when a developer has looked at a correct line but is about to move on.","The 75/25 concentration result quantifies how little code context humans actually use, which can inform how much context debuggers or AI assistants should surface.","Because self-reported attention was high (91% of probe responses indicated focus), the observed success/failure differences are not primarily explained by mind wandering or external distraction in this setting."],"supporting_citations":[{"why":"Supplies the taxonomy of memory bug types (uninitialized data, buffer overflows, memory leaks) used to justify the bug selection, with double frees substituted for uninitialized data.","marker":"[18]"},{"why":"Defines regression rate and provides the processing scripts on which the eye-tracking analysis builds.","marker":"[81]"},{"why":"Provides the practical eye-tracking guidance the study follows for drift, recalibration, and best practices.","marker":"[67]"},{"why":"Supplies the Eclipse-based eye-tracking plugin used to record gaze and interaction data during the tasks.","marker":"[89]"},{"why":"The toolkit pipeline used to map fixations to source lines and tokens.","marker":"[9]"},{"why":"Supplies the IVT fixation-detection algorithm whose thresholds (velocity 50, duration 80 ms) define what counts as a fixation.","marker":"[2]"},{"why":"Supplies the consensus procedure used to resolve disagreements between the two graders of bug-cause and bug-location accuracy.","marker":"[51]"}],"fun_headline_variants":["Rereading code, not just seeing it, finds C memory bugs","Bug fixers reread more, scan less, eye study finds","Attention patterns separate C bug hunters who succeed","To find memory bugs, reread: eye tracking shows why","Eye tracking: rereading beats scanning for C memory bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's strongest statistical claim assumes each participant-task contributes an independent observation, even though the same people and the same bugs appear many times; if success-related attention is really driven by who the participant is or how hard the bug is, the reported differences would shrink.","fun_headline_variants_meta":{"raw":{"variants":["Rereading code, not just seeing it, finds C memory bugs","Bug fixers reread more, scan less, eye study finds","Attention patterns separate C bug hunters who succeed","To find memory bugs, reread: eye tracking shows why","Eye tracking: rereading beats scanning for C memory bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3149,"prompt_tokens":875,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2200}},"tokens_in":491,"tokens_out":2274,"duration_ms":14092,"temperature":1.0,"reasoning_tokens":2200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:59:07.227513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the released eye-tracking data with a mixed-effects model that includes random intercepts for participant and bug; if the regression-rate and fixation-distance differences disappear or drop below significance once individual skill and bug difficulty are accounted for, the central claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of memory bug types (uninitialized data, buffer overflows, memory leaks) used to justify the bug selection, with double frees substituted for uninitialized data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines regression rate and provides the processing scripts on which the eye-tracking analysis builds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the consensus procedure used to resolve disagreements between the two graders of bug-cause and bug-location accuracy."}],"review_version":1}