Pith. sign in

REVIEW 3 major objections 4 minor 93 references

An empirical study of 319 real Python and Java bugs argues that LLM repair failures are driven by attention misallocation, not just bug difficulty.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:13 UTC pith:M5JGAZVE

load-bearing objection Useful empirical study with a genuinely new developer-attention dataset, but the headline claim about Diffused/Localized attention patterns rests on an under-specified, single-annotator visual labeling step; worth refereeing with required revisions. the 3 major comments →

arxiv 2607.25873 v1 pith:M5JGAZVE submitted 2026-07-28 cs.SE cs.AI

How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

Ramtin Ehsani , Irene Manotas , Saurabh Pujar , Luca Buratti , Preetha Chatterjee This is my paper
classification cs.SE cs.AI
keywords large language modelsautomated program repairattention analysisbug reportsperturbation-based attributiondeveloper attentionmodel interpretabilitypatch correctness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish why an LLM repairs one bug correctly but fails on a closely related one, even when the input structure is the same. Using perturbation-based attribution over 319 real-world Python and Java bugs, it argues that repair success tracks where the model concentrates while reading the bug report: successful repairs spread attention across diagnostic components such as the bug description, stack traces, and test snippets, while failures concentrate on peripheral metadata such as version numbers. It also claims that the closer a model's attention is to what human developers flag as important, the more likely the repair succeeds. If correct, the results give practitioners a concrete lever—prompt, retrieval, and context design—for reducing LLM repair failures before a patch is generated.

Core claim

The paper claims that attention misallocation is a key factor in LLM-based automated program repair failures. Across 319 real-world Python and Java bugs, successful repairs are characterized by diffused attention spanning multiple diagnostic components—bug descriptions, stack traces, and test snippets—whereas unsuccessful repairs tend to over-localize attention on peripheral metadata such as version information. The paper further reports that model attention aligned with developer-identified important sections and phrases is significantly associated with repair success, with stronger section-level Spearman correlation (0.60 vs 0.34) and higher Hit@1 for successful repairs. These findings are

What carries the argument

Perturbation-based attention analysis, in which a bug report is systematically masked and the change in the generated patch is measured. Section-level contributions are aggregated using Shapley-value attribution (Kernel SHAP) over masks, with CodeBLEU quantifying patch change; fine-grained components (natural-language sentences, stack traces, test snippets, class and method definitions, imports and variables) are masked one at a time and scored by UniXcoder semantic similarity. The analysis yields three named attention structures—diffused, localized, and no-attention—plus per-component attention scores compared against human developer annotations.

Load-bearing premise

The whole argument depends on the assumption that removing a piece of a bug report and seeing the patch change reveals what the model was paying attention to; if masked sections instead break the prompt's formatting or logic in some other way, the attention story collapses.

What would settle it

Repeat the perturbation analysis but replace each masked section with a same-length placeholder, or shuffle section order instead of deleting content; if patch changes vanish while repair outcomes stay the same, the reported attention signal is an artifact of formatting rather than semantic prioritization. Conversely, if steering a model's attention toward developer-important sections fails to change repair rates, the causal claim is weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Attention patterns can serve as an early-warning signal: a model fixating on version metadata while ignoring the description is likely to produce an incorrect patch.
  • APR prompt, retrieval, and preprocessing pipelines can be redesigned to foreground bug descriptions, stack traces, and tests, and to demote or reformat version information.
  • The annotated developer-attention data can be used to fine-tune or align APR models so their information prioritization matches expert reasoning.
  • Because the perturbation method is model-agnostic, the same attention diagnostics extend to agentic repair systems that do not expose internal model weights.
  • The observed associations persist after controlling for bug difficulty and appear across Python, Java, and multiple studied models, indicating a general failure mode rather than a benchmark-specific artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If attention misallocation causes failures, then rewriting or reordering bug reports to lead with diagnostic content should change repair success without any model retraining; that is a direct, testable extension the paper does not run.
  • The diffused-versus-localized axis may transfer to other long-input software tasks such as issue triage, code review, or vulnerability analysis, where the same occlusion-style attention probes could be applied.
  • Because masking deletes content rather than replacing it, some measured 'attention' could actually be sensitivity to prompt formatting; a placeholder-controlled replication would separate semantic attention from format sensitivity.
  • Developer attention alignment could become a training signal for reinforcement learning or preference optimization, but the paper only proposes it as future supervision, not as a proven recipe.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies how LLMs distribute attention to sections and fine-grained components of bug reports during automated program repair. Using perturbation-based attribution (Kernel SHAP and UniXcoder similarity), the authors compute importance scores for six report sections and ten code/NL component types across 319 Python and Java bugs from SWE-bench Verified and Multi-SWE-bench, for three LLMs in RQ1 and one LLM (plus a Java-only replication) in RQ2/RQ3. They classify attention into three structures (Diffused, Localized, No-attention) and relate these to repair success. They also collect developer annotations on 100 reports and measure alignment between model attention and developer-identified sections/phrases. The paper reports that successful repairs show diffused attention and attention to descriptions/stacktraces, while failures show localized attention to version metadata; and that model–developer alignment predicts success.

Significance. If the results were robust, the paper would provide a useful new empirical perspective on LLM-based APR failures and a reusable developer-attention dataset. However, the central quantitative claims rest on a subjective, non-operationalized label mapping (§3.1.2) and on an occlusion-based measure of attention that the authors themselves concede is approximate (§6). These issues affect the interpretation of the headline odds ratios and need to be resolved before the conclusions can be accepted. The paper's statistical care (FDR correction, difficulty controls, bootstrap CIs, replication package) is a strength, and the developer dataset is a valuable artifact.

major comments (3)
  1. [§3.1.2 (RQ2), Figures 2–4, Table 3] The transition from per-component importance scores to the three attention-structure labels is not operationalized. The text states only that 'We visualize component-level importance scores... We then analyze these visualizations to identify recurring attention patterns.' No quantitative rule defines 'distributed across multiple components' (Diffused), 'focuses heavily on a single component' (Localized), or 'no changes' (No-attention), and no inter-rater reliability is reported for this classification. Table 3, the paper's central quantitative evidence for RQ2, derives Fisher tests and odds ratios (Diffused OR=2.07; Localized OR=0.40) from these labels. Unlike the RQ3 developer annotations, no agreement metric (e.g., Cohen's kappa) is provided for the attention-pattern labels. The authors should operationalize the thresholds (e.g., entropy or Gini index of normalized importance scores, t
  2. [§2, §3.1.1, §6 (Construct validity), Tables 3 and 7] The paper equates 'attention' with output sensitivity to input perturbations. The §6 construct-validity paragraph concedes that this 'may not fully reflect model reasoning,' but the central conclusion—'attention misallocation is a key factor in LLM-based APR failures'—requires the proxy to be diagnostic. A more specific worry is circularity: the success/failure outcome is determined from the same generated patches that define sensitivity. A correct patch must, by construction, be responsive to the bug-report content that led to it, while an incorrect patch may be unresponsive for unrelated reasons (e.g., the model latched onto a wrong pattern or produced a generic patch). This could inflate the observed association between Diffused attention and successful repair. The No-attention row in Table 3 (OR=1.0) mitigates the simplest mechanistic version of this concern, but the Diffused/Localiz
  3. [§4.1, Table 2; Table 5] The cross-model consistency claim is overstated. The text states that 'Version information' shows large negative effect sizes for each model and that 'the direction and magnitude of the attention differences are consistent.' However, for gpt-oss-20b the effect for Version information is −0.20 and is not marked significant in Table 2 (unlike the values for claude-4-sonnet and qwen-3-32b). In the Java-only RQ2 replication with gpt-oss-20b (Table 5), attention to NL:Version information has OR=0.92 with p=0.683, i.e., no evidence of a negative association. The paper should temper the claim of model-invariant over-attention to version metadata and present the gpt-oss results as weaker/statistically nonsignificant. This matters because the abstract generalizes the finding across models.
minor comments (4)
  1. [§4.1, Table 1] The Hard category contains only 11 bugs; the chi-square p=0.06 and the difficulty-controlled logistic regressions should be read with this low power in mind. Also, the text and Table 1 use 'claude-4' and 'claude-4-sonnet' interchangeably.
  2. [Table 3] The No-attention row reports OR=1.0 [1.0,1.0] exactly. This is likely a rounding artifact; please report more decimal places and clarify the reference coding used for the logistic regression.
  3. [Table 7] Nine Mann-Whitney tests are reported without a multiple-testing correction. Reporting FDR-adjusted q-values would be more consistent with the paper's RQ2 methodology. The absolute effect sizes are also small (e.g., F1@20: 0.19 vs. 0.08); please discuss practical significance.
  4. [§3.1.1, §3.1.2] For reproducibility, specify the exact UniXcoder similarity computation (layer, pooling, normalization) and the CodeBLEU configuration used in the SHAP value estimation.

Circularity Check

0 steps flagged

No significant circularity: the attention–success associations are empirical correlations, not definitional reductions.

full rationale

The paper's central claim is an empirical association between perturbation-derived attention measures and repair success, not a derivation in which the dependent variable is defined in terms of the independent variable. RQ1 attention scores come from SHAP over CodeBLEU output differences; success is CodeBLEU against the ground-truth patch. RQ2 component importance comes from UniXcoder similarity between baseline and perturbed outputs; success remains CodeBLEU against ground truth. RQ3 compares developer annotations to model attention. In no equation is 'success' defined as 'diffuse attention' or vice versa, and no fitted parameter is relabeled as a prediction. The 'Diffused/Localized/No-attention' labels are assigned by visual inspection without an explicit threshold (§3.1.2), and §6 concedes perturbation 'may not fully reflect model reasoning'; these are construct-validity and reliability threats, not circularity. Notably, Table 3 reports No-attention OR = 1.0 with no association, which argues against the mechanical concern that failed repairs are simply insensitive to perturbations. The self-citations ([12], [13], [14]) are used for prompt design, fault localization, and motivating examples, not as load-bearing evidence for the attention–success claim. Therefore no circular step can be identified by the paper's own equations or definitions.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical or conceptual entities. Its contribution rests instead on interpreting occlusion sensitivity as attention and on a hand-assigned taxonomy of attention patterns. No numerical parameters are fitted to the outcome, but the pattern-classification thresholds are unspecified, which is the main unstated modeling choice.

free parameters (1)
  • Attention-pattern classification thresholds (Diffused/Localized/No-attention) = not reported
    RQ2 labels each bug's attention structure by visual inspection of importance-score visualizations; no numeric thresholds or automated decision rule are given, so the category boundaries are effectively hand-set. The central claim depends on these categories.
axioms (6)
  • domain assumption Occlusion sensitivity equals attention
    §2: if removing a portion significantly alters the output, that component 'likely played an important role'; §6 admits it 'may not fully reflect model reasoning'. All RQs depend on this proxy.
  • domain assumption Markdown headers reliably identify semantic bug-report sections
    §3: sections are identified 'using the headers present in the markdown'; real bug reports are heterogeneous and may have nonstandard formatting.
  • domain assumption CodeBLEU and UniXcoder capture relevant output change
    §3.1: SHAP values are computed over CodeBLEU differences in RQ1, and RQ2 uses UniXcoder semantic distance. These metrics determine all attention scores.
  • domain assumption Single-developer annotation captures developer importance
    §3.1.3: each report is annotated by one developer by design; agreement is checked only against the first author's re-annotation, not against independent developers.
  • domain assumption Zero-temperature deterministic decoding isolates perturbation effects
    §3: temperature is set to zero and reasoning modes disabled; the paper assumes output differences are due to prompt perturbation rather than sampling.
  • domain assumption SWE-bench Verified and Multi-SWE-bench reports are representative real-world bug reports
    §3 and §6: benchmarks are manually curated and may not generalize to other languages, projects, or less structured reports; the authors acknowledge this external-validity threat.

pith-pipeline@v1.3.0-alltime-deepseek · 23213 in / 10278 out tokens · 94341 ms · 2026-08-01T01:13:17.248441+00:00 · methodology

0 comments
read the original abstract

Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent. Even when provided with the same contextual information, an LLM may generate a correct patch for one bug but fail on another closely related bug. Why this happens remains poorly understood, and it is unclear how LLMs prioritize the diverse information in bug reports and whether model attention affects repair success. In this paper, we present the first empirical study of attention patterns in LLM-based program repair, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair. We analyze 319 real-world Python and Java bugs from SWE-bench Verified and Multi-SWE-bench to study (RQ1) how model attention is distributed across bug report sections, (RQ2) how attention patterns within each section differ between successful and unsuccessful repairs, and (RQ3) how these patterns compare to information developers consider important for bug fixing. We find that successful repairs are characterized by diffused attention across multiple diagnostic components such as bug descriptions, stacktraces, and test cases, while failures often exhibit over-localized attention toward metadata such as version information. We further observe that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success. Our results provide the first empirical evidence that attention misallocation is a key factor in LLM-based APR failures, and offer actionable insights for designing more interpretable and reliable future APR systems.

Figures

Figures reproduced from arXiv: 2607.25873 by Irene Manotas, Luca Buratti, Preetha Chatterjee, Ramtin Ehsani, Saurabh Pujar.

Figure 1
Figure 1. Figure 1: Generated patches for bug instances mat￾plotlib#16344 and #16420. This behavior is not unique to this example. Such inconsisten￾cies are frequently observed in LLM-based program repair. Even when models are provided with similar information, their repair performance can vary substantially across seemingly similar types of bugs [12–14, 63]. This unpredictability is widely recognized as one of the central ch… view at source ↗
Figure 2
Figure 2. Figure 2: Example of Diffused Attention Pattern in LLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of Localized Attention Pattern in LLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example of No-attention Pattern in LLMs. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of Bug Description and Version Informa [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of Diffused Attention Throughout the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Two Examples of Localized Attention in Reports of [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Misaligned attention between developer and LLM [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Aligned attention between developer and LLM for [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 6 canonical work pages

  1. [1]

    Nikta Akbarpour, Mahdieh Sadat Benis, Fatemeh Hendijani Fard, Ali Ouni, and Mohamed Aymen Saied. 2025. Collaborative Agents for Automated Program Repair in Ruby. arXiv:2511.03925 [cs.SE]

    arXiv 2025

  2. [2]

    Anthropic. 2025. https://www.anthropic.com/news/claude-4

    2025

  3. [3]

    Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretabil- ity room: Why use attention as explanation when we have saliency methods? arXiv:2010.05607 [cs.CL] https://arxiv.org/abs/2010.05607

    Pith/arXiv arXiv 2020

  4. [4]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.Journal of the Royal Statistical Society Series B: Statistical Methodology57, 1 (Jan. 1995), 289–300

    1995

  5. [5]

    Sicong Cao, Xiaobing Sun, Ratnadira Widyasari, David Lo, Xiaoxue Wu, Lili Bo, Jiale Zhang, Bin Li, Wei Liu, Di Wu, and Yixin Chen. 2025. A Systematic Literature Review on Explainability for ML/DL-based Software Engineering.ACM Comput. Surv.58, 4, Article 95 (Oct. 2025), 34 pages. doi:10.1145/3763230

    work page doi:10.1145/3763230 2025

  6. [6]

    Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. 2019. Assessing the quality of the steps to reproduce in bug reports. InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Ta...

    2019

  7. [7]

    Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. 2017. Detecting missing information in bug descriptions. InProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering(Paderborn, Germany)(ESEC/FSE 2017). Association for Computing Machinery, New York, NY,...

    2017

  8. [8]

    Preetha Chatterjee, Minji Kong, and Lori Pollock. 2020. Finding Help with Programming Errors: An Exploratory Study of Novice Software Engineers’ Focus in Stack Overflow Posts.Journal of Systems and Software159 (2020), 110454

    2020

  9. [9]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What Does BERT Look at? An Analysis of BERT’s Attention. InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes (Eds.). Association for Computational Linguistics,...

    2019

  10. [10]

    William G. Cochran. 1977.Sampling Techniques(3rd ed.). John Wiley & Sons, New York, NY

    1977

  11. [11]

    Ximing Dong, Shaowei Wang, Dayi Lin, Gopi Krishnan Rajbahadur, and Ahmed E. Hassan. 2025. Promptexp: Multi-Granularity Prompt Explanation of Large Lan- guage Models. In2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware). 01–10. doi:10.1109/AIware69974.2025.00027

    arXiv 2025

  12. [12]

    Ramtin Ehsani, Esteban Parra, Sonia Haiduc, and Preetha Chatterjee. 2025. Hi- erarchical Knowledge Injection for Improving LLM-based Program Repair. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1440–1452. doi:10.1109/ASE63991.2025.00122

    arXiv 2025

  13. [13]

    Ramtin Ehsani, Sakshi Pathak, and Preetha Chatterjee. 2025. Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 699–711. doi:10.1109/MSR66628.2025.00107

    arXiv 2025

  14. [14]

    Ramtin Ehsani, Sakshi Pathak, Esteban Parra, Sonia Haiduc, and Preetha Chatter- jee. 2025. What characteristics make ChatGPT effective for software issue reso- lution? An empirical study of task, project, and conversational signals in GitHub issues.Empirical Software Engineering31, 1 (Nov. 2025). doi:10.1007/s10664-025- 10745-8

    work page doi:10.1007/s10664-025- 2025

  15. [15]

    An Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL]

    Pith/arXiv arXiv 2025

  16. [16]

    Daoguang Zan et al. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv:2504.02605 [cs.SE]

    Pith/arXiv arXiv 2025

  17. [17]

    Md Tahmid Rahman Laskar et al. 2024. A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommen- dations. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miam...

    2024

  18. [18]

    OpenAI et al. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL] https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  19. [19]

    Stella Biderman et al. 2024. Lessons from the Trenches on Reproducible Evalua- tion of Language Models. arXiv:2405.14782 [cs.CL]

    Pith/arXiv arXiv 2024

  20. [20]

    Xingyao Wang et al. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv:2407.16741 [cs.SE]

    Pith/arXiv arXiv 2025

  21. [21]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan

  22. [22]

    Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar, Avraham Shinnar, Ziv Nevo, and Martin Hirzel. 2026. Resolving Java Code Repository Issues with iSWE Agent. arXiv:2603.11356 [cs.SE] https://arxiv.org/abs/2603.11356

    arXiv 2026

  23. [23]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin

  24. [24]

    Mirazul Haque, Petr Babkin, Farima Farmahinifarahani, and Manuela Veloso. 2025. Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs. InProceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing. Association for Computational Linguis- tics, Albuquerque, New Mexico, USA, 160–179

    2025

  25. [25]

    Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. 2025. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. arXiv:2504.07086 [cs.LG]

    arXiv 2025

  26. [26]

    Haichuan Hu, Ye Shang, Weifeng Sun, and Quanjun Zhang. 2025. TSAPR: A Tree Search Framework For Automated Program Repair. arXiv:2507.01827 [cs.SE]

    Pith/arXiv arXiv 2025

  27. [27]

    Yuheng Huang, Lei Ma, Keizaburo Nishikino, and Takumi Akazaki. 2025. Risk as- sessment framework for code llms via leveraging internal states. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 432–443

    2025

  28. [28]

    Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. InPro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minn...

    2019

  29. [29]

    Joseph James. 2026. Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation. arXiv:2603.06865 [cs.CL]

    arXiv 2026

  30. [30]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 1430–1442

    2023

  31. [31]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  32. [32]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(San Francisco, CA, USA)(ESEC/FSE 2023). Association for Computing...

    2023

  33. [33]

    Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Ivan Radiček, and Gust Verbruggen. 2023. Repair is nearly generation: multilingual program repair with LLMs. InProceedings of the Thirty-Seventh AAAI Conference on Artifi- cial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposiu...

    2023

  34. [34]

    2024.AI-powered patching: the future of auto- mated vulnerability fixes

    Jan Keller and Jan Nowakowski. 2024.AI-powered patching: the future of auto- mated vulnerability fixes. Technical Report

    2024

  35. [35]

    Hae-Young Kim. 2017. Statistical notes for clinical researchers: Chi-squared test and Fisher’s exact test.Restorative Dentistry & Endodontics42, 2 (May 2017), 152–155. doi:10.5395/rde.2017.42.2.152

    work page doi:10.5395/rde.2017.42.2.152 2017

  36. [36]

    Shunsuke Kitada and Hitoshi Iyatomi. 2021. Attention Meets Perturbations: Robust and Interpretable Attention With Adversarial Training.IEEE Access9 (2021), 92974–92985. doi:10.1109/ACCESS.2021.3093456

    arXiv 2021

  37. [37]

    Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong. 2017. Robust Statistical Methods for Empirical Software Engineering.Empirical Software Engineering22, 2 (April 2017), 579–630. doi:10.1007/s10664-016-9437-5 How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs...

    doi:10.1007/s10664-016-9437-5 2017

  38. [38]

    Ko, Brad A

    Amy J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant In- formation during Software Maintenance Tasks.IEEE Transactions on Software Engineering32, 12 (2006), 971–987. doi:10.1109/TSE.2006.116

    doi:10.1109/tse.2006.116 2006

  39. [39]

    Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, and Tianyi Zhang. 2024. Do large language models pay similar attention like human programmers when generating code?Proceedings of the ACM on Software Engineering1, FSE (2024), 2261–2284

    2024

  40. [40]

    Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the Dark Secrets of BERT. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Associatio...

    doi:10.18653/v1/d19-1445 2019

  41. [41]

    Stefano Lambiase, Gemma Catolino, Fabio Palomba, and Filomena Ferrucci. 2024. Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review.ACM Comput. Surv.57, 4, Article 93 (Dec. 2024), 37 pages. doi:10.1145/3704806

    doi:10.1145/3704806 2024

  42. [42]

    Thanh Le-Cong, Bach Le, and Toby Murray. 2025. Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generation. arXiv:2410.16655 [cs.SE] https://arxiv.org/abs/2410.16655

    arXiv 2025

  43. [43]

    Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. 2025. Evaluating the Generalizability of LLMs in Automated Program Repair. arXiv:2503.09217 [cs.SE]

    Pith/arXiv arXiv 2025

  44. [44]

    Hongyan Li, Meng Yan, Weifeng Sun, Xiao Liu, and Yunsong Wu. 2023. A first look at bug report templates on GitHub.Journal of Systems and Software202 (Aug. 2023), 111709. doi:10.1016/j.jss.2023.111709

    arXiv 2023

  45. [45]

    Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Team- mates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE]

    Pith/arXiv arXiv 2025

  46. [46]

    Jiliang Li, Yifan Zhang, Zachary Karas, Collin McMillan, Kevin Leach, and Yu Huang. 2024. Do machines and humans focus on similar code? exploring ex- plainability of large language models in code summarization. InProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 47–51

    2024

  47. [47]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Informa- tion Processing Systems37 (2024), 22947–22970

    2024

  48. [48]

    Zhong Li, Chong Zhang, Minxue Pan, Tian Zhang, and Xuandong Li. 2024. AACEGEN: Attention Guided Adversarial Code Example Generation for Deep Code Models. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machinery, New York, NY, USA, 1245–1257

    2024

  49. [49]

    Shanchao Liang, Nan Jiang, Yiran Hu, and Lin Tan. 2025. Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (Eds.). Association for Computat...

    doi:10.18653/v1/2025.acl-long.1204 2025

  50. [50]

    Shanchao Liang, Nan Jiang, Shangshu Qian, and Lin Tan. 2025. WAFFLE: Fine- tuning Multi-Modal Model for Automated Front-End Development. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Com...

    work page doi:10.18653/v1/2025.acl-long.1208 2025

  51. [51]

    Rensis Likert. 1932. A technique for the measurement of attitudes.Archives of Psychology22, 140 (1932), 1–55

    1932

  52. [52]

    Bissyandé, Dongsun Kim, Jacques Klein, and Yves Le Traon

    Kui Liu, Anil Koyuncu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, and Yves Le Traon. 2019. You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems. In2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). 102–113. doi:10.1109/ICST.2019.00020

    arXiv 2019

  53. [53]

    Gianluigi Lopardo, Frederic Precioso, and Damien Garreau. 2024. At- tention Meets Post-hoc Interpretability: A Mathematical Perspective. arXiv:2402.03485 [stat.ML] https://arxiv.org/abs/2402.03485

    arXiv 2024

  54. [54]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 4768–4777

    2017

  55. [55]

    Mary McHugh. 2012. Interrater reliability: The kappa statistic.Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB22 (10 2012), 276–82

    2012

  56. [56]

    Mary L. McHugh. 2013. The chi-square test of independence.Biochemia Medica 23, 2 (2013), 143–149. doi:10.11613/bm.2013.018

    doi:10.11613/bm.2013.018 2013

  57. [57]

    Niklas Metzger, Christopher Hahn, Julian Siber, Frederik Schmitt, and Bernd Finkbeiner. 2022. Attention Flows for General Transformers. arXiv:2205.15389 [cs.LG] https://arxiv.org/abs/2205.15389

    Pith/arXiv arXiv 2022

  58. [58]

    Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. 2020. Rethinking Self-Attention: Towards Interpretability in Neural Parsing. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Com- putational Linguistics, Online, 731–742. doi:1...

    doi:10.18653/v1/2020.findings-emnlp.65 2020

  59. [59]

    Zheng Ning, Yuan Tian, Zheng Zhang, Tianyi Zhang, and Toby Jia-Jun Li. 2024. Insights into natural language database query errors: From attention misalign- ment to user handling strategies.ACM Transactions on Interactive Intelligent Systems14, 4 (2024), 1–32

    2024

  60. [60]

    NLTK. 2025. https://www.nltk.org/

    2025

  61. [61]

    Replication Package. 2026. https://doi.org/10.5281/zenodo.21381449

    work page doi:10.5281/zenodo.21381449 2026

  62. [62]

    Henley, and Albert Ziegler

    Matteo Paltenghi, Rahul Pandita, Austin Z. Henley, and Albert Ziegler. 2024. Follow-Up Attention: An Empirical Study of Developer and Neural Model Code Exploration.IEEE Transactions on Software Engineering50, 10 (2024), 2568–2582

    2024

  63. [63]

    Barr, and Sergey Mechtaev

    Nikhil Parasaram, Huijie Yan, Boyu Yang, Zineb Flahy, Abriele Qudsi, Damian Ziaber, Earl T. Barr, and Sergey Mechtaev. 2025. The Fact Selection Problem in LLM-Based Program Repair. InProceedings of the IEEE/ACM 47th International Conference on Software Engineering(Ottawa, Ontario, Canada)(ICSE ’25). IEEE Press, 2574–2586. doi:10.1109/ICSE55347.2025.00162

    arXiv 2025

  64. [64]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE]

    Pith/arXiv arXiv 2020

  65. [65]

    Sofia Serrano and Noah A. Smith. 2019. Is Attention Interpretable?. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 2931–2951. doi:10.18653/v1/P19-1282

    doi:10.18653/v1/p19-1282 2019

  66. [66]

    Lin Shi, Fangwen Mu, Yumin Zhang, Ye Yang, Junjie Chen, Xiao Chen, Hanzhi Jiang, Ziyou Jiang, and Qing Wang. 2022. BugListener: identifying and syn- thesizing bug reports from collaborative live chats. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New Yo...

    2022

  67. [67]

    Yu Shi, Abdul Ali Bangash, Emad Fallahzadeh, Bram Adams, and Ahmed E. Hassan. 2025. HAFix: History-Augmented Large Language Models for Bug Fixing. arXiv:2501.09135 [cs.SE] https://arxiv.org/abs/2501.09135

    arXiv 2025

  68. [68]

    Mozhan Soltani, Felienne Hermans, and Thomas Bäck. 2020. The significance of bug report elements.Empirical Software Engineering25, 6 (Nov. 2020), 5255–5294

    2020

  69. [69]

    Sonar. 2025. https://www.sonarsource.com

    2025

  70. [70]

    Stackoverflow. 2025. https://survey.stackoverflow.co/2025/technology

    2025

  71. [71]

    Streamlit. 2025. https://streamlit.io/

    2025

  72. [72]

    Caizhi Tang, Qing Cui, Longfei Li, and Jun Zhou. 2023. GINT: A Generative Interpretability method via perturbation in the latent space.Expert Systems with Applications232 (Dec. 2023), 120570. doi:10.1016/j.eswa.2023.120570

    arXiv 2023

  73. [73]

    Graham J. G. Upton. 1992. Fisher’s Exact Test.Journal of the Royal Statistical Society. Series A (Statistics in Society)155, 3 (1992), 395–402. doi:10.2307/2982890

    work page doi:10.2307/2982890 1992

  74. [74]

    Thomas Valentin, Ardi Madadi, Gaetano Sapia, and Marcel Böhme. 2025. In- coherence as Oracle-less Measure of Error in LLM-Based Code Generation. arXiv:2507.00057 [cs.PL] https://arxiv.org/abs/2507.00057

    arXiv 2025

  75. [75]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  76. [76]

    Chunqiu Steven Xia, Yifeng Ding, and Lingming Zhang. 2024. The Plastic Surgery Hypothesis in the Era of Large Language Models. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering(Echter- nach, Luxembourg)(ASE ’23). IEEE Press, 522–534

    2024

  77. [77]

    Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang

  78. [78]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-Trained Language Models. InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia)(ICSE ’23). IEEE Press, 1482–1494

    2023

  79. [79]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis(Vienna, Austria)(ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 819–831. doi:10.1145/3650212.3680323

    arXiv 2024

  80. [80]

    Boyang Yang, Luyao Ren, Xin Yin, Jiadong Ren, Haoye Tian, and Shunfu Jin. 2025. Input Reduction Enhanced LLM-based Program Repair. arXiv:2507.15251 [cs.SE]

    arXiv 2025

Showing first 80 references.