Pith. sign in

REVIEW 4 major objections 5 minor 52 references

This paper shows that the behavioral cues in bug reports can be reliably extracted and tracked through tests and patches, making semantic alignment a measurable engineering signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:03 UTC pith:KTO646CB

load-bearing objection Good descriptive study of semantic drift with real human grounding, but the headline engineering claim leans on an unvalidated proxy chain and unevaluated algorithms. the 4 major comments →

arxiv 2607.18550 v1 pith:KTO646CB submitted 2026-07-20 cs.SE

Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

Wendk\^uuni C. Ou\'edraogo , Yinghua Li , Xueqi Dang , Pawe{\l} Borsukiewicz , Liang Xiao , Lingfeng Bao , Anil Koyuncu , Jacques Klein
show 2 more authors
David Lo Tegawend\'e F. Bissyand\'e
This is my paper
classification cs.SE
keywords semantic alignmentbug reporttest generationprogram repairfault localizationLLM evaluationbehavioral anchorssemantic drift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Desc2Fix, a framework that quantifies how behavioral signals in bug reports (exceptions, APIs, input conditions, reproduction steps, expected vs. actual behavior) propagate into triggering tests and corrective patches. It claims that LLMs can extract these structured anchors with high completeness and cross-model stability, and that alignment between artifacts is measurable but not reducible to similarity: lexical overlap is near zero, embedding similarity is moderate, full diffs give the most stable report–patch signal, and structured summaries shift from surface similarity to predicate-level salience. It also documents that two LLM judges systematically over-score alignment relative to humans by 1–2 points on 5-point scales and have only modest rank agreement, so alignment scores require bias-aware calibration. If correct, this turns alignment into a controllable signal for ranking tests, filtering candidate patches, guiding fault localization, and improving bug report authoring.

Core claim

The central claim is that cross-artifact behavioral alignment can be operationalized, measured, and used as an engineering signal, and that it is representation-sensitive rather than intrinsic. Using a six-field structured schema (exception, api_involved, input_conditions, reproduction_steps, expected, actual) and four Likert dimensions (coverage, correctness, specificity, alignment), the authors find that LLM extraction achieves 83–90% completeness with near-ceiling semantic fidelity on populated fields (F1 ≥ 0.93) and 92–94% cross-model fill/abstain agreement. Report–test alignment is predominantly semantic rather than lexical (embedding cosine ≈ 0.47–0.59 vs. ROUGE-L ≤ 0.07), and OpenAI e

What carries the argument

The load-bearing mechanism is the structured behavioral anchor schema: a six-field JSON representation (exception, api_involved, input_conditions, reproduction_steps, expected, actual) that LLMs extract from bug reports, combined with a taxonomy of alignment dimensions (Entity, Scenario, Behavioral) and a four-axis Likert judgment protocol (coverage, correctness, specificity, alignment). This schema converts free-form reports into a normalized 'semantic input contract' that can be compared to tests and patches through deterministic metrics (ROUGE, Jaccard, SBERT, CodeBERT, OpenAI embeddings) and LLM-based scoring. The framework additionally varies the patch view (full diff vs. additions vs.

Load-bearing premise

The entire measurement depends on the six-field schema and four Likert dimensions adequately capturing what developers mean by 'behavioral alignment'; if this operationalization omits or misweights semantic cues, every alignment score, proxy, and optimism finding is an artifact of that schema.

What would settle it

Take a random sample of report–test–patch triplets, have human experts rate them with an independent schema or free-form judgment, and compare rankings to Desc2Fix's embedding-cosine and LLM-derived scores. If the rank correlation is near zero or negative—for instance, if a patch rated highly aligned is judged by developers as not restoring the reported behavior—the alignment construct fails to capture developer intent. A concrete observation: find a triplet where the test passes and the patch is minimal but the report's expected/actual contrast is not enforced by the test; Desc2Fix would like

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Test generation pipelines can rank and filter candidate tests using embedding cosine between structured report anchors and test representations, augmented by anchor-coverage checks, instead of lexical overlap or raw LLM scores.
  • Repair validation should evaluate candidate patches as full diffs with embedding-based semantic alignment plus anchor checks, providing a complement to fail-to-pass criteria that can detect overfitting fixes.
  • LLM-based alignment scores should be calibrated against human references or treated as heuristics, since systematic optimism and weak rank agreement make raw scores unsuitable as decision signals.
  • Bug report authors can improve actionability by writing explicit behavioral anchors (APIs, reproduction steps, expected vs. actual contrast) rather than longer narratives, which dilute perceived alignment.
  • Fault localization can use structured anchors as stable semantic queries, avoiding the inflation that verbosity causes in surface-similarity measures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework's alignment numbers are conditional on the six-field schema; a different schema could yield different rankings, and a schema-ablation study could reveal how much of the measured alignment is an artifact of the chosen fields.
  • The systematic LLM optimism observed here suggests that using LLM alignment scores as reward signals for fine-tuning repair models would require explicit bias correction; uncorrected, they may over-reward patches that merely look aligned.
  • The full-diff stability finding may transfer to agentic repair pipelines, where evaluating intermediate edits as isolated hunks could misjudge whether the patch restores the reported behavior.
  • The OpenAI embedding cosine proxy could serve as a cheap, deterministic surrogate for human alignment judgments in triage workflows, but its rank agreement with human raters should be established per dataset before deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Desc2Fix, a framework for measuring semantic alignment between bug reports, triggering tests, and human-written patches across 2,857 triplets from Defects4J and SWT-Bench. Alignment is operationalized through a six-field structured extraction schema (exception, API, input conditions, reproduction steps, expected, actual), four Likert dimensions (coverage, correctness, specificity, alignment), deterministic similarity metrics (ROUGE, Jaccard, SBERT, CodeBERT, OpenAI embeddings), and LLM judgments from GPT-4o and DeepSeek-Chat, with human annotations on 400 reports as a reference. The study reports three main results: (RQ1) LLM-based structured extraction is dense and semantically stable, with near-ceiling content-only fidelity; (RQ2) report–test alignment is moderately strong but judge- and representation-dependent, LLM judges are optimistically biased and weakly rank-aligned with humans, and OpenAI embedding cosine is the best deterministic proxy for LLM judgments; (RQ3) report–patch alignment is similarly framing-dependent, with full diffs providing the most stable signal, and LLM judges again showing systematic optimism. Based on these findings, the paper proposes two reference algorithms (Algorithms 1 and 2) that use structured anchors and embedding-based proxies for test generation and patch ranking. The headline claim is that behavioral alignment is measurable, not reducible to similarity, and that structured anchors plus embedding proxies provide re

Significance. If the main claims hold, the paper would provide a large-scale, cross-language empirical characterization of how behavioral signals propagate across bug reports, tests, and patches, and would offer actionable, model-agnostic signals for test generation, fault localization, and patch ranking. The descriptive findings—representation sensitivity, LLM optimism, lexical insufficiency of alignment, and the advantage of full diffs—are valuable and are supported by triangulation across two datasets, two LLM families, multiple metric families, and human annotations on 400 reports. The release of an annotated corpus and a fully reproducible research package is a concrete strength that will benefit the community. However, the central engineering claim—that the deterministic embedding proxy (OpenAI cosine) is a human-grounded, reproducible signal suitable for deployment—is not established by the evidence as presented. The proxy is validated only against LLM judgments that the paper itself shows are weakly correlated with human ratings (Tables 10 and 17). The paper is honest about this limitation in places, but the abstract and Section 4.2 Finding 8 state the actionable claim more strongly than

major comments (4)
  1. [Section 4.2, Finding 8; Table 13] The central engineering claim that OpenAI embedding cosine is a ‘reproducible signal for ranking and filtering tests and candidate patches’ (abstract; Algorithm 1 line 12) is supported only by correlation with LLM alignment scores, not with human ratings. Tables 10 and 17 show that the same LLM scores correlate weakly with human judgments (ρ mostly in [-0.13, 0.25] for report–test; similar or lower for report–patch), and the paper itself warns (Finding 6) that LLM judges should not be used out-of-the-box. Validating a proxy against a criterion that is itself weakly aligned with the human reference does not establish that the proxy is human-grounded; it may merely reproduce LLM optimism. The 400-report human sample (Section 3.9) is never used to compute the correlation between OpenAI cosine and human alignment scores. This is a circular validation chain for the paper’s main actionable sig
  2. [Section 4.3, Table 17; Abstract] The abstract states that both LLMs show systematic optimism ‘relative to humans (1–2 points on 5-point scales).’ Table 17 reports mean biases up to +3.03 (DeepSeek-Chat, Defects4J/RAW-remove) and MAE up to 3.05, with most RAW settings exceeding +2.0. The ‘1–2 points’ characterization understates the observed worst case by about a factor of 1.5–3. This is not merely a cosmetic discrepancy: the magnitude of the bias is directly relevant to the advice that LLM scores should be bias-calibrated before use (Section 5.3). Please revise the abstract and any summary statements to report the full range of observed bias and MAE, or add a qualifier such as ‘1–3 points.’
  3. [Section 4.3, Findings 10–11; Table 17] The paper reports that full-diff views yield the highest and most stable LLM alignment scores (Finding 10), and that ‘structured behavioral cues combined with addition-focused diffs partially recover human-consistent ranking’ (Section 4.3, paragraph before Table 17). However, Table 17 shows that even the best condition (SWT-Bench STRUCTURED+add) reaches ρ≈0.33 for GPT-4o, far below inter-human agreement (Table 16: ρ up to 0.98), while bias remains high (+2.04). The statement in Section 5.1 that ‘full diffs yield the most stable alignment signal’ is supported by metric-based analysis, but the recommendation to use embedding-based proximity for patch ranking (Algorithm 2) rests on the same unvalidated proxy issue as comment 1. The paper should explicitly separate (i) descriptive findings about what affects LLM judgments, which are well-supported, from (ii) prescriptive recommendations abou
  4. [Section 3.4; Section 5.4 (Construct validity)] The entire measurement depends on the six-field schema and four Likert dimensions adequately operationalizing ‘behavioral alignment.’ The paper acknowledges this in Section 5.4, but the threat is under-addressed for the central claim. Specifically, the human annotations in Section 3.9 use the same schema to guide human ratings, so the high inter-annotator agreement may reflect the annotators’ adherence to the provided schema rather than independent confirmation that the schema captures developer intent. A more convincing construct validation would compare the framework’s alignment scores with an independent developer-level outcome, such as whether tests that score higher under the framework are more likely to detect real regressions, or whether patches with higher alignment scores are preferred by developers in a blind comparison. Such evidence is not currently provided, and without it t
minor comments (5)
  1. [Section 3.2] The example JSON output is helpful, but the field names use underscores (api_involved, input_conditions, reproduction_steps) while the text and tables sometimes use spaces or different capitalization (e.g., ‘Expected’, ‘Actual’ in Figure 2). Please standardize the naming throughout for clarity and reproducibility.
  2. [Table 16] In Table 16(b), several SWT-Bench RAW cells report ‘–’ for Spearman/Kendall due to constant inputs, but the MAE is reported as 0.00. It would be useful to note explicitly that MAE=0 in these cells means both annotators assigned identical scores, not that the metric is undefined. Also consider adding the number of pairs used for each view, as done elsewhere.
  3. [Section 3.9] The paper says that annotators ‘qualitatively inspected representative report–test and report–patch triplets to validate LLM alignment scores’ and that this ‘informed minor prompt clarifications while preserving evaluation independence.’ It is not clear when these prompt changes occurred relative to the main LLM evaluation runs. If the prompt clarifications were applied after the initial runs, or if the human-in-the-loop inspection influenced the final prompts, the reported LLM scores may not reflect a fully pre-registered protocol. Please provide a timeline or clarify that the prompts were fixed before the main runs.
  4. [Section 4.2, Finding 8] The sentence ‘OpenAI embedding cosine provides a stable and practical proxy for LLM alignment judgments’ is accurate, but the wording ‘practical proxy’ is later used to justify replacing LLM judges for ranking/filtering tests. Even if the proxy is a good approximation of LLM judgments, the previous comment about human grounding applies. Consider using ‘proxy for LLM judgments’ consistently to avoid overclaiming.
  5. [Algorithm 1 and Algorithm 2] The hyperparameters (α, β, γ, λ, τ_a, τ_s, N, top-k) are introduced without a sensitivity analysis or guidance for selecting them. Since the paper emphasizes reproducibility, stating default values and a small sensitivity check (e.g., varying α/β over a reasonable range) would strengthen the practical contribution. At minimum, list the values used in the reference implementation.

Circularity Check

0 steps flagged

No circular derivation: extraction and alignment findings are anchored in independent human annotations; the embedding proxy is a disclosed empirical predictor of LLM scores, not a fitted parameter renamed as a prediction.

full rationale

The paper's derivation chain is self-contained. RQ1 extraction quality is evaluated against 400 independently human-annotated reports with high inter-annotator agreement (Section 3.9, Tables 5-6), and RQ2/RQ3 LLM alignment judgments are explicitly compared with human ratings, with the paper reporting weak Human-LLM rank agreement and systematic optimism (Findings 6 and 11, Tables 10 and 17). The OpenAI embedding cosine proxy (Finding 8, Table 13) is selected by empirical correlation with LLM alignment scores, but it is not defined in terms of those scores, nor is it fitted to a subset of data and then presented as a prediction of human alignment; it is explicitly described as a proxy for LLM judgments. The paper discloses that LLM scores 'should not be used out-of-the-box' and that each proxy 'captures only part of the phenomenon' (Sections 4.2 and 5.4). Author self-citations (e.g., [18,33,34,36]) appear only as background motivation, not as load-bearing justification for the central empirical claims. The limitation that the embedding proxy is validated only against weakly human-aligned LLM judgments is a construct-validity gap, not a circular derivation, and the paper acknowledges this threat. Therefore no specific circular reduction can be exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

No parameters are fitted to make the descriptive analysis work; the only hand-chosen constants appear in the two unexecuted pseudo-algorithms. The main axioms are the representativeness of the Defects4J/SWT-Bench triplets, the reliability of two-expert human annotation, the construct validity of the six-field/four-dimension operationalization, and the stability of stochastic LLM ratings.

free parameters (2)
  • Algorithm 1 hyperparameters (α, β, γ, τ_a, τ_s, N, top-k)
    Algorithm 1 defines an adequacy score as α·ŝ_embed + β·ŝ_anch − γ(1−s_oracle) with thresholds, but no values or sensitivity analysis are given; the algorithm is not executed.
  • Algorithm 2 hyperparameters (α, β, λ, N, top-k)
    Algorithm 2 similarly defines utility as α·u_embed + β·u_anch + λ·u_min without calibration or evaluation; these are hand-chosen constants in an untested blueprint.
axioms (4)
  • domain assumption Defects4J v3.0.1 and SWT-Bench triplets are correctly reconstructed and representative of real bug-resolution workflows.
    Section 3.7 builds the corpus from public benchmarks; if the report-test-patch links are mispaired or unrepresentative, all alignment measurements inherit that bias.
  • domain assumption Two expert annotators on a 400-report sample provide reliable ground truth for extraction and alignment.
    Section 3.9 reports high inter-annotator agreement, but n=2 and the same fixed sample underlies all human calibration; unseen annotator or sampling variation could shift the optimism findings.
  • ad hoc to paper The six-field schema plus four Likert dimensions adequately operationalize behavioral alignment.
    Sections 3.2-3.4 define alignment through these fields and scores; this is a construct created for the paper, not independently established by prior literature.
  • domain assumption GPT-4o and DeepSeek-Chat API responses at temperature=1.0, top_p=1.0 are stable enough for the reported ratings.
    Sections 3.6 and 3.10 use stochastic decoding with no repeated-sampling reliability analysis; some reported correlations could be affected by sampling noise.
invented entities (1)
  • Desc2Fix framework and structured behavioral-anchor schema no independent evidence
    purpose: Operationalizes semantic alignment between bug reports, tests, and patches via six JSON fields and LLM/human/metric scores.
    The framework is a measurement construct introduced by the paper; it produces a benchmark, but no independent falsifiable prediction outside the paper's own corpus.

pith-pipeline@v1.3.0-alltime-deepseek · 36589 in / 11772 out tokens · 123310 ms · 2026-08-01T15:03:00.812437+00:00 · methodology

0 comments
read the original abstract

Bug resolution is a cross-artifact process: natural-language reports must expose actionable behavioral cues that can be reproduced by tests and corrected by patches. Yet the extent to which these signals are preserved across artifacts remains largely unquantified. We introduce Desc2Fix, a framework for measuring semantic alignment between bug reports, triggering tests, and developer-written fixes. Alignment is operationalized through structured behavioral anchors (e.g., reproduction steps, API/exception cues, expected vs. actual behavior), deterministic similarity metrics (ROUGE, SBERT, CodeBERT, OpenAI embeddings), and LLM-based judgments grounded in coverage, correctness, and specificity. We analyze 2,857 report-test-patch triplets from Defects4J and SWT-Bench using GPT-4o and DeepSeek-Chat. LLMs reliably extract structured signals (up to 90% completeness) and exhibit strong cross-model consistency, yielding a stable semantic input contract for downstream reasoning. However, alignment is highly representation-sensitive: lexical similarity alone is insufficient; full diffs provide the most stable basis for judging report-patch correspondence; and structured summaries trade surface similarity for predicate-level salience. Across more than 182,000 LLM alignment ratings, both models show systematic optimism relative to humans (1-2 points on 5-point scales) and only modest rank agreement, motivating bias-aware evaluation. Our results demonstrate that behavioral alignment is measurable but not reducible to similarity, and that structured anchors combined with embedding-based proxies provide reproducible signals for ranking and filtering tests and candidate patches. By transforming alignment into a controllable engineering signal, Desc2Fix enables more reliable test generation, semantics-aware fault localization, principled patch ranking, and improved bug report authoring.

Figures

Figures reproduced from arXiv: 2607.18550 by Anil Koyuncu, David Lo, Jacques Klein, Liang Xiao, Lingfeng Bao, Pawe{\l} Borsukiewicz, Tegawend\'e F. Bissyand\'e, Wendk\^uuni C. Ou\'edraogo, Xueqi Dang, Yinghua Li.

Figure 1
Figure 1. Figure 1: Overview of the general workflow of Desc2Fix. 3.2 Textual and Semantic Signal Extraction We extract high-level semantic signals from bug reports to support alignment analyses with triggering tests and corrective patches. In our setting, bug reports are treated as pre-fix problem descriptions—i.e., the developer-facing specification available before (or at the time) a failing test and a corrective patch are… view at source ↗
Figure 2
Figure 2. Figure 2: Semantic similarity between GPT-4o and DeepSeek-Chat by field and dataset. summaries (Expected, Actual) display the lowest lexical overlap (ROUGE-L ≈ 0.50) while maintaining moderate embedding similarity (cosine ≈ 0.70–0.75), reflecting natural summarization variability rather than semantic divergence. This gradient, from near-canonical entities to loosely paraphrased behaviors, confirms that inter-model d… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 12 linked inside Pith

  1. [1]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches.arXiv preprint arXiv:2502.05368(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Alif Al Hasan, Subarna Saha, Mia Mohammad Imran, and Tarannum Shaila Zaman. 2025. LLPut: Investigating Large Language Models for Bug Report-Based Input Generation. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1652–1659. , Vol. 1, No. 1, Article . Publication date: July 2026. 32 Ouédraogo et al

    2025

  3. [3]

    Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. 308–318

    2008

  4. [4]

    Tegawendé F Bissyandé, David Lo, Lingxiao Jiang, Laurent Réveillere, Jacques Klein, and Yves Le Traon. 2013. Got issues? who cares about it? a large scale investigation of issue trackers from github. In2013 IEEE 24th international symposium on software reliability engineering (ISSRE). IEEE, 188–197

    2013

  5. [5]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200

    2025

  6. [6]

    Partha Chakraborty, Mahmoud Alfadel, and Meiyappan Nagappan. 2024. RLocator: Reinforcement learning for bug localization.IEEE Transactions on Software Engineering(2024)

    2024

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  8. [8]

    Fan Fang, John Wu, Yanyan Li, Xin Ye, Wajdi Aljedaani, and Mohamed Wiem Mkaouer. 2021. On the classification of bug reports to improve bug localization.Soft Computing25, 11 (2021), 7307–7323

    2021

  9. [9]

    Sidong Feng and Chunyang Chen. 2024. Prompting is all you need: Automated android bug replay with large language models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13

    2024

  10. [10]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  11. [11]

    Qing Gao, Hansheng Zhang, Jie Wang, Yingfei Xiong, Lu Zhang, and Hong Mei. 2015. Fixing recurring crash bugs via analyzing q&a sites (t). In2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 307–318

    2015

  12. [12]

    Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A deep dive into large language models for automated bug localization and repair.Proceedings of the ACM on Software Engineering1, FSE (2024), 1471–1493

    2024

  13. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    Pith/arXiv arXiv 2023

  15. [15]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with llms. InProceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering. 1646–1656

    2023

  16. [16]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. InProceedings of the 2014 international symposium on software testing and analysis. 437–440

    2014

  17. [17]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2312–2323

    2023

  18. [18]

    Anil Koyuncu, Kui Liu, Tegawendé F Bissyandé, Dongsun Kim, Martin Monperrus, Jacques Klein, and Yves Le Traon

  19. [19]

    Ahmed Lamkanfi, Serge Demeyer, Emanuel Giger, and Bart Goethals. 2010. Predicting the severity of a reported bug. 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)(2010), 1–10. https://api.semanticscholar. org/CorpusID:9312951

    2010

  20. [20]

    Ahmed Lamkanfi, Serge Demeyer, Quinten David Soetens, and Tim Verdonck. 2011. Comparing mining algorithms for predicting the severity of a reported bug. In2011 15th European Conference on Software Maintenance and Reengineering. IEEE, 249–258

    2011

  21. [21]

    Jaekwon Lee, Dongsun Kim, Tegawendé F Bissyandé, Woosung Jung, and Yves Le Traon. 2018. Bench4bl: reproducibility study on the performance of ir-based bug localization. InProceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis. 61–72

    2018

  22. [22]

    Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. InProceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. 169–180

    2019

  23. [23]

    Yuning Li, Wenkang Zhong, Zongwen Shen, Chuanyi Li, Xiang Chen, Jidong Ge, and Bin Luo. 2025. An empirical study on the code naturalness modeling capability for LLMs in automated patch correctness assessment.Automated Software Engineering32, 2 (2025), 35. , Vol. 1, No. 1, Article . Publication date: July 2026. Semantic Drift in Bug Resolution: How Behavio...

    2025

  24. [24]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  25. [25]

    Chen Liu, Jinqiu Yang, Lin Tan, and Munawar Hafiz. 2013. R2Fix: Automatically generating bug fixes from bug reports. In2013 IEEE Sixth international conference on software testing, verification and validation. IEEE, 282–291

    2013

  26. [26]

    Xiangxin Meng, Zexiong Ma, Pengfei Gao, and Chao Peng. 2024. An Empirical Study on LLM-based Agents for Automated Bug Fixing.arXiv preprint arXiv:2411.10213(2024)

    arXiv 2024

  27. [27]

    Manish Motwani and Yuriy Brun. 2023. Better automatic program repair by using bug reports and tests together. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1225–1237

    2023

  28. [28]

    Niels Mündler, Mark Müller, Jingxuan He, and Martin Vechev. 2024. SWT-bench: Testing and validating real-world bug-fixes with code agents.Advances in Neural Information Processing Systems37 (2024), 81857–81887

    2024

  29. [29]

    Noor Nashid, Islem Bouzenia, Michael Pradel, and Ali Mesbah. 2025. Issue2test: Generating reproducing test cases from issue reports.arXiv preprint arXiv:2503.16320(2025)

    arXiv 2025

  30. [30]

    Anh Tuan Nguyen, Tung Thanh Nguyen, Hoan Anh Nguyen, and Tien N Nguyen. 2012. Multi-layered approach for recovering links between bug reports and fixes. InProceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11

    2012

  31. [31]

    Feifei Niu, Chuanyi Li, Kui Liu, Xin Xia, and David Lo. 2025. When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey.Comput. Surveys57, 11 (2025), 1–41

    2025

  32. [32]

    Yuki Noyori, Hironori Washizaki, Yoshiaki Fukazawa, Keishi Ooshima, Hideyuki Kanuka, and Shuhei Nojiri. 2023. Deep learning and gradient-based extraction of bug report features related to bug fixing time.Frontiers in Computer Science5 (2023), 1032440

    2023

  33. [33]

    Wendkûuni C Ouédraogo, Laura Plein, Kader Kaboré, Andrew Habib, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Extracting Relevant Test Inputs from Bug Reports for Automatic Test Case Generation. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 406–407

    2024

  34. [34]

    Wendkûuni C Ouédraogo, Laura Plein, Kader Kabore, Andrew Habib, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Enriching automatic test case generation by extracting relevant test inputs from bug reports.Empirical Software Engineering30, 3 (2025), 85

    2025

  35. [35]

    Avinash Patil, Kihwan Han, and Aryan Jadon. 2023. A comparative study of text embedding models for semantic text similarity in bug reports.arXiv preprint arXiv:2308.09193(2023)

    Pith/arXiv arXiv 2023

  36. [36]

    Laura Plein, Wendkûuni C Ouédraogo, Jacques Klein, and Tegawendé F Bissyandé. 2024. Automatic generation of test cases based on bug reports: a feasibility study with large language models. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 360–361

    2024

  37. [37]

    Sarah Rastkar, Gail C Murphy, and Gabriel Murray. 2010. Summarizing software artifacts: a case study of bug reports. InProceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. 505–514

    2010

  38. [38]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    Shuai Shao and Tingting Yu. 2024. Enhancing IR-based Fault Localization using Large Language Models.arXiv preprint arXiv:2412.03754(2024)

    Pith/arXiv arXiv 2024

  40. [40]

    Asankhaya Sharma. 2024. Patched RTC: evaluating LLMs for diverse software development tasks.arXiv preprint arXiv:2407.16557(2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Jiajun Sun, Fengjie Li, Xinzhu Qi, Hongyu Zhang, and Jiajun Jiang. 2025. Empirical Evaluation of Large Language Models in Automated Program Repair.arXiv preprint arXiv:2506.13186(2025)

    Pith/arXiv arXiv 2025

  42. [42]

    Shaowei Wang and David Lo. 2014. Version history, similar report, and structure: Putting them together for improved bug localization. InProceedings of the 22nd international conference on program comprehension. 53–63

    2014

  43. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  44. [44]

    Aidan ZH Yang, Sophia Kolak, Vincent J Hellendoorn, Ruben Martins, and Claire Le Goues. 2024. Revisiting un- naturalness for automated program repair in the era of large language models.arXiv preprint arXiv:2404.15236 (2024)

    Pith/arXiv arXiv 2024

  45. [45]

    Klaus Changsun Youm, June Ahn, Jeongho Kim, and Eunseok Lee. 2015. Bug localization based on code change histories and bug reports. In2015 Asia-Pacific Software Engineering Conference (APSEC). IEEE, 190–197

    2015

  46. [46]

    Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair.arXiv preprint arXiv:2405.01466(2024)

    arXiv 2024

  47. [47]

    Wen Zhang, Ziqiang Li, Qing Wang, and Juan Li. 2019. FineLocator: A novel approach to method-level fine-grained bug localization by query expansion.Information and Software Technology110 (2019), 121–135. , Vol. 1, No. 1, Article . Publication date: July 2026. 34 Ouédraogo et al

    2019

  48. [48]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement, 2024.arXiv preprint arXiv:2404.05427(2024)

    Pith/arXiv arXiv 2024

  49. [49]

    Cheng Zhou, Bin Li, Xiaobing Sun, and Hongjing Guo. 2018. Recognizing software bug-specific named entity in software bug repository. InProceedings of the 26th Conference on Program Comprehension. 108–119

    2018

  50. [50]

    Cheng Zhou, Bin Li, Xiaobing Sun, and Sheng Yu. 2023. Leveraging multi-level embeddings for knowledge-aware bug report reformulation.Journal of Systems and Software198 (2023), 111617

    2023

  51. [51]

    Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? more accurate information retrieval- based bug localization based on bug reports. In2012 34th International conference on software engineering (ICSE). IEEE, 14–24. A ADDITIONAL TABLES Table 21.Top bug-report attribute per deterministic metric (Spearman 𝜌), by dataset, scenario, a...

    2012

  52. [2019]

    InProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering

    iFixR: Bug report driven program repair. InProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 314–325

    2019