Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Is Your Automated Software Engineer Trustworthy?

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM coding agents almost never refuse to act on vague bug reports or wrong patches, and BouncerBench is a new benchmark built to measure that failure.

desk verdict First abstention benchmark for coding agents with a real artifact and a plausible headline finding, but the input ground truth needs agreement statistics and the O-Score's range claim is wrong. read the letter →

arxiv 2506.17812 v1 pith:QGMG5TIY submitted 2025-06-21 cs.SE

Noble Saji Mathews , Meiyappan Nagappan This is my paper · ORCID
classification cs.SE
keywords LLMagentsabstentionBouncerBenchsoftwareengineeringbenchmarkvagueissueticketspatchvalidationtrustworthyAIselectiveprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that current LLM-based software agents are untrustworthy in a specific, measurable way: they rarely abstain. They generate a patch for every issue, even when the report is too vague to act on, and they ship patches even when those patches are wrong. To make this failure visible, the authors introduce BouncerBench, with two tasks: input bouncing, which asks a system to reject underspecified tickets, and output bouncing, which asks it to reject incorrect patches. On the benchmark's 1,699 input tasks and 642 output tasks, the best configurations still catch only a fraction of the cases that should be bounced, and the paper concludes that large language models cannot yet be trusted to make correct decisions in real software-engineering workflows.

What carries the argument

The central object is the bouncer, a gatekeeping agent placed around an LLM-based coding system. An input bouncer decides whether an issue ticket is specific enough to work on; an output bouncer decides whether a generated patch should reach the user. BouncerBench turns both decisions into binary classification tasks with ground truth: for inputs, a highest-severity ensemble of three human annotators' labels; for outputs, the pass/fail behavior of the dataset's unit tests. The paper's main metric is macro-averaged F-measure, chosen so an 'always bounce' or 'always accept' strategy cannot inflate scores, with fine-grained I-Score and O-Score that weight each decision by ambiguity or by the fraction of tests passed.

What would settle it

Re-run the input-bouncing task with ground truth defined by majority vote instead of the highest severity label, and check whether the model rankings and the 38% 'should bounce' rate change substantially; if they do, the input task is an artifact of the ensemble rule.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that abstention is a missing capability in current coding agents, and BouncerBench is the first benchmark to isolate and score it. The input side treats a ticket as underspecified if human annotators rated it vague or nearly impossible to solve, and the output side treats a patch as incorrect unless it passes all fail-to-pass and pass-to-pass tests. Across the evaluated LLMs, the strongest input bouncer bounced only 26.8% of the tickets that should be bounced, while the strongest output bouncer caught 61.0% of incorrect patches but also rejected 16.9% of correct ones. Combining both bouncers on a balanced subset raised the fraction of correct patches in the final output from about 36.9% to 57.4%, yet still left 142 wrong patches among 333 responses. The paper concludes that, while bouncing helps, current models are far from the reliability needed for real workflows.

Load-bearing premise

The benchmark's ground truth for a vague ticket is the harshest label among three human annotators, so any ticket that one annotator found ambiguous is counted as underspecified; if those labels are noisy or the aggregation is too aggressive, the input-bouncing scores are not a clean measure of abstention.

Editorial extensions

If this is right

  • Resolution-rate leaderboards overstate agent usefulness because they reward answering every ticket and ignore the cost of wrong patches, which BouncerBench is designed to correct.
  • Reasoning models and codebase access improve bouncing relative to plain non-reasoning models, but even the best configuration agrees with human annotators only slightly beyond chance.
  • Deploying both bouncers in sequence raises the share of correct patches in the final output and cuts the number of wrong patches a developer reviews, while discarding some correct patches.
  • Abstention behaves like a distinct capability: models that resolve tickets well are not automatically good at deciding when to abstain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the highest-severity ensemble rule likely inflates the number of 'vague' tickets, so a majority-vote version of the benchmark could rank models differently.
  • Beyond the paper's claims, the output oracle is the dataset's own unit tests, which the annotations themselves flag as imperfect; output-bouncer scores therefore partly measure test quality rather than pure patch trustworthiness.
  • Beyond the paper's claims, the bouncer pattern generalizes: any LLM agent taking consequential actions could be gated by analogous input and output checks, and BouncerBench-style tasks could quantify abstention in those settings.
  • Beyond the paper's claims, a natural next experiment is to measure developer time saved by bouncing, not just the accuracy of the filtered patch set.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces BouncerBench, a benchmark for measuring whether LLM-based software agents can abstain from acting on underspecified issue reports (input bouncing) and from returning incorrect patches (output bouncing). The input task uses 1,699 SWE-Bench issues with human annotation labels 0-3, where ground truth 'should bounce' is assigned if any of three annotators gave the highest severity labels 2 or 3, yielding 650 vague tickets. The output task draws 642 LLM-generated patches from leaderboard submissions and labels each as correct or incorrect from unit-test outcomes, with the caveat that only incorrect patches are sampled for the 109 input-vague instances. The paper proposes macro-averaged F-measure, I-Score, and O-Score, and evaluates six LLM baselines plus a Codex CLI agent. Results show low recall on vague tickets (best 26.8% for o4-mini) and output-bouncing recall up to 61% for Codex, leading the authors to conclude that current models cannot yet abstain reliably.

Significance. BouncerBench addresses an overlooked and practically important failure mode in code agents, and the public leaderboard and replication package are good contributions. The paper ships a concrete benchmark with a clear task formulation and baseline implementations that will likely be useful to the community. The headline finding that current models seldom abstain, even when instructions ask them to, is plausible and largely supported by the low absolute recall values. However, several construction and metric choices currently weaken the force of the quantitative conclusions; these are fixable in revision.

major comments (4)
  1. [II-A1 and VII] The input-bouncing ground truth is defined as the maximum severity across three annotators, which labels 650 of 1,699 tickets (38.26%) as vague, yet the paper reports no inter-annotator agreement for the vague/clear distinction. Because a single dissenting annotator is sufficient to mark a ticket as 'should bounce', the low recall values in Table II may reflect disagreement with an aggressive labeling policy rather than an inability to detect underspecification. Please report agreement statistics (e.g., Fleiss' kappa for the binary decision) and a sensitivity analysis using majority-vote or unanimous-vote thresholds; the threats-to-validity section acknowledges reliance on annotation soundness but does not quantify it.
  2. [II-A2] For the output-bouncing task, the 109 instances that are input-vague are populated exclusively with incorrect patches, and 6 instances with no incorrect submissions are discarded. This means that for 109 of 642 tasks (17%), the correct label is 'bounce' by construction rather than by inspection of the patch, so a model that uses ticket vagueness as a shortcut will appear to perform well for reasons unrelated to output evaluation. The conditional sampling also changes the class distribution relative to the full population of patches. Please either include correct patches for vague tickets (and model the decision as needing both input and output judgments) or report the output-bouncing results separately for the well-specified subset.
  3. [II-B and IV] The two fine-grained scores are not calibrated so that 0 corresponds to the natural baseline, and the stated O-Score range is incorrect. For the input task, an 'accept all' policy achieves I-Score = (2/3)(1.5 - mean_label) ≈ 0.191 under the Table I label distribution, so Claude-3.7-sonnet (0.209) and Gemma3 (0.198) are barely above a trivial constant classifier. For the output task, an incorrect patch cannot have PassedTests/TotalTests = 1, so the maximum O-Score is strictly below +1 whenever the dataset contains incorrect patches, contradicting the claim of a [-1, +1] range; moreover, a constant-accept policy has a nonzero expected O-Score. Please report always-accept and always-bounce baselines for all metrics and reinterpret the scores against these baselines.
  4. [Tables II-III and IV-E] Model comparisons are reported as point estimates without confidence intervals or significance tests. For example, the input macro-F gap between o4-mini (0.592) and Codex (0.561), and the output gap between Codex (0.690) and o4-mini (0.612), are used to support claims of improvement, but on a fixed task set a bootstrap or McNemar test would be inexpensive and appropriate. Without such tests, the relative statements in Sections IV-B, IV-D, and IV-E are not statistically supported; the central 'low absolute performance' claim is less affected by this issue.
minor comments (5)
  1. [IV-C] The sentence 'their is no human annotation for the patches' contains a typo; it should read 'there is no human annotation for the patches'.
  2. [VI] The related-work section contains the typo 'acheiving' in 'acheiving over 90% alignment with Human Judges'; it should be 'achieving'.
  3. [Abstract] The abstract says BouncerBench 'aims to improve precision', but as a benchmark it measures rather than improves systems; consider rewording to 'measure precision'.
  4. [II-A2] The term 'BouncerBench Lite' is introduced in the text but never explicitly defined; please state up front that it is the 642-task subset with equal numbers of input and output tasks.
  5. [IV-A] The statement 'We sample at temperature 0 for all the non-reasoning models' is immediately followed by exceptions for reasoning models; please consolidate the temperature and sampling settings for each model into a single clear specification.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: BouncerBench is an empirical benchmark construction evaluated against external human annotations and test outcomes, not a derivation from fitted inputs.

full rationale

BouncerBench is an empirical benchmark construction and evaluation paper rather than a derivation paper. The input-bouncing ground truth comes from OpenAI's externally produced three-annotator labels on SWE-Bench instances, and the output-bouncing ground truth comes from externally defined SWE-Bench Fail-to-Pass and Pass-to-Pass test outcomes. The paper fits no parameter to these labels and then predicts a quantity equivalent to the fit. The baseline bouncers use fixed prompts and structured-output configurations, and their outputs are measured against the external labels rather than derived from them. The closest structural issue, that the input-bouncer prompt closely mirrors the OpenAI annotation instructions, is an intentional construct-alignment choice: the task is to reproduce human specificity judgments, and agreement with those judgments is the evaluation criterion, not an input to a derivation. The conservative 'highest-severity label' ensemble is a labeling policy that determines ground truth, but it does not make the benchmark's conclusions circular; it is a validity or robustness concern that the paper itself partially acknowledges in Section VII: 'We also rely on the soundness of the human annotations provided by OpenAI when curating SWE-Bench Verified.' No load-bearing self-citation, no fitted-input-called-prediction pattern, no uniqueness theorem imported from prior work, and no renaming of a known result as a new derivation appear. Therefore no circularity is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's ground-truth labels rest on external data: OpenAI's human annotations for input vagueness and unit-test outcomes for patch correctness, plus the assumption that the sampled patches represent real LLM output. No parameters were fitted; the only design choices are the highest-severity ensemble and the sampling rules.

assumptions (4)
  • domain assumption The human annotations (labels 0-3 for issue specificity and test validity) from the SWE-Bench Verified curation are accurate enough to serve as ground truth.
    The entire input-bouncing label set is built from these annotations, with the highest severity label across three annotators taken as ground truth (Section II-A).
  • domain assumption Unit test outcomes (Fail-to-Pass and Pass-to-Pass) are a valid oracle for patch correctness after discarding instances with severe test-validity issues.
    Output-bouncing labels are binary based on test outcomes; any patch failing at least one test is incorrect (Section II-A).
  • domain assumption The sampled leaderboard patches, one per instance, are representative of LLM-generated patch distributions.
    The output-bouncing dataset is a random sample of available patches per instance, with all incorrect-only sampling for vague tickets (Section II-A).
  • domain assumption Model knowledge cutoffs prevent the task labels from being memorized, so the evaluations measure abstention ability rather than memorization.
    Discussed in Section VII as a threat; the authors argue evidence does not show memorization, but it remains an assumption for output bouncing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Your Automated Software Engineer Trustworthy?." pith.science (2026). https://pith.science/paper/QGMG5TIY

@misc{pith2026250617812,
  author       = {Pith},
  title        = {Pith review of: Is Your Automated Software Engineer Trustworthy?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGMG5TIY}},
  note         = {Machine review of arXiv:2506.17812}
}
read the original abstract

Large Language Models (LLMs) are being increasingly used in software engineering tasks, with an increased focus on bug report resolution over the past year. However, most proposed systems fail to properly handle uncertain or incorrect inputs and outputs. Existing LLM-based tools and coding agents respond to every issue and generate a patch for every case, even when the input is vague or their own output is incorrect. There are no mechanisms in place to abstain when confidence is low. This leads to unreliable behaviour, such as hallucinated code changes or responses based on vague issue reports. We introduce BouncerBench, a benchmark that evaluates whether LLM-based software agents can refuse to act when inputs are ill-defined or refuse to respond when their own outputs are likely to be incorrect. Unlike prior benchmarks that implicitly incentivize models to generate responses even when uncertain, BouncerBench aims to improve precision by targeting two overlooked failure points: (1) vague or underspecified issue descriptions in tickets and (2) logically or functionally incorrect code patches created by the system. It measures whether proposed systems can distinguish actionable issues from vague tickets and valid patches from untrustworthy ones. We also implement a basic input and output bouncer, evaluating how well current LLMs can abstain when needed. Our results show that most models fail to abstain from underspecified inputs or incorrect outputs. Hence, we conclude that there is significant room for improvement before LLMs can be trusted to make correct decisions and recommendations in real-world software engineering workflows. BouncerBench provides a first step toward evaluating and building more cautious, trustworthy code agents. The replication package, dataset, and leaderboard can be found at bouncerbench.com

Figures

Figures reproduced from arXiv: 2506.17812 by the authors.

Figure 1
Figure 1. Overview of Input and Output bouncing tasks for an AI-based Ticket Resolution System. Dotted components are optional. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Impact of Issue Length on True Positive Rate and False Positive Rate [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Impact of Patch Length on True Positive Rate and False Positive Rate [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Sankey diagram depicting effects of the o4-Mini based Input Bouncer and the codex based Output Bouncer on BouncerBench Lite [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 18 canonical work pages

  1. [1]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    arXiv 2024

  2. [2]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “OpenHands: An Open Platform for AI Software Developers as Generalist Agents,” 2024. [Online]. Available: https://arxiv.org/ab...

    arXiv 2024

  3. [3]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” arXiv preprint arXiv:2408.02232 , 2024

    arXiv 2024

  4. [4]

    Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,

    M. S. Rashid, C. Bock, Y . Zhuang, A. Buchholz, T. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, A. Deoras, G. Zappella, and L. Callot, “Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,” 2025. [Online]. Available: https://arxiv.org/abs/2504.08703

    arXiv 2025

  5. [5]

    SWE-bench: Can language models resolve real- world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real- world github issues?” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66

    2024

  6. [6]

    SWT-bench: Testing and validating real-world bug-fixes with code agents,

    N. M ¨undler, M. N. Mueller, J. He, and M. Vechev, “SWT-bench: Testing and validating real-world bug-fixes with code agents,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id= 9Y8zUO11EQ

    work page 2024

  7. [7]

    Consistent estimators for learning to defer to an expert,

    H. Mozannar and D. Sontag, “Consistent estimators for learning to defer to an expert,” in International conference on machine learning. PMLR, 2020, pp. 7076–7087

    work page 2020

  8. [8]

    Improving the effectiveness of peer code review in identifying security defects,

    R. Paul, “Improving the effectiveness of peer code review in identifying security defects,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 1645–1649

    work page 2021

Show all 43 references
  1. [9]

    A research agenda for assessing the economic impacts of code generation models,

    S. Manning, P. Mishkin, G. Hadfield, T. Eloundou, and E. Eisner, “A research agenda for assessing the economic impacts of code generation models,” 2022

    2022

  2. [10]

    Automated unit test improvement using large language models at meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” in Companion Proceedings of the 32nd ACM International Conference on the Founda- tions of So...

    2024

  3. [11]

    Examining the use and impact of an ai code assistant on developer productivity and experience in the enterprise,

    J. D. Weisz, S. V . Kumar, M. Muller, K.-E. Browne, A. Goldberg, K. E. Heintze, and S. Bajpai, “Examining the use and impact of an ai code assistant on developer productivity and experience in the enterprise,” in Proceedings of the Extended Abstracts of the CHI Conference on H...

    2025

  4. [12]

    Exploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wang, Z. Yang, L. Zhang, Z. Li, and Y . Ma, “Exploring and evaluating hallucinations in llm-powered code generation,” arXiv preprint arXiv:2404.00971 , 2024

    2024

  5. [13]

    Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,

    A. Sergeyuk, Y . Golubev, T. Bryksin, and I. Ahmed, “Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,” Information and Software Technology , vol. 178, p. 107610, 2025

    2025

  6. [14]

    How we refactor, and how we know it,

    E. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,” IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 5–18, 2011

    2011

  7. [15]

    Make your tools sparkle with trust: The picse framework for trust in software tools,

    B. Johnson, C. Bird, D. Ford, N. Forsgren, and T. Zimmermann, “Make your tools sparkle with trust: The picse framework for trust in software tools,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 202...

    2023

  8. [16]

    Selective question answering under domain shift,

    A. Kamath, R. Jia, and P. Liang, “Selective question answering under domain shift,” arXiv preprint arXiv:2006.09462 , 2020

    2006 arXiv

  9. [17]

    Is that your final answer? test-time scaling improves selective question answering,

    W. Jurayj, J. Cheng, and B. Van Durme, “Is that your final answer? test-time scaling improves selective question answering,” arXiv preprint arXiv:2502.13962, 2025

    2025 arXiv

  10. [18]

    Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and confor- mal ensembles,

    R. Verma, D. Barrej ´on, and E. Nalisnick, “Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and confor- mal ensembles,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2023, pp. 11 415–11 434

    2023

  11. [19]

    Introducing SWE-bench Verified — openai.com,

    “Introducing SWE-bench Verified — openai.com,” https://openai.com/ index/introducing-swe-bench-verified, [Accessed 30-05-2025]

    2025

  12. [20]

    https://cdn.openai.com/introducing-swe-bench-verified/ swe-b-annotation-instructions.pdf, [Accessed 30-05-2025]

    2025

  13. [21]

    https://openai.com/index/o3-o4-mini-system-card, [Accessed 30-05- 2025]

    2025

  14. [22]

    https://www.anthropic.com/news/claude-3-7-sonnet, [Accessed 30-05- 2025]

    2025

  15. [23]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al. , “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  16. [24]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ere et al. , “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786 , 2025

    2025 arXiv

  17. [25]

    A survey on llm-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu et al. , “A survey on llm-as-a-judge,” arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  18. [26]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Pro- ceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  19. [27]

    SWE-bench multimodal: Do ai systems generalize to visual software domains?

    J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. I. Wang, and O. Press, “SWE-bench multimodal: Do ai systems generalize to visual software domains?” in The Thirteenth International Conference o...

    2025

  20. [28]

    Swe-bench-java: A github issue resolving benchmark for java,

    D. Zan, Z. Huang, A. Yu, S. Lin, Y . Shi, W. Liu, D. Chen, Z. Qi, H. Yu, L. Yu et al., “Swe-bench-java: A github issue resolving benchmark for java,” arXiv preprint arXiv:2408.14354 , 2024

    2024 arXiv

  21. [29]

    Swe-lancer: Can frontier llms earn 1 million from real-world freelance software engineering?

    S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke, “Swe-lancer: Can frontier llms earn 1 million from real-world freelance software engineering?” arXiv preprint arXiv:2502.12115 , 2025

    2025 arXiv

  22. [30]

    Baxbench: Can llms generate correct and secure backends?

    M. Vero, N. M ¨undler, V . Chibotaru, V . Raychev, M. Baader, N. Jo- vanovi´c, J. He, and M. Vechev, “Baxbench: Can llms generate correct and secure backends?” 2025

    2025

  23. [31]

    Sota on swe-bench verified with inference- time scaling and critic model,

    X. Wang, “Sota on swe-bench verified with inference- time scaling and critic model,” All Hands AI Blog , April 2025. [Online]. Available: https://www.all-hands.dev/blog/ sota-on-swe-bench-verified-with-inference-time-scaling-and-critic-model

    2025

  24. [32]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  25. [33]

    Codejudge- eval: Can large language models be good judges in code understanding?

    Y . Zhao, Z. Luo, Y . Tian, H. Lin, W. Yan, A. Li, and J. Ma, “Codejudge- eval: Can large language models be good judges in code understanding?” arXiv preprint arXiv:2408.10718 , 2024

    2024 arXiv

  26. [34]

    Agent-as-a-judge: Evaluate agents with agents,

    M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y . Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y . Tian et al., “Agent-as-a-judge: Evaluate agents with agents,” arXiv preprint arXiv:2410.10934 , 2024

    2024 arXiv

  27. [35]

    What makes a good bug report?

    N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308–318

    2008

  28. [36]

    Deep learning based valid bug reports determination and explanation,

    J. He, L. Xu, Y . Fan, Z. Xu, M. Yan, and Y . Lei, “Deep learning based valid bug reports determination and explanation,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2020, pp. 184–194

    2020

  29. [37]

    Chaff from the wheat: Characterizing and determining valid bug reports,

    Y . Fan, X. Xia, D. Lo, and A. E. Hassan, “Chaff from the wheat: Characterizing and determining valid bug reports,” IEEE transactions on software engineering , vol. 46, no. 5, pp. 495–525, 2018

    2018

  30. [38]

    Industrial adoption of machine learning techniques for early identification of invalid bug reports,

    M. Laiq, N. b. Ali, J. B ¨orstler, and E. Engstr ¨om, “Industrial adoption of machine learning techniques for early identification of invalid bug reports,” Empirical Software Engineering , vol. 29, no. 5, p. 130, 2024

    2024

  31. [39]

    Can we enhance bug report quality using llms?: An empirical study of llm-based bug report generation,

    J. Acharya and G. Ginde, “Can we enhance bug report quality using llms?: An empirical study of llm-based bug report generation,” 2025. [Online]. Available: https://arxiv.org/abs/2504.18804

    2025 arXiv

  32. [40]

    Chatbr: Automated assessment and improvement of bug report quality using chatgpt,

    L. Bo, W. Ji, X. Sun, T. Zhang, X. Wu, and Y . Wei, “Chatbr: Automated assessment and improvement of bug report quality using chatgpt,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’24. New York, NY , USA: Associatio...

    2024

  33. [41]

    Agentless: De- mystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: De- mystifying llm-based software engineering agents,” arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  34. [42]

    Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks,

    X. Zhou, M. Weyssow, R. Widyasari, T. Zhang, J. He, Y . Lyu, J. Chang, B. Zhang, D. Huang, and D. Lo, “Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks,” arXiv preprint arXiv:2502.06215 , 2025

    2025 arXiv

  35. [43]

    Are large language models memorizing bug benchmarks?

    D. Ramos, C. Mamede, K. Jain, P. Canelas, C. Gamboa, and C. L. Goues, “Are large language models memorizing bug benchmarks?” arXiv preprint arXiv:2411.13323 , 2024

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.