Pith. sign in

REVIEW 3 major objections 6 minor 83 references

ReCatcher: Towards LLMs Regression Testing for Code Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReCatcher claims to be the first regression-testing framework for LLM code generation, comparing models on logic, static quality, and execution speed and finding regressions of up to 80%.

desk verdict ReCatcher is a coherent and useful framework for LLM code-generation regression testing, with plausible headline findings, but the execution-time claims outrun the input scale and the significance statistics are thin in places. read the letter →

arxiv 2507.19390 v1 pith:XSWWZT3D submitted 2025-07-25 cs.SE cs.AI

Altaf Allah Abbassi , Leuson Da Silva , Amin Nikanjam , Foutse Khomh This is my paper · ORCID
classification cs.SEcs.AI
keywords regressiontestingLLMcodegenerationqualityexecutionperformancefine-tuningmodelmergingGPT-4ostaticanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReCatcher is proposed as the first regression-testing framework aimed specifically at LLM-generated Python code. Instead of asking whether a new model is more accurate, it asks whether swapping in a candidate model for the current one introduces regressions in logical correctness, static code quality, or execution performance. Applied to fine-tuned, merged, and newly released CodeLlama, DeepSeek-Coder, and GPT-family models, the framework reports that cross-language fine-tuning raises syntax errors by up to 12.93%, merging with a general-purpose model can reduce logical correctness by up to 18.72%, GPT-4o misses imports or declarations up to 51.64% more often than GPT-3.5-turbo on BigCodeBench, and GPT-4o-mini is slower than GPT-4o on 80.49% of HumanEval+ tasks. The point of the paper is that more capable or newer models are not automatically better on every axis, and that systematic regression testing should precede model adoption.

What carries the argument

The load-bearing object is the three-part Test Suite: unit-test execution for logical correctness, Pylint (plus PMD-CPD for duplication and a custom rule for unnecessary conditional blocks) for static code issues, and repeated profiling of execution time and memory with the Mann-Whitney U test for performance. Each inefficiency category in the taxonomy of LLM-generated code inefficiencies is mapped to a detectable test or Pylint message, giving the framework a structured, reproducible way to turn code-quality concerns into pass or fail signals. The direct comparison of two models on the same tasks is what turns ordinary quality measurement into regression testing.

What would settle it

Run the same GPT-4o versus GPT-4o-mini comparison on a benchmark with large, real-world-shaped inputs or on a production workload and profile execution time; if the candidate is not slower on a substantial fraction of tasks, the headline execution-time regression claim would be confined to the benchmark setting. Similarly, repeat the fine-tuning scenario with a same-language dataset; if syntax errors do not rise, the cross-language mechanism fails to reproduce.

Watch

Extended reading notes

Core claim

ReCatcher's central claim is that regressions in LLM code generation can be detected systematically by comparing, per benchmark task, code generated by the deployed model and the candidate model across three dimensions. Logical correctness is judged by executing each snippet against the benchmark's unit tests; static code issues are detected by Pylint and PMD-CPD using a taxonomy of LLM-code inefficiencies; performance is profiled by repeated execution and compared with the Mann-Whitney U test to determine whether the candidate's code is statistically slower or faster. The framework emits a regression report with an inefficiency-difference percentage for logic and static issues and a task ratio of improvements or regressions for time and memory. Using it across fine-tuning, merging, and model-release scenarios, the paper reports the specific regressions listed above and concludes that logical correctness, syntax and import errors, and execution time are the most regression-prone aspects.

Load-bearing premise

The regressions, especially execution-time regressions, are measured only on tasks and small inputs drawn from two benchmarks because large inputs were not available for performance testing; if those inputs do not resemble real workloads, the performance findings may not transfer to practice.

Editorial extensions

If this is right

  • Teams planning to fine-tune a code model on data from another language should expect syntax regression and budget for syntax checks before deployment.
  • Merging a code model with a general-purpose model can cost up to roughly 18% in logical correctness and roughly 20-30% in syntax reliability, so merged models should be regression-tested rather than assumed beneficial.
  • Model releases within one family can trade correctness or import handling for speed or vice versa; GPT-4o-mini's 80.49% execution-time regression on HumanEval+ means cheaper models should be profiled on the intended workload.
  • Memory usage stayed relatively stable across most updates, while execution time, syntax errors, and missing imports were the volatile axes, so regression suites should prioritize these three.
  • More capable models are not universally better, so adopting a new model release or merged variant on the strength of leaderboard accuracy alone risks silent quality or performance losses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the regression patterns generalize, error-handling signals such as syntax errors and missing imports act as an early canary: they shift with fine-tuning and merging, so cheap static checks alone could serve as a low-cost gate before full unit-test and profiling runs.
  • The framework's reliance on unit-test inputs for performance means the 80.49% slowdown could be a small-input artifact; extending the same comparison to large, workload-shaped inputs is the natural next test, and ReCatcher's architecture already allows such inputs.
  • A direct extension would be to rank candidate models for a specific codebase by running ReCatcher on the team's own tasks, turning the framework from a post-hoc evaluator into a release gate in a CI pipeline.
  • The finding that an LLM judge agrees with test-based logic judgment only about 80% of the time and performs poorly on performance suggests a hybrid approach: test execution for logic, profiling for speed, and LLM judges only for aspects that resist automation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ReCatcher, a regression testing framework that systematically compares two LLMs for Python code generation across three dimensions: logical correctness, static code quality, and execution performance. The framework uses unit tests, Pylint, PMD-CPD, and profiling/tracemalloc, and generates a regression report based on repeated generation and statistical testing. It is applied to three model-update scenarios—fine-tuning, merging, and model release—using CodeLlama, DeepSeek-Coder, and GPT-family models on HumanEval+ and BigCodeBench. Headline findings include syntax-error regressions up to 12.93% after cross-language fine-tuning, logical-correctness regression up to 18.72% after merging CodeLlama with Llama2, missing-import regressions up to 51.64% for GPT-4o relative to GPT-3.5-turbo, and an 80.49% execution-time regression for GPT-4o-mini on HumanEval+. The paper also compares ReCatcher with LLM-as-judge baselines and standard metrics for a small subset of models.

Significance. If the headline results are correct, ReCatcher is a practical quality gate for deciding whether to adopt a new LLM for code generation, and the findings usefully demonstrate that more capable or cost-efficient models are not universally better in correctness, quality, or performance. The paper has concrete strengths: the framework is open-sourced, a dataset of generated snippets is released, the correctness oracles are external (unit tests, Pylint, CPD, profiler) rather than fitted to the conclusions, and the performance comparison uses a non-parametric test. The claim of being the first LLM regression testing framework for code generation is plausible relative to RETAIN and RACE, although RACE is closely related. However, the execution-time headline is currently not adequately supported, and the statistical reporting of the main regression rates is too thin to distinguish real effects from noise.

major comments (3)
  1. [Section 4.2.2 and Section 5.5.3, Table 1] The 80.49% execution-time regression for GPT-4o-mini on HumanEval+ is the fraction of tasks where a Mann-Whitney U test found a significant difference, not a measured 80% slowdown; no effect sizes or confidence intervals are reported for any performance entry. Performance testing used only unit-test inputs because 'large inputs were not available' (Section 4.2.2), and the comparison is restricted to tasks where both models produced unit-test-passing code. On the tiny, correctness-oriented inputs of HumanEval+, timing can be dominated by interpreter overhead or noise, so an algorithmically worse solution can appear faster. The paper's own large-input mechanism was not exercised, leaving the practical recommendation to prefer GPT-4o for algorithmic tasks unsubstantiated. Please report effect sizes (e.g., common-language effect size or median ratios), state how many tasks were excluded per scenario, and either use large inputs or limit the performance claims to unit-test-scale inputs.
  2. [Table 1 and Sections 5.5.1-5.5.3] All correctness and static-code regression rates are reported as point estimates without confidence intervals, significance tests, or effect sizes. Many entries are at the level of 0.61% to 3%, which corresponds to 1 to 5 tasks out of 164 or 1,140; with only 10 generations per task, sampling variability is substantial. As a result, the paper cannot distinguish genuine regressions from noise, and conclusions such as 'readability and maintainability were relatively stable' rest on very small differences. Please provide task-level rates with binomial confidence intervals or otherwise quantify the uncertainty of the reported percentages.
  3. [Section 5.6 and Table 2] The abstract's claim that ReCatcher achieves 'better and consistent accuracy' against baseline solutions is not supported by the design of the comparison. For logical correctness and performance, ReCatcher's accuracy is 1.0 by construction because unit tests and the Mann-Whitney U test define the ground truth; the experiment measures only the LLM-as-judge baseline's accuracy against that ground truth. No comparison is made to RACE, RETAIN, or any other regression-testing framework, and the 'static code issues' baseline uses only cyclomatic complexity and maintainability index, which measure different properties. Please restate the comparison as an evaluation of LLM-as-judge and two standard metrics, or conduct a head-to-head comparison with an alternative framework before claiming superior accuracy.
minor comments (6)
  1. [Section 5.5.2] The text states that the CodeLlama merged variant 'exhibited a regression of 30.77% in Syntax Error on BigCodeBench and 18.72% on HumanEval+', but Table 1 reports Syntax Error as -9.02% on HumanEval+; the -18.72% value in the table is the General Logic regression. Please align the narrative with the table.
  2. [Sections 4.2.1 and 5.3.2] The repetition variables are used inconsistently: Section 4.2.1 defines generation repetitions as m and Section 5.3.2 labels them n, while performance repetitions are labeled m in Section 5.3.2. Please use consistent notation.
  3. [Table 1] The table header contains typos: 'Fintuned' should be 'Fine-tuned' and 'Errros' should be 'Errors'.
  4. [Section 5.5.2] The sentence 'We observed overall improvements across all code aspects except for a minor minor exception exception regression regression of 1.28%' contains duplicated words; please clean up the prose.
  5. [Abstract and Section 5.5.3] The phrase 'suffers up to 80% performance degradation' is imprecise; the reported 80.49% is the percentage of tasks with a statistically significant execution-time difference, not the magnitude of the slowdown. Please phrase this as 'shows a significant execution-time regression on up to 80% of tasks' or report the actual magnitude.
  6. [Section 3.3] The library name is misspelled as 'tarcemalloc' in the sentence about memory measurement; it should be 'tracemalloc'.

Circularity Check

1 steps flagged · score 5.0 of 10

Performance-baseline comparison is self-referential: LLM-as-judge accuracy is scored against ground truth produced by ReCatcher's own Mann-Whitney U analysis, so ReCatcher wins by construction; headline regression findings remain independent.

  1. self definitional [Section 5.6, 'Performance' paragraph (ground truth definition for the LLM-as-judge comparison)]
    "To establish the ground truth for each pair, we applied the Mann-Whitney U test to compare the distributions of performance metrics (e.g., execution time and memory consumption). Based on the statistical outcomes, we determined whether a significant difference existed. These results served as the reference to evaluate the accuracy of the LLM's judgments."

    ReCatcher's Analyzer already defines performance regression using exactly the same procedure: Section 4.2.3 states, 'we apply the Mann-Whitney U test...' on the same execution-time and memory-usage distributions. Scoring the LLM-judge baseline against ground truth generated by that identical pipeline means the baseline is being measured against ReCatcher's own output. ReCatcher therefore matches the ground truth by definition, making the abstract claim that ReCatcher 'presents better and consistent accuracy across ... performance aspects' true by construction rather than by independent evidence. The logical-correctness part of the comparison uses external unit tests and is not circular, but the performance and memory dimensions of the baseline evaluation are self-referential.

full rationale

The headline regression findings are not circular. The fine-tuning, merging, and release regressions are computed from external oracles: HumanEval+ and BigCodeBench unit tests for logical correctness, Pylint and PMD-CPD for static code issues, and direct profiling with the Mann-Whitney U test for performance. No parameter is fitted to the target conclusions, and the main percentages (12.93% syntax regression, 18.72% logical regression, 51.64% missing-import regression, 80.49% execution-time regression) are all measured against independent tasks and test cases. The taxonomy citation [2] is a self-citation, but it is used to organize static-code checks, not to derive the regression numbers, so it is not load-bearing in the sense that would make the central results circular. The genuine circularity is confined to Section 5.6's comparison with baseline solutions: for performance and memory, the ground truth used to score the LLM-as-judge baseline is produced by the same Mann-Whitney U analysis that ReCatcher itself uses, so ReCatcher's superior 'accuracy' on those dimensions is definitional. This does not undermine the independent regression measurements, but it does mean one advertised contribution—the claim that ReCatcher beats baselines on performance evaluation—should be read as an internal consistency check rather than an externally validated result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several domain assumptions: unit tests and benchmark tasks are valid correctness oracles, the authors' taxonomy-to-linter mapping detects the intended static issues, unit-test inputs represent real performance, and the selected model variants represent their adaptation techniques. These assumptions are stated and acknowledged in Section 8, but they are not independently validated in this paper.

free parameters (3)
  • CPD duplication token threshold = 10 tokens
    Configurable default in ReCatcher for detecting code duplication; changing this threshold would alter Code Duplication regression rates in Table 1.
  • Repetition counts for generation and performance = m=10, n=5
    Chosen as defaults to manage randomness and cost; these determine the statistical precision of every reported regression rate.
  • Generation hyperparameters = temperature=0.1, top-p=0.95, max_tokens=2048
    Fixed generation settings used across all models; they affect which code snippets are produced and therefore all measured regressions.
assumptions (5)
  • domain assumption Unit tests in HumanEval+ and BigCodeBench are valid oracles for logical correctness of generated code.
    Section 3.1 uses unit tests as the proxy for correctness; coverage limitations are acknowledged in Section 8 but not quantified.
  • domain assumption The taxonomy of inefficiencies from the authors' prior work [2], and its mapping to Pylint and PMD-CPD messages, correctly captures the intended static code quality dimensions.
    Section 3.2 maps each taxonomy subcategory to linter messages; no independent validation of the mapping is provided in this paper.
  • domain assumption Performance measured on unit-test inputs is representative of execution performance differences between models.
    Section 4.2.2 states large inputs were not available; Section 8 acknowledges unit tests alone do not fully capture performance bottlenecks.
  • domain assumption The single highest-download fine-tuned and merged variants selected per open-source base model are representative of those adaptation techniques.
    Section 5.1 selects one variant per scenario; Section 8 concedes other variants may behave differently.
  • standard math The Mann-Whitney U test assumptions hold for comparing independent performance samples per task.
    Section 4.2.3 describes the samples as independent and non-paired; the test detects stochastic dominance but not effect size.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReCatcher: Towards LLMs Regression Testing for Code Generation." pith.science (2026). https://pith.science/paper/XSWWZT3D

@misc{pith2026250719390,
  author       = {Pith},
  title        = {Pith review of: ReCatcher: Towards LLMs Regression Testing for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSWWZT3D}},
  note         = {Machine review of arXiv:2507.19390}
}
read the original abstract

Large Language Models (LLMs) for code generation evolve rapidly through fine-tuning, merging, or new model releases. However, such updates can introduce regressions, not only in correctness but also in code quality and performance. To address this, we present ReCatcher, a regression testing framework for Python code generation. ReCatcher systematically compares two LLMs, typically a current model and a candidate update, across three dimensions: logical correctness, static code quality, and execution performance. We apply ReCatcher to assess regressions across three update scenarios, fine-tuning, merging, and model release, using CodeLlama, DeepSeek-Coder, and GPT-4o. Our evaluation shows that fine-tuning with cross-language datasets increases syntax errors by up to 12%. Merging with general-purpose models like Llama2 leads to regressions in correctness by up to 18%. GPT-4o introduces regressions of up to 50% in handling missing imports compared to GPT-3.5-turbo, while GPT-4o-mini suffers up to 80% performance degradation in execution time versus GPT-4o. Overall, logical correctness, performance, and error handling (e.g., syntax errors and missing imports) are the most regression-prone areas. Comparing ReCatcher with baseline solutions, it presents better and consistent accuracy across logical and performance aspects. ReCatcher highlights the importance of systematic regression evaluation before adopting new models, while assisting researchers and practitioners in making more informed update decisions.

Figures

Figures reproduced from arXiv: 2507.19390 by the authors.

Figure 1
Figure 1. ReCatcher Architecture: LLM Regression Testing Framework for Code Generation [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Code snippet generated by CodeLlama (Original version). [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Code snippet generated by CodeLlama (Finetuned version). [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 39 canonical work pages

  1. [1]

    Altaf Allah Abbassi. 2025. ReCatcher: LLM-generated Code Snippets. https://zenodo.org/records/14997627

    work page arXiv 2025

  2. [2]

    Altaf Allah Abbassi, Leuson Da Silva, Amin Nikanjam, and Foutse Khomh. 2025. Unveiling Inefficiencies in LLM-Generated Code: Toward a Comprehensive Taxonomy.arXiv preprint arXiv:2503.06327(2025)

    arXiv 2025

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774(2023)

    arXiv 2023

  4. [4]

    Online Appendix. 2025. ReCatcher: Towards Regression Testing for LLMs in Code Generation - Replication Package. https://github.com/ AltafAllahAbbassi/ReCatcher

    2025

  5. [5]

    Alexandre Bergel, Felipe Banados, Romain Robbes, and David Röthlisberger. 2012. Spy: A flexible code profiling framework.Computer Languages, Systems & Structures38, 1 (2012), 16–28

    2012

  6. [6]

    Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. Unit test generation using generative AI: A comparative performance analysis of autogeneration tools. InProceedings of the 1st International Workshop on Large Language Models for Code. 54–61

    2024

  7. [7]

    Swarnendu Biswas, Rajib Mall, Manoranjan Satpathy, and Srihari Sukumaran. 2011. Regression test selection techniques: A survey.Informatica35, 3 (2011)

    work page 2011

  8. [8]

    Alexandru Calotoiu, Torsten Hoefler, Marius Poke, and Felix Wolf. 2013. Using automated performance modeling to find scalability bugs in complex codes. InProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. 1–12

    work page 2013

Show all 83 references
  1. [9]

    Don Coleman, Dan Ash, Bruce Lowther, and Paul Oman. 1994. Using metrics to evaluate software system maintainability.Computer27, 8 (1994), 44–49

    1994

  2. [10]

    Domenico Cotroneo, Alessio Foggia, Cristina Improta, Pietro Liguori, and Roberto Natella. 2024. Automating the correctness assessment of AI-generated code for security contexts.Journal of Systems and Software(2024), 112113

    2024

  3. [11]

    Han Cui, Menglei Xie, Ting Su, Chengyu Zhang, and Shin Hwei Tan. 2024. An Empirical Study of False Negatives and Positives of Static Code Analyzers From the Perspective of Historical Issues.arXiv preprint arXiv:2408.13855(2024)

    2024 arXiv

  4. [12]

    Leuson Da Silva, Jordan Samhi, and Foutse Khomh. 2025. LLMs and Stack Overflow discussions: Reliability, impact, and challenges.Journal of Systems and Software(2025), 112541

    2025

  5. [13]

    Meghdad Dehghan, Jie JW Wu, Fatemeh H Fard, and Ali Ouni. 2024. Mergerepair: An exploratory study on merging task-specific adapters in code llms for automated program repair.arXiv preprint arXiv:2408.09568(2024)

    2024 arXiv

  6. [14]

    Tanay Dixit, Daniel Lee, Sally Fang, Sai Sree Harsha, Anirudh Sureshan, Akash Maharaj, and Yunyao Li. 2024. RETAIN: Interactive Tool for Regression Testing Guided LLM Migration.arXiv preprint arXiv:2409.03928(2024)

    2024 arXiv

  7. [15]

    Rodrigo Magalhães dos Santos and Marco Aurélio Gerosa. 2018. Impacts of coding practices on readability. InProceedings of the 26th Conference on Program Comprehension(Gothenburg, Sweden)(ICPC ’18). Association for Computing Machinery, New York, NY, USA, 277–285. doi:10.1145/ 3...

    2018

  8. [16]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al

  9. [17]

    2025.EvalPlus Leaderboard

    EvalPlus. 2025.EvalPlus Leaderboard. https://evalplus.github.io/leaderboard.html

    2025

  10. [18]

    João Pascoal Faria and Rui Abreu. 2023. Case studies of development of verified programs with Dafny for accessibility assessment. InInternational Conference on Fundamentals of Software Engineering. Springer, 25–39

    2023

  11. [19]

    Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419

    2011

  12. [20]

    Zachary P Fry, Bryan Landau, and Westley Weimer. 2012. A human study of patch maintainability. InProceedings of the 2012 International Symposium on Software Testing and Analysis. 177–187

    2012

  13. [21]

    Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and Jinfu Chen. 2023. Security weaknesses of copilot generated code in github.arXiv preprint arXiv:2310.02059(2023)

    2023 arXiv

  14. [22]

    Ivo Gomes, Pedro Morgado, Tiago Gomes, and Rodrigo Moreira. 2009. An overview on the static code analysis approach in software development. Faculdade de Engenharia da Universidade do Porto, Portugal16 (2009)

    2009

  15. [23]

    Alex Groce, Iftekhar Ahmed, Josselin Feist, Gustavo Grieco, Jiri Gesi, Mehran Meidani, and Qihong Chen. 2021. Evaluating and improving static analysis tools via differential mutation analysis. In2021 IEEE 21st International Conference on Software Quality, Reliability and Secur...

    2021

  16. [24]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594(2024)

    2024 arXiv

  17. [25]

    Hristina Gulabovska and Zoltán Porkoláb. 2019. Survey on Static Analysis Tools of Python Programs.. InSQAMIA

    2019

  18. [26]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  19. [27]

    Sarra Habchi, Xavier Blanc, and Romain Rouvoy. 2018. On adopting linters to deal with performance concerns in android apps. InProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 6–16

    2018

  20. [28]

    Hassan Bapeer Hassan, Qusay Idrees Sarhan, and Árpád Beszédes. 2024. Evaluating Python Static Code Analysis Tools Using FAIR Principles.IEEE Access12 (2024), 173647–173659

    2024

  21. [29]

    Hugging Face. [n. d.]. Hugging Face Hub. https://huggingface.co Accessed: 14-Mar-2025

    2025

  22. [30]

    2024.Top Programming Languages 2024

    IEEE Spectrum. 2024.Top Programming Languages 2024. https://spectrum.ieee.org/top-programming-languages-2024

    2024

  23. [31]

    Jasmin Jahić and Ashkan Sami. 2024. State of Practice: LLMs in Software Engineering and Software Architecture. In2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C). IEEE, 311–318

    2024

  24. [32]

    JetBrains. 2021. KExercises. https://huggingface.co/datasets/JetBrains/KExercises

    2021

  25. [33]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation.arXiv preprint arXiv:2406.00515(2024)

    2024 arXiv

  26. [34]

    Junjie Li, Aseem Sangalay, Cheng Cheng, Yuan Tian, and Jinqiu Yang. 2024. Fine Tuning Large Language Model for Secure Code Generation. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering. 86–90

    2024

  27. [35]

    Feng Lin, Dong Jae Kim, et al. 2024. When llm-based code generation meets the software development process.arXiv e-prints(2024), arXiv–2403

    2024

  28. [36]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al

  29. [37]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems36 (2024)

    2024

  30. [38]

    Shuqi Liu, Han Wu, Bowei He, Xiongwei Han, Mingxuan Yuan, and Linqi Song. 2025. Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models. arXiv:2502.12420 [cs.CL] https://arxiv.org/abs/2502.12420

    2025 arXiv

  31. [39]

    Wei Lu, Rachel K Luu, and Markus J Buehler. 2024. Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities.arXiv preprint arXiv:2409.03444(2024)

    2024 arXiv

  32. [40]

    Wanqin Ma, Chenyang Yang, and Christian Kästner. 2024. (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs. InProceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 166–171

    2024

  33. [41]

    Desmarais, and Zhen Ming (Jack) Jiang

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability?Journal of Systems and Software203 (2023), 111734. doi:10.1016/j.jss.2023.111734

    2023

  34. [42]

    Atharva Naik. 2024. On the limitations of embedding based methods for measuring functional correctness for code generation.arXiv preprint arXiv:2405.01580(2024)

    2024 arXiv

  35. [43]

    Anh-Tu Phuong Nguyen, Van-Dung Hoang, et al. 2024. Development of Code Evaluation System based on Abstract Syntax Tree.Journal of Technical Education Science19, Special Issue 01 (2024), 15–24

    2024

  36. [44]

    Jernej Novak, Andrej Krajnc, et al. 2010. Taxonomy of static code analysis tools. InThe 33rd international convention MIPRO. IEEE, 418–422

    2010

  37. [45]

    Peter W O’Hearn. 2019. Incorrectness logic.Proceedings of the ACM on Programming Languages4, POPL (2019), 1–32

    2019

  38. [46]

    Michael Olan. 2003. Unit testing: test early, test often.J. Comput. Sci. Coll.19, 2 (Dec. 2003), 319–328

    2003

  39. [47]

    OpenAI. 2023. GPT-3.5 Model Documentation. https://platform.openai.com/docs/models/gpt-3-5 Accessed: 2025-02-21

    2023

  40. [48]

    OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Accessed: 2024-01-18

    2024

  41. [49]

    Wendkûuni C Ouédraogo, Yinghua Li, Kader Kaboré, Xunzhu Tang, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Test smells in LLM-Generated Unit Tests.arXiv preprint arXiv:2410.10628(2024)

    2024 arXiv

  42. [50]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–28

    2025

  43. [51]

    Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. 2024. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities.arXiv pr...

    2024 arXiv

  44. [52]

    PMD Developers. [n. d.]. PMD: An extensible cross-language static code analyzer. https://docs.pmd-code.org/latest/index.html. Accessed: 2025-01-23

    2025

  45. [53]

    Pylint. 2025. Pylint: Python Code Static Checker. https://pylint.pycqa.org/. Accessed: 2025-01-23

    2025

  46. [54]

    Python Software Foundation. [n. d.]. tracemalloc — Trace memory allocations. https://docs.python.org/3/library/tracemalloc.html Accessed: 14-Mar-2025

    2025

  47. [55]

    Willard Rafnsson, Rosario Giustolisi, Mark Kragerup, and Mathias Høyrup. 2020. Fixing vulnerabilities automatically with linters. InNetwork and System Security: 14th International Conference, NSS 2020, Melbourne, VIC, Australia, November 25–27, 2020, Proceedings 14. Springer, 224–244

    2020

  48. [56]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020). Manuscript submitted to ACM 24 Abbassi et al

    2020 arXiv

  49. [57]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al

  50. [58]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm.Proceedings of the ACM on Software Engineering1, FSE (2024...

    2024

  51. [59]

    Mohammed Latif Siddiq, Lindsay Roney, Jiahao Zhang, and Joanna Cecilia Da Silva Santos. 2024. Quality Assessment of ChatGPT Generated Code and their Use by Developers. InProceedings of the 21st International Conference on Mining Software Repositories(Lisbon, Portugal)(MSR ’24)...

    2024

  52. [60]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950(2023)

    2023 arXiv

  53. [61]

    SonarSource. [n. d.]. SonarQube: Continuous Code Quality and Security. https://www.sonarqube.org/. Accessed: 2025-01-23

    2025

  54. [62]

    Jeffrey Svajlenko and Chanchal K Roy. 2015. Evaluating clone detection tools with bigclonebench. In2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 131–140

    2015

  55. [63]

    Devarshi Singh, Varun Ramachandra Sekar, Kathryn T Stolee, and Brittany Johnson. 2017. Evaluating how static analysis tools can reduce code review effort. In2017 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 101–105

    2017

  56. [64]

    Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the ultimate programming assistant–how far is it?arXiv preprint arXiv:2304.11938(2023)

    2023 arXiv

  57. [65]

    John Törnblom and Simin Nadjm-Tehrani. 2021. Scaling up memory-efficient formal verification tools for tree ensembles.arXiv preprint arXiv:2105.02595(2021)

    2021 arXiv

  58. [66]

    Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2024. Bugs in large language models generated code.arXiv preprint arXiv:2403.08937(2024)

    2024 arXiv

  59. [67]

    Luca Traini, Daniele Di Pompeo, Michele Tucci, Bin Lin, Simone Scalabrino, Gabriele Bavota, Michele Lanza, Rocco Oliveto, and Vittorio Cortellessa

  60. [69]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  61. [70]

    Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. How and Why LLMs Use Deprecated APIs in Code Completion? An Empirical Study.arXiv preprint arXiv:2406.09834(2024)

    2024 arXiv

  62. [71]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems36 (2024)

    2024

  63. [72]

    Yongda Yu, Guoping Rong, Haifeng Shen, He Zhang, Dong Shao, Min Wang, Zhao Wei, Yong Xu, and Juhong Wang. 2024. Fine-tuning large language models to improve accuracy and comprehensibility of automated code review.ACM transactions on software engineering and methodology 34, 1 (...

    2024

  64. [73]

    Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. 2024. Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning. arXiv:2407.05040 [cs.SE] https://arxiv.org/abs/2407.05040

    2024 arXiv

  65. [74]

    Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. 2024. CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?arXiv preprint arXiv:2408.10718(2024)

    2024 arXiv

  66. [75]

    Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2024. Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models.arXiv preprint arXiv:2407.11470(2024)

    2024 arXiv

  67. [76]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al

  68. [77]

    Morteza Zakeri-Nasrabadi, Saeed Parsa, Mohammad Ramezani, Chanchal Roy, and Masoud Ekhtiarzadeh. 2023. A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges.Journal of Systems and Software(2023), 111796

    2023

  69. [78]

    Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. 2023. A survey of large language models for code: Evolution, benchmarking, and future trends.arXiv preprint arXiv:2311.10372(2023)

    2023 arXiv

  70. [79]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406...

    2024 arXiv

  71. [81]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  72. [82]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement.arXiv preprint arXiv:2402.14658(2024)

    2024 arXiv

  73. [2021]

    How software refactoring impacts execution time.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 2 (2021), 1–23

    2021

  74. [2023]

    Domain specialization as the key to make large language models disruptive: A comprehensive survey.arXiv preprint arXiv:2305.18703(2023)

    2023 arXiv

  75. [2024]

    What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study.arXiv preprint arXiv:2407.06153(2024)

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.