Pith. sign in

REVIEW 1 major objections 6 minor 43 references

SemaDiff's central claim is that running the same generated tests on both sides of a commit can distinguish semantic-preserving from semantic-changing commits, and its evaluation reports 100% precision on flagging semantic-changing commits.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:20 UTC pith:7JYPCWPB

load-bearing objection SemaDiff has a genuinely new idea—generate dependent code so the same tests run on both commit versions—but the 100% precision claim is weakened by a test-generation prompt that steers the model toward the expected verdict. the 1 major comments →

arxiv 2607.13111 v1 pith:7JYPCWPB submitted 2026-07-14 cs.SE cs.AI

SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

Maha Ayub , Michael Konstantinou , Ahmed Khanfir , Nikolaos Tsantalis , Mike Papadakis This is my paper
classification cs.SE cs.AI
keywords SemaDiffsemantic-preserving commitssemantic-changing commitsbehavioral analysistest generationlarge language modelscommit miningrefactoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Refactoring tools can detect structural edits but cannot tell whether a commit that looks like a refactoring actually changed program behavior. SemaDiff attacks this by generating, with a large language model, a synthetic dependent class that calls the modified code and compiles on both sides of the commit, then generating tests for that class and running the identical suite on the pre- and post-commit versions. If every test produces the same result on both versions, the commit is classified as semantic-preserving; if any test differs, it is semantic-changing. On a manually labeled dataset of 183 Java commits, the paper reports accuracy of about 76% and a precision of 100% for semantic-changing detection. If the approach holds up, repository-mining tasks such as bug-dataset construction, rollback analysis, and backporting could use it to obtain a commit-level signal that is conservative in the right direction: any flagged behavioral change really is one.

Core claim

SemaDiff's central claim is that behavioral equivalence across a commit can be decided by comparing execution results of the same generated test suite on both versions, where the suite targets a generated wrapper class that calls the commit's modified code through stable public APIs. The approach first parses the diff to locate changed code and any unchanged code that calls it; when no caller exists, or when the caller is too deeply embedded to be tested, an LLM generates a dependent class that compiles against both versions. Tests are then generated from the dependent code plus the diff, with prompts that instruct the model to write failing tests when behavior may change and passing tests o

What carries the argument

The generative-dependent-class mechanism is the load-bearing component. SemaDiff asks a large language model to synthesize a Java class that only calls the commit-modified code through public methods and is required to compile and run on both the parent and modified versions. Because the generated class isolates the changed lines behind a simple callable surface, the same LLM-generated tests can exercise the change in both versions, making the behavioral comparison possible. The paper's own ablation shows this component is essential: generated code exposes about 98% of the detected semantic-changing commits, whereas tests against the original developer-written dependent code alone expose onl

Load-bearing premise

The threads-to-validity section concedes reliance on generated tests and external parsing tools, but the load-bearing premise is that the LLM-generated tests, once they compile and run on both versions, are a faithful and sufficient behavioral oracle for the commit's changes — if the model's reading of the diff is wrong or shallow, identical outcomes across versions do not truly prove semantic preservation.

What would settle it

A single semantic-preserving commit for which any generated test fails on exactly one of the two versions would produce a false positive; the paper reports zero such cases across its 95 semantic-preserving commits, so one clean counterexample on a fresh sample would settle the precision claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mining pipelines can treat SemaDiff's 'semantic-changing' label as high-confidence evidence: with 100% precision on the benchmark, every flagged commit actually changes behavior, making it safe to use as a filter before costly analysis such as fault localization or backporting.
  • Semantic-preserving predictions are not equally reliable — recall is about 59% — so a commit labeled 'semantic-preserving' cannot be taken as proof of purity; the approach is a one-sided detector rather than a complete classifier.
  • The generated code, not the repository's existing callers, does the work: roughly 85% of detected semantic-changing commits are exposed only through the synthesized dependent class, implying that static caller-based testing alone is insufficient for this task.
  • LLM-generated tests dominate a rule-based regression-test generator in both coverage and accuracy, and the approach subsumes the rule-based variant's correct predictions — evidence that prompt-directed test generation, not just any automated testing, is what makes the behavioral comparison effective.
  • Compared with static baselines that inspect the diff without execution, SemaDiff trades some raw accuracy (about 76% versus up to about 87% for a static LLM) for a qualitative difference: 100% precision instead of roughly 81–84%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit is to combine SemaDiff's 100%-precision positive signal with a high-recall static classifier: the static model would select candidates and SemaDiff would confirm or reject them, giving a pipeline that keeps near-perfect precision while raising recall above 59%.
  • Because precision is perfect but recall is limited, the method's real utility is as a conservative alarm rather than a balanced classifier; this suggests task design should treat 'not flagged' as 'no evidence of change,' not 'semantically identical,' a distinction the paper itself does not stress.
  • The paper's dependence on LLM-generated tests means the oracle is ultimately the model's judgment about what behaviors matter; a testable extension would run multiple independently generated test suites and require agreement, or replace the assertion-based tests with differential fuzzing across the two versions, to see whether the 100% precision survives without prompt guidance.
  • The manual annotation treats any added functionality as semantic-changing, which is a conservative labeling choice; if the benchmark were re-annotated with a runtime-based oracle, reported precision and recall could shift, so the 76% accuracy should be interpreted relative to that labeling convention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes SemaDiff, a technique for classifying commits as semantic-preserving or semantic-changing. Given a commit, SemaDiff parses the diff, identifies modified code and unchanged callers, uses an LLM to generate a dependent class that calls the modified code, generates tests for that dependent class, and executes the same tests on the pre- and post-commit versions. A commit is classified as semantic-preserving only if all generated tests produce identical outcomes across versions. The evaluation uses a manually labeled dataset of 183 Java commits from seven open-source projects. The paper reports about 76% accuracy with 100% precision for detecting semantic-changing commits, and compares against PurityChecker, static LLM classification, and a Randoop-based variant.

Significance. If the 100% precision claim were well supported, SemaDiff would be a practically valuable tool for repository-mining tasks such as bug-dataset construction and backporting, where false positives are costly. The paper has several strengths: a manually inspected dataset, evaluation across three LLMs, a Randoop baseline, coverage analysis, and a stated replication package. However, the central empirical claim is currently undermined by a prompt-induced circularity in the test-generation procedure and by a subjective ground-truth labeling policy. These issues directly affect the headline 100% precision result and the comparison with static LLM baselines.

major comments (1)
  1. [§VI.C (RQ3) and Figure 6] The text and Figure 6 are inconsistent. The figure appears to show 1.9% of semantic-changing commits detected only by tests targeting real dependent code, 84.9% detected in both, and 13.2% detected only by tests targeting generated code. This implies tests targeting real dependent code participate in 86.8% of detections, not the '≈15%' stated in the text. The statement that '≈85% can only be identified through the generated dependent code' also contradicts the figure, which shows 84.9% as shared, not exclusive. Since RQ3 is used to justify the dependent-code-generation design decision, this discrepancy must be corrected before the RQ3 conclusion can be assessed.
minor comments (6)
  1. [§V.A and Table I] The counts are inconsistent. Table I sums to 88 semantic-preserving and 95 semantic-changing commits, while the text says '95 semantic-preserving and 88 semantic-changing commits.' Please correct this and ensure all later references (including the conclusion) use consistent numbers.
  2. [§VI.D (RQ4) and Table V] The text says SemaDiff_Rand's accuracy drops to 22.34%, but Table V reports 34.9%. Clarify whether 22.34% refers to a different computation, and align the text with the table.
  3. [Conclusion] The conclusion states an average accuracy of about 78%, but Table II reports 75.95%. Please align the quoted number with the reported result.
  4. [§VI.A (RQ1)] The sentence 'SEMADIFF achieved 100% accuracy on the preserving ones' should probably say '100% precision on semantic-changing commits' or 'no false positives.' As written, it is misleading because accuracy is a combined metric.
  5. [Throughout] There are several typos: 'accross' (abstract), 'orignal' (§VI.C), 'RefactorMinor' (§V.B) should be 'RefactoringMiner,' and 'anaylsis' (§VII). A careful proofreading pass is needed.
  6. [References] References [7] and [15] appear to refer to the same paper by Silva, Alves, and Andrade. They should be merged or distinctly cited.

Circularity Check

0 steps flagged

No significant circularity: dynamic test execution provides an independent oracle, and the Randoop baseline reproduces the 100% precision result.

full rationale

The paper's central claim is an empirical result, not a derivation that reduces to its inputs by construction. The behavioral oracle is real execution: SemaDiff generates dependent code and tests, runs them on both commit versions, and classifies a commit as semantic-changing only if a generated test produces different outcomes across versions. The test-generation prompt does ask the LLM to produce failing assertions for changed behavior and only passing tests for pure refactorings, which can steer coverage and may partly explain the high precision. However, this is not circularity: the LLM's belief does not determine the execution outcome. A test written while believing a diff is a pure refactoring can still fail at runtime if hidden behavioral changes exist, and the paper's own Figure 1 example is exactly such a case. Moreover, the paper's RQ4 experiment with Randoop — a rule-based, feedback-guided generator with no semantic prior from the diff — also achieves 100% precision on its successful runs, showing that the precision result is not an artifact of the LLM prompt alone. The comparison against static LLM baselines (ChatGPT 84.47% precision vs. SemaDiff 100%) is an empirical comparison, not a fitted prediction. The dataset is manually annotated by the authors and is independent of the tool's output; the approach also reports meaningful false negatives (recall 58.89%), which is inconsistent with a claim that the prediction is forced by construction. No load-bearing self-citation chain exists: RefactoringMiner/PurityChecker are used as baselines, not as justification for the main claim. The paper is therefore self-contained for the claims it makes, and the prompt-bias concern is a validity threat (correctly handled by the execution-based oracle and the Randoop comparison), not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or theoretical entities. Its free parameters are the LLM selection and prompt design, which are chosen by the authors and materially affect results. The core inference depends on the assumptions that test equivalence implies semantic equivalence, that LLM-generated tests provide sufficient coverage, and that the manually constructed ground truth is correct.

free parameters (2)
  • LLM model choice (GPT-5 vs DeepSeek vs CodeLlama) = GPT-5
    The paper selects ChatGPT (GPT-5) as the default after comparing variants; the choice impacts accuracy and failure rates substantially, so it is a hand-picked parameter that affects the central result.
  • LLM prompt templates = Prompts shown in Section III
    The exact prompts for dependent code generation and test generation are manually crafted and influence the behavior of the LLM; they are not derived from a formal principle.
axioms (3)
  • domain assumption A test suite that produces identical outcomes on both commit versions implies the versions are behaviorally equivalent for the tested behaviors.
    The core inference in Section III-C equates identical test results with semantic preservation; this is only valid under complete coverage and deterministic execution.
  • ad hoc to paper LLM-generated dependent code and tests exercise the modified code sufficiently to reveal semantic changes.
    The approach assumes that the LLM will produce harnesses that cover relevant edge cases; the paper's own RQ5 shows LLMs are imperfect at this task.
  • domain assumption The manual annotation of the 183 commits is accurate.
    Ground truth was labeled by three authors with discussion for ambiguous cases; the paper acknowledges subjectivity (e.g., labeling functionality additions as changing).

pith-pipeline@v1.3.0-alltime-deepseek · 16022 in / 7569 out tokens · 75820 ms · 2026-08-02T06:20:05.105753+00:00 · methodology

0 comments
read the original abstract

Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining. While existing approaches detect refactoring commits accurately, they cannot ensure that a commit is purely semantic-preserving, without any interleaving behaviour-changing modification. This limitation can impact several tasks, such as debugging, fault localisation, bug dataset construction, rollback analysis, and bug fixes backporting. To fill this gap, we propose SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis; comparison of similar test execution on pre- and post-commit versions. As code impacted by the refactoring is often hard to test and different accross both versions, we propose generating additional calling methods to that code, which serve as testing target. Given a commit, SemaDiff analyses the diff to identify modified code and extracts unchanged dependent code that calls it. It then generates an additional dependent class using a large language model to exercise the changed code in both versions, and automatically generates tests for the dependent code. This way, we obtain the same tests for the different code versions, enabling the behavioural-difference detection. The commit is classified as semantic-preserving only if all generated tests produce identical outcomes across the two versions. To evaluate SemaDiff, we construct and annotate manually a dataset of 183 commits, gathered from well-known open-source Java projects. The obtained results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.

Figures

Figures reproduced from arXiv: 2607.13111 by Ahmed Khanfir, Maha Ayub, Michael Konstantinou, Mike Papadakis, Nikolaos Tsantalis.

Figure 1
Figure 1. Figure 1: Example of semantic changing refactoring commit. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Refactoring semantic detection flow. such regression tools to ensure semantic equivalence after refactoring provides limited assurance. Beyond refactoring engines and regression testing tools, Large Language Models (LLMs) have recently been adopted in software engineering tasks. LLMs can generate functional code directly from natural language problem descriptions with relatively high success rates [28]. Th… view at source ↗
Figure 3
Figure 3. Figure 3: Example of generated dependent. Once the code is generated, we add it to both versions of the project and compile it. If it does not compile, we repeat code generation, i.e. by asking the model again; otherwise, it is kept as additional target-dependent code to test. Subfigure 3a illustrates an example commit diff2 , which renames the private method defensiveCopy to clone. As can be seen in Subfigure 3b, t… view at source ↗
Figure 4
Figure 4. Figure 4: Percentage of the top 10 refactoring types in semantically and non-semantically preserving commits [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Confidence of manually labeling conflicting commits [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Code coverage of the modified code by test cases [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Percentage of detected semantic changing commits [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Generated and developer-written code for calling a [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average modified code coverage of tests generated [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [1]

    Available: https://github.com/

    “Github.” [Online]. Available: https://github.com/

  2. [2]

    What is backporting? the process & how it works — crowdstrike,

    K. Imtiaz, “What is backporting? the process & how it works — crowdstrike,” Aug. 2025. [Online]. Available: https://www.crowdstrike. com/en-us/cybersecurity-101/exposure-management/backporting/

    2025

  3. [3]

    Enhancing oss patch backporting with semantics,

    S. Yang, Y . Xiao, Z. Xu, C. Sun, C. Ji, and Y . Zhang, “Enhancing oss patch backporting with semantics,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 2366–2380. [Online]. Available: https://doi.org/10.1145/3576915.3623188

    arXiv 2023

  4. [4]

    Characterizing architectural information in commit messages: an exploratory study,

    T. O. Motta, R. R. Gomes e Souza, and C. Sant’Anna, “Characterizing architectural information in commit messages: an exploratory study,” inProceedings of the XXXII Brazilian Symposium on Software Engineering, ser. SBES ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 12–21. [Online]. Available: https://doi.org/10.1145/3266237.3266260

    arXiv 2018

  5. [5]

    On the co-occurrence of refactoring of test and source code,

    N. A. Nagy and R. Abdalkareem, “On the co-occurrence of refactoring of test and source code,” inProceedings of the 19th International Conference on Mining Software Repositories, ser. MSR ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 122–126. [Online]. Available: https://doi.org/10.1145/3524842.3528529

    arXiv 2022

  6. [6]

    Automated testing of refactoring engines,

    B. Daniel, D. Dig, K. Garcia, and D. Marinov, “Automated testing of refactoring engines,” inProceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering, ser. ESEC-FSE ’07. New York, NY , USA: Association for Computing Machinery, 2007, p. 185–194. [Online]...

    arXiv 2007

  7. [8]

    A comparative study of manual and automated refactorings,

    S. Negara, N. Chen, M. Vakilian, R. E. Johnson, and D. Dig, “A comparative study of manual and automated refactorings,” inECOOP 2013 – Object-Oriented Programming, G. Castagna, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 552–576

    2013

  8. [9]

    A field study of refactoring challenges and benefits,

    M. Kim, T. Zimmermann, and N. Nagappan, “A field study of refactoring challenges and benefits,” inProceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ser. FSE ’12. New York, NY , USA: Association for Computing Machinery, 2012. [Online]. Available: https://doi.org/10.1145/2393596.2393655

    arXiv 2012

  9. [10]

    Text-219

    “Text-219.” [Online]. Available: https://issues.apache.org/jira/projects/ TEXT/issues/TEXT-219?filter=allopenissues

  10. [11]

    Github - tsantalis/refactoringminer

    Tsantalis, “Github - tsantalis/refactoringminer.” [Online]. Available: https://github.com/tsantalis/RefactoringMiner

  11. [12]

    Github - aserg-ufmg/refdiff: A tool to mine refactorings in the commit history of git repositories

    Aserg-Ufmg, “Github - aserg-ufmg/refdiff: A tool to mine refactorings in the commit history of git repositories.” [Online]. Available: https://github.com/aserg-ufmg/RefDiff

  12. [13]

    Refactoringminer/documentation/purity.md at master · tsantalis/refactoringminer

    Tsantalis, “Refactoringminer/documentation/purity.md at master · tsantalis/refactoringminer.” [Online]. Available: https://github.com/ tsantalis/RefactoringMiner/blob/master/documentation/purity.md

  13. [14]

    Syntactic versus semantic similarity of artificial and real faults in mutation testing studies,

    M. Ojdanic, A. Garg, A. Khanfir, R. Degiovanni, M. Papadakis, and Y . Le Traon, “Syntactic versus semantic similarity of artificial and real faults in mutation testing studies,”IEEE Transactions on Software Engineering, vol. 49, no. 7, pp. 3922–3938, 2023

    2023

  14. [15]

    Analyzing automatic test generation tools for refactoring validation,

    I. P. Silva, E. L. Alves, and W. L. Andrade, “Analyzing automatic test generation tools for refactoring validation,” in2017 IEEE/ACM 12th International Workshop on Automation of Software Testing (AST), 2017, pp. 38–44

    2017

  15. [16]

    What is refactoring? — agile alliance,

    D. Ops, “What is refactoring? — agile alliance,” Aug. 2022. [Online]. Available: https://agilealliance.org/glossary/refactoring/

    2022

  16. [17]

    When does a refactoring induce bugs? an empirical study,

    G. Bavota, B. De Carluccio, A. De Lucia, M. Di Penta, R. Oliveto, and O. Strollo, “When does a refactoring induce bugs? an empirical study,” in2012 IEEE 12th International Working Conference on Source Code Analysis and Manipulation, 2012, pp. 104–113

    2012

  17. [18]

    On the relationship between refactoring actions and bugs: a differentiated replication,

    M. Di Penta, G. Bavota, and F. Zampetti, “On the relationship between refactoring actions and bugs: a differentiated replication,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2020. New York, NY , USA: Association for Computing Machinery, 20...

    arXiv 2020

  18. [19]

    Untangling fine-grained code changes,

    M. Dias, A. Bacchelli, G. Gousios, D. Cassou, and S. Ducasse, “Untangling fine-grained code changes,” 2015. [Online]. Available: https://arxiv.org/abs/1502.06757

    Pith/arXiv arXiv 2015

  19. [20]

    Utango: untangling commits with context-aware, graph-based, code change clustering learning model,

    Y . Li, S. Wang, and T. N. Nguyen, “Utango: untangling commits with context-aware, graph-based, code change clustering learning model,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY , USA: Association for Computing Machinery, 2022, p. ...

    arXiv 2022

  20. [21]

    Manual refactoring changes with automated refactoring validation,

    X. Ge and E. Murphy-Hill, “Manual refactoring changes with automated refactoring validation,” inProceedings of the 36th International Conference on Software Engineering, ser. ICSE 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 1095–1105. [Online]. Available: https://doi.org/10.1145/2568225.2568280

    arXiv 2014

  21. [22]

    Saferefactor – tool for checking refactoring safety,

    G. Soares, D. Cavalcanti, R. Gheyi, T. Massoni, D. Serey, and M. Corne- lio, “Saferefactor – tool for checking refactoring safety,” inTools Session at SBES, 2009, pp. 49–54

    2009

  22. [23]

    Towards understanding refactoring engine bugs,

    H. Wang, Z. Xu, H. Zhang, N. Tsantalis, and S. H. Tan, “Towards understanding refactoring engine bugs,”ACM Trans. Softw. Eng. Methodol., Jul. 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3747289

    doi:10.1145/3747289 2025

  23. [24]

    Detecting refactoring commits in machine learning python projects: A machine learning-based approach,

    S. Noei, H. Li, and Y . Zou, “Detecting refactoring commits in machine learning python projects: A machine learning-based approach,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, Feb. 2025. [Online]. Available: https://doi.org/10.1145/3705309

    doi:10.1145/3705309 2025

  24. [25]

    Refactoringminer 2.0,

    N. Tsantalis, A. Ketkar, and D. Dig, “Refactoringminer 2.0,”IEEE Transactions on Software Engineering, vol. 48, no. 3, pp. 930–950, 2022

    2022

  25. [26]

    Refdiff: Detecting refactorings in version histories,

    D. Silva and M. T. Valente, “Refdiff: Detecting refactorings in version histories,” in2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017, pp. 269–279

    2017

  26. [27]

    Evaluating the effectiveness of regression test suites for extract method validation,

    L. Gomes, C. Cordeiro, and E. L. G. Alves, “Evaluating the effectiveness of regression test suites for extract method validation,” inProceedings of the 7th Brazilian Symposium on Systematic and Automated Software Testing, ser. SAST ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1–8. [Online]. Available: https://doi.org/10.1145/3559...

    arXiv 2022

  27. [28]

    Test-driven development and llm-based code generation,

    N. S. Mathews and M. Nagappan, “Test-driven development and llm-based code generation,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 1583–1594. [Online]. Available: https://doi.org/10. 1145/3691620.3695527

    arXiv 2024

  28. [29]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, ser. FSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 572–576. [Online]. Available: https://doi.org/10.1145/3...

    arXiv 2024

  29. [30]

    Maintaining unit tests during refactoring,

    H. Passier, L. Bijlsma, and C. Bockisch, “Maintaining unit tests during refactoring,” inProceedings of the 13th International Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools, ser. PPPJ ’16. New York, NY , USA: Association for Computing Machinery, 2016. [Online]. Available: https://doi.org/...

    arXiv 2016

  30. [31]

    A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,

    S. B. Dristi and M. B. Dwyer, “A differential fuzzing-based evaluation of functional equivalence in llm-generated code refactorings,” 2026. [Online]. Available: https://arxiv.org/abs/2602.15761

    arXiv 2026

  31. [32]

    Changeguard: Validating code changes via pairwise learning-guided execution,

    L. Gr ¨oninger, B. Souza, and M. Pradel, “Changeguard: Validating code changes via pairwise learning-guided execution,”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715760

    work page doi:10.1145/3715760 2025

  32. [33]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschelet al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  33. [34]

    Untangling changes,

    K. Herzig and A. Zeller, “Untangling changes,”Unpublished manuscript, September, vol. 37, pp. 38–40, 2011

    2011

  34. [35]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2203.02155

    Pith/arXiv arXiv 2022

  35. [36]

    Code llama: Open foundation models for code

    “Code llama: Open foundation models for code.” [Online]. Available: https://arxiv.org/html/2308.12950v3#bib

    Pith/arXiv arXiv

  36. [37]

    Deepseek llm: Scaling open-source language models with longtermism,

    DeepSeek-AI, :, X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y . He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y . Li, Y . K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y . Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y . Pi...

    Pith/arXiv arXiv 2024

  37. [38]

    Evaluating large language models for software testing,

    Y . Li, P. Liu, H. Wang, J. Chu, and W. E. Wong, “Evaluating large language models for software testing,”Computer Standards & Interfaces, vol. 93, p. 103942, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0920548924001119

    2025

  38. [39]

    How well llm- based test generation techniques perform with newer llm versions?

    M. Konstantinou, R. Degiovanni, and M. Papadakis, “How well llm- based test generation techniques perform with newer llm versions?” arXiv preprint arXiv:2601.09695, 2026

    Pith/arXiv arXiv 2026

  39. [40]

    SPOON: A library for implementing analyses and transformations of java source code,

    R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “SPOON: A library for implementing analyses and transformations of java source code,”Softw. Pract. Exp., vol. 46, no. 9, pp. 1155–1179, 2016

    2016

  40. [41]

    Release randoop version 4.3.4 · randoop/randoop

    Randoop, “Release randoop version 4.3.4 · randoop/randoop.” [Online]. Available: https://github.com/randoop/randoop/releases/tag/v4.3.4

  41. [42]

    Introducing gpt-5,

    OpenAI, “Introducing gpt-5,” Aug. 2025. [Online]. Available: https: //openai.com/index/introducing-gpt-5/

    2025

  42. [43]

    Your first api call — deepseek api docs

    “Your first api call — deepseek api docs.” [Online]. Available: https://api-docs.deepseek.com/

  43. [44]

    Llama 3.3 70b api — together ai

    “Llama 3.3 70b api — together ai.” [Online]. Available: https: //www.together.ai/models/llama-3-3-70b