Pith. sign in

REVIEW 3 major objections 5 minor 71 references

LLM-Based Invariant Testing for Software Functional Bugs

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage LLM pipeline that synthesizes API call sequences and then inserts documentation-grounded invariants detects 12 of 25 re-introduced functional bugs in C/C++ libraries, outperforming both prior LLM unit testing and crash-based fuz

desk verdict LISA's two-stage decomposition is a real contribution, but the headline bug-detection claim is demonstrated only against a known-good reference build, and the paper doesn't own that gap. read the letter →

arxiv 2607.18711 v1 pith:4JLIKMCN submitted 2026-07-21 cs.SE

Ruogu Yang , Yifeng He , Yundi Xu , Yuqing Wei , Hao Chen This is my paper · ORCID
classification cs.SE
keywords functionalbugdetectioninvarianttestingLLM-basedtestgenerationunitAPIn-gramfeedbackoracleC/C++libraries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that functional bugs—silent logic errors that return wrong results without crashing—can be detected automatically by having a large language model generate executable program invariants. The authors propose LISA, which first synthesizes valid sequences of API calls using an n-gram co-occurrence feedback loop, then inserts invariants at chunk boundaries, grounding them in API documentation and filtered dynamic-analysis candidates. This decomposition separates the problem of reaching deep code states from the problem of knowing what correct behavior looks like, sidestepping the oracle problem that makes exact input–output tests brittle. On 25 re-introduced historical bugs across five C/C++ libraries, LISA detects 12 (48%), versus 3 for a state-of-the-art LLM unit test generator and 2 for a crash-focused fuzzing platform, with statistically significant p < 0.05 for both comparisons and higher average line and branch coverage.

What carries the argument

The central mechanism is the two-stage decomposition of unit test generation. In stage one, API-sequence synthesis, an API n-gram model (order 3) captures co-occurrence patterns among calls, and an adaptive energy-normalization scheme (ACNE) balances exploration of under-used APIs against exploitation of known-successful combinations; a bounded repair loop fixes compilation and execution errors. In stage two, chunk-invariant reasoning partitions each sequence into semantically coherent chunks and prompts the LLM to insert invariants at chunk boundaries, supported by a knowledge base of API contracts assembled from official documentation and filtered dynamic-analysis candidates. A verificatio

What would settle it

Measure, across the 25-bug corpus, how often the invariant-repair loop weakens or deletes an assertion that would otherwise fail on the buggy build but passes on the reference build; or run LISA on a library whose documentation is known to be wrong and check whether the resulting invariants fail to flag the corresponding bug.

Watch

Extended reading notes

Core claim

The core discovery is that invariant testing—using partial, documentation-grounded assertions as oracles instead of exact expected outputs—turns LLM-generated tests into high-confidence detectors of functional bugs. An invariant counts as semantically valid when it is consistent with the documented contract of the APIs in scope and holds on the reference build; a later violation on a target build is then reported as a bug candidate for developer confirmation. The authors show that the two-stage decomposition works: feedback-guided API-sequence generation achieves high validity and coverage, and chunk-level invariant insertion with a verify-and-repair loop produces assertions that catch silen

Load-bearing premise

The load-bearing premise is that an invariant is a valid oracle if it matches the API documentation and holds on the reference build; if the documentation is inaccurate or the reference build embodies the bug, the verify-and-repair loop can weaken or discard the very assertion that would catch the fault.

Editorial extensions

If this is right

  • LISA's generated invariant-enriched tests can serve as regression suites for C/C++ libraries, catching behavioral regressions in future commits without developer-written oracles.
  • The two-stage design suggests that other test-generation tasks can improve by separating reachability (exploring valid API interactions) from oracle construction (deciding what correct behavior means).
  • Because the invariants are grounded in documentation, any library with public API documentation is a candidate target, extending automated functional-bug detection broadly across the software supply chain.
  • The benchmark of 25 re-introduced historical functional bugs provides a reusable evaluation protocol for comparing functional-bug detectors beyond crash coverage.
  • The n-gram feedback mechanism offers a lightweight alternative to explicit API dependency graphs, with sensitivity analysis showing order 3 maximizes validity and coverage under a fixed budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method inherits a blind spot: bugs that violate undocumented conventions will escape, since invariants are only generated for behaviors the documentation or filtered candidates make explicit; mining usage patterns or existing test suites could lift this ceiling.
  • The verification-and-repair loop that weakens failing assertions could mask real faults when the target build itself is buggy; quantifying how often repair deletes bug-catching assertions would sharpen the reported recall.
  • The two-stage decomposition is likely portable to managed languages (Python, Java) and to other domains with API-sequence structures, though the C/C++-specific repair rules would need adaptation.
  • Replacing the n-gram co-occurrence model with richer API dependency information could raise sequence validity further, but at the cost of the framework's current simplicity and model-agnostic gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LISA, a two-stage LLM-based framework for generating invariant-enriched unit tests for C/C++ libraries. The first stage synthesizes straight-line API call sequences under n-gram feedback and an error-repair loop; the second partitions each sequence into chunks and uses an LLM, grounded in API documentation and Daikon-mined candidates, to insert executable invariants at chunk boundaries, with a verification-and-repair loop. The evaluation uses a new benchmark, LISA-BENCH, of 25 re-introduced historical functional bugs across five libraries. The headline result is that LISA detects 12/25 (48%) bugs versus 3/25 (12%) for CITYWALK and 2/25 (8%) for OSS-Fuzz, with Fisher exact p-values below 0.05, and reports higher average line/branch coverage than OSS-Fuzz over seven libraries.

Significance. If the empirical claims are correct, LISA's two-stage decomposition—separating API-sequence reachability from oracle construction—is a useful contribution to functional-bug testing, and the paper offers a rare attempt to create a ground-truth benchmark for this setting. The evaluation has notable strengths: historical bugs are re-introduced deterministically, the selection protocol is fixed and described, the pass-on-patched/fail-on-buggy definition is explicit, ablations isolate component contributions, and the artifact is promised. However, the central capability claim is stated more broadly than the evidence supports: the method's invariant validation assumes a known-good reference build, and the main bug-detection comparison gives LISA access to oracle information (a curated documentation knowledge base and the patched build) that the baselines do not receive. The significance of the result therefore depends on whether the claims are repositioned as regression-testing capability rather than general automated functional-bug detection.

major comments (3)
  1. [§III-F, §III-G2, §IV-C2] Semantic validity is defined as an invariant holding on the reference build (§III-F), and bug detection is counted as pass-on-patched/fail-on-buggy (§IV-C2). In deployment against a single target build without a known-good reference, the verification-and-repair loop in §III-G2 executes each candidate against the target; a bug-revealing invariant fails validation and is weakened or removed by the repair prompt, or is only flagged as a 'potential unknown bug' after K=5 failed attempts. The paper does not quantify how often this masks real faults, and §VI-B's threat model discusses over-strong assertions and documentation coverage but not this reference-build circularity. The headline result therefore demonstrates regression testing with a known-good build, not the claimed capability of detecting functional bugs in an arbitrary target library. Please either reposition the contribution as re
  2. [§IV-C2, §V-D, Table VI] The bug-detection comparison gives LISA oracle information that is unavailable to CITYWALK and OSS-Fuzz: a manually curated API knowledge base (Section III-G1 reports 5.0 minutes per API for 20 APIs) and the patched build used for invariant validation. Table VI shows that removing the knowledge base drops detection from 12 to 3 (w/o Knowledge), and Daikon-only or doc-only variants reach only 5 and 4, respectively. This suggests a substantial portion of the reported gain is due to information sources the baselines do not receive, rather than to the architecture alone. To support the automated-detection claim, the comparison should either provide comparable oracle information to baselines or report a fully autonomous LISA variant as the headline configuration.
  3. [§V-D] The Fisher exact p-values (p=0.0036, p=0.0121) treat the 25 historical bugs as independent trials. The bugs are clustered within five libraries, share library-specific API sequences, knowledge-base entries, and repair behavior, and the per-library counts are inconsistent (cJSON 3/5, lcms 3/5, zlib 2/5, sqlite 2/5, libpng 2/5). Independence is therefore unlikely to hold. A cluster-aware test (e.g., permutation by library or mixed-effects model) is needed to support the significance claim; as reported, the p-values are likely overconfident.
minor comments (5)
  1. [§III-E1, Eq. (1)] The term 'newly discovered 3-gram' is not formally defined. Does novelty mean first occurrence across all generated traces in a run, or across all programs? Clarify the scope of the energy update.
  2. [§V-A, Table I] The n-gram order sensitivity analysis is reported only for zlib. The text points to the artifact for the full sweep, but the paper would be easier to assess if the other libraries' sensitivity at least summarized in an appendix.
  3. [§IV-C2] The displayed equation for AUVC is garbled in the plain text ('AUVC= P over t∈Tpass U(t)/|T_total|'). Please typeset it correctly.
  4. [Data Availability] The artifact links are inconsistent: the text gives two GitHub URLs under different names (SecurityLab-UCD/CNTG and SecurityLab-UCD/CGNTG) plus a Zenodo DOI. Please verify and unify these references.
  5. [§I, Contributions] The claim 'first to recast functional-bug detection as a decoupled two-stage problem' is strong. Since the components are drawn from PROMPTFUZZ, Daikon, SmartInv, and prior chunk-based prompting, please phrase the novelty claim more precisely, e.g., 'first to combine these in this decomposition for C/C++ library functional-bug detection.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LISA's headline bug-detection result is an empirical differential-testing evaluation against held-out historical bugs, not a derivation from fitted inputs or a self-citation chain.

full rationale

The central claims are empirical and externally grounded. LISA-BENCH uses 25 re-introduced historical bugs chosen by a fixed protocol (reverse-chronological scan of developer-fixed commits, deterministic reproduction, non-crashing functional symptom), and a bug counts as detected only when a generated test passes on the patched branch and fails on the buggy branch (Section IV-C2). The invariant filter ('holds on the reference build') is what makes an assertion a valid regression oracle; it does not by construction force detection, as evidenced by LISA finding only 12/25 and by the ablation variants dropping to 4/25 or fewer (Table VI). Hyperparameters N and alpha_min were chosen from coverage/sensitivity analysis (Table I, Appendix B-B), not from the bug corpus. Daikon candidates are mined from LISA-generated sequences, but only as oracle seeds, and the buggy target version is not an input to invariant construction. The paper's stated reliance on a correct reference build is a deployment limitation (Section VI-B), not a circular derivation: without a known-good reference, the repair loop could weaken bug-revealing assertions, but this is a validity threat and does not make the demonstrated result equivalent to its inputs. Self-citations (e.g., [17], [18], [41], [69]) are contextual and non-load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

LISA proposes no new physical or theoretical entities. Its inventions are a method decomposition, an energy-feedback scheme (ACNE), and a benchmark corpus; these are artifacts, not postulated entities requiring independent falsifiable handles.

free parameters (5)
  • N (n-gram order) = 3
    Set from a coverage sweep on zlib only (Table I); N=2 under-constrains, N=4 too sparse. Applied to all libraries and to bug detection without separate validation.
  • alpha_min (max condensation factor) = 0.5
    Derived in Appendix B-B from an arbitrary 10x selection-ratio target (log10/log100); controls the exploration/exploitation trade-off in API sampling.
  • epsilon (normalization floor) = 0.01
    Hand-set constant in Eq. 2 to keep a nonzero sampling probability for every API function.
  • K (invariant repair attempts) = 5
    Cap on the invariant-repair loop (Section III-G2a); chosen because 'gains plateau beyond that', affects the trade-off between salvaging over-strong assertions and discarding real bug signals.
  • LLM model choices = gpt-5-mini (LISA API), gpt-5.1 (LISA inv)
    Model capability directly affects sequence validity and invariant quality; closed APIs limit exact reproducibility.
assumptions (5)
  • domain assumption API n-gram co-occurrence in successful generated programs is a useful proxy for semantic API dependencies and guides exploration toward deep, under-tested states.
    Section III-E. If common n-grams are shallow, feedback may reinforce trivial patterns instead of deep state exploration; the ablation on 3 libraries suggests it helps but does not prove generality.
  • domain assumption Documentation (plus Daikon candidates filtered by hand) correctly captures intended API semantics, and invariants that match docs and pass the reference build are valid oracles.
    Sections III-F and III-G2. If documentation is stale or wrong, or if the reference build contains the target bug, the invariants will be wrong or the repair loop may mask failures.
  • domain assumption Straight-line API sequences (no branches or loops in the driver) still reach deep library states and are sufficient for functional-bug detection.
    Appendix C-B. The paper acknowledges this trade-off; state-dependent bugs that require branchy setup may be systematically missed.
  • domain assumption Deterministic reproduction on the authors' toolchain and a fix message flagging a non-crashing functional defect define a valid functional-bug corpus.
    Section IV-C2. The selection protocol excludes non-deterministic and environment-dependent bugs, biasing the benchmark toward assertion-friendly bugs.
  • standard math Standard calculus facts used in Proposition 1 (monotonicity, concavity of x^alpha) are correct.
    Appendix B-A. Derivative and concavity properties are standard; no issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Based Invariant Testing for Software Functional Bugs." pith.science (2026). https://pith.science/paper/4JLIKMCN

@misc{pith2026260718711,
  author       = {Pith},
  title        = {Pith review of: LLM-Based Invariant Testing for Software Functional Bugs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4JLIKMCN}},
  note         = {Machine review of arXiv:2607.18711}
}
read the original abstract

Manually writing unit tests to uncover functional bugs in software libraries is not only time-consuming but also requires a deep understanding of the intended semantics of the APIs. Heuristic-based test generation methods suffer from low usability because they cannot reason about program semantics or interpret source code and documentation as humans do. Traditional fuzzing techniques like OSS-Fuzz often rely on crashes to detect bugs, but functional bugs do not always cause crashes. To overcome these limitations, we present LISA, a novel LLM-based invariant testing framework for software functional bugs. LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.

Figures

Figures reproduced from arXiv: 2607.18711 by the authors.

Figure 1
Figure 1. Overview of LISA’s iterative feedback-guided API sequence generation. parsing the generated JSON-formatted AST. All types are then canonicalized into a standardized representation that preserves pointer mutability and array size information. This metadata is serialized into structured schemas that serve as the foundation for API-sequence generation, not fuzzing-driver synthesis. C. API Sequence Generation LISA adopt… view at source ↗
Figure 2
Figure 2. API knowledge database preparation. than soundness in formal verification: LISA does not prove φ over all inputs, only that φ is consistent with the documented contract and with observed reference behavior. In that sense LISA’s invariants are weaker as specifications, since they are partial and unproven, yet more robust as oracles, since asserting a stable property is less brittle than predicting an exact output [18… view at source ↗
Figure 3
Figure 3. Overview of LISA’s incremental invariant generation pipeline. mated Extraction: Daikon [43] mines likely invariants from executable API sequences, and an LLM extracts intended in￾variants from official API documentation [44] (e.g., “read-only” or “no internal state modified”). 2) Manual Filtering: We keep a Daikon candidate only if it does not contradict the documentation and does not cause assertion failures when a… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of AUVC between LISA and CITYWALK. sqlite3 (31.06% vs. 14.40%). This confirms LISA’s n-gram feedback and repair mechanisms are essential for guiding LLM beyond shallow happy paths to explore deep, unseen code. b) Comparison with Fuzzing: Compared to the indu…
Figure 5
Figure 5. Figure 5: The structured prompt template employed in our [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: The structured repair prompt template for correcting [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 1 canonical work pages

  1. [1]

    Quantitative analysis of faults and failures in a complex software system

    N.E. Fenton and N. Ohlsson. “Quantitative analysis of faults and failures in a complex software system”. In: IEEE Transactions on Software Engineering26.8 (2000), pp. 797–814

    2000

  2. [2]

    An empirical study of operating systems errors

    Andy Chou et al. “An empirical study of operating systems errors”. In:SIGOPS Oper. Syst. Rev.35.5 (Oct. 2001), pp. 73–88

    2001

  3. [3]

    Hao-Nan Zhu et al.From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets. 2025. arXiv: 2504.17977[cs.SE]

    arXiv 2025

  4. [4]

    Continuous Fuzzing with libFuzzer and AddressSanitizer

    Kosta Serebryany. “Continuous Fuzzing with libFuzzer and AddressSanitizer”. In:2016 IEEE Cybersecurity Development (SecDev). 2016, pp. 157–157

    2016

  5. [5]

    AFL++ combining incremental steps of fuzzing research

    Andrea Fioraldi et al. “AFL++ combining incremental steps of fuzzing research”. In:Proceedings of the 14th USENIX Conference on Offensive Technologies. 2020, pp. 10–10

    2020

  6. [6]

    Directed Greybox Fuzzing

    Marcel Böhme et al. “Directed Greybox Fuzzing”. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. CCS ’17. Dal- las, Texas, USA: Association for Computing Machinery, 2017, pp. 2329–2344. 1https://github.com/SecurityLab-UCD/CNTG and https://github.com/ SecurityLab-UCD/CGNTG

    2017

  7. [7]

    OSS-Fuzz - Google’s continuous fuzzing service for open source software

    Kostya Serebryany. “OSS-Fuzz - Google’s continuous fuzzing service for open source software”. In: Vancouver, BC: USENIX Association, Aug. 2017

    2017

  8. [8]

    SmartInv: Multimodal Learning for Smart Contract Invariant Inference

    Sally Junsong Wang, Kexin Pei, and Junfeng Yang. “ SmartInv: Multimodal Learning for Smart Contract Invariant Inference”. In:2024 IEEE Symposium on Security and Privacy (SP). Los Alamitos, CA, USA: IEEE Computer Society, May 2024, pp. 2217–2235

    2024

Show all 71 references
  1. [9]

    Simon and Schuster, 2020

    Vladimir Khorikov.Unit Testing Principles, Practices, and Patterns. Simon and Schuster, 2020

    2020

  2. [10]

    The impact of continuous integration on other software development practices: A large-scale empirical study

    Yangyang Zhao et al. “The impact of continuous integration on other software development practices: A large-scale empirical study”. In:2017 32nd IEEE/ACM International Conference on Automated Software Engi- neering (ASE). 2017

    2017

  3. [11]

    When, How, and Why Developers (Do Not) Test in Their IDEs

    Moritz Beller et al. “When, How, and Why Developers (Do Not) Test in Their IDEs”. In:Proceedings of the 2015 10th Joint Meeting on Foundations of Software En- gineering. ESEC/FSE 2015. Bergamo, Italy: Association for Computing Machinery, 2015, pp. 179–190

    2015

  4. [12]

    Feedback-Directed Random Test Generation

    Carlos Pacheco et al. “Feedback-Directed Random Test Generation”. In:29th International Conference on Software Engineering (ICSE’07). 2007, pp. 75–84

    2007

  5. [13]

    EvoSuite: automatic test suite generation for object-oriented software

    Gordon Fraser and Andrea Arcuri. “EvoSuite: automatic test suite generation for object-oriented software”. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. ESEC/FSE ’11. Szeged, Hungary: Association for ...

    2011

  6. [14]

    How Do Automatically Generated Unit Tests Influence Software Maintenance?

    Sina Shamshiri et al. “How Do Automatically Generated Unit Tests Influence Software Maintenance?” In:2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 2018, pp. 250–261

    2018

  7. [15]

    Revisiting test smells in automatically generated tests: limitations, pitfalls, and opportunities

    Annibale Panichella et al. “Revisiting test smells in automatically generated tests: limitations, pitfalls, and opportunities”. In:2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE. 2020, pp. 523–533

    2020

  8. [16]

    CAT-LM Training Language Models on Aligned Code and Tests

    Nikitha Rao et al. “CAT-LM Training Language Models on Aligned Code and Tests”. In:Proceedings of the 38th IEEE/ACM International Conference on Automated Soft- ware Engineering. ASE ’23. Echternach, Luxembourg: IEEE Press, 2024, pp. 409–420

    2024

  9. [17]

    UniTSyn: A Large-Scale Dataset Capa- ble of Enhancing the Prowess of Large Language Models for Program Testing

    Yifeng He et al. “UniTSyn: A Large-Scale Dataset Capa- ble of Enhancing the Prowess of Large Language Models for Program Testing”. In:Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA 2024. Vienna, Austria: Association for Comp...

    2024

  10. [18]

    FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation

    Yifeng He et al. “FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation”. In:Findings of the Association for Computational Lin- guistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 15642–15655

    2025

  11. [19]

    CITYWALK: Enhancing LLM- Based C++ Unit Test Generation via Project-Dependency Awareness and Language-Specific Knowledge

    Yuwei Zhang et al. “CITYWALK: Enhancing LLM- Based C++ Unit Test Generation via Project-Dependency Awareness and Language-Specific Knowledge”. In:ACM Trans. Softw. Eng. Methodol.(Aug. 2025). Just Accepted

    2025

  12. [20]

    Demystifying LLM-Based Software Engineering Agents

    Chunqiu Steven Xia et al. “Demystifying LLM-Based Software Engineering Agents”. In:Proc. ACM Softw. Eng.2.FSE (June 2025)

    2025

  13. [21]

    Learning Deep Semantics for Test Completion

    Pengyu Nie et al. “Learning Deep Semantics for Test Completion”. In:Proceedings of the 45th International Conference on Software Engineering. ICSE ’23. Mel- bourne, Victoria, Australia: IEEE Press, 2023, pp. 2111– 2123

    2023

  14. [22]

    TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark

    Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. “TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark”. In:The Thirteenth International Conference on Learning Representations. 2025

    2025

  15. [23]

    Theoretical and Empirical Studies of Program Testing

    William E. Howden. “Theoretical and Empirical Studies of Program Testing”. In:Proceedings of the 3rd Inter- national Conference on Software Engineering (ICSE). ICSE ’78. Atlanta, Georgia, USA: IEEE Press, 1978, pp. 305–311

    1978

  16. [24]

    On Testing Non-testable Programs

    Elaine J. Weyuker. “On Testing Non-testable Programs”. In:The Computer Journal25.4 (1982), pp. 465–470

    1982

  17. [25]

    The Oracle Problem in Software Testing: A Survey

    Earl T. Barr et al. “The Oracle Problem in Software Testing: A Survey”. In:IEEE Transactions on Software Engineering41.5 (2015), pp. 507–525

    2015

  18. [26]

    Prompt Fuzzing for Fuzz Driver Generation

    Yunlong Lyu et al. “Prompt Fuzzing for Fuzz Driver Generation”. In:Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. CCS ’24. Salt Lake City, UT, USA: Association for Computing Machinery, 2024, pp. 3793–3807

    2024

  19. [27]

    Can Large Language Models Reason about Program Invariants?

    Kexin Pei et al. “Can Large Language Models Reason about Program Invariants?” In:Proceedings of the 40th International Conference on Machine Learning. V ol. 202. Proceedings of Machine Learning Research. PMLR, 23– 29 Jul 2023, pp. 27496–27520

    2023

  20. [28]

    A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

    Al Bessey et al. “A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World”. In:Communications of the ACM53.2 (2010), pp. 66–75

    2010

  21. [29]

    KLEE: Unassisted and Automatic Generation of High- Coverage Tests for Complex Systems Programs

    Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. “KLEE: Unassisted and Automatic Generation of High- Coverage Tests for Complex Systems Programs”. In:8th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2008, pp. 209–224

    2008

  22. [30]

    Dafny: An Automatic Program Verifier for Functional Correctness

    K. Rustan M. Leino. “Dafny: An Automatic Program Verifier for Functional Correctness”. In:16th Interna- tional Conference on Logic for Programming, Artificial Intelligence, and Reasoning (LPAR). Springer, 2010, pp. 348–370

    2010

  23. [31]

    QuickCheck: A Lightweight Tool for Random Testing of Haskell Pro- grams

    Koen Claessen and John Hughes. “QuickCheck: A Lightweight Tool for Random Testing of Haskell Pro- grams”. In:5th ACM SIGPLAN International Confer- ence on Functional Programming (ICFP). ACM, 2000, pp. 268–279

    2000

  24. [32]

    Coverage Guided, Property Based Testing

    Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. “Coverage Guided, Property Based Testing”. In:Proceedings of the ACM on Programming Languages (OOPSLA)3 (2019), pp. 1–29

    2019

  25. [33]

    https://github.com/ llvm/llvm-project, accessed 2025-10-12

    The LLVM Compiler Infrastructure. https://github.com/ llvm/llvm-project, accessed 2025-10-12

    2025

  26. [34]

    Dynamically discovering likely program invariants to support program evolution

    M.D. Ernst et al. “Dynamically discovering likely program invariants to support program evolution”. In: IEEE Transactions on Software Engineering27.2 (2001), pp. 99–123

    2001

  27. [35]

    LLVM: A Com- pilation Framework for Lifelong Program Analysis & Transformation

    Chris Lattner and Vikram Adve. “LLVM: A Com- pilation Framework for Lifelong Program Analysis & Transformation”. In:International Symposium on Code Generation and Optimization (CGO). IEEE, 2004, pp. 75–86

    2004

  28. [36]

    Notes on Structured Programming

    Edsger W. Dijkstra. “Notes on Structured Programming”. In:Structured Programming. Academic Press, 1972, pp. 1–82

    1972

  29. [37]

    Stepwise API usage assistance using n-gram language models

    André L. Santos et al. “Stepwise API usage assistance using n-gram language models”. In:Journal of Systems and Software131 (2017), pp. 461–474

    2017

  30. [38]

    Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM

    Gabriel Ryan et al. “Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM”. In:Proc. ACM Softw. Eng.1.FSE (July 2024)

    2024

  31. [39]

    Better Zero-Shot Reasoning with Self-Adaptive Prompting

    Xingchen Wan et al. “Better Zero-Shot Reasoning with Self-Adaptive Prompting”. In:Findings of the Associa- tion for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, July 2023, pp. 3493–3514

    2023

  32. [40]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn et al. “Reflexion: language agents with verbal reinforcement learning”. In:Thirty-seventh Con- ference on Neural Information Processing Systems. 2023

    2023

  33. [41]

    Jicheng Wang, Yifeng He, and Hao Chen.RepoGen- Reflex: Enhancing Repository-Level Code Completion with Verbal Reinforcement and Retrieval-Augmented Generation. 2024. arXiv: 2409.13122[cs.SE]

    2024 arXiv

  34. [42]

    Hopper: Interpretative Fuzzing for Libraries

    Peng Chen et al. “Hopper: Interpretative Fuzzing for Libraries”. In:Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Secu- rity. CCS ’23. Copenhagen, Denmark: Association for Computing Machinery, 2023, pp. 1600–1614

    2023

  35. [43]

    Quickly detecting relevant program invariants

    Michael D. Ernst et al. “Quickly detecting relevant program invariants”. In:Proceedings of the 22nd Inter- national Conference on Software Engineering. ICSE ’00. Limerick, Ireland: Association for Computing Machinery, 2000, pp. 449–458

    2000

  36. [44]

    DocTer: documentation-guided fuzzing for testing deep learning API functions

    Danning Xie et al. “DocTer: documentation-guided fuzzing for testing deep learning API functions”. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ISSTA

  37. [45]

    Lost in the Middle: How Language Models Use Long Contexts

    Nelson F. Liu et al. “Lost in the Middle: How Language Models Use Long Contexts”. In:Transactions of the Association for Computational Linguistics12 (2024), pp. 157–173

    2024

  38. [46]

    FuzzBench: an open fuzzer benchmarking platform and service

    Jonathan Metzman et al. “FuzzBench: an open fuzzer benchmarking platform and service”. In:Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Founda- tions of Software Engineering. ESEC/FSE 2021. Athens, Greece: Associatio...

    2021

  39. [47]

    SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering

    John Yang et al. “SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering”. In: Advances in Neural Information Processing Systems (NeurIPS). 2024

    2024

  40. [48]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents

    Xingyao Wang et al. “OpenHands: An Open Platform for AI Software Developers as Generalist Agents”. In: The Thirteenth International Conference on Learning Representations (ICLR). 2025

    2025

  41. [49]

    Test Intention Guided LLM-based Unit Test Generation

    Zifan Nan et al. “Test Intention Guided LLM-based Unit Test Generation”. In:Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). 2025

    2025

  42. [50]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E. Jimenez et al. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” In:The Twelfth International Conference on Learning Represen- tations (ICLR). 2024

    2024

  43. [51]

    Magma: A Ground-Truth Fuzzing Benchmark

    Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. “Magma: A Ground-Truth Fuzzing Benchmark”. In: vol. 4. 3. New York, NY , USA: Association for Com- puting Machinery, Nov. 2020

    2020

  44. [52]

    https://zlib.net/

    Jean-loup Gailly and Mark Adler.ZLIB DATA COM- PRESSION LIBRARY. https://zlib.net/. 2025

    2025

  45. [53]

    https : / / github

    PNG Development Group.LIBPNG: Portable Network Graphics Reference Library. https : / / github . com / pnggroup/libpng. 2025

    2025

  46. [54]

    https://github.com/the-tcpdump-group/libpcap

    The Tcpdump Group.libpcap: Portable Packet Capture Library. https://github.com/the-tcpdump-group/libpcap. 2025

    2025

  47. [55]

    https://github.com/ sqlite/sqlite

    SQLite Development Team.SQLite. https://github.com/ sqlite/sqlite. 2025

    2025

  48. [56]

    https://github.com/mm2/Little-CMS

    Marti Maria.Little CMS: A Free Color Management Engine. https://github.com/mm2/Little-CMS. 2025

    2025

  49. [57]

    https://github.com/DaveGamble/cJSON

    Dave Gamble.cJSON: Ultralightweight JSON Parser in ANSI C. https://github.com/DaveGamble/cJSON. 2025

    2025

  50. [58]

    https://github.com/google/re2

    Google team.RE2: Efficient Regular Expression Match- ing. https://github.com/google/re2. 2025

    2025

  51. [59]

    An Empirical Evaluation of Using Large Language Models for Automated Unit Test Gener- ation

    Max Schäfer et al. “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Gener- ation”. In:IEEE Transactions on Software Engineering 50.1 (2024), pp. 85–105

    2024

  52. [60]

    Runlin Liu et al.LLM-based Unit Test Generation for Dynamically-Typed Programs. 2025. arXiv: 2503.14000 [cs.SE]

    2025

  53. [61]

    Is mutation an appropriate tool for testing experiments?

    J. H. Andrews, L. C. Briand, and Y . Labiche. “Is mutation an appropriate tool for testing experiments?” In:Proceedings of the 27th International Conference on Software Engineering. ICSE ’05. St. Louis, MO, USA: Association for Computing Machinery, 2005, pp. 402–411

    2005

  54. [62]

    Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empiri- cal Study on the Relationship Between Mutants and Real Faults

    Mike Papadakis et al. “Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empiri- cal Study on the Relationship Between Mutants and Real Faults”. In:2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 2018, pp. 537–548

    2018

  55. [63]

    LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing

    Hongxiang Zhang et al. “LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing”. In: ACM/IEEE International Conference on Automation of Software Test (AST). 2026

    2026

  56. [64]

    On learning meaningful assert statements for unit test cases

    Cody Watson et al. “On learning meaningful assert statements for unit test cases”. In:Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ICSE ’20. Seoul, South Korea: Association for Computing Machinery, 2020, pp. 1398–1409

    2020

  57. [65]

    TOGA: A Neural Method for Test Oracle Generation

    Elizabeth Dinella et al. “TOGA: A Neural Method for Test Oracle Generation”. In:Proceedings of the 44th International Conference on Software Engineering. ICSE ’22. Pittsburgh, Pennsylvania: Association for Computing Machinery, 2022, pp. 2130–2141

    2022

  58. [66]

    TestEval: Benchmarking Large Language Models for Test Case Generation

    Wenhan Wang et al. “TestEval: Benchmarking Large Language Models for Test Case Generation”. In:Find- ings of the Association for Computational Linguistics: NAACL 2025. Albuquerque, New Mexico: Association for Computational Linguistics, Apr. 2025, pp. 3547– 3562

    2025

  59. [67]

    SWT-Bench: Testing and Vali- dating Real-World Bug-Fixes with Code Agents

    Niels Mündler et al. “SWT-Bench: Testing and Vali- dating Real-World Bug-Fixes with Code Agents”. In: Advances in Neural Information Processing Systems. V ol. 37. Curran Associates, Inc., 2024, pp. 81857–81887

    2024

  60. [68]

    Code2Inv: A Deep Learning Frame- work for Program Verification

    Xujie Si et al. “Code2Inv: A Deep Learning Frame- work for Program Verification”. In:Computer Aided Verification: 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21–24, 2020, Proceedings, Part II. Los Angeles, CA, USA: Springer-Verlag, 2020, pp. 151–164

    2020

  61. [69]

    Evaluating Program Semantics Rea- soning with Type Inference in System $F$

    Yifeng He et al. “Evaluating Program Semantics Rea- soning with Type Inference in System $F$”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2025

    2025

  62. [70]

    int test_{project}_api_sequence()

    Ruogu Yang et al.Lisa Artifacts. Zenodo, https://doi. org/10.5281/zenodo.20839021. Jan. 2026. APPENDIXA PROMPTTEMPLATES Your task is to write a complete, logically correct C++ function named “int test_{project}_api_sequence()” using the {project} library. The API sequence shou...

    2026 doi

  63. [2022]

    Virtual, South Korea: Association for Computing Machinery, 2022, pp. 176–188

    2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.