Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Evaluating LLM-Based Regression Test Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that an LLM prompted only with a commit message and a code diff can generate regression tests that find and reproduce real bugs in minutes, and that those same tests, used as fuzz seeds, can beat a state-of-the-art…

desk verdict A useful, narrowly scoped empirical study whose abstract overstates the body; the ClevFuzz-vs-WAFLGo claim is one bug away from vanishing. read the letter →

arxiv 2501.11086 v2 pith:676HFBBX submitted 2025-01-19 cs.SE cs.AI

Jing Liu , Seongmin Lee , Eleonora Losiouk , Marcel Böhme This is my paper · ORCID
classification cs.SEcs.AI
keywords regressiontestgenerationlargelanguagemodelszero-shotfeedback-directedpromptinggreyboxfuzzingseedstructuredinputscommittesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large language model, prompted only with a commit message and a code diff, can generate regression tests for programs whose inputs are highly structured and human-readable, such as XML files, JavaScript programs, and PDF documents. The practical stake is CI/CD: instead of running a 24-hour directed fuzzing campaign that needs a corpus of valid seed inputs, a pipeline could ask an LLM for candidate tests in minutes. The paper claims that this zero-shot feedback loop, Cleverest, finds as many bugs as the state-of-the-art directed fuzzer WAFLGo in a fraction of the time, and that when the generated tests are handed to a generic fuzzer as seeds (ClevFuzz), the zero-shot pipeline finds more bugs than WAFLGo in roughly half the wall-clock time. A secondary claim is that the expressiveness of the commit message is a decisive input: vague messages cut effectiveness sharply, while adding about 17 informative words restored it.

What carries the argument

The load-bearing mechanism is the feedback-directed generation loop. A prompt synthesizer turns the task, commit information, and prior attempt history into a prompt; the LLM module returns an input, and optionally the command line; the execution analyzer compiles the program with a sanitizer, runs the input on the before and after versions, and classifies the result using the RIPR (Reaching, Infecting, Propagating, Revealing) model: whether the input reaches the changed lines, changes observable output or return code, or triggers the sanitizer. That classification is appended to the next prompt, so the model can repair near-miss inputs. The second mechanism is integration: a reaching or output-changing Cleverest input becomes the sole seed for a vanilla coverage-guided fuzzer, which mutates it into a bug-triggering input.

What would settle it

Take twenty fresh bug-introducing and bug-fixing commits from several other structured-input programs, such as a YAML parser, a JSON schema validator, a SQL engine, and a template engine, all with post-2024 fixes so the LLM cannot have memorized the answers, and run Cleverest's default five-iteration loop followed by one 24-hour AFL++ campaign per reaching input. If the combined ClevFuzz pipeline finds fewer bugs than a directed fuzzer that starts from a curated seed corpus, or if Cleverest alone finds none, the claimed advantage of zero-shot LLM regression test generation for this class of programs is refuted.

Watch

Extended reading notes

Core claim

The discovery is that regression test generation for structured-input programs can be treated as a machine-translation problem and solved zero-shot. Cleverest's prompt carries only the commit message, the diff, and the input format name; the model emits a candidate input in that format; the execution analyzer runs it on the program before and after the commit, classifies the outcome as bug triggered, output changed, commit reached, or none, and feeds that classification back into the next prompt. On 22 commits spanning Mujs, Libxml2, and Poppler, the loop found bugs in 3 of 11 bug-introducing commits and reproduced bugs in 4 of 11 bug-fixing commits in under three minutes per commit. That matches WAFLGo's reproduction count and trails its finding count by two, but WAFLGo started from a seed corpus that already revealed the bug in four cases. Used as seeds for a vanilla AFL++, the Cleverest-generated inputs (the ClevFuzz configuration) found 5 of 11 bugs in both scenarios, beating WAFLGo, while taking about 6 hours to WAFLGo's 13 to 15.

Load-bearing premise

The load-bearing assumption is that the 11 bugs taken from the reference benchmark represent the broader class of regression bugs in programs with structured human-readable inputs; if those bugs are unusually reachable or their commit messages unusually informative, the reported effectiveness and the ClevFuzz advantage will not transfer.

Editorial extensions

If this is right

  • A CI/CD pipeline can use an LLM call of a few minutes as a first regression-test pass for structured-input programs, reserving long fuzzing campaigns for cases the LLM does not settle.
  • Tests that only reach the changed code or change its output are still assets: they are human-readable starting points for developers and seeds that a fuzzer can finish.
  • Commit-message quality is a measurable lever: a vague message degrades automated test generation, and a short, informative rewrite significantly increases the number of bugs found.
  • The zero-shot ClevFuzz recipe, LLM-generated seeds plus a stock coverage-guided fuzzer, can outperform a few-shot directed fuzzer that starts from a curated seed corpus, in both bugs found and total time.
  • The approach is bounded by input readability: compact formats like PDF resist direct generation, though the near-miss inputs stay close enough to be repaired.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the seed-generator result is likely the more durable contribution, because using LLM output to bootstrap coverage-guided fuzzing is a recipe that should transfer to other structured-input domains as long as the format's validity constraints can be represented in the prompt.
  • Editorial inference: the sensitivity to commit-message expressiveness suggests a cheap experiment, rewording vague commit messages neutrally and re-running the pipeline; if effectiveness tracks specificity, commit-message quality becomes a low-cost lever for both automated testing and human maintenance.
  • Editorial inference: the PDF failures point to a natural repair loop, feeding parse-error location back to the model, which the paper's own error analysis shows would have been enough to turn near-miss PDFs into reaching inputs.
  • Editorial inference: the zero-shot claim would be stronger with a post-cutoff bug set, since the paper's similarity check against available bug reports is suggestive but memorization can be ruled out decisively only on bugs fixed after the model's training data ends.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Cleverest, a zero-shot LLM-based regression test generator that iteratively prompts GPT-4o with a commit message, a code diff, and execution feedback to produce system-level inputs (JavaScript, XML, PDF) for two tasks: finding bugs introduced by a commit and reproducing bugs fixed by a commit. The evaluation covers 22 commits (11 bugs, both bug-introducing and bug-fixing commits) from three programs selected from the WAFLGo benchmark, with five repetitions per configuration, using sanitizers, output and return-code comparison, and GCOV coverage to classify outcomes. The paper also evaluates ClevFuzz, which feeds Cleverest-generated inputs to AFL++, and makes three main claims: Cleverest is fast (about two minutes) and effective on human-readable formats but weak on PDF; commit-message expressiveness and execution feedback matter; and ClevFuzz, as a zero-shot pipeline, outperforms WAFLGo, whose initial seeds are shown to be close to bug-revealing. The body is framed as a case study and ships a replication package, but the abstract claims a much larger evaluation on 72 commits and eight programs and describes a commit-message-rewriting experiment that does not appear in the body.

Significance. If the central comparative claim held, the paper would make a practical contribution: LLM-generated inputs as zero-shot seeds for directed greybox fuzzing of structured-input parsers and interpreters within CI/CD time budgets. The evaluation has real strengths: it executes all generated inputs under sanitizers rather than relying on LLM self-scoring; the Levenshtein-based leakage check is a good-faith memorization probe; Section 5.2's analysis of how close WAFLGo's seeds are to bug-revealing is an honest and useful fairness check; and the discovery of a bug fixed after the model's cut-off date is a persuasive non-memorization data point. However, the head-to-head advantage over WAFLGo rests on aggregate counts of 5/11 vs 5/11 and 5/11 vs 4/11 over 11 bugs with no significance testing, and the abstract overstates the evaluated scope. The significance is therefore that of a suggestive, well-executed case study whose specific superiority claims are not yet established.

major comments (4)
  1. [Abstract; §2.3, Table 1; Table 4] The abstract does not describe the study reported in the body. It says 'Testing 72 commits to Mujs, Libxml2, Poppler, JerryScript, Z3, PHP, JQ, and MicroPython', but the body evaluates 22 commits on only Mujs, Libxml2, and Poppler, and the other five programs never appear in the experiments. It says 'Cleverest found as many bugs as ... WAFLGo did in 24 hours' in under 2 minutes on average, but Table 4 reports Cleverest finding bugs in 3/11 bug-introducing commits versus WAFLGo's 5/11, with equality holding only in the bug-reproduction scenario (4/11 vs 4/11). It says fuzzing 'doubles the number of bugs found', but Table 4 shows increases from 3/11 to 5/11 and from 4/11 to 5/11. Finally, the abstract describes a commit-message-rewriting experiment ('adding 17 words on average (max. 43)') that does not appear anywhere in the body. These discrepancies must be resolved: either the abstract should be rewritten to match the 22-commit, three-program evaluation actually reported, or the additional programs and experiments claimed in the abstract must be added to the body.
  2. [§5.1, Table 4; §5.2 and RQ3 summary] The load-bearing claim that the zero-shot ClevFuzz 'outperforms' WAFLGo is not supported by the reported data. The aggregate counts are 5/11 vs 5/11 in bug finding (a tie) and 5/11 vs 4/11 in bug reproduction (a one-bug difference), and the per-bug counts driving the difference are at 4/5 (#141 and #1282 in bug finding; #65, #141, and #145 in bug reproduction), so a single trial flip reverses or erases the advantage. No significance test, confidence interval, effect size, or paired per-bug comparison is reported despite five repetitions per bug per scenario, and the paper uses 'significant' loosely elsewhere (Section 4: 'model size has a significant impact on the performance of Cleverest'). I ask for a paired analysis (e.g., McNemar's test or a bootstrap over per-bug success counts) or, failing significance, a calibrated statement such as 'comparable on this benchmark'.
  3. [§5.1, Table 4, footnote 6] The aggregate time claims are not reproducible from the table. The ClevFuzz total is stated as 'roughly 6 hours ... including the time-out of the fuzzing campaign, which is 24 hours', yet the table lists several T.O. entries (e.g., #166 and #550 in bug finding, and over half the Poppler rows in both scenarios); if any of those campaigns ran to the 24-hour timeout, the total can only be six hours if unsuccessful campaigns are excluded. Similarly, the WAFLGo aggregate times 13:16:04 and 15:18:03 are hard to reconcile with per-bug entries that include 24-hour timeouts. The authors should define the T.O. semantics, report total wall-clock budgets including timeouted and failed trials, and state whether the aggregates are sums, means, or medians.
  4. [§2.3, Table 1; §6; §8] The evidence base is 11 memory-related bugs from the WAFLGo benchmark across three C programs, and Section 6 properly frames the study as a case study. However, the abstract and Section 8 ('we can safely recommend the use of LLMs') generalize well beyond this base, and the abstract's eight-program claim is backed by no experiment in the body. I ask the authors to either add the additional programs and commits claimed in the abstract or consistently restrict the conclusions to the three evaluated programs, and to add a sensitivity discussion of how commit-message expressiveness (shown in Section 4 to be influential, e.g., bug-reproduction effectiveness dropping to 0.66 with only the message) may moderate the results on other benchmarks.
minor comments (5)
  1. [§5.2, Listing 1] The diff in Listing 1 shows 'if(a =Error== b)', which is not valid JavaScript and appears to be a garbled token; please verify the listing against the actual commit.
  2. [Table 3] The slider-based cells make Table 3 very difficult to read in print; consider printing the numeric average score for each cell.
  3. [Table 4 caption] State explicitly that the ClevFuzz 'Bug' column counts only the trials that were fuzzed (e.g., 1/1 for #145 in bug finding, 2/3 for #65 in bug reproduction) and that the 'Bug all' column includes Cleverest's direct hits.
  4. [§6, Construct Validity] The Levenshtein check compares generated inputs only with the bug-triggering test case from the bug report; please state the exact comparison corpus and discuss the maximum-40% similarity case explicitly, as it is a potential memorization candidate.
  5. [Abstract; §5.1] The framing 'WAFLGo did in 24 hours' is misleading because Section 5.1 reports that WAFLGo found most bugs within 5-15 minutes; the time comparison in the abstract should use actual time-to-exposure. Equivalent wording is also used in the RQ3 summary.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: effectiveness is measured by external execution, not by LLM self-assessment or by fitting.

full rationale

The paper's central evaluation is self-contained against external execution oracles. Cleverest's generated inputs are executed on real program versions before and after each commit, and outcomes are classified by sanitizer reports, output/return-code differences, and GCOV coverage (Sections 2.2.3 and 3). The RQ3 comparison to WAFLGo is an external benchmark comparison; the ClevFuzz extension runs a vanilla AFL++ on Cleverest-generated seeds and compares bug counts to WAFLGo (Table 4). No parameter is fitted to the target metric, and the LLM never scores its own output; the only self-referential check is the Levenshtein-based leakage analysis (Section 6), which tests memorization rather than defining effectiveness. The cited prior work by the authors ([4], [38]) is used to contextualize WAFLGo's predecessor fuzzers, not as an assumption from which the results are derived. The aggregate one-bug margin in Table 4 is a statistical-robustness concern, not a circularity concern. Hence no circular step is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small benchmark chosen by hand (11 bugs, 22 commits), on proprietary LLM behavior, and on an oracle definition. No new physical or formal entities are introduced. The hyperparameters are hand-chosen budgets, and several are ablated, so they are costs rather than fitted parameters.

free parameters (6)
  • LLM temperature = 0.5 default; 1.0 in ablation
    Hand-chosen; the ablation finds no significant effect of temperature.
  • number of feedback iterations = 5 default; 10 in ablation
    Hand-chosen budget; 10 iterations slightly improve bug reproduction at double cost.
  • max tokens for LLM output = 4096
    Set to balance performance and cost; not swept.
  • per-generation timeout = 30 seconds
    Hand-chosen; affects total runtime.
  • experiment repetitions = 5
    Small budget for binary outcomes; raw counts without confidence intervals.
  • language model choice = GPT-4o default; GPT-4o mini ablation
    Model size strongly affects effectiveness; the choice of GPT-4o is an input assumption, not derived.
assumptions (5)
  • domain assumption The 11 bugs from the WAFLGo benchmark represent regression bugs in programs with human-readable structured inputs.
    Section 2.3 selects all applicable programs from WAFLGo; the general recommendation depends on this representativeness.
  • domain assumption AddressSanitizer and output/return-code comparison are sufficient oracles for 'bug revealed' and 'behavior changed'.
    Section 2.2.3; standard in fuzzing, but an assumption about oracle completeness for memory bugs.
  • domain assumption The commit message and diff give the LLM enough context to infer the changed feature.
    Core premise of Cleverest; the paper shows effectiveness drops when messages are vague (Section 4, Issue #145), so this assumption is not always met.
  • domain assumption Levenshtein ratio to public bug-triggering inputs is a valid check against training-data memorization.
    Section 6 Construct Validity; this only compares against known bug report tests, not the full training corpus.
  • domain assumption The BIC/BFC commit labels in the WAFLGo dataset are correct.
    Section 2.3 relies on the dataset's commit identification; an error here would mislabel bug-finding vs bug-reproduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating LLM-Based Regression Test Generation." pith.science (2026). https://pith.science/paper/676HFBBX

@misc{pith2026250111086,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLM-Based Regression Test Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/676HFBBX}},
  note         = {Machine review of arXiv:2501.11086}
}
read the original abstract

Large Language Models (LLMs) have shown tremendous promise in automated software engineering. In this paper, we investigate LLMs for just-in-time regression test generation for programs, like parsers, interpreters, or compilers, that take highly structured, human-readable inputs. When a bug fix or code change is committed, the repository (as part of CI/CD) runs an LLM for a few minutes to generate regression tests that exercise the changed code and potentially trigger bugs. We frame LLM-based regression test generation as a machine translation task that takes the developer-provided commit message, the code change, and the input format name (e.g., XML), and produces regression tests for the described change in that format. Testing 72 commits to Mujs, Libxml2, Poppler, JerryScript, Z3, PHP, JQ, and MicroPython, our feedback-directed, zero-shot prototype Cleverest performed well, even without the code change. In under 2 minutes on average, Cleverest found as many bugs as the state-of-the-art directed greybox fuzzer WAFLGo did in 24 hours, even though WAFLGo started with a commit-reaching seed corpus in most cases. Using the Cleverest-generated tests as a seed corpus in coverage-guided greybox fuzzing doubles the number of bugs found; we call this integration ClevFuzz. We also find that some commit messages are more expressive than others, thus we ask how it impacts Cleverest's effectiveness. Cleverest picks up on the change intention: e.g., given a commit message that a patch changes how floating point variables are treated in the Mujs JavaScript interpreter, Cleverest generates JavaScript programs that contain floating point variables. Minimally changing commit messages to reduce or increase their information substantially impacts effectiveness: adding 17 words on average (max. 43) to make ineffective commit messages more expressive significantly increased the number of bugs found.

Figures

Figures reproduced from arXiv: 2501.11086 by the authors.

Figure 1
Figure 1. Cleverest: Our zero-shot feedback-guided LLM-based regression test generation methodology. programs that take highly structured, human-readable inputs, like JavaScript programs or XML files. Given only the code changes and/or a description of these changes (i.e., a commit or pull request), we want to find out how well an LLM performs in generating test cases that reveal bugs that might have been introduced or patche… view at source ↗
Figure 2
Figure 2. The prompt generated by the prompt synthesizer for the LLM to generate the test input for Mujs. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The result of the ClevFuzz. (a) The outer circle shows the number of cases where the fuzzer finds the bug-triggering input for the regression test generation scenario: ‘✓’ indicates the fuzzer finds the bug￾triggering input and ‘-’ indicates otherwise. The inner circle shows the original result the Cleverest-generated input: ‘➣’ indicates the input reaches the commit and ‘✐’ indicates the input changes the program o… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PROGRESS: Property-Guided Regression Search for Semantic Falsification

    cs.SE 2026-07 conditional novelty 7.0 of 10

    PROGRESS embeds LLM-generated jqwik properties as search objectives in EvoSuite's DynaMOSA, detecting 58% of injected Java bugs that regression assertions miss and satisfying hard preconditions 4x more often than stan...

  2. Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Ammann and J

    P. Ammann and J. Offutt. 2016. Introduction to Software Testing. Cambridge University Press. https://books.google.de/ books?id=58LeDQAAQBAJ

    work page 2016

  2. [2]

    Max Bachmann. 2024. Levenshtein.ratio. https://rapidfuzz.github.io/Levenshtein/levenshtein.html#ratio

    work page 2024

  3. [3]

    Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code. arXiv:2206.01335 [cs.SE] https://arxiv.org/abs/2206.01335

    arXiv 2022

  4. [4]

    Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed Greybox Fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, New York, NY, USA, 2329–2344. https://doi.org/10.1145/3133956.3134020

    arXiv 2017

  5. [5]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. arXiv:2207.10397 [cs.CL] https://arxiv.org/abs/2207.10397

    arXiv 2022

  6. [6]

    Zhengjie Du, Yuekang Li, Yang Liu, and Bing Mao. 2022. WindRanger: a directed greybox fuzzer driven by deviation basic blocks. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2440–2451. https://doi.org/10.1145/3510003.3510197

    arXiv 2022

  7. [7]

    Michael Fu, Chakkrit Tantithamthavorn, Van Nguyen, and Trung Le. 2023. ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We? arXiv:2310.09810 [cs.SE] https://arxiv.org/abs/2310.09810

    work page Pith review arXiv 2023

  8. [8]

    Subbarao Kambhampati. 2024. Can large language models reason and plan? Annals of the New York Academy of Sciences 1534, 1 (2024), 15–18. https://doi.org/10.1111/nyas.15125

    doi:10.1111/nyas.15125 2024

Show all 39 references
  1. [9]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . 2312–2323. https://doi.org/10.1109/ICSE48619.2023.00194

    2023

  2. [10]

    Tomasz Kuchta, Hristina Palikareva, and Cristian Cadar. 2018. Shadow Symbolic Execution for Testing Software Patches. ACM Trans. Softw. Eng. Methodol. 27, 3, Article 10 (sep 2018), 32 pages. https://doi.org/10.1145/3208952

    2018 doi

  3. [11]

    Shuvendu K. Lahiri, Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madanlal Musuvathi, Piali Choudhury, Curtis von Veh, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2023. Interactive Code Generation via Test-Driven User-Intent Formalization. arXiv...

    2023 arXiv

  4. [12]

    Nan Li and Jeff Offutt. 2017. Test Oracle Strategies for Model-Based Testing. IEEE Transactions on Software Engineering 43, 4 (2017), 372–395. https://doi.org/10.1109/TSE.2016.2597136

    2017

  5. [13]

    Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, Peng Cheng, Kangjie Lu, and Ting Wang. 2021. UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers. In 30th USENIX Security Symposium...

    2021

  6. [14]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. arXiv:2409.02977 [cs.SE] https://arxiv.org/abs/ 2409.02977

    2024 arXiv

  7. [15]

    Changhua Luo, Wei Meng, and Penghui Li. 2023. SelectFuzz: Efficient Directed Fuzzing with Selective Path Exploration. In 2023 IEEE Symposium on Security and Privacy (SP) . 2693–2707. https://doi.org/10.1109/SP46215.2023.10179296

    2023

  8. [16]

    Yunlong Lyu, Yuxuan Xie, Peng Chen, and Hao Chen. 2023. Prompt Fuzzing for Fuzz Driver Generation. arXiv:2312.17677 [cs.CR]

    2023 arXiv

  9. [17]

    Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large Language Model guided Protocol Fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS)

    2024

  10. [18]

    Christian Munley, Aaron Jarmusch, and Sunita Chandrasekaran. 2024. LLM4VV: Developing LLM-driven testsuite for compiler validation. Future Generation Computer Systems 160 (2024), 1–13. https://doi.org/10.1016/j.future.2024.05.034

    2024 doi

  11. [19]

    Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng Cai. 2024. Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities. arXiv:2402.17230 [cs.CR] https://arxiv.org/abs/2402.17230

    2024 arXiv

  12. [20]

    Suzette Person, Guowei Yang, Neha Rungta, and Sarfraz Khurshid. 2011. Directed incremental symbolic execution. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (San Jose, California, USA) (PLDI ’11) . Association for Computing...

    2011

  13. [21]

    Juan Altmayer Pizzorno and Emery D. Berger. 2024. CoverUp: Coverage-Guided LLM-Based Test Generation. arXiv:2403.16218 [cs.SE] https://arxiv.org/abs/2403.16218

    2024 arXiv

  14. [22]

    Mohamadreza Rostami, Marco Chilese, Shaza Zeitouni, Rahul Kande, Jeyavijayan Rajendran, and Ahmad-Reza Sadeghi

  15. [23]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. , Vol. 1, No. 1, Article . Publication date: January 2025...

    2024 doi

  16. [24]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85–105. https: //doi.org/10.1109/TSE.2023.3334955

    2024

  17. [25]

    Chaofan Shou, Jing Liu, Doudou Lu, and Koushik Sen. 2024. LLM4Fuzz: Guided Fuzzing of Smart Contracts with Large Language Models. arXiv:2401.11108 [cs.CR] https://arxiv.org/abs/2401.11108

    2024 arXiv

  18. [26]

    Barr, and Wei Le

    Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Earl T. Barr, and Wei Le. 2024. A Comprehensive Study of the Capabilities of Large Language Models for Vulnerability Detection. arXiv:2403.17218 [cs.SE] https://arxiv.org/abs/2403.17218

    2024 arXiv

  19. [27]

    Karl Tamberg and Hayretdin Bahsi. 2024. Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study. arXiv:2405.15614 [cs.CR] https://arxiv.org/abs/2405.15614

    2024 arXiv

  20. [28]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE] https://arxiv.org/abs/2009.05617

    2021 arXiv

  21. [29]

    Ullah, M

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini. 2024. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. In 2024 IEEE Symposium on Security and Privacy (SP) . IEEE Computer...

    2024

  22. [30]

    Zejun Wang, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. https://arxiv.org/pdf/2408.11324v1

    2024 arXiv

  23. [31]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE ’20) . ACM. https://doi.org/10.1145/3...

    2020

  24. [32]

    Chun Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2023. Fuzz4ALL: Universal Fuzzing with Large Language Models. 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2023), 1547–1559. https://api.semanticscholar.org/CorpusID:260735598

    2023

  25. [33]

    Yi Xiang, Xuhong Zhang, Peiyu Liu, Shouling Ji, Hong Liang, Jiacheng Xu, and Wenhai Wang. 2024. Critical Code Guided Directed Greybox Fuzzing for Commits. In 33rd USENIX Security Symposium (USENIX Security 24) . USENIX Association, Philadelphia, PA, 2459–2474. https://www.usen...

    2024

  26. [34]

    Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. 2024. Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. arXiv:2308.12697 [cs.SE] https://arxiv.org/abs/2308.12697

    2024 arXiv

  27. [35]

    Hongxiang Zhang, Yuyang Rong, Yifeng He, and Hao Chen. 2024. LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing. arXiv:2406.07714 [cs.CR] https://arxiv.org/abs/2406.07714

    2024

  28. [36]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. arXiv:2404.05427 [cs.SE]

    2024 arXiv

  29. [37]

    Han Zheng, Jiayuan Zhang, Yuhang Huang, Zezhong Ren, He Wang, Chunjie Cao, Yuqing Zhang, Flavio Toffalini, and Mathias Payer. 2023. FISHFUZZ: Catch Deeper Bugs by Throwing Larger Nets. In 32nd USENIX Security Sym- posium (USENIX Security 23) . USENIX Association, Anaheim, CA, ...

    2023

  30. [38]

    Xiaogang Zhu and Marcel Böhme. 2021. Regression Greybox Fuzzing. InProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (Virtual Event, Republic of Korea) (CCS ’21). Association for Computing Machinery, New York, NY, USA, 2169–2182. https://doi...

    2021

  31. [2024]

    In 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)

    Beyond Random Inputs: A Novel ML-Based Hardware Fuzzing. In 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE). 1–6. https://doi.org/10.23919/DATE58400.2024.10546625

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.