{"id":"3df713ef-f936-4056-8500-a0a0e97f39ae","arxiv_id":"2501.11086","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A feedback-driven LLM tool generates regression tests for commits in minutes and, when its outputs seed a fuzzer, finds more bugs than a 24-hour state-of-the-art fuzzer on the same benchmark.","lead":"This paper evaluates Cleverest, which uses a large language model with execution feedback to write regression tests for software commits, studying 22 commits to three programs. For JavaScript and XML inputs it finds or reproduces bugs in minutes, and its generated tests, when used as fuzz seeds, beat a state-of-the-art targeted fuzzer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed ClevFuzz advantage over WAFLGo rests on aggregate counts that differ by one bug (5/11 vs 5/11 and 5/11 vs 4/11) with no significance testing or confidence intervals; a single bug flip reverses the conclusion.","rationale":"Good-faith reading: the paper's engineering contribution is real — Cleverest is a feedback-directed LLM test generator with a replication package, real sanitizer execution, and a plausible zero-shot seed-generation story. The central claim, however, is the comparative one: that ClevFuzz outperforms WAFLGo. That claim requires the difference to be larger than the noise of GPT-4o sampling and AFL++ nondeterminism. The data as reported cannot establish this: the aggregate advantage is one bug, per-bug counts are 4/5 or 5/5, and no variance measure accompanies the headline counts. The reader's benchmark-representativeness concern is real but secondary; even granting that the 11 bugs are representative, the 'outperforms' wording overstates a 5-vs-4 count. The abstract additionally introduces a 72-commit/8-program experiment and a 'doubles' statement that are absent from the body, so the strongest version of the claim cannot be verified from the manuscript. I agree with the conditional verdict, but the condition should be explicit: either add a paired significance test or confidence interval, or soften the comparative conclusion to 'comparable.'","tokens_in":21486,"tokens_out":9136,"duration_ms":87434,"concrete_test":"Treat each of the 11 bugs as a paired observation and compute McNemar's exact test (or a sign test with continuity correction) for ClevFuzz vs WAFLGo in each scenario; additionally, bootstrap the per-repetition data (5 trials per bug) to obtain a 95% confidence interval for the aggregate bug-count difference. If the two-sided p-value exceeds 0.05 or the CI includes zero, the conclusion should be reported as 'comparable' rather than 'outperforms,' and Table 4 should include explicit per-bug denominators and full Cleverest+ClevFuzz time accounting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4 is the only evidence for the RQ3/ClevFuzz superiority claim, but the raw aggregates are 5/11 vs 5/11 (bug finding) and 5/11 vs 4/11 (bug reproduction). A one-bug change in either direction flips or erases the advantage. The per-bug columns show the fragility: in bug finding, #141 is 4/5 and #1282 is 4/5; if either of those five-repetition counts had been 3/5, ClevFuzz's aggregate would drop to 4/11, tying or losing to WAFLGo. No confidence interval, effect size, or paired test is reported, even though the paper uses language like 'significant' elsewhere. The same issue affects the abstract's claim that fuzzing 'doubles' the number of bugs: from Table 4 the improvements are 3/11 to 5/11 and 4/11 to 5/11, not a doubling. Because the entire 'zero-shot outperforms few-shot' conclusion rests on aggregate counts that are within stochastic noise on 11 bugs, the claim is not yet supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Cleverest, a zero-shot LLM-based regression test generator that iteratively prompts GPT-4o with a commit message, a code diff, and execution feedback to produce system-level inputs (JavaScript, XML, PDF) for two tasks: finding bugs introduced by a commit and reproducing bugs fixed by a commit. The evaluation covers 22 commits (11 bugs, both bug-introducing and bug-fixing commits) from three programs selected from the WAFLGo benchmark, with five repetitions per configuration, using sanitizers, output and return-code comparison, and GCOV coverage to classify outcomes. The paper also evaluates ClevFuzz, which feeds Cleverest-generated inputs to AFL++, and makes three main claims: Cleverest is fast (about two minutes) and effective on human-readable formats but weak on PDF; commit-message expressiveness and execution feedback matter; and ClevFuzz, as a zero-shot pipeline, outperforms WAFLGo, whose initial seeds are shown to be close to bug-revealing. The body is framed as a case study and ships a replication package, but the abstract claims a much larger evaluation on 72 commits and eight programs and describes a commit-message-rewriting experiment that does not appear in the body.","tokens_in":21619,"tokens_out":17053,"duration_ms":156534,"significance":"If the central comparative claim held, the paper would make a practical contribution: LLM-generated inputs as zero-shot seeds for directed greybox fuzzing of structured-input parsers and interpreters within CI/CD time budgets. The evaluation has real strengths: it executes all generated inputs under sanitizers rather than relying on LLM self-scoring; the Levenshtein-based leakage check is a good-faith memorization probe; Section 5.2's analysis of how close WAFLGo's seeds are to bug-revealing is an honest and useful fairness check; and the discovery of a bug fixed after the model's cut-off date is a persuasive non-memorization data point. However, the head-to-head advantage over WAFLGo rests on aggregate counts of 5/11 vs 5/11 and 5/11 vs 4/11 over 11 bugs with no significance testing, and the abstract overstates the evaluated scope. The significance is therefore that of a suggestive, well-executed case study whose specific superiority claims are not yet established.","major_comments":[{"comment":"The abstract does not describe the study reported in the body. It says 'Testing 72 commits to Mujs, Libxml2, Poppler, JerryScript, Z3, PHP, JQ, and MicroPython', but the body evaluates 22 commits on only Mujs, Libxml2, and Poppler, and the other five programs never appear in the experiments. It says 'Cleverest found as many bugs as ... WAFLGo did in 24 hours' in under 2 minutes on average, but Table 4 reports Cleverest finding bugs in 3/11 bug-introducing commits versus WAFLGo's 5/11, with equality holding only in the bug-reproduction scenario (4/11 vs 4/11). It says fuzzing 'doubles the number of bugs found', but Table 4 shows increases from 3/11 to 5/11 and from 4/11 to 5/11. Finally, the abstract describes a commit-message-rewriting experiment ('adding 17 words on average (max. 43)') that does not appear anywhere in the body. These discrepancies must be resolved: either the abstract should be rewritten to match the 22-commit, three-program evaluation actually reported, or the additional programs and experiments claimed in the abstract must be added to the body.","section":"Abstract; §2.3, Table 1; Table 4"},{"comment":"The load-bearing claim that the zero-shot ClevFuzz 'outperforms' WAFLGo is not supported by the reported data. The aggregate counts are 5/11 vs 5/11 in bug finding (a tie) and 5/11 vs 4/11 in bug reproduction (a one-bug difference), and the per-bug counts driving the difference are at 4/5 (#141 and #1282 in bug finding; #65, #141, and #145 in bug reproduction), so a single trial flip reverses or erases the advantage. No significance test, confidence interval, effect size, or paired per-bug comparison is reported despite five repetitions per bug per scenario, and the paper uses 'significant' loosely elsewhere (Section 4: 'model size has a significant impact on the performance of Cleverest'). I ask for a paired analysis (e.g., McNemar's test or a bootstrap over per-bug success counts) or, failing significance, a calibrated statement such as 'comparable on this benchmark'.","section":"§5.1, Table 4; §5.2 and RQ3 summary"},{"comment":"The aggregate time claims are not reproducible from the table. The ClevFuzz total is stated as 'roughly 6 hours ... including the time-out of the fuzzing campaign, which is 24 hours', yet the table lists several T.O. entries (e.g., #166 and #550 in bug finding, and over half the Poppler rows in both scenarios); if any of those campaigns ran to the 24-hour timeout, the total can only be six hours if unsuccessful campaigns are excluded. Similarly, the WAFLGo aggregate times 13:16:04 and 15:18:03 are hard to reconcile with per-bug entries that include 24-hour timeouts. The authors should define the T.O. semantics, report total wall-clock budgets including timeouted and failed trials, and state whether the aggregates are sums, means, or medians.","section":"§5.1, Table 4, footnote 6"},{"comment":"The evidence base is 11 memory-related bugs from the WAFLGo benchmark across three C programs, and Section 6 properly frames the study as a case study. However, the abstract and Section 8 ('we can safely recommend the use of LLMs') generalize well beyond this base, and the abstract's eight-program claim is backed by no experiment in the body. I ask the authors to either add the additional programs and commits claimed in the abstract or consistently restrict the conclusions to the three evaluated programs, and to add a sensitivity discussion of how commit-message expressiveness (shown in Section 4 to be influential, e.g., bug-reproduction effectiveness dropping to 0.66 with only the message) may moderate the results on other benchmarks.","section":"§2.3, Table 1; §6; §8"}],"minor_comments":[{"comment":"The diff in Listing 1 shows 'if(a =Error== b)', which is not valid JavaScript and appears to be a garbled token; please verify the listing against the actual commit.","section":"§5.2, Listing 1"},{"comment":"The slider-based cells make Table 3 very difficult to read in print; consider printing the numeric average score for each cell.","section":"Table 3"},{"comment":"State explicitly that the ClevFuzz 'Bug' column counts only the trials that were fuzzed (e.g., 1/1 for #145 in bug finding, 2/3 for #65 in bug reproduction) and that the 'Bug all' column includes Cleverest's direct hits.","section":"Table 4 caption"},{"comment":"The Levenshtein check compares generated inputs only with the bug-triggering test case from the bug report; please state the exact comparison corpus and discuss the maximum-40% similarity case explicitly, as it is a potential memorization candidate.","section":"§6, Construct Validity"},{"comment":"The framing 'WAFLGo did in 24 hours' is misleading because Section 5.1 reports that WAFLGo found most bugs within 5-15 minutes; the time comparison in the abstract should use actual time-to-exposure. Equivalent wording is also used in the RQ3 summary.","section":"Abstract; §5.1"}],"recommendation":"major_revision","confidential_remarks":"The abstract describes experiments (72 commits across eight programs; a commit-message-rewriting study) that do not exist in the body, and the 'doubles' claim in the abstract contradicts Table 4. Before sending the paper to a second round of review, I recommend that the editor ask the authors to confirm against the replication package which experiments were actually executed: if the 72-commit evaluation exists, it must be added to the body with full tables; otherwise the abstract must be corrected. I also recommend insisting on a paired significance analysis for the ClevFuzz-versus-WAFLGo comparison, since the current one-bug margin is within the noise of five repetitions on 11 bugs. The body of the paper is otherwise honest and well-executed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the body, not the abstract. The abstract describes 72 commits to 8 programs and a commit-message rewriting experiment that do not appear in the paper. The body is a much narrower, honestly executed study on 22 commits to three programs, and that study is worth taking seriously.\n\nWhat is actually new: Cleverest is a feedback-directed, zero-shot LLM pipeline that generates regression tests from a commit diff and message, and the paper shows those generated tests can serve as seeds for vanilla AFL++ and find bugs that Cleverest alone misses. That seed-generation result is real and practically interesting. The methodology is sound where it counts: generated inputs are actually executed, bugs are confirmed with sanitizers, outcomes are classified using the RIPR model, and each experiment is repeated five times with a replication package supplied. The Levenshtein-based leakage check is weak but at least an attempt, and the authors do not hide the fact that WAFLGo's initial seeds were often already close to bug-revealing.\n\nThe soft spots are real, and the largest one is the abstract itself. The 72-commit/8-program claims and the commit-message augmentation experiment are simply absent from the body. That is a mechanical integrity problem, not just a style issue; the paper cannot be accepted as-is until abstract and body agree. The 'doubles the number of bugs' claim is also not supported by Table 4, where the improvements are 3/11 to 5/11 and 4/11 to 5/11.\n\nThe main empirical claim—ClevFuzz outperforming WAFLGo—is fragile. The aggregate difference is 5/11 vs 5/11 in bug finding and 5/11 vs 4/11 in bug reproduction. One bug flip erases the result. There are no confidence intervals, effect sizes, or paired tests, even though the word 'significant' appears in the text. On 11 bugs with five repetitions, this is within stochastic noise. The authors should either provide proper statistical analysis or soften the claim.\n\nMinor but worth noting: the ablation table (Table 3) is not legible—only averaged sliders, no per-repetition counts—and the benchmark is small and inherited from WAFLGo, so external validity is limited even for structured-input programs. The discussion sections acknowledge this, which is fair.\n\nWho is this for? Researchers and practitioners working on LLM-based testing, especially those interested in using LLMs as fuzz seed generators. The paper deserves a serious referee, but with the expectation of major revision: align the abstract, add statistical grounding, and make the ablation table readable.","headline":"A useful, narrowly scoped empirical study whose abstract overstates the body; the ClevFuzz-vs-WAFLGo claim is one bug away from vanishing.","tokens_in":22288,"tokens_out":1570,"would_cite":true,"duration_ms":18393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that an LLM prompted only with a commit message and a code diff can generate regression tests that find and reproduce real bugs in minutes, and that those same tests, used as fuzz seeds, can beat a state-of-the-art…","keywords":["regression test generation","large language models","zero-shot","feedback-directed prompting","greybox fuzzing","seed generation","structured inputs","commit testing"],"falsifier":"Take twenty fresh bug-introducing and bug-fixing commits from several other structured-input programs, such as a YAML parser, a JSON schema validator, a SQL engine, and a template engine, all with post-2024 fixes so the LLM cannot have memorized the answers, and run Cleverest's default five-iteration loop followed by one 24-hour AFL++ campaign per reaching input. If the combined ClevFuzz pipeline finds fewer bugs than a directed fuzzer that starts from a curated seed corpus, or if Cleverest alone finds none, the claimed advantage of zero-shot LLM regression test generation for this class of programs is refuted.","tokens_in":21174,"feed_emoji":"🐛","tokens_out":8683,"duration_ms":86998,"temperature":0.7,"pith_summary":"The paper sets out to show that a large language model, prompted only with a commit message and a code diff, can generate regression tests for programs whose inputs are highly structured and human-readable, such as XML files, JavaScript programs, and PDF documents. The practical stake is CI/CD: instead of running a 24-hour directed fuzzing campaign that needs a corpus of valid seed inputs, a pipeline could ask an LLM for candidate tests in minutes. The paper claims that this zero-shot feedback loop, Cleverest, finds as many bugs as the state-of-the-art directed fuzzer WAFLGo in a fraction of the time, and that when the generated tests are handed to a generic fuzzer as seeds (ClevFuzz), the zero-shot pipeline finds more bugs than WAFLGo in roughly half the wall-clock time. A secondary claim is that the expressiveness of the commit message is a decisive input: vague messages cut effectiveness sharply, while adding about 17 informative words restored it.","feed_headline":"LLM-generated tests beat a 24-hour fuzzer in minutes","feed_subtitle":"Zero-shot generation plus fuzzing finds more bugs than a state-of-the-art directed fuzzer, without a seed corpus.","key_machinery":"The load-bearing mechanism is the feedback-directed generation loop. A prompt synthesizer turns the task, commit information, and prior attempt history into a prompt; the LLM module returns an input, and optionally the command line; the execution analyzer compiles the program with a sanitizer, runs the input on the before and after versions, and classifies the result using the RIPR (Reaching, Infecting, Propagating, Revealing) model: whether the input reaches the changed lines, changes observable output or return code, or triggers the sanitizer. That classification is appended to the next prompt, so the model can repair near-miss inputs. The second mechanism is integration: a reaching or output-changing Cleverest input becomes the sole seed for a vanilla coverage-guided fuzzer, which mutates it into a bug-triggering input.","core_discovery":"The discovery is that regression test generation for structured-input programs can be treated as a machine-translation problem and solved zero-shot. Cleverest's prompt carries only the commit message, the diff, and the input format name; the model emits a candidate input in that format; the execution analyzer runs it on the program before and after the commit, classifies the outcome as bug triggered, output changed, commit reached, or none, and feeds that classification back into the next prompt. On 22 commits spanning Mujs, Libxml2, and Poppler, the loop found bugs in 3 of 11 bug-introducing commits and reproduced bugs in 4 of 11 bug-fixing commits in under three minutes per commit. That matches WAFLGo's reproduction count and trails its finding count by two, but WAFLGo started from a seed corpus that already revealed the bug in four cases. Used as seeds for a vanilla AFL++, the Cleverest-generated inputs (the ClevFuzz configuration) found 5 of 11 bugs in both scenarios, beating WAFLGo, while taking about 6 hours to WAFLGo's 13 to 15.","pith_inferences":["Editorial inference: the seed-generator result is likely the more durable contribution, because using LLM output to bootstrap coverage-guided fuzzing is a recipe that should transfer to other structured-input domains as long as the format's validity constraints can be represented in the prompt.","Editorial inference: the sensitivity to commit-message expressiveness suggests a cheap experiment, rewording vague commit messages neutrally and re-running the pipeline; if effectiveness tracks specificity, commit-message quality becomes a low-cost lever for both automated testing and human maintenance.","Editorial inference: the PDF failures point to a natural repair loop, feeding parse-error location back to the model, which the paper's own error analysis shows would have been enough to turn near-miss PDFs into reaching inputs.","Editorial inference: the zero-shot claim would be stronger with a post-cutoff bug set, since the paper's similarity check against available bug reports is suggestive but memorization can be ruled out decisively only on bugs fixed after the model's training data ends."],"forward_implications":["A CI/CD pipeline can use an LLM call of a few minutes as a first regression-test pass for structured-input programs, reserving long fuzzing campaigns for cases the LLM does not settle.","Tests that only reach the changed code or change its output are still assets: they are human-readable starting points for developers and seeds that a fuzzer can finish.","Commit-message quality is a measurable lever: a vague message degrades automated test generation, and a short, informative rewrite significantly increases the number of bugs found.","The zero-shot ClevFuzz recipe, LLM-generated seeds plus a stock coverage-guided fuzzer, can outperform a few-shot directed fuzzer that starts from a curated seed corpus, in both bugs found and total time.","The approach is bounded by input readability: compact formats like PDF resist direct generation, though the near-miss inputs stay close enough to be repaired."],"supporting_citations":[{"why":"Supplies the state-of-the-art directed greybox fuzzer WAFLGo, the benchmark dataset of 11 bugs and 22 commits, and the direct comparison baseline.","marker":"[33]"},{"why":"Provides the seed corpus that WAFLGo starts from, which the paper analyzes to show how close those seeds already are to bug-revealing.","marker":"[13]"},{"why":"Supplies the RIPR model used by the execution analyzer to classify each generated test input.","marker":"[12]"},{"why":"Provides the Levenshtein ratio metric used to check whether generated tests could be memorized rather than derived from commit information.","marker":"[2]"}],"fun_headline_variants":["LLM tests in minutes match fuzzer's 24-hour bug haul","Zero-shot LLM test generator outruns fuzzer by hours","Machine translation for regression tests: faster than fuzzing","LLM-generated seeds double fuzzing bug discovery","Regression tests from commit messages: LLM beats directed fuzzer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 11 bugs taken from the reference benchmark represent the broader class of regression bugs in programs with structured human-readable inputs; if those bugs are unusually reachable or their commit messages unusually informative, the reported effectiveness and the ClevFuzz advantage will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LLM tests in minutes match fuzzer's 24-hour bug haul","Zero-shot LLM test generator outruns fuzzer by hours","Machine translation for regression tests: faster than fuzzing","LLM-generated seeds double fuzzing bug discovery","Regression tests from commit messages: LLM beats directed fuzzer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2702,"prompt_tokens":1142,"completion_tokens":1560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":758,"completion_tokens_details":{"reasoning_tokens":1471}},"tokens_in":758,"tokens_out":1560,"duration_ms":12351,"temperature":1.0,"reasoning_tokens":1471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:39:54.884025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take twenty fresh bug-introducing and bug-fixing commits from several other structured-input programs, such as a YAML parser, a JSON schema validator, a SQL engine, and a template engine, all with post-2024 fixes so the LLM cannot have memorized the answers, and run Cleverest's default five-iteration loop followed by one 24-hour AFL++ campaign per reaching input. If the combined ClevFuzz pipeline finds fewer bugs than a directed fuzzer that starts from a curated seed corpus, or if Cleverest alone finds none, the claimed advantage of zero-shot LLM regression test generation for this class of programs is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the state-of-the-art directed greybox fuzzer WAFLGo, the benchmark dataset of 11 bugs and 22 commits, and the direct comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the seed corpus that WAFLGo starts from, which the paper analyzes to show how close those seeds already are to bug-revealing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Levenshtein ratio metric used to check whether generated tests could be memorized rather than derived from commit information."}],"review_version":1}