{"id":"88a84846-0a5f-4fbc-8fa2-c74afbd9dfa3","arxiv_id":"2505.03425","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"HGFuzzer uses an LLM to generate harnesses, reachable seeds, and custom mutators for directed greybox fuzzing, and reports triggering 17/20 known vulnerabilities plus finding 9 new ones with CVE IDs.","lead":"HGFuzzer is a new fuzzing tool that uses a large language model to build a custom test harness, a first input, and a targeted mutator for a given vulnerable function. It reports triggering 17 of 20 known bugs and discovering 9 new CVEs, but the evaluation is weakened because the model likely saw the benchmark code during training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation leaks the vulnerability description into HGFuzzer's prompts, so the 24.8x speedup over DGF baselines is explained by information asymmetry rather than by the method.","rationale":"The paper's central claim is an empirical superiority result, so the evaluation must compare like with like. It does not: HGFuzzer's prompts embed the vulnerability description from CVE records, so the LLM is given the answer it is supposed to find. This is not a speculation about training data—it is a design feature of the prompt templates in Section 3.3 and the mutator-generation procedure in Section 3.5. The baselines get only the target function and must explore to find triggering conditions. The resulting 24.8x speedup is therefore an oracle comparison. I partially agree with the reader: the reader attributed the problem to LLM memorization (Section 5), but the more fundamental issue is the explicit CVE description in the prompts, which would be a confound even for a freshly trained LLM. The RQ3 ablation does not test this, since all variants keep the description. RQ4's nine new CVE IDs are real evidence of some capability and are appropriately credited, but they do not support the quantitative speedup claim against SOTA. A simple ablation that removes or redacts target descriptions would settle the question; until then the evidence does not establish the abstract's claim. I therefore concur with the reader's REJECT verdict.","tokens_in":24576,"tokens_out":4938,"duration_ms":45202,"concrete_test":"Run HGFuzzer on the same 20-CVE benchmark in four conditions: (A) original prompts; (B) prompts with the target_description removed from both harness and mutator generation; (C) prompts with target_description replaced by a neutral phrase like 'target function to test'; (D) a holdout set of CVEs disclosed after Claude-3.5-Sonnet's training cutoff, with descriptions removed. If the median TTE in (B)/(C) degrades by more than 10x relative to (A), or if the success rate falls from 17/20 toward the baselines' 5-6/20, the claimed 24.8x speedup is an artifact of leaked vulnerability information. The same comparison also separates memorization (D) from prompt-level leakage (B/C).","verdict_should_be":"REJECT","load_bearing_attack":"The headline speedup is not supported by the evaluation design. For each known CVE, HGFuzzer's harness prompt includes a 'target description' that 'may come from prior vulnerability reports (e.g., CVE records)' (Section 3.3), and the mutator prompt explicitly asks the LLM to analyze the root cause of the vulnerability from this description (Section 3.5). The DGF baselines (AFLGo, Beacon, SelectFuzz) receive only a target location and must discover the bug-triggering conditions through fuzzing. HGFuzzer is therefore effectively told what bug to trigger, making the reported speedup an oracle-aided result. Section 5's admission that the benchmark libraries are 'likely included in the training data' further compounds the asymmetry: even if the description were removed, Claude-3.5-Sonnet may have memorized the vulnerable code and known PoCs. The 17/20 triggering rate and '11 within the first minute' are consistent with retrieval of known vulnerability knowledge rather than with the framework's call-chain analysis, reachable-input generation, and custom mutators. The RQ3 ablation does not address this, because every variation retains the target description. Consequently, the central empirical claim does not demonstrate that HGFuzzer outperforms SOTA DGF on a level playing field.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HGFuzzer, a directed greybox fuzzing framework that uses LLMs to generate target-specific harnesses, reachable seed inputs, and custom mutators for C/C++ libraries. The pipeline consists of static call-chain analysis, LLM-based execution-condition inference, harness generation with RAG-based compilation repair, Python-script-based reachable input generation, and LLM-generated custom mutators integrated into AFL++. The evaluation compares HGFuzzer against AFLGo, Beacon, SelectFuzz, and AFL++ on 20 known CVEs from nine open-source libraries, reporting that HGFuzzer triggers 17 of 20 vulnerabilities, 11 within the first minute, with at least a 24.8x speedup over baselines. The paper also reports discovering nine previously unknown vulnerabilities with assigned CVE IDs. The central claim is that HGFuzzer substantially improves directed greybox fuzzing efficiency by using LLMs to reduce exploration and exploitation randomness.","tokens_in":24884,"tokens_out":3995,"duration_ms":43768,"significance":"If the empirical claims were well supported, HGFuzzer would be a meaningful contribution to directed greybox fuzzing and to the growing line of work on LLM-assisted fuzzing. The decomposition of the problem into harness generation, reachable-seed generation, and target-specific mutator generation is sensible, and the paper makes an effort to ablate these components. The fact that the reported new vulnerabilities received CVE IDs is concrete evidence of some practical bug-finding capability. However, the evaluation does not provide a level playing field between HGFuzzer and the baselines, and the paper explicitly acknowledges that the benchmark libraries are likely in the LLM's training data. As a result, the headline speedup and success-rate numbers are consistent with query-specific memorization and oracle-style vulnerability descriptions rather than with a generalizable method. The contribution is therefore not currently established at the level claimed.","major_comments":[{"comment":"The comparison with AFLGo, Beacon, and SelectFuzz is not apples-to-apples. Section 3.3 states that the harness prompt includes a 'Target Description' that 'may come from prior vulnerability reports (e.g., CVE records)', and Section 3.5's mutator prompt explicitly asks the LLM to 'analyze the root cause of the vulnerability' from this description. The DGF baselines receive only a target location and must infer the bug-triggering conditions by fuzzing. HGFuzzer thus receives information that directly encodes the vulnerability to be triggered. The 24.8x speedup and the 17/20 success rate in Table 1 are therefore explained by this information asymmetry, not by the framework's call-chain analysis, reachable-input generation, or custom mutators alone. A fair evaluation would either remove CVE descriptions and root-cause hints from all HGFuzzer prompts or give the baselines equivalent semantic information.","section":"§3.3, §3.5, Table 1"},{"comment":"Section 5 concedes that the benchmark open-source libraries 'are also likely included in the training data of the selected LLM.' This is a direct threat to the central claim: Claude-3.5-Sonnet may have memorized the vulnerable source code, the CVE records, and even public proof-of-concept inputs. Under that hypothesis, the generated harnesses, reachable seeds, and mutators are acts of retrieval or near-retrieval rather than evidence that the proposed pipeline solves the path-constraint and mutation-design problems. The paper does not include any experiment on a post-training-cutoff library, a closed-source target, or a sufficiently transformed benchmark to rule out memorization. The acknowledgment in Section 5 is framed as a limitation, but it undermines the generalizability of the headline results and requires new evidence before the central claim can be accepted.","section":"§5, 'Expanding Applicability'"},{"comment":"The ablation study does not address the information-asymmetry problem. All three variations (Without Input, Without Mutator, and Harness-only) still receive the target description and the root-cause-oriented prompts defined in Sections 3.3 and 3.5. The ablation therefore shows only how the components contribute when the LLM is given oracle-style vulnerability knowledge; it does not show that the speedup over baselines survives when that knowledge is removed. To make RQ3 informative, the authors would need to ablate the target description itself, for example by replacing it with only the target function and call chain.","section":"§4.3, Table 3"},{"comment":"The claim that HGFuzzer 'discovered 9 previously unknown vulnerabilities' should be interpreted with the evaluation design in mind. The targets for these experiments were selected either from root-cause locations of recent CVEs in the CVE database or from locations identified by manual expert review. This is a plausible way to search for related bugs, but it is not an end-to-end discovery of vulnerabilities from code alone. The nine CVEs also come from only two libraries (libming and lcms2), and most are memory leaks, so the breadth of the finding is limited. The result is still useful evidence of practical bug-finding ability, but it does not support the stronger reading that HGFuzzer independently discovers new vulnerabilities.","section":"§4.4, Table 4"}],"minor_comments":[{"comment":"The legend contains the typo 'Imiediately Exploit'; it should be 'Immediately Exploit', and the abbreviation 'I.E.' should be written as something less confusable with the punctuation 'i.e.'.","section":"Table 1"},{"comment":"The row header 'A VG' appears to be an unintended spacing of 'AVG' or 'Average'; please correct it.","section":"Table 2"},{"comment":"The phrase 'all-chain analysis' in the discussion of hallucinations should be 'call-chain analysis'.","section":"§5"},{"comment":"The prompt template uses a placeholder {target_info} and the text says this information 'may come from prior vulnerability reports,' but the paper never states how many of the 20 benchmark CVEs had such a description available and whether the same description was provided to the baselines in any form. Clarifying this would help the reader evaluate the fairness of the comparison.","section":"§3.3, Figure 5"},{"comment":"The paper appears to use an unmodified ACM template placeholder ('Conference acronym ’XX', 2018 dates) and does not provide an artifact link; please update the camera-ready format and, ideally, release the code and benchmark scripts to support reproducibility.","section":"Formatting"}],"recommendation":"reject","confidential_remarks":"The core issue is not the idea but the evaluation: the paper's own text in Sections 3.3, 3.5, and 5 makes the information asymmetry and training-data contamination explicit. I do not see a path to acceptance within a normal revision cycle unless the authors redo the central experiments with held-out targets, remove oracle-style descriptions, and add an LLM-based baseline that receives the same information. The discovery of nine CVEs is a positive signal, but it is attached to a target-selection procedure that already points at suspicious code, so it does not compensate for the invalid head-to-head comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: HGFuzzer is a cleanly engineered LLM-directed fuzzing pipeline, but the headline 24.8x speedup is not supported by the evaluation as designed. The comparison leaks the vulnerability description to HGFuzzer and nothing to the baselines, and the authors themselves admit the libraries are probably in the LLM training data.\n\nWhat's actually new: the combination of call-chain analysis, LLM-generated harness, Python-scripted reachable seed, and target-specific custom mutator. I haven't seen that exact package elsewhere; Magneto, ISC4DGF, and CKGFuzzer overlap on pieces. The ablation study is useful: each component improves time-to-exploit, and the harness-only variant does worst. The RQ4 findings—9 CVEs in libming and lcms—are real artifacts, even if the memory leaks are shallow.\n\nWhere it breaks down: the evaluation is not a level playing field. For each known CVE, HGFuzzer's prompts include a target description that \"may come from prior vulnerability reports (e.g., CVE records)\" (Section 3.3), and the mutator prompt asks the LLM to analyze the root cause from that description (Section 3.5). The baselines receive only target locations. So the speedup, the 17/20 trigger rate, and the \"11 within the first minute\" are exactly what you'd predict from an oracle-aided system. RQ3 doesn't fix this: every ablation variant keeps the target description. Section 5's admission that the benchmark libraries are \"likely included in the training data\" of Claude-3.5-Sonnet compounds the problem—even without the description, the LLM may know the bug and its PoC. The RQ4 new-CVE results are less contaminated because they target recent vulnerabilities, but the method still gets root-cause hints from the CVE database or the expert, so they don't demonstrate a general decontaminated method either.\n\nThe paper is honest about the training-data issue, which I credit. It's clearly written and the implementation is real. But the central empirical claim—outperforming SOTA DGF—is not established. The right fix is a decontaminated benchmark: libraries released after the LLM cutoff, or at least runs where the target description is withheld. The authors should also release the harnesses, seeds, and mutators so the community can check for memorization.\n\nWho this is for: researchers working on LLM-guided fuzzing, especially those designing evaluations for such systems. The paper is a good case study in how easy it is to leak target knowledge into LLM prompts. I'd send it to peer review—it deserves referee time—but I'd expect major revision or rejection unless the decontamination comes through.","headline":"HGFuzzer is a real system with a plausible pipeline, but its headline speedup is not credible because the evaluation leaks target descriptions to the LLM and the authors admit training-data contamination.","tokens_in":25384,"tokens_out":3353,"would_cite":false,"duration_ms":32742,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HGFuzzer turns directed fuzzing's two bottlenecks, path explosion and random mutation, into LLM code-generation tasks, and reports triggering 17 of 20 real-world vulnerabilities with a speedup of at least 24.8x.","keywords":["Fuzzing","Directed Greybox Fuzzing","Large Language Model","Vulnerability","Open-source Library","Test Harness","Reachable Input Generation","Custom Mutator"],"falsifier":"Run HGFuzzer on a target that postdates the LLM's knowledge cutoff, such as a vulnerability in a library release made after the model's training date or in a proprietary library never published online, and compare time-to-trigger with AFLGo; if the speedup vanishes, memorization rather than the framework carried the result.","tokens_in":24394,"feed_emoji":"🐛","tokens_out":10347,"duration_ms":87939,"temperature":0.7,"pith_summary":"Directed greybox fuzzing tries to steer a fuzzer toward a specific target, such as a known vulnerable function, but wastes effort exploring irrelevant paths and mutating inputs blindly. This paper claims that a large language model can remove both inefficiencies: the LLM reads the call chain leading to the target, writes a small harness that fixes the path, generates a seed input that satisfies the path conditions, and writes a custom mutator aimed at the vulnerability's trigger conditions. The resulting system, HGFuzzer, triggered 17 of 20 known vulnerabilities in a benchmark of open-source C/C++ libraries, 11 of them within the first minute, and achieved at least a 24.8x speedup over three directed fuzzers. It also reported 9 previously unknown vulnerabilities in two libraries, each assigned a CVE ID. If the approach generalizes, it would make directed fuzzing dramatically cheaper for patch testing, crash reproduction, and vulnerability discovery.","feed_headline":"LLM-driven fuzzer triggers 17 real bugs, finds 9 new CVEs","feed_subtitle":"HGFuzzer turns path constraints into harnesses, seeds, and custom mutators, cutting time-to-trigger by at least 24.8x.","key_machinery":"The load-bearing mechanism is the transformation of path-constraint solving into LLM-guided code generation along a chosen call chain. Given a target, HGFuzzer selects the shortest call chain that begins at main or at a declared external function, and asks the LLM to produce three artifacts: a harness that constrains execution to that chain, a Python script that generates a seed satisfying all branch conditions along the chain (verified with afl-cov), and a custom mutator written against the AFL++ custom-mutator API that biases inputs toward the vulnerability's trigger conditions. The harness removes irrelevant path exploration, the seed removes the need for random path discovery, and the mutator removes blind mutation; together they convert directed fuzzing from a search problem into a guided generation problem.","core_discovery":"The central discovery is that the two bottlenecks of directed greybox fuzzing, path explosion during exploration and randomness during exploitation, can be recast as code-generation problems that an LLM can solve. Given a target function, HGFuzzer uses static analysis to extract the shortest call chain reaching it, then asks the LLM to infer the execution conditions at each call and to produce three artifacts: an executable harness that pins the path to that chain, a Python script that synthesizes a seed satisfying every condition, and a custom C/C++ mutator aimed at the vulnerability's root cause. On 20 real-world CVEs across nine open-source C/C++ libraries, HGFuzzer triggered 17 of them, versus 5 or 6 for the three directed-fuzzer baselines, with 11 triggered in the first minute and a minimum speedup of 24.8x. It additionally reported 9 previously unknown vulnerabilities, all assigned CVE IDs, and raised the target-function hit rate from the best baseline's 37.22% to 64.75%.","pith_inferences":["A direct consequence of the paper's own training-data caveat is that the 24.8x speedup is likely an upper bound on what the pipeline would achieve on code outside the LLM's training distribution; the first decisive test is to run HGFuzzer on a library released after the model's knowledge cutoff.","Because HGFuzzer fixes on a single shortest call chain, vulnerabilities reachable through multiple calling contexts may be missed or fuzzed along a non-triggering path; enumerating several chains and running the pipeline per chain is a natural extension the paper leaves implicit.","The LLM-generated custom mutator is effectively a small, vulnerability-specific grammar; combining it with grammar-aware mutation or format-aware parsers could transfer the benefit to binary-heavy input formats without regenerating seeds each time."],"forward_implications":["HGFuzzer's preparation phase drops block-distance computation and symbolic execution: the paper reports static analysis as call-chain querying that completes in roughly 6-7 seconds per target in its benchmark.","Given only a target location, such as a recent CVE root cause or a reviewer-identified suspicious function, the same pipeline discovered 9 previously unknown vulnerabilities in two open-source libraries, all assigned CVE IDs.","The ablation study shows that removing the reachable-input generator or the custom mutator lowers success from 17/20 to 13-14/20 and stretches time-to-trigger from minutes to hours, so both components are necessary for the reported speedup.","Because every stage is prompt-driven, using a stronger LLM or adding a cross-check on inferred constraints should directly improve harness quality and reduce failures from misinterpreted branch conditions, as the paper notes in its discussion."],"supporting_citations":[{"why":"AFLGo, the original directed greybox fuzzer whose distance-based guidance is the main comparison baseline for time-to-trigger.","marker":"[10]"},{"why":"BEACON, a directed fuzzer with provable path pruning, used as a state-of-the-art baseline.","marker":"[21]"},{"why":"SelectFuzz, a selective path-exploration directed fuzzer, used as a baseline and the best prior performer on target-hit rate.","marker":"[39]"},{"why":"AFL++, the engine that HGFuzzer extends and whose custom mutator API the generated mutators must implement.","marker":"[2]"},{"why":"afl-cov, used to verify that a generated reachable input actually drives execution to the target function.","marker":"[1]"},{"why":"OSS-Fuzz, one of the two sources of the 20-vulnerability benchmark dataset.","marker":"[6]"},{"why":"Claude-3.5-Sonnet, the specific large language model that performs all code-generation and condition-analysis steps.","marker":"[9]"}],"fun_headline_variants":["LLM fuzzer triggers 17 real bugs, uncovers 9 new CVEs","HGFuzzer: LLM-based fuzzing finds 9 CVEs, 24.8x faster","Directed fuzzing via LLM: 17 bugs triggered, 9 new CVEs","LLM fuzzing recasts path constraints, snags 17 real bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's own caveat in Section 5, that the benchmark libraries are likely in the LLM's training data, makes the results hinge on the assumption that the model's reasoning, not its memory of known CVEs, produces the speedups.","fun_headline_variants_meta":{"raw":{"variants":["LLM fuzzer triggers 17 real bugs, uncovers 9 new CVEs","HGFuzzer: LLM-based fuzzing finds 9 CVEs, 24.8x faster","Directed fuzzing via LLM: 17 bugs triggered, 9 new CVEs","LLM fuzzing recasts path constraints, snags 17 real bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1619,"prompt_tokens":971,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":550}},"tokens_in":587,"tokens_out":648,"duration_ms":5722,"temperature":1.0,"reasoning_tokens":550,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:51:21.434537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HGFuzzer on a target that postdates the LLM's knowledge cutoff, such as a vulnerability in a library release made after the model's training date or in a proprietary library never published online, and compare time-to-trigger with AFLGo; if the speedup vanishes, memorization rather than the framework carried the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AFL++, the engine that HGFuzzer extends and whose custom mutator API the generated mutators must implement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"afl-cov, used to verify that a generated reachable input actually drives execution to the target function."},{"cited_title":"OSS-Fuzz","cited_arxiv_id":null,"evidence_quote":"OSS-Fuzz, one of the two sources of the 20-vulnerability benchmark dataset."}],"review_version":1}