{"id":"e76141b4-a992-4d58-94e8-3b0f8321af94","arxiv_id":"2607.08949","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An agentic LLM pipeline generates sink-targeted fuzzing seeds that on Magma give geomean crash-time speedups of 11.51-14.66x and unlock 16 ARVO bugs unreachable with default seeds.","lead":"SeedSmith is an LLM agent pipeline that reads a target program's sink function, traces a path to it, and writes concrete crash-triggering seed inputs, which are then handed to any fuzzer. In tests on two fuzzing benchmarks, the seeds cut crash-finding time by roughly an order of magnitude and reached 16 bugs that default seeds never trigger.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Memorization of public Magma/ARVO PoCs may drive the headline gains; no held-out or contamination-controlled evaluation supports the general front-end claim.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the agentic analysis may be reconstructing reachability/preconditions, or it may be recalling benchmark-specific code/PoCs. I agree. Other issues (1-second convention, 24h substitution, missing baselines, no artifact) are secondary: they affect the magnitude and confidence of the quantitative claims, but they do not undermine the core mechanism if contamination is absent. Conversely, if contamination is present, even a perfectly executed evaluation would overstate generality, because the same LLM would fail on any target not in its training data. The paper's own §7 acknowledges dependence on LLM code understanding but does not discuss training-data leakage, and the AIxCC zero-day claim in the same section is merely asserted. The proposed held-out test directly separates reasoning from recall: synthetic or post-cutoff vulnerabilities cannot be memorized, so performance on them isolates the agentic pipeline's actual capability. This is not an accusation; it is the standard required control for any LLM evaluation on public security benchmarks. If the authors pass it, the central claim is substantially supported. If they fail or cannot run it, the quantitative headline should be treated as unvalidated for novel targets.","tokens_in":28954,"tokens_out":6993,"duration_ms":73720,"concrete_test":"Construct a held-out set of 10–20 new Magma-style vulnerabilities (e.g., injected via Magma's framework at sink locations not corresponding to any public CVE, or taken from bugs reported after Claude Sonnet 4's knowledge cutoff), each requiring a non-obvious indirect-call resolution or crash precondition. Run SeedSmith on these with the same settings and measure the one-shot crash rate, downstream-fuzzer speedups, and ARVO-style unlock count. If the one-shot rate and speedups are comparable to the reported Magma/ARVO results, contamination is not the explanation; if they collapse, the benchmark results are likely memorization artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SeedSmith is a fuzzer-agnostic seed front-end that improves any downstream fuzzer on novel targets—rests on the unstated assumption that the LLM reasons about reachability and crash preconditions rather than recalling known exploits. Magma bugs are injected from public CVEs and ARVO is a public dataset with PoCs; both are almost certainly in Claude Sonnet 4's training data. The pipeline feeds the exact sink function code (often the vulnerable function from the CVE) and harness into the LLM, so a one-shot seed can simply be the memorized crash input. The 5 Magma one-shot crashes and many of the 16 ARVO unlocks are concentrated in precisely those well-known bugs. The evaluation never controls for contamination: no comparison against a one-shot LLM prompt without agentic search, no temporal split, no refactored code variants, and no held-out synthetic bugs. If memorization is the driver, the claimed speedups and 'previously unreachable' bugs are not evidence that SeedSmith generalizes; the ablation and cost analysis would still be valid, but the central contribution would be reduced to recalling public PoCs. This is a standard risk in LLM-security evaluation and must be resolved before accepting the quantitative claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SeedSmith is an agentic LLM-based seed-generation front-end for directed fuzzing of C/C++ programs. Starting from a harness and a user-provided sink function, an Analysis Agent uses a context-aware code-search tool to reconstruct a path from the harness to the sink, resolve indirect calls that static call-graph analysis misses, and identify crash preconditions; a Seed Generation Agent then emits Python scripts that construct concrete seeds, which are validated against a sanitizer-instrumented binary in a refinement loop of up to ten rounds. The resulting seeds are added to the initial corpus of a downstream fuzzer without modifying the fuzzer. The paper evaluates SeedSmith on 23 Magma bugs with AFL++, FairFuzz, AFLGo, and AFLRun, and on 115 ARVO targets with AFL++ and AFLRun, comparing Default, SeedSmith-only, and SeedSmith-combined seed configurations. It reports geometric-mean crash-time speedups of 11.51x (AFL++) to 14.66x (AFLGo) on Magma, 16 previously un-triggered ARVO bugs across 10 projects, a comparison against Locus on Magma, and ablations of the scan strategy and CodeQL-based control-flow support.","tokens_in":29269,"tokens_out":7555,"duration_ms":80959,"significance":"If the quantitative claims hold, SeedSmith addresses a real and well-motivated problem: directed fuzzers frequently fail to trigger crashes because static call graphs miss indirect edges and because crash preconditions are invisible to coverage feedback. The intervention at the seed-corpus level is clean, fuzzer-agnostic, and supported by a large evaluation that is more extensive than most work in this area: 23 Magma bugs, 115 ARVO targets, four fuzzers, ten trials per configuration, per-bug Mann-Whitney p-values, and validation of seeds against sanitizer-instrumented binaries. The cost analysis (Table 4) and the appendices with full prompts and a sample analysis report also strengthen reproducibility. The principal risk is that the central generalization claim is not separated from the possibility that the LLM recalls public PoCs and vulnerable code from training data, since Magma and ARVO are public benchmarks and the pipeline feeds exact sink code into the model. That risk is material and needs to be resolved before the headline claims can be accepted at face value.","major_comments":[{"comment":"The central claim—that agentic code exploration recovers reachability and crash preconditions that static analysis misses—is not separated from the possibility that Claude Sonnet 4 recalls public PoCs and vulnerable code from its training data. Magma bugs are injected from public CVEs and ARVO is a public dataset with PoCs; the pipeline gives the model the exact sink function code and the harness. The five Magma one-shot crashes and the 16 ARVO unlocks could be produced by memorization rather than by the agent's iterative search and reasoning. The evaluation contains no contamination control: no held-out bugs introduced after the model training cutoff, no refactored/variable-renamed code variants, no comparison against a one-shot LLM prompt (or non-agentic retrieval baseline), and no recall probe. Since the abstract claims the seeds improve any downstream mutation-based fuzzer on novel t","section":"§4.2, §6.2.2, Table 3, and §7"},{"comment":"The speedup computation treats '1-shot' crashes as 1 s of fuzzer time while ignoring the 678 s average seed-generation cost per target reported in Table 4. For the five Magma one-shot cases, the per-bug ratio over a 24h default timeout is 86400 under the current convention; including the 678s generation cost reduces each such ratio to about 127. Because these entries are the largest terms in the geometric mean, the headline 11.51x–14.66x speedups are materially dependent on this convention. The paper discloses the convention, and the cost may be small relative to a full 24h campaign, but the metric is labeled 'crash time' and the abstract presents the speedups without qualification. Please recompute all geomeans with end-to-end time (seed generation + fuzzing) or clearly separate fuzzer-only and end-to-end numbers in every table and in the abstract.","section":"§6.1 Metrics; Table 4"},{"comment":"The abstract's '16 previously unreachable bugs' is stronger than what the experiments show: the comparison is against AFL++ and AFLRun with default seeds within a 24h budget. These bugs may be reachable by other fuzzers or with more time, so the claim should be 'not triggered by the evaluated baseline configurations within 24h.' The statistical support for the ARVO speedup is also uneven: the 3.09x/3.02x geomeans are dominated by the 16 unlock cases, while on the 21/31 bugs that both configurations trigger the speedup is 1.46x/1.71x and not significant (p=0.50 and p=0.18, respectively). This distinction should be reflected in the abstract, not only buried in §6.2.2. Additionally, the conclusion reports 'p=0.58' for the shared-bug comparison, which does not match the p-values given in §6.2.2.","section":"§6.2.2, Table 3, and Conclusion"}],"minor_comments":[{"comment":"The context-window thresholds (X=10, Y=2, 50K-token cap, 30-tool-call halt, 10 refinement rounds) are described as empirically chosen defaults, but no sensitivity analysis is reported. Since these parameters affect both cost and effectiveness, reporting one or two variations (e.g., X ∈ {5, 20} or rounds ∈ {5, 15}) would make the robustness of the pipeline easier to assess.","section":"§5 Implementation"},{"comment":"The cost analysis is reported only for Magma. The ARVO evaluation uses 115 targets across 26 projects, each requiring an analysis report and seed generation; reporting total or average ARVO cost would help readers judge end-to-end deployability.","section":"§6.2.3 / Table 4"},{"comment":"The control-flow-support ablation is computed on only 8 Magma bugs where CodeQL succeeds, and the geomeans in Table 6 are dominated by a few timeout-to-one-shot ratios. For FairFuzz, the 540.9x figure is based on two non-timeout ratios and is explicitly non-comparable. The raw per-bug ratios should be shown so the reader can see how fragile these marginal speedups are.","section":"§6.4, Tables 5 and 6"},{"comment":"Minor text issues: 'Geneartion' in Figure 3; 'the the vulnerability' in §4.2; 'they they' in §1; 'less ineffective' in §7; and the inconsistency in the p-value for the shared ARVO subset between §6.2.2 (p=0.50/0.18) and the Conclusion (p=0.58). An artifact/reproducibility statement with code links and exact model versions would also help.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The contamination/memorization concern is the main barrier to acceptance. The manuscript is otherwise well executed, with an unusually large and careful evaluation, and the seed-front-end idea is timely and likely to have impact. I would make a contamination-control experiment a condition for acceptance, and I would also ask the authors to rerun the headline speedup numbers with seed-generation time included or clearly separated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing you need to know: this paper describes a genuinely useful system, but the headline speedups are not yet credible because the evaluation never rules out the LLM recalling public PoCs from Magma and ARVO. The underlying idea — a two-stage agentic pipeline that analyzes the codebase then generates seeds — is a real step beyond one-shot prompting, and the evaluation is larger than most in this area. But the quantitative claims rest on a convention and a substitution rule that flatter the results, and the memorization confound is not addressed.\n\nWhat's genuinely new: the separation of analysis and generation, with a heavyweight model doing iterative code exploration and a cheap model emitting Python scripts that construct seeds. The context-aware retrieval rules are a sensible fix to the fixed-line-window problem. The path optimization over an over-approximated call graph is clever, and the ablation study gives partial credit to each component. The cost transparency is refreshing: $5.28 and about 11 minutes per target on average. On the core claim — that sink-targeted seeds help downstream fuzzers — I think the evidence is there, even if the magnitude is uncertain.\n\nThe soft spots are real. Counting a one-shot crash as 1 second of fuzzer time inflates geomean speedups by a lot, and substituting 24 hours for censored trials pushes the ratio up further. The paper doesn't compare against a simple one-shot LLM prompt, so we don't know what the agentic search adds. No artifact is provided, and several hyperparameters are tuned on the evaluation set. The AIxCC zero-day claim in Section 7 is unsupported and should be removed or substantiated.\n\nThe big one is contamination. Magma bugs come from public CVEs and ARVO ships PoCs. Claude Sonnet 4 was almost certainly trained on much of this. The pipeline feeds the sink function — often the CVE's vulnerable function — directly to the LLM. So a one-shot seed may simply be a memorized crash input. The paper does no temporal split, no refactored variants, no held-out synthetic bugs, and no comparison against a single-shot prompt that would control for memorization. Without that, the 16 ARVO unlocks and the 5 one-shot Magma crashes are not convincing evidence that SeedSmith generalizes to new targets. The case study is nice, but a memorized PoC can produce the same trace.\n\nNone of this means the paper is junk. The engineering is sound, the evaluation is big, and the failure model is plausible. But the central quantitative claims should not be taken at face value until the contamination question is resolved. I'd send it to peer review, and ask the authors to release code/data and redo the evaluation with a contamination-controlled design. This is a paper for the fuzzing and LLM-security community; it will generate discussion. If the numbers survive a clean evaluation, it could be important.","headline":"SeedSmith is a solid engineering contribution, but the headline speedups are not yet trustworthy because the LLM could be recalling public PoCs rather than reasoning.","tokens_in":29840,"tokens_out":5650,"would_cite":true,"duration_ms":55494,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Direct fuzzing misses crashes because its starting seeds are blind; this paper argues that an agentic LLM pipeline can write crash-triggering seeds up front, cutting crash time by 11.5x to 14.7x and unlocking 16 previously unreachable bugs.","keywords":["directed fuzzing","seed generation","large language models","agentic LLM","indirect calls","crash preconditions","vulnerability discovery","seed corpus"],"falsifier":"Run the pipeline on a set of sink functions whose crashing inputs were created after the LLM's training cutoff (e.g., freshly injected vulnerabilities or private targets) and compare the one-shot crash rate and crash-time speedups against the reported numbers; a large drop would indicate that memorization, not reconstruction, is the mechanism. A second check is to strip the analysis agent's code-search tool and feed only the static call-graph hint: if one-shot crashes persist without any tool access, the LLM alone is doing the work.","tokens_in":28790,"feed_emoji":"🐛","tokens_out":8836,"duration_ms":73970,"temperature":0.7,"pith_summary":"This paper claims that directed fuzzers fail for two reasons: static analysis misses indirect calls, leaving the target path invisible to distance guidance, and crash preconditions are too precise for blind mutation to satisfy. The proposed fix is not a new fuzzer but a better initial seed corpus: seeds that already encode the correct control-flow path and key crash conditions turn the fuzzer's task into local refinement. To build such seeds, the paper presents a two-stage LLM pipeline that mimics a security analyst: one agent iteratively searches the codebase to resolve indirect calls and extract crash preconditions, and a second agent converts that analysis into concrete seed files, refined by execution feedback. In evaluation, seeds from the pipeline cut crash time by over an order of magnitude in geometric mean across four fuzzers on a widely used benchmark, and triggered 16 real-world bugs that default seeds never reached across 10 projects.","feed_headline":"LLM-written seeds make fuzzers find bugs 11.5x to 14.7x faster","feed_subtitle":"Two AI agents write crash-triggering seeds that work with any fuzzer, unlocking 16 previously unreachable bugs.","key_machinery":"The load-bearing mechanism is the two-stage agentic pipeline. The analysis agent searches a project index with a context-aware retrieval tool that returns whole function bodies, type definitions, or configuration files rather than fixed line windows; this lets it resolve indirect calls by reading type information and function bodies, and discover crash preconditions by tracing data flow. A path-optimization step compresses the over-approximated static call graph into a single linearized path with a connector node for divergent middles, keeping the search within context limits. The seed generation agent then converts the analysis report into Python scripts that build inputs with format librar","core_discovery":"The central claim is that the bottlenecks of directed fuzzing—missing indirect call edges and semantically demanding crash preconditions—can be bypassed at the seed layer rather than in the fuzzer. The system's analysis agent starts from the sink function and, using a context-aware code-search tool, iteratively reconstructs the execution path from the harness, resolves function-pointer and virtual-dispatch calls that static call graphs omit, and identifies the input-level conditions that must hold for the crash to fire. A second agent consumes the resulting report and writes Python scripts that construct structurally valid seeds; those seeds are run against a sanitizer-instrumented binary an","pith_inferences":["Because the evaluation does not control for training-data memorization, the strongest non-obvious inference is that the pipeline's real-world ceiling depends on whether the LLM is reconstructing reachability from code or recalling benchmark-specific exploits; a test on freshly created, non-public vulnerabilities would separate these.","The analysis/seed-generation split suggests the design can ride the trend of improving code-reasoning models: a stronger analysis model should improve seed quality with no pipeline change—and equally, the system is hostage to a single model's weaknesses.","Pairing the seed generator with an automatic sink locator is the natural route from known-vulnerability reproduction to zero-day discovery; the paper reports doing this in a competition setting but evaluates only the N-day case.","The pattern of results implies an editing or triage workflow: run the seed generator first, then fuzz, and focus remaining compute on the few targets that even good seeds do not crack."],"forward_implications":["Seeds are fuzzer-agnostic: any mutation-based fuzzer, directed or coverage-guided, can drop them into its initial corpus without modification and inherit the crash-time speedups.","The effect is front-loaded: several of the 23 benchmark bugs crashed on the first generated seed, before any mutation, and cumulative crash curves plateau within the first hour.","The approach gives general-purpose fuzzers directed-crash capability: a coverage-guided fuzzer using the seeds outperformed a dedicated directed fuzzer at exposing targeted crashes.","On real-world vulnerabilities, the primary benefit is expanding reachable bugs (16 previously unreachable crashes across 10 projects) rather than merely accelerating already-reachable ones.","The upfront cost is small relative to a 24-hour fuzzing campaign—roughly $5 and 11 minutes per target on average—so the pipeline is practical as a pre-fuzzing step."],"fun_headline_variants":["LLM seeds cut crash times 11.5x–14.7x in directed fuzzing","Agentic LLM seed synthesis speeds directed fuzzing 14.7x","LLM seeds unlock 16 bugs that were unreachable by fuzzing","Seed-level AI bypasses fuzzing bottlenecks, finds 16 unreachable bugs","Fuzzer-agnostic LLM seeds speed crash discovery 11.5–14.7x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a pretrained LLM genuinely reconstructs reachability and crash preconditions from the code through iterative search, rather than recalling benchmark-specific code or proof-of-concept inputs from its training data; the paper tests the pipeline end-to-end but never isolates this factor.","fun_headline_variants_meta":{"raw":{"variants":["LLM seeds cut crash times 11.5x–14.7x in directed fuzzing","Agentic LLM seed synthesis speeds directed fuzzing 14.7x","LLM seeds unlock 16 bugs that were unreachable by fuzzing","Seed-level AI bypasses fuzzing bottlenecks, finds 16 unreachable bugs","Fuzzer-agnostic LLM seeds speed crash discovery 11.5–14.7x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001131,"raw_usage":{"total_tokens":4569,"prompt_tokens":812,"completion_tokens":3757,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":3642}},"tokens_in":556,"tokens_out":3757,"duration_ms":24753,"temperature":1.0,"reasoning_tokens":3642,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:43:41.354153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a set of sink functions whose crashing inputs were created after the LLM's training cutoff (e.g., freshly injected vulnerabilities or private targets) and compare the one-shot crash rate and crash-time speedups against the reported numbers; a large drop would indicate that memorization, not reconstruction, is the mechanism. A second check is to strip the analysis agent's code-search tool and feed only the static call-graph hint: if one-shot crashes persist without any tool access, the LLM alone is doing the work.","supporting_citations":[],"review_version":2}