{"id":"3e3c0663-907f-434b-81e1-eeed44dc723c","arxiv_id":"2507.15241","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A staged LLM agent that reasons about data flow and branch conditions generated valid proof-of-vulnerability tests for 16 of 100 Java, C, and C++ projects, versus 9 for the CodeAct 2.1 baseline.","lead":"FaultLine is a three-stage LLM agent that traces data from user input to a vulnerable sink, infers constraints on the input from branch conditions, then generates and repairs a proof-of-vulnerability test. On 100 known Java, C, and C++ vulnerabilities it produced correct tests for 16 projects versus 9 for the CodeAct 2.1 baseline, though the paper's evaluation omits a fixed-state check.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A PoV test counts as correct only if it fails at the vulnerable commit and touches a fixed method; Section 7 admits no fixed-commit pass check. Tests that fail for unrelated reasons could inflate the 16-vs-9 gap.","rationale":"Reader's weakest_assumption is exactly the fixed-state pass check, and the paper's own Section 7 confirms it. This is load-bearing because the definition of a PoV test is the entire basis of the evaluation; without the fixed-state check, 'correct' is not the correct notion. The paper argues that dependency mismatches make the check hard, but that does not remove the need. I considered other concerns: the ablation in RQ3 is run only on the 16 successful projects, which cannot establish that flow/branch reasoning are essential; the reported numbers in Section 5.1 for vulnerable-function coverage are inconsistent (17 vs 19 in different places); and there is no repeated-run significance testing for 16 vs 9. These are real but secondary. The fixed-commit check is the single most direct threat to the central claim, and it is cheap to test on 25 examples. Therefore verdict remains CONDITIONAL: contribution plausible but the central quantitative claim needs this validation.","tokens_in":18473,"tokens_out":3949,"duration_ms":40859,"concrete_test":"For each of the 16 FaultLine and 9 CodeAct successful tests, checkout the provided fixed commit (CWE-Bench-Java gives fixed hashes; for PrimeVul use the fix commit from the CVE or PrimeVul metadata), add the generated test file, rebuild the Docker image using the same Dockerfile and the fixed source, and run the container. Record whether the test exits 0 (pass). If a substantial number of either tool's tests fail on the fixed commit for reasons other than environment/dependency mismatch (e.g., the test itself is buggy), then those tests are not valid PoV tests and both success counts must be reduced; recompute the 16-vs-9 comparison. Ideally also fix any Dockerfiles that break on the fixed commit so that the check is a fair test of pass/fail semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FaultLine generates 'correct' PoV tests for 16 projects versus 9 for CodeAct. Correctness is defined in Section 4.4 by three automated checks: (1) build at the vulnerable commit, (2) run exits non-zero, (3) execution reaches a method touched by the fix, plus a manual CWE-specific inspection. There is no check that the test passes (exit 0) when the vulnerability is fixed. This is a necessary property of a PoV test and is explicitly acknowledged as missing in Section 7 (Limitations). Consequences: a test that crashes at the vulnerable commit because of a wrong API assumption, a missing dependency, or an environment issue is indistinguishable from a test that actually demonstrates the vulnerability. The manual inspection checks only that, e.g., embedded code is executed in the vulnerable state; it does not verify that a patched program would neutralize the exploit. If the 16 FaultLine successes include even a few tests that would also fail on the fixed commit for reasons unrelated to the vulnerability, the headline 16-vs-9 gap and the 77% relative improvement are not supported. Since both tools are scored by the same flawed criterion, the relative gap could survive or disappear depending on the distribution of false positives; there is no evidence either way.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FaultLine, an LLM-agent workflow for automatically generating proof-of-vulnerability (PoV) tests for known vulnerabilities. The workflow has three stages: data-flow reasoning from a source to a sink, control-flow or branch reasoning that derives input constraints, and test generation plus feedback-driven repair inside Docker. The authors evaluate FaultLine on a newly collated dataset of 100 vulnerabilities from CWE-Bench-Java (70 Java projects) and PrimeVul (30 C/C++ projects), comparing against the CodeAct 2.1 agent in OpenHands with the same underlying model (Claude 3.7 Sonnet) and the same budget. They report 16 correct PoV tests for FaultLine versus 9 for CodeAct, and 31 versus 19 tests that reach the vulnerable functions. They also report ablation results suggesting that the flow and branch reasoning components contribute to performance. The code, prompts, and dataset are released publicly.","tokens_in":18664,"tokens_out":10954,"duration_ms":126083,"significance":"If the reported results hold, this is a useful step toward automated PoV test generation: the multi-stage reasoning workflow is language-agnostic, the benchmark is a public resource, and the comparison uses the same model and budget for both systems. The paper also ships reproducible Docker-based builds, detailed prompts in the appendix, and explicit failure-stage accounting, which are strengths. The headline improvement over CodeAct is meaningful and the problem is clearly hard: only 16 of 100 vulnerabilities receive a PoV test. However, the evaluation has load-bearing gaps: correctness is never checked in the fixed state, each agent is run only once despite LLM stochasticity, and the ablation study is restricted to the 16 successful projects. These gaps mean the quantitative claims are plausible but not yet established. The benchmark and agent design are valuable regardless, and the limitations are partly acknowledged in Section 7, but the paper needs further evaluation work before the central claims can be accepted.","major_comments":[{"comment":"The correctness definition omits the fixed-state pass check, which is a necessary property of a PoV test. Section 4.4 requires only that the test builds at the vulnerable commit, exits non-zero, reaches a method touched by the fix, and passes a manual CWE-specific inspection. The Introduction defines a PoV test as failing when the vulnerability exists and succeeding when it is fixed, and Appendix Listing 5 explicitly instructs the agent that the test should PASS when the vulnerability is fixed, yet the evaluation never verifies this. Section 7 concedes this absence. Consequently, a test that fails at the vulnerable commit because of a wrong API assumption, a missing dependency, or an environment issue is indistinguishable from a test that actually demonstrates the vulnerability. The CWE-Bench-Java subset was already filtered using fixed-commit builds, so adding a fixed-state run is feasible for at least the Java portion. The authors should either add this check and report the fixed-state pass rate, or relabel the results as 'vulnerable-state failing tests' and temper the abstract and contribution claims accordingly.","section":"Section 4.4 and Section 7"},{"comment":"The ablation study is run only on the 16 projects for which the full FaultLine configuration succeeded, which is selection on the dependent variable. This design cannot support the claim that both flow and branch reasoning are essential: the configurations without flow or branch reasoning are never given the opportunity to succeed on the other 84 projects, so the comparison only shows what happens on the already-solved subset. Moreover, the counts 9, 11, and 16 differ by only a few examples and are based on a single run per configuration, so the differences may be within stochastic noise. Please run the ablations on a representative sample of the full 100-project benchmark, or at least on a fixed random subset that is not defined by the full system's successes, and report repeated runs.","section":"Section 5.3, Table 2"},{"comment":"The headline result of 16 versus 9 correct PoV tests rests on a single run of each stochastic LLM agent on each project. No temperature, seed, or randomization control is reported, and no statistical significance test or confidence interval accompanies the 7-project difference. With 100 paired instances and small counts, the observed gap could easily shift under repeated sampling. Please report multiple runs (e.g., 3 to 5 seeds) and a paired significance test, or otherwise justify that the stochasticity does not affect the comparison.","section":"Section 5.1"},{"comment":"The manual inspection criteria are the same CWE-specific criteria that are inserted verbatim into the test-generation prompt (Figure 4 and the corresponding appendix prompt). As a result, the manual check is not an independent oracle: it verifies that the agent followed the instructions it was given, rather than independently validating that the generated test demonstrates an exploit. The automated checks alone are too weak, because any non-zero exit at the vulnerable commit is accepted, so the manual inspection is the only guard against shallow tests. The authors should make the inspection more independent, report who performed it, and state whether the evaluators were blind to which system generated each test; this is especially important because the fixed-state check is absent.","section":"Section 3.3 and Section 4.4"}],"minor_comments":[{"comment":"The Command Injection row lists a total of 16 even though the two columns contain 6 and 4, which sum to 10; with the printed row totals, the overall sum becomes 106 instead of 100. Please correct the row and verify all totals.","section":"Table 1"},{"comment":"The vulnerable-function coverage counts are inconsistent: the Abstract reports 31 versus 21, the Section 5.1 summary reports 31 versus 19, and the in-text discussion reports 28 versus 17 for CWE-Bench-Java. The values derived from Figure 5's stage counts are again different, so the authors should audit these numbers and use one consistent set.","section":"Abstract and Section 5.1"},{"comment":"The text says 'CWE-94 corresponds to Code Injection vulnerabilities, such as SQL injection,' but SQL injection is a distinct CWE (CWE-89). A CWE-94 example such as expression-language injection or template injection would be more accurate.","section":"Section 2.1"},{"comment":"The authors say they selected 30 PrimeVul vulnerabilities at random, but they do not report the random seed or the selection procedure. A reproducible selection process is needed for the benchmark to be reusable.","section":"Section 4.1"},{"comment":"The numeric labels in the funnel diagrams are hard to parse and do not obviously match the prose in Section 5.1 (for example, the 'Succeeded 70 9 63 19 16' label). Please redraw the figure so each stage count and the transition from total to successful tests is explicit.","section":"Figure 5"},{"comment":"The manual inspection is described as 'manual' but the paper does not report how many annotators performed it, what their agreement was, or whether they were blinded to the tool that generated each test. Even without a full inter-rater reliability study, reporting these details would strengthen the evaluation.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution with a useful benchmark and a thoughtful agent design, and the code and dataset release is a plus. The main barrier is evaluation validity: the missing fixed-state oracle, the single-run stochastic evaluation, and the selected-subset ablation all touch the central quantitative claims. These are fixable within the manuscript's scope if the authors can run the fixed-state check (at least for the CWE-Bench-Java portion) and add repeated runs. I also recommend an internal audit of the result tables, since Table 1 and the coverage counts contain arithmetic inconsistencies. If the authors supply fixed-state pass rates, repeated-run evidence, and appropriately qualified claims, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. FaultLine is one of the more honest and usable pieces I've seen on LLM agents for security testing. The staged decomposition—flow tracing, branch-condition extraction, input-condition inference, feedback-driven repair—is a real contribution for PoV generation, which is currently mostly handwork. The 100-vuln Java/C/C++ benchmark with public code and data is genuinely useful, and the appendix prompts make the method reproducible. The authors also admit the big threats: possible data leakage, no fixed-state check, manual inspection, and single runs. That candor earns credit. The citation pattern is appropriate—they cover the vulnerability datasets and concurrent work like PoCGen.\n\nThe soft spots are real, though. The biggest is the PoV definition. The paper says a PoV must pass on the fixed commit, but the evaluation only checks build at the vulnerable commit, non-zero exit, and coverage of a changed method, plus a manual category-specific inspection. The manual inspection catches tests that don't actually exploit anything, but it doesn't check the test would pass after the fix. So a test that crashes for an unrelated reason could be counted as a success. Section 7 acknowledges this and explains why they skipped it—dependency mismatches at the fixed commit, unknown exceptions—but that means the 16 vs 9 gap can't be taken at face value. The stress-test note on this point is accurate. The direction is probably right, because the gap in reaching vulnerable functions (31 vs 19, though the intro says 21) is more robust, but the magnitude is uncertain.\n\nSecond, single-run evaluation with no significance test. Sixteen vs nine on 100 instances is not obviously outside noise for stochastic LLM agents. Repeated runs and a McNemar test would settle it.\n\nThird, the RQ3 ablation runs only on the 16 full-system successes. That's selection bias, and the results are odd: branch-only gets 9, neither gets 11. The paper's explanation is hand-wavy. The claim that both components are essential isn't as clean as the abstract suggests.\n\nAlso, the reach-vulnerable-functions numbers disagree between the intro (21) and Section 5.1 (19). Minor, but sloppy.\n\nWho's this for? People building LLM agents for vulnerability reproduction, and anyone designing benchmarks for code reasoning. It deserves a serious referee; the evaluation needs strengthening, not a rewrite. The core workflow and benchmark are worth engaging with. If I were editing, I'd send it to review and ask for repeated runs, a fixed-state check where feasible, and a cleaner ablation.","headline":"A genuinely new staged-agent pipeline for PoV tests plus a reusable 100-vuln benchmark, but the 16-vs-9 headline is only as strong as a PoV criterion that never checks the fixed state; worth a serious referee.","tokens_in":19240,"tokens_out":7497,"would_cite":true,"duration_ms":69609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A staged LLM workflow writes correct proof-of-vulnerability tests for 16 of 100 known vulnerabilities.","keywords":["proof-of-vulnerability tests","LLM agents","test generation","data flow reasoning","control flow reasoning","software vulnerabilities","agentic workflow"],"falsifier":"Run each of the 16 accepted FaultLine tests against the corresponding fixed commit (the commit that patches the vulnerability) and count how many build successfully and exit with code 0. If most of them still fail on the fixed version, the claim that these are genuine proof-of-vulnerability tests, and with it the 16-versus-9 headline, is weakened.","tokens_in":18234,"feed_emoji":"🧪","tokens_out":8321,"duration_ms":76667,"temperature":0.7,"pith_summary":"Vulnerability reports rarely include a proof-of-vulnerability (PoV) test: a failing test that demonstrates the exploit, validates a fix, and guards against regressions. Writing such a test is hard because it requires tracing attacker-controlled data from an external entry point to the vulnerability sink, and constructing an input that satisfies every branch condition along the way. FaultLine claims that an LLM agent can do this substantially better by making those two reasoning steps explicit, first extracting a source-to-sink data flow, then deriving the input constraints implied by the branch conditions, before generating and repairing the test in a feedback loop. On a curated benchmark of 100 known vulnerabilities in Java, C, and C++, it reports correct PoV tests for 16 projects versus 9 for the CodeAct 2.1 baseline, a 77% relative improvement. The paper reads the result as evidence that hierarchical reasoning helps LLM agents on this task, while noting that the problem remains largely unsolved.","feed_headline":"LLM workflow writes exploit tests for 16 of 100 known flaws","feed_subtitle":"It beats a general-purpose agent 16 to 9 by reasoning about data flow and branch conditions first.","key_machinery":"The machinery is a three-stage hierarchical reasoning workflow, with the intermediate artifacts, not the final test, doing the work. Stage 1 (data flow reasoning) produces a labeled source-to-sink path from the vulnerability report and repository exploration tools such as find and grep. Stage 2 (control flow reasoning) converts that path into a sequence of branch conditions (if, switch, try-except) and then into a distilled set of input conditions, so the generated test input can walk the tightrope through the branches. Stage 3 generates a test that must actually execute the vulnerable code according to category-specific success criteria (for example, a CWE-94 test must inject code that is executed), runs it in Docker, and repairs it from the build or run output in a loop. The workflow deliberately uses no language-specific static or dynamic analysis, which is what lets the same prompts run across Java, C, and C++.","core_discovery":"The central claim is that decomposing PoV test generation into explicit reasoning stages lets an LLM agent outperform a general-purpose coding agent on real-world vulnerabilities. FaultLine first prompts the model to reconstruct the vulnerable flow from source to sink, labeling each program point with its file, variable, and role. It then prompts the model to enumerate every branch condition along that flow and to compose them into a list of concrete constraints an input must satisfy. Finally, a test-generation prompt, seeded with the flow and the input constraints, writes a test in a Docker-contained project and repairs it iteratively from build and run feedback. The paper reports 16 correct PoV tests on 100 vulnerabilities, versus 9 for the CodeAct baseline, and 31 tests that reach the vulnerable function, versus 19; ablations show that removing either the flow or the branch reasoning component drops the success count to between 9 and 11.","pith_inferences":["We infer that the reported 16-vs-9 gap is an upper bound on genuine PoV quality: since the evaluation never runs the tests against the patched code, some of the 16 may be failing for build, dependency, or environment reasons rather than because they trigger the vulnerability; the paper's own limitation discussion grants this possibility.","We infer that the biggest wins from the workflow come from the branch-condition stage, since the motivating failure of the baseline (an input with the wrong number of cron fields) is exactly what the input-constraint step catches; a testable extension would be to replace that step with a lightweight constraint solver or symbolic execution and measure whether the gap widens or shrinks.","We infer that the same decomposition could be applied to other security tasks, such as regression-test generation for patched code or fuzzing seed construction, where the value of an input depends on controlling a long path through branches."],"forward_implications":["Correct PoV test generation becomes a staged reasoning problem: future agent designs should expose flow paths and branch-derived input constraints as first-class artifacts rather than expecting end-to-end generation to work.","The same prompt structure, without language-specific tools, transfers across Java, C, and C++, so the workflow can be pointed at other CWE categories and languages with minimal changes.","Reaching the vulnerable function (31 cases) is much easier than building a correct exploit test (16 cases), so improving payload construction and semantic verification is the next bottleneck to attack.","The released 100-vulnerability benchmark gives the community a reproducible way to compare PoV-generation agents on a harder task than general bug reproduction."],"supporting_citations":[{"why":"Supplies the 70 Java vulnerabilities with build metadata and fix-commit information that make up most of the evaluation set.","marker":"[20]"},{"why":"Supplies the 30 C and C++ vulnerabilities sampled for the second benchmark.","marker":"[11]"},{"why":"Defines the CodeAct 2.1 baseline agent against which FaultLine's 16 vs 9 success counts are measured.","marker":"[29]"},{"why":"Provides the OpenHands framework that hosts the baseline agent under the same model and budget constraints.","marker":"[31]"},{"why":"Provides the National Vulnerability Database reports that FaultLine consumes as input for flow and branch reasoning.","marker":"[24]"}],"fun_headline_variants":["FaultLine agents reason source-to-sink to write exploit tests","LLM workflow turns 100 flaws into 16 working PoV tests","Hierarchical reasoning boosts LLM exploit-test hits 77%","Data-flow reasoning helps LLMs generate more vulnerability tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The counting rule for a correct test is that it builds at the vulnerable commit, exits non-zero when run, and calls a method touched by the fix, but there is no check that the same test passes once the vulnerability is patched, so a test that fails for unrelated reasons is still counted as a success.","fun_headline_variants_meta":{"raw":{"variants":["FaultLine agents reason source-to-sink to write exploit tests","LLM workflow turns 100 flaws into 16 working PoV tests","Hierarchical reasoning boosts LLM exploit-test hits 77%","Data-flow reasoning helps LLMs generate more vulnerability tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2561,"prompt_tokens":1071,"completion_tokens":1490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":1418}},"tokens_in":687,"tokens_out":1490,"duration_ms":11735,"temperature":1.0,"reasoning_tokens":1418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:36:33.462260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the 16 accepted FaultLine tests against the corresponding fixed commit (the commit that patches the vulnerability) and count how many build successfully and exit with code 0. If most of them still fail on the fixed version, the claim that these are genuine proof-of-vulnerability tests, and with it the 16-versus-9 headline, is weakened.","supporting_citations":[{"cited_title":"Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H","cited_arxiv_id":null,"evidence_quote":"Provides the OpenHands framework that hosts the baseline agent under the same model and budget constraints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the National Vulnerability Database reports that FaultLine consumes as input for flow and branch reasoning."}],"review_version":1}