{"id":"9ddb3c49-3c6c-4b82-8f95-955a4279521a","arxiv_id":"2608.04217","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"POVGEN generates proof-of-vulnerability inputs for roughly three quarters of real-world C/C++ memory-safety CVEs using fine-tuned open-weight LLMs guided by symbolic path analysis.","lead":"This paper presents POVGEN, a system that automatically creates inputs that trigger known software vulnerabilities, using fine-tuned open-weight language models combined with symbolic path analysis. It reports successfully generating working proof-of-vulnerability inputs for about 79% of benchmark vulnerabilities and 75% of 250 real-world CVEs that lack public exploits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ5 never defines what makes a PoV 'valid' for the 250 CVEs without public PoVs; the 74.80% real-world headline depends on an unstated success criterion that cannot use ground-truth matching.","rationale":"The reader's weakest assumption was that the fine-tuned constraint models generalize to real-world CVE constraints. That is a legitimate external-validity concern, but the more immediate threat to the central claim is measurement validity: Section V-F reports the headline 74.80% real-world success rate without specifying how 'valid' is judged when no ground-truth PoV exists. The paper's strict ARVO criterion in Section V-B cannot apply to the 250-CVE set by construction, and no alternative criterion is given. This is an internal omission, not merely a distributional extrapolation risk. It matters because the 250-CVE result is the paper's main claim of practical scalability beyond curated benchmarks; if successes were scored as 'any sanitizer crash' or by an unstated heuristic, the number could be substantially inflated. The paper does have independent support: the ARVO evaluation uses ground-truth PoVs with a strict matching criterion, the fine-tuning and evaluation sets are disjoint as described, and the ablations show the components contribute meaningfully. Those strengths make the central ARVO claim plausible, but they do not transfer automatically to the unvalidated RQ5 number. A concrete re-check with blinded manual verification and patched-version re-runs would settle whether the concern lands. If the authors can document a strict RQ5 validation procedure, the concern dissolves and the conditional verdict can stand; if not, the 74.80% claim should be downgraded or marked unverified. Since the reader already recommended CONDITIONAL, my read does not change the verdict, but it adds a specific condition that should be satisfied before acceptance.","tokens_in":19228,"tokens_out":8366,"duration_ms":75004,"concrete_test":"Take 50 randomly sampled successes from the 250-CVE set. For each, (1) run the generated PoV against the patched version of the same project; a valid PoV must not trigger the same sanitizer failure on the fixed build. (2) Have two independent security researchers, blinded to the CVE ID, verify from the CVE description and patch that the PoV triggers the specific documented vulnerability rather than any crash. Pre-register thresholds (e.g., >10% rejected or >5% still crashing on patched build means the success criterion was too lax). Also ask the authors to state the exact RQ5 validation procedure; if no such procedure exists, the 74.80% claim is unverifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B defines a reproduction as successful only if the sanitizer error type and crash location match the ground-truth PoV. Section V-F then reports 74.80% 'valid PoVs' on 250 real-world CVEs explicitly chosen because they have no public PoVs. The paper never states the success criterion for this set. It cannot reuse ground-truth matching, and it does not say whether the generated input was run against both the vulnerable and patched builds, whether the crash site was required to match the CVE/patch, or whether independent confirmation was used. If the criterion is merely 'any sanitizer crash on the vulnerable binary,' then unrelated bugs inflate the headline number. If the criterion is 'crash site matches the patch,' the motivating example (Fig. 1, CVE-2022-23537) shows the manifestation point can be hundreds of lines away from the patch, so this criterion would reject genuine PoVs or accept wrong ones. Either way, the 74.80% number is not interpretable from the manuscript as written. This is load-bearing because the ARVO 78.98% result has ground truth and cannot validate the RQ5 claim, which is the paper's central real-world extension.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"POVGEN is a neuro-symbolic pipeline for generating Proof-of-Vulnerability (PoV) inputs for disclosed C/C++ memory-safety vulnerabilities. It combines (1) a fine-tuned open-weight LLM for vulnerability manifestation localization, (2) anchored random walks over the interprocedural CFG to extract candidate paths, and (3) fine-tuned constraint-extraction and constraint-solving models with an SMT fallback to produce concrete inputs. The paper reports 78.98% success on a 490-sample ARVO subset in patch-guided mode and 65.10% without patch, outperforming fuzzing baselines (up to 50.20%) and KLEE (2.45%). On 250 real-world CVEs without public PoVs, it reports 74.80% valid PoVs. The generated PoVs allegedly exposed six flawed patches and five previously unreported vulnerabilities.","tokens_in":19481,"tokens_out":7910,"duration_ms":61675,"significance":"If the results hold, POVGEN would be a substantial step toward automating PoV generation at scale with locally deployable models. The ARVO evaluation uses a strict, ground-truth-matched success criterion; the 490-sample sampling with confidence interval, disjointness checks against fine-tuning data, and extensive ablations (Tables IV–VI) are methodological strengths. The decomposition of the problem into localization, path exploration, and constraint reasoning, plus the SMT fallback, is a sensible and potentially generalizable design. However, the real-world claim (RQ5) lacks a defined success criterion, so the paper's headline external-validity result is not yet interpretable.","major_comments":[{"comment":"The success criterion for the 250 real-world CVEs without public PoVs is never defined. Section V-B defines success for ARVO as a sanitizer error whose vulnerability type and crash location match the ground-truth PoV, but RQ5 has no ground truth by construction. The manuscript does not state whether the generated input was run against both the vulnerable and patched builds, whether the crash type or site was required to align with the CVE or patch, or whether independent confirmation was used. Without this, the 74.80% figure cannot be distinguished from 'any sanitizer crash on the vulnerable binary' and is uninterpretable as a vulnerability-specific success rate. A concrete success criterion for this set must be specified, and the evaluation re-run or re-reported under it. The motivating example in Fig. 1 also shows that the manifestation point can be far from the patch, so requiring crash-site match to the patch would need careful justification against the paper's own definition of reproduction.","section":"§V-F (RQ5), Table VII"},{"comment":"The abstract states: 'On 250 real-world CVEs without public PoVs, it generates valid PoVs for 74.80% of cases and reproduces 65.1% when without patch information.' In the body, the 65.10% patch-free figure is reported only for the 490 ARVO vulnerabilities (Table I, 'Overall' row), while §V-F and Table VII report only the 74.80% value for the 250-CVE set. No patch-free experiment is reported for the real-world set. The abstract therefore either misattributes the ARVO result or announces an experiment that is absent from the evaluation. This must be corrected, and if the patch-free claim is meant for the CVE set, the experiment must be added.","section":"Abstract and §V-B/§V-F"},{"comment":"The threshold MAX_ITER_MLM for slice expansion is set to three 'based on statistics from InterPVD [24] and ARVO [6]'. ARVO is one of the two evaluation datasets used in RQ1–RQ4. Choosing a pipeline hyperparameter from the evaluation distribution risks optimistic results, even if the effect is likely modest. The paper should either fix this threshold using only training or validation data, or report sensitivity of the main result to this parameter.","section":"§III-B1, Algorithm 1"}],"minor_comments":[{"comment":"There are numerous rendering artifacts and missing spaces, e.g., 'AProof-of-Vulnerability' in the abstract, 'viasemantic' and 'viaopen' in §I, and section headers such as 'V .' and 'F .'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"Algorithm 3 uses MAX_ITER_WALK, but its value is not specified; §IV says 'extract up to 200 candidate paths', so state explicitly whether MAX_ITER_WALK equals 200 or is a separate parameter.","section":"Algorithm 3 and §IV"},{"comment":"The column header 'No S1 VulTrigger S1' is ambiguous; clarify that this setting replaces the fine-tuned MLM with VulTrigger as the localization step.","section":"Table V"},{"comment":"The 52-sample subset used for direct frontier-LLM comparison is selected by context-window fit; report how representative this subset is of the full 490-sample ARVO set (e.g., project size and CWE distribution) to allow assessment of selection bias.","section":"§V-C, Table IV"},{"comment":"The criteria for classifying a patch as 'incomplete' versus 'incorrect' are not defined; specify how the patched builds were obtained and how unrelated crashes were ruled out before labeling a patch as flawed.","section":"§VI-B and Table VIII"}],"recommendation":"major_revision","confidential_remarks":"The reader's report correctly identifies the RQ5 criterion gap; in my reading, the abstract's placement of the 65.1% figure next to the 250-CVE sentence is a genuine misattribution, not a stylistic issue. The use of ARVO statistics to set MAX_ITER_MLM should also be clarified in revision. The ARVO result itself is well-supported, so the paper is fixable within scope. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the ARVO result is the real contribution; the 74.80% real-world CVE number is the load-bearing weakness.\n\nWhat's new: the specific combination of semantic manifestation localization, anchored random walks on ICFGs, and fine-tuned open-weight constraint models with an SMT fallback is a sensible decomposition of PoV generation, and the task specialization of open-weight models is a useful direction. The ARVO evaluation is well done: 490 sampled vulnerabilities, a strict success criterion (sanitizer type and crash location must match the ground truth PoV), fine-tuning data disjoint from the evaluation sets, and thorough ablations. The 78.98% patch-guided and 65.10% patch-free rates are credible, and the cost report (98 minutes per sample, $0.04 in API calls) is honest. I also credit the explicit disclosure that ARVO vulnerabilities were originally fuzzer-found, which makes the fuzzing comparison fair.\n\nThe weaknesses are fixable but one is load-bearing. The RQ5 claim — 74.80% valid PoVs on 250 real-world CVEs without public PoVs — never defines what counts as a valid PoV for that set. Ground-truth matching is impossible, and the paper doesn't state whether the input was run against both vulnerable and patched builds, whether the crash site had to match the CVE, or whether independent manual confirmation was done. With no criterion, the number is uninterpretable and could capture unrelated crashes. That matters because this is the headline real-world result. The authors need to specify the criterion, ideally with patch-validation checks and manual inspection.\n\nTwo smaller issues: the abstract implies the 65.1% patch-free rate applies to the 250 CVEs, but it's the ARVO patch-free rate; and 'no per-sample API cost' ignores the $0.04/sample spent on GLM-5.2 in slicing. Both are easy to correct. Also, the artifact link is a bare Figshare URL with no code or commit hash, which is weak for a reproducibility-focused paper.\n\nThe memorization risk is real but mitigated: the ablations show base Llama and Gemini underperform the fine-tuned models by a large margin, and the fine-tuning data is disjoint from evaluation. I'd accept that argument.\n\nWho should read it: researchers working on vulnerability reproduction, PoV generation, or LLM-guided program analysis. It deserves peer review, but with a major-revision request focused on the RQ5 success criterion and the abstract's overstatements. I'd bring it to reading group; there's a good methodological lesson about under-specified success metrics when ground truth is absent.","headline":"The ARVO numbers are credible; the 74.80% real-world claim lacks a defined success criterion and needs revision.","tokens_in":20042,"tokens_out":4387,"would_cite":true,"duration_ms":33038,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"POVGEN claims that a three-stage neuro-symbolic pipeline—manifestation localization, anchored path exploration, and LLM-guided constraint solving with an SMT fallback—generates valid proofs of vulnerability for 78.98% of benchmarked…","keywords":["proof-of-vulnerability generation","neuro-symbolic","manifestation localization","path-sensitive exploration","LLM-guided constraint solving","open-weight models","CVE reproduction","patch validation"],"falsifier":"Run POVGEN on a fresh set of CVEs from projects not represented in its training data and see whether the success rate stays near 74.80%; if it drops sharply and the fallback solver rescues few of the failures, the generalization claim is not supported.","tokens_in":18987,"feed_emoji":"🛡️","tokens_out":8633,"duration_ms":68285,"temperature":0.7,"pith_summary":"Disclosed vulnerabilities usually arrive with a patch but not with a concrete input that triggers the flaw, so the vulnerability and its fix remain hard to validate. This paper argues that such inputs—proofs of vulnerability—can be generated automatically by a three-stage pipeline: localize where the vulnerability manifests, explore a small set of program paths that reach that point, and extract and solve the path constraints a triggering input must satisfy. The system, POVGEN, performs the constraint reasoning with two fine-tuned open-weight models backed by an SMT solver, keeping per-sample model cost near zero for the core constraint steps. On a 490-case sample of a recent memory-safety benchmark it succeeds on 78.98% of cases in patch-guided mode, versus up to 50.20% for fuzzing and 2.45% for symbolic execution, and on 250 real-world CVEs without public PoVs it reaches 74.80%. If the result holds, disclosed vulnerabilities can be turned into reproducible evidence, patch-validation tests, and regression tests at scale without relying on proprietary model APIs.","feed_headline":"Pipeline generates triggering inputs for 79% of disclosed flaws","feed_subtitle":"Neuro-symbolic path search plus open-weight models beats fuzzing and symbolic execution on C/C++ memory-safety CVEs.","key_machinery":"The load-bearing mechanism is the pairing of three fine-tuned open-weight models with an SMT fallback: a Manifestation Localization Model (MLM) that predicts the line where the vulnerability manifests, a Constraints Extraction Model (CEM) that converts a candidate path into SMT constraints, and a Constraints Solving Model (CSM) that emits a Python script producing the triggering input. The exploration that feeds them is an anchored random walk on the interprocedural control-flow graph, biased by backward shortest-path distances toward the manifestation point (and the patch location when available), which turns whole-program search into a focused reachability problem over a few dozen candidate paths.","core_discovery":"POVGEN's central claim is that vulnerability-targeted input synthesis becomes tractable when the search is anchored at the site where the vulnerability actually manifests, rather than at the patch location, because patches often sit far from the triggering condition. Given a target program and optionally a patch, POVGEN first uses a fine-tuned localization model to mark the manifestation point, then runs anchored random walks over the interprocedural control-flow graph to collect a small set of candidate paths from program entry to that point, and finally converts each path into SMT constraints via a constraint-extraction model and into a concrete PoV via a constraint-solving model, with an SMT solver as fallback. The paper reports that this decomposition lets task-specialized open-weight models match frontier commercial LLMs on the core constraint-reasoning subtasks while running locally, and that the end-to-end system outperforms both fuzzing and symbolic-execution baselines on a common benchmark and succeeds on 74.80% of real-world CVEs without public PoVs. Applying the generated PoVs to patched versions of those CVEs surfaced six flawed patches, all later fixed, and five previously unreported vulnerabilities in newer project versions.","pith_inferences":["Editorial inference: the generalization claim is only as strong as the training distribution; the constraint models were fine-tuned on benchmark suites plus four synthetic templates, so a fresh CVE whose path constraints do not resemble those motifs may fall through to the SMT fallback and fail if the constraints are not solver-friendly.","Editorial inference: the reported 74.80% real-world figure is measured on projects whose build environments and harnesses already exist; applying POVGEN to arbitrary new codebases would add non-trivial setup cost that the paper explicitly leaves out of scope.","Editorial inference: a sharper ablation would swap CSM for a deterministic input generator that ignores the constraints and retains only the SMT fallback; the paper's partial solver-only comparison suggests the solver alone is weaker, but the exact contribution of the solving model is not isolated end-to-end."],"forward_implications":["If POVGEN's success rate holds on new data, every patched CVE can be checked by replaying a generated PoV against the patched build, which is exactly how the paper identified six flawed patches.","For CVEs without public PoVs, defenders and vendors can obtain a concrete triggering input without waiting for an exploit disclosure, making the PoV available as a regression test.","Because the core constraint models are pinned and run locally, vulnerability research no longer depends on the continued availability or pricing of proprietary models for this task.","The paper's modular design implies the same localization-exploration-constraint pipeline can be adapted to other vulnerability classes and languages, though the evaluation only covers C/C++ memory-safety."],"supporting_citations":[{"why":"Supplies the benchmark of reproducible memory-safety vulnerabilities with ground-truth triggering inputs used for the main effectiveness evaluation.","marker":"[6]"},{"why":"Serves both as the symbolic-execution baseline that POVGEN outperforms and as the engine that generates path constraints and satisfying assignments for constraint-reasoning fine-tuning.","marker":"[12]"},{"why":"Provides the labeled manifestation-point dataset for localization fine-tuning and the empirical depth bound used in on-demand slicing.","marker":"[24]"},{"why":"Contributes static-analysis benchmark programs used to build the constraint-extraction and constraint-solving training data.","marker":"[34]"},{"why":"Supplies memory-safety test samples that make up the bulk of the constraint-reasoning fine-tuning data.","marker":"[35]"},{"why":"Implements the SMT solving used both to create training PoVs in Phase 1 and as the fallback solver in Step 3.","marker":"[32]"},{"why":"Is the directed-fuzzing baseline whose patch-targeted search is compared against POVGEN's anchored exploration.","marker":"[39]"},{"why":"Represents the closed-model LLM-based PoC generation approach that POVGEN positions itself against on cost and controllability.","marker":"[18]"}],"fun_headline_variants":["Neuro-symbolic PoV generator hits 75% on real CVEs","Six flawed patches exposed by synthetic PoVs","POVGEN finds PoVs for 65% of CVEs with no patch info","Open-weight models match paid LLMs in PoV reasoning","POVGEN uses open-weight models to crack 79% of benchmark vulns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fine-tuned models that translate a candidate execution path into logical conditions, and then into a concrete input, were trained on benchmark programs with a few added synthetic patterns; the whole result depends on those models handling the conditions found in real-world vulnerabilities they never saw during training.","fun_headline_variants_meta":{"raw":{"variants":["Neuro-symbolic PoV generator hits 75% on real CVEs","Six flawed patches exposed by synthetic PoVs","POVGEN finds PoVs for 65% of CVEs with no patch info","Open-weight models match paid LLMs in PoV reasoning","POVGEN uses open-weight models to crack 79% of benchmark vulns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001184,"raw_usage":{"total_tokens":4951,"prompt_tokens":1068,"completion_tokens":3883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":3790}},"tokens_in":684,"tokens_out":3883,"duration_ms":24274,"temperature":1.0,"reasoning_tokens":3790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:13:06.777912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run POVGEN on a fresh set of CVEs from projects not represented in its training data and see whether the success rate stays near 74.80%; if it drops sharply and the fallback solver rescues few of the failures, the generalization claim is not supported.","supporting_citations":[{"cited_title":"KLEE: unassisted and auto- matic generation of high-coverage tests for complex systems programs","cited_arxiv_id":null,"evidence_quote":"Serves both as the symbolic-execution baseline that POVGEN outperforms and as the engine that generates path constraints and satisfying assignments for constraint-reasoning fine-tuning."},{"cited_title":"On the effectiveness of function-level vulnerability detectors for inter- procedural vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"Provides the labeled manifestation-point dataset for localization fine-tuning and the empirical depth bound used in on-demand slicing."},{"cited_title":"Software verification: 10th comparative evaluation (SV- COMP 2021),","cited_arxiv_id":null,"evidence_quote":"Contributes static-analysis benchmark programs used to build the constraint-extraction and constraint-solving training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies memory-safety test samples that make up the bulk of the constraint-reasoning fine-tuning data."},{"cited_title":"Z3: An efficient smt solver,","cited_arxiv_id":null,"evidence_quote":"Implements the SMT solving used both to create training PoVs in Phase 1 and as the fallback solver in Step 3."},{"cited_title":"Directed greybox fuzzing,","cited_arxiv_id":null,"evidence_quote":"Is the directed-fuzzing baseline whose patch-targeted search is compared against POVGEN's anchored exploration."}],"review_version":1}