{"id":"ff5a94a0-9412-42ff-b33b-4a79a344bd49","arxiv_id":"2608.06471","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CyberForge generates and validates 1,034 synthetic repository-level vulnerabilities that, when used to fine-tune LLM agents, improve real-world patch repair by up to 14.7 points and transfer across languages.","lead":"Researchers built CyberForge, a system that automatically injects security flaws into open-source C/C++ projects and verifies each flaw with a test that crashes only the modified version. The resulting 1,034 verified examples improved LLM agents' ability to patch real vulnerabilities, with gains that carried over to other programming languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Project-level overlap between CyberForge's OSS-Fuzz training projects and SEC-bench's OSS-Fuzz instances is unreported and likely substantial, undermining the SEC-bench generalization claim.","rationale":"The reader's weakest assumption is exactly the one I judge most load-bearing. The paper's only explicit disjointness statement is that PatchEval contains languages 'none of which appear in our C/C++ corpus' (Section 4); there is no project-level disjointness statement for SEC-bench. Since both CyberForge and SEC-bench's 'oss split' draw from OSS-Fuzz, the potential overlap is structural rather than hypothetical. This matters because the central claim is not just 'the corpus has valid instances' but 'training on it transfers to real benchmarks'; if the benchmark shares projects, the transfer claim is confounded. I do not see an internal inconsistency that would justify rejection: the differential validation, ablations, and scaling curve are supportive, and PatchEval provides some cross-language evidence. However, the missing overlap report is a concrete, addressable condition. I therefore keep the reader's CONDITIONAL recommendation rather than moving it.","tokens_in":23747,"tokens_out":8446,"duration_ms":70418,"concrete_test":"Extract SEC-bench metadata (24 project names, 150 instance base commits and touched files) from the SEC-bench release, and CyberForge metadata (80 project names and secure_base_commit from project.json in the released corpus). Compute: (a) the number of SEC-bench projects in CyberForge's 80; (b) the number of SEC-bench instances whose project is in CyberForge; (c) the number of SEC-bench instances that touch at least one file also touched by any CyberForge injection. Then rerun the six CyberForge fine-tuned students and the three base models on the SEC-bench subset whose project is not in CyberForge, and report Strict and PoV scores for that disjoint subset. If the +3.3 to +14.7 SEC-bench gains persist on the disjoint subset, the contamination concern is resolved; if they vanish or fall within noise, the central claim needs to be restated as project-overlap-sensitive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning on CyberForge trajectories improves SEC-bench patch repair, and that the gains are not memorization. The load-bearing condition is that SEC-bench is disjoint from the training corpus at the project/instance level. That condition is asserted for PatchEval (by language) but never for SEC-bench. CyberForge deliberately builds on OSS-Fuzz projects (Section 3.1), and Appendix A.4 reveals that SEC-bench's 300 real instances are split into '200 from the cve split and 100 from the oss split'—that is, a substantial part of SEC-bench is OSS-Fuzz bugs from the same project pool CyberForge uses. The paper reports all 80 CyberForge projects in Table 5 but no comparison against SEC-bench's 24 projects. If common projects exist (likely: brotli, libarchive, libpng, openssl and similar appear in both), training on synthetic vulnerabilities in those repositories can inflate repair via project-specific file and format familiarity, or via near-duplicate OSS instances, rather than via a general vulnerability-repair capability. The PatchEval cross-language gains mitigate the concern for transfer, but the headline SEC-bench numbers (+3.3 to +14.7 in all six configurations) remain unvalidated against a disjoint subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CyberForge, a framework that generates repository-level vulnerability-injection training data for cybersecurity agents. CyberForge injects weaknesses into real C/C++ OSS-Fuzz projects, requires each injected build to pass the project's unit tests, and validates each instance by a differential proof-of-vulnerability oracle (PoV triggers on the injected build but not on the clean one). The released corpus contains 1034 validated vulnerabilities across 80 projects and 63 CWE categories. The authors collect agent trajectories on these instances using two teachers (GPT-5.4-mini and Gemma 4 31B) and fine-tune three Gemma 4 students (E4B, 12B, 31B). They report that all six student-teacher configurations improve SEC-bench patch repair by +3.3 to +14.7 percentage points, with the strongest 31B student reaching 72.7% against its teacher's 74.0%, and that all configurations also improve on PatchEval, a Go/JavaScript/Python benchmark disjoint in language from the C/C++ training corpus. The paper also claims that the injected edits have edit-locality statistics similar to real CVE patches under a real-versus-real noise floor.","tokens_in":23987,"tokens_out":5489,"duration_ms":48523,"significance":"If the empirical claims hold, this is a useful and timely contribution: it decouples security training-data generation from the rate of public CVE disclosure, provides reproducible containerized environments, validates every instance with an executable differential oracle rather than static heuristics, and shows consistent downstream gains across model scales and teachers. The cross-language PatchEval results and the self-distillation results are valuable pieces of evidence. The paper is also transparent about pipeline costs and reports a detailed cost accounting. The two main open issues are (1) unquantified project-level overlap between the CyberForge OSS-Fuzz training projects and the SEC-bench evaluation instances, which is load-bearing for the headline in-domain claim, and (2) the absence of repeated-seed or significance information behind the 'all six configurations improve' statement. These issues are fixable and do not undermine the framework's potential, but they must be resolved before the central claims can be accepted at face value.","major_comments":[{"comment":"The paper never reports the project-level overlap between the 80 CyberForge training projects listed in Table 5 and the 24 SEC-bench projects. Because CyberForge deliberately targets OSS-Fuzz projects (§3.1) and Appendix A.4 states that SEC-bench's 300 real instances are split into '200 from the cve split and 100 from the oss split,' the two corpora plausibly share several projects (for example brotli, libarchive, libpng, or openssl). If common projects exist, the SEC-bench repair gains of +3.3 to +14.7 points (§5.1, Table 2) could reflect project-specific familiarity (file layout, build conventions, format-specific input quirks) rather than a general vulnerability-repair capability. This is load-bearing for the central claim that the gains are not memorization. Please report the exact project-level and instance-level overlap and re-evaluate on the disjoint subset of SEC-bench; PatchEval provides partial transfer evidence, but it does not validate the headline in-domain numbers on a disjoint C/C++ set.","section":"§3.1, Appendix A.4, Table 5"},{"comment":"Every fine-tuned model is trained once with seed 42 (Table 13) and evaluated on fixed benchmark sets, so there is no variance information and no significance testing behind the statement that all six student-teacher configurations improve. The smallest reported gains (+3.3 and +4.7 percentage points at E4B) are within the range that a single training run could flip under a different initialization or trajectory-sampling order, particularly given the small absolute number of solved instances (the E4B base solves only 9 of 150 SEC-bench instances). Please report multiple seeds (or permutation/bootstrap intervals over benchmark instances) and show that the improvements hold with accompanying uncertainty intervals.","section":"§5.1, Table 2, Appendix D"},{"comment":"The 'near-realistic' edit-locality claim is asserted from a single KS distance on functions changed (0.165 versus a real-to-real baseline of 0.190), but the same appendix reports injected-to-real distances of 0.123 for files touched and 0.243 for hunks, against baselines of 0.065 and 0.205. On the files-touched axis the injected corpus is nearly twice as far from real patches as two real corpora are from each other, so the statement that 'injected edits broadly resemble real patches' is not uniformly supported by the reported numbers. Please report all five axes with uncertainty estimates and either temper the realism claim or provide a multivariate assessment.","section":"§5.2 and Appendix A.4"}],"minor_comments":[{"comment":"The pie-chart slices are described with percentages in the text but the figure itself does not show the values; adding percentage labels would make the failure-stage analysis self-contained.","section":"Figure 4"},{"comment":"The caption says 'All 100 qualified OSS-Fuzz projects' but several rows show zero P1/P2 instances and zero CWE categories; please distinguish clearly between qualified projects and projects that contributed at least one validated instance, or mark the non-contributing rows explicitly.","section":"Table 5"},{"comment":"The release metadata uses the value 'fuzz_poc_guided' as a producer identifier, while the paper consistently calls this pipeline 'fuzzer-guided'; please standardize the terminology between the paper and the released corpus metadata.","section":"Appendix E.5"},{"comment":"The text says the deleted line is shown in red, but the listing is monochrome; please use an explicit deletion marker or a footnote so the removed line is unambiguous in print.","section":"Listing 1"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially strong systems and dataset contribution for security-agent training. The main technical risk is evaluation contamination via project overlap between the OSS-Fuzz training projects and SEC-bench's OSS-Fuzz-derived instances; this must be addressed with an explicit overlap table and a disjoint-subset evaluation. The paper also makes many strong claims from single-seed runs; the editor should require the authors to add at least a small number of repeated seeds or bootstrap intervals before publication. The manuscript uses several 2026-dated references and model names that may be unfamiliar to readers; the editor may wish to verify that these citations and named systems are publicly verifiable as of the submission date."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. CyberForge is the closest thing yet to a scalable source of execution-validated, repository-level vulnerability training data. Two complementary injection pipelines, 1034 instances across 80 OSS-Fuzz projects, and a differential PoV oracle that requires tests to pass and the trigger to fire only on the injected build: that is real engineering and a real corpus. The downstream story is also credible: all six student-teacher configurations improve on SEC-bench, and the gains transfer to a Go/JS/Python benchmark the corpus never saw. The trajectory-behavior analysis is a nice extra; it actually explains where the gains come from (coverage vs solve-rate) rather than just reporting accuracy.\n\nThe main soft spot is the one the stress test flags. The paper never reports project overlap between the 80 CyberForge projects and SEC-bench's 24 projects. Appendix A.4 says SEC-bench's 300 instances split into 200 from the cve split and 100 from the oss split, and CyberForge intentionally builds on OSS-Fuzz. From Table 5, projects like brotli, libarchive, libpng, and openssl appear in both pools. If those are SEC-bench's OSS instances, training on synthetic bugs in those repositories could inflate repair gains through project-specific familiarity rather than a general repair skill. This matters for the headline SEC-bench numbers. The PatchEval results are out-of-distribution and mitigate the transfer concern, but they do not validate the in-domain claim. The fix is straightforward: report the overlap and rerun the headline numbers on a disjoint project subset. This is addressable, and I do not think it breaks the central claim.\n\nSecond soft spot: every configuration is one fine-tuning run, seed 42, with no variance or significance testing. That is a real weakness for a paper making empirical claims about gains, though the monotonic scaling curve and consistent direction across six setups soften it.\n\nThe realism check (KS distance 0.165 vs a 0.190 real-to-real baseline) is suggestive rather than strong; the two distributions are not far apart, and the baseline is not a no-injection control. Minor.\n\nCitation pattern looks fine. The comparison table is honest about what prior injection work lacks, and the cost accounting is unusually thorough.\n\nBottom line: a solid, useful paper that deserves a serious referee. The overlap question and single-seed runs should be addressed before acceptance. I would cite the corpus if I were working on security-agent training.","headline":"Solid, well-engineered corpus paper; the training-data claim mostly holds, but the SEC-bench generalization claim needs an explicit project-overlap check.","tokens_in":743,"tokens_out":1642,"would_cite":true,"duration_ms":29984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CyberForge claims that execution-validated synthetic vulnerabilities can replace CVE mining as training data, improving repair agents by up to 14.7 points and transferring to other languages.","keywords":["vulnerability injection","proof-of-vulnerability","synthetic training data","LLM agent fine-tuning","patch repair","C/C++ security","differential validation","security benchmarks"],"falsifier":"Compute the project-level overlap between the 80 training projects and the 24 SEC-bench projects (and any PatchEval repositories), retrain after deleting all training instances from overlapping projects, and check whether the SEC-bench and PatchEval improvements survive; if the gains collapse on fully disjoint projects, the generalization claim is falsified.","tokens_in":23563,"feed_emoji":"🛡️","tokens_out":6711,"duration_ms":53195,"temperature":0.7,"pith_summary":"CyberForge's central claim is that security training data for defensive agents does not have to be mined from disclosed vulnerabilities: it can be generated by injecting weaknesses into real C/C++ projects and keeping only instances that are validated by execution. Each accepted instance must compile, pass the project's own unit tests on both the original and injected build, and come with a proof-of-vulnerability that triggers only on the injected build. On this corpus the paper collected agent trajectories from two teachers and fine-tuned three open-weight student models, and all six configurations improved patch repair on the SEC-bench task by +3.3 to +14.7 points, with the largest student reaching 72.7% against its teacher's 74.0%. The same training improved repair on PatchEval, a benchmark in Go, JavaScript, and Python, despite a C/C++ training corpus. If the claim holds, it removes the main bottleneck limiting security-agent training: scarcity of reproducible, executable vulnerability data.","feed_headline":"Patch repair gains 14.7 points from verified synthetic bugs","feed_subtitle":"A 1,034-example C/C++ corpus trains agents that also fix Go, JavaScript, and Python bugs.","key_machinery":"The load-bearing mechanism is a differential proof-of-vulnerability (PoV) oracle, applied to every candidate instance. It enforces two conditions: the injected build must pass the project's existing unit tests, and a generated PoV must trigger on the injected build while leaving the clean build unaffected. This oracle is shared by two complementary generation pipelines: a fuzzer-guided pipeline that targets functions reachable by existing harnesses, and an agentic in-context pipeline that selects sites by static analysis, retrieval of similar historical code patterns, and autonomous exploration. The oracle turns a synthetic bug into a certified training signal: it guarantees the injected project is still buildable and test-passing, that the weakness is latent under normal execution, and that it is demonstrably reachable by an adversarial input. The paper also uses the oracle to decide which teacher trajectories to keep, discarding any trajectory whose claimed success is not verified.","core_discovery":"On its own terms, the paper establishes that vulnerability injection with differential validation is a viable and scalable substitute for CVE mining. The core result is empirical: fine-tuning on oracle-verified trajectories over 1,034 injected vulnerabilities, across 80 projects and 63 weakness categories, improves a repair agent's success rate on the SEC-bench benchmark in every configuration tried. The improvements are not confined to the training distribution; every configuration also improves on PatchEval, whose Go, JavaScript, and Python instances are absent from the C/C++ corpus. The paper also claims the injected edits resemble real CVE patches in locality, with a two-sample Kolmogorov–Smirnov distance of 0.165 against a 0.190 real-versus-real baseline. The mechanism it credits is not better injection alone but the validation machinery: in the agentic pipeline, a naive single pass compiles and passes tests in 68.2% of attempts but validates 0% of them, while the full workflow raises validated yield to 7.5%.","pith_inferences":["A testable extension not explored in the paper is to measure project-level overlap between the training projects and evaluation projects; if gains persist after excluding any overlapping repositories, that would strengthen the generalization claim beyond what is currently reported.","Because validation is execution-based, the same differential oracle could be reused as a filter or reward signal for other synthetic-data generators, including those targeting other languages or non-memory-safety bug classes.","The paper's edit-locality comparison suggests a cheap quality probe: monitor the Kolmogorov–Smirnov distance between injected and real patches over time as the pipeline scales; if distance grows, the corpus may drift toward idiosyncratic, easily caught bugs.","One could test whether the trajectories' benefit comes from the successful-patch examples or from the failed attempts discarded by the oracle; training on filtered failures alone might reveal whether negative signal matters."],"forward_implications":["Corpus growth no longer depends on the rate of public vulnerability disclosure; any buildable project with a fuzzing harness can supply new training instances.","Smaller students gain the most: the 12B model roughly doubles its SEC-bench score, and self-distillation improves every scale, so even a strong model can learn from its own verified trajectories.","Fine-tuning transfers repair behavior, not just scores: students adopt the teacher's edit-and-verify workflow, with verification coverage rising from 20.7% to 82.7% at 12B and format-error rates collapsing.","The gains carry to out-of-distribution languages (Go, JavaScript, Python), suggesting the corpus teaches general vulnerability-repair concepts rather than encoding C/C++ syntax patterns.","Ensembling two students trained under different teachers beats either single student and even beats the stronger teacher (82.0% vs 74.0% at 31B), a post-hoc result the paper computes from the same runs."],"supporting_citations":[{"why":"Defines repository-level task construction from real pull requests, the template CyberForge follows for task formation.","marker":"Jimenez et al. 2024"},{"why":"SEC-bench, the C/C++ evaluation benchmark whose task format and scoring the training pipeline matches.","marker":"Lee et al. 2025"},{"why":"CVE-Factory, whose task-formation and successful-trajectory-training methodology CyberForge adopts.","marker":"Luo et al. 2026"},{"why":"SWE-Gym, supplying the trajectory-mining approach: keep only successful oracle-verified trajectories for fine-tuning.","marker":"Pan et al. 2025"},{"why":"Source of the containerized C/C++ projects with build scripts and libFuzzer harnesses whose environment CyberForge reuses.","marker":"Google 2016"},{"why":"PrimeVul supplies historical vulnerable-function pairs used for retrieval-based selection of injection sites in the agentic pipeline.","marker":"Ding et al. 2024"},{"why":"VGX, the function-level injection baseline the paper contrasts with repository-level, execution-validated injection.","marker":"Nong et al. 2023"},{"why":"Cyber-Zero, a simulation-only trajectory generator whose lack of execution grounding CyberForge argues against.","marker":"Zhuo et al. 2025a"}],"fun_headline_variants":["Verified injection of 1,034 synthetic bugs lifts patch repair","Synthetic vulnerability injection: up to 14.7-point patch gains","CyberForge: scaled training data from validated synthetic bugs","1,034 injected bugs train agents that fix cross-language patches","Injecting validated bugs boosts security agents beyond C/C++"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmarks and the training projects may share source-level ancestry, so the reported gains could partly reflect memorization of familiar codebases rather than a general vulnerability-repair skill.","fun_headline_variants_meta":{"raw":{"variants":["Verified injection of 1,034 synthetic bugs lifts patch repair","Synthetic vulnerability injection: up to 14.7-point patch gains","CyberForge: scaled training data from validated synthetic bugs","1,034 injected bugs train agents that fix cross-language patches","Injecting validated bugs boosts security agents beyond C/C++"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3401,"prompt_tokens":1051,"completion_tokens":2350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2264}},"tokens_in":667,"tokens_out":2350,"duration_ms":14144,"temperature":1.0,"reasoning_tokens":2264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:32:41.176802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the project-level overlap between the 80 training projects and the 24 SEC-bench projects (and any PatchEval repositories), retrain after deleting all training instances from overlapping projects, and check whether the SEC-bench and PatchEval improvements survive; if the gains collapse on fully disjoint projects, the generalization claim is falsified.","supporting_citations":[{"cited_title":"2025 , url =","cited_arxiv_id":null,"evidence_quote":"SEC-bench, the C/C++ evaluation benchmark whose task format and scoring the training pipeline matches."},{"cited_title":"2026 , note =","cited_arxiv_id":null,"evidence_quote":"CVE-Factory, whose task-formation and successful-trajectory-training methodology CyberForge adopts."},{"cited_title":"Training Software Engineering Agents and Verifiers with","cited_arxiv_id":null,"evidence_quote":"SWE-Gym, supplying the trajectory-mining approach: keep only successful oracle-verified trajectories for fine-tuning."},{"cited_title":"2024 , url =","cited_arxiv_id":null,"evidence_quote":"PrimeVul supplies historical vulnerable-function pairs used for retrieval-based selection of injection sites in the agentic pipeline."}],"review_version":2}