Pith. sign in

REVIEW 3 major objections 4 minor 23 references

From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a trigger signal on a vulnerable build is not evidence of CVE-specific reproduction unless a clean patched counterfactual is present, and that embedded oracles are unreliable (sensitivity 60%, specificity 45%).

desk verdict The qualitative core is right and useful, but the headline oracle-reliability numbers overstate the case because some 'patched' builds are not actually patched and calibration cases are mixed into the full-corpus matrix. read the letter →

arxiv 2608.09567 v1 pith:ZNN56TI3 submitted 2026-08-10 cs.CR

Bo Chen This is my paper · ORCID
classification cs.CR
keywords reproducibilityauditvulnerabilityvalidationartifactsLLMagentspatchedcounterfactualoraclereliabilityCVEreproductionartifactevaluationsecuritybenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a security artifact that publicly exists, runs, and prints an alarming signal is still a long way from having reproduced the claimed CVE. It argues that artifact-embedded oracles—scripts that declare "VULNERABILITY TRIGGERED"—are systematically unreliable, because they rarely check the one thing that would make a signal CVE-specific: a clean patched counterfactual. On a 102-case benchmark plus an 18-paper sample, 20 of 30 signal-producing cases still fired on the patched build, 7 of 19 matched negative controls fired on benign input, and the oracle confusion matrix showed 60% sensitivity and 45% specificity. A sympathetic reader should care because claimed per-case success rates in this literature are widely used to compare agents and to certify vulnerability-reproduction skill, and this audit says those claims can overstate confirmation by a wide margin.

What carries the argument

The machinery is a four-layer semantic-confirmation ladder: candidates at the "available" level, "runnable" at R0 or after environment-only R1 repair, "signal-producing" when a run yields a candidate marker (G1), and "semantically confirmed" when the signal matches a CVE-specific post-condition (G2), is absent on benign input (G3a negative control), and is absent on the patched build (G3b patched counterfactual). The patched-counterfactual oracle is the load-bearing instrument: it turns a yes/no "did it print something scary" check into a test of whether the signal is actually CVE-specific, and it is what generates the paper's confusion matrix.

What would settle it

Run the 30 G3b cases again with patched builds whose fix status is independently confirmed (for example, compile from the official fixed release tags and verify the vulnerable code path is gone via diff), then recompute the dirty rate and the oracle confusion matrix; if the dirty rate drops well below 20/30, the oracle-unreliability conclusion weakens.

Watch

Extended reading notes

Core claim

The central discovery is that availability, runnability, signal production, and semantic confirmation form a strict ladder, and the gap between layers is large. On the anchor corpus, the repository was reachable, the workflow completed at R0 for only 10 of 18 papers (11 of 18 after environment-only R1 repair), 34 of 87 executed runs (39.1%) produced a candidate signal, 23 of those 34 satisfied their pre-registered post-condition, and only 10 of 30 audited cases survived the patched-counterfactual check. A trigger on the vulnerable build is therefore not evidence of CVE-specific reproduction unless the patched counterfactual is clean. The paper also found that 58 of 102 anchor cases contain a script-internal CVE identifier that differs from the directory's declared CVE, meaning the target being tested can differ from the target claimed.

Load-bearing premise

The patched-counterfactual verdicts are treated as ground truth for whether a signal is CVE-specific, but some patched builds are constructed by swapping image tags or pinned versions and may still contain the vulnerability, so a dirty verdict can mean "not actually patched" rather than "oracle unreliable."

Editorial extensions

If this is right

  • Artifact-embedded trigger checks should be treated as screening tools, not confirmation, until a patched-counterfactual check is included.
  • Published per-case success rates for LLM/agent vulnerability validation are upper bounds; the true semantically-confirmed rate on the anchor corpus is far lower, with only two strict E1 confirmations out of the signal-producing corpus.
  • Benchmark maintainers need to reconcile script-internal CVE identifiers with declared targets before using case outcomes for agent evaluation.
  • Reproducibility audits of security papers should add a semantic-confirmation layer rather than stopping at buildability.
  • The pre-registered protocol (R0/R1 ladder, G1–G3 evidence levels, patched-counterfactual oracle) offers a reusable template for future audits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same patched-counterfactual discipline likely applies to traditional (non-LLM) PoC exploits; a cheaper spot-check of existing exploit databases against patched builds would test this directly.
  • Editorial inference: the 56.9% CVE-ID divergence rate, if it generalizes beyond the anchor corpus, means any benchmark evaluated by matching directory labels to trigger strings inherits label noise that could silently inflate agent scores.
  • Editorial inference: oracle false positives concentrated in manually forced crashes (NULL dereferences, designed-in panics) suggest a concrete fix—require PoCs to exercise the vulnerable code path rather than any crash.
  • Editorial inference: the study's calibration E1 rate bounds (1/20 to 5/20) could be sharpened by completing G3a/G3b audits on the remaining cases; the pre-registered bounds policy already gives readers worst/best cases rather than single imputed numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a pre-registered reproducibility audit of LLM/agent-driven vulnerability validation artifacts. It builds a 104-paper consensus corpus, finds 59 reachable artifacts, executes 18 paper-level artifacts at R0/R1, executes all 102 cases of the anchor benchmark (arXiv:2509.24037), and audits 30 signal-producing cases with patched-counterfactual (G3b) builds and 19 with matched negative controls (G3a). The three headline findings are: 58/102 anchor cases have a script-internal CVE identifier that diverges from the declared directory CVE; 10/18 (55.6%) artifacts complete their workflow at R0, rising to 11/18 after R1; and artifact-embedded oracles appear unreliable, with 20/30 patched-counterfactual verdicts dirty and an oracle confusion matrix of sensitivity 60% and specificity 45%. The paper concludes that a trigger on the vulnerable build is not evidence of CVE-specific reproduction unless the patched counterfactual is clean.

Significance. If the quantitative measurement were fully valid, this would be a valuable contribution: it operationalizes semantic confirmation for security reproducibility, provides a reusable G1-G3 evidence ladder, and documents concrete failure modes of artifact-embedded oracles. The qualitative ladder claim is convincingly supported by the CVE-2020-1967 case in Section 5.4, where a clean segmentation fault occurs on both the vulnerable and the patched OpenSSL builds, showing that signal production does not imply CVE-specific reproduction. The frozen protocol, evidence bundles, per-case execution logs, and the explicit execution ledger in Table 2 are real strengths that make the audit auditable. However, the headline oracle-reliability rates currently rest on a patched-counterfactual ground truth that is not valid for a subset of the included cases, so the quantitative findings need revision before the paper's central rates can be accepted as stated.

major comments (3)
  1. The G3b patched-counterfactual verdicts are not valid ground truth for the EOL apt-level cases, yet these cases are retained in the 20/30 dirty rate and in Table 5. Section 5.4 explicitly lists among the 20 dirty verdicts 'apt-level cases whose EOL sources carry no fix.' For those cases, the patched build was constructed by apt upgrade injection, but since the base image is EOL, no fix is actually available; the resulting 'patched' build may still be vulnerable. A signal that persists on such a build is therefore not evidence that the oracle is non-specific. Section 8.2 itself states that 'EOL base images carry no apt-level fixes, so patched-counterfactual construction failed,' yet those same cases are counted as dirty rather than as unbuildable or unassessable. The authors should either exclude these cases from the dirty set, classify them as unbuildable, or demonstrate for each retained EOL apt-level case that a genuine patched build was available and applied; the 20/30 rate, the specificity estimate of 45%, and the confusion matrix must then be recomputed.
  2. Table 5's confusion matrix pools non-random calibration cases with full-corpus cases, yet reports Wilson 95% confidence intervals as though the 30 verdicts form a well-defined random sample. Section 5.4 states that the calibration G3b subset 'is not a random sample, so the 4/5 rate below is not an estimate of a population proportion.' The G3b-executed set comprises 29 of the 34 full-corpus G1 signals plus four additional calibration-only cases (Section 5.4), so the n=30 matrix is a convenience set rather than a probability sample. The confidence intervals for sensitivity and specificity are therefore not sampling-based estimates for a defined population. The authors should present the full-corpus and calibration-only verdicts separately, or clearly label the matrix as a descriptive summary of the audited cases and omit inferential intervals.
  3. The abstract and Section 5.6 state that artifact-embedded oracles 'prove unreliable' and headline the 60%/45% matrix, but Section 8.2 correctly cautions that these are single-corpus, partial-coverage results that 'should not be read as a population estimate until the multi-paper sample replicates it.' This internal tension is load-bearing because the G3b coverage is incomplete (5 of 34 G1 signals lack a patched counterfactual), the E1 confirmation set contains only two cases, and the G3b ground-truth issues described above directly affect the matrix. The authors should align the abstract and conclusion with the stated exploratory scope, or provide a formal justification for why the audited 30 cases support a systematic-reliability claim despite the non-random sampling and the EOL construction failures.
minor comments (4)
  1. The bullet 'F ailure taxonomy' contains a spacing typo and should read 'Failure taxonomy.'
  2. The inconsistent spacing in 'PASS/F AIL' and 'PASS/F AIL' should be normalized to 'PASS/FAIL' throughout the manuscript and tables.
  3. The paper describes the protocol as pre-registered but provides only self-hosted paths such as 'protocol/search protocol v1.json' and no external registry, DOI, or third-party timestamp. Please provide a public, immutable deposit or registry entry so readers can independently verify the pre-registration claim.
  4. The exploratory claim gap on the calibration cases (claimed success 0.80 vs. E1 rate 0.20, computed on five G3b-verdict cases) should be explicitly labeled as a pilot illustration rather than an estimate, especially since the preceding paragraph states that the four-fifths calibration dirty rate is not a population proportion.

Circularity Check

1 steps flagged · score 4.0 of 10

Patched-counterfactual 'dirty' verdicts include EOL apt-level cases whose patched build could not be constructed, making the 20/30 oracle-unreliability rate and 60%/45% matrix partly an artifact of the paper's own labeling.

  1. self definitional [Section 5.4 (Patched-counterfactual (G3b) audit) and Section 8.2 (Limitations)]
    "Verdicts were reached on 30 cases: 10 clean ... and 20 dirty—including ... apt-level cases whose EOL sources carry no fix. / EOL base images carry no apt-level fixes, so patched-counterfactual construction failed for a small number of cases—an EOL effect that itself recurs as a runnability failure mode."

    G3b 'dirty' is defined as the signal persisting on the patched build. For apt-level EOL cases, no fix exists and the paper admits patched-counterfactual construction failed; the signal persists on a still-vulnerable build, so by the paper's own definition these cases are not dirty. Yet they are counted in the 20/30 dirty rate and in the oracle matrix's FP=11. The headline 'oracles unreliable' result is therefore partially generated by assigning a verdict label to cases that do not satisfy the verdict's defining condition, rather than by the patched-counterfactual evidence. The qualitative conclusion has independent support, but the quantitative matrix is not fully independent of this labeling.

full rationale

The paper is largely a pre-registered empirical audit with no self-citation chain and no fitted parameter renamed as prediction. The only load-bearing definitional issue is the G3b patched-counterfactual labeling of EOL apt-level cases: Section 5.4 counts them among the 20 dirty even though Section 8.2 says patched-counterfactual construction failed for such cases. This makes the 20/30 and 60%/45% numbers partially dependent on the authors' labeling rather than on a true patched counterfactual. The central qualitative claim—that a trigger on a vulnerable build is insufficient evidence without a clean patched counterfactual—is independently supported by concrete examples (e.g., CVE-2020-1967, CVE-2021-31162), so the circularity is partial, not total. No other pattern (self-citation, imported uniqueness, renamed known result) applies.

Assumptions & free parameters 0 free parameters · 5 assumptions · 3 invented entities

The central measurements rest on hand-defined evidence levels and the author's own adjudications; the numbers carry no fitted parameters, but the protocol constructs are the paper's invention, and several domain assumptions (NVD fidelity, patch-version correctness, corpus completeness, environment fidelity) are load-bearing.

assumptions (5)
  • domain assumption NVD CVE descriptions provide accurate ground truth for pre-registered post-conditions.
    Section 4.3 derives post-conditions from NVD semantics before execution; if NVD is incomplete, G2 verdicts and the E1 set are unreliable.
  • domain assumption Patched builds constructed via image-tag/package/crate version replacement match the official fix.
    Section 5.4; EOL apt sources carry no fix, so some patched builds may remain vulnerable, biasing G3b dirty verdicts.
  • domain assumption The search protocol (single query over arXiv, DBLP, OpenAlex, 2023-01-01 to 2026-08-08) captures the relevant universe of papers.
    Section 3.1; non-indexed venues and off-window work are excluded.
  • domain assumption The 18-paper execution sample is representative of reachable artifacts.
    Section 4.1; stratified by year and category with deterministic seed 20260808 but no formal random sampling description or confidence intervals.
  • domain assumption The Docker Desktop LinuxKit environment (2 CPU, 2 GB, per-case network policy) does not materially distort runnability.
    Section 5.2 and Section 8.2; resource and network choices are safety-driven and may affect port-dependent or resource-hungry artifacts.
invented entities (3)
  • G1/G2/G3a/G3b evidence ladder independent evidence
    purpose: Grades how much an executed signal supports a claimed CVE reproduction: candidate signal, post-condition match, negative control, patched counterfactual.
    Defined in Section 2.2 and applied to 102 cases; each level is an observable condition an independent auditor can check, giving a falsifiable handle.
  • R0/R1 repair ladder independent evidence
    purpose: Separates clean-snapshot runnability from environment-only repair (dependency installs, EOL source redirects).
    Defined in Section 2.2; the R0/R1 distinction is deterministic and reproducible.
  • E1/E2/E3 verdict classes independent evidence
    purpose: Derived confirmation categories for a case: E1 requires G2+G3a+G3b, E2 requires G2+G3a, E3 is incomplete evidence.
    Section 2.2; these categories are applied consistently to the executed cases and are checkable by re-audit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts." pith.science (2026). https://pith.science/paper/ZNN56TI3

@misc{pith2026260809567,
  author       = {Pith},
  title        = {Pith review of: From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNN56TI3}},
  note         = {Machine review of arXiv:2608.09567}
}
read the original abstract

Security research artifacts---repositories, PoC exploits, and validation pipelines---are increasingly produced by LLM/agent-driven vulnerability workflows, yet the gap between \emph{publicly available}, \emph{runnable}, \emph{signal-producing}, and \emph{semantically confirmed} artifacts is poorly measured. We conduct a pre-registered reproducibility audit of this literature. A search covering 2023--2026 with dual screening yields a 104-paper consensus corpus, of which 59 papers (56.7\%) have a publicly reachable artifact. We execute an 18-paper sample at R0/R1 and all 102 cases of the anchor benchmark (arXiv:2509.24037), with patched-counterfactual verdicts on 30 signal-producing cases and matched-negative-control verdicts on 19. Three findings stand out. First, 58/102 (56.9\%) anchor cases contain a script-internal CVE identifier that diverges from the declared directory CVE. Second, only 10/18 (55.6\%) paper-level artifacts complete their declared workflow at R0, rising to 11/18 (61.1\%) after environment-only R1 repair. Third, artifact-embedded oracles prove unreliable: 20/30 patched-counterfactual audits still produce the claimed signal on the patched build, 7/19 matched negative controls still trigger on benign input, and the oracle confusion matrix has sensitivity 60\% and specificity 45\%. A trigger on the vulnerable build is not evidence of CVE-specific reproduction without a clean patched counterfactual. These are exploratory results from a pre-registered protocol, and our protocol---pre-registered post-conditions, R0/R1 repair ladder, G1--G3 semantic evidence levels, and patched-counterfactual oracles---is a reusable template for the security reproducibility community.

Figures

Figures reproduced from arXiv: 2608.09567 by the authors.

Figure 1
Figure 1. Screening funnel: raw search records to the 104-paper consensus corpus (frozen 2026-08- [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Artifact availability on the consensus set ( [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

  1. [1]

    Repeatability in computer systems research.Com- munications of the ACM, 59(3):62–69, 2016

    Christian Collberg and Todd Proebsting. Repeatability in computer systems research.Com- munications of the ACM, 59(3):62–69, 2016

    work page 2016

  2. [2]

    PentestGPT: Evaluating and harnessing large lan- guage models for automated penetration testing

    Gelei Deng, Yi Liu, V ´ ıctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: Evaluating and harnessing large lan- guage models for automated penetration testing. InProceedings of the 33rd USENIX Security Symposium (USENIX Security 2024), pages 847–864, 2024

    work page 2024

  3. [3]

    PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy

    Phan The Duy, Khoa Ngo-Khanh, Nguyen Huu Quyen, and Van-Hau Pham. PoC-Adapt: Semantic-aware automated vulnerability reproduction with LLM multi-agents and reinforce- ment learning-driven adaptive policy, 2026. arXiv:2604.06618

    work page Pith review arXiv 2026

  4. [4]

    Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generation

    David Jin, Qian Fu, and Yuekang Li. Good News for Script Kiddies? evaluating large lan- guage models for automated exploit generation, 2025. arXiv:2505.01065; also in IEEE S&P Workshops 2025, pp. 278–282

    work page Pith review arXiv 2025

  5. [5]

    SEC-bench: Automated bench- marking of LLM agents on real-world software security tasks, 2025

    Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. SEC-bench: Automated bench- marking of LLM agents on real-world software security tasks, 2025. arXiv:2506.11791; NeurIPS 2025

    arXiv 2025

  6. [6]

    Automated vulnerability validation and verification: A large language model approach, 2025

    Alireza Lotfi, Charalampos Katsis, and Elisa Bertino. Automated vulnerability validation and verification: A large language model approach, 2025. arXiv:2509.24037; anchor calibration corpus (102 cases)

    arXiv 2025

  7. [7]

    Shell or nothing: Real-world benchmarks and memory-activated agents for automated penetration testing, 2025

    Wuyuao Mai, Geng Hong, Qi Liu, Jinsong Chen, Jiarun Dai, Xudong Pan, Yuan Zhang, and Min Yang. Shell or nothing: Real-world benchmarks and memory-activated agents for automated penetration testing, 2025. arXiv:2509.09207; TermiBench and TermiAgent. 14

    arXiv 2025

  8. [8]

    Cve-2020-1967

    National Institute of Standards and Technology. Cve-2020-1967. National Vulnerability Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2020-1967, 2020. OpenSSL TLS 1.3signature algorithms certdenial of service; fixed in OpenSSL 1.1.1g

    work page 2020

Show all 23 references
  1. [9]

    Cve-2021-31162

    National Institute of Standards and Technology. Cve-2021-31162. National Vulnerabil- ity Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2021-31162, 2021. Rust std::vec::Vec::from iterdouble drop on iterator panic; fixed in Rust 1.52

    2021

  2. [10]

    Cve-2021-44228

    National Institute of Standards and Technology. Cve-2021-44228. National Vulnerability Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2021-44228, 2021. Log4Shell: Apache Log4j JNDI remote code execution; fixed in Log4j 2.15.0

    2021

  3. [11]

    Cve-2022-22816

    National Institute of Standards and Technology. Cve-2022-22816. National Vulnerability Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2022-22816, 2022. Python Pil- low/ImageIO path traversal

    2022

  4. [12]

    Cve-2023-0217

    National Institute of Standards and Technology. Cve-2023-0217. National Vulnerability Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2023-0217, 2023. OpenSSL NULL pointer dereference inX.509certificate verification

    2023

  5. [13]

    Cve-2023-25676

    National Institute of Standards and Technology. Cve-2023-25676. National Vulnerability Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2023-25676, 2023. TensorFlow QuantizeAndDequantizeV2 null-pointer dereference

    2023

  6. [14]

    Cve-2025-30223

    National Institute of Standards and Technology. Cve-2025-30223. National Vulnerabil- ity Database (NVD),https://nvd.nist.gov/vuln/detail/CVE-2025-30223, 2025. Beego cross-site scripting; fixed in Beego 2.3.6

    2025

  7. [15]

    FaultLine: Automated proof-of-vulnerability generation using LLM agents, 2025

    Vikram Nitin, Baishakhi Ray, and Roshanak Zilouchian Moghaddam. FaultLine: Automated proof-of-vulnerability generation using LLM agents, 2025. arXiv:2507.15241

    2025 arXiv

  8. [16]

    ”Get in Researchers; We’re Measuring Reproducibility”: A reproducibility study of machine learning papers in tier 1 se- curity conferences

    Daniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren, Cole Kitroser, Alejandro Pas- cual, Divyajyoti Ukirde, Kevin Butler, and Patrick Traynor. ”Get in Researchers; We’re Measuring Reproducibility”: A reproducibility study of machine learning papers in tier 1 se- curity...

    2023

  9. [17]

    Artificial intelligence as the new hacker: Developing agents for offensive security, 2024

    Leroy Jacob Valencia. Artificial intelligence as the new hacker: Developing agents for offensive security, 2024. arXiv:2406.07561; ReaperAI autonomous offensive-security agent

    2024 arXiv

  10. [18]

    Contract- Tinker: LLM-empowered vulnerability repair for real-world smart contracts

    Che Wang, Jiashuo Zhang, Jianbo Gao, Libin Xia, Zhi Guan, and Zhong Chen. Contract- Tinker: LLM-empowered vulnerability repair for real-world smart contracts. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE 2024), pages 2350–2...

    2024

  11. [19]

    PATCHEV AL: A new benchmark for evaluating LLMs on patching real-world vulnerabilities, 2025

    Zichao Wei, Jun Zeng, Ming Wen, Zeliang Yu, Kai Cheng, Yiding Zhu, Jingyi Guo, Shiqi Zhou, Le Yin, Xiaodong Su, and Zhechao Ma. PATCHEV AL: A new benchmark for evaluating LLMs on patching real-world vulnerabilities, 2025. arXiv:2511.11019

    2025

  12. [20]

    AutoPT: How far are we from the End2End automated web penetration testing?, 2024

    Benlong Wu, Guoqiang Chen, Kejiang Chen, Xiuwei Shang, Jiapeng Han, Yanru He, Weim- ing Zhang, and Nenghai Yu. AutoPT: How far are we from the End2End automated web penetration testing?, 2024. arXiv:2411.01236. 15

    2024 arXiv

  13. [21]

    Patch validation in automated vulnerability repair, 2026

    Zheng Yu, Wenxuan Shi, Xinqian Sun, Zheyun Feng, Meng Xu, and Xinyu Xing. Patch validation in automated vulnerability repair, 2026. arXiv:2603.06858; PVBench: 40%+ of basic-test-validated patches fail rigorous testing

    2026

  14. [22]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. InProceedings of the 13th International Conference on Learning Representations (ICLR 2025),

    2025

  15. [23]

    CVE-Bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities, 2025

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-Bench: A benchmark for AI agents’ ability to exploit real-worl...

    2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.