Pith. sign in

REVIEW 3 major objections 5 minor 22 references

A minimal non-frontier LLM analyzer rediscovers up to 65 of 95 real AI-found CVEs under strict identity matching.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 13:16 UTC pith:IJKHGFV5

load-bearing objection Solid vendor-built benchmark with real measurements: cheap models plus repeats rediscover most of their own AI-found CVEs under a file-scoped, strict-identity setup—useful if you read the caveats. the 3 major comments →

arxiv 2607.27030 v1 pith:IJKHGFV5 submitted 2026-07-29 cs.CR cs.LG

HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

Petr Simecek , Elnaz Babayeva , Jiri Balhar , Michal Bida , Michal Buran , Vaclav Cadek , Luigino Camastra , Tomas Dulka
show 18 more authors
Michal Janocko Tomas Klohna Pavel Kohout Ondrej Kokes Adam Krivka Jakub Kubik Patrik Mada Igor Morgenstern Marek Pavelka Joshua Rogers Petr Stastny Jan Tattermusch Dmitrijs Trizna Martin Votruba Guido Vranken Jakub Zikl Evelina Gabasova Stanislav Fort
This is my paper
classification cs.CR cs.LG
keywords vulnerability rediscoveryLLM security scannersCVE benchmarknon-frontier modelsrepeated-pass reliabilitysource-conditioned detectionstrict identity matchingtriage burden
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper builds HoF-Bench from 95 public CVEs that AI systems originally found in eight mature open-source projects, pinned at vulnerable commits. Scanners get the repository and designated target files but no CVE ids, descriptions, or fixes. Credit is granted only when a detector-blinded judge agrees the finding matches the same code path, root cause, attack condition, and impact. Inside a deliberately simple fixed scaffold, ten inexpensive open-weight and flash-tier models—never frontier detectors—recover as many as 68% of the set after four repeated passes. Repeated scanning and mixing models add more rediscoveries than generated context or deeper triage, while the bugs every model misses concentrate in large C infrastructure code. The result is a compact, auditable test bed for measuring how reliably scanners rediscover real CVE-worthy bugs and how many candidate findings they dump on reviewers.

Core claim

Under a strict, detector-blinded protocol on HoF-Bench, a deliberately minimal LLM-based analyzer using only non-frontier detectors rediscovers up to 65 of 95 real AI-discovered CVEs (68% at pass@4). Six of ten model families reach at least 50 CVEs in some four-pass condition; model diversity and repetition raise coverage more than optional generated context or multi-round triage.

What carries the argument

HoF-Bench plus strict vulnerability-identity scoring: a blinded frontier-model judge credits a finding only when it matches the ground-truth code path or component, root cause, attacker-controlled condition, and security impact—not mere file proximity or CWE class.

Load-bearing premise

Giving the scanner ground-truth-derived target files and trusting an automated judge on four identity dimensions is a fair enough proxy for genuine CVE rediscovery.

What would settle it

A human audit of the judge’s strict-match labels on the stored findings, or a rerun that withholds target-file scope and still recovers a comparable fraction of the 95 CVEs, would confirm or break the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scanner evaluations should report cumulative recall over repeated runs plus review burden, not single-pass scores alone.
  • A mixed portfolio of inexpensive models covers more CVEs than repeating the single strongest model the same number of times.
  • Generated context and deeper triage mainly raise candidate volume; they are not automatic recall wins.
  • Residual hard cases concentrate in large, stateful C infrastructure rather than web-application bugs.
  • Eight pinned checkouts and a public manifest are enough for others to compare tools under the same strict judge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If language tracks difficulty more than vulnerability class, scaffolds tuned for visible web trust boundaries may systematically under-serve cross-call C state bugs.
  • Cost-aware default practice may shift from one large-model scan toward portfolios of small models with few passes each.
  • Because every task is positive and public, future holdout or post-cutoff CVE sets would separate true rediscovery skill from training-data familiarity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces HoF-Bench, a benchmark of 95 public AI-attributed CVEs across eight pinned open-source repositories (C, PHP, JavaScript). Under a fixed minimal scaffold (optional generated context, detection, multi-round triage), ten non-frontier detector models are evaluated over four repeated passes with and without context (7,600 pass records). A detector-blinded frontier judge awards strict credit only when a finding matches code path, root cause, attack condition, and impact. The strongest configuration rediscovers 65/95 CVEs (pass@4); repeated passes and cross-model diversity help more than context or deeper triage; difficulty is strongly language-structured, with universal misses concentrated in C. The dataset and artifacts are released.

Significance. If the results hold under the stated protocol, the work supplies a compact, reproducible test bed for real CVE rediscovery rather than snippet classification, and shows that inexpensive non-frontier models inside a systematic scaffold can recover a large fraction of previously AI-found bugs. Strengths include the fixed scaffold with full failure accounting, success@k over all pass subsets, triage-depth replay, Wilson intervals, a Rasch Beta-Binomial reliability model with LOO model comparison, explicit ethics disclosure of vendor construction, and public release of the dataset and run artifacts. These make the study useful for comparing scanners, reliability, and review burden even if the population is AISLE-conditioned.

major comments (3)
  1. [§2.2 Task Definition; Abstract] §2.2 and Abstract: The task supplies 1–7 ground-truth-derived target files (median 911 LOC, primary vulnerable file included). The paper correctly calls this source-conditioned and not whole-repository localization, yet the abstract headline (“rediscovers up to 65 of the 95 CVEs”) does not carry that qualifier. Because localization is largely removed, the 68% figure measures mechanism recognition inside a narrowed window. Please align abstract/title claims with §2.2 (e.g., “target-conditioned rediscovery”) and, if feasible, report at least one broader-scope or no-oracle sensitivity on a subset so readers can gauge how much the file oracle drives the rate—especially on C, where Discussion already notes cross-file state.
  2. [§4.3 Evaluation; §6 Discussion] §4.3 and Discussion: Strict outcomes rest on an automated, human-unaudited judge from the same vendor family as several detectors. Blinding to detector identity does not remove stylistic affinity. A modest human audit (e.g., stratified sample of match/partial/no-match verdicts, with agreement rates) is load-bearing for trusting the primary metric; without it, the 65/95 and language-gap claims remain only weakly validated. Please add such an audit or clearly demote automated strict recall to a provisional score.
  3. [§5 Results; §6 Discussion] §5–§6: The benchmark and reference scanner are AISLE-conditioned (population, scaffold lineage, author affiliation). Ethics discloses this, but Results still generalize to “vulnerability discovery is a systems problem” and to scanner comparison broadly. Keep causal claims inside the ten configurations of one scaffold on one vendor’s CVE slice; frame external validity via the Villa et al. census comparison already cited, rather than implying product-level ranking of ten independent tools.
minor comments (5)
  1. [Table 2; §5.1] Table 2 portfolio row is labeled exploratory; ensure the abstract and §5.1 do not read the 68 expected CVEs as a primary confirmed result on par with single-model pass@4.
  2. [§5.2; Supplement B] Supplement B triage replay caveats (later rounds conditioned on earlier reviews; no counterfactual arbiter) should be cross-referenced in the main-text triage paragraph so readers do not over-read the 974→1007 shift.
  3. [Figure 3; Table 2] Figure 3 caption and success@k definition are clear; consider adding raw pass@1 means in Table 2 for readers who will not parse the curves.
  4. [§3 Related Work] Related Work: briefly contrast scoring strictness with RepoPairBench/JITVul/RealVuln so the four-dimension identity criterion is easier to place.
  5. [§1; Table 1] Typos/consistency: “ACVEis” spacing in the introduction; ensure model identifiers in Table 1 match supplement/run artifacts exactly.

Circularity Check

1 steps flagged

Empirical rediscovery study with vendor-conditioned population; no derivation-by-construction circularity.

specific steps
  1. self citation load bearing [§1 Introduction; §2.1 Construction; §7 Ethics]
    "HoF-Bench takes its name from AISLE’s public Hall of Fame and asks that question on 95 recent public CVEs attributed by AISLE to its systems and listed there [1]. ... Readers should treat HoF-Bench as a vendor-constructed benchmark: we have therefore released the dataset, the scanner-visible manifest, and the raw run artifacts so that the selection and scoring decisions can be audited independently."

    The benchmark population is defined as AISLE-attributed CVEs and the reference scaffold is inspired by AISLE’s nano-analyzer, with all authors AISLE-affiliated. This is self-referential population/instrument choice, not a forced numerical identity: rediscovery counts still come from blinded experimental runs rather than from equating outputs to inputs by definition. Minor and acknowledged; does not make the 65/95 figure tautological.

full rationale

HoF-Bench is an empirical systems/benchmark paper, not a first-principles derivation. The headline result (up to 65/95 strict pass@4 rediscovery with non-frontier detectors) is obtained from 7,600 recorded analyzer runs scored by a detector-blinded judge under an explicitly source-conditioned protocol; it is not algebraically or statistically forced by a fitted parameter, uniqueness theorem, or definitional identity. Self-citations to AISLE’s Hall of Fame, nano-analyzer scaffold, and prior reliability papers define the population and instrument and are openly labeled vendor-constructed, with dataset and artifacts released for audit. That is selection/conditioning bias and a validity caveat (target-file oracle, same-family judge, unaudited labels), not circularity of the enumerated kinds: nothing equates a claimed prediction to its inputs by construction. Score 1 only for minor load-adjacent self-reference in population framing; central quantitative claim remains an independent experimental measurement under stated assumptions.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

This is an empirical systems/benchmark paper, not a derivation from axioms. Load-bearing commitments are methodological: what counts as the same vulnerability, what the scanner is allowed to see, and that automated judging plus AISLE-conditioned sampling still support the stated rediscovery claims. Free parameters are engineering knobs of the fixed scaffold rather than fitted physical constants.

free parameters (6)
  • Number of repeated detector passes (k=4) = 4
    Chosen experimental budget; success@k curves and pass@4 headline metrics depend on this repetition depth.
  • Max raw findings per pass = 5
    Hard cap in the scaffold that shapes recall/volume tradeoff and triage load.
  • Target-file prompt size cap = 300,000 characters
    Truncation/ cap policy affects how much vulnerable context the detector actually sees.
  • Context-stage repository searches = up to 3
    Limits extra evidence gathering in context-on runs; part of the context ablation.
  • Triage depth and arbiter policy = 3 rounds + arbiter (replayed shallower policies)
    Determines which findings survive to the judge; replay estimates depend on stored sequential reviews.
  • Rasch/Beta-Binomial concentration κ and ability/difficulty effects = posterior-fitted (Supplement C)
    Statistical model parameters fit to the 7,600-pass outcomes to quantify overdispersion and relative strengths; not used as the primary 65/95 claim but support reliability conclusions.
axioms (6)
  • ad hoc to paper Strict CVE identity requires same code path/component, root cause, attacker-controlled condition/trust boundary, and security impact.
    Primary scoring definition in §2.2/§4.3; file proximity or CWE class alone is insufficient by protocol choice.
  • domain assumption Ground-truth-derived target-file scope is acceptable for a source-conditioned rediscovery benchmark.
    Stated task design: tests analysis given localization oracle, not autonomous repo-wide finding.
  • domain assumption A detector-blinded frontier LLM judge can label strict match / partial / no match / insufficient evidence well enough for primary metrics.
    Evaluation depends on automated judging without human audit (§4.3, Discussion).
  • domain assumption Unmatched findings are not counted as false positives; reported metric is recall-only on all-positive tasks.
    Explicit metric scope; precision/FPR are out of scope by construction.
  • domain assumption Public AISLE Hall of Fame CVEs grounded at one common vulnerable commit per repo form a useful fixed population.
    Construction filter in §2.1; excludes cases that could not be pinned or verified.
  • standard math Binomial/Beta-Binomial item-response style model with scanner strength and CVE difficulty is an adequate reliability description.
    Supplement C applies standard Rasch/2PL and overdispersion machinery to repeated passes.
invented entities (2)
  • HoF-Bench v1 independent evidence
    purpose: Fixed 95-CVE, eight-repo benchmark with scanner-visible/evaluator-only field split for strict rediscovery scoring.
    New dataset/protocol artifact; independent handle is the public HF/GitHub release and pinned commits.
  • Reference nano-analyzer-style scaffold (context → detect → skeptical triage/arbiter) independent evidence
    purpose: Controlled comparison instrument holding pipeline fixed while swapping detector backbones and conditions.
    Engineering system, not a physical entity; authors disclaim it as a scientific contribution beyond instrumentation.

pith-pipeline@v1.2.0-daily-grok45 · 17022 in / 3918 out tokens · 80695 ms · 2026-07-30T13:16:45.442127+00:00 · methodology

0 comments
read the original abstract

LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS. We introduce HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 of these public AI-discovered CVEs across eight repositories pinned at vulnerable commits. Analyzers receive source and target-file scope but not CVE identifiers, descriptions, fixes, or expected mechanisms; a detector-blinded frontier-model judge credits only findings that identify the same code path, root cause, attack condition, and impact. A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol. No frontier model performs detection anywhere in the study. The ten detector backbones are five open-weight models (21B--284B total parameters, 3--13B active) and five proprietary small or "flash"-tier models. All of them run in the fixed scaffold with four repeated passes, an optional generated-context stage, and a replayable multi-round triage stage (7,600 model--CVE pass records). Difficulty is strongly structured by language; the CVEs missed by every model concentrate in C infrastructure code. HoF-Bench provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create. The dataset is available at https://huggingface.co/datasets/aisleinc/HoF-Bench.

Figures

Figures reproduced from arXiv: 2607.27030 by Adam Krivka, Dmitrijs Trizna, Elnaz Babayeva, Evelina Gabasova, Guido Vranken, Igor Morgenstern, Jakub Kubik, Jakub Zikl, Jan Tattermusch, Jiri Balhar, Joshua Rogers, Luigino Camastra, Marek Pavelka, Martin Votruba, Michal Bida, Michal Buran, Michal Janocko, Ondrej Kokes, Patrik Mada, Pavel Kohout, Petr Simecek, Petr Stastny, Stanislav Fort, Tomas Dulka, Tomas Klohna, Vaclav Cadek.

Figure 1
Figure 1. Figure 1: HoF-Bench contains 95 public CVEs across eight repositories and three primary [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Reference-scanner pipeline and information boundary. The detector never receives [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Expected strict rediscovery rate: the percentage of the 95 benchmark CVEs found by [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Strict pass@4 recall by model and repository. Rows are ordered by overall recall; [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Posterior analyzer strength under the one-dimensional Rasch Beta-Binomial model. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Attributed API cost per benchmark CVE versus strict rediscovery. This is an [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 1 canonical work pages

  1. [1]

    CVE discoveries

    AISLE. CVE discoveries. Public Hall of Fame, 2026. URL https://aisle.com/ cve-discoveries. Accessed July 2026

    2026

  2. [2]

    Paul E. Black. The software assurance reference dataset (SARD). Technical Report NIST IR 8561, National Institute of Standards and Technology, 2025

    2025

  3. [3]

    Boland, Jr

    Frederick E. Boland, Jr. and Paul E. Black. The juliet 1.1 C/C++ and java test suite. Computer, 45(10):88–90, 2012. doi: 10.1109/MC.2012.345

    doi:10.1109/mc.2012.345 2012

  4. [4]

    Benchmarks as random variables—modeling overdispersion in LLM evaluation

    Michal Buran and Vaclav Cadek. Benchmarks as random variables—modeling overdispersion in LLM evaluation. InStructured Probabilistic Inference & Generative Modeling (SPIGM), ICML 2026 Workshop, 2026. URLhttps://openreview.net/forum?id=g5jezzrnct

    2026

  5. [5]

    Wagner, Baishakhi Ray, and Yizheng Chen

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David A. Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we? InProceedings of the 47th IEEE/ACM International Conference on Software Engineering, pages 1729–1741. IEEE, 2025. doi: 10.1109/ICSE55347.2025.00038

    arXiv 2025

  6. [6]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. A C/C++ code vulnerability dataset with code changes and CVE summaries. InProceedings of the 17th International Conference on Mining Software Repositories, pages 508–512. ACM, 2020. doi: 10.1145/3379597.3387501

    arXiv 2020

  7. [7]

    System over model: Zero-day discovery at the jagged frontier

    Stanislav Fort. System over model: Zero-day discovery at the jagged frontier. AISLE blog, April 2026. URL https://aisle.com/blog/ system-over-model-zero-day-discovery-at-the-jagged-frontier . Published April 14, 2026

    2026

  8. [8]

    AI cybersecurity after Mythos: The jagged fron- tier

    Stanislav Fort. AI cybersecurity after Mythos: The jagged fron- tier. AISLE blog, April 2026. URL https://aisle.com/blog/ ai-cybersecurity-after-mythos-the-jagged-frontier. Published April 7, 2026

    2026

  9. [9]

    Hoffman and Andrew Gelman

    Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo.Journal of Machine Learning Research, 15(47): 1593–1623, 2014

    2014

  10. [10]

    SEC-bench: Automated benchmarking of LLM agents on real-world software security tasks

    Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. SEC-bench: Automated benchmarking of LLM agents on real-world software security tasks. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  11. [11]

    Lord.Applications of Item Response Theory to Practical Testing Problems

    Frederic M. Lord.Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum Associates, Hillsdale, NJ, 1980. ISBN 0-89859-006-X

    1980

  12. [12]

    RealVuln: Benchmarking rule-based, general-purpose LLM, and security-specialized scanners on real-world code

    John Pellew and Faizan Raza. RealVuln: Benchmarking rule-based, general-purpose LLM, and security-specialized scanners on real-world code. arXiv preprint arXiv:2604.13764, 2026. URLhttps://arxiv.org/abs/2604.13764. 10

    Pith/arXiv arXiv 2026

  13. [13]

    Danish Institute for Educational Research, Copenhagen, Denmark, 1960

    Georg Rasch.Probabilistic Models for Some Intelligence and Attainment Tests. Danish Institute for Educational Research, Copenhagen, Denmark, 1960

    1960

  14. [14]

    Reckase.Multidimensional Item Response Theory

    Mark D. Reckase.Multidimensional Item Response Theory. Springer, New York, NY, 2009. doi: 10.1007/978-0-387-89976-3

    work page doi:10.1007/978-0-387-89976-3 2009

  15. [15]

    Silent failures in LLM vulnerability detection: The expensive model trap

    Petr Simecek, Vaclav Cadek, and Michal Buran. Silent failures in LLM vulnerability detection: The expensive model trap. InFailure Modes in Agentic AI (F AGEN), ICML 2026 Workshop, 2026. URLhttps://openreview.net/forum?id=0xklkDx3U1

    2026

  16. [16]

    DREA: Decoupled reasoning and exploration agents for repository-level vulnerability detection

    Mingyang Sun and Guozhu Meng. DREA: Decoupled reasoning and exploration agents for repository-level vulnerability detection. arXiv preprint arXiv:2607.13439, 2026. URL https://arxiv.org/abs/2607.13439

    Pith/arXiv arXiv 2026

  17. [17]

    Snyk vulnbench JS 1.0: Can LLMs find the same bugs twice? arXiv preprint arXiv:2606.15762, June 2026

    Liran Tal, Johannes Kloos, Arsenii Rudich, Stephen Thoemmes, and Manoj Nair. Snyk vulnbench JS 1.0: Can LLMs find the same bugs twice? arXiv preprint arXiv:2606.15762, June 2026. URLhttps://arxiv.org/abs/2606.15762

    arXiv 2026

  18. [18]

    Practical bayesian model evaluation using leave-one-out cross-validation and waic.Statistics and Computing, 27(5):1413–1432, 2017

    Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leave-one-out cross-validation and waic.Statistics and Computing, 27(5):1413–1432, 2017. doi: 10.1007/s11222-016-9696-4

    doi:10.1007/s11222-016-9696-4 2017

  19. [19]

    The agentic vulner- ability coverage map

    Corban Villa, Darya Kaviani, Alice Cheriton, and Raluca Ada Popa. The agentic vulner- ability coverage map. Berkeley Vulnerability Initiative, UC Berkeley EECS, 2026. URL https://vuln.cs.berkeley.edu/. Accessed July 2026

    2026

  20. [20]

    Teo, Yiling Lou, Yebo Feng, Chong Wang, and Dinil Mon Divakaran

    Alperen Yildiz, Sin G. Teo, Yiling Lou, Yebo Feng, Chong Wang, and Dinil Mon Divakaran. Benchmarking LLMs and LLM-based agents in practical vulnerability detection for code repositories. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30848–30865, Vienna, Austria, 2025. Association f...

    doi:10.18653/v1/2025.acl-long.1490 2025

  21. [21]

    Zhang et al

    Andy K. Zhang et al. BountyBench: Dollar impact of AI agent attackers and defenders on real-world cybersecurity systems. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  22. [22]

    Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. InAdvances in Neural Information Processing Systems, volume 32, pages 10197– 10207, 2019. 11 A Supplement: Benchmark Format Each scanner task includes repository URL, scan...

    2019