Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

In the largest autonomous-cybersecurity contest to date, the winning system won not by finding more bugs but by staying available and accurate across all seven phases while its closest rivals plateaued or crashed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:30 UTC pith:7E2SU5KM

load-bearing objection First real SoK of AIxCC, with a solid taxonomy and per-CPV baselines; the stability claim holds up, but the per-CPV 'no activity' annotations rest on telemetry the paper itself calls unreliable, and data is not yet public. the 3 major comments →

arxiv 2602.07666 v4 pith:7E2SU5KM submitted 2026-02-07 cs.CR cs.AI

SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

Cen Zhang , Younggi Park , Fabian Fleischer , Yu-Fu Fu , Jiho Kim , Dongkwan Kim , Youngjoon Kim , Qingxiao Xu
show 13 more authors
Andrew Chin Ze Sheng Hanqing Zhao Michael Pelican David J. Musliner Jeff Huang Jon Silliman Mikel Mcdaniel Jefferson Casavant Isaac Goldthwaite Nicholas Vidovich Matthew Lehman Taesoo Kim
This is my paper
classification cs.CR cs.AI
keywords AIxCCcyber reasoning systemsLLM-based vulnerability discoveryautomated patchingfuzzingcompetition designpatch correctnesssystem stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is the first systematic study of AIxCC, the largest competition to date for fully autonomous, LLM-driven systems that find and patch vulnerabilities in real-world open-source software. Drawing on all seven finalist codebases, the full competition database, and organizer and team discussions, it argues that the deciding factor in the final standings was not superior vulnerability-finding technique but system stability and submission accuracy: the winning system stayed available and accurate across all seven phases while its closest rivals plateaued or crashed. The authors separate technique from reliability by re-running foundational tools—a standard parallel fuzzer and two minimal LLM patch agents—against the same 63 challenge vulnerabilities under ideal conditions, finding that roughly half were solvable by a single baseline and that a system which reliably applied such baselines would have ranked in the top three. The upshot: engineering reliability is as decisive as AI capability in autonomous security software, and automated patch correctness remains an open problem.

Core claim

The paper's central finding is that in AIxCC, stability and accuracy were deciding factors in cyber reasoning system (CRS) performance. The winning system did not dominate because its vulnerability-detection technique was categorically superior; it led because it stayed available and accurate across all seven phases, while its closest rivals plateaued after phases 3 and 4 and other competitors suffered system-wide crashes or submission bugs. The authors support this by grounding scores in per-vulnerability analysis against foundational baselines—parallel fuzzing and two minimal LLM patch agents run under ideal conditions—which shows that roughly half of the 63 challenge vulnerabilities were

What carries the argument

The load-bearing object is the competition's scoring design—time-decayed rewards for proof-of-vulnerability inputs, patches, SARIF verdicts, and bundles, plus a quartic accuracy multiplier that harshly penalizes inaccurate submissions—paired with the authors' per-CPV annotation method: re-running a standard parallel fuzzer and two minimal LLM patch agents (a security-specialized retrieval agent and a general coding agent) against all 63 vulnerabilities under ideal laboratory conditions to label which challenges are solvable by foundational techniques. This annotation lets the authors distinguish capability gaps from reliability gaps: when a CRS misses a baseline-solvable challenge, the cause

Load-bearing premise

The stability conclusion rests on treating the absence of telemetry after certain phases as evidence that a CRS stopped working; the authors note that telemetry was selectively uploaded and unordered, so a system that kept running but stopped logging would be misread as crashed.

What would settle it

Re-run a similar multi-phase contest with mandatory OS-level liveness logging and post-competition forensic recovery of each system's final state, then check whether competitors that 'plateaued' had actually crashed or had merely stopped uploading logs; a second check would give every team an identical baseline CRS and measure score variance attributable to reliability alone.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Future autonomous security systems should treat uptime, submission accuracy, and resource scheduling as first-class design goals; a system that reliably applies off-the-shelf fuzzing and patch agents would place near the top of a similar contest.
  • Roughly half of real-world OSS challenge vulnerabilities are solvable by single foundational techniques, so the near-term frontier is less about new detection methods than about reliable end-to-end integration.
  • LLM-driven code understanding delivers measurable gains: CRSs solved 22 PoVs that parallel fuzzing could not, including inputs with complex grammars, logical constraints, and indirect calls.
  • Automated patch correctness remains unsolved: a large fraction of patches that pass build, reproduction, and functional tests are semantically wrong, so semantic validation will require manual review or new automated methods.
  • Competition telemetry should be designed for post-hoc analysis—logging intermediate outputs, decision traces, and environmental snapshots—because current telemetry is too incomplete to attribute failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If stability is decisive, then competition rankings measure engineering discipline as much as AI capability; a re-run with mandatory, complete liveness telemetry could substantially reshuffle the order.
  • Editorial: The scoring rules—time decay plus a quartic accuracy penalty—reward conservative, punctual submission, which may explain why the winning architecture favored ensembles of proven components over a single high-risk technique; a track that scores exploration rather than reliability would likely favor different architectures.
  • Editorial: Before autonomous patching can be trusted in production, evaluation needs a semantic-correctness oracle beyond build-and-test checks; the authors' two-reviewer manual validation could serve as a template for such benchmarks.
  • Editorial: The dominant causes of missed easy bugs were ecosystem frictions—nonstandard harnesses, self-downloading tools, and multi-hundred-megabyte build artifacts—suggesting that taming open-source build complexity, not AI reasoning, is the near-term bottleneck.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper is a Systematization of Knowledge study of DARPA's AI Cyber Challenge (AIxCC) final competition. It analyzes competition design, challenge construction, and scoring; taxonomizes the architectures of the seven finalist CRSs across PoV generation, patch generation, SARIF validation, and bundling; and evaluates per-CPV results against foundational baselines (a parallel fuzzer, Claude Code, and a multi-retrieval patch agent). The central findings are that stability and accuracy were the deciding factors in CRS performance (KF1), that a CRS reliably applying foundational techniques would have ranked in the top three (KF2), that LLM-driven techniques found PoVs that fuzzing could not (KF3), and that automated patch accuracy remains an open problem (KF4). The paper proposes lessons for future competitions and for deployment of autonomous CRSs.

Significance. If the causal claims are supported, this is a valuable first systematic study of the largest autonomous vulnerability discovery and repair competition to date. Strengths include the rare access to the official competition database, the 8,906 CPU-hour fuzzing annotation study, two-person-week manual patch validation, cross-validation by multiple authors, and a rich taxonomy of real CRS architectures. The scoreboard data themselves (Table 7) are hard evidence and will be a useful reference. However, the paper's headline conclusions depend on inferences from incomplete telemetry and on a counterfactual that the annotation experiments do not directly test. With revision, this could become the standard reference for AIxCC.

major comments (3)
  1. KF1 ('Stability and accuracy were deciding factors') is the manuscript's central claim, but its key support rests on an inference from missing telemetry. §7.1 says TB and TI 'plateaued after P3 and P4 respectively (telemetry shows no further activity after these points),' and Figure 3 marks CPVs as 'No CRS Activity' when the CRS 'did not send any log messages.' The paper itself concedes in §7.2 that competition telemetry is 'inherently incomplete and unordered' and that teams 'selectively uploaded logs for performance reasons.' Absence of uploaded logs is therefore also consistent with a system that continued running but stopped logging, or continued working but stopped submitting. The scoreboard flattening in Figure 2 only proves that scoring stopped; it does not establish the causal claim that TB/TI suffered system-wide failures. Because KF1, the 'system-wide failures' attribution, and
  2. KF2 ('A CRS that reliably applies foundational techniques in real-world conditions would rank among the top three') is a counterfactual that the annotation experiments do not support. PF, MR, and CC are run in isolation under ideal conditions: PF receives only vulnerable harnesses with 16 cores and shared seeds; MR/CC receive ground-truth PoVs with sanitizer logs. As the paper notes, these baselines 'do not need to handle the challenges of fully autonomous systems' such as self-discovery, deduplication, scheduling, and cross-challenge resource allocation. A CRS that 'reliably applies' these techniques in the AFC environment would have to solve exactly the integration problems that the paper identifies as hard. The top-three inference also ignores time-decay scoring, accuracy multipliers, and bundle timing, all of which affect final rank. I recommend either testing the counterfactual with
  3. The per-CPV 'No CRS Activity' annotation conflates absence of log messages with absence of activity. This is the same issue as Major Comment 1 but in a form that feeds all subsequent analysis: the columns labeled 'no CRS activity' are used to attribute individual CPV misses to 'system-wide failures' or 'critical bugs' rather than to technique capability. Because the annotation is a derived label and not raw scoreboard data, it should either be backed by corroborating non-log evidence (e.g., final submission timestamps and system heartbeat records) or renamed to 'no logged activity,' with the inference to system failure explicitly flagged as an assumption. As written, the annotation legend in Figure 3 and the narrative in §7.2 overstate what the telemetry can establish.
minor comments (6)
  1. The abstract says 'discussions with organizers and all finalist teams,' while §2 states 'questionnaires and meetings with most finalist teams.' Please reconcile these descriptions.
  2. Table 16 and Figures 4–5 are based on OTEL logs that the paper says are 'incomplete due to the competition environment.' Please label these as lower-bound estimates and state whether missingness is likely correlated with team, phase, or provider; otherwise the resource-utilization comparison (e.g., AT's higher budget use) may be biased by uneven log collection.
  3. CC was capped at 3 patch-generation attempts per CPV while MR determined iterations autonomously. The paper does not state whether this asymmetry affects the MR/CC comparison or the derived annotations; please discuss or control for it.
  4. The author list includes members of Team Atlanta and Kudu Dynamics, and the baseline tools used in §7.2 include the authors' own OSS-CRS/MR. This is disclosed in the acknowledgments but there is no formal conflict-of-interest statement. Please add one, and clarify the provenance of MR in the text.
  5. Minor typography: '42skips project tests' should read '42 skips project tests'; the table's '–' and '×' markers could be explained in a dedicated legend.
  6. The multi-panel figure is dense. Consider a high-resolution supplementary version or an interactive table so per-CPV annotations can be inspected independently.

Circularity Check

0 steps flagged

No derivation reduces to its inputs; the scoreboard is external and the stability conclusion is an interpretation, not a constructed prediction. Minor self-citations in baseline tools are noted but not load-bearing.

full rationale

This is an empirical SoK, not a formal derivation chain: there is no equation that is fitted and then re-presented as a prediction, and no definition smuggled in via self-citation. The central claim KF1 ('Stability and accuracy were deciding factors in CRS performance') rests on the organizer-produced scoreboard (Table 7) and score-vs-time curves (Figure 2), which are external data, not artifacts of the authors' own construction. The per-CPV annotations use PF/MR/CC baselines, including tools from the authors' own OSS-CRS project [54] and the Team Atlanta MR agent [57]; this is a methodological self-reference, but the comparisons are independently checkable and are not equivalent to the conclusions they support. The paper itself flags a serious limitation at §7.2: 'Competition telemetry—traces, logs, and events submitted to the infrastructure—is inherently incomplete and unordered: teams selectively uploaded logs for performance reasons.' That caveat directly weakens the §7.1 inference that TB and TI 'plateaued' because 'telemetry shows no further activity after these points,' since absence of log messages may not equal absence of activity or functionality. However, this is an evidence-quality and interpretive problem, not a circular reduction: the stability claim does not redefine its inputs or treat the telemetry inference as a derived prediction. No self-definitional, fitted-input-called-prediction, imported-uniqueness, or ansatz-smuggling patterns are present. Score 2 reflects only minor, non-load-bearing self-citations in the baseline and related-work apparatus.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters or invented entities appear because this is an empirical/observational SoK rather than a fitted theoretical model. The load-bearing assumptions are about data completeness, telemetry interpretation, baseline validity, and manual patch verification, all of which are domain assumptions rather than mathematical axioms.

axioms (4)
  • domain assumption The official AFC scoreboard and organizer-provided competition database are accurate and complete.
    All performance data 'originates from AFC competition results' and is provided by the organizer, who is a co-author of this paper; the paper does not independently re-verify the official scores. This underlies every score-based claim in §7.
  • domain assumption The absence of telemetry activity indicates a CRS stopped functioning.
    The paper states in §7.2 that telemetry is 'inherently incomplete and unordered' and teams selectively uploaded logs, yet §7.1 uses telemetry to conclude 'TB and TI ... plateaued after P3 and P4 respectively' and 'telemetry shows no further activity.' If an active system simply stopped uploading logs, the 'stability wins' claim would be over-attributed.
  • domain assumption Foundational baseline annotations (PF, MR, CC) under ideal lab conditions are a valid measure of inherent CPV difficulty.
    CPVs are labeled fuzz-discoverable if any of three parallel fuzzing runs succeeds, and patchable if any of three MR/CC runs produces a manually verified valid patch. These unions are generous, but they assume that ideal-condition success transfers to 'solvable by a foundational technique' in the competition context, which the paper uses to argue that many unsolved CPVs reflect system failures rather than capability gaps.
  • domain assumption Manual review by two independent authors is a reliable ground truth for patch semantic correctness.
    The high rates of semantically incorrect patches (CC: 20/53, MR: 26/57) rest on manual verification by two reviewers, with no reported inter-rater reliability metric or adjudication process. This is standard practice but is still an unverified judgment call that affects KF4.

pith-pipeline@v1.3.0-alltime-deepseek · 33467 in / 8712 out tokens · 95511 ms · 2026-08-03T03:30:11.382879+00:00 · methodology

0 comments
read the original abstract

DARPA's AI Cyber Challenge (AIxCC, 2023--2025) is the largest competition to date for building fully autonomous cyber reasoning systems (CRSs) that leverage recent advances in AI -- particularly large language models (LLMs) -- to discover and remediate vulnerabilities in real-world open-source software. This paper presents the first systematic analysis of AIxCC. Drawing on design documents, source code, execution traces, and discussions with organizers and competing teams, we examine the competition's structure and key design decisions, characterize the architectural approaches of finalist CRSs, and analyze competition results beyond the final scoreboard. Our analysis reveals the factors that truly drove CRS performance, identifies genuine technical advances achieved by teams, and exposes limitations that remain open for future research. We conclude with lessons for organizing future competitions and broader insights toward deploying autonomous CRSs in practice.

Figures

Figures reproduced from arXiv: 2602.07666 by Andrew Chin, Cen Zhang, David J. Musliner, Dongkwan Kim, Fabian Fleischer, Hanqing Zhao, Isaac Goldthwaite, Jefferson Casavant, Jeff Huang, Jiho Kim, Jon Silliman, Matthew Lehman, Michael Pelican, Mikel Mcdaniel, Nicholas Vidovich, Qingxiao Xu, Taesoo Kim, Younggi Park, YoungJoon Kim, Yu-Fu Fu, Ze Sheng.

Figure 1
Figure 1. Figure 1: AFC workflow. GitHub webhooks trigger challenge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Score per time (top) and phase (bottom) axes. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Team performance per CPV (CWE-wise breakdowns are in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Token consumption (input + output) per model by team. OpenAI Anthropic Google xAI Other gpt-4.1 gpt-4.1-mini gpt-4.1-nano gpt-4o gpt-4o-mini o3 o3-mini o4-mini claude-3-haiku claude-3.5-haiku claude-3.5-sonnet claude-3.7-sonnet claude-opus-4 claude-sonnet-4 gemini-1.5-pro gemini-2.0-flash gemini-2.5-flash gemini-2.5-pro grok-3 custom AT TB TI FB SP 42 LC 12.3 22.9 15.6 22.4 4.4 9.0 6.0 7.9 6.3 4.7 4.7 37.5… view at source ↗
Figure 5
Figure 5. Figure 5: I/O token ratio per model by team. from 80%–90%, with patch rates being below average; some teams achieved 100% accuracy in certain scoring categories. Appendix N. CWE-Wise Performance Analysis [PITH_FULL_IMAGE:figures/full_fig_p025_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Submission timing by type. Each task has a fixed time window (12h for full-mode, 6h for delta-mode); 0% is [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CWE-wise PoV generation performance heatmap. Note: zero-day CPVs are excluded as they are still under [PITH_FULL_IMAGE:figures/full_fig_p026_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: CWE-wise patch generation performance heatmap. Note: zero-day CPVs are excluded as they are still under [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SeedSmith: LLM-Driven Seed Synthesis for Directed Fuzzing

    cs.CR 2026-07 accept novelty 7.0

    An agentic LLM seed generator that recovers missing call edges and crash preconditions yields 11–15× geometric-mean crash-time speedups on Magma and unlocks 16 previously unreachable ARVO bugs.

  2. SeedSmith: LLM-Driven Seed Synthesis for Directed Fuzzing

    cs.CR 2026-07 conditional novelty 7.0

    An agentic LLM pipeline generates sink-targeted fuzzing seeds that on Magma give geomean crash-time speedups of 11.51-14.66x and unlock 16 ARVO bugs unreachable with default seeds.

  3. OverrideFuzz: Semantic-Aware Grammar Fuzzing for Script-Runtime Vulnerabilities

    cs.CR 2026-05 conditional novelty 7.0

    OverrideFuzz uses semantic-aware grammar fuzzing with reflection to model override hooks and dynamic rebinding, producing coverage growth and inputs that match known vulnerability patterns on CPython, Lua, and QuickJS...

  4. Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilities

    cs.CR 2026-06 unverdicted novelty 6.0

    Chai uses AI to enhance differential testing for cryptographic misuse, cataloging library-level flaws and propagating them to find over 100 vulnerabilities including a critical one in a widely deployed SSL library.

  5. Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases

    cs.CR 2026-06 unverdicted novelty 6.0

    Revelio combines LLMs, static analysis, and sanitizer-verified PoVs to scalably discover memory safety vulnerabilities in repository-scale code, finding 19 new bugs in long-fuzzed projects at low cost.

  6. Quality-Assured Fuzz Harness Generation via the Four Principles Framework

    cs.CR 2026-05 unverdicted novelty 6.0

    QuartetFuzz introduces the Four Principles framework for harness correctness and deploys an autonomous LLM agent that produces verified harnesses, yielding 29 confirmed bugs across 23 projects and identifying violatio...

Reference graph

Works this paper leans on

72 extracted references · 5 linked inside Pith · cited by 5 Pith papers

  1. [1]

    GigE Vision standard

    AIA (Association for Advancing Automation). GigE Vision standard. Accessed: 2026-02-05. URL: https://www.automate.org/vision/vision -standards/vision-standards-gige-vision

    2026

  2. [2]

    Claude Code: An agentic coding tool that lives in your terminal, 2025

    Anthropic. Claude Code: An agentic coding tool that lives in your terminal, 2025. https://github.com/anthropics/claude-code

    2025

  3. [3]

    Apache Maven Project

    Apache Software Foundation. Apache Maven Project. https://maven. apache.org/, 2025. Accessed: 2025-12-12

    2025

  4. [4]

    OSS-Fuzz

    Abhishek Arya, Oliver Chang, Jonathan Metzman, Kostya Serebryany, and Dongge Liu. OSS-Fuzz. https://github.com/google/oss-fuzz

  5. [5]

    Nautilus: Fishing for deep bugs with grammars

    Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. Nautilus: Fishing for deep bugs with grammars. InNDSS, volume 19, page 337, 2019

    2019

  6. [6]

    IJON: Exploring deep state spaces via fuzzing

    Cornelius Aschermann, Sergej Schumilo, Ali Abbasi, and Thorsten Holz. IJON: Exploring deep state spaces via fuzzing. In2020 IEEE Symposium on Security and Privacy (SP), pages 1597–1612. IEEE, 2020

    2020

  7. [7]

    LiteLLM: Call 100+ LLM APIs in OpenAI format

    BerriAI. LiteLLM: Call 100+ LLM APIs in OpenAI format. https: //github.com/BerriAI/litellm, 2026. Accessed: 2026

    2026

  8. [8]

    AURORA: Statistical crash analysis for automated root cause explanation

    Tim Blazytko, Moritz Schlögel, Cornelius Aschermann, Ali Abbasi, Joel Frank, Simon Wörner, and Thorsten Holz. AURORA: Statistical crash analysis for automated root cause explanation. In29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2020

    2020

  9. [9]

    Directed Greybox Fuzzing

    Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. Directed Greybox Fuzzing. InProceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS’17), pages 2329–2344. ACM, 2017. doi:10.1145/ 3133956.3134020. 13

    arXiv 2017

  10. [10]

    KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs

    Cristian Cadar, Daniel Dunbar, and Dawson Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In8th USENIX Symposium on Operating Systems Design and Implementation (OSDI’08), pages 209–224. USENIX Association, 2008

    2008

  11. [11]

    ccache: A fast C/C++ compiler cache

    ccache developers. ccache: A fast C/C++ compiler cache. https: //ccache.dev/, 2025. Accessed: 2025-12-12

    2025

  12. [12]

    LangChain, 2022

    Harrison Chase. LangChain, 2022. https://github.com/langchain-ai/la ngchain

    2022

  13. [13]

    Enfuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers

    Yuanliang Chen, Yu Jiang, Fuchen Ma, Jie Liang, Mingzhe Wang, Chijin Zhou, Xun Jiao, and Zhuo Su. Enfuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers. InProceedings of the 28th USENIX Security Symposium (Security), Santa Clara, CA, August 2019

    2019

  14. [14]

    Universal Ctags. ctags. https://github.com/universal-ctags/ctags, 2025. Accessed: 2025-07-27

    2025

  15. [15]

    curl-fuzzer: Quality assurance testing for the curl project

    curl project. curl-fuzzer: Quality assurance testing for the curl project. https://github.com/curl/curl-fuzzer, 2025. Accessed: 2025-12-12

    2025

  16. [16]

    ast-grep: A cli tool for code structural search, lint, and rewriting

    Herrington Darkholme. ast-grep: A cli tool for code structural search, lint, and rewriting. https://github.com/ast-grep/ast-grep, 2026. Accessed: 2026

    2026

  17. [17]

    Cyber Grand Challenge, 2016

    DARPA. Cyber Grand Challenge, 2016. https://www.darpa.mil/resear ch/programs/cyber-grand-challenge

    2016

  18. [18]

    AIxCC Semifinal Competition, 2024

    DARPA. AIxCC Semifinal Competition, 2024. https://aicyberchallen ge.com/semifinal-competition/

    2024

  19. [19]

    AI Cyber Challenge (AIxCC), 2025

    DARPA. AI Cyber Challenge (AIxCC), 2025. https://aicyberchallen ge.com/

    2025

  20. [20]

    AIxCC Archive, 2025

    DARPA. AIxCC Archive, 2025. https://archive.aicyberchallenge.com/

    2025

  21. [21]

    JaCoCo: Java code coverage library

    EclEmma. JaCoCo: Java code coverage library. https://www.eclemm a.org/jacoco/, 2026. Accessed: 2026

    2026

  22. [22]

    AFL++: Combining incremental steps of fuzzing research

    Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. AFL++: Combining incremental steps of fuzzing research. In14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association, August 2020

    2020

  23. [23]

    LibAFL: A Framework to Build Modular and Reusable Fuzzers

    Andrea Fioraldi, Dominik Maier, Dongjia Zhang, and Davide Balzarotti. LibAFL: A Framework to Build Modular and Reusable Fuzzers. InProceedings of the 29th ACM Conference on Com- puter and Communications Security, CCS ’22, pages 1331–1347, New York, NY , USA, 2022. Association for Computing Machinery. doi:10.1145/3548606.3560602

    arXiv 2022

  24. [24]

    GDB: The GNU project debugger

    Free Software Foundation. GDB: The GNU project debugger. https: //sourceware.org/gdb/, 2026. Accessed: 2026

    2026

  25. [25]

    autofz: Automated Fuzzer Composition at Runtime

    Yu-Fu Fu, Jaehyuk Lee, and Taesoo Kim. autofz: Automated Fuzzer Composition at Runtime. InProceedings of the 32nd USENIX Security Symposium (Security), Anaheim, CA, August 2023

    2023

  26. [26]

    Aider: AI pair programming in your terminal, 2025

    Paul Gauthier. Aider: AI pair programming in your terminal, 2025. https://aider.chat/

    2025

  27. [27]

    CodeQL: The libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security, 2025

    GitHub. CodeQL: The libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security, 2025. https://codeql.github.com/

    2025

  28. [28]

    SARIF support for code scanning, 2025

    GitHub. SARIF support for code scanning, 2025. https://docs.github. com/en/code-security/reference/code-scanning/sarif-support-for-cod e-scanning

    2025

  29. [29]

    Clusterfuzz: Scalable fuzzing infrastructure

    Google. Clusterfuzz: Scalable fuzzing infrastructure. https://github.c om/google/clusterfuzz, 2026. Accessed: 2026-02-01

    2026

  30. [30]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Vibe Coding, 2025

    Andrej Karpathy. Vibe Coding, 2025. https://x.com/karpathy/status/ 1886192184808149383

    2025

  32. [32]

    Patchisland: Orchestration of llm agents for continuous vulnerability repair, 2026

    Wonyoung Kim, Seunggi Min, Minjae Gwon, Dowoo Baik, Haein Lee, Hyeon Heo, Minjae Lee, Min Woo Baek, Yonghwi Jin, Younggi Park, Yunjae Choi, Taesoo Kim, Sangdon Park, and Insu Yun. Patchisland: Orchestration of llm agents for continuous vulnerability repair, 2026. URL: https://arxiv.org/abs/2601.17471,arXiv:2601.17471

    Pith/arXiv arXiv 2026

  33. [33]

    Logs in, patches out: Automated vulnerability repair via {Tree-of- Thought}{LLM} analysis

    Youngjoon Kim, Sunguk Shin, Hyoungshick Kim, and Jiwon Yoon. Logs in, patches out: Automated vulnerability repair via {Tree-of- Thought}{LLM} analysis. In34th USENIX Security Symposium (USENIX Security 25), pages 4401–4419, 2025

    2025

  34. [34]

    LangGraph: Build resilient language agents as graphs,

    LangChain, Inc. LangGraph: Build resilient language agents as graphs,

  35. [35]

    LLVM: A compilation framework for lifelong program analysis and transformation

    Chris Lattner and Vikram Adve. LLVM: A compilation framework for lifelong program analysis and transformation. InInternational Symposium on Code Generation and Optimization (CGO’04), pages 75–86. IEEE, 2004

    2004

  36. [36]

    VULCANBOOST: Boosting ReDoS Fixes through Symbolic Representation and Feature Normalization

    Yeting Li, Yecheng Sun, Zhiwu Xu, Haiming Chen, Xinyi Wang, Hengyu Yang, Huina Chao, Cen Zhang, Yang Xiao, Yanyan Zou, et al. VULCANBOOST: Boosting ReDoS Fixes through Symbolic Representation and Feature Normalization. In34th USENIX Security Symposium (USENIX Security 25), pages 4463–4479, 2025

    2025

  37. [37]

    UndefinedBehaviorSanitizer, 2025

    LLVM Project. UndefinedBehaviorSanitizer, 2025. https://clang.llvm .org/docs/UndefinedBehaviorSanitizer.html

    2025

  38. [38]

    FuzzedDataProvider: A helper class for fuzz targets

    LLVM Project. FuzzedDataProvider: A helper class for fuzz targets. https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/ fuzzer/FuzzedDataProvider.h, 2026. Accessed: 2026

    2026

  39. [39]

    Infer: A static analyzer for Java, C, C++, and Objective-C,

    Meta. Infer: A static analyzer for Java, C, C++, and Objective-C,

  40. [40]

    Language Server Protocol specification

    Microsoft. Language Server Protocol specification. https://microsoft. github.io/language-server-protocol/, 2026. Accessed: 2026

    2026

  41. [41]

    Common Weakness Enumeration (CWE)

    MITRE. Common Weakness Enumeration (CWE). https://cwe.mitre. org/, 2026. Accessed: 2026-02-03

    2026

  42. [42]

    Fuzzbomb: Fully-autonomous detection and repair of cyber vulnerabilities.International Journal on Advances in Security, 9(3-4), 2016

    David J Musliner, Scott E Friedman, Michael Boldt, J Benton, Max Schuchard, and Peter Keller. Fuzzbomb: Fully-autonomous detection and repair of cyber vulnerabilities.International Journal on Advances in Security, 9(3-4), 2016

    2016

  43. [43]

    Musliner, Scott E

    David J. Musliner, Scott E. Friedman, Jeffrey M. Rye, and Tom Marble. Meta-control for adaptive cybersecurity in FUZZBUSTER. InProc. IEEE Int’l Conf. on Self-Adaptive and Self-Organizing Systems, sep 2013

    2013

  44. [44]

    Musliner, Michael J

    David J. Musliner, Michael J. S. Pelican, Matthew McLure, Steven Johnston, Richard G. Freedman, and Corey Knutson. OpenMIND: Planning and adapting in domains with novelty. InAdvances in Cognitive Systems, November 2021. URL: https://www.sift.net/sites/d efault/files/publications/main_5.pdf

    2021

  45. [45]

    Static Analysis Results Interchange Format (SARIF) Version 2.1.0, 2020

    OASIS. Static Analysis Results Interchange Format (SARIF) Version 2.1.0, 2020. https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0. html

    2020

  46. [46]

    Using Logprobs

    OpenAI. Using Logprobs. https://developers.openai.com/cookbook/e xamples/using_logprobs, 2023. Accessed: 2026-02-01

    2023

  47. [47]

    jdb: The Java debugger

    Oracle. jdb: The Java debugger. https://docs.oracle.com/en/java/java se/21/docs/specs/man/jdb.html, 2026. Accessed: 2026

    2026

  48. [48]

    Symbolic execution with SymCC: Don’t interpret, compile! In29th USENIX Security Symposium (USENIX Security 20), pages 181–198, Berkeley, CA, USA, August 2020

    Sebastian Poeplau and Aurélien Francillon. Symbolic execution with SymCC: Don’t interpret, compile! In29th USENIX Security Symposium (USENIX Security 20), pages 181–198, Berkeley, CA, USA, August 2020. USENIX Association. URL: https://www.usenix .org/conference/usenixsecurity20/presentation/poeplau

    2020

  49. [49]

    Semgrep: Lightweight static analysis for many lan- guages, 2025

    Semgrep, Inc. Semgrep: Lightweight static analysis for many lan- guages, 2025. https://semgrep.dev/

    2025

  50. [50]

    {AddressSanitizer}: A fast address sanity checker

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. {AddressSanitizer}: A fast address sanity checker. In2012 USENIX annual technical conference (USENIX ATC 12), pages 309–318, 2012. 14

    2012

  51. [51]

    Bandfuzz: An ml-powered collaborative fuzzing framework, 2025

    Wenxuan Shi, Hongwei Li, Jiahao Yu, Xinqian Sun, Wenbo Guo, and Xinyu Xing. Bandfuzz: An ml-powered collaborative fuzzing framework, 2025. URL: https://arxiv.org/abs/2507.10845, arXiv: 2507.10845

    Pith/arXiv arXiv 2025

  52. [52]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL: https://openreview.net/f orum?id=vAElhFcKW6

    2023

  53. [53]

    SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis

    Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Audrey Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, and Giovanni Vigna. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. In2016 IEEE Symposium on Security and Privacy (SP), pages 138–157. IEEE, 2016.doi:10.1109/SP.2016.17

    doi:10.1109/sp.2016.17 2016

  54. [54]

    OSS-CRS: Open-Source Software Cyber Reasoning System

    SSLab, Georgia Tech. OSS-CRS: Open-Source Software Cyber Reasoning System. https://github.com/sslab-gatech/oss-crs, 2025. Accessed: 2025-12-12

    2025

  55. [55]

    DSPy: The framework for programming—not prompt- ing—language models, 2025

    Stanford NLP. DSPy: The framework for programming—not prompt- ing—language models, 2025. https://github.com/stanfordnlp/dspy

    2025

  56. [56]

    SVF: Interprocedural static value-flow analysis in LLVM

    Yulei Sui and Jingling Xue. SVF: Interprocedural static value-flow analysis in LLVM. InProceedings of the 25th International Conference on Compiler Construction, CC ’16, pages 265–266. ACM, 2016. doi:10.1145/2892208.2892235

    arXiv 2016

  57. [57]

    (Team Atlanta)

    Taesoo Kim et al. (Team Atlanta). Atlantis: Ai-driven threat localiza- tion, analysis, and triage intelligence system, 2025. Cross-listed in cs.AI. URL: https://arxiv.org/abs/2509.14589, arXiv:2509.14589, doi:10.48550/arXiv.2509.14589

    doi:10.48550/arxiv.2509.14589 2025

  58. [58]

    42-b3yond-6ug Open Letter

    Team 42 (b3yond 6ug). 42-b3yond-6ug Open Letter. https://docs.goo gle.com/document/d/1-1TexnOwQGj2KJ8rrLtk-Vgl-PqO5MZe7k0q rHQPC_0/, 2025

    2025

  59. [59]

    Branch Flipper: Unlocking fuzz blockers with coverage- grounded LLMs

    Theori. Branch Flipper: Unlocking fuzz blockers with coverage- grounded LLMs. https://theori-io.github.io/aixcc-public/afc/Branch %20Flipper.pdf, 2025. AIxCC Technical Report

    2025

  60. [60]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  61. [61]

    Tree-sitter: An incremental parsing system for program- ming tools, 2025

    Tree-sitter. Tree-sitter: An incremental parsing system for program- ming tools, 2025. https://github.com/tree-sitter/tree-sitter

    2025

  62. [62]

    W ALA: T.J

    W ALA. W ALA: T.J. Watson libraries for analysis. https://github.com /wala/W ALA, 2026. Accessed: 2026

    2026

  63. [63]

    Machine vs

    Mike Walker. Machine vs. Machine: Lessons from the First Year of Cyber Grand Challenge. In24th USENIX Security Symposium (USENIX Security 15), Washington, D.C., August 2015. USENIX Association

    2015

  64. [64]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR), 2023. URL: https://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  65. [65]

    Modeling and Discovering Vulnerabilities with Code Property Graphs

    Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and Discovering Vulnerabilities with Code Property Graphs. In2014 IEEE Symposium on Security and Privacy, pages 590–604. IEEE, 2014.doi:10.1109/SP.2014.44

    doi:10.1109/sp.2014.44 2014

  66. [66]

    GNU Global: Source code tagging system

    Shigio Yamaguchi. GNU Global: Source code tagging system. https: //www.gnu.org/software/global/, 2026. Accessed: 2026

    2026

  67. [67]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: agent-computer interfaces enable automated software engineering. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA, 2024. Curran Associates Inc

    2024

  68. [68]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  69. [69]

    Patchagent: A practical program repair agent mimicking human expertise

    Zheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu, Meng Xu, Dongliang Mu, Yan Chen, and Xinyu Xing. Patchagent: A practical program repair agent mimicking human expertise. In34rd USENIX Security Symposium (USENIX Security 25), 2025

    2025

  70. [70]

    QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing

    Insu Yun, Sangho Lee, Meng Xu, Yeongjin Jang, and Taesoo Kim. QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing. InProceedings of the 27th USENIX Security Symposium (Security), Baltimore, MD, August 2018. 15 Appendix A. Ethical Considerations Stakeholder Identification.We identify three primary stake- holder groups.(1) Researchers an...

    2018

  71. [2024]

    https://github.com/langchain-ai/langgraph

  72. [2025]

    https://fbinfer.com/