Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read At the largest autonomous-cyber competition to date, reliability beat raw technique, LLM-driven analysis found 22 vulnerabilities fuzzing missed, and automated patching still demands human review.

desk verdict First systematic SoK of AIxCC with real new measurements; the stability/accuracy thesis holds, but the LLM-attribution claim for the 22 non-PF PoVs is not established and Table 7's penalty arithmetic is inconsistent. read the letter →

arxiv 2602.07666 v5 pith:7E2SU5KM submitted 2026-02-07 cs.CR cs.AI

Cen Zhang , Younggi Park , Fabian Fleischer , Yu-Fu Fu , Jiho Kim , Dongkwan Kim , Youngjoon Kim , Qingxiao Xu
show 13 more authors
Andrew Chin Ze Sheng Hanqing Zhao Michael Pelican David J. Musliner Jeff Huang Jon Silliman Mikel Mcdaniel Jefferson Casavant Isaac Goldthwaite Nicholas Vidovich Matthew Lehman Taesoo Kim
This is my paper · ORCID
classification cs.CRcs.AI
keywords AIxCCcyberreasoningsystemsLLMvulnerabilitydiscoveryautomatedpatchingparallelfuzzingcompetitiondesignsystemreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The AI Cyber Challenge (AIxCC) was the largest test yet of fully autonomous cyber reasoning systems — software that uses large language models to discover and patch vulnerabilities in real open-source C and Java projects while running unattended for roughly 143 hours. The paper's central assertion is that the outcome was decided by reliability, not raw technique: the winning system outscored second place by about 80%, and most of the margin accumulated after rival systems crashed or plateaued mid-competition. Its second claim is that LLM-based analysis contributed genuinely: the seven finalists together produced proof-of-vulnerability inputs for 22 planted vulnerabilities that a dedicated six-hour parallel-fuzzing baseline could not trigger, clustered in code diffs, indirect calls, grammar-heavy formats, and logic-guarded paths — and every team found at least one true, previously unknown vulnerability. Its third claim is that automated patching is the open problem: even under ideal laboratory conditions, off-the-shelf repair agents produced semantically incorrect patches in 38–46% of cases while passing every automated check, and competition systems likewise emitted bursts of invalid patches. If the paper is right, the field's real bottleneck is not model capability but operational reliability, submission accuracy, and patch trust.

What carries the argument

The argument's load-bearing machinery is a difficulty-annotation methodology: foundational techniques were run under ideal laboratory conditions on the same 63 planted vulnerabilities, and the finalists' results were overlaid on top to separate 'hard for everyone' from 'solvable in principle, unsolved in practice.' The discovery baseline was six hours of parallel fuzzing (16 cores per harness, three runs), which marked 34 of 63 vulnerabilities discoverable — 75% of C but only 17% of Java. The patch baselines were two off-the-shelf repair agents given ground-truth reports, marking 31 and 33 vulnerabilities patchable, with every passing patch manually reviewed. This lets the paper attribute ou

What would settle it

An ablation on the 22 beyond-fuzzing vulnerabilities: run the relevant finalist pipelines with all LLM components replaced by deterministic substitutes while keeping their non-LLM enhancements; if a substantial share still yield triggering inputs, the 'LLM-driven code understanding' attribution fails. A check already available in the paper's data: tabulate which teams produced each of the 22 finds — the most fuzzing-centric finalist had the second-highest discovery score, so the distribution alone would indicate how much of the effect is genuinely LLM-attributable.

Watch

Extended reading notes

Core claim

On competition-scale evidence, the paper establishes three things. First, stability and submission accuracy outweighed analytical capability: decisive score gaps came from systems that stopped functioning or accumulated accuracy penalties, and a hypothetical CRS that merely applied off-the-shelf techniques reliably would have ranked in the top three. Second, LLM-driven code understanding produced measurable discovery gains beyond fuzzing — 22 proof-of-vulnerability inputs across code diffs, indirect calls, grammar-constrained formats, and logic-guarded paths, plus 25 genuine zero-days found by the seven finalists — reasoning that feedback-driven fuzzing cannot match. Third, automated patchin

Load-bearing premise

The load-bearing premise is the paper's causal attribution: that the 22 vulnerabilities found beyond parallel fuzzing were found thanks to LLM code understanding, rather than the non-LLM enhancements (concolic execution, directed fuzzing, grammars, dictionaries, better sanitizers) that several finalists also deployed — a distinction the paper does not test per vulnerability.

Editorial extensions

If this is right

  • A system that simply applied off-the-shelf techniques — parallel fuzzing plus a standard repair agent — would have ranked in the top three of the final competition, so dependable execution, not novel analysis, is the near-term path to deployable autonomous security.
  • LLM-based vulnerability analysis earns its keep precisely where fuzzing is structurally weak: delta-mode code changes, indirect calls, grammar-constrained inputs, and guards such as regexes, encodings, and compression — the 22 beyond-fuzzing discoveries concentrate in exactly those cases.
  • Machine-generated patches must be treated as candidate fixes, not fixes: a large share pass every automated validation yet change behavior subtly or miss the root cause, so human review remains load-bearing.
  • Competition telemetry should be designed for retrospective analysis, not just live scoring; the paper's main analytical obstacle was unordered, incomplete execution traces that made per-technique attribution impossible.
  • Every finalist discovered at least one genuine zero-day — 25 distinct previously unknown vulnerabilities across ten projects, twelve of them patched — showing autonomous systems already contribute real discovery at scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's attribution of the 22 beyond-fuzzing finds to 'LLM-driven code understanding' is a causal claim its own data only partially supports: the same teams also ran non-LLM techniques (concolic execution, directed and grammar-aware fuzzing, dictionaries, improved sanitizers), the most fuzzing-centric finalist posted the second-highest discovery score, and the paper concedes attribution to any
  • If reliability is the binding constraint, expect near-term deployments to be conservative: deterministic tools doing the heavy lifting with LLM components gated behind verification, rather than the fully autonomous agent fleets the competition showcased.
  • Only about half of the planted real-world-style vulnerabilities yielded to today's automated discovery, and fewer to trustworthy patching; that ratio implies a near-term division of labour in which machines triage and propose while humans review and dispose.
  • The baseline-annotation method could be productized as a benchmark axis: pre-rating each challenge by whether off-the-shelf tooling solves it, and scoring systems on baseline-hard vulnerabilities, would change what winning a cyber competition measures — from volume to genuine advance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This SoK paper presents the first systematic analysis of DARPA's AI Cyber Challenge (AIxCC) final competition, covering competition design, the architectures of all seven finalist CRSs, and a per-CPV analysis of results using three foundational baselines: parallel fuzzing (PF), the MultiRetrieval patch agent (MR), and Claude Code (CC). The main findings are: (KF1) stability and accuracy, not raw capability, decided final scores; (KF2) technical sophistication and engineering reliability are distinct challenges; (KF3) CRSs found 22 PoVs that PF could not, attributed to LLM-driven code understanding; and (KF4) automated patch accuracy remains an open problem. The paper draws on organizer data, team codebases, telemetry, and discussions with all finalist teams, and includes substantial new annotation experiments with documented costs and manual patch validation.

Significance. If the results hold, this is an important and timely reference for competition design, autonomous vulnerability discovery, and LLM-based security research. The paper is strongest in its descriptive and taxonomical material: the architecture comparison across all seven finalists is careful and cross-validated, the scoring analysis is grounded in external competition data, and the annotation experiments are concrete, with specified CPU hours, LLM costs, run counts, and two-person-week manual patch validation. The central qualitative message — that winning AIxCC was primarily an engineering-reliability achievement and that automated patching remains semantically unreliable — is credible and significant. The main weaknesses are the unsupported causal attribution in KF3 and an internal arithmetic inconsistency in the key score table. Both are fixable within the scope of a revision.

major comments (3)
  1. [Table 7 / §7.1] The Final column is inconsistent with the Pen row as printed. For TI, the component rows sum to 54.3 + 101.6 + 4.9 + 49.8 = 210.7; applying the documented -16.3 penalty would give 194.4, not the printed Final of 210.7. The text says TI's pre-penalty score was higher than TB's, which requires a pre-penalty total of 227.0, not the component sum. The same arithmetic issue appears for every team (e.g., AT 392.8 with Pen -0.4). Since KF1's 'accuracy matters' evidence depends on the size and rank effect of the penalty, please clarify whether the component rows and Final are pre- or post-penalty, and make the table internally consistent under one stated convention.
  2. [§7.3, KF3] The existence of 22 non-PF PoVs is supported by the annotation experiment, but the causal claim 'thanks to LLM-driven code understanding' is not established. Table 3 shows that AT, SP, and 42 also fielded non-LLM enhancements (concolic execution, directed fuzzing, dictionary generation, improved sanitizers, added fuzzers), and the paper reports no per-CPV attribution of which component produced the first successful PoV for each of the 22 CPVs. 'Improvement over PF is observed across most teams' is a team-level statement, not a CPV-level one. The FB example is a clean natural experiment only for FB's subset. Please add per-CPV attribution from telemetry where available, or at least a sensitivity analysis restricted to teams without non-LLM fuzzing enhancements, or rewrite KF3 to the weaker, supported claim that CRSs collectively exceeded PF and that FB demonstrates LLM-driven generation c
  3. [§7.2, baseline provenance] The foundational baselines PF and MR are taken from OSS-CRS [54], developed by the authors' own lab, and the winning CRS (AT) is from the same group. The scoreboard data are external, so the central KF1/KF4 claims are not circular, but the 'foundational' annotations underlying KF3/KF4 may be sensitive to baseline selection. Please disclose this overlap explicitly, justify the neutrality of the chosen baseline (e.g., fixed public configuration, standard OSS-Fuzz settings), and ideally add an independent fuzzing baseline such as stock libFuzzer or AFL++ with the same harnesses. Without this, the exact counts 34/63 PF-solvable and 36/63 MR/CC-patchable cannot be independently audited by readers.
minor comments (4)
  1. [§7.2] The statement that 'a CRS that reliably applies foundational techniques in real-world conditions would rank among the top three' is asserted without a scoring simulation. Ranking depends on time decay, bundling, SARIF scoring, and accuracy penalties, not only on the number of PF/MR/CC-annotatable CPVs. Please derive this from the actual score model or soften the claim.
  2. [Figure 3 / §7.3] The per-CPV matrices are very dense, and the markers for PF/CC/MR annotations and the zero-day submatrices are hard to distinguish in print. Consider providing a machine-readable table of per-CPV annotations so the counts (8 C + 14 Java = 22 beyond PF) can be verified exactly.
  3. [Appendix C] The accuracy multiplier formula AM = 1 - (1 - r)^4 with r = n_acc/(n_acc + n_inacc) is undefined when a team has no counted submissions in a category. Table 17 uses dashes for LC's SARIF and Bundle; please specify the convention for AM in that case.
  4. [§7.4 / Table 17] The SP patch accuracy of 100% is based on only 11 counted submissions. The paper correctly attributes this to strict filtering and challenge distribution, but adding raw counts with confidence intervals would help readers calibrate the comparison with TI (63 submissions, 31.7%) and FB (60 submissions, 23.3%).

Circularity Check

0 steps flagged · score 2.0 of 10

Central claims rest on external competition data and independent baselines; no definitional circularity. The only self-citation concern is the OSS-CRS/Atlantis baselines, which are not load-bearing.

full rationale

KF1 ('Stability and accuracy were deciding factors in CRS performance') is grounded in the external AFC scoreboard and telemetry (Figure 2, Table 7), not in the paper's own outputs, so it is not circular. KF3 ('CRSs find 22 PoVs that PF cannot, thanks to LLM-driven code understanding') is an empirical set-difference against a concrete parallel-fuzzing run (8,906 CPU hours) and is not equivalent to its input by construction. The causal attribution is an inference, not a definitional reduction; the paper itself concedes that 'CRS architectures ensemble multiple techniques... making attribution to any single technique inherently difficult' (§7.2), which weakens the causal claim but is a correctness/evidence concern, not circularity. The main self-citation is the use of atlantis-multilang-given_fuzzer and MultiRetrieval from OSS-CRS [54]/Atlantis [57], the authors' own lab-based baseline. This is a minor self-citation, but it is code-executed, and the same comparisons are corroborated by the independent Claude Code (CC) baseline and by the official competition database. KF4 similarly rests on manual two-reviewer validation of patches plus external CC results. Thus there is no specific circular step in which a prediction reduces by construction to a fitted parameter or a self-referential definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical study, not a derivation, so there are no fitted constants. The free_parameters list captures hand-chosen analytical thresholds that calibrate the paper's central benchmark comparisons. The axioms are the interpretive commitments the analysis depends on: log-based inactivity inference, ideal-condition annotations as difficulty measures, acceptance of organizer ground truth, and the disclosed 5-CP exclusion. No new entities (particles, forces, dimensions) are introduced; the 'annotation' concept is a method, not an entity.

free parameters (3)
  • PF solvability criterion = any of 3 runs succeeds (union)
    §7.2: 'a CPV is marked fuzz-discoverable if any of the three runs succeeds.' Hand-chosen; a majority/strict criterion would shrink the 34/63 baseline and inflate the 22-CPV CRS-over-PF delta in KF3.
  • CC patch generation budget = at most 3 per CPV
    §7.2: 'CC tries at most 3 patch generations per CPV.' Hand-chosen; bounds the 33/63 patchable annotation and the 37.7% semantic-incorrectness rate.
  • MR iteration budget = self-determined (autonomous)
    §7.2: 'MR determines iterations autonomously.' The 31/63 MR annotation therefore depends on agent self-stopping behavior that is not held fixed.
assumptions (4)
  • domain assumption Absence of log messages is treated as absence of CRS activity
    §7.2 admits telemetry is 'inherently incomplete and unordered' and that 'teams selectively uploaded logs for performance reasons'; Figure 3's diagonal 'no CRS activity' markings and the 'TB and TI plateaued after P3/P4 (telemetry shows no further activity)' narrative in §7.1 rely on this inference.
  • domain assumption Foundational-technique performance under ideal single-task conditions reflects inherent CPV difficulty
    Stated in §7.2: 'The rationale is that their performance under ideal laboratory conditions reflects the inherent difficulty characteristics of CPVs.' This premise legitimizes every CRS-vs-annotation comparison in §7.3-§7.4.
  • domain assumption Organizer-provided competition data is accepted as ground truth
    §7 uses the organizers' CPV set, CWE labels, scores, and scoring rules without independent re-derivation; the paper never re-evaluates any team's CRS (stated in Appendix A).
  • ad hoc to paper Five unharnessed CPs are excluded as unscored
    §4: 'We exclude five CPs for which the organizer did not provide fuzzing harnesses, as they were unscored.' Defensible scoping, but openssl appears both in the excluded list and as os1□ in Table 12, and the resulting '48 CPs / 16 full-mode / 32 delta' bookkeeping is ambiguous.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned." pith.science (2026). https://pith.science/paper/7E2SU5KM

@misc{pith2026260207666,
  author       = {Pith},
  title        = {Pith review of: SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7E2SU5KM}},
  note         = {Machine review of arXiv:2602.07666}
}
read the original abstract

DARPA's AI Cyber Challenge (AIxCC, 2023--2025) is the largest competition to date for building fully autonomous cyber reasoning systems (CRSs) that leverage recent advances in AI -- particularly large language models (LLMs) -- to discover and remediate vulnerabilities in real-world open-source software. This paper presents the first systematic analysis of AIxCC. Drawing on design documents, source code, execution traces, and discussions with organizers and competing teams, we examine the competition's structure and key design decisions, characterize the architectural approaches of finalist CRSs, and analyze competition results beyond the final scoreboard. Our analysis reveals the factors that truly drove CRS performance, identifies genuine technical advances achieved by teams, and exposes limitations that remain open for future research. We conclude with lessons for organizing future competitions and broader insights toward deploying autonomous CRSs in practice.

Figures

Figures reproduced from arXiv: 2602.07666 by the authors.

Figure 1
Figure 1. AFC workflow. GitHub webhooks trigger challenge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Score per time (top) and phase (bottom) axes. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Team performance per CPV (CWE-wise breakdowns are in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Token consumption (input + output) per model by team. OpenAI Anthropic Google xAI Other gpt-4.1 gpt-4.1-mini gpt-4.1-nano gpt-4o gpt-4o-mini o3 o3-mini o4-mini claude-3-haiku claude-3.5-haiku claude-3.5-sonnet claude-3.7-sonnet claude-opus-4 claude-sonnet-4 gemini-1.5-…
Figure 5
Figure 5. Figure 5: I/O token ratio per model by team. from 80%–90%, with patch rates being below average; some teams achieved 100% accuracy in certain scoring categories. Appendix N. CWE-Wise Performance Analysis [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: Submission timing by type. Each task has a fixed time window (12h for full-mode, 6h for delta-mode); 0% is [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: CWE-wise PoV generation performance heatmap. Note: zero-day CPVs are excluded as they are still under [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: CWE-wise patch generation performance heatmap. Note: zero-day CPVs are excluded as they are still under [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SeedSmith: LLM-Driven Seed Synthesis for Directed Fuzzing

    cs.CR 2026-07 conditional novelty 7.0 of 10

    An agentic LLM seed generator that recovers missing call edges and crash preconditions yields 11–15× geometric-mean crash-time speedups on Magma and unlocks 16 previously unreachable ARVO bugs.

  2. Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilities

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    Chai uses AI to enhance differential testing for cryptographic misuse, cataloging library-level flaws and propagating them to find over 100 vulnerabilities including a critical one in a widely deployed SSL library.

  3. Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    Revelio combines LLMs, static analysis, and sanitizer-verified PoVs to scalably discover memory safety vulnerabilities in repository-scale code, finding 19 new bugs in long-fuzzed projects at low cost.

Reference graph

Works this paper leans on

72 extracted references · 5 linked inside Pith · cited by 3 Pith papers

  1. [54]

    OSS-CRS: Open-Source Software Cyber Reasoning System

    SSLab, Georgia Tech. OSS-CRS: Open-Source Software Cyber Reasoning System. https://github.com/sslab-gatech/oss-crs, 2025. Accessed: 2025-12-12

    2025

  2. [1]

    GigE Vision standard

    AIA (Association for Advancing Automation). GigE Vision standard. Accessed: 2026-02-05. URL: https://www.automate.org/vision/vision -standards/vision-standards-gige-vision

    2026

  3. [2]

    Claude Code: An agentic coding tool that lives in your terminal, 2025

    Anthropic. Claude Code: An agentic coding tool that lives in your terminal, 2025. https://github.com/anthropics/claude-code

    2025

  4. [3]

    Apache Maven Project

    Apache Software Foundation. Apache Maven Project. https://maven. apache.org/, 2025. Accessed: 2025-12-12

    2025

  5. [4]

    OSS-Fuzz

    Abhishek Arya, Oliver Chang, Jonathan Metzman, Kostya Serebryany, and Dongge Liu. OSS-Fuzz. https://github.com/google/oss-fuzz

  6. [5]

    Nautilus: Fishing for deep bugs with grammars

    Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. Nautilus: Fishing for deep bugs with grammars. InNDSS, volume 19, page 337, 2019

    2019

  7. [6]

    IJON: Exploring deep state spaces via fuzzing

    Cornelius Aschermann, Sergej Schumilo, Ali Abbasi, and Thorsten Holz. IJON: Exploring deep state spaces via fuzzing. In2020 IEEE Symposium on Security and Privacy (SP), pages 1597–1612. IEEE, 2020

    2020

  8. [7]

    LiteLLM: Call 100+ LLM APIs in OpenAI format

    BerriAI. LiteLLM: Call 100+ LLM APIs in OpenAI format. https: //github.com/BerriAI/litellm, 2026. Accessed: 2026

    2026

Show all 72 references
  1. [8]

    AURORA: Statistical crash analysis for automated root cause explanation

    Tim Blazytko, Moritz Schlögel, Cornelius Aschermann, Ali Abbasi, Joel Frank, Simon Wörner, and Thorsten Holz. AURORA: Statistical crash analysis for automated root cause explanation. In29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2020

    2020

  2. [9]

    Directed Greybox Fuzzing

    Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. Directed Greybox Fuzzing. InProceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS’17), pages 2329–2344. ACM, 2017. doi:10.1145/ 3133956.3134020. 13

    2017

  3. [10]

    KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs

    Cristian Cadar, Daniel Dunbar, and Dawson Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In8th USENIX Symposium on Operating Systems Design and Implementation (OSDI’08), pages 209–224. USENIX Association, 2008

    2008

  4. [11]

    ccache: A fast C/C++ compiler cache

    ccache developers. ccache: A fast C/C++ compiler cache. https: //ccache.dev/, 2025. Accessed: 2025-12-12

    2025

  5. [12]

    LangChain, 2022

    Harrison Chase. LangChain, 2022. https://github.com/langchain-ai/la ngchain

    2022

  6. [13]

    Enfuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers

    Yuanliang Chen, Yu Jiang, Fuchen Ma, Jie Liang, Mingzhe Wang, Chijin Zhou, Xun Jiao, and Zhuo Su. Enfuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers. InProceedings of the 28th USENIX Security Symposium (Security), Santa Clara, CA, August 2019

    2019

  7. [14]

    Universal Ctags. ctags. https://github.com/universal-ctags/ctags, 2025. Accessed: 2025-07-27

    2025

  8. [15]

    curl-fuzzer: Quality assurance testing for the curl project

    curl project. curl-fuzzer: Quality assurance testing for the curl project. https://github.com/curl/curl-fuzzer, 2025. Accessed: 2025-12-12

    2025

  9. [16]

    ast-grep: A cli tool for code structural search, lint, and rewriting

    Herrington Darkholme. ast-grep: A cli tool for code structural search, lint, and rewriting. https://github.com/ast-grep/ast-grep, 2026. Accessed: 2026

    2026

  10. [17]

    Cyber Grand Challenge, 2016

    DARPA. Cyber Grand Challenge, 2016. https://www.darpa.mil/resear ch/programs/cyber-grand-challenge

    2016

  11. [18]

    AIxCC Semifinal Competition, 2024

    DARPA. AIxCC Semifinal Competition, 2024. https://aicyberchallen ge.com/semifinal-competition/

    2024

  12. [19]

    AI Cyber Challenge (AIxCC), 2025

    DARPA. AI Cyber Challenge (AIxCC), 2025. https://aicyberchallen ge.com/

    2025

  13. [20]

    AIxCC Archive, 2025

    DARPA. AIxCC Archive, 2025. https://archive.aicyberchallenge.com/

    2025

  14. [21]

    JaCoCo: Java code coverage library

    EclEmma. JaCoCo: Java code coverage library. https://www.eclemm a.org/jacoco/, 2026. Accessed: 2026

    2026

  15. [22]

    AFL++: Combining incremental steps of fuzzing research

    Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. AFL++: Combining incremental steps of fuzzing research. In14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association, August 2020

    2020

  16. [23]

    LibAFL: A Framework to Build Modular and Reusable Fuzzers

    Andrea Fioraldi, Dominik Maier, Dongjia Zhang, and Davide Balzarotti. LibAFL: A Framework to Build Modular and Reusable Fuzzers. InProceedings of the 29th ACM Conference on Com- puter and Communications Security, CCS ’22, pages 1331–1347, New York, NY , USA, 2022. Association ...

    2022

  17. [24]

    GDB: The GNU project debugger

    Free Software Foundation. GDB: The GNU project debugger. https: //sourceware.org/gdb/, 2026. Accessed: 2026

    2026

  18. [25]

    autofz: Automated Fuzzer Composition at Runtime

    Yu-Fu Fu, Jaehyuk Lee, and Taesoo Kim. autofz: Automated Fuzzer Composition at Runtime. InProceedings of the 32nd USENIX Security Symposium (Security), Anaheim, CA, August 2023

    2023

  19. [26]

    Aider: AI pair programming in your terminal, 2025

    Paul Gauthier. Aider: AI pair programming in your terminal, 2025. https://aider.chat/

    2025

  20. [27]

    CodeQL: The libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security, 2025

    GitHub. CodeQL: The libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security, 2025. https://codeql.github.com/

    2025

  21. [28]

    SARIF support for code scanning, 2025

    GitHub. SARIF support for code scanning, 2025. https://docs.github. com/en/code-security/reference/code-scanning/sarif-support-for-cod e-scanning

    2025

  22. [29]

    Clusterfuzz: Scalable fuzzing infrastructure

    Google. Clusterfuzz: Scalable fuzzing infrastructure. https://github.c om/google/clusterfuzz, 2026. Accessed: 2026-02-01

    2026

  23. [30]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  24. [31]

    Vibe Coding, 2025

    Andrej Karpathy. Vibe Coding, 2025. https://x.com/karpathy/status/ 1886192184808149383

    2025

  25. [32]

    Patchisland: Orchestration of llm agents for continuous vulnerability repair, 2026

    Wonyoung Kim, Seunggi Min, Minjae Gwon, Dowoo Baik, Haein Lee, Hyeon Heo, Minjae Lee, Min Woo Baek, Yonghwi Jin, Younggi Park, Yunjae Choi, Taesoo Kim, Sangdon Park, and Insu Yun. Patchisland: Orchestration of llm agents for continuous vulnerability repair, 2026. URL: https://...

    2026 arXiv

  26. [33]

    Logs in, patches out: Automated vulnerability repair via {Tree-of- Thought}{LLM} analysis

    Youngjoon Kim, Sunguk Shin, Hyoungshick Kim, and Jiwon Yoon. Logs in, patches out: Automated vulnerability repair via {Tree-of- Thought}{LLM} analysis. In34th USENIX Security Symposium (USENIX Security 25), pages 4401–4419, 2025

    2025

  27. [34]

    LangGraph: Build resilient language agents as graphs,

    LangChain, Inc. LangGraph: Build resilient language agents as graphs,

  28. [35]

    LLVM: A compilation framework for lifelong program analysis and transformation

    Chris Lattner and Vikram Adve. LLVM: A compilation framework for lifelong program analysis and transformation. InInternational Symposium on Code Generation and Optimization (CGO’04), pages 75–86. IEEE, 2004

    2004

  29. [36]

    VULCANBOOST: Boosting ReDoS Fixes through Symbolic Representation and Feature Normalization

    Yeting Li, Yecheng Sun, Zhiwu Xu, Haiming Chen, Xinyi Wang, Hengyu Yang, Huina Chao, Cen Zhang, Yang Xiao, Yanyan Zou, et al. VULCANBOOST: Boosting ReDoS Fixes through Symbolic Representation and Feature Normalization. In34th USENIX Security Symposium (USENIX Security 25), pag...

    2025

  30. [37]

    UndefinedBehaviorSanitizer, 2025

    LLVM Project. UndefinedBehaviorSanitizer, 2025. https://clang.llvm .org/docs/UndefinedBehaviorSanitizer.html

    2025

  31. [38]

    FuzzedDataProvider: A helper class for fuzz targets

    LLVM Project. FuzzedDataProvider: A helper class for fuzz targets. https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/ fuzzer/FuzzedDataProvider.h, 2026. Accessed: 2026

    2026

  32. [39]

    Infer: A static analyzer for Java, C, C++, and Objective-C,

    Meta. Infer: A static analyzer for Java, C, C++, and Objective-C,

  33. [40]

    Language Server Protocol specification

    Microsoft. Language Server Protocol specification. https://microsoft. github.io/language-server-protocol/, 2026. Accessed: 2026

    2026

  34. [41]

    Common Weakness Enumeration (CWE)

    MITRE. Common Weakness Enumeration (CWE). https://cwe.mitre. org/, 2026. Accessed: 2026-02-03

    2026

  35. [42]

    Fuzzbomb: Fully-autonomous detection and repair of cyber vulnerabilities.International Journal on Advances in Security, 9(3-4), 2016

    David J Musliner, Scott E Friedman, Michael Boldt, J Benton, Max Schuchard, and Peter Keller. Fuzzbomb: Fully-autonomous detection and repair of cyber vulnerabilities.International Journal on Advances in Security, 9(3-4), 2016

    2016

  36. [43]

    Musliner, Scott E

    David J. Musliner, Scott E. Friedman, Jeffrey M. Rye, and Tom Marble. Meta-control for adaptive cybersecurity in FUZZBUSTER. InProc. IEEE Int’l Conf. on Self-Adaptive and Self-Organizing Systems, sep 2013

    2013

  37. [44]

    Musliner, Michael J

    David J. Musliner, Michael J. S. Pelican, Matthew McLure, Steven Johnston, Richard G. Freedman, and Corey Knutson. OpenMIND: Planning and adapting in domains with novelty. InAdvances in Cognitive Systems, November 2021. URL: https://www.sift.net/sites/d efault/files/publicatio...

    2021

  38. [45]

    Static Analysis Results Interchange Format (SARIF) Version 2.1.0, 2020

    OASIS. Static Analysis Results Interchange Format (SARIF) Version 2.1.0, 2020. https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0. html

    2020

  39. [46]

    Using Logprobs

    OpenAI. Using Logprobs. https://developers.openai.com/cookbook/e xamples/using_logprobs, 2023. Accessed: 2026-02-01

    2023

  40. [47]

    jdb: The Java debugger

    Oracle. jdb: The Java debugger. https://docs.oracle.com/en/java/java se/21/docs/specs/man/jdb.html, 2026. Accessed: 2026

    2026

  41. [48]

    Symbolic execution with SymCC: Don’t interpret, compile! In29th USENIX Security Symposium (USENIX Security 20), pages 181–198, Berkeley, CA, USA, August 2020

    Sebastian Poeplau and Aurélien Francillon. Symbolic execution with SymCC: Don’t interpret, compile! In29th USENIX Security Symposium (USENIX Security 20), pages 181–198, Berkeley, CA, USA, August 2020. USENIX Association. URL: https://www.usenix .org/conference/usenixsecurity2...

    2020

  42. [49]

    Semgrep: Lightweight static analysis for many lan- guages, 2025

    Semgrep, Inc. Semgrep: Lightweight static analysis for many lan- guages, 2025. https://semgrep.dev/

    2025

  43. [50]

    {AddressSanitizer}: A fast address sanity checker

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. {AddressSanitizer}: A fast address sanity checker. In2012 USENIX annual technical conference (USENIX ATC 12), pages 309–318, 2012. 14

    2012

  44. [51]

    Bandfuzz: An ml-powered collaborative fuzzing framework, 2025

    Wenxuan Shi, Hongwei Li, Jiahao Yu, Xinqian Sun, Wenbo Guo, and Xinyu Xing. Bandfuzz: An ml-powered collaborative fuzzing framework, 2025. URL: https://arxiv.org/abs/2507.10845, arXiv: 2507.10845

    2025 arXiv

  45. [52]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL: https://openreview.net/f orum?id=vAElhFcKW6

    2023

  46. [53]

    SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis

    Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Audrey Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, and Giovanni Vigna. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. In2016 IEEE Symposium o...

    2016 doi

  47. [55]

    DSPy: The framework for programming—not prompt- ing—language models, 2025

    Stanford NLP. DSPy: The framework for programming—not prompt- ing—language models, 2025. https://github.com/stanfordnlp/dspy

    2025

  48. [56]

    SVF: Interprocedural static value-flow analysis in LLVM

    Yulei Sui and Jingling Xue. SVF: Interprocedural static value-flow analysis in LLVM. InProceedings of the 25th International Conference on Compiler Construction, CC ’16, pages 265–266. ACM, 2016. doi:10.1145/2892208.2892235

    2016

  49. [57]

    (Team Atlanta)

    Taesoo Kim et al. (Team Atlanta). Atlantis: Ai-driven threat localiza- tion, analysis, and triage intelligence system, 2025. Cross-listed in cs.AI. URL: https://arxiv.org/abs/2509.14589, arXiv:2509.14589, doi:10.48550/arXiv.2509.14589

    2025 doi

  50. [58]

    42-b3yond-6ug Open Letter

    Team 42 (b3yond 6ug). 42-b3yond-6ug Open Letter. https://docs.goo gle.com/document/d/1-1TexnOwQGj2KJ8rrLtk-Vgl-PqO5MZe7k0q rHQPC_0/, 2025

    2025

  51. [59]

    Branch Flipper: Unlocking fuzz blockers with coverage- grounded LLMs

    Theori. Branch Flipper: Unlocking fuzz blockers with coverage- grounded LLMs. https://theori-io.github.io/aixcc-public/afc/Branch %20Flipper.pdf, 2025. AIxCC Technical Report

    2025

  52. [60]

    LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language...

    2023 arXiv

  53. [61]

    Tree-sitter: An incremental parsing system for program- ming tools, 2025

    Tree-sitter. Tree-sitter: An incremental parsing system for program- ming tools, 2025. https://github.com/tree-sitter/tree-sitter

    2025

  54. [62]

    W ALA: T.J

    W ALA. W ALA: T.J. Watson libraries for analysis. https://github.com /wala/W ALA, 2026. Accessed: 2026

    2026

  55. [63]

    Machine vs

    Mike Walker. Machine vs. Machine: Lessons from the First Year of Cyber Grand Challenge. In24th USENIX Security Symposium (USENIX Security 15), Washington, D.C., August 2015. USENIX Association

    2015

  56. [64]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR), 2023. URL: https://arxiv.org/a...

    2023 arXiv

  57. [65]

    Modeling and Discovering Vulnerabilities with Code Property Graphs

    Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and Discovering Vulnerabilities with Code Property Graphs. In2014 IEEE Symposium on Security and Privacy, pages 590–604. IEEE, 2014.doi:10.1109/SP.2014.44

    2014 doi

  58. [66]

    GNU Global: Source code tagging system

    Shigio Yamaguchi. GNU Global: Source code tagging system. https: //www.gnu.org/software/global/, 2026. Accessed: 2026

    2026

  59. [67]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: agent-computer interfaces enable automated software engineering. InProceedings of the 38th International Conference on Neural Information Processing System...

    2024

  60. [68]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  61. [69]

    Patchagent: A practical program repair agent mimicking human expertise

    Zheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu, Meng Xu, Dongliang Mu, Yan Chen, and Xinyu Xing. Patchagent: A practical program repair agent mimicking human expertise. In34rd USENIX Security Symposium (USENIX Security 25), 2025

    2025

  62. [70]

    QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing

    Insu Yun, Sangho Lee, Meng Xu, Yeongjin Jang, and Taesoo Kim. QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing. InProceedings of the 27th USENIX Security Symposium (Security), Baltimore, MD, August 2018. 15 Appendix A. Ethical Considerations Stakeholder ...

    2018

  63. [2024]

    https://github.com/langchain-ai/langgraph

  64. [2025]

    https://fbinfer.com/

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.