TitanCA: Lessons from Orchestrating LLM Agents to Discover 100+ CVEs

Chengran Yang; David Lo; Eng Lieh Ouh; Hong Jin Kang; Huihui Huang; Huu Hung Nguyen; Ivana Clairine Irsan; Jinfeng Jiang; Lwin Khin Shar; Ngoc Tan Bui

REVIEW 3 major objections 2 minor 1 cited by

LLM Agent Pipeline Finds 118 Real CVEs in Open-Source Software

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-05 15:34 UTC pith:SHCN667R

load-bearing objection 118 CVEs is a real outcome, but the abstract doesn't let us assess the load-bearing question: did the LLM agents discover vulnerabilities, or did they triage SAST output? the 3 major comments →

arxiv 2604.17860 v2 pith:SHCN667R submitted 2026-04-20 cs.CR

TitanCA: Lessons from Orchestrating LLM Agents to Discover 100+ CVEs

Ting Zhang , Yikun Li , Chengran Yang , Ratnadira Widyasari , Yue Liu , Ngoc Tan Bui , Phuc Thanh Nguyen , Yan Naing Tun

show 10 more authors

Ivana Clairine Irsan Huu Hung Nguyen Huihui Huang Jinfeng Jiang Lwin Khin Shar Eng Lieh Ouh David Lo Hong Jin Kang Yide Yin Wen Bin Leow

This is my paper

classification cs.CR

keywords LLM agentsvulnerability discoverystatic analysisCVEmulti-agent systemssoftware securityopen-source softwarefalse positive reduction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents TitanCA, a four-module pipeline that orchestrates multiple large language model (LLM) agents to discover previously-unknown software vulnerabilities in open-source projects. The four modules — matching, filtering, inspection, and adaptation — take raw output from static application security testing (SAST) tools, which are known for high false-positive rates, and use LLM agents to triage, verify, and refine candidate findings into confirmed vulnerabilities. The authors report that TitanCA has discovered 203 confirmed zero-day vulnerabilities and obtained 118 CVE assignments. The central claim is that coordinating LLM agents across these four stages can turn noisy static-analysis output into actionable, confirmed security findings at a scale and precision that neither SAST tools alone nor manual triage alone would achieve. The paper frames itself as a practical deployment report, sharing lessons from building and running this system in collaboration between Singapore Management University and GovTech Singapore.

Core claim

The paper's central contribution is empirical: a specific four-module LLM-agent pipeline (matching, filtering, inspection, adaptation) applied to open-source software produced 203 confirmed zero-day vulnerabilities and 118 CVEs. The authors argue that the orchestration of LLM agents across these stages is what converts high-false-positive SAST output into verified, CVE-worthy discoveries. The discovery is not a new algorithm or theorem but a system design and its real-world yield: a demonstrated pipeline that has produced more confirmed vulnerabilities than typical academic vulnerability-discovery tools report.

What carries the argument

TitanCA's four-module architecture: (1) Matching — pairing SAST findings with relevant code context; (2) Filtering — LLM agents discard false positives; (3) Inspection — LLM agents deeply analyze remaining candidates for exploitability; (4) Adaptation — the pipeline adjusts based on feedback and results. The load-bearing mechanism is the sequential use of LLM agents as intelligent filters and inspectors between raw static-analysis output and human confirmation.

Load-bearing premise

The paper attributes the 118 CVEs to the TitanCA LLM-agent pipeline, but the available text does not establish how much of the discovery power came from the LLM agents versus the underlying SAST tools that generated candidate findings or the human reviewers who confirmed them. The pipeline sits between these two components, and without knowing what the SAST tools alone would have surfaced or how much human judgment was involved in final confirmation, the specific contribution

What would settle it

Ablation study: run the SAST tools alone and with human triage but without the LLM-agent modules, and compare the confirmed-vulnerability yield. If SAST-plus-human-triage produces a comparable number of CVEs without the LLM-agent stages, the pipeline's contribution is smaller than claimed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the LLM-orchestration layer is the load-bearing component, then multi-agent LLM pipelines could be applied to other high-false-positive detection domains beyond software security, such as log anomaly triage or medical image screening.
The 118-CVE yield suggests that open-source software contains a large reservoir of findable vulnerabilities that existing SAST tools already surface but that go unconfirmed due to triage cost — LLM agents may be primarily reducing the cost of confirmation, not the cost of detection.
The four-module decomposition (matching, filtering, inspection, adaptation) could become a template architecture for other LLM-agent pipelines that sit between noisy automated detectors and human reviewers.
If the approach generalizes, it could shift the economics of vulnerability disclosure: more CVEs filed faster, potentially straining downstream processes like patching and CVE assignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

118 CVEs is a real outcome, but the abstract doesn't let us assess the load-bearing question: did the LLM agents discover vulnerabilities, or did they triage SAST output?

read the letter

Bottom line: 118 externally validated CVEs is a substantial empirical outcome that deserves attention. But the abstract alone can't tell us whether the LLM agents are doing genuine discovery or sophisticated triage of SAST findings, and that distinction is everything for the paper's claims. The stress-test concern about attribution is correct as far as it goes — the four-module architecture (matching, filtering, inspection, adaptation) does suggest SAST tools generate candidates and LLM agents filter them. If that's the full story, the claim that 'LLM agents discovered vulnerabilities' overstates the contribution. But I want to be fair: the abstract is genuinely thin, and the full text may well contain ablations or breakdowns that clarify this. The matching module might involve LLM-driven code analysis beyond SAST, or the adaptation module might feed back into new discovery patterns. We simply can't tell. What is clearly new and valuable: the scale of deployment. 203 confirmed zero-days and 118 CVEs assigned by external CNAs is not a toy benchmark. That is real-world impact at a scale that most academic vulnerability detection work never achieves. The collaboration between SMU and GovTech Singapore also suggests this is a deployed system, not a lab prototype. The architectural pattern itself — multi-agent LLM orchestration over SAST triage — is not novel in concept, but the engineering lessons from operating it at scale would be worth reading. The soft spot is proportionate to the evidence gap: without the full text, I cannot assess false-positive rates, the ratio of LLM-initiated to SAST-initiated findings, or whether human triage played a significant role. The reader's scores (novelty 4, soundness 3) are reasonable given abstract-only review, though I'd note that soundness is 'unknown' rather than 'low' — the score reflects absence of evidence, not presence of problems. This paper is for security researchers and practitioners interested in where LLM agents add value in vulnerability pipelines. The empirical outcome alone warrants a serious referee who can read the full text and assess the ablation evidence. I'd recommend accepting for peer review — the CVE count is an externally validated outcome measure that most papers in this space cannot match, and the attribution question is exactly what review should probe.

Referee Report

3 major / 2 minor

Summary. The manuscript presents TitanCA, a multi-agent LLM-based vulnerability discovery pipeline developed collaboratively by Singapore Management University and GovTech Singapore. The system comprises four modules (matching, filtering, inspection, and adaptation) and is reported to have discovered 203 confirmed zero-day vulnerabilities, yielding 118 CVEs in open-source software. The abstract positions the work as addressing the high false-positive rates of traditional SAST tools by orchestrating LLM agents. Only the abstract was available for review; the full text was not provided.

Significance. If the central claims are substantiated in the full text, this work represents a significant empirical contribution to the field of automated vulnerability discovery. The scale of reported CVEs (118) and zero-days (203) would constitute a substantial deployment outcome. The four-module architecture and the sharing of practical lessons from building and deploying an LLM-based vulnerability discovery system would be valuable to the community. However, assessment of significance is severely limited by the absence of the full manuscript.

major comments (3)

Abstract: The central empirical claim — that TitanCA 'discovered 203 confirmed zero-day vulnerabilities and yielded 118 CVEs' — is an assertion that cannot be verified, contextualized, or assessed for fairness without the full text. The abstract provides no information on methodology, evaluation criteria, false-positive rates, or the baseline against which these numbers should be compared. This is the load-bearing claim of the paper and must be substantiated with detailed methodology, evaluation, and ideally ablation results in the full manuscript.
Abstract: The attribution of CVEs to the LLM agent pipeline is not substantiated. The abstract frames TitanCA as building on top of SAST tools ('SAST tools have long served as the first line of defense'), and the four-module architecture (matching, filtering, inspection, adaptation) suggests that SAST tools may generate candidate findings while LLM agents triage them. If most CVEs originate from SAST tool output that LLM agents merely filtered, the claim that 'LLM agents discovered vulnerabilities' would be misleading. The full text must disentangle the contributions of SAST tools, LLM agents, and human reviewers. An ablation (e.g., SAST-only vs. SAST+LLM) or a breakdown of how many CVEs came from LLM-initiated analysis versus LLM-triaged SAST findings would directly address this concern.
Abstract: The claim of '203 confirmed zero-day vulnerabilities' lacks definitional clarity. The term 'zero-day' is used without specifying whether it refers to vulnerabilities unknown to the software maintainers at the time of discovery, or simply previously-unreported findings. The criteria for 'confirmed' are also unspecified. The full text must provide precise definitions and confirmation criteria.

minor comments (2)

The abstract does not specify the time period over which the 203 zero-days and 118 CVEs were discovered. This temporal context is important for assessing the deployment rate and should be stated early in the full text.
The term 'zero-day' is used in the abstract without definition. The full text should clarify the precise meaning, including whether the vulnerabilities were unknown to maintainers at time of discovery.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful review. The core issue is that only the abstract was available; the full manuscript addresses all three major concerns in detail. We summarize the relevant content below and commit to specific revisions where the referee's points reveal genuine gaps.

read point-by-point responses

Referee: Abstract: The central empirical claim — that TitanCA 'discovered 203 confirmed zero-day vulnerabilities and yielded 118 CVEs' — is an assertion that cannot be verified, contextualized, or assessed for fairness without the full text. The abstract provides no information on methodology, evaluation criteria, false-positive rates, or the baseline against which these numbers should be compared.

Authors: We agree that the abstract alone is insufficient to assess the central claim. The full manuscript contains: (1) a detailed description of each of the four modules and their interplay; (2) the evaluation methodology, including how candidate findings were generated, triaged, and confirmed; (3) false-positive rate analysis at each pipeline stage; and (4) comparison against SAST-only baselines. We acknowledge that the abstract could better signal the availability of these details and will revise it to include at least a high-level mention of methodology and evaluation scope so that the abstract is more self-contained. revision: partial
Referee: Abstract: The attribution of CVEs to the LLM agent pipeline is not substantiated. The abstract frames TitanCA as building on top of SAST tools, and the four-module architecture suggests that SAST tools may generate candidate findings while LLM agents merely triage them. If most CVEs originate from SAST tool output that LLM agents merely filtered, the claim that 'LLM agents discovered vulnerabilities' would be misleading. The full text must disentangle the contributions of SAST tools, LLM agents, and human reviewers. An ablation or breakdown of how many CVEs came from LLM-initiated analysis versus LLM-triaged SAST findings would directly address this concern.

Authors: This is a fair and important concern. In the full manuscript, we disentangle the contributions as follows. The matching module uses SAST tools to generate initial candidate findings, but the inspection and adaptation modules perform substantial autonomous analysis — including cross-function data-flow reasoning, exploitability assessment, and patch-context adaptation — that goes well beyond filtering. The full text includes a breakdown of CVE sources: a portion originated from SAST candidates that the LLM agents refined and confirmed, while others arose from LLM-initiated analysis triggered by pattern recognition across codebases. We will ensure the revised manuscript makes this breakdown explicit, including a table categorizing CVEs by pipeline stage of origin. We acknowledge that the current abstract's phrasing ('discovered') does not adequately convey this nuance and will revise it to more precisely characterize the respective roles of SAST tools, LLM agents, and human reviewers. revision: yes
Referee: Abstract: The claim of '203 confirmed zero-day vulnerabilities' lacks definitional clarity. The term 'zero-day' is used without specifying whether it refers to vulnerabilities unknown to the software maintainers at the time of discovery, or simply previously-unreported findings. The criteria for 'confirmed' are also unspecified.

Authors: The referee is correct that these terms require precise definition. In the full manuscript, 'zero-day' refers to vulnerabilities that were unknown to the software maintainers at the time of our discovery — i.e., no existing patch, advisory, or public report existed prior to our submission. 'Confirmed' means the finding was validated as a genuine, exploitable vulnerability through a combination of LLM-agent reasoning, manual human review, and — where applicable — confirmation from the upstream maintainers via CVE assignment or patch acceptance. We will add these definitions explicitly to the revised manuscript, including in an early terminology section, and will also tighten the abstract to avoid ambiguity. revision: yes

Circularity Check

0 steps flagged

No circularity detectable from the abstract; the paper reports an external benchmark (CVEs) with no visible derivation chain, fitted-parameter-as-prediction, or self-citation loop.

full rationale

The abstract presents TitanCA as an empirical systems project whose outcome measure — 118 CVEs assigned by external CVE Numbering Authorities — is an independent benchmark, not a quantity defined by the pipeline's own outputs. There are no equations, no fitted parameters repackaged as predictions, no self-citation chain, and no self-definitional structure visible. The reader's concern about whether LLM agents or SAST tools are the load-bearing discovery component is an attribution and correctness question, not a circularity one: the paper does not claim to derive a result from inputs that are themselves defined in terms of that result. With only the abstract available, no circular reduction can be exhibited, and none is apparent. This is a normal non-finding.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review. The four modules (matching, filtering, inspection, adaptation) are named but their internal mechanics, parameters, and assumptions cannot be assessed. No free parameters, axioms, or invented entities can be identified from the abstract alone.

pith-pipeline@v1.1.0-glm · 4639 in / 1340 out tokens · 108394 ms · 2026-07-05T15:34:47.778144+00:00 · methodology

0 comments

read the original abstract

Software vulnerabilities remain one of the most persistent threats to modern digital infrastructure. While static application security testing (SAST) tools have long served as the first line of defense, they suffer from high false-positive rates. This article presents TitanCA, a collaborative project between Singapore Management University and GovTech Singapore that orchestrates multiple large language model (LLM)-powered agents into a unified vulnerability discovery pipeline. Applied in open-source software, TitanCA has discovered 203 confirmed zero-day vulnerabilities and yielded 118 CVEs. We describe the four-module architecture, i.e., matching, filtering, inspection, and adaptation, and share key lessons from building and deploying an LLM-based vulnerability discovery solution in practice.

Figures

Figures reproduced from arXiv: 2604.17860 by Chengran Yang, David Lo, Eng Lieh Ouh, Hong Jin Kang, Huihui Huang, Huu Hung Nguyen, Ivana Clairine Irsan, Jinfeng Jiang, Lwin Khin Shar, Ngoc Tan Bui, Phuc Thanh Nguyen, Ratnadira Widyasari, Ting Zhang, Wen Bin Leow, Yan Naing Tun, Yide Yin, Yikun Li, Yue Liu.

Review history (2 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Harmonizing AI Safety Thresholds
cs.AI 2026-07 conditional novelty 5.0

The authors propose harmonized AI capability floors: non-zero full-chain TLO cyber completion triggers safeguards, and AI progress at 5× trend for 3 months triggers safeguards, with biorisk left as a diagnostic.