REVIEW 3 major objections 5 minor 10 cited by
RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read An autonomous LLM agent can audit an entire code repository without compiling it: it reproduces all 21 known bugs in a 15-project benchmark, finds 19 more, at 78.43% precision.
desk verdict A genuinely useful LLM auditing agent with externally grounded results, but the validator ablation is confounded and the 'first' claim is overstated; still deserves serious refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the agent memory plus the validator loop. The memory stores, for each function and program value, a set of pairs of a feasible program path and the data-flow facts discovered along that path; before re-analyzing a value in a function, the explorer consults this cache, and this is what keeps token and time costs low. The validator filters two kinds of hallucination: a parsing-based check rejects facts that violate control-flow order, and a second LLM prompt rejects inter-procedural paths whose branch conditions are unsatisfiable. Together with the function-analysis prompt, these checks turn per-function LLM reasoning into repository-level auditing without compiling the code.
What would settle it
Build a labeled set of inter-procedural paths with known feasibility, run only the validator's path-feasibility prompt on them, and compare its answers to ground truth; if the validator is no more accurate than the underlying LLM's unvalidated guesses, the false-positive reduction is not an independent check.
Extended reading notes
Core claim
RepoAudit's central claim is that path-sensitive, source-to-sink bug detection at repository scale can be achieved by a demand-driven LLM agent rather than by pouring a whole project into one prompt or by building a compiler-grade static analyzer. The initiator locates source values (for example, null assignments for null pointer dereference detection); the explorer prompts the LLM to abstract each relevant function, handle pointer aliasing, enumerate feasible intra-procedural paths, and emit data-flow facts per path, storing them in memory keyed by function and value; values that escape through returns, parameters, or globals drive further exploration through the call graph. The validator then rejects data-flow facts that violate control-flow order and rejects bug reports whose inter-procedural path conditions are contradictory. On the 15-project benchmark the design reproduces every previously reported bug (21), adds 19 new true bugs (14 already fixed in the latest commit), and reaches 78.43% precision; on nine additional high-profile repositories it reports 185 new bugs, 174 confirmed or fixed by developers.
Load-bearing premise
The reported precision assumes the LLM used to check whether inter-procedural paths are feasible is reliable enough to reject hallucinated paths, even though it is the same class of model that the paper shows hallucinates on path reasoning.
Editorial extensions
If this is right
- If the central claim holds, development-time auditing becomes practical for codebases that cannot be built: the 15 benchmark projects, averaging 251 KLoC, were each audited in about 0.44 hours and $2.54, while seven of them could not be handled by a compilation-dependent industrial analyzer in the evaluation.
- The same explorer prompts cover null pointer dereference, memory leak, and use-after-free because all three reduce to source-to-sink data-flow reachability along feasible paths, so extending the tool to other weakness classes mainly means adding source and sink patterns in the initiator.
- The validator is load-bearing: disabling it raises false positives in the benchmark by 245.45% (from 11 to 31), so precision comes from filtering hallucinated facts rather than from the raw reasoning of the LLM.
- Caching in agent memory is what makes the cost numbers possible: without caching, prompting rounds and financial cost rise 3-4 times on average, and one project exceeds 72 hours of analysis time.
- The design holds across different LLMs: with three other models, precision ranges from 82.35% to 88.46%, suggesting the agent architecture, not a single model, drives the result.
Reading between the lines
- Editorial inference: the same explore-cache-validate loop should transfer to other path-sensitive bug classes such as taint-style vulnerabilities or integer-overflow flows, because the explorer's core prompt targets generic data-flow reachability; the initiator's source and sink matchers would be the main new component.
- Editorial inference: the LLM-based feasibility validator is the least independent link in the chain; replacing it with a symbolic feasibility check wherever path constraints are expressible would turn the false-positive reduction from a second opinion into a verifiable guarantee.
- Editorial inference: a direct measurement of the validator, by running its path-feasibility prompt on hand-labeled feasible and infeasible inter-procedural paths, would show how much of the precision gain is genuine filtering rather than the explorer's own accuracy.
- Editorial inference: the cost structure suggests a general recipe for LLM program analysis: externalize the repository graph, cache per-function summaries, and prompt with one function at a time, which is a testable hypothesis for other graph-shaped analyses such as dependency or taint tracking.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RepoAudit is an LLM-based agent for repository-level auditing of three bug types (null-pointer dereference, memory leak, and use-after-free). It combines a source-initiator, a demand-driven explorer that prompts the LLM one function at a time to produce data-flow facts along feasible paths, an agent memory with caching, and a two-part validator that checks control-flow alignment of data-flow facts and inter-procedural path feasibility. The evaluation on 15 projects reports 40 true positives (21 previously published bugs and 19 new ones), 11 false positives, precision 78.43%, and average cost 0.44 hours and $2.54 per project. Ablations assess the effects of abstraction, validation, and caching; a further scan of additional open-source projects is reported as 185 new bugs with 174 confirmed or fixed by developers. Comparisons are made with CoT prompting, LLMDFA, Meta Infer, and Amazon CodeGuru.
Significance. The external grounding of the main evaluation is a genuine strength: the 21 reproduced bugs come from prior publications, and the 174 maintainer confirmations/fixes on newly reported bugs make the headline result difficult to explain away as an artifact of the LLM's priors. The reported cost figures and the open-source release are also useful to the community. If the validation claims are substantiated, RepoAudit would be a meaningful step toward practical LLM-driven repository auditing. The robustness checks across models and temperatures further support the empirical claims. However, the paper's explanation of why precision is high is not fully supported by the reported ablation, and the field-study table contains internal inconsistencies that need correction.
major comments (3)
- [Section 3.3 and Appendix C] The ablation used to attribute the 245.45% false-positive reduction to the validator does not isolate the path-feasibility component. REPOAUDIT-NOVAL, as described in Appendix C, removes 'the validation of the data-flow facts discovered by the explorer and also skips examining the bug reports'—that is, it disables both the deterministic control-flow-order alignment check and the LLM-based feasibility check simultaneously. Since the alignment check is a parsing-based, rule-driven component, the observed increase in false positives could be entirely due to it. No accuracy measurement of the feasibility validator itself (e.g., on a labeled set of feasible/infeasible inter-procedural paths) is reported, and Section 2.2 documents that the same class of model hallucinates on path-sensitive reasoning. I therefore request either an ablation that toggles the two validators independently, or a direct evaluation of the feasibility validator against an oracle such as SMT solving or manually labeled paths. Without this, the paper's explanation of its precision advantage is not established.
- [Section 4.2 and Table 4] The real-world impact numbers are internally inconsistent. The Introduction says ten actively maintained repositories were scanned, while Section 4.2 and Table 4 report nine projects, and Table 4 lists only nine rows. The text states that 95 and 79 bugs were confirmed and fixed respectively, but summing the Con and Fix columns in Table 4 yields 96 confirmed and 78 fixed. In addition, the table's TP and FP totals (185 TPs and 29 FPs) give a precision of 86.45%, not the reported 85.71%. These are headline empirical claims, so they should be corrected and reconciled before publication.
- [Appendix F, Listing 2] Listing 2 contains the non-code text 'Nothing to see, move along here' inside the vrf_get function. This appears to be a placeholder or artifact and should be removed; as printed, it makes the example inconsistent with the surrounding explanation.
minor comments (5)
- [Section 2.2] The controlled experiment that motivates the paper reports only qualitative results ('substantial hallucinations,' 'false positives and incorrect explanations'). Appendix A provides comparisons with single-function and multiple-function detectors, but not the quantitative outcome of the specific five-function direct-prompting experiment described in Section 2.2. Please add the numbers or a pointer to the corresponding table.
- [Section 3.3 and Figure 5] The feasibility-validation prompt template is underspecified: it is unclear what '[PATH]' contains (source code for all involved functions, a list of path conditions, or both) and how the LLM's 'Yes or No' answer is converted into a discard decision. A brief description of the actual prompt content and parsing of the answer would improve reproducibility.
- [Table 4] The table caption does not define the Con and Fix columns. The text explains them, but a one-line caption definition would make the table self-contained.
- [Section 4.2] The phrase '95 and 79 of which have been confirmed and fixed by developers, respectively' should be reconciled with the table sums. Please also clarify whether '174 confirmed or fixed' is the union of the two sets or the sum.
- [Appendix A and Appendix C] There are several typos and grammatical slips, e.g., 'filed2json' in Section 3.2.1, 'desgin' and 'senarios' in Appendix A, and 'concurruption' in the Introduction. These do not affect the technical content but should be cleaned up.
Circularity Check
No significant circularity: benchmark ground truth and developer confirmations are external to the method, and no fitted parameter or self-citation chain defines the reported results.
full rationale
RepoAudit's headline results are empirical measurements against external labels: the 21 previously reported bugs are taken from prior published bug reports, the 40 TPs and 11 FPs are compared against those reports and manual inspection, and the 185 new bugs are confirmed or fixed by independent developers. No parameter of the system is fitted to these labels, and no equation defines the predicted bugs in terms of the evaluation set. The method's internal components—the initiator's tree-sitter matchers, the explorer's prompts, and the validator—are not defined in terms of the benchmark outcomes. The path-feasibility validator is implemented by prompting an LLM (Section 3.3 and Figure 5), and the NOVAL ablation removes both the deterministic alignment check and the LLM feasibility check together, so the 245.45% FP increase does not isolate the feasibility validator; however, that is a confounded-ablation and robustness concern about how much of the precision gain the validator explains, not a circular derivation. The paper's citations to the authors' own LLMDFA and LLMSAN work are used as baselines and related work, not as load-bearing premises for RepoAudit's correctness. Under the stated definition of circularity (derivation equivalent to input by construction, fitted parameter renamed as prediction, or uniqueness imported from self-citation), no circular step is present.
Assumptions & free parameters
free parameters (2)
- K (calling context bound) =
4
- temperature =
0.0
assumptions (4)
- domain assumption tree-sitter-based call graph and control-flow order are accurate
- domain assumption LLM program abstraction preserves relevant data-flow facts within a function
- domain assumption LLM-based feasibility validation correctly judges path condition satisfiability
- domain assumption Developer confirmations and fixes are reliable ground truth for the new bugs
Cite this review
Pith. "Pith review of RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing." pith.science (2026). https://pith.science/paper/YAXXFRAX
@misc{pith2026250118160,
author = {Pith},
title = {Pith review of: RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing},
year = {2026},
howpublished = {\url{https://pith.science/paper/YAXXFRAX}},
note = {Machine review of arXiv:2501.18160}
}
abstract
Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability. This work introduces an LLM-based agent, RepoAudit, designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project. Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 10 Pith papers
-
ProSec: Fortifying Code LLMs with Proactive Security Alignment
ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.
-
ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments
Adaptive adversarial source-code comments evade four LLM vulnerability detectors at >90% success on 125 real null-pointer bugs without altering program behavior.
-
Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries
Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.
-
Validating Network Protocol Parsers with Traceable RFC Document Interpretation
An LLM-based pipeline that extracts formal protocol formats from RFC documents detected 69 parser bugs (36 confirmed) across nine network protocols.
-
OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review
OpenCodeReview reports up to 2.17x higher SEM-F1 than Claude Code and Codex on AACR-Bench, using 5-15x fewer tokens, by constraining agent review with rules, bounded tools, and diff-only reflection.
-
Antares: Foundation Models for Agentic Vulnerability Localization
Antares-3B, a 3B model trained with SFT plus GRPO, matches GPT-5.5 on repository-scale vulnerability localization at roughly 1/100th the inference cost.
-
VulWeaver: Weaving Broken Semantics for Grounded Vulnerability Detection
VulWeaver improves Java vulnerability detection to 0.75 F1 by enhancing dependency graphs with LLM semantic fixes, extracting full context from slices plus implicit usage info, and applying type-specific meta-promptin...
-
Large Language Models for Validating Network Protocol Parsers
PARVAL uses multiple LLM agents to extract format specifications from both RFC 5880 and the FRRouting BFD parser and flags 34 true inconsistencies, including seven unique bugs and two RFC documentation issues.
-
Position: Intelligent Coding Systems Should Write Programs with Justifications
A position paper advocating that intelligent coding systems should accompany code with justified explanations that are cognitively aligned and semantically faithful.
-
Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Models
Reasoning depth in code LLMs should be managed as a controllable resource across synthetic data generation, benchmarking, and deployment, rather than left implicit.
Reference graph
Works this paper leans on
-
[1]
static int __init damon_reclaim_init(void){
-
[2]
ctx = damon_new_ctx()
-
[5]
if (damon_select_ops(ctx, DAMON_OPS_PADDR))
-
[7]
Large Language Models for Validating Network Protocol Parsers
[Online; accessed 29-Jan-2025]. Mingwei Zheng, Danning Xie, and Xiangyu Zhang. Large language models for validating network proto- col parsers, 2025. URL https://arxiv.org/abs/ 2504.13515. Mingwei Zheng, Danning Xie, Qingkai Shi, Chengpeng Wang, and Xiangyu Zhang. Validating network protocol parsers with traceable rfc document interpretation. In Proceedin...
work page Pith review arXiv 2025
-
[8]
ctx->callback.after_aggregation = damon_reclaim_after_aggregation
-
[9]
target = damon_new_target()
-
[11]
ctx->callback.after_wmarks_check = damon_reclaim_after_wmarks_check
-
[14]
damon_destroy_ctx(ctx)
Show all 18 references
-
[15]
damon_add_target(ctx, target)
-
[16]
schedule_delayed_work(&damon_reclaim_timer, 0)
-
[17]
damon_reclaim_initialized = true
-
[18]
Error allocating memory for context
} Table 5. The ratios of the prompting rounds (Prompt) and input token costs (In token) of LLMDFA under two settings Bug TypeID LLMDFA-PATHSCAN LLMDFA-SRCSCAN Prompt InToken Prompt InToken NPD N1 115.39 81.23 871.61 522.40 N2 217.00 120.23 2,839.00 2,087.27 N3 6.83 3.13 255.57...
2025
-
[132]
Springer, 1984. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,
1984 arXiv
-
[899]
doi: 10.1145/3620666.3651366
ACM, 2024. doi: 10.1145/3620666.3651366. Anshunkang Zhou, Chengfeng Ye, Heqing Huang, Yuandao Cai, and Charles Zhang. Plankton: Reconciling binary code and debug information. In Rajiv Gupta, Nael B. Abu- Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 2...
2024
-
[928]
doi: 10.1145/3620665.3640382
ACM, 2024. doi: 10.1145/3620665.3640382. Brittany Johnson, Yoonki Song, Emerson R. Murphy-Hill, and Robert W. Bowdidge. Why don’t software devel- opers use static analysis tools to find bugs? In David Notkin, Betty H. C. Cheng, and Klaus Pohl, editors, 35th International Confe...
2024
-
[2008]
Springer, 2008
Proceedings, volume 4963 of Lecture Notes in Computer Science, pages 337–340. Springer, 2008. doi: 10.1007/978-3-540-78800-3 \ 24. Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: Analyzing dataflow in code with large language models. ...
2008
-
[2023]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al
doi: 10.48550/ARXIV .2303.08774. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pa...
-
[2025]
12 Amazon
[Online; accessed 29-Jan-2025]. 12 Amazon. Code Review Tool: Amazon CodeGuru Se- curity. https://aws.amazon.com/codeguru/,
2025
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.