Pith. sign in

REVIEW 3 major objections 5 minor 10 cited by

RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read An autonomous LLM agent can audit an entire code repository without compiling it: it reproduces all 21 known bugs in a 15-project benchmark, finds 19 more, at 78.43% precision.

desk verdict A genuinely useful LLM auditing agent with externally grounded results, but the validator ablation is confounded and the 'first' claim is overstated; still deserves serious refereeing. read the letter →

arxiv 2501.18160 v3 pith:YAXXFRAX submitted 2025-01-30 cs.SE cs.PL

classification cs.SEcs.PL
keywords LLMagentcodeauditingdata-flowanalysispath-sensitivebugdetectionnullpointerdereferencememoryleakuse-after-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that repository-level code auditing, the hunt for bugs whose causes and symptoms span many functions and files, can be done by an autonomous LLM agent that never compiles or executes the code. The agent breaks the repository into per-function analysis steps, stores discovered data-flow facts in an agent memory, and validates candidate bug reports before presenting them. The claimed payoff is scale and cost: across 15 real-world projects averaging 251 KLoC it reproduces all 21 previously known bugs and finds 19 new ones at 78.43% precision, spending $2.54 and 0.44 hours per project; across nine additional active repositories it reports 185 new bugs, 174 of which developers confirmed or fixed. If the claim holds, development-time auditing becomes practical for codebases that are uncompilable or under active change, where conventional static analyzers and industrial tools miss the bugs or cannot run.

What carries the argument

The load-bearing mechanism is the agent memory plus the validator loop. The memory stores, for each function and program value, a set of pairs of a feasible program path and the data-flow facts discovered along that path; before re-analyzing a value in a function, the explorer consults this cache, and this is what keeps token and time costs low. The validator filters two kinds of hallucination: a parsing-based check rejects facts that violate control-flow order, and a second LLM prompt rejects inter-procedural paths whose branch conditions are unsatisfiable. Together with the function-analysis prompt, these checks turn per-function LLM reasoning into repository-level auditing without compiling the code.

What would settle it

Build a labeled set of inter-procedural paths with known feasibility, run only the validator's path-feasibility prompt on them, and compare its answers to ground truth; if the validator is no more accurate than the underlying LLM's unvalidated guesses, the false-positive reduction is not an independent check.

Watch

Extended reading notes

Core claim

RepoAudit's central claim is that path-sensitive, source-to-sink bug detection at repository scale can be achieved by a demand-driven LLM agent rather than by pouring a whole project into one prompt or by building a compiler-grade static analyzer. The initiator locates source values (for example, null assignments for null pointer dereference detection); the explorer prompts the LLM to abstract each relevant function, handle pointer aliasing, enumerate feasible intra-procedural paths, and emit data-flow facts per path, storing them in memory keyed by function and value; values that escape through returns, parameters, or globals drive further exploration through the call graph. The validator then rejects data-flow facts that violate control-flow order and rejects bug reports whose inter-procedural path conditions are contradictory. On the 15-project benchmark the design reproduces every previously reported bug (21), adds 19 new true bugs (14 already fixed in the latest commit), and reaches 78.43% precision; on nine additional high-profile repositories it reports 185 new bugs, 174 confirmed or fixed by developers.

Load-bearing premise

The reported precision assumes the LLM used to check whether inter-procedural paths are feasible is reliable enough to reject hallucinated paths, even though it is the same class of model that the paper shows hallucinates on path reasoning.

Editorial extensions

If this is right

  • If the central claim holds, development-time auditing becomes practical for codebases that cannot be built: the 15 benchmark projects, averaging 251 KLoC, were each audited in about 0.44 hours and $2.54, while seven of them could not be handled by a compilation-dependent industrial analyzer in the evaluation.
  • The same explorer prompts cover null pointer dereference, memory leak, and use-after-free because all three reduce to source-to-sink data-flow reachability along feasible paths, so extending the tool to other weakness classes mainly means adding source and sink patterns in the initiator.
  • The validator is load-bearing: disabling it raises false positives in the benchmark by 245.45% (from 11 to 31), so precision comes from filtering hallucinated facts rather than from the raw reasoning of the LLM.
  • Caching in agent memory is what makes the cost numbers possible: without caching, prompting rounds and financial cost rise 3-4 times on average, and one project exceeds 72 hours of analysis time.
  • The design holds across different LLMs: with three other models, precision ranges from 82.35% to 88.46%, suggesting the agent architecture, not a single model, drives the result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same explore-cache-validate loop should transfer to other path-sensitive bug classes such as taint-style vulnerabilities or integer-overflow flows, because the explorer's core prompt targets generic data-flow reachability; the initiator's source and sink matchers would be the main new component.
  • Editorial inference: the LLM-based feasibility validator is the least independent link in the chain; replacing it with a symbolic feasibility check wherever path constraints are expressible would turn the false-positive reduction from a second opinion into a verifiable guarantee.
  • Editorial inference: a direct measurement of the validator, by running its path-feasibility prompt on hand-labeled feasible and infeasible inter-procedural paths, would show how much of the precision gain is genuine filtering rather than the explorer's own accuracy.
  • Editorial inference: the cost structure suggests a general recipe for LLM program analysis: externalize the repository graph, cache per-function summaries, and prompt with one function at a time, which is a testable hypothesis for other graph-shaped analyses such as dependency or taint tracking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. RepoAudit is an LLM-based agent for repository-level auditing of three bug types (null-pointer dereference, memory leak, and use-after-free). It combines a source-initiator, a demand-driven explorer that prompts the LLM one function at a time to produce data-flow facts along feasible paths, an agent memory with caching, and a two-part validator that checks control-flow alignment of data-flow facts and inter-procedural path feasibility. The evaluation on 15 projects reports 40 true positives (21 previously published bugs and 19 new ones), 11 false positives, precision 78.43%, and average cost 0.44 hours and $2.54 per project. Ablations assess the effects of abstraction, validation, and caching; a further scan of additional open-source projects is reported as 185 new bugs with 174 confirmed or fixed by developers. Comparisons are made with CoT prompting, LLMDFA, Meta Infer, and Amazon CodeGuru.

Significance. The external grounding of the main evaluation is a genuine strength: the 21 reproduced bugs come from prior publications, and the 174 maintainer confirmations/fixes on newly reported bugs make the headline result difficult to explain away as an artifact of the LLM's priors. The reported cost figures and the open-source release are also useful to the community. If the validation claims are substantiated, RepoAudit would be a meaningful step toward practical LLM-driven repository auditing. The robustness checks across models and temperatures further support the empirical claims. However, the paper's explanation of why precision is high is not fully supported by the reported ablation, and the field-study table contains internal inconsistencies that need correction.

major comments (3)
  1. [Section 3.3 and Appendix C] The ablation used to attribute the 245.45% false-positive reduction to the validator does not isolate the path-feasibility component. REPOAUDIT-NOVAL, as described in Appendix C, removes 'the validation of the data-flow facts discovered by the explorer and also skips examining the bug reports'—that is, it disables both the deterministic control-flow-order alignment check and the LLM-based feasibility check simultaneously. Since the alignment check is a parsing-based, rule-driven component, the observed increase in false positives could be entirely due to it. No accuracy measurement of the feasibility validator itself (e.g., on a labeled set of feasible/infeasible inter-procedural paths) is reported, and Section 2.2 documents that the same class of model hallucinates on path-sensitive reasoning. I therefore request either an ablation that toggles the two validators independently, or a direct evaluation of the feasibility validator against an oracle such as SMT solving or manually labeled paths. Without this, the paper's explanation of its precision advantage is not established.
  2. [Section 4.2 and Table 4] The real-world impact numbers are internally inconsistent. The Introduction says ten actively maintained repositories were scanned, while Section 4.2 and Table 4 report nine projects, and Table 4 lists only nine rows. The text states that 95 and 79 bugs were confirmed and fixed respectively, but summing the Con and Fix columns in Table 4 yields 96 confirmed and 78 fixed. In addition, the table's TP and FP totals (185 TPs and 29 FPs) give a precision of 86.45%, not the reported 85.71%. These are headline empirical claims, so they should be corrected and reconciled before publication.
  3. [Appendix F, Listing 2] Listing 2 contains the non-code text 'Nothing to see, move along here' inside the vrf_get function. This appears to be a placeholder or artifact and should be removed; as printed, it makes the example inconsistent with the surrounding explanation.
minor comments (5)
  1. [Section 2.2] The controlled experiment that motivates the paper reports only qualitative results ('substantial hallucinations,' 'false positives and incorrect explanations'). Appendix A provides comparisons with single-function and multiple-function detectors, but not the quantitative outcome of the specific five-function direct-prompting experiment described in Section 2.2. Please add the numbers or a pointer to the corresponding table.
  2. [Section 3.3 and Figure 5] The feasibility-validation prompt template is underspecified: it is unclear what '[PATH]' contains (source code for all involved functions, a list of path conditions, or both) and how the LLM's 'Yes or No' answer is converted into a discard decision. A brief description of the actual prompt content and parsing of the answer would improve reproducibility.
  3. [Table 4] The table caption does not define the Con and Fix columns. The text explains them, but a one-line caption definition would make the table self-contained.
  4. [Section 4.2] The phrase '95 and 79 of which have been confirmed and fixed by developers, respectively' should be reconciled with the table sums. Please also clarify whether '174 confirmed or fixed' is the union of the two sets or the sum.
  5. [Appendix A and Appendix C] There are several typos and grammatical slips, e.g., 'filed2json' in Section 3.2.1, 'desgin' and 'senarios' in Appendix A, and 'concurruption' in the Introduction. These do not affect the technical content but should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark ground truth and developer confirmations are external to the method, and no fitted parameter or self-citation chain defines the reported results.

full rationale

RepoAudit's headline results are empirical measurements against external labels: the 21 previously reported bugs are taken from prior published bug reports, the 40 TPs and 11 FPs are compared against those reports and manual inspection, and the 185 new bugs are confirmed or fixed by independent developers. No parameter of the system is fitted to these labels, and no equation defines the predicted bugs in terms of the evaluation set. The method's internal components—the initiator's tree-sitter matchers, the explorer's prompts, and the validator—are not defined in terms of the benchmark outcomes. The path-feasibility validator is implemented by prompting an LLM (Section 3.3 and Figure 5), and the NOVAL ablation removes both the deterministic alignment check and the LLM feasibility check together, so the 245.45% FP increase does not isolate the feasibility validator; however, that is a confounded-ablation and robustness concern about how much of the precision gain the validator explains, not a circular derivation. The paper's citations to the authors' own LLMDFA and LLMSAN work are used as baselines and related work, not as load-bearing premises for RepoAudit's correctness. Under the stated definition of circularity (derivation equivalent to input by construction, fitted parameter renamed as prediction, or uniqueness imported from self-citation), no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The system depends on unproven engineering assumptions: the parser's call graph and control-flow order are accurate, the LLM's program abstraction retains all relevant data-flow, the LLM-based feasibility validator is reliable, and maintainer responses are accurate labels. No free parameters are fitted to data except hand-chosen system settings (K=4, temperature=0.0).

free parameters (2)
  • K (calling context bound) = 4
    Explorer investigates data-flow across at most four functions; chosen by hand in Section 4, affects recall and cost.
  • temperature = 0.0
    Set to 0.0 to reduce randomness; ablations show precision varies from 72.92% to 78.43% across temperature settings.
assumptions (4)
  • domain assumption tree-sitter-based call graph and control-flow order are accurate
    Explorer selects functions and validator checks control-flow order based on tree-sitter parse (Sections 3.1 and 3.3); parse errors would break exploration and validation.
  • domain assumption LLM program abstraction preserves relevant data-flow facts within a function
    Explorer prompts the LLM to reduce functions to critical statements (Section 3.2.1); if abstraction drops a relevant path, bugs are missed.
  • domain assumption LLM-based feasibility validation correctly judges path condition satisfiability
    Validator discards bug reports when the LLM finds contradictions in inter-procedural path conditions (Section 3.3); false accept or reject changes precision.
  • domain assumption Developer confirmations and fixes are reliable ground truth for the new bugs
    Real-world impact claims count issues confirmed or fixed by maintainers (Section 4.2, Table 4); confirmations may be noisy but fixes are stronger evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing." pith.science (2026). https://pith.science/paper/YAXXFRAX

@misc{pith2026250118160,
  author       = {Pith},
  title        = {Pith review of: RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YAXXFRAX}},
  note         = {Machine review of arXiv:2501.18160}
}
abstract

Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability. This work introduces an LLM-based agent, RepoAudit, designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project. Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit.

Figures

Figures reproduced from arXiv: 2501.18160 by the authors.

Figure 1
Figure 1. A simplified code snippet from the project sofa-pbrpc contains a real NPD bug found by REPOAUDIT. The blue dashed arrows indicate the edges in the call graph. The red solid arrows show the data-flow facts indicating the null value propagation. The call graph of the project contains 1,508 nodes and 6,196 edges, while its data dependence graph contains 160,875 nodes and 360,096 edges. 2.1. Auditing Entails Path-Sensit… view at source ↗
Figure 2
Figure 2. Three examples of showing intrinsic strengths of LLMs To validate our speculation, we conducted a controlled ex￾periment where we prompt Claude 3.5 Sonnet with all the five functions shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The architecture of REPOAUDIT facts of the return value of the function. The model accu￾rately identifies the two possible points-to facts within the function. It even identifies the path constraints that make the corresponding points-to facts hold. In contrast, the sym￾bolic static analyzer SVF (Sui and Xue, 2016) computes the points-to facts without path conditions by default due to its inherent limitations in sem… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The prompt template for analyzing individual functions in the agent memory. The explorer performs several actions, each guided by corresponding prompts. These actions in￾clude: analyzing individual functions, selecting functions for exploration, and generating bug repo…
Figure 5
Figure 5. Figure 5: The prompt template for feasibility validation reaches a sink value, i.e., the dereferenced pointer at line 8. Hence, the explorer identifies a potential bug and report it by concatenation the data-flow facts 1 , 2 , 3 , and 4 . For several bug types, such as MLK, the …
Figure 6
Figure 6. Figure 6: The comparison results with single-function level and multiple-function level bug detectors bug detection is widely adopted and evaluated by many re￾cent studies (Chen et al., 2023; Ding et al., 2024). These techniques are applicable for models with a limited context l…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ProSec: Fortifying Code LLMs with Proactive Security Alignment

    cs.CR 2024-11 conditional novelty 8.0 of 10

    ProSec synthesizes vulnerability-inducing coding scenarios from CWE definitions and uses preference learning on model-generated fixes to make code LLMs up to 35.4% more secure on the PurpleLlama benchmark.

  2. ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Adaptive adversarial source-code comments evade four LLM vulnerability detectors at >90% success on 125 real null-pointer bugs without altering program behavior.

  3. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.

  4. Validating Network Protocol Parsers with Traceable RFC Document Interpretation

    cs.SE 2025-04 conditional novelty 7.0 of 10

    An LLM-based pipeline that extracts formal protocol formats from RFC documents detected 69 parser bugs (36 confirmed) across nine network protocols.

  5. OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review

    cs.SE 2026-08 conditional novelty 6.0 of 10

    OpenCodeReview reports up to 2.17x higher SEM-F1 than Claude Code and Codex on AACR-Bench, using 5-15x fewer tokens, by constraining agent review with rules, bounded tools, and diff-only reflection.

  6. Antares: Foundation Models for Agentic Vulnerability Localization

    cs.CR 2026-08 conditional novelty 6.0 of 10

    Antares-3B, a 3B model trained with SFT plus GRPO, matches GPT-5.5 on repository-scale vulnerability localization at roughly 1/100th the inference cost.

  7. VulWeaver: Weaving Broken Semantics for Grounded Vulnerability Detection

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    VulWeaver improves Java vulnerability detection to 0.75 F1 by enhancing dependency graphs with LLM semantic fixes, extracting full context from slices plus implicit usage info, and applying type-specific meta-promptin...

  8. Large Language Models for Validating Network Protocol Parsers

    cs.SE 2025-04 conditional novelty 6.0 of 10

    PARVAL uses multiple LLM agents to extract format specifications from both RFC 5880 and the FRRouting BFD parser and flags 34 true inconsistencies, including seven unique bugs and two RFC documentation issues.

  9. Position: Intelligent Coding Systems Should Write Programs with Justifications

    cs.SE 2025-08 conditional novelty 4.0 of 10

    A position paper advocating that intelligent coding systems should accompany code with justified explanations that are cognitively aligned and semantically faithful.

  10. Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Models

    cs.SE 2025-06 conditional novelty 4.0 of 10

    Reasoning depth in code LLMs should be managed as a controllable resource across synthetic data generation, benchmarking, and deployment, rather than left implicit.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages · cited by 10 Pith papers

  1. [1]

    static int __init damon_reclaim_init(void){

  2. [2]

    ctx = damon_new_ctx()

  3. [5]

    if (damon_select_ops(ctx, DAMON_OPS_PADDR))

  4. [7]

    Large Language Models for Validating Network Protocol Parsers

    [Online; accessed 29-Jan-2025]. Mingwei Zheng, Danning Xie, and Xiangyu Zhang. Large language models for validating network proto- col parsers, 2025. URL https://arxiv.org/abs/ 2504.13515. Mingwei Zheng, Danning Xie, Qingkai Shi, Chengpeng Wang, and Xiangyu Zhang. Validating network protocol parsers with traceable rfc document interpretation. In Proceedin...

  5. [8]

    ctx->callback.after_aggregation = damon_reclaim_after_aggregation

  6. [9]

    target = damon_new_target()

  7. [11]

    ctx->callback.after_wmarks_check = damon_reclaim_after_wmarks_check

  8. [14]

    damon_destroy_ctx(ctx)

Show all 18 references
  1. [15]

    damon_add_target(ctx, target)

  2. [16]

    schedule_delayed_work(&damon_reclaim_timer, 0)

  3. [17]

    damon_reclaim_initialized = true

  4. [18]

    Error allocating memory for context

    } Table 5. The ratios of the prompting rounds (Prompt) and input token costs (In token) of LLMDFA under two settings Bug TypeID LLMDFA-PATHSCAN LLMDFA-SRCSCAN Prompt InToken Prompt InToken NPD N1 115.39 81.23 871.61 522.40 N2 217.00 120.23 2,839.00 2,087.27 N3 6.83 3.13 255.57...

  5. [132]

    Springer, 1984. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,

  6. [899]

    doi: 10.1145/3620666.3651366

    ACM, 2024. doi: 10.1145/3620666.3651366. Anshunkang Zhou, Chengfeng Ye, Heqing Huang, Yuandao Cai, and Charles Zhang. Plankton: Reconciling binary code and debug information. In Rajiv Gupta, Nael B. Abu- Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 2...

  7. [928]

    doi: 10.1145/3620665.3640382

    ACM, 2024. doi: 10.1145/3620665.3640382. Brittany Johnson, Yoonki Song, Emerson R. Murphy-Hill, and Robert W. Bowdidge. Why don’t software devel- opers use static analysis tools to find bugs? In David Notkin, Betty H. C. Cheng, and Klaus Pohl, editors, 35th International Confe...

  8. [2008]

    Springer, 2008

    Proceedings, volume 4963 of Lecture Notes in Computer Science, pages 337–340. Springer, 2008. doi: 10.1007/978-3-540-78800-3 \ 24. Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: Analyzing dataflow in code with large language models. ...

  9. [2023]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al

    doi: 10.48550/ARXIV .2303.08774. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pa...

  10. [2025]

    12 Amazon

    [Online; accessed 29-Jan-2025]. 12 Amazon. Code Review Tool: Amazon CodeGuru Se- curity. https://aws.amazon.com/codeguru/,

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.