{"id":"d6b4ae38-d860-4760-b7ef-0a934457ace3","arxiv_id":"2501.18160","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A demand-driven LLM agent with memory and validation detects 40 bugs in 15 real-world projects at 78.43% precision, plus 185 new bugs in high-profile repositories, 174 confirmed or fixed.","lead":"RepoAudit is an LLM agent that audits whole code repositories by tracing data-flow paths function by function, then filters its bug reports with a validator. In benchmarks it found 40 real bugs in 15 projects with 78.43% precision at about $2.54 and 0.44 hours per project, and maintainers confirmed or fixed 174 of the bugs it reported in popular open-source projects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FP reduction attributed to the validator is not isolated: REPOAUDIT-NOVAL disables both the deterministic alignment check and the LLM feasibility check together, so the 245.45% FP increase does not show the LLM path-feasibility validator actually works.","rationale":"I agree with the reader's identification: the validator is the weakest link in the explanation of precision. The external evidence (21/21 reproduced, 174 maintainer confirmations) makes the empirical headline credible, so I am not proposing rejection; however, the paper's mechanism claim is not yet supported. The ablation confound is concrete: REPOAUDIT-NOVAL removes both validators, so the 245.45% FP increase could be driven entirely by the deterministic alignment check, which is not the controversial component. The paper also uses \"satisfiability\" language in the abstract and Section 3.3 while the implementation is an LLM prompt, which overstates the rigor. This does not change the reader's CONDITIONAL verdict but strengthens the conditions: isolate validator components, report validator accuracy, and provide the artifact and bug labels for reproduction. The paper's own Limitations section acknowledges unsoundness and the four-function bound, which further supports a conditional rather than an accept verdict. The count inconsistencies (185 vs 186, nine vs ten projects) are minor editorial issues and not the load-bearing concern; correcting them is still worthwhile for a revision.","tokens_in":25074,"tokens_out":9926,"duration_ms":96667,"concrete_test":"Run two additional ablations on the same 15 benchmark projects: (a) alignment-only, keeping the parsing-based control-flow-order validator while removing the LLM path-feasibility prompt; and (b) feasibility-only, keeping the LLM prompt while removing the alignment validator. Compare TP/FP counts and precision against the full system and against REPOAUDIT-NOVAL. If variant (a) already yields 11 FPs and 40 TPs, or a similar reduction, then the LLM feasibility validator contributes little and the claimed mechanism is not established. If variant (b) shows a large FP reduction without dropping TPs, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central precision claim is empirical, but its explanation of why precision is high rests on Section 3.3's validator, whose \"path feasibility validation\" is implemented by prompting an LLM (Figure 5) to detect contradictions in inter-procedural path conditions. This is not a satisfiability check in the formal sense used in Section 2.3.3, and it is the same class of model that Section 2.2 shows hallucinates on path-sensitive reasoning. The only ablation evidence for this validator is Appendix C's REPOAUDIT-NOVAL, which removes \"the validation of the data-flow facts discovered by the explorer and also skips examining the bug reports\" — i.e., it disables both the deterministic control-flow-order alignment validator and the LLM feasibility validator simultaneously. The reported 245.45% FP increase therefore cannot be attributed to the LLM feasibility component; the deterministic alignment check alone could account for most or all of the improvement. No accuracy measurement of the feasibility validator itself (e.g., on a labeled set of feasible/infeasible inter-procedural paths) is reported, so there is no direct evidence that the LLM feasibility prompt is more reliable than the explorer it is meant to check. If the feasibility validator is unreliable, the FP reduction and the 78.43% precision may be specific to the benchmark/model rather than a general property of the method, and the paper's claim that the validator \"check[s] the satisfiability of path conditions\" is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RepoAudit is an LLM-based agent for repository-level auditing of three bug types (null-pointer dereference, memory leak, and use-after-free). It combines a source-initiator, a demand-driven explorer that prompts the LLM one function at a time to produce data-flow facts along feasible paths, an agent memory with caching, and a two-part validator that checks control-flow alignment of data-flow facts and inter-procedural path feasibility. The evaluation on 15 projects reports 40 true positives (21 previously published bugs and 19 new ones), 11 false positives, precision 78.43%, and average cost 0.44 hours and $2.54 per project. Ablations assess the effects of abstraction, validation, and caching; a further scan of additional open-source projects is reported as 185 new bugs with 174 confirmed or fixed by developers. Comparisons are made with CoT prompting, LLMDFA, Meta Infer, and Amazon CodeGuru.","tokens_in":25369,"tokens_out":7745,"duration_ms":70076,"significance":"The external grounding of the main evaluation is a genuine strength: the 21 reproduced bugs come from prior publications, and the 174 maintainer confirmations/fixes on newly reported bugs make the headline result difficult to explain away as an artifact of the LLM's priors. The reported cost figures and the open-source release are also useful to the community. If the validation claims are substantiated, RepoAudit would be a meaningful step toward practical LLM-driven repository auditing. The robustness checks across models and temperatures further support the empirical claims. However, the paper's explanation of why precision is high is not fully supported by the reported ablation, and the field-study table contains internal inconsistencies that need correction.","major_comments":[{"comment":"The ablation used to attribute the 245.45% false-positive reduction to the validator does not isolate the path-feasibility component. REPOAUDIT-NOVAL, as described in Appendix C, removes 'the validation of the data-flow facts discovered by the explorer and also skips examining the bug reports'—that is, it disables both the deterministic control-flow-order alignment check and the LLM-based feasibility check simultaneously. Since the alignment check is a parsing-based, rule-driven component, the observed increase in false positives could be entirely due to it. No accuracy measurement of the feasibility validator itself (e.g., on a labeled set of feasible/infeasible inter-procedural paths) is reported, and Section 2.2 documents that the same class of model hallucinates on path-sensitive reasoning. I therefore request either an ablation that toggles the two validators independently, or a direct evaluation of the feasibility validator against an oracle such as SMT solving or manually labeled paths. Without this, the paper's explanation of its precision advantage is not established.","section":"Section 3.3 and Appendix C"},{"comment":"The real-world impact numbers are internally inconsistent. The Introduction says ten actively maintained repositories were scanned, while Section 4.2 and Table 4 report nine projects, and Table 4 lists only nine rows. The text states that 95 and 79 bugs were confirmed and fixed respectively, but summing the Con and Fix columns in Table 4 yields 96 confirmed and 78 fixed. In addition, the table's TP and FP totals (185 TPs and 29 FPs) give a precision of 86.45%, not the reported 85.71%. These are headline empirical claims, so they should be corrected and reconciled before publication.","section":"Section 4.2 and Table 4"},{"comment":"Listing 2 contains the non-code text 'Nothing to see, move along here' inside the vrf_get function. This appears to be a placeholder or artifact and should be removed; as printed, it makes the example inconsistent with the surrounding explanation.","section":"Appendix F, Listing 2"}],"minor_comments":[{"comment":"The controlled experiment that motivates the paper reports only qualitative results ('substantial hallucinations,' 'false positives and incorrect explanations'). Appendix A provides comparisons with single-function and multiple-function detectors, but not the quantitative outcome of the specific five-function direct-prompting experiment described in Section 2.2. Please add the numbers or a pointer to the corresponding table.","section":"Section 2.2"},{"comment":"The feasibility-validation prompt template is underspecified: it is unclear what '[PATH]' contains (source code for all involved functions, a list of path conditions, or both) and how the LLM's 'Yes or No' answer is converted into a discard decision. A brief description of the actual prompt content and parsing of the answer would improve reproducibility.","section":"Section 3.3 and Figure 5"},{"comment":"The table caption does not define the Con and Fix columns. The text explains them, but a one-line caption definition would make the table self-contained.","section":"Table 4"},{"comment":"The phrase '95 and 79 of which have been confirmed and fixed by developers, respectively' should be reconciled with the table sums. Please also clarify whether '174 confirmed or fixed' is the union of the two sets or the sum.","section":"Section 4.2"},{"comment":"There are several typos and grammatical slips, e.g., 'filed2json' in Section 3.2.1, 'desgin' and 'senarios' in Appendix A, and 'concurruption' in the Introduction. These do not affect the technical content but should be cleaned up.","section":"Appendix A and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an ML venue as an applied LLM-agent system. The externally grounded bug findings and maintainer confirmations suggest the core empirical result will survive, but the validator ablation conflates two mechanisms and the field-study table has inconsistencies in headline counts. A revision that isolates the validators and corrects the counts would make the paper much stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result here is real and worth taking seriously: RepoAudit reproduces 21/21 known bugs, reports 40 true positives across 15 benchmark projects at 78.43% precision, and — the strongest evidence — 174 of 185 newly reported bugs in popular repositories were confirmed or fixed by maintainers. That is external ground truth, not self-measurement, and it makes the central claim credible. The system is also a coherent engineering contribution: demand-driven function-level exploration, agent memory with caching, and a two-part validator. The cost numbers ($2.54, 0.44 hours per project) are attractive, and the code is open-sourced. Credit where due.\n\nThe soft spots are real but not fatal. The stress-test note is right: the 245.45% FP increase in the no-validator ablation cannot be attributed to the LLM path-feasibility check, because REPOAUDIT-NOVAL disables both the deterministic control-flow-order alignment and the LLM feasibility prompt simultaneously. The deterministic check alone could explain most of the improvement. The paper never measures the feasibility validator in isolation, e.g., on a labeled set of feasible vs. infeasible inter-procedural paths. Given that Section 2.2 documents the same class of model hallucinating on path-sensitive reasoning, calling the LLM feasibility check a satisfiability check is unsupported, and the precision numbers would feel more robust with that component pinned down.\n\nThere are also minor mechanical inconsistencies — 185 vs. 186 bugs, nine vs. ten projects — that should be fixed. The \"first purely LLM-driven repo-level auditor\" claim is positioning; the paper builds directly on LLMDFA and LLMSAN, and the authors know it.\n\nWho is this for? Researchers in LLM-for-code and practical program analysis. It will not shift paradigms, but it is a useful, reproducible system with strong real-world evidence. I would want the validator ablation unconfounded and the feasibility check evaluated directly before trusting the precision explanation, but the maintainer-confirmation numbers stand on their own.\n\nRecommendation: send it to peer review. A serious referee will ask for the isolated validator analysis and consistency cleanup, not for a rewrite of the core idea.","headline":"A genuinely useful LLM auditing agent with externally grounded results, but the validator ablation is confounded and the 'first' claim is overstated; still deserves serious refereeing.","tokens_in":25902,"tokens_out":1100,"would_cite":true,"duration_ms":13211,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An autonomous LLM agent can audit an entire code repository without compiling it: it reproduces all 21 known bugs in a 15-project benchmark, finds 19 more, at 78.43% precision.","keywords":["LLM agent","code auditing","data-flow analysis","path-sensitive analysis","bug detection","null pointer dereference","memory leak","use-after-free"],"falsifier":"Build a labeled set of inter-procedural paths with known feasibility, run only the validator's path-feasibility prompt on them, and compare its answers to ground truth; if the validator is no more accurate than the underlying LLM's unvalidated guesses, the false-positive reduction is not an independent check.","tokens_in":24891,"feed_emoji":"🐛","tokens_out":12140,"duration_ms":107138,"temperature":0.7,"pith_summary":"The paper argues that repository-level code auditing, the hunt for bugs whose causes and symptoms span many functions and files, can be done by an autonomous LLM agent that never compiles or executes the code. The agent breaks the repository into per-function analysis steps, stores discovered data-flow facts in an agent memory, and validates candidate bug reports before presenting them. The claimed payoff is scale and cost: across 15 real-world projects averaging 251 KLoC it reproduces all 21 previously known bugs and finds 19 new ones at 78.43% precision, spending $2.54 and 0.44 hours per project; across nine additional active repositories it reports 185 new bugs, 174 of which developers confirmed or fixed. If the claim holds, development-time auditing becomes practical for codebases that are uncompilable or under active change, where conventional static analyzers and industrial tools miss the bugs or cannot run.","feed_headline":"LLM agent audits whole repos: 40 real bugs, $2.54 per project","feed_subtitle":"No compiler needed: it finds path-sensitive bugs in large C/C++ repos, 174 of them confirmed or fixed.","key_machinery":"The load-bearing mechanism is the agent memory plus the validator loop. The memory stores, for each function and program value, a set of pairs of a feasible program path and the data-flow facts discovered along that path; before re-analyzing a value in a function, the explorer consults this cache, and this is what keeps token and time costs low. The validator filters two kinds of hallucination: a parsing-based check rejects facts that violate control-flow order, and a second LLM prompt rejects inter-procedural paths whose branch conditions are unsatisfiable. Together with the function-analysis prompt, these checks turn per-function LLM reasoning into repository-level auditing without compiling the code.","core_discovery":"RepoAudit's central claim is that path-sensitive, source-to-sink bug detection at repository scale can be achieved by a demand-driven LLM agent rather than by pouring a whole project into one prompt or by building a compiler-grade static analyzer. The initiator locates source values (for example, null assignments for null pointer dereference detection); the explorer prompts the LLM to abstract each relevant function, handle pointer aliasing, enumerate feasible intra-procedural paths, and emit data-flow facts per path, storing them in memory keyed by function and value; values that escape through returns, parameters, or globals drive further exploration through the call graph. The validator then rejects data-flow facts that violate control-flow order and rejects bug reports whose inter-procedural path conditions are contradictory. On the 15-project benchmark the design reproduces every previously reported bug (21), adds 19 new true bugs (14 already fixed in the latest commit), and reaches 78.43% precision; on nine additional high-profile repositories it reports 185 new bugs, 174 confirmed or fixed by developers.","pith_inferences":["Editorial inference: the same explore-cache-validate loop should transfer to other path-sensitive bug classes such as taint-style vulnerabilities or integer-overflow flows, because the explorer's core prompt targets generic data-flow reachability; the initiator's source and sink matchers would be the main new component.","Editorial inference: the LLM-based feasibility validator is the least independent link in the chain; replacing it with a symbolic feasibility check wherever path constraints are expressible would turn the false-positive reduction from a second opinion into a verifiable guarantee.","Editorial inference: a direct measurement of the validator, by running its path-feasibility prompt on hand-labeled feasible and infeasible inter-procedural paths, would show how much of the precision gain is genuine filtering rather than the explorer's own accuracy.","Editorial inference: the cost structure suggests a general recipe for LLM program analysis: externalize the repository graph, cache per-function summaries, and prompt with one function at a time, which is a testable hypothesis for other graph-shaped analyses such as dependency or taint tracking."],"forward_implications":["If the central claim holds, development-time auditing becomes practical for codebases that cannot be built: the 15 benchmark projects, averaging 251 KLoC, were each audited in about 0.44 hours and $2.54, while seven of them could not be handled by a compilation-dependent industrial analyzer in the evaluation.","The same explorer prompts cover null pointer dereference, memory leak, and use-after-free because all three reduce to source-to-sink data-flow reachability along feasible paths, so extending the tool to other weakness classes mainly means adding source and sink patterns in the initiator.","The validator is load-bearing: disabling it raises false positives in the benchmark by 245.45% (from 11 to 31), so precision comes from filtering hallucinated facts rather than from the raw reasoning of the LLM.","Caching in agent memory is what makes the cost numbers possible: without caching, prompting rounds and financial cost rise 3-4 times on average, and one project exceeds 72 hours of analysis time.","The design holds across different LLMs: with three other models, precision ranges from 82.35% to 88.46%, suggesting the agent architecture, not a single model, drives the result."],"supporting_citations":[{"why":"LLMDFA is the agent-centric data-flow baseline that RepoAudit compares against; it summarizes data-flow facts for all functions, making the cost and demand-driven design load-bearing.","marker":"Wang et al., 2024a"},{"why":"Supplies the direct-prompting methodology and controlled setup showing that LLMs hallucinate on multi-function null-pointer reasoning, motivating the agent architecture.","marker":"Fang et al., 2024"},{"why":"Defines the data dependence graph that gives the formal notion of data-flow facts along paths that RepoAudit collects and stores in memory.","marker":"Ferrante et al., 1984"},{"why":"Z3 is the conventional satisfiability oracle for path conditions, the baseline against which the paper's LLM-based feasibility validation is positioned.","marker":"de Moura and Bjørner, 2008"},{"why":"SVF provides the traditional points-to and value-flow analysis the paper contrasts with LLM pointer handling and path-condition reasoning.","marker":"Sui and Xue, 2016"},{"why":"DiverseVul is one of the function-level vulnerability datasets behind the single-function chain-of-thought baseline that RepoAudit outperforms.","marker":"Chen et al., 2023"},{"why":"Provides the benchmark and evaluation methodology for function-level LLM vulnerability detection used as a baseline.","marker":"Ding et al., 2024"},{"why":"One source of the previously reported bugs that RepoAudit must reproduce in the 15-project benchmark, anchoring the recall claim.","marker":"Huang et al., 2024"},{"why":"Another source of benchmark bugs plus a path-sensitive sparse analysis baseline against which RepoAudit's approach is framed.","marker":"Shi et al., 2021"},{"why":"Pinpoint contributes previously reported bugs used in the reproduction benchmark and represents the compiler-based value-flow analysis the paper compares with.","marker":"Shi et al., 2018b"}],"fun_headline_variants":["Autonomous LLM auditor finds 40 real bugs in 15 repos","RepoAudit: LLM agent finds 40 true bugs, 78% precision, no compiler","LLM agent audits repos on demand: 40 true bugs, $2.54 per repo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported precision assumes the LLM used to check whether inter-procedural paths are feasible is reliable enough to reject hallucinated paths, even though it is the same class of model that the paper shows hallucinates on path reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Autonomous LLM auditor finds 40 real bugs in 15 repos","RepoAudit: LLM agent finds 40 true bugs, 78% precision, no compiler","LLM agent audits repos on demand: 40 true bugs, $2.54 per repo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001431,"raw_usage":{"total_tokens":5798,"prompt_tokens":999,"completion_tokens":4799,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":4726}},"tokens_in":615,"tokens_out":4799,"duration_ms":33457,"temperature":1.0,"reasoning_tokens":4726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:26:56.187889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a labeled set of inter-procedural paths with known feasibility, run only the validator's path-feasibility prompt on them, and compare its answers to ground truth; if the validator is no more accurate than the underlying LLM's unvalidated guesses, the false-positive reduction is not an independent check.","supporting_citations":[],"review_version":1}