{"id":"ce31aaae-49aa-4ed6-8b4b-d04b4d78efdf","arxiv_id":"2502.06348","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-LLM pipeline (knowledge synthesis, prompt generation, auditing) is reported to detect 2.58x more price oracle manipulation bugs than GPTScan on 60 known DeFi vulnerabilities.","lead":"The paper builds a three-stage LLM pipeline that writes its own vulnerability audit prompts, then uses the prompts to find price oracle manipulation bugs in Solidity contracts. It reports a 2.58x recall gain over GPTScan on 60 known vulnerabilities, but with high false-positive rates and evaluation caveats that make the precise gain uncertain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline recall gain is measured on the same DeFiHacks set used to select the best LLM combination, so the 2.58x improvement may largely reflect selection optimism; out-of-sample Code4Rena results do not reproduce the claimed precision.","rationale":"The reader's CONDITIONAL verdict is appropriate: the framework is plausible, the Code4Rena result provides some out-of-sample evidence of a recall advantage, and the central concern is not that the pipeline is broken but that its headline numbers are not credible as estimates for a fixed detector. I agree with the reader that evaluation validity is the weak point, but I locate the load-bearing issue slightly earlier than the reader's stated weakest assumption: the unspecified function-matching rule is a serious interpretability problem, yet the model-selection-on-the-test-set protocol is what directly inflates the abstract's 2.58x claim. The paper never reports variance, confidence intervals, or a split between configuration selection and final evaluation; instead, Section 4.4 selects and reports on the same 36 DeFiHacks vulnerabilities. The Code4Rena evaluation is a useful check, and it does show a recall advantage, but it also shows that precision is not 'comparable' in any practically meaningful sense: both systems are at roughly 9% precision. Therefore, the central claim should be re-estimated with a fixed evaluation protocol before the specific 0.667/0.313 numbers are used. This does not require rejecting the paper; it requires conditional acceptance with a re-evaluation. Since the reader already recommended CONDITIONAL, my read does not change the verdict.","tokens_in":20324,"tokens_out":5069,"duration_ms":42769,"concrete_test":"Pre-register a function-level matching rule (exact contract/function name match, with a written reconciliation policy for renames and partial matches), then rerun Section 4.4 with leave-one-project-out cross-validation on DeFiHacks: for each held-out project, select the best Knowledge-Synthesizer/Prompt-Generator/Auditor combination on the remaining projects and score the selected combination on the held-out project. Report average held-out recall, precision, and F1 with 95% bootstrap confidence intervals over projects, and score GPTScan under the same matching rule. If the selected combination's held-out recall is not significantly above GPTScan or its precision falls below GPTScan's, the claimed 2.58x recall improvement is selection optimism rather than a property of the detector.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the best configuration identified by AiRacleX, Haiku-Haiku-4o-mini, achieves 0.667 recall and 0.313 precision versus GPTScan's 0.259 and 0.311 on the DeFiHacks benchmark. But this configuration was not evaluated as a pre-specified detector; it was selected on the same 36 vulnerabilities. Section 4.4 first chooses the best Auditor on DeFiHacks (Table 4), then the best Prompt Generator on DeFiHacks (Table 5 and Figure 2), then the best Knowledge Synthesizer on DeFiHacks (Table 6). Finding 3 then reports the winning cell's metrics as the result. Because every model-selection step is scored against the same ground truth, the reported recall and precision are in-sample estimates of the maximum over 16 configurations, not estimates of a fixed detector's performance. The only out-of-sample evidence is Table 7 on Code4Rena, where the same configuration achieves recall 0.54 and precision 0.092, while GPTScan achieves recall 0.13 and precision 0.094. The recall advantage persists out-of-sample, but 'comparable precision' does not: both tools have very low precision, and the absolute F1 is 0.157 versus 0.107. A secondary issue is that Section 4.1 says findings were manually checked, while Section 4.4.1 acknowledges that GPTScan counts vulnerability types and the authors count per-function true positives; the exact rule for matching an AiRacleX JSON finding to a ground-truth vulnerability or for handling renamed or partial functions is never specified. Without a fixed matching rule, even a correct evaluation protocol is hard to interpret.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AiRacleX, a three-stage LLM pipeline for detecting price oracle manipulation (POM) vulnerabilities in Solidity smart contracts. A Knowledge Synthesizer distills domain knowledge from seven academic papers, a Prompt Generator converts the synthesized knowledge into structured chain-of-thought prompts, and an Auditor LLM applies the prompts to contract code and outputs JSON findings. The authors evaluate 16 model configurations on a DeFiHacks dataset (31 projects, 36 vulnerabilities) and select the best combination (Claude Haiku as synthesizer and prompt generator, GPT-4o-mini as auditor). On this same dataset, the selected configuration achieves recall 0.667 and precision 0.313, versus GPTScan's 0.259 and 0.311. On a separate Code4Rena dataset (14 projects, 24 vulnerabilities), the same configuration achieves recall 0.54 and precision 0.092, versus GPTScan's 0.13 and 0.094. The paper claims a 2.58x recall improvement over GPTScan with comparable precision.","tokens_in":20619,"tokens_out":5738,"duration_ms":45320,"significance":"If the results are representative, AiRacleX would be a meaningful advance in automated smart-contract auditing: it removes the need for hand-crafted rules, shows that a combination of small models can outperform a larger single model, and the Code4Rena evaluation provides a partially out-of-sample check in which the selected configuration more than quadruples GPTScan's recall. The paper is also commendable for making its prompts and curated knowledge available in the appendix, which supports reproducibility. However, the headline claim rests on in-sample model selection, the evaluation pipeline lacks an explicit finding-to-ground-truth matching rule, and the knowledge sources overlap with the test incidents, so the current evidence does not establish the claimed improvement at the stated magnitude.","major_comments":[{"comment":"The best configuration is selected on DeFiHacks and then the same DeFiHacks set is used to report the headline '2.58-times improvement' (Finding 3; abstract). This is a selection-on-the-test-set procedure: the reported 0.667 recall / 0.313 precision is the maximum over 16 configurations evaluated on the same 36 vulnerabilities, not the performance of a pre-specified detector. The Code4Rena results (Table 7) are the only out-of-sample evidence, and they show precision of 0.092 for both tools, which does not support the 'comparable precision' claim in the abstract. Please report the selected configuration's performance on a held-out set as the primary claim, or use nested selection and report the selection bias.","section":"§4.4–§4.5, Tables 4–6"},{"comment":"The paper never specifies how an AiRacleX JSON finding (contract name, function name, beneficiary, victim, reason) is matched to a ground-truth vulnerability. The note in Section 4.4.1 says GPTScan counts vulnerability types while the authors count per-function true positives, but the actual matching rule—including how renamed/partial functions, multiple functions in one finding, and findings spread over the three repeated runs are handled—is absent. Without this rule, the reported precision and recall are not reproducible, and they could be artifacts of a lenient or ambiguous matching function. Please publish the matching procedure and the raw per-run outputs.","section":"§4.4.1–§4.4.2"},{"comment":"The knowledge sources used by the Knowledge Synthesizer include refs [40] and [44], which analyze the same real-world attack incidents and Code4Rena projects that constitute the test sets. For example, Section 4.1.1 draws 20 DeFiHacks projects from the SoK [44], and Section 4.1.2 draws 6 Code4Rena projects from Zhang et al. [40]. Since the synthesized knowledge may therefore contain incident-specific details (project names, attack procedures) that are directly relevant to the test instances, the measured improvement over GPTScan may partly reflect leakage rather than generalizable detection. Please demonstrate that the distilled knowledge does not contain test-specific information, or evaluate on incidents whose analyses are excluded from the knowledge sources.","section":"§3.2.1 and §4.1"},{"comment":"GPTScan is evaluated after two modifications: replacing ChatGPT-3.5 with ChatGPT-4o-mini and changing the tool's code to enforce structured JSON output. The comparison is therefore against a modified version of the published SOTA tool, not the tool as originally reported, and the original GPTScan's metric (vulnerability types) differs from the per-function TP metric used here (Section 4.4.1). Please justify that the modifications are neutral or favorable to GPTScan, and report original GPTScan results under both counting conventions.","section":"§4.2"},{"comment":"Each experiment is run three times and only averages are shown; no variance, confidence intervals, or significance tests are reported. Several configuration differences that drive the selection (e.g., F1 0.424 vs 0.421 in Section 4.4.2) are smaller than the typical run-to-run variability of temperature=1.0 sampling of an LLM auditor. Please report per-run values and a measure of dispersion or a statistical test, so that the model-selection decisions are distinguishable from noise.","section":"§4.3 and Tables 4–6"}],"minor_comments":[{"comment":"In the hybrid oracles bullet, 'Extra Finace2' should be 'Extra Finance' and the footnote marker is malformed.","section":"§2.2"},{"comment":"The phrase 'Constant Product Formula (CPF)??' contains stray question marks and should be cleaned up.","section":"§2.1"},{"comment":"The exclusion criteria ('excluded two projects that were unrelated', 'excluded two misclassified projects and three incomplete ones') are post-hoc and the excluded project names are not listed; please list them for transparency.","section":"§4.1.1–§4.1.2"},{"comment":"The prose refers to text highlighted in orange, brown, blue and red, but the manuscript renders these examples in plain black text; please add visible formatting or color legends.","section":"§3.2.2"},{"comment":"The contribution bullet '20 bugs which can only be detected by AiRacleX but not by GPTScan' is not directly supported by a table; the TP differences in Tables 4 and 7 suggest a different number, and a detailed breakdown would help verify the claim.","section":"§1, Contributions"},{"comment":"The column header layout of Table 5 is hard to read; consider a multi-level header or separate tables per auditor.","section":"Table 5"},{"comment":"Reference [41] is cited as 'Zhang et al.' but the reference entry lists Zheng et al.; please correct the citation.","section":"§3.2.2"},{"comment":"The abstract says 'demonstrate' where 'demonstrates' would agree with the singular subject; a light grammar pass would improve readability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the framework is clearly described, with prompts and knowledge included in appendices. The main risk is that the headline result is selected on the same set used for reporting, and the missing matching rule makes the evaluation non-reproducible. I believe these issues are fixable within the scope of a revision: the authors could re-run the Code4Rena evaluation as the primary out-of-sample claim, specify the matching procedure, and report variance. The overlap between knowledge sources and test incidents is a more delicate concern; a clean way to address it is to evaluate on incidents not analyzed in the knowledge papers or to ablate the knowledge source. I would not recommend rejection, but the current version's central quantitative claim is not yet established at the stated strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper shows a new and sensible pipeline: one LLM distills academic literature into a definition of price oracle manipulation, a second generates chain-of-thought audit prompts, and a third audits Solidity contracts. That specific three-stage orchestration is not in the prior GPTScan, GPTLens, or zero-shot baselines. It is also honest enough to show that small, cheap models (Haiku-Haiku-4o-mini) match or beat stronger ones, and that the knowledge synthesizer can replace human-curated knowledge with no loss in F1. The Code4Rena evaluation is a real out-of-sample check: the winning configuration achieves recall 0.54 versus GPTScan's 0.13, so the recall advantage is not solely an artifact of tuning on DeFiHacks.\n\nNow the soft spots, in rough order of importance. First, the headline result is selected on the same set used for reporting. Section 4.4 picks the best auditor, then the best prompt generator, then the best knowledge synthesizer, all on DeFiHacks, and Finding 3 reports that winning cell as the result. The 2.58x recall improvement is therefore the maximum over about sixteen configurations, not the expected performance of a fixed detector. The out-of-sample Code4Rena numbers are the more honest headline, but there precision collapses to 0.092 (GPTScan: 0.094), so 'comparable precision' is true only in the sense that both tools are mostly noise. Second, the matching rule between an AiRacleX JSON finding and a ground-truth vulnerability is never specified: how are renamed functions, partial matches, or multiple findings per function handled? Without that, the reported recall and precision are not fully reproducible. Third, the GPTScan baseline was modified (ChatGPT-3.5 replaced by 4o-mini, structured JSON enforced); defensible given the deprecation, but no code or data is released, so I cannot verify the comparison. Fourth, no confidence intervals or variance across the three runs are reported, and the knowledge source includes papers [40] and [44] that analyzed some of the same DeFiHacks incidents used in the test set, which risks leakage.\n\nOverall, the framework is worth building on and the paper deserves peer review. I would send it out, but I would ask for major revision: report the out-of-sample Code4Rena result as the primary evidence, specify the matching rule, release the artifacts, quantify variance, and reframe the abstract so it does not lead with an in-sample selection winner. The reader's conditional verdict is about right.","headline":"A genuinely new three-LLM pipeline for price oracle manipulation detection with a real out-of-sample recall gain over GPTScan, but the headline 2.58x number is selected on the same set that produced it and the matching procedure is underspecified.","tokens_in":21236,"tokens_out":2633,"would_cite":true,"duration_ms":24495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a three-LLM pipeline lifts price-oracle-manipulation recall from 0.259 to 0.667 versus GPTScan at comparable precision.","keywords":["price oracle manipulation","smart contract vulnerability detection","large language models","chain-of-thought prompting","prompt engineering","domain knowledge synthesis","DeFi security","audit automation"],"falsifier":"Run the Haiku-Haiku-4o-mini configuration on the dataset and count a true positive only when the model's JSON contract and function names exactly match the ground-truth vulnerable function; if recall falls toward GPTScan's 0.259, the 2.58x improvement is an artifact of loose function matching.","tokens_in":20059,"feed_emoji":"🛡️","tokens_out":8308,"duration_ms":64338,"temperature":0.7,"pith_summary":"The paper claims that price oracle manipulation (POM) detection can be fully automated as a three-stage LLM pipeline: a Knowledge Synthesizer distills a definition of POM from excerpts of seven academic papers, a Prompt Generator turns that definition into a structured chain-of-thought audit prompt, and an Auditor follows the prompt to flag vulnerable functions in Solidity code. On 60 known vulnerabilities from 46 real DeFi attacks and audit contests, the best LLM combination (Claude Haiku for knowledge and prompt generation, GPT-4o-mini for auditing) reaches 0.667 recall against 0.259 for GPTScan on the attack dataset, a 2.58x improvement at comparable precision, and 0.54 recall against 0.13 on the Code4Rena set. The paper also claims the LLM-written knowledge summary slightly outperforms human-curated knowledge (F1 0.426 vs 0.424), so the framework needs no price-oracle expert to run. A sympathetic reader would care because this is a bug class that static analysis tools and plain LLM audits mostly miss, and the result suggests the bottleneck is prompt design and knowledge grounding, not model scale.","feed_headline":"Three-LLM pipeline finds 2.6x more price-oracle bugs than GPTScan","feed_subtitle":"A knowledge-synthesizer, prompt-generator, auditor pipeline lifts recall from 0.259 to 0.667 on 60 known DeFi bugs.","key_machinery":"The load-bearing object is the generated chain-of-thought prompt. Rather than handing the Auditor a generic 'find vulnerabilities' instruction, AiRacleX has the Prompt Generator write a structured protocol with numbered steps and per-step questions (identify the price oracle; determine whether the contract relies on it without cross-checking; evaluate financial impact; recommend mitigations), grounded only in the definition produced by the Knowledge Synthesizer. The knowledge step matters because it replaces web-scale pre-training noise with externally validated descriptions drawn from seven peer-reviewed papers on oracle manipulation. The Auditor is run with temperature 1.0 and the same prompt three times, and outputs are collected as JSON so findings can be attributed to contract and function.","core_discovery":"The central discovery, on the paper's own terms, is that the hard part of POM detection is not the detector but the instruction: giving the model a step-by-step audit protocol written from a distilled problem definition lifts recall from 0.259 to 0.667 while precision stays roughly level. The pipeline's best configuration, Haiku as both Knowledge Synthesizer and Prompt Generator with 4o-mini as Auditor, also beats GPTScan on the more complex Code4Rena benchmark (recall 0.54 vs 0.13, F1 0.157 vs 0.107) and finds 20 vulnerabilities that GPTScan misses. The paper attributes the gain to three prompt-engineering moves: role-based framing, positive and negative prompts that suppress pre-training noise (e.g., not treating an oracle owner's ability to change a feed as a POM bug), and an output schema that forces the model to name beneficiary, victim, and reason before returning a verdict. It also reports that smaller models produce less conservative output, yielding more findings and fewer false negatives, which is why the best configuration uses no flagship model.","pith_inferences":["The reported 2.58x recall ratio should be read with caution until the paper publishes its finding-matching rule: the authors count true positives per function while GPTScan counted vulnerability types, and Section 4.1 says bugs were manually checked but does not say how a JSON finding is matched to a ground-truth bug; lenient matching would inflate the gain.","A direct test of the transferability claim would be to point the same three-LLM pipeline at other machine-unauditable bug categories from the same taxonomy the paper cites, such as inconsistent state updates, and measure whether recall behaves like the POM results.","Because the Auditor is stochastic (temperature 1.0, three repeats), deployment would need per-run stability reporting; a single averaged run could overstate or understate detection rates for a specific contract.","The framework's low precision on Code4Rena suggests a practical extension the authors do not build: a fourth stage that clusters or ranks the Auditor's findings to let a human review a short list, rather than hundreds of alarms."],"forward_implications":["If the reported recall gain is real, an auditor using Haiku-Haiku-4o-mini would catch about two-thirds of POM vulnerabilities in the benchmarked attack dataset, versus one-quarter for GPTScan, at roughly the same false-alarm rate.","Because the knowledge synthesizer takes academic excerpts as input, the framework is transferable to other machine-unauditable bug classes by swapping the source papers, without writing new rules or static-analysis validators.","The best performance from smaller, cheaper models implies that the design bottleneck is prompt quality rather than model capability, which lowers the cost of deploying the detector.","The structured output (contract, function, beneficiary, victim, reason) makes each finding directly interpretable and removes the need for the static-analysis validation workflow that GPTScan relies on.","On Code4Rena, precision is only 0.092, so in realistic multi-file contracts the framework is a high-recall pre-filter whose many false positives still require human triage."],"supporting_citations":[{"why":"supplies the GPTScan baseline and 11 of the DeFiHack projects; the paper re-evaluates it under a per-function true-positive counting rule.","marker":"[29]"},{"why":"earlier evaluation showing plain ChatGPT-4 finds only 33% of POM bugs with an 87% false-positive rate, the motivation for a dedicated prompt pipeline.","marker":"[11]"},{"why":"source of six Code4Rena projects and the finding that POM is 34.3% of machine-unauditable exploits, framing the gap AiRacleX targets.","marker":"[40]"},{"why":"SoK of DeFi attacks that supplies 20 of the 31 real-world attacked projects in the dataset.","marker":"[44]"},{"why":"zero-shot chain-of-thought prompting ('let's think step by step'), the technique the Prompt Generator builds on and the baseline prompt mimics.","marker":"[15]"},{"why":"chain-of-thought prompting method used to structure the generated audit prompts.","marker":"[34]"},{"why":"DeFiRanger, a pattern-matching POM detector whose post-mortem, transaction-based approach the framework positions against.","marker":"[37]"},{"why":"DeFiTainter, a taint-tracking POM detector requiring on-chain data, cited as an alternative that differs from the static framework.","marker":"[16]"},{"why":"ProMutator, a mutated-transaction POM detector and one of the seven papers whose excerpts feed the Knowledge Synthesizer.","marker":"[32]"}],"fun_headline_variants":["3-LLM trick: better prompts catch 2.6x more oracle bugs","LLM trio finds 2.6x more smart contract oracle exploits","Prompt engineering boosts oracle bug recall by 2.58x","Open-source LLMs match commercial for oracle audit tasks","AI pipeline catches 20 bugs GPTScan misses in DeFi"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the manually checked ground-truth labels and on an unpublished rule for deciding when an AiRacleX finding (contract name plus function name) counts as a hit; if that rule is lenient, the reported recall and precision are artifacts of matching rather than of detection quality.","fun_headline_variants_meta":{"raw":{"variants":["3-LLM trick: better prompts catch 2.6x more oracle bugs","LLM trio finds 2.6x more smart contract oracle exploits","Prompt engineering boosts oracle bug recall by 2.58x","Open-source LLMs match commercial for oracle audit tasks","AI pipeline catches 20 bugs GPTScan misses in DeFi"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1740,"prompt_tokens":1059,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":590}},"tokens_in":675,"tokens_out":681,"duration_ms":5966,"temperature":1.0,"reasoning_tokens":590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:45:05.210302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Haiku-Haiku-4o-mini configuration on the dataset and count a true positive only when the model's JSON contract and function names exactly match the ground-truth vulnerable function; if recall falls toward GPTScan's 0.259, the 2.58x improvement is an artifact of loose function matching.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"earlier evaluation showing plain ChatGPT-4 finds only 33% of POM bugs with an 87% false-positive rate, the motivation for a dedicated prompt pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"source of six Code4Rena projects and the finding that POM is 34.3% of machine-unauditable exploits, framing the gap AiRacleX targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SoK of DeFi attacks that supplies 20 of the 31 real-world attacked projects in the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeFiTainter, a taint-tracking POM detector requiring on-chain data, cited as an alternative that differs from the static framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ProMutator, a mutated-transaction POM detector and one of the seven papers whose excerpts feed the Knowledge Synthesizer."}],"review_version":1}