Pith. sign in

REVIEW 5 major objections 6 minor 46 references

Adaptive Plan-Execute Framework for Smart Contract Security Auditing

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SmartAuditFlow claims that an adaptive plan-execute workflow detects 100 percent of common smart contract vulnerabilities and all 13 tested CVEs.

desk verdict Solid system-building paper whose headline numbers all run through an unvalidated LLM judge; the framework deserves referee time, but the evaluation needs human-validated labels and honest recall/precision reporting. read the letter →

arxiv 2505.15242 v2 pith:RB5AK47R submitted 2025-05-21 cs.CR cs.AI

Zhiyuan Wei , Jing Sun , Zijian Zhang , Zhe Hou , Zixiao Zhao This is my paper
classification cs.CRcs.AI
keywords smartcontractauditingLLMsecurityanalysisPlan-ExecuteframeworkadaptiveauditplanningvulnerabilitydetectionRetrieval-AugmentedGenerationpromptoptimizationCVE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SmartAuditFlow, an LLM-driven smart contract auditor built on a Plan-Execute loop: it first reads the contract and builds a tailored audit plan, then executes each planned check with a separate review-and-calibration step, and lets intermediate findings reshape later checks. The authors claim this adaptive structure fixes two weaknesses of plain LLM auditing: hallucinated findings and fixed, shallow workflows. On the SmartBugs standard dataset it reports 100% detection of common vulnerabilities; on 72 real-world audited projects it reports 41.2% top-max accuracy with only 6.2 findings per contract; and on a 13-CVE benchmark it reports catching all 13, including several that prior LLM auditors missed. The intended payoff is a scalable, low-cost audit that behaves more like a methodical human auditor than a checklist.

What carries the argument

The load-bearing mechanism is the adaptive audit plan: a structured Plan-Execute loop in which the LLM first produces a context-aware understanding of the contract, turns that understanding into a prioritized list of sub-tasks, and then treats each sub-task as a review-then-calibrate pair, keeping only findings whose confidence exceeds a threshold. The two supporting mechanisms are the evolutionary prompt optimizer, which searches for instructions that maximize an output-alignment-plus-log-likelihood score, and the RAG-enhanced calibration, which retrieves vulnerability-pattern documents so the validation step can ground its verdicts in external evidence. Static-analysis tool outputs enter at the initial-analysis stage as additional context. Together these mechanisms implement the claim that a workflow can compensate for an LLM's individual weaknesses.

What would settle it

Take a sample of contracts from the Standard Vulnerability Set and the Real-World Set, have independent human auditors re-label each SmartAuditFlow finding against the same ground truth, and compare their labels with the LLM evaluator's Exact/Partial/Incorrect outputs; if human experts reject a material share of the matches the evaluator counted as true positives, the reported detection rates are not reproducible.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that decomposing an LLM audit into dynamically generated, prioritized sub-tasks, and then validating each sub-task's result before aggregating it, substantially improves both recall and precision over prompting an LLM once. The workflow runs five stages: context-aware initial analysis, adaptive audit planning, multi-task execution with confidence-thresholded validation, cross-cutting synthesis, and report generation. Two auxiliary mechanisms carry most of the weight: an evolutionary prompt-optimization loop that tunes the instructions used at the planning stages, and external grounding through static-analysis outputs and retrieval-augmented generation during calibration. The authors further claim that their LLM-powered evaluator, which maps SmartAuditFlow findings to expert reports using Exact, Partial, and Incorrect categories, makes the comparison fair and reproducible. In their experiments, SmartAuditFlow reaches 100% top-max accuracy on the common-vulnerability set, 41.2% on the real-world set with the best ranking quality among compared methods, and 13/13 on the CVE benchmark.

Load-bearing premise

The reported accuracy figures all pass through the LLM-Powered Audit Evaluator's judgment of which findings match the expert ground truth; if that judge is lenient, biased, or imprecise, every headline number—100%, 41.2%, and 13/13—can be inflated.

Editorial extensions

If this is right

  • If the 100% common-vulnerability result holds, an adaptive LLM workflow can serve as a high-recall first pass for standard vulnerability classes, leaving humans to focus on business-logic flaws.
  • If the 41.2% real-world accuracy holds with only about 6.2 findings per contract, automated audits become cheap enough to run repeatedly during development instead of once before deployment.
  • If the 13/13 CVE detection holds, the framework can catch known historical exploits that static tools and earlier LLM methods miss, making it useful for regression checks on codebases with known risk patterns.
  • If the modularity result holds, teams can choose their backbone LLM by priority: higher accuracy with Gemini-2.5-pro at 37.2%, better ranking with GPT-4o at MAP 0.334, or a balanced cost-quality tradeoff with Claude-3.7-sonnet.
  • If the ablation result holds, combining static analysis and RAG raises top-max accuracy by 4.0 percentage points over the baseline, supporting the claim that external grounding reduces false positives without inflating output volume.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the evaluator's match labeling is validated against human judgment and the reported numbers survive, the framework's deeper contribution would be its ranking and conciseness gains, because those are the properties that reduce audit cost in practice.
  • The same plan-execute loop with RAG-calibrated validation could transfer to other property-rich code domains, such as Rust-based smart contracts or upgradeability-proxy analysis, though the paper does not test that transfer.
  • A cheap test of the weakest assumption would be to audit the evaluator itself: feed it pairs of known-matching and known-distinct findings, measure its Exact/Partial/Incorrect agreement, and then decide whether any downstream accuracy figure is trustworthy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SmartAuditFlow, a Plan-Execute framework for LLM-based smart contract auditing. The framework performs context-aware initial analysis (A1), adaptive audit planning (A2), multi-task execution with calibration and RAG (A3), findings synthesis (A4), and report generation (A5); prompts for A1/A2 are optimized with an evolutionary algorithm, Slither is used for static grounding, and a curated knowledge base supports retrieval-augmented calibration. The authors evaluate on the SmartBugs standard set (143 contracts), a Code4rena real-world set (72 projects containing 784 high/medium-severity findings), and 13 smart-contract CVEs, claiming 100% accuracy on the standard set, 41.2% top-max accuracy on real-world contracts, and 13/13 CVE detection, outperforming static tools and standalone LLMs. The paper also reports ablations on external knowledge and different LLM backbones.

Significance. SmartAuditFlow is architecturally interesting: the Plan-Execute decomposition is principled, Algorithm 2 gives a concrete prompt-optimization procedure, and the integration of Slither and RAG is clearly described. The authors state that data, annotations, and processing scripts are open, which supports reproducibility. If the claimed detection rates were confirmed, this would be a meaningful step for automated smart-contract auditing. However, all headline metrics flow through an LLM-based evaluator that is not validated against human experts, the reported 'accuracy' is actually recall per ground-truth instance, and several numeric claims are internally inconsistent. The central empirical claim therefore cannot be accepted on the evidence presented.

major comments (5)
  1. [Section 3.4, Section 4.2.2] The LLM-Powered Audit Evaluator is the sole arbiter of True Positives and False Positives, but the paper provides no evidence that its match classifications agree with human expert judgments. Since the auditor and the evaluator are both LLM outputs, the reported 100%, 41.2%, 310 TPs, and 13/13 CVEs may reflect self-consistent over-matching rather than genuine detection. The 'Partial Match' category is defined but its treatment in the quantitative metrics is never specified. The authors must validate the evaluator against human annotations (e.g., Cohen's kappa or precision/recall of the judge), provide the judge prompt and rubric, state how partial matches are scored, and disclose the confidence thresholds used in Algorithm 1.
  2. [Section 4.2.2, Section 4.4.1] The paper calls its main metric 'accuracy,' but the procedure counts a ground-truth vulnerability as detected if it is 'reported anywhere in the output'; this is recall per instance, not accuracy. Additionally, the definition of False Positive is actually a false negative ('the framework fails to detect a vulnerability that is present'), and no precision or false-positive rate is reported anywhere; Tables 1-4 report only recall-oriented top-max accuracy, MRR, MAP, and output counts. Without precision or F1, the claims about 'minimizing false positives' and about 'accuracy' are unsupported. The authors should rename the metric, report false-positive counts/precision/F1, and give confidence intervals or variance across repeated runs.
  3. [Section 4.4.6, Table 5, Section 6, Abstract, Section 4.4.2] The headline numbers are internally inconsistent and cannot be reconstructed from the text. The conclusion reports 97.2% where the abstract and Section 4.4.1 report 100%; Table 1 shows 100% top-max but Section 4.4.2 says 99.2%; Table 5 marks 10/13 CVEs for PropertyGPT while the text says 9/13, and marks 10/13 for GPT-o3 while the text says 11/13; and 310 TPs out of 784 instances is 39.5%, not the claimed 41.2%. These discrepancies should be resolved by recomputing every headline number from the raw outputs and checking it against the corresponding table.
  4. [Section 4.2.1, Section 3.2.2] The Prompt Optimization Set (1,000 contracts) is used to evolve prompts for Stages A1 and A2, but the paper never states whether this set is disjoint from the SmartBugs, Code4rena, and CVE evaluation sets. If the optimized prompts were tuned on contracts that later appear in the evaluation, the comparisons in Tables 1-5 are inflated. The paper must confirm disjointness (e.g., no overlapping contract addresses or code hashes) or re-optimize on a genuinely separate set. The selection of 13 CVEs also needs a justification that the set is representative, rather than merely inherited from PropertyGPT.
  5. [Section 4.4.3-4.4.5, Tables 2-4] The real-world evaluation reports point estimates for a single attempt (pass@1) without error bars or multiple runs, despite LLM stochasticity; the 'average outputs' and MAP values also omit standard deviations. Furthermore, Section 4.4.5's baseline uses Gemini-2.5-pro (Table 4) while Section 4.4.3's headline uses GPT-4o (Table 2), so the 41.2% full-system figure is not directly comparable to the 32.4% GPT-4o result. Reporting distributions and a consistent baseline is necessary to support RQ3-RQ5.
minor comments (6)
  1. [Section 1] There is a typo in 'dvanced performance metrics' in the introduction.
  2. [Section 4.2.1] The SmartBugs and Code4rena datasets are cited to Durieux et al. [2020], but that reference is an empirical review of automated analysis tools, not the dataset source; please cite the original dataset or competition.
  3. [Table 7] The title of Table 7 is 'Prompt Optimization Dataset Statistics', but the table reports instruction scores; the title should be changed to reflect the content.
  4. [Appendix A] There are typos in the appendix, including 'expexted result' and 'Guithub repositories'.
  5. [Table 5] The note in Table 5 says '× indicates an incorrect detection (FP)', but a missed ground-truth vulnerability is a false negative, not a false positive; the terminology should be corrected.
  6. [Section 3.3.2] Equations (5) and (6) are referenced in the text but are not numbered; please add equation numbers.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline metrics are counts produced by an unvalidated LLM judge whose match rubric defines TP; ground truth is external but the measurement is not independent.

  1. self definitional [Section 3.4 (LLM-Powered Audit Evaluator) and Section 4.2.2 (Evaluation Criteria, TP definition)]
    "True Positive (TP): The framework correctly identifies a vulnerability that exists in the contract, as validated against the ground truth through our LLM-Powered Audit Evaluator."

    The paper's central claims—100% on the SmartBugs set, 41.2% on the real-world set, 310 TPs, and 13/13 CVEs—are all counts of TPs. TP is defined as 'validated ... through our LLM-Powered Audit Evaluator.' That evaluator is itself an LLM applying a subjective Exact/Partial/Incorrect rubric with no reported inter-rater agreement against human experts. Thus the reported accuracies are not independent measurements of SmartAuditFlow's outputs versus external truth; they are restatements of the LLM judge's match categorizations. Because the auditor and the judge are the same kind of model, observed agreement reflects LLM self-consistency as much as real detection, so the headline numbers are partially determined by the evaluator's own definitions.

full rationale

The framework's derivation chain is otherwise self-contained: prompts are optimized on a dedicated 1,000-contract set and evaluated on separate SmartBugs, Code4rena, and CVE benchmarks; there is no leakage of test labels into the optimization objective, and the paper's self-citations (e.g., Wei et al. 2023) are background only. The single load-bearing circularity is the measurement instrument. Section 3.4 creates an LLM-Powered Audit Evaluator, and Section 4.2.2 defines every True Positive as a finding 'validated against the ground truth through' that evaluator. No experiment demonstrates that the LLM judge's Exact/Partial/Incorrect categorizations agree with human expert auditors, and the paper does not state how Partial Matches are scored in the quantitative metrics. Consequently, the headline accuracies, MRR, MAP, and 13/13 CVE count are counts of the evaluator's judgments rather than independent evidence, warranting a partial circularity score of 4 rather than 0. If the judge were validated against human labels or if partial-match scoring were specified, this concern would drop and the score would be 0-2.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework introduces no physical or formal entities; it is a composition of existing techniques (LLM calls, RAG, static analysis, prompt optimization). The main unwarranted reliance is on unverified assumptions about data, ground truth, and the judge rather than on new postulated entities.

free parameters (4)
  • THRESHOLD_CONFIDENCE = unspecified
    Algorithm 1 (line 8) filters findings by a confidence threshold, but no value or calibration method is given; this filter directly determines which findings appear in the final report and affects every accuracy figure.
  • Prompt optimization hyperparameters = k=20, ke=10, T=10, tau=0.7, beta=0.3, eps=0.1, alpha=0.3, lambda=0.01
    Hand-set values in Appendix A control which prompts are selected for the Plan phase; different hyperparameters yield different prompts and therefore different audit results.
  • Scoring weights (w_exec, w_log, w_cov, w_det) = 0.7, 0.3, 0.6, 0.4
    Weights in Eq. 3 and Appendix A define the prompt-quality objective used to select the final instructions; they are choices, not data-derived.
  • RAG top-k = unspecified
    Section 3.3.3 retrieves top-k document snippets for calibration but never specifies k or the retrieval configuration, both of which influence calibration quality and the final findings.
assumptions (5)
  • domain assumption The SmartBugs-curated dataset annotations are correct and complete.
    Section 4.2.1 treats this dataset as ground truth for the 100% claim; any annotation error propagates directly into the reported accuracy.
  • domain assumption The LLM-Powered Audit Evaluator produces valid and unbiased match classifications.
    Section 3.4 uses an LLM to categorize findings as exact, partial, or incorrect matches; without human-validated agreement, all TP/FP counts depend on this assumption.
  • ad hoc to paper The prompt optimization set is disjoint from the evaluation benchmarks.
    The paper does not test or state whether the 1,000 contracts used for prompt selection overlap with SmartBugs, Code4rena, or CVE contracts; overlap would inflate performance.
  • domain assumption A single pass@1 sample per contract is representative.
    Section 4.3 uses one attempt per contract, but LLM outputs are stochastic; repeated runs could produce materially different results, which are not reported.
  • ad hoc to paper The 13 selected CVEs are a representative and unbiased benchmark.
    Section 4.2.1 selects these CVEs 'following PropertyGPT' without specified selection criteria, so table-based comparisons may not generalize to the broader CVE population.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Plan-Execute Framework for Smart Contract Security Auditing." pith.science (2026). https://pith.science/paper/RB5AK47R

@misc{pith2026250515242,
  author       = {Pith},
  title        = {Pith review of: Adaptive Plan-Execute Framework for Smart Contract Security Auditing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RB5AK47R}},
  note         = {Machine review of arXiv:2505.15242}
}
read the original abstract

Large Language Models (LLMs) have shown great promise in code analysis and auditing; however, they still struggle with hallucinations and limited context-aware reasoning. We introduce SmartAuditFlow, a novel Plan-Execute framework that enhances smart contract security analysis through dynamic audit planning and structured execution. Unlike conventional LLM-based auditing approaches that follow fixed workflows and predefined steps, SmartAuditFlow dynamically generates and refines audit plans based on the unique characteristics of each smart contract. It continuously adjusts its auditing strategy in response to intermediate LLM outputs and newly detected vulnerabilities, ensuring a more adaptive and precise security assessment. The framework then executes these plans step by step, applying a structured reasoning process to enhance vulnerability detection accuracy while minimizing hallucinations and false positives. To further improve audit precision, SmartAuditFlow integrates iterative prompt optimization and external knowledge sources, such as static analysis tools and Retrieval-Augmented Generation (RAG). This ensures audit decisions are contextually informed and backed by real-world security knowledge, producing comprehensive security reports. Extensive evaluations across multiple benchmarks demonstrate that SmartAuditFlow outperforms existing methods, achieving 100 percent accuracy on common and critical vulnerabilities, 41.2 percent accuracy for comprehensive coverage of known smart contract weaknesses in real-world projects, and successfully identifying all 13 tested CVEs. These results highlight SmartAuditFlow's scalability, cost-effectiveness, and superior adaptability over traditional static analysis tools and contemporary LLM-based approaches, establishing it as a robust solution for automated smart contract auditing.

Figures

Figures reproduced from arXiv: 2505.15242 by the authors.

Figure 1
Figure 1. Overview of the SmartAuditFlow system. Initially, the LLM leverages prompt PA1 with c and Dctx to generate an initial contextual understanding (s1). Subse￾quently, this understanding s1 informs the LLM, via prompt PA2 and code c, in creating a prioritized audit plan (s2), which is then decomposed into specific sub-tasks (t). Each sub-task t j then undergoes a multi-faceted execution and validation process; here, the… view at source ↗
Figure 2
Figure 2. (a) Basic Calibration directly uses LLM to validate the initial review finding; (b) RAG-Enhanced Calibration [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Evaluation of Smart Contract Vulnerability Detection Tools - A Comparative Analysis [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Trade-off Between Accuracy (top-max) and Efficiency (Average Number of Predictions) [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Trade-off Between Accuracy (top-max) and Efficiency (Average Number of Predictions) [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 19 canonical work pages

  1. [1]

    Demystifying exploitable bugs in smart contracts

    Zhuo Zhang, Brian Zhang, Wen Xu, and Zhiqiang Lin. Demystifying exploitable bugs in smart contracts. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 615--627. IEEE, 2023

    work page 2023

  2. [2]

    Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1--13, 2024 a

    work page 2024

  3. [3]

    Propertygpt: Llm-driven formal verification of smart contracts through retrieval-augmented property generation

    Ye Liu, Yue Xue, Daoyuan Wu, Yuqiang Sun, Yi Li, Miaolei Shi, and Yang Liu. Propertygpt: Llm-driven formal verification of smart contracts through retrieval-augmented property generation. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025 . The Internet Society, 2025

    work page 2025

  4. [4]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36: 0 21558--21572, 2023 a

    work page 2023

  5. [5]

    Do you still need a manual smart contract audit? arXiv preprint arXiv:2306.12338, 2023

    Isaac David, Liyi Zhou, Kaihua Qin, Dawn Song, Lorenzo Cavallaro, and Arthur Gervais. Do you still need a manual smart contract audit? arXiv preprint arXiv:2306.12338, 2023

    arXiv 2023

  6. [6]

    Survey on quality assurance of smart contracts

    Zhiyuan Wei, Jing Sun, Zijian Zhang, Xianhao Zhang, Xiaoxuan Yang, and Liehuang Zhu. Survey on quality assurance of smart contracts. ACM Computing Surveys, 2023

    work page 2023

  7. [7]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623, 2021

    2021

  8. [8]

    Evaluation of openai o1: Opportunities and challenges of agi

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486, 2024

    arXiv 2024

Show all 46 references
  1. [9]

    Deliberative alignment: Reasoning enables safer language models

    Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Heylar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339, 2024

    2024 arXiv

  2. [10]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36: 0 38154--38180, 2023

    2023

  3. [11]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18 0 (6): 0 186345, 2024

    2024

  4. [12]

    A survey of smart contract formal specification and verification

    Palina Tolmach, Yi Li, Shang-Wei Lin, Yang Liu, and Zengxiang Li. A survey of smart contract formal specification and verification. ACM Computing Surveys (CSUR), 54 0 (7): 0 1--38, 2021

    2021

  5. [13]

    Combining fine-tuning and llm-based agents for intuitive smart contract auditing with justifications

    Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, and Yang Liu. Combining fine-tuning and llm-based agents for intuitive smart contract auditing with justifications. arXiv preprint arXiv:2403.16073, 2024

    2024 arXiv

  6. [14]

    Auditgpt: Auditing smart contracts with chatgpt

    Shihao Xia, Shuai Shao, Mengting He, Tingting Yu, Linhai Song, and Yiying Zhang. Auditgpt: Auditing smart contracts with chatgpt. arXiv preprint arXiv:2404.04306, 2024

    2024 arXiv

  7. [15]

    Stefanos Chaliasos, Marcos Antonios Charalambous, Liyi Zhou, Rafaila Galanopoulou, Arthur Gervais, Dimitris Mitropoulos, and Benjamin Livshits. Smart contract and defi security tools: Do they meet the needs of practitioners? In Proceedings of the 46th IEEE/ACM International Co...

    2024

  8. [16]

    Empirical review of automated analysis tools on 47,587 ethereum smart contracts

    Thomas Durieux, Jo \ a o F Ferreira, Rui Abreu, and Pedro Cruz. Empirical review of automated analysis tools on 47,587 ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International conference on software engineering, pages 530--541, 2020

    2020

  9. [17]

    When chatgpt meets smart contract vulnerability detection: How far are we? arXiv preprint arXiv:2309.05520, 2023 a

    Chong Chen, Jianzhong Su, Jiachi Chen, Yanlin Wang, Tingting Bi, Jianxing Yu, Yanli Wang, Xingwei Lin, Ting Chen, and Zibin Zheng. When chatgpt meets smart contract vulnerability detection: How far are we? arXiv preprint arXiv:2309.05520, 2023 a

    2023 arXiv

  10. [18]

    Large language models and the perils of their hallucinations

    Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler. Large language models and the perils of their hallucinations. Critical Care, 27 0 (1): 0 120, 2023

    2023

  11. [19]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, pages...

    2023

  12. [20]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  13. [21]

    Camel: Communicative agents for" mind" exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36: 0 51991--52008, 2023

    2023

  14. [22]

    Chain-of-thought reasoning without prompting

    Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024

    2024 arXiv

  15. [23]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  16. [24]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Confe...

    2024

  17. [25]

    Gptswarm: Language agents as optimizable graphs

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J \"u rgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning

  18. [26]

    Data interpreter: An llm agent for data science

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024

    2024 arXiv

  19. [27]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024

    2024 arXiv

  20. [28]

    Large language models are better reasoners with self-verification

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561, 2022

    2022 arXiv

  21. [29]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022

    2022 arXiv

  22. [30]

    Black-box prompt optimization: Aligning large language models without model training

    Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155, 2023

    2023 arXiv

  23. [31]

    Efficient mini-batch training for stochastic optimization

    Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 661--670, 2014

    2014

  24. [32]

    Mini-batch optimization enables training of ode models on large-scale datasets

    Paul Stapor, Leonard Schmiester, Christoph Wierling, Simon Merkt, Dilan Pathirana, Bodo MH Lange, Daniel Weindl, and Jan Hasenauer. Mini-batch optimization enables training of ode models on large-scale datasets. Nature Communications, 13 0 (1): 0 34, 2022

    2022

  25. [33]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  26. [34]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--...

    2024

  27. [35]

    Semantic-enriched code knowledge graph to reveal unknowns in smart contract code reuse

    Qing Huang, Dianshu Liao, Zhenchang Xing, Zhengkang Zuo, Changjing Wang, and Xin Xia. Semantic-enriched code knowledge graph to reveal unknowns in smart contract code reuse. ACM Transactions on Software Engineering and Methodology, 32 0 (6): 0 1--37, 2023

    2023

  28. [36]

    Storage state analysis and extraction of ethereum blockchain smart contracts

    Maha Ayub, Tania Saleem, Muhammad Janjua, and Talha Ahmad. Storage state analysis and extraction of ethereum blockchain smart contracts. ACM Transactions on Software Engineering and Methodology, 32 0 (3): 0 1--32, 2023

    2023

  29. [37]

    Exploring \ ChatGPT's \ capabilities on vulnerability management

    Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. Exploring \ ChatGPT's \ capabilities on vulnerability management. In 33rd USENIX Security Symposium (USENIX Security 24), pages 811--828, 2024

    2024

  30. [38]

    Multitask-based evaluation of open-source llm on software vulnerability

    Xin Yin, Chao Ni, and Shaohua Wang. Multitask-based evaluation of open-source llm on software vulnerability. IEEE Transactions on Software Engineering, 2024

    2024

  31. [39]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  32. [40]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  33. [41]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023

    2023 arXiv

  34. [42]

    G-eval: Nlg evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023 b

    2023 arXiv

  35. [43]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    2024 arXiv

  36. [44]

    Understanding the effectiveness of large language models in detecting security vulnerabilities

    Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. Understanding the effectiveness of large language models in detecting security vulnerabilities. arXiv preprint arXiv:2311.16169, 2023

    2023 arXiv

  37. [45]

    Llm-assisted static analysis for detecting security vulnerabilities

    Ziyang Li, Saikat Dutta, and Mayur Naik. Llm-assisted static analysis for detecting security vulnerabilities. arXiv preprint arXiv:2405.17238, 2024

    2024 arXiv

  38. [46]

    Llm4vuln: A unified evaluation framework for decoupling and enhancing llms' vulnerability reasoning

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. Llm4vuln: A unified evaluation framework for decoupling and enhancing llms' vulnerability reasoning. arXiv preprint arXiv:2401.16185, 2024 b

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.