REVIEW 4 cited by
When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the development of blockchain technology, smart contracts have become an important component of blockchain applications. Despite their crucial role, the development of smart contracts may introduce vulnerabilities and potentially lead to severe consequences, such as financial losses. Meanwhile, large language models, represented by ChatGPT, have gained great attentions, showcasing great capabilities in code analysis tasks. In this paper, we presented an empirical study to investigate the performance of ChatGPT in identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT's effectiveness using a publicly available smart contract dataset. Our findings discover that while ChatGPT achieves a high recall rate, its precision in pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT's performance varies when detecting different vulnerability types. We delved into the root causes for the false positives generated by ChatGPT, and categorized them into four groups. Second, by comparing ChatGPT with other state-of-the-art smart contract vulnerability detection tools, we found that ChatGPT's F-score is lower than others for 3 out of the 7 vulnerabilities. In the case of the remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these tools. Finally, we analyzed the limitation of ChatGPT in smart contract vulnerability detection, revealing that the robustness of ChatGPT in this field needs to be improved from two aspects: its uncertainty in answering questions; and the limited length of the detected code. In general, our research provides insights into the strengths and weaknesses of employing large language models, specifically ChatGPT, for the detection of smart contract vulnerabilities.
Forward citations
Cited by 4 Pith papers
-
SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detection
SAEL combines LLM predictions, LLM-generated explanations, and code features via an adaptive mixture-of-experts gate and reports state-of-the-art F1 on four smart contract vulnerability benchmarks.
-
Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection
A LLaMA-3.1-8B model trained with continual pre-training, supervised fine-tuning, and direct preference optimization reports state-of-the-art accuracy and F1 for smart contract vulnerability detection and explanation.
-
Adaptive Plan-Execute Framework for Smart Contract Security Auditing
SmartAuditFlow claims 100% detection on a standard smart contract benchmark and all 13 tested CVEs via a plan-execute LLM workflow, though the supporting evaluation has major reproducibility and validation gaps.
-
Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques
A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.
Discussion (0). Sign in to comment.