Pith. sign in

REVIEW 3 cited by

When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.05520 v4 pith:TPMGEJBO submitted 2023-09-11 cs.SE

Chong Chen , Jianzhong Su , Jiachi Chen , Yanlin Wang , Tingting Bi , Jianxing Yu , Yanli Wang , Xingwei Lin
show 2 more authors
Ting Chen Zibin Zheng
This is my paper
classification cs.SE
keywords chatgptsmartcontractvulnerabilitiesdetectionvulnerabilityblockchaincode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the development of blockchain technology, smart contracts have become an important component of blockchain applications. Despite their crucial role, the development of smart contracts may introduce vulnerabilities and potentially lead to severe consequences, such as financial losses. Meanwhile, large language models, represented by ChatGPT, have gained great attentions, showcasing great capabilities in code analysis tasks. In this paper, we presented an empirical study to investigate the performance of ChatGPT in identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT's effectiveness using a publicly available smart contract dataset. Our findings discover that while ChatGPT achieves a high recall rate, its precision in pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT's performance varies when detecting different vulnerability types. We delved into the root causes for the false positives generated by ChatGPT, and categorized them into four groups. Second, by comparing ChatGPT with other state-of-the-art smart contract vulnerability detection tools, we found that ChatGPT's F-score is lower than others for 3 out of the 7 vulnerabilities. In the case of the remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these tools. Finally, we analyzed the limitation of ChatGPT in smart contract vulnerability detection, revealing that the robustness of ChatGPT in this field needs to be improved from two aspects: its uncertainty in answering questions; and the limited length of the detected code. In general, our research provides insights into the strengths and weaknesses of employing large language models, specifically ChatGPT, for the detection of smart contract vulnerabilities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detection

    cs.CR 2025-07 conditional novelty 5.0 of 10

    SAEL combines LLM predictions, LLM-generated explanations, and code features via an adaptive mixture-of-experts gate and reports state-of-the-art F1 on four smart contract vulnerability benchmarks.

  2. Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection

    cs.CR 2025-06 conditional novelty 5.0 of 10

    A LLaMA-3.1-8B model trained with continual pre-training, supervised fine-tuning, and direct preference optimization reports state-of-the-art accuracy and F1 for smart contract vulnerability detection and explanation.

  3. Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques

    cs.CR 2025-07 conditional novelty 4.0 of 10

    A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.

Pith tools