Ai-Driven Vulnerability Analysis in Smart Contracts: Trends, Challenges and Future Directions

Mesut Ozdag

REVIEW 4 major objections 7 minor 20 references

Ai-Driven Vulnerability Analysis in Smart Contracts: Trends, Challenges and Future Directions

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-driven tools now lead smart contract vulnerability detection, survey argues

desk verdict A useful but flawed survey; the unsourced comparison table undercuts the quantitative claims and needs fixing before this is citable. read the letter →

arxiv 2506.06735 v1 pith:RTYVX4KI submitted 2025-06-07 cs.CR cs.AI

Mesut Ozdag This is my paper

classification cs.CRcs.AI

keywords smartcontractsvulnerabilitydetectionartificialintelligencemachinelearningdeepgraphneuralnetworkstransformersEthereum

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey paper argues that AI-driven techniques have reshaped smart contract vulnerability detection, moving the field beyond manual audits and formal verification. The paper organizes the methods into machine learning, deep learning, graph neural network, transformer, and hybrid categories, and it claims that transformer-based models such as SmartBERT and SolTrans reach F1-scores above 0.90, outperforming earlier CNN-based approaches. The intended contribution is a comparative map that helps developers and auditors choose among methods by accuracy, interpretability, computational cost, and real-time suitability. A sympathetic reading is that the paper aims to establish AI as the most viable path to securing smart contracts at scale, while acknowledging unresolved issues in explainability and dataset quality.

What carries the argument

The organizing device is a five-family taxonomy of AI methods, each tied to a distinct code representation: random forests over execution traces (dynamic ML), CNNs over token embeddings (deep learning), GNNs over control-flow graphs and heterogeneous transaction graphs, transformers over tokenized Solidity source or bytecode, and hybrids that combine these representations. The paper maps each family to the vulnerability classes it handles best and uses a single comparison table (Table 1) with precision, recall, F1, accuracy, and inference time to rank the approaches. That table, rather than any new experiment, is the mechanism that carries the paper's performance claims.

What would settle it

Locate the original papers for SmartBERT, SolTrans, ContractGraph, and Sereum and re-run their detectors on a common benchmark such as SmartBugs; if the true F1-scores fall below 0.90 for the transformer models, or if the relative ordering in Table 1 changes materially, the paper's central performance claims are not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that AI-based vulnerability detection has advanced from lightweight anomaly detectors to context-aware models that can match or exceed traditional static analysis. Its central assertion is that transformer-based models, fine-tuned or pre-trained on Solidity code, achieve the best detection performance, with F1-scores above 0.90, while GNN-based models offer structural interpretability and hybrid ensembles balance accuracy with efficiency. The paper also claims that these methods complement rather than replace traditional auditing, and that the remaining obstacles are explainability, dataset imbalance, generalization across languages and chains, and deployment cost.

Load-bearing premise

The paper's ranking and recommendations rest on the accuracy of Table 1, which lists precision, recall, F1, accuracy, and inference times for methods like Sereum and SmartBERT without citing a source, benchmark protocol, or error bars; if those numbers are unreliable or not comparable, the comparative conclusions lose their evidence base.

Editorial extensions

If this is right

If transformer-based models really reach F1-scores above 0.90 on standard benchmarks, auditors could use them as a first-pass screening layer before manual review.
If GNN methods trace vulnerable paths through control-flow graphs, their visual outputs could serve as audit evidence for specific exploit scenarios.
If lightweight models such as Sereum detect reentrancy in real time, they could be embedded into development pipelines or monitoring tools wherever transformer latency is prohibitive.
If hybrid ensembles combine multiple code representations effectively, they would be the most resilient option for obfuscated or adversarially crafted contracts.
If the listed challenges get resolved, AI-based detection could extend from Ethereum and Solidity to other chains and languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not show that the methods in Table 1 were evaluated on the same data or under the same protocol; a shared benchmark with unified metrics would be needed before treating the F1 ordering as a real ranking.
The reported F1-scores above 0.90 likely reflect binary classification on curated vulnerability datasets, not the harder open-world task of finding novel vulnerabilities in unlabeled production contracts.
Attention-weight visualizations, which the paper cites as explainability, may only highlight tokens correlated with vulnerable code rather than explain why a contract is unsafe, so using them as audit evidence would require validation.
The case studies imply a testable extension: feed the DAO and Parity contract code to current detectors and measure whether they flag the exact vulnerable functions before any patch is applied.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: this is a survey, so don't expect new results. What it does well is organize the field into ML, DL, GNN, transformer, and hybrid families, and it gives decent, accessible explanations of how each works—the equations for CNN, LSTM, attention, and GNN message passing are standard and correctly presented. The case studies on the DAO and Parity are accurate, and the dataset overview is a reasonable starting point. If you're new to the area, you'd walk away with a fair map of the landscape.

The problem is the quantitative layer, specifically Table 1. It reports precision, recall, F1, accuracy, and inference time for seven methods, but there are no citations, no dataset or split information, no error bars, and no indication of how the numbers were gathered. The text even contradicts the table: Section 3.3 says transformer models increase inference times up to 3–4 seconds per contract, while SmartBERT is listed at 110 ms. And the prose says SolTrans achieves F1 > 0.90, but there is no SolTrans row in the table. The conclusions and the future-work recommendations lean on this table, so this is not a minor omission—it's the evidentiary base for the paper's central comparative claims, and it's currently unsupported.

The qualitative sections can stand. The taxonomy is standard but coherent, and the challenges section (XAI, benchmarks, cross-chain) is accurate. The survey would be genuinely useful if the author either replaced Table 1 with sourced numbers from the original papers or removed it entirely and stuck to qualitative comparisons. As it stands, I wouldn't cite the numbers.

Who's this for? Practitioners and grad students wanting a quick orientation. It doesn't open any research question, but it's a fine entry point if they understand the table's limits.

Should it get a serious referee? Yes—the topic matters and the survey is fixable. I'd send it to review with a clear instruction that the table must be sourced or cut. After that, it's a passable survey.

Referee Report

4 major / 7 minor

Summary. The paper is a survey of AI-based vulnerability detection for smart contracts. It organizes methods into machine learning, deep learning, graph neural network, transformer, and hybrid families in Section 2, presents a quantitative comparison in Table 1, discusses three case studies in Section 4, and describes three datasets in Section 5. It concludes in Section 6 that AI-based techniques have 'significantly transformed' smart-contract vulnerability detection and that AI-driven detection offers 'the best hope' for securing contracts at scale, with future research directions listed in Section 7. The central quantitative assertions are that transformer-based models such as SmartBERT and SolTrans reach F1-scores above 0.90 and that Table 1 provides a reliable performance and efficiency ranking.

Significance. If the comparative claims held, the paper would be a useful entry point for researchers and practitioners, particularly because it covers a broad set of method families, includes worked case studies of the DAO and Parity incidents, and identifies relevant open problems such as multimodal learning, cross-chain detection, lightweight real-time detection, explainable AI, continual learning, and standardized benchmarks. The qualitative taxonomy and the collection of model architectures, datasets, and challenges are the stronger parts of the contribution. However, the quantitative layer is not currently reproducible: Table 1 has no per-row sources, no evaluation protocol, and no error information, and the paper does not state how studies were selected for inclusion. Credit is due for the breadth of coverage and for making the survey's scope explicit through named model families, but the numerical evaluation needs substantial revision before the paper's central claims can be accepted.

major comments (4)

[3.1, Table 1] Table 1 is the sole quantitative support for the paper's central claims, but it provides no source for any of its seven rows: no citation to the original study, no dataset name per row, no class-level breakdown, and no error bars or variance information. The prose in Section 3.1 and the conclusions in Section 6 then use these numbers to assert that transformer-based models reach F1-scores above 0.90 and that AI has significantly transformed the field. As a result, the ranking and recommendations are unfalsifiable. Please add per-row citations and a precise evaluation protocol (datasets, splits, vulnerability classes, hardware), or clearly re-label the table as a subjective synthesis with explicit caveats.
[3.3 vs. Table 1] Section 3.3 states that transformer models 'increase inference times up to 3–4 seconds per contract,' while the SmartBERT row of Table 1 reports an inference time of 110 ms. These two statements cannot both be true for the same evaluation setting. Please reconcile the prose with the table by specifying the workload, hardware, and preprocessing steps, or correct one of the two values.
[3.1] SolTrans is named in Section 3.1 as one of the two transformer models achieving F1-score > 0.90, yet no SolTrans row appears in Table 1. Because this F1 claim is part of the paper's headline quantitative result, the table needs either a SolTrans row with its source or a restatement of the claim that does not assert a specific score that is not displayed.
[2–3 (survey methodology)] The survey never describes its literature search, inclusion and exclusion criteria, quality assessment, or data-extraction procedure. For a comparison-based survey, that methodology is load-bearing: the reader cannot judge whether the seven methods in Table 1 are a representative sample or a selective subset, nor whether the cited F1 scores were measured under comparable conditions. Please add a methodology subsection or an explicit scope statement, and make the selection criteria for the table reproducible.

minor comments (7)

[Abstract] The abstract contains a typo: 'Vul nerabilities' should be 'Vulnerabilities'.
[1] In Section 1, the text cites Oyente as [17] and Mythril as [18], but reference [17] is SmartCheck and reference [18] is Osiris; these citations need to be corrected.
[2.2.3, 2.4] Equations (4), (5), (7), and (8) are referenced in the text but the displayed formulas are missing in the manuscript; the full formulas and their notation should be restored.
[4, Table 2] Table 2 lists 'Secure Templates (2019+)' as a case-study incident, but it is not an incident comparable to the DAO or Parity events; consider renaming the column or moving this row to a separate application example.
[3.3, 5] The paper uses 'EtherScan' throughout; the correct product name is 'Etherscan'.
[5, Table 3] Table 3, like Table 1, gives dataset sizes (e.g., '>1M contracts') without a source; please add citations for these numbers or state that they are approximate orders of magnitude.
[References] Reference [5] attributes Sereum to 'Mueller, B.', but the Sereum paper is by Rodler et al.; please check all references for author and venue accuracy, since several entries (e.g., [11], [12], [16]) appear inconsistent with the published literature.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found: the survey makes no derivation and fits no parameters, so its unsupported performance numbers are a rigor concern rather than a circularity concern.

full rationale

This is a literature survey with no derivation chain, no fitted parameters, and no predictive claim generated from an input. Section 2 restates standard model equations (embedding lookup, convolution, LSTM recurrence, GNN aggregation, self-attention) as descriptions of existing methods, but none of these equations is used to derive a result, and no quantity in Table 1 is computed from them. Section 3.1's claim that SmartBERT and SolTrans exceed 0.90 F1 is presented as a report of prior studies, and Table 1 lists seven method/performance tuples, but the table has no source column, no dataset/split definition, and no citations to individual rows; additionally, the prose in Section 3.3 says transformer models 'increase inference times up to 3-4 seconds per contract,' which conflicts with the SmartBERT row's 110 ms, and SolTrans, which the text names, has no row. These are serious evidence and internal-consistency defects that should be treated as correctness risk, but they are not circularity: the paper does not fit the numbers to its own model, define a concept in terms of the conclusion, or rely on a self-citation as the load-bearing justification. There are no author self-citations at all, and no uniqueness theorem or ansatz is imported from the authors' prior work. The paper's central 'significantly transformed' conclusion in Section 6 is an interpretive opinion about the surveyed literature, not a result derived from the paper's own inputs, so it cannot reduce to itself by construction.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new parameters or entities. Its conclusions depend on the representativeness and accuracy of the surveyed literature, especially the unsourced Table 1.

assumptions (2)

domain assumption The surveyed models are representative of the state of the art.
The paper selects a handful of models per family without a systematic search criteria, yet uses them to draw general conclusions about ML, DL, GNN, and transformer approaches.
domain assumption The performance figures in Table 1 accurately reflect the cited models.
No source or benchmark is given for the table, so the survey assumes these numbers are valid for its comparisons.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ai-Driven Vulnerability Analysis in Smart Contracts: Trends, Challenges and Future Directions." pith.science (2026). https://pith.science/paper/RTYVX4KI

@misc{pith2026250606735,
  author       = {Pith},
  title        = {Pith review of: Ai-Driven Vulnerability Analysis in Smart Contracts: Trends, Challenges and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RTYVX4KI}},
  note         = {Machine review of arXiv:2506.06735}
}

read the original abstract

Smart contracts, integral to blockchain ecosystems, enable decentralized applications to execute predefined operations without intermediaries. Their ability to enforce trustless interactions has made them a core component of platforms such as Ethereum. Vulnerabilities such as numerical overflows, reentrancy attacks, and improper access permissions have led to the loss of millions of dollars throughout the blockchain and smart contract sector. Traditional smart contract auditing techniques such as manual code reviews and formal verification face limitations in scalability, automation, and adaptability to evolving development patterns. As a result, AI-based solutions have emerged as a promising alternative, offering the ability to learn complex patterns, detect subtle flaws, and provide scalable security assurances. This paper examines novel AI-driven techniques for vulnerability detection in smart contracts, focusing on machine learning, deep learning, graph neural networks, and transformer-based models. This paper analyzes how each technique represents code, processes semantic information, and responds to real world vulnerability classes. We also compare their strengths and weaknesses in terms of accuracy, interpretability, computational overhead, and real time applicability. Lastly, it highlights open challenges and future opportunities for advancing this domain.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

A Survey of Attacks on Ethereum Smart Contracts,

Atzei, N., Bartoletti, M., &Cimoli, T., "A Survey of Attacks on Ethereum Smart Contracts," Proceedings of International Conference on Principles of Security and Trust, 2017

work page 2017
[2]

Formal V erification of Smart Contracts,

Bhargavan, K., et al., "Formal V erification of Smart Contracts," ACM CCS, 2016

work page 2016
[3]

Making Smart Contracts Smarter,

Luu, L., et al., "Making Smart Contracts Smarter," Proceedings of ACM CCS, 2016

work page 2016
[4]

Blockchains and Smart Contracts for the Internet of Things,

Christidis, K., Devetsikiotis, M., "Blockchains and Smart Contracts for the Internet of Things," IEEE Access, 2016

work page 2016
[5]

The distinguishing number and distinguishing chromatic number for posets

Mueller, B., "Sereum: Protecting Smart Contracts from Re -Entrancy Attacks," arXiv preprint arXiv:1905.09858, 2019

work page Pith review arXiv 1905
[6]

MadMax: Surviving Out -of-Gas Conditions in Ethereum Smart Contracts,

Brent, L., Jurisevic, A., Kong, M., Liu, L., and Scholz, B., "MadMax: Surviving Out -of-Gas Conditions in Ethereum Smart Contracts," Proceedings of ACM OOPSLA, 2018

work page 2018
[7]

Drug repurposing prediction for COVID-19 using probabilistic networks and crowdsourced curation

Chen, T., et al., "SmartEmbed: A Deep Learning Framework for Smart Contract Vulnerability Detection," arXiv preprint arXiv:2005.11088, 2020

work page Pith review arXiv 2005
[8]

Hydrogenous content identification in heterogeneous cargoes via multiple monoenergetic neutron radiography

Zhou, Y ., Zhang, Z., and Jin, H., "Learning to Detect Vulnerabilities with Graph Neural Networks," arXiv preprint arXiv:1909.06920, 2019

work page Pith review arXiv 1909

Show all 20 references

[9]

VulnSniffer: Detecting Vulnerabilities in Smart Contracts Through Sequence Modeling,

Sun, X., et al., "VulnSniffer: Detecting Vulnerabilities in Smart Contracts Through Sequence Modeling," Journal of Systems Architecture, 2020

2020
[10]

ContractGraph: GNN -Based Vulnerability Detection in Ethereum Smart Contracts,

Zhang, Y ., Xu, H., "ContractGraph: GNN -Based Vulnerability Detection in Ethereum Smart Contracts," arXiv preprint arXiv:2103.15120, 2021

2021 arXiv
[11]

ETH2V ec: Heterogeneous Graph Embedding for Smart Contracts,

Li, F., Zhu, H., and Ren, Z., "ETH2V ec: Heterogeneous Graph Embedding for Smart Contracts," IEEE Transactions on Dependable and Secure Computing, 2022

2022
[12]

SolGraph: Graph Attention Networks for Smart Contract Vulnerability Detection,

Huang, Z., et al., "SolGraph: Graph Attention Networks for Smart Contract Vulnerability Detection," Proceedings of AAAI Conference on Artificial Intelligence, 2022

2022
[13]

SmartBERT: A Pre -trained Model for Sma rt Contract Vulnerability Detection,

Wang, L., et al., "SmartBERT: A Pre -trained Model for Sma rt Contract Vulnerability Detection," arXiv preprint arXiv:2201.12877, 2022

2022 arXiv
[14]

SolTrans: Transformer-Based Smart Contract Vulnerability Detection,

Xu, X., Jiang, J., Wang, F., "SolTrans: Transformer-Based Smart Contract Vulnerability Detection," Proceedings of IEEE BigData Conference, 2023

2023
[15]

DeepSolid: Deep Learning and Graph Neural Networks for Smart Contracts,

Peng, C., et al., "DeepSolid: Deep Learning and Graph Neural Networks for Smart Contracts," Proceedings of IEEE Blockchain, 2021

2021
[16]

HybridVulDetect: Ensemble Learning for Vulnerability Detection in Smart Contracts,

Rao, S., et al., "HybridVulDetect: Ensemble Learning for Vulnerability Detection in Smart Contracts," IEEE Transactions on Software Engineering, 2023

2023
[17]

SmartCheck: Static Analysis of Ethereum Smart Contracts,

Tikhomirov, S., et al., "SmartCheck: Static Analysis of Ethereum Smart Contracts," Proceedings of ACM W orkshop on Blockchain, Cryptocurrencies and Contracts, 2018

2018
[18]

Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts,

Torres, C., Steichen, M., and State, R., "Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts," Proceedings of ACM AsiaCCS, 2018

2018
[19]

Semantic -Based Smart Contract Vulnerability Detection with Machine Learning,

Liao, X., et al., "Semantic -Based Smart Contract Vulnerability Detection with Machine Learning," Proceedings of IEEE COMPSAC, 2020

2020
[20]

Scalable Detection of Vulnerabilities in Smart Contracts Using Machine Learning,

Feng, Q., et al., "Scalable Detection of Vulnerabilities in Smart Contracts Using Machine Learning," IEEE Transactions on Information Forensics and Security, 2021. International Journal of Artificial Intelligence and Applications (IJAIA), V ol.16, No.3, May 2025 82 AUTHOR Dr. ...

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A Survey of Attacks on Ethereum Smart Contracts,

Atzei, N., Bartoletti, M., &Cimoli, T., "A Survey of Attacks on Ethereum Smart Contracts," Proceedings of International Conference on Principles of Security and Trust, 2017

work page 2017

[2] [2]

Formal V erification of Smart Contracts,

Bhargavan, K., et al., "Formal V erification of Smart Contracts," ACM CCS, 2016

work page 2016

[3] [3]

Making Smart Contracts Smarter,

Luu, L., et al., "Making Smart Contracts Smarter," Proceedings of ACM CCS, 2016

work page 2016

[4] [4]

Blockchains and Smart Contracts for the Internet of Things,

Christidis, K., Devetsikiotis, M., "Blockchains and Smart Contracts for the Internet of Things," IEEE Access, 2016

work page 2016

[5] [5]

The distinguishing number and distinguishing chromatic number for posets

Mueller, B., "Sereum: Protecting Smart Contracts from Re -Entrancy Attacks," arXiv preprint arXiv:1905.09858, 2019

work page Pith review arXiv 1905

[6] [6]

MadMax: Surviving Out -of-Gas Conditions in Ethereum Smart Contracts,

Brent, L., Jurisevic, A., Kong, M., Liu, L., and Scholz, B., "MadMax: Surviving Out -of-Gas Conditions in Ethereum Smart Contracts," Proceedings of ACM OOPSLA, 2018

work page 2018

[7] [7]

Drug repurposing prediction for COVID-19 using probabilistic networks and crowdsourced curation

Chen, T., et al., "SmartEmbed: A Deep Learning Framework for Smart Contract Vulnerability Detection," arXiv preprint arXiv:2005.11088, 2020

work page Pith review arXiv 2005

[8] [8]

Hydrogenous content identification in heterogeneous cargoes via multiple monoenergetic neutron radiography

Zhou, Y ., Zhang, Z., and Jin, H., "Learning to Detect Vulnerabilities with Graph Neural Networks," arXiv preprint arXiv:1909.06920, 2019

work page Pith review arXiv 1909