{"id":"34e00ed0-65da-4561-b339-4a66ce50403f","arxiv_id":"2501.02229","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A comparison of LSTM, DistilBERT, and BERT for smart contract vulnerability classification claims high accuracy for fine-tuned LLMs, but the abstract's >90% figure conflicts with the 87-93% numbers in the results.","lead":"This paper compares LSTM, DistilBERT, and BERT models for classifying smart contract code into four vulnerability types using a public dataset of 2,217 annotated contracts. It reports that a fine-tuned LLM reaches around 90% accuracy, a result relevant to choosing automated security tools for blockchain applications.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own reported results contradict the 'over 90%' claim; Section V lists LSTM and BERT at 89% and DistilBERT at 87%, so the headline accuracy is unsupported even before considering the iterative-testing leakage in Section IV-C.","rationale":"The reader's verdict is REJECT, and I agree that rejection is correct, but for a slightly different reason. The reader's weakest assumption is test-set leakage from 'iterative testing' in Section IV-C. That is a valid and serious concern. However, the more immediate load-bearing problem is that the paper's own numbers do not support the headline: the prose reports no accuracy above 90%, and the one 0.93 figure in Table I is ambiguous. Even a perfectly clean experiment described as achieving 87-89% would not support an abstract claiming 'over 90%' and 'surpasses the accuracy of any other model.' The internal inconsistency is objective and does not depend on assumptions about author intent or hidden tuning. I also note the appearance of 'DeBERTa' in Section V without introduction, which suggests the results text was assembled from multiple sources and not reconciled. The combination of (a) missing or nonexistent >90% result, (b) contradictory statements about which model was best, and (c) no code, data, or validation-set description means the central claim is not verifiable. A concrete reproduction with a pre-registered held-out test set is the decisive check; until then the paper should not be cited as demonstrating LLM superiority in smart contract vulnerability detection.","tokens_in":9714,"tokens_out":3889,"duration_ms":35069,"concrete_test":"Obtain the authors' code/data (or re-run on the Liu et al. dataset with the same preprocessing) and reproduce the test-set evaluation with a fixed random seed and an 80/10/10 train/validation/test split, selecting hyperparameters only on the validation fold. Then directly compare the reproduced test accuracy for LSTM, DistilBERT, and BERT against Table I and Section V: if no model reaches >90% on the held-out test set, the abstract's claim is false; if DistilBERT does reach >90%, Section V's 87% figure must be corrected and the leakage concern from the iterative-testing procedure must be addressed by showing the test set was untouched during tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is that a fine-tuned LLM actually achieves >90% accuracy on an unbiased test set. The paper does not establish this. Section V reports LSTM accuracy 89%, DistilBERT 87%, and BERT 89%, and then says 'The best overall accuracy was obtained by DistilBERT' while also stating LSTM and BERT each at 89%—no model exceeds 90% in the prose. The only >90% value is the ambiguous 0.93 in Table I, which is not clearly attributed; the table layout mixes class-wise metrics with model-level accuracy rows. If 0.93 is DistilBERT's test accuracy, the prose is wrong; if it is a per-class or macro value, the abstract's 'over 90%' has no supporting number. Section IV-C adds a second independent problem: 'feedback from the model performance helped adjust the dataset and model parameters' during 'iterative testing,' with no separate validation set described, so any reported test accuracy may reflect test-set feedback rather than generalization. The unexplained mention of 'DeBERTa' in Section V—a model never introduced earlier—reinforces that the results section is internally inconsistent. Either the numerical contradiction or the leakage risk is enough to sink the headline; together they make the central comparison unverifiable from the manuscript alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative evaluation of classical machine learning and fine-tuned language models for smart contract vulnerability detection. Using the annotated smart contract dataset of Liu et al., the authors train an LSTM, a DistilBERT model, and a BERT model to classify four vulnerability classes: Dangerous Delegatecall, Integer Overflow, Reentrancy, and Timestamp Dependency. The abstract claims that a fine-tuned LLM surpasses all other models by achieving an accuracy of over 90%, and the discussion and conclusion assert general advantages for LLMs in capturing context-rich vulnerability patterns. The body reports per-class precision, recall, F1, support, accuracy figures of 89%, 87%, and 89% for LSTM, DistilBERT, and BERT respectively, plus training curves and confusion matrices. The paper does not describe a held-out test protocol, variance estimates, or code release.","tokens_in":10050,"tokens_out":4550,"duration_ms":42217,"significance":"If the headline claim of over 90% accuracy were verified on an unbiased test set, the comparison of encoder-only transformer models with an LSTM baseline on a public smart contract dataset would be a useful empirical datapoint for the vulnerability-detection community. The paper addresses practically relevant vulnerability classes and reports class-wise metrics, confusion matrices, and training curves, which are appropriate ingredients for a comparative study. However, the central quantitative claim is internally contradicted by the Results section, the experimental protocol appears to allow test-set feedback during tuning, and no uncertainty or significance analysis is provided. As written, the reported numbers do not support the paper's main conclusion, so the contribution is not currently usable as a benchmark result.","major_comments":[{"comment":"The abstract's central claim that \"our fine-tuned LLM surpasses the accuracy of any other model by achieving an accuracy of over 90%\" is not supported by the Results section. Section V reports LSTM accuracy of 89%, DistilBERT accuracy of 87%, and BERT accuracy of 89%, and the Conclusion states \"The best overall accuracy was obtained by DistilBERT\" even though 87% is the lowest reported figure. Table I's Accuracy row contains four unlabeled numbers (0.89, 0.93, 0.89, 0.93) that do not match the prose values; if 0.93 is intended as DistilBERT's test accuracy then the prose is wrong, and if it is a per-class figure then no model exceeds 90%. This direct numerical contradiction makes the headline result unverifiable from the manuscript alone.","section":"Abstract / §V, Table I"},{"comment":"The \"Iterative testing\" paragraph states that \"feedback from the model performance helped adjust the dataset and model parameters\" during testing, yet no separate validation set or final frozen test set is described anywhere in §IV, and the split proportion, stratification, and random seeds are not reported. If the test set was consulted to adjust the dataset or hyperparameters, all reported accuracies are optimistically biased and the comparison is not an unbiased estimate of generalization. The authors must specify a proper train/validation/test protocol and state that the test set was used exactly once.","section":"§IV-C"},{"comment":"The experimental comparison lacks any variance or significance analysis. The support values in Table I are 10 for the DD class and 31 for the TD class per model, and the total test support appears to be 222 samples, but no confidence intervals, multiple seed runs, or statistical tests are reported. Given the class imbalance and small supports, differences of a few percentage points between models (87% vs. 89%) are within the range of sampling noise, so the conclusions about which model is \"best\" are not supported by the evidence presented.","section":"§V, Table I"},{"comment":"Section V introduces a \"DeBERTa\" matrix without any prior mention of DeBERTa in the methodology, the model list, or Table I. The sentence \"Another matrix (DeBERTa) has achieved improved accuracy across most categories\" is unexplained, and no DeBERTa results are tabulated. Together with the other inconsistencies, the results section does not currently describe a coherent, reproducible experiment.","section":"§V"}],"minor_comments":[{"comment":"The sentence \"several LLMs alongside various traditional machine learning algorithms such as DistilBERT model is trained and tested\" has a subject-verb agreement error and does not clearly identify which models are traditional ML algorithms.","section":"Abstract"},{"comment":"The dataset description says the data has 5 columns but then lists only four: filename, code snippet, vulnerability label, and encoded label.","section":"§IV-A"},{"comment":"Table I is difficult to parse: the Accuracy row values are not attached to model names, and the macro and weighted average rows are repeated for each class column; the table should be reformatted with one column per model.","section":"Table I"},{"comment":"The terms \"Dangerous Delegatecall,\" \"DC,\" and \"DD\" are used inconsistently; the authors should define a single acronym and use it consistently.","section":"§V and Table I"},{"comment":"The rhetorical question \"How do we trust, accept or understand security outcome that we are aware of it generates more false alarms?\" is not appropriate in a research report and should be removed or rewritten.","section":"§III"},{"comment":"Reference [26] is cited as the source of the dataset, but the linked arXiv paper is a fuzzing paper; the authors should verify that the citation and dataset provenance are correct.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript has the character of an early workshop-style report, but the central claim is directly contradicted by the reported results and the methodology admits possible test-set leakage. A revision that merely rewrites the abstract would not remedy the lack of a proper validation protocol, and the experiments would need to be redone before the contribution could be assessed. On this basis I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's own results don't support its headline accuracy, and the methodology section describes an iterative testing loop that sounds like test-set leakage. This is a straightforward LSTM/BERT/DistilBERT comparison on Liu et al.'s public dataset, with no new architecture, data, or analysis. It is not ready for publication.\n\nWhat the paper does well: it uses a recognized public benchmark dataset, reports per-class precision/recall/F1 and support, and the LSTM vs. BERT comparison is a reasonable thing to document. The figures and confusion matrices are fine as far as they go.\n\nThe soft spots are load-bearing. The abstract claims >90% accuracy for a fine-tuned LLM, but the Results section gives LSTM 89%, DistilBERT 87%, and BERT 89%. The only 0.93 in Table I is not clearly attributed to a specific model and conflicts with the prose. Section IV-C says 'feedback from the model performance helped adjust the dataset and model parameters' during 'iterative testing,' with no separate validation set described; that alone undermines any generalization claim. On top of that, a 'DeBERTa' matrix appears in Section V without being introduced anywhere, and no code, data, or error estimates are provided. These aren't cosmetic issues; they make the central comparison unverifiable from the manuscript.\n\nThe paper does not show clear thinking on its own terms—the numbers are internally inconsistent, and the leakage-prone setup is described without acknowledgment. A serious editor could send this to review, but I'd desk reject: the authors need to redo the experiments with a fixed train/validation/test split and release code and results before any of these claims can be evaluated. The topic is timely, but this version doesn't add anything beyond what refs [8] and [24] already cover. My advice: skip it, and don't cite it.","headline":"Routine benchmark comparison undermined by internal numerical contradictions and a leakage-prone iterative testing procedure; the >90% claim is unsupported.","tokens_in":10585,"tokens_out":3250,"would_cite":false,"duration_ms":31380,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning a large language model on annotated smart contract source code yields over 90% accuracy in detecting vulnerabilities, surpassing a classical LSTM baseline.","keywords":["smart contract","large language model","machine learning","vulnerability detection","fine-tuning","Ethereum","DistilBERT","LSTM"],"falsifier":"Run LSTM, DistilBERT, and BERT on the same dataset with a fixed 80/10/10 split, a validation set for early stopping, and a test set touched exactly once, then compare accuracies; if the fine-tuned LLM does not exceed the LSTM's accuracy, or if no model reaches over 90%, the central claim fails.","tokens_in":9548,"feed_emoji":"🛡️","tokens_out":8541,"duration_ms":71500,"temperature":0.7,"pith_summary":"The paper sets out to show that fine-tuned large language models can classify smart contract vulnerabilities more accurately than classical machine learning. It trains an LSTM, a DistilBERT model, and a BERT model on an annotated dataset of 2,217 Solidity contracts covering Reentrancy, Integer Overflow, Timestamp Dependency, and Dangerous Delegatecall. The authors report that the fine-tuned LLM exceeds 90% accuracy and beats the other models, arguing that the transformer's bidirectional attention captures subtle, context-dependent code patterns that the LSTM misses. If correct, this would give blockchain security teams a practical, automated screening tool for known vulnerability types, complementing static analysis.","feed_headline":"Fine-tuned LLMs claim 90%+ accuracy on smart contract flaws","feed_subtitle":"Paper compares BERT, DistilBERT, and LSTM on annotated Solidity code to automate blockchain security auditing.","key_machinery":"The load-bearing mechanism is fine-tuning of encoder-only transformer language models—BERT and its distilled variant DistilBERT—on tokenized Solidity source snippets, so that the model learns to map each snippet to one of four vulnerability labels. The paper argues that the bidirectional self-attention over the entire token sequence is what lets these models capture long-range, context-dependent patterns such as the call-and-state-update ordering behind Reentrancy, whereas the LSTM baseline (a bidirectional LSTM with attention and convolution layers) builds a sequential latent representation that is more limited in this respect. The dataset of 2,217 annotated contracts supplies the supervised signal for the fine-tuning.","core_discovery":"The central claim is that fine-tuning a pre-trained encoder-only transformer on a labeled smart contract dataset produces a vulnerability classifier superior to a classical recurrent baseline. Concretely, the authors assert that their fine-tuned DistilBERT model outperforms both BERT and a bidirectional LSTM with attention, achieving an accuracy above 90% and stronger precision-recall balance on the majority of the four vulnerability classes. The paper attributes this advantage to the LLM's ability to attend to the full code context and recognize dependencies, such as the ordering of calls and state updates that define Reentrancy, which sequential models tend to miss. The authors present this as evidence that LLMs can advance existing vulnerability detection benchmarks rather than merely match them.","pith_inferences":["Because the dataset is heavily imbalanced (Reentrancy has over 1,200 samples while Dangerous Delegatecall has under 100), the 'over 90% accuracy' headline may reflect majority-class performance; macro-averaged precision/recall would be the more reliable metric, and the paper's own macro F1 numbers are lower than the overall accuracy.","The iterative-testing procedure described in Section IV-C, where model feedback was used to adjust the dataset and parameters, means an independent replication with a pre-registered train/validation/test split would be needed to confirm that the LLM advantage is not an artifact of test-set feedback.","If the fine-tuning approach transfers, it could be extended to bytecode-level detection or to severity scoring, enabling detection on contracts whose source code is not public.","The same fine-tuned LLM framework could be applied to other blockchain platforms, such as Rust-based smart contracts, where annotated vulnerability datasets are scarcer."],"forward_implications":["If the reported accuracy holds, fine-tuned LLMs can serve as automated first-pass auditors that flag Reentrancy, Integer Overflow, Timestamp Dependency, and Dangerous Delegatecall in Solidity code.","The comparison establishes a benchmark on this annotated dataset that future LLM-based detectors can measure against.","The observed BERT overfitting suggests that larger, more diverse annotated datasets would improve generalization and reduce the gap between training and validation performance.","The better handling of context-dependent classes supports pairing LLM detectors with static analysis tools to lower false alarms in real-world security workflows."],"supporting_citations":[{"why":"Supplies the annotated dataset of 2,217 smart contracts with four vulnerability labels that all models are trained and tested on.","marker":"[26]"},{"why":"Provides the BERT architecture that the paper fine-tunes for vulnerability classification.","marker":"[11]"},{"why":"Provides the DistilBERT architecture, the distilled variant the paper reports as the strongest performer.","marker":"[10]"},{"why":"Defines the LSTM baseline that classical machine learning performance is measured against.","marker":"[9]"},{"why":"Supplies the bidirectional LSTM with attention and convolution layer design used as the classical baseline model.","marker":"[29]"},{"why":"Demonstrates prior fine-tuning of transformer encoder models for smart contract vulnerability detection, the approach this paper extends and compares.","marker":"[8]"}],"fun_headline_variants":["Fine-tuned LLM beats LSTM for smart contract flaws","DistilBERT tops 90% accuracy on Solidity vulnerabilities","LLM detects Reentrancy, Overflow with 90%+ accuracy","Fine-tuned BERT outshines classical ML on contract bugs","Smart contract security: LLM edges out classic models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands only if the reported accuracies were measured on a test set that never influenced the model or dataset adjustments during the iterative testing described in Section IV-C.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLM beats LSTM for smart contract flaws","DistilBERT tops 90% accuracy on Solidity vulnerabilities","LLM detects Reentrancy, Overflow with 90%+ accuracy","Fine-tuned BERT outshines classical ML on contract bugs","Smart contract security: LLM edges out classic models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1232,"prompt_tokens":971,"completion_tokens":261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":587,"tokens_out":261,"duration_ms":2534,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:37.402889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LSTM, DistilBERT, and BERT on the same dataset with a fixed 80/10/10 split, a validation set for early stopping, and a test set touched exactly once, then compare accuracies; if the fine-tuned LLM does not exceed the LSTM's accuracy, or if no model reaches over 90%, the central claim fails.","supporting_citations":[{"cited_title":"Rethinking Smart Contract Fuzzing: Fuzzing With Invocation Ordering and Important Branch Revisiting","cited_arxiv_id":"2301.03943","evidence_quote":"Supplies the annotated dataset of 2,217 smart contracts with four vulnerability labels that all models are trained and tested on."},{"cited_title":"Bidirectional lstm with attention mechanism and convolutional layer for text classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the bidirectional LSTM with attention and convolution layer design used as the classical baseline model."}],"review_version":1}