{"id":"57400029-b9be-405e-b35d-5ad58b45431e","arxiv_id":"2506.18245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A LLaMA-3.1-8B model trained with continual pre-training, supervised fine-tuning, and direct preference optimization reports state-of-the-art accuracy and F1 for smart contract vulnerability detection and explanation.","lead":"The authors trained a LLaMA-based model to find and explain vulnerabilities in Ethereum smart contracts, using extra pre-training, fine-tuning, and preference optimization. The model reports large accuracy and F1 gains over prior tools, which matters because contract bugs have caused millions of dollars in losses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on a self-assembled, self-cleaned test set with no demonstrated deduplication against training data; near-duplicate leakage could account for the reported margins.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the clean-evaluation assumption. I sharpen it by noting that the test set is not a pre-existing independent benchmark; it is constructed and filtered by the same authors from the same source pools, and no deduplication analysis is presented. This is directly load-bearing because the headline claim is numeric superiority, and the effect size is concentrated in small positive samples where even a few leaked contracts can change F1 by several points. The paper does provide real supporting evidence: released code/data/weights, ablations with internally consistent module effects, and human evaluation. Therefore rejection is not warranted, but the conditional acceptance should explicitly require a deduplication audit and a check of label-cleaning neutrality. Other concerns, such as the omitted Smart-LLaMA baseline, reuse of Llama-3.1-70B as both curator and evaluator, and absence of confidence intervals, are real but secondary: they affect interpretation of the margins, whereas leakage would invalidate the margins themselves. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":26096,"tokens_out":6582,"duration_ms":72236,"concrete_test":"Run a deduplication audit between the 3,542 test contracts and every training contract (SFT, DPO, and CPT). Use exact source hash plus normalized token-based Jaccard at thresholds 0.5, 0.7, and 0.9, consistent with the similarity filtering described in Section 3.1.1. For each test contract with a near-duplicate in training, remove that test contract and recompute Tables 2-4 for Smart-LLaMA-DPO and the best baseline per type. Report how many test contracts are affected and whether the average F1/accuracy advantages survive. A clean result should show either zero near-duplicates or unchanged margins; if the margins drop materially below the reported 10.43% F1 and 7.87% accuracy improvements, the SOTA claim is not established. As a secondary check, re-score the original uncleaned labels and verify that cleaning did not selectively remove baseline false positives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is the independence of the evaluation set. Section 4.2 assembles the 3,542-contract test set from the same public pools used for training ([49], [77], Etherscan, GitHub, blogs, 2020-2024) and then systematically cleans it by removing labels the authors deem incorrect, e.g., RE cases with no state change after call.value. No exact-match or fuzzy deduplication between the test set and the SFT/DPO/CPT corpora is reported, even though CPT alone uses 186,397 Etherscan contracts and both training and test include token-level ERC20 reentrancy samples from the same sources. If test contracts appear verbatim or as near-duplicates in training, an 8B model can exploit that familiarity, and the effect is amplified on the small positive sets (DE: 76 vulnerable; CI: 10; AV: 10) where a handful of predictions moves F1 by several points. The cleaning step is also not independent of the method: the removal criterion (no state changes after call.value) is exactly the kind of sequencing judgment the model is trained to make, so ground truth and model reasoning are coupled. Because the central claim is a quantitative SOTA improvement, this unverified independence is the single most load-bearing weakness; the omitted Smart-LLaMA baseline and missing error bars are secondary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Smart-LLaMA-DPO, a smart contract vulnerability detector and explainer built on LLaMA-3.1-8B. The method combines continual pre-training (CPT) on a large Ethereum source-code corpus, supervised fine-tuning (SFT) on a newly constructed multi-type vulnerability dataset with labels, explanations, and locations, and Direct Preference Optimization (DPO) on expert-constructed preferred/rejected explanation pairs. The authors evaluate on a self-assembled test set of 3,542 contracts covering reentrancy, timestamp dependence, integer overflow/underflow, delegatecall, and seven machine-unauditable vulnerability types, reporting state-of-the-art F1 and accuracy gains over a wide range of baselines (claimed averages of +10.43% F1 and +7.87% accuracy). They also report LLM and human evaluations arguing that their explanations are more correct, thorough, and clear than those of iAudit and other baselines. The paper includes ablation studies, a case study, and promises public code, data, and model weights.","tokens_in":26321,"tokens_out":2810,"duration_ms":31536,"significance":"If the reported results are valid, the paper makes a useful contribution: it is, to my knowledge, the first application of DPO to explainable smart contract vulnerability detection, and it provides a relatively broad dataset covering five vulnerability families including machine-unauditable types. The pipeline (CPT then SFT then DPO) is clearly motivated by a concrete failure mode (execution-order misunderstandings), and the ablations suggest that CPT and DPO contribute differently across vulnerability types. The evaluation is unusually extensive for this area, including many rule-based, neural, and LLM baselines, plus human and LLM explanation ratings. The release of code, data, and weights is a concrete strength. However, the central SOTA claim rests on a self-curated test set whose independence from the training corpora is asserted but not demonstrated, and the explanation-quality evaluation uses the same LLM judge that participated in training-data curation; these issues must be resolved before the quantitative claims can be accepted at face value.","major_comments":[{"comment":"The independence of the evaluation set from the training corpora is load-bearing for the SOTA claim and is not established. The evaluation set is assembled from the same public pools used for training (Qian et al. [49], Zhang et al. [77], Etherscan, GitHub, blogs, 2020-2024), and the paper asserts 'no overlap between evaluation data and training data' without reporting any exact-match or fuzzy deduplication procedure. Given that the CPT corpus alone contains 186,397 Etherscan contracts and the SFT/DPO corpora include token-level ERC20 reentrancy samples from the same sources, near-duplicate code across the split could plausibly account for a large fraction of the reported margins. I request a concrete deduplication analysis (e.g., token-level Jaccard similarity distributions between evaluation and all training corpora, with a stated threshold) or, preferably, an additional evaluation on an independently held-out third-party benchmark.","section":"Section 4.2 (Dataset)"},{"comment":"The test-set cleaning step appears coupled to the method. The authors state that they performed 'systematic cleaning to remove incorrect labels (e.g., cases mistakenly labeled as vulnerable where there were no state changes after call.value)'; this criterion is exactly the kind of execution-order judgment the model is trained to make. If the cleaning decisions were informed by the authors' own vulnerability criteria or by model outputs, the ground-truth labels are not independent of the evaluated model. The paper should specify who performed the cleaning, whether it was blind to model predictions, how many contracts were relabeled or removed, and should release the list of removed/relabeled contracts for external audit.","section":"Section 4.2 (Dataset cleaning)"},{"comment":"The performance comparisons lack error bars, confidence intervals, or significance tests, which is particularly important for the small positive sets. In Table 4, the CI and AV columns have only 10 vulnerable samples each, and in Table 3 the delegatecall positive set has 76 samples; in such cases the difference between F1 = 84.85% and the runner-up values can be driven by a handful of predictions. I request bootstrap confidence intervals for all metrics and pairwise significance tests (e.g., McNemar or approximate randomization) against the strongest baselines (DMT, PSCVFinder, iAudit) on the same test sets.","section":"Tables 2-4"},{"comment":"The explanation-quality claim is weakened by using the same judge model (Llama-3.1-70B-Instruct) both to select high-scoring explanations for the training dataset and to grade the final explanations of Smart-LLaMA-DPO versus baselines. The agreement validation of this judge was performed on only 40 samples. Although the detection claim is not directly affected by this circularity, the RQ4 claim about explanation superiority needs an independent judge model or a human-only evaluation on a larger sample, together with inter-annotator agreement statistics.","section":"Sections 3.2.2 and 3.6.2"},{"comment":"The paper omits Smart-LLaMA (Yu et al., reference [73]), the authors' own two-stage post-trained model and the closest methodological competitor, from all quantitative comparisons. Since the paper explicitly positions Smart-LLaMA-DPO as an advance over Smart-LLaMA, the missing baseline makes the stated 'state-of-the-art' claim incomplete. I request that Smart-LLaMA be evaluated on the same test set (using the released weights if available) or that a clear justification for its exclusion be provided.","section":"Section 3 and Tables 2-4"}],"minor_comments":[{"comment":"Typos: 'Futhermore' and 'ultilizing' should be corrected; the term 'machine-unauditable' should be defined at first use in the abstract or introduction.","section":"Abstract and Section 2"},{"comment":"Figure 1 is hard to read as rendered: the code and explanation boxes are cramped, and the arrow labels inside the boxes are not typeset cleanly. Figure 2 would benefit from a clearer separation of the data-construction and training flows.","section":"Figures 1 and 2"},{"comment":"The introduction claims an F1 improvement of 25.98% over iAudit for machine-unauditable vulnerabilities, but Table 4 reports Ours Total F1 = 83.4% versus iAudit Total F1 = 66.2%, which is a difference of 17.2 percentage points, not 25.98%. Please reconcile the numbers and the exact baseline used.","section":"Section 4.2, Table 4"},{"comment":"Equation (1) is typeset as a running fraction with the summation index over n but no explicit upper bound; please reformat so that the token-level CE loss is unambiguous.","section":"Section 3.3, Eq. (1)"},{"comment":"The internal-validity discussion notes that the model sometimes produces redundant output and that truncation is used as a post-processing step. Please specify the truncation rule and its effect on the reported explanation-quality scores, since the human and LLM raters evaluated the truncated outputs.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is strong in breadth of evaluation and in releasing artifacts, but the central empirical claim currently hinges on the integrity of a self-cleaned, self-assembled test set. The missing deduplication analysis and the unblinded cleaning are the primary reasons for major revision rather than accept. The omission of the authors' own Smart-LLaMA baseline is also worth flagging to the editor as a completeness issue in the SOTA comparison. If the authors can provide the deduplication evidence, a blinded-cleaning statement, and error bars, the paper could become a solid ISSTA contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, well-executed empirical study that deserves a real referee slot. The combination of CPT, SFT, and DPO on LLaMA-3.1-8B for smart-contract vulnerability detection is a legitimate new application, and the expert-curated preference dataset is a real artifact others will use. The ablations are thoughtful, the human evaluation is a genuine effort, and the practical gains on machine-unauditable bugs, if real, matter.\n\nWhere it gets soft is exactly where the stress-test points: the SOTA claim rests on a test set the authors assembled and cleaned themselves from the same source pools as the training data. They assert no overlap, but no dedup procedure is reported. For an 8B model, near-duplicate leakage could plausibly produce the reported margins, especially on small positive sets like DE (76), CI (10), and AV (10). The cleaning criterion (e.g., no state change after call.value) is the same sequencing judgment the model is trained to make, so ground truth and model reasoning are coupled. I'd want to see token-level or fuzzy dedup evidence and a breakdown of how many contracts were dropped or relabeled during cleaning.\n\nAlso missing: the authors' own Smart-LLaMA baseline never appears in the comparison tables. Given the paper extends that line, omitting it weakens the \"SOTA\" claim. And there are no error bars or significance tests, so I can't tell if the TD improvement of 1.54 F1 is meaningful. The reuse of Llama-3.1-70B as both data-curation judge and final explanation judge is worth flagging, though the human evaluation partially mitigates it.\n\nNone of these are fatal. The method is specified well enough to replicate, the central qualitative story (CPT helps IO, DPO helps RE/MU, CoT helps MU) is consistent with the ablations, and the artifacts are public. I'd send this to review with a request for major revisions: dedup analysis, Smart-LLaMA baseline, and uncertainty quantification. The paper is a useful contribution for the smart-contract LLM auditing community and for anyone building preference-based explainability pipelines.","headline":"Solid and useful, but the SOTA margins need independent test-set hygiene before I'd trust them.","tokens_in":26934,"tokens_out":2522,"would_cite":true,"duration_ms":27480,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single 8-billion-parameter language model, trained with continual pre-training, supervised fine-tuning, and direct preference optimization, outperforms all prior smart-contract vulnerability detectors and explainers in this benchmark.","keywords":["smart contract vulnerability detection","large language models","direct preference optimization","reentrancy","timestamp dependence","integer overflow/underflow","delegatecall","machine-unauditable vulnerabilities"],"falsifier":"Obtain a fresh set of 2020-2024 Etherscan and GitHub contracts with independent expert labels, run the released Smart-LLaMA-DPO weights on them without any cleaning, and compare against iAudit and DMT; if the F1 gaps over the prior best shrink to near zero, or the machine-unauditable gap drops from 25.98% toward 0%, the generalization claim would be refuted.","tokens_in":25856,"feed_emoji":"🛡️","tokens_out":8002,"duration_ms":76380,"temperature":0.7,"pith_summary":"The paper tries to establish that one relatively small language model can be made to beat every earlier smart-contract vulnerability detector while also explaining its decisions. The recipe is a three-stage post-training of LLaMA-3.1-8B: continual pre-training on roughly 186,000 real Ethereum contracts, supervised fine-tuning on a human-verified dataset of labels, explanations, and code locations, and direct preference optimization on pairs of expert-preferred and expert-written suboptimal explanations. On a 3,542-contract evaluation set spanning reentrancy, timestamp dependence, integer overflow/underflow, delegatecall, and seven machine-unauditable vulnerability types (bugs that automated tools miss), the paper reports average gains of 10.43% in F1 and 7.87% in accuracy over the previous best baselines, with the largest gain (25.98% F1) on machine-unauditable vulnerabilities. Human and LLM evaluators also rate the generated explanations more correct, thorough, and clear than those of the baselines. If the claim holds, a single model—not a pipeline of separate detectors and explainers—is enough for practical auditing, and the reason is that preference learning fixes the specific error that remains after earlier training stages: misunderstanding the order of external calls and state changes.","feed_headline":"Single 8B model beats all smart-contract bug detectors","feed_subtitle":"Three-stage training adds 10.43% F1 and fixes explanations that misorder state changes","key_machinery":"The load-bearing mechanism is Direct Preference Optimization (DPO): a training objective that raises the probability of an expert-preferred explanation and lowers that of a rejected explanation relative to a frozen reference model, without training a separate reward model. DPO is the third stage after continual pre-training on 186,397 real contracts plus general-domain data (620.56M tokens) and supervised fine-tuning on 7,549 human-verified instances with labels, explanations, and locations (8.90M tokens). The preference pairs (1,442 pairs, 1.98M tokens) carry the critical ordering lesson: preferred reentrancy explanations name the exact checks-effects-interactions sequence, trigger conditions, and attack path, while rejected explanations merely flag the risk or state the update order wrongly. The pairing is what lets the model learn subtle security-critical distinctions that absolute labels cannot express.","core_discovery":"After continual pre-training and supervised fine-tuning, an LLM can be right about whether a contract is vulnerable and still wrong about why: it mixes up execution order, for example claiming that a balance is updated before an external call when the code makes the call first. The paper's central claim is that direct preference optimization on expert-verified high/low quality explanation pairs removes this residual error and, as a side effect, sharpens detection itself. The result is a single LLaMA-3.1-8B model that reports F1 improvements of 7.51%, 1.54%, 11.07%, and 6.06% over the previous best on reentrancy, timestamp dependence, overflow/underflow, and delegatecall, respectively, plus a 25.98% F1 improvement on machine-unauditable vulnerabilities, and positive human ratings (4 or 3) on 81.15% of correctness, 83.88% of thoroughness, and 94.63% of clarity judgments.","pith_inferences":["A direct test the paper leaves implicit is to run the released weights on an independently and freshly labeled sample of the same 2020-2024 contract sources, without the authors' label-cleaning step, and see whether the 10.43% F1 margin survives; that would isolate the model's true generalization from the benchmark curation.","Because the DPO 'suboptimal' answers were written by the same experts who wrote the preferred answers, the learned preference is bounded by that group's notion of quality; a different audit team could produce different pairs and shift what the model learns.","The same pipeline could be applied to other high-stakes code, such as DeFi protocol invariants, cross-chain bridges, or administrative key-management logic, where the ablation suggests DPO weight should be higher for reasoning-heavy bug classes and CPT weight higher for syntax-heavy ones.","The model's joint label-plus-explanation output suggests an inexpensive triage tool: contracts flagged vulnerable without a concrete exploit path in the explanation could be routed to human auditors first."],"forward_implications":["A single 8B model can act as both detector and explainer, so auditors no longer need to reconcile labels from one tool with justifications from another.","The three-stage recipe transfers to other domain-specific languages such as Bash, SQL, and SysML, because only the training corpus and annotation guidelines need adaptation.","Preference data becomes a required ingredient for security-focused LLM benchmarks: the ablation shows DPO, not CPT or SFT, is what removes the residual execution-order errors.","Dropping either CPT or DPO costs more than a full retrain of the other: CPT carries overflow/underflow understanding while DPO carries reentrancy and machine-unauditable judgments, so they are complementary rather than interchangeable."],"supporting_citations":[{"why":"Supplies the Direct Preference Optimization objective and the Bradley-Terry reformulation that the training loss is built on.","marker":"[51]"},{"why":"Provides the cross-modality evaluation methodology and the four-vulnerability benchmark distribution used for RE, TD, IO, and DE.","marker":"[49]"},{"why":"Supplies the machine-unauditable vulnerability taxonomy and the Code4rena-sourced MU dataset used for training and evaluation.","marker":"[77]"},{"why":"Provides manually verified vulnerable contracts used as SFT training instances.","marker":"[33]"},{"why":"Supplies the SmartBugs-derived contracts, including 1,634 call.value contracts, that expand the RE, TD, IO, and DE training set.","marker":"[74]"},{"why":"Provides the SmartBugs framework and curated Solidity vulnerability contracts underlying parts of the SFT dataset.","marker":"[22]"},{"why":"Provides the 186,397-contract Ethereum corpus used for continual pre-training.","marker":"[55]"},{"why":"Is the strongest explainable-auditing baseline and the source of the two-stage detection-then-explanation design the paper argues against.","marker":"[37]"},{"why":"Supplies the reentrancy annotations on SmartBugs contracts used in SFT data construction.","marker":"[71]"},{"why":"Is the prior two-stage Smart-LLaMA approach whose CPT+SFT limitations and evaluation design this work extends with DPO.","marker":"[73]"}],"fun_headline_variants":["8B LLaMA wins on smart-contract bugs and explains them right","One 8B LLM beats bug detectors and clarifies its reasoning","Smart-LLaMA: 8B model outscores detectors, explains why","8B LLaMA tops smart-contract detectors and explains its logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margins rest on the assumption that the 3,542-contract test set, which the authors cleaned by removing 'incorrect labels' and drew from the same source pools as the training data, is free of near-duplicate overlap with training code; if that assumption fails, the 10.43% F1 gain would overstate how well the model detects unseen real-world contracts.","fun_headline_variants_meta":{"raw":{"variants":["8B LLaMA wins on smart-contract bugs and explains them right","One 8B LLM beats bug detectors and clarifies its reasoning","Smart-LLaMA: 8B model outscores detectors, explains why","8B LLaMA tops smart-contract detectors and explains its logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000356,"raw_usage":{"total_tokens":1995,"prompt_tokens":1074,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":840}},"tokens_in":690,"tokens_out":921,"duration_ms":10443,"temperature":1.0,"reasoning_tokens":840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:24.195061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain a fresh set of 2020-2024 Etherscan and GitHub contracts with independent expert labels, run the released Smart-LLaMA-DPO weights on them without any cleaning, and compare against iAudit and DMT; if the F1 gaps over the prior best shrink to near zero, or the machine-unauditable gap drops from 25.98% toward 0%, the generalization claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cross-modality evaluation methodology and the four-vulnerability benchmark distribution used for RE, TD, IO, and DE."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SmartBugs-derived contracts, including 1,634 call.value contracts, that expand the RE, TD, IO, and DE training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SmartBugs framework and curated Solidity vulnerability contracts underlying parts of the SFT dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 186,397-contract Ethereum corpus used for continual pre-training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reentrancy annotations on SmartBugs contracts used in SFT data construction."}],"review_version":1}