{"id":"76c85d5f-656e-47c4-b4c0-1e7848b1ee04","arxiv_id":"2505.08542","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A finite-state-machine intermediate representation, fine-tuning, and compilation and security feedback improve LLM-generated smart contract compilation rates and lower vulnerability scores on a synthetic benchmark.","lead":"This paper describes a framework that turns user requirements into a finite state machine, then uses that machine to guide an LLM in writing Ethereum smart contracts, with automatic compilation and security feedback loops. A smart generalist might read it to see one approach to adding structure and validation to LLM-generated code, though the evaluation has important limitations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.3 tests on requirements sampled from the same dataset used for fine-tuning, so the headline CPR/VRS gains may reflect distribution overlap rather than generalizable generation quality.","rationale":"The reader identified the same load-bearing assumption: the test set is sampled from the fine-tuning dataset. My read of Sections 3.2, 4.3, and Appendix B confirms this is the most serious threat to the central claim. The paper never states that the 1,000 test requirements are held out, and the phrase 'from dataset' in Section 4.3 naturally refers to the 30k fine-tuning dataset described in Section 3.2. If the test and fine-tuning sets share the same synthetic source, the comparison is not a test of generalization. No independent evidence offsets this: there is no released code, no link to the claimed open-source dataset, and no formal verification. The VRS metric's dependence on compilation success is a real but secondary issue; even if corrected, it would not resolve the training/test overlap. Therefore the reader's REJECT verdict remains appropriate. A re-evaluation on a genuinely disjoint, human-written requirement set with decoupled security metrics could change the verdict to CONDITIONAL, because the controlled ablations (FSM vs. Direct, FSM-SCG* with feedback, ablation of fine-tuning) suggest the framework may have some real effect. But as written, the central empirical support is not established.","tokens_in":14509,"tokens_out":4329,"duration_ms":44935,"concrete_test":"Request the exact train/test split from the authors, or reconstruct it if the dataset is released, and verify that the 1,000 test requirements do not overlap with fine-tuning examples by exact match or embedding-based near-duplicate detection. Then rerun FSM-SCG against Direct and FSM-SCG* on a truly held-out set of human-written smart-contract requirements (not GPT-4o-generated from the same contract pool). If the CPR gap remains close to the reported 95.1% versus 47.5% and the VRS gap remains substantial, the concern is resolved; if the gap collapses, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the 1,000 test requirements being independent of the fine-tuning data. Section 4.3 says: 'We sample 1,000 high-quality requirements from dataset for testing.' Section 3.2 and Appendix B define the dataset as the fine-tuning set of ~30k items built by using GPT-4o to generate requirements and FSMs from Etherscan contract code. No separate held-out test set or train/test split is described anywhere in the paper. If those 1,000 requirements are drawn from the same pool as D_f t, then the fine-tuned FSM-SCG has already been trained on the same synthetic requirement style and, potentially, on requirements derived from the same contracts. The reported improvements in CPR and VRS (Figure 4 and Table 2) would then reflect distribution overlap and possible data leakage, not a generalizable advantage of FSM-guided generation. This is load-bearing because every headline number is measured on this test set. A secondary, related weakness: VRS is assigned a value of 10 for non-compiling contracts, so security improvements are partially entangled with compilation success, and the security claim is not independent of the effectiveness claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FSM-SCG, a pipeline that transforms natural-language user requirements into an enhanced finite-state-machine representation (SmartFSM), uses a fine-tuned LLM to generate smart contracts from the FSM, and then iteratively refines the code using compilation and Slither-based security feedback. The main claims are that FSM-guided generation plus fine-tuning and feedback substantially improves compilation success rate (CPR) and reduces vulnerability risk score (VRS) relative to direct generation and to intermediate-representation baselines, with headline numbers of 95.1% CPR and a 68% VRS reduction reported on LLaMa3.1-8B. The authors also describe the construction of a 30k-item fine-tuning dataset generated by GPT-4o from Etherscan contract code, and they report ablations on SmartFSM, feedback, fine-tuning, and the A2C sub-dataset.","tokens_in":14724,"tokens_out":7142,"duration_ms":66203,"significance":"If the evaluation were sound, the paper would make a useful contribution: SmartFSM is a reasonable way to inject structure into LLM-based contract generation, the ablation study isolates the contribution of individual components, and an open fine-tuning dataset would help reproducibility. However, the current test protocol does not support the headline quantitative claims. The test requirements are drawn from the same synthetic dataset used for fine-tuning, so the CPR and VRS numbers may reflect train/test distribution overlap rather than a generalizable improvement; in addition, the VRS metric assigns 10 to non-compiling contracts, entangling security with compilation success. The core idea is promising and the flaws are addressable, but the evidence as presented is not sufficient for acceptance.","major_comments":[{"comment":"The test set is sampled from the same dataset used for fine-tuning, creating a direct train/test overlap. Section 4.3 states \"We sample 1,000 high-quality requirements from dataset for testing,\" and Section 3.2 defines that dataset as the fine-tuning set built by having GPT-4o generate requirements and FSMs from existing contract code. No separate held-out set or contract-level split is described anywhere in the paper. Because the fine-tuned model has been trained on requirements of the same synthetic style and possibly derived from the same contracts, the reported CPR and VRS improvements in Figure 4 and Table 2 may reflect distribution overlap rather than a generalizable advantage. This is load-bearing for every headline number; the authors must evaluate on an independent test set of human-written or otherwise held-out requirements and must describe how the 1,000 test requirements were selected and verified to be disjoint from the fine-tuning data.","section":"§3.2, §4.3"},{"comment":"The VRS metric assigns a score of 10 to every non-compiling contract. Since FSM-SCG achieves much higher CPR than the baselines (Table 1 and Figure 4), part of the reported 68% VRS reduction may be a mechanical consequence of more contracts compiling, not of improved security. The paper should report VRS computed only over compiled contracts, or make ZRCP and HRCP the primary security metrics, so that the security effect is separated from the compilation effect.","section":"§4.2, VRS definition"},{"comment":"The fine-tuning dataset construction also limits external validity. Because the \"user requirements\" are reverse-engineered by GPT-4o from existing contract code, the model is trained and tested on a distribution that may not match real user requirements, which are often incomplete, ambiguous, or expressed differently. A random split of this dataset would not fully solve the problem. The authors should test on independently collected human-written requirements, or at minimum provide a contract-level split and a qualitative comparison showing that the synthetic requirements resemble real-world usage.","section":"§3.2, §4.3"}],"minor_comments":[{"comment":"The text says the CPR of LLaMa3.1-8B rises from 36.9% to 95.3%, but Table 1 and Figure 4 report 95.1%; please correct the inconsistency.","section":"§4.3"},{"comment":"Figure 4 appears to include FSM-SCG on GPT-4o, but Section 4.3 states that FSM-SCG is applied only to LLaMa3.1-8B and Qwen2.5-7B; clarify which bars correspond to FSM-SCG in the figure.","section":"§4.1, Figure 4"},{"comment":"The FSM regeneration loop in lines 4-6 calls M_f t(R) rather than M_f t(P_{R→F}), and neither loop has an iteration cap; align the pseudocode with the experimental setting of one feedback round described in Section 4.1.","section":"Appendix A, Algorithm 1"},{"comment":"The graph check requires that all transitions satisfy t ≠ s, but a self-loop can model a legitimate event that does not change the contract state; please justify or remove this condition.","section":"§3.4"},{"comment":"The paper claims to release an open-source fine-tuning dataset but gives no URL or repository; include a link or release mechanism for reproducibility.","section":"§3.2"},{"comment":"Minor typos and naming inconsistencies should be fixed, including \"LlaMa3.1\" (should be \"LLaMA3.1\") and \"GeMini1.5\" (should be \"Gemini1.5\").","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"I would not support acceptance until the evaluation is redone on a genuinely held-out test set; the current self-referential test protocol is the central obstacle. The VRS-compilation entanglement also needs to be addressed before the security claims can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou asked about the FSM-SCG paper. Here's my honest take: the framework is a sensible engineering idea, but the evaluation is compromised in a way that breaks the headline claim. The test set is sampled from the same dataset used to build the fine-tuning data, so the reported gains are likely inflated by distribution overlap.\n\nWhat is actually new? SmartFSM, a five-part FSM representation that captures states, variables, functions, and events, is a thoughtful addition. The feedback loop that feeds compilation errors and Slither security warnings back into the LLM is practical. Running across six LLMs and including ablations for each component is also a plus. The paper does show that the FSM-only variant (no fine-tuning) beats direct prompting on several models, which suggests the core idea has merit.\n\nThe problem is in Section 4.3. The 1,000 test requirements are sampled from 'dataset,' which is the same pool used for fine-tuning. No separate held-out set is described. So the fine-tuned model has already seen the synthetic requirement style, and potentially requirements derived from the same contracts. This is a load-bearing flaw. The 48% CPR improvement and 68% VRS reduction are measured against baselines that have not been fine-tuned on the test distribution. Not an apples-to-apples comparison.\n\nThere are smaller issues too. VRS assigns a score of 10 to non-compiling contracts, tangling the security metric with compilation success. The paper says the dataset is released, but no link appears. And while the FSM-SCG* feedback-only variant also improves over baselines, it is still evaluated on the same leaked test set.\n\nNone of this kills the underlying idea. The FSM-guided approach could well be useful in practice. But the empirical support as stated is not credible. A proper re-evaluation on a genuinely held-out, human-written requirement set, with decoupled security metrics and accessible code/data, could change my mind.\n\nThis paper deserves peer review—not a desk reject—because the engineering contribution is real and a competent referee could guide the authors toward a valid evaluation. It's also a good reading-group example of why test-set independence matters.\n\nBest, [Your name]","headline":"Promising engineering idea for FSM-guided smart contract generation, but the evaluation leaks the training distribution into the test set, so the headline gains are not credible as stated.","tokens_in":15278,"tokens_out":3585,"would_cite":false,"duration_ms":32951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Interposing a finite state machine between requirements and code, then repairing the code with compiler and security feedback, yields smart contracts that compile far more often and carry far fewer vulnerabilities.","keywords":["finite state machine","smart contract generation","large language models","Solidity","compilation pass rate","vulnerability risk score","fine-tuning feedback","state machine prompting"],"falsifier":"Run FSM-SCG on 1,000 Solidity requirements written by humans rather than generated by an LLM from existing contract code; if the compilation pass rate falls back toward the 37–50% baseline range, the reported gains come from test/training distribution overlap, not from the method itself.","tokens_in":14270,"feed_emoji":"📜","tokens_out":12796,"duration_ms":108439,"temperature":0.7,"pith_summary":"Large language models generate Solidity code that often fails to compile and frequently carries exploitable vulnerabilities. The paper's bet is that the bottleneck is a missing intermediate structure: the model is asked to jump from a natural-language requirement straight to code. The proposed FSM-SCG pipeline instead has a fine-tuned LLM first translate each requirement into SmartFSM, an augmented finite state machine recording states, variables, functions, and events, then generate the contract from that machine, and finally repair the code using compiler errors and static vulnerability reports as feedback. On 1,000 sampled requirements with Llama-3.1-8B, the pipeline raises the compilation pass rate from 36.9% (direct generation) to 95.1% and lowers the average vulnerability risk score from 7.44 to 2.36. The paper's claim is that this decomposition plus repair loop is what makes the difference, and the topic matters because smart contracts cannot be patched after deployment.","feed_headline":"State machine guide lifts smart contract compile rate to 95%","feed_subtitle":"The intermediate state machine scaffold plus compiler and security feedback cuts vulnerability risk about 68%.","key_machinery":"The load-bearing object is SmartFSM, a five-section extension of a Mealy state machine—basic information, states, variables, functions, and events—that keeps the transition function $\\delta: S \\times X \\to S$ while adding the information a contract needs. It works by giving the LLM a fixed JSON scaffold that it can be fine-tuned to fill and that can be machine-checked before any code is written. Around that object, the framework wraps two repair loops: compilation feedback from the Solidity compiler and security feedback from Slither, both prompting the model to regenerate until checks pass. The format and graph checks (initial state defined, targets defined, all states reachable from the initial state, no self-loops) are what make the intermediate representation trustworthy enough to guide generation.","core_discovery":"The paper's central claim is that the bottleneck in LLM-based smart contract generation is not the language model itself but the absence of a checkable intermediate structure. FSM-SCG therefore inserts a SmartFSM—an augmented finite state machine carrying five sections (basic information, states, variables, functions, events)—between the user's natural-language requirements and the Solidity code. A fine-tuned model first produces the machine, checks it for format and reachability, then writes the contract from it; compiler errors and static vulnerability findings are fed back into the model to patch the code. On the paper's own experiments this raises the compilation pass rate for Llama-3.1-8B from 36.9% to 95.1% and cuts the average vulnerability risk score from 7.44 to 2.36, with qualitatively similar gains on five other models.","pith_inferences":["A consequence the authors leave implicit: because the test requirements are drawn from the same synthetic distribution used for fine-tuning, the reported CPR/VRS gains are likely an upper bound on what the pipeline would deliver on genuinely novel user requirements; an independent human-written test set is needed.","The graph check accepts any reachable graph with no self-loops, which is too weak to catch many logical errors; pairing SmartFSM with invariant checking or a model checker could reduce vulnerabilities further than the one-round Slither feedback loop.","The same requirement-to-FSM-to-code decomposition could transfer to other low-resource contract languages such as Vyper or Move, where syntax errors are the dominant failure mode; the paper only demonstrates Solidity.","A policy of one compilation feedback round plus one security feedback round is a bounded repair loop; on out-of-distribution requirements, repair may require many rounds or may never terminate, so a budget-aware stopping rule would make the framework more robust."],"forward_implications":["On Llama-3.1-8B, the full FSM-SCG pipeline reaches a 95.1% compilation pass rate and a 2.36 vulnerability risk score, beating direct generation (36.9% / 7.44) and both intermediate-DSL baselines.","The benefit transfers across model families: fine-tuned Qwen2.5-7B reaches 93.6% CPR and VRS 2.48, and even the no-fine-tuning variant FSM-SCG* beats all baselines on the six tested models.","Every ablated component matters: removing feedback drops Llama-3.1-8B CPR from 95.1% to 91.3% and raises VRS from 2.36 to 3.11; removing the whole fine-tuning step drops CPR to 71.3%.","SmartFSM itself, without fine-tuning or feedback, already beats plain Mealy FSM and the CML/IContractML baselines, so the five-section representation is a genuine improvement over a bare state machine."],"supporting_citations":[{"why":"Provides Slither, the static analyzer whose vulnerability reports supply the security feedback signal in FSM-SCG.","marker":"[Feist et al., 2019]"},{"why":"Chain-of-thought prompting; motivates decomposing generation into requirement-to-FSM and FSM-to-code subtasks.","marker":"[Wei et al., 2022]"},{"why":"Structured chain-of-thought for code generation; supports the use of a structured intermediate representation.","marker":"[Li♂ et al., 2023]"},{"why":"IContractML baseline; a model-driven LLM generation approach with an intermediate model that FSM-SCG must beat.","marker":"[Petrović and Al-Azzoni, 2023]"},{"why":"CML baseline; a high-level DSL for smart contracts used as a rival intermediate representation.","marker":"[Wöhrer and Zdun, 2020]"},{"why":"Chat2Code baseline; a chatbot and DSL-based contract generation approach used for comparison.","marker":"[Qasse et al., 2023]"},{"why":"Direct LLM generation baseline; also the source of the observation that LLMs overlook security.","marker":"[Chatterjee and Ramamurthy, 2024]"},{"why":"Direct requirements-to-code LLM baseline that FSM-SCG outperforms.","marker":"[Napoli et al., 2024]"},{"why":"LoRA; the alternative fine-tuning method shown inferior to full-parameter fine-tuning in the ablation.","marker":"[Hu et al., 2022]"}],"fun_headline_variants":["FSM scaffold boosts smart contract compile rate to 95%","State machine bridges LLM and smart contracts, cut risk 68%","SmartFSM guide lifts Solidity pass rate from 37% to 95%","LLM smart contracts: FSM intermediate cuts vulnerabilities 68%","Checkable FSM steers LLMs to safer, compilable contracts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 1,000 test requirements, sampled from the same dataset used to build the fine-tuning set, are independent of that set and representative of real user requests.","fun_headline_variants_meta":{"raw":{"variants":["FSM scaffold boosts smart contract compile rate to 95%","State machine bridges LLM and smart contracts, cut risk 68%","SmartFSM guide lifts Solidity pass rate from 37% to 95%","LLM smart contracts: FSM intermediate cuts vulnerabilities 68%","Checkable FSM steers LLMs to safer, compilable contracts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3211,"prompt_tokens":886,"completion_tokens":2325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2228}},"tokens_in":502,"tokens_out":2325,"duration_ms":14379,"temperature":1.0,"reasoning_tokens":2228,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:51:16.349568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FSM-SCG on 1,000 Solidity requirements written by humans rather than generated by an LLM from existing contract code; if the compilation pass rate falls back toward the 37–50% baseline range, the reported gains come from test/training distribution overlap, not from the method itself.","supporting_citations":[{"cited_title":"Slither: a static analysis framework for smart contracts","cited_arxiv_id":null,"evidence_quote":"Provides Slither, the static analyzer whose vulnerability reports supply the security feedback signal in FSM-SCG."},{"cited_title":"Model-driven smart contract generation lever- aging chatgpt","cited_arxiv_id":null,"evidence_quote":"IContractML baseline; a model-driven LLM generation approach with an intermediate model that FSM-SCG must beat."},{"cited_title":"Domain specific language for smart contract devel- opment","cited_arxiv_id":null,"evidence_quote":"CML baseline; a high-level DSL for smart contracts used as a rival intermediate representation."},{"cited_title":"Chat2code: A chatbot for model specification and code generation, the case of smart contracts","cited_arxiv_id":null,"evidence_quote":"Chat2Code baseline; a chatbot and DSL-based contract generation approach used for comparison."},{"cited_title":"Efficacy of Various Large Language Models in Generating Smart Contracts","cited_arxiv_id":"2407.11019","evidence_quote":"Direct LLM generation baseline; also the source of the observation that LLMs overlook security."},{"cited_title":"Lever- aging large language models for automatic smart contract generation","cited_arxiv_id":null,"evidence_quote":"Direct requirements-to-code LLM baseline that FSM-SCG outperforms."}],"review_version":1}