REVIEW 4 major objections 6 minor 27 references
Generating Move Smart Contracts based on Concepts
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read ConMover claims that Move smart contracts can be generated from natural language using a concept knowledge graph and iterative self-refinement, lifting small open LLMs' pass rates by up to about 47 percent relative to their self-refined…
desk verdict Plausible engineering framework for low-resource code generation, but the evaluation doesn't establish the central claim because the train/test split is unverified and the cited data source is a taxonomy, not a code benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a three-part loop: a knowledge graph of Move concepts extracted from official documentation, a small set of correct Move examples used by a planning agent that emits confidence-scored plans, and a debug-refinement stage in which a model fine-tuned on faulty code paired with compiler execution traces rewrites the failed code through a fixed template that concatenates the natural-language description, the faulty code, and the execution feedback. The knowledge graph is also used as a retrieval-augmented validator during planning, and the whole pipeline is orchestrated as an adaptive traversal that cycles through the next-best plan when debugging exhausts its refinement budget.
What would settle it
Run ConMover's self-refinement on a fresh Move benchmark whose tasks are provably disjoint from the 313 faulty examples used for fine-tuning and from the 20 example categories; if the pass-rate improvements over the direct baseline shrink to near zero or fall below the RAG-with-correct-code baseline, the reported gains are in-distribution artifacts rather than general self-correction.
Extended reading notes
Core claim
On its own terms, the paper claims that ConMover can generate Move smart contracts based on concepts without pre-training an LLM on a vast existing code corpus. The discovery is that a knowledge graph of Move concepts, built from documentation, plus roughly 20 categories of correct Move examples, can substitute for large training corpora when they are injected through a planning agent and when a small code model is fine-tuned to self-refine using faulty code and compiler feedback. The reported evidence is a set of pass-rate comparisons: on a 780-test Move dataset, ConMover's self-refinement improves the 2B-parameter open model from 12.2 percent to 20.7 percent, the 7B model from 18.3 percent to 22.0 percent, and the 9B model from 21.4 percent to 29.3 percent; for two large proprietary models it improves direct pass rates from 27.8 percent to 37.6 percent and from 37.8 percent to 56.9 percent.
Load-bearing premise
The results stand or fall on the assumption that the 780-test evaluation set does not overlap with the 313 faulty examples used to build the fine-tuning data, since the paper reports no train/test split.
Editorial extensions
If this is right
- Fine-tuning a code model on faulty code plus execution feedback teaches self-refinement that does not arise from pre-training alone, since adding RAG with correct code gives much smaller gains.
- Small open models can match or approach the one-time pass rates of much larger models once they are equipped with ConMover's planning and debugging loop, so deployment with a 2B-parameter model becomes practical.
- A knowledge base of roughly 20 correctly coded categories and a concept graph from documentation can be enough to ground code generation in a low-resource language, lowering data-engineering cost.
- The improvement transfers to large proprietary models when the same planning and self-refinement pipeline is applied, suggesting the method is model-agnostic rather than tied to the fine-tuned open model.
- Repeated self-refinement with compiler feedback converges within a few iterations, so the accuracy gain does not require unbounded inference cost.
Reading between the lines
- Editorial inference: if the train/test separation is clean, the same concept-graph-plus-debug-loop recipe could plausibly transfer to other verification-oriented low-resource languages by swapping the documentation source and the fine-tuning traces, though the paper only demonstrates Move.
- Editorial inference: the design suggests a testable extension where static-analysis warnings or formal-verification counterexamples replace runtime failures as the debugging feedback, which could matter for safety-critical smart contracts.
- Editorial inference: because the largest relative gains appear on the smallest model, ConMover points toward on-device or privacy-preserving code generation with small open-weight models, a direction the paper does not evaluate.
- Editorial inference: the claim would be strengthened by an ablation that disables confidence-ranked planning; if random plan order performs about as well, the ranking mechanism is not carrying the improvement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ConMover, a multi-agent framework for generating Move smart contracts. It builds a knowledge graph from Sui Move documentation, uses concept retrieval, planning, coding, and debugging agents, and fine-tunes ConMoverLM on faulty code and execution traces. The framework is evaluated on a scraped dataset of 780 Move tasks with several open-source LLMs, and the paper reports improvements in both one-time generation and iterative self-refinement settings.
Significance. If the reported numbers are valid, the framework would be a useful contribution to low-resource code generation: it suggests that a small open-weight model can be augmented to outperform direct use of larger proprietary models (e.g., Gemini 1.5 002 rising from 37.8 to 56.9 in Table 2). The paper's strengths include a clearly described multi-agent pipeline, comparison against chain-of-thought, self-planning, and self-debugging baselines, and a deliberate focus on small models. However, the paper ships no code, data, or model weights, reports no train/test split, and the evaluation is the sole support for the main claims; in its present form the evidence is not verifiable.
major comments (4)
- [Sections 2.2, 5.3, and 7] The evaluation is not shown to be disjoint from the training data. Section 2.2 states that 313 code examples 'from the whole dataset' are used to create the wrong-solution corpus; Section 5.3 states that ConMoverLM is fine-tuned on faulty code, execution traces, and compilation errors from that corpus; Section 7 evaluates on 'our scraped dataset in Move[2]' with 780 tests. No train/test split is reported. Because the fine-tuned model is the self-refinement engine, the gains in Tables 1 and 2 could reflect memorization of training tasks rather than generalization. The paper must report the split, release the data, and re-run the evaluation on a held-out set.
- [Section 7, reference [2]] The data source cited for the evaluation is electric-capital/crypto-ecosystems, a taxonomy of blockchain ecosystems, not a Move code benchmark. This makes it impossible to determine what the 780 tests contain, how they were generated, or whether they are independent of the fine-tuning examples. The authors should replace this citation with the actual benchmark or dataset and describe its composition, including how the 313 faulty examples were selected and how they relate to the 780 test tasks.
- [Table 1] The one-time generation results do not support a general improvement claim. Gemma-2 9B decreases from 21.9 to 21.4 with ConMover, CodeLlama-7B stays at 18.3, and Gemma-2 2B improves only from 12.2 to 14.0. The headline result therefore rests on the self-refinement numbers, which are exactly the numbers most exposed to the contamination risk described in the first major comment. No error bars or significance tests are provided for any of the tables, so the reader cannot tell whether differences such as 23.8 versus 29.3 are stable or due to sampling noise.
- [Sections 5.2, 5.3, and 6.2] The fine-tuning and self-refinement procedure is underspecified for reproducibility. The paper reports hyperparameters (batch size 512 examples, top-p 0.95, learning rates, 5% masking, 25% self-refined data), but it does not state the number of training examples, the source of canonical solutions, the exact split between faulty-code fine-tuning and instruction tuning, or how the roughly 313 examples relate to the 780-task test set. No model weights or code are released. Without this information the experimental claims cannot be independently checked.
minor comments (6)
- [Section 4] The framework name is written as 'CoMover' at the start of Section 4; the rest of the paper uses 'ConMover', so the spelling should be unified.
- [Section 7] The sentence 'comparing its performance to MOve for various sizes' should read 'Move'; the same typo appears in the first line of the evaluation section.
- [Section 2.1] The phrase 'apros chain' should be corrected to 'Aptos chain'.
- [Section 7] The passage 'We conducted evaluations using the Pass@1 metric, In the set k iterations (10 in our experiment)' is grammatically unclear; please define Pass@1 and clarify the relationship between the 10 iterations and the four or five refinement attempts described in Section 5.3.
- [Section 7] The sentence 'Instead of training a new code language model (LM) from the ground up,We also compared different self-refining techniques and ours on the state-of-the-art models.ed a vast amount of information about code structure, syntax, and common patterns' is garbled and should be rewritten.
- [Table 1] The table heading says 'Move Test (780 Tests)' and the columns are 'One-time' and 'Self-Refine', but the units are not explicit; clarify that the numbers are percentages (Pass@1).
Circularity Check
Self-refinement gains may be in-distribution: ConMoverLM is fine-tuned on faulty Move code from the same scraped dataset [2] that is later used as the 780-test evaluation set, with no reported train/test split.
-
fitted input called prediction
[Sections 2.2, 5.3, 7 and Table 1]
"We use [2] as our coding data source to generate task descriptions for the LLMs to solve. We use 313 code examples from the whole dataset to create our wrong solution dataset. ... ConMoverLM is fine-tuned on faulty code execution traces and its compilation errors. ... This section evaluates ConMover’s code generation capabilities, comparing its performance to MOve for various sizes across our scraped dataset in Move[2]."
Fine-tuning input and evaluation output are drawn from the same source: the 313 code examples used to build the faulty-code/trace corpus come from dataset [2], and the 780-test evaluation is described as 'our scraped dataset in Move[2].' The paper never reports a split between the fine-tuning examples and the test tasks. Consequently the self-refinement accuracy in Tables 1 and 2 is not shown to be an out-of-distribution prediction; it is a score on data whose overlap with the training corpus is neither documented nor controlled. If any of the 780 test tasks correspond to the 313 fine-tuning seeds (or to tasks generated from the same descriptions), the reported improvement is forced by memorization rather than measured generalization.
full rationale
The central empirical claim is that ConMover's self-refinement substantially improves Move code generation. The load-bearing evidence is Table 1's self-refinement column and Table 2's ConMover results. The derivation chain for that evidence is: (i) Section 2.2 creates a wrong-solution corpus from 313 code examples scraped from [2]; (ii) Section 5.3 fine-tunes ConMoverLM on the resulting faulty code, execution traces, and compilation errors; (iii) Section 7 and Table 1 score ConMover on 'our scraped dataset in Move[2]' with 780 tests. No train/test split, no overlap analysis, and no external benchmark is reported. Thus the 'prediction' of self-refinement accuracy is not shown to be independent of the fine-tuning input; if the 780 tests include any of the 313 seeds (or tasks generated from the same source), the gains are in-distribution and partly forced. This is a fitted-input-called-prediction concern rather than a definitional identity. I did not find other load-bearing circularity: self-citations to [13] are motivational or comparative and do not by themselves force the accuracy result; no uniqueness theorem or ansatz-by-citation is used. The severity is capped at 6 because the paper's text does not explicitly state that the test tasks are identical to the training tasks; it simply omits the split, leaving the reduction conditional. If a clean split is later documented, this circularity concern would disappear.
Assumptions & free parameters
free parameters (3)
- top_p (nucleus sampling) =
0.95
- maximum refinement iterations =
4 to 5
- self-refinement training mixture =
25 percent self-refined samples, 5 percent masked input
assumptions (3)
- domain assumption Compiler and unit tests are a correct and complete oracle for Move code quality.
- ad hoc to paper The Sui Move documentation plus 20 example categories cover the concepts needed for all 780 test tasks.
- ad hoc to paper Fine-tuning on faulty Move code and execution traces transfers to unseen Move tasks.
Cite this review
Pith. "Pith review of Generating Move Smart Contracts based on Concepts." pith.science (2026). https://pith.science/paper/TKB43EOM
@misc{pith2026241212513,
author = {Pith},
title = {Pith review of: Generating Move Smart Contracts based on Concepts},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKB43EOM}},
note = {Machine review of arXiv:2412.12513}
}
read the original abstract
The growing adoption of formal verification for smart contracts has spurred the development of new verifiable languages like Move. However, the limited availability of training data for these languages hinders effective code generation by large language models (LLMs). This paper presents ConMover, a novel framework that enhances LLM-based code generation for Move by leveraging a knowledge graph of Move concepts and a small set of verified code examples. ConMover integrates concept retrieval, planning, coding, and debugging agents in an iterative process to refine generated code. Evaluations with various open-source LLMs demonstrate substantial accuracy improvements over baseline models. These results underscore ConMover's potential to address low-resource code generation challenges, bridging the gap between natural language descriptions and reliable smart contract development.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[2]
electric-capital/crypto-ecosystems: A taxonomy for open source cryptocurrency, blockchain, and decentralized ecosystems (2024), https://github.com/electric-capital/crypto-ecosystems
work page 2024
-
[1]
blog/starchat-alpha.md at main · huggingface/blog · github (2024), https://github.com/huggingface/ blog/blob/main/starchat-alpha.md
work page 2024
-
[3]
Move concepts | sui documentation (2024), https://docs.sui.io/concepts/sui-move-concepts
work page 2024
-
[4]
sui/examples/move at main · mystenlabs/sui (2024), https://github.com/MystenLabs/sui/tree/main/ examples/move
work page 2024
-
[5]
arXiv preprint arXiv:2108.07732 (2021)
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)
arXiv 2021
-
[6]
Blackshear, S., Cheng, E., Dill, D.L., Gao, V ., Maurer, B., Nowacki, T., Pott, A., Qadeer, S., Rain, D.R., Sezer, S., et al.: Move: A language with programmable resources. Libra Assoc p. 1 (2019)
work page 2019
-
[7]
arXiv preprint arXiv:2107.03374 (2021)
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)
arXiv 2021
-
[8]
arXiv preprint arXiv:2304.05128 (2023)
Chen, X., Lin, M., Schärli, N., Zhou, D.: Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023)
arXiv 2023
Show all 27 references
-
[9]
arXiv preprint arXiv:2405.11514 (2024)
Eniser, H.F., Zhang, H., David, C., Wang, M., Paulsen, B., Dodds, J., Kroening, D.: Towards translating real- world code with llms: A study of translating to rust. arXiv preprint arXiv:2405.11514 (2024)
2024 arXiv
-
[10]
In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security
He, J., Vechev, M.: Large language models for code: Security hardening and adversarial testing. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. pp. 1865–1879 (2023)
2023
-
[11]
ACM Transactions on Software Engineering and Methodology (2023)
Jiang, X., Dong, Y ., Wang, L., Zheng, F., Shang, Q., Li, G., Jin, Z., Jiao, W.: Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology (2023)
2023
-
[12]
arXiv preprint arXiv:2407.05202 (2024)
Karanjai, R., Hussain, A., Rabin, M.R.I., Xu, L., Shi, W., Alipour, M.A.: Harnessing the power of llms: Au- tomating unit test generation for high-performance computing. arXiv preprint arXiv:2407.05202 (2024)
2024 arXiv
-
[13]
In: Proceedings of the 1st ACM International Conference on AI-Powered Software
Karanjai, R., Xu, L., Shi, W.: Solmover: Smart contract code translation based on concepts. In: Proceedings of the 1st ACM International Conference on AI-Powered Software. pp. 112–121 (2024)
2024
-
[14]
arXiv preprint arXiv:2211.15533 (2022) 11 ConMover: Generating Move Smart Contracts based on Concepts A PREPRINT
Kocetkov, D., Li, R., Allal, L.B., Li, J., Mou, C., Ferrandis, C.M., Jernite, Y ., Mitchell, M., Hughes, S., Wolf, T., et al.: The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533 (2022) 11 ConMover: Generating Move Smart Contracts based on Con...
2022 arXiv
-
[15]
Science 378(6624), 1092–1097 (2022)
Li, Y ., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al.: Competition-level code generation with alphacode. Science 378(6624), 1092–1097 (2022)
2022
-
[16]
arXiv preprint arXiv:2203.13474 (2022)
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y ., Savarese, S., Xiong, C.: Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)
2022 arXiv
-
[17]
arXiv preprint arXiv:2308.12950 (2023)
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)
2023 arXiv
-
[18]
Advances in Neural Information Processing Systems 36 (2024)
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., Scialom, T.: Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36 (2024)
2024
-
[19]
Sun, Q., Luo, Y ., Zhang, W., Li, S., Li, J., Niu, K., Kong, X., Liu, W.: Docs2kg: Unified knowledge graph construction from heterogeneous documents assisted by large language models (2024)
2024
-
[20]
arXiv preprint arXiv:2307.13018 (2023)
Tarassow, A.: The potential of llms for coding with low-resource and domain-specific programming languages. arXiv preprint arXiv:2307.13018 (2023)
2023 arXiv
-
[21]
arXiv preprint arXiv:2408.00118 (2024)
Team, G., Riviere, M., Pathak, S., Sessa, P.G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al.: Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)
2024 arXiv
-
[22]
arXiv preprint arXiv:2305.07922 (2023)
Wang, Y ., Le, H., Gotmare, A.D., Bui, N.D., Li, J., Hoi, S.C.: Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023)
2023 arXiv
-
[23]
Advances in neural information processing systems 35, 24824–24837 (2022)
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V ., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)
2022
-
[24]
In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming
Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V .J.: A systematic evaluation of large language models of code. In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. pp. 1–10 (2022)
2022
-
[25]
arXiv preprint arXiv:2309.06275 (2023)
Xu, X., Tao, C., Shen, T., Xu, C., Xu, H., Long, G., Lou, J.g.: Re-reading improves reasoning in language models. arXiv preprint arXiv:2309.06275 (2023)
2023 arXiv
-
[26]
arXiv preprint arXiv:2211.12561 (2022)
Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.t.: Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561 (2022)
2022 arXiv
-
[27]
arXiv preprint arXiv:2303.17568 (2023) 12
Zheng, Q., Xia, X., Zou, X., Dong, Y ., Wang, S., Xue, Y ., Wang, Z., Shen, L., Wang, A., Li, Y ., et al.: Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023) 12
2023 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.