Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Generating Move Smart Contracts based on Concepts

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read ConMover claims that Move smart contracts can be generated from natural language using a concept knowledge graph and iterative self-refinement, lifting small open LLMs' pass rates by up to about 47 percent relative to their self-refined…

desk verdict Plausible engineering framework for low-resource code generation, but the evaluation doesn't establish the central claim because the train/test split is unverified and the cited data source is a taxonomy, not a code benchmark. read the letter →

arxiv 2412.12513 v1 pith:TKB43EOM submitted 2024-12-17 cs.SE

classification cs.SE
keywords Movesmartcontractscodegenerationlargelanguagemodelsself-refinementknowledgegraphretrieval-augmentedlow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a smart-contract language with little training data, Move, can still be generated reliably from natural language if the model is guided by a knowledge graph of concepts and a small set of correct examples rather than by a large code corpus. The proposed framework, ConMover, chains planning, coding, and debugging agents, and fine-tunes a code model on faulty code together with compiler execution traces so the model learns to repair its own output. The paper reports that self-refinement raises pass rates on 780 Move tests from 12.2 percent to 20.7 percent for the smallest open model tested, and from 37.8 percent to 56.9 percent for a large proprietary model. If the evaluation is genuinely out-of-distribution, this would mean low-resource language generation is more about feedback and concept structure than about parameter count or training data size.

What carries the argument

The load-bearing machinery is a three-part loop: a knowledge graph of Move concepts extracted from official documentation, a small set of correct Move examples used by a planning agent that emits confidence-scored plans, and a debug-refinement stage in which a model fine-tuned on faulty code paired with compiler execution traces rewrites the failed code through a fixed template that concatenates the natural-language description, the faulty code, and the execution feedback. The knowledge graph is also used as a retrieval-augmented validator during planning, and the whole pipeline is orchestrated as an adaptive traversal that cycles through the next-best plan when debugging exhausts its refinement budget.

What would settle it

Run ConMover's self-refinement on a fresh Move benchmark whose tasks are provably disjoint from the 313 faulty examples used for fine-tuning and from the 20 example categories; if the pass-rate improvements over the direct baseline shrink to near zero or fall below the RAG-with-correct-code baseline, the reported gains are in-distribution artifacts rather than general self-correction.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that ConMover can generate Move smart contracts based on concepts without pre-training an LLM on a vast existing code corpus. The discovery is that a knowledge graph of Move concepts, built from documentation, plus roughly 20 categories of correct Move examples, can substitute for large training corpora when they are injected through a planning agent and when a small code model is fine-tuned to self-refine using faulty code and compiler feedback. The reported evidence is a set of pass-rate comparisons: on a 780-test Move dataset, ConMover's self-refinement improves the 2B-parameter open model from 12.2 percent to 20.7 percent, the 7B model from 18.3 percent to 22.0 percent, and the 9B model from 21.4 percent to 29.3 percent; for two large proprietary models it improves direct pass rates from 27.8 percent to 37.6 percent and from 37.8 percent to 56.9 percent.

Load-bearing premise

The results stand or fall on the assumption that the 780-test evaluation set does not overlap with the 313 faulty examples used to build the fine-tuning data, since the paper reports no train/test split.

Editorial extensions

If this is right

  • Fine-tuning a code model on faulty code plus execution feedback teaches self-refinement that does not arise from pre-training alone, since adding RAG with correct code gives much smaller gains.
  • Small open models can match or approach the one-time pass rates of much larger models once they are equipped with ConMover's planning and debugging loop, so deployment with a 2B-parameter model becomes practical.
  • A knowledge base of roughly 20 correctly coded categories and a concept graph from documentation can be enough to ground code generation in a low-resource language, lowering data-engineering cost.
  • The improvement transfers to large proprietary models when the same planning and self-refinement pipeline is applied, suggesting the method is model-agnostic rather than tied to the fine-tuned open model.
  • Repeated self-refinement with compiler feedback converges within a few iterations, so the accuracy gain does not require unbounded inference cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the train/test separation is clean, the same concept-graph-plus-debug-loop recipe could plausibly transfer to other verification-oriented low-resource languages by swapping the documentation source and the fine-tuning traces, though the paper only demonstrates Move.
  • Editorial inference: the design suggests a testable extension where static-analysis warnings or formal-verification counterexamples replace runtime failures as the debugging feedback, which could matter for safety-critical smart contracts.
  • Editorial inference: because the largest relative gains appear on the smallest model, ConMover points toward on-device or privacy-preserving code generation with small open-weight models, a direction the paper does not evaluate.
  • Editorial inference: the claim would be strengthened by an ablation that disables confidence-ranked planning; if random plan order performs about as well, the ranking mechanism is not carrying the improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ConMover, a multi-agent framework for generating Move smart contracts. It builds a knowledge graph from Sui Move documentation, uses concept retrieval, planning, coding, and debugging agents, and fine-tunes ConMoverLM on faulty code and execution traces. The framework is evaluated on a scraped dataset of 780 Move tasks with several open-source LLMs, and the paper reports improvements in both one-time generation and iterative self-refinement settings.

Significance. If the reported numbers are valid, the framework would be a useful contribution to low-resource code generation: it suggests that a small open-weight model can be augmented to outperform direct use of larger proprietary models (e.g., Gemini 1.5 002 rising from 37.8 to 56.9 in Table 2). The paper's strengths include a clearly described multi-agent pipeline, comparison against chain-of-thought, self-planning, and self-debugging baselines, and a deliberate focus on small models. However, the paper ships no code, data, or model weights, reports no train/test split, and the evaluation is the sole support for the main claims; in its present form the evidence is not verifiable.

major comments (4)
  1. [Sections 2.2, 5.3, and 7] The evaluation is not shown to be disjoint from the training data. Section 2.2 states that 313 code examples 'from the whole dataset' are used to create the wrong-solution corpus; Section 5.3 states that ConMoverLM is fine-tuned on faulty code, execution traces, and compilation errors from that corpus; Section 7 evaluates on 'our scraped dataset in Move[2]' with 780 tests. No train/test split is reported. Because the fine-tuned model is the self-refinement engine, the gains in Tables 1 and 2 could reflect memorization of training tasks rather than generalization. The paper must report the split, release the data, and re-run the evaluation on a held-out set.
  2. [Section 7, reference [2]] The data source cited for the evaluation is electric-capital/crypto-ecosystems, a taxonomy of blockchain ecosystems, not a Move code benchmark. This makes it impossible to determine what the 780 tests contain, how they were generated, or whether they are independent of the fine-tuning examples. The authors should replace this citation with the actual benchmark or dataset and describe its composition, including how the 313 faulty examples were selected and how they relate to the 780 test tasks.
  3. [Table 1] The one-time generation results do not support a general improvement claim. Gemma-2 9B decreases from 21.9 to 21.4 with ConMover, CodeLlama-7B stays at 18.3, and Gemma-2 2B improves only from 12.2 to 14.0. The headline result therefore rests on the self-refinement numbers, which are exactly the numbers most exposed to the contamination risk described in the first major comment. No error bars or significance tests are provided for any of the tables, so the reader cannot tell whether differences such as 23.8 versus 29.3 are stable or due to sampling noise.
  4. [Sections 5.2, 5.3, and 6.2] The fine-tuning and self-refinement procedure is underspecified for reproducibility. The paper reports hyperparameters (batch size 512 examples, top-p 0.95, learning rates, 5% masking, 25% self-refined data), but it does not state the number of training examples, the source of canonical solutions, the exact split between faulty-code fine-tuning and instruction tuning, or how the roughly 313 examples relate to the 780-task test set. No model weights or code are released. Without this information the experimental claims cannot be independently checked.
minor comments (6)
  1. [Section 4] The framework name is written as 'CoMover' at the start of Section 4; the rest of the paper uses 'ConMover', so the spelling should be unified.
  2. [Section 7] The sentence 'comparing its performance to MOve for various sizes' should read 'Move'; the same typo appears in the first line of the evaluation section.
  3. [Section 2.1] The phrase 'apros chain' should be corrected to 'Aptos chain'.
  4. [Section 7] The passage 'We conducted evaluations using the Pass@1 metric, In the set k iterations (10 in our experiment)' is grammatically unclear; please define Pass@1 and clarify the relationship between the 10 iterations and the four or five refinement attempts described in Section 5.3.
  5. [Section 7] The sentence 'Instead of training a new code language model (LM) from the ground up,We also compared different self-refining techniques and ours on the state-of-the-art models.ed a vast amount of information about code structure, syntax, and common patterns' is garbled and should be rewritten.
  6. [Table 1] The table heading says 'Move Test (780 Tests)' and the columns are 'One-time' and 'Self-Refine', but the units are not explicit; clarify that the numbers are percentages (Pass@1).

Circularity Check

1 steps flagged · score 6.0 of 10

Self-refinement gains may be in-distribution: ConMoverLM is fine-tuned on faulty Move code from the same scraped dataset [2] that is later used as the 780-test evaluation set, with no reported train/test split.

  1. fitted input called prediction [Sections 2.2, 5.3, 7 and Table 1]
    "We use [2] as our coding data source to generate task descriptions for the LLMs to solve. We use 313 code examples from the whole dataset to create our wrong solution dataset. ... ConMoverLM is fine-tuned on faulty code execution traces and its compilation errors. ... This section evaluates ConMover’s code generation capabilities, comparing its performance to MOve for various sizes across our scraped dataset in Move[2]."

    Fine-tuning input and evaluation output are drawn from the same source: the 313 code examples used to build the faulty-code/trace corpus come from dataset [2], and the 780-test evaluation is described as 'our scraped dataset in Move[2].' The paper never reports a split between the fine-tuning examples and the test tasks. Consequently the self-refinement accuracy in Tables 1 and 2 is not shown to be an out-of-distribution prediction; it is a score on data whose overlap with the training corpus is neither documented nor controlled. If any of the 780 test tasks correspond to the 313 fine-tuning seeds (or to tasks generated from the same descriptions), the reported improvement is forced by memorization rather than measured generalization.

full rationale

The central empirical claim is that ConMover's self-refinement substantially improves Move code generation. The load-bearing evidence is Table 1's self-refinement column and Table 2's ConMover results. The derivation chain for that evidence is: (i) Section 2.2 creates a wrong-solution corpus from 313 code examples scraped from [2]; (ii) Section 5.3 fine-tunes ConMoverLM on the resulting faulty code, execution traces, and compilation errors; (iii) Section 7 and Table 1 score ConMover on 'our scraped dataset in Move[2]' with 780 tests. No train/test split, no overlap analysis, and no external benchmark is reported. Thus the 'prediction' of self-refinement accuracy is not shown to be independent of the fine-tuning input; if the 780 tests include any of the 313 seeds (or tasks generated from the same source), the gains are in-distribution and partly forced. This is a fitted-input-called-prediction concern rather than a definitional identity. I did not find other load-bearing circularity: self-citations to [13] are motivational or comparative and do not by themselves force the accuracy result; no uniqueness theorem or ansatz-by-citation is used. The severity is capped at 6 because the paper's text does not explicitly state that the test tasks are identical to the training tasks; it simply omits the split, leaving the reduction conditional. If a clean split is later documented, this circularity concern would disappear.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework is a software system, not a mathematical derivation, so there are no physical or mathematical entities. The main ledger entries are hand-chosen hyperparameters and domain assumptions about the validity and coverage of the Move compiler, documentation, and test suite.

free parameters (3)
  • top_p (nucleus sampling) = 0.95
    Set by hand in Section 6.2 for all ConMover generations; it controls output diversity and is not derived from theory.
  • maximum refinement iterations = 4 to 5
    Section 6.2 limits inference self-refinement to 4 steps while Section 5.3 says the iterative process goes through five times; the choice affects the reported Pass@1 results.
  • self-refinement training mixture = 25 percent self-refined samples, 5 percent masked input
    Chosen in Section 6.2 to build the fine-tuning set; the reported gains depend on this manually chosen mix.
assumptions (3)
  • domain assumption Compiler and unit tests are a correct and complete oracle for Move code quality.
    The framework treats passing visible and hidden test cases as success (Sections 4.1, 5.3, and 7), but no evidence is given that the scraped test suite covers the intended behavior or that the tests are not flawed.
  • ad hoc to paper The Sui Move documentation plus 20 example categories cover the concepts needed for all 780 test tasks.
    Sections 2.1 and 8.2 describe the knowledge base as built from reference [3] and 20 official Move categories, and the framework's retrieval relies on this coverage; the paper does not measure concept coverage.
  • ad hoc to paper Fine-tuning on faulty Move code and execution traces transfers to unseen Move tasks.
    Section 5.3 trains ConMoverLM on faulty code plus traces, and Section 8.3 interprets the gains as a general corrective capability; the transfer claim is load-bearing but not tested on a disjoint benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Move Smart Contracts based on Concepts." pith.science (2026). https://pith.science/paper/TKB43EOM

@misc{pith2026241212513,
  author       = {Pith},
  title        = {Pith review of: Generating Move Smart Contracts based on Concepts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKB43EOM}},
  note         = {Machine review of arXiv:2412.12513}
}
read the original abstract

The growing adoption of formal verification for smart contracts has spurred the development of new verifiable languages like Move. However, the limited availability of training data for these languages hinders effective code generation by large language models (LLMs). This paper presents ConMover, a novel framework that enhances LLM-based code generation for Move by leveraging a knowledge graph of Move concepts and a small set of verified code examples. ConMover integrates concept retrieval, planning, coding, and debugging agents in an iterative process to refine generated code. Evaluations with various open-source LLMs demonstrate substantial accuracy improvements over baseline models. These results underscore ConMover's potential to address low-resource code generation challenges, bridging the gap between natural language descriptions and reliable smart contract development.

Figures

Figures reproduced from arXiv: 2412.12513 by the authors.

Figure 1
Figure 1. Concept Generation [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Planning Agent 4 Architecture of ConMover CoMover adopts an agentic approach of holistically generating move code. Each agent has multiple parts inside it not only a LLM. 4.1 Planner Agent The planner agent encapsulated the concept generator inside it. Its workflow looks [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Feedback Loop any additional information. During the debug stage, however, the system matches and retrieves relevant samples from the memory pool based on task similarity. The matching mechanism considers the following key components: Q: The provided task description. P: The plan generated by the Planning Agent. K: A set of keywords extracted from the task description (Q) and the generated code (C). E: The execution… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Template The quality of code in these datasets varies significantly. High-quality code, often from well-maintained projects, aligns well with its documentation, while noisy code, potentially from less experienced developers or unfinished projects, may contain vulnerabi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 12 canonical work pages

  1. [2]

    electric-capital/crypto-ecosystems: A taxonomy for open source cryptocurrency, blockchain, and decentralized ecosystems (2024), https://github.com/electric-capital/crypto-ecosystems

  2. [1]

    blog/starchat-alpha.md at main · huggingface/blog · github (2024), https://github.com/huggingface/ blog/blob/main/starchat-alpha.md

  3. [3]

    Move concepts | sui documentation (2024), https://docs.sui.io/concepts/sui-move-concepts

  4. [4]

    sui/examples/move at main · mystenlabs/sui (2024), https://github.com/MystenLabs/sui/tree/main/ examples/move

  5. [5]

    arXiv preprint arXiv:2108.07732 (2021)

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

  6. [6]

    Libra Assoc p

    Blackshear, S., Cheng, E., Dill, D.L., Gao, V ., Maurer, B., Nowacki, T., Pott, A., Qadeer, S., Rain, D.R., Sezer, S., et al.: Move: A language with programmable resources. Libra Assoc p. 1 (2019)

  7. [7]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

  8. [8]

    arXiv preprint arXiv:2304.05128 (2023)

    Chen, X., Lin, M., Schärli, N., Zhou, D.: Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 (2023)

Show all 27 references
  1. [9]

    arXiv preprint arXiv:2405.11514 (2024)

    Eniser, H.F., Zhang, H., David, C., Wang, M., Paulsen, B., Dodds, J., Kroening, D.: Towards translating real- world code with llms: A study of translating to rust. arXiv preprint arXiv:2405.11514 (2024)

  2. [10]

    In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security

    He, J., Vechev, M.: Large language models for code: Security hardening and adversarial testing. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. pp. 1865–1879 (2023)

  3. [11]

    ACM Transactions on Software Engineering and Methodology (2023)

    Jiang, X., Dong, Y ., Wang, L., Zheng, F., Shang, Q., Li, G., Jin, Z., Jiao, W.: Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology (2023)

  4. [12]

    arXiv preprint arXiv:2407.05202 (2024)

    Karanjai, R., Hussain, A., Rabin, M.R.I., Xu, L., Shi, W., Alipour, M.A.: Harnessing the power of llms: Au- tomating unit test generation for high-performance computing. arXiv preprint arXiv:2407.05202 (2024)

  5. [13]

    In: Proceedings of the 1st ACM International Conference on AI-Powered Software

    Karanjai, R., Xu, L., Shi, W.: Solmover: Smart contract code translation based on concepts. In: Proceedings of the 1st ACM International Conference on AI-Powered Software. pp. 112–121 (2024)

  6. [14]

    arXiv preprint arXiv:2211.15533 (2022) 11 ConMover: Generating Move Smart Contracts based on Concepts A PREPRINT

    Kocetkov, D., Li, R., Allal, L.B., Li, J., Mou, C., Ferrandis, C.M., Jernite, Y ., Mitchell, M., Hughes, S., Wolf, T., et al.: The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533 (2022) 11 ConMover: Generating Move Smart Contracts based on Con...

  7. [15]

    Science 378(6624), 1092–1097 (2022)

    Li, Y ., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al.: Competition-level code generation with alphacode. Science 378(6624), 1092–1097 (2022)

  8. [16]

    arXiv preprint arXiv:2203.13474 (2022)

    Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y ., Savarese, S., Xiong, C.: Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)

  9. [17]

    arXiv preprint arXiv:2308.12950 (2023)

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

  10. [18]

    Advances in Neural Information Processing Systems 36 (2024)

    Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., Scialom, T.: Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36 (2024)

  11. [19]

    Sun, Q., Luo, Y ., Zhang, W., Li, S., Li, J., Niu, K., Kong, X., Liu, W.: Docs2kg: Unified knowledge graph construction from heterogeneous documents assisted by large language models (2024)

  12. [20]

    arXiv preprint arXiv:2307.13018 (2023)

    Tarassow, A.: The potential of llms for coding with low-resource and domain-specific programming languages. arXiv preprint arXiv:2307.13018 (2023)

  13. [21]

    arXiv preprint arXiv:2408.00118 (2024)

    Team, G., Riviere, M., Pathak, S., Sessa, P.G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al.: Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

  14. [22]

    arXiv preprint arXiv:2305.07922 (2023)

    Wang, Y ., Le, H., Gotmare, A.D., Bui, N.D., Li, J., Hoi, S.C.: Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023)

  15. [23]

    Advances in neural information processing systems 35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V ., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837 (2022)

  16. [24]

    In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming

    Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V .J.: A systematic evaluation of large language models of code. In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. pp. 1–10 (2022)

  17. [25]

    arXiv preprint arXiv:2309.06275 (2023)

    Xu, X., Tao, C., Shen, T., Xu, C., Xu, H., Long, G., Lou, J.g.: Re-reading improves reasoning in language models. arXiv preprint arXiv:2309.06275 (2023)

  18. [26]

    arXiv preprint arXiv:2211.12561 (2022)

    Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.t.: Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561 (2022)

  19. [27]

    arXiv preprint arXiv:2303.17568 (2023) 12

    Zheng, Q., Xia, X., Zou, X., Dong, Y ., Wang, S., Xue, Y ., Wang, Z., Shen, L., Wang, A., Li, Y ., et al.: Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023) 12

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.