{"id":"e8235602-fde4-4709-a87c-f71d1daef9f2","arxiv_id":"2412.12513","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A concept-based knowledge graph and iterative planning, coding, and debugging agents improve LLM-generated Move smart contracts, with reported relative gains up to 47 percent over baselines.","lead":"ConMover is a multi-agent framework that helps large language models write Move smart contracts by pulling concepts from a knowledge graph and iteratively debugging generated code. The paper reports large accuracy gains over standard prompting, especially for small models, but it does not release code or data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim depends on an unverified train/test split: ConMoverLM is fine-tuned on faulty Move code drawn from the same scraped Move dataset used for the 780-test evaluation, so the reported self-refinement gains may be in-distribution artifacts.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the evaluation may not be out-of-distribution because the fine-tuning corpus and the test set are both drawn from the same scraped Move dataset, with no reported split. I agree with that assessment. The central claim is not a theoretical construction or a machine-checked proof; it is an empirical accuracy improvement, so the fairness and disjointness of the evaluation are decisive. The paper provides no code, no data, no model weights, no error bars, and cites a non-code taxonomy as the data source, which compounds the uncertainty. Given these issues, the reader's REJECT verdict is appropriate. A revised version that releases artifacts and demonstrates a clean train/test split, ideally with an independent held-out evaluation, could change the verdict, but as written the central claim is not supported.","tokens_in":10386,"tokens_out":2816,"duration_ms":26278,"concrete_test":"Request that the authors release the full data pipeline, then compute the overlap between the 313 faulty examples used to create the fine-tuning corpus (Section 2.2/5.3) and the 780-task evaluation set (Section 7). If the Jaccard overlap of task descriptions or code examples is greater than zero, rerun Tables 1 and 2 on the disjoint held-out subset and compare. A cleaner alternative: run ConMoverLM on a fresh set of Move tasks from the Sui examples repository that were explicitly excluded from all fine-tuning and report the self-refinement gain over direct generation; if the gain drops materially, the reported improvements are in-distribution artifacts. Either check would settle whether the central empirical claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is empirical: ConMover's concept-based planning and self-refinement substantially improve Move code generation. The most load-bearing condition is that the evaluation set is disjoint from all data used to build the framework. The text does not establish this. Section 2.2 says 313 code examples from the dataset [2] are used to create the wrong-solution corpus; Section 5.3 says ConMoverLM is fine-tuned on faulty code execution traces and compilation errors from that corpus; Section 7 evaluates on 'our scraped dataset in Move[2]' with 780 tests. No train/test split is reported anywhere. Furthermore, reference [2] is electric-capital/crypto-ecosystems, a repository taxonomy of blockchain ecosystems, not a Move code benchmark, which makes it unclear what the evaluation set actually contains and undermines the claim that it is a meaningful out-of-distribution test. If the 313 fine-tuning seeds overlap with the 780 evaluation tasks, the accuracy gains in Tables 1 and 2 are partly memorization rather than generalization. Since no code, data, or model weights are released, this contamination risk cannot currently be checked post hoc. The one-time generation improvements are small or even slightly negative (e.g., Gemma-2 9B goes from 21.9 to 21.4 in Table 1), so the entire headline result rests on the self-refinement numbers being computed on tasks that may have appeared in the fine-tuning corpus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConMover, a multi-agent framework for generating Move smart contracts. It builds a knowledge graph from Sui Move documentation, uses concept retrieval, planning, coding, and debugging agents, and fine-tunes ConMoverLM on faulty code and execution traces. The framework is evaluated on a scraped dataset of 780 Move tasks with several open-source LLMs, and the paper reports improvements in both one-time generation and iterative self-refinement settings.","tokens_in":10716,"tokens_out":5447,"duration_ms":46510,"significance":"If the reported numbers are valid, the framework would be a useful contribution to low-resource code generation: it suggests that a small open-weight model can be augmented to outperform direct use of larger proprietary models (e.g., Gemini 1.5 002 rising from 37.8 to 56.9 in Table 2). The paper's strengths include a clearly described multi-agent pipeline, comparison against chain-of-thought, self-planning, and self-debugging baselines, and a deliberate focus on small models. However, the paper ships no code, data, or model weights, reports no train/test split, and the evaluation is the sole support for the main claims; in its present form the evidence is not verifiable.","major_comments":[{"comment":"The evaluation is not shown to be disjoint from the training data. Section 2.2 states that 313 code examples 'from the whole dataset' are used to create the wrong-solution corpus; Section 5.3 states that ConMoverLM is fine-tuned on faulty code, execution traces, and compilation errors from that corpus; Section 7 evaluates on 'our scraped dataset in Move[2]' with 780 tests. No train/test split is reported. Because the fine-tuned model is the self-refinement engine, the gains in Tables 1 and 2 could reflect memorization of training tasks rather than generalization. The paper must report the split, release the data, and re-run the evaluation on a held-out set.","section":"Sections 2.2, 5.3, and 7"},{"comment":"The data source cited for the evaluation is electric-capital/crypto-ecosystems, a taxonomy of blockchain ecosystems, not a Move code benchmark. This makes it impossible to determine what the 780 tests contain, how they were generated, or whether they are independent of the fine-tuning examples. The authors should replace this citation with the actual benchmark or dataset and describe its composition, including how the 313 faulty examples were selected and how they relate to the 780 test tasks.","section":"Section 7, reference [2]"},{"comment":"The one-time generation results do not support a general improvement claim. Gemma-2 9B decreases from 21.9 to 21.4 with ConMover, CodeLlama-7B stays at 18.3, and Gemma-2 2B improves only from 12.2 to 14.0. The headline result therefore rests on the self-refinement numbers, which are exactly the numbers most exposed to the contamination risk described in the first major comment. No error bars or significance tests are provided for any of the tables, so the reader cannot tell whether differences such as 23.8 versus 29.3 are stable or due to sampling noise.","section":"Table 1"},{"comment":"The fine-tuning and self-refinement procedure is underspecified for reproducibility. The paper reports hyperparameters (batch size 512 examples, top-p 0.95, learning rates, 5% masking, 25% self-refined data), but it does not state the number of training examples, the source of canonical solutions, the exact split between faulty-code fine-tuning and instruction tuning, or how the roughly 313 examples relate to the 780-task test set. No model weights or code are released. Without this information the experimental claims cannot be independently checked.","section":"Sections 5.2, 5.3, and 6.2"}],"minor_comments":[{"comment":"The framework name is written as 'CoMover' at the start of Section 4; the rest of the paper uses 'ConMover', so the spelling should be unified.","section":"Section 4"},{"comment":"The sentence 'comparing its performance to MOve for various sizes' should read 'Move'; the same typo appears in the first line of the evaluation section.","section":"Section 7"},{"comment":"The phrase 'apros chain' should be corrected to 'Aptos chain'.","section":"Section 2.1"},{"comment":"The passage 'We conducted evaluations using the Pass@1 metric, In the set k iterations (10 in our experiment)' is grammatically unclear; please define Pass@1 and clarify the relationship between the 10 iterations and the four or five refinement attempts described in Section 5.3.","section":"Section 7"},{"comment":"The sentence 'Instead of training a new code language model (LM) from the ground up,We also compared different self-refining techniques and ours on the state-of-the-art models.ed a vast amount of information about code structure, syntax, and common patterns' is garbled and should be rewritten.","section":"Section 7"},{"comment":"The table heading says 'Move Test (780 Tests)' and the columns are 'One-time' and 'Self-Refine', but the units are not explicit; clarify that the numbers are percentages (Pass@1).","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early preprint with severe evaluation gaps. The reader's stress-test concern about train/test overlap is well-founded: the fine-tuning corpus is drawn from the same dataset used for evaluation, the cited data source is not a code benchmark, and no code or data are released. I would not recommend acceptance without a fully re-run evaluation on a clearly held-out dataset, corrected data citations, and release of code and data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead ConMover. The architecture is a sensible integration of known pieces: a knowledge graph over Move docs, a planner that generates multiple plans with confidence scores, a coder, and a debugger that uses execution feedback, all orchestrated in an iterative loop within MapCoder. The genuinely new bit is the idea of using the extracted concepts as feedback for self-refinement rather than as direct context, and the ConMoverLM fine-tuning on faulty code plus execution traces. That's a plausible direction for low-resource languages.\n\nWhat the paper does well: it gives a clear account of the agents and the training scheme, compares against reasonable baselines (COT, self-planning, self-debugging, RAG with correct code), and spans model sizes from 2B to 9B. The finding that concepts help most as a debugging signal is interesting and worth probing.\n\nThe problem is the evidence. The paper fine-tunes ConMoverLM on faulty Move code and execution traces collected from a corpus (Section 5.3), then evaluates on 'our scraped dataset in Move' (Section 7) with 780 tests, and never reports a train/test split. The data source cited for that corpus is [2], which is electric-capital/crypto-ecosystems — a taxonomy of blockchain ecosystems, not a code benchmark. That mismatch makes it hard to know what the evaluation set actually contains. If the 313 seed examples used for fine-tuning overlap with the 780 test tasks, the self-refinement gains in Tables 1 and 2 are in-distribution artifacts. Since no code, data, or weights are released, the contamination risk can't be checked. One-time generation gains are small or slightly negative (Gemma-2 9B: 21.9 to 21.4), so the entire headline rests on the self-refinement numbers, which are exactly the ones most exposed to this leak.\n\nThere are also smaller issues: the paper promises an RL reward scheme in the intro but never describes it in the methods; there are typos and a few unfinished sentences. Those are fixable, not fatal.\n\nNet: the engineering contribution is plausible and the idea is worth pursuing, but as written the central empirical claim is not supported because the evaluation's independence from the training data is not established. This deserves a serious referee to sort out whether the authors can release artifacts and clarify the split, but I would not take the accuracy numbers at face value.","headline":"Plausible engineering framework for low-resource code generation, but the evaluation doesn't establish the central claim because the train/test split is unverified and the cited data source is a taxonomy, not a code benchmark.","tokens_in":11225,"tokens_out":2465,"would_cite":false,"duration_ms":20741,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConMover claims that Move smart contracts can be generated from natural language using a concept knowledge graph and iterative self-refinement, lifting small open LLMs' pass rates by up to about 47 percent relative to their self-refined…","keywords":["Move","smart contracts","code generation","large language models","self-refinement","knowledge graph","retrieval-augmented generation","low-resource languages"],"falsifier":"Run ConMover's self-refinement on a fresh Move benchmark whose tasks are provably disjoint from the 313 faulty examples used for fine-tuning and from the 20 example categories; if the pass-rate improvements over the direct baseline shrink to near zero or fall below the RAG-with-correct-code baseline, the reported gains are in-distribution artifacts rather than general self-correction.","tokens_in":10168,"feed_emoji":"📈","tokens_out":9338,"duration_ms":78588,"temperature":0.7,"pith_summary":"This paper tries to establish that a smart-contract language with little training data, Move, can still be generated reliably from natural language if the model is guided by a knowledge graph of concepts and a small set of correct examples rather than by a large code corpus. The proposed framework, ConMover, chains planning, coding, and debugging agents, and fine-tunes a code model on faulty code together with compiler execution traces so the model learns to repair its own output. The paper reports that self-refinement raises pass rates on 780 Move tests from 12.2 percent to 20.7 percent for the smallest open model tested, and from 37.8 percent to 56.9 percent for a large proprietary model. If the evaluation is genuinely out-of-distribution, this would mean low-resource language generation is more about feedback and concept structure than about parameter count or training data size.","feed_headline":"Self-refinement lifts Move code accuracy up to 47 percent","feed_subtitle":"ConMover's concept graph and debug loop improve even a 2B open model, without large Move training corpora.","key_machinery":"The load-bearing machinery is a three-part loop: a knowledge graph of Move concepts extracted from official documentation, a small set of correct Move examples used by a planning agent that emits confidence-scored plans, and a debug-refinement stage in which a model fine-tuned on faulty code paired with compiler execution traces rewrites the failed code through a fixed template that concatenates the natural-language description, the faulty code, and the execution feedback. The knowledge graph is also used as a retrieval-augmented validator during planning, and the whole pipeline is orchestrated as an adaptive traversal that cycles through the next-best plan when debugging exhausts its refinement budget.","core_discovery":"On its own terms, the paper claims that ConMover can generate Move smart contracts based on concepts without pre-training an LLM on a vast existing code corpus. The discovery is that a knowledge graph of Move concepts, built from documentation, plus roughly 20 categories of correct Move examples, can substitute for large training corpora when they are injected through a planning agent and when a small code model is fine-tuned to self-refine using faulty code and compiler feedback. The reported evidence is a set of pass-rate comparisons: on a 780-test Move dataset, ConMover's self-refinement improves the 2B-parameter open model from 12.2 percent to 20.7 percent, the 7B model from 18.3 percent to 22.0 percent, and the 9B model from 21.4 percent to 29.3 percent; for two large proprietary models it improves direct pass rates from 27.8 percent to 37.6 percent and from 37.8 percent to 56.9 percent.","pith_inferences":["Editorial inference: if the train/test separation is clean, the same concept-graph-plus-debug-loop recipe could plausibly transfer to other verification-oriented low-resource languages by swapping the documentation source and the fine-tuning traces, though the paper only demonstrates Move.","Editorial inference: the design suggests a testable extension where static-analysis warnings or formal-verification counterexamples replace runtime failures as the debugging feedback, which could matter for safety-critical smart contracts.","Editorial inference: because the largest relative gains appear on the smallest model, ConMover points toward on-device or privacy-preserving code generation with small open-weight models, a direction the paper does not evaluate.","Editorial inference: the claim would be strengthened by an ablation that disables confidence-ranked planning; if random plan order performs about as well, the ranking mechanism is not carrying the improvement."],"forward_implications":["Fine-tuning a code model on faulty code plus execution feedback teaches self-refinement that does not arise from pre-training alone, since adding RAG with correct code gives much smaller gains.","Small open models can match or approach the one-time pass rates of much larger models once they are equipped with ConMover's planning and debugging loop, so deployment with a 2B-parameter model becomes practical.","A knowledge base of roughly 20 correctly coded categories and a concept graph from documentation can be enough to ground code generation in a low-resource language, lowering data-engineering cost.","The improvement transfers to large proprietary models when the same planning and self-refinement pipeline is applied, suggesting the method is model-agnostic rather than tied to the fine-tuned open model.","Repeated self-refinement with compiler feedback converges within a few iterations, so the accuracy gain does not require unbounded inference cost."],"supporting_citations":[{"why":"Supplies the base instruction-tuning set that is modified with compilation errors to train the self-refinement model.","marker":"[1]"},{"why":"Provides the task descriptions and the 313 faulty-code seed examples used to build the dataset that is later evaluated as a 780-test Move set.","marker":"[2]"},{"why":"Is the stated sole source of Move documentation converted into the knowledge graph and the retrieval-augmented validator.","marker":"[3]"},{"why":"Supplies the small set of correct Move examples used as planning schemas by the planner agent.","marker":"[4]"},{"why":"Defines the self-debugging prompting method that ConMover compares against and whose iterative feedback idea it extends.","marker":"[8]"},{"why":"Supplies the self-planning code-generation approach that ConMover builds on for multiple confidence-scored plans.","marker":"[11]"},{"why":"Is the prior concept-based translation work for Move that motivates the framework and serves as a comparison point.","marker":"[13]"},{"why":"Is one of the open code models used as a base for ConMover and as a baseline in the evaluation.","marker":"[17]"},{"why":"Provides the knowledge-graph construction technique used to create the Move concept knowledge base.","marker":"[19]"},{"why":"Provides the small and medium open models used as bases and baselines in the evaluation.","marker":"[21]"}],"fun_headline_variants":["Concept graph lifts Move code accuracy across LLMs","ConMover boosts Move code generation with knowledge graph","Self-refinement improves Move contracts without big data","Move code generation enhanced by concept planning and debug","Concept-driven approach narrows Move training data gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on the assumption that the 780-test evaluation set does not overlap with the 313 faulty examples used to build the fine-tuning data, since the paper reports no train/test split.","fun_headline_variants_meta":{"raw":{"variants":["Concept graph lifts Move code accuracy across LLMs","ConMover boosts Move code generation with knowledge graph","Self-refinement improves Move contracts without big data","Move code generation enhanced by concept planning and debug","Concept-driven approach narrows Move training data gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2717,"prompt_tokens":874,"completion_tokens":1843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":1771}},"tokens_in":490,"tokens_out":1843,"duration_ms":15303,"temperature":1.0,"reasoning_tokens":1771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:59:58.951735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ConMover's self-refinement on a fresh Move benchmark whose tasks are provably disjoint from the 313 faulty examples used for fine-tuning and from the 20 example categories; if the pass-rate improvements over the direct baseline shrink to near zero or fall below the RAG-with-correct-code baseline, the reported gains are in-distribution artifacts rather than general self-correction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base instruction-tuning set that is modified with compilation errors to train the self-refinement model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the task descriptions and the 313 faulty-code seed examples used to build the dataset that is later evaluated as a 780-test Move set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the stated sole source of Move documentation converted into the knowledge graph and the retrieval-augmented validator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the small set of correct Move examples used as planning schemas by the planner agent."},{"cited_title":"ACM Transactions on Software Engineering and Methodology (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the self-planning code-generation approach that ConMover builds on for multiple confidence-scored plans."},{"cited_title":"In: Proceedings of the 1st ACM International Conference on AI-Powered Software","cited_arxiv_id":null,"evidence_quote":"Is the prior concept-based translation work for Move that motivates the framework and serves as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the knowledge-graph construction technique used to create the Move concept knowledge base."}],"review_version":1}