{"id":"63af7e77-14a7-46cb-954a-d9a0a7052318","arxiv_id":"2505.08686","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned LLM converts template-style text prompts into ezdxf Python code that produces editable, annotated Dxf CAD files.","lead":"CAD-Coder turns natural language prompts into Python script code that generates editable Dxf CAD files, including dimension annotations. The authors build a synthetic dataset of 29,130 text-code-Dxf triples and fine-tune an 8B Llama model with LoRA, reporting gains over generic LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported scores are measured on prompts sampled from the same parent-template generator as training, so they may demonstrate template inversion rather than robust text-guided CAD generation.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: training and evaluation prompts are generated from the same parent-code templates, so the reported metrics do not establish generalization to arbitrary natural language or unseen shape classes. I agree with that assessment. The central claim requires evidence that the model can interpret new phrasings and new geometric configurations; the current protocol, which randomly samples test prompts from the same synthetic pool used for training, cannot provide such evidence. This is not an internal inconsistency in the pipeline, but it is a correctness risk for the advertised capability: the model may be a high-quality template instantiator rather than a general text-to-CAD system. The missing comparisons to Text2CAD, CAD-LLM, and other text-to-CAD methods strengthen the concern about the 'first interactive model' claim, though they are secondary to the evaluation-distribution issue. Because the flaw is addressable with a proper held-out evaluation and does not by itself disprove the method's potential, the reader's CONDITIONAL verdict remains appropriate. No change to the verdict is required, but the authors should be required to provide an out-of-template evaluation before the central claim is accepted.","tokens_in":15306,"tokens_out":2992,"duration_ms":33618,"concrete_test":"Perform a template-level holdout evaluation. Assign every CFSC sample the identity of its parent-code template (shape class) and split the dataset so that no test sample shares a parent template with any training sample. For the test prompts, additionally use human-written paraphrases or prompts from an external source such as Text2CAD, rather than the auto-generated template strings. Retrain and evaluate CAD-Coder on this split. If pass@1 or ACC-G on unseen templates drops sharply relative to the values in Table 1, the reported results are explained by template inversion rather than by natural-language-driven CAD generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that CAD-Coder maps arbitrary natural language instructions to editable, annotated Dxf files. For that claim to hold, the evaluation must show the model generalizes beyond its training distribution. Section 4.3 does not do this: the 485 test prompts are randomly selected from the same synthetic CFSC dataset described in Section 3.2, where every prompt and code pair is produced by hand-authored parent-code templates with randomly sampled parameters. Figure A makes the template structure explicit: the prompt is literally 'please draw a rectangle for me, the width of the rectangle is {width}, the height is {height} and the origin is {(x0,y0)}', and the answer is the same template with those values hard-coded. The model can therefore achieve high pass@1 by learning the template layout and filling in sampled numbers, without any evidence of compositional language understanding, paraphrase robustness, or generalization to unseen shape classes. The paper also does not report any deduplication between the randomly selected test prompts and the training set, so near-duplicate training examples may further inflate the scores. In addition, the comparison in Table 1 omits the most relevant text-to-CAD baselines (Text2CAD, CAD-LLM, LLM4CAD), so the 'first' and 'superior' claims are not supported. The core weakness is that the evaluation protocol never tests the model on a prompt distribution that differs from the training distribution, which is exactly the condition needed to support the central claim of text-guided interactive CAD generation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAD-Coder, a framework that fine-tunes DeepSeek-R1-Distill-Llama-8B with LoRA on a newly constructed CFSC dataset of 29,130 Python/ezdxf script codes paired with Dxf files and natural-language descriptions, so that a user's text prompt can be converted into runnable Python code that produces an editable, annotated Dxf file. The authors report pass@1 of 0.40, ACC-G of 0.68, and ACC-A of 0.77 on a randomly selected 485-prompt test set, and they compare against four generic LLMs on these metrics. The paper also demonstrates cross-platform compatibility of the generated Dxf files and includes ablation studies on annotation strategy, fine-tuning method, and base models.","tokens_in":15568,"tokens_out":3926,"duration_ms":38855,"significance":"If the claims were fully supported, CAD-Coder would be a practically useful contribution: it would be an interactive text-to-CAD system whose output is not a fixed mesh or command sequence but a universal, editable Dxf file with geometric annotations, which is absent from most earlier text-guided CAD work. The CFSC dataset, if released, would also be a resource for the community. The paper's strengths include the focus on editability and annotations, the construction of a large paired dataset from parent-code templates, and the demonstration that the fine-tuned model outperforms several generic LLMs on the authors' own metrics. However, the evaluation as presented does not establish the central claim of robust text-guided generalization, because the test prompts are generated by the same template process as the training data and the most relevant text-to-CAD baselines are not compared. The reported metric values are therefore not yet convincing evidence for the paper's headline capabilities.","major_comments":[{"comment":"The evaluation is performed on prompts drawn from the same template-based generator used to create the training set. In Figure A, the natural-language prompt is literally the filename produced by the parent code, e.g., 'please draw a rectangle for me, the width of the rectangle is {width}, the height is {height} and the origin is {(x0,y0)}'. Section 4.3 states that the 485 test prompts were randomly selected from this same dataset. Consequently, high pass@k and ACC scores may reflect the model's ability to invert the template generator and fill in parameter values, rather than its ability to understand arbitrary user text, paraphrases, or shape classes not present in the training templates. The authors should evaluate on a held-out set of human-written prompts or on prompts describing shape classes and configurations not seen during training, and report the performance separately for such out-of-distribution inputs.","section":"Sec. 3.2 and Sec. 4.3"},{"comment":"The related-work section cites Text2CAD, CAD-LLM, and LLM4CAD as the closest text-to-CAD methods, but Table 1 compares CAD-Coder only against generic LLMs (Qwen2.5-Coder-14b, ChatGPT-4, Deepseek-V3, Llama3.3-70b). Without comparison to the state-of-the-art text-to-CAD systems, the abstract's claim of showing 'superior interactive capabilities' and the introduction's claim of being 'the first interactive model capable of generating annotated CAD files from natural language descriptions' are not established. The authors should either include these baselines in the quantitative comparison or clearly explain why they cannot be directly compared, and should temper the novelty claims accordingly.","section":"Table 1 and Sec. 2"},{"comment":"ACC-G and ACC-A are not operationalized. Equation (8) defines ACC-G as the indicator that 'the graph is correct' and Eq. (9) as 'the annotation is correct', but no concrete decision procedure is given: is correctness determined by an automated geometric comparison, by human raters, by rendering and visual inspection, or by some unit test? Without a specific protocol, the reported ACC-G=0.68 and ACC-A=0.77 are not reproducible or independently verifiable. The authors need to specify the exact evaluation pipeline, including any thresholds, comparison functions, and inter-annotator agreement if humans are involved.","section":"Sec. 4.1, Eqs. (8) and (9)"},{"comment":"The reported pass@1 for the main CAD-Coder model is inconsistent across tables. Table 1 lists pass@1=0.40 for CAD-Coder; Table 2 lists pass@1=0.33 for 'CAD-Coder with annotation'; and Table 3 lists pass@1=0.33 for 'CAD-Coder with LoRA' while Table 1's CAD-Coder is also described as using LoRA in Sec. 3.3. If these numbers correspond to different test sets, training configurations, or evaluation conditions, the paper must state this explicitly and present a single consolidated set of results for the final model; otherwise the headline accuracy is contradictory.","section":"Tables 1, 2, and 3"},{"comment":"The metric APR (Average Parsing Rate) is used repeatedly in the tables and discussed in Sec. 4.3, but it is never defined in Sec. 4.1 or anywhere in the paper. Also, Sec. 4.3 refers to a 'substantial lead in ACC-L', but no metric named ACC-L is defined; this appears to be a typo for ACC-A but must be corrected. All metrics, including APR and any ACC-L, need precise definitions and formulas.","section":"Sec. 4.3 and Tables 1, 2, 3, A, B"}],"minor_comments":[{"comment":"The notation in the ACC-F formula is unclear: 'f_i and f_hat_i is the set of functions' mixes singular and plural, and the indicator compares sets while the text says 'function names are the same'. Please clarify whether the comparison is over function names or full function signatures.","section":"Eq. (6)"},{"comment":"Equation (7) contains typographical errors: the summation limits use 'e' and 'Ne' without definition, and the index 'p' is used inconsistently. Please revise the formula and define all symbols.","section":"Eq. (7)"},{"comment":"There is a typo: 'Ithe graph is comect(xi)' should be 'I[the graph is correct(x_i)]' or similar; also, the subscript/superscript formatting is inconsistent.","section":"Eq. (8)"},{"comment":"The qualitative comparison with VQ-CAD in Figure 6 is of limited value because VQ-CAD is not a text-guided method; the comparison is apples-to-oranges unless the authors clarify that this is only a visual sanity check of geometric quality, not a comparison of text-following ability.","section":"Fig. 6 and Sec. 4.2"},{"comment":"In the parent-code example in Figure A, the file names and prompts are identical to the natural-language descriptions; the paper should explicitly acknowledge that the dataset's language is template-generated and discuss the implications for language diversity and realism.","section":"Sec. 3.2 and Supplemental F"},{"comment":"The sentence 'For anonymous reason, this dataset will be released upon acceptance of the paper' is grammatically awkward and should be rewritten; more importantly, the authors should state a clear release plan for code and dataset to support reproducibility.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and potentially useful application, but the evaluation section needs substantial strengthening before the claims can be considered supported. The most serious concern is the train/test distribution overlap from the template generator; adding a human-written or held-out evaluation set is essential. The novelty claim of 'first' also needs careful qualification given existing Text2CAD and CAD-LLM work. The inconsistency in pass@1 across tables should be resolved clearly. I would encourage the editor to seek a revised version that addresses these points, as the underlying idea has merit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the genuinely useful part is the CFSC dataset and the demonstration that LoRA-fine-tuning an 8B LLM on templated ezdxf code produces runnable code for annotated Dxf files. The pipeline is sound. But the evaluation does not support the headline claim of text-guided interactive CAD generation, because both training and test prompts come from the same parent-code templates. The reported numbers are likely close to ceiling for template inversion.\n\nWhat's good: using Dxf as a universal, editable output format is a legitimate practical choice. Building 29k script+Dxf pairs with annotations is real work; if released, the dataset would be useful for benchmarking. The code generation itself works — the qualitative figures show shapes that match prompts, and the ablations (comments, LoRA vs full fine-tune) are informative. The cross-platform demo is a nice touch.\n\nSoft spots, in order: (1) The test set is randomly sampled from the same synthetic distribution as training, with no deduplication reported, so pass@1 of 0.40 is not evidence of generalization to human phrasing. A single hold-out template family or a paraphrased prompt set would have been easy to add and is necessary. (2) No comparison against Text2CAD, CAD-LLM, LLM4CAD, or vision-language CAD generation; the baselines are generic LLMs that were never prompted with a code template, which flatters CAD-Coder. The \"first\" claim is also unsupported given those works exist. (3) ACC-G and ACC-A are defined with \"correct\" left to a human judge, with no judging protocol, so 0.68/0.77 are not independently checkable. (4) Minor: the metric formulas have typos (ACC-P sums inconsistently over N_c and N_p), and the ablation showing annotation hurts all metrics is honest but undercuts the claim of \"uniquely providing annotations\" as an unqualified win.\n\nNet: the engineering is credible, the dataset could be a contribution, but the paper as written overclaims. A revision with an out-of-distribution test, proper baselines, and a released dataset could turn this into a useful paper. I'd send it to a serious referee — the dataset and system deserve scrutiny — but I wouldn't accept it without substantial changes.","headline":"Useful dataset and a plausible fine-tuned LLM for Dxf code generation, but the evaluation only proves template inversion, not robust text-guided CAD.","tokens_in":16121,"tokens_out":2140,"would_cite":false,"duration_ms":21695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAD-Coder's central claim is that natural-language instructions can be turned into runnable Python scripts that generate editable, annotated CAD files, with a reported 40% pass@1 on its own test set.","keywords":["CAD generation","natural language to code","DXF files","ezdxf","geometric annotations","code generation","LoRA fine-tuning","text-guided design"],"falsifier":"Take the trained model and test it on one hundred prompts written by humans describing the same shapes in their own words, plus fifty prompts requesting shapes that never appear in the parent-code library. If pass@1 falls to near zero on the out-of-library shapes or drops below the generic-LLM baseline on the paraphrased prompts, the robustness claim fails; if it holds, the claim is supported.","tokens_in":15071,"feed_emoji":"📐","tokens_out":7206,"duration_ms":65720,"temperature":0.7,"pith_summary":"CAD-Coder aims to make CAD drafting as easy as typing a sentence: a user describes a shape, and the model outputs runnable Python code that, when executed, produces a Dxf file with editable geometry and dimension annotations. The paper's claim is that this is the first interactive system to generate annotated CAD files straight from natural language, because prior text-guided methods output meshes, point clouds, or custom command sequences that engineers cannot directly open or edit. To support the claim, the authors build a synthetic dataset of 29,130 Dxf files with matching script code and natural-language prompts, generated by randomly filling in parameters of hand-written parent-code templates. On a 485-prompt test set drawn from the same synthetic distribution, they report pass@1 of 0.40, graphic accuracy of 0.68, and annotation accuracy of 0.77, beating the generic LLMs they compare against. If the approach generalizes beyond its template distribution, it would give engineers a fast, editable, cross-platform route from text to CAD.","feed_headline":"Text to editable CAD code: 40% one-shot success","feed_subtitle":"Fine-tuned 8B model writes Python scripts for annotated Dxf drawings, beating generic LLMs on its own benchmark.","key_machinery":"The load-bearing object is the CFSC dataset together with its parent-code randomization procedure. For each shape, the authors hand-write a parametric script template; a randomization algorithm assigns legal values to the parameters, yielding many concrete script-code/Dxf pairs, and the script structure is standardized so that every sample has the same three-part layout: library imports, a construction function, and a main function with dimension parameters. The templates are also annotated with comments, which the paper shows helps the model disambiguate similar code fragments. Training uses the DeepSeek-R1-Distill-Llama-8B model fine-tuned with LoRA, so the argument rests on the model inverting the template generator rather than on a new architectural mechanism.","core_discovery":"On its own terms, the paper establishes a pipeline: natural language prompt to standardized Python/ezdxf script to executable Dxf file that preserves geometric annotations. The central discovery claimed is that a compact, instruction-tuned language model can learn this mapping well enough to outperform much larger generic LLMs on the authors' benchmark, and that the Dxf output format solves the editability and annotation gaps left by earlier CAD generators. The paper also reports that annotated generation is harder than unannotated generation, that comment-injected training code improves generation, and that LoRA tuning beats full fine-tuning on this dataset.","pith_inferences":["The reported metrics are relative to the same synthetic template distribution used for training, so the headline numbers should not be read as generalization to arbitrary user prompts; a human-written prompt benchmark would be the direct test.","The parent-code randomization recipe could be reused for other output formats or CAD kernels, and a natural extension is to test whether the model can compose multiple template shapes in one prompt, which the current dataset does not appear to exercise.","The contrast between annotation-free and annotated generation suggests that the bottleneck is not geometry but the extra linguistic and geometric bookkeeping of dimensions; future work might separate these skills rather than train them jointly.","Since the output is standard code, the same model could potentially be integrated into agentic CAD workflows that iterate on user feedback, but editability of the resulting dimensions is not directly measured by the paper's metrics."],"forward_implications":["Users could go from a typed description to an editable, dimensioned Dxf file without manual drafting, and open the result in mainstream CAD platforms.","Engineers could request annotated parts, with radius, angle, tolerance, chamfer, or surface-roughness dimensions, directly from text, reducing re-drafting for manufacturing.","Because the output is executable code, CAD-Coder could be embedded in interactive loops where the user inspects the generated Dxf and asks for changes, though the paper only demonstrates one-shot generation.","The ablation suggests that adding explanatory comments to training code is a cheap way to improve code generation, which may transfer to other program-synthesis tasks."],"supporting_citations":[{"why":"Supplies the ezdxf library used to construct Dxf files and their geometric annotations; the central output mechanism.","marker":"[9]"},{"why":"Defines the Pass@k metric used to score generated script code in all experiments.","marker":"[15]"},{"why":"Text2CAD established text-guided CAD generation and the cross-modal dataset concept this work extends and compares against.","marker":"[22]"},{"why":"Provides the DeepSeek-R1-Distill-Llama-8B backbone whose reasoning abilities are transferred to the CAD task.","marker":"[25]"},{"why":"Supports the premise that Dxf is a universal, editable CAD exchange format, justifying the output choice.","marker":"[34]"},{"why":"VQ-CAD serves as the baseline in the qualitative 2D sketch comparison, representing prior non-text-guided CAD generation.","marker":"[38]"},{"why":"Qwen2.5-Coder is one of the generic code LLMs quantitatively compared, providing the baseline CAD-Coder is claimed to beat.","marker":"[44]"}],"fun_headline_variants":["Prompt to editable Dxf: 8B model beats larger LLMs","Text-to-CAD code: LoRA tuning wins for editable sketches","CAD-Coder: 29k annotated Dxf files unlock editable CAD","Efficient CAD generation: 8B model writes annotated Dxf"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training and test prompts both come from the same hand-written parent-code templates, so the reported scores measure how well the model reverses that template generator, not how well it understands arbitrary user language or shape classes outside the template library.","fun_headline_variants_meta":{"raw":{"variants":["Prompt to editable Dxf: 8B model beats larger LLMs","Text-to-CAD code: LoRA tuning wins for editable sketches","CAD-Coder: 29k annotated Dxf files unlock editable CAD","Efficient CAD generation: 8B model writes annotated Dxf"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2700,"prompt_tokens":846,"completion_tokens":1854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1777}},"tokens_in":462,"tokens_out":1854,"duration_ms":12676,"temperature":1.0,"reasoning_tokens":1777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:49:14.717253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained model and test it on one hundred prompts written by humans describing the same shapes in their own words, plus fifty prompts requesting shapes that never appear in the parent-code library. If pass@1 falls to near zero on the out-of-library shapes or drops below the generic-LLM baseline on the paraphrased prompts, the robustness claim fails; if it holds, the claim is supported.","supporting_citations":[{"cited_title":"readthedocs","cited_arxiv_id":null,"evidence_quote":"Supplies the ezdxf library used to construct Dxf files and their geometric annotations; the central output mechanism."},{"cited_title":"Text2cad: Generating sequential cad designs from beginner- to-expert level text prompts.Advances in Neural Information Processing Systems, 37:7552–7579, 2025","cited_arxiv_id":null,"evidence_quote":"Text2CAD established text-guided CAD generation and the cross-modal dataset concept this work extends and compares against."},{"cited_title":"Analysis of dxf file with an application to 3d graphic display","cited_arxiv_id":null,"evidence_quote":"Supports the premise that Dxf is a universal, editable CAD exchange format, justifying the output choice."},{"cited_title":"Vq-cad: Computer-aided design model generation with vector quantized diffusion.Computer Aided Geometric Design, 111:102327, 2024","cited_arxiv_id":null,"evidence_quote":"VQ-CAD serves as the baseline in the qualitative 2D sketch comparison, representing prior non-text-guided CAD generation."}],"review_version":1}