{"id":"264bbf21-37d8-4458-befd-61236ef39d50","arxiv_id":"2505.14646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning a LLaVA-style vision-language model on 163k synthetic image-CadQuery pairs yields a model that compiles every test script and matches CAD solids better than general VLMs.","lead":"CAD-Coder is a vision-language model fine-tuned to turn images into editable CadQuery code, and it reports a perfect syntax rate and higher shape similarity than GPT-4.5 or Qwen2.5-VL on a 100-example CAD test set. The authors also release a 163k-pair dataset of CAD images and code and show limited generalization to photos of real 3D-printed parts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Superiority over GPT-4.5/Qwen2.5-VL rests on one 100-example subset; without confidence intervals or demonstrated deduplication of ABC-derived data, the reported margins could be sampling or leakage artifacts.","rationale":"The reader correctly identifies the 100-sample evaluation and split/leakage risk as load-bearing; this is the empirical foundation of the paper's headline quantitative claim. I agree that representativeness is the single most important unresolved issue, but I disagree slightly on the specific mechanism: the dataset split numbers in §3.1 (147,289 train / 7,355 test / 9,027 val, summing to 163,671 scripts) indicate the split is at the CAD-program level, so image-level leakage through the five rendered views is less likely than duplicate-geometry contamination inherited from ABC/DeepCAD. The paper's own citation of ABC duplicates [24-26] makes the latter plausible and unaddressed. I also found manuscript-internal evidence that strengthens the concern: Fig 2 shows a strongly right-skewed complexity distribution, so a random 100-example subset could easily overrepresent simple solids, and §4.3.2 explicitly reports that the main CAD-Coder cannot perform the unseen fillet operation, contradicting the abstract's generalization claim. These issues do not invalidate the dataset or the fine-tuning approach, but they do mean the current evidence supports 'promising on a small benchmark subset' rather than the broad superiority claim in the abstract. Since the reader's verdict is already CONDITIONAL, I see no need to change it, but the requested full-test-set/CI/dedup audit would either substantiate the claim or require a scaled-back conclusion.","tokens_in":20309,"tokens_out":7044,"duration_ms":66529,"concrete_test":"Run the full §3.3 evaluation on all 7,355 GenCAD-Code test programs (or a pre-registered stratified sample of at least 300 with fixed seeds), computing paired per-sample IOUbest and VSR for CAD-Coder and GPT-4.5/Qwen2.5-VL-72B, and report bootstrap 95% CIs for the mean differences. In the same pass, hash the normalized meshes or parameter sequences of the 100 test programs against the 147,289 training programs to detect exact or near-duplicate solids; if any duplicate is found, re-split, retrain, and re-report. If the IOUbest advantage falls below the CI or a test-training duplicate is found, the headline claim should be scaled back to 'competitive on a small benchmark subset.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 and §4.2 compare all models on a single 100-image random subset of the 7,355-example test set, with no reported seed, no per-sample scores, and no confidence intervals. The dataset is right-skewed toward simple solids (Fig 2), so a lucky draw can inflate both the 100% VSR and the 0.675 IOUbest. The 0.151 gap over GPT-4.5 is meaningful only if it survives sampling variability, which is not shown. In addition, GenCAD-Code inherits ABC/DeepCAD data that the paper itself notes contains duplicates [24-26]; the paper does not state that duplicates were removed or that the split is at CAD-program level, so a test solid may have a near-identical training counterpart. Finally, the abstract's claim of 'executing CAD operations unseen during fine-tuning' is contradicted by §4.3.2, where the main CAD-Coder cannot add fillets; only the LowLR variant succeeds with a prescriptive prompt, and it fails with abstract prompts. This overstatement does not directly drive Table 2, but it makes the broader 'state-of-the-art' framing stronger than the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CAD-Coder, a LLaVA-1.5-style vision-language model fine-tuned on GenCAD-Code, a new dataset of 163,671 rendered-CAD-image and CadQuery-code pairs converted from the GenCAD dataset. The model is evaluated on a 100-example subset of the GenCAD-Code test set against six open- and closed-source VLMs, and the authors report a 100% valid syntax rate (VSR) and an IOUbest of 0.675, outperforming GPT-4.5 and Qwen2.5-VL-72B. Additional experiments examine generalization to photographs of real 3D-printed objects and to the fillet operation, which was not present in the fine-tuning data. The paper also releases the model weights, evaluation prompts, and the dataset.","tokens_in":20542,"tokens_out":5722,"duration_ms":48414,"significance":"If the reported results are robust, the work is valuable: it provides a relatively large open dataset of image-CadQuery pairs, an open fine-tuned VLM for editable CAD code generation, and a formal alignment metric for solid-shape comparison. The comparison with strong external VLMs addresses a timely question, and the release of weights and exact prompts supports reproducibility. The Appendix's derivation of the optimal rigid-body alignment is a genuine technical contribution. However, the headline quantitative claims rest on a single 100-example evaluation subset, on an unvalidated conversion script that defines ground truth, and on dataset construction choices whose leakage risk is not quantified; the significance is therefore conditional on these issues being resolved.","major_comments":[{"comment":"The central state-of-the-art claim rests on a single 100-example subset of the 7,355-example test set. The paper reports no sampling seed, no sample identifiers, and no confidence intervals. Given the right-skewed complexity distribution in Figure 2, the 100% VSR and the 0.151 IOUbest margin over GPT-4.5 may be sampling artifacts. Please report bootstrap or exact binomial confidence intervals (for VSR with 100/100 successes, the 95% Clopper-Pearson interval is approximately [96.4%, 100%]), release the exact sample list, and, if feasible, report results on the full test set.","section":"§3.3, Table 2"},{"comment":"The paper does not state whether the train/test/validation split is at the CAD-program level or at the image level, and it does not describe any deduplication of the underlying ABC/DeepCAD/GenCAD data, even though Section 2.3 notes that ABC includes duplicates. Because GenCAD supplies five rendered images per CAD program, an image-level split could place different views of the same solid in both training and test, directly inflating VSR and IOUbest. Please specify the split granularity, implement and describe solid-level deduplication, or provide a quantitative analysis of train/test similarity to rule out leakage.","section":"§3.1, §3.3"},{"comment":"The GenCAD-Code conversion script defines both the training targets and the evaluation ground truth, but the paper reports no validation that the converted CadQuery scripts reproduce the original GenCAD/ABC solids. A systematic conversion error (for example, in arc or line parameter ordering) would be invisible to the reported metrics because the same converter generates both the prediction targets and the reference solids. Please validate the conversion on a sample by comparing rendered images or STL meshes of converted solids against the original GenCAD outputs and report the failure rate.","section":"§3.1"},{"comment":"IOUbest is computed only on scripts that compile successfully, while VSR differs across models (100% for CAD-Coder versus 82-94% for the baselines). This makes the IOU comparison conditional on successful execution and can favor a model with a low VSR if its few successful outputs happen to be accurate. Please also report an unconditional metric, such as treating failed scripts as IOU=0, and discuss how the model ranking changes under that metric.","section":"§3.3.2, Table 2"},{"comment":"The abstract's claim that CAD-Coder succeeds at 'executing CAD operations unseen during fine-tuning' is contradicted by the body of the paper. Section 4.3.2 states that the main CAD-Coder model cannot add fillets, and that only CAD-Coder-Qwen2.5-14B-LowLR succeeds, and only when given a prescriptive prompt; more abstract prompts do not elicit correct behavior. Please soften the abstract, the introduction, and contribution 3 so that they match the actual results and specify the prompt conditions under which the capability appears.","section":"§4.3.2, Abstract"}],"minor_comments":[{"comment":"The statement that Qwen2.5-VL-72B is the next-best open-source model is incorrect: Ovis2-34B has IOUbest 0.408 and InternVL2_5-78B-MPO has 0.379, both higher than Qwen2.5-VL's 0.352. The relative improvement figure is also arithmetically incorrect: 0.675 versus 0.352 is about a 92% relative improvement, not 60%.","section":"§4.2"},{"comment":"The real-image generalization evidence is qualitative and limited to five 3D-printed objects, with no quantitative metric or comparison to baselines. Please either add quantitative scores for these five examples or explicitly describe the result as anecdotal.","section":"§4.3.1"},{"comment":"The baseline selection criterion ('top-three models on leaderboards as of March 17, 2025') is not reproducible because leaderboard rankings change over time. Please report exact model version identifiers or checkpoints for each baseline.","section":"§3.3.1"},{"comment":"The fillet experiment relies on a two-query setup with a prescriptive prompt, but the exact prompt text is only summarized in Figure 5. Please include the verbatim prompts in the appendix or repository.","section":"§4.3.2"},{"comment":"Training details do not mention the learning-rate schedule, warmup, or random seed for Stage 2 fine-tuning. This information is needed to reproduce the LowLR variant and the reported results.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's dataset builds directly on GenCAD, a prior paper by the same group, which creates a mild self-citation and novelty concern, but the external baseline comparison and the release of weights and prompts mitigate this. The main risk to the paper's central claim is statistical and data-hygiene related rather than a fundamental methodological flaw; with a larger or properly intervaled evaluation, split/dedup clarification, and validation of the conversion script, the paper could be a solid fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful dataset plus a competent fine-tuning recipe, and the paper is honest about most of its own weaknesses. But the headline claim of beating GPT-4.5 rests on 100 test images with no variance estimate, and the abstract overstates the generalization results.\n\nThe new contribution is GenCAD-Code, 163k image-CadQuery pairs converted from GenCAD, and an open-source LLaVA-style model fine-tuned end-to-end. The authors ship weights, code, and prompts, which makes the work reproducible and easy for others to build on. The dataset is the real asset; previous image-to-CAD-code datasets were much smaller.\n\nThe paper also does some things well. The baselines include strong closed- and open-source VLMs. The limitations section is candid: the model struggles with real images, with complex solids, and the fillet experiment honestly shows the main model cannot do unseen operations, with only a LowLR variant succeeding under a prescriptive prompt. The IOU alignment derivation in the appendix is careful.\n\nThe soft spots are real but not disqualifying. First, the core comparison is on a random 100-image subset of a 7,355-image test set, with no seed, no per-sample scores, and no confidence intervals. A 0.151 IOU gap over GPT-4.5 could shift with a different draw, especially given the dataset's right-skewed complexity. Second, IOUbest is only computed on scripts that compile, which inflates scores for models that rarely run. Third, the train/test split granularity is never stated, and the paper itself cites known duplicates in ABC/DeepCAD. If the split is at image level or duplicates are not removed, the 100% VSR and high IOU could be optimistic. Fourth, the abstract says the model is 'executing CAD operations unseen during fine-tuning,' which contradicts the body: the main CAD-Coder cannot fillet. The text is careful; the abstract is not.\n\nAll that said, the central direction of the result, that fine-tuning a VLM on paired CAD code works and beats generic VLMs, is credible and useful. The dataset alone justifies attention. I would send this to a serious referee, with a request to expand the evaluation, add confidence intervals, and fix the abstract. It is not a desk reject.","headline":"Useful open dataset and an honest fine-tuning recipe, but the headline numbers rest on 100 test images with no variance estimate and the abstract overclaims generalization.","tokens_in":21072,"tokens_out":2296,"would_cite":true,"duration_ms":20715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fine-tuning a LLaVA-style vision-language model on 163k image-CadQuery pairs produces a model that writes valid CadQuery code from images with 100% syntax success and the best 3D solid similarity.","keywords":["CAD code generation","vision-language model","CadQuery","image-conditioned CAD","fine-tuning","GenCAD-Code dataset","3D solid similarity","editable CAD"],"falsifier":"Re-run the evaluation on all 7,355 test-set images, or on a larger stratified sample, and check whether VSR remains 100% and IOUbest stays near 0.675; additionally, verify that each sampled test CAD program and its five rendered views are absent from the training split. If the 100-example subset contains only simple solids or shares programs with training, the reported margins would shrink.","tokens_in":20131,"feed_emoji":"📐","tokens_out":5776,"duration_ms":47270,"temperature":0.7,"pith_summary":"The paper tries to show that a general-purpose vision-language model, fine-tuned on a large dataset of CAD images paired with CadQuery scripts, can replace the manual step of turning a picture into an editable 3D CAD model. It introduces CAD-Coder, built on a LLaVA-1.5-style architecture, and GenCAD-Code, a dataset of 163,671 image-CadQuery pairs converted from the GenCAD dataset. On a 100-image subset of the test set, the model generates syntactically valid CadQuery code in 100% of cases and reaches the highest intersection-over-union similarity to ground-truth solids, ahead of closed and open VLMs including much larger ones. The paper also reports partial generalization to photographs of real 3D-printed objects and, for a lower-learning-rate variant, to a fillet operation not present in the fine-tuning data. A sympathetic reader would take the central claim to be that domain-specific fine-tuning of a foundation VLM is a viable route to accurate, editable, image-conditioned CAD generation.","feed_headline":"Fine-tuned VLM turns images into valid CAD code 100% of the time","feed_subtitle":"It writes runnable CadQuery scripts from rendered images and real photos, beating larger open and closed baselines.","key_machinery":"The load-bearing machinery is the pairing of a LLaVA-1.5-style vision-language architecture with a two-stage training recipe: first align a CLIP-ViT-L-336px vision encoder to a Vicuna-13B language model through a two-layer MLP on 595k image-caption pairs, then fine-tune the MLP and LLM end-to-end on 147k image-CadQuery pairs with the vision encoder frozen. The data side is GenCAD-Code, obtained by converting the command-vector CAD programs of GenCAD/DeepCAD into CadQuery scripts, which gives a complete, executable, editable code representation rather than a restricted DSL. Evaluation uses a normalized IOUbest metric in which solids are centered and scale-normalized by the radius of gyration, then aligned by principal axes with the best of four valid rotations, justified by a Procrustes-style optimal alignment argument.","core_discovery":"CAD-Coder's central claim is that an open-source VLM fine-tuned end-to-end on the GenCAD-Code dataset can generate executable, editable CadQuery Python code directly from an input image, and that this method outperforms state-of-the-art image-conditioned code-generating baselines. In the paper's evaluation on 100 randomly sampled test examples, CAD-Coder attains a 100% valid syntax rate (VSR) and an IOUbest score of 0.675, compared with 0.524 for GPT-4.5 and 0.352 for Qwen2.5-VL-72B, the next-best baselines on each metric. The paper further claims that the model shows signs of generalization beyond its fine-tuning distribution: it produces reasonable CAD code from photographs of real objects, and one variant trained with a halved learning rate can apply a fillet operation that never appeared in the fine-tuning data. The authors position this as evidence that fine-tuned foundation models, rather than bespoke CAD-only networks or DSL-based systems, are a promising path for automating editable CAD modeling.","pith_inferences":["The headline numbers rest on 100 test examples, so a natural extension is to run the same evaluation on the full 7,355-example test set; if the 100-sample subset is biased toward simpler solids, VSR and IOU would likely drop.","Because the paper does not specify whether the train/test split is at the CAD-program level, the same solid might appear in training through one of its five rendered views; checking program-level exclusivity would settle whether part of the gain is memorization.","The real-photo generalization was tested on only five objects, so a larger real-image benchmark paired with ground-truth CAD code would be needed to quantify the domain gap.","The fillet generalization is prompt-dependent, so more robust methods for preserving pretrained knowledge during fine-tuning could expand the set of usable CAD operations."],"forward_implications":["If CAD-Coder's results hold, engineers could hand a photo or rendering to a model and get a starting CadQuery script that compiles, cutting the manual CAD modeling step down to editing generated code.","A 100% VSR means generated scripts can be run automatically, enabling pipelines that batch-generate CAD candidates from image collections.","The open release of GenCAD-Code (163k image-code pairs) gives other researchers a large English-code dataset for further fine-tuning beyond this specific model.","The generalization experiments suggest foundation-model pretraining can partially compensate for a narrow fine-tuning distribution, and the fillet result on the low-LR variant shows hyperparameter choice controls how much pretrained CAD knowledge survives.","The IOUbest metric offers a more principled replacement for chamfer-distance comparisons in editable-CAD evaluation."],"supporting_citations":[{"why":"Supplies the GenCAD dataset of CAD programs and rendered images that GenCAD-Code converts into CadQuery scripts.","marker":"[9]"},{"why":"Defines the DeepCAD command representation from which GenCAD and GenCAD-Code derive their CAD programs.","marker":"[8]"},{"why":"Introduces the visual instruction tuning method that CAD-Coder's two-stage training is based on.","marker":"[10]"},{"why":"Defines the LLaVA 1.5 architecture that CAD-Coder uses as its model backbone.","marker":"[27]"},{"why":"Provides the improved baseline recipe for the two-stage LLaVA training procedure.","marker":"[28]"},{"why":"Documents that general VLMs without domain training hallucinate geometry and produce syntax errors, motivating the need for fine-tuning.","marker":"[11]"},{"why":"Demonstrates text-conditioned fine-tuning for CadQuery code generation, a baseline approach this work extends to the image-conditioned setting.","marker":"[20]"},{"why":"Shows a fine-tuned VLM outputting a custom DSL for image-conditioned CAD, a comparison point for CAD-Coder's full-code representation.","marker":"[14]"},{"why":"Supplies the orthogonal Procrustes solution used to justify the principal-axis alignment in the IOUbest metric.","marker":"[35]"}],"fun_headline_variants":["100% valid CAD code from images: open VLM does it","Open VLM beats GPT-4.5 at image-to-CAD code generation","Image to CadQuery: open VLM achieves 100% syntax rate","CAD-Coder: open-source VLM turns images into editable CAD","Open VLM generates runnable CAD code from real photos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative claims rest on the assumption that the 100 randomly sampled test examples fairly represent the full 7,355-example test set and that the train/test split separates CAD programs, not just rendered images, so no test solid appeared in training through a different view.","fun_headline_variants_meta":{"raw":{"variants":["100% valid CAD code from images: open VLM does it","Open VLM beats GPT-4.5 at image-to-CAD code generation","Image to CadQuery: open VLM achieves 100% syntax rate","CAD-Coder: open-source VLM turns images into editable CAD","Open VLM generates runnable CAD code from real photos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3593,"prompt_tokens":1020,"completion_tokens":2573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2479}},"tokens_in":636,"tokens_out":2573,"duration_ms":17744,"temperature":1.0,"reasoning_tokens":2479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:30:14.310491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on all 7,355 test-set images, or on a larger stratified sample, and check whether VSR remains 100% and IOUbest stays near 0.675; additionally, verify that each sampled test CAD program and its five rendered views are absent from the training split. If the 100-example subset contains only simple solids or shares programs with training, the reported margins would shrink.","supporting_citations":[{"cited_title":"Deep- cad: A deep generative network for computer-aided design models","cited_arxiv_id":null,"evidence_quote":"Defines the DeepCAD command representation from which GenCAD and GenCAD-Code derive their CAD programs."},{"cited_title":"Visual Instruction Tuning","cited_arxiv_id":null,"evidence_quote":"Introduces the visual instruction tuning method that CAD-Coder's two-stage training is based on."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Defines the LLaVA 1.5 architecture that CAD-Coder uses as its model backbone."},{"cited_title":"LLM4CAD:Multi-ModalLargeLanguageModelsfor3D Computer-Aided Design Generation","cited_arxiv_id":null,"evidence_quote":"Documents that general VLMs without domain training hallucinate geometry and produce syntax errors, motivating the need for fine-tuning."},{"cited_title":"LargeLan- guage Models for Computer-Aided Design (LLM4CAD) Fine-Tuned: Dataset and Experiments","cited_arxiv_id":null,"evidence_quote":"Demonstrates text-conditioned fine-tuning for CadQuery code generation, a baseline approach this work extends to the image-conditioned setting."},{"cited_title":"OpenECAD: Anefficientvisuallanguagemodelforeditable3D-CADde- sign","cited_arxiv_id":null,"evidence_quote":"Shows a fine-tuned VLM outputting a custom DSL for image-conditioned CAD, a comparison point for CAD-Coder's full-code representation."}],"review_version":1}