REVIEW 3 major objections 5 minor 25 references
A semantically wrong design-intent header lowers geometric adherence below the no-header baseline in conditional CAD completion, and the harm requires the learned header-to-program mapping.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 03:20 UTC pith:MGYUJSSQ
load-bearing objection Genuinely careful causal-control study of header conditioning in CAD, but the headline claim that a wrong header is worse than none only holds against an unconditioned model — the same model with the header masked is just as degraded. the 3 major comments →
Wrong Design Intent Is Worse Than None: A Derangement-Control Diagnosis of Header Conditioning in CAD Program Completion
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that wrong conditioning content is not neutral noise in conditional CAD completion: on programs the model can render unconditioned, a semantically wrong design-intent header yields executable geometry that adheres to the true intent less than the no-header baseline. The derangement control — trained on shuffled ground-truth headers so the header marginal is byte-identical but the header-to-program correlation is destroyed — remains a competent generator yet shows no correct-to-wrong drop (interaction positive on 3/3 seeds, p ≤ 4.2×10⁻³), localizing the harm to the learned mapping rather than to receiving a rare or malformed input. A metric that executes the produced
What carries the argument
CADCON: a five-feature design-intent header (circle, polygon, thin, tall, multi-part) prepended to sketch-extrude CAD programs during LoRA fine-tuning of a 1.5B code language model. The paper's causal instrument is the derangement-trained control: a twin model trained on the same programs with every header permuted to a different program, keeping the header marginal identical while destroying the content correlation; if a wrong header harms the standard model but not the control, the harm requires the learned mapping. The independent scoring path executes each generated program and reads realized geometric features from the resulting solid, sharing no code with the regex extractor that defin
Load-bearing premise
The headline wrong-vs-none contrast rests on a single frozen 100-program held-out set and only about 15–25 non-tied pairs per seed; the paper's own appendix shows the NGON+THIN profile moving opposite to the headline (wrong 0.708 vs baseline 0.667, n=8), and its integrity note admits 2–6-sample discrepancies in regex-side cells of a prior run — if the frozen set, the pre-registration, or the pipeline's internal consistency is not exactly as described, the pooled p-values and
What would settle it
Recompute the wrong-vs-none contrast on a larger held-out set drawn from the same corpus with the same pre-registered matrix; if the wrong-header arm does not stay below the no-header baseline for polygonal and thin profiles, or if the derangement control shows a correct-to-wrong drop on the executable intersection, the central dissociation collapses. A reader could also audit the 2–6-sample regex-side discrepancies: if the internally-consistent run's regex values for correct/masked/baseline cells do not reproduce from the released raw data, the metric-deflation claim is undermined.
If this is right
- In conditional CAD completion, a well-formed but semantically wrong intent header is actively misdirecting: it lowers realized geometric adherence below having no header, so such models should not be deployed with untrusted intent fields without a fallback.
- The harm is not a mechanical reaction to unusual input: a model trained on the same header distribution with no content correlation stays competent and unaffected, placing the failure in the learned conditioning interface.
- Conditioning benefits measured with the same feature detector that builds the conditioning signal are inflated; an independent, execution-level metric is needed to see the real gain.
- The hazard lives in the intended primary workflow (completing or editing a partial program), not in from-scratch generation, where the baseline cannot generate valid CAD at all.
- For builders, the mitigation belongs at the conditioning interface — e.g., training with header dropout or decode-time backoff to unconditioned behavior — not in input validation, since a well-formed wrong header passes any format check.
Where Pith is reading between the lines
- Editorial extension: the wrong-vs-none ordering suggests a general 'trust interface' risk for any conditional generator — a field the model relies on should be treated as untrusted input; the same audit pair (independent scoring + derangement control) could be applied to other conditioning signals, such as retrieved plans or natural-language prompts.
- Editorial extension: the token-header collapse from +0.21 to +0.02 indicates that surface-form token conditioning may largely teach the model to echo the header rather than to realize geometry; a testable follow-up is whether longer, semantically richer headers show the same deflation.
- Editorial extension: the paper leaves open whether the mechanism is graded semantic reading or learned consistency-checking; an intervention that varies the header's distance from the true intent (e.g., one-feature vs. two-feature corruption) with more power could separate those.
- Editorial extension: since only about 15–25 non-tied pairs carry the headline contrast, a larger held-out set or a public executable CAD benchmark could confirm the effect survives beyond this frozen 100-program set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether a fine-tuned code LLM that receives a five-feature design-intent header in parametric CAD program completion actually uses the header's semantic content. It introduces CADCON, a LoRA-fine-tuned Qwen2.5-Coder-1.5B, and evaluates it under a pre-registered {0%, 40%} prefix × {correct, wrong, masked} header matrix with two header tokenizations and three seeds. Adherence is measured both by the training-side regex extractor and by an independent executable geometric-assertion metric on the produced B-rep solid. A derangement-trained control model (shuffled header–program pairings) is used to test whether wrong-header harm requires the learned header→program mapping. The paper reports four findings: (i) at 40% prefix a wrong header degrades adherence below the never-header-trained unconditioned baseline; (ii) the derangement control is competent but immune to wrong headers, with a significant interaction on 3/3 seeds; (iii) the independent metric deflates the apparent benefit of a correct token header; and (iv) the harm is regime-specific, disappearing at 0% prefix where the baseline cannot generate valid CAD.
Significance. If the central claim were supported as stated, the paper would provide a valuable, methodologically careful demonstration that a learned conditioning interface can be actively misdirected by wrong content in structured geometric generation. The independent scoring metric, the derangement control, pre-registered decision rules, and release of raw scores are genuine strengths. However, the headline claim 'wrong intent is worse than none' is only established against an unconditioned model, not against the same model with the header absent at inference; a renderable profile moves in the opposite direction; and the statistical base is smaller than the pooled numbers suggest. With appropriate reframing and additional within-model analyses, the diagnostic contribution would be solid, but the current central claim is overstated.
major comments (3)
- [§4.2, Table 2] The 'wrong worse than none' claim is operationalized against the never-header-trained unconditioned baseline (0.425), not against the same model with the header masked at inference. Table 2 shows wrong-text 0.305 vs masked-text 0.263 and wrong-token 0.215 vs masked-token 0.237, so the within-model contrast is mixed and no paired wrong-vs-masked test is reported. The derangement Test D (§4.4) establishes only that M's correct→wrong drop exceeds R's; it does not compare wrong content to field absence. Thus the title's 'worse than none' and the abstract's 'if the field were absent' are unsupported for the learned interface. Please add a within-model wrong-vs-masked paired test and, if the result is not significant or is in the wrong direction, restrict the claim to 'worse than the unconditioned baseline' and revise the title and abstract accordingly.
- [Appendix C, Table 8] The manuscript claims the harm is established on intents the model can render unconditioned, specifically polygonal and thin geometries. However, the NGON+THIN profile (n=8) shows wrong 0.708 > baseline 0.667, i.e., the wrong header helps rather than hurts. Since NGON+THIN is both polygonal and thin, this contradicts the claim as stated in §4.2 and the abstract. The pre-registered profile guardrail uses the mean profile effect (−0.078), which can pass while a counter-profile is hidden. Please report per-profile wrong-vs-baseline tests and either restrict the claim to the specific profiles for which it holds or explain why NGON+THIN is excluded from the 'renderable intents' set.
- [§6 and §4.2] The wrong-vs-baseline contrast rests on only 15–25 non-tied pairs per seed, and the pooled 76-program numbers include pure-CIRCLE and TALL profiles that are floored at 0 in both arms, so the headline numbers (0.425→0.305/0.215) mix uninformative profiles with the renderable ones. More importantly, the integrity note in §6 states that regex-side values for correct/masked/baseline cells at 40% differ from an earlier run by 2–6 samples per cell, while the decision-critical wrong cells reproduce exactly. This is not fully compatible with the 'frozen protocol' framing unless the released analysis scripts deterministically produce the published Table 2. Please clarify which run produced the published numbers, whether any frozen script was re-run, and report the primary wrong-vs-baseline comparison on the renderable-intent subset (excluding floor profiles) as the headline statistic.
minor comments (5)
- [§4.2] The text-side wrong-header test is significant on only 2/3 seeds (p = 0.073 on seed 0). The phrase 'direction-unanimous' is accurate but could be misinterpreted as 'significant on 3/3'; please state the seed-level significances explicitly in the abstract or the finding.
- [Table 2 caption] The caption should state explicitly that regex-recall includes MULTI_PART and is averaged over all non-empty-intent samples, while geom-recall is four-feature and execution-inclusive. The convention-matched regex footnote is useful and should also appear in the main text.
- [§3.7] The residual ~9% correct-by-chance in the derangement control is handled honestly in §6; consider stating its conservative direction (biasing R toward content-sensitivity) in §3.7 itself, since it is important for interpreting the control's immunity.
- [Figure 4] The caption should define 'identical realized feature set' — ideally noting that this refers to the four primary features and that MULTI_PART is excluded from the primary metric. The main text does explain this, but the caption alone is ambiguous.
- [§6] The integrity note should be moved or expanded to explain the provenance of the Table 2 numbers and the exact relationship between the 'original run' and the frozen analysis scripts. This is more than a presentation issue, but it is currently only in limitations.
Circularity Check
No significant circularity: the central contrasts are empirical and the metric-independence move is scoped honestly.
full rationale
The derivation chain is self-contained. The header extractor φ defines the training signal and the task target, but evaluation executes generated programs and asserts realized features on the produced B-rep solid with separate code; the paper explicitly states: 'The intent target remains φ(GT), which defines the task; only the produced-side scoring changes, breaking the produced-side circularity.' The wrong-vs-none and derangement-control comparisons are behavioral observations under frozen, pre-registered protocols, not fitted quantities renamed as predictions. The masked arm, non-tied-pair counts, profile-level floors, and the dose-response criterion's failure are all reported, so no result is hidden by construction. No load-bearing self-citation appears: the sole author does not cite prior work by the same author, and the integrity note explicitly disclaims that the built-in check is 'not an independent reproduction.' The residual sharing of the five-feature vocabulary between header and target is acknowledged by the authors and is inherent to the task definition, so it does not reduce a prediction to an input by construction.
Axiom & Free-Parameter Ledger
free parameters (4)
- THIN aspect-ratio threshold =
0.30 (depth / largest planar extent)
- TALL aspect-ratio threshold =
2.0
- Evaluation prefix fraction =
0.40
- Competence-gate adherence floor =
0.30
axioms (6)
- domain assumption DeepCAD-derived held-out set is representative of the intended CAD program distribution
- domain assumption The deterministic transpiler from DeepCAD JSON to CadQuery preserves the five geometric features
- domain assumption The regex extractor phi correctly labels ground-truth design intents
- domain assumption CadQuery execution and the five geometric assertions reliably measure realized geometry
- standard math Paired Wilcoxon (pratt) is appropriate for the tie-heavy per-sample differences
- domain assumption The pre-registration was genuinely frozen before evaluation data existed
read the original abstract
Fine-tuned code LLMs can be conditioned on a lightweight design-intent header to steer parametric CAD generation, but whether the model actually reads the header's content has not been tested under a metric independent of the conditioning itself, nor with a causal control. We study CADCON, a five-feature design-intent header prepended to CadQuery-style sketch-extrude programs during LoRA fine-tuning of Qwen2.5-Coder-1.5B, re-scored by executable geometric assertions on the produced B-rep solid, sharing no code with the header-defining regex extractor. Across three seeds and a pre-registered {0%, 40%}-prefix $\times$ {correct, wrong, masked}-header matrix we find: (i) in conditional completion (40% prefix), a semantically wrong header degrades adherence below the no-header baseline (0.43 $\to$ 0.30/0.21 text/token) on design intents the model can render unconditioned -- polygonal and thin geometries; circle and tall intents sit at a baseline generation floor for this checkpoint ($\approx$0 in both compared arms) and are uninformative for this contrast; (ii) a derangement control -- retrained with shuffled ground-truth headers, identical header marginal but destroyed content correlation -- remains competent yet is immune to wrong headers, while the standard model is not (text headers; interaction significant on 3/3 seeds, p $\leq$ $4.2\times10^{-3}$): the harm requires the learned header$\to$program mapping, excluding the marginal/mechanical distribution-shift confound; (iii) the independent metric deflates the apparent benefit of a correct header (token: +0.21 regex $\to$ +0.02 geometry), quantifying metric circularity; (iv) the harm is regime-specific -- at 0% prefix the unconditioned baseline cannot generate valid CAD at all. Wrong intent is not noise: it actively misdirects generation.
Figures
Reference graph
Works this paper leans on
-
[1]
Qlora: Efficient finetuning of quantized llms
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. InNeural Information Processing Systems,
-
[3]
Cadmium: Fine-tuning code language models for text-driven sequential cad design.Trans
Prashant Govindarajan, Davide Baldelli, Jay Pathak, Quentin Fournier, and Sarath Chandar. Cadmium: Fine-tuning code language models for text-driven sequential cad design.Trans. Mach. Learn. Res., 2025. doi: 10.48550/arXiv.2507.09792. URLhttps://www.semanticscholar.org/ paper/eb088897b07b26473068534bc6d31e3a6ad48696
-
[4]
Guan, Xilin Wang, Xingxia Ming, Jing Zhang, Dong Xu, and Qian Yu
Y. Guan, Xilin Wang, Xingxia Ming, Jing Zhang, Dong Xu, and Qian Yu. Cad- coder: Text-to-cad generation with chain-of-thought and geometric reward.arXiv.org,
-
[5]
Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. URLhttps://arxiv.org/abs/2207.12598. 15
Pith/arXiv arXiv 2022
-
[6]
Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186, 2024
Binyuan Hui, Jian Yang, Zeyu Cui, et al. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186, 2024. URLhttps://arxiv.org/abs/2409.12186
Pith/arXiv arXiv 2024
-
[7]
P. Jayaraman, J. Lambourne, Nishkrit Desai, Karl D. D. Willis, Aditya Sanghi, and Nigel Morris. Solidgen: An autoregressive model for direct b-rep synthesis.Trans. Mach. Learn. Res., 2022. doi: 10.48550/arXiv.2203.13944. URLhttps://www.semanticscholar.org/paper/ 301bf1ecb28ae1035e6dcef5ed67add79271c252
-
[8]
Man Ho Lam, Chaozheng Wang, Jen tse Huang, and Michael R. Lyu. Codecrash: Exposing llm fragility to misleading natural language in code reasoning. InNeural Information Processing Systems, 2025. URLhttps://arxiv.org/abs/2504.14119
arXiv 2025
-
[9]
Text-to-cad evaluation with cadtests.arXiv preprint arXiv:2605.07807, 2026
Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz, Elisa Ricci, Anis Kacem, and Djamila Aouada. Text-to-cad evaluation with cadtests.arXiv preprint arXiv:2605.07807, 2026. URL https://arxiv.org/abs/2605.07807
Pith/arXiv arXiv 2026
-
[10]
How does misinformation affect large language model behaviors and preferences? InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025
Miao Peng, Nuo Chen, Jianheng Tang, and Jia Li. How does misinformation affect large language model behaviors and preferences? InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. URLhttps://aclanthology.org/2025.acl-long.674/
2025
-
[11]
Rukhovich, Elona Dupont, Dimitrios Mallis, K
D. Rukhovich, Elona Dupont, Dimitrios Mallis, K. Cherenkova, Anis Kacem, and D. Aouada. Cad-recode: Reverse engineering cad code from point clouds. InIEEE International Con- ference on Computer Vision, 2024. doi: 10.1109/ICCV51701.2025.00914. URL https: //www.semanticscholar.org/paper/05bd4fe6c0ace2cb6bd51a433bf97f03218bdcef
arXiv 2024
-
[12]
Stay on topic with classifier-free guidance.arXiv preprint arXiv:2306.17806, 2023
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-free guidance.arXiv preprint arXiv:2306.17806, 2023. URLhttps://arxiv.org/abs/2306.17806
Pith/arXiv arXiv 2023
-
[13]
No-worse context-aware decoding: Preventing neutral regression in context-conditioned generation
Yufei Tao and Ameeta Agrawal. No-worse context-aware decoding: Preventing neutral regression in context-conditioned generation. InFindings of the Association for Computational Linguistics: ACL 2026, 2026. URLhttps://arxiv.org/abs/2604.16686
Pith/arXiv arXiv 2026
-
[14]
On code-induced reasoning in llms.arXiv preprint arXiv:2509.21499, 2025
Abdul Waheed, Zhen Wu, Carolyn Rosé, and Daphne Ippolito. On code-induced reasoning in llms.arXiv preprint arXiv:2509.21499, 2025. URLhttps://arxiv.org/abs/2509.21499
arXiv 2025
-
[15]
Retrieval-augmented generation with conflicting evidence
Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Retrieval-augmented generation with conflicting evidence. InConference on Language Modeling (COLM), 2025. URL https://arxiv.org/abs/2504.13079
Pith/arXiv arXiv 2025
-
[16]
Deepcad: A deep generative network for computer-aided design models
Rundi Wu, Chang Xiao, and Changxi Zheng. Deepcad: A deep generative network for computer-aided design models. InIEEE International Conference on Computer Vision,
-
[17]
Haoyang Xie and Feng Ju. Text-to-cadquery: A new paradigm for cad generation with scalable large model capabilities.arXiv.org, 2025. doi: 10.48550/arXiv.2505.06507. URL https: //www.semanticscholar.org/paper/5ac4c1a706d5bc49d483f5f8c63323b0fd780a82
-
[18]
Xiang Xu, Karl D. D. Willis, J. Lambourne, Chin-Yi Cheng, P. Jayaraman, and Yasutaka Furukawa. Skexgen: Autoregressive generation of cad construction se- quences with disentangled codebooks. InInternational Conference on Machine Learning,
-
[19]
Bo Yuan, Zelin Zhao, Petr Molodyk, Bin Hu, and Yongxin Chen. Clarify before you draw: Proactive agents for robust text-to-cad generation.arXiv preprint arXiv:2602.03045, 2026. URL https://arxiv.org/abs/2602.03045. 16
Pith/arXiv arXiv 2026
-
[20]
Zhanwei Zhang, Shizhao Sun, Wenxiao Wang, Deng Cai, and Jiang Bian. Flexcad: Unified and versatile controllable cad generation with fine-tuned large language models.arXiv (Cornell University), 2024. doi: 10.48550/arxiv.2411.05823. URLhttps://doi.org/10.48550/arxiv. 2411.05823
-
[21]
Cad-hllm: Generating executable cad from text with hierarchical llm planning
Zhuo Zuo, Yantao Gan, Junfeng Long, and Xianggen Liu. Cad-hllm: Generating executable cad from text with hierarchical llm planning. InProceedings of the Asian Conference on Machine Learning (ACML), volume 304 ofProceedings of Machine Learning Research, 2025. 17 Appendix A Pre-registration record and report-only probes Freeze chain.The six-cell protocol (d...
2025
-
[2021]
URL https://www.semanticscholar.org/paper/ 57d71b30f2ff68bde4a0c50322bb93a5c3358ee0
doi: 10.1109/ICCV48922.2021.00670. URL https://www.semanticscholar.org/paper/ 57d71b30f2ff68bde4a0c50322bb93a5c3358ee0
arXiv 2021
-
[2022]
URL https://www.semanticscholar.org/paper/ 816a64e5b7247ed0b8d29d300e2cae787674e174
doi: 10.48550/arXiv.2207.04632. URL https://www.semanticscholar.org/paper/ 816a64e5b7247ed0b8d29d300e2cae787674e174
-
[2023]
URL https://www.semanticscholar.org/paper/ 32ac52069e562d4f900afee70bdca63f53461481
doi: 10.48550/arXiv.2305.14314. URL https://www.semanticscholar.org/paper/ 32ac52069e562d4f900afee70bdca63f53461481
-
[2025]
URL https://www.semanticscholar.org/paper/ dfd5941b9f60a97de58b6bc4da56c0a22b348580
doi: 10.48550/arXiv.2505.19713. URL https://www.semanticscholar.org/paper/ dfd5941b9f60a97de58b6bc4da56c0a22b348580
-
[2026]
URLhttps://arxiv.org/abs/2604.07960
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.