Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Proactive clarification before code synthesis—asking targeted questions about missing or conflicting dimensions—makes text-to-CAD generation substantially more robust, cutting mean geometric error by about 80% and invalid programs from 4.8%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 05:05 UTC pith:TDPHGDNH

load-bearing objection Real step forward for text-to-CAD robustness, but the abstract oversells what the controlled comparisons back and the evaluation is too synthetic to be the last word—still worth a serious referee. the 3 major comments →

arxiv 2602.03045 v2 pith:TDPHGDNH submitted 2026-02-03 cs.LG

Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation

Bo Yuan , Zelin Zhao , Petr Molodyk , Bin Hu , Yongxin Chen This is my paper
classification cs.LG
keywords text-to-CADCadQueryproactive clarificationambiguous promptsprogram synthesisagentic supervised fine-tuningparametric CADlarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the main failure point in text-to-CAD is not code writing but specification: when a user prompt omits a dimension or contains conflicting values, reactive models guess and hallucinate. ProCAD pairs a clarifying agent that audits the prompt and asks only necessary questions with a coding agent that translates the cleaned specification into CadQuery. The central claim is that resolving ambiguity before synthesis, rather than post-hoc correction from rendered images, yields large gains: on 2,469 ambiguous prompts, the full system reduces mean Chamfer distance by about 80% and the share of invalid programs from 4.8% to 0.9%, while asking few questions. A sympathetic reader would care because it shifts the text-to-CAD problem from model scale to specification quality and interaction design.

Core claim

The core claim is that a two-stage 'clarify before you draw' decomposition—an auditing agent that decides whether the specification is self-consistent and asks targeted questions only when needed, followed by a coding agent that writes the CadQuery program—outperforms single-model reactive generation and even another strong model acting as clarifier. The paper demonstrates that a 7B-parameter open-weight model fine-tuned on only 1.6K curated unambiguous pairs and on synthetic clarification trajectories reaches a 0.9% invalid ratio and mean Chamfer distance of 0.63e-3 on ambiguous prompts, whereas the reactive baseline leaves roughly one in seven prompts invalid or geometrically distorted. Th

What carries the argument

The load-bearing mechanism is the proactive clarifying agent modeled as a finite-horizon Markov decision process: at each turn it either ACCEPTs the current prompt (outputting a standardized self-consistent specification) or ASKs a minimal set of targeted questions in a single round. Its reward balances negative Chamfer distance against an interaction-cost term, and it is trained with supervised fine-tuning on full agent trajectories generated by perturbing verified specifications (omitting or conflicting dimensions) and simulating user answers. The coding agent then converts the standardized specification into executable CadQuery. This decomposition makes ambiguity resolution explicit and a

Load-bearing premise

The assumption that users can and will give correct answers to clear clarification questions—and that synthetic ambiguous prompts resemble real human phrasing—is the load-bearing premise; if it fails, the gains may not transfer.

What would settle it

A human study where real users type ambiguous CAD requests and answer the system's questions: if mean Chamfer distance or invalid ratio on those interactions approaches the single-model baseline, or if users find the questions unclear, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the two-agent pattern holds, ambiguity handling should precede code synthesis rather than be recovered by iterative execution feedback or visual refinement.
  • The coding agent's strong results from 1.6K clean samples imply that prompt/specification quality, not training-set size, is the main bottleneck; better data pipelines may transfer more than more parameters.
  • The clarifying agent generalizes when the user simulator is swapped to a different out-of-distribution model, suggesting the learned question-asking policy is not overfit to one answer style.
  • Keeping the interaction to a single round of targeted questions makes the system practical: users are not asked to re-specify everything, only the missing or conflicting dimensions.
  • The same two-agent framework can be instantiated with different base models, so improvements compound with better future coding agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: if human users answer clarifying questions less consistently than the simulated user, the reported 80% reduction may shrink; the paper's assumption of always-correct answers is the key thing to stress-test.
  • My inference: the clarify-first pattern should transfer to other underspecified program-synthesis domains (e.g., circuit schematics, architectural details, machining plans) where missing constraints cause silent hallucinations.
  • My inference: the dataset-construction pipeline—generate verified descriptions from programs, filter by leakage and reconstruction, retry—could itself be a reusable asset; the 1.6K sample effectiveness suggests it is the actual contribution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ProCAD, a two-agent system for text-to-CAD (CadQuery) generation. A clarifying agent audits user prompts for ambiguity and asks targeted clarification questions; a coding agent translates the clarified specification into CadQuery code. The authors construct a 10K text-to-CadQuery dataset by generating natural descriptions from CadQuery code and multi-view renderings using GPT-5-mini, with leakage and completeness checks. They fine-tune Qwen2.5-7B-Instruct as the coding agent on 1.6K samples and as the clarifying agent via agentic SFT on 6,063 synthetic trajectories. Ambiguous prompts are created by GPT-5-mini perturbing verified specifications with two ambiguity types (under-specified and inconsistent). Experiments on 2,469 ambiguous prompts show that the two-agent system with ProCAD-clarifier and ProCAD-coder yields mean Chamfer distance 0.63e-3, invalid ratio 0.9%, and efficiency/resolution scores ~0.97/~0.93, outperforming baselines including Claude Sonnet 4.5. An out-of-distribution test swaps the user answer generator to Claude Haiku and shows consistent gains.

Significance. Proactive clarification before synthesis is a sensible and underexplored direction for text-to-CAD. The paper's data pipeline and agentic SFT are clearly described, and the code and datasets are promised to be public. The controlled comparisons on the synthetic benchmark are internally consistent and show that the two-agent framework improves over single-model and off-the-shelf clarifiers under the same coding agent. The lightweight fine-tuning result (1.6K samples competitive with 150K-sample prior work) is interesting. However, the evaluation rests entirely on synthetic ambiguity, and the headline abstract numbers do not correspond to a single controlled comparison, which tempers the significance until corrected.

major comments (3)
  1. [Abstract; Table 4] The abstract claims 'reducing the mean Chamfer distance by 79.9% and lowering the invalidity ratio from 4.8% to 0.9%.' In Table 4, 4.8% is the IR of Claude Sonnet 4.5 as clarifier with coding=Claude Sonnet 4.5, while 0.9% is ProCAD-clarifier with coding=ProCAD-coder. The 79.9% CD reduction is also close to comparing ProCAD+ProCAD-coder (0.63) against Claude+Claude Sonnet (3.10), again different coding agents. For a controlled comparison with coding=ProCAD-coder, the mean CD reduction is 76.6% (2.69→0.63) and IR reduction is 2.3%→0.9%. The abstract should report a single controlled row or explicitly state which systems are compared.
  2. [§6.3, Table 5] The central claim of robustness to ambiguous prompts is evaluated entirely on synthetic ambiguity. All ambiguous prompts are produced by GPT-5-mini perturbing verified specifications with exactly two predefined ambiguity types (under-specified and inconsistent). The out-of-distribution test in Table 5 only changes the user answer simulator to Claude Haiku; the prompt distribution and perturbation taxonomy remain the same. Real-world ambiguity may involve different error types, vaguer phrasing, and unreliable user answers, so the reported gains may not transfer. The paper should either add a small human-annotated pilot with naturally occurring ambiguous prompts or explicitly restrict the claims to the synthetic distribution.
  3. [Appendix G] The Efficiency and Resolution metrics are computed by GPT-5-mini as an LLM judge. The same model generated the ambiguous prompts and simulated the user answers during training and evaluation. This creates a risk of circularity: the judge may reward the clarifier for matching the perturbation schema rather than for human interaction quality. No human agreement study is reported for these scores. Since the headline CD and IR numbers are objective, this does not invalidate the main comparison, but the interaction-quality claims should be interpreted with caution.
minor comments (5)
  1. [§6.2 vs Table 1] The text states the invalidity ratio drops 'from nearly 86.9% to 0.9%' when fine-tuning on the new dataset, but Table 1 reports zero-shot Qwen2.5-7B-Instruct on the Ours dataset at 82.9%. The 86.9% figure corresponds to Text2CAD. Please clarify which baseline is used for the improvement.
  2. [Throughout] Model naming is inconsistent: 'Claude 4.5 Sonnet' and 'Claude Sonnet 4.5' are both used (e.g., Table 1 vs Table 4). Please standardize.
  3. [Table 3] The LLM-judge win rates report only the 'Ours' and 'Text2CAD' percentages, which sum to 100 in some rows, but no tie rates are given. Given the large position-bias effect (98.4% vs 66.0% depending on order), explicitly reporting tie rates would strengthen the interpretation.
  4. [Figure 1 caption] The caption mentions 'the radius of the inner icicle,' which appears to be a typo for 'inner circle.'
  5. [Appendix J.3 heading] The heading 'Prompts for ambiguous prompt synteacis generation' contains a typo: 'synteacis' should be 'synthesis.'

Circularity Check

0 steps flagged

No significant circularity: synthetic user simulation is a validity limitation, not a circular derivation.

full rationale

The central coding-agent result is grounded externally: the 10K text-to-CadQuery dataset is generated from CadQuery programs and multi-view renderings, then filtered by an independent completeness check that regenerates CadQuery with GPT-5-mini and compares Chamfer distance to ground-truth meshes (Sec. 4). ProCAD-coder's held-out performance on unambiguous prompts (Table 2) is evaluated against those external meshes, not against the training text. The clarification-agent evaluation is synthetic: Sec. 6.3 states 'we use GPT-5-mini as a user simulator', and Sec. 5.2 assumes 'the user can provide correct answers to any asked question as long as the question itself is clear.' This is a genuine external-validity limitation—real users may answer differently and phrase ambiguities differently—but it is not circularity. The ambiguous test set is generated by the same GPT-5-mini perturbation process used for training, so some in-distribution inversion is measured, but the test set is held out, baseline clarifiers (Claude Sonnet 4.5, GPT-4o-mini) share the same simulator, and Table 5 replaces the answer generator with Claude Haiku as an OOD check. No fitted parameter is renamed as a prediction, and no equation reduces to its own input. The anonymized 'authors, 2025' reference (PR-CAD) is a possible self-citation, but it appears only as a baseline and is not load-bearing for the paper's claims. Overall, the derivation chain is empirical and self-contained against external geometry, so no significant circularity is found.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced. The load-bearing premises are domain assumptions about ground-truth code quality, metric sufficiency, and user-simulation fidelity; the CD threshold and degradation ratio are hand-chosen filters that define the benchmark.

free parameters (3)
  • Chamfer distance quality threshold = 2e-4
    Used to filter CadQuery reconstructions, select original/perturbed prompt pairs, and define completeness checks (Sections 6.1, 6.3). Hand-chosen; affects dataset composition and benchmark difficulty.
  • Ambiguity degradation ratio = 10
    Samples kept only if perturbed-prompt CD is at least 10x the original CD (Section 6.3). This guarantees detectable ambiguity and may inflate measured improvement.
  • Retry limit = 3
    Data annotation retries each generated description up to three times before human escalation (Section 4). Chosen by hand; affects dataset diversity and human effort.
axioms (5)
  • domain assumption CadQuery programs from Rukhovich et al. (2025) are valid ground truth for DeepCAD shapes after CD filtering.
    The entire dataset and evaluation inherit this code as ground truth (Sections 4, 6.1). Errors in reconstruction propagate to text descriptions and metrics.
  • ad hoc to paper Users can and will answer any clear clarification question with correct values.
    Stated in Section 5.2: 'we assume that the user can provide correct answers to any asked question as long as the question itself is clear.' This reduces multi-round interaction to a two-round policy and is not validated with human users.
  • ad hoc to paper GPT-5-mini is a faithful user simulator and a reliable generator of human-like ambiguous prompts.
    Section 6.3 explicitly substitutes simulated users for real human behavior; the OOD test uses Claude Haiku, but both are LLM simulators rather than human interaction data.
  • domain assumption Chamfer distance between generated and ground-truth mesh is a sufficient fidelity metric for CAD generation.
    Used throughout as the primary metric; it is a point-cloud distance, not a manufacturability or functional check.
  • domain assumption LLM-as-judge scores (efficiency, resolution, clarity, human-likeness) are valid proxies for interaction quality and data quality.
    Appendix G defines these metrics; they are not validated against human judgments, although the authors note and attempt to control position and length biases.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation." pith.science (2026). https://pith.science/paper/TDPHGDNH

@misc{pith2026260203045,
  author       = {Pith},
  title        = {Pith review of: Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TDPHGDNH}},
  note         = {Machine review of arXiv:2602.03045}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models have recently enabled text-to-CAD systems that synthesize parametric CAD programs (e.g., CadQuery) from natural-language prompts. In practice, however, geometric descriptions can be under-specified or internally inconsistent: critical dimensions may be missing and constraints may conflict. However, existing fine-tuned models tend to reactively follow the user instructions and hallucinate dimensions when the text is ambiguous. To address this, we propose a proactive agentic framework for text-to-CadQuery generation, named as ProCAD, that resolves specification issues before code synthesis. Our framework pairs a proactive clarifying agent, which audits the prompt and asks targeted clarification questions only when necessary to produce a self-consistent specification, with a CAD coding agent that translates the specification into an executable CadQuery program. We fine-tune the coding agent based on a curated high-quality text-to-CadQuery dataset and train the clarifying agent via agentic SFT on clarification trajectories. Experiments show that proactive clarification significantly improves robustness to ambiguous prompts while keeping interaction overhead low. ProCAD outperforms frontier closed-source models, including Claude Sonnet 4.5, reducing the mean Chamfer distance by 79.9% and lowering the invalidity ratio from 4.8% to 0.9%. Our code and datasets are made publicly available on https://github.com/BoYuanVisionary/Pro-CAD.

Figures

Figures reproduced from arXiv: 2602.03045 by Bin Hu, Bo Yuan, Petr Molodyk, Yongxin Chen, Zelin Zhao.

Figure 1
Figure 1. Figure 1: Diagram of our two-agent text-to-CadQuery pipeline. A proactive clarifying agent audits [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Semi-automatic annotation pipeline: render each shape and use its CadQuery to prompt [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: One failure example in Text2CAD for text-to-CadQuery generation. [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison with the coding agent fixed as [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing

    cs.AI 2026-06 unverdicted novelty 7.0

    IterCAD introduces a closed-loop multimodal agent for CAD generation and editing, trained via progressive SFT and geometry-aware RL with viable-prefix masking, and evaluated on IterCAD-Bench using a new CD-TR curve an...

  2. P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0

    P3D-Bench is a benchmark with three task families that scores MLLMs on generating executable parametric 3D programs, finding failures in precise geometry and part assembly.

  3. IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing

    cs.AI 2026-06 unverdicted novelty 6.0

    IterCAD is a multimodal agent framework using progressive SFT and geometry-aware RL for CAD tasks, with a new data pipeline, IterCAD-Bench, and CD-TR metric showing outperformance in executability and precision.

  4. WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling

    cs.CV 2026-07 conditional novelty 5.5

    A BAGEL-based unified VLAW model with Fourier action tokenization outperforms separate-weight and non-world-model VLA baselines on LIBERO, Language Table, and Franka.

  5. Wrong Design Intent Is Worse Than None: A Derangement-Control Diagnosis of Header Conditioning in CAD Program Completion

    cs.LG 2026-07 conditional novelty 5.0

    A wrong design-intent header degrades CAD completion below the no-header baseline, and a derangement-trained control shows the harm comes from the learned header-program mapping.

Reference graph

Works this paper leans on

19 extracted references · cited by 4 Pith papers

  1. [1]

    Theoriginal correct prompt(ground truth)

  2. [2]

    is_misleading

    Conflicting dimensions: two or more measurements or descriptions that contradict each other. 3.Geometrically impossible dimensions: measurements that cannot form a valid solid. If the prompt is CLEAR and unambiguous, respond with: { "is_misleading": false, "standardized_prompt": "<standardized prompt>" } If the prompt is AMBIGUOUS or MISLEADING, respond w...

  3. [3]

    approxi- mately

    Ensure the code is correct, complete, and follows CadQuery best practices Requirements: •Start with:import cadquery as cq •Store the final result in variable’r’ •Use CadQuery operations only (no other libraries) •Match the dimensions and features described in the text •Output only the Python code, no explanations or markdown J.2 System prompts in data ann...

  4. [4]

    Provide concise and specific answers

    Amisleading promptthat the user actually provided (with ambiguities or errors) 3.Clarification questionsasked by an AI assistant Your task is to answer each question based strictly on the original correct prompt. Provide concise and specific answers. Answer each question clearly and concisely. Use explicit numbers and dimensions from the original prompt w...

  5. [5]

    Analyze the provided text description of a 3D CAD model

  6. [6]

    Generate equivalent CadQuery Python code that creates the described shape

  7. [8]

    a hollow rectangular frame,

    General shape:several sentences naming the part and its main features using engineering terms (e.g., “a hollow rectangular frame,” “a mounting plate with through-holes,” “a stepped bracket with a boss”)

  8. [9]

    Setup:one sentence stating the sketch/workplane and any relevant transforms (origin shift, rotation, translation)

  9. [10]

    Build description:a few sentences describing how to sketch the base profile, define key cutouts, then extrude and apply boolean operations, including only essential dimensions and locations. In constructing our 10K text-to-CadQuery dataset, we use the following prompt to instruct GPT-5-mini as an LLM judge to detect whether a generated natural-language de...

  10. [11]

    The original CadQuery Python code

  11. [12]

    origin” and “workplane

    A modified natural-language prompt that is supposed to describe the same shape Your task.Return a JSON decision on whether the modified prompt containsanyraw code or code-like syntax lifted from the original script. Key principle. •Geometry precision is OK (numbers, tuples, ranges, planes). • CadQuery/Python surface form isNOTOK (API tokens, method calls,...

  12. [13]

    MISLEADING DESCRIPTION Provide the rewritten description with exactly K ambiguities

  13. [14]

    WHAT I CHANGED A bullet list with exactly K bullets. Each bullet: •names the ambiguity type used (must match an item from AMBIGUITY TYPES), •quotes the specific phrase you inserted/edited (short quote), •explains in 1 sentence why it is ambiguous

  14. [15]

    AMBIGUITY SCAN (brief, structured rationale) •List exactly K items. •Each item must include: –Trigger phrase: (quote the exact phrase from MISLEADING DESCRIPTION) 32 –Why it’s unclear: (1 short sentence describing the plausible interpretations) Do NOT label anything as “wrong”; only describe uncertainty

  15. [16]

    Each question must directly resolve one ambiguity you introduced

    QUESTIONS TO ASK Provide exactly K questions, one per ambiguity. Each question must directly resolve one ambiguity you introduced. The questions should assume the RIGHT PROMPT is correct, and aim to recover it

  16. [17]

    Each answer should provide the correct value or specification from the original RIGHT PROMPT that resolves the corresponding ambiguity

    ANSWER TO QUESTIONS Provide exactly K answers, one per question. Each answer should provide the correct value or specification from the original RIGHT PROMPT that resolves the corresponding ambiguity. Format as a bullet list matching the order of QUESTIONS TO ASK. Selection policy • If multiple ambiguity types are provided, diversify across types unless t...

  17. [18]

    Clarity and completeness.Which description more clearly and completely specifies the object in the image? Consider: •Accurate coverage of visible parts and features •Precise dimensions and proportions, with no missing critical measurements •Clear spatial relationships between components •No ambiguity and no misleading or incorrect information

  18. [19]

    clarity winner

    Human-likeness.Which description sounds more natural and human-written? Consider: •Natural flow and readability •Appropriate level of detail (not overly verbose or overly terse) •Use of common engineering terminology without unnecessary jargon Important rules. • Units do not matter:ignore differences in measurement units (e.g., mm vs. inches). Judge geome...

  19. [200]

    What are the coordinates of the shifted workplane origin (the exact shift vector)?

    The cylinder’s axis is aligned with the workplane normal (i.e., normal to the XY plane). Setup: work on the XY workplane whose origin has been shifted. Build description: On that shifted XY workplane, sketch a circle whose center is placed at (19 , 0) relative to the workplane origin. Finalize the sketch and extrude the circle 200 in the positive normal d...

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.