{"id":"e48afe4d-ea9f-4843-a2b2-4f598363127c","arxiv_id":"2602.03045","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Proactively asking clarification questions before CAD code generation reduces geometry error and invalid-code rate on ambiguous text prompts, in simulated-user evaluations.","lead":"A two-agent text-to-CAD system where one model asks the user for missing or conflicting dimensions before a second model writes the CAD code. On a synthetic benchmark with simulated users, it beats a frontier closed-source model on geometric fidelity while asking fewer questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic user simulation may not transfer to real-world ambiguity; the OOD check only swaps the answer generator, not the prompt distribution.","rationale":"The reader's weakest assumption—that synthetic user simulations faithfully approximate real-world prompts and answers—is the most load-bearing threat to the paper's central empirical claim. The paper itself acknowledges this limitation in Sections 6.3 and 7, and the OOD simulator check only varies the answer generator, not the prompt distribution. The controlled comparisons with a fixed coding agent (Table 4) are internally consistent and do show ProCAD-clarifier outperforming Claude Sonnet as clarifier, so the central claim holds within the synthetic environment. However, because both the training data and the evaluation rely on GPT-5-mini-generated ambiguity types and perfect user responses, external validity is unproven. The abstract's headline numbers mixing rows from different fixed-coding-agent settings is a reporting inconsistency, but it does not undermine the controlled comparison that supports the core claim. Therefore a conditional acceptance is appropriate, unchanged from the reader's verdict.","tokens_in":23068,"tokens_out":5409,"duration_ms":69234,"concrete_test":"Run a human-in-the-loop study: collect 100 ambiguous CAD prompts written by human annotators (not GPT-5-mini) for a fixed set of target objects, have separate human participants answer the clarification questions, and evaluate ProCAD-clarifier + ProCAD-coder against Claude Sonnet clarifier + ProCAD-coder on mean Chamfer distance, invalidity ratio, and interaction rounds. If the CD/IR improvement over the Claude clarifier on human prompts is substantially smaller than the improvement on the synthetic set (e.g., less than 50% of the CD reduction), the central claim is not supported. Only a real human study can settle whether the synthetic user simulation faithfully approximates real-world conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that proactive clarification improves robustness to ambiguous prompts—is evaluated entirely on synthetic ambiguity. Section 6.3 generates all ambiguous prompts by having GPT-5-mini perturb verified specifications into exactly two controlled ambiguity types (under-specified and inconsistent), and simulates every user answer with GPT-5-mini. Section 5.2 explicitly assumes 'the user can provide correct answers to any asked question as long as the question itself is clear.' Section 7 defers real-human data to future work. The OOD check in Table 5 replaces the answer generator with Claude Haiku, but the prompts remain the same GPT-5-mini-generated set with the same ambiguity taxonomy. Thus the reported gains (e.g., mean CD 0.63 vs 2.69, IR 0.9% vs 2.3% in Table 4) may be an artifact of optimizing for synthetic user behavior. Real users phrase ambiguities differently (vaguer, with unmodeled error types) and may answer inconsistently, so the clarifier could ask the wrong questions or trust unreliable answers, and the headline robustness improvement may not transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProCAD, a two-agent system for text-to-CAD (CadQuery) generation. A clarifying agent audits user prompts for ambiguity and asks targeted clarification questions; a coding agent translates the clarified specification into CadQuery code. The authors construct a 10K text-to-CadQuery dataset by generating natural descriptions from CadQuery code and multi-view renderings using GPT-5-mini, with leakage and completeness checks. They fine-tune Qwen2.5-7B-Instruct as the coding agent on 1.6K samples and as the clarifying agent via agentic SFT on 6,063 synthetic trajectories. Ambiguous prompts are created by GPT-5-mini perturbing verified specifications with two ambiguity types (under-specified and inconsistent). Experiments on 2,469 ambiguous prompts show that the two-agent system with ProCAD-clarifier and ProCAD-coder yields mean Chamfer distance 0.63e-3, invalid ratio 0.9%, and efficiency/resolution scores ~0.97/~0.93, outperforming baselines including Claude Sonnet 4.5. An out-of-distribution test swaps the user answer generator to Claude Haiku and shows consistent gains.","tokens_in":23335,"tokens_out":6724,"duration_ms":64178,"significance":"Proactive clarification before synthesis is a sensible and underexplored direction for text-to-CAD. The paper's data pipeline and agentic SFT are clearly described, and the code and datasets are promised to be public. The controlled comparisons on the synthetic benchmark are internally consistent and show that the two-agent framework improves over single-model and off-the-shelf clarifiers under the same coding agent. The lightweight fine-tuning result (1.6K samples competitive with 150K-sample prior work) is interesting. However, the evaluation rests entirely on synthetic ambiguity, and the headline abstract numbers do not correspond to a single controlled comparison, which tempers the significance until corrected.","major_comments":[{"comment":"The abstract claims 'reducing the mean Chamfer distance by 79.9% and lowering the invalidity ratio from 4.8% to 0.9%.' In Table 4, 4.8% is the IR of Claude Sonnet 4.5 as clarifier with coding=Claude Sonnet 4.5, while 0.9% is ProCAD-clarifier with coding=ProCAD-coder. The 79.9% CD reduction is also close to comparing ProCAD+ProCAD-coder (0.63) against Claude+Claude Sonnet (3.10), again different coding agents. For a controlled comparison with coding=ProCAD-coder, the mean CD reduction is 76.6% (2.69→0.63) and IR reduction is 2.3%→0.9%. The abstract should report a single controlled row or explicitly state which systems are compared.","section":"Abstract; Table 4"},{"comment":"The central claim of robustness to ambiguous prompts is evaluated entirely on synthetic ambiguity. All ambiguous prompts are produced by GPT-5-mini perturbing verified specifications with exactly two predefined ambiguity types (under-specified and inconsistent). The out-of-distribution test in Table 5 only changes the user answer simulator to Claude Haiku; the prompt distribution and perturbation taxonomy remain the same. Real-world ambiguity may involve different error types, vaguer phrasing, and unreliable user answers, so the reported gains may not transfer. The paper should either add a small human-annotated pilot with naturally occurring ambiguous prompts or explicitly restrict the claims to the synthetic distribution.","section":"§6.3, Table 5"},{"comment":"The Efficiency and Resolution metrics are computed by GPT-5-mini as an LLM judge. The same model generated the ambiguous prompts and simulated the user answers during training and evaluation. This creates a risk of circularity: the judge may reward the clarifier for matching the perturbation schema rather than for human interaction quality. No human agreement study is reported for these scores. Since the headline CD and IR numbers are objective, this does not invalidate the main comparison, but the interaction-quality claims should be interpreted with caution.","section":"Appendix G"}],"minor_comments":[{"comment":"The text states the invalidity ratio drops 'from nearly 86.9% to 0.9%' when fine-tuning on the new dataset, but Table 1 reports zero-shot Qwen2.5-7B-Instruct on the Ours dataset at 82.9%. The 86.9% figure corresponds to Text2CAD. Please clarify which baseline is used for the improvement.","section":"§6.2 vs Table 1"},{"comment":"Model naming is inconsistent: 'Claude 4.5 Sonnet' and 'Claude Sonnet 4.5' are both used (e.g., Table 1 vs Table 4). Please standardize.","section":"Throughout"},{"comment":"The LLM-judge win rates report only the 'Ours' and 'Text2CAD' percentages, which sum to 100 in some rows, but no tie rates are given. Given the large position-bias effect (98.4% vs 66.0% depending on order), explicitly reporting tie rates would strengthen the interpretation.","section":"Table 3"},{"comment":"The caption mentions 'the radius of the inner icicle,' which appears to be a typo for 'inner circle.'","section":"Figure 1 caption"},{"comment":"The heading 'Prompts for ambiguous prompt synteacis generation' contains a typo: 'synteacis' should be 'synthesis.'","section":"Appendix J.3 heading"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within scope and the core method is reasonable. The main issues are (i) the abstract's headline numbers do not correspond to a controlled comparison and should be corrected, and (ii) the evaluation is entirely synthetic, with the OOD test only swapping the answer simulator, not the prompt distribution. These are addressable but require either added evidence (e.g., a small human-pilot) or careful rephrasing of the claims. I also note the reference to 'authors, 2025' (an anonymous OpenReview submission) in the bibliography; this is an unusual citation style that should be cleaned up for a published version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine contribution, but the headline numbers overstate what the controlled experiments support, and the synthetic evaluation should not be treated as the final word. \n\nWhat's new and good: the paper is the first to tackle ambiguity in text-to-CAD with a proactive clarification agent. The data pipeline is careful—generating descriptions from both multi-view renderings and CadQuery code, with leakage checks, completeness checks, retry, and human escalation. Training the coder on only 1.6K samples and getting invalid rates down to 0.9% on their own test set is striking if it holds. The two-agent decomposition (clarifier then coder) is natural, and the controlled comparison in Table 4 is internally consistent: keeping the coding agent fixed, ProCAD-clarifier beats Claude Sonnet 4.5 as clarifier on mean Chamfer distance and invalid ratio. The case studies show real qualitative differences in question quality.\n\nSoft spots: the abstract reports a 79.9% reduction in mean Chamfer distance and an invalid ratio improvement from 4.8% to 0.9%, but those numbers mix two different settings—the 4.8% comes from Claude-as-clarifier with Claude-as-coder, and the 0.9% comes from ProCAD-clarifier with ProCAD-coder. No single controlled row produces both. Second, the entire ambiguity evaluation is synthetic. GPT-5-mini perturbs verified prompts into two ambiguity types, GPT-5-mini simulates the user, and the test set is filtered to cases where the coder fails badly before clarification. The Haiku out-of-distribution check swaps only the answer generator, not the prompt distribution, so it does not test whether real users phrase ambiguity the way the perturbations do. The paper explicitly acknowledges this in Sections 5.2 and 6.3, which is honest, but it still means the robustness gains are not yet demonstrated outside the synthetic regime. Third, no code or data artifacts are actually available yet, so I cannot verify the 1.6K-vs-150K claim or recompute the Chamfer numbers. Fourth, there are no variances or error bars anywhere; the efficiency and resolution scores come from GPT-5-mini as judge.\n\nThe Chamfer distance against external ground-truth shapes is the one metric that cannot be gamed by the simulator, and the central claim—that proactive clarification helps—likely holds. But the magnitude in the abstract is not backed by a single controlled comparison.\n\nBottom line: this deserves peer review. Ask the authors to fix the abstract to match one controlled row, release the code and data with a commit hash, add variances, and ideally run a small human interaction study or at least evaluate on human-written ambiguous prompts. I would not cite the quantitative claims as-is, but the framework and dataset are worth building on.","headline":"Real step forward for text-to-CAD robustness, but the abstract oversells what the controlled comparisons back and the evaluation is too synthetic to be the last word—still worth a serious referee.","tokens_in":23792,"tokens_out":3011,"would_cite":true,"duration_ms":32434,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Proactive clarification before code synthesis—asking targeted questions about missing or conflicting dimensions—makes text-to-CAD generation substantially more robust, cutting mean geometric error by about 80% and invalid programs from 4.8%","keywords":["text-to-CAD","CadQuery","proactive clarification","ambiguous prompts","program synthesis","agentic supervised fine-tuning","parametric CAD","large language models"],"falsifier":"A human study where real users type ambiguous CAD requests and answer the system's questions: if mean Chamfer distance or invalid ratio on those interactions approaches the single-model baseline, or if users find the questions unclear, the central claim fails.","tokens_in":22948,"feed_emoji":"📐","tokens_out":5662,"duration_ms":56373,"temperature":0.7,"pith_summary":"The paper argues that the main failure point in text-to-CAD is not code writing but specification: when a user prompt omits a dimension or contains conflicting values, reactive models guess and hallucinate. ProCAD pairs a clarifying agent that audits the prompt and asks only necessary questions with a coding agent that translates the cleaned specification into CadQuery. The central claim is that resolving ambiguity before synthesis, rather than post-hoc correction from rendered images, yields large gains: on 2,469 ambiguous prompts, the full system reduces mean Chamfer distance by about 80% and the share of invalid programs from 4.8% to 0.9%, while asking few questions. A sympathetic reader would care because it shifts the text-to-CAD problem from model scale to specification quality and interaction design.","feed_headline":"Clarify first, then draw: CAD errors drop 80%","feed_subtitle":"Asking only the needed dimension questions before writing CAD code beats reactive models on ambiguous prompts.","key_machinery":"The load-bearing mechanism is the proactive clarifying agent modeled as a finite-horizon Markov decision process: at each turn it either ACCEPTs the current prompt (outputting a standardized self-consistent specification) or ASKs a minimal set of targeted questions in a single round. Its reward balances negative Chamfer distance against an interaction-cost term, and it is trained with supervised fine-tuning on full agent trajectories generated by perturbing verified specifications (omitting or conflicting dimensions) and simulating user answers. The coding agent then converts the standardized specification into executable CadQuery. This decomposition makes ambiguity resolution explicit and a","core_discovery":"The core claim is that a two-stage 'clarify before you draw' decomposition—an auditing agent that decides whether the specification is self-consistent and asks targeted questions only when needed, followed by a coding agent that writes the CadQuery program—outperforms single-model reactive generation and even another strong model acting as clarifier. The paper demonstrates that a 7B-parameter open-weight model fine-tuned on only 1.6K curated unambiguous pairs and on synthetic clarification trajectories reaches a 0.9% invalid ratio and mean Chamfer distance of 0.63e-3 on ambiguous prompts, whereas the reactive baseline leaves roughly one in seven prompts invalid or geometrically distorted. Th","pith_inferences":["My inference: if human users answer clarifying questions less consistently than the simulated user, the reported 80% reduction may shrink; the paper's assumption of always-correct answers is the key thing to stress-test.","My inference: the clarify-first pattern should transfer to other underspecified program-synthesis domains (e.g., circuit schematics, architectural details, machining plans) where missing constraints cause silent hallucinations.","My inference: the dataset-construction pipeline—generate verified descriptions from programs, filter by leakage and reconstruction, retry—could itself be a reusable asset; the 1.6K sample effectiveness suggests it is the actual contribution."],"forward_implications":["If the two-agent pattern holds, ambiguity handling should precede code synthesis rather than be recovered by iterative execution feedback or visual refinement.","The coding agent's strong results from 1.6K clean samples imply that prompt/specification quality, not training-set size, is the main bottleneck; better data pipelines may transfer more than more parameters.","The clarifying agent generalizes when the user simulator is swapped to a different out-of-distribution model, suggesting the learned question-asking policy is not overfit to one answer style.","Keeping the interaction to a single round of targeted questions makes the system practical: users are not asked to re-specify everything, only the missing or conflicting dimensions.","The same two-agent framework can be instantiated with different base models, so improvements compound with better future coding agents."],"fun_headline_variants":["Ask first, draw later: CAD errors drop 80%","ProCAD clarifies before coding, cutting CAD errors 80%","Proactive clarification slashes CAD invalidity to 0.9%","Clarify-before-CAD reduces failures by 80%","Why proactive CAD agents beat reactive models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The assumption that users can and will give correct answers to clear clarification questions—and that synthetic ambiguous prompts resemble real human phrasing—is the load-bearing premise; if it fails, the gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Ask first, draw later: CAD errors drop 80%","ProCAD clarifies before coding, cutting CAD errors 80%","Proactive clarification slashes CAD invalidity to 0.9%","Clarify-before-CAD reduces failures by 80%","Why proactive CAD agents beat reactive models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1599,"prompt_tokens":777,"completion_tokens":822,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":751}},"tokens_in":521,"tokens_out":822,"duration_ms":8036,"temperature":1.0,"reasoning_tokens":751,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:05:22.027054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human study where real users type ambiguous CAD requests and answer the system's questions: if mean Chamfer distance or invalid ratio on those interactions approaches the single-model baseline, or if users find the questions unclear, the central claim fails.","supporting_citations":[],"review_version":1}