Pith. sign in

REVIEW 3 major objections 6 minor 21 references

Foundation Models for Automatic CAD Generation

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Compact instruction-tuned models match larger ones at text-to-CAD, reaching ~99% mesh success on canonical mechanical parts under iterative critique.

desk verdict Clean multi-model CAD baseline: top instruction-tuned models already hit ~99% watertight meshes on a narrow but useful 97-problem set; scope is the real limit, not the numbers. read the letter →

arxiv 2607.05573 v1 pith:FC3LBIR4 submitted 2026-07-06 cs.AI cs.CE

J de Curtò , Victoria Guillén , I. de Zarzà This is my paper
classification cs.AIcs.CE
keywords LargeLanguageModelsVision-LanguageText-to-CADParametricDesignFoundationEngineeringAutomationIterativeRefinementMeshGeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that modern foundation models can turn plain-language engineering specs into valid parametric CAD geometry for mechanical parts, and that multi-round critique is what makes the process reliable. The authors build LLMForge, a shared pipeline that forces models to emit a fixed JSON CAD schema, converts it into a mesh, scores the result on schema validity, mesh soundness, feature match, and visual fidelity, then feeds structured feedback back for up to three refinement rounds. They test two critique styles: a fast analytic ray-trace scorer (IterTracer) and a vision-language model that judges rendered views for design intent (IterVision). On a 97-problem benchmark of plates with holes, boxes, cylinders, and L-brackets, four of seven models form a tight high-performance cluster near 0.89 overall score with 98.97% mesh success under analytic critique; under the VLM critic, one model reaches 100% watertight meshes while cylinders remain systematically harder. A sympathetic reader cares because the work turns “describe the part in English” into a reproducible first-pass geometry generator that industrial design workflows could actually plug in.

What carries the argument

LLMForge’s multi-round iterative refinement loop under two critique regimes—IterTracer (analytic Phong ray-trace metrics: silhouette IoU, hole visibility, edge clearance, aspect ratio) and IterVision (Qwen2.5-VL-72B chain-of-thought semantic critic on rendered views)—combined with a fixed JSON CAD schema and composite weighted scores that select the best-of-N geometry.

What would settle it

Run the same seven models and both critique pipelines on a held-out set of industrial parts that include freeform surfaces, multi-body assemblies, or manufacturing constraints (GD&T, threads, fillets) and measure whether the top-four cluster still sits near 0.89 overall with ≥98% watertight mesh success, or whether the ranking and saturation collapse.

Watch

Extended reading notes

Core claim

Under analytic visual critique, four instruction-tuned foundation models (DeepSeek-V3.2, Qwen3-235B-A22B, Llama-3.3-70B, Gemma-3-27B) reach essentially the same high overall score band [0.885, 0.890] and 98.97% mesh success on the 97-problem benchmark of four canonical geometry families; adding a VLM semantic critic restores score headroom, yields 100% watertight mesh success for Gemma-3-27B, and exposes residual difficulty on rotationally symmetric parts where visual and semantic scores diverge most.

Load-bearing premise

The claim that success on 97 relatively simple parts from only four geometry families, scored by a fixed JSON schema and the authors’ weighted composite metrics, is a fair proxy for real industrial text-to-CAD performance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript presents LLMForge, a unified multi-model text-to-CAD evaluation framework that converts natural-language part specifications into a fixed JSON schema, synthesizes meshes, and iterates generation under two critique regimes. IterTracer supplies analytic visual feedback (silhouette IoU, hole visibility, edge clearance, aspect-ratio conformance, section score) from Phong-shaded renders; IterVision adds a Qwen2.5-VL-72B chain-of-thought semantic critic. On a curated 97-problem benchmark spanning four geometry families (plates with holes/bolt circles, multi-feature boxes, flanged cylinders, L-brackets), seven foundation models are compared under identical temperature, token budget, and refinement protocol. The central empirical claims are that under IterTracer four models form a tight cluster (μ overall ∈ [0.885, 0.890], 98.97% mesh success), that compact instruction-tuned models match larger systems on this task, and that IterVision yields 100% watertight mesh success for Gemma-3-27B while exposing residual semantic difficulty on rotationally symmetric (cylinder) parts. Failure modes, round-level convergence, and industrial implications are discussed.

Significance. If the reported numbers hold under the stated protocol, the work supplies a reproducible, multi-axis baseline for language-guided parametric CAD generation and a practical dual-critique methodology (analytic vs. VLM-as-judge) that the community can reuse. Strengths that should be credited explicitly include: (i) a fixed, publicly documented JSON schema and geometry engine; (ii) full mesh watertightness checks and best-of-N multi-round trajectories (679 scored attempts per system); (iii) Bonferroni-corrected nonparametric tests and axis-wise correlation matrices; and (iv) public code, benchmark, and result files. The industrial-applicability claims are appropriately tempered by the Scope and Limitations section. Absolute impact is bounded by the four simple geometry families, but as a controlled empirical study of off-the-shelf foundation models for structured CAD output the contribution is solid and timely.

major comments (3)
  1. [Section 3.1, Eqs. (1)–(2)] Section 3.1, Eqs. (1)–(2): The composite weights (IterTracer 0.25/0.15/0.20/0.40; IterVision 0.20/0.10/0.20/0.30/0.20) and the visual sub-signal weights (0.30/0.30/0.20/0.10/0.10) are fixed a priori with no sensitivity or leave-one-axis-out analysis. Under IterTracer, s_vis correlates r = 0.978 with s_overall (Fig. 4a), so the reported tight top-four band [0.885, 0.890] and the saturation claim in Section 6 could be partly weight-driven. A short ablation (equal weights; or reweighting s_vis ∈ {0.2, 0.3, 0.4}) on the same 97 problems would show whether the cluster and ranking order are robust.
  2. [Section 5] Section 5 and Section 2: The evaluation compares only the seven listed foundation models against one another. No specialized parametric CAD baseline (DeepCAD, SkexGen) and no strong general-purpose reference (e.g., GPT-4 / GPT-4o under the same JSON schema and refinement loop) is reported on the 97-problem set. Without at least one external anchor, absolute performance and the language of “saturation” (Section 6) remain relative to the chosen seven models only. Adding one or two such baselines on the same protocol would substantially strengthen the comparative claims.
  3. [Section 3.2, Table 1] Section 3.2 / Table 1: Mesh success rates of 98.97% and 100% are reported from a single generation trajectory per model–problem at fixed T = 0.15 (with multi-round refinement but no multi-seed replication). LLM sampling variance is non-negligible even at low temperature; for the headline success percentages, at least a 2–3-seed check on a stratified subset (or bootstrap CIs over problems) would make the 98.97%/100% figures more defensible as population estimates rather than single-run outcomes.
minor comments (6)
  1. [Abstract / Introduction] The manuscript repeatedly calls itself a “chapter.” For journal submission, reframe as an article and ensure the abstract/introduction do not assume book-chapter framing.
  2. [Section 3.3] Model identifiers INTELLECT and MiniMax-M2.1 are less standard than the others; a one-line description (size, instruction-tuning status, provider) in Section 3.3 would aid readers.
  3. [Figures 2–4] Figure 2 and Figure 4 captions refer to both regimes but the panel labels in the text (“(a) IterTracer” / “(b) IterVision”) should be cross-checked against the actual figure files for consistency in the camera-ready version.
  4. [Section 3.2] The early-exit threshold 0.92 and the 1.5× hole-radius clearance rule are stated in Section 3.2 but not justified; a brief sentence on how they were chosen (or that they are fixed protocol constants) would help reproducibility.
  5. [Section 2] Related Work cites DeepCAD and SkexGen but does not discuss more recent LLM-to-CAD or CAD-as-code systems that also emit structured operations; a short paragraph situating LLMForge against those would improve coverage without changing the experimental design.
  6. [Title page] Minor typography: “de Curt` o” / “de Zarz` a” accent rendering is inconsistent across the author block and affiliations; standardize for production.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical rankings rest on independent analytic/VLM scoring of model-generated JSON, not on fits or self-definitional reductions.

full rationale

LLMForge is an empirical multi-model evaluation study, not a first-principles derivation. Models emit structured JSON under a fixed schema; a deterministic geometry engine builds meshes; scores are computed from a priori axes (schema validation, mesh soundness, analytic feature adherence to ground-truth records, ray-trace visual metrics, and optionally a separate VLM critic). Composite weights in Eqs. (1)–(2) are stated up front and not fitted to produce the reported rankings. Best-of-N selection and multi-round refinement use those external scores as feedback, which is ordinary iterative evaluation rather than circular prediction. Self-citations [7, 8] appear only in Related Work (multi-agent / engineering LLM context) and do not underwrite the central performance claims (top-four cluster µ ∈ [0.885, 0.890], 98.97% mesh success under IterTracer; Gemma-3-27B 100% watertight under IterVision). No uniqueness theorem, fitted parameter renamed as prediction, or self-definitional reduction of the reported results is present. Scope limits of the 97-problem, four-family benchmark are acknowledged and do not constitute circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claims rest on standard LLM/VLM capabilities, a fixed JSON CAD schema, a deterministic mesh engine, and two hand-chosen composite scoring formulas. No new physical entities or free parameters are fitted to data; the free parameters are the a-priori score weights and the early-exit threshold. Domain assumptions about what constitutes a valid mechanical part (watertight mesh, clearance rules, metric hole mappings) are conventional engineering practice.

free parameters (4)
  • Composite score weights (IterTracer: 0.25/0.15/0.20/0.40; IterVision: 0.20/0.10/0.20/0.30/0.20) = 0.25 val + 0.15 mesh + 0.20 feat + 0.40 vis (and reweighted variant)
    Hand-chosen linear weights that define the overall ranking; not fitted to an external gold standard of human CAD quality.
  • Early-exit threshold 0.92 = 0.92
    Arbitrary composite-score cutoff that can halt refinement before round 3; affects average rounds and best-of-N selection.
  • Visual sub-signal weights (0.30/0.30/0.20/0.10/0.10) = silhouette 0.30, hole vis 0.30, edge 0.20, aspect 0.10, section 0.10
    Hand-chosen weights inside the analytic visual score; directly influence s_vis and therefore overall ranking under both regimes.
  • Temperature 0.15 (LLM) / 0.05 (VLM) and max 2048 tokens = T=0.15 / 0.05, 2048 tokens
    Fixed inference hyperparameters that affect generation stochasticity and completeness; held constant but still free choices.
assumptions (5)
  • domain assumption A JSON object conforming to the fixed schema can be deterministically converted by Trimesh/Shapely into a watertight solid mesh that is a faithful realization of the intended parametric part.
    Stated in Section 3 (geometry engine); underpins every mesh-success and visual-score claim.
  • ad hoc to paper The four analytic visual sub-signals (silhouette IoU, hole visibility, edge clearance, aspect-ratio conformance) plus section score are adequate proxies for geometric fidelity of the generated part.
    Defined in Section 3.1; no external validation against human CAD inspection is provided.
  • domain assumption Qwen2.5-VL-72B chain-of-thought judgments of rendered views constitute a valid semantic critic of design intent.
    Introduced in Section 4 (IterVision); treated as complementary ground truth for s_vlm.
  • domain assumption Metric-to-clearance hole mappings (M3→3.4 mm … M8→9.0 mm) and 1.5 imes radius edge-clearance rule are the correct engineering conventions for the benchmark.
    Enforced in the system prompt (Section 3); standard but still an assumption about manufacturing practice.
  • domain assumption Temperature 0.15 and three retries with exponential back-off produce comparable samples across the seven models.
    Section 3.3; necessary for fair multi-model ranking.
invented entities (2)
  • LLMForge framework (IterTracer + IterVision critique regimes) independent evidence
    purpose: Unified multi-model text-to-CAD evaluation pipeline with multi-round refinement and dual critique modalities.
    New system name and protocol introduced by the paper; the components are known techniques recombined.
  • 97-problem CAD benchmark spanning four geometry families independent evidence
    purpose: Provide a fixed, stratified test suite with ground-truth feature records for analytic scoring.
    New dataset constructed for this study; independent of any prior public CAD benchmark cited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Foundation Models for Automatic CAD Generation." pith.science (2026). https://pith.science/paper/FC3LBIR4

@misc{pith2026260705573,
  author       = {Pith},
  title        = {Pith review of: Foundation Models for Automatic CAD Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FC3LBIR4}},
  note         = {Machine review of arXiv:2607.05573}
}
read the original abstract

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) enable the automatic generation of parametric 3D designs from natural-language specifications. This chapter presents an empirical study of foundation models for automatic Computer-Aided Design (CAD) generation of mechanical parts, using a unified evaluation pipeline and a curated benchmark of 97 engineering design problems. We introduce LLMForge, a multi-model text-to-CAD framework integrating JSON-schema validation, analytic feature scoring, mesh synthesis, and multi-round iterative refinement, studied under two critique regimes. IterTracer uses a Phong-shaded ray-trace renderer with analytic visual metrics (silhouette IoU, hole visibility, edge clearance, aspect-ratio conformance) for lightweight geometry-aware feedback across rounds. IterVision replaces the analytic scorer with a VLM semantic critic (Qwen2.5-VL-72B) that evaluates rendered views via chain-of-thought visual reasoning, assessing spatial coherence and design intent. On a benchmark spanning four canonical geometry families (plates with holes and bolt circles, multi-feature boxes, flanged cylinders, and L-brackets), we evaluate seven foundation models: DeepSeek-V3.2, Qwen3-235B-A22B, Llama-3.3-70B, Gemma-3-27B, GLM-4.5, MiniMax-M2.1, and INTELLECT. Under IterTracer, the four highest-ranked models form a tight cluster (overall mean in [0.885, 0.890]) with 98.97% mesh success, showing that compact instruction-tuned models can match substantially larger systems. VLM-based critique in IterVision yields 100% watertight mesh generation on the leading model while surfacing systematic difficulty on rotationally symmetric geometries such as cylinders, where visual and semantic scoring diverge most. We discuss benchmark design, failure modes, CAD-oriented prompting, and implications for industrial workflows and scalable automated mechanical design.

Figures

Figures reproduced from arXiv: 2607.05573 by the authors.

Figure 1
Figure 1. Example output meshes generated by DeepSeek-V3.2 under IterVision (best-round STL exports), [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Mean score per axis per model under both critique regimes (97 problems each). Comparing (a) and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Pearson correlation matrices (pooled across all models and problems) for the two critique regimes. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Per-round score trajectories (±SEM) under both critique regimes. The qualitative reversal between (a) and (b) for top-tier models demonstrates that VLM feedback sustains productive refinement beyond round 1 whereas analytic feedback saturates. 6 Discussion Critique mod…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems35, 23716–23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems35, 23716–23736 (2022)

    2022

  2. [2]

    arXiv preprint arXiv:2108.07732 (2021)

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    In: Advances in Neural Information Processing Systems

    Brown, T., Mann, B., Ryder, N., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems. vol. 33, pp. 1877–1901 (2020)

    1901

  4. [4]

    ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology15(3), 1–45 (2024)

    2024

  5. [5]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  6. [6]

    In: International Confer- ence on Learning Representations (2024)

    Chen, X., Lin, M., Sch¨ arli, N., Zhou, D.: Teaching large language models to self-debug. In: International Confer- ence on Learning Representations (2024)

    2024

  7. [7]

    In: Jezic, G., Chen-Burger, Y.H., Kuˇ sek, M.,ˇSperka, R., Howlett, R.J., Jain, L.C

    de Curt` o, J., de Zarz` a, I., Calafate, C.T.: Llm multi-agent decision optimization. In: Jezic, G., Chen-Burger, Y.H., Kuˇ sek, M.,ˇSperka, R., Howlett, R.J., Jain, L.C. (eds.) Agents and Multi-agent Systems: Technologies and Applications 2024. pp. 3–15. Springer Nature Singapore, Singapore (2025)

    2024

  8. [8]

    Procedia Computer Science270, 733–743 (2025), 29th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2025)

    de Curt` o, J., de Zarz` a, I., Calafate, C.T.: Integrating polyglot persistence with large language models for scalable social network applications. Procedia Computer Science270, 733–743 (2025), 29th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES 2025)

    2025

Show all 21 references
  1. [9]

    arXiv preprint arXiv:2407.21783 (2024)

    Grattafiori, A., Dubey, A., Jauhri, A., et al.: The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  2. [10]

    In: Advances in Neural Information Processing Systems

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Advances in Neural Information Processing Systems. vol. 36 (2023)

    2023

  3. [11]

    Advances in Neural Information Processing Systems36(2023)

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al.: Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems36(2023)

    2023

  4. [12]

    Makatura, L., Foshey, M., Wang, B., H¨ ahnlein, F., Ma, P., Deng, B., Tjandrasuwita, M., Spielberg, A., Owens, J.B., Chen, P.Y., et al.: How can large language models help humans in design and manufacturing? arXiv preprint arXiv:2307.14377 (2023)

    2023 arXiv

  5. [13]

    In: International Conference on Learning Representations (2023)

    Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., Xiong, C.: CodeGen: An open large language model for code with multi-turn program synthesis. In: International Conference on Learning Representations (2023)

    2023

  6. [14]

    arXiv preprint arXiv:2303.08774 (2023)

    OpenAI: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  7. [15]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  8. [16]

    In: Advances in Neural Information Processing Systems

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30, pp. 5998–6008 (2017)

    2017

  9. [17]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  10. [18]

    Advances in Neural Information Processing Systems37, 121475–121499 (2024)

    Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., et al.: CogVLM: Visual expert for pretrained language models. Advances in Neural Information Processing Systems37, 121475–121499 (2024)

    2024

  11. [19]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wu, R., Xiao, C., Zheng, C.: DeepCAD: A deep generative network for computer-aided design models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6772–6782 (2021)

    2021

  12. [20]

    In: Proceedings of the 39th International Conference on Machine Learning

    Xu, X., Willis, K.D., Lambourne, J.G., Cheng, C.Y., Jayaraman, P.K., Furukawa, Y.: SkexGen: Autoregressive generation of cad construction sequences with disentangled codebooks. In: Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Lea...

    2022

  13. [21]

    arXiv preprint arXiv:2406.15877 (2024)

    Zhuo, T.Y., Vu, M.C., Chim, J., Hu, H., Liu, W., et al.: BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 (2024)

    2024 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.