REVIEW 4 major objections 5 minor 1 cited by
Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper argues that the fidelity of LLM-generated 3D CAD models improves with the semantic richness of the input, and that a quantitative, human-in-the-loop evaluation suite lets designers reach the ground-truth geometry faster than visu
desk verdict The framework has a decent kernel, but the headline result is hand-built, not measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a dual-metric evaluation suite: structural complexity metrics (feature count Cf, surface-area-to-volume ratio Cs, Euler-characteristic-based topological complexity Ct, combined into a composite score C) and geometric similarity metrics (dimensional accuracy Sd, volumetric similarity Sv, surface similarity Sa, Hausdorff distance Hd, PCA alignment score Sp, ICP alignment score Si, combined into a final similarity score Sf). These numbers are fed back to a human reviewer, who uses them to refine the prompt or the OpenSCAD code and decide when a modality has plateaued, forming the human-in-the-loop iteration. The paper's central object is therefore the numeric feedb
What would settle it
Run the same four-modality L-bracket study but compute the PCA alignment score after rotating the generated model's top arm by 90 degrees; if the score does not drop below its unrotated value (or if a false positive appears in a condition that should fail), the alignment metric is not actually capturing the failure it diagnoses. A second test: repeat the iteration loop with only visual feedback (no numeric scores) on the same modalities; if it reaches the ground truth in the same number of steps, the claimed 'faster convergence' premise would be undercut.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that 'input expressiveness plays a crucial role in the accuracy of LLM-generated models' and that a quantitative evaluation strategy 'enables significantly faster convergence to the ground truth geometry compared to traditional, visually guided methods.' The evidence is the L-bracket case study: across four input conditions the generated OpenSCAD models were scored by the proposed metrics, showing a monotonic improvement in fidelity. In the geometric-structure condition the LLM produced the correct volume, surface area, and dimensions, but a single rotation of the upper bracket remained; PCA and ICP alignment scores flagged exactly that error, a
Load-bearing premise
The whole ranking and the rotated-arm diagnosis rest on the assumption that the PCA alignment score and the ICP alignment score correctly measure geometric alignment—yet the PCA score in Eq. 9 can go negative, and the ICP score's range is asserted without a defining equation.
Editorial extensions
If this is right
- If correct, structured symbolic inputs (geometric structure trees or CAD code) should be the default for LLM-based geometry generation when precision matters.
- The metric suite gives a common yardstick to compare different LLMs and generation pipelines, replacing qualitative scoring in CAD studies.
- Alignment metrics (PCA/ICP) can act as diagnostics that isolate specific transformation errors—like a rotated feature—that volume/surface/dimension metrics cannot see.
- Quantitative feedback can shrink iteration cycles, because the human knows exactly which property to fix instead of relying on visual inspection.
- The framework extends from single parts to multi-part assemblies and parametrized geometry, as the authors propose for future work.
Reading between the lines
- The framework's modularity suggests a direct extension: plug the same metrics into any generative 3D pipeline (diffusion, mesh deformation, GANs) to benchmark against a CAD ground truth without changing the evaluation code.
- The perfect code-level result hints that LLMs excel at exploiting grammar-like constraints; the same quantitative loop could set a new test suite for structured code generation beyond geometry, such as programmatic generation of parametric circuits.
- A concrete next experiment the paper leaves for future work is applying the metrics to a multi-part assembly, where alignment scores would be needed to check relative pose between components, not just the global shape.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a human-in-the-loop framework for quantitatively evaluating 3D CAD models generated by large language models. It defines structural complexity metrics (feature, surface, topological, and a composite score) and geometric similarity metrics (dimensional accuracy, volumetric similarity, surface similarity, Hausdorff distance, PCA alignment, ICP alignment, and a weighted final score). Using an L-bracket case study, it compares four input modalities—2D orthographic views, isometric sketches, geometric structure trees, and code-level correction prompts—and reports that increasing semantic richness improves fidelity, with code-level prompts achieving perfect reconstruction. It further claims that the quantitative evaluation enables significantly faster convergence to ground truth than visual inspection alone.
Significance. If the results were valid, the proposed framework could be a useful building block for benchmarking LLM-based CAD generation: it assembles standard metrics in a modular fashion, includes an OpenSCAD appendix that aids reproducibility, and explicitly targets human-in-the-loop refinement, which is a practical need in engineering workflows. However, the central empirical claims are not supported by the reported experiments. The code-level 'perfect reconstruction' is produced by manual human code edits rather than by the LLM, the two alignment metrics that ground the main diagnosis are either undefined or mathematically mis-specified, and the 'faster convergence' claim is asserted without a comparison experiment. These are load-bearing issues, not presentation defects.
major comments (4)
- [Section 4.1, Table 1, Abstract] The headline result is confounded. Section 4.1 states: 'For code-level correction prompts, errors identified through quantitative evaluation are addressed directly through manual code adjustments, without further involvement of the LLM.' Section 4.2 similarly describes the geometric-structure case as requiring 'a few parameter modifications in the OpenSCAD code' by the human. The perfect scores in the Code row of Table 1 therefore measure human correction, not LLM generation from a code-level prompt. The abstract's claim that 'code level prompts achieving perfect reconstruction across all metrics' is not supported by the experiment as designed. The modality ranking is not a controlled comparison of LLM capabilities.
- [Section 3.2, Eq. (9), Table 1] The PCA alignment score is not a valid similarity score as defined. Eq. (9) gives S_p = 1 - ||C_g - C_t|| / ||C_t||, which can be negative whenever the principal-component distance exceeds the norm of the target vector. Table 1 reports exactly this: -0.2516, -0.1547, and -0.1567 for the first three modalities, yet these are presented as 'PCA Align. Score' values. No clipping, normalization, or interpretation of negative values is given. The ICP alignment score S_i is asserted to lie in [0,1] but no equation or algorithm is provided. Since the 'rotated arm' diagnosis in Section 4.2 relies on PCA and ICP scores, the central diagnostic conclusion for the geometric-structure case is unsupported.
- [Section 5] The claim that the proposed quantitative evaluation enables 'significantly faster convergence to the ground truth geometry compared to traditional, visually guided methods' is not evidenced anywhere in the paper. No experiment compares the quantitative loop to a visually guided loop; no iteration counts, wall-clock times, or convergence curves are reported. Table 1 reports only final scores. This is a stated key contribution, so the absence of any comparative data is a major gap.
- [Section 4, Table 1] The case study consists of a single L-bracket with one output per modality and no repeated trials, error bars, or statistical analysis. Given the manual intervention in the code-level condition, the reported monotonic trend across modalities cannot be distinguished from the amount of human effort invested. Even setting aside the metric issues, the evidence is too thin to support the general conclusion that 'increased semantic richness' improves LLM generation fidelity.
minor comments (5)
- [Section 3.1, Eq. (4)] The weights K_1, K_2, K_3 for the composite complexity score are never given numerical values. The reported value C = 255 in Section 4.2 is therefore not reproducible from the text.
- [Section 3.2, Eq. (10)] The final similarity score S_f uses weights K_1..K_5, but their values are not specified. The 'Gen. Score' column of Table 1 presumably corresponds to S_f, yet the reader cannot verify the aggregation.
- [Figure 4] The figure mixes similarity scores (bounded or unbounded) with Hausdorff distance (a distance, not a similarity). It is unclear whether a secondary axis is used and what the units are. Please clarify.
- [Section 4.2] The text says the geometric-structure model 'exactly matched the ground truth in terms of volume, dimension, and surface area' while the arm was rotated. This is internally consistent only if 'dimension' excludes orientation, but the phrase should be clarified to avoid the appearance of contradiction.
- [Throughout] Some metric names lack equation numbers for the ICP score, and the paragraph after Eq. (9) reads as if a formula was omitted. Please ensure all formally introduced metrics have a numbered definition.
Circularity Check
Code-level 'perfect reconstruction' is produced by manual human code edits, not LLM generation; the central fidelity claim reduces to its human-corrected input by construction.
-
fitted input called prediction
[Section 4.1 (paragraph on code-level correction prompts); Section 4.2 (paragraph beginning 'Crucially'); Table 1 'Code' row; Abstract]
"For code-level correction prompts, errors identified through quantitative evaluation are addressed directly through manual code adjustments, without further involvement of the LLM."
The paper's headline result—'code-level prompts achieving perfect reconstruction across all metrics' (Abstract; Table 1 Code row = 1.0000 across all metrics, Hausdorff 0.0000)—is generated by a human editing the OpenSCAD code until the metrics match the ground truth. The LLM is explicitly not involved in these corrections. Thus the code row is not an LLM prediction from a code-level prompt; it is a hand-fitted solution to the target geometry, so the reported perfect scores are forced by construction. The central claim that increasing semantic richness improves LLM generation fidelity therefore rests on comparing unedited LLM outputs (3-view, isometric, structure tree) with a human-corrected code file, making the modality ranking an artifact of how much direct correction the human supplied.
full rationale
The metric framework (complexity and similarity equations) is independent and not circular: the formulas in Section 3 are standard geometric comparisons and are not defined in terms of the LLM outputs. The composite weights and C=255 baseline are hand-chosen but do not themselves smuggle in the conclusion. No load-bearing self-citation or uniqueness theorem is used. However, the central empirical claim is circular in the experimental sense: the code-level condition, which anchors the conclusion 'improved generation fidelity with increased semantic richness' and 'perfect reconstruction across all metrics', was produced by direct manual code adjustment based on the ground truth, with explicit statement that the LLM was not involved in those corrections. The Table 1 Code row (1.0000 across all metrics, Hausdorff 0.0000) is therefore constructed, not predicted. The 'faster convergence vs visually guided methods' claim is unsupported rather than circular. Overall, the derivation chain for the headline result reduces to the human-corrected input by construction.
Assumptions & free parameters
free parameters (3)
- Complexity weights K1, K2, K3 =
unspecified (Cf prioritized)
- Similarity weights K1..K5 =
unspecified (volume and surface prioritized)
- Iteration stopping threshold =
not quantified
assumptions (4)
- standard math Euler characteristic with E=1.5F for triangular meshes is valid for the STL models
- domain assumption Face count, surface/volume ratio, and Euler characteristic are valid proxies for geometric complexity
- domain assumption A single L-bracket run per modality is representative of LLM generation performance
- domain assumption Quantitative feedback converges faster than qualitative visual inspection
Cite this review
Pith. "Pith review of Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models." pith.science (2026). https://pith.science/paper/N3JH6UE3
@misc{pith2026250907010,
author = {Pith},
title = {Pith review of: Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/N3JH6UE3}},
note = {Machine review of arXiv:2509.07010}
}
read the original abstract
Large Language Models are increasingly capable of interpreting multimodal inputs to generate complex 3D shapes, yet robust methods to evaluate geometric and structural fidelity remain underdeveloped. This paper introduces a human in the loop framework for the quantitative evaluation of LLM generated 3D models, supporting applications such as democratization of CAD design, reverse engineering of legacy designs, and rapid prototyping. We propose a comprehensive suite of similarity and complexity metrics, including volumetric accuracy, surface alignment, dimensional fidelity, and topological intricacy, to benchmark generated models against ground truth CAD references. Using an L bracket component as a case study, we systematically compare LLM performance across four input modalities: 2D orthographic views, isometric sketches, geometric structure trees, and code based correction prompts. Our findings demonstrate improved generation fidelity with increased semantic richness, with code level prompts achieving perfect reconstruction across all metrics. A key contribution of this work is demonstrating that our proposed quantitative evaluation approach enables significantly faster convergence toward the ground truth, especially compared to traditional qualitative methods based solely on visual inspection and human intuition. This work not only advances the understanding of AI assisted shape synthesis but also provides a scalable methodology to validate and refine generative models for diverse CAD applications.
Forward citations
Cited by 1 Pith paper
-
3D-CBM: A Framework for Concept-Based Interpretability in Generative 3D Modeling
Introduces 3D-CBM framework mapping raw 3D inputs to multi-tiered interpretable concepts, achieving 88.8% concept accuracy and test-time intervention on PartNet and ShapeNet.
Reference graph
Works this paper leans on
-
[1]
P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning rep- resentations and generative models for 3d point clouds. InInternational conference on machine learning, pages 40–49. PMLR, 2018
work page 2018
-
[2]
K. Alrashedy, P. Tambwekar, Z. Zaidi, M. Langwasser, W. Xu, and M. Gombolay. Generating cad code with vision-language models for 3d designs.arXiv preprint arXiv:2410.05340, 2024
arXiv 2024
-
[3]
K. Amadori, M. Tarkian, J. Ölvander, and P. Krus. Flexible and robust CAD models for design automation.Advanced Engineering Informat- ics, 26(2):180–195, 2012. ISSN 1474-0346. doi: 10.1016/j.aei.2012. 01.004
- [4]
-
[5]
N. Dommaraju, M. Bujny, S. Menzel, M. Olhofer, and F. Duddeck. Evaluation of geometric similarity metrics for structural clusters gener- ated using topology optimization.Applied Intelligence, 53(1):904–929, 2023
work page 2023
-
[6]
A. C. Doris, D. Grandi, R. Tomich, M. F. Alam, M. Ataei, H. Cheong, and F. Ahmed. Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation. Journal of Computing and Information Science in Engineering, 25(2): 021009, 2025
work page 2025
-
[7]
L. Graening and B. Sendhoff. Shape mining: A holistic data mining approach for engineering design.Advanced Engineering Informatics, 28(2):166–185, 2014. ISSN 14740346. doi: 10.1016/j.aei.2014.03.002
-
[8]
Y . Jadhav and A. B. Farimani. Large Language Model Agent as a Me- chanical Designer, 2024. arXiv:2404.17525 [cs]
arXiv 2024
Show all 38 references
-
[9]
M. D. Johnson, L. M. Valverde, and W. D. Thomison. An investigation and evaluation of computer-aided design model complexity metrics. Computer-Aided Design and Applications, 15(1):61–75, 2018. ISSN 1686-4360. doi: 10.1080/16864360.2017.1353729
2018
-
[10]
Joshi and B
D. Joshi and B. Ravi. Quantifying the Shape Complexity of Cast Parts. Computer-Aided Design and Applications, 7(5):685–700, 2010. ISSN 1686-4360. doi: 10.3722/cadaps.2010.685-700
2010 doi
-
[11]
Kerbrat, P
O. Kerbrat, P. Mognol, and J. Hascoet. Manufacturability analy- sis to combine additive and subtractive processes.Rapid Prototyp- ing Journal, 16(1):63–72, 2010. ISSN 1355-2546. doi: 10.1108/ 13552541011011721
2010
-
[12]
Lara López, A
G. Lara López, A. Peña Pérez Negrón, A. De Antonio Jiménez, J. Ramírez Rodríguez, and R. Imbert Paredes. Comparative analy- sis of shape descriptors for 3D objects.Multimedia Tools and Ap- plications, 76(5):6993–7040, 2017. ISSN 1573-7721. doi: 10.1007/ s11042-016-3330-5
2017
-
[13]
X. Li, Y . Sun, and Z. Sha. LLM4CAD: Multi-Modal Large Lan- guage Models for 3D Computer-Aided Design Generation. Ameri- can Society of Mechanical Engineers Digital Collection, 2024. doi: 10.1115/DETC2024-143740
2024 doi
-
[14]
Makatura, M
L. Makatura, M. Foshey, B. Wang, F. HähnLein, P. Ma, B. Deng, M. Tjandrasuwita, A. Spielberg, C. E. Owens, P. Y . Chen, et al. How can large language models help humans in design and manufacturing? arXiv preprint arXiv:2307.14377, 2023
2023 arXiv
-
[15]
Mandolini, C
M. Mandolini, C. Favi, M. Germani, M. Marconi, and R. Raffaeli. An Analytical Cost Estimation Approach for Generic Sheet Metal 3D Mod- els.Computer-Aided Design and Applications, 16(5):936–950, 2019. ISSN 16864360. doi: 10.14733/cadaps.2019.936-950
2019 doi
-
[16]
Ni and M
B. Ni and M. J. Buehler. MechAgents: Large language model multi- agent collaborations can solve mechanics problems, generate new data, and integrate knowledge.Extreme Mechanics Letters, 67:102131, 2024. ISSN 23524316. doi: 10.1016/j.eml.2024.102131
2024
-
[17]
Ocker, S
F. Ocker, S. Menzel, A. Sadik, and T. Rios. From idea to cad: A lan- guage model-driven multi-agent system for collaborative design.arXiv preprint arXiv:2503.04417, 2025
2025 arXiv
-
[18]
Paviot, V
T. Paviot, V . Fortineau, and S. Lamouri. Iterative diagnosis-driven augmented generation (iddag) for programmatic 3d cad. InPro- ceedings of the 13th International Conference on Model-Based Soft- ware and Systems Engineering (MODELSWARD 2025), pages 474–
2025
-
[19]
Picard, K
C. Picard, K. M. Edwards, A. C. Doris, B. Man, G. Giannone, M. F. Alam, and F. Ahmed. From concept to manufacturing: Evaluat- ing vision-language models for engineering design.arXiv preprint arXiv:2311.12668, 2023
2023 arXiv
-
[20]
T. Rios, S. Menzel, and B. Sendhoff. Large language and text-to-3d models for engineering design optimization. In2023 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1704–1711. IEEE, 2023
2023
-
[21]
A. R. Sadik, S. Brulin, and M. Olhofer. Coding by design: Gpt-4 empowers agile model driven development.arXiv preprint arXiv:2310.04304, 2023
2023 arXiv
-
[22]
A. R. Sadik, A. Ceravola, F. Joublin, and J. Patra. Analysis of chatgpt on source code.arXiv preprint arXiv:2306.00597, 2023
2023 arXiv
-
[23]
Salehi and C
V . Salehi and C. McMahon. Methodological Integration of Parametric Associative CAD Systems in Product Lifecycle Management (PLM) Environment. InV olume 2: 29th Computers and Information in En- gineering Conference, Parts A and B, pages 505–514, San Diego, California, USA, 200...
2009 doi
-
[24]
J. D. Summers and J. J. Shah. Mechanical Engineering Design Com- plexity Metrics: Size, Coupling, and Solvability.Journal of Mechani- cal Design, 132(2):021004, 2010. ISSN 1050-0472, 1528-9001. doi: 10.1115/1.4000759
2010 doi
-
[25]
J. Tian, J. Hou, Z. Wu, P. Shu, Z. Liu, Y . Xiang, B. Gu, N. Filla, Y . Li, N. Liu, et al. Assessing large language models in mechanical engineer- ing education: A study on mechanics-focused conceptual understand- ing.arXiv preprint arXiv:2401.12983, 2024
2024 arXiv
-
[26]
X. Wang, N. Anwer, Y . Dai, and A. Liu. ChatGPT for design, man- ufacturing, and education.Procedia CIRP, 119:7–14, 2023. ISSN 22128271. doi: 10.1016/j.procir.2023.04.001
2023 doi
-
[27]
Wollstadt, M
P. Wollstadt, M. Bujny, S. Ramnath, J. J. Shah, D. Detwiler, and S. Men- zel. CarHoods10k: An Industry-Grade Data Set for Representation Learning and Design Optimization in Engineering Applications.IEEE Transactions on Evolutionary Computation, 26(6):1221–1235, 2022. ISSN 1941...
2022
-
[28]
M. Wong, T. Rios, S. Menzel, and Y . S. Ong. Generative AI- based Prompt Evolution Engineering Design Optimization With Vision- Language Model. In2024 IEEE Congress on Evolutionary Computa- tion (CEC), pages 1–8, 2024. doi: 10.1109/CEC60901.2024.10611898
2024 arXiv
-
[29]
S. Wu, A. Khasahmadi, M. Katz, P. K. Jayaraman, Y . Pu, K. Willis, and B. Liu. Cad-llm: Large language model for cad generation. In Proceedings of the neural information processing systems conference. neurIPS, 2023
2023
-
[30]
Yoshimura and H
K. Yoshimura and H. Oishi. Variability management for large language model tasks: Practical insights from an industrial application. InPro- ceedings of the 28th ACM International Systems and Software Product Line Conference, pages 148–152, 2024
2024
-
[31]
M. S. Yousaf, M. Bujny, N. Zurbrugg, D. Detwiler, and F. Duddeck. Similarity control in topology optimization under static and crash load- ing scenarios.Engineering Optimization, 53(9):1523–1538, 2021
2021
-
[32]
M. S. Yousaf, D. Detwiler, F. Duddeck, S. Menzel, S. Ramnath, N. Zur- brugg, and M. Bujny. Similarity-Driven Topology Optimization for Statics and Crash via Energy Scaling Method.Journal of Mechanical Design, 145(101707), 2023. ISSN 1050-0472. doi: 10.1115/1.4062943
2023 doi
-
[33]
Z. Yuan, H. Lan, Q. Zou, and J. Zhao. 3d-premise: Can large language models generate 3d shapes with sharp features and parametric control? arXiv preprint arXiv:2401.06437, 2024
2024 arXiv
-
[34]
Z. Yuan, J. Shi, and Y . Huang. OpenECAD: An efficient visual lan- guage model for editable 3D-CAD design.Computers & Graphics, 124:104048, 2024. ISSN 0097-8493. doi: 10.1016/j.cag.2024.104048
2024
-
[35]
Zhu and J
Q. Zhu and J. Luo. Generative Pre-Trained Transformer for Design Concept Generation: An Exploration.Proceedings of the Design Soci- ety, 2:1825–1834, 2022. ISSN 2732-527X. doi: 10.1017/pds.2022.185
2022 doi
-
[36]
Zhu and J
Q. Zhu and J. Luo. Generative Transformers for Design Concept Gen- eration.Journal of Computing and Information Science in Engineering, 23(041003), 2023. ISSN 1530-9827. doi: 10.1115/1.4056220
2023 doi
-
[37]
Q. Zhu, X. Zhang, and J. Luo. Biologically Inspired Design Con- cept Generation Using Generative Pre-Trained Transformers.Journal of Mechanical Design, 145(041409), 2023. ISSN 1050-0472. doi: 10.1115/1.4056598
2023 doi
-
[480]
ISBN 978-989-758-729-0
SCITEPRESS – Science and Technology Publications, Lda, 2025. ISBN 978-989-758-729-0. doi: 10.5220/0013443500003896
2025 doi
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.