REVIEW 1 major objections 5 minor 24 references
CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation
T0 review · 1 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read CADEngBench argues that CAD evaluation must test engineering behavior instead of appearance, and shows that current models fail most such checks.
desk verdict A serious, unusually honest CAD benchmark that moves evaluation from appearance to engineering behavior; the assembly track has a disclosed ground-truth completeness issue that should temper the headline, not sink the paper. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a layered evaluation hierarchy in which a candidate passes a stage only after satisfying all earlier stages. The parametric track maps a program $C(\theta)$ to a solid $B = C(\theta)$ and tests: L0 executes and re-imports a STEP solid; L1 checks engineering requirements and DFM screens (minimum wall 1.0 mm, hole diameter at least 2.0 mm, depth-to-diameter ratio at most 8); L2-Z rebuilds the same $\text{build}(\text{params})$ function over a fixed set of parameter values and requires every state to keep the declared parameter-to-geometry relation while preserving protected invariants; L2-E applies a source-native edit and checks the requested target plus hidden preservation checks; L3 compares generated and reference solids under identical materials, supports, loads, and mesh profiles, using log-ratio disagreement $\delta = |\log((q(C)+\epsilon)/(q(R)+\epsilon))|$ on 95th-percentile von Mises stress, normalized displacement, and compliance or stress concentration. The assembly track takes a hypothesis $h = (j, e_A, e_B)$ with a joint family $j$ and exact face or edge entities from each body, scores role-aware retrieval (Entity@k and Typed@k), then predicts the joint frame and executes allowed and blocked motion in a rigid-body simulator. This machinery is what separates “looks right” from “behaves right”.
What would settle it
Take a sample of assembly pairs and ask a human engineer to list every mechanically plausible joint between the two bodies without seeing the source records; if a large share of engineer-approved mates are not in the recorded set, then CADEngBench-A's A1/A2 pass rates understate true assembly competence. In the opposite direction, exhibit one scored-miss rank-1 hypothesis that is kinematically valid in simulation, meaning the predicted joint type, entity pair, and frame allow exactly the permitted motions and block the forbidden ones; a single such case shows the metric can reject a working hypothesis.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that each evaluation layer removes a large share of apparent successes, and the layers behave like nearly independent capabilities. Executable code can violate design intent; parameter families that rebuild successfully can still diverge from the reference stress and deformation response; and assembly predictions that localize the correct region often fail to recover the recorded joint type and exact B-Rep entities. The evidence includes 1,030 of 2,400 generated programs passing L0 execution but only 432 also passing L1 engineering and DFM requirements; editing supplied CAD passing far more often than zero-to-CAD generation (1,071 item–model pairs pass editing only versus 231 generation only); L3 matched FEA pair pass reaching 46.3% for the best model and being uncorrelated with how many simulations a model can run (ρs = 0.024); and in assembly, 542 of 960 requests putting a correct B-Rep entity pair at rank 1, but only 143 of those also specifying the correct joint family and body ordering, with end-to-end kinematic pass at 15.8% for the best model. The conclusion is that CAD evaluation must measure engineering behavior, not plausible shape.
Load-bearing premise
The assembly scores treat the source-recorded joint alternatives as the complete set of correct answers, so a mechanically valid but unrecorded mating is counted as a miss; the paper notes that absence of a relation is not treated as a verified negative example.
Editorial extensions
If this is right
- A benchmark that scores only executability or shape similarity overstates capability: 42% of executable generated programs fail stated engineering or DFM requirements, so requirement checks are needed to detect failures of design intent.
- Parameter-family testing is necessary because a parameter can appear correct at its default value while being inert or coupled to the wrong feature elsewhere; L2-Z requires every evaluated state to behave.
- A successful FEA solve does not establish correct physics: L3 reach is uncorrelated with agreement to the reference (ρs = 0.024), so benchmarks should compare generated and reference structural response under the same analysis contract.
- Assembly evaluation should require exact B-Rep entity grounding and typed joint recovery rather than region location; the gap between Entity@1 and Typed@1 shows that most correct localizations still miss the recorded joint.
- Capability ranks are weakly associated across generation, editing, parametric, and assembly scores, so no single model ranking describes CAD ability and multi-stage evaluation is required.
Reading between the lines
- If the source joint records are incomplete, A1/A2 pass rates likely underestimate assembly competence; a human-annotation study that counts mechanically valid but unrecorded mates as correct would quantify that ceiling.
- The pin-slot 0% result rests on six items, which the paper itself flags as too few for a broad claim; a larger pin-slot subset could change family-level conclusions.
- The L2-Z perturbation protocol could transfer to other generative design domains, such as layouts, structural frames, or circuit boards, where a plausible default hides wrong parameter wiring.
- The “editing is easier than generation” result may be partly a function of construction-history coupling, so future benchmark design should stratify edit tasks by how deeply the edited feature is joined or cut into existing geometry.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CADEngBench, a two-track benchmark for evaluating CAD generation and assembly behavior beyond appearance. CADEngBench-P comprises 300 parametric parts used for 600 zero-to-CAD and functional-editing tasks, evaluated through B-Rep validity (L0), engineering and DFM requirements (L1), parametric perturbation and functional editing (L2-Z and L2-E), and matched linear-static FEA in CalculiX (L3). CADEngBench-A comprises 150 body pairs (120 evaluation pairs after development exclusion) scored on ranked joint retrieval, exact face/edge grounding, joint-frame prediction, and kinematic verification in PyBullet. Eight multimodal models are evaluated. The reported results show that executable code frequently violates engineering requirements, matched FEA agreement is rare, editing is easier than generation, and assembly predictions often locate the correct region but rarely recover the source-recorded joint and mating entities.
Significance. If the evaluation is valid, the benchmark addresses a genuine gap: existing CAD benchmarks often stop at shape similarity or executable-code checks, whereas CADEngBench attempts to measure behavioral properties such as parametric control, functional editing, physical response, and assembly grounding. The paper has notable methodological strengths: scoring is based on deterministic B-Rep measurement and replay; thresholds and analysis cards are disclosed; confidence intervals use item-clustered bootstraps; pairwise comparisons use exact McNemar tests with Holm correction; and limitations are stated explicitly. The two-track design and the L0-L3 gated hierarchy are useful contributions. The principal risk is the completeness of the assembly ground-truth set: because source-recorded joint alternatives are treated as the accepted answers, mechanically valid but unrecorded mating relations are scored as misses, which could overstate the assembly-failure conclusions.
major comments (1)
- [Supplement A.2, F.1, F.2; main text 'Complex Assembly Behaviour'] The L2-Z results are reported with model-specific denominators (N_s = 287-300 in Table 2), while the experimental protocol states that malformed responses, timeouts, and build failures are retained as failures. If an item is excluded from the denominator because a model's output does not satisfy the required interface, the reported pass rates are not directly comparable across models and are not strict item pass rates. The paper should either report L2-Z over the fixed 300-item denominator for every model, with excluded cases counted as failures, or justify why model-specific evaluability is appropriate and show the sensitivity of the conclusions to the choice of denominator.
minor comments (5)
- [Results, Fig. 7] The main text states that '554 solve but violate engineering limits or disagree with the reference stress or deformation,' but Figure 7 shows 270 engineering-limit failures and 593 FEA-quantity mismatches, which sum to 863. Please correct the number or clarify which subset of outcome categories is included in the 554 figure.
- [Results, 'Generation and Editing'] The sentence 'Functional-edit L2 also falls from these slices show that the remaining difficulty is not merely parsing an edit request' is grammatically incomplete. It should be rewritten to state the intended observation, for example that the functional-edit gap is concentrated in histories involving joins, cuts, or multiple bodies.
- [Supplement B.5 and F.1] When describing the source relations in the A1 prompt, the manuscript says the dataset 'consolidates alternative joints observed for identical pairs of parts across different assemblies'; this is helpful and should also be stated in the main text near the assembly results, since it directly qualifies how the 14.8% result should be read.
- [Supplement D.3 and main Eq. (3)] The log-ratio tolerance is described in the main text as a multiplicative comparison, and the supplement gives equivalent ratio intervals in Table 8. The connection between them is clear, but a one-sentence note that log(1.25) corresponds to the [0.8, 1.25] interval would prevent reader confusion, especially for the asymmetric body-acceleration interval [2/3, 1.5].
- [Table 2 caption] The caption for Table 2 says 'P uses 300 tasks except L2-Z,' but the L3 reach and pair-pass columns also have model-dependent denominators. Please clarify that L3 pair pass is conditional on comparable generated/reference state pairs, as is done in the text.
Circularity Check
No significant circularity: scoring is grounded in external source datasets and deterministic replay, and the acknowledged joint-alternative incompleteness is a coverage caveat, not a circular step.
full rationale
The paper's derivation chain is self-contained against external references rather than against its own outputs. In CADEngBench-P, ground truth is produced by deterministic replay of source CadQuery programs and Fusion 360 histories, by direct B-Rep measurement with OpenCascade, and by matched CalculiX FEA on the source reference solids; model programs are executed and compared to these external references. Supplement A.3 explicitly bars LLM/VLM annotations from scoring: 'These annotations help construct the dataset’s task descriptions but are not reference answers.' L2-Z and L2-E check measured geometric outcomes, not whether a model repeats its own prompt. In CADEngBench-A, the hidden references are source-recorded joint families, entity pairs, frames, and motion semantics from the Fusion 360 Assembly-Joint dataset (Jones et al. 2021), and the models are never fitted to those records; A1 and A2 are exact role-aware comparisons to those external labels. The paper's only notable caveat, stated in Supplement A.2 and F.2, is that 'the source records provide positive joint alternatives rather than an exhaustive catalogue of every mechanically feasible relation,' so a mechanically valid but unrecorded mating could be scored as a miss. That is a benchmark-coverage limitation that may understate model competence, but it does not make the reported scores equivalent to the benchmark's inputs by construction. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors' own prior work, and no ansatz smuggled in via self-citation. The thresholds and FEA limits are disclosed hand-chosen criteria applied identically to reference and candidate geometry, which is standard evaluation practice rather than circularity.
Assumptions & free parameters
free parameters (10)
- DFM minimum wall thickness =
1.0 mm
- DFM minimum hole diameter =
2.0 mm
- DFM maximum hole depth/diameter ratio =
8
- L3 surface-load response ratio tolerance =
[0.8, 1.25], log-error 0.2231
- L3 body-acceleration stress/concentration ratio tolerance =
[2/3, 1.5]
- L3 absolute engineering limits =
stress utilization <=1.0; normalized displacement <=0.02
- A2 joint origin tolerance =
1% of Body-A bounding-box diagonal
- A2 direction/rotation tolerance =
5 degrees
- L3 material card elastic modulus and yield strength =
E=70,000 MPa, nu=0.33, yield=250 MPa, safety factor 2.0
- Log-ratio stabilization epsilon =
not disclosed
assumptions (6)
- domain assumption Source CAD programs and Fusion histories replay deterministically, recovering the same B-Reps and parameters used for scoring.
- domain assumption Recorded joint relations in the Fusion 360 Assembly-Joint dataset are the accepted ground truth for A1 and A2 scoring.
- domain assumption Linear-static FEA with the benchmark-assigned material card and boundary selectors is a valid proxy for engineering behavior of the part family.
- domain assumption OpenCascade B-Rep measurements and the certified global-thickness extractor are accurate.
- domain assumption PyBullet joint adapters faithfully represent the allowed and blocked motions of the seven joint families.
- domain assumption LLM and VLM annotation constraints prevent benchmark prompts from leaking scoring answers.
Cite this review
Pith. "Pith review of CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation." pith.science (2026). https://pith.science/paper/ESZFJVJS
@misc{pith2026260809296,
author = {Pith},
title = {Pith review of: CADEngBench: It Looks Like CAD, but Does It Work? Evaluating Parametric Design, Assembly Reasoning, and Physics Simulation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ESZFJVJS}},
note = {Machine review of arXiv:2608.09296}
}
read the original abstract
A CAD model is not engineering-grade merely because it looks correct. It must satisfy design requirements, respond predictably to parameter changes, support controlled edits, match a reference structural response under a declared analysis, and connect to other parts through valid joints. We present CADEngBench, a two-track benchmark for these capabilities. CADEngBench-P evaluates 300 parametric parts, each used for one zero-to-CAD task and one functional-editing task (600 tasks in total), through boundary-representation (B-Rep) validity, engineering and DFM checks, parameter-family perturbations, functional editing, and matched linear-static FEA in CalculiX. CADEngBench-A evaluates 150 body pairs through ranked joint retrieval, exact face-and-edge grounding, joint-frame prediction, and kinematic verification. Across eight multimodal, code-capable models, editing supplied CAD is substantially easier than generating it, while complex edits and matched FEA remain difficult. Assembly predictions often locate the relevant region but fail to recover the recorded joint or mating entities. These results show that CAD evaluation must test engineering behavior rather than appearance alone.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[2]
BenDFM: A Tax- onomy and Synthetic CAD Dataset for Manufacturability Assessment in Sheet Metal Bending. arXiv:2603.13102. Berger, E.; Usama, M.; Mehlstäubl, J.; Saske, B.; and Paetzold-Byhain, K
-
[3]
Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering De- sign. arXiv:2605.19717. Dhondt, G. 2004.The Finite Element Method for Three- Dimensional Thermomechanical Applications. Wiley. Dong, X.; Li, C.; Zheng, P.; Han, C.; Jing, J.; Shen, H.; Song,Y.;andYang,Z.2026. HistCAD:AConstraint-Aware ParametricHistory-BasedCADRepresentat...
arXiv 2004
-
[5]
CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation
CADBench: A Multi- modal Benchmark for AI-Assisted CAD Program Genera- tion. arXiv:2605.10873. Elistratov, M.; Barannikov, M.; Ivanov, G.; Khrulkov, V.; Konushin,A.;Kuznetsov,A.;andZhemchuzhnikov,D.2026. CADEvolve:CreatingRealisticCADviaProgramEvolution. arXiv:2602.16317. Geuzaine, C.; and Remacle, J.-F
work page Pith review arXiv 2026
-
[6]
IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing. arXiv:2606.13368. Jones, B.; Hildreth, D.; Chen, D.; Baran, I.; Kim, V. G.; and Schulz, A
- [10]
-
[11]
KimiK2.5:VisualAgenticIntelligence
MoonshotAI.2026. KimiK2.5:VisualAgenticIntelligence. arXiv:2602.02276. OpenAI
arXiv 2026
-
[12]
neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing
neuralCAD-Edit: An Expert Benchmark for Multimodal- Instructed 3D CAD Model Editing. arXiv:2604.16170. Pyatov, V.; Bobrovskikh, G.; Galochkin, S.; Boldyrev, N.; Voynov, O.; Filippov, A.; Ferrer, G.; Wonka, P.; and Bur- naev, E
-
[13]
CADFS: A Big CAD Program Dataset and Framework for Computer-Aided Design with Large Lan- guage Models. arXiv:2605.01925. QwenTeam.2026. Qwen3.5-35B-A3BModelCard. https:// huggingface.co/Qwen/Qwen3.5-35B-A3B. Accessed
arXiv 2026
Show all 24 references
-
[14]
arXiv:2604.10992
ArtiCAD: Articulated CAD Assembly Design via Multi-Agent Code Generation. arXiv:2604.10992. Son, G.; Park, J.; Park, S.; Ahn, S.; and Yu, Y
-
[15]
arXiv:2605.17448
Self- Improving CAD Generation Agents with Finite Element Analysis as Feedback. arXiv:2605.17448. Tian,Y.;Willis,K.D.D.;Omari,B.A.;Luo,J.;Ma,P.;Li,Y.; Javid, F.; Gu, E.; Jacob, J.; Sueda, S.; Li, H.; Chitta, S.; and Matusik, W
-
[16]
arXiv:2309.16909
ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility. arXiv:2309.16909. Tian, Y.; Xu, J.; Li, Y.; Luo, J.; Sueda, S.; Li, H.; Willis, K. D.D.;andMatusik,W.2022. AssembleThemAll:Physics- BasedPlanningforGeneralizableAssemblybyDisassembly. arX...
2022 arXiv
-
[17]
arXiv:2605.18430
Text2CAD-Bench: A Bench- mark for LLM-based Text-to-Parametric CAD Generation. arXiv:2605.18430. Willis, K. D. D.; Jayaraman, P. K.; Chu, H.; Tian, Y.; Li, Y.; Grandi, D.; Sanghi, A.; Tran, L.; Lambourne, J. G.; Solar-Lezama, A.; and Matusik, W
-
[22]
grade A or grade B
The complete set is covered by seven joint families: rigid, revolute, slider, cylindrical, pin-slot, planar, and ball relations. Familycountsmayoverlapbecauseonepaircancontainseveralrecordedalternatives.Thesourcerecordsprovidepositivejoint alternatives rather than an exhaustiv...
2021
-
[24]
arXiv:2105.12238
AutoMate: A Dataset and Learning Approach for Automatic Mating of CAD Assemblies. arXiv:2105.12238. Khan, M. S.; Sinha, S.; Sheikh, T. U.; Stricker, D.; Ali, S. A.; and Afzal, M. Z
-
[25]
arXiv:2409.17106
Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts. arXiv:2409.17106. Li, D.; Zhang, J.; Egger, B.; Chatterjee, M.; Lohit, S.; Marks, T. K.; and Cherian, A
-
[26]
arXiv:2605.12845
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects. arXiv:2605.12845. Willis, K. D. D.; Jayaraman, P. K.; Chu, H.; Tian, Y.; Li, Y.; Grandi, D.; Sanghi, A.; Tran, L.; Lambourne, J. G.; Solar-Lezama, A.; and Matusik, W
-
[27]
arXiv:2111.12772
JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints. arXiv:2111.12772. Willis, K. D. D.; Pu, Y.;Luo, J.; Chu, H.; Du, T.; Lambourne, J. G.;Solar-Lezama, A.; and Matusik, W
-
[28]
arXiv:2010.02392
Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences. arXiv:2010.02392. Wu, R.; Xiao, C.; and Zheng, C
2010 arXiv
-
[29]
arXiv:2105.09492
DeepCAD: A Deep Generative Network for Computer-Aided Design Models. arXiv:2105.09492. Zhang, H.; Liu, K.; Chen, M.; Li, L.; Yang, S.; Peng, C.; and Chen, H
-
[30]
arXiv:2605.10865
BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD. arXiv:2605.10865
-
[200]
Ball joints are tested about all three predicted axes
Allowed-motion probes prescribe0.25rad rotation or0.01m translation, clamped inside recorded source limits. Ball joints are tested about all three predicted axes. Where applicable, unintended rotation must remain below 10−7. Blocked-motion probes apply off-axis forces and torq...
2025
-
[2019]
arXiv:1812.06216
ABC: A Big CAD Model Dataset For Geometric Deep Learning. arXiv:1812.06216. Li, D.; Zhang, J.; Egger, B.; Chatterjee, M.; Lohit, S.; Marks, T. K.; and Cherian, A. 2026a. AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects. arXiv:2605.12845. Li, J.; Zhang, Q.; C...
-
[2025]
arXiv:2505.14646
CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation. arXiv:2505.14646. Doris, A. C.; Sony, J. T.; Nehme, G.; Syla, E.; No- bari, A. H.; and Ahmed, F
-
[2026]
arXiv:2604.24479
Zero-to-CAD: Agentic Synthesis of Inter- pretableCADProgramsatMillion-ScaleWithoutRealData. arXiv:2604.24479. Ballegeer, M.; and Benoit, D. F
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.