REVIEW 5 major objections 6 minor 13 references
A single 27B multimodal model trained on six industrial CAD corpora becomes a shared substrate that beats both its base and a frontier closed model on CAD vision, CadQuery, COM APIs, and assemblies—without forgetting general skills.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 19:24 UTC pith:MTIRAEIJ
load-bearing objection Solid applied SFT paper: real CadQuery/COM jumps on a scarce industrial stack, with the usual sibling-eval and small-n caveats—not a methods breakthrough, but worth engaging if you build CAD agents. the 5 major comments →
IndustryForge-27B: A Domain-Enhanced Multimodal Foundation Model for Industrial CAD
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Multi-task supervised fine-tuning of a 27B vision-language model on six integrated industrial-CAD sub-corpora (~52k samples) produces a single foundation that substantially raises CAD-domain performance—mean +33.65 pp over the base, winning all four domain benchmarks against GPT-5.4—while retaining and slightly improving general capability (+1.56 pp mean on eleven public benchmarks, no catastrophic forgetting), thereby serving as a usable common substrate for full-stack industrial CAD agents.
What carries the argument
Unified multi-task SFT on six industrial-CAD sub-corpora (CAD-VQA, text2cadquery, text2cadquery-assembly, com_2d, com_3d, com_assembly), mixed flat and trained with LoRA, DeepSpeed ZeRO-3, sequence parallel, and padding-free packing, with sandbox execution, Chamfer-Distance filtering, and UUID/AST decontamination as quality gates.
Load-bearing premise
The four CAD benchmarks—especially the small COM set and the still-low assembly pass rate—plus decontamination against related pipelines are assumed independent and representative enough that the large gains prove a generally superior industrial CAD foundation rather than in-distribution wins on author-adjacent tests.
What would settle it
Re-evaluate IndustryForge-27B and the same baselines on a held-out industrial CAD suite built from unrelated plant drawings, COM workflows, and multi-part assemblies never seen in the six sub-corpora or sibling agent pipelines; if the mean domain lead over the base and over GPT-5.4 collapses or assembly pass rate stays near zero under that protocol, the substrate claim fails.
If this is right
- Downstream CAD agents can start from one shared checkpoint for drawing understanding, CadQuery, COM operation, and assembly instead of each re-solving basic CAD literacy.
- COM-as-code operation of SolidWorks/Inventor becomes a first-class foundation skill rather than a rare, near-zero capability of general models.
- Assembly-level generation moves from near-total failure (0–4%) to a non-zero usable baseline (~15%) that closed-loop agents can iterate on.
- CAD-oriented multi-task SFT can spill over positively into geometric visual reasoning and structured figure-text tasks on general benchmarks without wiping out broad knowledge.
- A planned feedback loop can recycle high-quality agent traces back into the next substrate round, linking single-shot SFT to closed-loop self-improvement.
Where Pith is reading between the lines
- If the substrate pattern holds, other heavy Windows professional apps with scarce public COM/API data (beyond CAD) may benefit from the same curated sandbox-executed multi-task recipe.
- The still-dominant single-part mix and ~15% assembly pass rate imply that scale and verification of assembly data—not just more SFT steps—will be the next bottleneck for production-ready multi-part systems.
- Positive transfer on math_vision and chart-like tasks suggests industrial drawing SFT is a practical way to strengthen multi-view spatial reasoning in general multimodal models.
- Open release of the substrate could shrink duplicated CAD-foundation work across teams and make fair head-to-head agent comparisons easier.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents IndustryForge-27B, a 27B multimodal model obtained by multi-task supervised fine-tuning of Qwen3.5-VL-27B on six industrial-CAD sub-corpora (~52k samples) covering CAD visual QA, CadQuery part/assembly code, and SolidWorks/Inventor COM APIs. It reports large gains on four author-curated CAD-domain benchmarks (mean +33.65 pp over the base; wins 4/4 vs GPT-5.4), with CadQuery rising 7.82%→77.88% and COM 28.33%→73.33%, while eleven general benchmarks show a small mean gain (+1.56 pp) and no catastrophic forgetting. The stated contribution is not an end-to-end CAD system but a shared foundation-model substrate for downstream agents (IterCAD, AssemCAD, ComAct, SimLoop).
Significance. If the domain gains transfer beyond the authors’ sibling evaluation stacks, the work would be a useful systems contribution: a single open multimodal checkpoint that jointly covers drawing understanding, parametric CadQuery, Windows COM operation, and a non-zero assembly pass rate—capabilities that general VLMs and single-task CAD fine-tunes currently split. Strengths include sandbox execution plus Chamfer-Distance filtering of training code, UUID/AST/Chamfer decontamination against listed benches, a clear pass@CD≤1e-3 metric, and explicit measurement of general-capability retention. The absolute assembly result (15.38%) and the COM lift where public data are scarcest are practically interesting if they hold under independent evaluation.
major comments (5)
- [§3.2.6, §4.1–4.2] §3.2.2–3.2.4 and §4.1–4.2: Three of the four domain benches (Zero-To-CAD / text2cadquery, com_QA_mini, CadQuery Assembly) are co-sourced with the same IterCAD / ComAct / AssemCAD pipelines that supply SFT data. UUID and AST-normalised / Chamfer decontamination (§3.2.6) block exact and trivial-variant leakage but do not address shared generators, COM helper idioms, geometry families, or template structure. The headline “+33.65 pp / wins 4/4 vs GPT-5.4” and the “common substrate” claim are load-bearing on transfer; the manuscript needs either an external or held-out plant-style split, or a quantitative analysis showing that pass@CD and COM artefact-match are not inflated by in-family homology.
- [§4.1, Figure 4] §4.1 and Figure 4: COM CAD is evaluated on com_QA_mini_0311 with only 60 samples. A jump 28.33%→73.33% (+45 pp) and +43.33 pp over GPT-5.4 is central to the differentiation narrative, but with n=60 binomial uncertainty is large and a few template families could dominate. Report confidence intervals, item-level breakdown by API family (2D/3D/assembly), and ideally a larger or independently sourced COM set before treating this as a settled closed-source gap.
- [§4.2, Table 1, Limitations] §4.2 and Limitations: Assembly pass@CD≤1e-3 reaches 15.38% from 0/3.85%, which is directionally important but still low for the claim of a “first usable non-zero” industrial foundation and for supporting upper-layer assembly agents. The training mix is only ~1k CadQuery-assembly + 5k COM-assembly against ~42k single-part samples (Table 1, Figure 3). Either strengthen the assembly evaluation (more tasks, partial-credit / constraint metrics, failure taxonomy) or temper the substrate/assembly claims to match the absolute level and data imbalance.
- [§1.2, §3.3.3, §4] §3.3 and §4: The paper argues multi-task SFT is necessary because single-task fine-tuning is “too narrow” for agent call diversity (§1.2), yet there is no ablation of joint training vs. single-corpus specialists (or vs. mixture-of-adapters) on the four domain benches and on cross-task interference. Without that, the unified-recipe claim and the flat sampling choice over a highly unbalanced mix (com_2d 20k vs assembly 1k) remain untested relative to the obvious alternative the introduction criticises.
- [§3.2.1, Table 1, References] §3.2.1 and Table 1: CAD-VQA is listed as “Independent [TODO]” with construction details and source still unfinished; References [6–9] are also TODO placeholders. CAD-VQA barely moves (84.10→88.23) while carrying one of the four domain axes and the “drawing understanding” skill in Figure 1. Incomplete data provenance for a reported benchmark and missing citations for the sibling systems the substrate is defined against are not presentation nits—they block reproducibility of a core claimed skill.
minor comments (6)
- [Title page, §4.1] Date line and arXiv stamp say July 31, 2026 / 30 Jul 2026 while citing “gpt-5.4” as the contemporary closed-source comparator; clarify model identity and evaluation date so readers can interpret the comparison.
- [§3.3] §3.3.3 states a “flat proportion” mix but does not give epochs, effective tokens per subset, LoRA rank/targets, learning rate, or IMAGE_MAX_TOKEN_NUM. These are needed to reproduce the v4.1 recipe on 8×A100.
- [§4.1, Table 2] Table 2: gpt-5.4 column is populated on the general suite in the table but the text says gpt-5.4 was not evaluated on the general suite (§4.1). Resolve the contradiction.
- [Figure 1, §1.3] Figure 1 and §1.3 count “four shared skills” but radiate five capability tags (COM split by application). Align the taxonomy in text and figure.
- [§4.4] Case studies (Figures 5–9) are described in captions but the qualitative evidence would be stronger with full prompts, full code, and CD values in an appendix or supplement.
- [Throughout] Minor prose issues: “Qwen3.5-27B” vs “Qwen3.5-VL-27B” naming inconsistency; “outperformsthe” spacing; author line “Y urui Dong”.
Circularity Check
Empirical multi-task SFT paper: measured gains are not forced by definition; only mild ecosystem self-citation for data provenance, not a circular derivation.
full rationale
IndustryForge-27B’s central claims are empirical: multi-task SFT of Qwen3.5-VL-27B on ~52k curated CAD/COM samples, then head-to-head measurement on four domain benches and eleven public general benches versus the untouched base and gpt-5.4. There is no algebraic or definitional chain in which a fitted quantity is renamed a prediction, no uniqueness theorem imported from the authors that forces the architecture or scores, and no ansatz smuggled in as a theorem. Training data is co-sourced with IterCAD / AssemCAD / ComAct and those works are cited as sibling arXiv preprints (several still TODO), which creates ecosystem dependence and a legitimate contamination/generalization concern—but that is eval-homology risk, not circularity of derivation. The paper states UUID-, AST-, and Chamfer-level decontamination against the reported benches and reports absolute metrics (e.g. assembly 15.38%, COM n=60) rather than deriving them from the training mix. General-suite retention (+1.56 pp) is likewise measured on external public benchmarks. Under the circularity criteria, this is a normal non-finding with at most minor non-load-bearing self-citation for data lineage.
Axiom & Free-Parameter Ledger
free parameters (4)
- Chamfer Distance pass threshold (10^-3) =
≤ 1e-3
- Flat multi-task sampling ratios over six unbalanced corpora =
uniform/flat mix (unstated exact probabilities)
- LoRA / sequence_parallel_size / IMAGE_MAX_TOKEN_NUM and related SFT hyperparameters =
sequence_parallel_size=8; other values not fully reported
- COM and assembly corpus acceptance filters (sandbox success + structural checks) =
execution-success and structural/geometry gates (thresholds not fully numeric)
axioms (6)
- domain assumption Supervised fine-tuning on explicit multimodal QA/code pairs is sufficient to install industrial CAD skills without continued pretraining of the base distribution.
- domain assumption Sandbox execution plus Chamfer/structural match is an adequate proxy for manufacturable, assemblable engineering correctness.
- domain assumption UUID-level and AST-normalized dedup (plus Chamfer NN on COM artefacts) eliminates meaningful train/test leakage relative to sibling benchmarks.
- domain assumption Windows COM automation of SolidWorks/Inventor/AutoCAD is the right native action interface for industrial software agents (COM-as-Action).
- standard math Standard transformer multimodal SFT optimization (next-token on packed instruction data) with LoRA and ZeRO-3 converges to the reported skills without hidden curriculum tricks.
- ad hoc to paper GPT-5.4 is a fair strong closed-source comparator for public CAD code/vision tasks at time of writing.
invented entities (2)
-
IndustryForge-27B (unified industrial-CAD multimodal substrate)
no independent evidence
-
CAD-VQA-4k corpus
no independent evidence
read the original abstract
Automating industrial CAD design and manufacturing places distinctive demands on multimodal foundation models: the model must see engineering drawings and 3D geometry screenshots, write correct parametric-modelling scripts and Windows COM API code, and cover the full range from single parts to assemblies. General-purpose multimodal models fall short on these tasks, while single-task fine-tuning is too narrow to support the diverse calls that upper-layer agents issue. We build IndustryForge-27B on top of Qwen3.5-VL-27B by curating and integrating six industrial-CAD sub-corpora totalling $\sim$52k multimodal samples---CAD Visual QA (CAD-VQA), parametric CAD code (text2cadquery), assembly-level CAD code (text2cadquery-assembly), and three COM sub-corpora for Inventor / SolidWorks (com_2d / com_3d / com_assembly)---and training with a unified multi-task SFT recipe. Across four CAD-domain benchmarks IndustryForge-27B lifts the base model by $+33.65$~pp on average and outperforms the strong closed-source model GPT-5.4 on all four; across eleven general-capability benchmarks it retains, and slightly improves upon, the base model ($+1.56$~pp mean, no catastrophic forgetting). IndustryForge-27B will serve as the common substrate for downstream industrial-agent projects, providing a unified starting point for a full-stack industrial agent that spans from CAD design to industrial-software operation, from parts to assemblies, and from single-shot generation to closed-loop self-improvement.
Figures
Reference graph
Works this paper leans on
-
[1]
CadQuery: a python parametric CAD scripting framework based on OpenCascade, 2024
CadQuery Team. CadQuery: a python parametric CAD scripting framework based on OpenCascade, 2024. https://github.com/CadQuery/cadquery
2024
-
[2]
OpenSCAD: The programmers solid 3d cad modeller, 2024.https://openscad.org
OpenSCAD Team. OpenSCAD: The programmers solid 3d cad modeller, 2024.https://openscad.org
2024
-
[3]
DeepCAD: A deep generative network for computer-aided design models.ICCV, 2021
Rundi Wu, Chang Xiao, and Changxi Zheng. DeepCAD: A deep generative network for computer-aided design models.ICCV, 2021
2021
-
[4]
Text2CAD: Generating sequential CAD designs from beginner-to-expert level text prompts.NeurIPS, 2024
Mohammad Sadil Khan and Others. Text2CAD: Generating sequential CAD designs from beginner-to-expert level text prompts.NeurIPS, 2024
2024
-
[5]
Xiang Xu, Karl D. D. Peng, Karl D. D. Willis, and Others. SkexGen: Autoregressive generation of CAD construction sequences with disentangled codebooks.ICML, 2022
2022
-
[6]
ComAct: COM-as-Action for industrial CAD software agents.arXiv preprint, 2026
ComAct Team. ComAct: COM-as-Action for industrial CAD software agents.arXiv preprint, 2026. [TODO: replace with final citation]
2026
-
[7]
IterCAD: Closed-loop CAD code generation with compiler and visual feedback.arXiv preprint, 2026
IterCAD Team. IterCAD: Closed-loop CAD code generation with compiler and visual feedback.arXiv preprint, 2026. [TODO: replace with final citation]
2026
-
[8]
AssemCAD: Production-ready assembly generation via what-how-verify decomposition
AssemCAD Team. AssemCAD: Production-ready assembly generation via what-how-verify decomposition. arXiv preprint, 2026. [TODO: replace with final citation]
2026
-
[9]
CAD-LLM: Leveraging large language models for parametric CAD generation.arXiv preprint,
Anonymous. CAD-LLM: Leveraging large language models for parametric CAD generation.arXiv preprint,
-
[10]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023
Pith/arXiv arXiv 2023
-
[11]
Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and Others. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025
Pith/arXiv arXiv 2025
-
[12]
InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and Others. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.CVPR, 2024. 14
2024
-
[2024]
[TODO: replace with final citation]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.