{"id":"47e38ed1-742e-4915-a141-e8618b1e759e","arxiv_id":"2607.28050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Multi-task SFT of Qwen3.5-VL-27B on six CAD corpora (~52k samples) raises four CAD benchmarks by +33.65 pp on average and beats GPT-5.4 on all four while holding general scores.","lead":"IndustryForge-27B is a 27B multimodal model fine-tuned on ~52k industrial CAD samples so it can read engineering drawings and write CadQuery and SolidWorks/Inventor COM code. It is meant as a shared base model for CAD agents rather than a finished design system.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Domain gains may reflect sibling-pipeline eval homology more than transferable industrial-CAD competence.","rationale":"The paper’s empirical story is coherent as scoped systems work: multi-task SFT on ~52k CAD/COM samples produces large CadQuery and COM jumps without wrecking the 11 general benches. Nothing in the text is internally contradictory. The load-bearing soft spot is exactly the reader’s: eval scale and independence relative to sibling data pipelines, not algebraic circularity. Decontamination is stated but unverifiable from the PDF; COM n=60 and assembly 15.38% limit how much “wins 4/4” and “common substrate” can be read as general industrial competence. TODOs on CAD-VQA provenance and sibling citations, plus promised rather than linked artifacts, keep confidence moderate. That already justifies CONDITIONAL rather than ACCEPT or REJECT; this pass does not move the needle further. A third-party held-out re-eval (or a public nearest-neighbor similarity audit of eval vs train) is the single check that would settle whether the concern lands.","tokens_in":11751,"tokens_out":630,"duration_ms":29321,"concrete_test":"Build or obtain one third-party held-out suite (e.g., ≥100 COM tasks and ≥50 assembly tasks from public STEP/assemblies or vendor tutorials never run through ComForge/IterCAD/AssemCAD generators), re-score IndustryForge-27B, Qwen3.5-27B, and GPT-5.4 with the same sandbox+CD/structural protocol. If IndustryForge’s mean domain lead falls below ~15 pp or it loses ≥2/4 benches to GPT-5.4, the substrate/general-superiority claim does not transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (usable common substrate; +33.65 pp mean; wins 4/4 vs GPT-5.4) rests on four author-defined domain benches whose data and metrics are tightly coupled to the same sibling stacks that supply SFT data: text2cadquery/Zero-To-CAD with IterCAD, com_QA_mini (n=60) with ComAct/ComForge, and CadQuery Assembly with AssemCAD (§3.2.2–3.2.4, §4.1–4.2). UUID/AST/Chamfer decontamination (§3.2.6) blocks exact and trivial-variant leakage but does not rule out shared generators, templates, COM helper idioms, or geometry families that make pass@CD≤1e-3 and COM artefact-match much easier in-family than on independent plant workflows. Absolute assembly remains 15.38%, COM n is tiny, CAD-VQA barely moves (84.10→88.23), and no downstream agent has yet integrated the checkpoint (future-tense substrate claim). If benches are in-distribution to the curation stack, the headline deltas overstate general industrial-CAD foundation quality.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents IndustryForge-27B, a 27B multimodal model obtained by multi-task supervised fine-tuning of Qwen3.5-VL-27B on six industrial-CAD sub-corpora (~52k samples) covering CAD visual QA, CadQuery part/assembly code, and SolidWorks/Inventor COM APIs. It reports large gains on four author-curated CAD-domain benchmarks (mean +33.65 pp over the base; wins 4/4 vs GPT-5.4), with CadQuery rising 7.82%→77.88% and COM 28.33%→73.33%, while eleven general benchmarks show a small mean gain (+1.56 pp) and no catastrophic forgetting. The stated contribution is not an end-to-end CAD system but a shared foundation-model substrate for downstream agents (IterCAD, AssemCAD, ComAct, SimLoop).","tokens_in":12029,"tokens_out":1738,"duration_ms":38812,"significance":"If the domain gains transfer beyond the authors’ sibling evaluation stacks, the work would be a useful systems contribution: a single open multimodal checkpoint that jointly covers drawing understanding, parametric CadQuery, Windows COM operation, and a non-zero assembly pass rate—capabilities that general VLMs and single-task CAD fine-tunes currently split. Strengths include sandbox execution plus Chamfer-Distance filtering of training code, UUID/AST/Chamfer decontamination against listed benches, a clear pass@CD≤1e-3 metric, and explicit measurement of general-capability retention. The absolute assembly result (15.38%) and the COM lift where public data are scarcest are practically interesting if they hold under independent evaluation.","major_comments":[{"comment":"§3.2.2–3.2.4 and §4.1–4.2: Three of the four domain benches (Zero-To-CAD / text2cadquery, com_QA_mini, CadQuery Assembly) are co-sourced with the same IterCAD / ComAct / AssemCAD pipelines that supply SFT data. UUID and AST-normalised / Chamfer decontamination (§3.2.6) block exact and trivial-variant leakage but do not address shared generators, COM helper idioms, geometry families, or template structure. The headline “+33.65 pp / wins 4/4 vs GPT-5.4” and the “common substrate” claim are load-bearing on transfer; the manuscript needs either an external or held-out plant-style split, or a quantitative analysis showing that pass@CD and COM artefact-match are not inflated by in-family homology.","section":"§3.2.6, §4.1–4.2"},{"comment":"§4.1 and Figure 4: COM CAD is evaluated on com_QA_mini_0311 with only 60 samples. A jump 28.33%→73.33% (+45 pp) and +43.33 pp over GPT-5.4 is central to the differentiation narrative, but with n=60 binomial uncertainty is large and a few template families could dominate. Report confidence intervals, item-level breakdown by API family (2D/3D/assembly), and ideally a larger or independently sourced COM set before treating this as a settled closed-source gap.","section":"§4.1, Figure 4"},{"comment":"§4.2 and Limitations: Assembly pass@CD≤1e-3 reaches 15.38% from 0/3.85%, which is directionally important but still low for the claim of a “first usable non-zero” industrial foundation and for supporting upper-layer assembly agents. The training mix is only ~1k CadQuery-assembly + 5k COM-assembly against ~42k single-part samples (Table 1, Figure 3). Either strengthen the assembly evaluation (more tasks, partial-credit / constraint metrics, failure taxonomy) or temper the substrate/assembly claims to match the absolute level and data imbalance.","section":"§4.2, Table 1, Limitations"},{"comment":"§3.3 and §4: The paper argues multi-task SFT is necessary because single-task fine-tuning is “too narrow” for agent call diversity (§1.2), yet there is no ablation of joint training vs. single-corpus specialists (or vs. mixture-of-adapters) on the four domain benches and on cross-task interference. Without that, the unified-recipe claim and the flat sampling choice over a highly unbalanced mix (com_2d 20k vs assembly 1k) remain untested relative to the obvious alternative the introduction criticises.","section":"§1.2, §3.3.3, §4"},{"comment":"§3.2.1 and Table 1: CAD-VQA is listed as “Independent [TODO]” with construction details and source still unfinished; References [6–9] are also TODO placeholders. CAD-VQA barely moves (84.10→88.23) while carrying one of the four domain axes and the “drawing understanding” skill in Figure 1. Incomplete data provenance for a reported benchmark and missing citations for the sibling systems the substrate is defined against are not presentation nits—they block reproducibility of a core claimed skill.","section":"§3.2.1, Table 1, References"}],"minor_comments":[{"comment":"Date line and arXiv stamp say July 31, 2026 / 30 Jul 2026 while citing “gpt-5.4” as the contemporary closed-source comparator; clarify model identity and evaluation date so readers can interpret the comparison.","section":"Title page, §4.1"},{"comment":"§3.3.3 states a “flat proportion” mix but does not give epochs, effective tokens per subset, LoRA rank/targets, learning rate, or IMAGE_MAX_TOKEN_NUM. These are needed to reproduce the v4.1 recipe on 8×A100.","section":"§3.3"},{"comment":"Table 2: gpt-5.4 column is populated on the general suite in the table but the text says gpt-5.4 was not evaluated on the general suite (§4.1). Resolve the contradiction.","section":"§4.1, Table 2"},{"comment":"Figure 1 and §1.3 count “four shared skills” but radiate five capability tags (COM split by application). Align the taxonomy in text and figure.","section":"Figure 1, §1.3"},{"comment":"Case studies (Figures 5–9) are described in captions but the qualitative evidence would be stronger with full prompts, full code, and CD values in an appendix or supplement.","section":"§4.4"},{"comment":"Minor prose issues: “Qwen3.5-27B” vs “Qwen3.5-VL-27B” naming inconsistency; “outperformsthe” spacing; author line “Y urui Dong”.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is tightly coupled to a cluster of concurrent, still-unreleased sibling arXiv TODOs (IterCAD, AssemCAD, ComAct). That is a legitimate systems programme, but for a journal it raises novelty-boundary and independent-eval concerns: much of the claimed advance may be “same group’s data + SFT + same group’s benches.” I would ask the editor to require either external benches or a clear scope statement that this is an internal substrate release with limited external validity until those integrations and independent tests exist. Fit is reasonable for an applied AI / systems venue; weaker for a venue that expects new learning methodology."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: multi-task SFT of Qwen3.5-VL-27B on ~52k CAD-VQA + CadQuery + assembly + SolidWorks/Inventor COM samples produces large, directionally consistent gains on author CAD benches (CadQuery 7.82→77.88, COM 28.33→73.33, assembly 0→15.38) while general scores hold or tick up (+1.56 pp). That is the result.\n\nWhat is actually new is the integrated industrial scope, not the recipe. Domain multi-task SFT on a strong VL backbone is standard. Putting scarce Windows COM (sandbox-executed in ComForge), parametric CadQuery, assembly, and drawing QA into one open 27B substrate—and positioning it explicitly under IterCAD/AssemCAD/ComAct/SimLoop rather than as another single-task generator—is the contribution. They did the unglamorous work: execution filters, Chamfer gates, UUID/AST decontamination, sequence-parallel long COM scripts, per-subset validation. The CadQuery and COM absolute jumps are the evidence that matters; CAD-VQA barely moves because the base was already strong. Case studies are concrete (correct COM helpers, shaft centering, math_vision spillover).\n\nSoft spots, in proportion. COM eval is n=60; assembly pass@CD≤1e-3 is still only 15%. Data and benches are co-sourced with the sibling agent lines the model is meant to serve. UUID/AST/Chamfer decontam blocks exact and trivial leakage but not shared templates, COM idioms, or geometry families—so the “wins 4/4 vs GPT-5.4” and “common substrate” claims are still partly in-family until independent plant workflows appear. Several citations and the CAD-VQA pipeline are TODO; release is promised, not linked; no agent has integrated the checkpoint yet (future tense throughout). None of that contradicts the empirical claim as scoped; it just caps how far the headlines transfer.\n\nMath is metric definition, not derivation—fine for this genre. Citation pattern is honest about DeepCAD/Text2CAD/Qwen-VL and the sibling stacks.\n\nThis is for people building industrial CAD agents or COM automation who need a starting checkpoint, not for theory readers. I would bring it to reading group if the group cares about applied multimodal systems. It deserves peer review: important enough applied artifact, clear enough protocol, large enough domain deltas. Engage if the weights and data cards land; treat the substrate claim as provisional until external evals exist.","headline":"Solid applied SFT paper: real CadQuery/COM jumps on a scarce industrial stack, with the usual sibling-eval and small-n caveats—not a methods breakthrough, but worth engaging if you build CAD agents.","tokens_in":12773,"tokens_out":634,"would_cite":true,"duration_ms":13454,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A single 27B multimodal model trained on six industrial CAD corpora becomes a shared substrate that beats both its base and a frontier closed model on CAD vision, CadQuery, COM APIs, and assemblies—without forgetting general skills.","keywords":["industrial CAD","multimodal foundation model","multi-task SFT","CadQuery","COM API","assembly generation","CAD visual QA","industrial agents"],"falsifier":"Re-evaluate IndustryForge-27B and the same baselines on a held-out industrial CAD suite built from unrelated plant drawings, COM workflows, and multi-part assemblies never seen in the six sub-corpora or sibling agent pipelines; if the mean domain lead over the base and over GPT-5.4 collapses or assembly pass rate stays near zero under that protocol, the substrate claim fails.","tokens_in":12548,"feed_emoji":"⚙️","tokens_out":1116,"duration_ms":20156,"temperature":0.7,"pith_summary":"Industrial CAD automation needs models that can read engineering drawings and 3D screenshots, emit correct parametric scripts and Windows COM API code, and handle both single parts and assemblies. General multimodal models fail on those axes, and single-task fine-tunes are too narrow for the mixed calls real agents make. This paper builds IndustryForge-27B by multi-task supervised fine-tuning of Qwen3.5-VL-27B on about 52k curated multimodal samples spanning CAD visual QA, CadQuery parts and assemblies, and COM code for Inventor and SolidWorks. On four CAD-domain benchmarks the model gains roughly 34 percentage points over its base and outperforms GPT-5.4 on every task, including the first usable non-zero assembly pass rate among the compared models, while slightly improving average scores on eleven general benchmarks. The authors present the result not as an end-to-end CAD system but as a common foundation so downstream agents need not re-solve basic CAD reading and coding from scratch.","feed_headline":"27B CAD model beats base and GPT-5.4 on four industrial tasks","feed_subtitle":"Six curated corpora teach drawing reading, CadQuery, COM APIs, and assemblies without general-skill loss","key_machinery":"Unified multi-task SFT on six industrial-CAD sub-corpora (CAD-VQA, text2cadquery, text2cadquery-assembly, com_2d, com_3d, com_assembly), mixed flat and trained with LoRA, DeepSpeed ZeRO-3, sequence parallel, and padding-free packing, with sandbox execution, Chamfer-Distance filtering, and UUID/AST decontamination as quality gates.","core_discovery":"Multi-task supervised fine-tuning of a 27B vision-language model on six integrated industrial-CAD sub-corpora (~52k samples) produces a single foundation that substantially raises CAD-domain performance—mean +33.65 pp over the base, winning all four domain benchmarks against GPT-5.4—while retaining and slightly improving general capability (+1.56 pp mean on eleven public benchmarks, no catastrophic forgetting), thereby serving as a usable common substrate for full-stack industrial CAD agents.","pith_inferences":["If the substrate pattern holds, other heavy Windows professional apps with scarce public COM/API data (beyond CAD) may benefit from the same curated sandbox-executed multi-task recipe.","The still-dominant single-part mix and ~15% assembly pass rate imply that scale and verification of assembly data—not just more SFT steps—will be the next bottleneck for production-ready multi-part systems.","Positive transfer on math_vision and chart-like tasks suggests industrial drawing SFT is a practical way to strengthen multi-view spatial reasoning in general multimodal models.","Open release of the substrate could shrink duplicated CAD-foundation work across teams and make fair head-to-head agent comparisons easier."],"forward_implications":["Downstream CAD agents can start from one shared checkpoint for drawing understanding, CadQuery, COM operation, and assembly instead of each re-solving basic CAD literacy.","COM-as-code operation of SolidWorks/Inventor becomes a first-class foundation skill rather than a rare, near-zero capability of general models.","Assembly-level generation moves from near-total failure (0–4%) to a non-zero usable baseline (~15%) that closed-loop agents can iterate on.","CAD-oriented multi-task SFT can spill over positively into geometric visual reasoning and structured figure-text tasks on general benchmarks without wiping out broad knowledge.","A planned feedback loop can recycle high-quality agent traces back into the next substrate round, linking single-shot SFT to closed-loop self-improvement."],"fun_headline_variants":["IndustryForge-27B lifts CAD scores 33.65 pp, tops GPT-5.4 on four tasks","27B multimodal model wins four CAD benchmarks over GPT-5.4","Multi-task SFT on 52k CAD samples beats base and GPT-5.4","One 27B model covers drawings, CadQuery, COM APIs, and assemblies","IndustryForge-27B gains 33.65 pp on CAD without general-skill loss"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The four CAD benchmarks—especially the small COM set and the still-low assembly pass rate—plus decontamination against related pipelines are assumed independent and representative enough that the large gains prove a generally superior industrial CAD foundation rather than in-distribution wins on author-adjacent tests.","fun_headline_variants_meta":{"raw":{"variants":["IndustryForge-27B lifts CAD scores 33.65 pp, tops GPT-5.4 on four tasks","27B multimodal model wins four CAD benchmarks over GPT-5.4","Multi-task SFT on 52k CAD samples beats base and GPT-5.4","One 27B model covers drawings, CadQuery, COM APIs, and assemblies","IndustryForge-27B gains 33.65 pp on CAD without general-skill loss"]},"model":"grok-4.5","effort":"low","cost_usd":0.004721,"raw_usage":{"total_tokens":1454,"prompt_tokens":892,"num_sources_used":0,"completion_tokens":100,"cost_in_usd_ticks":47208000,"prompt_tokens_details":{"text_tokens":892,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":462,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":892,"tokens_out":100,"duration_ms":7812,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T19:24:08.641295+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-evaluate IndustryForge-27B and the same baselines on a held-out industrial CAD suite built from unrelated plant drawings, COM workflows, and multi-part assemblies never seen in the six sub-corpora or sibling agent pipelines; if the mean domain lead over the base and over GPT-5.4 collapses or assembly pass rate stays near zero under that protocol, the substrate claim fails.","supporting_citations":[],"review_version":1}