{"id":"3167e5ec-197b-4c5e-9861-d923ed65d860","arxiv_id":"2608.03062","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TraceCAD improves agentic CAD repair by keeping persistent trace state, searching bounded local edits, and reusing repair skills, cutting retries and improving geometry.","lead":"TraceCAD wraps an AI CAD generator in a repair layer that remembers which features were requested, which modeling step failed, and what repairs worked before. It then searches small nearby code edits instead of regenerating the whole model, cutting retries and token cost while matching or beating prior CAD agents on geometric quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal attribution to persistent/localized/reusable recovery rests on single-run ablations with no variance estimates, in a stochastic, order-sensitive pipeline; repeated-seed and order-permutation checks are required before Table 2 can ground the central claim.","rationale":"I considered three candidate concerns. (1) Fault localizability: Eq. (2)'s one-to-two-hop upstream region could miss global faults. However, the paper acknowledges this in Limitations, and the included geared-motor case shows a genuinely local diagnosis and a 19-line local patch; localizability is a scope boundary, not a demonstrated failure of the central claim on the chosen benchmark. (2) Baseline geometric advantage: TraceCAD's mean IoU exceeds CADDesigner but the paired test is not significant (p=0.7884). The paper only claims 'competitive' geometry, however, so this does not sink the repair-reliability contribution. (3) Single-run stochastic ablations plus order-dependent skill accumulation: this is the load-bearing issue. The paper gives no way to distinguish each component's effect from one stochastic trajectory. The large differences could be real, but publication-quality causal attribution requires repeated runs. Because the Reader already rendered a CONDITIONAL verdict largely for reproducibility and statistical robustness, my concern does not move the verdict; it reinforces it. No code or data release strengthens the need for the repeat-seed experiment. If the repeated-seed test confirms Table 2, the paper's central claim should be accepted; if not, the appropriate verdict would remain conditional until such evidence is provided.","tokens_in":16809,"tokens_out":6421,"duration_ms":65148,"concrete_test":"Run every Table 2 configuration (full cold-start, w/o persistent state, w/o localized search, w/o visual feedback, w/o skill store, warm-up) on the same 200 cases under at least 5 independent LLM sampling seeds and at least 5 shuffled task orders (stratified per command-count interval), reusing the fixed seed 20260705 for dataset construction but varying the evaluation order and sampling seed. Report paired 95% confidence intervals and Wilcoxon effect sizes for Rec., Reg., AVG Re, Scope, and Reuse for each ablation contrast. If full-vs-ablation differences exclude zero in most orders/seeds (e.g., the Rec. gap remains >0.2), the causal claim is supported; if confidence intervals overlap zero or signs flip across orders, the paper must downgrade the attribution to an order- and seed-dependent observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'persistent, localized, and reusable recovery improves final CAD quality and repair reliability'—is supported almost entirely by Table 2. Table 2 differences are large (Rec. 0.9167 vs 0.4872/0.4865/0.4583; Reg. 0.3456 vs 0.7296), but the paper reports no variance, confidence interval, or significance test for any of these ablation contrasts. The methods text (Supplementary B.2) states: 'Each configuration is executed once per selected case' and 'hosted LLM endpoints do not provide a consistently enforceable sampling seed, so generation remains stochastic.' With stochastic LLM calls at temperature 0.0–0.2, a single run per configuration cannot separate the effect of the component from sampling noise or from the particular 200-case draw/order. This is compounded by the cold-start skill store: skills are induced from earlier evaluation cases and reused on later ones, so every cold-start result depends on task order. The paper's own Limitations says 'online cold-start accumulation makes reuse sensitive to task order and parallel scheduling.' Thus the headline ablations are one trajectory of a stochastic, order-dependent process. The 1K geometric comparison does not rescue this: against CADDesigner, the paired IoU difference is not significant (p=0.7884, Table 7), so the credibility of TraceCAD's contribution rests on the unverified ablation contrasts. Localizability (the Reader's weakest assumption) is a real boundary condition, but it is acknowledged and qualitatively supported by the case study; the missing repeat-seed/order analysis is more load-bearing because it undermines causal attribution for every ablation component, not just the local-search edge case.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TraceCAD proposes a persistent recovery layer for LLM-based CAD agents. The system maintains feature and step state across the correction loop, diagnoses a likely faulty modeling step from execution/visual evidence, generates bounded local patches in a dependency region, and accumulates reusable repair skills with reuse statistics. The paper evaluates TraceCAD on DeepCAD-derived 200-model and 1K-model subsets, reporting geometric fidelity (IoU, Chamfer distance, Hausdorff distance), repair behavior (Recovery Score, Regression, Scope, Reuse), and efficiency (retry count, tokens, latency). The central claim is that persistent state, localized search, and skill memory improve final CAD quality and repair reliability, supported mainly by the ablation contrasts in Table 2.","tokens_in":17056,"tokens_out":6827,"duration_ms":65831,"significance":"The paper addresses a genuine problem in agentic CAD generation: correction loops lose evidence about satisfied requirements, faulty operations, and prior repairs, leading to regressions and inefficient retries. The proposed recovery layer — persistent feature/step state, bounded localized repair search, and reuse-aware skill memory — is a well-motivated and potentially reusable design. The evaluation goes beyond final geometry by introducing repair-specific metrics (Recovery Score, Regression, Scope, Reuse), and the supplement provides detailed metric definitions, backend sensitivity, cost comparisons, and paired Wilcoxon tests for baseline comparisons. If the ablation results are robust, the system would be a useful contribution to CAD agents and to agentic program repair more broadly. However, the central causal attribution currently rests on single-run ablations with no uncertainty quantification, and the paper itself acknowledges order dependence from online skill accumulation. These issues must be addressed before the main claim is fully supported.","major_comments":[{"comment":"The central claim rests on single-run ablation contrasts with no uncertainty quantification. Supp. B.2 states that each configuration is executed once per selected case and that hosted LLM endpoints do not provide a consistently enforceable sampling seed, so generation remains stochastic. Since the cold-start skill store accumulates from earlier evaluation cases and can be reused later (as the Limitations section explicitly notes), the reported differences (e.g., Recovery 0.9167 vs 0.4872; Regression 0.3456 vs 0.7296) are one trajectory of an order-dependent stochastic process. The paper should report repeated runs (e.g., multiple seeds and order permutations) with confidence intervals, or at least per-case paired tests for the ablations analogous to the Wilcoxon tests in Supp. B.5, before attributing the differences to the ablated components.","section":"§Experiments (Ablations), Table 2, and Supp. B.2"},{"comment":"The Recovery Score is averaged only over cases whose initial generation fails, and the Regression metric only over cases with distinct pre- and post-repair artifacts, but the paper does not report the cardinalities |F| and |V_Reg|. If the initial failure rate is low, the stated Recovery differences may be based on a very small number of cases, making them fragile. The paper should report these denominators and, ideally, the per-case distribution of recovery outcomes for every variant in Table 2.","section":"§A.2 (Eq. 7) and Table 2"},{"comment":"The text says TraceCAD 'achieves the strongest overall geometric fidelity' and frames the comparison with CADDesigner favorably, but the paired statistics in Table 7 show the IoU difference from CADDesigner is not significant (Holm-adjusted p=0.7884), and the HD difference is also not significant (p=0.7884). Only the Text2CAD differences are significant after correction. The paper should either weaken the wording to 'comparable' or provide additional evidence before claiming superiority over the strongest baseline.","section":"§Comparison with Existing Methods, Table 4, and Supp. B.5"},{"comment":"The locality hyperparameters — hop budget h (starting at 1, expanding to 2), at most three candidates over at most two target steps, visual-score threshold of five points, and skill retrieval top-5 — are fixed with no sensitivity analysis. Because the central mechanism is bounded local search, the reader cannot tell whether the benefit is robust to the repair-region size or is tuned to the reported setting. The paper should include an ablation over h and beta (e.g., h=0, 1, 2, 3, or unrestricted), and report how often the bounded search exhausts its region or fails to find a local candidate. This would also partly address the acknowledged localizability limitation.","section":"§Method (Localized Repair Search, Eq. 2) and §Setup"},{"comment":"The visual-feedback ablation removes rendered semantic evidence from diagnosis and repair validation, but it also changes the termination and promotion rule: without visual evidence, the run stops as soon as execution and export succeed, which lowers AVG Re (1.4), tokens, and latency. The comparison therefore conflates the absence of visual evidence with a different retry budget. To isolate the contribution of visual feedback, the paper should hold the retry budget fixed (or report results as a function of budget) so that the lower cost reflects the evidence source rather than an earlier stopping policy.","section":"§Ablations, Table 2 (w/o visual feedback)"}],"minor_comments":[{"comment":"A dash is said to indicate absent evidence, but for the 'w/o localized repair search' row the absence of Scope and Reuse should be explained more explicitly, since full-program repairs still produce promoted candidates in this variant.","section":"Table 2 caption"},{"comment":"a_i in Eq. (7) is defined as the number of subsequent code-agent invocations, whereas AVG Re in Eq. (13) counts the initial generation as one invocation; the paper should state this distinction clearly in one place to avoid misinterpretation of the two metrics.","section":"§A.2 (Eq. 7) and §A.3 (Eq. 13)"},{"comment":"The phrase 'retry budget' is used as an umbrella term, but the individual limits in Table 5 (30 main-agent tool steps, 20 internal steps, 180-second timeout, five consecutive failures) are numerous; a brief sentence connecting these to Eq. (1)'s beta would help the reader understand the search budget.","section":"§Setup and Table 5"},{"comment":"The claim that 'many failures are local even when their visual effects are global' is plausible but is not referenced to prior CAD or program-repair literature; a citation or a small empirical count from the failures observed in this study would strengthen it.","section":"Introduction"},{"comment":"Figure 2 draws skill retrieval before diagnosis, which is consistent with Algorithm 1 (line 7 retrieves skills, line 8 diagnoses), but the prose in §Method describes diagnosis before skill retrieval; align the presentation order in the text.","section":"Figure 2 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the recovery-layer idea is worth publishing if the statistical grounding is fixed. The main barrier is that the central claim rests on a single-run, order-dependent ablation. I would support acceptance after the authors provide repeated-run ablation results with confidence intervals or permutation-based tests, report the denominators for Repair-specific metrics, and moderate the comparison with CADDesigner to 'comparable' given the non-significant paired tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TraceCAD is a real contribution to the agentic CAD repair subfield. The StepTrace abstraction, persistent feature-state linking, bounded dependency-localized repair search, and skill memory with reuse statistics are new relative to the cited systems, and the paper gives them a clear, formal treatment. The evaluation is unusually careful in some ways: the metric definitions in the supplement are precise, the arithmetic in Table 2 checks out, and the paired Wilcoxon tests against baselines are honest (the IoU difference vs CADDesigner is not significant, and they say so). The case study is detailed enough to show the mechanism working.\n\nThe soft spot is exactly what the stress-test note says. The headline ablation contrasts in Table 2—persistent state nearly halving recovery score, localized search doubling regression—come from a single run of each configuration on a 200-case subset, with non-seedable hosted LLMs at temperature 0–0.2 and a cold-start skill store whose content depends on evaluation order. One trajectory cannot support causal attribution for the components. The paper acknowledges order sensitivity in the limitations, but the abstract and conclusion assert the causal claim without that caveat. That is a genuine gap, not a manufactured one. The localizability assumption is also load-bearing, but the authors are upfront about it, and the case study provides plausible qualitative support. Missing code and data make independent verification harder, though the metric definitions are sufficiently detailed to reimplement.\n\nI don't think this is a desk-reject. The framework is coherent, the problem is well-motivated, and the paper is a meaningful extension of automated program repair to CAD. But the central empirical claim needs repeated-seed and order-permutation experiments, plus variance or confidence intervals on the ablation contrasts, before it can be taken as established. I'd send it to a serious referee with the expectation of major revision.","headline":"Solid systems contribution to CAD repair whose central causal claim needs repeated-seed and order-permutation evidence before it lands.","tokens_in":17708,"tokens_out":1734,"would_cite":true,"duration_ms":19568,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Persistent, localized, reusable repair—not another regeneration—drives reliable CAD recovery.","keywords":["CAD generation","program repair","LLM agents","persistent state","localized search","skill memory","parametric CAD","agentic recovery"],"falsifier":"Run TraceCAD on a suite of CAD models whose reference mismatch is deliberately spread across many steps—for instance, a boolean union whose evaluation order corrupts the entire solid, or a program whose step scopes have been scrambled so the recorded intents no longer match the code regions. If bounded two-hop search recovers almost none of these while full-program regeneration does, the localizability assumption fails and the recovery-score gains vanish.","tokens_in":16544,"feed_emoji":"🛠️","tokens_out":6897,"duration_ms":63715,"temperature":0.7,"pith_summary":"TraceCAD wraps an executable CAD generation agent with a persistent recovery layer: it records which requested features are satisfied, which modeling step failed, and which repairs were tried, then uses that record to diagnose one or two faulty steps, search bounded edits in their dependency neighborhood, and validate candidates by executing them and checking that unrelated geometry is preserved. The paper claims that this persistent, localized, reusable recovery is what makes repair reliable: removing persistent state nearly halves the recovery score (from 0.9167 to 0.4872), and removing localized search more than doubles geometric regression (from 0.3456 to 0.7296). The point of caring is that CAD models are parametric programs, not just shapes: a repair that silently rewrites valid construction history is almost as bad as the original failure, and TraceCAD's design is explicitly aimed at avoiding that.","feed_headline":"Persistent local repair lifts CAD recovery from 0.49 to 0.92","feed_subtitle":"Keeping feature and step evidence, bounding edits, and reusing skills cuts geometric regression and retries.","key_machinery":"The load-bearing mechanism is the persistent recovery record, which has two linked anchors: feature state F, tracking the lifecycle of each requested feature such as a body, hole pattern, edge treatment, or export, and StepTrace T, an ordered trace of instrumented code scopes each annotated with intent, execution status, and exception evidence. Repair runs on top of this record: a dependency graph built from variables defined and used by each step, a diagnosis selecting a likely faulty step, the bounded edit region R_h(s) = {s} ∪ Upstream(s, h) with hop budget h starting at one and expanding at most once to two, and a candidate selection that favors fewer changed step scopes and requires execution plus preservation evidence. Successful or failed trajectories induce reusable repair skills k = (σ, d, π, τ, ε, u)—retrieval key, transferable diagnosis, policy, patch template, evidence, and reuse statistics—so later failures are guided by prior outcomes.","core_discovery":"On its own terms, TraceCAD's central claim is that a CAD agent's correction loop should be backed by an explicit, persistent record—linking requested features, modeling steps, failure evidence, and candidate outcomes—so that recovery becomes a localized, evidence-backed edit rather than another unconstrained generation attempt. Concretely, the paper defines feature state F and a StepTrace T over instrumented code scopes, diagnoses a likely faulty step, builds a repair region R_h(s) = {s} ∪ Upstream(s, h) starting at one hop and expanding at most once to two, and promotes a patch only when execution, artifact export, semantic and visual agreement, preservation of unrelated geometry, and edit locality all pass. The ablation results are the direct support: the full cold-start system reaches a Recovery Score of 0.9167 with Geometric Regression 0.3456 and a Repair Scope Ratio of 0.3899, whereas removing persistent state drops recovery to 0.4872 and removing localized search raises regression to 0.7296 and average retries to 3.0. The paper concludes that persistent, localized, and reusable recovery improves both final CAD geometry and repair reliability, and that this holds across LLM backends while reducing token cost and latency when the skill store is warm-started on disjoint training models.","pith_inferences":["The paper's localizability premise could be stress-tested by engineering failures that are truly global—for example, a boolean operation whose effect corrupts the entire solid—and measuring whether bounded two-hop search still recovers them; the paper itself notes that poorly decomposed code makes step attribution ambiguous.","The same persistent-state recipe—feature obligations, step traces, bounded dependency search, and skill memory—could apply to any agent that produces structured executable artifacts such as scripts, assembly plans, or shader programs, whenever a repair must preserve already-correct parts.","Recovery Score, defined as success weighted by the inverse number of repair invocations, is a natural general metric for agentic repair; adopting it across code-generation benchmarks would make repair reliability comparable across systems."],"forward_implications":["If TraceCAD's central claim is right, then recovery quality in agentic CAD should be measured not by final shape alone but by whether the intended defect was fixed without rewriting valid construction history; Recovery Score and Geometric Regression capture that distinction.","Keeping feature and step evidence across attempts is what enables reliable recovery: the state ablation shows Recovery Score falling from 0.9167 to 0.4872 when that state is removed.","Bounding repairs to a diagnosed step's dependency neighborhood prevents broad, damaging rewrites: without localized search, Geometric Regression rises from 0.3456 to 0.7296 and average code-agent invocations double to 3.0.","Reusing successful and failed repair outcomes in a skill store cuts retries, token cost, and latency: warm-starting on 1K disjoint training models lowers average retries from 1.5 to 1.2 and tokens from 103.6K to 70.3K.","The recovery layer is generator-agnostic: it wraps initial generation without changing decoding, so gains from persistent state, localized search, and skill memory should transfer to other executable CAD generators."],"supporting_citations":[{"why":"Supplies the DeepCAD dataset and construction-history format from which the benchmark subsets are built.","marker":"(Wu, Xiao, and Zheng 2021)"},{"why":"Provides the deduplication and preprocessing pipeline and the train–test split convention used to construct the 200-model and 1K-model evaluations.","marker":"(Xu et al. 2022)"},{"why":"Defines the CAD modeling environment and benchmark protocol that TraceCAD adopts, and is the strongest agentic baseline.","marker":"(Fan et al. 2026)"},{"why":"Text2CAD is the learning-based baseline whose sequential CAD program generation TraceCAD compares against.","marker":"(Khan et al. 2024)"},{"why":"CADCodeVerify is the agentic baseline that iteratively validates and revises CAD code using visual-language feedback.","marker":"(Alrashedy et al. 2025)"},{"why":"Supplies the program-repair framing of localization, patching, and validation that TraceCAD adapts to CAD geometry.","marker":"(Le Goues et al. 2012)"},{"why":"Self-Refine motivates iterative self-feedback in LLM agents, which TraceCAD extends with persistent state.","marker":"(Madaan et al. 2023)"},{"why":"ReAct grounds the reasoning-action trace and tool-observation loop that TraceCAD's agent runtime builds on.","marker":"(Yao et al. 2023)"}],"fun_headline_variants":["TraceCAD's trace state doubles CAD recovery to 0.92","Localized repair cuts geometric regression in half","Reusable repair skills cut retries and lift CAD quality","TraceCAD: recovery 0.92, regression 0.35 after trace repair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that residual CAD failures are local: the faulty operation can be traced to one or two modeling steps within a small dependency neighborhood, and patching that neighborhood fixes the model without disturbing already-correct geometry.","fun_headline_variants_meta":{"raw":{"variants":["TraceCAD's trace state doubles CAD recovery to 0.92","Localized repair cuts geometric regression in half","Reusable repair skills cut retries and lift CAD quality","TraceCAD: recovery 0.92, regression 0.35 after trace repair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000904,"raw_usage":{"total_tokens":3905,"prompt_tokens":976,"completion_tokens":2929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2857}},"tokens_in":592,"tokens_out":2929,"duration_ms":23691,"temperature":1.0,"reasoning_tokens":2857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:59:11.466123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TraceCAD on a suite of CAD models whose reference mismatch is deliberately spread across many steps—for instance, a boolean union whose evaluation order corrupts the entire solid, or a program whose step scopes have been scrambled so the recorded intents no longer match the code regions. If bounded two-hop search recovers almost none of these while full-program regeneration does, the localizability assumption fails and the recovery-score gains vanish.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Self-Refine motivates iterative self-feedback in LLM agents, which TraceCAD extends with persistent state."}],"review_version":1}