{"id":"357efbdc-cc95-4bc3-af25-e6c671077132","arxiv_id":"2608.09706","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Geometric consensus selection, which returns the most representative compiled CAD model in a sampled pool, outperforms a VLM verifier on geometry metrics and matches it on topology.","lead":"This paper shows that when an AI generates many CAD designs, picking the one most similar to the rest of the group beats using a separate judge model for geometry, at no extra training cost. The result points to a cheaper way to make text-to-CAD systems reliable, by exploiting the candidate pool itself.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shared systematic errors, not independent mistakes, are the load-bearing risk: consensus selects the majority error cluster, and the paper only shows aggregate wins without per-prompt agreement-error correlation.","rationale":"The reader's verdict is CONDITIONAL with MODERATE confidence and identifies the consensus premise as the weakest assumption; I agree. The single most load-bearing issue for the central claim is that the method's mechanism, agreement-as-correctness, is only supported by aggregate results. The paper honestly reports limitations, which is credit, but those limitations are not mere caveats: they describe a failure regime that is plausible for LLM-generated pools. The concrete check is feasible with released data and ground truth: a per-prompt correlation directly tests whether the selection score tracks correctness, and the synthetic ablation isolates the conceded shared-error failure. If the per-prompt correlation is high and the synthetic ablation does not select the injected error, the concern is resolved and the verdict can stand upon data release. If not, the claim that consensus is a reliable verifier-free alternative would need substantial qualification. I do not find other issues that would change the verdict: the same-pool comparison is fair, the MBR framing is appropriate, and the scaling experiment is informative. The watertight restriction is transparent and motivated by topology fairness, and the cross-model table's lack of error bars is a reproducibility concern rather than a threat to the Table 1 central claim.","tokens_in":793,"tokens_out":765,"duration_ms":114030,"concrete_test":"Using the Table 1 pools (n=197), compute for each prompt the Spearman correlation across candidates between the consensus score (mean pairwise Chamfer distance from Eq. 1) and the candidate's true Chamfer distance to the ground-truth model. Report the mean and the fraction of prompts with positive correlation. Then run a synthetic ablation: take one prompt's pool, duplicate its most common erroneous candidate so that it forms 60% of the pool, and verify whether consensus selects that error and the per-prompt correlation drops. If the mean per-prompt correlation is weak or the synthetic ablation selects the injected error, the 'mistakes are random' premise fails exactly where the method is claimed to work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is stated in Section 3: 'correct features appear in many candidates, while individual mistakes are random.' The paper's own Section 5 concedes that if the same error appears in most candidates, that error becomes part of the consensus. This is not a remote possibility: all candidates in a pool come from the same LLM with the same few-shot examples and prompt, so errors are strongly correlated across samples. A model with a systematic bias (e.g., omitting a class of features, wrong parameter ranges, or a common misinterpretation of the prompt) will produce a majority cluster that is confidently wrong, and Equation (1) will select inside it. The reported aggregate wins do not rule this out: a medoid can beat the verifier on average while being worse than the pool median on many individual prompts, if its wins are concentrated. The paper provides no per-prompt analysis of whether agreement scores correlate with true error; without that, the central claim that consensus provides a reliable verifier-free signal is not established outside the specific aggregate numbers. The concern is not that the Table 1 numbers are wrong, but that the mechanism claimed to explain them is unvalidated and is exactly the place the method can fail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a verifier-free selection method for text-to-CAD generation. Instead of using a vision-language verifier to choose among sampled CAD programs, the method compiles all candidates to 3D models and returns the candidate with greatest agreement with the rest of the pool: a geometric medoid under Chamfer distance, or a majority vote over Euler-characteristic classes. The method is framed as minimum Bayes risk decoding, is training-free, and is evaluated on the CADPrompt benchmark. On EvoCAD's initial candidate pools, geometric consensus improves all three geometric metrics over the verifier (with reported p-values), while topological consensus matches the verifier on topology. Across four LLMs and two prompt variants, geometric consensus is reported to improve over random selection by 1--10% in Chamfer distance, with an oracle still clearly better. The paper also studies scaling with pool size and discusses limitations, including the risk of selecting systematically shared errors.","tokens_in":10672,"tokens_out":6481,"duration_ms":54804,"significance":"If the empirical claims hold, this is a useful, low-cost alternative to VLM-based verifiers for CAD candidate selection, relevant to test-time scaling in code generation and CAD. The controlled same-pool comparison in Table 1 is a genuine strength: the candidates are identical and only the selection rule varies, making the geometric improvements credible. The method has no fitted parameters, is grounded in established MBR decoding ideas, and is clearly presented as an extension of execution-based consensus to compiled 3D models. The main risk is that the central mechanism — that correct features are common and mistakes are random — is only validated in aggregate, not per prompt, and the paper's own Section 5 identifies the systematic-error failure mode. If the method is to be positioned as a general replacement for verifiers, that mechanism needs direct support.","major_comments":[{"comment":"The claim that geometric consensus improves over random selection for every tested LLM and prompt variant rests on point estimates without confidence intervals or significance tests. The smallest improvement is 0.7% (Gemini 3 Flash with measurements: CD 0.0546 vs 0.0550), which is plausibly within sampling noise given n=125 prompts and 1000 random subsets. Please report paired significance tests (e.g., bootstrap or Wilcoxon) and confidence intervals for each row, or qualify the claim to reflect that the gains are not all statistically significant.","section":"Section 4.4, Table 2"},{"comment":"The method's justification in Section 3 is that 'correct features appear in many candidates, while individual mistakes are random.' Section 5 correctly concedes that if the same error appears in most candidates, that error becomes part of the consensus. Since all candidates come from the same LLM, the same few-shot examples, and the same prompt, correlated errors are a concrete possibility rather than a remote one. The paper reports only aggregate improvements and gives no per-prompt analysis of whether agreement scores correlate with true geometric error. Without such an analysis, the central claim that consensus provides a reliable verifier-free signal is not established beyond the specific aggregate numbers. I recommend adding a per-prompt correlation analysis between the consensus score and the candidate's error, or a breakdown of failure cases, to test the stated principle.","section":"Section 3, Section 5"},{"comment":"The evaluation protocol for Table 2 is ambiguous. The text says 'evaluate selection at N=15 with 1000 random subsets per prompt,' but the mean candidate count ('Cand.') is below 15 for several models (e.g., Gemma 3 12B: 13.3). It is unclear how subsets of size 15 are constructed when fewer than 15 candidates compile, whether prompts without enough candidates are excluded (and how many), or whether sampling is with replacement. Please specify the subset construction and report the number of prompts actually used per row.","section":"Section 4.4"},{"comment":"The abstract and introduction state that consensus selection is evaluated 'on the exact candidate pools' of EvoCAD. In fact, the controlled comparison restricts all selection methods to the watertight subset of each initial pool (Section 4.2, Protocol). This is a reasonable choice for comparing topological consensus, but it changes the verifier's selection space relative to its original protocol. The claim should be qualified as applying to watertight subsets, and the verifier's performance on the full pool could be reported as a reference to assess the effect of this restriction.","section":"Section 4.2, Abstract"}],"minor_comments":[{"comment":"The voxel IoU metric is described as 'sample 10,000 points from each aligned mesh and convert them into a solid voxel grid with voxel size 0.1.' Please clarify how the points are converted to occupancy (e.g., whether each voxel is occupied if it contains at least one point) and whether the voxel grid origin is aligned across models.","section":"Section 4.1"},{"comment":"The significance tests are reported only as p-values without specifying the statistical test used (e.g., paired bootstrap, permutation test, Wilcoxon). Please state the test and whether any multiple-comparison correction was applied when testing several metrics.","section":"Section 4.2"},{"comment":"The shaded bands are described as ±1 standard deviation over random subset draws, which is informative. The right panel's sample size is given only as '180 with watertight ground truth'; please also state the number of prompts for the left panel explicitly in the caption (the text says 182 for CD, but the caption should match).","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CE and the central idea is attractive. The major concerns are the lack of significance testing in Table 2, the ambiguity in the N=15 evaluation protocol, and the absence of per-prompt validation of the consensus principle, which is explicitly concedes a failure mode in Section 5. These are addressable with additional analyses rather than requiring a different approach. The watertight-subset restriction also deserves clearer framing relative to the 'exact candidate pools' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honestly reported empirical paper. The core result—geometric consensus (Chamfer medoid) selecting from the same candidate pools as EvoCAD's VLM verifier beats that verifier on all three geometry metrics, with p≤0.022, and topological consensus matches it on topology—is credible from the numbers. That is a genuinely useful data point for test-time scaling in CAD: you can replace a costly judge with a free deterministic rule, at least on these pools.\n\nThe paper also does the right things around framing. It explicitly connects the rule to minimum Bayes risk decoding and cites the execution-consensus literature (MBR-EXEC, self-consistency). The same-pool comparison is well controlled, and Figure 2's scaling analysis with an oracle bound is informative. The method is training-free and has no fitted parameters, which lowers the circularity risk. I agree with the reader that the Chamfer-distance-as-both-selection-and-evaluation alignment is objective alignment, not circularity.\n\nSoft spots, in order of importance. First, the mechanism: the consensus principle that correct features are shared while mistakes are random is load-bearing. The paper concedes in Section 5 that if the same error appears in most candidates, that error becomes the consensus. That is not a remote failure mode here, because all candidates come from the same LLM, same prompt, same few-shot pool—errors are likely correlated. The paper shows aggregate wins but no per-prompt correlation between agreement and true error. So the claimed 'verifier-free signal' is only demonstrated on these specific pools, not as a general property. This is a limitation, not a fatal flaw, but the authors should either add the per-prompt analysis or soften the framing.\n\nSecond, Table 2 reports improvements over random selection across models but has no confidence intervals or significance tests, and the smallest gain is 0.7%. The abstract's '1–10%' range is accurate but overstates the consistency across models. Third, reproducibility depends on several parameters not fully specified in the text: point cloud sample count, ICP alignment settings, voxel size for IoU, and how the N=15 subsets were drawn. The paper mentions 1000 random subsets, but the exact sampling and the common subset of 125 prompts need clarity.\n\nWho benefits: people working on text-to-CAD pipelines, test-time scaling, and anyone comparing verifier-based selection against cheap baselines. It deserves a serious referee. I'd send it to review, conditional on the authors shipping code and data and adding statistical detail to Table 2 and a per-prompt agreement-error analysis.","headline":"Agreement-based selection beats a VLM verifier on the same pools, but the mechanism's failure mode under systematic error is acknowledged, not tested.","tokens_in":11277,"tokens_out":2474,"would_cite":true,"duration_ms":21326,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Geometric consensus among generated CAD models selects better shapes than a vision-language verifier from the same candidate pool.","keywords":["test-time scaling","text-to-CAD generation","consensus selection","minimum Bayes risk decoding","parametric CAD","Chamfer distance","Euler characteristic","verifier-free selection"],"falsifier":"A controlled experiment that injects a shared systematic error into most candidates—for example, all but one generated model omit a slot that the prompt requires—would settle the claim: if geometric consensus then picks the erroneous majority and its Chamfer distance to the ground truth is worse than random selection or the oracle on those prompts, the central assumption fails; if it still tracks correctness, the claim survives. Concretely, compute per-prompt the rank correlation between a candidate's mean distance to the pool and its distance to the ground-truth model; prompts with strong negative correlation would be direct counterexamples.","tokens_in":10237,"feed_emoji":"📐","tokens_out":9064,"duration_ms":69505,"temperature":0.7,"pith_summary":"The paper asks whether the candidate pool itself is enough to select a good parametric CAD model when no ground truth and no verifier is available. It proposes consensus selection: sample N CAD programs, compile them to 3D models, and return the candidate with the lowest average distance to all other candidates. On the exact candidate pools of a state-of-the-art verifier-based CAD agent, geometric consensus improves all three geometric metrics over that agent's verifier, while a topological consensus variant matches the verifier on topology. Across every tested LLM and prompt variant, geometric consensus also beats random selection from the same pool. This makes agreement a cheaper, training-free alternative to vision-language judges for choosing among CAD candidates.","feed_headline":"CAD consensus beats a vision-language judge on same candidates","feed_subtitle":"Sampling N CAD programs and picking the most central one improves geometry over verifier-based selection on CADPrompt.","key_machinery":"The load-bearing object is the consensus selection rule, Equation (1), which turns a pool of compiled CAD models into a single pick without ground truth or a verifier. The rule is instantiated with two distances: the symmetric Chamfer distance $d_{CD}$ between uniformly sampled, ICP-aligned surface point clouds, which yields the geometric medoid of the pool; and a 0-1 distance $d_\\chi$ on the Euler characteristic of watertight meshes, which yields a majority vote over topology classes, with geometric consensus breaking ties. The choice of distance determines which evaluation metric improves, and the rule is training-free, requiring only a compiler plus $O(|V|^2)$ pairwise distance evaluations.","core_discovery":"The paper's central claim is that for text-to-CAD generation, selection by agreement over compiled 3D models—rather than by a learned or vision-language verifier—is sufficient to improve geometric accuracy. Formally, the rule is\n$$i^* = \\arg\\min_{i\\in V} \\frac{1}{|V|-1}\\sum_{j\\in V\\setminus\\{i\\}} d(S_i,S_j),$$\nwith $d$ either the symmetric Chamfer distance between ICP-aligned, uniformly sampled surface point clouds (geometric consensus) or a 0-1 distance on Euler characteristic classes (topological consensus). On the shared watertight subsets of the EvoCAD initial pools, geometric consensus achieves Chamfer distance 0.0610, Hausdorff distance 0.176, and voxel IoU 0.703, versus 0.0627, 0.181, and 0.695 for the verifier, all significant at $p \\le 0.022$; topological consensus shows no significant difference from the verifier on topology ($p = 0.82$ for Euler-characteristic error, $p = 1.00$ for topology-correct percentage). The paper interprets this as minimum Bayes risk decoding with the empirical distribution over samples, and notes that the oracle over the same pool remains clearly better.","pith_inferences":["A natural next test is to preserve outliers alongside the consensus candidate: returning the medoid plus the most distant candidate would recover high-quality outliers that agreement-based selection misses, a gap the paper attributes to the oracle.","The paper's loss-determines-metric result suggests other CAD-relevant properties—dimensions, manufacturability, or finite-element simulation outputs—could be plugged into the same rule; the authors mention this only as future work.","Whether consensus can form across sequential refinements of one program rather than parallel samples remains untested; if refinements share features, the same agreement rule might apply to iterative generation."],"forward_implications":["CAD verifiers used for candidate selection should be compared against consensus selection on the same candidate pool; on the EvoCAD pools, consensus matches or beats a vision-language verifier without any extra model calls.","Test-time scaling in CAD is not only about drawing more samples: selection quality saturates around $N \\approx 9$, so larger pools mainly add coverage, while the oracle over the same pool stays well ahead.","The loss used for agreement determines which metric improves: geometric consensus with Chamfer distance helps geometry, and topological consensus with Euler characteristic helps topology, so the rule can be targeted to a desired property.","Because alignment normalizes away absolute dimensions, consensus does not currently exploit dimensional information even when prompts specify measurements; a direct corollary of the method's preprocessing."],"supporting_citations":[{"why":"It supplies the verifier-based CAD agent, its initial candidate pools, and the few-shot generation setup used for the controlled comparison, plus the Euler-characteristic topology evaluation.","marker":"[Preintner et al., 2025]"},{"why":"It provides the CADPrompt benchmark with 200 text-to-CAD prompts and ground-truth models, the evaluation dataset for all experiments.","marker":"[Alrashedy et al., 2025]"},{"why":"It defines minimum Bayes risk decoding, the mathematical frame that identifies consensus selection as selecting the candidate with lowest expected loss under the empirical distribution.","marker":"[Kumar and Byrne, 2004]"},{"why":"It justifies approximating the output distribution by the empirical distribution over drawn samples in sampling-based MBR decoding.","marker":"[Eikema and Aziz, 2022]"},{"why":"It establishes execution-based voting (MBR-EXEC) as the direct analog, which the paper adapts by compiling CAD programs to 3D models and voting on their properties.","marker":"[Shi et al., 2022]"},{"why":"It provides self-consistency as a majority-voting consensus precedent for LLM outputs, cited as the text-domain counterpart.","marker":"[Wang et al., 2023]"},{"why":"It establishes the coverage-versus-selection gap for repeated sampling, which the paper uses to interpret why an oracle over the same pool beats consensus selection.","marker":"[Brown et al., 2024]"}],"fun_headline_variants":["Consensus selection beats verifier for CAD generation","Pick the most agreed CAD model, no verifier needed","Training-free consensus outperforms vision-language verifier","Most central CAD model wins: verifier-free selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rule rests on the premise that correct features recur across candidates while individual mistakes are random; if a pool shares systematic errors, consensus will select those errors, a failure the paper concedes in Section 5 and supports only with empirical evidence on CADPrompt, not a distributional argument.","fun_headline_variants_meta":{"raw":{"variants":["Consensus selection beats verifier for CAD generation","Pick the most agreed CAD model, no verifier needed","Training-free consensus outperforms vision-language verifier","Most central CAD model wins: verifier-free selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2769,"prompt_tokens":1030,"completion_tokens":1739,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":1677}},"tokens_in":646,"tokens_out":1739,"duration_ms":13054,"temperature":1.0,"reasoning_tokens":1677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:17:15.622071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that injects a shared systematic error into most candidates—for example, all but one generated model omit a slot that the prompt requires—would settle the claim: if geometric consensus then picks the erroneous majority and its Chamfer distance to the ground truth is worse than random selection or the oracle on those prompts, the central assumption fails; if it still tracks correctness, the claim survives. Concretely, compute per-prompt the rank correlation between a candidate's mean distance to the pool and its distance to the ground-truth model; prompts with strong negative correlation would be direct counterexamples.","supporting_citations":[],"review_version":1}