{"id":"735ee0bd-fad4-4213-affe-9ffdc57f5146","arxiv_id":"2608.03506","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"CALVER is a training-free symbolic verifier that selects the best-of-K LLM causal answer by checking each trace against graph-validity criteria, outperforming plurality when multiple valid answers exist.","lead":"A causal-reasoning verifier, CALVER, scores each sampled LLM trace against Pearl's graphical criteria and picks the highest-scoring candidate instead of voting. It beats plurality, reward models, and LLM judges on answer-multiplicity tasks, with gains widening as more samples are drawn.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline CLEAR gain is scored against the verifier's own predicate, not the benchmark's listed answer; exact-match re-scoring is needed to test whether the claimed advantage transfers.","rationale":"The paper is transparent about its correctness criterion, and the construction-graph/ATE/DoVerifier experiments partially separate the verifier from the grader. However, the central empirical headline is the supplied-graph CLEAR comparison, where scoring and grading use the same predicate. This makes the headline result self-referential with respect to the definition of correctness. The reader's weakest assumption identifies exactly this issue, and the proposed exact-match re-scoring is a concrete, feasible check. The remaining concerns (missing code/data, omitted meaning-based baselines) are secondary and do not change the conditional verdict.","tokens_in":35224,"tokens_out":6046,"duration_ms":72858,"concrete_test":"Re-score all 1,111 CLEAR records in Table 4 (and the 377 SFT records in Table 2) with correctness defined as exact match to CLEAR's listed answer, using the same answer normalization already applied to exact plurality. Compute CALVER, exact plurality, reward model, and judge accuracies under this metric on the identical frozen pools. If CALVER's gain over plurality persists (e.g., remains >5 pp), the concern is resolved; if it shrinks to noise or reverses, the central claim should be re-scoped as applying only to validity-class correctness, not answer-matched accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the definition of correctness in the headline comparison. In supplied-graph CLEAR, the same executable predicate is used both to score candidates and to grade them: the Supplementary states that 'both the checker and the grader evaluate the published graph predicate instead of matching CLEAR's listed example.' The 42.1% figure in Figure 2/Table 4 is therefore a measure of how well CALVER ranks candidates by its own validity criterion, not by the benchmark's reference-answer standard. The paper explicitly notes that in an audited subset, 11 of 21 graph-valid CALVER selections differ from the listed answer; under exact-match grading those are errors. The constructed-graph, ATE, and DoVerifier experiments use distinct verifier and grader objects and provide external grounding, but the headline claim—and the abstract's '42.1% where plurality... remain near 30%'—is reported on the self-graded supplied-graph regime. The transfer of the headline numbers to the community-standard exact-match metric is therefore unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CALVER, a training-free symbolic verifier that scores sampled LLM reasoning traces against executable causal criteria (d-separation, backdoor adjustment, intervention, etc.) and selects the highest-scoring trace for best-of-K causal reasoning. The authors argue that when a query has multiple valid answers, plurality voting fragments valid mass and can select an invalid repeated answer. CALVER instead checks each candidate against a fixed graph predicate. The paper contributes theoretical results: a soundness theorem for a strict ATE certificate, a Hoeffding-style validity-fragmentation theorem showing when plurality converges to an invalid mode, an exact finite-K selection law, and a query-local graph transfer result. Empirically, on a 126-item clean core of CLEAR find-one-valid tasks, CALVER reaches 42.1% accuracy versus 30.5% for the best generic scorer, with gains reproduced across eight policies, ten bnlearn networks, text-to-graph construction settings, an independent do-calculus prover, and a Knights-and-Knaves truth-table task. The paper emphasizes that correctness is defined as satisfying the graph predicate, not as matching the benchmark's listed answer.","tokens_in":35513,"tokens_out":5793,"duration_ms":70480,"significance":"If the headline empirical claim holds, the paper makes a useful contribution: it demonstrates that executable validity checking can outperform answer-frequency, reward-model, and LLM-judge selection in a regime where multiple answers are correct. The theoretical results are standard but correctly presented; the fragmentation bound is a clean formal explanation of why plurality fails under answer multiplicity, and the exact selection law clarifies when larger sampling budgets help. The experimental design is unusually thorough, with structure-only controls, tie-rule ablations, graph-corruption tests, node-renaming invariance checks, independent DoVerifier and truth-table validators, and exact-ground-truth ATE evaluation. The central caveat is that the headline CLEAR comparison is graded by the same executable predicate used by the verifier, rather than by exact match to the benchmark's listed answer. The authors disclose this, but the abstract's headline numbers are reported without the caveat, and the transfer of those numbers to the community-standard exact-match metric is unestablished. The external experiments (ATE, K&K, graph-from-text) use distinct verifier/grader objects and","major_comments":[{"comment":"The headline claim that CALVER 'reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%' is computed under the paper's own correctness definition: a selected answer is correct if it satisfies the published graph predicate, not if it equals CLEAR's listed answer. The Supplementary states that 'both the checker and the grader evaluate the published graph predicate instead of matching CLEAR's listed example,' and the audited subset shows 11 of 21 CALVER selections differ from the listed answer. Under the standard CLEAR exact-match metric, those 11 selections would be scored as errors, and the reported ranking advantage may not transfer. Please report exact-match accuracy on the same frozen pools for all selectors, along with exact-match candidate coverage. Without these numbers, the abstract's headline comparison is not comparable to conventional CL","section":"Abstract; Experiments, Figure 2/Table 4; Supplementary 'What the checker sees and what grades it'"},{"comment":"The per-pool decomposition reports 197 repairs and 54 new errors, yielding a net gain of 24.8 percentage points, and this is presented as the fragmentation mechanism working as predicted. This decomposition is also computed under the predicate-validity correctness criterion. Under exact-match grading, the sets of 'plurality errors' and 'valid answers' change, and the repair/new-error ratio could be substantially different. Since this decomposition is used to locate the mechanism, it should be repeated under the exact-match criterion, or the interpretation should be explicitly restricted to the multi-validity setting. This is load-bearing because the claimed net gain is a central empirical quantity.","section":"Experiments, 'Where does CALVER repair plurality's errors?' and Supplementary Table 10"}],"minor_comments":[{"comment":"The abstract reads 'including -separation, backdoor adjustment, and intervention' where 'd-separation' is intended. The missing 'd' is a typographical error.","section":"Abstract"},{"comment":"The per-policy n values differ (360 to 378) due to a parse-admissibility rule. The paper states that the rule is applied identically across selectors, but it would help to clarify in the caption how many of the 126 problems are lost per policy and whether the common-row comparison (Table 4) is the primary analysis. The current presentation invites over-reading of small absolute differences.","section":"Table 2 and Experimental Setup"},{"comment":"The candidate-coverage curve is computed from held-out labels after generation and is a diagnostic ceiling. The caption says this, but the main text could more clearly state that coverage is not available to any selector and that the gap to coverage is not a comparison with a competing method.","section":"Figure 2"},{"comment":"Theorem 9 (Supplementary numbering) is stated for i.i.d. candidates. The main text claims the widening gap through K=32 is 'consistent with the fragmentation mechanism in Theorem 2.' That is plausible, but the empirical curves in Table 8 mix policy effects (SFT, GRPO) with selection mechanics; a sentence acknowledging that policy changes across rows are not controlled would improve precision.","section":"Supplementary, 'Exact finite-K selection law'"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope and the core idea is interesting. The main concern is that the headline CLEAR numbers are self-graded; the authors should be asked to supply exact-match re-scoring. If the exact-match advantage is much smaller or absent, the paper's contribution would still be meaningful but the abstract and title claims would need substantial revision. I also note that the Code and Data Package is only 'being prepared for public release,' which is acceptable for a revision but should be resolved before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CALVER is a real contribution. The paper makes a sharp, testable point: plurality aggregation fails when valid answers fragment across strings, and executable validity checking can recover that signal. The six-slot trace contract is a practical way to make LLM outputs machine-checkable, and the theory is standard but correctly applied: the fragmentation bound is a clean Hoeffding argument, the strict ATE certificate is proper backdoor adjustment with explicit guards, and the exact selection law is a useful clarification of how best-of-K behaves. The query-local transfer analysis is the most conceptually interesting piece, and the DoVerifier and truth-table experiments show the principle generalizes beyond their own checker. The ablation suite, including structure-only, tie-rule, graph corruption, node renaming, and property tests, is exactly what a skeptical reader wants. The main soft spot is the one the stress test flagged. The headline 42.1% on CLEAR is graded with the same predicate the verifier optimizes. The supplementary is transparent about this, and the task is genuinely find-one-valid with multiple correct answers, so predicate satisfaction is arguably the right criterion. But the community's default metric for CLEAR is exact match to the listed answer, and the paper never reports those numbers. The abstract's framing could easily mislead. If exact-match accuracy is much lower, the headline claim is over-strong. This is fixable by reporting both scorings, or at least a sensitivity table. Two other soft spots: code and data are promised but not yet public, and meaning-based self-consistency baselines (soft self-consistency, semantic clustering) are absent. The set medoid is a reasonable proxy but is not the same as soft voting over semantic equivalence. None of this undermines the central mechanism. The paper is honest about its regime boundaries, includes a null result on CLadder where no gain is expected, and the external experiments (bnlearn, graph-from-text, independent prover) use distinct verifier and grader objects, which grounds the transfer claims. This deserves a serious referee. I would accept it with a request for exact-match re-scoring and public code and data.","headline":"Solid, careful paper: CALVER is a real selection signal for causal reasoning, but the headline CLEAR result is self-graded and needs an exact-match re-scoring before the claim is fully transferable.","tokens_in":703,"tokens_out":802,"would_cite":true,"duration_ms":33005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Verifier beats majority voting on multi-answer causal tasks by 11 points","keywords":["causal reasoning","symbolic verification","best-of-K selection","self-consistency","answer multiplicity","d-separation","backdoor adjustment","LLM inference-time scaling"],"falsifier":"Re-grade the same frozen pools and selected indices using exact string match to the dataset's reference answer. If CALVER's advantage over plurality and the reward model disappears under that grading rule, the headline result is an artifact of the graph-validity criterion rather than a property of causal selection.","tokens_in":35149,"feed_emoji":"✅","tokens_out":5800,"duration_ms":58406,"temperature":0.7,"pith_summary":"The paper argues that when a causal-reasoning question admits several equally valid answers, test-time scaling by majority voting fails: the valid answers split the sampled votes while one familiar invalid answer becomes the largest mode. To fix this, it introduces CALVER, a training-free symbolic verifier that scores each sampled reasoning trace against executable causal criteria—graph-theoretic tests such as d-separation, the backdoor criterion, and intervention surgery—and returns the earliest highest-scoring candidate without ever seeing a reference answer. On a frozen pool of find-one-valid causal queries, CALVER reaches 42.1% selected-answer accuracy where exact plurality, a reward model, an LLM judge, and model confidence all sit near 30%, and the gap grows with the number of samples through K=32. The paper also proves a validity-fragmentation theorem explaining why plurality converges to an invalid answer, proves soundness of a strict average-treatment-effect certificate, and shows the same selection principle transfers to logic puzzles via a truth-table checker.","feed_headline":"Verifier beats majority voting on multi-answer causal tasks by 11 points","feed_subtitle":"When several answers are valid, sampled-reasoning votes split; CALVER checks each candidate against the causal graph instead.","key_machinery":"The load-bearing object is the trace contract: each sampled reasoning trace is typed into six slots—graph, query, strategy, derivation record, computed result, and answer—and scored by six deterministic bits covering parsing/binding, graphical validity via d-separation or m-separation for mixed graphs, the backdoor adjustment test, provenance and format, numerical recomputation, and answer consistency. Selection is the earliest trace achieving the maximum deployed score of six. Two theorems carry the argument: a validity-fragmentation bound showing that exact plurality's probability of returning a valid answer decays exponentially when one invalid answer outweighs every individual valid answ","core_discovery":"The central claim is that executable causal-validity checking is a stronger target-label-free selection signal for best-of-K causal reasoning than agreement-based or score-based aggregation. When a query asks for any single object satisfying a causal predicate—any valid backdoor adjustment set, any d-separating set, any intervention witness—the set of valid answers is a decidable class, so validity can be checked per candidate rather than inferred from consensus. CALVER implements this by parsing each sampled trace into six typed slots and returning the earliest trace whose six deterministic checks all pass on the verification graph. On the same frozen candidate pools the symbolic selector r","pith_inferences":["Editorial inference: the same candidate-wise validity check could apply to other test-time scaling regimes, such as code generation where acceptance tests are the predicate, or theorem proving with a proof checker.","Editorial inference: the paper's grading criterion—graph-predicate validity rather than the dataset's listed answer—is the faithful reading of 'find-one-valid', but it means headline accuracies are not comparable to exact-match benchmark scores; re-grading the same pools under exact match would be the decisive test.","Editorial inference: a distilled process-reward model trained to imitate the symbolic scores fails to recover the selection gain, suggesting the mechanism depends on exact recomputation rather than any learnable proxy.","Editorial inference: the monotone gain with K suggests a testable extension—hybrid routing that estimates extraction reliability and switches between exact solving and candidate-wise verification; the paper's data on correlated misreads predicts such a gate should not rely on edge-set agreement between traces."],"forward_implications":["If CALVER's results hold, best-of-K selection in causal reasoning should use executable validity rather than plurality whenever a causal graph is available or can be built from text.","The advantage should continue to widen with sampling budget because validation pools validity across answer strings, while plurality saturates once an invalid answer is the largest mode; the paper observes this through K=32.","Generic scorers—reward models, model confidence, and LLM judges—cannot recover the signal by scaling; a 72B judge given the same graph remains statistically tied with plurality.","The selection principle generalizes to any domain with a decidable validity predicate, demonstrated on knights-and-knaves puzzles with a truth-table checker.","When text-to-graph extraction is reliable, extracting one graph and solving exactly beats candidate-wise verification; the paper gives an empirical crossover rule for choosing between the two routes."],"supporting_citations":[{"why":"Defines self-consistency, the exact-plurality aggregation the paper argues fails under answer multiplicity.","marker":"Wang et al. 2023"},{"why":"Supplies the find-one-valid causal benchmark subset containing multiple graph-valid answers used as the primary testbed.","marker":"Chen et al. 2024"},{"why":"Provides the backdoor criterion and d-separation definitions that the verifier executes.","marker":"Pearl 2009"},{"why":"Supplies the causal-inference background and separation criteria used in the graph checks.","marker":"Peters, Janzing, and Schölkopf 2017"},{"why":"Defines m-separation for acyclic directed mixed graphs, used for ADMG queries.","marker":"Richardson 2003"},{"why":"Supplies an independent do-calculus prover used as a reference-free candidate-wise selector under a separate implementation.","marker":"He et al. 2026"},{"why":"Provides the knights-and-knaves benchmark used to test transfer of the selection principle to logic via a truth-table checker.","marker":"Xie et al. 2025"},{"why":"Supplies the learned reward model that serves as a strong generic comparator.","marker":"Liu et al. 2026"}],"fun_headline_variants":["CALVER verifies causal validity per candidate, beating majority vote by 11 points","When votes split, causal checks pick the valid trace: CALVER wins by 11 pts","Symbolic verifier beats majority voting on multi-answer causal tasks","Causal validity beats consensus: CALVER outperforms voting on best-of-K","Voting fails when many answers are valid; CALVER verifies instead"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline comparison defines an answer as correct when it satisfies the causal-graph predicate, not when it matches the benchmark's listed answer; under exact-match grading the reported 42.1% versus ~30% margins would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["CALVER verifies causal validity per candidate, beating majority vote by 11 points","When votes split, causal checks pick the valid trace: CALVER wins by 11 pts","Symbolic verifier beats majority voting on multi-answer causal tasks","Causal validity beats consensus: CALVER outperforms voting on best-of-K","Voting fails when many answers are valid; CALVER verifies instead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4405,"prompt_tokens":798,"completion_tokens":3607,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":3515}},"tokens_in":542,"tokens_out":3607,"duration_ms":28924,"temperature":1.0,"reasoning_tokens":3515,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:41:06.815147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-grade the same frozen pools and selected indices using exact string match to the dataset's reference answer. If CALVER's advantage over plurality and the reward model disappears under that grading rule, the headline result is an artifact of the graph-validity criterion rather than a property of causal selection.","supporting_citations":[{"cited_title":"2023 , publisher=","cited_arxiv_id":null,"evidence_quote":"Defines self-consistency, the exact-plurality aggregation the paper argues fails under answer multiplicity."},{"cited_title":"Findings of the Association for Computational Linguistics: EMNLP 2024 , pages =","cited_arxiv_id":null,"evidence_quote":"Supplies the find-one-valid causal benchmark subset containing multiple graph-valid answers used as the primary testbed."},{"cited_title":"Scandinavian Journal of Statistics , volume =","cited_arxiv_id":null,"evidence_quote":"Defines m-separation for acyclic directed mixed graphs, used for ADMG queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the knights-and-knaves benchmark used to test transfer of the selection principle to logic via a truth-table checker."},{"cited_title":"2026 , url=","cited_arxiv_id":null,"evidence_quote":"Supplies the learned reward model that serves as a strong generic comparator."}],"review_version":1}