{"id":"e9a5df1e-29b8-44ff-95b4-d2141bcfb43b","arxiv_id":"2607.27783","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Combining multiple LLMs' reasoning traces into weighted DAGs gives an auditable consensus graph that matches self-consistency and modestly improves on majority voting.","lead":"Large language models' chain-of-thought reasoning is merged here into a weighted graph: each reasoning step gets a score from how many independent traces support it. The result is an inspectable 'consensus reasoning' that slightly beats majority voting on several benchmarks and exposes alternative justifications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'matched-budget' baseline is not matched: accuracy-weighting in Table 1(a) uses 50 held-out labels per dataset to set per-model priors that the unweighted majority-vote baseline never sees, so the headline claim that structural aggregation beats majority vote is confounded.","rationale":"The method is plausible, code is promised, and the qualitative/explanatory results are useful. But the paper's most important empirical assertion—that structural DAG aggregation, not answer voting, is what beats the baseline—rests on a confounded comparison. The reader's chosen weakest assumption (extractor fidelity) is real but secondary: it is acknowledged, and cross-trace attestation mitigates random extraction noise. The accuracy-prior confound is not acknowledged and is directly testable. If the accuracy-weighted MV baseline performs as well as the DAG ensemble, the contribution reduces to inspectability and diversity, not accuracy. I therefore keep the reader's CONDITIONAL verdict, but for a different reason; no shift to ACCEPT/REJECT is warranted from this pass.","tokens_in":16596,"tokens_out":10939,"duration_ms":102878,"concrete_test":"Add to Table 1(a) a baseline that uses the same held-out accuracy priors: for each of the 20 traces, weight its final answer by its source model's W_acc^m, sum weights per answer, and pick the argmax. If the DAG ensemble no longer outperforms this accuracy-weighted MV on 6/6 (or if differences are within noise), re-scope the claim to 'structural aggregation adds value beyond an accuracy prior' and drop 'matched-budget' from the accuracy-weighted comparison. Report paired differences across the 5 seeds with a Wilcoxon signed-rank test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 defines a 50-example held-out split 'used to estimate per-model accuracy priors'; Section 3.4 then sets w_t = W_acc^{m(t)} for accuracy weighting. The Table 1(a) baseline is an unweighted 4-model majority vote over the same 20 traces. The comparison therefore varies two factors at once: (i) DAG merging/scoring vs. voting, and (ii) use of 50 labeled examples per dataset. The paper says the matched-budget comparison 'isolat[es] the contribution of structural aggregation: any difference reflects how traces are combined,' but that is true only for the simple-weighting rows. Those rows beat MV on just 3/6 datasets and lose on SARA, MuSR-OP, and FOLIO. The 6/6 claim appears only after the accuracy prior is injected. FOLIO also ties the baseline exactly (0.6405 vs 0.6405), so 'every dataset' is not supported even at the point-estimate level. The missing control is an accuracy-weighted majority vote using the same W_acc^m; without it, the central accuracy gain cannot be attributed to structure.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for ensembling LLM reasoning at the structural level. It extracts a typed, bundle-structured DAG from each chain-of-thought trace, merges nodes across traces via an embedding filter plus an LLM judge, weights merged nodes by cross-trace attestation (simple or accuracy-weighted), and selects the highest-weighted conclusion together with its maximally supported proof subgraph. The framework is evaluated on six reasoning benchmarks (SARA, GPQA-D, three MuSR domains, FOLIO) using a four-model pool with ≤12B parameters, comparing against majority voting and self-consistency at equal trace budgets. The paper reports accuracy gains over the majority-vote baseline (max +3.1% on MuSR-MM), matches or exceeds self-consistency on single-model pools, and provides evidence via LLM-judge and human evaluation that the consensus subgraphs are preferred as justifications.","tokens_in":16915,"tokens_out":3538,"duration_ms":35429,"significance":"If the accuracy claims were established, the framework would be a useful contribution to auditable multi-model reasoning: it generalizes self-consistency from answers to reasoning structure, requires no trained process reward model, and returns an inspectable weighted graph with alternative justifications. The method is clearly specified and reproducible: open-source models, fixed prompts in the appendix, and a stated random seed. The qualitative/explainability results—consensus graphs preferred over length-matched alternatives in 54.4–65.4% of comparisons and moderate Spearman correlations with an independent judge—support the qualitative value of the approach. However, the headline quantitative claim—'outperforms matched-budget majority-vote baseline on every dataset'—is not currently supported by the reported experiments, for the reasons detailed below.","major_comments":[{"comment":"The claimed 'matched-budget' comparison is confounded for the accuracy-weighted rows. Section 4 states that a 50-example held-out split is used to estimate per-model accuracy priors W_acc^m, and Section 3.4 uses these priors for the accuracy-based weights. The baseline is an unweighted 4-model majority vote over the same 20 traces, so the comparison varies two factors simultaneously: (i) DAG aggregation vs. voting, and (ii) access to 50 labeled examples per dataset. The paper's assertion that the difference 'reflects how traces are combined' (§4) is therefore valid only for the simple-weighting rows, which beat majority vote on only 3/6 datasets and lose/tie on SARA, MuSR-OP, and FOLIO. The missing control is an accuracy-weighted majority vote using the same W_acc^m; without it, the 6/6 accuracy gain cannot be attributed to structural aggregation.","section":"§4, §5.1, Table 1(a)"},{"comment":"The claim that the accuracy-weighted ensemble 'outperforms' the majority-vote baseline on 'every dataset' is not statistically supported. For SARA (0.6387±0.0080 vs 0.6279±0.0158), GPQA-D (0.2784±0.0495 vs 0.2662±0.0421), MuSR-OP (0.4922±0.0174 vs 0.4874±0.0184), and MuSR-TA (0.2890±0.0167 vs 0.2830±0.0057), the differences are within one standard deviation of the baseline. FOLIO ties the baseline exactly at the point estimate (0.6405 vs 0.6405), contradicting the 'outperforms' wording in the abstract and Section 1. Only MuSR-MM shows a difference (59.6 vs 56.5) that exceeds the reported standard deviations. The abstract and conclusion should be revised to reflect the specific datasets where gains are statistically meaningful, or additional evidence (e.g., paired tests across seeds) should be provided.","section":"§5.1, Table 1(a)"},{"comment":"The 'training-free' characterization of the accuracy-weighting scheme is potentially misleading. The method uses 50 ground-truth labels per dataset to set per-model priors (Section 4), which is not 'training' in the model-parameters sense, but it does consume labeled data. This is a minor terminology issue, but it matters for the comparison to majority vote, which sees no labels. The distinction should be stated clearly: the simple-weighting variant is label-free; the accuracy-weighted variant is not.","section":"§3.4, §8"}],"minor_comments":[{"comment":"Table 1(a) is difficult to parse because the baseline row is placed after two ensemble rows; consider listing the baseline first or adding a separator. Also, the 'Bold/underline' note says 'best/second-best per column in each block' but the table does not bold any values; either apply the formatting or remove the note.","section":"§5.1, Table 1 caption"},{"comment":"The Spearman correlations and win rates are computed on a per-question basis and then averaged; the paper should state whether these averages are over questions or over seeds, and how many questions are included. The 'n=5 subgraphs per question' is mentioned, but the total number of questions per dataset is not.","section":"§5.3, Table 2"},{"comment":"The definition of the merged support function is clear, but the notation π^{-1}(c) is ambiguous when clusters are built from a clustering π: V → C; consider using [c] or a subscripted class notation to avoid confusion with an inverse function.","section":"§3.3, Eq. (2)"},{"comment":"The temperature of the trace generators is reported as 0.6, but the number of decoding samples per trace is not; the 'temperature' is a free parameter and the reader cannot reproduce the exact sampling distribution without the seed/sample count. Please report the number of independent generations per model per question.","section":"§4"},{"comment":"The strong-model homogeneous ensemble uses Qwen 2.5 32B as trace generator and the same model as DAG extractor (Section 4). This shared identity could introduce a systematic bias in the extracted structure. At minimum, a sentence acknowledging this and arguing why it is not circular would be helpful.","section":"§5.2, Table 1(c)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the structural aggregation framework, which is well-specified and has strong qualitative/explainability support. The main concern is that the accuracy-advantage claim is not yet established: the only comparison that isolates structure (simple weighting) beats majority vote on only half the datasets, and the accuracy-weighted comparison is confounded by the use of held-out labels. The authors should add the missing accuracy-weighted majority-vote control and either report paired significance tests or temper the 'every dataset' claim. The referee report should emphasize these points; the paper is otherwise solid and the framework is worth publishing after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea is genuinely new and worth your time: generalize self-consistency by extracting a typed DAG from each CoT trace, merging them with a bundle-preserving operator, and scoring nodes by cross-trace attestation. The pipeline is carefully specified—prompts in the appendix, open-weight models, code link—and the single-model results showing parity with self-consistency while returning an inspectable graph are the strongest evidence. That is a real contribution.\n\nThe soft spot is the headline accuracy claim. The abstract says the ensemble outperforms a matched-budget majority vote on every dataset with max +3.1%. Table 1(a) tells a different story: accuracy-weighted beats the unweighted baseline on five datasets and ties on FOLIO, and most gaps are within one standard deviation of the baseline. Only MuSR-MM's 3.1% is clearly outside noise. The stress-test confound is real: accuracy weighting uses 50 held-out labels per dataset to set per-model priors that the unweighted baseline never sees. The simple-weighting row, which is the actual structural aggregation comparison, beats majority vote on only 3 datasets and loses on the other three. So the 'every dataset' claim rests on injecting the held-out prior, not on structure alone. The missing control is an accuracy-weighted majority vote using the same priors; without it you cannot attribute the gains to DAG aggregation. The paper's own sentence—'simple weighting also beats majority vote on three datasets'—implicitly concedes this.\n\nThe dependence on a 32B extractor to faithfully convert prose to DAGs is acknowledged in the Limitations and is a real fragility, though they do validate and discard malformed extractions. The LLM-judge correlations are moderate and the human eval is small but consistent.\n\nWho is this for? Practitioners in high-stakes legal/scientific settings who want auditability and are willing to pay for 20 chains of compute. It is a solid systems paper with a novel mechanism. But the accuracy claims need softening or better support. I would send it to peer review—a good referee will ask for the accuracy-weighted majority-vote control and significance tests—but I would not repeat the 'beats majority vote' claim until that control is run.\n\nRecommendation: engage with it, especially for the structural-ensembling idea, but treat the headline result as unverified.","headline":"Novel structural ensembling, but the 'beats majority vote' claim is confounded and mostly within noise.","tokens_in":17394,"tokens_out":4357,"would_cite":true,"duration_ms":39080,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structural ensembling of reasoning graphs — not just final answers — improves LLM accuracy and returns an auditable consensus chain.","keywords":["LLM reasoning","chain-of-thought","ensemble methods","reasoning graphs","DAG aggregation","self-consistency","interpretability","multi-model voting"],"falsifier":"Take a dataset where majority voting clearly outperforms any single model, run the framework with a deliberately corrupted extractor (e.g., one that randomly swaps or drops nodes), and check whether ensemble accuracy collapses to or below the majority-vote baseline while node weights still look plausible. Alternatively, on a logic benchmark with gold proofs, compare the extracted consensus graph's inferred AND/OR structure against the ground-truth proof: if high-accuracy answers frequently ride on incorrectly reconstructed structure, the claimed mechanism is not doing the work.","tokens_in":16489,"feed_emoji":"🧠","tokens_out":3971,"duration_ms":38084,"temperature":0.7,"pith_summary":"The paper argues that when multiple language models reason aloud, the reasoning steps themselves carry a consensus signal that final answers discard. It extracts a directed acyclic graph from each chain of thought, merges graphs across models while preserving logical 'bundles' of jointly supporting premises, weights each merged node by how many traces independently attest to it, and returns the highest-weighted conclusion together with its strongest supporting subgraph. Across six benchmarks spanning statutory interpretation, graduate science, narrative multi-hop reasoning, and first-order logic, this structural ensemble beats majority voting over the same traces at the same budget (up to +3.1% accuracy), and matches or exceeds self-consistency on a single model. Its stated value beyond accuracy is inspectability: users see which steps are well-supported, which alternatives were considered, and how much support each conclusion received.","feed_headline":"Structural voting beats answer voting across six LLM benchmarks","feed_subtitle":"Step-by-step consensus from multiple models outperforms majority vote and stays inspectable.","key_machinery":"The load-bearing mechanism is bundle-preserving weighted DAG aggregation. Each reasoning trace is converted into an AND/OR DAG whose nodes carry typed labels (Planning, Fact, Reasoning, Conclusion) and whose edges are grouped into bundles: a support bundle is a conjunctive set of premises that jointly justify a node, and multiple bundles are disjunctive alternatives. A merge operator maps nodes across traces to clusters (pruned by embedding similarity, verified by an LLM judge) and fuses identical bundles, so each merged node accumulates an attestation count. Node scores are computed topologically: a bundle's strength is the weakest-link minimum over its members, OR gates take the max over a","core_discovery":"The central claim is that aggregating the internal reasoning structure of LLM traces—rather than only their final answers—produces both accuracy gains and an auditable artifact. Specifically, the paper claims that a weighted merge of DAGs extracted from multiple traces, with weights derived purely from cross-trace attestation, outperforms a matched-budget majority-vote baseline on every dataset tested (maximum +3.1% on a narrative multi-hop benchmark) and matches or exceeds self-consistency on single-model trace pools. The resulting consensus subgraph is preferred by human and LLM judges over random alternative justifications leading to the same answer, with win rates 54.4–65.4% in five of s","pith_inferences":["The DAG representation invites a natural user-intervention interface: if a specific step is wrong, a human could edit or delete that node and re-propagate weights; the paper does not test this, but the structure makes it possible.","Attestation weighting is a training-free proxy for process supervision; if extraction noise is controlled, it might transfer to single-model settings that use diverse prompts or temperatures to simulate multiple traces.","The AND/OR bundle semantics could be linked to probabilistic logic or argumentation theory to yield calibrated confidence intervals on each step, a direction the paper only touches qualitatively.","One testable extension: on a logic benchmark with gold proofs, compare the extracted consensus graph's logical form against the ground-truth proof to see whether accuracy gains actually come from correct structure reconstruction rather than from answer-frequency artifacts."],"forward_implications":["At a fixed trace budget, structural ensembling outperforms answer-level majority voting on all six benchmarks, with the largest gain on narrative multi-hop reasoning.","On a single model, the framework matches or exceeds self-consistency while additionally providing an inspectable reasoning graph.","Ensemble weights track LLM-judge rankings of reasoning quality (Spearman rho = 0.30–0.51), so the weight signal is not just repetition.","The framework preserves competing conclusions and alternative justifications, useful for audit and for open-ended tasks like argument generation.","Because step-level confidence is derived from cross-trace agreement, no process reward model or step-level labels are needed; it scales to low-resource domains."],"fun_headline_variants":["Reasoning DAGs beat majority vote on six benchmarks","Vote on the steps, not just the answers","Consensus reasoning graphs edge out majority vote","Structural ensemble wins across six LLM tests","Step-level consensus outperforms answer voting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole pipeline assumes a small extractor model faithfully converts free-form chains of thought into typed, bundle-structured DAGs; if extraction introduces systematic errors, the attestation counts and consensus graphs inherit them.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning DAGs beat majority vote on six benchmarks","Vote on the steps, not just the answers","Consensus reasoning graphs edge out majority vote","Structural ensemble wins across six LLM tests","Step-level consensus outperforms answer voting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1039,"prompt_tokens":782,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":187}},"tokens_in":526,"tokens_out":257,"duration_ms":3161,"temperature":1.0,"reasoning_tokens":187,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:26:48.756819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset where majority voting clearly outperforms any single model, run the framework with a deliberately corrupted extractor (e.g., one that randomly swaps or drops nodes), and check whether ensemble accuracy collapses to or below the majority-vote baseline while node weights still look plausible. Alternatively, on a logic benchmark with gold proofs, compare the extracted consensus graph's inferred AND/OR structure against the ground-truth proof: if high-accuracy answers frequently ride on incorrectly reconstructed structure, the claimed mechanism is not doing the work.","supporting_citations":[],"review_version":1}