{"id":"6a5f490e-fcd5-4fa9-b857-de047cb513cc","arxiv_id":"2505.15817","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Jointly training and voting across natural language, code, and truth-table reasoning modalities improves LLM logical reasoning accuracy by up to 11.7 percentage points.","lead":"This paper introduces Mixture-of-Thought (MoT), a training and inference framework that makes large language models reason in three formats at once: natural language, Python code, and truth tables. On FOLIO and ProofWriter benchmarks, MoT lifts accuracy by up to 11.7 percentage points over single-modality chain-of-thought baselines, allowing a 9B open model to match GPT-4 with an external solver on FOLIO.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own truth-table illustration mis-grounds the 'destroyer' premise (Table 1), so the load-bearing assumption that LLM-generated truth tables are logically faithful is unsupported.","rationale":"This is the same weakest assumption the reader identified, and it is load-bearing: the truth-table modality is the paper's main novelty, and its claimed value rests on faithful grounding and pruning. The Table 1 mis-grounding is not cosmetic, because the quality filter (Eq. 2) never checks logical faithfulness, so similar errors can survive in training data and test traces whenever the final label matches. The paper does have independent support: the MoT-vs-SoT training ablation (Table 3), pass@k scaling curves, and consistent gains across three base models suggest the empirical method works even if the mechanism is not yet verified. However, those results do not establish that truth-table reasoning is systematically enumerating cases; they are compatible with a stochastic voting benefit from additional noisy rationales. A conditional verdict with a faithfulness audit is therefore the right call, and the current reader's CONDITIONAL verdict should stand.","tokens_in":27308,"tokens_out":8066,"duration_ms":69012,"concrete_test":"Have two logicians independently verify the model-generated truth-table groundings on 100 randomly sampled FOLIO/ProofWriter test items, using a faithful first-order formalization as gold standard. As an immediate canary, re-derive Table 1 with the destroyer premise formalized as DestroyerBreaksBridge→¬C and Thor not a destroyer; if the conclusion is Uncertain rather than True, the paper's own example is invalid. If a substantial share of correct-answer truth tables (e.g., >10%) are unfaithful, the claimed systematic complementarity is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that truth-table reasoning systematically enumerates logical cases and fixes NL bottlenecks depends on the LLM's first-order grounding step being faithful. The paper's own illustrative truth table (Table 1) violates this. It encodes 'If a destroyer breaks a bridge, Peter is not a civilian' as B→¬C with B='bridge breaks'. But the original premise is about a destroyer breaking a bridge; neither Thor (explicitly a god, hence not a destroyer) nor the Hulk is asserted to be a destroyer. From T→H, H→A, A→B one only gets that the Hulk breaks a bridge; B→¬C therefore does not follow, so ¬C, S, and U do not follow. The correct three-way answer is Uncertain, not True. Because the quality filter in Eq. (2) checks only final-answer equality and tag/format validity, exactly this kind of unfaithful grounding can be retained whenever the predicted label is right. Thus the empirical gains attributed to 'truth-table complementarity' may be majority-vote noise rather than sound symbolic enumeration, and the mechanism claimed in the abstract and Sec. 2.1 is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixture-of-Thought (MoT), a framework that trains a single LLM to reason in three modalities—natural-language chain-of-thought, code-based reasoning, and a newly introduced truth-table modality—via a self-evolving loop that filters self-generated rationales by final-answer correctness and format validity, and then combines the modalities by majority vote at inference. Experiments on FOLIO and ProofWriter (and ProverQA in further analyses) report consistent gains over untrained base models, over single-thought training, and in test-time scaling, with a 9B open model reaching 78.9% on FOLIO, matching the paper's reported GPT-4 + Logic-LM number. The paper also includes ablations on policy strategy, mixing strategy, data-volume controls, and an error analysis of natural-language reasoning failures.","tokens_in":27545,"tokens_out":7135,"duration_ms":62824,"significance":"The central idea—that jointly training on multiple reasoning modalities can be more effective than selecting or augmenting a single modality—is timely and potentially valuable for improving logical reasoning in smaller open models. The paper ships code and includes several useful controlled experiments: Appendix E.5 compares MoT data against an equal volume of diverse NL-only data, Table 6 ablates on-policy versus off-policy training, and the uniqueness/complementarity analysis in Section 4.2 quantifies cross-modal synergy. If the mechanism claims were fully supported, the work would be a meaningful step toward modality-diverse self-training. However, the paper's own illustrative truth table is logically unfaithful and yields an incorrect answer to the stated problem, which directly undermines the claim that the truth-table modality 'systematically enumerates logical cases.' That issue, together with missing statistical rigor and some uncontrolled comparisons, means the current evidence supports a more modest claim: that training and voting over three model-generated reasoning formats improves accuracy, not that the symbolic enumeration mechanism is the cause.","major_comments":[{"comment":"The illustrative truth-table reasoning is logically unfaithful and produces the wrong answer. The premises imply T→H, H→A, and A→B (the Hulk breaks a bridge), and the premise 'If a destroyer breaks a bridge, Peter is not a civilian' cannot be applied because neither Thor (explicitly a god, hence not a destroyer) nor the Hulk is asserted to be a destroyer. Therefore B→¬C is not entailed, and the derived chain ¬C, S, U does not follow; the correct answer to the question is (C) Uncertain, not (A) True. This is not a minor presentational slip: the reward function in Eq. (2) checks only final-answer equality and tag/format validity, so exactly this kind of unfaithful grounding can be retained in the training set whenever the predicted label matches. The paper's claim that truth-table reasoning 'systematically enumerates logical cases' is therefore not supported by its own central example, and the mechanism attributed to the truth-table modality is not established.","section":"Section 2.1, Table 1"},{"comment":"The headline comparison 'MoT training outperforms single-thought training' is confounded by training-data volume. Each Single-Thought row in Table 3 is trained on roughly one modality's worth of data (N traces), whereas Mixture-of-Thought (Default, All) is trained on approximately 3N traces, one set per modality. The 'Three Models Combined' row uses three separate models, which is a different resource trade-off. Appendix E.5 does provide an equal-volume control for NL-only data (3N NL_CoT versus 3N MoT data), but this control is not in the main table. Please make the equal-volume comparison the primary evidence in Table 3, or add an equal-volume single-thought row, so the reader can separate modality-mixture effects from data-quantity effects.","section":"Section 3.3, Table 3"},{"comment":"The abstract and Section 3.2 claim that MoT 'consistently and significantly outperforms' strong baselines, but Table 2 reports a single run per condition with a fixed seed and no error bars, confidence intervals, or significance tests. The phrase 'significantly' is thus not substantiated; please report variance across multiple seeds or bootstrap CIs, and ideally a paired test on the FOLIO and ProofWriter instances. In addition, the +11.7pp average gain is computed against untrained 3-shot base models (Table 2, rows B-D), so it conflates the effect of MoT with the effect of any supervised fine-tuning on 1000 examples. The controlled comparisons in Table 3 and Appendix E.5 are more informative for isolating MoT's contribution and should be foregrounded.","section":"Section 3.2, Table 2"},{"comment":"The statement that 'our 9B model achieves 78.9% accuracy on FOLIO, matching the performance of Logic-LM' is selectively framed. Table 5 reports GPT-4 with Symbolic CoT at 83.3% and GPT-4 with Logic-of-Thoughts at 88.2% on FOLIO, both well above 78.9%. Matching Logic-LM specifically is true, but the broader wording in the abstract and introduction ('matches the results of GPT-4 + Logic-LM on FOLIO') risks implying parity with state-of-the-art GPT-4 methods. Please either restrict the claim to the Logic-LM baseline or acknowledge the stronger GPT-4 numbers in the same sentence.","section":"Section 3.2 and Table 5"},{"comment":"The paper's central motivation—that nearly two thirds of natural-language CoT errors are missing-branch or invalid-converse errors—rests on an automatic error-detection pipeline whose accuracy is not validated. Figure 1(c) and Section 4.2 report these percentages, and Section 4.2 mentions a human evaluation, but no agreement statistics, sample sizes, or precision/recall of the detector are given. Because the truth-table modality is introduced specifically to address these two error types, the authors should report the error detector's agreement with human annotations on a labeled sample, or provide the human-evaluation details, before using these percentages as the primary design rationale.","section":"Section 4.2 and Appendix D"}],"minor_comments":[{"comment":"The sentence 'outperforms the corresponding base models by an average of 11.7pp (from 41.1% to 61.9% ...)' lists only the Gemma-2-2B numbers in parentheses; the 11.7pp is an average across all three models, so the parenthetical should be clarified to avoid implying that 41.1% to 61.9% is the average.","section":"Section 3.2"},{"comment":"The text states that there are 10 runs for each setting, but Figure 3 shows no error bars or confidence bands; adding them would make the test-time scaling comparison considerably more convincing.","section":"Figure 3"},{"comment":"The rows labeled 'MoT(0-Shot) Single-Thought Best' should define how the 'Best' modality is selected for each model and dataset, and the 'Mixture-of-Thought All' row should state how ties are broken in the three-way majority vote.","section":"Table 2"},{"comment":"The code-reasoning example contains inline annotations such as '⇒Correct' and '⇒Incorrect' inside the Python code, which make the displayed code non-executable; these annotations should be moved to a separate column or footnote so the code example is clearly distinct from the human commentary.","section":"Appendix G.2"},{"comment":"ProverQA appears in Figure 4 and the associated analysis but is not introduced until Appendix C.1; a one-sentence definition of ProverQA in the main text would help readers interpret the depth-based results.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the journal and the topic is timely. My main concern is that the truth-table mechanism, which is the paper's most novel element, is not actually validated: the illustrative example in Table 1 is logically wrong, and the training filter Eq. (2) does not check logical faithfulness. The empirical gains may still hold, but the paper's central explanatory claim is currently unsupported. The authors should either add a verification step for truth-table grounding, or substantially soften the mechanistic claims and reposition the contribution as 'training and voting over diverse self-generated reasoning formats.' I would also encourage the editor to ask for multi-seed results or significance tests, given the strong language in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before spending an evening on it. First, the empirical core is genuinely useful: the authors train one model to reason in natural language, code, and a new truth-table format, and the ablations show that joint training beats single-format training on the same data, with consistent gains across three base models. Second, the paper's central story about why truth-table reasoning helps is not established. The illustrative truth table in Table 1 mis-grounds a first-order premise, and that error is load-bearing.\n\nThe new stuff: the truth-table CoT modality is new to my knowledge, and the self-evolving training recipe—generate traces in three formats, filter on answer correctness and tag validity, fine-tune on the survivors, repeat on-policy—is a sensible and clean extension of STaR. The paper ships code. The ablation in Table 3 is the right experiment: MoT training outperforms single-thought training, and adding modalities helps. I buy that result. The +11.7pp gain over the untrained base models is less meaningful because the baselines never see the 1000 training examples, but the Table 3 comparison controls for that. The fact that the gain reproduces across Gemma-2-2B, Gemma-2-9B, and Qwen-2.5-7B is real evidence.\n\nThe soft spots. Table 1 encodes 'If a destroyer breaks a bridge, Peter is not a civilian' as B→¬C with B='bridge breaks'. The chain only gets you to 'the Hulk breaks a bridge,' and no premise says the Hulk is a destroyer. The correct verdict for the question is Uncertain, not True. That mistake is not cosmetic. The paper claims truth-table reasoning systematically enumerates logical cases and fixes NL bottlenecks; if the grounding step can produce tables like this, the quality filter in Eq. (2) won't catch it—it only checks final-answer equality and tags. So the complementarity attributed to truth tables may be partly voting noise. The paper also overclaims 'significantly' without error bars or significance tests. The claims are consistent across models, which mitigates this, but I wouldn't use the word 'significant' in the abstract.\n\nWho should read this? Anyone working on self-training or multi-modal reasoning for LLMs. The training recipe is worth borrowing. The truth-table modality needs a fidelity study before I'd trust it as the paper describes. Deserves a serious referee: yes, with a request for a corrected example and some statistical grounding. I'd engage with it.","headline":"Solid multi-modality training results, but the truth-table mechanism is undercut by the paper's own faulty example; worth reviewing with revisions.","tokens_in":28066,"tokens_out":3347,"would_cite":true,"duration_ms":29984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a single LLM to reason in natural language, code, and truth tables, then majority-voting across the three at inference, beats single-modality chain-of-thought by up to 11.7 percentage points on FOLIO and…","keywords":["mixture-of-thought","logical reasoning","chain-of-thought","truth-table reasoning","code reasoning","self-evolving training","multi-modality reasoning","LLM reasoning"],"falsifier":"Run the trained MoT model on a random sample of FOLIO and ProofWriter problems, independently ground each problem with an off-the-shelf logical proof checker, and compare the model's truth-table rows and final labels with the checker's; the central claim fails if the accuracy gain over natural-language CoT does not concentrate in cases where the model's table is faithful, or if faithful tables do not disproportionately rescue missing-branch and invalid-converse errors.","tokens_in":27109,"feed_emoji":"🧠","tokens_out":10168,"duration_ms":81164,"temperature":0.7,"pith_summary":"The paper argues that no single reasoning modality—natural language, code, or symbolic truth tables—is enough for LLM logical reasoning, and that training a single model across all three makes it more reliable. It introduces Mixture-of-Thought (MoT), in which the model generates filtered rationales in each modality over iterative self-training rounds and then answers each problem in all three modes and takes a majority vote. On FOLIO and ProofWriter, MoT reports average accuracy gains of up to 11.7 percentage points over single-modality chain-of-thought, with a 9-billion-parameter model matching a GPT-4-based pipeline that calls an external logical solver. If the claim holds, open-weight models can approach formal-reasoning performance without an external prover, simply by learning complementary ways to think.","feed_headline":"Three reasoning modes beat one mode by up to 11.7 points","feed_subtitle":"A 9B open model reaches the level of GPT-4 with a symbolic solver.","key_machinery":"The load-bearing object is the truth-table thought paradigm, a symbolic reasoning format the model builds itself: it grounds first-order premises into a finite set of propositional predicates, enumerates truth assignments, prunes assignments that violate any premise, and labels the conclusion true if every surviving assignment satisfies it, false if none do, and uncertain otherwise. The surrounding machinery is the self-evolving MoT training loop, which filters self-generated traces by answer correctness and format validity and finetunes on-policy each round, plus inference-time majority voting across the three modalities. The mechanism that carries the argument is complementarity: the truth table explicitly covers the branches and directions that natural-language reasoning tends to skip or reverse.","core_discovery":"The paper's central claim is that reasoning modalities are complementary in a specific, error-targeted way: natural-language CoT is flexible but systematically misses disjunction branches and commits invalid-converse errors; code forces structured object-state tracking but can commit prematurely; truth-table reasoning enumerates possible assignments and therefore repairs exactly those natural-language failures. Self-evolving MoT training finetunes the model on filtered self-generated traces from all three modalities, and MoT inference aggregates the three outputs by majority vote. The paper reports that this recipe consistently beats single-modality CoT across three base models and two benchmarks, that the gain grows with reasoning depth, and that each modality solves some instances no other modality solves, with the truth-table modality uniquely rescuing cases involving transposition and disjunction.","pith_inferences":["Because the truth table is generated by the model itself with no independent verification, a version that checks or repairs tables against a lightweight logical checker is the natural next test; it would both improve accuracy and directly confirm that table faithfulness, not prompt diversity, drives the gain.","The same three-way design should transfer to other settings with a natural enumeration modality, such as math word problems with case splits or planning problems with state spaces, though the paper only evaluates logical reasoning.","The paper's error taxonomy implies a sharp diagnostic: if MoT's gain over natural-language reasoning disappears on instances where no missing-branch or invalid-converse error occurs, then the complementarity mechanism is confirmed; if not, the gain may be mostly ensemble averaging."],"forward_implications":["Single-model training on all three modalities outperforms training separate single-modality models, so complementarity is captured without tripling parameters.","The gains concentrate on deep, multi-step problems, so Mixture-of-Thought is most valuable exactly where chain-of-thought is known to fail.","Adding truth-table reasoning to code-plus-natural-language raises the oracle upper bound on both benchmarks, so the new modality is the source of coverage, not just another vote.","A 9-billion-parameter open model reaches the FOLIO accuracy of a GPT-4-based neuro-symbolic baseline, suggesting an external solver is not required for that level of performance."],"supporting_citations":[{"why":"Defines chain-of-thought prompting, the single-modality baseline MoT must beat.","marker":"[1]"},{"why":"Establishes self-consistency voting, the inference mechanism MoT extends across modalities.","marker":"[4]"},{"why":"Provides the Logic-LM neuro-symbolic baseline and the ProofWriter depth-5 split MoT compares against.","marker":"[8]"},{"why":"Shows natural-language and code modalities can complement one another, motivating the modality mix.","marker":"[11]"},{"why":"Supplies the ProofWriter benchmark used for evaluation.","marker":"[19]"},{"why":"Supplies the FOLIO benchmark and the missing-branch and invalid-converse error categories used in the analysis.","marker":"[20]"},{"why":"Introduces the self-evolving bootstrapping procedure MoT adapts with on-policy training and modality filtering.","marker":"[23]"}],"fun_headline_variants":["Truth tables + code + text: up to +11.7 points on logic","Self-evolving Mixture-of-Thought: up to 11.7pp on logical reasoning","Mix of three thought modes lifts logic reasoning by up to 11.7 points","Three reasoning modalities beat one by up to 11.7 points","MoT: three thought modes, up to +11.7pp accuracy on logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the model's self-generated truth tables faithfully capture the original logical premises, since nothing outside the model checks whether the grounding and pruning are correct.","fun_headline_variants_meta":{"raw":{"variants":["Truth tables + code + text: up to +11.7 points on logic","Self-evolving Mixture-of-Thought: up to 11.7pp on logical reasoning","Mix of three thought modes lifts logic reasoning by up to 11.7 points","Three reasoning modalities beat one by up to 11.7 points","MoT: three thought modes, up to +11.7pp accuracy on logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001909,"raw_usage":{"total_tokens":7482,"prompt_tokens":953,"completion_tokens":6529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":6423}},"tokens_in":569,"tokens_out":6529,"duration_ms":43926,"temperature":1.0,"reasoning_tokens":6423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:10:19.774912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained MoT model on a random sample of FOLIO and ProofWriter problems, independently ground each problem with an off-the-shelf logical proof checker, and compare the model's truth-table rows and final labels with the checker's; the central claim fails if the accuracy gain over natural-language CoT does not concentrate in cases where the model's table is faithful, or if faithful tables do not disproportionately rescue missing-branch and invalid-converse errors.","supporting_citations":[{"cited_title":"Hybridmind: Meta selection of natural language and symbolic language for enhanced llm reasoning","cited_arxiv_id":null,"evidence_quote":"Shows natural-language and code modalities can complement one another, motivating the modality mix."}],"review_version":1}