Pith. sign in

REVIEW 5 major objections 6 minor 71 references

A reward that scores how much a chain of thought raises the correct answer's likelihood lifts a 7B model's deductive accuracy from 13% to 60% in 400 training steps.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Interesting reward idea and a useful new benchmark, but the central mechanism is confounded by mismatched prompts and the evaluation lacks the ablations needed to trust the headline numbers. the 5 major comments →

arxiv 2509.06024 v1 pith:JFXACJAK submitted 2025-09-07 cs.AI

Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL

Haoyang He , Zihua Rong , Kun Ji , Chenyang Li , Qing Huang , Chong Xia , Lan Yang , Honggang Zhang This is my paper
classification cs.AI
keywords DRERreinforcement learningchain-of-thoughtreasoning quality rewardlog-likelihood margindeductive reasoning benchmarkLogicTreelength advantage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard rule-based RL rewards for language models check only whether the final answer is right and well formatted, so they never tell the model whether its chain of thought actually helped. This paper proposes DRER, a reward that measures the log-likelihood margin between the probability the model assigns to the correct answer with and without the chain of thought, plus a length-based advantage that penalizes abnormally short or verbose responses. The central claim is that maximizing this margin steers the policy toward chains that genuinely contribute to correct deduction, and the paper backs this with a new benchmark, LogicTree, built from seven formal inference rules. On LogicTree, a 7B model trained for 400 steps goes from 13% to 60% average accuracy, retains 31% accuracy at depth 8 where frontier models fall near zero, and shows improved confidence and generalization to AIME24, ZebraLogic, and ProntoQA. The reason to care: if the margin measures reasoning quality, it gives RL a cheap, rule-free signal for logical training that does not require per-step human annotation.

Core claim

The paper's central discovery is that the difference in log-likelihood of the ground-truth answer tokens with versus without a chain of thought — Δ(x) = ℓ_CoT − ℓ_NoCoT — is a usable learning signal for reasoning quality. When a CoT is genuinely useful, the model assigns higher probability to the correct answer conditional on it, so Δ is positive; when the chain is spurious, Δ is negative. DRER converts this margin into a bounded reward R_q = tanh(Δ), adds it to the task reward (answer correctness plus format), and multiplies the group-normalized advantage by a length attenuation factor g_i derived from validation-set length quantiles. Training with this composite reward on the LogicTree dat

What carries the argument

The load-bearing object is the log-likelihood margin Δ(x) = ℓ_CoT − ℓ_NoCoT, the per-token average log-probability the current policy assigns to the ground-truth answer tokens when the prompt includes the generated chain of thought versus when it does not. Squashed through tanh it becomes the Reasoning Quality Reward; multiplied into the group advantage it becomes the Dynamic Length Advantage via g_i = exp(−max{0, L_min − ℓ_i, ℓ_i − L_max}/τ) with bounds taken from the validation set. The margin is doing the work: it is the only part of the reward that links intermediate reasoning tokens to the answer, and it is what lets DRER prefer chains that "demonstrably raise the likelihood of the corr

Load-bearing premise

The load-bearing premise is that the margin Δ(x) — how much more likely the current model finds the correct answer when the chain of thought is present — actually measures whether the chain helped reasoning, and not merely whether the model became more confident or leaned on memorized surface cues; the model computing the margin is also the one being trained, so the signal is not stationary.

What would settle it

Generate LogicTree prompts where a filler chain that restates the answer without valid deduction is inserted between premises and answer; if R_q is positive for such filler chains and accuracy on held-out depth-8 trees still rises substantially, the margin is rewarding confidence rather than deductive utility. A cheaper check on the same logic: compute the WR-group margin (+2.46 nats on GSM8K, +1.81 on LogicTree) on adversarial CoTs that state the answer before deriving it — if the margin stays positive, Eq. (11)'s sign convention is not tracking logical validity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the margin is a faithful quality signal, any RL pipeline that can sample CoT and no-CoT rollouts can upgrade outcome-only rewards to reasoning-sensitive ones without human annotation or a process reward model.
  • Training on formal deduction transfers: the same 400-step 7B model improves on AIME24, MMLU-redux, ZebraLogic, and ProntoQA, suggesting logical RL generalizes beyond the training domain.
  • The length attenuation suppresses the length-inflation tendency of GRPO-style group advantages, so reasoning quality can improve while token consumption falls by about 75% relative to the DAPO/GRPO baselines.
  • Consistency across paraphrases of the same logical structure rises from 0.01 to 0.41 on average, implying the model learns logical paradigms rather than surface patterns, though it still collapses at depth 7-8.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The margin Δ is computed with the current policy, which is itself being trained; the paper validates the sign convention post-hoc. A natural test the authors did not run: freeze the reference policy and compare credit assignment under frozen vs. live policy, since non-stationarity could mean early high-Δ chains are the ones the model already agrees with rather than the ones that teach new inferenc
  • Because the reward needs only the correct answer tokens and two forward passes, it should extend to any verifiable domain beyond True/False deduction — multiple-choice QA or code with testable prefixes — with the length advantage recalibrated per bucket.
  • The consistency metric (same logic, different words) could double as a reward-hacking detector: if accuracy rises while consistency stays flat, DRER is fitting the benchmark's wording distribution rather than the inference paradigm.
  • The scalar margin collapses CoT utility into one number; a finer variant would decompose the margin per inference step to say which premise the chain exploited, connecting to process reward models without their annotation cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DRER, a reinforcement-learning reward shaping method for LLM chain-of-thought reasoning. It defines a reasoning-quality reward R_q = tanh(ℓ_CoT − ℓ_NoCoT), computed as the per-token log-likelihood margin of the ground-truth answer when conditioning on a generated CoT versus a No-CoT prompt, and combines it with a task-level correctness reward. A secondary component, the Dynamic Length Advantage, multiplies the group-normalized advantage by an exponential penalty for responses whose lengths fall outside validation-derived quantiles. The authors also release LogicTree, a dynamically generated deductive-reasoning dataset with controllable depth and width, and report experiments training Qwen2.5-7B-Instruct-1M with DAPO/GRPO for 400 steps, claiming LogicTree accuracy rises from 13% to 60%, confidence gains of 30–60%, and generalization to ZebraLogic, ProntoQA, AIME24, and MMLU-redux.

Significance. If the central mechanism is validated, DRER is a potentially valuable plug-and-play reward for reasoning-quality RL: it is rule-based, requires no learned reward model, and directly targets whether CoT tokens increase the likelihood of the correct answer. The LogicTree dataset is a genuinely useful contribution: it is synthetically generated, semantically decoupled from world knowledge, and offers controlled reasoning depth, consistency metrics, and exact prompt templates. The paper also releases code and data, which supports reproducibility. However, the empirical support for the reward mechanism is currently incomplete: the reported gains are not ablated against the length penalty, the CoT/No-CoT prompts are confounded by template differences, and all headline results come from a single training run without error bars or multiple seeds.

major comments (5)
  1. [Appendix D, Tables 4–5; §3.1, Eqs. (9)–(12)]
  2. [§4.2, §4.5; Eqs. (13)–(14)]
  3. [§4.4; §3.1, Eqs. (9)–(12)]
  4. [Abstract; Introduction; Table 1]
  5. [§4.1; Tables 1, 11, 12]
minor comments (6)
  1. [§5]
  2. [Eqs. (9), (14)]
  3. [Throughout]
  4. [Appendix D]
  5. [§2, Eqs. (3)–(4)]
  6. [Table 12]

Circularity Check

1 steps flagged

The reasoning-quality reward's increase is by construction; the central accuracy claim remains independently measured.

specific steps
  1. self definitional [Section 4.4, Figure 5; Eqs. (11)-(13)]
    "We train the same base model with two popular base algorithms, GRPO and DAPO on DRER framework. Picture 5 shows that reasoning quality reward increases sharply ,which proves that the CoT tokens are becoming positive for model reasoning."

    R_q is defined in Eq. (12) as tanh(Δ(x)) with Δ(x)=ℓ_CoT−ℓ_NoCoT (Eq. 11), and the training objective (Eq. 13) explicitly maximizes this R_q. Therefore a rising 'reasoning quality reward' curve is a monotone consequence of optimizing the very margin that defines the reward; it cannot independently 'prove that the CoT tokens are becoming positive.' The same Δ is also used in Section 4.4 (Tables 9–10) to validate the hypothesis in Eq. (10). Thus the evidence offered for the mechanism is the objective itself, making the reported increase partly by construction rather than an independent confirmation of reasoning-quality improvement.

full rationale

The central empirical claim—DRER raising LogicTree accuracy from 13% to 60% on a held-out test split (Table 1)—is measured independently of the reward function and is not circular. However, the paper's secondary claim that DRER improves 'reasoning quality' relies on the log-likelihood margin Δ(x), which is the exact quantity that R_q = tanh(Δ) is designed to maximize. Figure 5's increase in R_q is therefore a direct consequence of optimizing that objective, not an independent verification that CoT tokens are becoming more useful. Additionally, the CoT and No-CoT prompts (Tables 4–5) differ in system instructions and output-format requirements, so Δ may partly reflect template effects; this is a confound rather than a circularity but further weakens the interpretation of the reward curve. No load-bearing self-citation chains or imported uniqueness theorems are present. The accuracy improvements on LogicTree and generalization to other benchmarks provide independent content, so the paper is only partially circular.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim leans on an assumed equivalence between a model-confidence margin and reasoning quality, plus standard RL machinery and the validity of the synthetic LogicTree annotations. The length bounds are derived from validation data, and the reward constants (lambda_q, tau, task reward weights) are hand-chosen without sensitivity analysis.

free parameters (4)
  • lambda_q (reasoning reward weight) = 1
    Weight in composite reward R = R_task + lambda_q * R_q (Eq. 13); set to 1 with no reported sensitivity analysis.
  • tau (length attenuation temperature) = 8
    Decay temperature in Eq. (14); paper states tau in [5,10] and chooses 8, with no ablation.
  • L_min^(d), L_max^(d) per difficulty bucket = 5th and 95th percentiles of validation-set response lengths
    Dynamic bounds in Eq. (14) are recomputed from the validation set, so the length penalty is fit to held-out data.
  • Task reward constants S_format and S_answer = 1/-1 and 2/-1.5/-2
    Hand-designed rule-based reward weights in Section 4.1; not tuned or reported with sensitivity.
axioms (6)
  • standard math Propositional logic inference rules (modus ponens, modus tollens, etc.) are valid and sufficient to determine truth values in LogicTree.
    LogicTree is generated from these seven rules and evaluated by an automatic verifier; the paper assumes this formal grounding in Appendix B.1.
  • ad hoc to paper The log-likelihood margin ell_CoT minus ell_NoCoT is a valid measure of whether CoT tokens are beneficial.
    Eqs. (9)-(12) define the reasoning-quality reward from this margin; Section 4.4 gives post-hoc supporting evidence, but the identity 'confidence gain equals reasoning quality' is assumed.
  • domain assumption Ground-truth answers are available and correctly formatted for all LogicTree training questions.
    Algorithm 1 (lines 5-6) inserts the ground-truth answer into the CoT trajectory to compute rewards; any annotation error would propagate to reward and evaluation.
  • ad hoc to paper Validation-derived length quantiles L_min and L_max remain representative during training.
    Eq. (14) applies a length attenuation based on percentiles from the previous validation round; the paper does not analyze the effect of distribution shift as the policy changes.
  • standard math The policy gradient and GRPO/DAPO objectives are unbiased under the stated MDP with deterministic transitions.
    Section 2 relies on standard PPO/GRPO/DAPO theory; no new proof is provided.
  • domain assumption Model confidence scores are comparable across prompts and models.
    Section 4.4 compares average log-probability gains across GSM8K and LogicTree; no calibration analysis is given.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL." pith.science (2026). https://pith.science/paper/JFXACJAK

@misc{pith2026250906024,
  author       = {Pith},
  title        = {Pith review of: Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFXACJAK}},
  note         = {Machine review of arXiv:2509.06024}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) has recently become the dominant paradigm for strengthening the reasoning abilities of large language models (LLMs). Yet the rule-based reward functions commonly used on mathematical or programming benchmarks assess only answer format and correctness, providing no signal as to whether the induced Chain-of-Thought (CoT) actually improves the answer. Furthermore, such task-specific training offers limited control over logical depth and therefore may fail to reveal a model's genuine reasoning capacity. We propose Dynamic Reasoning Efficiency Reward (DRER) -- a plug-and-play RL reward framework that reshapes both reward and advantage signals. (i) A Reasoning Quality Reward assigns fine-grained credit to those reasoning chains that demonstrably raise the likelihood of the correct answer, directly incentivising the trajectories with beneficial CoT tokens. (ii) A Dynamic Length Advantage decays the advantage of responses whose length deviates from a validation-derived threshold, stabilising training. To facilitate rigorous assessment, we also release Logictree, a dynamically constructed deductive reasoning dataset that functions both as RL training data and as a comprehensive benchmark. Experiments confirm the effectiveness of DRER: our 7B model attains GPT-o3-mini level performance on Logictree with 400 trianing steps, while the average confidence of CoT-augmented answers rises by 30%. The model further exhibits generalisation across diverse logical-reasoning datasets, and the mathematical benchmark AIME24. These results illuminate how RL shapes CoT behaviour and chart a practical path toward enhancing formal-reasoning skills in large language models. All code and data are available in repository https://github.com/Henryhe09/DRER.

Figures

Figures reproduced from arXiv: 2509.06024 by Chenyang Li, Chong Xia, Haoyang He, Honggang Zhang, Kun Ji, Lan Yang, Qing Huang, Zihua Rong.

Figure 1
Figure 1. Figure 1: Overview of the Dynamic Reasoning Efficiency Reward (DRER) framework. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The framework of LOGICTREE automatic construction pipeline. We first sample atomic logic structures and sentences from seven deduction logic rules and four sentential logics, then fill it with natural statements in filtered AscentKB [35], and eventually construct the nested argument tree. Those intermediate will be hidden and transformed into questions. LogicTree incorporates several key improvements: (1) … view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of the DAPO baseline with the DRER framework over 400 steps. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy on the LogicTree during post-training with DAPO (left) and GRPO (right), with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Reasoning quality reward on the LogicTree during post-training with DRER. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of response lengths over training steps between DAPO and DAPO+DRER. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of response lengths over training steps between GRPO and GRPO+DRER. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prediction distribution comparison between DAPO and DAPO+DRER under Chain-of [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prediction distribution comparison between our model (DAPO+DRER) and Qwen2.5-7B [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Word frequencies of seven deductive reasoning terms explicitly mentioned in LLMs [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Fβ-Score, Answer Rate and Precision metrics Comparison across various models. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 28 canonical work pages · 1 internal anchor

  1. [1]

    L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    2025

  2. [2]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    2024

  3. [3]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models.arXiv preprint arXiv:2501.14851, 2025

    Michael K Chen, Xikun Zhang, and Dacheng Tao. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models.arXiv preprint arXiv:2501.14851, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Empower- ing llms with logical reasoning: A comprehensive survey.arXiv preprint arXiv:2502.15652, 2025

    Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. Empower- ing llms with logical reasoning: A comprehensive survey.arXiv preprint arXiv:2502.15652, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Self- playing adversarial language game enhances llm reasoning.Advances in Neural Information Processing Systems, 37:126515–126543, 2024

    Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. Self- playing adversarial language game enhances llm reasoning.Advances in Neural Information Processing Systems, 37:126515–126543, 2024

    work page 2024

  7. [7]

    Transformers as soft reasoners over language.arXiv preprint arXiv:2002.05867, 2020

    Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language.arXiv preprint arXiv:2002.05867, 2020

    Pith/arXiv arXiv 2002

  8. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    Gemini 2.0 flash thinking, 2024

    Google DeepMind. Gemini 2.0 flash thinking, 2024

    2024

  10. [10]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    2021

  12. [12]

    Logicgame: Benchmarking rule-based reasoning abilities of large language models.arXiv preprint arXiv:2408.15778, 2024

    Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Logicgame: Benchmarking rule-based reasoning abilities of large language models.arXiv preprint arXiv:2408.15778, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Folio: Natural language reasoning with first-order logic

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021

    2021

  16. [16]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Reasoning-enhanced healthcare predictions with knowledge graph community retrieval.arXiv preprint arXiv:2410.04585, 2024

    Pengcheng Jiang, Cao Xiao, Minhao Jiang, Parminder Bhatia, Taha Kass-Hout, Jimeng Sun, and Jiawei Han. Reasoning-enhanced healthcare predictions with knowledge graph community retrieval.arXiv preprint arXiv:2410.04585, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Boardgameqa: A dataset for natural language reasoning with contradictory information

    Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. Boardgameqa: A dataset for natural language reasoning with contradictory information. Advances in Neural Information Processing Systems, 36:39052–39074, 2023

    work page 2023

  19. [19]

    Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019. 10

    work page 2019

  20. [20]

    Chain of code: Reasoning with a language model-augmented code emulator

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. InInternational Conference on Machine Learning, pages 28259–28277. PMLR, 2024

    work page 2024

  21. [21]

    Codei/o: Condensing reasoning patterns via code input-output prediction.arXiv preprint arXiv:2502.07316, 2025

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction.arXiv preprint arXiv:2502.07316, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Limr: Less is more for rl scaling, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, 2025

    2025

  23. [23]

    From system 1 to system 2: A survey of reasoning large language models, 2025

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Rui Mei, Junfeng Fang, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system 1 to system 2: A survey of reasoning large language models, 2025

    work page 2025

  24. [24]

    Zebralogic: On the scaling limits of llms for logical reasoning.arXiv preprint arXiv:2502.01100, 2025

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning.arXiv preprint arXiv:2502.01100, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Natural language inference in context-investigating contextual reasoning over long texts

    Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. Natural language inference in context-investigating contextual reasoning over long texts. InProceedings of the AAAI conference on artificial intelligence, volume 35, pages 13388–13396, 2021

    work page 2021

  26. [26]

    Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947–2962, 2023

    Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947–2962, 2023

    work page 2023

  27. [27]

    大模型逻辑推理研究综述 (survey on logical reasoning of large pre- trained language models)

    Hanmeng Liu and Yue Zhang. 大模型逻辑推理研究综述 (survey on logical reasoning of large pre- trained language models). In Zhao Xin, editor,Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 48–62, Taiyuan, China, July 2024. Chinese Information Processing Society of China

    work page 2024

  28. [28]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124, 2020

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124, 2020

    Pith/arXiv arXiv 2007

  29. [29]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025

    2025

  30. [30]

    Reasoning with large language models for medical question answering.Journal of the American Medical Informatics Association, 31(9):1964–1975, 2024

    Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. Reasoning with large language models for medical question answering.Journal of the American Medical Informatics Association, 31(9):1964–1975, 2024

    work page 1964

  31. [31]

    Improve mathematical reasoning in language models by automated process supervision, 2024

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024

    work page 2024

  32. [32]

    Exploring the limit of outcome reward for learning mathematical reasoning, 2025

    Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathematical reasoning, 2025

    2025

  33. [33]

    Real: Efficient rlhf training of large language models with parameter reallocation

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Real: Efficient rlhf training of large language models with parameter reallocation. InProceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. mlsys.org, 2025

    2025

  34. [34]

    Enhancing reasoning capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    work page 2024

  35. [35]

    Advanced semantics for commonsense knowledge extraction

    Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. Advanced semantics for commonsense knowledge extraction. InProceedings of the Web Conference 2021, pages 2636–2647, 2021

    work page 2021

  36. [36]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024

    2024

  37. [37]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  38. [38]

    Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning

    Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 15012–15032, 2024. 11

    work page 2024

  39. [39]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    2025

  40. [40]

    Qwq-32b: Embracing the power of reinforcement learning, 2024

    Qwen. Qwq-32b: Embracing the power of reinforcement learning, 2024

    2024

  41. [41]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.arXiv preprint arXiv:2210.01240, 2022

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.arXiv preprint arXiv:2210.01240, 2022

    Pith/arXiv arXiv 2022

  42. [42]

    Testing the general deductive reasoning capacity of large language models using ood examples.Advances in Neural Information Processing Systems, 36:3083–3105, 2023

    Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using ood examples.Advances in Neural Information Processing Systems, 36:3083–3105, 2023

    work page 2023

  43. [43]

    High-dimensional continuous control using generalized advantage estimation, 2018

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018

    2018

  44. [44]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  45. [45]

    Automatic solutions of logic puzzles

    Peter Sempolinski. Automatic solutions of logic puzzles. 2009

    work page 2009

  46. [46]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Satori: Reinforcement learning with chain-of-action-thought enhances llm reasoning via autoregressive search.arXiv preprint arXiv:2502.02508, 2025

    Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain-of-action-thought enhances llm reasoning via autoregressive search.arXiv preprint arXiv:2502.02508, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning.arXiv preprint arXiv:2409.12183, 2024

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning.arXiv preprint arXiv:2409.12183, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  50. [50]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?

    Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, and Chitta Baral. Step-by-step reasoning to solve grid puzzles: Where do llms falter?arXiv preprint arXiv:2407.14790, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, and Chitta Baral. Step-by-step reasoning to solve grid puzzles: Where do llms falter? InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19898–19915, 2024

    work page 2024

  53. [53]

    Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  54. [54]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y . Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024

    work page 2024

  55. [55]

    Legalreasoner: A multi-stage framework for legal judgment prediction via large language models and knowledge integration

    Xuran Wang, Xinguang Zhang, Vanessa Hoo, Zhouhang Shao, and Xuguang Zhang. Legalreasoner: A multi-stage framework for legal judgment prediction via large language models and knowledge integration. IEEE Access, 2024

    work page 2024

  56. [56]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  57. [57]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. InThe 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24. 12

    work page

  58. [58]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025

    2025

  59. [59]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering.arXiv preprint arXiv:1809.09600, 2018

    Pith/arXiv arXiv 2018

  60. [60]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    2025

  61. [61]

    Lawllm: Intelligent legal system with legal reasoning and verifiable retrieval

    Shengbin Yue, Shujun Liu, Yuxuan Zhou, Chenchen Shen, Siyuan Wang, Yao Xiao, Bingxuan Li, Yun Song, Xiaoyu Shen, Wei Chen, et al. Lawllm: Intelligent legal system with legal reasoning and verifiable retrieval. InInternational Conference on Database Systems for Advanced Applications, pages 304–321. Springer, 2024

    work page 2024

  62. [62]

    Rest-mcts*: Llm self-training via process reward guided tree search, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search, 2024

    2024

  63. [63]

    Rest-mcts*: Llm self-training via process reward guided tree search.arXiv preprint arXiv:2406.03816, 2024

    Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.arXiv preprint arXiv:2406.03816, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    R1-reward: Training multimodal reward model through stable reinforcement learning, 2025

    Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, and Liang Wang. R1-reward: Training multimodal reward model through stable reinforcement learning, 2025

    work page 2025

  65. [65]

    Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models.arXiv preprint arXiv:2502.16906, 2025

    Qin Zhu, Fei Huang, Runyu Peng, Keming Lu, Bowen Yu, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang, and Junyang Lin. Autologi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models.arXiv preprint arXiv:2502.16906, 2025. A Technical Appendices and Supplementary Material Table 2: An example of a logictree puzzle. An e...

    Pith/arXiv arXiv 2025

  66. [66]

    Analyze the logical structure of the given statements

    work page

  67. [67]

    Identify the conditions and implications in each statement

    work page

  68. [68]

    Determine the truth value of the given statements and how they relate to each other

    work page

  69. [69]

    Check the consistency of the statements with the provided premises

    work page

  70. [70]

    Deduce the implications of the given statements on the conclusion

    work page

  71. [71]

    Alice studies

    Evaluate the final conclusion based on the logical consistency and the given premises. </think> <answer>[True, False]</answer> Solution: True False B LogicTree: Formal Paradigms and Templates B.1 Seven Deductive Paradigms in LOGICTREE LogicTree centres on seven classic deductive paradigms that constitute the atomic reasoning units of every sample. Each pa...

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.