Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

ThinkTuning claims that adding a teacher model's feedback to a student's RL rollouts instills reflective reasoning that pure RL cannot, improving benchmarks by a few points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ThinkTuning uses teacher feedback grafted onto GRPO rollouts to teach small language models reflective reasoning, improving MATH-500, AIME, and GPQA-Diamond accuracy over vanilla GRPO.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection ThinkTuning is a plausible GRPO-plus-teacher-feedback recipe with modest, consistent gains, but the 'without distillation' claim is untested and the core AAS objective is not visible in the text. the 4 major comments →

arxiv 2508.07616 v2 pith:2X4G4LAL submitted 2025-08-11 cs.AI cs.CLcs.LG

ThinkTuning: Instilling Cognitive Reflections without Distillation

Aswin RRV , Jacob Dineen , Divij Handa , Md Nayem Uddin , Mihir Parmar , Chitta Baral , Ben Zhou This is my paper
classification cs.AI cs.CLcs.LG
keywords cognitive reflectionreinforcement learningGRPOteacher feedbackreasoningself-correctionexplorationtest-time scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ThinkTuning is a training method for language models that aims to make them reason reflectively by having a teacher model comment on their answers during reinforcement learning. In each round, a fraction of the student's own sampled answers is sent to a teacher, whose appended feedback highlights a misstep and shows a corrected line of thought; the teacher can express one of four reflective behaviors—self-conflict, self-agreement, self-critique, or self-consultancy. The paper's central claim is that this implicit supervision instills reflective reasoning behaviors that plain reinforcement learning cannot, and that it improves reasoning accuracy on math, science, and commonsense benchmarks. If true, the method gives smaller models a way to learn thinking skills from a same-size teacher rather than needing a much larger model, a human-written chain-of-thought corpus, or pure trial and error.

Core claim

Worded as the authors do: ThinkTuning is a two-stage procedure. First, a small set of few-shot exemplars fixes the format of the teacher's feedback, which always contains an <opinion> (correct or incorrect), a <reason> that models the student's internal dialogue, and a <phrase> that demonstrates the reflective behavior by solving the problem. Then, during GRPO training on a training set of grade-school math problems, the algorithm samples n rollouts per question, selects a random subset γ to send to the same-size teacher, appends the teacher's feedback to those rollouts, and uses all n rollouts to estimate group advantages. An Advantage-Aware Shaping weight scales the loss on teacher-augment

What carries the argument

The central mechanism is the augmented rollout: for a fraction γ of the n rollouts sampled per question, the student's own attempt is passed to a teacher that returns an <opinion>, <reason>, and <phrase> showing a reflective cognitive behavior; this feedback is appended to the rollout, and the combined sequence enters the GRPO advantage computation. To keep the off-policy teacher tokens from destabilizing training, an Advantage-Aware Shaping weight adjusts the policy-gradient update for those tokens according to both the rollout's advantage and the student's current probability of producing the token. The four-behavior taxonomy (self-conflict, self-agreement, self-critique, self-consultancy)

Load-bearing premise

The load-bearing premise is that the teacher's feedback acts as a hint that redirects the student's own reasoning rather than as a template to be copied; if the student is simply learning to reproduce the teacher's worked solutions, the claim of instilling cognitive reflection without distillation fails.

What would settle it

Train a copy of the student with plain supervised fine-tuning on the same teacher-augmented rollouts used in ThinkTuning, matching data volume; if SFT reaches or exceeds ThinkTuning's benchmark gains, the improvement is better explained by imitation of the teacher's feedback than by the RL-style shaping, and the 'implicit supervision' mechanism is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, weak reasoning models can acquire reflective behaviors through a same-size teacher's feedback during RL, reducing the need for a stronger, larger teacher or for hand-authored reasoning traces.
  • The guiding-exploration experiment implies that any behavior a teacher can verbalize—a stylistic tic, a verification habit, a formatting rule—can be injected into the student even if the student never samples it in its own rollouts.
  • Because rewards only check final answers, ThinkTuning shifts the burden of teaching intermediate reasoning from reward design to the teacher's feedback, making it easier to apply to tasks where process supervision is expensive.
  • The token-length analysis indicates ThinkTuning models spend more inference compute per question than GRPO, consistent with the idea that the learned behavior is genuinely longer reflective chains, which could matter for test-time scaling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitations section explicitly says the method depends on the teacher's ability to produce helpful guidance, uses only final-answer rewards, and was tested only on smaller models; these constraints bound the strength of the central claim.
  • The 'without distillation' framing is fragile because the teacher feedback in the appendix often contains fully worked solutions; a head-to-head against supervised fine-tuning on the same feedback would settle whether the student learns a transferable skill or copies an answer template.
  • The ethics statement's mention of proprietary LLMs used in the study suggests the 'same-size teacher' characterization should be read cautiously—at least some guidance may have come from models much larger than the 3B student.
  • The reported underperformance on GSM8K and StrategyQA suggests that reflective behaviors can turn into overthinking on simpler questions; a curriculum that lowers the guidance fraction γ on easy items is a natural testable follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ThinkTuning, a GRPO-based interactive training method in which a student model's rollouts are augmented with feedback from a teacher model. The teacher evaluates a student attempt, provides corrective feedback and a 'phrase' that demonstrates one of four cognitive behaviors (self-conflict, self-agreement, self-critique, self-consultancy). A subset of rollouts is randomly selected for teacher augmentation; the augmented and unaugmented rollouts are used together in the GRPO advantage computation, with an Advantage-Aware Shaping (AAS) weight intended to stabilize updates from off-policy teacher tokens. The method is evaluated by training Llama-3.2-3B-Instruct on GSM8K questions and testing on eight reasoning benchmarks, reporting gains over zero-shot baselines and over vanilla GRPO on several datasets. A separate experiment claims that ThinkTuning can instill a previously unknown behavior (ending responses with a Thalapathy Vijay quote) that pure RL cannot evoke. The conclusion is that ThinkTuning instills cognitive reflection 'without distillation.'

Significance. If the benchmark gains are robust, ThinkTuning offers a practical, low-overhead recipe for injecting reasoning behaviors into smaller models that do not already exhibit them. The method is simple, the code is released, and the token-length and error analyses are useful additions. The Thalapathy Vijay experiment is an interesting demonstration of exploration steering. However, the central 'without distillation' claim is not currently established. The teacher's feedback in Appendix A.1 contains fully worked solutions, so the augmented rollouts could simply act as supervised targets; the paper never compares against direct SFT or behavior cloning on the same trajectories. The reported gains are also small on several benchmarks and arrive without error bars or significance tests. These gaps are load-bearing for the paper's main claims, though they appear addressable with additional experiments.

major comments (4)
  1. [Appendix A.1, Section 3/4] The 'without distillation' claim is untested. The teacher's <phrase> in the Self-Conflict exemplar contains a complete solution ('20 chickens times 3 cups each gives 60 cups total...'), and these teacher-written tokens are appended to the student rollout and included in the GRPO/AAS update. A student could therefore improve simply by imitating the teacher's solution. The manuscript never compares against direct SFT or behavior cloning on the same teacher-feedback trajectories, so the observed gains may be supervised imitation rather than RL-instilled reflection. Adding this baseline is essential to support the title and the central claim.
  2. [Table 1, Abstract] The quantitative claims (e.g., 2.08%, 2.23%, and 3.99% over vanilla GRPO on MATH-500, AIME, and GPQA-Diamond) are reported without error bars, significance tests, or the number of independent runs. Given the small margins on several benchmarks, these numbers are not sufficient to establish that the improvements are systematic. Report mean and standard deviation over at least three seeds, and a paired significance test where appropriate.
  3. [Abstract vs. Ethics Statement] The abstract and introduction describe a teacher model 'of the same size,' but the Ethics Statement says GPT-4 and Gemini were used. If the teacher is a much larger proprietary model, the method is closer to distillation and the 'same-size' claim is false. The teacher model(s) and their sizes must be specified for each experiment. If the same-size claim is central, the teacher should be the same size as the student, or the claim should be revised.
  4. [Section 5, Figure 4] The Thalapathy Vijay experiment demonstrates exploration steering, but it is substantially forced by design: the reward explicitly pays for the target phrase and the teacher is prompted to emit it. This does not distinguish RL-instilled behavior from copying teacher-provided text. A control that trains on the same teacher-augmented rollouts with SFT, or with an advantage weight but without the RL update, would clarify whether the RL loop contributes anything beyond imitation.
minor comments (6)
  1. [Section 6] Typo: 'paves the the way' should be 'paves the way.'
  2. [Table 2] The error frequencies sum to well over 100%, which is presumably because multiple error types can co-occur in one instance. Please state this explicitly.
  3. [Throughout] The benchmark name is inconsistently written as 'GSM8k' and 'GSM8K.' Please standardize.
  4. [Section 5, reward definition] The reward equation for the Thalapathy Vijay experiment has garbled formatting with empty boxes. Please fix the equation so the reward components are legible.
  5. [Appendix A.1, Self-Consultancy example] The student response says '80% of the file' even though the question says '40% of the way.' This is presumably an intentional erroneous student attempt, but it may confuse readers; consider marking it as such.
  6. [Related Work] The paper cites Yan et al. (2025) on off-policy guidance but does not discuss it in the related-work narrative. Add a sentence positioning ThinkTuning relative to that work.

Circularity Check

1 steps flagged

Main benchmark derivation is self-contained and non-circular; the only forced-by-design element is the Thalapathy-Vijay 'instill unknown behavior' experiment, where the target phrase is part of both the reward and the teacher prompt.

specific steps
  1. self definitional [Section 5, 'Can THINKTUNING instill unknown behaviors?' (Figure 4 experiment)]
    "We define our reward function r(o) as follows: r(o) = [1 if the final answer is correct, 0 otherwise] + [1 if "Thalapathy Vijay" appears in the response, 0 otherwise]. ... For THINKTUNING, we prompt the teacher model to provide guidance that quotes Thalapathy Vijay in a movie-like dialogue about approaching the problem. ... a model trained with THINKTUNING was able to generate responses ending with a quote from Thalapathy Vijay ... This experiment shows that THINKTUNING can successfully instill a behavior that was completely unknown to the student model beforehand."

    The target string 'Thalapathy Vijay' is an explicit input to both components of the training signal: the reward function pays for it and the teacher is prompted to emit it. The augmented rollout therefore already contains the exact behavior that the experiment later reports as 'instilled.' The observed output is a restatement of the experimental construction (target phrase in reward + target phrase in teacher prompt => target phrase in output), not an independently discovered behavior. The word 'unknown' applies only to the student before training, but the behavior is fully specified in the reward and teacher prompt, so the demonstration reduces to an input-repetition check rather than a test of whether ThinkTuning can discover novel behaviors. The main benchmark evaluations are unaffected

full rationale

The central derivation chain — GRPO rollouts augmented by teacher feedback, AAS-weighted updates, and the reported benchmark gains — is self-contained and not circular. The MATH-500/AIME/GPQA-Diamond improvements over vanilla GRPO are external evaluations on held-out benchmarks, not reconstructions of training inputs; no fitted constant is renamed as a prediction. The self-citations in the reference list (PlanTuning, PlanGen, ToW, TriplePO) are background and are not load-bearing for the main claim. The genuine interpretive weakness is that the paper never compares against SFT/behavior cloning on the same teacher-augmented trajectories, so the 'without distillation' framing is untested; however, a missing control is a validity gap, not a circular reduction. The only step that is forced by the paper's own construction is the Thalapathy-Vijay demonstration: the target string is written into both the reward and the teacher prompt, so the model's adoption of the quote is an input repetition rather than an independently discovered behavior. This is a minor, localized circularity; it does not affect the benchmark results.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper does not introduce new physical or conceptual entities; it introduces a training procedure and a taxonomy of behaviors. The load-bearing assumptions are about teacher signal validity, off-policy correction, and the sufficiency of the four cognitive categories. The unstated AAS objective and the unstated teacher model identity are the main gaps in the ledger.

free parameters (4)
  • guidance fraction gamma = not reported
    Fraction of rollouts augmented with teacher feedback; controls how much off-policy signal enters GRPO and affects the central claim.
  • AAS weighting function parameters = not reported
    Advantage-aware shaping weights combine advantage and student token confidence; the exact form and any temperature or scale constants are not shown in the provided text.
  • number of rollouts n per question = not reported
    GRPO rollout count affects variance and exploration; not stated in the visible experimental section.
  • reward weights in behavior-steering experiment = binary reward for correctness plus binary reward for target phrase
    The toy experiment rewards the target quote explicitly, which makes the 'instill unknown behavior' result partly engineered.
axioms (4)
  • domain assumption GRPO advantage estimation remains valid when off-policy teacher tokens are included and corrected by AAS.
    The paper states the teacher guidance violates importance-sampling assumptions and asserts AAS fixes this, but the derivation is not visible in the provided text.
  • domain assumption Teacher feedback teaches a generalizable cognitive skill rather than leaking the final answer.
    Appendix exemplars show complete worked solutions inside the feedback, so the student sees the answer through the teacher. The paper assumes this instills reflection rather than answer copying.
  • domain assumption The four chosen cognitive behaviors (Self-Conflict, Self-Agreement, Self-Critique, Self-Consultancy) are sufficient to represent reflective reasoning.
    The authors select four behaviors because they are well defined, but no evidence is given that these categories capture the mechanisms behind the observed gains.
  • domain assumption The Gandhi et al. 2025 result that RL only draws out existing behaviors is accepted as motivation.
    The paper's framing depends on this external claim but does not independently verify it on the base model used here.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ThinkTuning: Instilling Cognitive Reflections without Distillation." pith.science (2026). https://pith.science/paper/2X4G4LAL

@misc{pith2026250807616,
  author       = {Pith},
  title        = {Pith review of: ThinkTuning: Instilling Cognitive Reflections without Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2X4G4LAL}},
  note         = {Machine review of arXiv:2508.07616}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. While RL drives this self-improvement paradigm, a recent study (Gandhi et al., 2025) shows that RL alone does not truly instill these new reasoning abilities - it merely draws out behaviors already present in the base models. This raises a question: How can we train the models that don't exhibit such thinking behavior to develop it in the first place? To this end, we propose ThinkTuning, a GRPO-based interactive training approach where we augment the rollouts of a student model with the guidance from a teacher model. A simple idea from classroom practice inspires our method: a teacher poses a problem, lets the student try an answer, then gives corrective feedback -- enough to point the mind in the right direction and then show the solution. Each piece of feedback reshapes the student's thoughts, leading them to arrive at the correct solution. Similarly, we find that this type of implicit supervision through feedback from a teacher model of the same size improves the reasoning capabilities of the student model. In particular, on average, our method shows a 3.85% improvement over zero-shot baselines across benchmarks, and on MATH-500, AIME and GPQA-Diamond it shows 2.08%, 2.23% and 3.99% improvements over the vanilla-GRPO baseline. Source code is available at https://github.com/3rdAT/ThinkTuning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

    cs.AI 2026-05 unverdicted novelty 5.0

    Mid-training LLMs on self-generated diverse reasoning paths improves subsequent RL performance on mathematical benchmarks and OOD tasks.

  2. RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systems

    cs.CL 2025-09 unverdicted novelty 5.0

    RECAP is an inference-time framework using cognitive appraisal theory to enhance emotional alignment and transparency in medical dialogue systems across model scales.

Reference graph

Works this paper leans on

44 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    B Camp and M Bash. 1978. Think aloud: Group manual (rev. ed.). Denver, CO: University of Colorado Medical School

    work page 1978

  2. [2]

    Anastasia Chouvalova, Anisha S Navlekar, Devin J Mills, Mikayla Adams, Sami Daye, Fatima De Anda, and Lisa B Limeri. 2024. Undergraduates’ reactions to errors mediates the association between growth mindset and study strategies. International Journal of STEM Education, 11(1):26

    work page 2024

  3. [3]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168, 9

    Pith/arXiv arXiv 2021

  5. [5]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. 2025. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307

    Pith/arXiv arXiv 2025

  6. [6]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346--361

    2021

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  8. [8]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  9. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  10. [10]

    Hubert JM Hermans. 2023. Dialogical self theory. In The Palgrave encyclopedia of the possible, pages 389--394. Springer

    work page 2023

  11. [11]

    Hubert JM Hermans and Thorsten Gieser. 2011. Handbook of dialogical self theory. Cambridge University Press

    work page 2011

  12. [12]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    Pith/arXiv arXiv 2023

  13. [13]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    Pith/arXiv arXiv 2024

  14. [14]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    Pith/arXiv arXiv 2020

  15. [15]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS

    2022

  16. [16]

    Training language models to self-correct via reinforcement learning, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. Training language models to self-correct via reinforcement learning, 2024. URL https://arxiv. org/abs/2409.12917

    Pith/arXiv arXiv 2024

  17. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  18. [18]

    Majumder, Katherine M

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Bodhisattwa P. Majumder, Katherine M. Hermann, Sean Welleck, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651

    Pith/arXiv arXiv 2023

  19. [19]

    Yeray Mera, Gabriel Rodr \' guez, and Eugenia Marin-Garcia. 2022. Unraveling the benefits of experiencing errors during learning: Definition, modulating factors, and explanatory theories. Psychonomic bulletin & review, 29(3):753--765

    work page 2022

  20. [20]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    Pith/arXiv arXiv 2025

  21. [21]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katherine Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedb...

    Pith/arXiv arXiv 2022

  22. [22]

    Amy A Overman, Joseph DW Stephens, and Mary F Bernhardt. 2021. Enhanced memory for context associated with corrective feedback: evidence for episodic processes in errorful learning. Memory, 29(8):1017--1042

    work page 2021

  23. [23]

    Mihir Parmar, Palash Goyal, Xin Liu, Yiwen Song, Mingyang Ling, Chitta Baral, Hamid Palangi, and Tomas Pfister. 2025 a . Plan-tuning: Post-training language models to learn step-by-step planning for complex problem solving. arXiv preprint arXiv:2507.07495

    Pith/arXiv arXiv 2025

  24. [24]

    Mihir Parmar, Xin Liu, Palash Goyal, Yanfei Chen, Long Le, Swaroop Mishra, Hossein Mobahi, Jindong Gu, Zifeng Wang, Hootan Nakhost, and 1 others. 2025 b . Plangen: A multi-agent framework for generating planning and reasoning trajectories for complex problem solving. arXiv preprint arXiv:2502.16111

    Pith/arXiv arXiv 2025

  25. [25]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290

    Pith/arXiv arXiv 2023

  26. [26]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  27. [27]

    Amir Saeidi, Shivanshu Verma, Aswin RRV, and Chitta Baral. 2024. Triple preference optimization: Achieving better alignment with less data in a single step optimization. arXiv preprint arXiv:2405.16681

    Pith/arXiv arXiv 2024

  28. [28]

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and 1 others. 2018. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835

    Pith/arXiv arXiv 2018

  29. [29]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438

    Pith/arXiv arXiv 2015

  30. [30]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  31. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Yankai Li, Yu Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  32. [32]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  33. [33]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. https://openreview.net/forum?id=4FWAwZtd2n Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning . In The Thirteenth International Conference on Learning Representations

    2025

  34. [34]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    Pith/arXiv arXiv 2018

  35. [35]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    Pith/arXiv arXiv 2023

  36. [36]

    Hemish Veeraboina. 2023. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024 Aime problem set 1983-2024

    work page 2023

  37. [37]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain-of-thought reasoning in language models. In ICLR

    work page 2023

  38. [38]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, volume 35, pages 24824--24837

    work page 2022

  39. [39]

    Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin RRV, Chitta Baral, and Ben Zhou. 2024. Tow: Thoughts of words improve reasoning in large language models. arXiv preprint arXiv:2410.16235

    Pith/arXiv arXiv 2024

  40. [40]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945

    Pith/arXiv arXiv 2025

  41. [41]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  42. [42]

    Yong Yao, Zhijie Xu, Eliana Reif, and Hang Zhao. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601

    Pith/arXiv arXiv 2023

  43. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  44. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.