Pith. sign in

REVIEW 4 major objections 4 minor 58 references

Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that test-time RL fails at both low and high consensus, and that sampling-stage hints from a power-transformed prefix distribution can steer rollout groups back to a healthy consensus interval, improving accuracy over…

desk verdict A genuinely new sampling-stage fix for TTRL consensus instability, backed by a broad empirical package but carrying an unproven steering premise and an over-confident 'consistently' claim. read the letter →

arxiv 2608.03545 v2 pith:PQH3AW5B submitted 2026-08-04 cs.CL

Kunbin Xu , Xingzuo Li , Xuefeng Bai , Kehai Chen This is my paper · ORCID
classification cs.CL
keywords test-timereinforcementlearningconsensusstrengthmajorityvotingpower-transformedsamplingMCMChintsamplerLLMreasoningGRPOlabel-freeadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the main source of instability in test-time reinforcement learning (TTRL) is not the reward rule itself but the distribution of rollouts that feed the majority vote. Its proposal, Hi-TTRL, samples half a group of answers, estimates consensus strength, and when that strength falls outside a healthy interval generates reasoning-prefix hints with a power-transformed MCMC sampler. A sharpened target pulls weak-consensus groups toward a common answer; a flattened target re-introduces diversity into saturated groups. The paper reports that this consistently improves over standard TTRL across five math benchmarks and three backbones, with the largest single gain of $+19.85$ points on MINERVA under greedy decoding.

What carries the argument

The central quantity is consensus strength $c(Y)=\frac{1}{m}\sum_{i=1}^m \mathbb{I}[a_i=\hat{a}]$, the fraction of rollouts in a group that share the majority answer. The carrying mechanism is the power-target hint sampler: prefixes $h$ are drawn from $q_\alpha(h|x)\propto \pi_{\theta_{\mathrm{old}}}(h|x)^\alpha$ using progressive blockwise Metropolis–Hastings updates, with acceptance ratio $\log A=\min\{0,(\alpha-1)(\ell_k(h'|x)-\ell_k(h|x))\}$, where $\ell_k$ is the prefix log-likelihood under the old policy. This makes $\alpha$ a direct control knob: higher-likelihood prefixes are favored when $\alpha>1$ and lower-likelihood prefixes gain mass when $\alpha<1$, while $\alpha=1$ recovers ordinary sampling. The old policy then completes each hint, so the steering signal comes from the same distribution that will be updated, and the final majority vote is taken over the adaptive group.

What would settle it

On a benchmark where two high-likelihood reasoning prefixes lead to different answers with comparable probability, run Hi-TTRL and check whether the $\alpha>1$ anchor hint consistently selects the prefix whose completion raises the wrong-promoted rate; if the low-consensus branch produces a higher wrong-promoted rate and lower final accuracy than standard TTRL with the same 32-rollout budget, the steering claim is contradicted.

Watch

Extended reading notes

Core claim

In majority-voting TTRL, the reward signal is governed by consensus strength, the frequency of the most common answer in a rollout group. Hi-TTRL's central discovery is that this quantity can be regulated before voting by steering the sampling distribution: it estimates consensus from half the rollouts, and for out-of-interval groups it invokes a blockwise Metropolis–Hastings sampler that targets the power-transformed prefix distribution $q_\alpha(h|x)\propto\pi_{\theta_{\mathrm{old}}}(h|x)^\alpha$. With $\alpha>1$ a long sharpened hint promotes convergence; with $\alpha<1$ several short flattened hints promote divergence. The old policy completes each hint, and the expanded rollout group is voted and used for the GRPO update. The paper's evidence is that this steering moves consensus toward the target interval, changes the majority label mostly in low-consensus groups, and translates into accuracy gains, particularly on the smallest math-specialized model.

Load-bearing premise

The load-bearing premise is that prefixes drawn from the power-transformed distribution $q_\alpha(h|x)\propto\pi_{\theta_{\mathrm{old}}}(h|x)^\alpha$ and then completed by the old policy shift the final-answer distribution in the intended direction, sharpening toward convergence and flattening toward diversity; the paper demonstrates this transfer only on one model-dataset pair, so if it fails elsewhere the hints could steer consensus away from the target interval.

Editorial extensions

If this is right

  • Because the change is at the sampling stage, Hi-TTRL can be layered on top of any majority-vote TTRL pipeline without altering the GRPO objective or the reward formula.
  • The two consensus branches are complementary: ablations on AMC-2023 show each branch alone improves over TTRL, while the adaptive routing gives the best accuracy in every column.
  • The mechanism is visible in distribution: after the hint stage the fraction of samples inside the target interval rises and the average distance to the interval falls.
  • The added cost is roughly 1.54x GPU time and 1.58x training tokens, concentrated in hint generation and completion, with peak memory and throughput essentially unchanged.
  • On GPQA, a graduate-level scientific benchmark, the same setup improves both general backbones over TTRL, indicating the consensus-steering effect is not limited to mathematics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the steering premise is verified on a single model-dataset pair, the most direct test is to run Hi-TTRL on tasks where the policy's top prefixes lead to different answers; if sharpening anchors on the wrong branch, the low-consensus branch should show a higher wrong-promoted rate than TTRL.
  • The adaptive update size (16 rollouts when in interval, 32 when triggered) leaves open whether gains come from the power target or simply from more samples for triggered groups; a control using random prefixes with the same adaptive budget would separate these effects.
  • The fixed target interval and exponents could in principle be tuned per benchmark or training stage, and the online consensus estimate could be replaced by a prediction of consensus from partial rollouts.
  • Consensus is defined over final answers; the same sampler could regulate agreement on intermediate reasoning steps, which may give finer control on tasks without a unique final answer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Hi-TTRL, a test-time reinforcement learning method that first samples half of a rollout group, estimates consensus strength c^(1) via majority voting (Eq. (1)), and, when c^(1) falls outside a target interval [τ_low, τ_high], invokes a block-wise MCMC sampler targeting the power-transformed prefix distribution q_α(h|x) ∝ π_{θ_old}(h|x)^α (Eq. (6)). Low-consensus groups receive one long sharpened hint (α>1) and high-consensus groups receive multiple short flattened hints (α<1); the old policy completes the hints before final voting, reward assignment, and GRPO update (Algorithms 1 and 2). Experiments on five math benchmarks with three backbones, plus GPQA, report average gains over standard TTRL, with ablations, sample-budget control, scaling, and cost analyses in appendices.

Significance. If the steering mechanism is accepted, Hi-TTRL identifies a new sampling-stage intervention point for TTRL and provides a concrete way to regulate consensus strength before pseudo-label construction. The empirical package is strong in several respects: it reports ten repeats with standard deviations (Table 4), includes a sample-budget control showing that gains are not simply from larger update batches (Appendix D), measures end-to-end compute overhead (Appendix G), and adds model-scale and non-math generalization checks (Appendices E and F). The central limitation is that the link from power-transformed prefix hints to final-answer consensus is asserted rather than derived, and the only direct evidence for it is one model-dataset pair (Qwen3-1.7B-Base on MINERVA, Figure 4). The method also introduces several free hyperparameters without sensitivity analysis, so the robustness of the reported configuration is not yet established. These issues are addressable and do not, in my view, invalidate the average improvements shown in Table 1.

major comments (4)
  1. [Power-Target MCMC Hint Sampling (Eqs. (6)–(7), Figure 4)] The load-bearing steering premise is that drawing prefixes from q_α and completing them with π_{θ_old} shifts final-answer consensus in the intended direction: sharpening (α>1) increases convergence and flattening (α<1) increases diversity. This is asserted rather than derived. Eq. (6) transforms only the prefix distribution, while Eq. (7) composes it with a nonlinear completion map; a high-likelihood prefix may be a wrong reasoning path that all completions follow, and a flattened prefix distribution may fail to change the majority answer if the answer is determined in the final tokens. The only direct evidence is Figure 4 for Qwen3-1.7B-Base on MINERVA, which reports in-target rate and distance before/after but does not isolate the second-stage consensus contribution, does not vary α or N_MCMC, and does not link increased low-branch consensus to per-query answer quality. Please provide a formal argument, or additional controlled experiments on at least two more model-dataset pairs, showing that the intended direction of consensus shift actually holds.
  2. [Appendix C, Algorithm 1] The MCMC approximation quality is unassessed. With N_MCMC=10 transitions per block and no mixing diagnostics, it is unknown whether the delivered hints approximate the power target q_α closely enough to support the mechanism. The MH acceptance rule in Eq. (16) is standard, but finite-step convergence for 128-token blocks and up to 512-token hints is not guaranteed. Please report acceptance rates and proposal statistics, and include a sensitivity analysis over N_MCMC (e.g., 1, 10, 50) showing that the steering effect and final accuracy are not artifacts of a poorly mixed chain.
  3. [Abstract and Table 4] The abstract's claim that Hi-TTRL 'consistently improves over standard TTRL' is contradicted by the paper's own full results in Table 4. On Qwen3-4B-Base, AIME-2024 mean@16 shows -0.21 (↓1.00%) and GAOKAO2023-en shows -0.52/↓0.73% (greedy@1) and -0.45/↓0.63% (mean@16). The average gains in Table 1 are credible, but the consistency claim should be qualified, for example as 'improves on average, with small degradations on some benchmark-backbone pairs.'
  4. [Table 2, Branch-Level Ablation] The branch-level ablation shows that the low-consensus sharpening branch is not universally beneficial: on Qwen3-4B-Base, the with-low variant is below TTRL on both greedy@1 (46.02 vs 46.69) and mean@16 (47.62 vs 48.19), while with-high is the dominant single branch. The paper concludes that adaptive use of both branches is more effective than either alone, but this conclusion is not fully supported without reporting how often each branch triggers on each backbone and whether the full method's gains come from the low branch, the high branch, or their interaction. Please report per-branch trigger rates and, if possible, an analysis of samples where both branches would have triggered.
minor comments (4)
  1. [Table 4] The Qwen3-4B-Base AIME-2024 greedy@1 entry for Hi-TTRL reports 23.33±0.00; given stochastic training over ten repetitions, a zero standard deviation requires explanation (e.g., deterministic decoding after fixed seeds, or rounding to two decimals).
  2. [References] References Lv et al. 2025a and Lv et al. 2025b appear to be the same paper with the same title and venue; please merge or differentiate them.
  3. [Figure 4] The mechanism analysis would be easier to interpret if the before/after consensus distributions were shown separately per branch, for example as histograms or box plots, with error bars or confidence bands over training steps.
  4. [Appendix H] The metric definitions in Appendix H are helpful; consider moving at least the definition of wrong-promoted rate into the main text where Figure 2 is first discussed.

Circularity Check

1 steps flagged · score 2.0 of 10

Headline gains over TTRL are externally benchmarked and not forced by construction; only mild self-referentiality in the mechanism-analysis metric, which validates the routing interval against itself.

  1. self definitional [Mechanism Analysis (Consensus-Steering Effect, Figure 4); Eqs. (4), (8); Appendix A hyperparameters.]
    "the in-target rate is markedly higher after the hint stage, indicating that hint-guided sampling moves a larger fraction of samples into the target interval... When c(1) < τlow, ... Hi-TTRL uses a sharpened power target with αlow > 1 and samples a single long anchor hint: Hlow(x) = {hconv}, hconv ∼ q(4B)αlow (·|x)... the single low-consensus anchor is shared by all G/2 second-stage rollouts."

    The steering-validation metric is defined on the same interval [τ_low, τ_high] = [0.25, 0.75] that triggers the hint stage (Eq. 4), so Figure 4 measures the method's own routing objective. For the low-consensus branch, the effect is partly arithmetic: Y_upd = Y^(1) ∪ Y^(2)_hint, and because one anchor h_conv (Eq. 8) is shared by all 16 second-stage rollouts, if those completions concentrate on one answer (the stated design goal), then c(Y_upd) = (16·c^(1)+16)/32 ∈ [0.5, 0.6] ⊂ [0.25, 0.75] for every triggering group with c^(1) < 0.25.

full rationale

The headline claim — that Hi-TTRL improves over standard TTRL — is measured against external baselines (Table 1: five math benchmarks x three backbones; Table 6: GPQA; Figure 6: 7B scale) and is not forced by construction. Appendix D rules out the sample-count confound (Hi-TTRL beats TTRL64/32, which uses at least as many update rollouts), and Table 3 shows that temperature/top-p sampling controls do not reproduce the gains. There is no fitted-input-called-prediction pattern: the thresholds and exponents (τ_low=0.25, τ_high=0.75, α_low=4.0, α_high=0.25, B=128, N_MCMC=10) are fixed once and applied across all benchmarks, and the reported gains are not fitted on the test sets. The load-bearing steering premise — that sampling prefixes from q_α(h|x) ∝ π_old(h|x)^α and completing them with π_old shifts final-answer consensus (Eqs. 6-7) — is an empirical transfer claim, not an identity; the paper tests it in Figure 4, and Appendix C derives the MH acceptance ratio (Eq. 16) correctly for the stated power target, so the sampler does not import its conclusion via citation. The skeptic's strongest objections (mechanism evidence on one model-dataset pair only; no mixing diagnostics for N_MCMC=10; hyperparameters ungrounded for the transfer) are evidence-strength and robustness limitations, partially acknowledged in the text ('finite-step approximate sampling', 'computational constraints', the GPQA scope note), not circularity. Self-citations exist (Xu et al. 2026; Lv et al. 2025a/b) but are related-work mentions and a benchmark citation; none is load-bearing. The one genuine self-referential element is the mechanism analysis: its in-target-rate metric is defined on the same [τ_low, τ_high] used for routing, and the low-branch in-target improvement is partly arithmetic because 16 shared-anchor completions are appended to a <0.25-consensus group, forcing the combined consensus toward 0.5-0.6 whenever those completions converge as designed. This is minor, does not force the headline gain, and coexists with external validation, hence score 2.

Assumptions & free parameters 8 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three load-bearing assumptions: the MCMC sampler approximates the power target well enough, the half-batch consensus estimate is reliable, and power-transformed prefixes transfer their sharpening or flattening to final-answer agreement. The paper provides empirical support for the last only on one configuration. The free parameters are hand-chosen and untested for sensitivity.

free parameters (8)
  • tau_low = 0.25
    Lower bound of the target consensus interval; chosen from the pilot-study categories (low < 0.25).
  • tau_high = 0.75
    Upper bound of the target consensus interval; chosen from the pilot-study categories (high > 0.75).
  • alpha_low = 4.0
    Power exponent for sharpening under low consensus; chosen by hand to concentrate prefixes.
  • alpha_high = 0.25
    Power exponent for flattening under high consensus; chosen by hand to diversify prefixes.
  • B = 128
    MCMC hint block size in tokens; chosen by hand.
  • N_MCMC = 10
    Number of Metropolis-Hastings transitions per block; chosen by hand.
  • L_low = 4
    Number of hint blocks for the low-consensus branch (one long anchor hint).
  • L_high = 2
    Number of hint blocks for the high-consensus branch (four short exploratory hints).
assumptions (3)
  • domain assumption Finite-step MCMC with warm starts targets the power-transformed prefix distribution sufficiently for downstream consensus steering.
    Inherited from Karan and Du (2025); used in Section 'Power-Target MCMC Hint Sampling' and Appendix C. The paper does not analyze convergence or approximation error for the prefix setting.
  • domain assumption First-stage consensus strength over G/2=16 rollouts reliably predicts full-group consensus and routing decisions.
    Used in Eqs. (2)-(4); the paper provides no error analysis of this estimate, yet it determines whether the costly second stage is triggered.
  • ad hoc to paper Higher-likelihood prefixes (sharpened) tend to converge final answers while lower-likelihood prefixes (flattened) tend to diversify them.
    Central mechanism behind Eq. (6) and consensus-adaptive hint configurations; supported only by the mechanism analysis in Figure 4 on one model-dataset pair, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning." pith.science (2026). https://pith.science/paper/PQH3AW5B

@misc{pith2026260803545,
  author       = {Pith},
  title        = {Pith review of: Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQH3AW5B}},
  note         = {Machine review of arXiv:2608.03545}
}
read the original abstract

Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.

Figures

Figures reproduced from arXiv: 2608.03545 by the authors.

Figure 1
Figure 1. Motivation for consensus-strength regulation in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diagnostic study of consensus-strength levels in majority-voting TTRL on AMC2023 with Qwen2.5-Math-1.5B. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of Hi-TTRL. Based on partial-rollout consensus, Hi-TTRL directly updates in-range groups and applies [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Mechanism analysis of adaptive hints for Qwen3-1.7B-Base on MINERVA. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Reward-hacking dynamics of Qwen2.5-Math-1.5B [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Generalization to a 7B-scale model on AMC-2023. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Computational cost comparison between Hi-TTRL and standard TTRL on AIME-2024 with Qwen2.5-Math-1.5B. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 44 canonical work pages

  1. [1]

    Blackboard Systems , year = 1986, address =

    work page 1986

  2. [2]

    , year = 1983, title =

    Clancey, William J. , year = 1983, title =. Proceedings of the Eighth International Joint Conference on Artificial Intelligence

    work page 1983

  3. [3]

    , year = 1984, title =

    Clancey, William J. , year = 1984, title =. Proceedings of the Fourth National Conference on Artificial Intelligence , pages =

    work page 1984

  4. [4]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  5. [5]

    , year = 1980, title =

    Robinson, Arthur L. , year = 1980, title =. Science , volume = 208, pages =

    work page 1980

  6. [6]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  7. [7]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas , year = 1983, title =. The International Journal of Man-Machine Studies , volume = 20, number = 1, pages =

    work page 1983

  8. [8]

    Rice, James , year = 1986, title =

    work page 1986

Show all 58 references
  1. [9]

    , year = 1979, title =

    Clancey, William J. , year = 1979, title =

    1979

  2. [10]

    , title =

    Clancey, William J. , title =

  3. [11]

    2017 , eprint =

    Attention Is All You Need , author =. 2017 , eprint =

    2017

  4. [12]

    Pluto: The 'Other' Red Planet , author =

  5. [13]

    The Eleventh International Conference on Learning Representations , year =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. The Eleventh International Conference on Learning Representations , year =

  6. [14]

    Advances in neural information processing systems , volume =

    Training language models to follow instructions with human feedback , author =. Advances in neural information processing systems , volume =

  7. [15]

    TTRL: Test-Time Reinforcement Learning , url =

    Zuo, Yuxin and Zhang, Kaiyan and Sheng, Li and Qu, Shang and Cui, Ganqu and Zhu, Xuekai and Li, Haozhan and zhang, yuchen and Long, Xinwei and Hua, Ermo and Qi, Biqing and Sun, Youbang and Ma, Zhiyuan and Yuan, Lifan and Ding, Ning and Zhou, Bowen , booktitle =. TTRL: Test-Tim...

  8. [16]

    2026 , eprint =

    Learning to Reason without External Rewards , author =. 2026 , eprint =

    2026

  9. [17]

    2025 , eprint =

    ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism , author =. 2025 , eprint =

    2025

  10. [18]

    2026 , eprint =

    ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning , author =. 2026 , eprint =

    2026

  11. [19]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages =

    Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages =

  12. [20]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    What if consensus lies? selective-complementary reinforcement learning at test time , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  13. [21]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Coverrl: Breaking the consensus trap in label-free reasoning via generator-verifier co-evolution , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  14. [22]

    2026 , eprint =

    SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization , author =. 2026 , eprint =

    2026

  15. [23]

    Forty-third International Conference on Machine Learning , year =

    PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching , author =. Forty-third International Conference on Machine Learning , year =

  16. [24]

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , volume =

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and others , year =. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , volume =. Nature , publis...

    doi

  17. [25]

    2025 , eprint =

    Qwen3 Technical Report , author =. 2025 , eprint =

    2025

  18. [26]

    2024 , eprint =

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author =. 2024 , eprint =

    2024

  19. [27]

    ACM Computing Surveys , volume =

    A survey on large language models for mathematical reasoning , author =. ACM Computing Surveys , volume =. 2026 , publisher =

    2026

  20. [28]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence in llms , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages =

    2025

  21. [29]

    2025 , eprint =

    Reasoning with Sampling: Your Base Model is Smarter Than You Think , author =. 2025 , eprint =

    2025

  22. [30]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Stephint: Multi-level stepwise hints enhance reinforcement learning to reason , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  23. [31]

    Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization , url =

    Zhang, Qingyang and Wu, Haitao and Zhang, Changqing and Zhao, Peilin and Bian, Yatao , booktitle =. Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization , url =

  24. [32]

    2026 , eprint =

    SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology , author =. 2026 , eprint =

    2026

  25. [33]

    2026 , eprint =

    Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning , author =. 2026 , eprint =

    2026

  26. [34]

    Proceedings of the AAAI Conference on Artificial Intelligence , number =

    Incorporating self-rewriting into large language model reasoning reinforcement , author =. Proceedings of the AAAI Conference on Artificial Intelligence , number =

  27. [35]

    2026 , eprint =

    R-Zero: Self-Evolving Reasoning LLM from Zero Data , author =. 2026 , eprint =

    2026

  28. [36]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Beyond majority voting: Towards fine-grained and more reliable reward signal for test-time reinforcement learning , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  29. [37]

    2017 , eprint =

    Proximal Policy Optimization Algorithms , author =. 2017 , eprint =

    2017

  30. [38]

    Advances in neural information processing systems , volume =

    Direct preference optimization: Your language model is secretly a reward model , author =. Advances in neural information processing systems , volume =

  31. [39]

    Advances in Neural Information Processing Systems , volume =

    Simpo: Simple preference optimization with a reference-free reward , author =. Advances in Neural Information Processing Systems , volume =

  32. [40]

    2026 , eprint =

    OpenAI o1 System Card , author =. 2026 , eprint =

    2026

  33. [41]

    2024 , eprint =

    On Designing Effective RL Reward at Training Time for LLM Reasoning , author =. 2024 , eprint =

    2024

  34. [42]

    Advances in Neural Information Processing Systems , volume =

    Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold , author =. Advances in Neural Information Processing Systems , volume =

  35. [43]

    2026 , eprint =

    Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models , author =. 2026 , eprint =

    2026

  36. [44]

    Reinforcement Learning for Reasoning in Large Language Models with One Training Example , url =

    Wang, Yiping and Yang, Qing and Zeng, Zhiyuan and Ren, Liliang and Liu, Liyuan and Peng, Baolin and Cheng, Hao and He, Xuehai and Wang, Kuan and Gao, Jianfeng and Chen, Weizhu and Wang, Shuohang and Du, Simon and shen, yelong , booktitle =. Reinforcement Learning for Reasoning...

  37. [45]

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning , url =

    Wang, Shenzhi and Yu, Le and Gao, Chang and Zheng, Chujie and Liu, Shixuan and Lu, Rui and Dang, Kai and Chen, Xiong-Hui and Yang, Jianxin and Zhang, Zhenru and Liu, Yuqiong and Yang, An and Zhao, Andrew and Yue, Yang and Song, Shiji and Yu, Bowen and Huang, Gao and Lin, Junya...

  38. [46]

    2026 , eprint =

    Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training , author =. 2026 , eprint =

    2026

  39. [47]

    2025 , eprint =

    RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks , author =. 2025 , eprint =

    2025

  40. [48]

    2024 , eprint =

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement , author =. 2024 , eprint =

    2024

  41. [49]

    2021 , eprint =

    Measuring Mathematical Problem Solving With the MATH Dataset , author =. 2021 , eprint =

    2021

  42. [50]

    Advances in neural information processing systems , volume =

    Solving quantitative reasoning problems with language models , author =. Advances in neural information processing systems , volume =

  43. [51]

    Hugging Face repository , volume =

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions , author =. Hugging Face repository , volume =

  44. [52]

    American Invitational Mathematics Examination (AIME) 2024 , author =

    2024

  45. [53]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    Mario: Math reasoning with code interpreter output-a reproducible pipeline , author =. Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    2024

  46. [54]

    2025 , eprint =

    The Hidden Link Between RLHF and Contrastive Learning , author =. 2025 , eprint =

    2025

  47. [55]

    The Hidden Link Between

    Xufei Lv and Kehai Chen and Haoyuan Sun and Xuefeng Bai and Min Zhang and Houde Liu and Kehai Chen , year =. The Hidden Link Between. 2506.22578 , archiveprefix =

  48. [56]

    The Fourteenth International Conference on Learning Representations , year =

    HiPO: Self-hint policy optimization for RLVR , author =. The Fourteenth International Conference on Learning Representations , year =

  49. [57]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    DUAL RM: Beyond Rule-based Preference Reward Modeling via Meta-Reward , author =. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  50. [58]

    arXiv preprint arXiv:2406.11191 , year =

    A survey on human preference learning for large language models , author =. arXiv preprint arXiv:2406.11191 , year =

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.